Commit Graph

9240 Commits

Author SHA1 Message Date
alexet
f9b6ca76e5 Python: Fix binding incorrect predicate. 2022-07-18 16:28:19 +01:00
Taus
bdd771989f Python: Fix bad join in syntactic_call_count
On certain databases, the evaluation of this predicate was running out
of memory due to the way the `count` aggregate was being used. Here's
an example of the tuple counts involved:

```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs/1@d2199bb8 after 1m27s:
595518502 ~521250%     {1} r1 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

26518709  ~111513%     {1} r2 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

622037211 ~498045%     {1} r3 = r1 UNION r2
                        return r3
```

and a timing report that looked like this:
```
time  | evals |   max @ iter | predicate
------|-------|--------------|----------
 5m8s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2@6d98d1nd
4m38s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#count_range@f5df1do4
3m51s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3@da3b4abf
1m58s |  7613 |  37ms @ 4609 | MRO::ClassListList::removedClassParts#f0820431#fffff#reorder_2_3_4_0_1@8155axyi
1m37s |  7613 |  33ms @ 3904 | MRO::ClassListList::bestMergeCandidate#f0820431#2#fff@8155a83w
1m27s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs@d2199bb8
 1m8s |  1825 |  63ms @ 404  | PointsTo::Expressions::equalityEvaluatesTo#741b54e2#fffff@8155aw7w
37.6s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs@e348fc1p
...
```

To make optimising this easier for the compiler, I moved the bodies of
the `count` aggregate into their own helper predicates (with size
linear in the number of `CallNode`s), and also factored out the many
calls to `f.getName()`.

The astute reader will notice that in writing this as a sum of `count`s
rather than a count of a disjunction, the intersection (if it exists)
will be counted twice, and so the semantics may be different. However,
since `method_call` and `function_call` require `AttrNode` and
`NameNode` functions respectively, and as these two types are disjoint,
there is no intersection, and so the semantics should be preserved.

After the change, the evaluation of `syntactic_call_count` now looks as
follows:
```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff/2@662dd8s0 after 216ms:
23960  ~0%     {1} r1 = @py_scope#f AND NOT py_Functions_0#antijoin_rhs(Lhs.0 's')
23960  ~0%     {2} r2 = SCAN r1 OUTPUT In.0 's', 0

276309 ~7%     {2} r3 = SCAN @py_scope#f OUTPUT In.0 's', "__init__"
11763  ~0%     {2} r4 = JOIN r3 WITH Scope::Scope::getName#dispred#f0820431#fb ON FIRST 2 OUTPUT Lhs.0 's', 1

35723  ~0%     {2} r5 = r2 UNION r4

252349 ~0%     {2} r6 = JOIN @py_scope#f WITH Function::Function::getName#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.0 's', Rhs.1

240586 ~0%     {2} r7 = SELECT r6 ON In.1 != "__init__"

131727 ~4%     {2} r8 = r7 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r9 = SCAN r8 OUTPUT In.1, In.0 's', 0

240586 ~0%     {2} r10 = SCAN r7 OUTPUT In.1, In.0 's'

108859 ~0%     {3} r11 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 's', Rhs.1

240586 ~0%     {3} r12 = r9 UNION r11
24100  ~0%     {2} r13 = JOIN r12 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs#1 ON FIRST 1 OUTPUT Lhs.1 's', (Rhs.1 + Lhs.2)

240586 ~0%     {2} r14 = SELECT r6 ON In.1 != "__init__"
131727 ~4%     {2} r15 = r14 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r16 = SCAN r15 OUTPUT In.0 's', In.1, 0

108859 ~4%     {3} r17 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.1 's', Lhs.0, Rhs.1

240586 ~4%     {3} r18 = r16 UNION r17
216486 ~2%     {3} r19 = r18 AND NOT project#PointsToContext::function_call#cf3039a0#ff(Lhs.1)
216486 ~0%     {2} r20 = SCAN r19 OUTPUT In.0 's', (0 + In.2)

240586 ~0%     {2} r21 = r13 UNION r20
276309 ~0%     {2} r22 = r5 UNION r21
                return r22
```
2022-07-18 13:58:00 +00:00
yo-h
d4443592eb Merge pull request #9776 from raulgarciamsft/azure-sdk-client-encryption-version
New queries to detect unsafe client side encryption in Azure Storage
2022-07-16 14:59:51 -04:00
Raul Garcia
6b17890e4f Fixing warning on usage of a deprecated feature. 2022-07-16 08:30:06 -07:00
Andrew Eisenberg
b897a40228 Move python contextual queries to lib folders
This will ensure that python projects can use jump to ref/def in
vscode when the core libraries are not installed.
2022-07-15 13:12:17 -07:00
Ahmed Farid
7406273346 Update TimingAttack.qhelp 2022-07-14 17:56:58 +01:00
Ahmed Farid
f4654136d6 Update TimingAttack.qhelp 2022-07-14 17:56:13 +01:00
github-actions[bot]
0ee476129a Post-release preparation for codeql-cli-2.10.1 2022-07-14 14:38:49 +00:00
Erik Krogh Kristensen
85a652f3d1 remove a bunch of repeated words 2022-07-14 12:42:48 +02:00
github-actions[bot]
d1aa0d7dd3 Release preparation for version 2.10.1 2022-07-14 08:56:03 +00:00
Raul Garcia
f7c47b6c75 Update python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.py
Co-authored-by: Taus <tausbn@github.com>
2022-07-13 08:34:48 -07:00
Erik Krogh Kristensen
595875ff98 remove redundant not-equals check 2022-07-13 12:06:12 +02:00
Erik Krogh Kristensen
a4262f8d91 add some more references to the overly-large-range qhelp 2022-07-13 11:20:24 +02:00
Erik Krogh Kristensen
c4f44bb67f sync files 2022-07-13 10:01:26 +02:00
Raul Garcia
0dbb03f732 Adding CVE information. 2022-07-12 21:49:19 -07:00
Raul Garcia
d929b1338b Addressing API::Node feedback for all predicates 2022-07-12 11:55:06 -07:00
Erik Krogh Kristensen
8e52fc97fc changes based on review by Shack 2022-07-12 16:02:50 +02:00
Erik Krogh Kristensen
220ff3cb2e convert tabs to spaces in qhelp 2022-07-12 16:02:50 +02:00
Erik Krogh Kristensen
aae3e2ddde other changes based on Esbens review 2022-07-12 16:02:50 +02:00
Erik Krogh Kristensen
ff25451699 rename query to overly-large-range, and rewrite the @description 2022-07-12 16:02:46 +02:00
Raul Garcia
d5791e2d56 Addressing feedback from the PR 2022-07-11 15:45:15 -07:00
Raul Garcia
ac05577966 Making various changes based on the feedback. Pending: 2 non-trivial fixes for Java & Python. 2022-07-11 13:25:35 -07:00
Raul Garcia
e5702d0e15 Update python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
Co-authored-by: Taus <tausbn@github.com>
2022-07-11 13:07:37 -07:00
Raul Garcia
7fc9ae6c49 Update python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
Co-authored-by: Taus <tausbn@github.com>
2022-07-11 13:07:20 -07:00
Taus
ec363166ba Python: Make UserInputMsgConfig public 2022-07-11 15:24:31 +02:00
Raul Garcia
dd1a9a22e3 Update UnsafeUsageOfClientSideEncryptionVersion.qhelp 2022-07-05 13:58:38 -07:00
Raul Garcia
e43e5810cf New queries to detect unsafe client side encryption in Azure Storage 2022-07-01 17:08:35 -07:00
yoff
f52d792b36 Merge branch 'main' of https://github.com/github/codeql into python-dataflow/flow-summaries-from-scratch 2022-07-01 12:01:07 +00:00
CodeQL CI
5b5a52fa25 Merge pull request #9551 from yoff/python/port-tarslip
Approved by RasmusWL
2022-07-01 12:58:25 +01:00
yoff
61523bd330 python: better names
- "Normal" instead of "NonSpecial"
- "NonLibrary" instead of "2"

I could not find a good replacement for "NonLibrary", nor for "Source",
but I added QLDocs in a few places to help the reading.
2022-07-01 11:55:20 +00:00
yoff
a0db438799 python: rename getACall2 -> getANonLibraryCall 2022-07-01 10:29:03 +00:00
yoff
f6af24894d python: recover isPackageUsed
- add `unknownAttribute` to pre-compute negation
- add `Node`-less formulation of "is imported"
2022-07-01 09:39:07 +00:00
yoff
71583bf6be python: fix import of AccessPathSyntax 2022-07-01 08:48:55 +00:00
yoff
3a80baf39c python: concession to get the code to compile
`isPackageUsed` now does no filtering
2022-07-01 07:06:09 +00:00
yoff
e54ada175d python: rewrite not away
A `LocalSourceNode` is either a `ModuleVariableNode`
or an `ExprNode`.
2022-07-01 07:03:14 +00:00
yoff
cf9b69b5f2 python: More helpful comment 2022-06-30 13:07:13 +00:00
yoff
b0a29b146a Update python/ql/lib/semmle/python/security/dataflow/TarSlipQuery.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2022-06-30 14:54:01 +02:00
yoff
df7ffb2880 Update python/ql/lib/semmle/python/security/dataflow/TarSlipCustomizations.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2022-06-30 14:53:49 +02:00
Andrew Eisenberg
fbeecd6c08 Merge pull request #9744 from github/aeisenberg/move-contextual-queries 2022-06-29 11:44:33 -07:00
Andrew Eisenberg
7864a7580e Fix import statements 2022-06-29 10:22:45 -07:00
Andrew Eisenberg
ddf06f8617 Add change notes and qldoc for moved files 2022-06-29 10:03:12 -07:00
Andrew Eisenberg
a3f4d1bf66 Move contextual queries from src to lib
With this change, users are now able to run View AST command in
vscode within vscode workspaces that do not include the core libraries.
The relevant core library only needs to be installed in the package
cache.
2022-06-29 07:51:26 -07:00
yoff
8988a02806 Merge pull request #9733 from tausbn/python-fix-bad-mro-flatten-list-join
Python: Fix bad join in MRO `flatten_list`
2022-06-29 13:29:48 +02:00
yoff
f122af81ea Merge pull request #9741 from tausbn/python-fix-bad-join-in-regexpbackref-getgroup
Python: Fix bad join in `RegExpBackRef::getGroup`
2022-06-29 13:23:07 +02:00
yoff
731f866242 Merge pull request #9717 from tausbn/python-fix-bad-mro-linearization-of-bases-join
Python: Fix bad join in MRO
2022-06-29 13:08:18 +02:00
Jeroen Ketema
55e052af26 Merge pull request #9686 from aschackmull/dataflow/no-node-scan
Dataflow performance: Avoid node scans
2022-06-29 10:38:56 +02:00
Ahmed Farid
f5d0791b4f Update TimingAttack.qll 2022-06-29 00:56:15 +01:00
Ahmed Farid
98909c2069 Update TimingAttackAgainstSensitiveInfo.ql 2022-06-29 00:55:21 +01:00
Ahmed Farid
41b4c06f2d Update TimingAttackAgainstSignature.ql 2022-06-29 00:54:44 +01:00
Ahmed Farid
e20fefc3ad Update TimingAttackAgainstHeader.ql 2022-06-29 00:54:03 +01:00