Commit Graph

5788 Commits

Author SHA1 Message Date
Anders Schack-Mulligen
abad133ab5 Dataflow: Fix identification of source PathNodes in the presence of source-to-source flow. 2022-08-10 15:02:56 +02:00
Rasmus Wriedt Larsen
40d25cb34c Merge pull request #9849 from tausbn/python-fix-bad-essa-getInput-join
Python: Fix bad join in ESSA `getInput`
2022-08-10 11:45:23 +02:00
Rasmus Wriedt Larsen
b541103b7f Merge pull request #9846 from tausbn/python-fix-bad-syntactic_call_count-join
Python: Fix bad join in `syntactic_call_count`
2022-08-10 10:09:51 +02:00
Erik Krogh Kristensen
559ec7ba56 Merge branch 'main' into repeatedWord 2022-08-09 21:22:47 +02:00
Rasmus Wriedt Larsen
f89b32183f Merge branch 'main' into typetracker-decorators 2022-08-08 11:52:09 +02:00
Anders Schack-Mulligen
3d47875b60 Dataflow: Generate shorter RA/DIL names. 2022-08-05 11:00:56 +02:00
Anders Schack-Mulligen
d3dcc3ce3a Dataflow: Sync. 2022-08-05 11:00:56 +02:00
Rasmus Wriedt Larsen
8fb85a98d8 Merge branch 'main' into post-release-prep/codeql-cli-2.10.2 2022-08-03 10:42:02 +02:00
Rasmus Wriedt Larsen
3d0c23e441 Python: Accept .expected for TarSlip
Changed after merging https://github.com/github/codeql/pull/9579,
which improved our handling of `not` for guards.
2022-08-03 09:52:11 +02:00
Alex Ford
8e3548efb3 Merge branch 'main' into post-release-prep/codeql-cli-2.10.2 2022-08-02 20:29:26 +01:00
Rasmus Wriedt Larsen
1737d08145 Merge pull request #9579 from yoff/python/more-logic-tests
Python: Improve `BarrierGuard`
2022-08-01 11:36:11 +02:00
github-actions[bot]
e8747d3176 Post-release preparation for codeql-cli-2.10.2 2022-07-28 20:00:09 +00:00
github-actions[bot]
212786ed91 Release preparation for version 2.10.2 2022-07-28 13:38:35 +00:00
Andrew Eisenberg
43ae5d4285 Merge pull request #9838 from github/aeisenberg/python-local-ref-def
Move python contextual queries to lib folders
2022-07-25 09:00:32 -07:00
Asger F
b9bdee6651 Merge branch 'main' into post-release-prep/codeql-cli-2.10.1 2022-07-19 16:24:35 +02:00
Taus
bfe90413e2 Merge pull request #9847 from alexet/alexet/fix-predicate-binding
Python: Fix binding incorrect predicate.
2022-07-19 13:59:13 +02:00
Taus
8c0725e8c6 Python: Fix bad join in ESSA getInput
Before:

```
Tuple counts for Essa::EssaEdgeRefinement::getInput#dispred#f0820431#ff/2@b84afc77 after 20.3s:
873421    ~0%     {3} r1 = JOIN Essa::TEssaEdgeDefinition#24e22a14#ffff_31#join_rhs WITH Essa::TEssaEdgeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'this'
181627951 ~0%     {3} r2 = JOIN r1 WITH Essa::EssaDefinition::getSourceVariable#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1, Lhs.2 'this'
873418    ~0%     {2} r3 = JOIN r2 WITH Essa::EssaDefinition::reachesEndOfBlock#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'
                return r3
```
It's perhaps not immediately obvious what's going on here (because of
the `...join_rhs` indirection), but basically we're joining together
`this` and `def` and their `getSourceVariable`, and only then actually
relating `this` and `def` through `reachesEndOfBlock`.

By unbinding `var`, we prevent this early join, which now encourages the
`reachesEndOfBlock` join to happen earlier:

```
Tuple counts for Essa::EssaEdgeRefinement::getInput#dispred#f0820431#ff/2@2d63e5lb after 2s
873421  ~0%     {2} r1 = SCAN Essa::TEssaEdgeDefinition#24e22a14#ffff OUTPUT In.3 'this', In.1
873421  ~0%     {3} r2 = JOIN r1 WITH Essa::TEssaEdgeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'this'
873421  ~0%     {3} r3 = JOIN r2 WITH Definitions::SsaSourceVariable#class#486534ab#f ON FIRST 1 OUTPUT Lhs.1, Lhs.2 'this', Lhs.0
8758877 ~0%     {3} r4 = JOIN r3 WITH Essa::EssaDefinition::reachesEndOfBlock#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.2, Lhs.1 'this'
873418  ~0%     {2} r5 = JOIN r4 WITH Essa::EssaDefinition::getSourceVariable#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'
                return r5
```
2022-07-18 20:21:39 +00:00
alexet
f9b6ca76e5 Python: Fix binding incorrect predicate. 2022-07-18 16:28:19 +01:00
Taus
bdd771989f Python: Fix bad join in syntactic_call_count
On certain databases, the evaluation of this predicate was running out
of memory due to the way the `count` aggregate was being used. Here's
an example of the tuple counts involved:

```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs/1@d2199bb8 after 1m27s:
595518502 ~521250%     {1} r1 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

26518709  ~111513%     {1} r2 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

622037211 ~498045%     {1} r3 = r1 UNION r2
                        return r3
```

and a timing report that looked like this:
```
time  | evals |   max @ iter | predicate
------|-------|--------------|----------
 5m8s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2@6d98d1nd
4m38s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#count_range@f5df1do4
3m51s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3@da3b4abf
1m58s |  7613 |  37ms @ 4609 | MRO::ClassListList::removedClassParts#f0820431#fffff#reorder_2_3_4_0_1@8155axyi
1m37s |  7613 |  33ms @ 3904 | MRO::ClassListList::bestMergeCandidate#f0820431#2#fff@8155a83w
1m27s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs@d2199bb8
 1m8s |  1825 |  63ms @ 404  | PointsTo::Expressions::equalityEvaluatesTo#741b54e2#fffff@8155aw7w
37.6s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs@e348fc1p
...
```

To make optimising this easier for the compiler, I moved the bodies of
the `count` aggregate into their own helper predicates (with size
linear in the number of `CallNode`s), and also factored out the many
calls to `f.getName()`.

The astute reader will notice that in writing this as a sum of `count`s
rather than a count of a disjunction, the intersection (if it exists)
will be counted twice, and so the semantics may be different. However,
since `method_call` and `function_call` require `AttrNode` and
`NameNode` functions respectively, and as these two types are disjoint,
there is no intersection, and so the semantics should be preserved.

After the change, the evaluation of `syntactic_call_count` now looks as
follows:
```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff/2@662dd8s0 after 216ms:
23960  ~0%     {1} r1 = @py_scope#f AND NOT py_Functions_0#antijoin_rhs(Lhs.0 's')
23960  ~0%     {2} r2 = SCAN r1 OUTPUT In.0 's', 0

276309 ~7%     {2} r3 = SCAN @py_scope#f OUTPUT In.0 's', "__init__"
11763  ~0%     {2} r4 = JOIN r3 WITH Scope::Scope::getName#dispred#f0820431#fb ON FIRST 2 OUTPUT Lhs.0 's', 1

35723  ~0%     {2} r5 = r2 UNION r4

252349 ~0%     {2} r6 = JOIN @py_scope#f WITH Function::Function::getName#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.0 's', Rhs.1

240586 ~0%     {2} r7 = SELECT r6 ON In.1 != "__init__"

131727 ~4%     {2} r8 = r7 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r9 = SCAN r8 OUTPUT In.1, In.0 's', 0

240586 ~0%     {2} r10 = SCAN r7 OUTPUT In.1, In.0 's'

108859 ~0%     {3} r11 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 's', Rhs.1

240586 ~0%     {3} r12 = r9 UNION r11
24100  ~0%     {2} r13 = JOIN r12 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs#1 ON FIRST 1 OUTPUT Lhs.1 's', (Rhs.1 + Lhs.2)

240586 ~0%     {2} r14 = SELECT r6 ON In.1 != "__init__"
131727 ~4%     {2} r15 = r14 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r16 = SCAN r15 OUTPUT In.0 's', In.1, 0

108859 ~4%     {3} r17 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.1 's', Lhs.0, Rhs.1

240586 ~4%     {3} r18 = r16 UNION r17
216486 ~2%     {3} r19 = r18 AND NOT project#PointsToContext::function_call#cf3039a0#ff(Lhs.1)
216486 ~0%     {2} r20 = SCAN r19 OUTPUT In.0 's', (0 + In.2)

240586 ~0%     {2} r21 = r13 UNION r20
276309 ~0%     {2} r22 = r5 UNION r21
                return r22
```
2022-07-18 13:58:00 +00:00
yo-h
d4443592eb Merge pull request #9776 from raulgarciamsft/azure-sdk-client-encryption-version
New queries to detect unsafe client side encryption in Azure Storage
2022-07-16 14:59:51 -04:00
Raul Garcia
6b17890e4f Fixing warning on usage of a deprecated feature. 2022-07-16 08:30:06 -07:00
Andrew Eisenberg
b897a40228 Move python contextual queries to lib folders
This will ensure that python projects can use jump to ref/def in
vscode when the core libraries are not installed.
2022-07-15 13:12:17 -07:00
github-actions[bot]
0ee476129a Post-release preparation for codeql-cli-2.10.1 2022-07-14 14:38:49 +00:00
Erik Krogh Kristensen
85a652f3d1 remove a bunch of repeated words 2022-07-14 12:42:48 +02:00
github-actions[bot]
d1aa0d7dd3 Release preparation for version 2.10.1 2022-07-14 08:56:03 +00:00
Raul Garcia
f7c47b6c75 Update python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.py
Co-authored-by: Taus <tausbn@github.com>
2022-07-13 08:34:48 -07:00
Erik Krogh Kristensen
c4f44bb67f sync files 2022-07-13 10:01:26 +02:00
Raul Garcia
0dbb03f732 Adding CVE information. 2022-07-12 21:49:19 -07:00
Raul Garcia
d929b1338b Addressing API::Node feedback for all predicates 2022-07-12 11:55:06 -07:00
Raul Garcia
d5791e2d56 Addressing feedback from the PR 2022-07-11 15:45:15 -07:00
Raul Garcia
ac05577966 Making various changes based on the feedback. Pending: 2 non-trivial fixes for Java & Python. 2022-07-11 13:25:35 -07:00
Raul Garcia
e5702d0e15 Update python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
Co-authored-by: Taus <tausbn@github.com>
2022-07-11 13:07:37 -07:00
Raul Garcia
7fc9ae6c49 Update python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
Co-authored-by: Taus <tausbn@github.com>
2022-07-11 13:07:20 -07:00
Raul Garcia
dd1a9a22e3 Update UnsafeUsageOfClientSideEncryptionVersion.qhelp 2022-07-05 13:58:38 -07:00
Raul Garcia
e43e5810cf New queries to detect unsafe client side encryption in Azure Storage 2022-07-01 17:08:35 -07:00
CodeQL CI
5b5a52fa25 Merge pull request #9551 from yoff/python/port-tarslip
Approved by RasmusWL
2022-07-01 12:58:25 +01:00
yoff
cf9b69b5f2 python: More helpful comment 2022-06-30 13:07:13 +00:00
yoff
b0a29b146a Update python/ql/lib/semmle/python/security/dataflow/TarSlipQuery.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2022-06-30 14:54:01 +02:00
yoff
df7ffb2880 Update python/ql/lib/semmle/python/security/dataflow/TarSlipCustomizations.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2022-06-30 14:53:49 +02:00
Andrew Eisenberg
fbeecd6c08 Merge pull request #9744 from github/aeisenberg/move-contextual-queries 2022-06-29 11:44:33 -07:00
Andrew Eisenberg
7864a7580e Fix import statements 2022-06-29 10:22:45 -07:00
Andrew Eisenberg
ddf06f8617 Add change notes and qldoc for moved files 2022-06-29 10:03:12 -07:00
Andrew Eisenberg
a3f4d1bf66 Move contextual queries from src to lib
With this change, users are now able to run View AST command in
vscode within vscode workspaces that do not include the core libraries.
The relevant core library only needs to be installed in the package
cache.
2022-06-29 07:51:26 -07:00
yoff
8988a02806 Merge pull request #9733 from tausbn/python-fix-bad-mro-flatten-list-join
Python: Fix bad join in MRO `flatten_list`
2022-06-29 13:29:48 +02:00
yoff
f122af81ea Merge pull request #9741 from tausbn/python-fix-bad-join-in-regexpbackref-getgroup
Python: Fix bad join in `RegExpBackRef::getGroup`
2022-06-29 13:23:07 +02:00
yoff
731f866242 Merge pull request #9717 from tausbn/python-fix-bad-mro-linearization-of-bases-join
Python: Fix bad join in MRO
2022-06-29 13:08:18 +02:00
Jeroen Ketema
55e052af26 Merge pull request #9686 from aschackmull/dataflow/no-node-scan
Dataflow performance: Avoid node scans
2022-06-29 10:38:56 +02:00
yoff
1105cd569b Merge branch 'main' into python/port-tarslip 2022-06-28 22:17:28 +02:00
yoff
6087bc6888 Merge branch 'main' into python/more-logic-tests 2022-06-28 22:16:38 +02:00
yoff
ac0c8d238f python: only clear taint on false-edge 2022-06-28 20:14:52 +00:00