Commit Graph

47078 Commits

Author SHA1 Message Date
Harry Maclean
7b8603c89b Ruby: Model Arel.sql 2022-07-19 11:27:15 +12:00
Taus
8c0725e8c6 Python: Fix bad join in ESSA getInput
Before:

```
Tuple counts for Essa::EssaEdgeRefinement::getInput#dispred#f0820431#ff/2@b84afc77 after 20.3s:
873421    ~0%     {3} r1 = JOIN Essa::TEssaEdgeDefinition#24e22a14#ffff_31#join_rhs WITH Essa::TEssaEdgeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'this'
181627951 ~0%     {3} r2 = JOIN r1 WITH Essa::EssaDefinition::getSourceVariable#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1, Lhs.2 'this'
873418    ~0%     {2} r3 = JOIN r2 WITH Essa::EssaDefinition::reachesEndOfBlock#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'
                return r3
```
It's perhaps not immediately obvious what's going on here (because of
the `...join_rhs` indirection), but basically we're joining together
`this` and `def` and their `getSourceVariable`, and only then actually
relating `this` and `def` through `reachesEndOfBlock`.

By unbinding `var`, we prevent this early join, which now encourages the
`reachesEndOfBlock` join to happen earlier:

```
Tuple counts for Essa::EssaEdgeRefinement::getInput#dispred#f0820431#ff/2@2d63e5lb after 2s
873421  ~0%     {2} r1 = SCAN Essa::TEssaEdgeDefinition#24e22a14#ffff OUTPUT In.3 'this', In.1
873421  ~0%     {3} r2 = JOIN r1 WITH Essa::TEssaEdgeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'this'
873421  ~0%     {3} r3 = JOIN r2 WITH Definitions::SsaSourceVariable#class#486534ab#f ON FIRST 1 OUTPUT Lhs.1, Lhs.2 'this', Lhs.0
8758877 ~0%     {3} r4 = JOIN r3 WITH Essa::EssaDefinition::reachesEndOfBlock#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.2, Lhs.1 'this'
873418  ~0%     {2} r5 = JOIN r4 WITH Essa::EssaDefinition::getSourceVariable#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'
                return r5
```
2022-07-18 20:21:39 +00:00
alexet
f9b6ca76e5 Python: Fix binding incorrect predicate. 2022-07-18 16:28:19 +01:00
thiggy1342
fc00e56058 Merge branch 'main' into experimental-manually-check-request-verb 2022-07-18 10:58:13 -04:00
Taus
bdd771989f Python: Fix bad join in syntactic_call_count
On certain databases, the evaluation of this predicate was running out
of memory due to the way the `count` aggregate was being used. Here's
an example of the tuple counts involved:

```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs/1@d2199bb8 after 1m27s:
595518502 ~521250%     {1} r1 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

26518709  ~111513%     {1} r2 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

622037211 ~498045%     {1} r3 = r1 UNION r2
                        return r3
```

and a timing report that looked like this:
```
time  | evals |   max @ iter | predicate
------|-------|--------------|----------
 5m8s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2@6d98d1nd
4m38s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#count_range@f5df1do4
3m51s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3@da3b4abf
1m58s |  7613 |  37ms @ 4609 | MRO::ClassListList::removedClassParts#f0820431#fffff#reorder_2_3_4_0_1@8155axyi
1m37s |  7613 |  33ms @ 3904 | MRO::ClassListList::bestMergeCandidate#f0820431#2#fff@8155a83w
1m27s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs@d2199bb8
 1m8s |  1825 |  63ms @ 404  | PointsTo::Expressions::equalityEvaluatesTo#741b54e2#fffff@8155aw7w
37.6s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs@e348fc1p
...
```

To make optimising this easier for the compiler, I moved the bodies of
the `count` aggregate into their own helper predicates (with size
linear in the number of `CallNode`s), and also factored out the many
calls to `f.getName()`.

The astute reader will notice that in writing this as a sum of `count`s
rather than a count of a disjunction, the intersection (if it exists)
will be counted twice, and so the semantics may be different. However,
since `method_call` and `function_call` require `AttrNode` and
`NameNode` functions respectively, and as these two types are disjoint,
there is no intersection, and so the semantics should be preserved.

After the change, the evaluation of `syntactic_call_count` now looks as
follows:
```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff/2@662dd8s0 after 216ms:
23960  ~0%     {1} r1 = @py_scope#f AND NOT py_Functions_0#antijoin_rhs(Lhs.0 's')
23960  ~0%     {2} r2 = SCAN r1 OUTPUT In.0 's', 0

276309 ~7%     {2} r3 = SCAN @py_scope#f OUTPUT In.0 's', "__init__"
11763  ~0%     {2} r4 = JOIN r3 WITH Scope::Scope::getName#dispred#f0820431#fb ON FIRST 2 OUTPUT Lhs.0 's', 1

35723  ~0%     {2} r5 = r2 UNION r4

252349 ~0%     {2} r6 = JOIN @py_scope#f WITH Function::Function::getName#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.0 's', Rhs.1

240586 ~0%     {2} r7 = SELECT r6 ON In.1 != "__init__"

131727 ~4%     {2} r8 = r7 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r9 = SCAN r8 OUTPUT In.1, In.0 's', 0

240586 ~0%     {2} r10 = SCAN r7 OUTPUT In.1, In.0 's'

108859 ~0%     {3} r11 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 's', Rhs.1

240586 ~0%     {3} r12 = r9 UNION r11
24100  ~0%     {2} r13 = JOIN r12 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs#1 ON FIRST 1 OUTPUT Lhs.1 's', (Rhs.1 + Lhs.2)

240586 ~0%     {2} r14 = SELECT r6 ON In.1 != "__init__"
131727 ~4%     {2} r15 = r14 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r16 = SCAN r15 OUTPUT In.0 's', In.1, 0

108859 ~4%     {3} r17 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.1 's', Lhs.0, Rhs.1

240586 ~4%     {3} r18 = r16 UNION r17
216486 ~2%     {3} r19 = r18 AND NOT project#PointsToContext::function_call#cf3039a0#ff(Lhs.1)
216486 ~0%     {2} r20 = SCAN r19 OUTPUT In.0 's', (0 + In.2)

240586 ~0%     {2} r21 = r13 UNION r20
276309 ~0%     {2} r22 = r5 UNION r21
                return r22
```
2022-07-18 13:58:00 +00:00
Arthur Baars
c9e5206396 Ruby: skip .git folder 2022-07-18 15:26:38 +02:00
Geoffrey White
541df9b550 Swift: Remove TODO comment. We have a test for this problem now. 2022-07-18 14:26:12 +01:00
Geoffrey White
336548f746 Swift: Improve comments. 2022-07-18 14:24:16 +01:00
Geoffrey White
9474e63faf Swift: Clean up isSink (4 - move common code out). 2022-07-18 14:24:15 +01:00
Geoffrey White
b136790efd Swift: Clean up isSink (3 - rename f -> funcDecl and move that out as well; in the other two cases this variable didn't exist, now it does). 2022-07-18 14:24:14 +01:00
Geoffrey White
0bd94a6307 Swift: Clean up isSink (2 - rename methodName -> funcName and move that out as well). 2022-07-18 14:24:13 +01:00
Geoffrey White
4854679a40 Swift: Clean up isSink (1 - move common variables to an outer exists). 2022-07-18 14:24:13 +01:00
Geoffrey White
39fb714ad1 Swift: Add test with substring declared differently. 2022-07-18 14:24:12 +01:00
Michael Nebel
6603024488 C#: Allow encryption- prefix for sinks in CsvValidation. 2022-07-18 14:32:31 +02:00
Michael Nebel
57ba0c4e5d C#: Move sinks into System.Security.Cryptography framework code. 2022-07-18 14:28:49 +02:00
Michael Nebel
c91d49a0fe C#: Add provenance column to CSV format for SymmetricAlgorithm. 2022-07-18 14:28:49 +02:00
Michael Nebel
66232a8054 C#: Fix typo. 2022-07-18 14:28:49 +02:00
Michael Nebel
a5b7e2a2e1 C#: Convert set Key of SymmetricAlgorithm to Csv sink. 2022-07-18 14:28:49 +02:00
Michael Nebel
032448041d C#: Convert CreateSymmetricKey to CSV sink. 2022-07-18 14:28:49 +02:00
Michael Nebel
1d405dba14 C#: Collapse Sink classes. 2022-07-18 14:28:49 +02:00
Michael Nebel
383ad51682 C#: Use CSV format for CreateEncryptor and CreateDecryptor sinks. 2022-07-18 14:28:49 +02:00
Michael Nebel
e6e82ef56d C#: Update test with Decrypt example. 2022-07-18 14:28:49 +02:00
Michael Nebel
52a9fb0de7 C#: Add test for decrypt. 2022-07-18 14:28:49 +02:00
Paolo Tranquilli
e1bd4a78ff Merge branch 'main' into redsun82/swift-type-repr-collapse 2022-07-18 14:05:43 +02:00
Paolo Tranquilli
10b7b1f183 Merge branch 'main' into redsun82/swift-file-label-caching 2022-07-18 14:02:57 +02:00
Paolo Tranquilli
410167671f Merge pull request #9795 from github/redsun82/swift-extraction
Swift: extract more entities
2022-07-18 13:37:43 +02:00
Nick Rolfe
eebba36b18 Merge pull request #9708 from github/nickrolfe/pathname
Ruby: model the standard library's `Pathname` class
2022-07-18 11:29:30 +01:00
Paolo Tranquilli
98fc8812fc Merge 'main' into redsun82/swift-extraction 2022-07-18 11:55:21 +02:00
Paolo Tranquilli
c779936ee8 Swift: commit forgotten files 2022-07-18 11:19:40 +02:00
Paolo Tranquilli
c08c3955d6 Swift: add UnresolvedPatternExpr test 2022-07-18 10:37:54 +02:00
Paolo Tranquilli
78fc356feb Swift: address review comments 2022-07-18 10:29:20 +02:00
Nick Rolfe
dbd6607875 Ruby: use ASCII dash in comment
Co-authored-by: Harry Maclean <hmac@github.com>
2022-07-18 08:54:58 +01:00
Harry Maclean
cc5f59f313 Merge pull request #9138 from hmac/hmac/array-inclusion-guard-local-flow
Ruby: Make StringArrayInclusion more sensitive
2022-07-18 10:11:49 +12:00
yo-h
d4443592eb Merge pull request #9776 from raulgarciamsft/azure-sdk-client-encryption-version
New queries to detect unsafe client side encryption in Azure Storage
2022-07-16 14:59:51 -04:00
Raul Garcia
6b17890e4f Fixing warning on usage of a deprecated feature. 2022-07-16 08:30:06 -07:00
Raul Garcia
eefa659503 Update java/ql/src/experimental/Security/CWE/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
Co-authored-by: yo-h <55373593+yo-h@users.noreply.github.com>
2022-07-16 08:23:59 -07:00
Raul Garcia
fe789c8aa9 Update java/ql/src/experimental/Security/CWE/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
Co-authored-by: yo-h <55373593+yo-h@users.noreply.github.com>
2022-07-16 08:22:18 -07:00
Andrew Eisenberg
b897a40228 Move python contextual queries to lib folders
This will ensure that python projects can use jump to ref/def in
vscode when the core libraries are not installed.
2022-07-15 13:12:17 -07:00
Andrew Eisenberg
2f50549184 Move definitions.ql back to src 2022-07-15 11:48:15 -07:00
thiggy1342
a1df1d1119 Merge branch 'main' into experimental-strong-params 2022-07-15 11:17:57 -04:00
thiggy1342
ee1c09329f Merge branch 'main' into add-activerecord-annotate 2022-07-15 11:17:48 -04:00
Chris Smowton
41ca791966 Implement is-underscore-parameter for old versions of Kotlin 2022-07-15 12:36:37 +01:00
Erik Krogh Kristensen
625e37a0da fix typo
Co-authored-by: intrigus-lgtm <60750685+intrigus-lgtm@users.noreply.github.com>
2022-07-14 21:53:21 +02:00
Aditya Sharad
d50816a284 Merge pull request #9802 from adityasharad/docs/language-pack-changelogs
Docs: Add links from query help to query pack changelog for each language
2022-07-14 08:52:50 -07:00
Chris Smowton
d4b0163c4c Kotlin: Don't extract a name for a '_' parameter
I can't reproduce the exact circumstances, but these sometimes get "<anonymous parameter X>" names and sometimes get "$noName_X" names. Whichever way, avoiding extracting a synthetic name seems safest; anyone finding the .class file and not reading the metadata indicating it came from a `_` will extract the binary name selected, or else QL will
invent a name.
2022-07-14 16:36:26 +01:00
github-actions[bot]
0ee476129a Post-release preparation for codeql-cli-2.10.1 2022-07-14 14:38:49 +00:00
Paolo Tranquilli
5e74df3882 Swift: cache file paths
This required a bit of a generalization of `TrapLabelStore` to not
work only with pointers.
2022-07-14 16:32:33 +02:00
Aditya Sharad
d13f9d5d71 Update docs/codeql/query-help/javascript.rst
Co-authored-by: Felicity Chapman <felicitymay@github.com>
2022-07-14 07:29:29 -07:00
Paolo Tranquilli
7fbe4f8547 Merge pull request #9815 from github/redsun82/swift-exclusive-file
Swift: trap output rework
2022-07-14 16:23:44 +02:00
thiggy1342
592ce3ec58 Merge branch 'main' into add-activerecord-annotate 2022-07-14 09:55:25 -04:00