Commit Graph

68010 Commits

Author SHA1 Message Date
Taus
6db7e72fb8 Python: Fix bad join in DataFlowDispatch
A case of bad magic. Rather than evaluating separately whether a class
has a method of some name, the compiler opted to magick in the fact
that this was done as part of the `findFunctionAccordingToMro`
predicate. Hilarity ensued.

However, _we_ know that magic really isn't needed in this case (the
number of results is bounded by `Class.getAMethod` since methods have
only a single name), so by factoring it out into a helper predicate, we
can help the join-orderer along.

Before
```
(377s) Starting to evaluate predicate _DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_DataFlowDispatch::getNextClassInMro/1#__#shared/3@i6#L3#f893bw2h (iteration 6)
(377s) Tuple counts for _DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_DataFlowDispatch::getNextClassInMro/1#__#shared/3@i6#L3#f893bw2h after 16ms:
33363  ~0%     {2} r1 = SCAN `DataFlowDispatch::getNextClassInMro/1#e1ee596a#prev_delta` OUTPUT In.1, In.0 'arg1'
159696 ~4%     {3}    | JOIN WITH `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev` ON FIRST 1 OUTPUT Rhs.1 'arg0', Lhs.1 'arg1', Rhs.2 'arg2'
               return r1
(377s) Starting to evaluate predicate _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs/3@i6#L4#f893bw2h (iteration 6)
(382s) Tuple counts for _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs/3@i6#L4#f893bw2h after 4.4s:
1770825904 ~4%     {4} r1 = JOIN `_DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_DataFlowDispatch::getNextClassInMro/1#__#shared` WITH `Function::Function.getName/0#dispred#033700ef_10#join_rhs` ON FIRST 1 OUTPUT Lhs.1 'arg0', Rhs.1, Lhs.0 'arg1', Lhs.2 'arg2'
34558      ~3%     {3}    | JOIN WITH `Class::Class.getAMethod/0#dispred#66416e47` ON FIRST 2 OUTPUT Lhs.0 'arg0', Lhs.2 'arg1', Lhs.3 'arg2'
                   return r1
...
(382s) Starting to evaluate predicate DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3/3@i6#f893b1xh (iteration 6)
(382s)                     - DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3_delta has 125138 rows (order for disjuncts: delta=<standard>).
(382s) Tuple counts for DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3/3@i6#f893b1xh after 12ms:
33363  ~0%     {2} r1 = SCAN `DataFlowDispatch::getNextClassInMro/1#e1ee596a#prev_delta` OUTPUT In.1, In.0 'cls'
159696 ~0%     {3}    | JOIN WITH `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev` ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.1 'name', Rhs.2 'result'
125138 ~1%     {3}    | AND NOT `_Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs`(FIRST 3)

0      ~0%     {3} r2 = JOIN `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_delta` WITH `DataFlowDispatch::getNextClassInMro/1#e1ee596a#reorder_1_0#prev` ON FIRST 1 OUTPUT Lhs.1 'name', Lhs.2 'result', Rhs.1 'cls'
               {3}    | AND NOT `_Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#__#antijoin_rhs`(FIRST 3)
0      ~0%     {3}    | SCAN OUTPUT In.2 'cls', In.0 'name', In.1 'result'

125138 ~1%     {3} r3 = r1 UNION r2
125138 ~1%     {3}    | AND NOT `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev`(FIRST 3)
               return r3
```

And now
```
(18s) Tuple counts for DataFlowDispatch::class_has_method/2#0d2ae9c0/2@ff66c1lr after 18ms:
202279 ~1%     {2} r1 = JOIN `Class::Class.getAMethod/0#dispred#66416e47_10#join_rhs` WITH `Function::Function.getName/0#dispred#033700ef` ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.1 'name'
               return r1
...
(490s) Tuple counts for DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3/3@i6#48b6c1xi after 54ms:
0      ~0%     {3} r1 = JOIN `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_delta` WITH `DataFlowDispatch::getNextClassInMro/1#e1ee596a#reorder_1_0#prev` ON FIRST 1 OUTPUT Rhs.1 'cls', Lhs.1 'name', Lhs.2 'result'
0      ~0%     {3}    | AND NOT `DataFlowDispatch::class_has_method/2#0d2ae9c0`(FIRST 2)

33363  ~0%     {2} r2 = SCAN `DataFlowDispatch::getNextClassInMro/1#e1ee596a#prev_delta` OUTPUT In.1, In.0 'cls'
159696 ~0%     {3}    | JOIN WITH `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev` ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.1 'name', Rhs.2 'result'
125138 ~1%     {3}    | AND NOT `DataFlowDispatch::class_has_method/2#0d2ae9c0`(FIRST 2)

125138 ~1%     {3} r3 = r1 UNION r2
125138 ~1%     {3}    | AND NOT `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev`(FIRST 3)
               return r3
```
2024-06-21 12:16:27 +00:00
Kasper Svendsen
988d0671bb Merge pull request #16734 from kaspersv/kaspersv/doc-intern-sets-builtin
Document builtin InternSets module
2024-06-21 12:06:40 +02:00
Tom Hvitved
dff3ce2a9f Merge pull request #16794 from hvitved/ruby/sinatra-flow
Ruby: Rework `Sinatra.FilterJumpStep`
2024-06-21 11:38:10 +02:00
Tom Hvitved
8ea4f85de3 Ruby: Rework Sinatra.FilterJumpStep 2024-06-21 08:57:59 +02:00
Tom Hvitved
95c764eff6 Fix Sinatra test to properly output pathgraph 2024-06-21 08:57:19 +02:00
Erik Krogh Kristensen
db768960f4 Merge pull request #15060 from am0o0/amammad-js-envinjection
JS: Env Injection query
2024-06-20 21:27:21 +02:00
Erik Krogh Kristensen
555d7e5958 Merge pull request #14293 from am0o0/amammad-js-CodeInjection_dynamic_import
JS: Dynamic import as code injection sink
2024-06-20 21:19:57 +02:00
Erik Krogh Kristensen
60ed51781e Merge pull request #16790 from github/max-schaefer-patch-1
JavaScript: Fix CodeQL alert in extractor
2024-06-20 20:20:00 +02:00
Erik Krogh Kristensen
e84028d01e Merge pull request #14088 from am0o0/amammad-js-JWT
JS: decoding JWT without signature verification
2024-06-20 20:13:40 +02:00
Jeroen Ketema
0e04a59c08 Merge pull request #16795 from jketema/test-cleanup
C++: Remove unneeded options from tests
2024-06-20 16:24:07 +02:00
Jeroen Ketema
4c4c15b425 C++: Remove unneeded options from tests 2024-06-20 14:21:34 +02:00
Asger F
a36e39359f Merge pull request #16739 from RasmusWL/js-array-steps
JS: Allow many Array steps to be used in type-tracking
2024-06-20 11:39:46 +02:00
Rasmus Wriedt Larsen
596102d3fb Update javascript/ql/lib/change-notes/2024-06-14-type-tracking-array-steps.md
Co-authored-by: Asger F <asgerf@github.com>
2024-06-20 10:07:49 +02:00
Max Schaefer
2be171746b JavaScript: Fix CodeQL alert in extractor
This doesn't make a difference in practice because we only run the method on arrays of even length, but we might as well fix it.
2024-06-19 17:13:01 +01:00
Tom Hvitved
6dbdc9e17f Merge pull request #16784 from github/redsun82/fix-warnings-in-ql-tests
C++/Java: Accept new warning format in ql tests
2024-06-19 13:05:50 +02:00
Paolo Tranquilli
b7a2ea8981 CI: accept other diagnostic format related test changes 2024-06-19 11:33:50 +02:00
Paolo Tranquilli
59f8f8a394 Merge branch 'main' into redsun82/fix-warnings-in-ql-tests 2024-06-19 11:21:36 +02:00
Tamás Vajk
45ece48b6f Merge pull request #16776 from tamasvajk/fix/source-generator-folder
C#: Make sure no file is added twice to the compilation
2024-06-19 10:09:50 +02:00
Paolo Tranquilli
919ddccfdb C++/Java: Accept new warning format in ql tests 2024-06-19 09:13:18 +02:00
Edward Minnix III
7adfa6bbed Merge pull request #16709 from egregius313/egregius313/go/df/threat-models/refactor-queries
Go: Refactor queries to use `ThreatModelFlowSource` instead of `RemoteFlowSource`
2024-06-18 13:56:00 -04:00
Ed Minnix
5bbd003dfc Reword change note 2024-06-18 12:27:21 -04:00
Ed Minnix
b53712cae0 Change note 2024-06-18 12:27:19 -04:00
Ed Minnix
6a0be6ad09 ExternalAPIs 2024-06-18 12:27:18 -04:00
Ed Minnix
46e16b88bb Refactor experimental queries to use ThreadModelFlowSource 2024-06-18 12:27:17 -04:00
Ed Minnix
cfd5f53eb0 Refactor Customizations libraries to use ThreatModelFlowSource 2024-06-18 12:27:15 -04:00
Edward Minnix III
8997f2cdf2 Merge pull request #16697 from egregius313/egregius313/go/dataflow/threat-modeling
Go: Introduce Threat Modeling
2024-06-18 12:25:33 -04:00
Taus
59a77a873c Merge pull request #16754 from github/tausbn/python-disregard-unused-imports-in-pytest-tests
Python: Disregard unused imports in `pytest` tests
2024-06-18 15:10:31 +02:00
Owen Mansel-Chan
9403bf25d8 Merge pull request #16667 from smowton/smowton/fix/global-variable-side-effect
Add support for flow through content of global variables
2024-06-18 13:41:57 +01:00
Tamas Vajk
6c8e391a63 C#: Make sure no file is added twice to the compilation 2024-06-18 13:45:09 +02:00
Tamas Vajk
5170585515 C#: Add integration test with file added multiple times in the same compilation 2024-06-18 13:45:06 +02:00
Max Schaefer
2c4a95bb5e Merge pull request #16777 from github/max-schaefer-patch-1
Swift: Add missing bracket in example
2024-06-18 11:39:42 +01:00
Alex Ford
6c3d90e8a0 Merge pull request #16650 from alexrford/rb/routing-improvements
Ruby: ActionDispatch - support `path => target` route format
2024-06-18 11:17:05 +01:00
Max Schaefer
9ca74de3e9 Swift: Add missing bracket in example 2024-06-18 11:00:59 +01:00
Joe Farebrother
33704779ea Merge pull request #16503 from joefarebrother/ruby-sensitive-sources
Ruby: Use additional sensitive data heuristics for CleartextSources
2024-06-18 10:57:55 +01:00
Michael Nebel
cd9d58fdc8 Merge pull request #16772 from michaelnebel/java/taintedpermissionthreatmodel
Java: Opt-in `java/tainted-permissions-check` to threat models.
2024-06-18 10:54:28 +02:00
Joe Farebrother
eee7f5a896 Use a combined regex for performance 2024-06-17 22:21:33 +01:00
Cornelius Riemenschneider
0f98d9e815 Merge pull request #16774 from github/criemen/cleanup
Remove unused bzl files.
2024-06-17 20:04:25 +02:00
Chris Smowton
4da5d6660a Add change note 2024-06-17 16:49:09 +01:00
Chris Smowton
38ee085782 Adjust test expectation 2024-06-17 16:46:49 +01:00
Chris Smowton
822f6eebfb Add support for flow through content of global variables 2024-06-17 16:42:23 +01:00
Ed Minnix
b4ecc81145 Fix provenance numbers in tests 2024-06-17 11:33:19 -04:00
Ed Minnix
fa2c50616b Remove getSourceType definitions 2024-06-17 10:57:11 -04:00
Ed Minnix
b6adff6377 Fix jsoniter test 2024-06-17 10:51:13 -04:00
Ed Minnix
53dd269f9f Go: Threat model tests 2024-06-17 10:51:11 -04:00
Ed Minnix
7f19f449eb Change note 2024-06-17 10:51:10 -04:00
Ed Minnix
49fb372eb9 Add getSourceType declarations to existing remote flow sources 2024-06-17 10:51:09 -04:00
Ed Minnix
df6449cfc7 Go: Add the SourceNode and ThreatModelFlowSource classes 2024-06-17 10:51:07 -04:00
Ed Minnix
b697068e9a Go: Add threat modeling shared library 2024-06-17 10:51:06 -04:00
Michael Nebel
5686efd25c Update java/ql/src/change-notes/2024-06-17-tainted-permissions-check.md
Co-authored-by: Owen Mansel-Chan <62447351+owen-mc@users.noreply.github.com>
2024-06-17 16:47:22 +02:00
Cornelius Riemenschneider
f41bd41bd0 Remove unused bzl files. 2024-06-17 15:15:08 +02:00