Commit Graph

87274 Commits

Author SHA1 Message Date
aegilops
fc6fba8d06 Fixed CWE tags 2024-07-01 14:25:47 +01:00
aegilops
d1d082982a More external references 2024-07-01 14:25:29 +01:00
am0o0
d31711bd89 merge all ne flow sources into one by extending current abstract class 2024-07-01 15:16:44 +02:00
am0o0
f1324a413a update qlhelp 2024-07-01 15:09:56 +02:00
am0o0
b360c8adb8 Update hardcodedCredentials query file to only exclude 'jwt key' kind from with the isTestFile predicate.
According to expected test results, with a new query, the jwt sinks of __test__/ dir have been exluded from query results.
2024-07-01 15:00:08 +02:00
Taus
446dbf67cc Python: Fix bad join in getImmediateModuleReference
The "most expensive predicates" report had the following line on a
certain database:

```
1m15s |    11 |   37s @ 4    | ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0@12bb4xdo
```

Investigating further revealed the following bad joins

```
(388s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i4#b2325xoe after 36.5s:
0         ~0%        {2} r1 = JOIN `ImportResolution::ImportResolution::sys_modules_module_with_name/1#134529bf#prev_delta` WITH `ImportResolution::ImportResolution::getReferenceToModuleName/1#bc5da225` ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1 'm'

74884348  ~0%        {3} r2 = JOIN `ImportResolution::ImportResolution::getModuleReference/1#28368ea4#prev_delta` WITH `ImportResolution::ImportResolution::potential_module_export/2#19340171` ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Lhs.0
5221604   ~0%        {3}    | JOIN WITH `Attributes::AttrRef.accesses/2#dispred#31929f12_120#join_rhs` ON FIRST 2 OUTPUT Rhs.2 'result', Lhs.2, Lhs.1
5219926   ~2%        {3}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.0 'result'
5300880   ~1%        {2}    | JOIN WITH `ImportResolution::ImportResolution::module_export/3#f2fc6a2a` ON FIRST 2 OUTPUT Rhs.2, Lhs.2 'result'
42211     ~5%        {2}    | JOIN WITH `ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0#prev` ON FIRST 1 OUTPUT Lhs.1 'result', Rhs.1 'm'

957042    ~4%        {3} r3 = JOIN `ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0#prev_delta` WITH `ImportResolution::ImportResolution::module_export/3#f2fc6a2a_201#join_rhs` ON FIRST 1 OUTPUT Rhs.1, Rhs.2, Lhs.1 'm'
957035    ~0%        {3}    | JOIN WITH `ImportResolution::ImportResolution::potential_module_export/2#19340171` ON FIRST 2 OUTPUT Lhs.1, Lhs.2 'm', Lhs.0
236753257 ~1%        {4}    | JOIN WITH `Attributes::AttrRef.accesses/2#dispred#31929f12_201#join_rhs` ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1 'm', Lhs.2, Rhs.2
199557145 ~2%        {4}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.2, Lhs.3, Lhs.1 'm', Lhs.0 'result'
1         ~0%        {2}    | JOIN WITH `ImportResolution::ImportResolution::getModuleReference/1#28368ea4#prev` ON FIRST 2 OUTPUT Lhs.3 'result', Lhs.2 'm'

15199013  ~1951%     {2} r4 = JOIN `ImportResolution::ImportResolution::getModuleReference/1#28368ea4#prev_delta` WITH `Module::Module.getPackageName/0#dispred#bb0c3872` ON FIRST 1 OUTPUT Lhs.1, Rhs.1
14707604  ~2136%     {3}    | JOIN WITH `Attributes::AttrRef.accesses/2#dispred#31929f12_102#join_rhs` ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1, Rhs.2

14623588  ~2190%     {4} r5 = JOIN r4 WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT _, Lhs.0 'result', Lhs.1, Lhs.2
14623588  ~2058%     {2}    | REWRITE WITH Tmp.0 := ".", Out.0 := (In.2 ++ Tmp.0 ++ In.3) KEEPING 2

14623588  ~2139%     {5} r6 = JOIN r4 WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT _, Lhs.0 'result', Lhs.1, Lhs.2, _
14623588  ~2092%     {2}    | REWRITE WITH Tmp.0 := ".", Tmp.0 := (In.2 ++ Tmp.0 ++ In.3), Tmp.4 := ".__init__", Out.0 := (Tmp.0 ++ Tmp.4) KEEPING 2

29247176  ~2099%     {2} r7 = r5 UNION r6
199786001 ~6922%     {2}    | JOIN WITH `Module::isPreferredModuleForName/2#5fb427f9_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'result'
199756923 ~7024%     {2}    | JOIN WITH `Module::Module.getFile/0#dispred#53eb9b1b_10#join_rhs` ON FIRST 1 OUTPUT Lhs.1 'result', Rhs.1 'm'

199799135 ~6954%     {2} r8 = r1 UNION r2 UNION r3 UNION r7
199793992 ~6954%     {2}    | AND NOT `ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0#prev`(FIRST 2)
                     return r8
```

Clearly, waiting to joining with `getModuleReference` last is not
healthy. To fix this, I opted to simply create a helper predicate for
the `accesses` construct.

After this change, here are the iteration timings

```
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i1#74f41yqa after 1.2s:
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i1#8a053ys7 after 1.3s:
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i2#74f41yqa after 20ms:
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i2#8a053ys7 after 20ms:
(337s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i4#74f41yqa after 8.5s:
(341s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i4#8a053ys7 after 3.2s:
(346s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i5#74f41yqa after 7.2s:
(349s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i6#74f41yqa after 3ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i5#8a053ys7 after 10s:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i8#74f41yqa after 37ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i9#74f41yqa after 0ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i10#74f41yqa after 0ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i11#74f41yqa after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i12#74f41yqa after 1ms:
(353s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i6#8a053ys7 after 1ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i8#8a053ys7 after 7ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i9#8a053ys7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i10#8a053ys7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i11#8a053ys7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i12#8a053ys7 after 0ms:
```

And the helper predicate itself is also quick to evaluate:

```
(327s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i1#74f41xqa after 0ms:
(327s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i1#8a053xs7 after 0ms:
(329s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i3#74f41xqa after 99ms:
(337s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i3#8a053xs7 after 98ms:
(338s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i4#74f41xqa after 679ms:
(341s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i4#8a053xs7 after 400ms:
(346s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i5#74f41xqa after 1ms:
(349s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i6#74f41xqa after 22ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i5#8a053xs7 after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i7#74f41xqa after 1.4s:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i8#74f41xqa after 8ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i9#74f41xqa after 0ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i10#74f41xqa after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i11#74f41xqa after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i12#74f41xqa after 1ms:
(353s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i13#74f41xqa after 806ms:
(353s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i6#8a053xs7 after 7ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i7#8a053xs7 after 870ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i8#8a053xs7 after 2ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i9#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i10#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i11#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i12#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i13#8a053xs7 after 276ms:
```

(I note that we appear to be evaluating this code twice, which is a bit
worrying. I'll leave that investigaton for later.)
2024-07-01 12:53:04 +00:00
am0o0
5a1877547f update test cases of __tests__/ dir
since we want to check if a jwt related sink is in this dir or not
2024-07-01 14:50:07 +02:00
am0o0
6ecd8b7ee8 add new default cred kind 2024-07-01 14:42:34 +02:00
Taus
d9b337cb2c Merge pull request #16804 from github/tausbn/python-fix-bad-join-in-dataflow-dispatch
Python: Fix bad join in `DataFlowDispatch`
2024-07-01 13:14:28 +02:00
Joe Farebrother
b81d41ba7b Add django header write models for direct subscript write 2024-07-01 11:26:54 +01:00
am0o0
fa8c457015 move the TextEncoder and Buffer jose.base64url taint steps to a local query taint step 2024-07-01 12:11:53 +02:00
am0o0
60aa711005 implement TextEncoderStep taint step with globalVarRef predicate 2024-07-01 11:59:05 +02:00
am0o0
65fdb8ccce move jose SharedTaintStep to a local taint step, add more additional steps with test cases, update test cases and expected test results 2024-07-01 11:38:17 +02:00
Mathias Vorreiter Pedersen
0b4459db75 C++: Add more QLDoc. 2024-07-01 10:13:32 +01:00
Arthur Baars
c6d02e4909 Merge pull request #16878 from github/aibaars/merge-3.14
Merge rc/3.14 into main
2024-07-01 11:04:57 +02:00
Tom Hvitved
c5c97aca50 C#: Move implicit entry definitions inside method bodies in SSA construction 2024-07-01 10:48:13 +02:00
Owen Mansel-Chan
0db4110449 Merge pull request #16879 from github/workflow/coverage/update
Update CSV framework coverage reports
2024-06-30 13:59:16 +01:00
github-actions[bot]
26194eb65f Add changed framework coverage reports 2024-06-30 00:19:16 +00:00
Owen Mansel-Chan
60b9d19d72 Add explicit relation between "reverse-dns" and "all" 2024-06-29 21:29:21 +01:00
Owen Mansel-Chan
390fecbef6 Merge pull request #16866 from owen-mc/go/add-go-jose-to-library-coverage-frameworks
Go: Add Go JOSE to library coverage frameworks
2024-06-29 07:30:36 +01:00
Rasmus Lerchedahl Petersen
e40ae2e52d Python: adjust test expectations
MaD row numbers in provenance column
2024-06-28 21:56:11 +02:00
Arthur Baars
b12b33c8f9 Merge remote-tracking branch 'upstream/main' into 'rc/3.14' 2024-06-28 19:50:35 +02:00
Jeroen Ketema
a85f73972c Merge pull request #16870 from jketema/preproc-more
C++: Update test expectations
2024-06-28 19:01:12 +02:00
Mathew Payne
ed314b1799 docs: Add Change Notes 2024-06-28 14:42:35 +01:00
Mathew Payne
1cf9714272 feat(python): Add Hardcoded Credentials MaD support 2024-06-28 14:30:36 +01:00
Rasmus Lerchedahl Petersen
77a00873a9 Python: add tests for loggers 2024-06-28 15:25:17 +02:00
Rasmus Lerchedahl Petersen
5ddfe75a0d Python: Add value steps for sequence elements
It would be nice to simplify to a single sequence content type..
2024-06-28 15:10:08 +02:00
Rasmus Lerchedahl Petersen
59f953269a Python: remove strange sink
It is not clear from the code how this could happen and
I do not remember the path I saw, perhaps it was unreasonable.
2024-06-28 14:42:24 +02:00
yoff
bbc3ff2dfe Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2024-06-28 14:39:03 +02:00
Asger F
1d267efb6b JS: Fix missing qldoc 2024-06-28 14:30:56 +02:00
Tom Hvitved
075a249f3b C#: Add SSA test for multi-bodied method 2024-06-28 14:27:51 +02:00
Jami
42925b56e3 Merge pull request #15921 from jcogs33/jcogs33/unsafe-url-forward-promotion-resource-and-file-methods
Java: add models for some resource-related methods
2024-06-28 08:05:50 -04:00
Erik Krogh Kristensen
60811116ab Merge pull request #16332 from erik-krogh/ts55
JS: upgrade TypeScript to 5.5
2024-06-28 13:59:52 +02:00
Tamas Vajk
199a9688af C#: Fix quality issues 2024-06-28 13:58:08 +02:00
Asger F
e5924c1f84 JS: Another messy test update 2024-06-28 13:08:38 +02:00
Asger F
14fc790617 Update DataFlowConsistency.expected 2024-06-28 13:08:09 +02:00
Tamás Vajk
1cf5e89b96 Merge pull request #16747 from tamasvajk/buildless/binary-log-extractor-2
C#: Add binlog support to buildless with source generator support
2024-06-28 12:39:54 +02:00
Alvaro Muñoz
39bff38d70 Bump qlpack versions 2024-06-28 12:32:18 +02:00
Alvaro Muñoz
a485528ebe Refactor bash script parsing to improve coverage of env var injection 2024-06-28 12:31:43 +02:00
Erik Krogh Kristensen
fd3089ee5a Merge pull request #14342 from maikypedia/maikypedia/javascript-cors
JS: Add Permissive CORS query (CWE-942)
2024-06-28 11:27:37 +02:00
Mathias Vorreiter Pedersen
72caadb7db Merge pull request #16869 from hvitved/shared/cfg/mermaid-output
Shared: Generate mermaid output in `View CFG` query
2024-06-28 10:21:02 +01:00
Tom Hvitved
e1c1314824 Shared: Generate mermaid output in View CFG query 2024-06-28 09:45:22 +02:00
Tamas Vajk
4db586f17d Add TSP diagnostic for binlog usage 2024-06-28 09:40:41 +02:00
Tamas Vajk
c8927447f5 Record diagnostics in binlog extraction test 2024-06-28 09:40:13 +02:00
Jami Cogswell
85a1e1a972 Java: update change note date 2024-06-27 22:11:01 -04:00
Jami Cogswell
be565288f2 Java: update more test cases due to shifted alert provenance line numbers 2024-06-27 22:08:38 -04:00
Jami Cogswell
c73af7f789 Java: update some test cases due to shifted alert provenance line numbers 2024-06-27 21:07:35 -04:00
Alvaro Muñoz
40a6f3bbee Make EnvVar and Path injection equivalent 2024-06-27 22:53:55 +02:00
Alvaro Muñoz
a9ea9a1f8a Update expected test files 2024-06-27 22:53:32 +02:00
Alvaro Muñoz
effa1e1356 Move ControlChecks to its own file 2024-06-27 22:53:20 +02:00