Commit Graph

9261 Commits

Author SHA1 Message Date
Rasmus Lerchedahl Petersen
c004ffaca8 python: move model to Stdlib.yml
There is already a model there so we add to that one.

We did observe that this existing model was blocked by the external MaD model.
This is concerning and needs to be cleared up.
2024-06-25 14:13:48 +02:00
Rasmus Lerchedahl Petersen
281ac05868 python: add modelling for urlib.parse
- `quote` together with `re.compile` recover regex injection alerts on haiwen/seahub
- `quote_plus` recovers the URL redirection alert on DemocracyClub/EveryElection
- `unquote` recovers path injection alerts on `cloudera/hue`
- it was tedious finding justifications for the rest..
2024-06-25 14:13:44 +02:00
Rasmus Lerchedahl Petersen
df406b4fca python: Start modelling using MaD
- empty models for now
- `summaryModel` of `codeql/python-all` will be added to shortly.
2024-06-25 14:13:41 +02:00
Rasmus Lerchedahl Petersen
aa4fd1992e Python: compact types in type models 2024-06-25 11:59:55 +02:00
Rasmus Lerchedahl Petersen
b902dd5680 Python: add change note 2024-06-25 11:54:30 +02:00
github-actions[bot]
fd385736e6 Post-release preparation for codeql-cli-2.17.6 2024-06-25 06:39:45 +00:00
Joe Farebrother
0901b3d0a6 Add change note 2024-06-24 21:43:09 +01:00
Joe Farebrother
d0f735ac28 Update tests for restframework 2024-06-24 20:52:09 +01:00
Joe Farebrother
c404f00a9b Add additional header write models for aiohttp and tornado + added qldoc 2024-06-24 17:27:25 +01:00
Joe Farebrother
79c0ed6074 Add additional fastapi mheader write models 2024-06-24 17:27:21 +01:00
Joe Farebrother
5ced5c010c Add django header writes 2024-06-24 17:27:15 +01:00
Joe Farebrother
7704801e47 Change fastapi raw cookie header models to header write models 2024-06-24 17:27:12 +01:00
Joe Farebrother
a0201e9c4f Update tests for new cookie write from headers 2024-06-24 17:27:06 +01:00
Joe Farebrother
6b8080a5b3 Update concept tests for header writes 2024-06-24 17:27:02 +01:00
Joe Farebrother
d11f58f768 Add cookie header write concept from experimental. 2024-06-24 17:26:56 +01:00
Joe Farebrother
b71ba7c30f Move Header Write derrived concepts to Concepts 2024-06-24 17:26:51 +01:00
github-actions[bot]
e32a587078 Release preparation for version 2.17.6 2024-06-24 14:33:10 +00:00
Anders Schack-Mulligen
8c23e21073 Dataflow: Cache compatibleTypes. 2024-06-24 13:35:48 +02:00
Rasmus Lerchedahl Petersen
00fbada41d Python: recognize fabric.operations 2024-06-24 10:54:59 +02:00
Taus
4a448f445e Merge pull request #15715 from am0o0/am0o0-python-codeExec
Python: New command execution sinks
2024-06-21 14:26:33 +02:00
Taus
6db7e72fb8 Python: Fix bad join in DataFlowDispatch
A case of bad magic. Rather than evaluating separately whether a class
has a method of some name, the compiler opted to magick in the fact
that this was done as part of the `findFunctionAccordingToMro`
predicate. Hilarity ensued.

However, _we_ know that magic really isn't needed in this case (the
number of results is bounded by `Class.getAMethod` since methods have
only a single name), so by factoring it out into a helper predicate, we
can help the join-orderer along.

Before
```
(377s) Starting to evaluate predicate _DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_DataFlowDispatch::getNextClassInMro/1#__#shared/3@i6#L3#f893bw2h (iteration 6)
(377s) Tuple counts for _DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_DataFlowDispatch::getNextClassInMro/1#__#shared/3@i6#L3#f893bw2h after 16ms:
33363  ~0%     {2} r1 = SCAN `DataFlowDispatch::getNextClassInMro/1#e1ee596a#prev_delta` OUTPUT In.1, In.0 'arg1'
159696 ~4%     {3}    | JOIN WITH `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev` ON FIRST 1 OUTPUT Rhs.1 'arg0', Lhs.1 'arg1', Rhs.2 'arg2'
               return r1
(377s) Starting to evaluate predicate _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs/3@i6#L4#f893bw2h (iteration 6)
(382s) Tuple counts for _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs/3@i6#L4#f893bw2h after 4.4s:
1770825904 ~4%     {4} r1 = JOIN `_DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_DataFlowDispatch::getNextClassInMro/1#__#shared` WITH `Function::Function.getName/0#dispred#033700ef_10#join_rhs` ON FIRST 1 OUTPUT Lhs.1 'arg0', Rhs.1, Lhs.0 'arg1', Lhs.2 'arg2'
34558      ~3%     {3}    | JOIN WITH `Class::Class.getAMethod/0#dispred#66416e47` ON FIRST 2 OUTPUT Lhs.0 'arg0', Lhs.2 'arg1', Lhs.3 'arg2'
                   return r1
...
(382s) Starting to evaluate predicate DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3/3@i6#f893b1xh (iteration 6)
(382s)                     - DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3_delta has 125138 rows (order for disjuncts: delta=<standard>).
(382s) Tuple counts for DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3/3@i6#f893b1xh after 12ms:
33363  ~0%     {2} r1 = SCAN `DataFlowDispatch::getNextClassInMro/1#e1ee596a#prev_delta` OUTPUT In.1, In.0 'cls'
159696 ~0%     {3}    | JOIN WITH `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev` ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.1 'name', Rhs.2 'result'
125138 ~1%     {3}    | AND NOT `_Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs`(FIRST 3)

0      ~0%     {3} r2 = JOIN `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_delta` WITH `DataFlowDispatch::getNextClassInMro/1#e1ee596a#reorder_1_0#prev` ON FIRST 1 OUTPUT Lhs.1 'name', Lhs.2 'result', Rhs.1 'cls'
               {3}    | AND NOT `_Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#__#antijoin_rhs`(FIRST 3)
0      ~0%     {3}    | SCAN OUTPUT In.2 'cls', In.0 'name', In.1 'result'

125138 ~1%     {3} r3 = r1 UNION r2
125138 ~1%     {3}    | AND NOT `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev`(FIRST 3)
               return r3
```

And now
```
(18s) Tuple counts for DataFlowDispatch::class_has_method/2#0d2ae9c0/2@ff66c1lr after 18ms:
202279 ~1%     {2} r1 = JOIN `Class::Class.getAMethod/0#dispred#66416e47_10#join_rhs` WITH `Function::Function.getName/0#dispred#033700ef` ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.1 'name'
               return r1
...
(490s) Tuple counts for DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3/3@i6#48b6c1xi after 54ms:
0      ~0%     {3} r1 = JOIN `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev_delta` WITH `DataFlowDispatch::getNextClassInMro/1#e1ee596a#reorder_1_0#prev` ON FIRST 1 OUTPUT Rhs.1 'cls', Lhs.1 'name', Lhs.2 'result'
0      ~0%     {3}    | AND NOT `DataFlowDispatch::class_has_method/2#0d2ae9c0`(FIRST 2)

33363  ~0%     {2} r2 = SCAN `DataFlowDispatch::getNextClassInMro/1#e1ee596a#prev_delta` OUTPUT In.1, In.0 'cls'
159696 ~0%     {3}    | JOIN WITH `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev` ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.1 'name', Rhs.2 'result'
125138 ~1%     {3}    | AND NOT `DataFlowDispatch::class_has_method/2#0d2ae9c0`(FIRST 2)

125138 ~1%     {3} r3 = r1 UNION r2
125138 ~1%     {3}    | AND NOT `DataFlowDispatch::findFunctionAccordingToMro/2#a610c0a3#prev`(FIRST 3)
               return r3
```
2024-06-21 12:16:27 +00:00
Rasmus Lerchedahl Petersen
280a9b4408 Python: Support Model Editor 2024-06-21 11:47:51 +02:00
Rasmus Lerchedahl Petersen
f0e68887d4 Python: autoformat 2024-06-20 10:59:39 +02:00
Rasmus Lerchedahl Petersen
5cb37f5c4c python: Document MaD format
- add a few tests reflecting the documentation
- make the mentioned sink-kinds have an effect on relevant queries
2024-06-19 17:00:15 +02:00
Paolo Tranquilli
b7a2ea8981 CI: accept other diagnostic format related test changes 2024-06-19 11:33:50 +02:00
am0o0
ccb923a436 fix formatting 2024-06-18 18:31:29 +02:00
am0o0
1f99559e9f Revert "update id of the query file"
This reverts commit 1f112467ce.
2024-06-18 17:33:07 +02:00
am0o0
8a7fdfa6fe fix conflict 2024-06-18 17:18:59 +02:00
Paolo Tranquilli
daea773fce Python: tests with false positives around match 2024-06-14 17:28:35 +02:00
Taus
b7b0f84e8b Python: Handle @pytest.fixture decorations with arguments as well
Not the prettiest of solutions, but it seems to work well enough.
2024-06-14 15:11:25 +00:00
Paolo Tranquilli
1046d03486 Python: update unused import test case for pytest 2024-06-14 16:55:05 +02:00
Taus
2f00a0d323 Python: Also test pytest fixture factories 2024-06-14 13:11:00 +00:00
Taus
78729180ad Python: Fix pytest fixture unused import FPs 2024-06-14 12:05:55 +00:00
Taus
f3a9c9a9dc Python: Add tests for pytest fixture unused import FPs 2024-06-14 12:03:43 +00:00
Joe Farebrother
f441c68f7e Merge pull request #16657 from joefarebrother/python-partial-ssrf-fp
Python: Add additional sanitizers to SSRF
2024-06-11 23:20:50 +01:00
Mathias Vorreiter Pedersen
3351b9547d Merge branch 'rc/3.14' into rc-3.14-mergeback 2024-06-11 16:21:08 +01:00
Joe Farebrother
93f10fcf14 Add sanitizers for compiled regexes 2024-06-11 15:44:16 +01:00
github-actions[bot]
8a25081a0e Post-release preparation for codeql-cli-2.17.5 2024-06-10 15:33:08 +00:00
Cornelius Riemenschneider
092bc6445d Rust/bazel: Port to bzlmod.
This gets rid of our last workspace dependency.
In particular, this change also gets rid of the checked-in extra
lock files that took forever to generate.
2024-06-10 17:03:58 +02:00
github-actions[bot]
877bfa2468 Release preparation for version 2.17.5 2024-06-10 13:40:39 +00:00
Anders Schack-Mulligen
68ddae2918 Python: Add support for pretty-printed provenace in tests. 2024-06-07 11:47:48 +02:00
Asger F
6e0f3df573 Merge pull request #14120 from asgerf/dynamic/typemodel-istypeused
Dynamic: add TypeModel.isTypeUsed
2024-06-06 15:31:16 +02:00
Joe Farebrother
6ff7fb2a70 Add change note 2024-06-04 09:52:57 +01:00
Joe Farebrother
9331c2c33a Add tests 2024-06-04 09:39:37 +01:00
Rasmus Wriedt Larsen
839171e557 Merge pull request #16646 from RasmusWL/url-redirect-qhelp
Python: Update url-redirect qhelp with `https:/example.com` handling
2024-06-04 10:17:37 +02:00
Rasmus Wriedt Larsen
dd8b65130e Merge pull request #16598 from jorgectf/jorgectf/opml-models
Python: Add models for `opml`
2024-06-04 10:16:26 +02:00
Joe Farebrother
6ac46b8436 Add additional sanitizers to SSRF for methods that restrict the contents of a string. 2024-06-03 23:23:25 +01:00
yoff
fe8f13e8dd Merge pull request #16654 from sidshank/sidshank/fix-incorrect-deprecation-comment
Python: Fixes an error in the deprecation notice for a taint tracking configuration
2024-06-03 21:46:35 +02:00
Sid Shankar
859e8db5f2 Fixes typo in deprecation notice 2024-06-03 16:31:29 +00:00
Cornelius Riemenschneider
c30cc0f665 Fix formatting. 2024-06-03 16:10:41 +02:00