Commit Graph

665 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
bdadf2b445 Python: Fix warnings 2022-04-13 10:30:59 +02:00
Rasmus Wriedt Larsen
4927f0018b Merge branch 'main' into django-filefield-uploadto 2022-04-13 10:22:28 +02:00
Anders Schack-Mulligen
4eaec3953a Merge pull request #8694 from aschackmull/dataflow/cleanup-unused
Dataflow: Cleanup unused column
2022-04-07 15:16:27 +02:00
Anders Schack-Mulligen
7beed570f2 Dataflow: Sync. 2022-04-07 13:53:48 +02:00
Erik Krogh Kristensen
7e4c76c63b revert API-graph change in Flask.qll 2022-04-07 13:52:14 +02:00
Erik Krogh Kristensen
bdfd6bdc79 fix a ql/field-only-used-in-charpred warning 2022-04-07 13:52:14 +02:00
Erik Krogh Kristensen
50bfc8eaa0 refactor uses of API::Node::getAUse() that should have been something else 2022-04-07 13:52:13 +02:00
Erik Krogh Kristensen
4e5afab082 refactor more python type-trackers to API-graphs 2022-04-07 13:51:40 +02:00
Rasmus Wriedt Larsen
f8f41428df Python: Minor refactor for FlaskViewClass 2022-04-06 15:15:42 +02:00
Rasmus Wriedt Larsen
1c2323eb85 Python: Refactor how we find a Class from API::Node
Using `getAnImmediateUse` might give better performance than `getAUse`.

Since all the changed code is about `API::Node`s that are found after
doing `.getASubclass*()`, this change is OK.

It's also nice to align how we actually do this.
2022-04-06 15:12:24 +02:00
Tom Hvitved
4099d1318f Data flow: Tweak two join-orders
Before
```
[2022-04-06 13:19:29] (96s) Tuple counts for DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff/2@i14#aa10f2wi after 4.4s:
                      10681    ~0%     {2} r1 = SCAN DataFlowImpl2::Stage1::revFlow#7ad53399#fff#prev_delta OUTPUT In.0, In.2 'config'
                      982      ~1%     {3} r2 = JOIN r1 WITH DataFlowImpl2::readSet#7ad53399#ffff_2301#join_rhs ON FIRST 2 OUTPUT Rhs.3, Lhs.1 'config', Rhs.2
                      83691528 ~2%     {3} r3 = JOIN r2 WITH DataFlowPublic::ContentSet::getAReadContent#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1 'config', Lhs.2, Rhs.1 'c'
                      83581763 ~2%     {3} r4 = r3 AND NOT DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff#prev(Lhs.2 'c', Lhs.0 'config')
                      83581763 ~0%     {3} r5 = SCAN r4 OUTPUT In.2 'c', In.0 'config', In.1
                      0        ~0%     {3} r6 = JOIN r5 WITH DataFlowImpl2::Stage1::fwdFlowConsCand#7ad53399#ff ON FIRST 2 OUTPUT Lhs.2, Lhs.1 'config', Lhs.0 'c'
                      0        ~0%     {2} r7 = JOIN r6 WITH DataFlowImpl2::Stage1::fwdFlow#7ad53399#2#fff_02#join_rhs ON FIRST 2 OUTPUT Lhs.2 'c', Lhs.1 'config'
                                       return r7
```

After
```
[2022-04-06 13:44:38] (6s) Tuple counts for DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff/2@i14#5abbf2wn after 6ms:
                      10681  ~0%     {2} r1 = SCAN DataFlowImpl2::Stage1::revFlow#7ad53399#fff#prev_delta OUTPUT In.0, In.2 'config'
                      982    ~1%     {3} r2 = JOIN r1 WITH DataFlowImpl2::readSet#7ad53399#ffff_2301#join_rhs ON FIRST 2 OUTPUT Rhs.3, Lhs.1 'config', Rhs.2
                      109765 ~0%     {3} r3 = JOIN r2 WITH DataFlowImpl2::Stage1::fwdFlowConsCandSet#7ad53399#fff#reorder_0_2_1 ON FIRST 2 OUTPUT Lhs.1 'config', Lhs.2, Rhs.2 'c'
                      0      ~0%     {3} r4 = r3 AND NOT DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff#prev(Lhs.2 'c', Lhs.0 'config')
                      0      ~0%     {3} r5 = SCAN r4 OUTPUT In.1, In.0 'config', In.2 'c'
                      0      ~0%     {2} r6 = JOIN r5 WITH DataFlowImpl2::Stage1::fwdFlow#7ad53399#2#fff_02#join_rhs ON FIRST 2 OUTPUT Lhs.2 'c', Lhs.1 'config'
                                     return r6
```
2022-04-06 13:52:30 +02:00
Rasmus Wriedt Larsen
b99767ef52 Merge pull request #8668 from RasmusWL/use-instanceof
Python: Rewrite concepts to use `extends ... instanceof ...`
2022-04-06 12:09:12 +02:00
Rasmus Wriedt Larsen
4d2a3b38d2 Merge pull request #8511 from RasmusWL/use-query-suffix
Python: Use `Query.qll` suffix for dataflow configuration definitions
2022-04-06 11:59:29 +02:00
Rasmus Wriedt Larsen
5b96db26b3 Python: Rewrite concepts to use extends ... instanceof ...
This solved performance problems experienced in
https://github.com/github/codeql/pull/8634, and this commit+PR is to
ensure we get this change in as fast as possible.
2022-04-05 12:34:15 +02:00
Tom Hvitved
57f2a74636 Python: Implement ContentSet 2022-04-04 13:51:44 +02:00
Tom Hvitved
c4fbc618a9 Data flow: Sync files 2022-04-04 13:51:44 +02:00
Tom Hvitved
50dc3820c6 Merge pull request #8589 from hvitved/regex/speedup-concretise 2022-04-03 17:56:07 +02:00
Chris Smowton
3119885a9b Merge pull request #8638 from smowton/smowton/docs/additional-flow-step-description
Improve wording of isAdditionalFlow/TaintStep qldoc
2022-04-01 16:41:04 +01:00
Chris Smowton
28fa49dcd6 dataflow -> data-flow 2022-04-01 13:22:58 +01:00
Rasmus Wriedt Larsen
ba011fb13f Merge pull request #8601 from zbazztian/recognize-flask-named-body-param
Python: Flask: Identify body contents passed via named response parameter in invocations of Response constructor
2022-04-01 14:19:28 +02:00
Erik Krogh Kristensen
ed7e1206ff rename isBeforeCode to isCommentAfterCode 2022-04-01 12:55:00 +02:00
Chris Smowton
3b0bd3bc0f Improve wording 2022-04-01 11:31:31 +01:00
Chris Smowton
99026a6071 Improve wording of isAdditionalFlow/TaintStep qldoc 2022-04-01 11:07:27 +01:00
Tom Hvitved
46d69cf544 Regex: Further tweaks to concretise computations 2022-03-31 12:52:43 +02:00
Tom Hvitved
5181544790 Sync shared files 2022-03-31 12:52:42 +02:00
Tom Hvitved
0fb28f4bc9 Sync shared files 2022-03-31 12:52:42 +02:00
Erik Krogh Kristensen
1218c4f4ed fix ql/name-casing, and drive-by QL-for-QL typo fix 2022-03-30 22:59:14 +02:00
Erik Krogh Kristensen
1847a5713b remove TODO 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
7ca6426ea5 revert the Taint stage, as it caused an alert for ql/abstract-class-import 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
7e4ab4c60b Revert "import all the frameworks that extend RegexString"
This reverts commit 84bc9042de4e876685f8f5ffdd88893383d1cfdc.

It caused ql/abstract-class-import alerts
2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
3b9335c051 nomagic on containsInScope 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
5caff81ff9 import all the frameworks that extend RegexString 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
b959705531 revert changes in MRO.qll 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
b74852ffd6 cache a bit more (again) 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
d9ced55e2c make private predicates private 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
040196f40d cache more basicblock predicates 2022-03-30 22:54:01 +02:00
Erik Krogh Kristensen
79713e0ef8 a bit more caching 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
7643aac207 revert bad nomagic 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
35c7fa58a7 joiner order fixes 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
88e896992e cache the remainder of the pointsto layer 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
79da0970cc various join order fixes 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
3e9ee887d4 fix bad mistake 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
758a5d7a85 few join order fixes 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
6eca4ba2d3 get around identical files by adding the ref() call somewhere else 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
4089788629 revert caching of some large predicates that caused the DB size to increase too much 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
0da80f90d3 rename the SSA stages to AST 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
c9e3a62953 cached stages iteration 5 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
a8f9a91e38 cached stages iteration 4 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
f68357a063 cached stages iteration 3.5 2022-03-30 22:54:00 +02:00
Erik Krogh Kristensen
37a9b41e26 cached stages iteration 3 2022-03-30 22:54:00 +02:00