Commit Graph

4612 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
e812029c03 Python: Add test for wsgiref.simple_server 2021-11-15 13:34:38 +01:00
Taus
c17560f948 Merge pull request #7096 from tausbn/python-fix-more-bad-joins
Python: Fix a bunch of performance issues
2021-11-15 12:10:27 +01:00
yoff
5beb681580 Merge pull request #7087 from RasmusWL/path-injection-fp
Python: Add interesting path-injection FP
2021-11-12 15:20:19 +01:00
yoff
9f614b1d98 Merge pull request #7016 from RasmusWL/django-rest-framework
Python: Model Django REST framework
2021-11-12 14:27:56 +01:00
Rasmus Wriedt Larsen
b11d11c0c9 Python: Add change-note 2021-11-12 14:27:01 +01:00
Rasmus Wriedt Larsen
491f72bb2a Python: Adjust generated code to be more familiar 2021-11-12 13:30:03 +01:00
Rasmus Wriedt Larsen
de69e4c645 Python: Expand on SubclassFinder implementation note 2021-11-12 13:29:03 +01:00
Rasmus Wriedt Larsen
f7b53321b9 Python: Remove copy-pasted comment 2021-11-12 13:19:20 +01:00
Taus
55ea715ce9 Merge pull request #7033 from RasmusWL/flask-admin 2021-11-12 12:18:56 +01:00
Rasmus Wriedt Larsen
860b1a5cc3 Python: Other minor QLDoc adjustment 2021-11-12 11:46:45 +01:00
Rasmus Wriedt Larsen
99081ea7e0 Python: Minor adjustment in QLDoc 2021-11-12 11:42:36 +01:00
Rasmus Wriedt Larsen
5e4b866f2b Python: Model rest_framework.exceptions.APIException 2021-11-12 11:37:54 +01:00
Rasmus Wriedt Larsen
62e58b534c Python: SubclassFinder: reorder + comment 2021-11-12 11:11:13 +01:00
Rasmus Wriedt Larsen
f48ecb1dc8 Python: Apply suggestions from code review
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-11-12 10:57:56 +01:00
Rasmus Wriedt Larsen
06cae3dac2 Merge pull request #7104 from yoff/python/model-aiomysql
Python: model aiomysql
2021-11-11 16:58:01 +01:00
Rasmus Lerchedahl Petersen
e2a2a42d59 Python: Fix api references 2021-11-11 13:20:57 +01:00
Erik Krogh Kristensen
b513033e0f Merge pull request #7021 from erik-krogh/cwe326
JS: Add insufficient key size query
2021-11-11 12:17:04 +01:00
Anders Schack-Mulligen
7ffd9b4f9e Dataflow: Include read/store steps when finding non-hidden return. 2021-11-11 11:26:21 +01:00
Anders Schack-Mulligen
6d9fb3ca43 Dataflow: Sync. 2021-11-10 15:11:13 +01:00
yoff
d23a920ed4 Merge branch 'main' into python/model-aiomysql 2021-11-10 14:32:36 +01:00
Rasmus Lerchedahl Petersen
57e7bfbdba Python: model aiomysql 2021-11-10 14:29:39 +01:00
Rasmus Lerchedahl Petersen
047cff0749 Python: test aiomysql 2021-11-10 14:24:45 +01:00
Rasmus Wriedt Larsen
de926dc2a1 Merge pull request #7085 from yoff/python/model-aiopg
Python: model aiopg
2021-11-10 13:10:30 +01:00
Rasmus Wriedt Larsen
1f90dcadf1 Python: Use backtracker for verify arg 2021-11-10 12:54:23 +01:00
Rasmus Lerchedahl Petersen
92a7114b72 Python: Add API references 2021-11-10 11:06:58 +01:00
Rasmus Lerchedahl Petersen
c6d285dd2a Python: Fix test 2021-11-10 11:06:45 +01:00
yoff
a856395d56 Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-10 10:51:40 +01:00
Rasmus Wriedt Larsen
77099fe9d0 Python: Always use @id py/
The two queries in CWE-020 are used for manual evaluation (is my
understanding), and the two IDE queries should work based on their tags,
and not on the query-id.
2021-11-10 10:40:17 +01:00
Taus
33135e909a Python: Add magic to named_argument_transfer
This predicate was materialised as a _big_, _cached_ relation:

```
(169s) Tuple counts for PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs/4@38ce07 after 53.4s:
25212     ~4%     {3} r1 = SCAN Function::Function::getArgByName_dispred#fff OUTPUT In.1, In.0 'arg1', In.2 'arg2'
159751200 ~0%     {4} r2 = JOIN r1 WITH Flow::CallNode::getArgByName_dispred#fff_102#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Lhs.1 'arg1', Lhs.2 'arg2', Rhs.2 'arg3'
                  return r2
```

... However it's only used in a single place (where it is immediately
joined with the points-to relation to relate the caller and argument),
none of these joins were ever larger than 2000 tuples. This made it
pretty clear that we could gain something by pushing in that points-to
join as a bit of manual magic.

However, doing so didn't actually fix anything, since the join-orderer
then decided to join `func.getArgByName(name)` with
`call.getArgByName(name)` on `name` as the first thing (which caused a
join of the same size as above).

Unbinding didn't work, since `name` would then be an unbound `string`,
so instead I factored out relating the function, parameter, and name
thereof into its own predicate. (I could also have done this with the
call, but I would expect there to be more calls than function
definitions in general.)

Overall, this resulted in going from

```
(709s)
Definitions.ql-7:PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs ......... 53.5s
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ............................... 35.3s (456 evaluations with max 136ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#0508e8)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf .................................. 27s (100 evaluations with max 12.8s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#fc1f7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ........................................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#0508eg)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff ................................ 13.4s (505 evaluations with max 50ms in Constants::ConstantObjectInternal::attribute#ffff/4@i153#0508e5)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ....................... 12.4s (99 evaluations with max 11.8s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#fc1f
7z)
...
```

to

```
(668s)
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ................... 35.4s (456 evaluations with max 140ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#bf4328)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf ...................... 27.4s (100 evaluations with max 13.3s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#679d7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ............................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#bf432g)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff .................... 14.4s (505 evaluations with max 51ms in Constants::ConstantObjectInternal::attribute#ffff/4@i140#bf4325)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ........... 12.3s (99 evaluations with max 11.7s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#679d
7z)
...
```
2021-11-09 21:39:32 +00:00
Taus
e2f79d8516 Python: Fix several bad getScope joins
It seems the optimiser has started getting the wrong end of the stick
whenever we write `foo.getScope() = bar.getScope()` for some expressions
`foo` and `bar`.

This lead to things like

```
(196s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@5ab278 after 2m33s:
2952757013 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
495693     ~0%     {2} r2 = JOIN r1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
453589     ~0%     {2} r3 = JOIN r2 WITH Definitions::ModuleVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
                   return r3
```

and

```
(315s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@a39328 after 1m57s:
...
1785275    ~3%       {2} r24 = Definitions::ModuleVariable::global_variable_callnode#ff#shared UNION Definitions::SsaSourceVariable::getAUse_dispred#ff#shared
3008614987 ~0%       {2} r25 = JOIN r24 WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
127        ~1%       {2} r26 = JOIN r25 WITH Definitions::NonLocalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
127        ~1%       {2} r27 = JOIN r26 WITH Variables::LocalVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
...
```

(Note the timings: 2m33s and 1m57s.)

Now we have the much more reasonable

```
(38s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@c53031 after 42ms:
453589 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Definitions::ModuleVariable::scope_as_global_variable#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
               return r1
```

and

```
(46s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@4b19de after 375ms:
...
```
2021-11-09 20:54:41 +00:00
Taus
24000a50e6 Python: Fix bad join in py/redundant-assignment
A minor thing, but still best avoided:
```
Tuple counts for RedundantAssignment::pyflakes_commented_line#bb/2@0477c7 after 14.2s:
160826   ~0%     {2} r1 = SCAN py_comments OUTPUT In.2, In.1
160826   ~0%     {3} r2 = JOIN r1 WITH project#Files::Location::hasLocationInfo_dispred#bfffff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Rhs.2 'result'
160826   ~3%     {3} r3 = JOIN r2 WITH files_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'result', Lhs.1, Rhs.1 'file'
46672419 ~4%     {4} r4 = JOIN r3 WITH Files::Location::getStartLine_dispred#bf_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.2 'file', Lhs.1, Lhs.0 'result'
3951     ~0%     {4} r5 = JOIN r4 WITH Files::Location::getFile_dispred#bf ON FIRST 2 OUTPUT Lhs.2, Lhs.3 'result', Lhs.1 'file', toLower(Lhs.2)
0        ~0%     {4} r6 = SELECT r5 ON In.3 contains "pyflakes"
0        ~0%     {2} r7 = SCAN r6 OUTPUT In.2 'file', In.1 'result'
                return r7
```
2021-11-09 20:31:44 +00:00
Rasmus Wriedt Larsen
985cd1ebdb Python: Port py/request-without-cert-validation to use API graphs 2021-11-09 16:37:50 +01:00
Rasmus Wriedt Larsen
59581690fd Python: Add py/request-without-cert-validation tests 2021-11-09 16:29:57 +01:00
Rasmus Wriedt Larsen
9710aeecbf Python/C#: Add CWE-1333 to redos queries
As is already done in JS and Ruby.
2021-11-09 16:10:38 +01:00
Rasmus Wriedt Larsen
f70e4fea55 Python: Add interesting path-injection FP 2021-11-09 14:53:32 +01:00
Rasmus Wriedt Larsen
1e31416049 Merge pull request #7031 from yoff/python/taint-through-with
Python: Taint through `async with`
2021-11-09 14:08:07 +01:00
Rasmus Lerchedahl Petersen
ac5a46f24f Python: split test as suggested in review 2021-11-09 13:04:52 +01:00
yoff
5f4aad40c1 Update python/ql/test/experimental/meta/InlineTaintTest.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-09 13:00:35 +01:00
Rasmus Lerchedahl Petersen
aa1541a5c3 Python: add changenote 2021-11-09 12:57:36 +01:00
Rasmus Lerchedahl Petersen
a58c47b07b Python: model aiopg.sa 2021-11-09 12:49:57 +01:00
Rasmus Lerchedahl Petersen
f53314019a Python: test aiopg.sa 2021-11-09 12:42:03 +01:00
Rasmus Lerchedahl Petersen
cd332a75fc Python: model aiopg 2021-11-09 12:32:21 +01:00
Rasmus Lerchedahl Petersen
cb8f1b4593 Python: Add tests for aiopg 2021-11-09 11:49:31 +01:00
Erik Krogh Kristensen
8727060ca7 add comment about modes of operation
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-09 11:15:12 +01:00
Rasmus Lerchedahl Petersen
3f4c2ba24e Python: Support debugging inline taint tests
The module `Conf` is created so that it can be imported
without importing the query predicates from the same file.
2021-11-08 14:08:11 +01:00
Rasmus Lerchedahl Petersen
624b794980 Python: separate taint sources in with 2021-11-04 17:06:36 +01:00
Rasmus Wriedt Larsen
9e2bc41648 Python: Improve hashlib.new modeling
By using a backwards type-tracker to find possible hashing algorithm
names.
2021-11-04 15:36:32 +01:00
Rasmus Wriedt Larsen
9e91f3a341 Python: Highlight shortcomings of hashlib.new modeling 2021-11-04 15:29:40 +01:00
Erik Krogh Kristensen
a19627c72f optionally ignore everything after a dash 2021-11-04 13:19:44 +01:00
Erik Krogh Kristensen
02f500b9c2 Merge branch 'main' into htmlReg 2021-11-04 12:58:42 +01:00