Commit Graph

2989 Commits

Author SHA1 Message Date
Erik Krogh Kristensen
b513033e0f Merge pull request #7021 from erik-krogh/cwe326
JS: Add insufficient key size query
2021-11-11 12:17:04 +01:00
Anders Schack-Mulligen
7ffd9b4f9e Dataflow: Include read/store steps when finding non-hidden return. 2021-11-11 11:26:21 +01:00
Anders Schack-Mulligen
6d9fb3ca43 Dataflow: Sync. 2021-11-10 15:11:13 +01:00
yoff
d23a920ed4 Merge branch 'main' into python/model-aiomysql 2021-11-10 14:32:36 +01:00
Rasmus Lerchedahl Petersen
57e7bfbdba Python: model aiomysql 2021-11-10 14:29:39 +01:00
Rasmus Wriedt Larsen
de926dc2a1 Merge pull request #7085 from yoff/python/model-aiopg
Python: model aiopg
2021-11-10 13:10:30 +01:00
Rasmus Lerchedahl Petersen
92a7114b72 Python: Add API references 2021-11-10 11:06:58 +01:00
Taus
33135e909a Python: Add magic to named_argument_transfer
This predicate was materialised as a _big_, _cached_ relation:

```
(169s) Tuple counts for PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs/4@38ce07 after 53.4s:
25212     ~4%     {3} r1 = SCAN Function::Function::getArgByName_dispred#fff OUTPUT In.1, In.0 'arg1', In.2 'arg2'
159751200 ~0%     {4} r2 = JOIN r1 WITH Flow::CallNode::getArgByName_dispred#fff_102#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Lhs.1 'arg1', Lhs.2 'arg2', Rhs.2 'arg3'
                  return r2
```

... However it's only used in a single place (where it is immediately
joined with the points-to relation to relate the caller and argument),
none of these joins were ever larger than 2000 tuples. This made it
pretty clear that we could gain something by pushing in that points-to
join as a bit of manual magic.

However, doing so didn't actually fix anything, since the join-orderer
then decided to join `func.getArgByName(name)` with
`call.getArgByName(name)` on `name` as the first thing (which caused a
join of the same size as above).

Unbinding didn't work, since `name` would then be an unbound `string`,
so instead I factored out relating the function, parameter, and name
thereof into its own predicate. (I could also have done this with the
call, but I would expect there to be more calls than function
definitions in general.)

Overall, this resulted in going from

```
(709s)
Definitions.ql-7:PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs ......... 53.5s
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ............................... 35.3s (456 evaluations with max 136ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#0508e8)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf .................................. 27s (100 evaluations with max 12.8s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#fc1f7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ........................................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#0508eg)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff ................................ 13.4s (505 evaluations with max 50ms in Constants::ConstantObjectInternal::attribute#ffff/4@i153#0508e5)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ....................... 12.4s (99 evaluations with max 11.8s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#fc1f
7z)
...
```

to

```
(668s)
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ................... 35.4s (456 evaluations with max 140ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#bf4328)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf ...................... 27.4s (100 evaluations with max 13.3s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#679d7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ............................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#bf432g)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff .................... 14.4s (505 evaluations with max 51ms in Constants::ConstantObjectInternal::attribute#ffff/4@i140#bf4325)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ........... 12.3s (99 evaluations with max 11.7s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#679d
7z)
...
```
2021-11-09 21:39:32 +00:00
Taus
e2f79d8516 Python: Fix several bad getScope joins
It seems the optimiser has started getting the wrong end of the stick
whenever we write `foo.getScope() = bar.getScope()` for some expressions
`foo` and `bar`.

This lead to things like

```
(196s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@5ab278 after 2m33s:
2952757013 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
495693     ~0%     {2} r2 = JOIN r1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
453589     ~0%     {2} r3 = JOIN r2 WITH Definitions::ModuleVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
                   return r3
```

and

```
(315s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@a39328 after 1m57s:
...
1785275    ~3%       {2} r24 = Definitions::ModuleVariable::global_variable_callnode#ff#shared UNION Definitions::SsaSourceVariable::getAUse_dispred#ff#shared
3008614987 ~0%       {2} r25 = JOIN r24 WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
127        ~1%       {2} r26 = JOIN r25 WITH Definitions::NonLocalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
127        ~1%       {2} r27 = JOIN r26 WITH Variables::LocalVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
...
```

(Note the timings: 2m33s and 1m57s.)

Now we have the much more reasonable

```
(38s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@c53031 after 42ms:
453589 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Definitions::ModuleVariable::scope_as_global_variable#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
               return r1
```

and

```
(46s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@4b19de after 375ms:
...
```
2021-11-09 20:54:41 +00:00
Rasmus Wriedt Larsen
1e31416049 Merge pull request #7031 from yoff/python/taint-through-with
Python: Taint through `async with`
2021-11-09 14:08:07 +01:00
Rasmus Lerchedahl Petersen
a58c47b07b Python: model aiopg.sa 2021-11-09 12:49:57 +01:00
Rasmus Lerchedahl Petersen
cd332a75fc Python: model aiopg 2021-11-09 12:32:21 +01:00
Erik Krogh Kristensen
8727060ca7 add comment about modes of operation
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-09 11:15:12 +01:00
Rasmus Wriedt Larsen
9e2bc41648 Python: Improve hashlib.new modeling
By using a backwards type-tracker to find possible hashing algorithm
names.
2021-11-04 15:36:32 +01:00
Erik Krogh Kristensen
a19627c72f optionally ignore everything after a dash 2021-11-04 13:19:44 +01:00
Erik Krogh Kristensen
02f500b9c2 Merge branch 'main' into htmlReg 2021-11-04 12:58:42 +01:00
Erik Krogh Kristensen
523c15cd72 don't include mode-of-operation into the algorithm names 2021-11-03 14:54:50 +01:00
Mathias Vorreiter Pedersen
4a2894a707 Merge pull request #7025 from MathiasVP/nomagic-parameterCand
Dataflow: Replace a 'noinline' pragma with a 'nomagic' pragma
2021-11-02 20:40:44 +00:00
Rasmus Wriedt Larsen
8cd9fdebf9 Python: Model flask_admin 2021-11-02 15:43:13 +01:00
Rasmus Lerchedahl Petersen
768932d7b3 Python: Add tainttracking step that was removed
when the correpsonding datadlow step was removed.
2021-11-02 15:01:47 +01:00
Erik Krogh Kristensen
5975e19f53 sync identical files 2021-11-02 14:45:33 +01:00
yoff
97625d7c2c Merge pull request #7023 from RasmusWL/toml
Python: Add modeling of `toml`
2021-11-02 14:42:06 +01:00
Rasmus Wriedt Larsen
cb6bcada4c Merge branch 'main' into django-rest-framework 2021-11-02 14:33:16 +01:00
yoff
0240631510 Merge pull request #6782 from RasmusWL/fastapi
Python: Model FastAPI
2021-11-02 14:16:12 +01:00
Rasmus Wriedt Larsen
c52e453342 Python: Minor rewrite 2021-11-02 13:37:50 +01:00
Anders Schack-Mulligen
7d0152f3c0 Merge pull request #6932 from aschackmull/dataflow/flow-features
Dataflow: Add support for call context restrictions on sources/sinks.
2021-11-02 13:24:17 +01:00
Mathias Vorreiter Pedersen
6f4107ff23 Dataflow: Replace a 'noinline' pragma with a 'nomagic' pragma. 2021-11-02 11:37:40 +00:00
Rasmus Wriedt Larsen
8ee804a8c2 Python: Add toml modeling 2021-11-02 11:57:15 +01:00
Tom Hvitved
302373d154 Merge pull request #6858 from hvitved/python/type-tracker-changes
Python: Type tracker changes
2021-11-02 11:47:01 +01:00
CodeQL CI
d5e2026a26 Merge pull request #6934 from erik-krogh/more-instanceof
Approved by MathiasVP, esbena, yoff
2021-11-02 03:46:23 -07:00
CodeQL CI
5d62aa5b29 Merge pull request #6994 from erik-krogh/redundant-cast
Approved by RasmusWL, aschackmull, esbena, geoffw0, hvitved, nickrolfe
2021-11-02 03:45:48 -07:00
Tom Hvitved
1e64893742 Update python/ql/lib/semmle/python/dataflow/new/internal/TypeTracker.qll
Co-authored-by: Taus <tausbn@github.com>
2021-11-02 11:16:32 +01:00
Tom Hvitved
660398aa78 Python: Introduce TypeBackTracker::getACompatibleTypeTracker() 2021-11-02 11:16:32 +01:00
Tom Hvitved
73fd66cfed Python: Cache TypeBackTracker::prepend 2021-11-02 11:16:32 +01:00
Rasmus Wriedt Larsen
83389be8e2 Python: Add some missing QLDocs 2021-11-02 11:02:51 +01:00
Rasmus Wriedt Larsen
a7e4e5ef83 Python: Add rest_framework Response modeling 2021-11-02 10:55:44 +01:00
Rasmus Wriedt Larsen
13815fe728 Python: Model known APIView subclasses
Added internal helper `.qll` file as well
2021-11-02 10:55:44 +01:00
Rasmus Wriedt Larsen
62d30630aa Python: Add rest_framework Request taint modeling 2021-11-02 10:55:44 +01:00
Rasmus Wriedt Larsen
5d77e62f3a Python: Add basic rest_framework Request modeling 2021-11-02 10:55:44 +01:00
Rasmus Wriedt Larsen
57e13c6066 Python: rest_framework.decorators.api_view handling
Had to expose even more things, and had to make the `DjangoRouteHandler`
modeling more flexible so I could extend the char-pred in a different
file.
2021-11-02 10:55:44 +01:00
Rasmus Wriedt Larsen
222db37c0d Python: Add initial rest_framework modeling
I had to make the Django and PrivateDjango modeling non-private :O
2021-11-02 10:55:44 +01:00
Rasmus Wriedt Larsen
b7b9120724 Python: Better handling of Pydantic models 2021-11-02 10:29:17 +01:00
Rasmus Wriedt Larsen
17da28118a Python: Small refactor to use extends .. instanceof 2021-11-02 10:06:11 +01:00
Erik Krogh Kristensen
0897b004eb revert removal of redundant inline casts in some python files 2021-10-29 14:40:27 +02:00
Erik Krogh Kristensen
d36c66cfca remove redundant inline casts in arguments where the type is inferred by the call target 2021-10-29 14:37:56 +02:00
yoff
1c78c792ff Merge pull request #6991 from RasmusWL/flask-blueprints
Python: Support `flask.blueprints.Blueprint`
2021-10-29 14:06:43 +02:00
Rasmus Wriedt Larsen
7e7c363e43 Python: Apply suggestions from code review
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-10-29 13:59:36 +02:00
Rasmus Wriedt Larsen
85f00fda19 Merge pull request #6776 from yoff/python/model-asyncpg
Python: Model `asyncpg`
2021-10-29 13:54:44 +02:00
Anders Schack-Mulligen
3a1836c9f6 Merge pull request #7000 from aschackmull/dataflow/interface-refactor
Dataflow: Refactor public references to DataFlowCallable
2021-10-29 12:21:13 +02:00
Anders Schack-Mulligen
5951ae79b9 Dataflow: Add language specific predicates. 2021-10-29 11:11:35 +02:00