Commit Graph

2111 Commits

Author SHA1 Message Date
yoff
17155b64f5 Merge pull request #4514 from tausbn/python-add-module-boundary-flow-steps
Python: Add module boundary flow steps
2020-10-20 14:36:10 +02:00
Taus Brock-Nannestad
a21c29507c Python: Fix false negative
I'm slightly suspicious of this fix -- it seems to work, but it makes
me wonder if we're potentially missing other kinds of flow, by not
handling other kinds of definitions.

Also, I feel like this should really be attached to an appropriate
post-update node of the given argument. As it is written now, the flow
will go from the argument _before_ the call, which obviously misses a
step if the argument is modified by the call. In practice, I would
expect this to be rather rare.
2020-10-20 13:16:54 +02:00
Taus Brock-Nannestad
860cafed4d Python: Mark failing test as false negative 2020-10-20 13:11:06 +02:00
Rasmus Wriedt Larsen
045a6c3cb5 Python: Add test for tricky module member for type-tracking
Local testing shows that the `getDefinition` result for this is a `SSA filter definition`,
and not an `AssignmentDefinition`.
2020-10-20 12:20:35 +02:00
Taus
f5ec548e68 Python: Fix typo in QLDoc
Co-authored-by: yoff <lerchedahl@gmail.com>
2020-10-19 23:51:38 +02:00
Taus Brock-Nannestad
7755993dd3 Python: Add jump steps for module attribute reads.
This is the quick-and-dirty solution, as discussed.

An even quicker-and-dirtier solution would have used
`ModuleValue::attr` and take the `getOrigin` of that as the source of
the jump step. However, this turns out to be a bad choice, since
`attr` might fail to have a value for the given attribute (for a
variety of reasons). Thus, we instead appeal to a helper predicate
that keeps track of which names are defined by which right-hand-sides
in a given module. (Observe that type tracking works correctly for `x`
in `mymodule.py`, even though `x` is never assigned a value in the
eyes of the Value API.)

This means that points-to is only used to actually figure out if the
object we're looking an attribute up on is a module or not. This is
the next thing to replace in order to eliminate the dependence on
points-to, but this will require some care to ensure that all module
lookups are handled correctly.

Only two test files needed to be changed for the tests to pass. The
first was the fixed false negative in the type tracker, and the other
was a bunch of missing flow in the regression test. I have manually
removed the `# Flow not found` annotations to make them consistent
with the output. Pay particular attention to the annotation on line
117 -- I believe it was misplaced and should have been on line 106
instead (where, indeed, we now have flow where none appeared before).
2020-10-19 19:13:32 +02:00
Anders Schack-Mulligen
b352605d12 Dataflow: Code review fixes. 2020-10-16 13:45:51 +02:00
Anders Schack-Mulligen
664f04020f Revert "Dataflow: Count callables instead of nodes for fieldFlowBranchLimit."
This reverts commit 1501a40de8.
2020-10-16 12:51:50 +02:00
Anders Schack-Mulligen
1501a40de8 Dataflow: Count callables instead of nodes for fieldFlowBranchLimit. 2020-10-16 12:51:17 +02:00
Anders Schack-Mulligen
6aae51fa4f Dataflow: Sync. 2020-10-16 12:51:17 +02:00
Taus Brock-Nannestad
60fcb5e7d5 Python: Add PEP-328 test example
Based on https://www.python.org/dev/peps/pep-0328/#guido-s-decision

Original "code" is in the Public Domain.
2020-10-16 12:03:43 +02:00
Tom Hvitved
27fc610c0d Python: Update expected test output 2020-10-16 09:09:06 +02:00
Tom Hvitved
5f01fda1ef Data flow: Sync files 2020-10-16 09:05:02 +02:00
Anders Schack-Mulligen
94f110f739 Sync. 2020-10-16 09:05:01 +02:00
Tom Hvitved
d608138c0c Data flow: Sync files 2020-10-16 09:03:13 +02:00
Rasmus Wriedt Larsen
5142bfaf01 Merge pull request #4453 from yoff/python-port-unsafe-deserialization
Python: port unsafe deserialization
2020-10-15 17:26:31 +02:00
Rasmus Wriedt Larsen
58baec5b06 Merge pull request #4364 from yoff/SharedDataflow_ArgumentPassing
Python: Shared dataflow, argument passing
2020-10-15 17:10:59 +02:00
Rasmus Lerchedahl Petersen
89f5352324 Python: fix QL format 2020-10-15 16:41:41 +02:00
Rasmus Lerchedahl Petersen
ef32488596 Merge branch 'main' of github.com:github/codeql into python-port-unsafe-deserialization 2020-10-15 15:45:35 +02:00
CodeQL CI
ab7d28b3fb Merge pull request #4482 from RasmusWL/promote-script
Approved by tausbn
2020-10-15 06:15:55 -07:00
Rasmus Wriedt Larsen
43cee8567c Python: Add script to promote experimental security queries 2020-10-15 13:25:01 +02:00
Rasmus Lerchedahl Petersen
cc7d32c27c Merge branch 'python-port-unsafe-deserialization' of github.com:yoff/codeql into python-port-unsafe-deserialization 2020-10-15 13:01:38 +02:00
Rasmus Lerchedahl Petersen
172e058438 Python: unsafe -> mayExecuteInput 2020-10-15 12:56:29 +02:00
Rasmus Lerchedahl Petersen
00566f0eee Python: Extend DataFlow::CfgNode when appropriate 2020-10-15 12:40:16 +02:00
yoff
c36ad7dd9b Apply suggestions from code review
Co-authored-by: Taus <tausbn@github.com>
2020-10-15 12:35:21 +02:00
Rasmus Lerchedahl Petersen
9c8e968cba Python: Fix bad merge 2020-10-15 11:47:34 +02:00
Taus
c8b93148a2 Merge pull request #4424 from RasmusWL/python-model-python2-specific-command-execution
Python: model Python 2 specific command execution
2020-10-15 10:52:43 +02:00
Rasmus Wriedt Larsen
c5810d623b Merge pull request #4474 from tausbn/python-fix-tostring-divergence
Python: Fix divergence in tuple/subscripted type `toString`
2020-10-15 10:29:33 +02:00
Rasmus Wriedt Larsen
ce967e1249 Merge branch 'main' into python-model-python2-specific-command-execution 2020-10-15 10:00:02 +02:00
Rasmus Lerchedahl Petersen
0766eef49b Merge branch 'main' of github.com:github/codeql into SharedDataflow_ArgumentPassing 2020-10-15 09:49:21 +02:00
Rasmus Lerchedahl Petersen
d2b90662a3 Python: implement ToString on mappings 2020-10-14 17:31:13 +02:00
Taus
466c22f4a8 Merge pull request #4435 from RasmusWL/python-port-code-injection
Python: port code injection query
2020-10-14 16:41:42 +02:00
Rasmus Lerchedahl Petersen
6a3aed337f Python self -> range 2020-10-14 16:35:43 +02:00
Rasmus Lerchedahl Petersen
352418cb5d Python: track safe loaders 2020-10-14 16:33:55 +02:00
Taus Brock-Nannestad
f8190feef2 Python: Fix divergence in tuple/subscripted type toString
A slightly more complicated version of the situation in
https://github.com/github/codeql/pull/2507 could cause the `toString`
calculation to diverge. Although the previous PR took tuples nested
inside tuples into account (and subscripted types cannot be nested
inside each other in our modelling), it did not account for having
this nesting be interleaved, and this is what caused the divergence.

I have not done the usual "test case first to show the problem
exists", since this would also diverge and take forever to fail. The
instance observed in `scipy` was likely caused by something akin to

```python
x = ()
while True:
    x = x[(x,)]
```

Finally, to prevent this from happening with other types, I went
through and checked each instance where the string representation of
an `ObjectInternal` might potentially contain a reference to
itself (and thus explode). I encapsulated this in a
`bounded_toString` helper predicate, and used this in all the cases
where I was able to determine that the above _could_ happen.
2020-10-14 16:13:03 +02:00
yoff
5f6f85c998 Merge pull request #4465 from tausbn/python-remove-essa-flow
Python: Remove flow between ESSA variables
2020-10-14 15:37:39 +02:00
Rasmus Lerchedahl Petersen
b8cba381cf Merge branch 'main' of github.com:github/codeql into python-port-unsafe-deserialization 2020-10-14 15:01:30 +02:00
Rasmus Lerchedahl Petersen
3a281a1bd6 Python: Adjust comments and tests 2020-10-14 14:40:11 +02:00
Rasmus Wriedt Larsen
5db4f906d0 Merge branch 'main' into python-port-code-injection 2020-10-14 14:22:02 +02:00
yoff
ffe79f688d Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-10-14 14:08:16 +02:00
Taus
92ccb795fd Merge pull request #4415 from RasmusWL/python-flask-routed-parameter
Python: Add support for routed parameters in flask
2020-10-14 13:29:51 +02:00
Rasmus Wriedt Larsen
1fde477a8f Python: Refactor argument matching 2020-10-14 13:22:35 +02:00
Rasmus Wriedt Larsen
680a6eb2a6 Python: Refactor argument matching (more) 2020-10-14 13:21:04 +02:00
Rasmus Wriedt Larsen
61ecec7d17 Merge pull request #4467 from tausbn/python-fix-import-type-tracking
Python: Fix unwanted module type tracking
2020-10-14 13:08:57 +02:00
Rasmus Lerchedahl Petersen
dc7e7890f0 Python: Clearer naming and comments (I hope) 2020-10-14 12:03:05 +02:00
Taus Brock-Nannestad
f3c07e3849 Python: Fix up import helper tests 2020-10-14 11:58:14 +02:00
Rasmus Wriedt Larsen
b0cfa1d92d Python: Make "..Call" modeling classes extend DataFlow::CfgNode 2020-10-14 10:53:18 +02:00
Rasmus Wriedt Larsen
bfa5d18476 Python: Use new importNode 2020-10-14 10:49:38 +02:00
Rasmus Wriedt Larsen
7d600e4e8e Merge branch 'main' into python-port-code-injection 2020-10-14 10:48:38 +02:00
Rasmus Wriedt Larsen
4d9d2155fc Python: Make "..Call" modeling classes extend DataFlow::CfgNode 2020-10-14 10:44:58 +02:00