Commit Graph

510 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
3e7dc12246 Python: Port taint tests to use inline expectations
The meat of this PR is described in the new python/ql/test/experimental/meta/InlineTaintTest.qll file:

> Defines a InlineExpectationsTest for checking whether any arguments in
> `ensure_tainted` and `ensure_not_tainted` calls are tainted.
>
> Also defines query predicates to ensure that:
> - if any arguments to `ensure_not_tainted` are tainted, their annotation is marked with `SPURIOUS`.
> - if any arguments to `ensure_tainted` are not tainted, their annotation is marked with `MISSING`.
>
> The functionality of this module is tested in `ql/test/experimental/meta/inline-taint-test-demo`.
2021-04-15 18:00:33 +02:00
Rasmus Lerchedahl Petersen
3eb1813584 Python: update test expectations 2021-04-15 10:47:49 +02:00
Rasmus Wriedt Larsen
1f5e52e822 Python: Cleanup "first" type-tracking predicate to be private
Since it's exposed nicely in the version that doesn't have a
`DataFlow::TypeTracker` parameter, these should be private.

Also found one instance where I had accidentially used DataFlow::Node instead of
LocalSourceNode
2021-03-23 16:40:56 +01:00
Taus Brock-Nannestad
978200e2ad Python: Distinguish between Python 2 and 3
Also moves the filtering on `name` to before the big disjunction in
`MkModuleImport`.
2021-03-12 12:35:23 +01:00
Taus Brock-Nannestad
c7b2b719cf Python: Support builtins in API graphs 2021-03-11 23:03:18 +01:00
Taus
53711dc82f Merge pull request #5238 from RasmusWL/no-flow-default-value
Python: Highlight missing flow from default value in functions
2021-02-23 13:27:41 +01:00
Rasmus Wriedt Larsen
e160c855ad Merge pull request #5233 from yoff/python-for-tuple-iteration
Python: `for`-iteration of tuples
2021-02-22 15:28:13 +01:00
Rasmus Wriedt Larsen
5249b54a9b Python: Highlight missing flow from default value in functions
Although it is becoming non-trivial to get an overview of what tests we have and
don't have, I didn't find any that highlighted this one

I used all 3 variants of parameters, just to be sure :)
2021-02-22 14:52:51 +01:00
Rasmus Lerchedahl Petersen
d23a8ad016 Python: elide test output 2021-02-21 13:12:54 +01:00
Rasmus Lerchedahl Petersen
46faba69ff Python: Fix for-iteration of tuples 2021-02-21 12:41:16 +01:00
Rasmus Lerchedahl Petersen
0aecf33fe6 Python: test iteration through overflow parameters
These are in a tuple, so the for-step does not fire
2021-02-21 12:33:04 +01:00
Taus Brock-Nannestad
04eb0c774c Python: Use LocalSourceNode in type tracker tests
One minor change to the tests results needed: there is no longer local
flow going into the `ModuleVariableNode` for `attr_ref` in the
`moduleattr.ql` test, but I think this is reasonable.
2021-02-16 18:25:54 +01:00
Rasmus Wriedt Larsen
bf401c7498 Merge pull request #5103 from tausbn/python-port-flask-to-api-graphs
Python: Port Flask models to use API graphs
2021-02-16 15:00:46 +01:00
Rasmus Wriedt Larsen
1d25184b32 Python: Add test for type-tracking through decorators
In general, if there is _some_ decorator on a function, it might not be safe to
track content out of it (since the decorator could do anything), but in this
case, we can see what the decorator does, so we should be able to handle it (but
we don't right now).

By my understanding of how type-tracking works, if we track content through
`my_decorator`, then we would also track content to the result of
`unrelated_func()`, which I wanted to make sure our tests would catch.

I found out the core of the problem seems to come from our lack of being able to
track to the inner scope, and added an explicit test for that.
2021-02-09 13:43:10 +01:00
Taus Brock-Nannestad
7f3c6acd08 Python: Handle class attribute references in API graph
This is slightly dubious, and should really be in the currently
unimplemented "def" counterpart to the "use" bits we already have.

However, it seems to work correctly, and in the spirit of moving
things along, this seemed like the easier solution. We can always
replace the implementation with the "proper" approach at a later point.
2021-02-05 21:54:35 +01:00
Taus Brock-Nannestad
ef600575ca Python: Add API graph support for subclasses 2021-02-05 16:52:58 +01:00
yoff
7fef1a8817 Merge pull request #5069 from tausbn/python-api-graphs
Python: Add support for API graphs
2021-02-05 13:17:09 +01:00
Taus Brock-Nannestad
f6e1ea5b2a Python: Fix missing global variable source nodes
In lieu of removing the offending flow (which would likely have
consequences for a lot of other tests), I opted to simply _include_
the relevant nodes directly.
2021-02-04 18:07:13 +01:00
Taus Brock-Nannestad
2524f23a46 Python: Add more test cases
There is now a bit of redundancy in the tests, but I thought it useful
to actually include some of the cases called out explicitly in the
documentation, so as to make it easy to see that the code actually
does what we expect (in these cases, anyway).
2021-02-04 18:05:33 +01:00
Taus
634041d2d7 Merge pull request #5047 from yoff/python-dataflow-unpacking-unifying-experiments
Python: dataflow, unify iterated unpacking
2021-02-04 12:57:43 +01:00
Taus
4627799c93 Python: Fix more typos
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-02-04 12:41:17 +01:00
Taus
e5ec1e105c Python: Fix typos in test files
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-02-04 12:18:07 +01:00
Rasmus Wriedt Larsen
ac0f2d37db Python: Fix small typo in test-output
Spotted by yoff in https://github.com/github/codeql/pull/5069#discussion_r570063207
2021-02-04 12:11:20 +01:00
Taus Brock-Nannestad
5974af661e Python: Update test file
Makes the `a.b.c.d` test more sensible.

Also adds a test that shows a case where we're currently _not_ getting
the right flow.
2021-02-03 22:43:21 +01:00
Rasmus Lerchedahl Petersen
a7ca065411 Python: Fix ForTarget 2021-02-03 22:14:15 +01:00
Taus Brock-Nannestad
e4c3544a3f Python: Add support for from foo.bar import baz
This turned out to be fairly simple. Given an import such as
```python
from foo.bar.baz import quux
```
we create an API-graph node for each valid dotted prefix of
`foo.bar.baz`, i.e. `foo`, `foo.bar`, and `foo.bar.baz`. For these, we
then insert nodes in the API graph, such that `foo` steps to `foo.bar`
along an edge labeled `bar`, etc.

Finally, we only allow undotted names to hang off of the API-graph
root. Thus, `foo` will have a `moduleImport` edge off of the root, and
a `getMember` edge for `bar` (which in turn has a `getMember` edge for
`baz`).

Relative imports are explicitly ignored.

Finally, this commit also adds inline tests for a variety of ways of
importing modules, including a copy of the "import-helper" tests (with
a few modifications to allow a single annotation per line, as these
get rather long quickly!).
2021-02-02 21:59:33 +01:00
Rasmus Lerchedahl Petersen
27fd46b855 Python: Update test expectation 2021-02-01 08:55:20 +01:00
Rasmus Lerchedahl Petersen
6730396ad6 Python: Remove tests from non-test directory 2021-02-01 08:52:00 +01:00
Rasmus Lerchedahl Petersen
f6fa1276a6 Python: Add consistency checks
to all data-flow test floders
2021-01-29 21:28:43 +01:00
Rasmus Lerchedahl Petersen
05a138694d Python: Fix crashing test 2021-01-29 21:12:44 +01:00
Rasmus Lerchedahl Petersen
182d435dc6 Python: Replace comprehension read-step by for
read-step. Add a version targetting sequence nodes.
2021-01-29 17:31:59 +01:00
Taus
cb195a0dc4 Merge pull request #4752 from yoff/python-dataflow-unpacking-assignment
Python: Dataflow, unpacking assignment
2021-01-29 14:15:28 +01:00
Taus
be5b7bb4c4 Merge pull request #5022 from yoff/python-split-lambdas
Python: Callable for lambdas
2021-01-29 14:12:26 +01:00
Rasmus Wriedt Larsen
5646af56dd Python: Fix too many results from DataFlow::importNode 2021-01-27 19:11:55 +01:00
Rasmus Wriedt Larsen
0d42e546a0 Python: Add deep import chain to import-helper tests 2021-01-27 19:09:09 +01:00
Rasmus Wriedt Larsen
44bb41e84b Python: Add extra type-tracking test for "long" import chain
While trying to debug an other problem related to full import of django view, I
stumbled upon this oddity. (yikes)
2021-01-27 19:06:51 +01:00
Rasmus Lerchedahl Petersen
d3e0e84c37 Python: Separate callable for lambdas
Since lambdas are split, but their children are not,
we use the Function as the callable.
2021-01-26 21:17:59 +01:00
Rasmus Lerchedahl Petersen
4526a1dd2f Python: test for split lambda 2021-01-26 21:17:59 +01:00
Rasmus Wriedt Larsen
902bade5ae Merge pull request #5015 from yoff/python-add-missing-postupdate-nodes
Python: add missing postupdate nodes
2021-01-26 14:39:29 +01:00
Taus
4c0f54f5d3 Merge pull request #5007 from yoff/python-disregard-comp-args 2021-01-26 12:53:33 +01:00
Rasmus Lerchedahl Petersen
2c58643fd1 Python: Test for parameters without nodes. 2021-01-26 11:28:31 +01:00
Rasmus Lerchedahl Petersen
7b9ca7171a Python: update test expectations 2021-01-26 09:47:48 +01:00
Rasmus Lerchedahl Petersen
dacc21d0b5 Python: update test expectation 2021-01-26 09:45:41 +01:00
Rasmus Wriedt Larsen
a8186be2fa Python: Add test of type-tracking self in methods 2021-01-25 17:20:11 +01:00
Rasmus Lerchedahl Petersen
ad39bfb2ff Python: Add postupdate nodes for subscripts.
This drops reverse read inconsistencies on saltstack from 14909 to 1353.
2021-01-25 17:01:25 +01:00
Rasmus Lerchedahl Petersen
361bee851a Python: Tests inspired by reverse read check 2021-01-25 17:01:25 +01:00
Rasmus Lerchedahl Petersen
4ff2c6d85a Python: fix test expectation
probably a copy-paste error..
2021-01-25 16:49:51 +01:00
Rasmus Lerchedahl Petersen
89e56707c3 Python: Omit all unresolved parameter nodes.
Drops the results further to 139.
2021-01-24 16:16:07 +01:00
Rasmus Lerchedahl Petersen
0d20a4cb4a Python: Simplify modelling 2021-01-22 19:40:34 +01:00
Rasmus Lerchedahl Petersen
f948ef8f27 Merge branch 'main' of github.com:github/codeql into python-dataflow-unpacking-assignment 2021-01-22 16:26:48 +01:00