Commit Graph

588 Commits

Author SHA1 Message Date
yoff
140dc1a61e merge in main 2022-06-23 09:05:32 +00:00
yoff
8bf60301da python: we have hidden isParameterOf
but now allow a clear alternative
2022-06-23 08:57:50 +00:00
yoff
fe0c5d8ee5 python: make ArgumentNode publicly usable
- add `getCall`
2022-06-23 08:48:55 +00:00
yoff
cedf9ef538 python: make DataFlowCall "publicly usable"
- add `getCallable`, `getArg` and `getNode`
- these are `none` for summary calls
- revert "external" uses (they had been changed to `DataFlowSourceCall`)
2022-06-23 08:32:23 +00:00
yoff
dd69100dcd python: ParameterNode -> SourceParameterNode 2022-06-21 12:55:22 +00:00
yoff
8b9915e372 Python: Let the user help us identifying callbacks 2022-05-23 11:07:47 +00:00
yoff
2822ed9594 Merge remote-tracking branch 'upstream/main' into python-dataflow/flow-summaries-from-scratch 2022-05-16 08:10:15 +00:00
yoff
0778d90ac1 python: fix implementation of lambdaCreation
- still identifying summarized callables by name.
I think ther shoudl perhaps be a `getAUse` next to `getACall`.
- also fix tests, adding a standard taint configuration
2022-05-12 11:35:57 +00:00
Rasmus Wriedt Larsen
795adf0566 Python: Fix API::moduleImport("foo.bar") 2022-05-12 13:33:00 +02:00
yoff
da3634188d python: variaous fixes
- sync summary files
- format files
- fix compilation
2022-05-10 12:48:42 +00:00
yoff
f14ee0e794 python: Flow summaries based on type tracking
Two classes have been inserted into the hierarchies:

- `NonLibraryDataFlowCallable` with a method `getACall2`.
This method implements "get a call, not considering flow summaries".
For `NonLibraryDataFlowCallable`s, `getACall` will defer to `getACall2`.
While you could have a synthesised call to such a callable,
it would not correspond to a `CallNode`.

- `NonLibraryDataFlowSourceCall` with methods
`getArg2` and `getCallable2`. These also refer to a call graph that
does not consider flow summaries.

`getArg2` is used to synthesise pre-update nodes for arguments.

`getCallable2` is used in `connects` to compute argument passing.
This is used to define data flow nodes for overflow arguments.

`getACall2` ensures that `LibraryCallableValue::getACall` is not called
when the charpred of `FunctionCall` is evaluated.
2022-05-10 12:48:42 +00:00
Rasmus Lerchedahl Petersen
506efcf051 python: refactor TDataFlowCall
- Branch predicates are made simple. In particular, they do not try to detect library calls.
- All branches based on `CallNode`s are gathered into one.
- That branch has been given a class `NonSpecialCall`, which is the new parent of call classes based on `CallNode`s. (Those classes now have more involved charpreds.)
- A new such class, 'LambdaCall` has been split out from `FunctionCall` to allow the latter to replace its
  general `CallNode` field with a specific `FunctionValue` one.
- `NonSpecialCall` is not an abstract class, but it has some abstract overrides. Therefor, it is not
  considered a resolved call in the test `UnresolvedCalls.qll`.
2022-05-10 12:48:42 +00:00
Rasmus Lerchedahl Petersen
177dea5307 python: use new syntax for flow summaries
also convert to inline tests
2022-05-10 12:48:42 +00:00
Rasmus Lerchedahl Petersen
4024ce4777 python: some summary flows 2022-05-10 12:48:42 +00:00
Rasmus Lerchedahl Petersen
828db3a392 python: Add summary nodes
allowing more `OutNode`s (not restricting to `CallNode`s),
gives more flow in the `classesCallGraph` test
2022-05-10 12:48:42 +00:00
Rasmus Lerchedahl Petersen
80175a9af5 Python: Compiles and mostly pass tests
- add flowsummaries shared files
- register in indentical files
- fix initial non-monotonic recursions
  - add DataFlowSourceCall
  - add resolvedCall
  - add SourceParameterNode

failing tests:
- 3/library-tests/with/test.ql
2022-05-10 12:48:42 +00:00
yoff
1d44694280 Merge pull request #8732 from RasmusWL/dataflow-imports
Python: Don't re-export `python` under `DataFlow::`
2022-05-02 12:08:28 +02:00
Taus
231def026f Merge pull request #8890 from tausbn/python-add-global-attribute-writes
Python: Add support for global attribute writes
2022-05-02 12:03:41 +02:00
yoff
c67b06b1fd Update python/ql/test/experimental/dataflow/typetracking/attribute_tests.py
Co-authored-by: Taus <tausbn@github.com>
2022-05-02 11:36:58 +02:00
Taus
b4a31e572f Python: Add global attribute writes 2022-04-27 16:45:00 +00:00
Taus
f71cf2e1fc Python: Add test 2022-04-27 15:48:11 +00:00
Rasmus Wriedt Larsen
084c8eb22e Python: Don't re-export python under DataFlow:: 2022-04-20 11:42:10 +02:00
Rasmus Wriedt Larsen
5dbbd17bb2 Python: Add test to ensure we keep DataFlow imports clean
Currently we're not in a good state :(
2022-04-20 11:41:01 +02:00
Erik Krogh Kristensen
50bfc8eaa0 refactor uses of API::Node::getAUse() that should have been something else 2022-04-07 13:52:13 +02:00
yoff
5efc19c39d Merge pull request #7806 from erik-krogh/pyDef
Python: Add def nodes to API graphs
2022-03-28 08:09:14 +02:00
Taus
5a8ba6a7af Python: Fix use of singleton set 2022-03-07 18:59:49 +00:00
yoff
d953382df9 Merge pull request #7807 from RasmusWL/dataflow-improvements
Python: Dataflow improvements
2022-02-28 16:24:00 +01:00
Rasmus Wriedt Larsen
0e0f159891 Python: Use Python 3 for investigation tests
Apparently there are minor differences with `test-6-max-import-depth-2`
where under Python 2 `isfile_no_problem.py` still works as before
2022-02-28 11:33:31 +01:00
Rasmus Wriedt Larsen
01d426dc58 Python: Replace rest of from testlib import *
I think we should write our tests in a way that puts points-to in the
best condition to resolve calls. Although this specific change did not
change much, it should help set us up for success in the future 👍
2022-02-28 10:58:44 +01:00
Rasmus Wriedt Larsen
ead0b658d2 Python: Fix fieldflow tests by increasing max-import-depth 2022-02-28 10:58:44 +01:00
Rasmus Wriedt Larsen
a0d1cea490 Python: Add investigation of field-flow problem
TL;DR; we used a too low value for `--max-import-depth` :(
2022-02-28 10:58:44 +01:00
Rasmus Wriedt Larsen
f988e1f0d8 Python: Improve field-flow by removing import *
Since that apparently impacts call graph resolution with points-to :O

Also interesting that global flow was only not working for those cases
because of the tricky ifs... still need to 100% figure out how those ifs
are messing up the analysis :|
2022-02-25 16:01:08 +01:00
Rasmus Wriedt Larsen
999af15bd5 Python: Show unresolved calls for field-flow tests 2022-02-25 15:58:07 +01:00
yoff
8b926f6859 Merge pull request #7873 from RasmusWL/fix-attribute-taint
Python: Fix attribute taint
2022-02-25 15:02:24 +01:00
Rasmus Wriedt Larsen
2d0034c40d Python: Replicate global field-flow failures 2022-02-25 14:14:00 +01:00
Rasmus Wriedt Larsen
faaa63a73c Python: Ensure no cross-talk in global tests
By giving all variables unique names

I also added a comment with the function name from the normal tests, so
its' easily visible what these tests are testing
2022-02-25 13:41:51 +01:00
Rasmus Wriedt Larsen
0642610ee9 Python: Global flow works when in own file???
This is very suspicious
2022-02-25 13:36:00 +01:00
Rasmus Wriedt Larsen
d83a9ef8d3 Python: Fix global field-flow for validTest.py 2022-02-25 13:35:43 +01:00
Rasmus Wriedt Larsen
abe4d8da62 Python: Accept global field-flow inconsistencies
Yikes
2022-02-24 15:07:18 +01:00
Rasmus Wriedt Larsen
94d23f3817 Python: Also do all field-flow tests in global scope
Notice that these tests don't pass, to show how they differ in the next
commit!
2022-02-24 15:06:40 +01:00
Rasmus Wriedt Larsen
2da4b39844 Python: Add global field-flow tests
I thought it was interesting that it did not propagate flow to the uses
inside the functions :O
2022-02-24 13:15:48 +01:00
Rasmus Wriedt Larsen
d2cd77aefb Merge branch 'main' into dataflow-improvements 2022-02-21 14:49:40 +01:00
Rasmus Wriedt Larsen
9d81fd3b95 Python: Improve sanitizer/guards tests
Based on review conversation
2022-02-18 14:12:41 +01:00
Rasmus Wriedt Larsen
7aa559f4aa Python: Restore dataflow consistency queries 2022-02-18 13:47:29 +01:00
Rasmus Wriedt Larsen
c5b6fb37b7 Python: Clean up NormalDataflowTest.qll 2022-02-18 13:47:29 +01:00
Rasmus Wriedt Larsen
67ca14876a Python: Apply suggestions from code review
Co-authored-by: yoff <lerchedahl@gmail.com>
2022-02-18 13:47:07 +01:00
Rasmus Wriedt Larsen
a8edd44a3c Python: Update .expected 2022-02-08 11:12:34 +01:00
Rasmus Wriedt Larsen
b276b2d48c Python: Clean up taint steps for attributes 2022-02-07 13:12:31 +01:00
Rasmus Wriedt Larsen
59160eeb24 Python: Add test showing taint for attr store
In `x.arg = TAINTED_STRING` there is a store step to the attribute `arg`
of `x`. In our taint modeling, we allow _any_ store step with the code
below. This means that we also say there is a taint-step directly from
`TAINTED_STRING` to `x` :|

```codeql
  // construction by literal
  // TODO: Not limiting the content argument here feels like a BIG hack, but we currently get nothing for free :|
  DataFlowPrivate::storeStep(nodeFrom, _, nodeTo)
```
2022-02-07 13:12:28 +01:00
Rasmus Wriedt Larsen
b2ce0fcb72 Python: Add post-update nodes to args of unresolved calls
Besides solving the problem with `setattr`, it also solved some old
problems with json library modeling (yay).
2022-02-04 11:51:53 +01:00