Also needed to fix up `TestUtil/UnresolvedCalls.qll` after a bad merge
conflict resolution. Since all calls are now DataFlowCall, and not JUST
the ones that can be resolved, we need to put in the restriction that
the callable can also be resolved.
This commit used to make sense to have here in the ordering of commits,
but due to various rebases it no longer changes any test output..
it's still a good change though, so I'll keep it.
When resolving merge conflict after flow-summaries was merged, this is
the original commit where I introduced ParameterNodeImpl, so this is the
commit where differences in that implementation was committed...
I removed TParameterNode, since I could not see we we gain anything from
having it.
This commit is a squash of 80 other commits. While developing, things
changed majorly 2-3 times, and it just wasn't feasible to go back and
write a really nice commit history.
My apologies for this HUGE commit.
Also, later on this is where I solved merge conflicts after flow-summaries
PR was merged.
For your amusement, I've included the original commit messages below.
Python: Add proper argument/parameter positions
Python: Handle normal function calls
Python: Reduce dataflow-consistency warnings
Previously there was a lot of failures for `uniqueEnclosingCallable` and
`argHasPostUpdate`
Removing the override of `getEnclosingCallable` in ParameterNode is
probably the most controversial... although from my point of view it's a
change for the better, since we're able to provide data-flow
ParameterNodes for more of the AST parameter nodes.
Python: Adjust `dataflow/calls` test
Python: Implement `isParameterOf`/`argumentOf`/`OutNode`
This makes the tests under `dataflow/basic` work as well 👍
(initially I had these as separate commits, but it felt like it was too much noise)
Python: Accept fix for `dataflow/consistency`
Python: Changes to `coverage/argumentRoutingTest.ql`
Notice we gain a few new resolved arguments.
We loose out on stuff due to:
1. not handling `*` or `**` in either arguments/parameters (yet)
2. not handling special calls (yet)
Python: Small fix for `TestUtil/RoutingTest.qll`
Since the helper predicates do not depend on this, moved outside class.
Python: Accept changes to `dataflow/coverage/NormalDataflowTest.ql`
Most of this is due to:
- not handling any kinds of methods yet
- not handling `*` or `**`
Python: Small investigation of `test_deep_callgraph`
Python: Accept changes to `coverage/localFlow.ql`
I don't fully understand why the .expected file changed.
Since we still have the desired flow, I'm not going to worry too much
about it.
with this commit, the `dataflow/coverage` tests passes 👍
Python: Minor doc update
Python: Add staticmethod/classmethod to `dataflow/calls`
Python: Handle method calls on class instances
without trying to deal with any class inheritance, or
staticmethod/classmethod at all.
Notice that with this change, we only have a DataFlowCall for the calls
that we can actually resolve. I'm not 100% sure if we need to add a
`UnresolvedCall` subclass of `DataFlowCall` for MaD in the future, but
it should be easy to do.
I'm still unsure about the value of `classesCallGraph`, but have just
accepted the changes.
Python: Handle direct method calls `C.foo(C, arg0)`
Python: Handle `@staticmethod`
Python: Handle class method calls... but the code is shit
WIP todo
Rewrite method calls to be better
also fixed a problem with `self` being an argument to the `x.staticmethod()` call :|
Python: Add subclass tests
Python: Split `class_advanced` test
Python: Rewrite call-graph tests to be inline expectation (1/2)
This adds inline expectations, next commit will remove old annotations
code... but I thought it would be easier to review like this.
Minor fixup
Python: Add simple subclass support
Python: more precise subclass lookup
Still not 100% precise.. but it's better
New ambiguous
Python: Add test for `self.m()` and `cls.m()` calls
Python: Handle `self.m()` and `cls.m()` calls
Python: Add tests for `__init__` and `__new__`
Python: Handle class calls
Python: Fix `self` argument passing for class calls
Now field-flow tests also pass 💪 (although the crosstalk
fieldflow test changes were due to this specific commit)
I also copied much of the setup for pre/post update nodes from Ruby,
specifically having the abstract `PostUpdateNodeImpl` in DataFlowPrivate
seemed like a nice change.
Same for the setup with `TNode` definition having the specification
directly in the body, instead of a `NeedsSyntheticPostUpdateNode` class.
Python: Add new crosstalk test WIP
Maybe needs a bit of refactoring, and to see how it all behaves with points-to
Python: Add `super()` call-graph tests
Python: Refactor MethodCall char-pred
In anticipation of supporting `super(MyClass, self).foo()`, where the
`self` argument doesn't come from an AttrNode, but from the second
argument to super.
Without `pragma[inline]` the optimizer found a terrible join-order --
this won't guarantee a good join-order for the future, but for now it
was just so simple and could let me move on with life.
Python: Add basic `super()` support
I debated a little (with myself) whether I should really do
`superTracker`, but I thought "why not" and just rolled with it. I did
not confirm whether it was actually needed anywhere, that is if anyone
does `ref = super; ref().foo()` -- although I certainly doubt it's very
wide-spread.
Python: InlineCallGraphTest: Allow non-unique callable name in different files
Python: more MRO tests
Python: Add MRO approximation for `super()`
Although it's not 100% accurate, it seems to be on level with the one in
points-to.
Python: Remove some spurious targets for direct calls
removal of TODO from refactoring
remove TODOs class call support
Python: Add contrived subclass call example
Python: Remove more spurious call targets
NOTE: I initially forgot to use
`findFunctionAccordingToMroKnownStartingClass` instead of
`findFunctionAccordingToMro` for __init__ and __new__, and since I did
make that mistake myself, I wanted to add something to the test to
highlight this fact, and make it viewable by PR reviewer... this will be
fixed in the next commit.
Python: Proper fix for spurious __init__ targets
Python: Add call-graph example of class decorator
Python: Support decorated classes in new call-graph
Python: Add call-graph tests for `type(obj).meth()`
Python: support `type(obj).meth()`
Python: Add test for callable defined in function
Python: Add test for callable as argument
Current'y we don't find these with type-tracking, which is super
mysterious. I did check that we have proper flow from the arguments to
the parameters.
Python: Found problem for callable as argument :| MAJOR WIP
WIP commit
IT WORKS AGAIN (but terrible performance)
remove pragma[inline]
remove oops
Fix performance problem
I tried to optimize it even further, but I didn't end up achieving anything :|
Fix call-graph comparison
add comparison version with easy lookup
incomplete missing call-graph tests
unhandled tests
trying to replicate missing call-edge due to missing imports ... but it's hard
also seems to be problems with the inline-expectation-value that I used, seems like it has both missing/unexpected results with same value
Python: Add import-problem test
Python: Add shadowing problem
some cleanup of rewrite fix
a little more cleanup
Add consistency queries to call-graph tests
Python: Add post-update nodes for `self` in implicit `super()` uses
But we do need to discuss whether this is the right approach :O
Fix for field-flow tests
This came from more precise argument passing
Fixed results in type-tracking
Comes from better argument passing with super() and handling of
functions with decorators
fix of inline call graph tests
Fixup call annotation test
Many minor cleanups/fixes
NewNormalCall -> NormalCall
Python: Major restructuring + qldoc writing
Python: Accept changes from pre/post update node .toString changes
Python: Reduce `super` complexity !! WIP !!
Python: Only pass self-reference if in same enclosing-callable
Python: Add call-graph test with nested class
This was inspired by the ImpliesDataflow test that showed missing flow
for q_super, but at least for the call-graph, I'm not able to reproduce
this missing result :|
Python: Restrict `super()` to function defined directly on class
Python: Accept fixes to ImpliesDataflow
Python: Expand field-flow crosstalk tests
- "Normal" instead of "NonSpecial"
- "NonLibrary" instead of "2"
I could not find a good replacement for "NonLibrary", nor for "Source",
but I added QLDocs in a few places to help the reading.
- add `getCallable`, `getArg` and `getNode`
- these are `none` for summary calls
- revert "external" uses (they had been changed to `DataFlowSourceCall`)
- still identifying summarized callables by name.
I think ther shoudl perhaps be a `getAUse` next to `getACall`.
- also fix tests, adding a standard taint configuration
Two classes have been inserted into the hierarchies:
- `NonLibraryDataFlowCallable` with a method `getACall2`.
This method implements "get a call, not considering flow summaries".
For `NonLibraryDataFlowCallable`s, `getACall` will defer to `getACall2`.
While you could have a synthesised call to such a callable,
it would not correspond to a `CallNode`.
- `NonLibraryDataFlowSourceCall` with methods
`getArg2` and `getCallable2`. These also refer to a call graph that
does not consider flow summaries.
`getArg2` is used to synthesise pre-update nodes for arguments.
`getCallable2` is used in `connects` to compute argument passing.
This is used to define data flow nodes for overflow arguments.
`getACall2` ensures that `LibraryCallableValue::getACall` is not called
when the charpred of `FunctionCall` is evaluated.
- Branch predicates are made simple. In particular, they do not try to detect library calls.
- All branches based on `CallNode`s are gathered into one.
- That branch has been given a class `NonSpecialCall`, which is the new parent of call classes based on `CallNode`s. (Those classes now have more involved charpreds.)
- A new such class, 'LambdaCall` has been split out from `FunctionCall` to allow the latter to replace its
general `CallNode` field with a specific `FunctionValue` one.
- `NonSpecialCall` is not an abstract class, but it has some abstract overrides. Therefor, it is not
considered a resolved call in the test `UnresolvedCalls.qll`.
I had to rewrite the SINK1-SINK7 definitions, since this new requirement
complained that we had to add this `MISSING: flow` annotation :D
Doing this implementation also revealed that there was a bug, since I
did not compare files when checking for these `MISSING:` annotations. So
fixed that up in the implementation for inline taint tests as well.
(extra whitespace in argumentPassing.py to avoid changing line numbers
for other tests)
I went with NormalDataflowTest to signify that if you don't know what
you're looking for, this is probably the one. I did not want to just
call it DataflowTest, since that becomes a big vague when there are also
`FlowTest.qll` and `MaximalFlowTest.qll` -- I'm open to renaming this
though 👍
Before, results from `dca` would look something like
## + py/meta/alerts/remote-flow-sources-reach
- django/django@c2250cf_cb8f: tests/messages_tests/urls.py:38:16:38:48
reachable with taint-tracking from RemoteFlowSource
- django/django@c2250cf_cb8f: tests/messages_tests/urls.py:38:9:38:12
reachable with taint-tracking from RemoteFlowSource
now it should make it easier to spot _what_ it is that actually changed,
since we pretty-print the node.
