mirror of
https://github.com/github/codeql.git
synced 2026-05-05 05:35:13 +02:00
Python: Implement call-graph with type-trackers
This commit is a squash of 80 other commits. While developing, things changed majorly 2-3 times, and it just wasn't feasible to go back and write a really nice commit history. My apologies for this HUGE commit. Also, later on this is where I solved merge conflicts after flow-summaries PR was merged. For your amusement, I've included the original commit messages below. Python: Add proper argument/parameter positions Python: Handle normal function calls Python: Reduce dataflow-consistency warnings Previously there was a lot of failures for `uniqueEnclosingCallable` and `argHasPostUpdate` Removing the override of `getEnclosingCallable` in ParameterNode is probably the most controversial... although from my point of view it's a change for the better, since we're able to provide data-flow ParameterNodes for more of the AST parameter nodes. Python: Adjust `dataflow/calls` test Python: Implement `isParameterOf`/`argumentOf`/`OutNode` This makes the tests under `dataflow/basic` work as well 👍 (initially I had these as separate commits, but it felt like it was too much noise) Python: Accept fix for `dataflow/consistency` Python: Changes to `coverage/argumentRoutingTest.ql` Notice we gain a few new resolved arguments. We loose out on stuff due to: 1. not handling `*` or `**` in either arguments/parameters (yet) 2. not handling special calls (yet) Python: Small fix for `TestUtil/RoutingTest.qll` Since the helper predicates do not depend on this, moved outside class. Python: Accept changes to `dataflow/coverage/NormalDataflowTest.ql` Most of this is due to: - not handling any kinds of methods yet - not handling `*` or `**` Python: Small investigation of `test_deep_callgraph` Python: Accept changes to `coverage/localFlow.ql` I don't fully understand why the .expected file changed. Since we still have the desired flow, I'm not going to worry too much about it. with this commit, the `dataflow/coverage` tests passes 👍 Python: Minor doc update Python: Add staticmethod/classmethod to `dataflow/calls` Python: Handle method calls on class instances without trying to deal with any class inheritance, or staticmethod/classmethod at all. Notice that with this change, we only have a DataFlowCall for the calls that we can actually resolve. I'm not 100% sure if we need to add a `UnresolvedCall` subclass of `DataFlowCall` for MaD in the future, but it should be easy to do. I'm still unsure about the value of `classesCallGraph`, but have just accepted the changes. Python: Handle direct method calls `C.foo(C, arg0)` Python: Handle `@staticmethod` Python: Handle class method calls... but the code is shit WIP todo Rewrite method calls to be better also fixed a problem with `self` being an argument to the `x.staticmethod()` call :| Python: Add subclass tests Python: Split `class_advanced` test Python: Rewrite call-graph tests to be inline expectation (1/2) This adds inline expectations, next commit will remove old annotations code... but I thought it would be easier to review like this. Minor fixup Python: Add simple subclass support Python: more precise subclass lookup Still not 100% precise.. but it's better New ambiguous Python: Add test for `self.m()` and `cls.m()` calls Python: Handle `self.m()` and `cls.m()` calls Python: Add tests for `__init__` and `__new__` Python: Handle class calls Python: Fix `self` argument passing for class calls Now field-flow tests also pass 💪 (although the crosstalk fieldflow test changes were due to this specific commit) I also copied much of the setup for pre/post update nodes from Ruby, specifically having the abstract `PostUpdateNodeImpl` in DataFlowPrivate seemed like a nice change. Same for the setup with `TNode` definition having the specification directly in the body, instead of a `NeedsSyntheticPostUpdateNode` class. Python: Add new crosstalk test WIP Maybe needs a bit of refactoring, and to see how it all behaves with points-to Python: Add `super()` call-graph tests Python: Refactor MethodCall char-pred In anticipation of supporting `super(MyClass, self).foo()`, where the `self` argument doesn't come from an AttrNode, but from the second argument to super. Without `pragma[inline]` the optimizer found a terrible join-order -- this won't guarantee a good join-order for the future, but for now it was just so simple and could let me move on with life. Python: Add basic `super()` support I debated a little (with myself) whether I should really do `superTracker`, but I thought "why not" and just rolled with it. I did not confirm whether it was actually needed anywhere, that is if anyone does `ref = super; ref().foo()` -- although I certainly doubt it's very wide-spread. Python: InlineCallGraphTest: Allow non-unique callable name in different files Python: more MRO tests Python: Add MRO approximation for `super()` Although it's not 100% accurate, it seems to be on level with the one in points-to. Python: Remove some spurious targets for direct calls removal of TODO from refactoring remove TODOs class call support Python: Add contrived subclass call example Python: Remove more spurious call targets NOTE: I initially forgot to use `findFunctionAccordingToMroKnownStartingClass` instead of `findFunctionAccordingToMro` for __init__ and __new__, and since I did make that mistake myself, I wanted to add something to the test to highlight this fact, and make it viewable by PR reviewer... this will be fixed in the next commit. Python: Proper fix for spurious __init__ targets Python: Add call-graph example of class decorator Python: Support decorated classes in new call-graph Python: Add call-graph tests for `type(obj).meth()` Python: support `type(obj).meth()` Python: Add test for callable defined in function Python: Add test for callable as argument Current'y we don't find these with type-tracking, which is super mysterious. I did check that we have proper flow from the arguments to the parameters. Python: Found problem for callable as argument :| MAJOR WIP WIP commit IT WORKS AGAIN (but terrible performance) remove pragma[inline] remove oops Fix performance problem I tried to optimize it even further, but I didn't end up achieving anything :| Fix call-graph comparison add comparison version with easy lookup incomplete missing call-graph tests unhandled tests trying to replicate missing call-edge due to missing imports ... but it's hard also seems to be problems with the inline-expectation-value that I used, seems like it has both missing/unexpected results with same value Python: Add import-problem test Python: Add shadowing problem some cleanup of rewrite fix a little more cleanup Add consistency queries to call-graph tests Python: Add post-update nodes for `self` in implicit `super()` uses But we do need to discuss whether this is the right approach :O Fix for field-flow tests This came from more precise argument passing Fixed results in type-tracking Comes from better argument passing with super() and handling of functions with decorators fix of inline call graph tests Fixup call annotation test Many minor cleanups/fixes NewNormalCall -> NormalCall Python: Major restructuring + qldoc writing Python: Accept changes from pre/post update node .toString changes Python: Reduce `super` complexity !! WIP !! Python: Only pass self-reference if in same enclosing-callable Python: Add call-graph test with nested class This was inspired by the ImpliesDataflow test that showed missing flow for q_super, but at least for the call-graph, I'm not able to reproduce this missing result :| Python: Restrict `super()` to function defined directly on class Python: Accept fixes to ImpliesDataflow Python: Expand field-flow crosstalk tests
This commit is contained in:
@@ -560,9 +560,9 @@ class With_getitem:
|
||||
|
||||
|
||||
def test_getitem():
|
||||
with_getitem = With_getitem() #$ arg1="SSA variable with_getitem" func=With_getitem.__getitem__
|
||||
with_getitem = With_getitem() #$ MISSING: arg1="SSA variable with_getitem" func=With_getitem.__getitem__
|
||||
arg2 = 0
|
||||
with_getitem[arg2] #$ arg2 func=With_getitem.__getitem__
|
||||
with_getitem[arg2] #$ MISSING: arg2 func=With_getitem.__getitem__
|
||||
|
||||
|
||||
# object.__setitem__(self, key, value)
|
||||
@@ -575,10 +575,10 @@ class With_setitem:
|
||||
|
||||
|
||||
def test_setitem():
|
||||
with_setitem = With_setitem() #$ arg1="SSA variable with_setitem" func=With_setitem.__setitem__
|
||||
with_setitem = With_setitem() #$ MISSING: arg1="SSA variable with_setitem" func=With_setitem.__setitem__
|
||||
arg2 = 0
|
||||
arg3 = ""
|
||||
with_setitem[arg2] = arg3 #$ arg2 arg3 func=With_setitem.__setitem__
|
||||
with_setitem[arg2] = arg3 #$ MISSING: arg2 arg3 func=With_setitem.__setitem__
|
||||
|
||||
|
||||
# object.__delitem__(self, key)
|
||||
@@ -590,9 +590,9 @@ class With_delitem:
|
||||
|
||||
|
||||
def test_delitem():
|
||||
with_delitem = With_delitem() #$ arg1="SSA variable with_delitem" func=With_delitem.__delitem__
|
||||
with_delitem = With_delitem() #$ MISSING: arg1="SSA variable with_delitem" func=With_delitem.__delitem__
|
||||
arg2 = 0
|
||||
del with_delitem[arg2] #$ arg2 func=With_delitem.__delitem__
|
||||
del with_delitem[arg2] #$ MISSING: arg2 func=With_delitem.__delitem__
|
||||
|
||||
|
||||
# object.__missing__(self, key)
|
||||
@@ -662,9 +662,9 @@ class With_add:
|
||||
|
||||
|
||||
def test_add():
|
||||
with_add = With_add() #$ arg1="SSA variable with_add" func=With_add.__add__
|
||||
with_add = With_add() #$ MISSING: arg1="SSA variable with_add" func=With_add.__add__
|
||||
arg2 = with_add
|
||||
with_add + arg2 #$ arg2 func=With_add.__add__
|
||||
with_add + arg2 #$ MISSING: arg2 func=With_add.__add__
|
||||
|
||||
|
||||
# object.__sub__(self, other)
|
||||
@@ -677,9 +677,9 @@ class With_sub:
|
||||
|
||||
|
||||
def test_sub():
|
||||
with_sub = With_sub() #$ arg1="SSA variable with_sub" func=With_sub.__sub__
|
||||
with_sub = With_sub() #$ MISSING: arg1="SSA variable with_sub" func=With_sub.__sub__
|
||||
arg2 = with_sub
|
||||
with_sub - arg2 #$ arg2 func=With_sub.__sub__
|
||||
with_sub - arg2 #$ MISSING: arg2 func=With_sub.__sub__
|
||||
|
||||
|
||||
# object.__mul__(self, other)
|
||||
@@ -692,9 +692,9 @@ class With_mul:
|
||||
|
||||
|
||||
def test_mul():
|
||||
with_mul = With_mul() #$ arg1="SSA variable with_mul" func=With_mul.__mul__
|
||||
with_mul = With_mul() #$ MISSING: arg1="SSA variable with_mul" func=With_mul.__mul__
|
||||
arg2 = with_mul
|
||||
with_mul * arg2 #$ arg2 func=With_mul.__mul__
|
||||
with_mul * arg2 #$ MISSING: arg2 func=With_mul.__mul__
|
||||
|
||||
|
||||
# object.__matmul__(self, other)
|
||||
@@ -707,9 +707,9 @@ class With_matmul:
|
||||
|
||||
|
||||
def test_matmul():
|
||||
with_matmul = With_matmul() #$ arg1="SSA variable with_matmul" func=With_matmul.__matmul__
|
||||
with_matmul = With_matmul() #$ MISSING: arg1="SSA variable with_matmul" func=With_matmul.__matmul__
|
||||
arg2 = with_matmul
|
||||
with_matmul @ arg2 #$ arg2 func=With_matmul.__matmul__
|
||||
with_matmul @ arg2 #$ MISSING: arg2 func=With_matmul.__matmul__
|
||||
|
||||
|
||||
# object.__truediv__(self, other)
|
||||
@@ -722,9 +722,9 @@ class With_truediv:
|
||||
|
||||
|
||||
def test_truediv():
|
||||
with_truediv = With_truediv() #$ arg1="SSA variable with_truediv" func=With_truediv.__truediv__
|
||||
with_truediv = With_truediv() #$ MISSING: arg1="SSA variable with_truediv" func=With_truediv.__truediv__
|
||||
arg2 = with_truediv
|
||||
with_truediv / arg2 #$ arg2 func=With_truediv.__truediv__
|
||||
with_truediv / arg2 #$ MISSING: arg2 func=With_truediv.__truediv__
|
||||
|
||||
|
||||
# object.__floordiv__(self, other)
|
||||
@@ -737,9 +737,9 @@ class With_floordiv:
|
||||
|
||||
|
||||
def test_floordiv():
|
||||
with_floordiv = With_floordiv() #$ arg1="SSA variable with_floordiv" func=With_floordiv.__floordiv__
|
||||
with_floordiv = With_floordiv() #$ MISSING: arg1="SSA variable with_floordiv" func=With_floordiv.__floordiv__
|
||||
arg2 = with_floordiv
|
||||
with_floordiv // arg2 #$ arg2 func=With_floordiv.__floordiv__
|
||||
with_floordiv // arg2 #$ MISSING: arg2 func=With_floordiv.__floordiv__
|
||||
|
||||
|
||||
# object.__mod__(self, other)
|
||||
@@ -752,9 +752,9 @@ class With_mod:
|
||||
|
||||
|
||||
def test_mod():
|
||||
with_mod = With_mod() #$ arg1="SSA variable with_mod" func=With_mod.__mod__
|
||||
with_mod = With_mod() #$ MISSING: arg1="SSA variable with_mod" func=With_mod.__mod__
|
||||
arg2 = with_mod
|
||||
with_mod % arg2 #$ arg2 func=With_mod.__mod__
|
||||
with_mod % arg2 #$ MISSING: arg2 func=With_mod.__mod__
|
||||
|
||||
|
||||
# object.__divmod__(self, other)
|
||||
@@ -788,9 +788,9 @@ def test_pow():
|
||||
|
||||
|
||||
def test_pow_op():
|
||||
with_pow = With_pow() #$ arg1="SSA variable with_pow" func=With_pow.__pow__
|
||||
with_pow = With_pow() #$ MISSING: arg1="SSA variable with_pow" func=With_pow.__pow__
|
||||
arg2 = with_pow
|
||||
with_pow ** arg2 #$ arg2 func=With_pow.__pow__
|
||||
with_pow ** arg2 #$ MISSING: arg2 func=With_pow.__pow__
|
||||
|
||||
|
||||
# object.__lshift__(self, other)
|
||||
@@ -803,9 +803,9 @@ class With_lshift:
|
||||
|
||||
|
||||
def test_lshift():
|
||||
with_lshift = With_lshift() #$ arg1="SSA variable with_lshift" func=With_lshift.__lshift__
|
||||
with_lshift = With_lshift() #$ MISSING: arg1="SSA variable with_lshift" func=With_lshift.__lshift__
|
||||
arg2 = with_lshift
|
||||
with_lshift << arg2 #$ arg2 func=With_lshift.__lshift__
|
||||
with_lshift << arg2 #$ MISSING: arg2 func=With_lshift.__lshift__
|
||||
|
||||
|
||||
# object.__rshift__(self, other)
|
||||
@@ -818,9 +818,9 @@ class With_rshift:
|
||||
|
||||
|
||||
def test_rshift():
|
||||
with_rshift = With_rshift() #$ arg1="SSA variable with_rshift" func=With_rshift.__rshift__
|
||||
with_rshift = With_rshift() #$ MISSING: arg1="SSA variable with_rshift" func=With_rshift.__rshift__
|
||||
arg2 = with_rshift
|
||||
with_rshift >> arg2 #$ arg2 func=With_rshift.__rshift__
|
||||
with_rshift >> arg2 #$ MISSING: arg2 func=With_rshift.__rshift__
|
||||
|
||||
|
||||
# object.__and__(self, other)
|
||||
@@ -833,9 +833,9 @@ class With_and:
|
||||
|
||||
|
||||
def test_and():
|
||||
with_and = With_and() #$ arg1="SSA variable with_and" func=With_and.__and__
|
||||
with_and = With_and() #$ MISSING: arg1="SSA variable with_and" func=With_and.__and__
|
||||
arg2 = with_and
|
||||
with_and & arg2 #$ arg2 func=With_and.__and__
|
||||
with_and & arg2 #$ MISSING: arg2 func=With_and.__and__
|
||||
|
||||
|
||||
# object.__xor__(self, other)
|
||||
@@ -848,9 +848,9 @@ class With_xor:
|
||||
|
||||
|
||||
def test_xor():
|
||||
with_xor = With_xor() #$ arg1="SSA variable with_xor" func=With_xor.__xor__
|
||||
with_xor = With_xor() #$ MISSING: arg1="SSA variable with_xor" func=With_xor.__xor__
|
||||
arg2 = with_xor
|
||||
with_xor ^ arg2 #$ arg2 func=With_xor.__xor__
|
||||
with_xor ^ arg2 #$ MISSING: arg2 func=With_xor.__xor__
|
||||
|
||||
|
||||
# object.__or__(self, other)
|
||||
@@ -863,9 +863,9 @@ class With_or:
|
||||
|
||||
|
||||
def test_or():
|
||||
with_or = With_or() #$ arg1="SSA variable with_or" func=With_or.__or__
|
||||
with_or = With_or() #$ MISSING: arg1="SSA variable with_or" func=With_or.__or__
|
||||
arg2 = with_or
|
||||
with_or | arg2 #$ arg2 func=With_or.__or__
|
||||
with_or | arg2 #$ MISSING: arg2 func=With_or.__or__
|
||||
|
||||
|
||||
# object.__radd__(self, other)
|
||||
|
||||
Reference in New Issue
Block a user