hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2026-04-29 18:55:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

Python: Implement call-graph with type-trackers

Browse Source 
This commit is a squash of 80 other commits. While developing, things
changed majorly 2-3 times, and it just wasn't feasible to go back and
write a really nice commit history.

My apologies for this HUGE commit.

Also, later on this is where I solved merge conflicts after flow-summaries
PR was merged.

For your amusement, I've included the original commit messages below.

Python: Add proper argument/parameter positions

Python: Handle normal function calls

Python: Reduce dataflow-consistency warnings

Previously there was a lot of failures for `uniqueEnclosingCallable` and
`argHasPostUpdate`

Removing the override of `getEnclosingCallable` in ParameterNode is
probably the most controversial... although from my point of view it's a
change for the better, since we're able to provide data-flow
ParameterNodes for more of the AST parameter nodes.

Python: Adjust `dataflow/calls` test

Python: Implement `isParameterOf`/`argumentOf`/`OutNode`

This makes the tests under `dataflow/basic` work as well 👍

(initially I had these as separate commits, but it felt like it was too much noise)

Python: Accept fix for `dataflow/consistency`

Python: Changes to `coverage/argumentRoutingTest.ql`

Notice we gain a few new resolved arguments.

We loose out on stuff due to:

1. not handling `*` or `**` in either arguments/parameters (yet)
2. not handling special calls (yet)

Python: Small fix for `TestUtil/RoutingTest.qll`

Since the helper predicates do not depend on this, moved outside class.

Python: Accept changes to `dataflow/coverage/NormalDataflowTest.ql`

Most of this is due to:

- not handling any kinds of methods yet
- not handling `*` or `**`

Python: Small investigation of `test_deep_callgraph`

Python: Accept changes to `coverage/localFlow.ql`

I don't fully understand why the .expected file changed.

Since we still have the desired flow, I'm not going to worry too much
about it.

with this commit, the `dataflow/coverage` tests passes 👍

Python: Minor doc update

Python: Add staticmethod/classmethod to `dataflow/calls`

Python: Handle method calls on class instances

without trying to deal with any class inheritance, or
staticmethod/classmethod at all.

Notice that with this change, we only have a DataFlowCall for the calls
that we can actually resolve. I'm not 100% sure if we need to add a
`UnresolvedCall` subclass of `DataFlowCall` for MaD in the future, but
it should be easy to do.

I'm still unsure about the value of `classesCallGraph`, but have just
accepted the changes.

Python: Handle direct method calls `C.foo(C, arg0)`

Python: Handle `@staticmethod`

Python: Handle class method calls... but the code is shit

WIP todo

Rewrite method calls to be better

also fixed a problem with `self` being an argument to the `x.staticmethod()` call :|

Python: Add subclass tests

Python: Split `class_advanced` test

Python: Rewrite call-graph tests to be inline expectation (1/2)

This adds inline expectations, next commit will remove old annotations
code... but I thought it would be easier to review like this.

Minor fixup

Python: Add simple subclass support

Python: more precise subclass lookup

Still not 100% precise.. but it's better

New ambiguous

Python: Add test for `self.m()` and `cls.m()` calls

Python: Handle `self.m()` and `cls.m()` calls

Python: Add tests for `__init__` and `__new__`

Python: Handle class calls

Python: Fix `self` argument passing for class calls

Now field-flow tests also pass 💪 (although the crosstalk
fieldflow test changes were due to this specific commit)

I also copied much of the setup for pre/post update nodes from Ruby,
specifically having the abstract `PostUpdateNodeImpl` in DataFlowPrivate
seemed like a nice change.

Same for the setup with `TNode` definition having the specification
directly in the body, instead of a `NeedsSyntheticPostUpdateNode` class.

Python: Add new crosstalk test WIP

Maybe needs a bit of refactoring, and to see how it all behaves with points-to

Python: Add `super()` call-graph tests

Python: Refactor MethodCall char-pred

In anticipation of supporting `super(MyClass, self).foo()`, where the
`self` argument doesn't come from an AttrNode, but from the second
argument to super.

Without `pragma[inline]` the optimizer found a terrible join-order --
this won't guarantee a good join-order for the future, but for now it
was just so simple and could let me move on with life.

Python: Add basic `super()` support

I debated a little (with myself) whether I should really do
`superTracker`, but I thought "why not" and just rolled with it. I did
not confirm whether it was actually needed anywhere, that is if anyone
does `ref = super; ref().foo()` -- although I certainly doubt it's very
wide-spread.

Python: InlineCallGraphTest: Allow non-unique callable name in different files

Python: more MRO tests

Python: Add MRO approximation for `super()`

Although it's not 100% accurate, it seems to be on level with the one in
points-to.

Python: Remove some spurious targets for direct calls

removal of TODO from refactoring

remove TODOs class call support

Python: Add contrived subclass call example

Python: Remove more spurious call targets

NOTE: I initially forgot to use
`findFunctionAccordingToMroKnownStartingClass` instead of
`findFunctionAccordingToMro` for __init__ and __new__, and since I did
make that mistake myself, I wanted to add something to the test to
highlight this fact, and make it viewable by PR reviewer... this will be
fixed in the next commit.

Python: Proper fix for spurious __init__ targets

Python: Add call-graph example of class decorator

Python: Support decorated classes in new call-graph

Python: Add call-graph tests for `type(obj).meth()`

Python: support `type(obj).meth()`

Python: Add test for callable defined in function

Python: Add test for callable as argument

Current'y we don't find these with type-tracking, which is super
mysterious. I did check that we have proper flow from the arguments to
the parameters.

Python: Found problem for callable as argument :| MAJOR WIP

WIP commit

IT WORKS AGAIN (but terrible performance)

remove pragma[inline]

remove oops

Fix performance problem

I tried to optimize it even further, but I didn't end up achieving anything :|

Fix call-graph comparison

add comparison version with easy lookup

incomplete missing call-graph tests

unhandled tests

trying to replicate missing call-edge due to missing imports ... but it's hard

also seems to be problems with the inline-expectation-value that I used, seems like it has both missing/unexpected results with same value

Python: Add import-problem test

Python: Add shadowing problem

some cleanup of rewrite fix

a little more cleanup

Add consistency queries to call-graph tests

Python: Add post-update nodes for `self` in implicit `super()` uses

But we do need to discuss whether this is the right approach :O

Fix for field-flow tests

This came from more precise argument passing

Fixed results in type-tracking

Comes from better argument passing with super() and handling of
functions with decorators

fix of inline call graph tests

Fixup call annotation test

Many minor cleanups/fixes

NewNormalCall -> NormalCall

Python: Major restructuring + qldoc writing

Python: Accept changes from pre/post update node .toString changes

Python: Reduce `super` complexity !! WIP !!

Python: Only pass self-reference if in same enclosing-callable

Python: Add call-graph test with nested class

This was inspired by the ImpliesDataflow test that showed missing flow
for q_super, but at least for the call-graph, I'm not able to reproduce
this missing result :|

Python: Restrict `super()` to function defined directly on class

Python: Accept fixes to ImpliesDataflow

Python: Expand field-flow crosstalk tests
This commit is contained in:
Rasmus Wriedt Larsen
2022-05-31 01:41:23 +02:00
parent aa78a43407
commit 9c275c177a
81 changed files with 2494 additions and 447 deletions
Download Patch File Download Diff File Expand all files Collapse all files

915
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatch.qll
View File

File diff suppressed because it is too large Load Diff

174
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll
View File

@@ -39,157 +39,47 @@ predicate isArgumentNode(ArgumentNode arg, DataFlowCall c, ArgumentPosition pos)
//--------
predicate isExpressionNode(ControlFlowNode node) { node.getNode() instanceof Expr }
/** DEPRECATED: Alias for `SyntheticPreUpdateNode` */
deprecated module syntheticPreUpdateNode = SyntheticPreUpdateNode;
class SyntheticPreUpdateNode extends Node, TSyntheticPreUpdateNode {
CallNode node;
/** A module collecting the different reasons for synthesising a pre-update node. */
module SyntheticPreUpdateNode {
class SyntheticPreUpdateNode extends Node, TSyntheticPreUpdateNode {
NeedsSyntheticPreUpdateNode post;
SyntheticPreUpdateNode() { this = TSyntheticPreUpdateNode(node) }
SyntheticPreUpdateNode() { this = TSyntheticPreUpdateNode(post) }
/** Gets the node for which this is a synthetic pre-update node. */
CfgNode getPostUpdateNode() { result.getNode() = node }
/** Gets the node for which this is a synthetic pre-update node. */
Node getPostUpdateNode() { result = post }
override string toString() { result = "[pre] " + node.toString() }
override string toString() { result = "[pre " + post.label() + "] " + post.toString() }
override Scope getScope() { result = node.getScope() }
override Scope getScope() { result = post.getScope() }
override Location getLocation() { result = post.getLocation() }
}
/** A data flow node for which we should synthesise an associated pre-update node. */
class NeedsSyntheticPreUpdateNode extends PostUpdateNode {
NeedsSyntheticPreUpdateNode() { this = objectCreationNode() }
override Node getPreUpdateNode() { result.(SyntheticPreUpdateNode).getPostUpdateNode() = this }
/**
* Gets the label for this kind of node. This will figure in the textual representation of the synthesized pre-update node.
*
* There is currently only one reason for needing a pre-update node, so we always use that as the label.
*/
string label() { result = "objCreate" }
}
/**
* Calls to constructors are treated as post-update nodes for the synthesized argument
* that is mapped to the `self` parameter. That way, constructor calls represent the value of the
* object after the constructor (currently only `__init__`) has run.
*/
CfgNode objectCreationNode() {
// TODO(call-graph): implement this!
none()
// result.getNode().(CallNode) = any(ClassCall c).getNode()
}
override Location getLocation() { result = node.getLocation() }
}
import SyntheticPreUpdateNode
/** DEPRECATED: Alias for `SyntheticPostUpdateNode` */
deprecated module syntheticPostUpdateNode = SyntheticPostUpdateNode;
/** A module collecting the different reasons for synthesising a post-update node. */
module SyntheticPostUpdateNode {
/** A post-update node is synthesized for all nodes which satisfy `NeedsSyntheticPostUpdateNode`. */
class SyntheticPostUpdateNode extends PostUpdateNode, TSyntheticPostUpdateNode {
NeedsSyntheticPostUpdateNode pre;
SyntheticPostUpdateNode() { this = TSyntheticPostUpdateNode(pre) }
override Node getPreUpdateNode() { result = pre }
override string toString() { result = "[post " + pre.label() + "] " + pre.toString() }
override Scope getScope() { result = pre.getScope() }
override Location getLocation() { result = pre.getLocation() }
}
/** A data flow node for which we should synthesise an associated post-update node. */
class NeedsSyntheticPostUpdateNode extends Node {
NeedsSyntheticPostUpdateNode() {
this = argumentPreUpdateNode()
or
this = storePreUpdateNode()
or
this = readPreUpdateNode()
}
/**
* Gets the label for this kind of node. This will figure in the textual representation of the synthesized post-update node.
* We favour being an arguments as the reason for the post-update node in case multiple reasons apply.
*/
string label() {
if this = argumentPreUpdateNode()
then result = "arg"
else
if this = storePreUpdateNode()
then result = "store"
else result = "read"
}
}
/**
* Gets the pre-update node for this node.
*
* An argument might have its value changed as a result of a call.
* Certain arguments, such as implicit self arguments are already post-update nodes
* and should not have an extra node synthesised.
*/
Node argumentPreUpdateNode() {
// TODO(call-graph): implement this!
none()
// result = any(FunctionCall c).getArg(_)
// or
// // Avoid argument 0 of method calls as those have read post-update nodes.
// exists(MethodCall c, int n | n > 0 | result = c.getArg(n))
// or
// result = any(SpecialCall c).getArg(_)
// or
// // Avoid argument 0 of class calls as those have non-synthetic post-update nodes.
// exists(ClassCall c, int n | n > 0 | result = c.getArg(n))
// or
// // any argument of any call that we have not been able to resolve
// exists(CallNode call | not call = any(DataFlowCall c).getNode() |
// result.(CfgNode).getNode() in [call.getArg(_), call.getArgByName(_)]
// )
}
/** Gets the pre-update node associated with a store. This is used for when an object might have its value changed after a store. */
CfgNode storePreUpdateNode() {
exists(Attribute a |
result.getNode() = a.getObject().getAFlowNode() and
a.getCtx() instanceof Store
)
}
/**
* Gets a node marking the state change of an object after a read.
*
* A reverse read happens when the result of a read is modified, e.g. in
* ```python
* l = [ mutable ]
* l[0].mutate()
* ```
* we may now have changed the content of `l`. To track this, there must be
* a postupdate node for `l`.
*/
CfgNode readPreUpdateNode() {
exists(Attribute a |
result.getNode() = a.getObject().getAFlowNode() and
a.getCtx() instanceof Load
)
or
result.getNode() = any(SubscriptNode s).getObject()
or
// The dictionary argument is read from if the callable has parameters matching the keys.
result.getNode().getNode() = any(Call call).getKwargs()
}
abstract class PostUpdateNodeImpl extends Node {
/** Gets the node before the state update. */
abstract Node getPreUpdateNode();
}
import SyntheticPostUpdateNode
class SyntheticPostUpdateNode extends PostUpdateNodeImpl, TSyntheticPostUpdateNode {
ControlFlowNode node;
SyntheticPostUpdateNode() { this = TSyntheticPostUpdateNode(node) }
override Node getPreUpdateNode() { result.(CfgNode).getNode() = node }
override string toString() { result = "[post] " + node.toString() }
override Scope getScope() { result = node.getScope() }
override Location getLocation() { result = node.getLocation() }
}
class NonSyntheticPostUpdateNode extends PostUpdateNodeImpl, CfgNode {
SyntheticPreUpdateNode pre;
NonSyntheticPostUpdateNode() { this = pre.getPostUpdateNode() }
override Node getPreUpdateNode() { result = pre }
}
class DataFlowExpr = Expr;

70
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPublic.qll
View File

@@ -31,10 +31,41 @@ newtype TNode =
or
node.getNode() instanceof Pattern
} or
/** A synthetic node representing the value of an object before a state change */
TSyntheticPreUpdateNode(NeedsSyntheticPreUpdateNode post) or
/** A synthetic node representing the value of an object after a state change. */
TSyntheticPostUpdateNode(NeedsSyntheticPostUpdateNode pre) or
/**
* A synthetic node representing the value of an object before a state change.
*
* For class calls we pass a synthetic self argument, so attribute writes in
* `__init__` is reflected on the resulting object (we need special logic for this
* since there is no `return` in `__init__`)
*/
// NOTE: since we can't rely on the call graph, but we want to have synthetic
// pre-update nodes for class calls, we end up getting synthetic pre-update nodes for
// ALL calls :|
TSyntheticPreUpdateNode(CallNode call) or
/**
* A synthetic node representing the value of an object after a state change.
* See QLDoc for `PostUpdateNode`.
*/
TSyntheticPostUpdateNode(ControlFlowNode node) {
exists(CallNode call |
node = call.getArg(_)
or
node = call.getArgByName(_)
)
or
node = any(AttrNode a).getObject()
or
node = any(SubscriptNode s).getObject()
or
// self parameter when used implicitly in `super()`
exists(Class cls, Function func, ParameterDefinition def |
func = cls.getAMethod() and
not hasStaticmethodDecorator(func) and
// this matches what we do in ParameterNode
def.getDefiningNode() = node and
def.getParameter() = func.getArg(0)
)
} or
/** A node representing a global (module-level) variable in a specific module. */
TModuleVariableNode(Module m, GlobalVariable v) {
v.getScope() = m and
@@ -270,13 +301,9 @@ class ExtractedParameterNode extends ParameterNodeImpl, CfgNode {
ExtractedParameterNode() { node = def.getDefiningNode() }
override predicate isParameterOf(DataFlowCallable c, ParameterPosition ppos) {
// TODO(call-graph): implement this!
none()
this = c.getParameter(ppos)
}
override DataFlowCallable getEnclosingCallable() { this.isParameterOf(result, _) }
/** Gets the `Parameter` this `ParameterNode` represents. */
override Parameter getParameter() { result = def.getParameter() }
}
@@ -294,16 +321,16 @@ abstract class ArgumentNode extends Node {
final ExtractedDataFlowCall getCall() { this.argumentOf(result, _) }
}
/** A data flow node that represents a call argument found in the source code. */
/**
* A data flow node that represents a call argument found in the source code,
* where the call can be resolved.
*/
class ExtractedArgumentNode extends ArgumentNode {
ExtractedArgumentNode() { this = any(ExtractedDataFlowCall c).getArgument(_) }
ExtractedArgumentNode() { getCallArg(_, _, _, this, _) }
final override predicate argumentOf(DataFlowCall call, ArgumentPosition pos) {
this.extractedArgumentOf(call, pos)
}
predicate extractedArgumentOf(ExtractedDataFlowCall call, ArgumentPosition pos) {
this = call.getArgument(pos)
this = call.getArgument(pos) and
call instanceof ExtractedDataFlowCall
}
}
@@ -312,16 +339,17 @@ class ExtractedArgumentNode extends ArgumentNode {
* changed its state.
*
* This can be either the argument to a callable after the callable returns
* (which might have mutated the argument), or the qualifier of a field after
* an update to the field.
* (which might have mutated the argument), the qualifier of a field after
* an update to the field, or a container such as a list/dictionary after an element
* update.
*
* Nodes corresponding to AST elements, for example `ExprNode`s, usually refer
* to the value before the update with the exception of `ObjectCreationNode`s,
* to the value before the update with the exception of class calls,
* which represents the value _after_ the constructor has run.
*/
abstract class PostUpdateNode extends Node {
class PostUpdateNode extends Node instanceof PostUpdateNodeImpl {
/** Gets the node before the state update. */
abstract Node getPreUpdateNode();
Node getPreUpdateNode() { result = super.getPreUpdateNode() }
}
/**

35
python/ql/lib/semmle/python/dataflow/new/internal/TypeTrackerSpecific.qll
View File

@@ -60,22 +60,6 @@ string getPossibleContentName() {
result = any(DataFlowPublic::AttrRef a).getAttributeName()
}
// /**
// * Gets a callable for the call where `nodeFrom` is used as the `i`'th argument.
// *
// * Helper predicate to avoid bad join order experienced in `callStep`.
// * This happened when `isParameterOf` was joined _before_ `getCallable`.
// */
// pragma[nomagic]
// private DataFlowPrivate::DataFlowCallable getCallableForArgument(
// DataFlowPublic::ExtractedArgumentNode nodeFrom, int i
// ) {
// exists(DataFlowPrivate::ExtractedDataFlowCall call |
// nodeFrom.extractedArgumentOf(call, i) and
// result = call.getCallable()
// )
// }
/**
* Holds if `nodeFrom` steps to `nodeTo` by being passed as a parameter in a call.
*
@@ -83,14 +67,17 @@ string getPossibleContentName() {
* recursion (or, at best, terrible performance), since identifying calls to library
* methods is done using API graphs (which uses type tracking).
*/
predicate callStep(DataFlowPublic::ArgumentNode nodeFrom, DataFlowPrivate::ParameterNodeImpl nodeTo) {
// TODO(call-graph): implement this!
none()
// // TODO: Support special methods?
// exists(DataFlowPrivate::DataFlowCallable callable, int i |
// callable = getCallableForArgument(nodeFrom, i) and
// nodeTo.isParameterOf(callable, i)
// )
predicate callStep(DataFlowPublic::ArgumentNode nodeFrom, DataFlowPublic::ParameterNode nodeTo) {
// TODO: Fix performance problem with pandas
exists(
DataFlowPrivate::DataFlowCall call, DataFlowPrivate::DataFlowCallable callable,
DataFlowPrivate::ArgumentPosition apos, DataFlowPrivate::ParameterPosition ppos
|
nodeFrom = call.getArgument(apos) and
nodeTo = callable.getParameter(ppos) and
DataFlowPrivate::parameterMatch(ppos, apos) and
callable = call.getCallable()
)
}
/** Holds if `nodeFrom` steps to `nodeTo` by being returned from a call. */

23
python/ql/src/meta/analysis-quality/CallGraphQuality.qll
View File

@@ -102,15 +102,26 @@ module TypeTrackingBasedCallGraph {
/** A call that can be resolved by type-tracking. */
class ResolvableCall extends RelevantCall {
TT::DataFlowCallable dataflowTarget;
ResolvableCall() { dataflowTarget = TT::viableCallable(TT::TNormalCall(this)) }
ResolvableCall() {
exists(TT::TNormalCall(this, _, _))
or
TT::resolveClassCall(this, _)
}
/** Gets a resolved target of this call. */
Target getTarget() {
result.(TargetFunction).getFunction() = dataflowTarget.(TT::DataFlowFunction).getScope()
// TODO: class calls
// result.(TargetClass).getClass()
exists(TT::DataFlowCall call, TT::CallType ct, Function targetFunc |
call = TT::TNormalCall(this, targetFunc, ct) and
not ct instanceof TT::CallTypeClass and
targetFunc = result.(TargetFunction).getFunction()
)
or
// a TT::TNormalCall only exists when the call can be resolved to a function.
// Since points-to just says the call goes directly to the class itself, and
// type-tracking based wants to resolve this to the constructor, which might not
// exist. So to do a proper comparison, we don't require the call to be resolve to
// a specific function.
TT::resolveClassCall(this, result.(TargetClass).getClass())
}
}

19
python/ql/src/meta/analysis-quality/TTCallGraphNewAmbiguous.ql Normal file
View File

@@ -0,0 +1,19 @@
/**
* @name New call graph edge from using type-tracking instead of points-to, that is ambiguous
* @kind problem
* @problem.severity recommendation
* @id py/meta/call-graph-new-ambiguous
* @tags meta
* @precision very-low
*/
import python
import CallGraphQuality
from CallNode call, Target target
where
target.isRelevant() and
not call.(PointsToBasedCallGraph::ResolvableCall).getTarget() = target and
call.(TypeTrackingBasedCallGraph::ResolvableCall).getTarget() = target and
1 < count(call.(TypeTrackingBasedCallGraph::ResolvableCall).getTarget())
select call, "NEW: $@ to $@", call, "Call", target, target.toString()

41
python/ql/test/experimental/dataflow/TestUtil/RoutingTest.qll
View File

@@ -26,29 +26,30 @@ abstract class RoutingTest extends InlineExpectationsTest {
element = fromNode.toString() and
(
tag = this.flowTag() and
if "\"" + tag + "\"" = this.fromValue(fromNode)
then value = ""
else value = this.fromValue(fromNode)
if "\"" + tag + "\"" = fromValue(fromNode) then value = "" else value = fromValue(fromNode)
or
// only have result for `func` tag if the function where `arg<n>` is used, is
// different from the function name of the call where `arg<n>` was specified as
// an argument
tag = "func" and
value = this.toFunc(toNode) and
not value = this.fromFunc(fromNode)
value = toFunc(toNode) and
not value = fromFunc(fromNode)
)
)
}
pragma[inline]
private string fromValue(DataFlow::Node fromNode) {
result = "\"" + prettyNode(fromNode).replaceAll("\"", "'") + "\""
}
pragma[inline]
private string fromFunc(DataFlow::ArgumentNode fromNode) {
result = fromNode.getCall().getNode().(CallNode).getFunction().getNode().(Name).getId()
}
pragma[inline]
private string toFunc(DataFlow::Node toNode) {
result = toNode.getEnclosingCallable().getCallableValue().getScope().getQualifiedName() // TODO: More robust pretty printing?
}
}
pragma[inline]
private string fromValue(DataFlow::Node fromNode) {
result = "\"" + prettyNode(fromNode).replaceAll("\"", "'") + "\""
}
pragma[inline]
private string fromFunc(DataFlow::ArgumentNode fromNode) {
result = fromNode.getCall().getNode().(CallNode).getFunction().getNode().(Name).getId()
}
pragma[inline]
private string toFunc(DataFlow::Node toNode) {
result = toNode.getEnclosingCallable().getQualifiedName()
}

1
python/ql/test/experimental/dataflow/basic/callGraphSinks.expected
View File

@@ -1,3 +1,2 @@
| file://:0:0:0:0 | parameter 0 of builtins.reversed |
| test.py:1:19:1:19 | ControlFlowNode for x |
| test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |

1
python/ql/test/experimental/dataflow/basic/callGraphSources.expected
View File

@@ -1,3 +1,2 @@
| file://:0:0:0:0 | [summary] to write: return (return) in builtins.reversed |
| test.py:4:10:4:10 | ControlFlowNode for z |
| test.py:7:19:7:19 | ControlFlowNode for a |

1
python/ql/test/experimental/dataflow/basic/global.expected
View File

@@ -1,4 +1,3 @@
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed | file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| test.py:1:1:1:21 | ControlFlowNode for FunctionExpr | test.py:1:5:1:17 | GSSA Variable obfuscated_id |
| test.py:1:1:1:21 | ControlFlowNode for FunctionExpr | test.py:7:5:7:17 | ControlFlowNode for obfuscated_id |
| test.py:1:5:1:17 | GSSA Variable obfuscated_id | test.py:7:5:7:17 | ControlFlowNode for obfuscated_id |

1
python/ql/test/experimental/dataflow/basic/globalStep.expected
View File

@@ -1,4 +1,3 @@
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed | file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| test.py:1:1:1:21 | ControlFlowNode for FunctionExpr | test.py:1:5:1:17 | GSSA Variable obfuscated_id |
| test.py:1:1:1:21 | ControlFlowNode for FunctionExpr | test.py:1:5:1:17 | GSSA Variable obfuscated_id |
| test.py:1:1:1:21 | ControlFlowNode for FunctionExpr | test.py:7:5:7:17 | ControlFlowNode for obfuscated_id |

8
python/ql/test/experimental/dataflow/basic/local.expected
View File

@@ -1,8 +1,3 @@
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed | file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed |
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed | file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| file://:0:0:0:0 | [summary] to write: return (return) in builtins.reversed | file://:0:0:0:0 | [summary] to write: return (return) in builtins.reversed |
| file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed | file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| file://:0:0:0:0 | parameter 0 of builtins.reversed | file://:0:0:0:0 | parameter 0 of builtins.reversed |
| test.py:0:0:0:0 | GSSA Variable __name__ | test.py:0:0:0:0 | GSSA Variable __name__ |
| test.py:0:0:0:0 | GSSA Variable __package__ | test.py:0:0:0:0 | GSSA Variable __package__ |
| test.py:0:0:0:0 | GSSA Variable b | test.py:0:0:0:0 | GSSA Variable b |
@@ -55,5 +50,6 @@
| test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() | test.py:7:1:7:1 | GSSA Variable b |
| test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() | test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |
| test.py:7:5:7:20 | GSSA Variable a | test.py:7:5:7:20 | GSSA Variable a |
| test.py:7:5:7:20 | [pre] ControlFlowNode for obfuscated_id() | test.py:7:5:7:20 | [pre] ControlFlowNode for obfuscated_id() |
| test.py:7:19:7:19 | ControlFlowNode for a | test.py:7:19:7:19 | ControlFlowNode for a |
| test.py:7:19:7:19 | [post arg] ControlFlowNode for a | test.py:7:19:7:19 | [post arg] ControlFlowNode for a |
| test.py:7:19:7:19 | [post] ControlFlowNode for a | test.py:7:19:7:19 | [post] ControlFlowNode for a |

1
python/ql/test/experimental/dataflow/basic/localStep.expected
View File

@@ -1,4 +1,3 @@
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed | file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| test.py:1:1:1:21 | ControlFlowNode for FunctionExpr | test.py:1:5:1:17 | GSSA Variable obfuscated_id |
| test.py:1:5:1:17 | GSSA Variable obfuscated_id | test.py:7:5:7:17 | ControlFlowNode for obfuscated_id |
| test.py:1:19:1:19 | ControlFlowNode for x | test.py:1:19:1:19 | SSA variable x |

7
python/ql/test/experimental/dataflow/basic/sinks.expected
View File

@@ -1,7 +1,3 @@
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed |
| file://:0:0:0:0 | [summary] to write: return (return) in builtins.reversed |
| file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| file://:0:0:0:0 | parameter 0 of builtins.reversed |
| test.py:0:0:0:0 | GSSA Variable __name__ |
| test.py:0:0:0:0 | GSSA Variable __package__ |
| test.py:0:0:0:0 | GSSA Variable b |
@@ -26,5 +22,6 @@
| test.py:7:5:7:17 | ControlFlowNode for obfuscated_id |
| test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |
| test.py:7:5:7:20 | GSSA Variable a |
| test.py:7:5:7:20 | [pre] ControlFlowNode for obfuscated_id() |
| test.py:7:19:7:19 | ControlFlowNode for a |
| test.py:7:19:7:19 | [post arg] ControlFlowNode for a |
| test.py:7:19:7:19 | [post] ControlFlowNode for a |

7
python/ql/test/experimental/dataflow/basic/sources.expected
View File

@@ -1,7 +1,3 @@
| file://:0:0:0:0 | [summary] read: argument 0.List element in builtins.reversed |
| file://:0:0:0:0 | [summary] to write: return (return) in builtins.reversed |
| file://:0:0:0:0 | [summary] to write: return (return).List element in builtins.reversed |
| file://:0:0:0:0 | parameter 0 of builtins.reversed |
| test.py:0:0:0:0 | GSSA Variable __name__ |
| test.py:0:0:0:0 | GSSA Variable __package__ |
| test.py:0:0:0:0 | GSSA Variable b |
@@ -26,5 +22,6 @@
| test.py:7:5:7:17 | ControlFlowNode for obfuscated_id |
| test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |
| test.py:7:5:7:20 | GSSA Variable a |
| test.py:7:5:7:20 | [pre] ControlFlowNode for obfuscated_id() |
| test.py:7:19:7:19 | ControlFlowNode for a |
| test.py:7:19:7:19 | [post arg] ControlFlowNode for a |
| test.py:7:19:7:19 | [post] ControlFlowNode for a |

13
python/ql/test/experimental/dataflow/callgraph_crosstalk/Arguments.expected Normal file
View File

@@ -0,0 +1,13 @@
| test.py:32:8:32:23 | CrosstalkTestX() | test.py:9:5:9:23 | Function __init__ | test.py:32:8:32:23 | [pre] ControlFlowNode for CrosstalkTestX() | self |
| test.py:33:8:33:23 | CrosstalkTestY() | test.py:21:5:21:23 | Function __init__ | test.py:33:8:33:23 | [pre] ControlFlowNode for CrosstalkTestY() | self |
| test.py:43:1:43:8 | func() | test.py:13:5:13:26 | Function setx | test.py:36:12:36:15 | ControlFlowNode for objx | self |
| test.py:43:1:43:8 | func() | test.py:13:5:13:26 | Function setx | test.py:43:6:43:7 | ControlFlowNode for IntegerLiteral | position 0 |
| test.py:43:1:43:8 | func() | test.py:25:5:25:26 | Function sety | test.py:38:12:38:15 | ControlFlowNode for objy | self |
| test.py:43:1:43:8 | func() | test.py:25:5:25:26 | Function sety | test.py:43:6:43:7 | ControlFlowNode for IntegerLiteral | position 0 |
| test.py:51:1:51:8 | func() | test.py:16:5:16:30 | Function setvalue | test.py:47:12:47:15 | ControlFlowNode for objx | self |
| test.py:51:1:51:8 | func() | test.py:16:5:16:30 | Function setvalue | test.py:51:6:51:7 | ControlFlowNode for IntegerLiteral | position 0 |
| test.py:51:1:51:8 | func() | test.py:28:5:28:30 | Function setvalue | test.py:49:12:49:15 | ControlFlowNode for objy | self |
| test.py:51:1:51:8 | func() | test.py:28:5:28:30 | Function setvalue | test.py:51:6:51:7 | ControlFlowNode for IntegerLiteral | position 0 |
| test.py:70:1:70:8 | func() | test.py:58:5:58:33 | Function foo | test.py:63:12:63:12 | ControlFlowNode for a | self |
| test.py:70:1:70:8 | func() | test.py:58:5:58:33 | Function foo | test.py:70:6:70:7 | ControlFlowNode for IntegerLiteral | position 0 |
| test.py:70:1:70:8 | func() | test.py:58:5:58:33 | Function foo | test.py:70:6:70:7 | ControlFlowNode for IntegerLiteral | self |

9
python/ql/test/experimental/dataflow/callgraph_crosstalk/Arguments.ql Normal file
View File

@@ -0,0 +1,9 @@
private import python
private import semmle.python.dataflow.new.internal.DataFlowPrivate
private import semmle.python.dataflow.new.internal.DataFlowPublic
from DataFlowCall call, DataFlowCallable callable, ArgumentNode arg, ArgumentPosition apos
where
callable = call.getCallable() and
arg = call.getArgument(apos)
select call, callable, arg, apos

19
python/ql/test/experimental/dataflow/callgraph_crosstalk/dataflow-consistency.expected Normal file
View File

@@ -0,0 +1,19 @@
uniqueEnclosingCallable
uniqueType
uniqueNodeLocation
missingLocation
uniqueNodeToString
missingToString
parameterCallable
localFlowIsLocal
compatibleTypesReflexive
unreachableNodeCCtx
localCallNodes
postIsNotPre
postHasUniquePre
uniquePostUpdate
postIsInSameCallable
reverseRead
argHasPostUpdate
postWithInFlow
viableImplInCallContextTooLarge

1
python/ql/test/experimental/dataflow/callgraph_crosstalk/dataflow-consistency.ql Normal file
View File

@@ -0,0 +1 @@
import semmle.python.dataflow.new.internal.DataFlowImplConsistency::Consistency

1
python/ql/test/experimental/dataflow/callgraph_crosstalk/options Normal file
View File

@@ -0,0 +1 @@
semmle-extractor-options: --max-import-depth=0

70
python/ql/test/experimental/dataflow/callgraph_crosstalk/test.py Normal file
View File

@@ -0,0 +1,70 @@
import random
cond = random.randint(0,1) == 1
# ------------------------------------------------------------------------------
# Calling different bound-methods based on conditional
# ------------------------------------------------------------------------------
class CrosstalkTestX:
def __init__(self):
self.x = None
self.y = None
def setx(self, value):
self.x = value
def setvalue(self, value):
self.x = value
class CrosstalkTestY:
def __init__(self):
self.x = None
self.y = None
def sety(self ,value):
self.y = value
def setvalue(self, value):
self.y = value
objx = CrosstalkTestX()
objy = CrosstalkTestY()
if cond:
func = objx.setx
else:
func = objy.sety
# What we're testing for is whether both objects are passed as self to both methods,
# which is wrong.
func(42)
if cond:
func = objx.setvalue
else:
func = objy.setvalue
func(43)
# ------------------------------------------------------------------------------
# Calling methods in different ways
# ------------------------------------------------------------------------------
class A(object):
def foo(self, arg="Default"):
print("A.foo", self, arg)
a = A()
if cond:
func = a.foo # `44` is passed as arg
else:
func = A.foo # `44` is passed as self
# What we're testing for is whether a single call ends up having both `a` and `44` is
# passed as self to `A.foo`, which is wrong.
func(44)

18
python/ql/test/experimental/dataflow/calls/DataFlowCallTest.ql
View File

@@ -1,6 +1,6 @@
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.internal.DataFlowPrivate
import semmle.python.dataflow.new.internal.DataFlowDispatch as DataFlowDispatch
import TestUtilities.InlineExpectationsTest
private import semmle.python.dataflow.new.internal.PrintNode
@@ -8,26 +8,28 @@ class DataFlowCallTest extends InlineExpectationsTest {
DataFlowCallTest() { this = "DataFlowCallTest" }
override string getARelevantTag() {
result in ["call", "qlclass"]
result in ["call", "callType"]
or
result = "arg_" + [0 .. 10]
result = "arg[" + any(DataFlowDispatch::ArgumentPosition pos).toString() + "]"
}
override predicate hasActualResult(Location location, string element, string tag, string value) {
exists(location.getFile().getRelativePath()) and
exists(DataFlowCall call |
exists(DataFlowDispatch::DataFlowCall call |
location = call.getLocation() and
element = call.toString()
|
value = prettyExpr(call.getNode().getNode()) and
tag = "call"
or
value = call.getAQlClass() and
tag = "qlclass"
value = call.(DataFlowDispatch::NormalCall).getCallType().toString() and
tag = "callType"
or
exists(int n, DataFlow::Node arg | arg = call.getArg(n) |
exists(DataFlowDispatch::ArgumentPosition pos, DataFlow::Node arg |
arg = call.getArgument(pos)
|
value = prettyNodeForInlineTest(arg) and
tag = "arg_" + n
tag = "arg[" + pos + "]"
)
)
}

56
python/ql/test/experimental/dataflow/calls/test.py
View File

@@ -14,24 +14,60 @@ class MyClass(object):
def my_method(self, arg):
pass
@staticmethod
def staticmethod(arg):
pass
@classmethod
def classmethod(cls, arg):
pass
def __getitem__(self, key):
pass
func(0) # $ call=func(..) arg[position 0]=0 callType=CallTypePlainFunction
func("foo") # $ call=func(..) qlclass=FunctionCall arg_0="foo"
x = MyClass(1) # $ call=MyClass(..) qlclass=ClassCall arg_0=[pre]MyClass(..) arg_1=1
x.my_method(2) # $ call=x.my_method(..) qlclass=MethodCall arg_0=x arg_1=2
x = MyClass(1) # $ call=MyClass(..) arg[self]=[pre]MyClass(..) arg[position 0]=1 callType=CallTypeClass
x.my_method(2) # $ call=x.my_method(..) arg[self]=x arg[position 0]=2 callType=CallTypeNormalMethod
mm = x.my_method
mm(2) # $ call=mm(..) qlclass=MethodCall arg_1=2 MISSING: arg_0=x
x[3] # $ call=x[3] qlclass=SpecialCall arg_0=x arg_1=3
mm(2) # $ call=mm(..) arg[self]=x arg[position 0]=2 callType=CallTypeNormalMethod
MyClass.my_method(x, 2) # $ call=MyClass.my_method(..) arg[position 0]=2 arg[self]=x callType=CallTypeMethodAsPlainFunction
x.staticmethod(3) # $ call=x.staticmethod(..) arg[position 0]=3 callType=CallTypeStaticMethod
MyClass.staticmethod(3) # $ call=MyClass.staticmethod(..) arg[position 0]=3 callType=CallTypeStaticMethod
x.classmethod(4) # $ call=x.classmethod(..) arg[position 0]=4 callType=CallTypeClassMethod
MyClass.classmethod(4) # $ call=MyClass.classmethod(..) arg[position 0]=4 arg[self]=MyClass callType=CallTypeClassMethod
x[5] # $ MISSING: call=x[5] arg[self]=x arg[position 0]=5
class Subclass(MyClass):
pass
y = Subclass(1) # $ call=Subclass(..) arg[self]=[pre]Subclass(..) arg[position 0]=1 callType=CallTypeClass
y.my_method(2) # $ call=y.my_method(..) arg[self]=y arg[position 0]=2 callType=CallTypeNormalMethod
mm = y.my_method
mm(2) # $ call=mm(..) arg[self]=y arg[position 0]=2 callType=CallTypeNormalMethod
Subclass.my_method(y, 2) # $ call=Subclass.my_method(..) arg[self]=y arg[position 0]=2 callType=CallTypeMethodAsPlainFunction
y.staticmethod(3) # $ call=y.staticmethod(..) arg[position 0]=3 callType=CallTypeStaticMethod
Subclass.staticmethod(3) # $ call=Subclass.staticmethod(..) arg[position 0]=3 callType=CallTypeStaticMethod
y.classmethod(4) # $ call=y.classmethod(..) arg[position 0]=4 callType=CallTypeClassMethod
Subclass.classmethod(4) # $ call=Subclass.classmethod(..) arg[self]=Subclass arg[position 0]=4 callType=CallTypeClassMethod
y[5] # $ MISSING: call=y[5] arg[self]=y arg[position 0]=5
try:
# These are included to show how we handle absent things with points-to where
# `mypkg.foo` is a `missing module variable`, but `mypkg.subpkg.bar` is compeltely
# ignored.
# These are included to show whether we have a DataFlowCall for things we can't
# resolve. Both are interesting since with points-to we used to have a DataFlowCall
# for _one_ but not the other
import mypkg
mypkg.foo(42) # $ call=mypkg.foo(..) qlclass=NormalCall
mypkg.subpkg.bar(43) # $ call=mypkg.subpkg.bar(..) qlclass=LibraryCall arg_0=43
mypkg.foo(42)
mypkg.subpkg.bar(43)
except:
pass

1
python/ql/test/experimental/dataflow/consistency/modeling-consistency.expected
View File

@@ -1 +0,0 @@
| test.py:239:27:239:27 | Parameter | There is no `ParameterNode` associated with this parameter. |

16
python/ql/test/experimental/dataflow/coverage/argumentPassing.py
View File

@@ -64,12 +64,12 @@ def argument_passing(
@expects(7)
def test_argument_passing1():
argument_passing(arg1, *(arg2, arg3, arg4), e=arg5, **{"f": arg6, "g": arg7}) #$ arg1 arg7 func=argument_passing MISSING: arg2 arg3="arg3 arg4 arg5 arg6
argument_passing(arg1, *(arg2, arg3, arg4), e=arg5, **{"f": arg6, "g": arg7}) #$ arg1 arg5 MISSING: arg2 arg3 arg4 arg6 arg7
@expects(7)
def test_argument_passing2():
argument_passing(arg1, arg2, arg3, f=arg6) #$ arg1 arg2 arg3
argument_passing(arg1, arg2, arg3, f=arg6) #$ arg1 arg2 arg3 arg6
def with_pos_only(a, /, b):
@@ -94,8 +94,8 @@ def with_multiple_kw_args(a, b, c):
def test_multiple_kw_args():
with_multiple_kw_args(b=arg2, c=arg3, a=arg1) #$ arg1 arg2 arg3
with_multiple_kw_args(arg1, *(arg2,), arg3) #$ arg1 MISSING: arg2 arg3
with_multiple_kw_args(arg1, **{"c": arg3}, b=arg2) #$ arg1 arg2 arg3 func=with_multiple_kw_args MISSING:
with_multiple_kw_args(**{"b": arg2}, **{"c": arg3}, **{"a": arg1}) #$ arg1 arg2 arg3 func=with_multiple_kw_args
with_multiple_kw_args(arg1, **{"c": arg3}, b=arg2) #$ arg1 arg2 MISSING: arg3
with_multiple_kw_args(**{"b": arg2}, **{"c": arg3}, **{"a": arg1}) #$ MISSING: arg1 arg2 arg3
def with_default_arguments(a=arg1, b=arg2, c=arg3): #$ arg1 arg2 arg3 func=with_default_arguments
@@ -109,7 +109,7 @@ def test_default_arguments():
with_default_arguments()
with_default_arguments(arg1) #$ arg1
with_default_arguments(b=arg2) #$ arg2
with_default_arguments(**{"c": arg3}) #$ arg3 func=with_default_arguments
with_default_arguments(**{"c": arg3}) #$ MISSING: arg3
# Nested constructor pattern
@@ -135,7 +135,7 @@ def grab_baz(baz):
@expects(4)
def test_grab():
grab_foo_bar_baz(baz=arg3, bar=arg2, foo=arg1) #$ arg1 arg2 arg3 func=grab_bar_baz func=grab_baz
grab_foo_bar_baz(baz=arg3, bar=arg2, foo=arg1) #$ arg1 MISSING: arg2 func=grab_bar_baz arg3 func=grab_baz
# All combinations
@@ -158,7 +158,7 @@ def test_pos_star():
if len(a) > 0:
SINK1(a[0])
with_star(arg1) #$ arg1 func=test_pos_star.with_star
with_star(arg1) #$ MISSING: arg1 func=test_pos_star.with_star
def test_pos_kw():
@@ -186,4 +186,4 @@ def test_kw_doublestar():
def with_doublestar(**a):
SINK1(a["a"])
with_doublestar(a=arg1) #$ arg1 func=test_kw_doublestar.with_doublestar
with_doublestar(a=arg1) #$ MISSING: arg1 func=test_kw_doublestar.with_doublestar

64
python/ql/test/experimental/dataflow/coverage/classes.py
View File

@@ -560,9 +560,9 @@ class With_getitem:
def test_getitem():
with_getitem = With_getitem() #$ arg1="SSA variable with_getitem" func=With_getitem.__getitem__
with_getitem = With_getitem() #$ MISSING: arg1="SSA variable with_getitem" func=With_getitem.__getitem__
arg2 = 0
with_getitem[arg2] #$ arg2 func=With_getitem.__getitem__
with_getitem[arg2] #$ MISSING: arg2 func=With_getitem.__getitem__
# object.__setitem__(self, key, value)
@@ -575,10 +575,10 @@ class With_setitem:
def test_setitem():
with_setitem = With_setitem() #$ arg1="SSA variable with_setitem" func=With_setitem.__setitem__
with_setitem = With_setitem() #$ MISSING: arg1="SSA variable with_setitem" func=With_setitem.__setitem__
arg2 = 0
arg3 = ""
with_setitem[arg2] = arg3 #$ arg2 arg3 func=With_setitem.__setitem__
with_setitem[arg2] = arg3 #$ MISSING: arg2 arg3 func=With_setitem.__setitem__
# object.__delitem__(self, key)
@@ -590,9 +590,9 @@ class With_delitem:
def test_delitem():
with_delitem = With_delitem() #$ arg1="SSA variable with_delitem" func=With_delitem.__delitem__
with_delitem = With_delitem() #$ MISSING: arg1="SSA variable with_delitem" func=With_delitem.__delitem__
arg2 = 0
del with_delitem[arg2] #$ arg2 func=With_delitem.__delitem__
del with_delitem[arg2] #$ MISSING: arg2 func=With_delitem.__delitem__
# object.__missing__(self, key)
@@ -662,9 +662,9 @@ class With_add:
def test_add():
with_add = With_add() #$ arg1="SSA variable with_add" func=With_add.__add__
with_add = With_add() #$ MISSING: arg1="SSA variable with_add" func=With_add.__add__
arg2 = with_add
with_add + arg2 #$ arg2 func=With_add.__add__
with_add + arg2 #$ MISSING: arg2 func=With_add.__add__
# object.__sub__(self, other)
@@ -677,9 +677,9 @@ class With_sub:
def test_sub():
with_sub = With_sub() #$ arg1="SSA variable with_sub" func=With_sub.__sub__
with_sub = With_sub() #$ MISSING: arg1="SSA variable with_sub" func=With_sub.__sub__
arg2 = with_sub
with_sub - arg2 #$ arg2 func=With_sub.__sub__
with_sub - arg2 #$ MISSING: arg2 func=With_sub.__sub__
# object.__mul__(self, other)
@@ -692,9 +692,9 @@ class With_mul:
def test_mul():
with_mul = With_mul() #$ arg1="SSA variable with_mul" func=With_mul.__mul__
with_mul = With_mul() #$ MISSING: arg1="SSA variable with_mul" func=With_mul.__mul__
arg2 = with_mul
with_mul * arg2 #$ arg2 func=With_mul.__mul__
with_mul * arg2 #$ MISSING: arg2 func=With_mul.__mul__
# object.__matmul__(self, other)
@@ -707,9 +707,9 @@ class With_matmul:
def test_matmul():
with_matmul = With_matmul() #$ arg1="SSA variable with_matmul" func=With_matmul.__matmul__
with_matmul = With_matmul() #$ MISSING: arg1="SSA variable with_matmul" func=With_matmul.__matmul__
arg2 = with_matmul
with_matmul @ arg2 #$ arg2 func=With_matmul.__matmul__
with_matmul @ arg2 #$ MISSING: arg2 func=With_matmul.__matmul__
# object.__truediv__(self, other)
@@ -722,9 +722,9 @@ class With_truediv:
def test_truediv():
with_truediv = With_truediv() #$ arg1="SSA variable with_truediv" func=With_truediv.__truediv__
with_truediv = With_truediv() #$ MISSING: arg1="SSA variable with_truediv" func=With_truediv.__truediv__
arg2 = with_truediv
with_truediv / arg2 #$ arg2 func=With_truediv.__truediv__
with_truediv / arg2 #$ MISSING: arg2 func=With_truediv.__truediv__
# object.__floordiv__(self, other)
@@ -737,9 +737,9 @@ class With_floordiv:
def test_floordiv():
with_floordiv = With_floordiv() #$ arg1="SSA variable with_floordiv" func=With_floordiv.__floordiv__
with_floordiv = With_floordiv() #$ MISSING: arg1="SSA variable with_floordiv" func=With_floordiv.__floordiv__
arg2 = with_floordiv
with_floordiv // arg2 #$ arg2 func=With_floordiv.__floordiv__
with_floordiv // arg2 #$ MISSING: arg2 func=With_floordiv.__floordiv__
# object.__mod__(self, other)
@@ -752,9 +752,9 @@ class With_mod:
def test_mod():
with_mod = With_mod() #$ arg1="SSA variable with_mod" func=With_mod.__mod__
with_mod = With_mod() #$ MISSING: arg1="SSA variable with_mod" func=With_mod.__mod__
arg2 = with_mod
with_mod % arg2 #$ arg2 func=With_mod.__mod__
with_mod % arg2 #$ MISSING: arg2 func=With_mod.__mod__
# object.__divmod__(self, other)
@@ -788,9 +788,9 @@ def test_pow():
def test_pow_op():
with_pow = With_pow() #$ arg1="SSA variable with_pow" func=With_pow.__pow__
with_pow = With_pow() #$ MISSING: arg1="SSA variable with_pow" func=With_pow.__pow__
arg2 = with_pow
with_pow ** arg2 #$ arg2 func=With_pow.__pow__
with_pow ** arg2 #$ MISSING: arg2 func=With_pow.__pow__
# object.__lshift__(self, other)
@@ -803,9 +803,9 @@ class With_lshift:
def test_lshift():
with_lshift = With_lshift() #$ arg1="SSA variable with_lshift" func=With_lshift.__lshift__
with_lshift = With_lshift() #$ MISSING: arg1="SSA variable with_lshift" func=With_lshift.__lshift__
arg2 = with_lshift
with_lshift << arg2 #$ arg2 func=With_lshift.__lshift__
with_lshift << arg2 #$ MISSING: arg2 func=With_lshift.__lshift__
# object.__rshift__(self, other)
@@ -818,9 +818,9 @@ class With_rshift:
def test_rshift():
with_rshift = With_rshift() #$ arg1="SSA variable with_rshift" func=With_rshift.__rshift__
with_rshift = With_rshift() #$ MISSING: arg1="SSA variable with_rshift" func=With_rshift.__rshift__
arg2 = with_rshift
with_rshift >> arg2 #$ arg2 func=With_rshift.__rshift__
with_rshift >> arg2 #$ MISSING: arg2 func=With_rshift.__rshift__
# object.__and__(self, other)
@@ -833,9 +833,9 @@ class With_and:
def test_and():
with_and = With_and() #$ arg1="SSA variable with_and" func=With_and.__and__
with_and = With_and() #$ MISSING: arg1="SSA variable with_and" func=With_and.__and__
arg2 = with_and
with_and & arg2 #$ arg2 func=With_and.__and__
with_and & arg2 #$ MISSING: arg2 func=With_and.__and__
# object.__xor__(self, other)
@@ -848,9 +848,9 @@ class With_xor:
def test_xor():
with_xor = With_xor() #$ arg1="SSA variable with_xor" func=With_xor.__xor__
with_xor = With_xor() #$ MISSING: arg1="SSA variable with_xor" func=With_xor.__xor__
arg2 = with_xor
with_xor ^ arg2 #$ arg2 func=With_xor.__xor__
with_xor ^ arg2 #$ MISSING: arg2 func=With_xor.__xor__
# object.__or__(self, other)
@@ -863,9 +863,9 @@ class With_or:
def test_or():
with_or = With_or() #$ arg1="SSA variable with_or" func=With_or.__or__
with_or = With_or() #$ MISSING: arg1="SSA variable with_or" func=With_or.__or__
arg2 = with_or
with_or | arg2 #$ arg2 func=With_or.__or__
with_or | arg2 #$ MISSING: arg2 func=With_or.__or__
# object.__radd__(self, other)

47
python/ql/test/experimental/dataflow/coverage/datamodel.py
View File

@@ -124,6 +124,40 @@ def test_staticmethod_call():
C.staticmethod(arg1, arg2) # $ func=C.staticmethod arg1 arg2
# subclass
class SC(C):
pass
sc = SC()
@expects(6)
def test_subclass_method_call():
func_obj = sc.method.__func__
sc.method(arg1, arg2) # $ func=C.method arg1 arg2
SC.method(sc, arg1, arg2) # $ func=C.method arg1 arg2
func_obj(sc, arg1, arg2) # $ MISSING: func=C.method arg1 arg2
@expects(6)
def test_subclass_classmethod_call():
c_func_obj = SC.classmethod.__func__
sc.classmethod(arg1, arg2) # $ func=C.classmethod arg1 arg2
SC.classmethod(arg1, arg2) # $ func=C.classmethod arg1 arg2
c_func_obj(SC, arg1, arg2) # $ MISSING: func=C.classmethod arg1 arg2
@expects(5)
def test_subclass_staticmethod_call():
try:
SC.staticmethod.__func__
except AttributeError:
print("OK")
sc.staticmethod(arg1, arg2) # $ func=C.staticmethod arg1 arg2
SC.staticmethod(arg1, arg2) # $ func=C.staticmethod arg1 arg2
# Generator functions
# A function or method which uses the yield statement (see section The yield statement) is called a generator function. Such a function, when called, always returns an iterator object which can be used to execute the body of the function: calling the iterator’s iterator.__next__() method will cause the function to execute until it provides a value using the yield statement. When the function executes a return statement or falls off the end, a StopIteration exception is raised and the iterator will have reached the end of the set of values to be returned.
def gen(x, count):
@@ -198,5 +232,16 @@ class Customized:
customized = Customized()
SINK(Customized.a) #$ MISSING:flow="SOURCE, l:-8 -> customized.a"
SINK_F(Customized.b)
SINK(customized.a) #$ MISSING:flow="SOURCE, l:-10 -> customized.a"
SINK(customized.a) #$ flow="SOURCE, l:-10 -> customized.a"
SINK(customized.b) #$ flow="SOURCE, l:-7 -> customized.b"
class Test2:
def __init__(self, arg):
self.x = SOURCE
self.y = arg
t = Test2(SOURCE)
SINK(t.x) # $ flow="SOURCE, l:-4 -> t.x"
SINK(t.y) # $ flow="SOURCE, l:-2 -> t.y"

6
python/ql/test/experimental/dataflow/coverage/localFlow.expected
View File

@@ -8,10 +8,4 @@
| test.py:187:1:187:53 | GSSA Variable SINK | test.py:189:5:189:8 | ControlFlowNode for SINK |
| test.py:187:1:187:53 | GSSA Variable SOURCE | test.py:188:25:188:30 | ControlFlowNode for SOURCE |
| test.py:188:5:188:5 | SSA variable x | test.py:189:10:189:10 | ControlFlowNode for x |
| test.py:188:9:188:68 | ControlFlowNode for .0 | test.py:188:9:188:68 | SSA variable .0 |
| test.py:188:9:188:68 | ControlFlowNode for ListComp | test.py:188:5:188:5 | SSA variable x |
| test.py:188:9:188:68 | SSA variable .0 | test.py:188:9:188:68 | ControlFlowNode for .0 |
| test.py:188:16:188:16 | SSA variable v | test.py:188:45:188:45 | ControlFlowNode for v |
| test.py:188:40:188:40 | SSA variable u | test.py:188:56:188:56 | ControlFlowNode for u |
| test.py:188:51:188:51 | SSA variable z | test.py:188:67:188:67 | ControlFlowNode for z |
| test.py:188:62:188:62 | SSA variable y | test.py:188:10:188:10 | ControlFlowNode for y |

2
python/ql/test/experimental/dataflow/coverage/localFlow.ql
View File

@@ -4,5 +4,5 @@ import semmle.python.dataflow.new.DataFlow
from DataFlow::Node nodeFrom, DataFlow::Node nodeTo
where
DataFlow::localFlowStep(nodeFrom, nodeTo) and
nodeFrom.getEnclosingCallable().getName().matches("%\\_with\\_local\\_flow")
nodeFrom.getEnclosingCallable().getQualifiedName().matches("%\\_with\\_local\\_flow")
select nodeFrom, nodeTo

64
python/ql/test/experimental/dataflow/coverage/test.py
View File

@@ -393,7 +393,7 @@ def test_call_unpack_iterable():
def test_call_unpack_mapping():
SINK(second(NONSOURCE, **{"b": SOURCE})) #$ flow="SOURCE -> second(..)"
SINK(second(NONSOURCE, **{"b": SOURCE})) #$ MISSING: flow="SOURCE -> second(..)"
def f_extra_pos(a, *b):
@@ -401,7 +401,7 @@ def f_extra_pos(a, *b):
def test_call_extra_pos():
SINK(f_extra_pos(NONSOURCE, SOURCE)) #$ flow="SOURCE -> f_extra_pos(..)"
SINK(f_extra_pos(NONSOURCE, SOURCE)) #$ MISSING: flow="SOURCE -> f_extra_pos(..)"
def f_extra_keyword(a, **b):
@@ -409,7 +409,7 @@ def f_extra_keyword(a, **b):
def test_call_extra_keyword():
SINK(f_extra_keyword(NONSOURCE, b=SOURCE)) #$ flow="SOURCE -> f_extra_keyword(..)"
SINK(f_extra_keyword(NONSOURCE, b=SOURCE)) #$ MISSING: flow="SOURCE -> f_extra_keyword(..)"
# return the name of the first extra keyword argument
@@ -509,17 +509,17 @@ def test_lambda_unpack_mapping():
def second(a, b):
return b
SINK(second(NONSOURCE, **{"b": SOURCE})) #$ flow="SOURCE -> second(..)"
SINK(second(NONSOURCE, **{"b": SOURCE})) #$ MISSING: flow="SOURCE -> second(..)"
def test_lambda_extra_pos():
f_extra_pos = lambda a, *b: b[0]
SINK(f_extra_pos(NONSOURCE, SOURCE)) #$ flow="SOURCE -> f_extra_pos(..)"
SINK(f_extra_pos(NONSOURCE, SOURCE)) #$ MISSING: flow="SOURCE -> f_extra_pos(..)"
def test_lambda_extra_keyword():
f_extra_keyword = lambda a, **b: b["b"]
SINK(f_extra_keyword(NONSOURCE, b=SOURCE)) #$ flow="SOURCE -> f_extra_keyword(..)"
SINK(f_extra_keyword(NONSOURCE, b=SOURCE)) #$ MISSING: flow="SOURCE -> f_extra_keyword(..)"
# call the function with our source as the name of the keyword argument
@@ -689,7 +689,7 @@ def test_iterable_star_unpacking_in_for_2():
def iterate_star_args(first, second, *args):
for arg in args:
SINK(arg) #$ flow="SOURCE, l:+5 -> arg" flow="SOURCE, l:+6 -> arg"
SINK(arg) #$ MISSING: flow="SOURCE, l:+5 -> arg" flow="SOURCE, l:+6 -> arg"
# FP reported here: https://github.com/github/codeql-python-team/issues/49
@expects(2)
@@ -697,9 +697,16 @@ def test_overflow_iteration():
s = SOURCE
iterate_star_args(NONSOURCE, NONSOURCE, SOURCE, s)
@expects(6)
def test_deep_callgraph():
# port of python/ql/test/library-tests/taint/general/deep.py
# based on the fact that `test_deep_callgraph_defined_in_module` works the problem
# seems to be that we're defining these functions inside another function and that
# the flow of these function definitions DOESN'T flow into the body of the `f<n>`
# functions (they DO flow into the body of `test_deep_callgraph`, otherwise the
# `f1` call wouldn't work).
def f1(arg):
return arg
@@ -720,8 +727,51 @@ def test_deep_callgraph():
x = f6(SOURCE)
SINK(x) #$ MISSING:flow="SOURCE, l:-1 -> x"
x = f5(SOURCE)
SINK(x) #$ MISSING:flow="SOURCE, l:-1 -> x"
x = f4(SOURCE)
SINK(x) #$ MISSING:flow="SOURCE, l:-1 -> x"
x = f3(SOURCE)
SINK(x) #$ MISSING:flow="SOURCE, l:-1 -> x"
x = f2(SOURCE)
SINK(x) #$ MISSING:flow="SOURCE, l:-1 -> x"
x = f1(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
def wat_f1(arg):
return arg
def wat_f2(arg):
return wat_f1(arg)
def wat_f3(arg):
return wat_f2(arg)
def wat_f4(arg):
return wat_f3(arg)
def wat_f5(arg):
return wat_f4(arg)
def wat_f6(arg):
return wat_f5(arg)
@expects(6)
def test_deep_callgraph_defined_in_module():
x = wat_f6(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
x = wat_f5(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
x = wat_f4(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
x = wat_f3(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
x = wat_f2(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
x = wat_f1(SOURCE)
SINK(x) #$ flow="SOURCE, l:-1 -> x"
@expects(2)
def test_dynamic_tuple_creation_1():
tup = tuple()

49
python/ql/test/experimental/dataflow/fieldflow/test.py
View File

@@ -84,10 +84,10 @@ def test_indirect_assign_bound_method():
sf = myobj.setFoo
sf(SOURCE)
SINK(myobj.foo) # $ MISSING: flow="SOURCE, l:-1 -> myobj.foo"
SINK(myobj.foo) # $ flow="SOURCE, l:-1 -> myobj.foo"
sf(NONSOURCE)
SINK_F(myobj.foo)
SINK_F(myobj.foo) # $ SPURIOUS: flow="SOURCE, l:-4 -> myobj.foo"
@expects(3) # $ unresolved_call=expects(..) unresolved_call=expects(..)(..)
@@ -167,6 +167,17 @@ def fields_with_local_flow(x):
def test_fields():
SINK(fields_with_local_flow(SOURCE)) # $ flow="SOURCE -> fields_with_local_flow(..)"
def call_with_source(func):
func(SOURCE)
def test_bound_method_passed_as_arg():
myobj = MyObj(NONSOURCE)
call_with_source(myobj.setFoo)
SINK(myobj.foo) # $ MISSING: flow="SOURCE, l:-5 -> foo.x"
# ------------------------------------------------------------------------------
# Nested Object
# ------------------------------------------------------------------------------
@@ -244,6 +255,9 @@ class CrosstalkTestX:
def setvalue(self, value):
self.x = value
def do_nothing(self, value):
pass
class CrosstalkTestY:
def __init__(self):
@@ -295,10 +309,10 @@ def test_potential_crosstalk_different_name(cond=True):
func(SOURCE)
SINK(objx.x) # $ MISSING: flow="SOURCE, l:-2 -> objx.x"
SINK(objx.x) # $ flow="SOURCE, l:-2 -> objx.x"
SINK_F(objx.y)
SINK_F(objy.x)
SINK(objy.y, not_present_at_runtime=True) # $ MISSING: flow="SOURCE, l:-5 -> objy.y"
SINK(objy.y, not_present_at_runtime=True) # $ flow="SOURCE, l:-5 -> objy.y"
@expects(8) # $ unresolved_call=expects(..) unresolved_call=expects(..)(..)
@@ -318,10 +332,10 @@ def test_potential_crosstalk_same_name(cond=True):
func(SOURCE)
SINK(objx.x) # $ MISSING: flow="SOURCE, l:-2 -> objx.x"
SINK(objx.x) # $ flow="SOURCE, l:-2 -> objx.x"
SINK_F(objx.y)
SINK_F(objy.x)
SINK(objy.y, not_present_at_runtime=True) # $ MISSING: flow="SOURCE, l:-5 -> objy.y"
SINK(objy.y, not_present_at_runtime=True) # $ flow="SOURCE, l:-5 -> objy.y"
@expects(10) # $ unresolved_call=expects(..) unresolved_call=expects(..)(..)
@@ -350,6 +364,27 @@ def test_potential_crosstalk_same_name_object_reference(cond=True):
SINK(obj.y, not_present_at_runtime=True) # $ flow="SOURCE, l:-8 -> obj.y"
@expects(4) # $ unresolved_call=expects(..) unresolved_call=expects(..)(..)
def test_potential_crosstalk_same_class(cond=True):
objx1 = CrosstalkTestX()
SINK_F(objx1.x)
objx2 = CrosstalkTestX()
SINK_F(objx2.x)
if cond:
func = objx1.setvalue
else:
func = objx2.do_nothing
# We want to ensure that objx2.x does not end up getting tainted, since that would
# be cross-talk between the self arguments are their functions.
func(SOURCE)
SINK(objx1.x) # $ flow="SOURCE, l:-2 -> objx1.x"
SINK_F(objx2.x)
# ------------------------------------------------------------------------------
# Global scope
# ------------------------------------------------------------------------------
@@ -400,7 +435,7 @@ SINK(obj2.foo) # $ flow="SOURCE, l:-1 -> obj2.foo"
# apparently these if statements below makes a difference :O
# but one is not enough
cond = os.urandom(1)[0] > 128
cond = os.urandom(1)[0] > 128 # $ unresolved_call=os.urandom(..)
if cond:
pass

10
python/ql/test/experimental/dataflow/typetracking/test.py
View File

@@ -91,7 +91,7 @@ def unrelated_func():
return "foo"
def use_funcs_with_decorators():
x = get_tracked2() # $ MISSING: tracked
x = get_tracked2() # $ tracked
y = unrelated_func()
# ------------------------------------------------------------------------------
@@ -117,11 +117,11 @@ class Foo(object):
def meth1(self):
do_stuff(self)
def meth2(self): # $ MISSING: tracked_self
do_stuff(self) # $ MISSING: tracked_self
def meth2(self): # $ tracked_self
do_stuff(self) # $ tracked_self
def meth3(self): # $ MISSING: tracked_self
do_stuff(self) # $ MISSING: tracked_self
def meth3(self): # $ tracked_self
do_stuff(self) # $ tracked_self
class Bar(Foo):

1
python/ql/test/experimental/library-tests/CallGraph-implicit-init/InlineCallGraphTest.expected
View File

@@ -1,5 +1,4 @@
failures
debug_callableNotUnique
pointsTo_found_typeTracker_notFound
| example.py:22:1:22:16 | ControlFlowNode for explicit_afunc() | explicit_afunc |
typeTracker_found_pointsTo_notFound

2
python/ql/test/experimental/library-tests/CallGraph-implicit-init/example.py
View File

@@ -19,4 +19,4 @@ from foo_explicit.bar.a import explicit_afunc
afunc() # $ MISSING: pt,tt=afunc
explicit_afunc() # $ pt=explicit_afunc MISSING: tt=explicit_afunc
explicit_afunc() # $ pt,tt="foo_explicit/bar/a.py:explicit_afunc"

5
python/ql/test/experimental/library-tests/CallGraph-imports/InlineCallGraphTest.expected Normal file
View File

@@ -0,0 +1,5 @@
failures
debug_callableNotUnique
pointsTo_found_typeTracker_notFound
| pkg/use.py:10:5:10:10 | ControlFlowNode for func() | func |
typeTracker_found_pointsTo_notFound

1
python/ql/test/experimental/library-tests/CallGraph-imports/InlineCallGraphTest.qlref Normal file
View File

@@ -0,0 +1 @@
../CallGraph/InlineCallGraphTest.ql

5
python/ql/test/experimental/library-tests/CallGraph-imports/README.md Normal file
View File

@@ -0,0 +1,5 @@
A testcase observed in real code, where mixing `from .this import that` with `from .other import *` (in that order) causes import resolution to not work properly.
This needs to be in a separate folder, since using relative imports requires a valid top-level package. We emulate real extractor behavior using `-R` extractor option.
From this directory, you can run the code with `python -m pkg.use`.

1
python/ql/test/experimental/library-tests/CallGraph-imports/options Normal file
View File

@@ -0,0 +1 @@
semmle-extractor-options: --max-import-depth=1 -R ./pkg/

0
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/__init__.py Normal file
View File

1
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/alias_only_direct.py Normal file
View File

@@ -0,0 +1 @@
from .func_def import func

2
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/alias_problem.py Normal file
View File

@@ -0,0 +1,2 @@
from .func_def import func
from .other import *

3
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/alias_problem_fixed.py Normal file
View File

@@ -0,0 +1,3 @@
# this ordering makes the problem go away
from .other import *
from .func_def import func

2
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/alias_star.py Normal file
View File

@@ -0,0 +1,2 @@
from .func_def import *
from .other import *

2
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/func_def.py Normal file
View File

@@ -0,0 +1,2 @@
def func():
print("func")

2
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/other.py Normal file
View File

@@ -0,0 +1,2 @@
def something():
pass

33
python/ql/test/experimental/library-tests/CallGraph-imports/pkg/use.py Normal file
View File

@@ -0,0 +1,33 @@
def test_direct_import():
from .func_def import func
func() # $ pt,tt="pkg/func_def.py:func"
test_direct_import() # $ pt,tt=test_direct_import
def test_alias_problem():
from .alias_problem import func
func() # $ pt="pkg/func_def.py:func" MISSING: tt="pkg/func_def.py:func"
test_alias_problem() # $ pt,tt=test_alias_problem
def test_alias_problem_fixed():
from .alias_problem_fixed import func
func() # $ pt,tt="pkg/func_def.py:func"
test_alias_problem_fixed() # $ pt,tt=test_alias_problem_fixed
def test_alias_star():
from .alias_star import func
func() # $ pt,tt="pkg/func_def.py:func"
test_alias_star() # $ pt,tt=test_alias_star
def test_alias_only_direct():
from .alias_only_direct import func
func() # $ pt,tt="pkg/func_def.py:func"
test_alias_only_direct() # $ pt,tt=test_alias_only_direct

53
python/ql/test/experimental/library-tests/CallGraph/InlineCallGraphTest.expected
View File

@@ -1,22 +1,39 @@
failures
debug_callableNotUnique
| code/class_advanced.py:18:5:18:18 | Function arg | Qualified function name 'B.arg' is not unique. Please fix. |
| code/class_advanced.py:23:5:23:25 | Function arg | Qualified function name 'B.arg' is not unique. Please fix. |
| code/class_properties.py:7:5:7:18 | Function arg | Qualified function name 'Prop.arg' is not unique within its file. Please fix. |
| code/class_properties.py:12:5:12:25 | Function arg | Qualified function name 'Prop.arg' is not unique within its file. Please fix. |
| code/class_properties.py:17:5:17:18 | Function arg | Qualified function name 'Prop.arg' is not unique within its file. Please fix. |
pointsTo_found_typeTracker_notFound
| code/class_simple.py:24:1:24:15 | ControlFlowNode for Attribute() | A.some_method |
| code/class_simple.py:25:1:25:21 | ControlFlowNode for Attribute() | A.some_staticmethod |
| code/class_simple.py:26:1:26:20 | ControlFlowNode for Attribute() | A.some_classmethod |
| code/class_simple.py:28:1:28:21 | ControlFlowNode for Attribute() | A.some_staticmethod |
| code/class_simple.py:29:1:29:20 | ControlFlowNode for Attribute() | A.some_classmethod |
| code/runtime_decision.py:18:1:18:6 | ControlFlowNode for func() | rd_bar |
| code/runtime_decision.py:18:1:18:6 | ControlFlowNode for func() | rd_foo |
| code/runtime_decision.py:26:1:26:7 | ControlFlowNode for func2() | rd_bar |
| code/runtime_decision.py:26:1:26:7 | ControlFlowNode for func2() | rd_foo |
| code/simple.py:15:1:15:5 | ControlFlowNode for foo() | foo |
| code/simple.py:16:1:16:14 | ControlFlowNode for indirect_foo() | foo |
| code/simple.py:17:1:17:5 | ControlFlowNode for bar() | bar |
| code/simple.py:18:1:18:5 | ControlFlowNode for lam() | lambda[simple.py:12:7] |
| code/underscore_prefix_func_name.py:18:5:18:19 | ControlFlowNode for some_function() | some_function |
| code/underscore_prefix_func_name.py:21:5:21:19 | ControlFlowNode for some_function() | some_function |
| code/underscore_prefix_func_name.py:24:1:24:21 | ControlFlowNode for _works_since_called() | _works_since_called |
| code/class_attr_assign.py:10:9:10:27 | ControlFlowNode for Attribute() | my_func |
| code/class_attr_assign.py:11:9:11:25 | ControlFlowNode for Attribute() | my_func |
| code/class_attr_assign.py:26:9:26:25 | ControlFlowNode for Attribute() | DummyObject.method |
| code/class_super.py:50:1:50:6 | ControlFlowNode for Attribute() | outside_def |
| code/conditional_in_argument.py:18:5:18:11 | ControlFlowNode for Attribute() | X.bar |
| code/func_defined_outside_class.py:21:1:21:11 | ControlFlowNode for Attribute() | A.foo |
| code/func_defined_outside_class.py:22:1:22:15 | ControlFlowNode for Attribute() | outside |
| code/func_defined_outside_class.py:24:1:24:14 | ControlFlowNode for Attribute() | outside_sm |
| code/func_defined_outside_class.py:25:1:25:14 | ControlFlowNode for Attribute() | outside_cm |
| code/func_defined_outside_class.py:38:11:38:21 | ControlFlowNode for _gen() | B._gen |
| code/func_defined_outside_class.py:39:11:39:21 | ControlFlowNode for _gen() | B._gen |
| code/func_defined_outside_class.py:42:1:42:7 | ControlFlowNode for Attribute() | B._gen.func |
| code/func_defined_outside_class.py:43:1:43:7 | ControlFlowNode for Attribute() | B._gen.func |
| code/type_tracking_limitation.py:8:1:8:3 | ControlFlowNode for x() | my_func |
typeTracker_found_pointsTo_notFound
| code/callable_as_argument.py:29:5:29:12 | ControlFlowNode for Attribute() | test_class.InsideTestFunc.sm |
| code/class_more_mro2.py:18:9:18:21 | ControlFlowNode for Attribute() | A.foo |
| code/class_more_mro2.py:21:1:21:8 | ControlFlowNode for Attribute() | A.foo |
| code/class_more_mro.py:24:9:24:21 | ControlFlowNode for Attribute() | A.foo |
| code/class_more_mro.py:34:1:34:16 | ControlFlowNode for Attribute() | A.foo |
| code/class_super.py:43:9:43:21 | ControlFlowNode for Attribute() | A.bar |
| code/class_super.py:44:9:44:27 | ControlFlowNode for Attribute() | A.bar |
| code/class_super.py:63:1:63:18 | ControlFlowNode for Attribute() | A.foo |
| code/class_super.py:78:9:78:28 | ControlFlowNode for Attribute() | A.foo |
| code/class_super.py:81:1:81:12 | ControlFlowNode for Attribute() | C.foo_on_A |
| code/class_super.py:92:9:92:21 | ControlFlowNode for Attribute() | X.foo |
| code/class_super.py:97:9:97:21 | ControlFlowNode for Attribute() | X.foo |
| code/class_super.py:97:9:97:21 | ControlFlowNode for Attribute() | Y.foo |
| code/class_super.py:101:1:101:7 | ControlFlowNode for Attribute() | Z.foo |
| code/class_super.py:108:1:108:8 | ControlFlowNode for Attribute() | Z.foo |
| code/def_in_function.py:22:5:22:11 | ControlFlowNode for Attribute() | test.A.foo |
| code/nested_class.py:83:9:83:16 | ControlFlowNode for Attribute() | X.class_def_in_func.Y.meth |
| code/underscore_prefix_func_name.py:14:5:14:19 | ControlFlowNode for some_function() | some_function |

45
python/ql/test/experimental/library-tests/CallGraph/InlineCallGraphTest.ql
View File

@@ -4,6 +4,8 @@ private import semmle.python.dataflow.new.internal.DataFlowDispatch as TT
/** Holds when `call` is resolved to `callable` using points-to based call-graph. */
predicate pointsToCallEdge(CallNode call, Function callable) {
exists(call.getLocation().getFile().getRelativePath()) and
exists(callable.getLocation().getFile().getRelativePath()) and
exists(PythonFunctionValue funcValue |
funcValue.getScope() = callable and
call = funcValue.getACall()
@@ -12,6 +14,8 @@ predicate pointsToCallEdge(CallNode call, Function callable) {
/** Holds when `call` is resolved to `callable` using type-tracking based call-graph. */
predicate typeTrackerCallEdge(CallNode call, Function callable) {
exists(call.getLocation().getFile().getRelativePath()) and
exists(callable.getLocation().getFile().getRelativePath()) and
exists(TT::DataFlowCallable dfCallable, TT::DataFlowCall dfCall |
dfCallable.getScope() = callable and
dfCall.getNode() = call and
@@ -19,6 +23,16 @@ predicate typeTrackerCallEdge(CallNode call, Function callable) {
)
}
/** Holds if the call edge is from a class call. */
predicate typeTrackerClassCall(CallNode call, Function callable) {
exists(call.getLocation().getFile().getRelativePath()) and
exists(callable.getLocation().getFile().getRelativePath()) and
exists(TT::NormalCall cc |
cc = TT::TNormalCall(call, _, any(TT::TCallType t | t instanceof TT::CallTypeClass)) and
TT::TFunction(callable) = TT::viableCallable(cc)
)
}
class CallGraphTest extends InlineExpectationsTest {
CallGraphTest() { this = "CallGraphTest" }
@@ -35,7 +49,20 @@ class CallGraphTest extends InlineExpectationsTest {
|
location = call.getLocation() and
element = call.toString() and
value = betterQualName(target)
if call.getLocation().getFile() = target.getLocation().getFile()
then value = betterQualName(target)
else
exists(string fixedRelativePath |
fixedRelativePath =
target
.getLocation()
.getFile()
.getRelativePath()
.regexpCapture(".*/CallGraph[^/]*/(.*)", 1)
|
// the value needs to be enclosed in quotes to allow special characters
value = "\"" + fixedRelativePath + ":" + betterQualName(target) + "\""
)
)
}
}
@@ -53,9 +80,15 @@ string betterQualName(Function func) {
}
query predicate debug_callableNotUnique(Function callable, string message) {
exists(Function f | f != callable and f.getQualifiedName() = callable.getQualifiedName()) and
exists(callable.getLocation().getFile().getRelativePath()) and
exists(Function f |
f != callable and
f.getQualifiedName() = callable.getQualifiedName() and
f.getLocation().getFile() = callable.getLocation().getFile()
) and
message =
"Qualified function name '" + callable.getQualifiedName() + "' is not unique. Please fix."
"Qualified function name '" + callable.getQualifiedName() +
"' is not unique within its file. Please fix."
}
query predicate pointsTo_found_typeTracker_notFound(CallNode call, string qualname) {
@@ -70,6 +103,10 @@ query predicate typeTracker_found_pointsTo_notFound(CallNode call, string qualna
exists(Function target |
not pointsToCallEdge(call, target) and
typeTrackerCallEdge(call, target) and
qualname = betterQualName(target)
qualname = betterQualName(target) and
// We filter out result differences for points-to and type-tracking for class calls,
// since otherwise it gives too much noise (these are just handled differently
// between the two).
not typeTrackerClassCall(call, target)
)
}

1
python/ql/test/experimental/library-tests/CallGraph/code/aliased_import.py Normal file
View File

@@ -0,0 +1 @@
from .simple import foo

16
python/ql/test/experimental/library-tests/CallGraph/code/bound_method_arg.py Normal file
View File

@@ -0,0 +1,16 @@
class Foo(object):
def meth(self, arg):
print("Foo.meth", arg)
@classmethod
def cm(cls, arg):
print("Foo.cm", arg)
def call_func(func):
func(42) # $ pt,tt=Foo.meth pt,tt=Foo.cm
foo = Foo()
call_func(foo.meth) # $ pt,tt=call_func
call_func(Foo.cm) # $ pt,tt=call_func

55
python/ql/test/experimental/library-tests/CallGraph/code/callable_as_argument.py Normal file
View File

@@ -0,0 +1,55 @@
# ==============================================================================
# function
# ==============================================================================
def call_func(f):
f() # $ pt,tt=my_func pt,tt=test_func.inside_test_func
def my_func():
print("my_func")
call_func(my_func) # $ pt,tt=call_func
def test_func():
def inside_test_func():
print("inside_test_func")
call_func(inside_test_func) # $ pt,tt=call_func
test_func() # $ pt,tt=test_func
# ==============================================================================
# class
# ==============================================================================
def class_func(cls):
cls.sm() # $ pt,tt=MyClass.sm tt=test_class.InsideTestFunc.sm
cls(42) # $ tt=MyClass.__init__ tt=test_class.InsideTestFunc.__init__
class MyClass(object):
def __init__(self, arg):
print(self, arg)
@staticmethod
def sm():
print("MyClass.staticmethod")
class_func(MyClass) # $ pt,tt=class_func
def test_class():
class InsideTestFunc(object):
def __init__(self, arg):
print(self, arg)
@staticmethod
def sm():
print("InsideTestFunc.staticmethod")
class_func(InsideTestFunc) # $ pt,tt=class_func
test_class() # $ pt,tt=test_class

40
python/ql/test/experimental/library-tests/CallGraph/code/class_advanced.py
View File

@@ -1,40 +0,0 @@
class B(object):
def __init__(self, arg):
print('B.__init__', arg)
self._arg = arg
def __str__(self):
print('B.__str__')
return 'B (arg={})'.format(self.arg)
def __add__(self, other):
print('B.__add__')
if isinstance(other, B):
return B(self.arg + other.arg)
return B(self.arg + other)
@property
def arg(self):
print('B.arg getter')
return self._arg
@arg.setter
def arg(self, value):
print('B.arg setter')
self._arg = value
b1 = B(1)
b2 = B(2)
b3 = b1 + b2
print('value printing:', str(b1))
print('value printing:', str(b2))
print('value printing:', str(b3))
b3.arg = 42
b4 = b3 + 100
# this calls `str(b4)` inside
print('value printing:', b4)

30
python/ql/test/experimental/library-tests/CallGraph/code/class_attr_assign.py Normal file
View File

@@ -0,0 +1,30 @@
def my_func():
print("my_func")
class Foo(object):
def __init__(self, func):
self.indirect_ref = func
self.direct_ref = my_func
def later(self):
self.indirect_ref() # $ pt=my_func MISSING: tt=my_func
self.direct_ref() # $ pt=my_func MISSING: tt=my_func
foo = Foo(my_func) # $ tt=Foo.__init__
foo.later() # $ pt,tt=Foo.later
class DummyObject(object):
def method(self):
print("DummyObject.method")
class Bar(object):
def __init__(self):
self.obj = DummyObject()
def later(self):
self.obj.method() # $ pt=DummyObject.method MISSING: tt=DummyObject.method
bar = Bar(my_func) # $ tt=Bar.__init__
bar.later() # $ pt,tt=Bar.later

66
python/ql/test/experimental/library-tests/CallGraph/code/class_construction.py Normal file
View File

@@ -0,0 +1,66 @@
class X(object):
def __init__(self, arg):
print("X.__init__", arg)
X(42) # $ tt=X.__init__
print()
class Y(X):
def __init__(self, arg):
print("Y.__init__", arg)
super().__init__(-arg) # $ pt,tt=X.__init__
Y(43) # $ tt=Y.__init__
print()
# ---
class WithNew(object):
def __new__(cls, arg):
print("WithNew.__new__", arg)
inst = super().__new__(cls)
assert isinstance(inst, cls)
inst.some_method() # $ MISSING: pt,tt=WithNew.some_method
return inst
def __init__(self, arg):
print("WithNew.__init__", arg)
def some_method(self):
print("WithNew.__init__")
WithNew(44) # $ tt=WithNew.__new__ tt=WithNew.__init__
print()
class ExtraCallToInit(object):
def __new__(cls, arg):
print("ExtraCallToInit.__new__", arg)
inst = super().__new__(cls)
assert isinstance(inst, cls)
# you're not supposed to do this, since it will cause the __init__ method will be run twice.
inst.__init__(1001) # $ MISSING: pt,tt=ExtraCallToInit.__init__
return inst
def __init__(self, arg):
print("ExtraCallToInit.__init__", arg, self)
ExtraCallToInit(1000) # $ tt=ExtraCallToInit.__new__ tt=ExtraCallToInit.__init__
print()
class InitNotCalled(object):
"""as described in https://docs.python.org/3/reference/datamodel.html#object.__new__
__init__ will only be called when the returned object from __new__ is an instance of
the `cls` parameter...
"""
def __new__(cls, arg):
print("InitNotCalled.__new__", arg)
return False
def __init__(self, arg):
print("InitNotCalled.__init__", arg)
InitNotCalled(2000) # $ tt=InitNotCalled.__new__ SPURIOUS: tt=InitNotCalled.__init__
print()

34
python/ql/test/experimental/library-tests/CallGraph/code/class_decorator.py Normal file
View File

@@ -0,0 +1,34 @@
# decorated class
def my_class_decorator(cls):
print("dummy decorator")
return cls
@my_class_decorator # $ pt=my_class_decorator tt=my_class_decorator
class A(object):
def foo(self):
pass
a = A()
a.foo() # $ pt,tt=A.foo
class B(A):
def bar(self):
self.foo() # $ pt,tt=A.foo
# decorated class, unknown decorator
from some_unknown_module import unknown_class_decorator
@unknown_class_decorator
class X(object):
def foo(self):
pass
x = X()
x.foo() # $ pt,tt=X.foo
class Y(X):
def bar(self):
self.foo() # $ pt,tt=X.foo

35
python/ql/test/experimental/library-tests/CallGraph/code/class_more_mro.py Normal file
View File

@@ -0,0 +1,35 @@
class Base(object):
def foo(self):
print("Base.foo")
class A(Base):
def foo(self):
print("A.foo")
super().foo() # $ pt,tt=Base.foo
class ASub(A):
pass
class B(Base):
def foo(self):
print("B.foo")
# NOTE: If this missing result is fixed, please update the QLDoc for
# `getNextClassInMro` as well
super().foo() # $ pt,tt=Base.foo MISSING: pt,tt=A.foo
class BSub(B):
def bar(self):
print("BSub.bar")
super().foo() # $ pt,tt=B.foo SPURIOUS: tt=A.foo
bs = BSub()
bs.foo() # $ pt,tt=B.foo
bs.bar() # $ pt,tt=BSub.bar
print("! Indirect")
class Indirect(BSub, ASub):
pass
Indirect().foo() # $ pt,tt=B.foo SPURIOUS: tt=A.foo
Indirect().bar() # $ pt,tt=BSub.bar

22
python/ql/test/experimental/library-tests/CallGraph/code/class_more_mro2.py Normal file
View File

@@ -0,0 +1,22 @@
class A(object):
def foo(self):
print("A.foo")
class B(A):
pass
b = B()
b.foo() # $ pt,tt=A.foo
class C(A):
def foo(self):
print("C.foo")
class BC(B, C):
def bar(self):
print("BC.bar")
super().foo() # $ pt,tt=C.foo SPURIOUS: tt=A.foo
bc = BC()
bc.foo() # $ pt,tt=C.foo SPURIOUS: tt=A.foo
bc.bar() # $ pt,tt=BC.bar

43
python/ql/test/experimental/library-tests/CallGraph/code/class_properties.py Normal file
View File

@@ -0,0 +1,43 @@
class Prop(object):
def __init__(self, arg):
self._arg = arg
self._arg2 = arg
@property
def arg(self):
print('Prop.arg getter')
return self._arg
@arg.setter
def arg(self, value):
print('Prop.arg setter')
self._arg = value
@arg.deleter
def arg(self):
print('Prop.arg deleter')
# haha, you cannot delete me!
def _arg2_getter(self):
print('Prop.arg2 getter')
return self._arg2
def _arg2_setter(self, value):
print('Prop.arg2 setter')
self._arg2 = value
def _arg2_deleter(self):
print('Prop.arg2 deleter')
# haha, you cannot delete me!
arg2 = property(_arg2_getter, _arg2_setter, _arg2_deleter)
prop = Prop(42) # $ tt=Prop.__init__
prop.arg
prop.arg = 43
del prop.arg
prop.arg2
prop.arg2 = 43
del prop.arg2

29
python/ql/test/experimental/library-tests/CallGraph/code/class_simple.py
View File

@@ -1,29 +0,0 @@
class A(object):
def __init__(self, arg):
print('A.__init__', arg)
self.arg = arg
def some_method(self):
print('A.some_method', self)
@staticmethod
def some_staticmethod():
print('A.some_staticmethod')
@classmethod
def some_classmethod(cls):
print('A.some_classmethod', cls)
# TODO: Figure out how to annotate class instantiation (and add one here).
# Current points-to says it's a call to the class (instead of __init__/__new__/metaclass-something).
# However, current test setup uses "callable" for naming, and expects things to be Function.
a = A(42)
a.some_method() # $ pt=A.some_method
a.some_staticmethod() # $ pt=A.some_staticmethod
a.some_classmethod() # $ pt=A.some_classmethod
A.some_staticmethod() # $ pt=A.some_staticmethod
A.some_classmethod() # $ pt=A.some_classmethod

29
python/ql/test/experimental/library-tests/CallGraph/code/class_special_methods.py Normal file
View File

@@ -0,0 +1,29 @@
class B(object):
def __init__(self, arg):
print('B.__init__', arg)
self.arg = arg
def __str__(self):
print('B.__str__')
return 'B (arg={})'.format(self.arg)
def __add__(self, other):
print('B.__add__')
if isinstance(other, B):
return B(self.arg + other.arg) # $ tt=B.__init__
return B(self.arg + other) # $ tt=B.__init__
b = B(1) # $ tt=B.__init__
print(str(b))
# this calls `str(b)` inside
print(b)
b2 = B(2) # $ tt=B.__init__
# __add__ is called
b + b2
b + 100

178
python/ql/test/experimental/library-tests/CallGraph/code/class_subclass.py Normal file
View File

@@ -0,0 +1,178 @@
class A(object):
def __init__(self, arg):
print('A.__init__', arg)
self.arg = arg
def some_method(self):
print('A.some_method', self)
@staticmethod
def some_staticmethod():
print('A.some_staticmethod')
@classmethod
def some_classmethod(cls):
print('A.some_classmethod', cls)
a = A(42) # $ tt=A.__init__
a.some_method() # $ pt,tt=A.some_method
a.some_staticmethod() # $ pt,tt=A.some_staticmethod
a.some_classmethod() # $ pt,tt=A.some_classmethod
A.some_method(a) # $ pt,tt=A.some_method
A.some_staticmethod() # $ pt,tt=A.some_staticmethod
A.some_classmethod() # $ pt,tt=A.some_classmethod
print("- type()")
type(a).some_method(a) # $ pt,tt=A.some_method
type(a).some_staticmethod() # $ pt,tt=A.some_staticmethod
type(a).some_classmethod() # $ pt,tt=A.some_classmethod
# Subclass test
print("\n! B")
class B(A):
pass
b = B(42) # $ tt=A.__init__
b.some_method() # $ pt,tt=A.some_method
b.some_staticmethod() # $ pt,tt=A.some_staticmethod
b.some_classmethod() # $ pt,tt=A.some_classmethod
B.some_method(b) # $ pt,tt=A.some_method
B.some_staticmethod() # $ pt,tt=A.some_staticmethod
B.some_classmethod() # $ pt,tt=A.some_classmethod
print("- type()")
type(b).some_method(b) # $ pt,tt=A.some_method
type(b).some_staticmethod() # $ pt,tt=A.some_staticmethod
type(b).some_classmethod() # $ pt,tt=A.some_classmethod
# Subclass with method override
print("\n! Subclass with method override")
class C(A):
def some_method(self):
print('C.some_method', self)
c = C(42) # $ tt=A.__init__
c.some_method() # $ pt,tt=C.some_method
class D(object):
def some_method(self):
print('D.some_method', self)
class E(C, D):
pass
e = E(42) # $ tt=A.__init__
e.some_method() # $ pt,tt=C.some_method
class F(D, C):
pass
f = F(42) # $ tt=A.__init__
f.some_method() # $ pt,tt=D.some_method
# ------------------------------------------------------------------------------
# self/cls in methods
# ------------------------------------------------------------------------------
class Base(object):
def foo(self):
print('Base.foo')
def bar(self):
print('Base.bar')
def call_stuff(self):
self.foo() # $ pt,tt=Base.foo pt,tt=Sub.foo pt,tt=Mixin.foo
self.bar() # $ pt,tt=Base.bar
self.sm() # $ pt,tt=Base.sm
self.cm() # $ pt,tt=Base.cm
self.sm2() # $ pt,tt=Base.sm2 pt,tt=Sub.sm2
self.cm2() # $ pt,tt=Base.cm2 pt,tt=Sub.cm2
type(self).sm2() # $ pt,tt=Base.sm2 pt,tt=Sub.sm2
type(self).cm2() # $ pt,tt=Base.cm2 pt,tt=Sub.cm2
@staticmethod
def sm():
print("Base.sm")
@classmethod
def cm(cls):
print("Base.cm")
@staticmethod
def sm2():
print("Base.sm2")
@classmethod
def cm2(cls):
print("Base.cm2")
@classmethod
def call_from_cm(cls):
cls.sm() # $ pt,tt=Base.sm
cls.cm() # $ pt,tt=Base.cm
cls.sm2() # $ pt,tt=Base.sm2 pt,tt=Sub.sm2
cls.cm2() # $ pt,tt=Base.cm2 pt,tt=Sub.cm2
base = Base()
print("! base.call_stuff()")
base.call_stuff() # $ pt,tt=Base.call_stuff
print("! Base.call_from_cm()")
Base.call_from_cm() # $ pt,tt=Base.call_from_cm
class Sub(Base):
def foo(self):
print("Sub.foo")
def foo_on_super(self):
sup = super()
sup.foo() # $ pt,tt=Base.foo
def also_call_stuff(self):
self.sm() # $ pt,tt=Base.sm
self.cm() # $ pt,tt=Base.cm
self.sm2() # $ pt,tt=Sub.sm2
self.cm2() # $ pt,tt=Sub.cm2
@staticmethod
def sm2():
print("Sub.sm2")
@classmethod
def cm2(cls):
print("Sub.cm2")
sub = Sub()
print("! sub.foo_on_super()")
sub.foo_on_super() # $ pt,tt=Sub.foo_on_super
print("! sub.call_stuff()")
sub.call_stuff() # $ pt,tt=Base.call_stuff
print("! sub.also_call_stuff()")
sub.also_call_stuff() # $ pt,tt=Sub.also_call_stuff
print("! Sub.call_from_cm()")
Sub.call_from_cm() # $ pt,tt=Base.call_from_cm
class Mixin(object):
def foo(self):
print("Mixin.foo")
class SubWithMixin(Mixin, Base):
# the ordering here means that in Base.call_stuff, the call to self.foo will go to Mixin.foo
pass
swm = SubWithMixin()
print("! swm.call_stuff()")
swm.call_stuff() # $ pt,tt=Base.call_stuff

38
python/ql/test/experimental/library-tests/CallGraph/code/class_subclass2.py Normal file
View File

@@ -0,0 +1,38 @@
class Base(object):
def foo(self):
print("Base.foo")
def call_stuff(self):
print("Base.call_stuff")
self.foo() # $ pt,tt=Base.foo pt,tt=X.foo
class X(object):
def __init__(self):
print("X.__init__")
def foo(self):
print("X.foo")
class Y(object):
def __init__(self):
print("Y.__init__")
def foo(self):
print("Y.foo")
class Contrived(X, Y, Base):
pass
contrived = Contrived() # $ tt=X.__init__
contrived.foo() # $ pt,tt=X.foo
contrived.call_stuff() # $ pt,tt=Base.call_stuff
# Ensure that we don't mix up __init__ resolution for Contrived() due to MRO
# approximation
class HasInit(object):
def __init__(self):
pass
class TryingToTrickYou(Contrived, HasInit):
pass

108
python/ql/test/experimental/library-tests/CallGraph/code/class_super.py Normal file
View File

@@ -0,0 +1,108 @@
def outside_def(self):
print("outside_def")
try:
super().foo()
except RuntimeError:
pass
class A(object):
def foo(self):
print("A.foo")
@classmethod
def bar(cls):
print("A.bar")
class B(A):
def foo(self):
print("B.foo")
def foo_on_super(self):
print("B.foo_on_super")
super().foo() # $ pt,tt=A.foo
super(B, self).foo() # $ pt,tt=A.foo
od = outside_def
@staticmethod
def sm():
try:
super().foo()
except RuntimeError:
print("B.sm")
pass
@classmethod
def bar(cls):
print("B.bar")
@classmethod
def bar_on_super(cls):
print("B.bar_on_super")
super().bar() # $ tt=A.bar
super(B, cls).bar() # $ tt=A.bar
b = B()
b.foo() # $ pt,tt=B.foo
b.foo_on_super() # $ pt,tt=B.foo_on_super
b.od() # $ pt=outside_def
b.sm() # $ pt,tt=B.sm
print("="*10, "static method")
B.bar() # $ pt,tt=B.bar
B.bar_on_super() # $ pt,tt=B.bar_on_super
print("="*10, "Manual calls to super")
super(B, b).foo() # $ pt,tt=A.foo
assert A.foo == super(B, B).foo
super(B, B).foo(b) # $ tt=A.foo
try:
super(B, 42).foo()
except TypeError:
pass
# For some reason, points-to isn't able to resolve any calls from here on. I've tried to
# comment out both try-except blocks, but that did not solve the problem :|
print("="*10, "C")
class C(B):
def foo_on_A(self):
print('C.foo_on_A')
super(B, self).foo() # $ tt=A.foo
c = C()
c.foo_on_A() # $ tt=C.foo_on_A
print("="*10, "Diamon hierachy")
class X(object):
def foo(self):
print('X.foo')
class Y(X):
def foo(self):
print('Y.foo')
super().foo() # $ tt=X.foo
class Z(X):
def foo(self):
print('Z.foo')
super().foo() # $ tt=X.foo tt=Y.foo
print("! z.foo()")
z = Z()
z.foo() # $ tt=Z.foo
class ZY(Z, Y):
pass
print("! zy.foo()")
zy = ZY()
zy.foo() # $ tt=Z.foo

36
python/ql/test/experimental/library-tests/CallGraph/code/conditional_in_argument.py Normal file
View File

@@ -0,0 +1,36 @@
class X(object):
def foo(self, *args):
print("X.foo", args)
def bar(self, *args):
print("X.bar", args)
def func(cond=True):
x = X()
# ok
x.foo() # $ pt,tt=X.foo
x.bar() # $ pt,tt=X.bar
# the conditional in the argument makes us stop tracking the class instance :|
x.foo(1 if cond else 0) # $ pt,tt=X.foo
x.bar() # $ pt=X.bar MISSING: tt=X.bar
func() # $ pt,tt=func
def func2(cond=True):
y = X()
# ok
y.foo() # $ pt,tt=X.foo
y.bar() # $ pt,tt=X.bar
if cond:
arg = 1
else:
arg = 0
y.foo(arg) # $ pt,tt=X.foo
y.bar() # $ pt,tt=X.bar

24
python/ql/test/experimental/library-tests/CallGraph/code/def_in_function.py Normal file
View File

@@ -0,0 +1,24 @@
def test():
def foo():
print("foo")
foo() # $ pt,tt=test.foo
def bar():
print("bar")
def baz():
print("baz")
baz() # $ pt,tt=test.bar.baz
return baz
baz_ref = bar() # $ pt,tt=test.bar
baz_ref() # $ pt,tt=test.bar.baz
class A(object):
def foo(self):
print("A.foo")
a = A()
a.foo() # $ tt=test.A.foo
test() # $ pt,tt=test

43
python/ql/test/experimental/library-tests/CallGraph/code/func_defined_outside_class.py Normal file
View File

@@ -0,0 +1,43 @@
def outside(self):
print("outside", self)
def outside_sm():
print("outside_sm")
def outside_cm(cls):
print("outside_cm", cls)
class A(object):
def foo(self):
print("A.foo")
foo_ref = foo
outside_ref = outside
outside_sm = staticmethod(outside_sm)
outside_cm = classmethod(outside_cm)
a = A()
a.foo_ref() # $ pt=A.foo
a.outside_ref() # $ pt=outside
a.outside_sm() # $ pt=outside_sm
a.outside_cm() # $ pt=outside_cm
# ===
print("\n! B")
# this pattern was seen in django
class B(object):
def _gen(value):
def func(self):
print("B._gen.func", value)
return func
foo = _gen("foo") # $ pt=B._gen
bar = _gen("bar") # $ pt=B._gen
b = B()
b.foo() # $ pt=B._gen.func
b.bar() # $ pt=B._gen.func

87
python/ql/test/experimental/library-tests/CallGraph/code/nested_class.py Normal file
View File

@@ -0,0 +1,87 @@
class A(object):
class B(object):
@staticmethod
def foo():
print("A.B.foo")
@staticmethod
def bar():
print("A.B.bar")
A.B.foo() # $ pt,tt=A.B.foo
A.B.bar() # $ pt,tt=A.B.bar
ab = A.B()
ab.bar() # $ pt,tt=A.B.bar
# ==============================================================================
class OuterBase(object):
def foo(self):
print("OuterBase.foo")
class InnerBase(object):
def foo(self):
print("InnerBase.foo")
class Outer(OuterBase):
def foo(self):
print("Outer.foo")
super().foo() # $ pt,tt=OuterBase.foo
class Inner(InnerBase):
def foo(self):
print("Inner.foo")
super().foo() # $ pt,tt=InnerBase.foo
outer = Outer()
outer.foo() # $ pt,tt=Outer.foo
inner = Outer.Inner()
inner.foo() # $ pt,tt=Outer.Inner.foo
# ==============================================================================
class Base(object):
def foo(self):
print("Base.foo")
class Base2(object):
def foo(self):
print("Base2.foo")
class X(Base):
def meth(self):
print("X.meth")
super().foo() # $ pt,tt=Base.foo
def inner_func():
print("inner_func")
try:
super().foo()
except RuntimeError:
print("RuntimeError, as expected")
inner_func() # $ pt,tt=X.meth.inner_func
def inner_func2(this_works):
print("inner_func2")
super().foo() # $ MISSING: tt=Base.foo
inner_func2(self) # $ pt,tt=X.meth.inner_func2
def class_def_in_func(self):
print("X.class_def_in_func")
class Y(Base2):
def meth(self):
print("Y.meth")
super().foo() # $ pt,tt=Base2.foo
y = Y()
y.meth() # $ tt=X.class_def_in_func.Y.meth
x = X()
x.meth() # $ pt,tt=X.meth
x.class_def_in_func() # $ pt=X.class_def_in_func tt=X.class_def_in_func

7
python/ql/test/experimental/library-tests/CallGraph/code/relative_import.py Normal file
View File

@@ -0,0 +1,7 @@
def test_relative_import():
from .simple import foo
foo() # $ pt,tt="code/simple.py:foo"
def test_aliased_relative_import():
from .aliased_import import foo
foo() # $ pt,tt="code/simple.py:foo"

4
python/ql/test/experimental/library-tests/CallGraph/code/runtime_decision.py
View File

@@ -15,7 +15,7 @@ if len(sys.argv) >= 2 and not sys.argv[1] in ['0', 'False', 'false']:
else:
func = rd_bar
func() # $ pt=rd_foo pt=rd_bar
func() # $ pt,tt=rd_foo pt,tt=rd_bar
# Random doesn't work with points-to :O
if random.random() < 0.5:
@@ -23,4 +23,4 @@ if random.random() < 0.5:
else:
func2 = rd_bar
func2() # $ pt=rd_foo pt=rd_bar
func2() # $ pt,tt=rd_foo pt,tt=rd_bar

22
python/ql/test/experimental/library-tests/CallGraph/code/shadowing.py Normal file
View File

@@ -0,0 +1,22 @@
def foo(n=0):
print("foo", n)
if n > 0:
foo(n-1) # $ pt,tt=foo
foo(1) # $ pt,tt=foo
def test():
def foo():
print("test.foo")
foo() # $ pt,tt=test.foo
class A(object):
def foo(self):
print("A.foo")
foo() # $ pt=foo MISSING: tt=foo
a = A()
a.foo() # $ pt,tt=A.foo

8
python/ql/test/experimental/library-tests/CallGraph/code/simple.py
View File

@@ -12,9 +12,9 @@ def bar():
lam = lambda: print("lambda called")
foo() # $ pt=foo
indirect_foo() # $ pt=foo
bar() # $ pt=bar
lam() # $ pt=lambda[simple.py:12:7]
foo() # $ pt,tt=foo
indirect_foo() # $ pt,tt=foo
bar() # $ pt,tt=bar
lam() # $ pt,tt=lambda[simple.py:12:7]
# python -m trace --trackcalls simple.py

6
python/ql/test/experimental/library-tests/CallGraph/code/through_content.py Normal file
View File

@@ -0,0 +1,6 @@
def my_func():
print("my_func")
funcs = [my_func]
for f in funcs:
f() # $ MISSING: tt=my_func

8
python/ql/test/experimental/library-tests/CallGraph/code/type_tracking_limitation.py Normal file
View File

@@ -0,0 +1,8 @@
def return_arg(arg):
return arg
def my_func():
print("my_func")
x = return_arg(my_func) # $ pt,tt=return_arg
x() # $ pt=my_func MISSING: tt=my_func

8
python/ql/test/experimental/library-tests/CallGraph/code/underscore_prefix_func_name.py
View File

@@ -11,14 +11,14 @@ def some_function():
def _ignored():
print('_ignored')
some_function()
some_function() # $ tt=some_function
def _works_since_called():
print('_works_since_called')
some_function() # $ pt=some_function
some_function() # $ pt,tt=some_function
def works_even_though_not_called():
some_function() # $ pt=some_function
some_function() # $ pt,tt=some_function
globals()['_ignored']()
_works_since_called() # $ pt=_works_since_called
_works_since_called() # $ pt,tt=_works_since_called

19
python/ql/test/experimental/library-tests/CallGraph/dataflow-consistency.expected Normal file
View File

@@ -0,0 +1,19 @@
uniqueEnclosingCallable
uniqueType
uniqueNodeLocation
missingLocation
uniqueNodeToString
missingToString
parameterCallable
localFlowIsLocal
compatibleTypesReflexive
unreachableNodeCCtx
localCallNodes
postIsNotPre
postHasUniquePre
uniquePostUpdate
postIsInSameCallable
reverseRead
argHasPostUpdate
postWithInFlow
viableImplInCallContextTooLarge

1
python/ql/test/experimental/library-tests/CallGraph/dataflow-consistency.ql Normal file
View File

@@ -0,0 +1 @@
import semmle.python.dataflow.new.internal.DataFlowImplConsistency::Consistency

5
python/ql/test/library-tests/PointsTo/new/ImpliesDataflow.expected
View File

@@ -5,11 +5,6 @@
| code/l_calls.py:12:1:12:20 | ControlFlowNode for ClassExpr | code/l_calls.py:25:16:25:16 | ControlFlowNode for a |
| code/l_calls.py:33:5:33:23 | ControlFlowNode for FunctionExpr | code/l_calls.py:39:1:39:3 | ControlFlowNode for Attribute |
| code/l_calls.py:48:5:48:30 | ControlFlowNode for FunctionExpr | code/l_calls.py:53:1:53:3 | ControlFlowNode for Attribute |
| code/q_super.py:10:18:10:21 | ControlFlowNode for self | code/q_super.py:4:22:4:25 | ControlFlowNode for self |
| code/q_super.py:26:14:26:17 | ControlFlowNode for self | code/q_super.py:22:32:22:35 | ControlFlowNode for self |
| code/q_super.py:31:14:31:17 | ControlFlowNode for self | code/q_super.py:22:32:22:35 | ControlFlowNode for self |
| code/q_super.py:37:14:37:17 | ControlFlowNode for self | code/q_super.py:22:32:22:35 | ControlFlowNode for self |
| code/q_super.py:37:14:37:17 | ControlFlowNode for self | code/q_super.py:27:32:27:35 | ControlFlowNode for self |
| code/q_super.py:48:5:48:17 | ControlFlowNode for ClassExpr | code/q_super.py:51:25:51:29 | ControlFlowNode for Attribute |
| code/q_super.py:63:5:63:17 | ControlFlowNode for ClassExpr | code/q_super.py:66:19:66:23 | ControlFlowNode for Attribute |
| code/t_type.py:3:1:3:16 | ControlFlowNode for ClassExpr | code/t_type.py:6:1:6:9 | ControlFlowNode for type() |

12
python/ql/test/library-tests/frameworks/django-orm/ReflectedXss.expected
View File

@@ -5,13 +5,13 @@ edges
| testapp/orm_security_tests.py:15:1:15:27 | [orm-model] Class Person [Attribute name] | testapp/orm_security_tests.py:47:14:47:53 | ControlFlowNode for Attribute() [Attribute name] |
| testapp/orm_security_tests.py:19:12:19:18 | ControlFlowNode for request | testapp/orm_security_tests.py:22:23:22:34 | ControlFlowNode for Attribute |
| testapp/orm_security_tests.py:19:12:19:18 | ControlFlowNode for request | testapp/orm_security_tests.py:23:22:23:33 | ControlFlowNode for Attribute |
| testapp/orm_security_tests.py:22:9:22:14 | [post store] ControlFlowNode for person [Attribute name] | testapp/orm_security_tests.py:23:9:23:14 | ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:22:9:22:14 | [post] ControlFlowNode for person [Attribute name] | testapp/orm_security_tests.py:23:9:23:14 | ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:22:23:22:34 | ControlFlowNode for Attribute | testapp/orm_security_tests.py:22:23:22:42 | ControlFlowNode for Subscript |
| testapp/orm_security_tests.py:22:23:22:42 | ControlFlowNode for Subscript | testapp/orm_security_tests.py:22:9:22:14 | [post store] ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:22:23:22:42 | ControlFlowNode for Subscript | testapp/orm_security_tests.py:22:9:22:14 | [post] ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:23:9:23:14 | ControlFlowNode for person [Attribute name] | testapp/orm_security_tests.py:28:9:28:14 | ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:23:9:23:14 | [post store] ControlFlowNode for person [Attribute age] | testapp/orm_security_tests.py:28:9:28:14 | ControlFlowNode for person [Attribute age] |
| testapp/orm_security_tests.py:23:9:23:14 | [post] ControlFlowNode for person [Attribute age] | testapp/orm_security_tests.py:28:9:28:14 | ControlFlowNode for person [Attribute age] |
| testapp/orm_security_tests.py:23:22:23:33 | ControlFlowNode for Attribute | testapp/orm_security_tests.py:23:22:23:40 | ControlFlowNode for Subscript |
| testapp/orm_security_tests.py:23:22:23:40 | ControlFlowNode for Subscript | testapp/orm_security_tests.py:23:9:23:14 | [post store] ControlFlowNode for person [Attribute age] |
| testapp/orm_security_tests.py:23:22:23:40 | ControlFlowNode for Subscript | testapp/orm_security_tests.py:23:9:23:14 | [post] ControlFlowNode for person [Attribute age] |
| testapp/orm_security_tests.py:28:9:28:14 | ControlFlowNode for person [Attribute age] | testapp/orm_security_tests.py:15:1:15:27 | [orm-model] Class Person [Attribute age] |
| testapp/orm_security_tests.py:28:9:28:14 | ControlFlowNode for person [Attribute name] | testapp/orm_security_tests.py:15:1:15:27 | [orm-model] Class Person [Attribute name] |
| testapp/orm_security_tests.py:42:13:42:18 | SSA variable person [Attribute age] | testapp/orm_security_tests.py:43:62:43:67 | ControlFlowNode for person [Attribute age] |
@@ -48,11 +48,11 @@ nodes
| testapp/orm_security_tests.py:15:1:15:27 | [orm-model] Class Person [Attribute age] | semmle.label | [orm-model] Class Person [Attribute age] |
| testapp/orm_security_tests.py:15:1:15:27 | [orm-model] Class Person [Attribute name] | semmle.label | [orm-model] Class Person [Attribute name] |
| testapp/orm_security_tests.py:19:12:19:18 | ControlFlowNode for request | semmle.label | ControlFlowNode for request |
| testapp/orm_security_tests.py:22:9:22:14 | [post store] ControlFlowNode for person [Attribute name] | semmle.label | [post store] ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:22:9:22:14 | [post] ControlFlowNode for person [Attribute name] | semmle.label | [post] ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:22:23:22:34 | ControlFlowNode for Attribute | semmle.label | ControlFlowNode for Attribute |
| testapp/orm_security_tests.py:22:23:22:42 | ControlFlowNode for Subscript | semmle.label | ControlFlowNode for Subscript |
| testapp/orm_security_tests.py:23:9:23:14 | ControlFlowNode for person [Attribute name] | semmle.label | ControlFlowNode for person [Attribute name] |
| testapp/orm_security_tests.py:23:9:23:14 | [post store] ControlFlowNode for person [Attribute age] | semmle.label | [post store] ControlFlowNode for person [Attribute age] |
| testapp/orm_security_tests.py:23:9:23:14 | [post] ControlFlowNode for person [Attribute age] | semmle.label | [post] ControlFlowNode for person [Attribute age] |
| testapp/orm_security_tests.py:23:22:23:33 | ControlFlowNode for Attribute | semmle.label | ControlFlowNode for Attribute |
| testapp/orm_security_tests.py:23:22:23:40 | ControlFlowNode for Subscript | semmle.label | ControlFlowNode for Subscript |
| testapp/orm_security_tests.py:28:9:28:14 | ControlFlowNode for person [Attribute age] | semmle.label | ControlFlowNode for person [Attribute age] |

1
python/ql/test/library-tests/fuck/options Normal file
View File

@@ -0,0 +1 @@
semmle-extractor-options: --max-import-depth=0

17
python/ql/test/library-tests/fuck/test.py Normal file
View File

@@ -0,0 +1,17 @@
def my_func(arg):
print("my_func", arg)
class Foo:
def foo(self, arg=42):
print("Foo.foo", self, arg)
my_func(43)
import random
if random.choice([True, False]):
func = my_func
else:
func = Foo.foo
func(44)

1
python/ql/test/library-tests/fuck/wat.expected Normal file
View File

@@ -0,0 +1 @@
| 1 |