hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2026-04-30 03:05:15 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into jorgectf/python/insecure-cookie

Browse Source
This commit is contained in:
Rasmus Wriedt Larsen
2022-05-11 11:13:47 +02:00
parent 840cded9b0 2b6e0cfb44
commit d127d2164a
7423 changed files with 705987 additions and 227271 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
python/ql/consistency-queries/qlpack.yml Normal file
View File

@@ -0,0 +1,5 @@
name: codeql/python-consistency-queries
groups: [python, test, consistency-queries]
dependencies:
codeql/python-all: "*"
extractor: python

6
python/ql/examples/qlpack.yml
View File

@@ -1,4 +1,6 @@
name: codeql/python-examples
version: 0.0.2
groups:
- python
- examples
dependencies:
codeql/python-all: "*"
codeql/python-all: "*"

101
python/ql/lib/CHANGELOG.md Normal file
View File

@@ -0,0 +1,101 @@
## 0.2.0
### Breaking Changes
* The signature of `allowImplicitRead` on `DataFlow::Configuration` and `TaintTracking::Configuration` has changed from `allowImplicitRead(DataFlow::Node node, DataFlow::Content c)` to `allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c)`.
## 0.1.0
### Breaking Changes
* The recently added flow-state versions of `isBarrierIn`, `isBarrierOut`, `isSanitizerIn`, and `isSanitizerOut` in the data flow and taint tracking libraries have been removed.
### Deprecated APIs
* Queries importing a data-flow configuration from `semmle.python.security.dataflow`
should ensure that the imported file ends with `Query`, and only import its top-level
module. For example, a query that used `CommandInjection::Configuration` from
`semmle.python.security.dataflow.CommandInjection` should from now use `Configuration`
from `semmle.python.security.dataflow.CommandInjectionQuery` instead.
### Major Analysis Improvements
* Added data-flow for Django ORM models that are saved in a database (no `models.ForeignKey` support).
### Minor Analysis Improvements
* Improved modeling of Flask `Response` objects, so passing a response body with the keyword argument `response` is now recognized.
## 0.0.13
## 0.0.12
### Breaking Changes
* The flow state variants of `isBarrier` and `isAdditionalFlowStep` are no longer exposed in the taint tracking library. The `isSanitizer` and `isAdditionalTaintStep` predicates should be used instead.
### Deprecated APIs
* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
* Some modules that started with a lowercase letter have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
### New Features
* The data flow and taint tracking libraries have been extended with versions of `isBarrierIn`, `isBarrierOut`, and `isBarrierGuard`, respectively `isSanitizerIn`, `isSanitizerOut`, and `isSanitizerGuard`, that support flow states.
### Minor Analysis Improvements
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.
## 0.0.11
### Minor Analysis Improvements
* Added new SSRF sinks for `httpx`, `pycurl`, `urllib`, `urllib2`, `urllib3`, and `libtaxii`. This improvement was [submitted by @haby0](https://github.com/github/codeql/pull/8275).
* The regular expression parser now groups sequences of normal characters. This reduces the number of instances of `RegExpNormalChar`.
* Fixed taint propagation for attribute assignment. In the assignment `x.foo = tainted` we no longer treat the entire object `x` as tainted, just because the attribute `foo` contains tainted data. This leads to slightly fewer false positives.
* Improved analysis of attributes for data-flow and taint tracking queries, so `getattr`/`setattr` are supported, and a write to an attribute properly stops flow for the old value in that attribute.
* Added post-update nodes (`DataFlow::PostUpdateNode`) for arguments in calls that can't be resolved.
## 0.0.10
### Deprecated APIs
* The old points-to based modeling has been deprecated. Use the new type-tracking/API-graphs based modeling instead.
## 0.0.9
## 0.0.8
### Deprecated APIs
* Moved the files defining regex injection configuration and customization, instead of `import semmle.python.security.injection.RegexInjection` please use `import semmle.python.security.dataflow.RegexInjection` (the same for `RegexInjectionCustomizations`).
* The `codeql/python-upgrades` CodeQL pack has been removed. All upgrades scripts have been merged into the `codeql/python-all` CodeQL pack.
## 0.0.7
## 0.0.6
## 0.0.5
### Minor Analysis Improvements
* Added modeling of many functions from the `os` module that uses file system paths, such as `os.stat`, `os.chdir`, `os.mkdir`, and so on.
* Added modeling of the `tempfile` module for creating temporary files and directories, such as the functions `tempfile.NamedTemporaryFile` and `tempfile.TemporaryDirectory`.
* Extended the modeling of FastAPI such that custom subclasses of `fastapi.APIRouter` are recognized.
* Extended the modeling of FastAPI such that `fastapi.responses.FileResponse` are considered `FileSystemAccess`.
* Added modeling of the `posixpath`, `ntpath`, and `genericpath` modules for path operations (although these are not supposed to be used), resulting in new sinks.
* Added modeling of `wsgiref.simple_server` applications, leading to new remote flow sources.
## 0.0.4
### Major Analysis Improvements
* Added modeling of `os.stat`, `os.lstat`, `os.statvfs`, `os.fstat`, and `os.fstatvfs`, which are new sinks for the _Uncontrolled data used in path expression_ (`py/path-injection`) query.
* Added modeling of the `posixpath`, `ntpath`, and `genericpath` modules for path operations (although these are not supposed to be used), resulting in new sinks for the _Uncontrolled data used in path expression_ (`py/path-injection`) query.
* Added modeling of `wsgiref.simple_server` applications, leading to new remote flow sources.
* Added modeling of `aiopg` for sinks executing SQL.
* Added modeling of HTTP requests and responses when using `flask_admin` (`Flask-Admin` PyPI package), which leads to additional remote flow sources.
* Added modeling of the PyPI package `toml`, which provides encoding/decoding of TOML documents, leading to new taint-tracking steps.

4
python/ql/lib/change-notes/2022-03-29-add-taint-for-StringIO.md Normal file
View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* Added taint propagation for `io.StringIO` and `io.BytesIO`. This addition was originally [submitted as part of an experimental query by @jorgectf](https://github.com/github/codeql/pull/6112).

4
python/ql/lib/change-notes/2022-04-20-export-python-under-DataFlow.md Normal file
View File

@@ -0,0 +1,4 @@
---
category: breaking
---
* The imports made available from `import python` are no longer exposed under `DataFlow::` after doing `import semmle.python.dataflow.new.DataFlow`, for example using `DataFlow::Add` will now cause a compile error.

5
python/ql/lib/change-notes/2022-05-02-flask-request-files-modeling.md Normal file
View File

@@ -0,0 +1,5 @@
---
category: minorAnalysis
---
The modeling of `request.files` in Flask has been fixed, so we now properly handle
assignments to local variables (such as `files = request.files; files['key'].filename`).

5
python/ql/lib/change-notes/released/0.0.10.md Normal file
View File

@@ -0,0 +1,5 @@
## 0.0.10
### Deprecated APIs
* The old points-to based modeling has been deprecated. Use the new type-tracking/API-graphs based modeling instead.

9
python/ql/lib/change-notes/released/0.0.11.md Normal file
View File

@@ -0,0 +1,9 @@
## 0.0.11
### Minor Analysis Improvements
* Added new SSRF sinks for `httpx`, `pycurl`, `urllib`, `urllib2`, `urllib3`, and `libtaxii`. This improvement was [submitted by @haby0](https://github.com/github/codeql/pull/8275).
* The regular expression parser now groups sequences of normal characters. This reduces the number of instances of `RegExpNormalChar`.
* Fixed taint propagation for attribute assignment. In the assignment `x.foo = tainted` we no longer treat the entire object `x` as tainted, just because the attribute `foo` contains tainted data. This leads to slightly fewer false positives.
* Improved analysis of attributes for data-flow and taint tracking queries, so `getattr`/`setattr` are supported, and a write to an attribute properly stops flow for the old value in that attribute.
* Added post-update nodes (`DataFlow::PostUpdateNode`) for arguments in calls that can't be resolved.

20
python/ql/lib/change-notes/released/0.0.12.md Normal file
View File

@@ -0,0 +1,20 @@
## 0.0.12
### Breaking Changes
* The flow state variants of `isBarrier` and `isAdditionalFlowStep` are no longer exposed in the taint tracking library. The `isSanitizer` and `isAdditionalTaintStep` predicates should be used instead.
### Deprecated APIs
* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
* Some modules that started with a lowercase letter have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
### New Features
* The data flow and taint tracking libraries have been extended with versions of `isBarrierIn`, `isBarrierOut`, and `isBarrierGuard`, respectively `isSanitizerIn`, `isSanitizerOut`, and `isSanitizerGuard`, that support flow states.
### Minor Analysis Improvements
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.

1
python/ql/lib/change-notes/released/0.0.13.md Normal file
View File

@@ -0,0 +1 @@
## 0.0.13

10
python/ql/lib/change-notes/released/0.0.4.md Normal file
View File

@@ -0,0 +1,10 @@
## 0.0.4
### Major Analysis Improvements
* Added modeling of `os.stat`, `os.lstat`, `os.statvfs`, `os.fstat`, and `os.fstatvfs`, which are new sinks for the _Uncontrolled data used in path expression_ (`py/path-injection`) query.
* Added modeling of the `posixpath`, `ntpath`, and `genericpath` modules for path operations (although these are not supposed to be used), resulting in new sinks for the _Uncontrolled data used in path expression_ (`py/path-injection`) query.
* Added modeling of `wsgiref.simple_server` applications, leading to new remote flow sources.
* Added modeling of `aiopg` for sinks executing SQL.
* Added modeling of HTTP requests and responses when using `flask_admin` (`Flask-Admin` PyPI package), which leads to additional remote flow sources.
* Added modeling of the PyPI package `toml`, which provides encoding/decoding of TOML documents, leading to new taint-tracking steps.

10
python/ql/lib/change-notes/released/0.0.5.md Normal file
View File

@@ -0,0 +1,10 @@
## 0.0.5
### Minor Analysis Improvements
* Added modeling of many functions from the `os` module that uses file system paths, such as `os.stat`, `os.chdir`, `os.mkdir`, and so on.
* Added modeling of the `tempfile` module for creating temporary files and directories, such as the functions `tempfile.NamedTemporaryFile` and `tempfile.TemporaryDirectory`.
* Extended the modeling of FastAPI such that custom subclasses of `fastapi.APIRouter` are recognized.
* Extended the modeling of FastAPI such that `fastapi.responses.FileResponse` are considered `FileSystemAccess`.
* Added modeling of the `posixpath`, `ntpath`, and `genericpath` modules for path operations (although these are not supposed to be used), resulting in new sinks.
* Added modeling of `wsgiref.simple_server` applications, leading to new remote flow sources.

1
python/ql/lib/change-notes/released/0.0.6.md Normal file
View File

@@ -0,0 +1 @@
## 0.0.6

1
python/ql/lib/change-notes/released/0.0.7.md Normal file
View File

@@ -0,0 +1 @@
## 0.0.7

6
python/ql/lib/change-notes/released/0.0.8.md Normal file
View File

@@ -0,0 +1,6 @@
## 0.0.8
### Deprecated APIs
* Moved the files defining regex injection configuration and customization, instead of `import semmle.python.security.injection.RegexInjection` please use `import semmle.python.security.dataflow.RegexInjection` (the same for `RegexInjectionCustomizations`).
* The `codeql/python-upgrades` CodeQL pack has been removed. All upgrades scripts have been merged into the `codeql/python-all` CodeQL pack.

1
python/ql/lib/change-notes/released/0.0.9.md Normal file
View File

@@ -0,0 +1 @@
## 0.0.9

21
python/ql/lib/change-notes/released/0.1.0.md Normal file
View File

@@ -0,0 +1,21 @@
## 0.1.0
### Breaking Changes
* The recently added flow-state versions of `isBarrierIn`, `isBarrierOut`, `isSanitizerIn`, and `isSanitizerOut` in the data flow and taint tracking libraries have been removed.
### Deprecated APIs
* Queries importing a data-flow configuration from `semmle.python.security.dataflow`
should ensure that the imported file ends with `Query`, and only import its top-level
module. For example, a query that used `CommandInjection::Configuration` from
`semmle.python.security.dataflow.CommandInjection` should from now use `Configuration`
from `semmle.python.security.dataflow.CommandInjectionQuery` instead.
### Major Analysis Improvements
* Added data-flow for Django ORM models that are saved in a database (no `models.ForeignKey` support).
### Minor Analysis Improvements
* Improved modeling of Flask `Response` objects, so passing a response body with the keyword argument `response` is now recognized.

5
python/ql/lib/change-notes/released/0.2.0.md Normal file
View File

@@ -0,0 +1,5 @@
## 0.2.0
### Breaking Changes
* The signature of `allowImplicitRead` on `DataFlow::Configuration` and `TaintTracking::Configuration` has changed from `allowImplicitRead(DataFlow::Node node, DataFlow::Content c)` to `allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c)`.

2
python/ql/lib/codeql-pack.release.yml Normal file
View File

@@ -0,0 +1,2 @@
---
lastReleaseVersion: 0.2.0

1
python/ql/lib/python.qll
View File

@@ -10,6 +10,7 @@ import semmle.python.Class
import semmle.python.Import
import semmle.python.Stmts
import semmle.python.Exprs
import semmle.python.Patterns
import semmle.python.Keywords
import semmle.python.Comprehensions
import semmle.python.Flow

6
python/ql/lib/qlpack.yml
View File

@@ -1,7 +1,7 @@
name: codeql/python-all
version: 0.0.2
version: 0.2.1-dev
groups: python
dbscheme: semmlecode.python.dbscheme
extractor: python
library: true
dependencies:
codeql/python-upgrades: 0.0.2
upgrades: upgrades

564
python/ql/lib/semmle/python/ApiGraphs.qll
View File

@@ -6,8 +6,10 @@
* directed and labeled; they specify how the components represented by nodes relate to each other.
*/
private import python
// Importing python under the `py` namespace to avoid importing `CallNode` from `Flow.qll` and thereby having a naming conflict with `API::CallNode`.
private import python as PY
import semmle.python.dataflow.new.DataFlow
private import semmle.python.internal.CachedStages
/**
* Provides classes and predicates for working with APIs used in a database.
@@ -39,6 +41,30 @@ module API {
)
}
/**
* Gets a data-flow node corresponding to the right-hand side of a definition of the API
* component represented by this node.
*
* For example, in the property write `foo.bar = x`, variable `x` is the the right-hand side
* of a write to the `bar` property of `foo`.
*
* Note that for parameters, it is the arguments flowing into that parameter that count as
* right-hand sides of the definition, not the declaration of the parameter itself.
* Consequently, in :
* ```python
* from mypkg import foo;
* foo.bar(x)
* ```
* `x` is the right-hand side of a definition of the first parameter of `bar` from the `mypkg.foo` module.
*/
DataFlow::Node getARhs() { Impl::rhs(this, result) }
/**
* Gets a data-flow node that may interprocedurally flow to the right-hand side of a definition
* of the API component represented by this node.
*/
DataFlow::Node getAValueReachingRhs() { result = Impl::trackDefNode(this.getARhs()) }
/**
* Gets an immediate use of the API component represented by this node.
*
@@ -55,7 +81,7 @@ module API {
/**
* Gets a call to the function represented by this API component.
*/
DataFlow::CallCfgNode getACall() { result = this.getReturn().getAnImmediateUse() }
CallNode getACall() { result = this.getReturn().getAnImmediateUse() }
/**
* Gets a node representing member `m` of this API component.
@@ -92,6 +118,29 @@ module API {
*/
Node getReturn() { result = this.getASuccessor(Label::return()) }
/**
* Gets a node representing the `i`th parameter of the function represented by this node.
*
* This predicate may have multiple results when there are multiple invocations of this API component.
* Consider using `getAnInvocation()` if there is a need to distingiush between individual calls.
*/
Node getParameter(int i) { result = this.getASuccessor(Label::parameter(i)) }
/**
* Gets the node representing the keyword parameter `name` of the function represented by this node.
*
* This predicate may have multiple results when there are multiple invocations of this API component.
* Consider using `getAnInvocation()` if there is a need to distingiush between individual calls.
*/
Node getKeywordParameter(string name) {
result = this.getASuccessor(Label::keywordParameter(name))
}
/**
* Gets the number of parameters of the function represented by this node.
*/
int getNumParameter() { result = max(int s | exists(this.getParameter(s))) + 1 }
/**
* Gets a node representing a subclass of the class represented by this node.
*/
@@ -114,13 +163,13 @@ module API {
* Gets a node such that there is an edge in the API graph between this node and the other
* one, and that edge is labeled with `lbl`.
*/
Node getASuccessor(string lbl) { Impl::edge(this, lbl, result) }
Node getASuccessor(Label::ApiLabel lbl) { Impl::edge(this, lbl, result) }
/**
* Gets a node such that there is an edge in the API graph between that other node and
* this one, and that edge is labeled with `lbl`
*/
Node getAPredecessor(string lbl) { this = result.getASuccessor(lbl) }
Node getAPredecessor(Label::ApiLabel lbl) { this = result.getASuccessor(lbl) }
/**
* Gets a node such that there is an edge in the API graph between this node and the other
@@ -137,7 +186,7 @@ module API {
/**
* Gets the data-flow node that gives rise to this node, if any.
*/
DataFlow::Node getInducingNode() { this = Impl::MkUse(result) }
DataFlow::Node getInducingNode() { this = Impl::MkUse(result) or this = Impl::MkDef(result) }
/**
* Holds if this element is at the specified location.
@@ -174,9 +223,8 @@ module API {
length = 0 and
result = ""
or
exists(Node pred, string lbl, string predpath |
exists(Node pred, Label::ApiLabel lbl, string predpath |
Impl::edge(pred, lbl, this) and
lbl != "" and
predpath = pred.getAPath(length - 1) and
exists(string dot | if length = 1 then dot = "" else dot = "." |
result = predpath + dot + lbl and
@@ -211,6 +259,17 @@ module API {
}
}
/** A node corresponding to the rhs of an API component. */
class Def extends Node, Impl::TDef {
override string toString() {
exists(string type | this = Impl::MkDef(_) and type = "Def " |
result = type + this.getPath()
or
not exists(this.getPath()) and result = type + "with no path"
)
}
}
/** Gets the root node. */
Root root() { any() }
@@ -226,6 +285,62 @@ module API {
/** Gets a node corresponding to the built-in with the given name, if any. */
Node builtin(string n) { result = moduleImport("builtins").getMember(n) }
/**
* An `CallCfgNode` that is connected to the API graph.
*
* Can be used to reason about calls to an external API in which the correlation between
* parameters and/or return values must be retained.
*
* The member predicates `getParameter`, `getKeywordParameter`, `getReturn`, and `getInstance` mimic
* the corresponding predicates from `API::Node`. These are guaranteed to exist and be unique to this call.
*/
class CallNode extends DataFlow::CallCfgNode {
API::Node callee;
CallNode() { this = callee.getReturn().getAnImmediateUse() }
/** Gets the API node for the `i`th parameter of this invocation. */
pragma[nomagic]
Node getParameter(int i) {
result = callee.getParameter(i) and
result = this.getAParameterCandidate(i)
}
/**
* Gets an API node where a RHS of the node is the `i`th argument to this call.
*/
pragma[noinline]
private Node getAParameterCandidate(int i) { result.getARhs() = this.getArg(i) }
/** Gets the API node for a parameter of this invocation. */
Node getAParameter() { result = this.getParameter(_) }
/** Gets the API node for the keyword parameter `name` of this invocation. */
Node getKeywordParameter(string name) {
result = callee.getKeywordParameter(name) and
result = this.getAKeywordParameterCandidate(name)
}
/** Gets the API node for the parameter that has index `i` or has keyword `name`. */
bindingset[i, name]
Node getParameter(int i, string name) {
result = this.getParameter(i)
or
result = this.getKeywordParameter(name)
}
pragma[noinline]
private Node getAKeywordParameterCandidate(string name) {
result.getARhs() = this.getArgByName(name)
}
/** Gets the API node for the return value of this call. */
Node getReturn() {
result = callee.getReturn() and
result.getAnImmediateUse() = this
}
}
/**
* Provides the actual implementation of API graphs, cached for performance.
*
@@ -304,6 +419,8 @@ module API {
* API graph node for the prefix `foo`), in accordance with the usual semantics of Python.
*/
private import semmle.python.internal.Awaited
cached
newtype TApiNode =
/** The root of the API graph. */
@@ -311,29 +428,33 @@ module API {
/** An abstract representative for imports of the module called `name`. */
MkModuleImport(string name) {
// Ignore the following module name for Python 2, as we alias `__builtin__` to `builtins` elsewhere
(name != "__builtin__" or major_version() = 3) and
(name != "__builtin__" or PY::major_version() = 3) and
(
imports(_, name)
or
// When we `import foo.bar.baz` we want to create API graph nodes also for the prefixes
// `foo` and `foo.bar`:
name = any(ImportExpr e | not e.isRelative()).getAnImportedModuleName()
name = any(PY::ImportExpr e | not e.isRelative()).getAnImportedModuleName()
)
or
// The `builtins` module should always be implicitly available
name = "builtins"
} or
/** A use of an API member at the node `nd`. */
MkUse(DataFlow::Node nd) { use(_, _, nd) }
MkUse(DataFlow::Node nd) { use(_, _, nd) } or
MkDef(DataFlow::Node nd) { rhs(_, _, nd) }
class TUse = MkModuleImport or MkUse;
class TDef = MkDef;
/**
* Holds if the dotted module name `sub` refers to the `member` member of `base`.
*
* For instance, `prefix_member("foo.bar", "baz", "foo.bar.baz")` would hold.
*/
private predicate prefix_member(TApiNode base, string member, TApiNode sub) {
cached
predicate prefix_member(TApiNode base, string member, TApiNode sub) {
exists(string sub_str, string regexp |
regexp = "(.+)[.]([^.]+)" and
base = MkModuleImport(sub_str.regexpCapture(regexp, 1)) and
@@ -349,141 +470,75 @@ module API {
* Ignores relative imports, such as `from ..foo.bar import baz`.
*/
private predicate imports(DataFlow::Node imp, string name) {
exists(ImportExprNode iexpr |
exists(PY::ImportExprNode iexpr |
imp.asCfgNode() = iexpr and
not iexpr.getNode().isRelative() and
name = iexpr.getNode().getImportedModuleName()
)
}
/** Gets the name of a known built-in. */
private string getBuiltInName() {
// These lists were created by inspecting the `builtins` and `__builtin__` modules in
// Python 3 and 2 respectively, using the `dir` built-in.
// Built-in functions and exceptions shared between Python 2 and 3
result in [
"abs", "all", "any", "bin", "bool", "bytearray", "callable", "chr", "classmethod",
"compile", "complex", "delattr", "dict", "dir", "divmod", "enumerate", "eval", "filter",
"float", "format", "frozenset", "getattr", "globals", "hasattr", "hash", "help", "hex",
"id", "input", "int", "isinstance", "issubclass", "iter", "len", "list", "locals", "map",
"max", "memoryview", "min", "next", "object", "oct", "open", "ord", "pow", "print",
"property", "range", "repr", "reversed", "round", "set", "setattr", "slice", "sorted",
"staticmethod", "str", "sum", "super", "tuple", "type", "vars", "zip", "__import__",
// Exceptions
"ArithmeticError", "AssertionError", "AttributeError", "BaseException", "BufferError",
"BytesWarning", "DeprecationWarning", "EOFError", "EnvironmentError", "Exception",
"FloatingPointError", "FutureWarning", "GeneratorExit", "IOError", "ImportError",
"ImportWarning", "IndentationError", "IndexError", "KeyError", "KeyboardInterrupt",
"LookupError", "MemoryError", "NameError", "NotImplemented", "NotImplementedError",
"OSError", "OverflowError", "PendingDeprecationWarning", "ReferenceError", "RuntimeError",
"RuntimeWarning", "StandardError", "StopIteration", "SyntaxError", "SyntaxWarning",
"SystemError", "SystemExit", "TabError", "TypeError", "UnboundLocalError",
"UnicodeDecodeError", "UnicodeEncodeError", "UnicodeError", "UnicodeTranslateError",
"UnicodeWarning", "UserWarning", "ValueError", "Warning", "ZeroDivisionError",
// Added for compatibility
"exec"
]
or
// Built-in constants shared between Python 2 and 3
result in ["False", "True", "None", "NotImplemented", "Ellipsis", "__debug__"]
or
// Python 3 only
result in [
"ascii", "breakpoint", "bytes", "exec",
// Exceptions
"BlockingIOError", "BrokenPipeError", "ChildProcessError", "ConnectionAbortedError",
"ConnectionError", "ConnectionRefusedError", "ConnectionResetError", "FileExistsError",
"FileNotFoundError", "InterruptedError", "IsADirectoryError", "ModuleNotFoundError",
"NotADirectoryError", "PermissionError", "ProcessLookupError", "RecursionError",
"ResourceWarning", "StopAsyncIteration", "TimeoutError"
]
or
// Python 2 only
result in [
"basestring", "cmp", "execfile", "file", "long", "raw_input", "reduce", "reload",
"unichr", "unicode", "xrange"
]
}
/**
* Gets a data flow node that is likely to refer to a built-in with the name `name`.
*
* Currently this is an over-approximation, and may not account for things like overwriting a
* built-in with a different value.
*/
private DataFlow::Node likely_builtin(string name) {
exists(Module m |
result.asCfgNode() =
any(NameNode n |
possible_builtin_accessed_in_module(n, name, m) and
not possible_builtin_defined_in_module(name, m)
)
)
}
/**
* Holds if a global variable called `name` (which is also the name of a built-in) is assigned
* a value in the module `m`.
*/
private predicate possible_builtin_defined_in_module(string name, Module m) {
global_name_defined_in_module(name, m) and
name = getBuiltInName()
}
/**
* Holds if `n` is an access of a global variable called `name` (which is also the name of a
* built-in) inside the module `m`.
*/
private predicate possible_builtin_accessed_in_module(NameNode n, string name, Module m) {
n.isGlobal() and
n.isLoad() and
name = n.getId() and
name = getBuiltInName() and
m = n.getEnclosingModule()
}
/**
* Holds if `n` is an access of a variable called `name` (which is _not_ the name of a
* built-in, and which is _not_ a global defined in the enclosing module) inside the scope `s`.
*/
private predicate name_possibly_defined_in_import_star(NameNode n, string name, Scope s) {
n.isLoad() and
name = n.getId() and
// Not already defined in an enclosing scope.
not exists(LocalVariable v |
v.getId() = name and v.getScope() = n.getScope().getEnclosingScope*()
) and
not name = getBuiltInName() and
s = n.getScope().getEnclosingScope*() and
exists(potential_import_star_base(s)) and
not global_name_defined_in_module(name, n.getEnclosingModule())
}
/** Holds if a global variable called `name` is assigned a value in the module `m`. */
private predicate global_name_defined_in_module(string name, Module m) {
exists(NameNode n |
not exists(LocalVariable v | n.defines(v)) and
n.isStore() and
name = n.getId() and
m = n.getEnclosingModule()
)
}
private import semmle.python.dataflow.new.internal.Builtins
private import semmle.python.dataflow.new.internal.ImportStar
/**
* Gets the API graph node for all modules imported with `from ... import *` inside the scope `s`.
*
* For example, given
*
* `from foo.bar import *`
* ```python
* from foo.bar import *
* ```
*
* this would be the API graph node with the path
*
* `moduleImport("foo").getMember("bar")`
*/
private TApiNode potential_import_star_base(Scope s) {
exists(DataFlow::Node ref |
ref.asCfgNode() = any(ImportStarNode n | n.getScope() = s).getModule() and
use(result, ref)
private TApiNode potential_import_star_base(PY::Scope s) {
exists(DataFlow::Node n |
n.asCfgNode() = ImportStar::potentialImportStarBase(s) and
use(result, n)
)
}
/**
* Holds if `rhs` is the right-hand side of a definition of a node that should have an
* incoming edge from `base` labeled `lbl` in the API graph.
*/
cached
predicate rhs(TApiNode base, Label::ApiLabel lbl, DataFlow::Node rhs) {
exists(DataFlow::Node def, DataFlow::LocalSourceNode pred |
rhs(base, def) and pred = trackDefNode(def)
|
// from `x` to a definition of `x.prop`
exists(DataFlow::AttrWrite aw | aw = pred.getAnAttributeWrite() |
lbl = Label::memberFromRef(aw) and
rhs = aw.getValue()
)
or
// TODO: I had expected `DataFlow::AttrWrite` to contain the attribute writes from a dict, that's how JS works.
exists(PY::Dict dict, PY::KeyValuePair item |
dict = pred.asExpr() and
dict.getItem(_) = item and
lbl = Label::member(item.getKey().(PY::StrConst).getS()) and
rhs.asExpr() = item.getValue()
)
or
exists(PY::CallableExpr fn | fn = pred.asExpr() |
not fn.getInnerScope().isAsync() and
lbl = Label::return() and
exists(PY::Return ret |
rhs.asExpr() = ret.getValue() and
ret.getScope() = fn.getInnerScope()
)
)
)
or
argumentPassing(base, lbl, rhs)
or
exists(DataFlow::LocalSourceNode src, DataFlow::AttrWrite aw |
use(base, src) and aw = trackUseNode(src).getAnAttributeWrite() and rhs = aw.getValue()
|
lbl = Label::memberFromRef(aw)
)
}
@@ -492,7 +547,7 @@ module API {
* `lbl` in the API graph.
*/
cached
predicate use(TApiNode base, string lbl, DataFlow::Node ref) {
predicate use(TApiNode base, Label::ApiLabel lbl, DataFlow::Node ref) {
exists(DataFlow::LocalSourceNode src, DataFlow::LocalSourceNode pred |
// First, we find a predecessor of the node `ref` that we want to determine. The predecessor
// is any node that is a type-tracked use of a data flow node (`src`), which is itself a
@@ -505,7 +560,7 @@ module API {
|
// Referring to an attribute on a node that is a use of `base`:
lbl = Label::memberFromRef(ref) and
ref = pred.getAnAttributeReference()
ref = pred.getAnAttributeRead()
or
// Calling a node that is a use of `base`
lbl = Label::return() and
@@ -514,28 +569,41 @@ module API {
// Subclassing a node
lbl = Label::subclass() and
exists(DataFlow::Node superclass | pred.flowsTo(superclass) |
ref.asExpr().(ClassExpr).getABase() = superclass.asExpr()
ref.asExpr().(PY::ClassExpr).getABase() = superclass.asExpr()
)
or
// awaiting
exists(Await await, DataFlow::Node awaitedValue |
exists(DataFlow::Node awaitedValue |
lbl = Label::await() and
ref.asExpr() = await and
await.getValue() = awaitedValue.asExpr() and
ref = awaited(awaitedValue) and
pred.flowsTo(awaitedValue)
)
)
or
exists(DataFlow::Node def, PY::CallableExpr fn |
rhs(base, def) and fn = trackDefNode(def).asExpr()
|
exists(int i |
lbl = Label::parameter(i) and
ref.asExpr() = fn.getInnerScope().getArg(i)
)
or
exists(string name |
lbl = Label::keywordParameter(name) and
ref.asExpr() = fn.getInnerScope().getArgByName(name)
)
)
or
// Built-ins, treated as members of the module `builtins`
base = MkModuleImport("builtins") and
lbl = Label::member(any(string name | ref = likely_builtin(name)))
lbl = Label::member(any(string name | ref = Builtins::likelyBuiltin(name)))
or
// Unknown variables that may belong to a module imported with `import *`
exists(Scope s |
exists(PY::Scope s |
base = potential_import_star_base(s) and
lbl =
Label::member(any(string name |
name_possibly_defined_in_import_star(ref.asCfgNode(), name, s)
ImportStar::namePossiblyDefinedInImportStar(ref.asCfgNode(), name, s)
))
)
}
@@ -551,7 +619,7 @@ module API {
)
or
// Ensure the Python 2 `__builtin__` module gets the name of the Python 3 `builtins` module.
major_version() = 2 and
PY::major_version() = 2 and
nd = MkModuleImport("builtins") and
imports(ref, "__builtin__")
or
@@ -573,6 +641,42 @@ module API {
exists(DataFlow::TypeTracker t2 | result = trackUseNode(src, t2).track(t2, t))
}
/**
* Holds if `arg` is passed as an argument to a use of `base`.
*
* `lbl` is represents which parameter of the function was passed. Either a numbered parameter, or a keyword parameter.
*/
private predicate argumentPassing(TApiNode base, Label::ApiLabel lbl, DataFlow::Node arg) {
exists(DataFlow::Node use, DataFlow::LocalSourceNode pred |
use(base, use) and pred = trackUseNode(use)
|
exists(int i |
lbl = Label::parameter(i) and
arg = pred.getACall().getArg(i)
)
or
exists(string name | lbl = Label::keywordParameter(name) |
arg = pred.getACall().getArgByName(name)
)
)
}
/**
* Gets a node that inter-procedurally flows into `nd`, which is a definition of some node.
*/
cached
DataFlow::LocalSourceNode trackDefNode(DataFlow::Node nd) {
result = trackDefNode(nd, DataFlow::TypeBackTracker::end())
}
private DataFlow::LocalSourceNode trackDefNode(DataFlow::Node nd, DataFlow::TypeBackTracker t) {
t.start() and
rhs(_, nd) and
result = nd.getALocalSource()
or
exists(DataFlow::TypeBackTracker t2 | result = trackDefNode(nd, t2).backtrack(t2, t))
}
/**
* Gets a data-flow node to which `src`, which is a use of an API-graph node, flows.
*
@@ -580,20 +684,26 @@ module API {
*/
cached
DataFlow::LocalSourceNode trackUseNode(DataFlow::LocalSourceNode src) {
Stages::TypeTracking::ref() and
result = trackUseNode(src, DataFlow::TypeTracker::end()) and
not result instanceof DataFlow::ModuleVariableNode
}
/**
* Holds if `rhs` is the right-hand side of a definition of node `nd`.
*/
cached
predicate rhs(TApiNode nd, DataFlow::Node rhs) { nd = MkDef(rhs) }
/**
* Holds if there is an edge from `pred` to `succ` in the API graph that is labeled with `lbl`.
*/
cached
predicate edge(TApiNode pred, string lbl, TApiNode succ) {
predicate edge(TApiNode pred, Label::ApiLabel lbl, TApiNode succ) {
/* There's an edge from the root node for each imported module. */
exists(string m |
pred = MkRoot() and
lbl = Label::mod(m)
|
lbl = Label::mod(m) and
succ = MkModuleImport(m) and
// Only allow undotted names to count as base modules.
not m.matches("%.%")
@@ -610,6 +720,11 @@ module API {
use(pred, lbl, ref) and
succ = MkUse(ref)
)
or
exists(DataFlow::Node rhs |
rhs(pred, lbl, rhs) and
succ = MkDef(rhs)
)
}
/**
@@ -621,36 +736,147 @@ module API {
cached
int distanceFromRoot(TApiNode nd) = shortestDistances(MkRoot/0, edge/2)(_, nd, result)
}
}
private module Label {
/** Gets the edge label for the module `m`. */
bindingset[m]
bindingset[result]
string mod(string m) { result = "moduleImport(\"" + m + "\")" }
/** Provides classes modeling the various edges (labels) in the API graph. */
module Label {
/** A label in the API-graph */
class ApiLabel extends TLabel {
/** Gets a string representation of this label. */
string toString() { result = "???" }
}
/** Gets the `member` edge label for member `m`. */
bindingset[m]
bindingset[result]
string member(string m) { result = "getMember(\"" + m + "\")" }
private import LabelImpl
/** Gets the `member` edge label for the unknown member. */
string unknownMember() { result = "getUnknownMember()" }
private module LabelImpl {
private import semmle.python.dataflow.new.internal.Builtins
private import semmle.python.dataflow.new.internal.ImportStar
/** Gets the `member` edge label for the given attribute reference. */
string memberFromRef(DataFlow::AttrRef pr) {
result = member(pr.getAttributeName())
or
not exists(pr.getAttributeName()) and
result = unknownMember()
newtype TLabel =
MkLabelModule(string mod) {
exists(Impl::MkModuleImport(mod)) and
not mod.matches("%.%") // only top level modules count as base modules
} or
MkLabelMember(string member) {
member = any(DataFlow::AttrRef pr).getAttributeName() or
exists(Builtins::likelyBuiltin(member)) or
ImportStar::namePossiblyDefinedInImportStar(_, member, _) or
Impl::prefix_member(_, member, _) or
member = any(PY::Dict d).getAnItem().(PY::KeyValuePair).getKey().(PY::StrConst).getS()
} or
MkLabelUnknownMember() or
MkLabelParameter(int i) {
exists(any(DataFlow::CallCfgNode c).getArg(i))
or
exists(any(PY::Function f).getArg(i))
} or
MkLabelKeywordParameter(string name) {
exists(any(DataFlow::CallCfgNode c).getArgByName(name))
or
exists(any(PY::Function f).getArgByName(name))
} or
MkLabelReturn() or
MkLabelSubclass() or
MkLabelAwait()
/** A label for a module. */
class LabelModule extends ApiLabel, MkLabelModule {
string mod;
LabelModule() { this = MkLabelModule(mod) }
/** Gets the module associated with this label. */
string getMod() { result = mod }
override string toString() { result = "moduleImport(\"" + mod + "\")" }
}
/** A label for the member named `prop`. */
class LabelMember extends ApiLabel, MkLabelMember {
string member;
LabelMember() { this = MkLabelMember(member) }
/** Gets the property associated with this label. */
string getMember() { result = member }
override string toString() { result = "getMember(\"" + member + "\")" }
}
/** A label for a member with an unknown name. */
class LabelUnknownMember extends ApiLabel, MkLabelUnknownMember {
override string toString() { result = "getUnknownMember()" }
}
/** A label for parameter `i`. */
class LabelParameter extends ApiLabel, MkLabelParameter {
int i;
LabelParameter() { this = MkLabelParameter(i) }
override string toString() { result = "getParameter(" + i + ")" }
/** Gets the index of the parameter for this label. */
int getIndex() { result = i }
}
/** A label for a keyword parameter `name`. */
class LabelKeywordParameter extends ApiLabel, MkLabelKeywordParameter {
string name;
LabelKeywordParameter() { this = MkLabelKeywordParameter(name) }
override string toString() { result = "getKeywordParameter(\"" + name + "\")" }
/** Gets the name of the parameter for this label. */
string getName() { result = name }
}
/** A label that gets the return value of a function. */
class LabelReturn extends ApiLabel, MkLabelReturn {
override string toString() { result = "getReturn()" }
}
/** A label that gets the subclass of a class. */
class LabelSubclass extends ApiLabel, MkLabelSubclass {
override string toString() { result = "getASubclass()" }
}
/** A label for awaited values. */
class LabelAwait extends ApiLabel, MkLabelAwait {
override string toString() { result = "getAwaited()" }
}
}
/** Gets the edge label for the module `m`. */
LabelModule mod(string m) { result.getMod() = m }
/** Gets the `member` edge label for member `m`. */
LabelMember member(string m) { result.getMember() = m }
/** Gets the `member` edge label for the unknown member. */
LabelUnknownMember unknownMember() { any() }
/** Gets the `member` edge label for the given attribute reference. */
ApiLabel memberFromRef(DataFlow::AttrRef ref) {
result = member(ref.getAttributeName())
or
not exists(ref.getAttributeName()) and
result = unknownMember()
}
/** Gets the `parameter` edge label for parameter `i`. */
LabelParameter parameter(int i) { result.getIndex() = i }
/** Gets the `parameter` edge label for the keyword parameter `name`. */
LabelKeywordParameter keywordParameter(string name) { result.getName() = name }
/** Gets the `return` edge label. */
LabelReturn return() { any() }
/** Gets the `subclass` edge label. */
LabelSubclass subclass() { any() }
/** Gets the `await` edge label. */
LabelAwait await() { any() }
}
/** Gets the `return` edge label. */
string return() { result = "getReturn()" }
/** Gets the `subclass` edge label. */
string subclass() { result = "getASubclass()" }
/** Gets the `await` edge label. */
string await() { result = "getAwaited()" }
}

75
python/ql/lib/semmle/python/AstExtended.qll
View File

@@ -1,6 +1,7 @@
import python
private import semmle.python.internal.CachedStages
/** Syntactic node (Class, Function, Module, Expr, Stmt or Comprehension) corresponding to a flow node */
/** A syntactic node (Class, Function, Module, Expr, Stmt or Comprehension) corresponding to a flow node */
abstract class AstNode extends AstNode_ {
/*
* Special comment for documentation generation.
@@ -17,9 +18,14 @@ abstract class AstNode extends AstNode_ {
* NOTE: For some statements and other purely syntactic elements,
* there may not be a `ControlFlowNode`
*/
ControlFlowNode getAFlowNode() { py_flow_bb_node(result, this, _, _) }
cached
ControlFlowNode getAFlowNode() {
Stages::AST::ref() and
py_flow_bb_node(result, this, _, _)
}
/** Gets the location for this AST node */
cached
Location getLocation() { none() }
/**
@@ -35,6 +41,7 @@ abstract class AstNode extends AstNode_ {
* Expr.getASubExpression(), Stmt.getASubStatement(), Stmt.getASubExpression() or
* Scope.getAStmt().
*/
cached
abstract AstNode getAChildNode();
/**
@@ -44,12 +51,16 @@ abstract class AstNode extends AstNode_ {
* Expr.getASubExpression(), Stmt.getASubStatement(), Stmt.getASubExpression() or
* Scope.getAStmt() applied to the parent.
*/
AstNode getParentNode() { result.getAChildNode() = this }
cached
AstNode getParentNode() {
Stages::AST::ref() and
result.getAChildNode() = this
}
/** Whether this contains `inner` syntactically */
predicate contains(AstNode inner) { this.getAChildNode+() = inner }
pragma[noinline]
pragma[nomagic]
private predicate containsInScope(AstNode inner, Scope scope) {
this.contains(inner) and
not inner instanceof Scope and
@@ -61,28 +72,34 @@ abstract class AstNode extends AstNode_ {
}
/* Parents */
/** Internal implementation class */
library class FunctionParent extends FunctionParent_ { }
/** The parent of a `Function`. Internal implementation class */
class FunctionParent extends FunctionParent_ { }
/** Internal implementation class */
library class ArgumentsParent extends ArgumentsParent_ { }
/** The parent of an `Arguments` node. Internal implementation class */
class ArgumentsParent extends ArgumentsParent_ { }
/** Internal implementation class */
library class ExprListParent extends ExprListParent_ { }
/** The parent of an `ExprList`. Internal implementation class */
class ExprListParent extends ExprListParent_ { }
/** Internal implementation class */
library class ExprContextParent extends ExprContextParent_ { }
/** The parent of an `ExprContext`. Internal implementation class */
class ExprContextParent extends ExprContextParent_ { }
/** Internal implementation class */
library class StmtListParent extends StmtListParent_ { }
/** The parent of a `StmtList`. Internal implementation class */
class StmtListParent extends StmtListParent_ { }
/** Internal implementation class */
library class StrListParent extends StrListParent_ { }
/** The parent of a `StrList`. Internal implementation class */
class StrListParent extends StrListParent_ { }
/** Internal implementation class */
library class ExprParent extends ExprParent_ { }
/** The parent of an `Expr`. Internal implementation class */
class ExprParent extends ExprParent_ { }
library class DictItem extends DictItem_, AstNode {
/** The parent of a `PatternList`. Internal implementation class */
class PatternListParent extends PatternListParent_ { }
/** The parent of a `Pattern`. Internal implementation class */
class PatternParent extends PatternParent_ { }
class DictItem extends DictItem_, AstNode {
override string toString() { result = DictItem_.super.toString() }
override AstNode getAChildNode() { none() }
@@ -100,9 +117,16 @@ class Comprehension extends Comprehension_, AstNode {
override string toString() { result = "Comprehension" }
override Location getLocation() { result = Comprehension_.super.getLocation() }
override Location getLocation() {
Stages::AST::ref() and
result = Comprehension_.super.getLocation()
}
override AstNode getAChildNode() { result = this.getASubExpression() }
pragma[nomagic]
override AstNode getAChildNode() {
Stages::AST::ref() and
result = this.getASubExpression()
}
Expr getASubExpression() {
result = this.getIter() or
@@ -114,7 +138,7 @@ class Comprehension extends Comprehension_, AstNode {
class BytesOrStr extends BytesOrStr_ { }
/**
* Part of a string literal formed by implicit concatenation.
* A part of a string literal formed by implicit concatenation.
* For example the string literal "abc" expressed in the source as `"a" "b" "c"`
* would be composed of three `StringPart`s.
*/
@@ -162,9 +186,12 @@ class ExprList extends ExprList_ {
/* syntax: Expr, ... */
}
library class DictItemList extends DictItemList_ { }
/** A list of patterns */
class PatternList extends PatternList_ { }
library class DictItemListParent extends DictItemListParent_ { }
class DictItemList extends DictItemList_ { }
class DictItemListParent extends DictItemListParent_ { }
/** A list of strings (the primitive type string not Bytes or Unicode) */
class StringList extends StringList_ { }

540
python/ql/lib/semmle/python/AstGenerated.qll
View File

File diff suppressed because it is too large Load Diff

6
python/ql/lib/semmle/python/Class.qll
View File

@@ -85,12 +85,6 @@ class ClassDef extends Assign {
/** The scope of a class. This is the scope of all the statements within the class definition */
class Class extends Class_, Scope, AstNode {
/**
* Use getADecorator() instead of getDefinition().getADecorator()
* Use getMetaClass() instead of getDefinition().getMetaClass()
*/
deprecated ClassExpr getDefinition() { result = this.getParent() }
/** Gets a defined init method of this class */
Function getInitMethod() { result.getScope() = this and result.isInitMethod() }

4
python/ql/lib/semmle/python/Comment.qll
View File

@@ -40,7 +40,7 @@ class Comment extends @py_comment {
private predicate comment_block_part(Comment start, Comment part, int i) {
not exists(Comment prev | prev.getFollowing() = part) and
exists(Comment following | part.getFollowing() = following) and
exists(part.getFollowing()) and
start = part and
i = 1
or
@@ -59,7 +59,7 @@ class CommentBlock extends @py_comment {
/** Gets a textual representation of this element. */
string toString() { result = "Comment block" }
/** The length of this comment block (in comments) */
/** Gets the length of this comment block (in comments) */
int length() { result = max(int i | comment_block_part(this, _, i)) }
/**

14
python/ql/lib/semmle/python/Comparisons.qll
View File

@@ -76,22 +76,22 @@ class CompareOp extends int {
}
}
/** The `CompareOp` for "equals". */
/** Gets the `CompareOp` for "equals". */
CompareOp eq() { result = 1 }
/** The `CompareOp` for "not equals". */
/** Gets the `CompareOp` for "not equals". */
CompareOp ne() { result = 2 }
/** The `CompareOp` for "less than". */
/** Gets the `CompareOp` for "less than". */
CompareOp lt() { result = 3 }
/** The `CompareOp` for "less than or equal to". */
/** Gets the `CompareOp` for "less than or equal to". */
CompareOp le() { result = 4 }
/** The `CompareOp` for "greater than". */
/** Gets the `CompareOp` for "greater than". */
CompareOp gt() { result = 5 }
/** The `CompareOp` for "greater than or equal to". */
/** Gets the `CompareOp` for "greater than or equal to". */
CompareOp ge() { result = 6 }
/* Workaround precision limits in floating point numbers */
@@ -514,7 +514,7 @@ class ComparisonControlBlock extends ConditionBlock {
Comparison getTest() { this.getLastNode() = result }
/** Whether this conditional guard implies that, in block `b`, the result of `that` is `thatIsTrue` */
/** Whether this conditional guard implies that, in block `b`, the result of `that` is `thatIsTrue` */
predicate impliesThat(BasicBlock b, Comparison that, boolean thatIsTrue) {
exists(boolean controlSense |
this.controls(b, controlSense) and

2
python/ql/lib/semmle/python/Comprehensions.qll
View File

@@ -1,6 +1,6 @@
import python
/** Base class for list, set and dictionary comprehensions, and generator expressions. */
/** The base class for list, set and dictionary comprehensions, and generator expressions. */
abstract class Comp extends Expr {
abstract Function getFunction();

599
python/ql/lib/semmle/python/Concepts.qll
View File

@@ -17,13 +17,9 @@ private import semmle.python.Frameworks
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `SystemCommandExecution::Range` instead.
*/
class SystemCommandExecution extends DataFlow::Node {
SystemCommandExecution::Range range;
SystemCommandExecution() { this = range }
class SystemCommandExecution extends DataFlow::Node instanceof SystemCommandExecution::Range {
/** Gets the argument that specifies the command to be executed. */
DataFlow::Node getCommand() { result = range.getCommand() }
DataFlow::Node getCommand() { result = super.getCommand() }
}
/** Provides a class for modeling new system-command execution APIs. */
@@ -48,13 +44,9 @@ module SystemCommandExecution {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `FileSystemAccess::Range` instead.
*/
class FileSystemAccess extends DataFlow::Node {
FileSystemAccess::Range range;
FileSystemAccess() { this = range }
class FileSystemAccess extends DataFlow::Node instanceof FileSystemAccess::Range {
/** Gets an argument to this file system access that is interpreted as a path. */
DataFlow::Node getAPathArgument() { result = range.getAPathArgument() }
DataFlow::Node getAPathArgument() { result = super.getAPathArgument() }
}
/** Provides a class for modeling new file system access APIs. */
@@ -78,14 +70,12 @@ module FileSystemAccess {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `FileSystemWriteAccess::Range` instead.
*/
class FileSystemWriteAccess extends FileSystemAccess {
override FileSystemWriteAccess::Range range;
class FileSystemWriteAccess extends FileSystemAccess instanceof FileSystemWriteAccess::Range {
/**
* Gets a node that represents data to be written to the file system (possibly with
* some transformation happening before it is written, like JSON encoding).
*/
DataFlow::Node getADataNode() { result = range.getADataNode() }
DataFlow::Node getADataNode() { result = super.getADataNode() }
}
/** Provides a class for modeling new file system writes. */
@@ -111,10 +101,9 @@ module Path {
* A data-flow node that performs path normalization. This is often needed in order
* to safely access paths.
*/
class PathNormalization extends DataFlow::Node {
PathNormalization::Range range;
PathNormalization() { this = range }
class PathNormalization extends DataFlow::Node instanceof PathNormalization::Range {
/** Gets an argument to this path normalization that is interpreted as a path. */
DataFlow::Node getPathArg() { result = super.getPathArg() }
}
/** Provides a class for modeling new path normalization APIs. */
@@ -123,16 +112,17 @@ module Path {
* A data-flow node that performs path normalization. This is often needed in order
* to safely access paths.
*/
abstract class Range extends DataFlow::Node { }
abstract class Range extends DataFlow::Node {
/** Gets an argument to this path normalization that is interpreted as a path. */
abstract DataFlow::Node getPathArg();
}
}
/** A data-flow node that checks that a path is safe to access. */
class SafeAccessCheck extends DataFlow::BarrierGuard {
SafeAccessCheck::Range range;
SafeAccessCheck() { this = range }
override predicate checks(ControlFlowNode node, boolean branch) { range.checks(node, branch) }
class SafeAccessCheck extends DataFlow::BarrierGuard instanceof SafeAccessCheck::Range {
override predicate checks(ControlFlowNode node, boolean branch) {
SafeAccessCheck::Range.super.checks(node, branch)
}
}
/** Provides a class for modeling new path safety checks. */
@@ -154,22 +144,18 @@ module Path {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `Decoding::Range` instead.
*/
class Decoding extends DataFlow::Node {
Decoding::Range range;
Decoding() { this = range }
class Decoding extends DataFlow::Node instanceof Decoding::Range {
/** Holds if this call may execute code embedded in its input. */
predicate mayExecuteInput() { range.mayExecuteInput() }
predicate mayExecuteInput() { super.mayExecuteInput() }
/** Gets an input that is decoded by this function. */
DataFlow::Node getAnInput() { result = range.getAnInput() }
DataFlow::Node getAnInput() { result = super.getAnInput() }
/** Gets the output that contains the decoded data produced by this function. */
DataFlow::Node getOutput() { result = range.getOutput() }
DataFlow::Node getOutput() { result = super.getOutput() }
/** Gets an identifier for the format this function decodes from, such as "JSON". */
string getFormat() { result = range.getFormat() }
string getFormat() { result = super.getFormat() }
}
/** Provides a class for modeling new decoding mechanisms. */
@@ -220,19 +206,15 @@ private class DecodingAdditionalTaintStep extends TaintTracking::AdditionalTaint
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `Encoding::Range` instead.
*/
class Encoding extends DataFlow::Node {
Encoding::Range range;
Encoding() { this = range }
class Encoding extends DataFlow::Node instanceof Encoding::Range {
/** Gets an input that is encoded by this function. */
DataFlow::Node getAnInput() { result = range.getAnInput() }
DataFlow::Node getAnInput() { result = super.getAnInput() }
/** Gets the output that contains the encoded data produced by this function. */
DataFlow::Node getOutput() { result = range.getOutput() }
DataFlow::Node getOutput() { result = super.getOutput() }
/** Gets an identifier for the format this function decodes from, such as "JSON". */
string getFormat() { result = range.getFormat() }
string getFormat() { result = super.getFormat() }
}
/** Provides a class for modeling new encoding mechanisms. */
@@ -274,13 +256,9 @@ private class EncodingAdditionalTaintStep extends TaintTracking::AdditionalTaint
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `Logging::Range` instead.
*/
class Logging extends DataFlow::Node {
Logging::Range range;
Logging() { this = range }
class Logging extends DataFlow::Node instanceof Logging::Range {
/** Gets an input that is logged. */
DataFlow::Node getAnInput() { result = range.getAnInput() }
DataFlow::Node getAnInput() { result = super.getAnInput() }
}
/** Provides a class for modeling new logging mechanisms. */
@@ -303,13 +281,9 @@ module Logging {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `CodeExecution::Range` instead.
*/
class CodeExecution extends DataFlow::Node {
CodeExecution::Range range;
CodeExecution() { this = range }
class CodeExecution extends DataFlow::Node instanceof CodeExecution::Range {
/** Gets the argument that specifies the code to be executed. */
DataFlow::Node getCode() { result = range.getCode() }
DataFlow::Node getCode() { result = super.getCode() }
}
/** Provides a class for modeling new dynamic code execution APIs. */
@@ -326,19 +300,53 @@ module CodeExecution {
}
}
/**
* A data-flow node that constructs an SQL statement.
*
* Often, it is worthy of an alert if an SQL statement is constructed such that
* executing it would be a security risk.
*
* If it is important that the SQL statement is indeed executed, then use `SQLExecution`.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `SqlConstruction::Range` instead.
*/
class SqlConstruction extends DataFlow::Node instanceof SqlConstruction::Range {
/** Gets the argument that specifies the SQL statements to be constructed. */
DataFlow::Node getSql() { result = super.getSql() }
}
/** Provides a class for modeling new SQL execution APIs. */
module SqlConstruction {
/**
* A data-flow node that constructs an SQL statement.
*
* Often, it is worthy of an alert if an SQL statement is constructed such that
* executing it would be a security risk.
*
* If it is important that the SQL statement is indeed executed, then use `SQLExecution`.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `SqlConstruction` instead.
*/
abstract class Range extends DataFlow::Node {
/** Gets the argument that specifies the SQL statements to be constructed. */
abstract DataFlow::Node getSql();
}
}
/**
* A data-flow node that executes SQL statements.
*
* If the context of interest is such that merely constructing an SQL statement
* would be valuabe to report, then consider using `SqlConstruction`.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `SqlExecution::Range` instead.
*/
class SqlExecution extends DataFlow::Node {
SqlExecution::Range range;
SqlExecution() { this = range }
class SqlExecution extends DataFlow::Node instanceof SqlExecution::Range {
/** Gets the argument that specifies the SQL statements to be executed. */
DataFlow::Node getSql() { result = range.getSql() }
DataFlow::Node getSql() { result = super.getSql() }
}
/** Provides a class for modeling new SQL execution APIs. */
@@ -346,6 +354,9 @@ module SqlExecution {
/**
* A data-flow node that executes SQL statements.
*
* If the context of interest is such that merely constructing an SQL statement
* would be valuabe to report, then consider using `SqlConstruction`.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `SqlExecution` instead.
*/
@@ -361,22 +372,18 @@ module SqlExecution {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `RegexExecution::Range` instead.
*/
class RegexExecution extends DataFlow::Node {
RegexExecution::Range range;
RegexExecution() { this = range }
class RegexExecution extends DataFlow::Node instanceof RegexExecution::Range {
/** Gets the data flow node for the regex being executed by this node. */
DataFlow::Node getRegex() { result = range.getRegex() }
DataFlow::Node getRegex() { result = super.getRegex() }
/** Gets a dataflow node for the string to be searched or matched against. */
DataFlow::Node getString() { result = range.getString() }
DataFlow::Node getString() { result = super.getString() }
/**
* Gets the name of this regex execution, typically the name of an executing method.
* This is used for nice alert messages and should include the module if possible.
*/
string getName() { result = range.getName() }
string getName() { result = super.getName() }
}
/** Provides classes for modeling new regular-expression execution APIs. */
@@ -402,6 +409,190 @@ module RegexExecution {
}
}
/** Provides classes for modeling XML-related APIs. */
module XML {
/**
* A data-flow node that constructs an XPath expression.
*
* Often, it is worthy of an alert if an XPath expression is constructed such that
* executing it would be a security risk.
*
* If it is important that the XPath expression is indeed executed, then use `XPathExecution`.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `XPathConstruction::Range` instead.
*/
class XPathConstruction extends DataFlow::Node instanceof XPathConstruction::Range {
/** Gets the argument that specifies the XPath expressions to be constructed. */
DataFlow::Node getXPath() { result = super.getXPath() }
/**
* Gets the name of this XPath expression construction, typically the name of an executing method.
* This is used for nice alert messages and should include the module if possible.
*/
string getName() { result = super.getName() }
}
/** Provides a class for modeling new XPath construction APIs. */
module XPathConstruction {
/**
* A data-flow node that constructs an XPath expression.
*
* Often, it is worthy of an alert if an XPath expression is constructed such that
* executing it would be a security risk.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `XPathConstruction` instead.
*/
abstract class Range extends DataFlow::Node {
/** Gets the argument that specifies the XPath expressions to be constructed. */
abstract DataFlow::Node getXPath();
/**
* Gets the name of this XPath expression construction, typically the name of an executing method.
* This is used for nice alert messages and should include the module if possible.
*/
abstract string getName();
}
}
/**
* A data-flow node that executes a xpath expression.
*
* If the context of interest is such that merely constructing an XPath expression
* would be valuabe to report, then consider using `XPathConstruction`.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `XPathExecution::Range` instead.
*/
class XPathExecution extends DataFlow::Node instanceof XPathExecution::Range {
/** Gets the data flow node for the XPath expression being executed by this node. */
DataFlow::Node getXPath() { result = super.getXPath() }
/**
* Gets the name of this XPath expression execution, typically the name of an executing method.
* This is used for nice alert messages and should include the module if possible.
*/
string getName() { result = super.getName() }
}
/** Provides classes for modeling new regular-expression execution APIs. */
module XPathExecution {
/**
* A data-flow node that executes a XPath expression.
*
* If the context of interest is such that merely constructing an XPath expression
* would be valuabe to report, then consider using `XPathConstruction`.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `XPathExecution` instead.
*/
abstract class Range extends DataFlow::Node {
/** Gets the data flow node for the XPath expression being executed by this node. */
abstract DataFlow::Node getXPath();
/**
* Gets the name of this xpath expression execution, typically the name of an executing method.
* This is used for nice alert messages and should include the module if possible.
*/
abstract string getName();
}
}
/**
* A kind of XML vulnerability.
*
* See overview of kinds at https://pypi.org/project/defusedxml/#python-xml-libraries
*
* See PoC at `python/PoCs/XmlParsing/PoC.py` for some tests of vulnerable XML parsing.
*/
class XmlParsingVulnerabilityKind extends string {
XmlParsingVulnerabilityKind() { this in ["XML bomb", "XXE", "DTD retrieval"] }
/**
* Holds for XML bomb vulnerability kind, such as 'Billion Laughs' and 'Quadratic
* Blowup'.
*
* While a parser could technically be vulnerable to one and not the other, from our
* point of view the interesting part is that it IS vulnerable to these types of
* attacks, and not so much which one specifically works. In practice I haven't seen
* a parser that is vulnerable to one and not the other.
*/
predicate isXmlBomb() { this = "XML bomb" }
/** Holds for XXE vulnerability kind. */
predicate isXxe() { this = "XXE" }
/** Holds for DTD retrieval vulnerability kind. */
predicate isDtdRetrieval() { this = "DTD retrieval" }
}
/**
* A data-flow node that parses XML.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `XmlParsing` instead.
*/
class XmlParsing extends Decoding instanceof XmlParsing::Range {
/**
* Holds if this XML parsing is vulnerable to `kind`.
*/
predicate vulnerableTo(XmlParsingVulnerabilityKind kind) { super.vulnerableTo(kind) }
}
/** Provides classes for modeling XML parsing APIs. */
module XmlParsing {
/**
* A data-flow node that parses XML.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `XmlParsing` instead.
*/
abstract class Range extends Decoding::Range {
/**
* Holds if this XML parsing is vulnerable to `kind`.
*/
abstract predicate vulnerableTo(XmlParsingVulnerabilityKind kind);
override string getFormat() { result = "XML" }
}
}
}
/** Provides classes for modeling LDAP-related APIs. */
module LDAP {
/**
* A data-flow node that executes an LDAP query.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `LDAPQuery::Range` instead.
*/
class LdapExecution extends DataFlow::Node instanceof LdapExecution::Range {
/** Gets the argument containing the filter string. */
DataFlow::Node getFilter() { result = super.getFilter() }
/** Gets the argument containing the base DN. */
DataFlow::Node getBaseDn() { result = super.getBaseDn() }
}
/** Provides classes for modeling new LDAP query execution-related APIs. */
module LdapExecution {
/**
* A data-flow node that executes an LDAP query.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `LDAPQuery` instead.
*/
abstract class Range extends DataFlow::Node {
/** Gets the argument containing the filter string. */
abstract DataFlow::Node getFilter();
/** Gets the argument containing the base DN. */
abstract DataFlow::Node getBaseDn();
}
}
}
/**
* A data-flow node that escapes meta-characters, which could be used to prevent
* injection attacks.
@@ -409,26 +600,23 @@ module RegexExecution {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `Escaping::Range` instead.
*/
class Escaping extends DataFlow::Node {
Escaping::Range range;
class Escaping extends DataFlow::Node instanceof Escaping::Range {
Escaping() {
this = range and
// escapes that don't have _both_ input/output defined are not valid
exists(range.getAnInput()) and
exists(range.getOutput())
exists(super.getAnInput()) and
exists(super.getOutput())
}
/** Gets an input that will be escaped. */
DataFlow::Node getAnInput() { result = range.getAnInput() }
DataFlow::Node getAnInput() { result = super.getAnInput() }
/** Gets the output that contains the escaped data. */
DataFlow::Node getOutput() { result = range.getOutput() }
DataFlow::Node getOutput() { result = super.getOutput() }
/**
* Gets the context that this function escapes for, such as `html`, or `url`.
*/
string getKind() { result = range.getKind() }
string getKind() { result = super.getKind() }
}
/** Provides a class for modeling new escaping APIs. */
@@ -459,8 +647,20 @@ module Escaping {
/** Gets the escape-kind for escaping a string so it can safely be included in HTML. */
string getHtmlKind() { result = "html" }
/** Gets the escape-kind for escaping a string so it can safely be included in HTML. */
/** Gets the escape-kind for escaping a string so it can safely be included in a regular expression. */
string getRegexKind() { result = "regex" }
/**
* Gets the escape-kind for escaping a string so it can safely be used as a
* distinguished name (DN) in an LDAP search.
*/
string getLdapDnKind() { result = "ldap_dn" }
/**
* Gets the escape-kind for escaping a string so it can safely be used as a
* filter in an LDAP search.
*/
string getLdapFilterKind() { result = "ldap_filter" }
// TODO: If adding an XML kind, update the modeling of the `MarkupSafe` PyPI package.
//
// Technically it claims to escape for both HTML and XML, but for now we don't have
@@ -474,7 +674,7 @@ module Escaping {
* `<p>{}</p>`.
*/
class HtmlEscaping extends Escaping {
HtmlEscaping() { range.getKind() = Escaping::getHtmlKind() }
HtmlEscaping() { super.getKind() = Escaping::getHtmlKind() }
}
/**
@@ -482,12 +682,31 @@ class HtmlEscaping extends Escaping {
* the body of a regex.
*/
class RegexEscaping extends Escaping {
RegexEscaping() { range.getKind() = Escaping::getRegexKind() }
RegexEscaping() { super.getKind() = Escaping::getRegexKind() }
}
/**
* An escape of a string so it can be safely used as a distinguished name (DN)
* in an LDAP search.
*/
class LdapDnEscaping extends Escaping {
LdapDnEscaping() { super.getKind() = Escaping::getLdapDnKind() }
}
/**
* An escape of a string so it can be safely used as a filter in an LDAP search.
*/
class LdapFilterEscaping extends Escaping {
LdapFilterEscaping() { super.getKind() = Escaping::getLdapFilterKind() }
}
/** Provides classes for modeling HTTP-related APIs. */
module HTTP {
import semmle.python.web.HttpConstants
/** Gets an HTTP verb, in upper case */
string httpVerb() { result in ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "HEAD"] }
/** Gets an HTTP verb, in lower case */
string httpVerbLower() { result = httpVerb().toLowerCase() }
/** Provides classes for modeling HTTP servers. */
module Server {
@@ -497,29 +716,25 @@ module HTTP {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `RouteSetup::Range` instead.
*/
class RouteSetup extends DataFlow::Node {
RouteSetup::Range range;
RouteSetup() { this = range }
class RouteSetup extends DataFlow::Node instanceof RouteSetup::Range {
/** Gets the URL pattern for this route, if it can be statically determined. */
string getUrlPattern() { result = range.getUrlPattern() }
string getUrlPattern() { result = super.getUrlPattern() }
/**
* Gets a function that will handle incoming requests for this route, if any.
*
* NOTE: This will be modified in the near future to have a `RequestHandler` result, instead of a `Function`.
*/
Function getARequestHandler() { result = range.getARequestHandler() }
Function getARequestHandler() { result = super.getARequestHandler() }
/**
* Gets a parameter that will receive parts of the url when handling incoming
* requests for this route, if any. These automatically become a `RemoteFlowSource`.
*/
Parameter getARoutedParameter() { result = range.getARoutedParameter() }
Parameter getARoutedParameter() { result = super.getARoutedParameter() }
/** Gets a string that identifies the framework used for this route setup. */
string getFramework() { result = range.getFramework() }
string getFramework() { result = super.getFramework() }
}
/** Provides a class for modeling new HTTP routing APIs. */
@@ -566,19 +781,15 @@ module HTTP {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `RequestHandler::Range` instead.
*/
class RequestHandler extends Function {
RequestHandler::Range range;
RequestHandler() { this = range }
class RequestHandler extends Function instanceof RequestHandler::Range {
/**
* Gets a parameter that could receive parts of the url when handling incoming
* requests, if any. These automatically become a `RemoteFlowSource`.
*/
Parameter getARoutedParameter() { result = range.getARoutedParameter() }
Parameter getARoutedParameter() { result = super.getARoutedParameter() }
/** Gets a string that identifies the framework used for this route setup. */
string getFramework() { result = range.getFramework() }
string getFramework() { result = super.getFramework() }
}
/** Provides a class for modeling new HTTP request handlers. */
@@ -634,16 +845,12 @@ module HTTP {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `HttpResponse::Range` instead.
*/
class HttpResponse extends DataFlow::Node {
HttpResponse::Range range;
HttpResponse() { this = range }
class HttpResponse extends DataFlow::Node instanceof HttpResponse::Range {
/** Gets the data-flow node that specifies the body of this HTTP response. */
DataFlow::Node getBody() { result = range.getBody() }
DataFlow::Node getBody() { result = super.getBody() }
/** Gets the mimetype of this HTTP response, if it can be statically determined. */
string getMimetype() { result = range.getMimetype() }
string getMimetype() { result = super.getMimetype() }
}
/** Provides a class for modeling new HTTP response APIs. */
@@ -689,13 +896,9 @@ module HTTP {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `HttpRedirectResponse::Range` instead.
*/
class HttpRedirectResponse extends HttpResponse {
override HttpRedirectResponse::Range range;
HttpRedirectResponse() { this = range }
class HttpRedirectResponse extends HttpResponse instanceof HttpRedirectResponse::Range {
/** Gets the data-flow node that specifies the location of this HTTP redirect response. */
DataFlow::Node getRedirectLocation() { result = range.getRedirectLocation() }
DataFlow::Node getRedirectLocation() { result = super.getRedirectLocation() }
}
/** Provides a class for modeling new HTTP redirect response APIs. */
@@ -721,25 +924,21 @@ module HTTP {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `HTTP::CookieWrite::Range` instead.
*/
class CookieWrite extends DataFlow::Node {
CookieWrite::Range range;
CookieWrite() { this = range }
class CookieWrite extends DataFlow::Node instanceof CookieWrite::Range {
/**
* Gets the argument, if any, specifying the raw cookie header.
*/
DataFlow::Node getHeaderArg() { result = range.getHeaderArg() }
DataFlow::Node getHeaderArg() { result = super.getHeaderArg() }
/**
* Gets the argument, if any, specifying the cookie name.
*/
DataFlow::Node getNameArg() { result = range.getNameArg() }
DataFlow::Node getNameArg() { result = super.getNameArg() }
/**
* Gets the argument, if any, specifying the cookie value.
*/
DataFlow::Node getValueArg() { result = range.getValueArg() }
DataFlow::Node getValueArg() { result = super.getValueArg() }
}
/** Provides a class for modeling new cookie writes on HTTP responses. */
@@ -770,6 +969,142 @@ module HTTP {
abstract DataFlow::Node getValueArg();
}
}
/**
* A data-flow node that enables or disables Cross-site request forgery protection
* in a global manner.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `CsrfProtectionSetting::Range` instead.
*/
class CsrfProtectionSetting extends DataFlow::Node instanceof CsrfProtectionSetting::Range {
/**
* Gets the boolean value corresponding to if CSRF protection is enabled
* (`true`) or disabled (`false`) by this node.
*/
boolean getVerificationSetting() { result = super.getVerificationSetting() }
}
/** Provides a class for modeling new CSRF protection setting APIs. */
module CsrfProtectionSetting {
/**
* A data-flow node that enables or disables Cross-site request forgery protection
* in a global manner.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `CsrfProtectionSetting` instead.
*/
abstract class Range extends DataFlow::Node {
/**
* Gets the boolean value corresponding to if CSRF protection is enabled
* (`true`) or disabled (`false`) by this node.
*/
abstract boolean getVerificationSetting();
}
}
/**
* A data-flow node that enables or disables Cross-site request forgery protection
* for a specific part of an application.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `CsrfLocalProtectionSetting::Range` instead.
*/
class CsrfLocalProtectionSetting extends DataFlow::Node instanceof CsrfLocalProtectionSetting::Range {
/**
* Gets a request handler whose CSRF protection is changed.
*/
Function getRequestHandler() { result = super.getRequestHandler() }
/** Holds if CSRF protection is enabled by this setting */
predicate csrfEnabled() { super.csrfEnabled() }
}
/** Provides a class for modeling new CSRF protection setting APIs. */
module CsrfLocalProtectionSetting {
/**
* A data-flow node that enables or disables Cross-site request forgery protection
* for a specific part of an application.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `CsrfLocalProtectionSetting` instead.
*/
abstract class Range extends DataFlow::Node {
/**
* Gets a request handler whose CSRF protection is changed.
*/
abstract Function getRequestHandler();
/** Holds if CSRF protection is enabled by this setting */
abstract predicate csrfEnabled();
}
}
}
/** Provides classes for modeling HTTP clients. */
module Client {
/**
* A data-flow node that makes an outgoing HTTP request.
*
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `HTTP::Client::Request::Range` instead.
*/
class Request extends DataFlow::Node instanceof Request::Range {
/**
* Gets a data-flow node that contributes to the URL of the request.
* Depending on the framework, a request may have multiple nodes which contribute to the URL.
*/
DataFlow::Node getAUrlPart() { result = super.getAUrlPart() }
/** Gets a string that identifies the framework used for this request. */
string getFramework() { result = super.getFramework() }
/**
* Holds if this request is made using a mode that disables SSL/TLS
* certificate validation, where `disablingNode` represents the point at
* which the validation was disabled, and `argumentOrigin` represents the origin
* of the argument that disabled the validation (which could be the same node as
* `disablingNode`).
*/
predicate disablesCertificateValidation(
DataFlow::Node disablingNode, DataFlow::Node argumentOrigin
) {
super.disablesCertificateValidation(disablingNode, argumentOrigin)
}
}
/** Provides a class for modeling new HTTP requests. */
module Request {
/**
* A data-flow node that makes an outgoing HTTP request.
*
* Extend this class to model new APIs. If you want to refine existing API models,
* extend `HTTP::Client::Request` instead.
*/
abstract class Range extends DataFlow::Node {
/**
* Gets a data-flow node that contributes to the URL of the request.
* Depending on the framework, a request may have multiple nodes which contribute to the URL.
*/
abstract DataFlow::Node getAUrlPart();
/** Gets a string that identifies the framework used for this request. */
abstract string getFramework();
/**
* Holds if this request is made using a mode that disables SSL/TLS
* certificate validation, where `disablingNode` represents the point at
* which the validation was disabled, and `argumentOrigin` represents the origin
* of the argument that disabled the validation (which could be the same node as
* `disablingNode`).
*/
abstract predicate disablesCertificateValidation(
DataFlow::Node disablingNode, DataFlow::Node argumentOrigin
);
}
}
// TODO: investigate whether we should treat responses to client requests as
// remote-flow-sources in general.
}
}
@@ -790,27 +1125,23 @@ module Cryptography {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `KeyGeneration::Range` instead.
*/
class KeyGeneration extends DataFlow::Node {
KeyGeneration::Range range;
KeyGeneration() { this = range }
class KeyGeneration extends DataFlow::Node instanceof KeyGeneration::Range {
/** Gets the name of the cryptographic algorithm (for example `"RSA"` or `"AES"`). */
string getName() { result = range.getName() }
string getName() { result = super.getName() }
/** Gets the argument that specifies the size of the key in bits, if available. */
DataFlow::Node getKeySizeArg() { result = range.getKeySizeArg() }
DataFlow::Node getKeySizeArg() { result = super.getKeySizeArg() }
/**
* Gets the size of the key generated (in bits), as well as the `origin` that
* explains how we obtained this specific key size.
*/
int getKeySizeWithOrigin(DataFlow::Node origin) {
result = range.getKeySizeWithOrigin(origin)
result = super.getKeySizeWithOrigin(origin)
}
/** Gets the minimum key size (in bits) for this algorithm to be considered secure. */
int minimumSecureKeySize() { result = range.minimumSecureKeySize() }
int minimumSecureKeySize() { result = super.minimumSecureKeySize() }
}
/** Provides classes for modeling new key-pair generation APIs. */
@@ -889,16 +1220,12 @@ module Cryptography {
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `CryptographicOperation::Range` instead.
*/
class CryptographicOperation extends DataFlow::Node {
CryptographicOperation::Range range;
CryptographicOperation() { this = range }
class CryptographicOperation extends DataFlow::Node instanceof CryptographicOperation::Range {
/** Gets the algorithm used, if it matches a known `CryptographicAlgorithm`. */
CryptographicAlgorithm getAlgorithm() { result = range.getAlgorithm() }
CryptographicAlgorithm getAlgorithm() { result = super.getAlgorithm() }
/** Gets an input the algorithm is used on, for example the plain text input to be encrypted. */
DataFlow::Node getAnInput() { result = range.getAnInput() }
DataFlow::Node getAnInput() { result = super.getAnInput() }
}
/** Provides classes for modeling new applications of a cryptographic algorithms. */

52
python/ql/lib/semmle/python/Exprs.qll
View File

@@ -1,6 +1,7 @@
import python
private import semmle.python.pointsto.PointsTo
private import semmle.python.objects.ObjectInternal
private import semmle.python.internal.CachedStages
/** An expression */
class Expr extends Expr_, AstNode {
@@ -8,7 +9,11 @@ class Expr extends Expr_, AstNode {
override Scope getScope() { py_scopes(this, result) }
/** Gets a textual representation of this element. */
override string toString() { result = "Expression" }
cached
override string toString() {
Stages::AST::ref() and
result = "Expression"
}
/** Gets the module in which this expression occurs */
Module getEnclosingModule() { result = this.getScope().getEnclosingModule() }
@@ -30,9 +35,6 @@ class Expr extends Expr_, AstNode {
/** Whether this expression is a constant */
predicate isConstant() { not this.isVariable() }
/** Use isParenthesized instead. */
deprecated override predicate isParenthesised() { this.isParenthesized() }
/** Whether the parenthesized property of this expression is true. */
predicate isParenthesized() { Expr_.super.isParenthesised() }
@@ -49,9 +51,6 @@ class Expr extends Expr_, AstNode {
/** Gets an immediate (non-nested) sub-expression of this expression */
Expr getASubExpression() { none() }
/** Use StrConst.getText() instead */
deprecated string strValue() { none() }
override AstNode getAChildNode() { result = this.getASubExpression() }
/**
@@ -190,7 +189,16 @@ class Call extends Call_ {
*/
Keyword getKeyword(int index) {
result = this.getNamedArg(index) and
not exists(DictUnpacking d, int lower | d = this.getNamedArg(lower) and lower < index)
(
not exists(this.getMinimumUnpackingIndex())
or
index <= this.getMinimumUnpackingIndex()
)
}
/** Gets the minimum index (if any) at which a dictionary unpacking (`**foo`) occurs in this call. */
private int getMinimumUnpackingIndex() {
result = min(int i | this.getNamedArg(i) instanceof DictUnpacking)
}
/**
@@ -315,7 +323,7 @@ class Ellipsis extends Ellipsis_ {
}
/**
* Immutable literal expressions (except tuples).
* An immutable literal expression (except tuples).
* Consists of string (both unicode and byte) literals and numeric literals.
*/
abstract class ImmutableLiteral extends Expr {
@@ -446,6 +454,8 @@ class Unicode extends StrConst {
}
/**
* Gets the quoted representation fo this string.
*
* The extractor puts quotes into the name of each string (to prevent "0" clashing with 0).
* The following predicate help us match up a string/byte literals in the source
* which the equivalent object.
@@ -620,8 +630,6 @@ class StrConst extends Str_, ImmutableLiteral {
)
}
deprecated override string strValue() { result = this.getS() }
override Expr getASubExpression() { none() }
override AstNode getAChildNode() { result = this.getAnImplicitlyConcatenatedPart() }
@@ -685,7 +693,7 @@ class False extends BooleanLiteral {
override boolean booleanValue() { result = false }
}
/** `None` */
/** The `None` constant. */
class None extends NameConstant {
/* syntax: None */
None() { name_consts(this, "None") }
@@ -718,24 +726,30 @@ class FormattedValue extends FormattedValue_ {
}
}
/** A guard in a case statement */
class Guard extends Guard_ {
/* syntax: if Expr */
override Expr getASubExpression() { result = this.getTest() }
}
/* Expression Contexts */
/** A context in which an expression used */
class ExprContext extends ExprContext_ { }
/** Load context, the context of var in len(var) */
/** The load context, the context of var in len(var) */
class Load extends Load_ { }
/** Store context, the context of var in var = 0 */
/** The store context, the context of var in var = 0 */
class Store extends Store_ { }
/** Delete context, the context of var in del var */
/** The delete context, the context of var in del var */
class Del extends Del_ { }
/** This is an artifact of the Python grammar which includes an AugLoad context, even though it is never used. */
library class AugLoad extends AugLoad_ { }
/** The context of an augmented load. This is an artifact of the Python grammar which includes an AugLoad context, even though it is never used. */
class AugLoad extends AugLoad_ { }
/** Augmented store context, the context of var in var += 1 */
/** The augmented store context, the context of var in var += 1 */
class AugStore extends AugStore_ { }
/** Parameter context, the context of var in def f(var): pass */
/** The parameter context, the context of var in def f(var): pass */
class Param extends Param_ { }

21
python/ql/lib/semmle/python/Files.qll
View File

@@ -2,12 +2,6 @@ import python
/** A file */
class File extends Container, @file {
/** DEPRECATED: Use `getAbsolutePath` instead. */
deprecated override string getName() { result = this.getAbsolutePath() }
/** DEPRECATED: Use `getAbsolutePath` instead. */
deprecated string getFullName() { result = this.getAbsolutePath() }
/**
* Holds if this element is at the specified location.
* The location spans column `startcolumn` of line `startline` to
@@ -115,9 +109,6 @@ private predicate occupied_line(File f, int n) {
/** A folder (directory) */
class Folder extends Container, @folder {
/** DEPRECATED: Use `getAbsolutePath` instead. */
deprecated override string getName() { result = this.getAbsolutePath() }
/**
* Holds if this element is at the specified location.
* The location spans column `startcolumn` of line `startline` to
@@ -156,9 +147,6 @@ class Folder extends Container, @folder {
abstract class Container extends @container {
Container getParent() { containerparent(result, this) }
/** Gets a child of this container */
deprecated Container getChild() { containerparent(this, result) }
/**
* Gets a textual representation of the path of this container.
*
@@ -166,8 +154,11 @@ abstract class Container extends @container {
*/
string toString() { result = this.getAbsolutePath() }
/** Gets the name of this container */
abstract string getName();
/**
* Gets the name of this container.
* DEPRECATED: Use `getAbsolutePath` instead.
*/
deprecated string getName() { result = this.getAbsolutePath() }
/**
* Gets the relative path of this file or folder from the root folder of the
@@ -339,7 +330,7 @@ abstract class Container extends @container {
* paths. The list of paths is composed of the paths passed to the extractor and
* `sys.path`.
*/
predicate isImportRoot(int n) { this.getName() = import_path_element(n) }
predicate isImportRoot(int n) { this.getAbsolutePath() = import_path_element(n) }
/** Holds if this folder is the root folder for the standard library. */
predicate isStdLibRoot(int major, int minor) {

86
python/ql/lib/semmle/python/Flow.qll
View File

@@ -1,5 +1,6 @@
import python
private import semmle.python.pointsto.PointsTo
private import semmle.python.internal.CachedStages
/*
* Note about matching parent and child nodes and CFG splitting:
@@ -82,30 +83,12 @@ class ControlFlowNode extends @py_flow_node {
toAst(this) instanceof NameConstant
}
/** Use NameNode.isLoad() instead */
deprecated predicate isUse() { toAst(this) instanceof Name and this.isLoad() }
/** Use NameNode.isStore() */
deprecated predicate isDefinition() { toAst(this) instanceof Name and this.isStore() }
/** Whether this flow node corresponds to an attribute expression */
predicate isAttribute() { toAst(this) instanceof Attribute }
/** Use AttrNode.isLoad() instead */
deprecated predicate isAttributeLoad() { toAst(this) instanceof Attribute and this.isLoad() }
/** Use AttrNode.isStore() instead */
deprecated predicate isAttributeStore() { toAst(this) instanceof Attribute and this.isStore() }
/** Whether this flow node corresponds to an subscript expression */
predicate isSubscript() { toAst(this) instanceof Subscript }
/** Use SubscriptNode.isLoad() instead */
deprecated predicate isSubscriptLoad() { toAst(this) instanceof Subscript and this.isLoad() }
/** Use SubscriptNode.isStore() instead */
deprecated predicate isSubscriptStore() { toAst(this) instanceof Subscript and this.isStore() }
/** Whether this flow node corresponds to an import member */
predicate isImportMember() { toAst(this) instanceof ImportMember }
@@ -140,7 +123,9 @@ class ControlFlowNode extends @py_flow_node {
AstNode getNode() { py_flow_bb_node(this, result, _, _) }
/** Gets a textual representation of this element. */
cached
string toString() {
Stages::DataFlow::ref() and
exists(Scope s | s.getEntryNode() = this | result = "Entry node for " + s.toString())
or
exists(Scope s | s.getANormalExit() = this | result = "Exit node for " + s.toString())
@@ -155,7 +140,7 @@ class ControlFlowNode extends @py_flow_node {
/** Whether this flow node is the first in its scope */
predicate isEntryNode() { py_scope_flow(this, _, -1) }
/** The value that this ControlFlowNode points-to. */
/** Gets the value that this ControlFlowNode points-to. */
predicate pointsTo(Value value) { this.pointsTo(_, value, _) }
/** Gets the value that this ControlFlowNode points-to. */
@@ -164,10 +149,10 @@ class ControlFlowNode extends @py_flow_node {
/** Gets a value that this ControlFlowNode may points-to. */
Value inferredValue() { this.pointsTo(_, result, _) }
/** The value and origin that this ControlFlowNode points-to. */
/** Gets the value and origin that this ControlFlowNode points-to. */
predicate pointsTo(Value value, ControlFlowNode origin) { this.pointsTo(_, value, origin) }
/** The value and origin that this ControlFlowNode points-to, given the context. */
/** Gets the value and origin that this ControlFlowNode points-to, given the context. */
predicate pointsTo(Context context, Value value, ControlFlowNode origin) {
PointsTo::pointsTo(this, context, value, origin)
}
@@ -209,7 +194,9 @@ class ControlFlowNode extends @py_flow_node {
BasicBlock getBasicBlock() { result.contains(this) }
/** Gets the scope containing this flow node */
cached
Scope getScope() {
Stages::AST::ref() and
if this.getNode() instanceof Scope
then
/* Entry or exit node */
@@ -317,7 +304,7 @@ class ControlFlowNode extends @py_flow_node {
exists(BasicBlock b, int i, int j | this = b.getNode(i) and other = b.getNode(j) and i < j)
}
/* Holds if this CFG node is a branch */
/** Holds if this CFG node is a branch */
predicate isBranch() { py_true_successors(this, _) or py_false_successors(this, _) }
ControlFlowNode getAChild() { result = this.getExprChild(this.getBasicBlock()) }
@@ -376,7 +363,7 @@ class CallNode extends ControlFlowNode {
)
}
/** Gets the flow node corresponding to the nth argument of the call corresponding to this flow node */
/** Gets the flow node corresponding to the n'th positional argument of the call corresponding to this flow node */
ControlFlowNode getArg(int n) {
exists(Call c |
this.getNode() = c and
@@ -439,12 +426,6 @@ class AttrNode extends ControlFlowNode {
)
}
/** Use getObject() instead */
deprecated ControlFlowNode getValue() { result = this.getObject() }
/** Use getObject(name) instead */
deprecated ControlFlowNode getValue(string name) { result = this.getObject(name) }
/**
* Gets the flow node corresponding to the object of the attribute expression corresponding to this flow node,
* with the matching name
@@ -507,18 +488,6 @@ class ImportStarNode extends ControlFlowNode {
class SubscriptNode extends ControlFlowNode {
SubscriptNode() { toAst(this) instanceof Subscript }
/**
* DEPRECATED: Use `getObject()` instead.
* This will be formally deprecated before the end 2018 and removed in 2019.
*/
deprecated ControlFlowNode getValue() {
exists(Subscript s |
this.getNode() = s and
s.getObject() = result.getNode() and
result.getBasicBlock().dominates(this.getBasicBlock())
)
}
/** flow node corresponding to the value of the sequence in a subscript operation */
ControlFlowNode getObject() {
exists(Subscript s |
@@ -650,7 +619,9 @@ class UnaryExprNode extends ControlFlowNode {
* and nodes implicitly assigned in class and function definitions and imports.
*/
class DefinitionNode extends ControlFlowNode {
cached
DefinitionNode() {
Stages::AST::ref() and
exists(Assign a | a.getATarget().getAFlowNode() = this)
or
exists(AnnAssign a | a.getTarget().getAFlowNode() = this and exists(a.getValue()))
@@ -709,6 +680,7 @@ abstract class SequenceNode extends ControlFlowNode {
ControlFlowNode getAnElement() { result = this.getElement(_) }
/** Gets the control flow node for the nth element of this sequence */
cached
abstract ControlFlowNode getElement(int n);
}
@@ -717,6 +689,7 @@ class TupleNode extends SequenceNode {
TupleNode() { toAst(this) instanceof Tuple }
override ControlFlowNode getElement(int n) {
Stages::AST::ref() and
exists(Tuple t | this.getNode() = t and result.getNode() = t.getElt(n)) and
(
result.getBasicBlock().dominates(this.getBasicBlock())
@@ -950,10 +923,6 @@ class NameNode extends ControlFlowNode {
/** A control flow node corresponding to a named constant, one of `None`, `True` or `False`. */
class NameConstantNode extends NameNode {
NameConstantNode() { exists(NameConstant n | py_flow_bb_node(this, n, _, _)) }
deprecated override predicate defines(Variable v) { none() }
deprecated override predicate deletes(Variable v) { none() }
/*
* We ought to override uses as well, but that has
* a serious performance impact.
@@ -1009,12 +978,6 @@ private module Scopes {
scope = n.getEnclosingModule()
}
private predicate maybe_defined(SsaVariable var) {
exists(var.getDefinition()) and not py_ssa_phi(var, _) and not var.getDefinition().isDelete()
or
exists(SsaVariable input | input = var.getAPhiInput() | maybe_defined(input))
}
private predicate maybe_undefined(SsaVariable var) {
not exists(var.getDefinition()) and not py_ssa_phi(var, _)
or
@@ -1044,11 +1007,13 @@ class BasicBlock extends @py_flow_node {
string toString() { result = "BasicBlock" }
/** Whether this basic block strictly dominates the other */
pragma[nomagic]
predicate strictlyDominates(BasicBlock other) { other.getImmediateDominator+() = this }
cached
predicate strictlyDominates(BasicBlock other) {
Stages::AST::ref() and
other.getImmediateDominator+() = this
}
/** Whether this basic block dominates the other */
pragma[nomagic]
predicate dominates(BasicBlock other) {
this = other
or
@@ -1057,6 +1022,7 @@ class BasicBlock extends @py_flow_node {
cached
BasicBlock getImmediateDominator() {
Stages::AST::ref() and
this.firstNode().getImmediateDominator().getBasicBlock() = result
}
@@ -1094,7 +1060,11 @@ class BasicBlock extends @py_flow_node {
}
/** Gets a successor to this basic block */
BasicBlock getASuccessor() { result = this.getLastNode().getASuccessor().getBasicBlock() }
cached
BasicBlock getASuccessor() {
Stages::AST::ref() and
result = this.getLastNode().getASuccessor().getBasicBlock()
}
/** Gets a predecessor to this basic block */
BasicBlock getAPredecessor() { result.getASuccessor() = this }
@@ -1164,7 +1134,11 @@ class BasicBlock extends @py_flow_node {
}
/** Holds if this basic block strictly reaches the other. Is the start of other reachable from the end of this. */
predicate strictlyReaches(BasicBlock other) { this.getASuccessor+() = other }
cached
predicate strictlyReaches(BasicBlock other) {
Stages::AST::ref() and
this.getASuccessor+() = other
}
/** Holds if this basic block reaches the other. Is the start of other reachable from the end of this. */
predicate reaches(BasicBlock other) { this = other or this.strictlyReaches(other) }

19
python/ql/lib/semmle/python/Frameworks.qll
View File

@@ -6,31 +6,50 @@
// `docs/codeql/support/reusables/frameworks.rst`
private import semmle.python.frameworks.Aioch
private import semmle.python.frameworks.Aiohttp
private import semmle.python.frameworks.Aiomysql
private import semmle.python.frameworks.Aiopg
private import semmle.python.frameworks.Asyncpg
private import semmle.python.frameworks.ClickhouseDriver
private import semmle.python.frameworks.Cryptodome
private import semmle.python.frameworks.Cryptography
private import semmle.python.frameworks.Dill
private import semmle.python.frameworks.Django
private import semmle.python.frameworks.Fabric
private import semmle.python.frameworks.FastApi
private import semmle.python.frameworks.Flask
private import semmle.python.frameworks.FlaskAdmin
private import semmle.python.frameworks.FlaskSqlAlchemy
private import semmle.python.frameworks.Httpx
private import semmle.python.frameworks.Idna
private import semmle.python.frameworks.Invoke
private import semmle.python.frameworks.Jmespath
private import semmle.python.frameworks.Ldap
private import semmle.python.frameworks.Ldap3
private import semmle.python.frameworks.Libtaxii
private import semmle.python.frameworks.Libxml2
private import semmle.python.frameworks.Lxml
private import semmle.python.frameworks.MarkupSafe
private import semmle.python.frameworks.Multidict
private import semmle.python.frameworks.Mysql
private import semmle.python.frameworks.MySQLdb
private import semmle.python.frameworks.Peewee
private import semmle.python.frameworks.Psycopg2
private import semmle.python.frameworks.Pycurl
private import semmle.python.frameworks.Pydantic
private import semmle.python.frameworks.PyMySQL
private import semmle.python.frameworks.Requests
private import semmle.python.frameworks.RestFramework
private import semmle.python.frameworks.Rsa
private import semmle.python.frameworks.RuamelYaml
private import semmle.python.frameworks.Simplejson
private import semmle.python.frameworks.SqlAlchemy
private import semmle.python.frameworks.Starlette
private import semmle.python.frameworks.Stdlib
private import semmle.python.frameworks.Toml
private import semmle.python.frameworks.Tornado
private import semmle.python.frameworks.Twisted
private import semmle.python.frameworks.Ujson
private import semmle.python.frameworks.Urllib3
private import semmle.python.frameworks.Yaml
private import semmle.python.frameworks.Yarl
private import semmle.python.frameworks.Xmltodict

16
python/ql/lib/semmle/python/Function.qll
View File

@@ -5,11 +5,11 @@ import python
* It is the syntactic entity that is compiled to a code object.
*/
class Function extends Function_, Scope, AstNode {
/** The expression defining this function */
/** Gets the expression defining this function */
CallableExpr getDefinition() { result = this.getParent() }
/**
* The scope in which this function occurs, will be a class for a method,
* Gets the scope in which this function occurs. This will be a class for a method,
* another function for nested functions, generator expressions or comprehensions,
* or a module for a plain function.
*/
@@ -18,7 +18,7 @@ class Function extends Function_, Scope, AstNode {
override Scope getScope() { result = this.getEnclosingScope() }
/** Whether this function is declared in a class */
predicate isMethod() { exists(Class cls | this.getEnclosingScope() = cls) }
predicate isMethod() { this.getEnclosingScope() instanceof Class }
/** Whether this is a special method, that is does its name have the form `__xxx__` (except `__init__`) */
predicate isSpecialMethod() {
@@ -167,24 +167,24 @@ class Function extends Function_, Scope, AstNode {
/** A def statement. Note that FunctionDef extends Assign as a function definition binds the newly created function */
class FunctionDef extends Assign {
FunctionExpr f;
/* syntax: def name(...): ... */
FunctionDef() {
/* This is an artificial assignment the rhs of which is a (possibly decorated) FunctionExpr */
exists(FunctionExpr f | this.getValue() = f or this.getValue() = f.getADecoratorCall())
this.getValue() = f or this.getValue() = f.getADecoratorCall()
}
override string toString() { result = "FunctionDef" }
/** Gets the function for this statement */
Function getDefinedFunction() {
exists(FunctionExpr func | this.containsInScope(func) and result = func.getInnerScope())
}
Function getDefinedFunction() { result = f.getInnerScope() }
override Stmt getLastStatement() { result = this.getDefinedFunction().getLastStatement() }
}
/** A function that uses 'fast' locals, stored in the frame not in a dictionary. */
class FastLocalsFunction extends Function {
/** A function that uses 'fast' locals, stored in the frame not in a dictionary. */
FastLocalsFunction() {
not exists(ImportStar i | i.getScope() = this) and
not exists(Exec e | e.getScope() = this)

19
python/ql/lib/semmle/python/Import.qll
View File

@@ -1,5 +1,6 @@
import python
private import semmle.python.types.Builtins
private import semmle.python.internal.CachedStages
/**
* An alias in an import statement, the `mod as name` part of `import mod as name`. May be artificial;
@@ -35,6 +36,8 @@ class ImportExpr extends ImportExpr_ {
}
/**
* Gets the level of this import.
*
* The language specifies level as -1 if relative imports are to be tried first, 0 for absolute imports,
* and level > 0 for explicit relative imports.
*/
@@ -165,13 +168,6 @@ class Import extends Import_ {
result = this.getAName().getValue().(ImportMember).getModule()
}
/**
* Use getAnImportedModuleName(),
* possibly combined with ModuleObject.importedAs()
* Gets a module imported by this import statement
*/
deprecated Module getAModule() { result.getName() = this.getAnImportedModuleName() }
/** Whether this a `from ... import ...` statement */
predicate isFromImport() { this.getAName().getValue() instanceof ImportMember }
@@ -208,7 +204,9 @@ class Import extends Import_ {
/** An import * statement */
class ImportStar extends ImportStar_ {
/* syntax: from modname import * */
cached
ImportExpr getModuleExpr() {
Stages::AST::ref() and
result = this.getModule()
or
result = this.getModule().(ImportMember).getModule()
@@ -216,13 +214,6 @@ class ImportStar extends ImportStar_ {
override string toString() { result = "from " + this.getModuleExpr().getName() + " import *" }
/**
* Use getAnImportedModuleName(),
* possibly combined with ModuleObject.importedAs()
* Gets the module imported by this import * statement
*/
deprecated Module getTheModule() { result.getName() = this.getImportedModuleName() }
override Expr getASubExpression() { result = this.getModule() }
override Stmt getASubStatement() { none() }

18
python/ql/lib/semmle/python/Metrics.qll
View File

@@ -18,7 +18,7 @@ class FunctionMetrics extends Function {
int getNumberOfLinesOfDocStrings() { py_docstringlines(this, result) }
/**
* Cyclomatic complexity:
* Gets the cyclomatic complexity of the function:
* The number of linearly independent paths through the source code.
* Computed as E - N + 2P,
* where
@@ -27,9 +27,9 @@ class FunctionMetrics extends Function {
* P = the number of connected components, which for a single function is 1.
*/
int getCyclomaticComplexity() {
exists(int E, int N |
N = count(BasicBlock b | b = this.getABasicBlock() and b.likelyReachable()) and
E =
exists(int e, int n |
n = count(BasicBlock b | b = this.getABasicBlock() and b.likelyReachable()) and
e =
count(BasicBlock b1, BasicBlock b2 |
b1 = this.getABasicBlock() and
b1.likelyReachable() and
@@ -39,7 +39,7 @@ class FunctionMetrics extends Function {
not b1.unlikelySuccessor(b2)
)
|
result = E - N + 2
result = e - n + 2
)
}
@@ -130,13 +130,13 @@ class ClassMetrics extends Class {
}
/**
* The afferent coupling of a class is the number of classes that
* Gets the afferent coupling of a class -- the number of classes that
* directly depend on it.
*/
int getAfferentCoupling() { result = count(ClassMetrics t | t.dependsOn(this)) }
/**
* The efferent coupling of a class is the number of classes that
* Gets the efferent coupling of a class -- the number of classes that
* it directly depends on.
*/
int getEfferentCoupling() { result = count(ClassMetrics t | this.dependsOn(t)) }
@@ -273,13 +273,13 @@ class ModuleMetrics extends Module {
int getNumberOfLinesOfDocStrings() { py_docstringlines(this, result) }
/**
* The afferent coupling of a class is the number of classes that
* Gets the afferent coupling of a class -- the number of classes that
* directly depend on it.
*/
int getAfferentCoupling() { result = count(ModuleMetrics t | t.dependsOn(this)) }
/**
* The efferent coupling of a class is the number of classes that
* Gets the efferent coupling of a class -- the number of classes that
* it directly depends on.
*/
int getEfferentCoupling() { result = count(ModuleMetrics t | this.dependsOn(t)) }

17
python/ql/lib/semmle/python/Module.qll
View File

@@ -1,6 +1,6 @@
import python
private import semmle.python.objects.ObjectAPI
private import semmle.python.objects.Modules
private import semmle.python.internal.CachedStages
/**
* A module. This is the top level element in an AST, corresponding to a source file.
@@ -22,12 +22,13 @@ class Module extends Module_, Scope, AstNode {
}
/**
* Gets the enclosing scope of this module (always none).
*
* This method will be deprecated in the next release. Please use `getEnclosingScope()` instead.
* The enclosing scope of this module (always none)
*/
override Scope getScope() { none() }
/** The enclosing scope of this module (always none) */
/** Gets the enclosing scope of this module (always none) */
override Scope getEnclosingScope() { none() }
/** Gets the statements forming the body of this module */
@@ -98,12 +99,6 @@ class Module extends Module_, Scope, AstNode {
/** Gets the metrics for this module */
ModuleMetrics getMetrics() { result = this }
/**
* Use ModuleObject.getAnImportedModule() instead.
* Gets a module imported by this module
*/
deprecated Module getAnImportedModule() { result.getName() = this.getAnImportedModuleName() }
string getAnImportedModuleName() {
exists(Import i | i.getEnclosingModule() = this | result = i.getAnImportedModuleName())
or
@@ -196,7 +191,7 @@ private predicate isPotentialSourcePackage(Folder f) {
private predicate isPotentialPackage(Folder f) {
exists(f.getFile("__init__.py"))
or
py_flags_versioned("options.respect_init", "False", _) and major_version() = 2
py_flags_versioned("options.respect_init", "False", _) and major_version() = 2 and exists(f)
}
private string moduleNameFromBase(Container file) {
@@ -226,7 +221,9 @@ private predicate transitively_imported_from_entry_point(File file) {
)
}
cached
string moduleNameFromFile(Container file) {
Stages::AST::ref() and
exists(string basename |
basename = moduleNameFromBase(file) and
legalShortName(basename)

6
python/ql/lib/semmle/python/Operations.qll
View File

@@ -1,6 +1,6 @@
import python
/** Base class for operators */
/** The base class for operators */
class Operator extends Operator_ {
/** Gets the name of the special method used to implement this operator */
string getSpecialMethodName() { none() }
@@ -98,7 +98,7 @@ class LShift extends LShift_ {
override string getSpecialMethodName() { result = "__lshift__" }
}
/** A modulo (`%`) binary operator, which includes string formatting */
/** A modulo (`%`) binary operator, which includes string formatting */
class Mod extends Mod_ {
override string getSpecialMethodName() { result = "__mod__" }
}
@@ -131,7 +131,7 @@ class Compare extends Compare_ {
}
}
/** List of comparison operators in a comparison */
/** A list of comparison operators in a comparison */
class CmpopList extends CmpopList_ { }
/** A comparison operator */

119
python/ql/lib/semmle/python/Patterns.qll Normal file
View File

@@ -0,0 +1,119 @@
/**
* Wrapping generated AST classes: `Pattern_` and subclasses.
*/
import python
/** A pattern in a match statement */
class Pattern extends Pattern_, AstNode {
/** Gets the scope of this pattern */
override Scope getScope() { result = this.getCase().getScope() }
/** Gets the case statement containing this pattern */
pragma[nomagic]
Case getCase() { result.contains(this) }
override string toString() { result = "Pattern" }
/** Gets the module enclosing this pattern */
Module getEnclosingModule() { result = this.getScope().getEnclosingModule() }
/** Whether the parenthesized property of this expression is true. */
predicate isParenthesized() { Pattern_.super.isParenthesised() }
override Location getLocation() { result = Pattern_.super.getLocation() }
/** Gets an immediate (non-nested) sub-expression of this pattern */
Expr getASubExpression() { none() }
/** Gets an immediate (non-nested) sub-statement of this pattern */
Stmt getASubStatement() { none() }
/** Gets an immediate (non-nested) sub-pattern of this pattern */
Pattern getASubPattern() { none() }
override AstNode getAChildNode() {
result = this.getASubExpression()
or
result = this.getASubStatement()
or
result = this.getASubPattern()
}
}
/** An as-pattern in a match statement: `<subpattern> as alias` */
class MatchAsPattern extends MatchAsPattern_ {
override Pattern getASubPattern() { result = this.getPattern() }
override Expr getASubExpression() { result = this.getAlias() }
override Name getAlias() { result = super.getAlias() }
}
/** An or-pattern in a match statement: `(<pattern1>|<pattern2>)` */
class MatchOrPattern extends MatchOrPattern_ {
override Pattern getASubPattern() { result = this.getAPattern() }
}
/** A literal pattern in a match statement: `42` */
class MatchLiteralPattern extends MatchLiteralPattern_ {
override Expr getASubExpression() { result = this.getLiteral() }
}
/** A capture pattern in a match statement: `var` */
class MatchCapturePattern extends MatchCapturePattern_ {
/* syntax: varname */
override Expr getASubExpression() { result = this.getVariable() }
/** Gets the variable that is bound by this capture pattern */
override Name getVariable() { result = super.getVariable() }
}
/** A wildcard pattern in a match statement: `_` */
class MatchWildcardPattern extends MatchWildcardPattern_ { }
/** A value pattern in a match statement: `Http.OK` */
class MatchValuePattern extends MatchValuePattern_ {
override Expr getASubExpression() { result = this.getValue() }
}
/** A sequence pattern in a match statement `<p1>, <p2>` */
class MatchSequencePattern extends MatchSequencePattern_ {
override Pattern getASubPattern() { result = this.getAPattern() }
}
/** A star pattern in a match statement: `(..., *)` */
class MatchStarPattern extends MatchStarPattern_ {
override Pattern getASubPattern() { result = this.getTarget() }
}
/** A mapping pattern in a match statement: `{'a': var}` */
class MatchMappingPattern extends MatchMappingPattern_ {
override Pattern getASubPattern() { result = this.getAMapping() }
}
/** A double star pattern in a match statement: `{..., **}` */
class MatchDoubleStarPattern extends MatchDoubleStarPattern_ {
override Pattern getASubPattern() { result = this.getTarget() }
}
/** A key-value pattern inside a mapping pattern: `a: var` */
class MatchKeyValuePattern extends MatchKeyValuePattern_ {
override Pattern getASubPattern() { result = this.getKey() or result = this.getValue() }
}
/** A class pattern in a match statement: `Circle(radius = 3)` */
class MatchClassPattern extends MatchClassPattern_ {
override Expr getASubExpression() { result = this.getClassName() }
override Pattern getASubPattern() {
result = this.getAPositional() or result = this.getAKeyword()
}
}
/** A keyword pattern inside a class pattern: `radius = 3` */
class MatchKeywordPattern extends MatchKeywordPattern_ {
override Expr getASubExpression() { result = this.getAttribute() }
override Pattern getASubPattern() { result = this.getValue() }
}

41
python/ql/lib/semmle/python/PrintAst.qll
View File

@@ -53,9 +53,6 @@ private newtype TPrintAstNode =
shouldPrint(list.getAnItem(), _) and
not list = any(Module mod).getBody() and
not forall(AstNode child | child = list.getAnItem() | isNotNeeded(child))
} or
TRegExpTermNode(RegExpTerm term) {
exists(StrConst str | term.getRootTerm() = getParsedRegExp(str) and shouldPrint(str, _))
}
/**
@@ -76,7 +73,7 @@ class PrintAstNode extends TPrintAstNode {
/**
* Gets a child of this node.
*/
final PrintAstNode getAChild() { result = getChild(_) }
final PrintAstNode getAChild() { result = this.getChild(_) }
/**
* Gets the parent of this node, if any.
@@ -94,7 +91,7 @@ class PrintAstNode extends TPrintAstNode {
*/
string getProperty(string key) {
key = "semmle.label" and
result = toString()
result = this.toString()
}
/**
@@ -103,7 +100,7 @@ class PrintAstNode extends TPrintAstNode {
* this.
*/
string getChildEdgeLabel(int childIndex) {
exists(getChild(childIndex)) and
exists(this.getChild(childIndex)) and
result = childIndex.toString()
}
}
@@ -157,13 +154,13 @@ class AstElementNode extends PrintAstNode, TElementNode {
override PrintAstNode getChild(int childIndex) {
exists(AstNode el | result.(AstElementNode).getAstNode() = el |
el = this.getChildNode(childIndex) and not el = getStmtList(_, _).getAnItem()
el = this.getChildNode(childIndex) and not el = this.getStmtList(_, _).getAnItem()
)
or
// displaying all `StmtList` after the other children.
exists(int offset | offset = 1 + max([0, any(int index | exists(this.getChildNode(index)))]) |
exists(int index | childIndex = index + offset |
result.(StmtListNode).getList() = getStmtList(index, _)
result.(StmtListNode).getList() = this.getStmtList(index, _)
)
)
}
@@ -299,7 +296,7 @@ class StmtListNode extends PrintAstNode, TStmtListNode {
private string getLabel() { this.getList() = any(AstElementNode node).getStmtList(_, result) }
override string toString() { result = "(StmtList) " + getLabel() }
override string toString() { result = "(StmtList) " + this.getLabel() }
override PrintAstNode getChild(int childIndex) {
exists(AstNode el | result.(AstElementNode).getAstNode() = el | el = list.getItem(childIndex))
@@ -430,32 +427,6 @@ class ParameterNode extends AstElementNode {
*/
class StrConstNode extends AstElementNode {
override StrConst element;
override PrintAstNode getChild(int childIndex) {
childIndex = 0 and result.(RegExpTermNode).getTerm() = getParsedRegExp(element)
}
}
/**
* A print node for a regular expression term.
*/
class RegExpTermNode extends TRegExpTermNode, PrintAstNode {
RegExpTerm term;
RegExpTermNode() { this = TRegExpTermNode(term) }
/** Gets the `RegExpTerm` for this node. */
RegExpTerm getTerm() { result = term }
override PrintAstNode getChild(int childIndex) {
result.(RegExpTermNode).getTerm() = term.getChild(childIndex)
}
override string toString() {
result = "[" + strictconcat(term.getPrimaryQLClass(), " | ") + "] " + term.toString()
}
override Location getLocation() { result = term.getLocation() }
}
/**

38
python/ql/lib/semmle/python/RegexTreeView.qll
View File

@@ -39,7 +39,12 @@ newtype TRegExpParent =
/** A special character */
TRegExpSpecialChar(Regex re, int start, int end) { re.specialCharacter(start, end, _) } or
/** A normal character */
TRegExpNormalChar(Regex re, int start, int end) { re.normalCharacter(start, end) } or
TRegExpNormalChar(Regex re, int start, int end) {
re.normalCharacterSequence(start, end)
or
re.escapedCharacter(start, end) and
not re.specialCharacter(start, end, _)
} or
/** A back reference */
TRegExpBackRef(Regex re, int start, int end) { re.backreference(start, end) }
@@ -440,6 +445,8 @@ class RegExpAlt extends RegExpTerm, TRegExpAlt {
override string getPrimaryQLClass() { result = "RegExpAlt" }
}
class RegExpCharEscape = RegExpEscape;
/**
* An escaped regular expression term, that is, a regular expression
* term starting with a backslash, which is not a backreference.
@@ -467,9 +474,10 @@ class RegExpEscape extends RegExpNormalChar {
or
this.getUnescaped() = "t" and result = "\t"
or
// TODO: Find a way to include a formfeed character
// this.getUnescaped() = "f" and result = " "
// or
this.getUnescaped() = "f" and result = 12.toUnicode()
or
this.getUnescaped() = "v" and result = 11.toUnicode()
or
this.isUnicode() and
result = this.getUnicode()
}
@@ -480,7 +488,7 @@ class RegExpEscape extends RegExpNormalChar {
override string getPrimaryQLClass() { result = "RegExpEscape" }
/** Gets the part of the term following the escape character. That is e.g. "w" if the term is "\w". */
private string getUnescaped() { result = this.getText().suffix(1) }
string getUnescaped() { result = this.getText().suffix(1) }
/**
* Gets the text for this escape. That is e.g. "\w".
@@ -535,9 +543,16 @@ private int toHex(string hex) {
result = 15 and hex = ["f", "F"]
}
/**
* A word boundary, that is, a regular expression term of the form `\b`.
*/
class RegExpWordBoundary extends RegExpSpecialChar {
RegExpWordBoundary() { this.getChar() = "\\b" }
}
/**
* A character class escape in a regular expression.
* That is, an escaped charachter that denotes multiple characters.
* That is, an escaped character that denotes multiple characters.
*
* Examples:
*
@@ -738,6 +753,9 @@ class RegExpGroup extends RegExpTerm, TRegExpGroup {
*/
int getNumber() { result = re.getGroupNumber(start, end) }
/** Holds if this is a capture group. */
predicate isCapture() { exists(this.getNumber()) }
/** Holds if this is a named capture group. */
predicate isNamed() { exists(this.getName()) }
@@ -801,7 +819,7 @@ class RegExpDot extends RegExpSpecialChar {
}
/**
* A dollar assertion `$` matching the end of a line.
* A dollar assertion `$` or `\Z` matching the end of a line.
*
* Example:
*
@@ -810,13 +828,13 @@ class RegExpDot extends RegExpSpecialChar {
* ```
*/
class RegExpDollar extends RegExpSpecialChar {
RegExpDollar() { this.getChar() = "$" }
RegExpDollar() { this.getChar() = ["$", "\\Z"] }
override string getPrimaryQLClass() { result = "RegExpDollar" }
}
/**
* A caret assertion `^` matching the beginning of a line.
* A caret assertion `^` or `\A` matching the beginning of a line.
*
* Example:
*
@@ -825,7 +843,7 @@ class RegExpDollar extends RegExpSpecialChar {
* ```
*/
class RegExpCaret extends RegExpSpecialChar {
RegExpCaret() { this.getChar() = "^" }
RegExpCaret() { this.getChar() = ["^", "\\A"] }
override string getPrimaryQLClass() { result = "RegExpCaret" }
}

2
python/ql/lib/semmle/python/SSA.qll
View File

@@ -86,7 +86,7 @@ class SsaVariable extends @py_ssa_var {
/** Gets the incoming edges for a Phi node. */
private BasicBlock getAPredecessorBlockForPhi() {
exists(getAPhiInput()) and
exists(this.getAPhiInput()) and
result.getASuccessor() = this.getDefinition().getBasicBlock()
}

3
python/ql/lib/semmle/python/Scope.qll
View File

@@ -9,11 +9,12 @@ class Scope extends Scope_ {
Module getEnclosingModule() { result = this.getEnclosingScope().getEnclosingModule() }
/**
* Gets the scope enclosing this scope (modules have no enclosing scope).
*
* This method will be deprecated in the next release. Please use `getEnclosingScope()` instead.
* The reason for this is to avoid confusion around use of `x.getScope+()` where `x` might be an
* `AstNode` or a `Variable`. Forcing the users to write `x.getScope().getEnclosingScope*()` ensures that
* the apparent semantics and the actual semantics coincide.
* [ Gets the scope enclosing this scope (modules have no enclosing scope) ]
*/
Scope getScope() { none() }

2
python/ql/lib/semmle/python/SelfAttribute.qll
View File

@@ -31,7 +31,7 @@ private predicate self_attribute(Attribute attr, Class cls) {
)
}
/** Helper class for UndefinedClassAttribute.ql &amp; MaybeUndefinedClassAttribute.ql */
/** A helper class for UndefinedClassAttribute.ql &amp; MaybeUndefinedClassAttribute.ql */
class SelfAttributeRead extends SelfAttribute {
SelfAttributeRead() {
this.getCtx() instanceof Load and

6
python/ql/lib/semmle/python/SpecialMethods.qll
View File

@@ -8,7 +8,7 @@
* Extend `SpecialMethod::Potential` to capture more cases.
*/
import python
private import python
/** A control flow node which might correspond to a special method call. */
class PotentialSpecialMethodCallNode extends ControlFlowNode {
@@ -107,11 +107,11 @@ class SpecialMethodCallNode extends PotentialSpecialMethodCallNode {
SpecialMethodCallNode() {
exists(SpecialMethod::Potential pot |
this.(SpecialMethod::Potential) = pot and
this = pot and
pot.getSelf().pointsTo().getClass().lookup(pot.getSpecialMethodName()) = resolvedSpecialMethod
)
}
/** The method that is called. */
/** Gets the method that is called. */
Value getResolvedSpecialMethod() { result = resolvedSpecialMethod }
}

31
python/ql/lib/semmle/python/Stmts.qll
View File

@@ -18,10 +18,15 @@ class Stmt extends Stmt_, AstNode {
/** Gets an immediate (non-nested) sub-statement of this statement */
Stmt getASubStatement() { none() }
/** Gets an immediate (non-nested) sub-pattern of this statement */
Pattern getASubPattern() { none() }
override AstNode getAChildNode() {
result = this.getASubExpression()
or
result = this.getASubStatement()
or
result = this.getASubPattern()
}
private ControlFlowNode possibleEntryNode() {
@@ -94,13 +99,13 @@ class AugAssign extends AugAssign_ {
* Gets the target of this augmented assignment statement.
* That is, the `a` in `a += b`.
*/
Expr getTarget() { result = this.getOperation().(BinaryExpr).getLeft() }
Expr getTarget() { result = this.getOperation().getLeft() }
/**
* Gets the value of this augmented assignment statement.
* That is, the `b` in `a += b`.
*/
Expr getValue() { result = this.getOperation().(BinaryExpr).getRight() }
Expr getValue() { result = this.getOperation().getRight() }
override Stmt getASubStatement() { none() }
}
@@ -318,7 +323,7 @@ class Raise extends Raise_ {
override Expr getASubExpression() { py_exprs(result, _, this, _) }
/**
* The expression immediately following the `raise`, this is the
* Gets the expression immediately following the `raise`. This is the
* exception raised, but not accounting for tuples in Python 2.
*/
Expr getException() {
@@ -327,7 +332,7 @@ class Raise extends Raise_ {
result = this.getExc()
}
/** The exception raised, accounting for tuples in Python 2. */
/** Gets the exception raised, accounting for tuples in Python 2. */
Expr getRaised() {
exists(Expr raw | raw = this.getException() |
if not major_version() = 2 or not exists(raw.(Tuple).getAnElt())
@@ -412,6 +417,24 @@ class With extends With_ {
override Stmt getLastStatement() { result = this.getBody().getLastItem().getLastStatement() }
}
/** A match statement */
class MatchStmt extends MatchStmt_ {
/* syntax: match subject: */
override Expr getASubExpression() { result = this.getSubject() }
override Stmt getASubStatement() { result = this.getCase(_) }
}
/** A case statement */
class Case extends Case_ {
/* syntax: case pattern if guard: */
override Expr getASubExpression() { result = this.getGuard() }
override Stmt getASubStatement() { result = this.getStmt(_) }
override Pattern getASubPattern() { result = this.getPattern() }
}
/** A plain text used in a template is wrapped in a TemplateWrite statement */
class TemplateWrite extends TemplateWrite_ {
override Expr getASubExpression() { result = this.getValue() }

4
python/ql/lib/semmle/python/Variables.qll
View File

@@ -57,7 +57,7 @@ class LocalVariable extends Variable {
override string toString() { result = "Local Variable " + this.getId() }
/** Whether this variable is a parameter */
override predicate isParameter() { exists(Parameter p | this.getAnAccess() = p) }
override predicate isParameter() { this.getAnAccess() instanceof Parameter }
/** Holds if this variable is the first parameter of a method. It is not necessarily called "self" */
override predicate isSelf() {
@@ -87,7 +87,7 @@ class NameLocalVariable extends LocalVariable {
/** A global (module-level) variable */
class GlobalVariable extends Variable {
GlobalVariable() { exists(Module m | m = this.getScope()) }
GlobalVariable() { this.getScope() instanceof Module }
override string toString() { result = "Global Variable " + this.getId() }
}

55
python/ql/lib/semmle/python/concepts/CryptoAlgorithms.qll
View File

@@ -1,55 +1,10 @@
/**
* Provides classes modeling cryptographic algorithms, separated into strong and weak variants.
*
* The classification into strong and weak are based on Wikipedia, OWASP and google (2017).
* The classification into strong and weak are based on Wikipedia, OWASP and Google (2021).
*/
/**
* Names of cryptographic algorithms, separated into strong and weak variants.
*
* The names are normalized: upper-case, no spaces, dashes or underscores.
*
* The names are inspired by the names used in real world crypto libraries.
*
* The classification into strong and weak are based on Wikipedia, OWASP and google (2017).
*/
private module AlgorithmNames {
predicate isStrongHashingAlgorithm(string name) {
name =
[
"DSA", "ED25519", "ES256", "ECDSA256", "ES384", "ECDSA384", "ES512", "ECDSA512", "SHA2",
"SHA224", "SHA256", "SHA384", "SHA512", "SHA3", "SHA3224", "SHA3256", "SHA3384", "SHA3512"
]
}
predicate isWeakHashingAlgorithm(string name) {
name =
[
"HAVEL128", "MD2", "MD4", "MD5", "PANAMA", "RIPEMD", "RIPEMD128", "RIPEMD256", "RIPEMD160",
"RIPEMD320", "SHA0", "SHA1"
]
}
predicate isStrongEncryptionAlgorithm(string name) {
name = ["AES", "AES128", "AES192", "AES256", "AES512", "RSA", "RABBIT", "BLOWFISH"]
}
predicate isWeakEncryptionAlgorithm(string name) {
name =
[
"DES", "3DES", "TRIPLEDES", "TDEA", "TRIPLEDEA", "ARC2", "RC2", "ARC4", "RC4", "ARCFOUR",
"ARC5", "RC5"
]
}
predicate isStrongPasswordHashingAlgorithm(string name) {
name = ["ARGON2", "PBKDF2", "BCRYPT", "SCRYPT"]
}
predicate isWeakPasswordHashingAlgorithm(string name) { none() }
}
private import AlgorithmNames
private import internal.CryptoAlgorithmNames
/**
* A cryptographic algorithm.
@@ -85,11 +40,13 @@ abstract class CryptographicAlgorithm extends TCryptographicAlgorithm {
/**
* Holds if the name of this algorithm matches `name` modulo case,
* white space, dashes, and underscores.
* white space, dashes, underscores, and anything after a dash in the name
* (to ignore modes of operation, such as CBC or ECB).
*/
bindingset[name]
predicate matchesName(string name) {
name.toUpperCase().regexpReplaceAll("[-_ ]", "") = getName()
[name.toUpperCase(), name.toUpperCase().regexpCapture("^(\\w+)(?:-.*)?$", 1)]
.regexpReplaceAll("[-_ ]", "") = getName()
}
/**

72
python/ql/lib/semmle/python/concepts/internal/CryptoAlgorithmNames.qll Normal file
View File

@@ -0,0 +1,72 @@
/**
* Names of cryptographic algorithms, separated into strong and weak variants.
*
* The names are normalized: upper-case, no spaces, dashes or underscores.
*
* The names are inspired by the names used in real world crypto libraries.
*
* The classification into strong and weak are based on Wikipedia, OWASP and Google (2021).
*/
/**
* Holds if `name` corresponds to a strong hashing algorithm.
*/
predicate isStrongHashingAlgorithm(string name) {
name =
[
"DSA", "ED25519", "ES256", "ECDSA256", "ES384", "ECDSA384", "ES512", "ECDSA512", "SHA2",
"SHA224", "SHA256", "SHA384", "SHA512", "SHA3", "SHA3224", "SHA3256", "SHA3384", "SHA3512"
]
}
/**
* Holds if `name` corresponds to a weak hashing algorithm.
*/
predicate isWeakHashingAlgorithm(string name) {
name =
[
"HAVEL128", "MD2", "MD4", "MD5", "PANAMA", "RIPEMD", "RIPEMD128", "RIPEMD256", "RIPEMD160",
"RIPEMD320", "SHA0", "SHA1"
]
}
/**
* Holds if `name` corresponds to a strong encryption algorithm.
*/
predicate isStrongEncryptionAlgorithm(string name) {
name =
[
"AES", "AES128", "AES192", "AES256", "AES512", "AES-128", "AES-192", "AES-256", "AES-512",
"ARIA", "BLOWFISH", "BF", "ECIES", "CAST", "CAST5", "CAMELLIA", "CAMELLIA128", "CAMELLIA192",
"CAMELLIA256", "CAMELLIA-128", "CAMELLIA-192", "CAMELLIA-256", "CHACHA", "GOST", "GOST89",
"IDEA", "RABBIT", "RSA", "SEED", "SM4"
]
}
/**
* Holds if `name` corresponds to a weak encryption algorithm.
*/
predicate isWeakEncryptionAlgorithm(string name) {
name =
[
"DES", "3DES", "DES3", "TRIPLEDES", "DESX", "TDEA", "TRIPLEDEA", "ARC2", "RC2", "ARC4", "RC4",
"ARCFOUR", "ARC5", "RC5"
]
}
/**
* Holds if `name` corresponds to a strong password hashing algorithm.
*/
predicate isStrongPasswordHashingAlgorithm(string name) {
name = ["ARGON2", "PBKDF2", "BCRYPT", "SCRYPT"]
}
/**
* Holds if `name` corresponds to a weak password hashing algorithm.
*/
predicate isWeakPasswordHashingAlgorithm(string name) { name = "EVPKDF" }
/**
* Holds if `name` corresponds to a weak block cipher mode of operation.
*/
predicate isWeakBlockMode(string name) { name = "ECB" }

41
python/ql/lib/semmle/python/dataflow/new/Regexp.qll Normal file
View File

@@ -0,0 +1,41 @@
/**
* Provides classes for working with regular expressions.
*/
private import semmle.python.RegexTreeView
private import semmle.python.regex
private import semmle.python.dataflow.new.DataFlow
/**
* Provides utility predicates related to regular expressions.
*/
module RegExpPatterns {
/**
* Gets a pattern that matches common top-level domain names in lower case.
*/
string getACommonTld() {
// according to ranking by http://google.com/search?q=site:.<<TLD>>
result = "(?:com|org|edu|gov|uk|net|io)(?![a-z0-9])"
}
}
/**
* A node whose value may flow to a position where it is interpreted
* as a part of a regular expression.
*/
class RegExpPatternSource extends DataFlow::CfgNode {
private Regex astNode;
RegExpPatternSource() { astNode = this.asExpr() }
/**
* Gets a node where the pattern of this node is parsed as a part of
* a regular expression.
*/
DataFlow::Node getAParse() { result = this }
/**
* Gets the root term of the regular expression parsed from this pattern.
*/
RegExpTerm getRegExpTerm() { result.getRegex() = astNode }
}

2
python/ql/lib/semmle/python/dataflow/new/SensitiveDataSources.qll
View File

@@ -227,7 +227,7 @@ private module SensitiveDataModeling {
}
/**
* Any kind of variable assignment (also including with/for) where the name indicates
* A variable assignment (also including with/for) where the name indicates
* it contains sensitive data.
*
* Note: We _could_ make any access to a variable with a sensitive name a source of

6
python/ql/lib/semmle/python/dataflow/new/TypeTracker.qll
View File

@@ -6,14 +6,14 @@
private import python
private import internal.TypeTracker as Internal
/** Any string that may appear as the name of an attribute or access path. */
/** A string that may appear as the name of an attribute or access path. */
class AttributeName = Internal::ContentName;
/** Either an attribute name, or the empty string (representing no attribute). */
/** An attribute name, or the empty string (representing no attribute). */
class OptionalAttributeName = Internal::OptionalContentName;
/**
* Summary of the steps needed to track a value to a given dataflow node.
* The summary of the steps needed to track a value to a given dataflow node.
*
* This can be used to track objects that implement a certain API in order to
* recognize calls to that API. Note that type-tracking does not by itself provide a

28
python/ql/lib/semmle/python/dataflow/new/internal/Attributes.qll
View File

@@ -92,6 +92,32 @@ private class AttributeAssignmentAsAttrWrite extends AttrWrite, CfgNode {
override string getAttributeName() { result = node.getName() }
}
/**
* An attribute assignment where the object is a global variable: `global_var.attr = value`.
*
* Syntactically, this is identical to the situation that is covered by
* `AttributeAssignmentAsAttrWrite`, however in this case we want to behave as if the object that is
* being written is the underlying `ModuleVariableNode`.
*/
private class GlobalAttributeAssignmentAsAttrWrite extends AttrWrite, CfgNode {
override AttributeAssignmentNode node;
override Node getValue() { result.asCfgNode() = node.getValue() }
override Node getObject() {
result.(ModuleVariableNode).getVariable() = node.getObject().getNode().(Name).getVariable()
}
override ExprNode getAttributeNameExpr() {
// Attribute names don't exist as `Node`s in the control flow graph, as they can only ever be
// identifiers, and are therefore represented directly as strings.
// Use `getAttributeName` to access the name of the attribute.
none()
}
override string getAttributeName() { result = node.getName() }
}
/** Represents `CallNode`s that may refer to calls to built-in functions or classes. */
private class BuiltInCallNode extends CallNode {
string name;
@@ -204,6 +230,8 @@ abstract class AttrRead extends AttrRef, Node, LocalSourceNode { }
private class AttributeReadAsAttrRead extends AttrRead, CfgNode {
override AttrNode node;
AttributeReadAsAttrRead() { node.isLoad() }
override Node getObject() { result.asCfgNode() = node.getObject() }
override ExprNode getAttributeNameExpr() {

93
python/ql/lib/semmle/python/dataflow/new/internal/Builtins.qll Normal file
View File

@@ -0,0 +1,93 @@
/** Provides predicates for reasoning about built-ins in Python. */
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.internal.ImportStar
module Builtins {
/** Gets the name of a known built-in. */
string getBuiltinName() {
// These lists were created by inspecting the `builtins` and `__builtin__` modules in
// Python 3 and 2 respectively, using the `dir` built-in.
// Built-in functions and exceptions shared between Python 2 and 3
result in [
"abs", "all", "any", "bin", "bool", "bytearray", "callable", "chr", "classmethod",
"compile", "complex", "delattr", "dict", "dir", "divmod", "enumerate", "eval", "filter",
"float", "format", "frozenset", "getattr", "globals", "hasattr", "hash", "help", "hex",
"id", "input", "int", "isinstance", "issubclass", "iter", "len", "list", "locals", "map",
"max", "memoryview", "min", "next", "object", "oct", "open", "ord", "pow", "print",
"property", "range", "repr", "reversed", "round", "set", "setattr", "slice", "sorted",
"staticmethod", "str", "sum", "super", "tuple", "type", "vars", "zip", "__import__",
// Exceptions
"ArithmeticError", "AssertionError", "AttributeError", "BaseException", "BufferError",
"BytesWarning", "DeprecationWarning", "EOFError", "EnvironmentError", "Exception",
"FloatingPointError", "FutureWarning", "GeneratorExit", "IOError", "ImportError",
"ImportWarning", "IndentationError", "IndexError", "KeyError", "KeyboardInterrupt",
"LookupError", "MemoryError", "NameError", "NotImplemented", "NotImplementedError",
"OSError", "OverflowError", "PendingDeprecationWarning", "ReferenceError", "RuntimeError",
"RuntimeWarning", "StandardError", "StopIteration", "SyntaxError", "SyntaxWarning",
"SystemError", "SystemExit", "TabError", "TypeError", "UnboundLocalError",
"UnicodeDecodeError", "UnicodeEncodeError", "UnicodeError", "UnicodeTranslateError",
"UnicodeWarning", "UserWarning", "ValueError", "Warning", "ZeroDivisionError",
// Added for compatibility
"exec"
]
or
// Built-in constants shared between Python 2 and 3
result in ["False", "True", "None", "NotImplemented", "Ellipsis", "__debug__"]
or
// Python 3 only
result in [
"ascii", "breakpoint", "bytes", "exec",
// Exceptions
"BlockingIOError", "BrokenPipeError", "ChildProcessError", "ConnectionAbortedError",
"ConnectionError", "ConnectionRefusedError", "ConnectionResetError", "FileExistsError",
"FileNotFoundError", "InterruptedError", "IsADirectoryError", "ModuleNotFoundError",
"NotADirectoryError", "PermissionError", "ProcessLookupError", "RecursionError",
"ResourceWarning", "StopAsyncIteration", "TimeoutError"
]
or
// Python 2 only
result in [
"basestring", "cmp", "execfile", "file", "long", "raw_input", "reduce", "reload", "unichr",
"unicode", "xrange"
]
}
/**
* Gets a data flow node that is likely to refer to a built-in with the name `name`.
*
* Currently this is an over-approximation, and may not account for things like overwriting a
* built-in with a different value.
*/
DataFlow::Node likelyBuiltin(string name) {
exists(Module m |
result.asCfgNode() =
any(NameNode n |
possible_builtin_accessed_in_module(n, name, m) and
not possible_builtin_defined_in_module(name, m)
)
)
}
/**
* Holds if a global variable called `name` (which is also the name of a built-in) is assigned
* a value in the module `m`.
*/
private predicate possible_builtin_defined_in_module(string name, Module m) {
ImportStar::globalNameDefinedInModule(name, m) and
name = getBuiltinName()
}
/**
* Holds if `n` is an access of a global variable called `name` (which is also the name of a
* built-in) inside the module `m`.
*/
private predicate possible_builtin_accessed_in_module(NameNode n, string name, Module m) {
n.isGlobal() and
n.isLoad() and
name = n.getId() and
name = getBuiltinName() and
m = n.getEnclosingModule()
}
}

558
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatchPointsTo.qll Normal file
View File

@@ -0,0 +1,558 @@
/**
* INTERNAL: Do not use.
*
* Points-to based call-graph.
*/
private import python
private import DataFlowPublic
private import semmle.python.SpecialMethods
/** A parameter position represented by an integer. */
class ParameterPosition extends int {
ParameterPosition() { exists(any(DataFlowCallable c).getParameter(this)) }
}
/** An argument position represented by an integer. */
class ArgumentPosition extends int {
ArgumentPosition() { exists(any(DataFlowCall c).getArg(this)) }
}
/** Holds if arguments at position `apos` match parameters at position `ppos`. */
pragma[inline]
predicate parameterMatch(ParameterPosition ppos, ArgumentPosition apos) { ppos = apos }
/**
* Computes routing of arguments to parameters
*
* When a call contains more positional arguments than there are positional parameters,
* the extra positional arguments are passed as a tuple to a starred parameter. This is
* achieved by synthesizing a node `TPosOverflowNode(call, callable)`
* that represents the tuple of extra positional arguments. There is a store step from each
* extra positional argument to this node.
*
* CURRENTLY NOT SUPPORTED:
* When a call contains an iterable unpacking argument, such as `func(*args)`, it is expanded into positional arguments.
*
* CURRENTLY NOT SUPPORTED:
* If a call contains an iterable unpacking argument, such as `func(*args)`, and the callee contains a starred argument, any extra
* positional arguments are passed to the starred argument.
*
* When a call contains keyword arguments that do not correspond to keyword parameters, these
* extra keyword arguments are passed as a dictionary to a doubly starred parameter. This is
* achieved by synthesizing a node `TKwOverflowNode(call, callable)`
* that represents the dictionary of extra keyword arguments. There is a store step from each
* extra keyword argument to this node.
*
* When a call contains a dictionary unpacking argument, such as `func(**kwargs)`, with entries corresponding to a keyword parameter,
* the value at such a key is unpacked and passed to the parameter. This is achieved
* by synthesizing an argument node `TKwUnpacked(call, callable, name)` representing the unpacked
* value. This node is used as the argument passed to the matching keyword parameter. There is a read
* step from the dictionary argument to the synthesized argument node.
*
* When a call contains a dictionary unpacking argument, such as `func(**kwargs)`, and the callee contains a doubly starred parameter,
* entries which are not unpacked are passed to the doubly starred parameter. This is achieved by
* adding a dataflow step from the dictionary argument to `TKwOverflowNode(call, callable)` and a
* step to clear content of that node at any unpacked keys.
*
* ## Examples:
* Assume that we have the callable
* ```python
* def f(x, y, *t, **d):
* pass
* ```
* Then the call
* ```python
* f(0, 1, 2, a=3)
* ```
* will be modeled as
* ```python
* f(0, 1, [*t], [**d])
* ```
* where `[` and `]` denotes synthesized nodes, so `[*t]` is the synthesized tuple argument
* `TPosOverflowNode` and `[**d]` is the synthesized dictionary argument `TKwOverflowNode`.
* There will be a store step from `2` to `[*t]` at pos `0` and one from `3` to `[**d]` at key
* `a`.
*
* For the call
* ```python
* f(0, **{"y": 1, "a": 3})
* ```
* no tuple argument is synthesized. It is modeled as
* ```python
* f(0, [y=1], [**d])
* ```
* where `[y=1]` is the synthesized unpacked argument `TKwUnpacked` (with `name` = `y`). There is
* a read step from `**{"y": 1, "a": 3}` to `[y=1]` at key `y` to get the value passed to the parameter
* `y`. There is a dataflow step from `**{"y": 1, "a": 3}` to `[**d]` to transfer the content and
* a clearing of content at key `y` for node `[**d]`, since that value has been unpacked.
*/
module ArgumentPassing {
/**
* Holds if `call` represents a `DataFlowCall` to a `DataFlowCallable` represented by `callable`.
*
* It _may not_ be the case that `call = callable.getACall()`, i.e. if `call` represents a `ClassCall`.
*
* Used to limit the size of predicates.
*/
predicate connects(CallNode call, CallableValue callable) {
exists(DataFlowCall c |
call = c.getNode() and
callable = c.getCallable().getCallableValue()
)
}
/**
* Gets the `n`th parameter of `callable`.
* If the callable has a starred parameter, say `*tuple`, that is matched with `n=-1`.
* If the callable has a doubly starred parameter, say `**dict`, that is matched with `n=-2`.
* Note that, unlike other languages, we do _not_ use -1 for the position of `self` in Python,
* as it is an explicit parameter at position 0.
*/
NameNode getParameter(CallableValue callable, int n) {
// positional parameter
result = callable.getParameter(n)
or
// starred parameter, `*tuple`
exists(Function f |
f = callable.getScope() and
n = -1 and
result = f.getVararg().getAFlowNode()
)
or
// doubly starred parameter, `**dict`
exists(Function f |
f = callable.getScope() and
n = -2 and
result = f.getKwarg().getAFlowNode()
)
}
/**
* A type representing a mapping from argument indices to parameter indices.
* We currently use two mappings: NoShift, the identity, used for ordinary
* function calls, and ShiftOneUp which is used for calls where an extra argument
* is inserted. These include method calls, constructor calls and class calls.
* In these calls, the argument at index `n` is mapped to the parameter at position `n+1`.
*/
newtype TArgParamMapping =
TNoShift() or
TShiftOneUp()
/** A mapping used for parameter passing. */
abstract class ArgParamMapping extends TArgParamMapping {
/** Gets the index of the parameter that corresponds to the argument at index `argN`. */
bindingset[argN]
abstract int getParamN(int argN);
/** Gets a textual representation of this element. */
abstract string toString();
}
/** A mapping that passes argument `n` to parameter `n`. */
class NoShift extends ArgParamMapping, TNoShift {
NoShift() { this = TNoShift() }
override string toString() { result = "NoShift [n -> n]" }
bindingset[argN]
override int getParamN(int argN) { result = argN }
}
/** A mapping that passes argument `n` to parameter `n+1`. */
class ShiftOneUp extends ArgParamMapping, TShiftOneUp {
ShiftOneUp() { this = TShiftOneUp() }
override string toString() { result = "ShiftOneUp [n -> n+1]" }
bindingset[argN]
override int getParamN(int argN) { result = argN + 1 }
}
/**
* Gets the node representing the argument to `call` that is passed to the parameter at
* (zero-based) index `paramN` in `callable`. If this is a positional argument, it must appear
* at an index, `argN`, in `call` wich satisfies `paramN = mapping.getParamN(argN)`.
*
* `mapping` will be the identity for function calls, but not for method- or constructor calls,
* where the first parameter is `self` and the first positional argument is passed to the second positional parameter.
* Similarly for classmethod calls, where the first parameter is `cls`.
*
* NOT SUPPORTED: Keyword-only parameters.
*/
Node getArg(CallNode call, ArgParamMapping mapping, CallableValue callable, int paramN) {
connects(call, callable) and
(
// positional argument
exists(int argN |
paramN = mapping.getParamN(argN) and
result = TCfgNode(call.getArg(argN))
)
or
// keyword argument
// TODO: Since `getArgName` have no results for keyword-only parameters,
// these are currently not supported.
exists(Function f, string argName |
f = callable.getScope() and
f.getArgName(paramN) = argName and
result = TCfgNode(call.getArgByName(unbind_string(argName)))
)
or
// a synthezised argument passed to the starred parameter (at position -1)
callable.getScope().hasVarArg() and
paramN = -1 and
result = TPosOverflowNode(call, callable)
or
// a synthezised argument passed to the doubly starred parameter (at position -2)
callable.getScope().hasKwArg() and
paramN = -2 and
result = TKwOverflowNode(call, callable)
or
// argument unpacked from dict
exists(string name |
call_unpacks(call, mapping, callable, name, paramN) and
result = TKwUnpackedNode(call, callable, name)
)
)
}
/** Currently required in `getArg` in order to prevent a bad join. */
bindingset[result, s]
private string unbind_string(string s) { result <= s and s <= result }
/** Gets the control flow node that is passed as the `n`th overflow positional argument. */
ControlFlowNode getPositionalOverflowArg(CallNode call, CallableValue callable, int n) {
connects(call, callable) and
exists(Function f, int posCount, int argNr |
f = callable.getScope() and
f.hasVarArg() and
posCount = f.getPositionalParameterCount() and
result = call.getArg(argNr) and
argNr >= posCount and
argNr = posCount + n
)
}
/** Gets the control flow node that is passed as the overflow keyword argument with key `key`. */
ControlFlowNode getKeywordOverflowArg(CallNode call, CallableValue callable, string key) {
connects(call, callable) and
exists(Function f |
f = callable.getScope() and
f.hasKwArg() and
not exists(f.getArgByName(key)) and
result = call.getArgByName(key)
)
}
/**
* Holds if `call` unpacks a dictionary argument in order to pass it via `name`.
* It will then be passed to the parameter of `callable` at index `paramN`.
*/
predicate call_unpacks(
CallNode call, ArgParamMapping mapping, CallableValue callable, string name, int paramN
) {
connects(call, callable) and
exists(Function f |
f = callable.getScope() and
not exists(int argN | paramN = mapping.getParamN(argN) | exists(call.getArg(argN))) and // no positional argument available
name = f.getArgName(paramN) and
// not exists(call.getArgByName(name)) and // only matches keyword arguments not preceded by **
// TODO: make the below logic respect control flow splitting (by not going to the AST).
not call.getNode().getANamedArg().(Keyword).getArg() = name and // no keyword argument available
paramN >= 0 and
paramN < f.getPositionalParameterCount() + f.getKeywordOnlyParameterCount() and
exists(call.getNode().getKwargs()) // dict argument available
)
}
}
import ArgumentPassing
/**
* IPA type for DataFlowCallable.
*
* A callable is either a function value, a class value, or a module (for enclosing `ModuleVariableNode`s).
* A module has no calls.
*/
newtype TDataFlowCallable =
TCallableValue(CallableValue callable) {
callable instanceof FunctionValue and
not callable.(FunctionValue).isLambda()
or
callable instanceof ClassValue
} or
TLambda(Function lambda) { lambda.isLambda() } or
TModule(Module m)
/** A callable. */
abstract class DataFlowCallable extends TDataFlowCallable {
/** Gets a textual representation of this element. */
abstract string toString();
/** Gets a call to this callable. */
abstract CallNode getACall();
/** Gets the scope of this callable */
abstract Scope getScope();
/** Gets the specified parameter of this callable */
abstract NameNode getParameter(int n);
/** Gets the name of this callable. */
abstract string getName();
/** Gets a callable value for this callable, if one exists. */
abstract CallableValue getCallableValue();
}
/** A class representing a callable value. */
class DataFlowCallableValue extends DataFlowCallable, TCallableValue {
CallableValue callable;
DataFlowCallableValue() { this = TCallableValue(callable) }
override string toString() { result = callable.toString() }
override CallNode getACall() { result = callable.getACall() }
override Scope getScope() { result = callable.getScope() }
override NameNode getParameter(int n) { result = getParameter(callable, n) }
override string getName() { result = callable.getName() }
override CallableValue getCallableValue() { result = callable }
}
/** A class representing a callable lambda. */
class DataFlowLambda extends DataFlowCallable, TLambda {
Function lambda;
DataFlowLambda() { this = TLambda(lambda) }
override string toString() { result = lambda.toString() }
override CallNode getACall() { result = this.getCallableValue().getACall() }
override Scope getScope() { result = lambda.getEvaluatingScope() }
override NameNode getParameter(int n) { result = getParameter(this.getCallableValue(), n) }
override string getName() { result = "Lambda callable" }
override FunctionValue getCallableValue() {
result.getOrigin().getNode() = lambda.getDefinition()
}
}
/** A class representing the scope in which a `ModuleVariableNode` appears. */
class DataFlowModuleScope extends DataFlowCallable, TModule {
Module mod;
DataFlowModuleScope() { this = TModule(mod) }
override string toString() { result = mod.toString() }
override CallNode getACall() { none() }
override Scope getScope() { result = mod }
override NameNode getParameter(int n) { none() }
override string getName() { result = mod.getName() }
override CallableValue getCallableValue() { none() }
}
/**
* IPA type for DataFlowCall.
*
* Calls corresponding to `CallNode`s are either to callable values or to classes.
* The latter is directed to the callable corresponding to the `__init__` method of the class.
*
* An `__init__` method can also be called directly, so that the callable can be targeted by
* different types of calls. In that case, the parameter mappings will be different,
* as the class call will synthesize an argument node to be mapped to the `self` parameter.
*
* A call corresponding to a special method call is handled by the corresponding `SpecialMethodCallNode`.
*
* TODO: Add `TClassMethodCall` mapping `cls` appropriately.
*/
newtype TDataFlowCall =
TFunctionCall(CallNode call) { call = any(FunctionValue f).getAFunctionCall() } or
/** Bound methods need to make room for the explicit self parameter */
TMethodCall(CallNode call) { call = any(FunctionValue f).getAMethodCall() } or
TClassCall(CallNode call) { call = any(ClassValue c | not c.isAbsent()).getACall() } or
TSpecialCall(SpecialMethodCallNode special)
/** A call. */
abstract class DataFlowCall extends TDataFlowCall {
/** Gets a textual representation of this element. */
abstract string toString();
/** Get the callable to which this call goes. */
abstract DataFlowCallable getCallable();
/**
* Gets the argument to this call that will be sent
* to the `n`th parameter of the callable.
*/
abstract Node getArg(int n);
/** Get the control flow node representing this call. */
abstract ControlFlowNode getNode();
/** Gets the enclosing callable of this call. */
abstract DataFlowCallable getEnclosingCallable();
/** Gets the location of this dataflow call. */
Location getLocation() { result = this.getNode().getLocation() }
}
/**
* A call to a function/lambda.
* This excludes calls to bound methods, classes, and special methods.
* Bound method calls and class calls insert an argument for the explicit
* `self` parameter, and special method calls have special argument passing.
*/
class FunctionCall extends DataFlowCall, TFunctionCall {
CallNode call;
DataFlowCallable callable;
FunctionCall() {
this = TFunctionCall(call) and
call = callable.getACall()
}
override string toString() { result = call.toString() }
override Node getArg(int n) { result = getArg(call, TNoShift(), callable.getCallableValue(), n) }
override ControlFlowNode getNode() { result = call }
override DataFlowCallable getCallable() { result = callable }
override DataFlowCallable getEnclosingCallable() { result.getScope() = call.getNode().getScope() }
}
/**
* Represents a call to a bound method call.
* The node representing the instance is inserted as argument to the `self` parameter.
*/
class MethodCall extends DataFlowCall, TMethodCall {
CallNode call;
FunctionValue bm;
MethodCall() {
this = TMethodCall(call) and
call = bm.getACall()
}
private CallableValue getCallableValue() { result = bm }
override string toString() { result = call.toString() }
override Node getArg(int n) {
n > 0 and result = getArg(call, TShiftOneUp(), this.getCallableValue(), n)
or
n = 0 and result = TCfgNode(call.getFunction().(AttrNode).getObject())
}
override ControlFlowNode getNode() { result = call }
override DataFlowCallable getCallable() { result = TCallableValue(this.getCallableValue()) }
override DataFlowCallable getEnclosingCallable() { result.getScope() = call.getScope() }
}
/**
* Represents a call to a class.
* The pre-update node for the call is inserted as argument to the `self` parameter.
* That makes the call node be the post-update node holding the value of the object
* after the constructor has run.
*/
class ClassCall extends DataFlowCall, TClassCall {
CallNode call;
ClassValue c;
ClassCall() {
this = TClassCall(call) and
call = c.getACall()
}
private CallableValue getCallableValue() { c.getScope().getInitMethod() = result.getScope() }
override string toString() { result = call.toString() }
override Node getArg(int n) {
n > 0 and result = getArg(call, TShiftOneUp(), this.getCallableValue(), n)
or
n = 0 and result = TSyntheticPreUpdateNode(TCfgNode(call))
}
override ControlFlowNode getNode() { result = call }
override DataFlowCallable getCallable() { result = TCallableValue(this.getCallableValue()) }
override DataFlowCallable getEnclosingCallable() { result.getScope() = call.getScope() }
}
/** A call to a special method. */
class SpecialCall extends DataFlowCall, TSpecialCall {
SpecialMethodCallNode special;
SpecialCall() { this = TSpecialCall(special) }
override string toString() { result = special.toString() }
override Node getArg(int n) { result = TCfgNode(special.(SpecialMethod::Potential).getArg(n)) }
override ControlFlowNode getNode() { result = special }
override DataFlowCallable getCallable() {
result = TCallableValue(special.getResolvedSpecialMethod())
}
override DataFlowCallable getEnclosingCallable() {
result.getScope() = special.getNode().getScope()
}
}
/** Gets a viable run-time target for the call `call`. */
DataFlowCallable viableCallable(DataFlowCall call) { result = call.getCallable() }
private newtype TReturnKind = TNormalReturnKind()
/**
* A return kind. A return kind describes how a value can be returned
* from a callable. For Python, this is simply a method return.
*/
class ReturnKind extends TReturnKind {
/** Gets a textual representation of this element. */
string toString() { result = "return" }
}
/** A data flow node that represents a value returned by a callable. */
class ReturnNode extends CfgNode {
Return ret;
// See `TaintTrackingImplementation::returnFlowStep`
ReturnNode() { node = ret.getValue().getAFlowNode() }
/** Gets the kind of this return node. */
ReturnKind getKind() { any() }
}
/** A data flow node that represents the output of a call. */
class OutNode extends CfgNode {
OutNode() { node instanceof CallNode }
}
/**
* Gets a node that can read the value returned from `call` with return kind
* `kind`.
*/
OutNode getAnOutNode(DataFlowCall call, ReturnKind kind) {
call.getNode() = result.getNode() and
kind = TNormalReturnKind()
}

2325
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl.qll
View File

File diff suppressed because it is too large Load Diff

2325
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl2.qll
View File

File diff suppressed because it is too large Load Diff

2325
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl3.qll
View File

File diff suppressed because it is too large Load Diff

2325
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl4.qll
View File

File diff suppressed because it is too large Load Diff

183
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll
View File

@@ -2,6 +2,53 @@ private import DataFlowImplSpecific::Private
private import DataFlowImplSpecific::Public
import Cached
module DataFlowImplCommonPublic {
/** A state value to track during data flow. */
class FlowState = string;
/**
* The default state, which is used when the state is unspecified for a source
* or a sink.
*/
class FlowStateEmpty extends FlowState {
FlowStateEmpty() { this = "" }
}
private newtype TFlowFeature =
TFeatureHasSourceCallContext() or
TFeatureHasSinkCallContext() or
TFeatureEqualSourceSinkCallContext()
/** A flow configuration feature for use in `Configuration::getAFeature()`. */
class FlowFeature extends TFlowFeature {
string toString() { none() }
}
/**
* A flow configuration feature that implies that sources have some existing
* call context.
*/
class FeatureHasSourceCallContext extends FlowFeature, TFeatureHasSourceCallContext {
override string toString() { result = "FeatureHasSourceCallContext" }
}
/**
* A flow configuration feature that implies that sinks have some existing
* call context.
*/
class FeatureHasSinkCallContext extends FlowFeature, TFeatureHasSinkCallContext {
override string toString() { result = "FeatureHasSinkCallContext" }
}
/**
* A flow configuration feature that implies that source-sink pairs have some
* shared existing call context.
*/
class FeatureEqualSourceSinkCallContext extends FlowFeature, TFeatureEqualSourceSinkCallContext {
override string toString() { result = "FeatureEqualSourceSinkCallContext" }
}
}
/**
* The cost limits for the `AccessPathFront` to `AccessPathApprox` expansion.
*
@@ -26,6 +73,18 @@ predicate accessPathCostLimits(int apLimit, int tupleLimit) {
tupleLimit = 1000
}
/**
* Holds if `arg` is an argument of `call` with an argument position that matches
* parameter position `ppos`.
*/
pragma[noinline]
predicate argumentPositionMatch(DataFlowCall call, ArgNode arg, ParameterPosition ppos) {
exists(ArgumentPosition apos |
arg.argumentOf(call, apos) and
parameterMatch(ppos, apos)
)
}
/**
* Provides a simple data-flow analysis for resolving lambda calls. The analysis
* currently excludes read-steps, store-steps, and flow-through.
@@ -35,25 +94,27 @@ predicate accessPathCostLimits(int apLimit, int tupleLimit) {
* calls. For this reason, we cannot reuse the code from `DataFlowImpl.qll` directly.
*/
private module LambdaFlow {
private predicate viableParamNonLambda(DataFlowCall call, int i, ParamNode p) {
p.isParameterOf(viableCallable(call), i)
pragma[noinline]
private predicate viableParamNonLambda(DataFlowCall call, ParameterPosition ppos, ParamNode p) {
p.isParameterOf(viableCallable(call), ppos)
}
private predicate viableParamLambda(DataFlowCall call, int i, ParamNode p) {
p.isParameterOf(viableCallableLambda(call, _), i)
pragma[noinline]
private predicate viableParamLambda(DataFlowCall call, ParameterPosition ppos, ParamNode p) {
p.isParameterOf(viableCallableLambda(call, _), ppos)
}
private predicate viableParamArgNonLambda(DataFlowCall call, ParamNode p, ArgNode arg) {
exists(int i |
viableParamNonLambda(call, i, p) and
arg.argumentOf(call, i)
exists(ParameterPosition ppos |
viableParamNonLambda(call, ppos, p) and
argumentPositionMatch(call, arg, ppos)
)
}
private predicate viableParamArgLambda(DataFlowCall call, ParamNode p, ArgNode arg) {
exists(int i |
viableParamLambda(call, i, p) and
arg.argumentOf(call, i)
exists(ParameterPosition ppos |
viableParamLambda(call, ppos, p) and
argumentPositionMatch(call, arg, ppos)
)
}
@@ -251,7 +312,7 @@ private module Cached {
predicate forceCachingInSameStage() { any() }
cached
predicate nodeEnclosingCallable(Node n, DataFlowCallable c) { c = n.getEnclosingCallable() }
predicate nodeEnclosingCallable(Node n, DataFlowCallable c) { c = nodeGetEnclosingCallable(n) }
cached
predicate callEnclosingCallable(DataFlowCall call, DataFlowCallable c) {
@@ -265,7 +326,10 @@ private module Cached {
predicate jumpStepCached(Node node1, Node node2) { jumpStep(node1, node2) }
cached
predicate clearsContentCached(Node n, Content c) { clearsContent(n, c) }
predicate clearsContentCached(Node n, ContentSet c) { clearsContent(n, c) }
cached
predicate expectsContentCached(Node n, ContentSet c) { expectsContent(n, c) }
cached
predicate isUnreachableInCallCached(Node n, DataFlowCall call) { isUnreachableInCall(n, call) }
@@ -286,7 +350,7 @@ private module Cached {
or
exists(ArgNode arg |
result.(PostUpdateNode).getPreUpdateNode() = arg and
arg.argumentOf(call, k.(ParamUpdateReturnKind).getPosition())
arg.argumentOf(call, k.(ParamUpdateReturnKind).getAMatchingArgumentPosition())
)
}
@@ -294,7 +358,7 @@ private module Cached {
predicate returnNodeExt(Node n, ReturnKindExt k) {
k = TValueReturn(n.(ReturnNode).getKind())
or
exists(ParamNode p, int pos |
exists(ParamNode p, ParameterPosition pos |
parameterValueFlowsToPreUpdate(p, n) and
p.isParameterOf(_, pos) and
k = TParamUpdate(pos)
@@ -312,17 +376,17 @@ private module Cached {
// For reads, `x.f`, we want to check that the tracked type after the read (which
// is obtained by popping the head of the access path stack) is compatible with
// the type of `x.f`.
read(_, _, n)
readSet(_, _, n)
}
cached
predicate parameterNode(Node n, DataFlowCallable c, int i) {
n.(ParameterNode).isParameterOf(c, i)
predicate parameterNode(Node p, DataFlowCallable c, ParameterPosition pos) {
isParameterNode(p, c, pos)
}
cached
predicate argumentNode(Node n, DataFlowCall call, int pos) {
n.(ArgumentNode).argumentOf(call, pos)
predicate argumentNode(Node n, DataFlowCall call, ArgumentPosition pos) {
isArgumentNode(n, call, pos)
}
/**
@@ -340,12 +404,12 @@ private module Cached {
}
/**
* Holds if `p` is the `i`th parameter of a viable dispatch target of `call`.
* The instance parameter is considered to have index `-1`.
* Holds if `p` is the parameter of a viable dispatch target of `call`,
* and `p` has position `ppos`.
*/
pragma[nomagic]
private predicate viableParam(DataFlowCall call, int i, ParamNode p) {
p.isParameterOf(viableCallableExt(call), i)
private predicate viableParam(DataFlowCall call, ParameterPosition ppos, ParamNode p) {
p.isParameterOf(viableCallableExt(call), ppos)
}
/**
@@ -354,9 +418,9 @@ private module Cached {
*/
cached
predicate viableParamArg(DataFlowCall call, ParamNode p, ArgNode arg) {
exists(int i |
viableParam(call, i, p) and
arg.argumentOf(call, i) and
exists(ParameterPosition ppos |
viableParam(call, ppos, p) and
argumentPositionMatch(call, arg, ppos) and
compatibleTypes(getNodeDataFlowType(arg), getNodeDataFlowType(p))
)
}
@@ -408,7 +472,7 @@ private module Cached {
// read
exists(Node mid |
parameterValueFlowCand(p, mid, false) and
read(mid, _, node) and
readSet(mid, _, node) and
read = true
)
or
@@ -596,8 +660,10 @@ private module Cached {
* Holds if `arg` flows to `out` through a call using only
* value-preserving steps and a single read step, not taking call
* contexts into account, thus representing a getter-step.
*
* This predicate is exposed for testing only.
*/
predicate getterStep(ArgNode arg, Content c, Node out) {
predicate getterStep(ArgNode arg, ContentSet c, Node out) {
argumentValueFlowsThrough(arg, TReadStepTypesSome(_, c, _), out)
}
@@ -720,28 +786,30 @@ private module Cached {
parameterValueFlow(p, n.getPreUpdateNode(), TReadStepTypesNone())
}
cached
predicate readSet(Node node1, ContentSet c, Node node2) { readStep(node1, c, node2) }
private predicate store(
Node node1, Content c, Node node2, DataFlowType contentType, DataFlowType containerType
) {
storeStep(node1, c, node2) and
contentType = getNodeDataFlowType(node1) and
containerType = getNodeDataFlowType(node2)
or
exists(Node n1, Node n2 |
n1 = node1.(PostUpdateNode).getPreUpdateNode() and
n2 = node2.(PostUpdateNode).getPreUpdateNode()
|
argumentValueFlowsThrough(n2, TReadStepTypesSome(containerType, c, contentType), n1)
exists(ContentSet cs | c = cs.getAStoreContent() |
storeStep(node1, cs, node2) and
contentType = getNodeDataFlowType(node1) and
containerType = getNodeDataFlowType(node2)
or
read(n2, c, n1) and
contentType = getNodeDataFlowType(n1) and
containerType = getNodeDataFlowType(n2)
exists(Node n1, Node n2 |
n1 = node1.(PostUpdateNode).getPreUpdateNode() and
n2 = node2.(PostUpdateNode).getPreUpdateNode()
|
argumentValueFlowsThrough(n2, TReadStepTypesSome(containerType, cs, contentType), n1)
or
readSet(n2, cs, n1) and
contentType = getNodeDataFlowType(n1) and
containerType = getNodeDataFlowType(n2)
)
)
}
cached
predicate read(Node node1, Content c, Node node2) { readStep(node1, c, node2) }
/**
* Holds if data can flow from `node1` to `node2` via a direct assignment to
* `f`.
@@ -828,7 +896,7 @@ private module Cached {
cached
newtype TReturnKindExt =
TValueReturn(ReturnKind kind) or
TParamUpdate(int pos) { exists(ParamNode p | p.isParameterOf(_, pos)) }
TParamUpdate(ParameterPosition pos) { exists(ParamNode p | p.isParameterOf(_, pos)) }
cached
newtype TBooleanOption =
@@ -871,16 +939,16 @@ class CastingNode extends Node {
}
private predicate readStepWithTypes(
Node n1, DataFlowType container, Content c, Node n2, DataFlowType content
Node n1, DataFlowType container, ContentSet c, Node n2, DataFlowType content
) {
read(n1, c, n2) and
readSet(n1, c, n2) and
container = getNodeDataFlowType(n1) and
content = getNodeDataFlowType(n2)
}
private newtype TReadStepTypesOption =
TReadStepTypesNone() or
TReadStepTypesSome(DataFlowType container, Content c, DataFlowType content) {
TReadStepTypesSome(DataFlowType container, ContentSet c, DataFlowType content) {
readStepWithTypes(_, container, c, _, content)
}
@@ -889,7 +957,7 @@ private class ReadStepTypesOption extends TReadStepTypesOption {
DataFlowType getContainerType() { this = TReadStepTypesSome(result, _, _) }
Content getContent() { this = TReadStepTypesSome(_, result, _) }
ContentSet getContent() { this = TReadStepTypesSome(_, result, _) }
DataFlowType getContentType() { this = TReadStepTypesSome(_, _, result) }
@@ -1020,9 +1088,9 @@ class ParamNode extends Node {
/**
* Holds if this node is the parameter of callable `c` at the specified
* (zero-based) position.
* position.
*/
predicate isParameterOf(DataFlowCallable c, int i) { parameterNode(this, c, i) }
predicate isParameterOf(DataFlowCallable c, ParameterPosition pos) { parameterNode(this, c, pos) }
}
/** A data-flow node that represents a call argument. */
@@ -1030,7 +1098,9 @@ class ArgNode extends Node {
ArgNode() { argumentNode(this, _, _) }
/** Holds if this argument occurs at the given position in the given call. */
final predicate argumentOf(DataFlowCall call, int pos) { argumentNode(this, call, pos) }
final predicate argumentOf(DataFlowCall call, ArgumentPosition pos) {
argumentNode(this, call, pos)
}
}
/**
@@ -1076,11 +1146,14 @@ class ValueReturnKind extends ReturnKindExt, TValueReturn {
}
class ParamUpdateReturnKind extends ReturnKindExt, TParamUpdate {
private int pos;
private ParameterPosition pos;
ParamUpdateReturnKind() { this = TParamUpdate(pos) }
int getPosition() { result = pos }
ParameterPosition getPosition() { result = pos }
pragma[nomagic]
ArgumentPosition getAMatchingArgumentPosition() { parameterMatch(pos, result) }
override string toString() { result = "param update " + pos }
}
@@ -1224,7 +1297,7 @@ class DataFlowCallOption extends TDataFlowCallOption {
}
}
/** Content tagged with the type of a containing object. */
/** A `Content` tagged with the type of a containing object. */
class TypedContent extends MkTypedContent {
private Content c;
private DataFlowType t;
@@ -1259,8 +1332,6 @@ abstract class AccessPathFront extends TAccessPathFront {
abstract boolean toBoolNonEmpty();
TypedContent getHead() { this = TFrontHead(result) }
predicate isClearedAt(Node n) { clearsContentCached(n, this.getHead().getContent()) }
}
class AccessPathFrontNil extends AccessPathFront, TFrontNil {

46
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplConsistency.qll
View File

@@ -9,6 +9,31 @@ private import tainttracking1.TaintTrackingParameter::Private
private import tainttracking1.TaintTrackingParameter::Public
module Consistency {
private newtype TConsistencyConfiguration = MkConsistencyConfiguration()
/** A class for configuring the consistency queries. */
class ConsistencyConfiguration extends TConsistencyConfiguration {
string toString() { none() }
/** Holds if `n` should be excluded from the consistency test `uniqueEnclosingCallable`. */
predicate uniqueEnclosingCallableExclude(Node n) { none() }
/** Holds if `n` should be excluded from the consistency test `uniqueNodeLocation`. */
predicate uniqueNodeLocationExclude(Node n) { none() }
/** Holds if `n` should be excluded from the consistency test `missingLocation`. */
predicate missingLocationExclude(Node n) { none() }
/** Holds if `n` should be excluded from the consistency test `postWithInFlow`. */
predicate postWithInFlowExclude(Node n) { none() }
/** Holds if `n` should be excluded from the consistency test `argHasPostUpdate`. */
predicate argHasPostUpdateExclude(ArgumentNode n) { none() }
/** Holds if `n` should be excluded from the consistency test `reverseRead`. */
predicate reverseReadExclude(Node n) { none() }
}
private class RelevantNode extends Node {
RelevantNode() {
this instanceof ArgumentNode or
@@ -31,8 +56,9 @@ module Consistency {
query predicate uniqueEnclosingCallable(Node n, string msg) {
exists(int c |
n instanceof RelevantNode and
c = count(n.getEnclosingCallable()) and
c = count(nodeGetEnclosingCallable(n)) and
c != 1 and
not any(ConsistencyConfiguration conf).uniqueEnclosingCallableExclude(n) and
msg = "Node should have one enclosing callable but has " + c + "."
)
}
@@ -53,6 +79,7 @@ module Consistency {
n.hasLocationInfo(filepath, startline, startcolumn, endline, endcolumn)
) and
c != 1 and
not any(ConsistencyConfiguration conf).uniqueNodeLocationExclude(n) and
msg = "Node should have one location but has " + c + "."
)
}
@@ -63,7 +90,8 @@ module Consistency {
strictcount(Node n |
not exists(string filepath, int startline, int startcolumn, int endline, int endcolumn |
n.hasLocationInfo(filepath, startline, startcolumn, endline, endcolumn)
)
) and
not any(ConsistencyConfiguration conf).missingLocationExclude(n)
) and
msg = "Nodes without location: " + c
)
@@ -85,13 +113,13 @@ module Consistency {
}
query predicate parameterCallable(ParameterNode p, string msg) {
exists(DataFlowCallable c | p.isParameterOf(c, _) and c != p.getEnclosingCallable()) and
exists(DataFlowCallable c | isParameterNode(p, c, _) and c != nodeGetEnclosingCallable(p)) and
msg = "Callable mismatch for parameter."
}
query predicate localFlowIsLocal(Node n1, Node n2, string msg) {
simpleLocalFlowStep(n1, n2) and
n1.getEnclosingCallable() != n2.getEnclosingCallable() and
nodeGetEnclosingCallable(n1) != nodeGetEnclosingCallable(n2) and
msg = "Local flow step does not preserve enclosing callable."
}
@@ -106,7 +134,7 @@ module Consistency {
query predicate unreachableNodeCCtx(Node n, DataFlowCall call, string msg) {
isUnreachableInCall(n, call) and
exists(DataFlowCallable c |
c = n.getEnclosingCallable() and
c = nodeGetEnclosingCallable(n) and
not viableCallable(call) = c
) and
msg = "Call context for isUnreachableInCall is inconsistent with call graph."
@@ -120,7 +148,7 @@ module Consistency {
n.(ArgumentNode).argumentOf(call, _) and
msg = "ArgumentNode and call does not share enclosing callable."
) and
n.getEnclosingCallable() != call.getEnclosingCallable()
nodeGetEnclosingCallable(n) != call.getEnclosingCallable()
}
// This predicate helps the compiler forget that in some languages
@@ -151,7 +179,7 @@ module Consistency {
}
query predicate postIsInSameCallable(PostUpdateNode n, string msg) {
n.getEnclosingCallable() != n.getPreUpdateNode().getEnclosingCallable() and
nodeGetEnclosingCallable(n) != nodeGetEnclosingCallable(n.getPreUpdateNode()) and
msg = "PostUpdateNode does not share callable with its pre-update node."
}
@@ -159,12 +187,13 @@ module Consistency {
query predicate reverseRead(Node n, string msg) {
exists(Node n2 | readStep(n, _, n2) and hasPost(n2) and not hasPost(n)) and
not any(ConsistencyConfiguration conf).reverseReadExclude(n) and
msg = "Origin of readStep is missing a PostUpdateNode."
}
query predicate argHasPostUpdate(ArgumentNode n, string msg) {
not hasPost(n) and
not isImmutableOrUnobservable(n) and
not any(ConsistencyConfiguration c).argHasPostUpdateExclude(n) and
msg = "ArgumentNode is missing PostUpdateNode."
}
@@ -177,6 +206,7 @@ module Consistency {
isPostUpdateNode(n) and
not clearsContent(n, _) and
simpleLocalFlowStep(_, n) and
not any(ConsistencyConfiguration c).postWithInFlowExclude(n) and
msg = "PostUpdateNode should not be the target of local flow."
}
}

6
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplSpecific.qll
View File

@@ -1,9 +1,15 @@
/**
* Provides Python-specific definitions for use in the data flow library.
*/
// we need to export `Unit` for the DataFlowImpl* files
private import python as Python
module Private {
import DataFlowPrivate
// import DataFlowDispatch
class Unit = Python::Unit;
}
module Public {

1178
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll
View File

File diff suppressed because it is too large Load Diff

96
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPublic.qll
View File

@@ -8,6 +8,7 @@ import semmle.python.dataflow.new.TypeTracker
import Attributes
import LocalSources
private import semmle.python.essa.SsaCompute
private import semmle.python.dataflow.new.internal.ImportStar
/**
* IPA type for data flow nodes.
@@ -24,13 +25,25 @@ newtype TNode =
/** A node corresponding to an SSA variable. */
TEssaNode(EssaVariable var) or
/** A node corresponding to a control flow node. */
TCfgNode(ControlFlowNode node) { isExpressionNode(node) } or
TCfgNode(ControlFlowNode node) {
isExpressionNode(node)
or
node.getNode() instanceof Pattern
} or
/** A synthetic node representing the value of an object before a state change */
TSyntheticPreUpdateNode(NeedsSyntheticPreUpdateNode post) or
/** A synthetic node representing the value of an object after a state change. */
TSyntheticPostUpdateNode(NeedsSyntheticPostUpdateNode pre) or
/** A node representing a global (module-level) variable in a specific module. */
TModuleVariableNode(Module m, GlobalVariable v) { v.getScope() = m and v.escapes() } or
TModuleVariableNode(Module m, GlobalVariable v) {
v.getScope() = m and
(
v.escapes()
or
isAccessedThroughImportStar(m) and
ImportStar::globalNameDefinedInModule(v.getId(), m)
)
} or
/**
* A node representing the overflow positional arguments to a call.
* That is, `call` contains more positional arguments than there are
@@ -70,7 +83,24 @@ newtype TNode =
* A synthetic node representing that there may be an iterable element
* for `consumer` to consume.
*/
TIterableElementNode(UnpackingAssignmentTarget consumer)
TIterableElementNode(UnpackingAssignmentTarget consumer) or
/**
* A synthetic node representing element content in a star pattern.
*/
TStarPatternElementNode(MatchStarPattern target) or
/**
* INTERNAL: Do not use.
*
* A synthetic node representing the data for an ORM model saved in a DB.
*/
// TODO: Limiting the classes here to the ones that are actually ORM models was
// non-trivial, since that logic is based on API::Node results, and trying to do this
// causes non-monotonic recursion, and makes the API graph evaluation recursive with
// data-flow, which might do bad things for performance.
//
// So for now we live with having these synthetic ORM nodes for _all_ classes, which
// is a bit wasteful, but we don't think it will hurt too much.
TSyntheticOrmModelNode(Class cls)
/** Helper for `Node::getEnclosingCallable`. */
private DataFlowCallable getCallableScope(Scope s) {
@@ -80,13 +110,19 @@ private DataFlowCallable getCallableScope(Scope s) {
result = getCallableScope(s.getEnclosingScope())
}
private import semmle.python.internal.CachedStages
/**
* An element, viewed as a node in a data flow graph. Either an SSA variable
* (`EssaNode`) or a control flow node (`CfgNode`).
*/
class Node extends TNode {
/** Gets a textual representation of this element. */
string toString() { result = "Data flow node" }
cached
string toString() {
Stages::DataFlow::ref() and
result = "Data flow node"
}
/** Gets the scope of this node. */
Scope getScope() { none() }
@@ -104,9 +140,11 @@ class Node extends TNode {
* For more information, see
* [Locations](https://codeql.github.com/docs/writing-codeql-queries/providing-locations-in-codeql-queries/).
*/
cached
predicate hasLocationInfo(
string filepath, int startline, int startcolumn, int endline, int endcolumn
) {
Stages::DataFlow::ref() and
this.getLocation().hasLocationInfo(filepath, startline, startcolumn, endline, endcolumn)
}
@@ -173,7 +211,7 @@ class CallCfgNode extends CfgNode, LocalSourceNode {
*/
Node getFunction() { result.asCfgNode() = node.getFunction() }
/** Gets the data-flow node corresponding to the i'th argument of the call corresponding to this data-flow node */
/** Gets the data-flow node corresponding to the i'th positional argument of the call corresponding to this data-flow node */
Node getArg(int i) { result.asCfgNode() = node.getArg(i) }
/** Gets the data-flow node corresponding to the named argument of the call corresponding to this data-flow node */
@@ -346,6 +384,8 @@ class ModuleVariableNode extends Node, TModuleVariableNode {
result.asCfgNode() = var.getALoad().getAFlowNode() and
// Ignore reads that happen when the module is imported. These are only executed once.
not result.getScope() = mod
or
this = import_star_read(result)
}
/** Gets an `EssaNode` that corresponds to an assignment of this global variable. */
@@ -358,6 +398,20 @@ class ModuleVariableNode extends Node, TModuleVariableNode {
override Location getLocation() { result = mod.getLocation() }
}
private predicate isAccessedThroughImportStar(Module m) { m = ImportStar::getStarImported(_) }
private ModuleVariableNode import_star_read(Node n) {
resolved_import_star_module(result.getModule(), result.getVariable().getId(), n)
}
pragma[nomagic]
private predicate resolved_import_star_module(Module m, string name, Node n) {
exists(NameNode nn | nn = n.asCfgNode() |
ImportStar::importStarResolvesTo(pragma[only_bind_into](nn), m) and
nn.getId() = name
)
}
/**
* The node holding the extra positional arguments to a call. This node is passed as a tuple
* to the starred parameter of the callable.
@@ -458,6 +512,21 @@ class IterableElementNode extends Node, TIterableElementNode {
override Location getLocation() { result = consumer.getLocation() }
}
/**
* A synthetic node representing element content of a star pattern.
*/
class StarPatternElementNode extends Node, TStarPatternElementNode {
CfgNode consumer;
StarPatternElementNode() { this = TStarPatternElementNode(consumer.getNode().getNode()) }
override string toString() { result = "StarPatternElement" }
override DataFlowCallable getEnclosingCallable() { result = consumer.getEnclosingCallable() }
override Location getLocation() { result = consumer.getLocation() }
}
/**
* A node that controls whether other nodes are evaluated.
*/
@@ -581,3 +650,20 @@ class AttributeContent extends TAttributeContent, Content {
override string toString() { result = "Attribute " + attr }
}
/**
* An entity that represents a set of `Content`s.
*
* The set may be interpreted differently depending on whether it is
* stored into (`getAStoreContent`) or read from (`getAReadContent`).
*/
class ContentSet instanceof Content {
/** Gets a content that may be stored into when storing into this set. */
Content getAStoreContent() { result = this }
/** Gets a content that may be read from when reading from this set. */
Content getAReadContent() { result = this }
/** Gets a textual representation of this content set. */
string toString() { result = super.toString() }
}

2
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowUtil.qll
View File

@@ -2,6 +2,7 @@
* Contains utility functions for writing data flow queries
*/
private import python
private import DataFlowPrivate
import DataFlowPublic
@@ -15,6 +16,7 @@ predicate localFlowStep(Node nodeFrom, Node nodeTo) { simpleLocalFlowStep(nodeFr
* Holds if data flows from `source` to `sink` in zero or more local
* (intra-procedural) steps.
*/
pragma[inline]
predicate localFlow(Node source, Node sink) { localFlowStep*(source, sink) }
/**

95
python/ql/lib/semmle/python/dataflow/new/internal/ImportStar.qll Normal file
View File

@@ -0,0 +1,95 @@
/** Provides predicates for reasoning about uses of `import *` in Python. */
private import python
private import semmle.python.dataflow.new.internal.Builtins
cached
module ImportStar {
/**
* Holds if `n` is an access of a variable called `name` (which is _not_ the name of a
* built-in, and which is _not_ a global defined in the enclosing module) inside the scope `s`.
*/
cached
predicate namePossiblyDefinedInImportStar(NameNode n, string name, Scope s) {
n.isLoad() and
name = n.getId() and
s = n.getScope().getEnclosingScope*() and
exists(potentialImportStarBase(s)) and
// Not already defined in an enclosing scope.
not isDefinedLocally(n.getNode())
}
/** Holds if `n` refers to a variable that is defined in the module in which it occurs. */
cached
private predicate isDefinedLocally(Name n) {
// Defined in an enclosing scope
enclosing_scope_defines_name(n.getScope(), n.getId())
or
// Defined as a built-in
n.getId() = Builtins::getBuiltinName()
or
// Defined as a global in this module
globalNameDefinedInModule(n.getId(), n.getEnclosingModule())
or
// A non-built-in that still has file-specific meaning
n.getId() in ["__name__", "__package__"]
}
pragma[nomagic]
private predicate enclosing_scope_defines_name(Scope s, string name) {
exists(LocalVariable v |
v.getId() = name and v.getScope() = s and not name = Builtins::getBuiltinName()
)
or
enclosing_scope_defines_name(s.getEnclosingScope(), name)
}
/** Holds if a global variable called `name` is assigned a value in the module `m`. */
cached
predicate globalNameDefinedInModule(string name, Module m) {
exists(NameNode n |
not exists(LocalVariable v | n.defines(v)) and
n.isStore() and
name = n.getId() and
m = n.getEnclosingModule()
)
}
/**
* Holds if `n` may refer to a global variable of the same name in the module `m`, accessible
* from the scope of `n` by a chain of `import *` imports.
*/
cached
predicate importStarResolvesTo(NameNode n, Module m) {
m = getStarImported+(n.getEnclosingModule()) and
globalNameDefinedInModule(n.getId(), m) and
not isDefinedLocally(n.getNode())
}
/**
* Gets a module that is imported from `m` via `import *`.
*/
cached
Module getStarImported(Module m) {
exists(ImportStar i |
i.getScope() = m and result = i.getModule().pointsTo().(ModuleValue).getScope()
)
}
/**
* Gets the data-flow node for a module imported with `from ... import *` inside the scope `s`.
*
* For example, given
*
* ```python
* from foo.bar import *
* from quux import *
* ```
*
* this would return the data-flow nodes corresponding to `foo.bar` and `quux`.
*/
cached
ControlFlowNode potentialImportStarBase(Scope s) {
result = any(ImportStarNode n | n.getScope() = s).getModule()
}
}

396
python/ql/lib/semmle/python/dataflow/new/internal/IterableUnpacking.qll Normal file
View File

@@ -0,0 +1,396 @@
/**
* The unpacking assignment takes the general form
* ```python
* sequence = iterable
* ```
* where `sequence` is either a tuple or a list and it can contain wildcards.
* The iterable can be any iterable, which means that (CodeQL modeling of) content
* will need to change type if it should be transferred from the LHS to the RHS.
*
* Note that (CodeQL modeling of) content does not have to change type on data-flow
* paths _inside_ the LHS, as the different allowed syntaxes here are merely a convenience.
* Consequently, we model all LHS sequences as tuples, which have the more precise content
* model, making flow to the elements more precise. If an element is a starred variable,
* we will have to mutate the content type to be list content.
*
* We may for instance have
* ```python
* (a, b) = ["a", SOURCE] # RHS has content `ListElementContent`
* ```
* Due to the abstraction for list content, we do not know whether `SOURCE`
* ends up in `a` or in `b`, so we want to overapproximate and see it in both.
*
* Using wildcards we may have
* ```python
* (a, *b) = ("a", "b", SOURCE) # RHS has content `TupleElementContent(2)`
* ```
* Since the starred variables are always assigned (Python-)type list, `*b` will be
* `["b", SOURCE]`, and we will again overapproximate and assign it
* content corresponding to anything found in the RHS.
*
* For a precise transfer
* ```python
* (a, b) = ("a", SOURCE) # RHS has content `TupleElementContent(1)`
* ```
* we wish to keep the precision, so only `b` receives the tuple content at index 1.
*
* Finally, `sequence` is actually a pattern and can have a more complicated structure,
* such as
* ```python
* (a, [b, *c]) = ("a", ["b", SOURCE]) # RHS has content `TupleElementContent(1); ListElementContent`
* ```
* where `a` should not receive content, but `b` and `c` should. `c` will be `[SOURCE]` so
* should have the content transferred, while `b` should read it.
*
* To transfer content from RHS to the elements of the LHS in the expression `sequence = iterable`,
* we use two synthetic nodes:
*
* - `TIterableSequence(sequence)` which captures the content-modeling the entire `sequence` will have
* (essentially just a copy of the content-modeling the RHS has)
*
* - `TIterableElement(sequence)` which captures the content-modeling that will be assigned to an element.
* Note that an empty access path means that the value we are tracking flows directly to the element.
*
*
* The `TIterableSequence(sequence)` is at this point superflous but becomes useful when handling recursive
* structures in the LHS, where `sequence` is some internal sequence node. We can have a uniform treatment
* by always having these two synthetic nodes. So we transfer to (or, in the recursive case, read into)
* `TIterableSequence(sequence)`, from which we take a read step to `TIterableElement(sequence)` and then a
* store step to `sequence`.
*
* This allows the unknown content from the RHS to be read into `TIterableElement(sequence)` and tuple content
* to then be stored into `sequence`. If the content is already tuple content, this inderection creates crosstalk
* between indices. Therefore, tuple content is never read into `TIterableElement(sequence)`; it is instead
* transferred directly from `TIterableSequence(sequence)` to `sequence` via a flow step. Such a flow step will
* also transfer other content, but only tuple content is further read from `sequence` into its elements.
*
* The strategy is then via several read-, store-, and flow steps:
* 1. a) [Flow] Content is transferred from `iterable` to `TIterableSequence(sequence)` via a
* flow step. From here, everything happens on the LHS.
*
* b) [Read] If the unpacking happens inside a for as in
* ```python
* for sequence in iterable
* ```
* then content is read from `iterable` to `TIterableSequence(sequence)`.
*
* 2. [Flow] Content is transferred from `TIterableSequence(sequence)` to `sequence` via a
* flow step. (Here only tuple content is relevant.)
*
* 3. [Read] Content is read from `TIterableSequence(sequence)` into `TIterableElement(sequence)`.
* As `sequence` is modeled as a tuple, we will not read tuple content as that would allow
* crosstalk.
*
* 4. [Store] Content is stored from `TIterableElement(sequence)` to `sequence`.
* Content type is `TupleElementContent` with indices taken from the syntax.
* For instance, if `sequence` is `(a, *b, c)`, content is written to index 0, 1, and 2.
* This is adequate as the route through `TIterableElement(sequence)` does not transfer precise content.
*
* 5. [Read] Content is read from `sequence` to its elements.
* a) If the element is a plain variable, the target is the corresponding essa node.
*
* b) If the element is itself a sequence, with control-flow node `seq`, the target is `TIterableSequence(seq)`.
*
* c) If the element is a starred variable, with control-flow node `v`, the target is `TIterableElement(v)`.
*
* 6. [Store] Content is stored from `TIterableElement(v)` to the essa variable for `v`, with
* content type `ListElementContent`.
*
* 7. [Flow, Read, Store] Steps 2 through 7 are repeated for all recursive elements which are sequences.
*
*
* We illustrate the above steps on the assignment
*
* ```python
* (a, b) = ["a", SOURCE]
* ```
*
* Looking at the content propagation to `a`:
* `["a", SOURCE]`: [ListElementContent]
*
* --Step 1a-->
*
* `TIterableSequence((a, b))`: [ListElementContent]
*
* --Step 3-->
*
* `TIterableElement((a, b))`: []
*
* --Step 4-->
*
* `(a, b)`: [TupleElementContent(0)]
*
* --Step 5a-->
*
* `a`: []
*
* Meaning there is data-flow from the RHS to `a` (an over approximation). The same logic would be applied to show there is data-flow to `b`. Note that _Step 3_ and _Step 4_ would not have been needed if the RHS had been a tuple (since that would have been able to use _Step 2_ instead).
*
* Another, more complicated example:
* ```python
* (a, [b, *c]) = ["a", [SOURCE]]
* ```
* where the path to `c` is
*
* `["a", [SOURCE]]`: [ListElementContent; ListElementContent]
*
* --Step 1a-->
*
* `TIterableSequence((a, [b, *c]))`: [ListElementContent; ListElementContent]
*
* --Step 3-->
*
* `TIterableElement((a, [b, *c]))`: [ListElementContent]
*
* --Step 4-->
*
* `(a, [b, *c])`: [TupleElementContent(1); ListElementContent]
*
* --Step 5b-->
*
* `TIterableSequence([b, *c])`: [ListElementContent]
*
* --Step 3-->
*
* `TIterableElement([b, *c])`: []
*
* --Step 4-->
*
* `[b, *c]`: [TupleElementContent(1)]
*
* --Step 5c-->
*
* `TIterableElement(c)`: []
*
* --Step 6-->
*
* `c`: [ListElementContent]
*/
private import python
private import DataFlowPublic
/**
* The target of a `for`, e.g. `x` in `for x in list` or in `[42 for x in list]`.
* This class also records the source, which in both above cases is `list`.
* This class abstracts away the differing representations of comprehensions and
* for statements.
*/
class ForTarget extends ControlFlowNode {
Expr source;
ForTarget() {
exists(For for |
source = for.getIter() and
this.getNode() = for.getTarget() and
not for = any(Comp comp).getNthInnerLoop(0)
)
or
exists(Comp comp |
source = comp.getIterable() and
this.getNode() = comp.getNthInnerLoop(0).getTarget()
)
}
Expr getSource() { result = source }
}
/** The LHS of an assignment, it also records the assigned value. */
class AssignmentTarget extends ControlFlowNode {
Expr value;
AssignmentTarget() {
exists(Assign assign | this.getNode() = assign.getATarget() | value = assign.getValue())
}
Expr getValue() { result = value }
}
/** A direct (or top-level) target of an unpacking assignment. */
class UnpackingAssignmentDirectTarget extends ControlFlowNode {
Expr value;
UnpackingAssignmentDirectTarget() {
this instanceof SequenceNode and
(
value = this.(AssignmentTarget).getValue()
or
value = this.(ForTarget).getSource()
)
}
Expr getValue() { result = value }
}
/** A (possibly recursive) target of an unpacking assignment. */
class UnpackingAssignmentTarget extends ControlFlowNode {
UnpackingAssignmentTarget() {
this instanceof UnpackingAssignmentDirectTarget
or
this = any(UnpackingAssignmentSequenceTarget parent).getAnElement()
}
}
/** A (possibly recursive) target of an unpacking assignment which is also a sequence. */
class UnpackingAssignmentSequenceTarget extends UnpackingAssignmentTarget instanceof SequenceNode {
ControlFlowNode getElement(int i) { result = super.getElement(i) }
ControlFlowNode getAnElement() { result = this.getElement(_) }
}
/**
* Step 1a
* Data flows from `iterable` to `TIterableSequence(sequence)`
*/
predicate iterableUnpackingAssignmentFlowStep(Node nodeFrom, Node nodeTo) {
exists(AssignmentTarget target |
nodeFrom.asExpr() = target.getValue() and
nodeTo = TIterableSequenceNode(target)
)
}
/**
* Step 1b
* Data is read from `iterable` to `TIterableSequence(sequence)`
*/
predicate iterableUnpackingForReadStep(CfgNode nodeFrom, Content c, Node nodeTo) {
exists(ForTarget target |
nodeFrom.asExpr() = target.getSource() and
target instanceof SequenceNode and
nodeTo = TIterableSequenceNode(target)
) and
(
c instanceof ListElementContent
or
c instanceof SetElementContent
)
}
/**
* Step 2
* Data flows from `TIterableSequence(sequence)` to `sequence`
*/
predicate iterableUnpackingTupleFlowStep(Node nodeFrom, Node nodeTo) {
exists(UnpackingAssignmentSequenceTarget target |
nodeFrom = TIterableSequenceNode(target) and
nodeTo.asCfgNode() = target
)
}
/**
* Step 3
* Data flows from `TIterableSequence(sequence)` into `TIterableElement(sequence)`.
* As `sequence` is modeled as a tuple, we will not read tuple content as that would allow
* crosstalk.
*/
predicate iterableUnpackingConvertingReadStep(Node nodeFrom, Content c, Node nodeTo) {
exists(UnpackingAssignmentSequenceTarget target |
nodeFrom = TIterableSequenceNode(target) and
nodeTo = TIterableElementNode(target) and
(
c instanceof ListElementContent
or
c instanceof SetElementContent
// TODO: dict content in iterable unpacking not handled
)
)
}
/**
* Step 4
* Data flows from `TIterableElement(sequence)` to `sequence`.
* Content type is `TupleElementContent` with indices taken from the syntax.
* For instance, if `sequence` is `(a, *b, c)`, content is written to index 0, 1, and 2.
*/
predicate iterableUnpackingConvertingStoreStep(Node nodeFrom, Content c, Node nodeTo) {
exists(UnpackingAssignmentSequenceTarget target |
nodeFrom = TIterableElementNode(target) and
nodeTo.asCfgNode() = target and
exists(int index | exists(target.getElement(index)) |
c.(TupleElementContent).getIndex() = index
)
)
}
/**
* Step 5
* For a sequence node inside an iterable unpacking, data flows from the sequence to its elements. There are
* three cases for what `toNode` should be:
* a) If the element is a plain variable, `toNode` is the corresponding essa node.
*
* b) If the element is itself a sequence, with control-flow node `seq`, `toNode` is `TIterableSequence(seq)`.
*
* c) If the element is a starred variable, with control-flow node `v`, `toNode` is `TIterableElement(v)`.
*/
predicate iterableUnpackingElementReadStep(Node nodeFrom, Content c, Node nodeTo) {
exists(
UnpackingAssignmentSequenceTarget target, int index, ControlFlowNode element, int starIndex
|
target.getElement(starIndex) instanceof StarredNode
or
not exists(target.getAnElement().(StarredNode)) and
starIndex = -1
|
nodeFrom.asCfgNode() = target and
element = target.getElement(index) and
(
if starIndex = -1 or index < starIndex
then c.(TupleElementContent).getIndex() = index
else
// This could get big if big tuples exist
if index = starIndex
then c.(TupleElementContent).getIndex() >= index
else c.(TupleElementContent).getIndex() >= index - 1
) and
(
if element instanceof SequenceNode
then
// Step 5b
nodeTo = TIterableSequenceNode(element)
else
if element instanceof StarredNode
then
// Step 5c
nodeTo = TIterableElementNode(element)
else
// Step 5a
nodeTo.asVar().getDefinition().(MultiAssignmentDefinition).getDefiningNode() = element
)
)
}
/**
* Step 6
* Data flows from `TIterableElement(v)` to the essa variable for `v`, with
* content type `ListElementContent`.
*/
predicate iterableUnpackingStarredElementStoreStep(Node nodeFrom, Content c, Node nodeTo) {
exists(ControlFlowNode starred | starred.getNode() instanceof Starred |
nodeFrom = TIterableElementNode(starred) and
nodeTo.asVar().getDefinition().(MultiAssignmentDefinition).getDefiningNode() = starred and
c instanceof ListElementContent
)
}
/** All read steps associated with unpacking assignment. */
predicate iterableUnpackingReadStep(Node nodeFrom, Content c, Node nodeTo) {
iterableUnpackingForReadStep(nodeFrom, c, nodeTo)
or
iterableUnpackingElementReadStep(nodeFrom, c, nodeTo)
or
iterableUnpackingConvertingReadStep(nodeFrom, c, nodeTo)
}
/** All store steps associated with unpacking assignment. */
predicate iterableUnpackingStoreStep(Node nodeFrom, Content c, Node nodeTo) {
iterableUnpackingStarredElementStoreStep(nodeFrom, c, nodeTo)
or
iterableUnpackingConvertingStoreStep(nodeFrom, c, nodeTo)
}
/** All flow steps associated with unpacking assignment. */
predicate iterableUnpackingFlowStep(Node nodeFrom, Node nodeTo) {
iterableUnpackingAssignmentFlowStep(nodeFrom, nodeTo)
or
iterableUnpackingTupleFlowStep(nodeFrom, nodeTo)
}

5
python/ql/lib/semmle/python/dataflow/new/internal/LocalSources.qll
View File

@@ -6,9 +6,10 @@
* local tracking within a function.
*/
import python
private import python
import DataFlowPublic
private import DataFlowPrivate
private import semmle.python.internal.CachedStages
/**
* A data flow node that is a source of local flow. This includes things like
@@ -33,6 +34,7 @@ private import DataFlowPrivate
class LocalSourceNode extends Node {
cached
LocalSourceNode() {
Stages::DataFlow::ref() and
this instanceof ExprNode and
not simpleLocalFlowStep(_, this)
or
@@ -176,6 +178,7 @@ private module Cached {
*/
cached
predicate hasLocalSource(Node sink, LocalSourceNode source) {
Stages::DataFlow::ref() and
source = sink
or
exists(Node second |

311
python/ql/lib/semmle/python/dataflow/new/internal/MatchUnpacking.qll Normal file
View File

@@ -0,0 +1,311 @@
/**
* There are a number of patterns available for the match statement.
* Each one transfers data and content differently to its parts.
*
* Furthermore, given a successful match, we can infer some data about
* the subject. Consider the example:
* ```python
* match choice:
* case 'Y':
* ...body
* ```
* Inside `body`, we know that `choice` has the value `'Y'`.
*
* A similar thing happens with the "as pattern". Consider the example:
* ```python
* match choice:
* case ('y'|'Y') as c:
* ...body
* ```
* By the binding rules, there is data flow from `choice` to `c`. But we
* can infer the value of `c` to be either `'y'` or `'Y'` if the match succeeds.
*
* We will treat such inferences separately as guards. First we will model the data flow
* stemming from the bindings and the matching of shape. Below, 'subject' is not necessarily the
* top-level subject of the match, but rather the part recursively matched by the current pattern.
* For instance, in the example:
* ```python
* match command:
* case ('quit' as c) | ('go', ('up'|'down') as c):
* ...body
* ```
* `command` is the subject of first the as-pattern, while the second component of `command`
* is the subject of the second as-pattern. As such, 'subject' refers to the pattern under evaluation.
*
* - as pattern: subject flows to alias as well as to the interior pattern
* - or pattern: subject flows to each alternative
* - literal pattern: flow from the literal to the pattern, to add information
* - capture pattern: subject flows to the variable
* - wildcard pattern: no flow
* - value pattern: flow from the value to the pattern, to add information
* - sequence pattern: each element reads from subject at the associated index
* - star pattern: subject flows to the variable, possibly via a conversion
* - mapping pattern: each value reads from subject at the associated key
* - double star pattern: subject flows to the variable, possibly via a conversion
* - key-value pattern: the value reads from the subject at the key (see mapping pattern)
* - class pattern: all keywords read the appropriate attribute from the subject
* - keyword pattern: the appropriate attribute is read from the subject (see class pattern)
*
* Inside the class pattern, we also find positional arguments. They are converted to
* keyword arguments using the `__match_args__` attribute on the class. We do not
* currently model this.
*/
private import python
private import DataFlowPublic
/**
* Holds when there is flow from the subject `nodeFrom` to the (top-level) pattern `nodeTo` of a `match` statement.
*
* The subject of a match flows to each top-level pattern
* (a pattern directly under a `case` statement).
*
* We could consider a model closer to use-use-flow, where the subject
* only flows to the first top-level pattern and from there to the
* following ones.
*/
predicate matchSubjectFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchStmt match, Expr subject, Pattern target |
subject = match.getSubject() and
target = match.getCase(_).(Case).getPattern()
|
nodeFrom.asExpr() = subject and
nodeTo.asCfgNode().getNode() = target
)
}
/**
* as pattern: subject flows to alias as well as to the interior pattern
* syntax (toplevel): `case pattern as alias:`
*/
predicate matchAsFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchAsPattern subject, Name alias | alias = subject.getAlias() |
// We make the subject flow to the interior pattern via the alias.
// That way, information can propagate from the interior pattern to the alias.
//
// the subject flows to the interior pattern
nodeFrom.asCfgNode().getNode() = subject and
nodeTo.asCfgNode().getNode() = subject.getPattern()
or
// the interior pattern flows to the alias
nodeFrom.asCfgNode().getNode() = subject.getPattern() and
nodeTo.asVar().getDefinition().(PatternAliasDefinition).getDefiningNode().getNode() = alias
)
}
/**
* or pattern: subject flows to each alternative
* syntax (toplevel): `case alt1 | alt2:`
*/
predicate matchOrFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchOrPattern subject, Pattern pattern | pattern = subject.getAPattern() |
nodeFrom.asCfgNode().getNode() = subject and
nodeTo.asCfgNode().getNode() = pattern
)
}
/**
* literal pattern: flow from the literal to the pattern, to add information
* syntax (toplevel): `case literal:`
*/
predicate matchLiteralFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchLiteralPattern pattern, Expr literal | literal = pattern.getLiteral() |
nodeFrom.asExpr() = literal and
nodeTo.asCfgNode().getNode() = pattern
)
}
/**
* capture pattern: subject flows to the variable
* syntax (toplevel): `case var:`
*/
predicate matchCaptureFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchCapturePattern capture, Name var | capture.getVariable() = var |
nodeFrom.asCfgNode().getNode() = capture and
nodeTo.asVar().getDefinition().(PatternCaptureDefinition).getDefiningNode().getNode() = var
)
}
/**
* value pattern: flow from the value to the pattern, to add information
* syntax (toplevel): `case Dotted.value:`
*/
predicate matchValueFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchValuePattern pattern, Expr value | value = pattern.getValue() |
nodeFrom.asExpr() = value and
nodeTo.asCfgNode().getNode() = pattern
)
}
/**
* sequence pattern: each element reads from subject at the associated index
* syntax (toplevel): `case [a, b]:`
*/
predicate matchSequenceReadStep(Node nodeFrom, Content c, Node nodeTo) {
exists(MatchSequencePattern subject, int index, Pattern element |
element = subject.getPattern(index)
|
nodeFrom.asCfgNode().getNode() = subject and
nodeTo.asCfgNode().getNode() = element and
(
// tuple content
c.(TupleElementContent).getIndex() = index
or
// list content
c instanceof ListElementContent
// set content is excluded from sequence patterns,
// see https://www.python.org/dev/peps/pep-0635/#sequence-patterns
)
)
}
/**
* star pattern: subject flows to the variable, possibly via a conversion
* syntax (toplevel): `case *var:`
*
* We decompose this flow into a read step and a store step. The read step
* reads both tuple and list content, the store step only stores list content.
* This way, we convert all content to list content.
*
* This is the read step.
*/
predicate matchStarReadStep(Node nodeFrom, Content c, Node nodeTo) {
exists(MatchSequencePattern subject, int index, MatchStarPattern star |
star = subject.getPattern(index)
|
nodeFrom.asCfgNode().getNode() = subject and
nodeTo = TStarPatternElementNode(star) and
(
// tuple content
c.(TupleElementContent).getIndex() >= index
or
// list content
c instanceof ListElementContent
// set content is excluded from sequence patterns,
// see https://www.python.org/dev/peps/pep-0635/#sequence-patterns
)
)
}
/**
* star pattern: subject flows to the variable, possibly via a conversion
* syntax (toplevel): `case *var:`
*
* We decompose this flow into a read step and a store step. The read step
* reads both tuple and list content, the store step only stores list content.
* This way, we convert all content to list content.
*
* This is the store step.
*/
predicate matchStarStoreStep(Node nodeFrom, Content c, Node nodeTo) {
exists(MatchStarPattern star |
nodeFrom = TStarPatternElementNode(star) and
nodeTo.asCfgNode().getNode() = star.getTarget() and
c instanceof ListElementContent
)
}
/**
* mapping pattern: each value reads from subject at the associated key
* syntax (toplevel): `case {"color": c, "height": x}:`
*/
predicate matchMappingReadStep(Node nodeFrom, Content c, Node nodeTo) {
exists(
MatchMappingPattern subject, MatchKeyValuePattern keyValue, MatchLiteralPattern key,
Pattern value
|
keyValue = subject.getAMapping() and
key = keyValue.getKey() and
value = keyValue.getValue()
|
nodeFrom.asCfgNode().getNode() = subject and
nodeTo.asCfgNode().getNode() = value and
c.(DictionaryElementContent).getKey() = key.getLiteral().(StrConst).getText()
)
}
/**
* double star pattern: subject flows to the variable, possibly via a conversion
* syntax (toplevel): `case {**var}:`
*
* Dictionary content flows to the double star, but all mentioned keys in the
* mapping pattern should be cleared.
*/
predicate matchMappingFlowStep(Node nodeFrom, Node nodeTo) {
exists(MatchMappingPattern subject, MatchDoubleStarPattern dstar | dstar = subject.getAMapping() |
nodeFrom.asCfgNode().getNode() = subject and
nodeTo.asCfgNode().getNode() = dstar.getTarget()
)
}
/**
* Bindings that are mentioned in a mapping pattern will not be available
* to a double star pattern in the same mapping pattern.
*/
predicate matchMappingClearStep(Node n, Content c) {
exists(
MatchMappingPattern subject, MatchKeyValuePattern keyValue, MatchLiteralPattern key,
MatchDoubleStarPattern dstar
|
keyValue = subject.getAMapping() and
key = keyValue.getKey() and
dstar = subject.getAMapping()
|
n.asCfgNode().getNode() = dstar.getTarget() and
c.(DictionaryElementContent).getKey() = key.getLiteral().(StrConst).getText()
)
}
/**
* class pattern: all keywords read the appropriate attribute from the subject
* syntax (toplevel): `case ClassName(attr = val):`
*/
predicate matchClassReadStep(Node nodeFrom, Content c, Node nodeTo) {
exists(MatchClassPattern subject, MatchKeywordPattern keyword, Name attr, Pattern value |
keyword = subject.getKeyword(_) and
attr = keyword.getAttribute() and
value = keyword.getValue()
|
nodeFrom.asCfgNode().getNode() = subject and
nodeTo.asCfgNode().getNode() = value and
c.(AttributeContent).getAttribute() = attr.getId()
)
}
/** All flow steps associated with match. */
predicate matchFlowStep(Node nodeFrom, Node nodeTo) {
matchSubjectFlowStep(nodeFrom, nodeTo)
or
matchAsFlowStep(nodeFrom, nodeTo)
or
matchOrFlowStep(nodeFrom, nodeTo)
or
matchLiteralFlowStep(nodeFrom, nodeTo)
or
matchCaptureFlowStep(nodeFrom, nodeTo)
or
matchValueFlowStep(nodeFrom, nodeTo)
or
matchMappingFlowStep(nodeFrom, nodeTo)
}
/** All read steps associated with match. */
predicate matchReadStep(Node nodeFrom, Content c, Node nodeTo) {
matchClassReadStep(nodeFrom, c, nodeTo)
or
matchSequenceReadStep(nodeFrom, c, nodeTo)
or
matchMappingReadStep(nodeFrom, c, nodeTo)
or
matchStarReadStep(nodeFrom, c, nodeTo)
}
/** All store steps associated with match. */
predicate matchStoreStep(Node nodeFrom, Content c, Node nodeTo) {
matchStarStoreStep(nodeFrom, c, nodeTo)
}
/**
* All clear steps associated with match
*/
predicate matchClearStep(Node n, Content c) { matchMappingClearStep(n, c) }

8
python/ql/lib/semmle/python/dataflow/new/internal/PrintNode.qll
View File

@@ -9,6 +9,7 @@
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.internal.DataFlowPrivate as DataFlowPrivate
/**
* INTERNAL: Do not use.
@@ -66,7 +67,12 @@ string prettyNodeForInlineTest(DataFlow::Node node) {
result = "[post]" + prettyExpr(e)
)
or
exists(Expr e | e = node.(DataFlowPrivate::SyntheticPreUpdateNode).getPostUpdateNode().asExpr() |
result = "[pre]" + prettyExpr(e)
)
or
not exists(node.asExpr()) and
not exists(Expr e | e = node.(DataFlow::PostUpdateNode).getPreUpdateNode().asExpr()) and
not exists(node.(DataFlow::PostUpdateNode).getPreUpdateNode().asExpr()) and
not exists(node.(DataFlowPrivate::SyntheticPreUpdateNode).getPostUpdateNode().asExpr()) and
result = node.toString()
}

50
python/ql/lib/semmle/python/dataflow/new/internal/TaintTrackingPrivate.qll
View File

@@ -10,6 +10,12 @@ private import semmle.python.ApiGraphs
*/
predicate defaultTaintSanitizer(DataFlow::Node node) { none() }
/**
* Holds if `guard` should be a sanitizer guard in all global taint flow configurations
* but not in local taint.
*/
predicate defaultTaintSanitizerGuard(DataFlow::BarrierGuard guard) { none() }
/**
* Holds if default `TaintTracking::Configuration`s should allow implicit reads
* of `c` at sinks and inputs to additional taint steps.
@@ -53,6 +59,8 @@ private module Cached {
DataFlowPrivate::iterableUnpackingStoreStep(nodeFrom, _, nodeTo)
or
awaitStep(nodeFrom, nodeTo)
or
asyncWithStep(nodeFrom, nodeTo)
}
}
@@ -159,8 +167,25 @@ predicate stringManipulation(DataFlow::CfgNode nodeFrom, DataFlow::CfgNode nodeT
*/
predicate containerStep(DataFlow::CfgNode nodeFrom, DataFlow::Node nodeTo) {
// construction by literal
// TODO: Not limiting the content argument here feels like a BIG hack, but we currently get nothing for free :|
DataFlowPrivate::storeStep(nodeFrom, _, nodeTo)
//
// TODO: once we have proper flow-summary modeling, we might not need this step any
// longer -- but there needs to be a matching read-step for the store-step, and we
// don't provide that right now.
DataFlowPrivate::listStoreStep(nodeFrom, _, nodeTo)
or
DataFlowPrivate::setStoreStep(nodeFrom, _, nodeTo)
or
DataFlowPrivate::tupleStoreStep(nodeFrom, _, nodeTo)
or
DataFlowPrivate::dictStoreStep(nodeFrom, _, nodeTo)
or
// comprehension, so there is taint-flow from `x` in `[x for x in xs]` to the
// resulting list of the list-comprehension.
//
// TODO: once we have proper flow-summary modeling, we might not need this step any
// longer -- but there needs to be a matching read-step for the store-step, and we
// don't provide that right now.
DataFlowPrivate::comprehensionStoreStep(nodeFrom, _, nodeTo)
or
// constructor call
exists(DataFlow::CallCfgNode call | call = nodeTo |
@@ -211,3 +236,24 @@ predicate copyStep(DataFlow::CfgNode nodeFrom, DataFlow::CfgNode nodeTo) {
predicate awaitStep(DataFlow::Node nodeFrom, DataFlow::Node nodeTo) {
nodeTo.asExpr().(Await).getValue() = nodeFrom.asExpr()
}
/**
* Holds if taint can flow from `nodeFrom` to `nodeTo` inside an `async with` statement.
*
* For example in
* ```python
* async with open("foo") as f:
* ```
* the variable `f` is tainted if the result of `open("foo")` is tainted.
*/
predicate asyncWithStep(DataFlow::Node nodeFrom, DataFlow::Node nodeTo) {
exists(With with, ControlFlowNode contextManager, ControlFlowNode var |
nodeFrom.(DataFlow::CfgNode).getNode() = contextManager and
nodeTo.(DataFlow::EssaNode).getVar().getDefinition().(WithDefinition).getDefiningNode() = var and
// see `with_flow` in `python/ql/src/semmle/python/dataflow/Implementation.qll`
with.getContextExpr() = contextManager.getNode() and
with.getOptionalVars() = var.getNode() and
with.isAsync() and
contextManager.strictlyDominates(var)
)
}

2
python/ql/lib/semmle/python/dataflow/new/internal/TaintTrackingPublic.qll
View File

@@ -14,12 +14,14 @@ private import semmle.python.Frameworks
* Holds if taint propagates from `source` to `sink` in zero or more local
* (intra-procedural) steps.
*/
pragma[inline]
predicate localTaint(DataFlow::Node source, DataFlow::Node sink) { localTaintStep*(source, sink) }
/**
* Holds if taint can flow from `e1` to `e2` in zero or more local (intra-procedural)
* steps.
*/
pragma[inline]
predicate localExprTaint(Expr e1, Expr e2) {
localTaint(DataFlow::exprNode(e1), DataFlow::exprNode(e2))
}

67
python/ql/lib/semmle/python/dataflow/new/internal/TypeTracker.qll
View File

@@ -3,7 +3,7 @@
private import TypeTrackerSpecific
/**
* Any string that may appear as the name of a piece of content. This will usually include things like:
* A string that may appear as the name of a piece of content. This will usually include things like:
* - Attribute names (in Python)
* - Property names (in JavaScript)
*
@@ -18,7 +18,7 @@ class ContentName extends string {
ContentName() { this = getPossibleContentName() }
}
/** Either a content name, or the empty string (representing no content). */
/** A content name, or the empty string (representing no content). */
class OptionalContentName extends string {
OptionalContentName() { this instanceof ContentName or this = "" }
}
@@ -34,7 +34,8 @@ private module Cached {
CallStep() or
ReturnStep() or
StoreStep(ContentName content) or
LoadStep(ContentName content)
LoadStep(ContentName content) or
JumpStep()
/** Gets the summary resulting from appending `step` to type-tracking summary `tt`. */
cached
@@ -49,6 +50,30 @@ private module Cached {
step = LoadStep(content) and result = MkTypeTracker(hasCall, "")
or
exists(string p | step = StoreStep(p) and content = "" and result = MkTypeTracker(hasCall, p))
or
step = JumpStep() and
result = MkTypeTracker(false, content)
)
}
/** Gets the summary resulting from prepending `step` to this type-tracking summary. */
cached
TypeBackTracker prepend(TypeBackTracker tbt, StepSummary step) {
exists(Boolean hasReturn, string content | tbt = MkTypeBackTracker(hasReturn, content) |
step = LevelStep() and result = tbt
or
step = CallStep() and hasReturn = false and result = tbt
or
step = ReturnStep() and result = MkTypeBackTracker(true, content)
or
exists(string p |
step = LoadStep(p) and content = "" and result = MkTypeBackTracker(hasReturn, p)
)
or
step = StoreStep(content) and result = MkTypeBackTracker(hasReturn, "")
or
step = JumpStep() and
result = MkTypeBackTracker(false, content)
)
}
@@ -92,12 +117,17 @@ class StepSummary extends TStepSummary {
exists(string content | this = StoreStep(content) | result = "store " + content)
or
exists(string content | this = LoadStep(content) | result = "load " + content)
or
this instanceof JumpStep and result = "jump"
}
}
pragma[noinline]
private predicate smallstepNoCall(Node nodeFrom, TypeTrackingNode nodeTo, StepSummary summary) {
jumpStep(nodeFrom, nodeTo) and
summary = JumpStep()
or
levelStep(nodeFrom, nodeTo) and
summary = LevelStep()
or
exists(string content |
@@ -182,7 +212,7 @@ module StepSummary {
private newtype TTypeTracker = MkTypeTracker(Boolean hasCall, OptionalContentName content)
/**
* Summary of the steps needed to track a value to a given dataflow node.
* A summary of the steps needed to track a value to a given dataflow node.
*
* This can be used to track objects that implement a certain API in order to
* recognize calls to that API. Note that type-tracking does not by itself provide a
@@ -329,7 +359,7 @@ module TypeTracker {
private newtype TTypeBackTracker = MkTypeBackTracker(Boolean hasReturn, OptionalContentName content)
/**
* Summary of the steps needed to back-track a use of a value to a given dataflow node.
* A summary of the steps needed to back-track a use of a value to a given dataflow node.
*
* This can for example be used to track callbacks that are passed to a certain API,
* so we can model specific parameters of that callback as having a certain type.
@@ -365,19 +395,7 @@ class TypeBackTracker extends TTypeBackTracker {
TypeBackTracker() { this = MkTypeBackTracker(hasReturn, content) }
/** Gets the summary resulting from prepending `step` to this type-tracking summary. */
TypeBackTracker prepend(StepSummary step) {
step = LevelStep() and result = this
or
step = CallStep() and hasReturn = false and result = this
or
step = ReturnStep() and result = MkTypeBackTracker(true, content)
or
exists(string p |
step = LoadStep(p) and content = "" and result = MkTypeBackTracker(hasReturn, p)
)
or
step = StoreStep(content) and result = MkTypeBackTracker(hasReturn, "")
}
TypeBackTracker prepend(StepSummary step) { result = prepend(this, step) }
/** Gets a textual representation of this summary. */
string toString() {
@@ -459,6 +477,19 @@ class TypeBackTracker extends TTypeBackTracker {
simpleLocalFlowStep(nodeFrom, nodeTo) and
this = result
}
/**
* Gets a forwards summary that is compatible with this backwards summary.
* That is, if this summary describes the steps needed to back-track a value
* from `sink` to `mid`, and the result is a valid summary of the steps needed
* to track a value from `source` to `mid`, then the value from `source` may
* also flow to `sink`.
*/
TypeTracker getACompatibleTypeTracker() {
exists(boolean hasCall | result = MkTypeTracker(hasCall, content) |
hasCall = false or this.hasReturn() = false
)
}
}
/** Provides predicates for implementing custom `TypeBackTracker`s. */

11
python/ql/lib/semmle/python/dataflow/new/internal/TypeTrackerSpecific.qll
View File

@@ -5,6 +5,7 @@
private import python
private import semmle.python.dataflow.new.internal.DataFlowPublic as DataFlowPublic
private import semmle.python.dataflow.new.internal.DataFlowPrivate as DataFlowPrivate
import semmle.python.internal.CachedStages
class Node = DataFlowPublic::Node;
@@ -12,13 +13,19 @@ class TypeTrackingNode = DataFlowPublic::TypeTrackingNode;
predicate simpleLocalFlowStep = DataFlowPrivate::simpleLocalFlowStep/2;
predicate jumpStep = DataFlowPrivate::jumpStep/2;
predicate jumpStep = DataFlowPrivate::jumpStepSharedWithTypeTracker/2;
/** Holds if there is a level step from `pred` to `succ`. */
predicate levelStep(Node pred, Node succ) { none() }
/**
* Gets the name of a possible piece of content. For Python, this is currently only attribute names,
* using the name of the attribute for the corresponding content.
*/
string getPossibleContentName() { result = any(DataFlowPublic::AttrRef a).getAttributeName() }
string getPossibleContentName() {
Stages::TypeTracking::ref() and // the TypeTracking::append() etc. predicates that we want to cache depend on this predicate, so we can place the `ref()` call here to get around identical files.
result = any(DataFlowPublic::AttrRef a).getAttributeName()
}
/**
* Gets a callable for the call where `nodeFrom` is used as the `i`'th argument.

138
python/ql/lib/semmle/python/dataflow/new/internal/readme.md
View File

@@ -1,138 +0,0 @@
# Using the shared dataflow library
## File organisation
The files currently live in `experimental` (whereas the existing implementation lives in `semmle\python\dataflow`).
In there is found `DataFlow.qll`, `DataFlow2.qll` etc. which refer to `internal\DataFlowImpl`, `internal\DataFlowImpl2` etc. respectively. The `DataFlowImplN`-files are all identical copies to avoid mutual recursion. They start off by including two files `internal\DataFlowImplCommon` and `internal\DataFlowImplSpecific`. The former contains all the language-agnostic definitions, while the latter is where we describe our favorite language. `Sepcific` simply forwards to two other files `internal\DataFlowPrivate.qll` and `internal\DataFlowPublic.qll`. Definitions in the former will be hidden behind a `private` modifier, while those in the latter can be referred to in data flow queries. For instance, the definition of `DataFlow::Node` should likely be in `DataFlowPublic.qll`.
## Define the dataflow graph
In order to use the dataflow library, we need to define the dataflow graph,
that is define the nodes and the edges.
### Define the nodes
The nodes are defined in the type `DataFlow::Node` (found in `DataFlowPublic.qll`).
This should likely be an IPA type, so we can extend it as needed.
Typical cases needed to construct the call graph include
- argument node
- parameter node
- return node
Typical extensions include
- postupdate nodes
- implicit `this`-nodes
### Define the edges
The edges split into local flow (within a function) and global flow (the call graph, between functions/procedures).
Extra flow, such as reading from and writing to global variables, can be captured in `jumpStep`.
The local flow should be obtainalble from an SSA computation.
Local flow nodes are generally either control flow nodes or SSA variables.
Flow from control flow nodes to SSA variables comes from SSA variable definitions, while flow from SSA variables to control flow nodes comes from def-use pairs.
The global flow should be obtainable from a `PointsTo` analysis. It is specified via `viableCallable` and
`getAnOutNode`. Consider making `ReturnKind` a singleton IPA type as in java.
Global flow includes local flow within a consistent call context. Thus, for local flow to count as global flow, all relevant nodes should implement `getEnclosingCallable`.
If complicated dispatch needs to be modelled, try using the `[reduced|pruned]viable*` predicates.
## Field flow
To track flow through fields we need to provide a model of fields, that is the `Content` class.
Field access is specified via `read_step` and `store_step`.
Work is being done to make field flow handle lists and dictionaries and the like.
`PostUpdateNode`s become important when field flow is used, as they track modifications to fields resulting from function calls.
## Type pruning
If type information is available, flows can be discarded on the grounds of type mismatch.
Tracked types are given by the class `DataFlowType` and the predicate `getTypeBound`, and compatibility is recorded in the predicate `compatibleTypes`.
If type pruning is not used, `compatibleTypes` should be implemented as `any`; if it is implemented, say, as `none`, all flows will be pruned.
Further, possible casts are given by the class `CastNode`.
---
# Plan
## Stage I, data flow
### Phase 0, setup
Define minimal IPA type for `DataFlow::Node`
Define all required predicates empty (via `none()`),
except `compatibleTypes` which should be `any()`.
Define `ReturnKind`, `DataFlowType`, and `Content` as singleton IPA types.
### Phase 1, local flow
Implement `simpleLocalFlowStep` based on the existing SSA computation
### Phase 2, local flow
Implement `viableCallable` and `getAnOutNode` based on the existing predicate `PointsTo`.
### Phase 3, field flow
Redefine `Content` and implement `read_step` and `store_step`.
Review use of post-update nodes.
### Phase 4, type pruning
Use type trackers to obtain relevant type information and redefine `DataFlowType` to contain appropriate cases. Record the type information in `getTypeBound`.
Implement `compatibleTypes` (perhaps simply as the identity).
If necessary, re-implement `getErasedRepr` and `ppReprType`.
If necessary, redefine `CastNode`.
### Phase 5, bonus
Review possible use of `[reduced|pruned]viable*` predicates.
Review need for more elaborate `ReturnKind`.
Review need for non-empty `jumpStep`.
Review need for non-empty `isUnreachableInCall`.
## Stage II, taint tracking
# Phase 0, setup
Implement all predicates empty.
# Phase 1, experiments
Try recovering an existing taint tracking query by implementing sources, sinks, sanitizers, and barriers.
---
# Status
## Achieved
- Copy of shared library; implemented enough predicates to make it compile.
- Simple flow into, out of, and through functions.
- Some tests, in particular a sceleton for something comprehensive.
## TODO
- Implementation has largely been done by finding a plausibly-sounding predicate in the python library to refer to. We should review that we actually have the intended semantics in all places.
- Comprehensive testing.
- The regression tests track the value of guards in order to eliminate impossible data flow. We currently have regressions because of this. We cannot readily replicate the existing method, as it uses the interdefinedness of data flow and taint tracking (there is a boolean taint kind). C++ [does something similar](https://github.com/github/codeql/blob/master/cpp/ql/src/semmle/code/cpp/controlflow/internal/ConstantExprs.qll#L27-L36) for eliminating impossible control flow, which we might be able to replicate (they infer values of "interesting" control flow nodes, which are those needed to determine values of guards).
- Flow for some syntactic constructs are done via extra taint steps in the existing implementation, we should find a way to get data flow for it. Some of this should be covered by field flow.
- A document is being written about proper use of the shared data flow library, this should be adhered to. In particular, we should consider replacing def-use with def-to-first-use and use-to-next-use in local flow.
- We seem to get duplicated results for global flow, as well as flow with and without type (so four times the "unique" results).
- We currently consider control flow nodes like exit nodes for functions, we should probably filter down which ones are of interest.
- We should probably override ToString for a number of data flow nodes.
- Test flow through classes, constructors and methods.
- What happens with named arguments? What does C# do?
- What should the enclosable callable for global variables be? C++ [makes it the variable itself](https://github.com/github/codeql/blob/master/cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowUtil.qll#L417), C# seems to not have nodes for these but only for their reads and writes.
- Is `yield` another return type? If not, how is it handled?
- Should `OutNode` include magic function calls?
- Consider creating an internal abstract class for nodes as C# does. Among other things, this can help the optimizer by stating that `getEnclosingCallable` [is functional](https://github.com/github/codeql/blob/master/csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowPublic.qll#L62).

68
python/ql/lib/semmle/python/dataflow/new/internal/tainttracking1/TaintTrackingImpl.qll
View File

@@ -61,15 +61,32 @@ abstract class Configuration extends DataFlow::Configuration {
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSource(DataFlow::Node source);
override predicate isSource(DataFlow::Node source) { none() }
/**
* Holds if `sink` is a relevant taint sink.
* Holds if `source` is a relevant taint source with the given initial
* `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSink(DataFlow::Node sink);
override predicate isSource(DataFlow::Node source, DataFlow::FlowState state) { none() }
/**
* Holds if `sink` is a relevant taint sink
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink) { none() }
/**
* Holds if `sink` is a relevant taint sink accepting `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink, DataFlow::FlowState state) { none() }
/** Holds if the node `node` is a taint sanitizer. */
predicate isSanitizer(DataFlow::Node node) { none() }
@@ -79,6 +96,16 @@ abstract class Configuration extends DataFlow::Configuration {
defaultTaintSanitizer(node)
}
/**
* Holds if the node `node` is a taint sanitizer when the flow state is
* `state`.
*/
predicate isSanitizer(DataFlow::Node node, DataFlow::FlowState state) { none() }
final override predicate isBarrier(DataFlow::Node node, DataFlow::FlowState state) {
this.isSanitizer(node, state)
}
/** Holds if taint propagation into `node` is prohibited. */
predicate isSanitizerIn(DataFlow::Node node) { none() }
@@ -93,12 +120,21 @@ abstract class Configuration extends DataFlow::Configuration {
predicate isSanitizerGuard(DataFlow::BarrierGuard guard) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard) {
this.isSanitizerGuard(guard)
this.isSanitizerGuard(guard) or defaultTaintSanitizerGuard(guard)
}
/**
* Holds if the additional taint propagation step from `node1` to `node2`
* must be taken into account in the analysis.
* Holds if taint propagation through nodes guarded by `guard` is prohibited
* when the flow state is `state`.
*/
predicate isSanitizerGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) {
this.isSanitizerGuard(guard, state)
}
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
*/
predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) { none() }
@@ -107,7 +143,25 @@ abstract class Configuration extends DataFlow::Configuration {
defaultAdditionalTaintStep(node1, node2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::Content c) {
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
* This step is only applicable in `state1` and updates the flow state to `state2`.
*/
predicate isAdditionalTaintStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
none()
}
final override predicate isAdditionalFlowStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
this.isAdditionalTaintStep(node1, state1, node2, state2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c) {
(this.isSink(node) or this.isAdditionalTaintStep(node, _)) and
defaultImplicitTaintRead(node, c)
}

68
python/ql/lib/semmle/python/dataflow/new/internal/tainttracking2/TaintTrackingImpl.qll
View File

@@ -61,15 +61,32 @@ abstract class Configuration extends DataFlow::Configuration {
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSource(DataFlow::Node source);
override predicate isSource(DataFlow::Node source) { none() }
/**
* Holds if `sink` is a relevant taint sink.
* Holds if `source` is a relevant taint source with the given initial
* `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSink(DataFlow::Node sink);
override predicate isSource(DataFlow::Node source, DataFlow::FlowState state) { none() }
/**
* Holds if `sink` is a relevant taint sink
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink) { none() }
/**
* Holds if `sink` is a relevant taint sink accepting `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink, DataFlow::FlowState state) { none() }
/** Holds if the node `node` is a taint sanitizer. */
predicate isSanitizer(DataFlow::Node node) { none() }
@@ -79,6 +96,16 @@ abstract class Configuration extends DataFlow::Configuration {
defaultTaintSanitizer(node)
}
/**
* Holds if the node `node` is a taint sanitizer when the flow state is
* `state`.
*/
predicate isSanitizer(DataFlow::Node node, DataFlow::FlowState state) { none() }
final override predicate isBarrier(DataFlow::Node node, DataFlow::FlowState state) {
this.isSanitizer(node, state)
}
/** Holds if taint propagation into `node` is prohibited. */
predicate isSanitizerIn(DataFlow::Node node) { none() }
@@ -93,12 +120,21 @@ abstract class Configuration extends DataFlow::Configuration {
predicate isSanitizerGuard(DataFlow::BarrierGuard guard) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard) {
this.isSanitizerGuard(guard)
this.isSanitizerGuard(guard) or defaultTaintSanitizerGuard(guard)
}
/**
* Holds if the additional taint propagation step from `node1` to `node2`
* must be taken into account in the analysis.
* Holds if taint propagation through nodes guarded by `guard` is prohibited
* when the flow state is `state`.
*/
predicate isSanitizerGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) {
this.isSanitizerGuard(guard, state)
}
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
*/
predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) { none() }
@@ -107,7 +143,25 @@ abstract class Configuration extends DataFlow::Configuration {
defaultAdditionalTaintStep(node1, node2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::Content c) {
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
* This step is only applicable in `state1` and updates the flow state to `state2`.
*/
predicate isAdditionalTaintStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
none()
}
final override predicate isAdditionalFlowStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
this.isAdditionalTaintStep(node1, state1, node2, state2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c) {
(this.isSink(node) or this.isAdditionalTaintStep(node, _)) and
defaultImplicitTaintRead(node, c)
}

68
python/ql/lib/semmle/python/dataflow/new/internal/tainttracking3/TaintTrackingImpl.qll
View File

@@ -61,15 +61,32 @@ abstract class Configuration extends DataFlow::Configuration {
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSource(DataFlow::Node source);
override predicate isSource(DataFlow::Node source) { none() }
/**
* Holds if `sink` is a relevant taint sink.
* Holds if `source` is a relevant taint source with the given initial
* `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSink(DataFlow::Node sink);
override predicate isSource(DataFlow::Node source, DataFlow::FlowState state) { none() }
/**
* Holds if `sink` is a relevant taint sink
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink) { none() }
/**
* Holds if `sink` is a relevant taint sink accepting `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink, DataFlow::FlowState state) { none() }
/** Holds if the node `node` is a taint sanitizer. */
predicate isSanitizer(DataFlow::Node node) { none() }
@@ -79,6 +96,16 @@ abstract class Configuration extends DataFlow::Configuration {
defaultTaintSanitizer(node)
}
/**
* Holds if the node `node` is a taint sanitizer when the flow state is
* `state`.
*/
predicate isSanitizer(DataFlow::Node node, DataFlow::FlowState state) { none() }
final override predicate isBarrier(DataFlow::Node node, DataFlow::FlowState state) {
this.isSanitizer(node, state)
}
/** Holds if taint propagation into `node` is prohibited. */
predicate isSanitizerIn(DataFlow::Node node) { none() }
@@ -93,12 +120,21 @@ abstract class Configuration extends DataFlow::Configuration {
predicate isSanitizerGuard(DataFlow::BarrierGuard guard) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard) {
this.isSanitizerGuard(guard)
this.isSanitizerGuard(guard) or defaultTaintSanitizerGuard(guard)
}
/**
* Holds if the additional taint propagation step from `node1` to `node2`
* must be taken into account in the analysis.
* Holds if taint propagation through nodes guarded by `guard` is prohibited
* when the flow state is `state`.
*/
predicate isSanitizerGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) {
this.isSanitizerGuard(guard, state)
}
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
*/
predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) { none() }
@@ -107,7 +143,25 @@ abstract class Configuration extends DataFlow::Configuration {
defaultAdditionalTaintStep(node1, node2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::Content c) {
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
* This step is only applicable in `state1` and updates the flow state to `state2`.
*/
predicate isAdditionalTaintStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
none()
}
final override predicate isAdditionalFlowStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
this.isAdditionalTaintStep(node1, state1, node2, state2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c) {
(this.isSink(node) or this.isAdditionalTaintStep(node, _)) and
defaultImplicitTaintRead(node, c)
}

68
python/ql/lib/semmle/python/dataflow/new/internal/tainttracking4/TaintTrackingImpl.qll
View File

@@ -61,15 +61,32 @@ abstract class Configuration extends DataFlow::Configuration {
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSource(DataFlow::Node source);
override predicate isSource(DataFlow::Node source) { none() }
/**
* Holds if `sink` is a relevant taint sink.
* Holds if `source` is a relevant taint source with the given initial
* `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
abstract override predicate isSink(DataFlow::Node sink);
override predicate isSource(DataFlow::Node source, DataFlow::FlowState state) { none() }
/**
* Holds if `sink` is a relevant taint sink
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink) { none() }
/**
* Holds if `sink` is a relevant taint sink accepting `state`.
*
* The smaller this predicate is, the faster `hasFlow()` will converge.
*/
// overridden to provide taint-tracking specific qldoc
override predicate isSink(DataFlow::Node sink, DataFlow::FlowState state) { none() }
/** Holds if the node `node` is a taint sanitizer. */
predicate isSanitizer(DataFlow::Node node) { none() }
@@ -79,6 +96,16 @@ abstract class Configuration extends DataFlow::Configuration {
defaultTaintSanitizer(node)
}
/**
* Holds if the node `node` is a taint sanitizer when the flow state is
* `state`.
*/
predicate isSanitizer(DataFlow::Node node, DataFlow::FlowState state) { none() }
final override predicate isBarrier(DataFlow::Node node, DataFlow::FlowState state) {
this.isSanitizer(node, state)
}
/** Holds if taint propagation into `node` is prohibited. */
predicate isSanitizerIn(DataFlow::Node node) { none() }
@@ -93,12 +120,21 @@ abstract class Configuration extends DataFlow::Configuration {
predicate isSanitizerGuard(DataFlow::BarrierGuard guard) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard) {
this.isSanitizerGuard(guard)
this.isSanitizerGuard(guard) or defaultTaintSanitizerGuard(guard)
}
/**
* Holds if the additional taint propagation step from `node1` to `node2`
* must be taken into account in the analysis.
* Holds if taint propagation through nodes guarded by `guard` is prohibited
* when the flow state is `state`.
*/
predicate isSanitizerGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) { none() }
final override predicate isBarrierGuard(DataFlow::BarrierGuard guard, DataFlow::FlowState state) {
this.isSanitizerGuard(guard, state)
}
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
*/
predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) { none() }
@@ -107,7 +143,25 @@ abstract class Configuration extends DataFlow::Configuration {
defaultAdditionalTaintStep(node1, node2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::Content c) {
/**
* Holds if taint may propagate from `node1` to `node2` in addition to the normal data-flow and taint steps.
* This step is only applicable in `state1` and updates the flow state to `state2`.
*/
predicate isAdditionalTaintStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
none()
}
final override predicate isAdditionalFlowStep(
DataFlow::Node node1, DataFlow::FlowState state1, DataFlow::Node node2,
DataFlow::FlowState state2
) {
this.isAdditionalTaintStep(node1, state1, node2, state2)
}
override predicate allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c) {
(this.isSink(node) or this.isAdditionalTaintStep(node, _)) and
defaultImplicitTaintRead(node, c)
}

10
python/ql/lib/semmle/python/dataflow/old/Configuration.qll
View File

@@ -119,16 +119,6 @@ module TaintTracking {
this.(TaintTrackingImplementation).hasFlowPath(src, sink)
}
/* Old query API */
/* deprecated */
deprecated predicate hasFlow(Source src, Sink sink) {
exists(PathSource psrc, PathSink psink |
this.hasFlowPath(psrc, psink) and
src = psrc.getNode().asCfgNode() and
sink = psink.getNode().asCfgNode()
)
}
/* New query API */
predicate hasSimpleFlow(DataFlow::Node src, DataFlow::Node sink) {
exists(PathSource psrc, PathSink psink |

36
python/ql/lib/semmle/python/dataflow/old/Implementation.qll
View File

@@ -79,7 +79,7 @@ abstract class AttributePath extends TAttributePath {
predicate noAttribute() { this = TNoAttribute() }
}
/** AttributePath for no attribute. */
/** The `AttributePath` for no attribute. */
class NoAttribute extends TNoAttribute, AttributePath {
override string toString() { result = "no attribute" }
@@ -88,7 +88,7 @@ class NoAttribute extends TNoAttribute, AttributePath {
override AttributePath fromAttribute(string name) { none() }
}
/** AttributePath for an attribute. */
/** The `AttributePath` for an attribute. */
class NamedAttributePath extends TAttribute, AttributePath {
override string toString() {
exists(string attr |
@@ -124,8 +124,8 @@ newtype TTaintTrackingNode =
}
/**
* Class representing the (node, context, path, kind) tuple.
* Used for context-sensitive path-aware taint-tracking.
* A class representing the (node, context, path, kind) tuple.
* Used for context-sensitive path-aware taint-tracking.
*/
class TaintTrackingNode extends TTaintTrackingNode {
/** Gets a textual representation of this element. */
@@ -503,7 +503,7 @@ class TaintTrackingImplementation extends string {
TaintKind kind, string edgeLabel
) {
exists(PythonFunctionValue init, EssaVariable self, TaintTrackingContext callee |
instantiationCall(node.asCfgNode(), src, init, context, callee) and
this.instantiationCall(node.asCfgNode(), src, init, context, callee) and
this.(EssaTaintTracking).taintedDefinition(_, self.getDefinition(), callee, path, kind) and
self.getSourceVariable().(Variable).isSelf() and
BaseFlow::reaches_exit(self) and
@@ -789,9 +789,9 @@ private class EssaTaintTracking extends string {
TaintTrackingNode src, PyEdgeRefinement defn, TaintTrackingContext context, AttributePath path,
TaintKind kind
) {
taintedPiNodeOneway(src, defn, context, path, kind)
this.taintedPiNodeOneway(src, defn, context, path, kind)
or
taintedPiNodeBothways(src, defn, context, path, kind)
this.taintedPiNodeBothways(src, defn, context, path, kind)
}
pragma[noinline]
@@ -802,7 +802,7 @@ private class EssaTaintTracking extends string {
exists(DataFlow::Node srcnode, ControlFlowNode use |
src = TTaintTrackingNode_(srcnode, context, path, kind, this) and
not this.(TaintTracking::Configuration).isBarrierTest(defn.getTest(), defn.getSense()) and
defn.getSense() = testEvaluates(defn, defn.getTest(), use, src)
defn.getSense() = this.testEvaluates(defn, defn.getTest(), use, src)
)
}
@@ -898,23 +898,7 @@ private class EssaTaintTracking extends string {
)
)
or
result = testEvaluates(defn, not_operand(test), use, src).booleanNot()
}
/**
* Holds if `test` is the test in a branch and `use` is that test
* with all the `not` prefixes removed.
*/
private predicate boolean_filter(ControlFlowNode test, ControlFlowNode use) {
any(PyEdgeRefinement ref).getTest() = test and
(
use = test
or
exists(ControlFlowNode notuse |
boolean_filter(test, notuse) and
use = not_operand(notuse)
)
)
result = this.testEvaluates(defn, not_operand(test), use, src).booleanNot()
}
}
@@ -991,7 +975,7 @@ int iterable_unpacking_descent(SequenceNode left_parent, ControlFlowNode left_de
}
module Implementation {
/* A call that returns a copy (or similar) of the argument */
/** Holds if `tonode` is a call that returns a copy (or similar) of the argument `fromnode` */
predicate copyCall(ControlFlowNode fromnode, CallNode tonode) {
tonode.getFunction().(AttrNode).getObject("copy") = fromnode
or

19
python/ql/lib/semmle/python/dataflow/old/Legacy.qll
View File

@@ -2,24 +2,7 @@ import semmle.python.dataflow.TaintTracking
private import semmle.python.objects.ObjectInternal
import semmle.python.dataflow.Implementation
/* For backwards compatibility -- Use `TaintTrackingContext` instead. */
deprecated class CallContext extends TaintTrackingContext {
TaintTrackingContext getCallee(CallNode call) { result.getCaller(call) = this }
predicate appliesToScope(Scope s) {
exists(PythonFunctionObjectInternal func, TaintKind param, AttributePath path, int n |
this = TParamContext(param, path, n) and
exists(TaintTrackingImplementation impl |
impl.callWithTaintedArgument(_, _, _, func, n, path, param) and
s = func.getScope()
)
)
or
this.isTop()
}
}
/* Backwards compatibility with config-less taint-tracking */
/** A configuration that provides backwards compatibility with config-less taint-tracking */
private class LegacyConfiguration extends TaintTracking::Configuration {
LegacyConfiguration() {
/* A name that won't be accidentally chosen by users */

1
python/ql/lib/semmle/python/dataflow/old/StateTracking.qll
View File

@@ -9,7 +9,6 @@
*/
import python
private import semmle.python.pointsto.Base
private import semmle.python.pointsto.PointsTo
private import semmle.python.pointsto.PointsToContext
private import semmle.python.objects.ObjectInternal

55
python/ql/lib/semmle/python/dataflow/old/TaintTracking.qll
View File

@@ -138,9 +138,6 @@ abstract class TaintKind extends string {
exists(TaintedNode n | n.getTaintKind() = this and n.getCfgNode() = expr)
}
/** DEPRECATED -- Use getType() instead */
deprecated ClassObject getClass() { none() }
/**
* Gets the class of this kind of taint.
* For example, if this were a kind of string taint
@@ -180,7 +177,7 @@ abstract class TaintKind extends string {
}
/**
* Alias of `TaintKind`, so the two types can be used interchangeably.
* An Alias of `TaintKind`, so the two types can be used interchangeably.
*/
class FlowLabel = TaintKind;
@@ -561,7 +558,7 @@ module DataFlowExtension {
ControlFlowNode getACalleeSuccessorNode(CallNode call) { none() }
}
/** Data flow variable that modifies the basic data-flow. */
/** A data flow variable that modifies the basic data-flow. */
class DataFlowVariable extends EssaVariable {
/**
* Gets a successor node for data-flow.
@@ -608,51 +605,11 @@ private import semmle.python.pointsto.PointsTo
*/
module DataFlow {
/**
* Generic taint kind, source and sink classes for convenience and
* The generic taint kind, source and sink classes for convenience and
* compatibility with other language libraries
*/
class Extension = DataFlowExtension::DataFlowNode;
abstract deprecated class Configuration extends string {
bindingset[this]
Configuration() { this = this }
abstract predicate isSource(ControlFlowNode source);
abstract predicate isSink(ControlFlowNode sink);
private predicate hasFlowPath(TaintedNode source, TaintedNode sink) {
source.getConfiguration() = this and
this.isSource(source.getCfgNode()) and
this.isSink(sink.getCfgNode()) and
source.flowsTo(sink)
}
predicate hasFlow(ControlFlowNode source, ControlFlowNode sink) {
exists(TaintedNode psource, TaintedNode psink |
psource.getCfgNode() = source and
psink.getCfgNode() = sink and
this.isSource(source) and
this.isSink(sink) and
this.hasFlowPath(psource, psink)
)
}
}
deprecated private class ConfigurationAdapter extends TaintTracking::Configuration {
ConfigurationAdapter() { this instanceof Configuration }
override predicate isSource(DataFlow::Node node, TaintKind kind) {
this.(Configuration).isSource(node.asCfgNode()) and
kind instanceof DataFlowType
}
override predicate isSink(DataFlow::Node node, TaintKind kind) {
this.(Configuration).isSink(node.asCfgNode()) and
kind instanceof DataFlowType
}
}
private newtype TDataFlowNode =
TEssaNode(EssaVariable var) or
TCfgNode(ControlFlowNode node)
@@ -672,9 +629,6 @@ module DataFlow {
abstract Location getLocation();
AstNode asAstNode() { result = this.asCfgNode().getNode() }
/** For backwards compatibility -- Use asAstNode() instead */
deprecated AstNode getNode() { result = this.asAstNode() }
}
class CfgNode extends Node, TCfgNode {
@@ -711,9 +665,10 @@ module DataFlow {
}
deprecated private class DataFlowType extends TaintKind {
// this only exists to avoid an empty recursion error in the type checker
DataFlowType() {
this = "Data flow" and
exists(DataFlow::Configuration c)
1 = 2
}
}

4
python/ql/lib/semmle/python/dependencies/Dependencies.qll
View File

@@ -158,13 +158,13 @@ private predicate defn_of_instance_attribute(Assign asgn, Class c, string name)
)
}
/* Whether asgn defines an attribute of a class */
/** Holds if asgn defines an attribute of a class */
private predicate defn_of_class_attribute(Assign asgn, Class c, string name) {
asgn.getScope() = c and
asgn.getATarget().(Name).getId() = name
}
/* Holds if `value` is a value assigned to the `name`d attribute of module `m`. */
/** Holds if `value` is a value assigned to the `name`d attribute of module `m`. */
private predicate defn_of_module_attribute(ControlFlowNode value, Module m, string name) {
exists(DefinitionNode def |
def.getScope() = m and

8
python/ql/lib/semmle/python/dependencies/TechInventory.qll
View File

@@ -14,16 +14,14 @@ string munge(File sourceFile, ExternalPackage package) {
result = "/" + sourceFile.getRelativePath() + "<|>" + package.getName() + "<|>unknown"
}
abstract class ExternalPackage extends Object {
ExternalPackage() { this instanceof ModuleObject }
abstract class ExternalPackage extends Object instanceof ModuleObject {
abstract string getName();
abstract string getVersion();
Object getAttribute(string name) { result = this.(ModuleObject).attr(name) }
Object getAttribute(string name) { result = super.attr(name) }
PackageObject getPackage() { result = this.(ModuleObject).getPackage() }
PackageObject getPackage() { result = super.getPackage() }
}
bindingset[text]

29
python/ql/lib/semmle/python/essa/Definitions.qll
View File

@@ -70,6 +70,10 @@ abstract class SsaSourceVariable extends @py_variable {
SsaSource::exception_capture(this, def)
or
SsaSource::with_definition(this, def)
or
SsaSource::pattern_capture_definition(this, def)
or
SsaSource::pattern_alias_definition(this, def)
}
/**
@@ -152,17 +156,17 @@ class NonLocalVariable extends SsaSourceVariable {
}
override ControlFlowNode getAnImplicitUse() {
result.(CallNode).getScope().getScope*() = this.(LocalVariable).getScope()
result.(CallNode).getScope().getScope*() = this.scope_as_local_variable()
}
override ControlFlowNode getScopeEntryDefinition() {
exists(Function f |
f.getScope+() = this.(LocalVariable).getScope() and
f.getScope+() = this.scope_as_local_variable() and
f.getEntryNode() = result
)
or
not this.(LocalVariable).isParameter() and
this.(LocalVariable).getScope().getEntryNode() = result
this.scope_as_local_variable().getEntryNode() = result
}
pragma[noinline]
@@ -215,13 +219,16 @@ class ModuleVariable extends SsaSourceVariable {
)
}
pragma[nomagic]
private Scope scope_as_global_variable() { result = this.(GlobalVariable).getScope() }
pragma[noinline]
CallNode global_variable_callnode() { result.getScope() = this.(GlobalVariable).getScope() }
CallNode global_variable_callnode() { result.getScope() = this.scope_as_global_variable() }
pragma[noinline]
ImportMemberNode global_variable_import() {
result.getScope() = this.(GlobalVariable).getScope() and
import_from_dot_in_init(result.(ImportMemberNode).getModule(this.getName()))
result.getScope() = this.scope_as_global_variable() and
import_from_dot_in_init(result.getModule(this.getName()))
}
override ControlFlowNode getAnImplicitUse() {
@@ -250,7 +257,7 @@ class ModuleVariable extends SsaSourceVariable {
override ControlFlowNode getScopeEntryDefinition() {
exists(Scope s | s.getEntryNode() = result |
/* Module entry point */
this.(GlobalVariable).getScope() = s
this.scope_as_global_variable() = s
or
/* For implicit use of __metaclass__ when constructing class */
class_with_global_metaclass(s, this)
@@ -286,13 +293,13 @@ class EscapingGlobalVariable extends ModuleVariable {
override ControlFlowNode getAnImplicitUse() {
result = ModuleVariable.super.getAnImplicitUse()
or
result.(CallNode).getScope().getScope+() = this.(GlobalVariable).getScope()
result.(CallNode).getScope().getScope+() = this.scope_as_global_variable()
or
result = this.innerScope().getANormalExit()
}
private Scope innerScope() {
result.getScope+() = this.(GlobalVariable).getScope() and
result.getScope+() = this.scope_as_global_variable() and
not result instanceof ImportTimeScope
}
@@ -306,7 +313,7 @@ class EscapingGlobalVariable extends ModuleVariable {
Scope scope_as_global_variable() { result = this.(GlobalVariable).getScope() }
override CallNode redefinedAtCallSite() {
result.(CallNode).getScope().getScope*() = this.scope_as_global_variable()
result.getScope().getScope*() = this.scope_as_global_variable()
}
}
@@ -332,7 +339,7 @@ class SpecialSsaSourceVariable extends SsaSourceVariable {
Scope scope_as_global_variable() { result = this.(GlobalVariable).getScope() }
override CallNode redefinedAtCallSite() {
result.(CallNode).getScope().getScope*() = this.scope_as_global_variable()
result.getScope().getScope*() = this.scope_as_global_variable()
}
}

120
python/ql/lib/semmle/python/essa/Essa.qll
View File

@@ -5,10 +5,11 @@
import python
private import SsaCompute
import semmle.python.essa.Definitions
private import semmle.python.internal.CachedStages
/** An (enhanced) SSA variable derived from `SsaSourceVariable`. */
class EssaVariable extends TEssaDefinition {
/** Gets the (unique) definition of this variable. */
/** Gets the (unique) definition of this variable. */
EssaDefinition getDefinition() { this = result }
/**
@@ -134,7 +135,7 @@ private newtype TEssaDefinition =
TPhiFunction(SsaSourceVariable v, BasicBlock b) { EssaDefinitions::phiNode(v, b) }
/**
* Definition of an extended-SSA (ESSA) variable.
* A definition of an extended-SSA (ESSA) variable.
* There is exactly one definition for each variable,
* and exactly one variable for each definition.
*/
@@ -171,6 +172,9 @@ abstract class EssaDefinition extends TEssaDefinition {
EssaVariable getVariable() { result.getDefinition() = this }
abstract BasicBlock getBasicBlock();
/** Gets the name of the primary QL class for this element. */
string getAPrimaryQlClass() { result = "EssaDefinition" }
}
/**
@@ -216,13 +220,15 @@ class EssaEdgeRefinement extends EssaDefinition, TEssaEdgeDefinition {
}
override string getRepresentation() {
result = this.getAQlClass() + "(" + this.getInput().getRepresentation() + ")"
result = this.getAPrimaryQlClass() + "(" + this.getInput().getRepresentation() + ")"
}
/** Gets the scope of the variable defined by this definition. */
override Scope getScope() { result = this.getPredecessor().getScope() }
override BasicBlock getBasicBlock() { result = this.getSuccessor() }
override string getAPrimaryQlClass() { result = "EssaEdgeRefinement" }
}
/** A Phi-function as specified in classic SSA form. */
@@ -265,6 +271,7 @@ class PhiFunction extends EssaDefinition, TPhiFunction {
/** Gets the input variable for this phi node on the edge `pred` -> `this.getBasicBlock()`, if any. */
cached
EssaVariable getInput(BasicBlock pred) {
Stages::AST::ref() and
result.getDefinition() = this.reachingDefinition(pred)
or
result.getDefinition() = this.inputEdgeRefinement(pred)
@@ -366,6 +373,8 @@ class PhiFunction extends EssaDefinition, TPhiFunction {
)
)
}
override string getAPrimaryQlClass() { result = "PhiFunction" }
}
/**
@@ -396,7 +405,7 @@ class EssaNodeDefinition extends EssaDefinition, TEssaNodeDefinition {
override Location getLocation() { result = this.getDefiningNode().getLocation() }
override string getRepresentation() { result = this.getAQlClass() }
override string getRepresentation() { result = this.getAPrimaryQlClass() }
override Scope getScope() {
exists(BasicBlock defb |
@@ -414,6 +423,8 @@ class EssaNodeDefinition extends EssaDefinition, TEssaNodeDefinition {
}
override BasicBlock getBasicBlock() { result = this.getDefiningNode().getBasicBlock() }
override string getAPrimaryQlClass() { result = "EssaNodeDefinition" }
}
/** A definition of an ESSA variable that takes another ESSA variable as an input. */
@@ -448,10 +459,10 @@ class EssaNodeRefinement extends EssaDefinition, TEssaNodeRefinement {
override Location getLocation() { result = this.getDefiningNode().getLocation() }
override string getRepresentation() {
result = this.getAQlClass() + "(" + this.getInput().getRepresentation() + ")"
result = this.getAPrimaryQlClass() + "(" + this.getInput().getRepresentation() + ")"
or
not exists(this.getInput()) and
result = this.getAQlClass() + "(" + this.getSourceVariable().getName() + "??)"
result = this.getAPrimaryQlClass() + "(" + this.getSourceVariable().getName() + "??)"
}
override Scope getScope() {
@@ -470,6 +481,8 @@ class EssaNodeRefinement extends EssaDefinition, TEssaNodeRefinement {
}
override BasicBlock getBasicBlock() { result = this.getDefiningNode().getBasicBlock() }
override string getAPrimaryQlClass() { result = "EssaNodeRefinement" }
}
pragma[noopt]
@@ -483,26 +496,22 @@ private EssaVariable potential_input(EssaNodeRefinement ref) {
)
}
/* For backwards compatibility */
deprecated class PyNodeDefinition = EssaNodeDefinition;
/* For backwards compatibility */
deprecated class PyNodeRefinement = EssaNodeRefinement;
/** An assignment to a variable `v = val` */
class AssignmentDefinition extends EssaNodeDefinition {
ControlFlowNode value;
AssignmentDefinition() {
SsaSource::assignment_definition(this.getSourceVariable(), this.getDefiningNode(), _)
SsaSource::assignment_definition(this.getSourceVariable(), this.getDefiningNode(), value)
}
ControlFlowNode getValue() {
SsaSource::assignment_definition(this.getSourceVariable(), this.getDefiningNode(), result)
}
ControlFlowNode getValue() { result = value }
override string getRepresentation() { result = this.getValue().getNode().toString() }
override string getAPrimaryQlClass() { result = "AssignmentDefinition" }
}
/** Capture of a raised exception `except ExceptionType ex:` */
/** A capture of a raised exception `except ExceptionType ex:` */
class ExceptionCapture extends EssaNodeDefinition {
ExceptionCapture() {
SsaSource::exception_capture(this.getSourceVariable(), this.getDefiningNode())
@@ -516,6 +525,8 @@ class ExceptionCapture extends EssaNodeDefinition {
}
override string getRepresentation() { result = "except " + this.getSourceVariable().getName() }
override string getAPrimaryQlClass() { result = "ExceptionCapture" }
}
/** An assignment to a variable as part of a multiple assignment `..., v, ... = val` */
@@ -536,6 +547,8 @@ class MultiAssignmentDefinition extends EssaNodeDefinition {
SsaSource::multi_assignment_definition(this.getSourceVariable(), this.getDefiningNode(), index,
lhs)
}
override string getAPrimaryQlClass() { result = "MultiAssignmentDefinition" }
}
/** A definition of a variable in a `with` statement */
@@ -543,6 +556,30 @@ class WithDefinition extends EssaNodeDefinition {
WithDefinition() { SsaSource::with_definition(this.getSourceVariable(), this.getDefiningNode()) }
override string getRepresentation() { result = "with" }
override string getAPrimaryQlClass() { result = "WithDefinition" }
}
/** A definition of a variable via a capture pattern */
class PatternCaptureDefinition extends EssaNodeDefinition {
PatternCaptureDefinition() {
SsaSource::pattern_capture_definition(this.getSourceVariable(), this.getDefiningNode())
}
override string getRepresentation() { result = "pattern capture" }
override string getAPrimaryQlClass() { result = "PatternCaptureDefinition" }
}
/** A definition of a variable via a pattern alias */
class PatternAliasDefinition extends EssaNodeDefinition {
PatternAliasDefinition() {
SsaSource::pattern_alias_definition(this.getSourceVariable(), this.getDefiningNode())
}
override string getRepresentation() { result = "pattern alias" }
override string getAPrimaryQlClass() { result = "PatternAliasDefinition" }
}
/** A definition of a variable by declaring it as a parameter */
@@ -576,6 +613,8 @@ class ParameterDefinition extends EssaNodeDefinition {
/** Gets the `Parameter` this `ParameterDefinition` represents. */
Parameter getParameter() { result = this.getDefiningNode().getNode() }
override string getAPrimaryQlClass() { result = "ParameterDefinition" }
}
/** A deletion of a variable `del v` */
@@ -583,10 +622,12 @@ class DeletionDefinition extends EssaNodeDefinition {
DeletionDefinition() {
SsaSource::deletion_definition(this.getSourceVariable(), this.getDefiningNode())
}
override string getAPrimaryQlClass() { result = "DeletionDefinition" }
}
/**
* Definition of variable at the entry of a scope. Usually this represents the transfer of
* A definition of variable at the entry of a scope. Usually this represents the transfer of
* a global or non-local variable from one scope to another.
*/
class ScopeEntryDefinition extends EssaNodeDefinition {
@@ -596,16 +637,20 @@ class ScopeEntryDefinition extends EssaNodeDefinition {
}
override Scope getScope() { result.getEntryNode() = this.getDefiningNode() }
override string getAPrimaryQlClass() { result = "ScopeEntryDefinition" }
}
/** Possible redefinition of variable via `from ... import *` */
/** A possible redefinition of variable via `from ... import *` */
class ImportStarRefinement extends EssaNodeRefinement {
ImportStarRefinement() {
SsaSource::import_star_refinement(this.getSourceVariable(), _, this.getDefiningNode())
}
override string getAPrimaryQlClass() { result = "ImportStarRefinement" }
}
/** Assignment of an attribute `obj.attr = val` */
/** An assignment of an attribute `obj.attr = val` */
class AttributeAssignment extends EssaNodeRefinement {
AttributeAssignment() {
SsaSource::attribute_assignment_refinement(this.getSourceVariable(), _, this.getDefiningNode())
@@ -617,12 +662,16 @@ class AttributeAssignment extends EssaNodeRefinement {
override string getRepresentation() {
result =
this.getAQlClass() + " '" + this.getName() + "'(" + this.getInput().getRepresentation() + ")"
this.getAPrimaryQlClass() + " '" + this.getName() + "'(" + this.getInput().getRepresentation()
+ ")"
or
not exists(this.getInput()) and
result =
this.getAQlClass() + " '" + this.getName() + "'(" + this.getSourceVariable().getName() + "??)"
this.getAPrimaryQlClass() + " '" + this.getName() + "'(" + this.getSourceVariable().getName() +
"??)"
}
override string getAPrimaryQlClass() { result = "AttributeAssignment" }
}
/** A use of a variable as an argument, `foo(v)`, which might modify the object referred to. */
@@ -636,15 +685,19 @@ class ArgumentRefinement extends EssaNodeRefinement {
ControlFlowNode getArgument() { result = argument }
CallNode getCall() { result = this.getDefiningNode() }
override string getAPrimaryQlClass() { result = "ArgumentRefinement" }
}
/** Deletion of an attribute `del obj.attr`. */
/** A deletion of an attribute `del obj.attr`. */
class EssaAttributeDeletion extends EssaNodeRefinement {
EssaAttributeDeletion() {
SsaSource::attribute_deletion_refinement(this.getSourceVariable(), _, this.getDefiningNode())
}
string getName() { result = this.getDefiningNode().(AttrNode).getName() }
override string getAPrimaryQlClass() { result = "EssaAttributeDeletion" }
}
/** A pi-node (guard) with only one successor. */
@@ -672,10 +725,12 @@ class SingleSuccessorGuard extends EssaNodeRefinement {
test = this.getDefiningNode() and
SsaSource::test_refinement(this.getSourceVariable(), use, test)
}
override string getAPrimaryQlClass() { result = "SingleSuccessorGuard" }
}
/**
* Implicit definition of the names of sub-modules in a package.
* An implicit definition of the names of sub-modules in a package.
* Although the interpreter does not pre-define these names, merely populating them
* as they are imported, this is a good approximation for static analysis.
*/
@@ -683,11 +738,13 @@ class ImplicitSubModuleDefinition extends EssaNodeDefinition {
ImplicitSubModuleDefinition() {
SsaSource::init_module_submodule_defn(this.getSourceVariable(), this.getDefiningNode())
}
override string getAPrimaryQlClass() { result = "ImplicitSubModuleDefinition" }
}
/** An implicit (possible) definition of an escaping variable at a call-site */
class CallsiteRefinement extends EssaNodeRefinement {
override string toString() { result = "CallsiteRefinement" }
override string toString() { result = "CallSiteRefinement" }
CallsiteRefinement() {
exists(SsaSourceVariable var, ControlFlowNode defn |
@@ -700,24 +757,31 @@ class CallsiteRefinement extends EssaNodeRefinement {
}
CallNode getCall() { this.getDefiningNode() = result }
override string getAPrimaryQlClass() { result = "CallsiteRefinement" }
}
/** An implicit (possible) modification of the object referred at a method call */
class MethodCallsiteRefinement extends EssaNodeRefinement {
MethodCallsiteRefinement() {
SsaSource::method_call_refinement(this.getSourceVariable(), _, this.getDefiningNode()) and
SsaSource::method_call_refinement(pragma[only_bind_into](this.getSourceVariable()), _,
this.getDefiningNode()) and
not this instanceof SingleSuccessorGuard
}
CallNode getCall() { this.getDefiningNode() = result }
override string getAPrimaryQlClass() { result = "MethodCallsiteRefinement" }
}
/** An implicit (possible) modification of `self` at a method call */
class SelfCallsiteRefinement extends MethodCallsiteRefinement {
SelfCallsiteRefinement() { this.getSourceVariable().(Variable).isSelf() }
override string getAPrimaryQlClass() { result = "SelfCallsiteRefinement" }
}
/** Python specific sub-class of generic EssaEdgeRefinement */
/** A Python specific sub-class of generic EssaEdgeRefinement */
class PyEdgeRefinement extends EssaEdgeRefinement {
override string getRepresentation() {
/*
@@ -732,4 +796,6 @@ class PyEdgeRefinement extends EssaEdgeRefinement {
}
ControlFlowNode getTest() { result = this.getPredecessor().getLastNode() }
override string getAPrimaryQlClass() { result = "PyEdgeRefinement" }
}

39
python/ql/lib/semmle/python/essa/SsaCompute.qll
View File

@@ -90,6 +90,7 @@
*/
import python
private import semmle.python.internal.CachedStages
cached
private module SsaComputeImpl {
@@ -139,7 +140,7 @@ private module SsaComputeImpl {
Liveness::liveAtEntry(v, succ)
}
/** A phi node for `v` at the beginning of basic block `b`. */
/** Holds if there is a phi node for `v` at the beginning of basic block `b`. */
cached
predicate phiNode(SsaSourceVariable v, BasicBlock b) {
(
@@ -175,8 +176,8 @@ private module SsaComputeImpl {
}
/**
* A ranking of the indices `i` at which there is an SSA definition or use of
* `v` in the basic block `b`.
* Holds if the `rankix`th definition or use of the SSA variable `v` in the basic block `b` occurs
* at index `i`.
*
* Basic block indices are translated to rank indices in order to skip
* irrelevant indices at which there is no definition or use when traversing
@@ -187,14 +188,14 @@ private module SsaComputeImpl {
i = rank[rankix](int j | variableDef(v, _, b, j) or variableUse(v, _, b, j))
}
/** A definition of a variable occurring at the specified rank index in basic block `b`. */
/** Holds if there is a definition of a variable occurring at the specified rank index in basic block `b`. */
cached
predicate defRank(SsaSourceVariable v, BasicBlock b, int rankix, int i) {
variableDef(v, _, b, i) and
defUseRank(v, b, rankix, i)
}
/** A variable access `use` of `v` in `b` at index `i`. */
/** Holds if there is a variable access `use` of `v` in `b` at index `i`. */
cached
predicate variableUse(SsaSourceVariable v, ControlFlowNode use, BasicBlock b, int i) {
(v.getAUse() = use or v.hasRefinement(use, _)) and
@@ -205,7 +206,7 @@ private module SsaComputeImpl {
}
/**
* A definition of an SSA variable occurring at the specified position.
* Holds if there is a definition of an SSA variable occurring at the specified position.
* This is either a phi node, a `VariableUpdate`, or a parameter.
*/
cached
@@ -227,7 +228,7 @@ private module SsaComputeImpl {
* dominance.
*/
/** The maximum rank index for the given variable and basic block. */
/** Gets the maximum rank index for the given variable and basic block. */
cached
int lastRank(SsaSourceVariable v, BasicBlock b) {
result = max(int rankix | defUseRank(v, b, rankix, _))
@@ -253,7 +254,7 @@ private module SsaComputeImpl {
i = piIndex()
}
/** The SSA definition reaches the rank index `rankix` in its own basic block `b`. */
/** Holds if the SSA definition reaches the rank index `rankix` in its own basic block `b`. */
cached
predicate ssaDefReachesRank(SsaSourceVariable v, BasicBlock b, int i, int rankix) {
ssaDefRank(v, b, rankix, i)
@@ -264,7 +265,7 @@ private module SsaComputeImpl {
}
/**
* The SSA definition of `v` at `def` reaches `use` in the same basic block
* Holds if the SSA definition of `v` at `def` reaches `use` in the same basic block
* without crossing another SSA definition of `v`.
*/
cached
@@ -303,11 +304,12 @@ private module SsaComputeImpl {
}
/**
* The SSA definition of `v` at `def` reaches the end of a basic block `b`, at
* Holds if the SSA definition of `v` at `def` reaches the end of a basic block `b`, at
* which point it is still live, without crossing another SSA definition of `v`.
*/
cached
predicate reachesEndOfBlock(SsaSourceVariable v, BasicBlock defbb, int defindex, BasicBlock b) {
Stages::AST::ref() and
Liveness::liveAtExit(v, b) and
(
defbb = b and
@@ -320,7 +322,7 @@ private module SsaComputeImpl {
}
/**
* The SSA definition of `v` at `(defbb, defindex)` reaches `use` without crossing another
* Holds if the SSA definition of `v` at `(defbb, defindex)` reaches `use` without crossing another
* SSA definition of `v`.
*/
cached
@@ -360,7 +362,7 @@ private module SsaComputeImpl {
i = rank[rankix](int j | variableDefine(v, _, b, j) or variableSourceUse(v, _, b, j))
}
/** A variable access `use` of `v` in `b` at index `i`. */
/** Holds if there is a variable access `use` of `v` in `b` at index `i`. */
cached
predicate variableSourceUse(SsaSourceVariable v, ControlFlowNode use, BasicBlock b, int i) {
v.getASourceUse() = use and
@@ -478,12 +480,11 @@ private module SsaComputeImpl {
predicate adjacentUseUse(ControlFlowNode use1, ControlFlowNode use2) {
adjacentUseUseSameVar(use1, use2)
or
exists(SsaSourceVariable v, EssaDefinition def, BasicBlock b1, int i1, BasicBlock b2, int i2 |
exists(SsaSourceVariable v, PhiFunction def, BasicBlock b1, int i1, BasicBlock b2, int i2 |
adjacentVarRefs(v, b1, i1, b2, i2) and
variableUse(v, use1, b1, i1) and
definesAt(def, v, b2, i2) and
firstUse(def, use2) and
def instanceof PhiFunction
variableUse(pragma[only_bind_into](v), use1, b1, i1) and
definesAt(def, pragma[only_bind_into](v), b2, i2) and
firstUse(def, use2)
)
}
@@ -495,8 +496,8 @@ private module SsaComputeImpl {
predicate firstUse(EssaDefinition def, ControlFlowNode use) {
exists(SsaSourceVariable v, BasicBlock b1, int i1, BasicBlock b2, int i2 |
adjacentVarRefs(v, b1, i1, b2, i2) and
definesAt(def, v, b1, i1) and
variableSourceUse(v, use, b2, i2)
definesAt(def, pragma[only_bind_into](v), b1, i1) and
variableSourceUse(pragma[only_bind_into](v), use, b2, i2)
)
or
exists(

27
python/ql/lib/semmle/python/essa/SsaDefinitions.qll
View File

@@ -4,13 +4,14 @@
*/
import python
private import semmle.python.pointsto.Base
private import semmle.python.internal.CachedStages
cached
module SsaSource {
/** Holds if `v` is used as the receiver in a method call. */
cached
predicate method_call_refinement(Variable v, ControlFlowNode use, CallNode call) {
Stages::AST::ref() and
use = v.getAUse() and
call.getFunction().(AttrNode).getObject() = use and
not test_contains(_, call)
@@ -40,6 +41,28 @@ module SsaSource {
)
}
/** Holds if `v` is defined by a capture pattern. */
cached
predicate pattern_capture_definition(Variable v, ControlFlowNode defn) {
exists(MatchCapturePattern capture, Name var |
capture.getVariable() = var and
var.getAFlowNode() = defn
|
var = v.getAStore()
)
}
/** Holds if `v` is defined by as the alias of an as-pattern. */
cached
predicate pattern_alias_definition(Variable v, ControlFlowNode defn) {
exists(MatchAsPattern pattern, Name var |
pattern.getAlias() = var and
var.getAFlowNode() = defn
|
var = v.getAStore()
)
}
/** Holds if `v` is defined by multiple assignment at `defn`. */
cached
predicate multi_assignment_definition(Variable v, ControlFlowNode defn, int n, SequenceNode lhs) {
@@ -127,7 +150,7 @@ module SsaSource {
not test_contains(_, call)
}
/** Holds if an attribute is deleted at `def` and `use` is the use of `v` for that deletion */
/** Holds if an attribute is deleted at `def` and `use` is the use of `v` for that deletion */
cached
predicate attribute_deletion_refinement(Variable v, NameNode use, DeletionNode def) {
use.uses(v) and

34
python/ql/lib/semmle/python/filters/GeneratedCode.qll
View File

@@ -14,7 +14,7 @@ abstract class GeneratedFile extends File {
* There is no formal reason for the above, it just seems to work well in practice.
*/
library class GenericGeneratedFile extends GeneratedFile {
class GenericGeneratedFile extends GeneratedFile {
GenericGeneratedFile() {
not this instanceof SpecificGeneratedFile and
(
@@ -32,18 +32,26 @@ library class GenericGeneratedFile extends GeneratedFile {
override string getTool() { lax_generated_by(this, result) or strict_generated_by(this, result) }
}
pragma[nomagic]
private int minStmtLine(File file) {
result =
min(int line |
line = any(Stmt s | s.getLocation().getFile() = file).getLocation().getStartLine()
)
}
pragma[nomagic]
private predicate isCommentAfterCode(Comment c, File f) {
f = c.getLocation().getFile() and
minStmtLine(f) < c.getLocation().getStartLine()
}
private string comment_or_docstring(File f, boolean before_code) {
exists(Comment c |
c.getLocation().getFile() = f and
result = c.getText()
|
if
exists(Stmt s |
s.getEnclosingModule().getFile() = f and
s.getLocation().getStartLine() < c.getLocation().getStartLine()
)
then before_code = false
else before_code = true
if isCommentAfterCode(c, f) then before_code = false else before_code = true
)
or
exists(Module m | m.getFile() = f |
@@ -103,7 +111,7 @@ private predicate auto_generated(File f) {
/**
* A file generated by a template engine
*/
abstract library class SpecificGeneratedFile extends GeneratedFile {
abstract class SpecificGeneratedFile extends GeneratedFile {
/*
* Currently cover Spitfire, Pyxl and Mako.
* Django templates are not compiled to Python.
@@ -112,7 +120,7 @@ abstract library class SpecificGeneratedFile extends GeneratedFile {
}
/** File generated by the spitfire templating engine */
/** A file generated by the spitfire templating engine */
class SpitfireGeneratedFile extends SpecificGeneratedFile {
SpitfireGeneratedFile() {
exists(Module m | m.getFile() = this and not m instanceof SpitfireTemplate |
@@ -127,14 +135,14 @@ class SpitfireGeneratedFile extends SpecificGeneratedFile {
override string getTool() { result = "spitfire" }
}
/** File generated by the pyxl templating engine */
/** A file generated by the pyxl templating engine */
class PyxlGeneratedFile extends SpecificGeneratedFile {
PyxlGeneratedFile() { this.getSpecifiedEncoding() = "pyxl" }
override string getTool() { result = "pyxl" }
}
/** File generated by the mako templating engine */
/** A file generated by the mako templating engine */
class MakoGeneratedFile extends SpecificGeneratedFile {
MakoGeneratedFile() {
exists(Module m | m.getFile() = this |
@@ -166,7 +174,7 @@ string from_mako_import(Module m) {
)
}
/** File generated by Google's protobuf tool. */
/** A file generated by Google's protobuf tool. */
class ProtobufGeneratedFile extends SpecificGeneratedFile {
ProtobufGeneratedFile() {
this.getAbsolutePath().regexpMatch(".*_pb2?.py") and

4
python/ql/lib/semmle/python/filters/Tests.qll
View File

@@ -15,7 +15,7 @@ class UnitTestClass extends TestScope {
abstract class Test extends TestScope { }
/** Class of test function that uses the `unittest` framework */
/** A test function that uses the `unittest` framework */
class UnitTestFunction extends Test {
UnitTestFunction() {
this.getScope+() instanceof UnitTestClass and
@@ -37,7 +37,7 @@ class NoseTestFunction extends Test {
}
}
/** Class of functions that are clearly tests, but don't belong to a specific framework */
/** A function that is clearly a test, but doesn't belong to a specific framework */
class UnknownTestFunction extends Test {
UnknownTestFunction() {
this.(Function).getName().matches("test%") and

58
python/ql/lib/semmle/python/frameworks/Aiohttp.qll
View File

@@ -243,7 +243,9 @@ module AiohttpWebModel {
/** A class that has a super-type which is an aiohttp.web View class. */
class AiohttpViewClassFromSuperClass extends AiohttpViewClass {
AiohttpViewClassFromSuperClass() { this.getABase() = View::subclassRef().getAUse().asExpr() }
AiohttpViewClassFromSuperClass() {
this.getParent() = View::subclassRef().getAnImmediateUse().asExpr()
}
}
/** A class that is used in a route-setup, therefore being considered an aiohttp.web View class. */
@@ -330,8 +332,8 @@ module AiohttpWebModel {
exists(Await await, DataFlow::CallCfgNode call, DataFlow::AttrRead read |
this.asExpr() = await
|
read.(DataFlow::AttrRead).getObject() = Request::instance() and
read.(DataFlow::AttrRead).getAttributeName() = "post" and
read.getObject() = Request::instance() and
read.getAttributeName() = "post" and
call.getFunction() = read and
await.getValue() = call.asExpr()
)
@@ -639,3 +641,53 @@ module AiohttpWebModel {
override DataFlow::Node getValueArg() { result = value }
}
}
/**
* Provides models for the web server part (`aiohttp.client`) of the `aiohttp` PyPI package.
* See https://docs.aiohttp.org/en/stable/client.html
*/
private module AiohttpClientModel {
/**
* Provides models for the `aiohttp.ClientSession` class
*
* See https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.
*/
module ClientSession {
/** Gets a reference to the `aiohttp.ClientSession` class. */
private API::Node classRef() {
result = API::moduleImport("aiohttp").getMember("ClientSession")
}
/** Gets a reference to an instance of `aiohttp.ClientSession`. */
private API::Node instance() { result = classRef().getReturn() }
/** A method call on a ClientSession that sends off a request */
private class OutgoingRequestCall extends HTTP::Client::Request::Range, DataFlow::CallCfgNode {
string methodName;
OutgoingRequestCall() {
methodName in [HTTP::httpVerbLower(), "request"] and
this = instance().getMember(methodName).getACall()
}
override DataFlow::Node getAUrlPart() {
result = this.getArgByName("url")
or
not methodName = "request" and
result = this.getArg(0)
or
methodName = "request" and
result = this.getArg(1)
}
override string getFramework() { result = "aiohttp.ClientSession" }
override predicate disablesCertificateValidation(
DataFlow::Node disablingNode, DataFlow::Node argumentOrigin
) {
// TODO: Look into disabling certificate validation
none()
}
}
}
}

111
python/ql/lib/semmle/python/frameworks/Aiomysql.qll Normal file
View File

@@ -0,0 +1,111 @@
/**
* Provides classes modeling security-relevant aspects of the `aiomysql` PyPI package.
* See
* - https://aiomysql.readthedocs.io/en/stable/index.html
* - https://pypi.org/project/aiomysql/
*/
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
/** Provides models for the `aiomysql` PyPI package. */
private module Aiomysql {
private import semmle.python.internal.Awaited
/**
* Gets a `ConnectionPool` that is created when the result of `aiomysql.create_pool()` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/pool.html
*/
API::Node connectionPool() {
result = API::moduleImport("aiomysql").getMember("create_pool").getReturn().getAwaited()
}
/**
* Gets a `Connection` that is created when
* - the result of `aiomysql.connect()` is awaited.
* - the result of calling `aquire` on a `ConnectionPool` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/connection.html#connection
*/
API::Node connection() {
result = API::moduleImport("aiomysql").getMember("connect").getReturn().getAwaited()
or
result = connectionPool().getMember("acquire").getReturn().getAwaited()
}
/**
* Gets a `Cursor` that is created when
* - the result of calling `cursor` on a `ConnectionPool` is awaited.
* - the result of calling `cursor` on a `Connection` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/cursors.html
*/
API::Node cursor() {
result = connectionPool().getMember("cursor").getReturn().getAwaited()
or
result = connection().getMember("cursor").getReturn().getAwaited()
}
/**
* A query. Calling `execute` on a `Cursor` constructs a query.
* See https://aiomysql.readthedocs.io/en/stable/cursors.html#Cursor.execute
*/
class CursorExecuteCall extends SqlConstruction::Range, API::CallNode {
CursorExecuteCall() { this = cursor().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "operation").getARhs() }
}
/**
* An awaited query. Awaiting the result of calling `execute` executes the query.
* See https://aiomysql.readthedocs.io/en/stable/cursors.html#Cursor.execute
*/
class AwaitedCursorExecuteCall extends SqlExecution::Range {
CursorExecuteCall executeCall;
AwaitedCursorExecuteCall() { this = executeCall.getReturn().getAwaited().getAnImmediateUse() }
override DataFlow::Node getSql() { result = executeCall.getSql() }
}
/**
* Gets an `Engine` that is created when the result of calling `aiomysql.sa.create_engine` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/sa.html#engine
*/
API::Node engine() {
result =
API::moduleImport("aiomysql")
.getMember("sa")
.getMember("create_engine")
.getReturn()
.getAwaited()
}
/**
* Gets an `SAConnection` that is created when the result of calling `aquire` on an `Engine` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/sa.html#connection
*/
API::Node saConnection() { result = engine().getMember("acquire").getReturn().getAwaited() }
/**
* A query. Calling `execute` on a `SAConnection` constructs a query.
* See https://aiomysql.readthedocs.io/en/stable/sa.html#aiomysql.sa.SAConnection.execute
*/
class SAConnectionExecuteCall extends SqlConstruction::Range, API::CallNode {
SAConnectionExecuteCall() { this = saConnection().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").getARhs() }
}
/**
* An awaited query. Awaiting the result of calling `execute` executes the query.
* See https://aiomysql.readthedocs.io/en/stable/sa.html#aiomysql.sa.SAConnection.execute
*/
class AwaitedSAConnectionExecuteCall extends SqlExecution::Range {
SAConnectionExecuteCall execute;
AwaitedSAConnectionExecuteCall() { this = execute.getReturn().getAwaited().getAnImmediateUse() }
override DataFlow::Node getSql() { result = execute.getSql() }
}
}

107
python/ql/lib/semmle/python/frameworks/Aiopg.qll Normal file
View File

@@ -0,0 +1,107 @@
/**
* Provides classes modeling security-relevant aspects of the `aiopg` PyPI package.
* See
* - https://aiopg.readthedocs.io/en/stable/index.html
* - https://pypi.org/project/aiopg/
*/
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
/** Provides models for the `aiopg` PyPI package. */
private module Aiopg {
private import semmle.python.internal.Awaited
/**
* Gets a `ConnectionPool` that is created when the result of `aiopg.create_pool()` is awaited.
* See https://aiopg.readthedocs.io/en/stable/core.html#pool
*/
API::Node connectionPool() {
result = API::moduleImport("aiopg").getMember("create_pool").getReturn().getAwaited()
}
/**
* Gets a `Connection` that is created when
* - the result of `aiopg.connect()` is awaited.
* - the result of calling `aquire` on a `ConnectionPool` is awaited.
* See https://aiopg.readthedocs.io/en/stable/core.html#connection
*/
API::Node connection() {
result = API::moduleImport("aiopg").getMember("connect").getReturn().getAwaited()
or
result = connectionPool().getMember("acquire").getReturn().getAwaited()
}
/**
* Gets a `Cursor` that is created when
* - the result of calling `cursor` on a `ConnectionPool` is awaited.
* - the result of calling `cursor` on a `Connection` is awaited.
* See https://aiopg.readthedocs.io/en/stable/core.html#cursor
*/
API::Node cursor() {
result = connectionPool().getMember("cursor").getReturn().getAwaited()
or
result = connection().getMember("cursor").getReturn().getAwaited()
}
/**
* A query. Calling `execute` on a `Cursor` constructs a query.
* See https://aiopg.readthedocs.io/en/stable/core.html#aiopg.Cursor.execute
*/
class CursorExecuteCall extends SqlConstruction::Range, API::CallNode {
CursorExecuteCall() { this = cursor().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "operation").getARhs() }
}
/**
* An awaited query. Awaiting the result of calling `execute` executes the query.
* See https://aiopg.readthedocs.io/en/stable/core.html#aiopg.Cursor.execute
*/
class AwaitedCursorExecuteCall extends SqlExecution::Range {
CursorExecuteCall execute;
AwaitedCursorExecuteCall() { this = execute.getReturn().getAwaited().getAnImmediateUse() }
override DataFlow::Node getSql() { result = execute.getSql() }
}
/**
* Gets an `Engine` that is created when the result of calling `aiopg.sa.create_engine` is awaited.
* See https://aiopg.readthedocs.io/en/stable/sa.html#engine
*/
API::Node engine() {
result =
API::moduleImport("aiopg").getMember("sa").getMember("create_engine").getReturn().getAwaited()
}
/**
* Gets an `SAConnection` that is created when the result of calling `aquire` on an `Engine` is awaited.
* See https://aiopg.readthedocs.io/en/stable/sa.html#connection
*/
API::Node saConnection() { result = engine().getMember("acquire").getReturn().getAwaited() }
/**
* A query. Calling `execute` on a `SAConnection` constructs a query.
* See https://aiopg.readthedocs.io/en/stable/sa.html#aiopg.sa.SAConnection.execute
*/
class SAConnectionExecuteCall extends SqlConstruction::Range, API::CallNode {
SAConnectionExecuteCall() { this = saConnection().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").getARhs() }
}
/**
* An awaited query. Awaiting the result of calling `execute` executes the query.
* See https://aiopg.readthedocs.io/en/stable/sa.html#aiopg.sa.SAConnection.execute
*/
class AwaitedSAConnectionExecuteCall extends SqlExecution::Range {
SAConnectionExecuteCall excute;
AwaitedSAConnectionExecuteCall() { this = excute.getReturn().getAwaited().getAnImmediateUse() }
override DataFlow::Node getSql() { result = excute.getSql() }
}
}

140
python/ql/lib/semmle/python/frameworks/Asyncpg.qll Normal file
View File

@@ -0,0 +1,140 @@
/**
* Provides classes modeling security-relevant aspects of the `asyncpg` PyPI package.
* See https://magicstack.github.io/asyncpg/.
*/
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
/** Provides models for the `asyncpg` PyPI package. */
private module Asyncpg {
private import semmle.python.internal.Awaited
/** Gets a `ConnectionPool` that is created when the result of `asyncpg.create_pool()` is awaited. */
API::Node connectionPool() {
result = API::moduleImport("asyncpg").getMember("create_pool").getReturn().getAwaited()
}
/**
* Gets a `Connection` that is created when
* - the result of `asyncpg.connect()` is awaited.
* - the result of calling `aquire` on a `ConnectionPool` is awaited.
*/
API::Node connection() {
result = API::moduleImport("asyncpg").getMember("connect").getReturn().getAwaited()
or
result = connectionPool().getMember("acquire").getReturn().getAwaited()
}
/** `Connection`s and `ConnectionPool`s provide some methods that execute SQL. */
class SqlExecutionOnConnection extends SqlExecution::Range, DataFlow::MethodCallNode {
string methodName;
SqlExecutionOnConnection() {
this = [connectionPool(), connection()].getMember(methodName).getACall() and
methodName in ["copy_from_query", "execute", "fetch", "fetchrow", "fetchval", "executemany"]
}
override DataFlow::Node getSql() {
methodName in ["copy_from_query", "execute", "fetch", "fetchrow", "fetchval"] and
result in [this.getArg(0), this.getArgByName("query")]
or
methodName = "executemany" and
result in [this.getArg(0), this.getArgByName("command")]
}
}
/** A model of `Connection` and `ConnectionPool`, which provide some methods that access the file system. */
class FileAccessOnConnection extends FileSystemAccess::Range, DataFlow::MethodCallNode {
string methodName;
FileAccessOnConnection() {
this = [connectionPool(), connection()].getMember(methodName).getACall() and
methodName in ["copy_from_query", "copy_from_table", "copy_to_table"]
}
// The path argument is keyword only.
override DataFlow::Node getAPathArgument() {
methodName in ["copy_from_query", "copy_from_table"] and
result = this.getArgByName("output")
or
methodName = "copy_to_table" and
result = this.getArgByName("source")
}
}
/**
* Provides models of the `PreparedStatement` class in `asyncpg`.
* `PreparedStatement`s are created when the result of calling `prepare(query)` on a connection is awaited.
* The result of calling `prepare(query)` is a `PreparedStatementFactory` and the argument, `query` needs to
* be tracked to the place where a `PreparedStatement` is created and then futher to any executing methods.
* Hence the two type trackers.
*/
module PreparedStatement {
class PreparedStatementConstruction extends SqlConstruction::Range, API::CallNode {
PreparedStatementConstruction() { this = connection().getMember("prepare").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").getARhs() }
}
class PreparedStatementExecution extends SqlExecution::Range, API::CallNode {
PreparedStatementConstruction prepareCall;
PreparedStatementExecution() {
this =
prepareCall
.getReturn()
.getAwaited()
.getMember(["executemany", "fetch", "fetchrow", "fetchval"])
.getACall()
}
override DataFlow::Node getSql() { result = prepareCall.getSql() }
}
}
/**
* Provides models of the `Cursor` class in `asyncpg`.
* `Cursor`s are created
* - when the result of calling `cursor(query)` on a connection is awaited.
* - when the result of calling `cursor()` on a prepared statement is awaited.
* The result of calling `cursor` in either case is a `CursorFactory` and the argument, `query` needs to
* be tracked to the place where a `Cursor` is created, hence the type tracker.
* The creation of the `Cursor` executes the query.
*/
module Cursor {
class CursorConstruction extends SqlConstruction::Range, API::CallNode {
CursorConstruction() { this = connection().getMember("cursor").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").getARhs() }
}
/** The creation of a `Cursor` executes the associated query. */
class CursorCreation extends SqlExecution::Range {
DataFlow::Node sql;
CursorCreation() {
exists(CursorConstruction c |
sql = c.getSql() and
this = c.getReturn().getAwaited().getAnImmediateUse()
)
or
exists(PreparedStatement::PreparedStatementConstruction prepareCall |
sql = prepareCall.getSql() and
this =
prepareCall
.getReturn()
.getAwaited()
.getMember("cursor")
.getReturn()
.getAwaited()
.getAnImmediateUse()
)
}
override DataFlow::Node getSql() { result = sql }
}
}
}

2
python/ql/lib/semmle/python/frameworks/ClickhouseDriver.qll
View File

@@ -20,7 +20,7 @@ private import semmle.python.frameworks.PEP249
*/
module ClickhouseDriver {
/**
* `clickhouse_driver` implements PEP249,
* A model of `clickhouse-driver`, which implements PEP249,
* providing ways to execute SQL statements against a database.
*/
class ClickHouseDriverPEP249 extends PEP249::PEP249ModuleApiNode {

120
python/ql/lib/semmle/python/frameworks/Cryptography.qll
View File

@@ -22,7 +22,7 @@ private module CryptographyModel {
* Gets a predefined curve class from
* `cryptography.hazmat.primitives.asymmetric.ec` with a specific key size (in bits).
*/
private API::Node predefinedCurveClass(int keySize) {
API::Node predefinedCurveClass(int keySize) {
exists(string curveName |
result =
API::moduleImport("cryptography")
@@ -73,41 +73,6 @@ private module CryptographyModel {
curveName = "BrainpoolP512R1" and keySize = 512
)
}
/** Gets a reference to a predefined curve class with a specific key size (in bits), as well as the origin of the class. */
private DataFlow::TypeTrackingNode curveClassWithKeySize(
DataFlow::TypeTracker t, int keySize, DataFlow::Node origin
) {
t.start() and
result = predefinedCurveClass(keySize).getAnImmediateUse() and
origin = result
or
exists(DataFlow::TypeTracker t2 |
result = curveClassWithKeySize(t2, keySize, origin).track(t2, t)
)
}
/** Gets a reference to a predefined curve class with a specific key size (in bits), as well as the origin of the class. */
DataFlow::Node curveClassWithKeySize(int keySize, DataFlow::Node origin) {
curveClassWithKeySize(DataFlow::TypeTracker::end(), keySize, origin).flowsTo(result)
}
/** Gets a reference to a predefined curve class instance with a specific key size (in bits), as well as the origin of the class. */
private DataFlow::TypeTrackingNode curveClassInstanceWithKeySize(
DataFlow::TypeTracker t, int keySize, DataFlow::Node origin
) {
t.start() and
result.(DataFlow::CallCfgNode).getFunction() = curveClassWithKeySize(keySize, origin)
or
exists(DataFlow::TypeTracker t2 |
result = curveClassInstanceWithKeySize(t2, keySize, origin).track(t2, t)
)
}
/** Gets a reference to a predefined curve class instance with a specific key size (in bits), as well as the origin of the class. */
DataFlow::Node curveClassInstanceWithKeySize(int keySize, DataFlow::Node origin) {
curveClassInstanceWithKeySize(DataFlow::TypeTracker::end(), keySize, origin).flowsTo(result)
}
}
// ---------------------------------------------------------------------------
@@ -179,9 +144,13 @@ private module CryptographyModel {
DataFlow::Node getCurveArg() { result in [this.getArg(0), this.getArgByName("curve")] }
override int getKeySizeWithOrigin(DataFlow::Node origin) {
this.getCurveArg() = Ecc::curveClassInstanceWithKeySize(result, origin)
or
this.getCurveArg() = Ecc::curveClassWithKeySize(result, origin)
exists(API::Node n |
n = Ecc::predefinedCurveClass(result) and origin = n.getAnImmediateUse()
|
this.getCurveArg() = n.getAUse()
or
this.getCurveArg() = n.getReturn().getAUse()
)
}
// Note: There is not really a key-size argument, since it's always specified by the curve.
@@ -202,9 +171,8 @@ private module CryptographyModel {
}
/** Gets a reference to a Cipher instance using algorithm with `algorithmName`. */
DataFlow::TypeTrackingNode cipherInstance(DataFlow::TypeTracker t, string algorithmName) {
t.start() and
exists(DataFlow::CallCfgNode call | result = call |
API::Node cipherInstance(string algorithmName) {
exists(API::CallNode call | result = call.getReturn() |
call =
API::moduleImport("cryptography")
.getMember("hazmat")
@@ -216,47 +184,6 @@ private module CryptographyModel {
call.getArg(0), call.getArgByName("algorithm")
]
)
or
exists(DataFlow::TypeTracker t2 | result = cipherInstance(t2, algorithmName).track(t2, t))
}
/** Gets a reference to a Cipher instance using algorithm with `algorithmName`. */
DataFlow::Node cipherInstance(string algorithmName) {
cipherInstance(DataFlow::TypeTracker::end(), algorithmName).flowsTo(result)
}
/** Gets a reference to the encryptor of a Cipher instance using algorithm with `algorithmName`. */
DataFlow::TypeTrackingNode cipherEncryptor(DataFlow::TypeTracker t, string algorithmName) {
t.start() and
result.(DataFlow::MethodCallNode).calls(cipherInstance(algorithmName), "encryptor")
or
exists(DataFlow::TypeTracker t2 | result = cipherEncryptor(t2, algorithmName).track(t2, t))
}
/**
* Gets a reference to the encryptor of a Cipher instance using algorithm with `algorithmName`.
*
* You obtain an encryptor by using the `encryptor()` method on a Cipher instance.
*/
DataFlow::Node cipherEncryptor(string algorithmName) {
cipherEncryptor(DataFlow::TypeTracker::end(), algorithmName).flowsTo(result)
}
/** Gets a reference to the dncryptor of a Cipher instance using algorithm with `algorithmName`. */
DataFlow::TypeTrackingNode cipherDecryptor(DataFlow::TypeTracker t, string algorithmName) {
t.start() and
result.(DataFlow::MethodCallNode).calls(cipherInstance(algorithmName), "decryptor")
or
exists(DataFlow::TypeTracker t2 | result = cipherDecryptor(t2, algorithmName).track(t2, t))
}
/**
* Gets a reference to the decryptor of a Cipher instance using algorithm with `algorithmName`.
*
* You obtain an decryptor by using the `decryptor()` method on a Cipher instance.
*/
DataFlow::Node cipherDecryptor(string algorithmName) {
cipherDecryptor(DataFlow::TypeTracker::end(), algorithmName).flowsTo(result)
}
/**
@@ -267,11 +194,12 @@ private module CryptographyModel {
string algorithmName;
CryptographyGenericCipherOperation() {
exists(DataFlow::Node object, string method |
object in [cipherEncryptor(algorithmName), cipherDecryptor(algorithmName)] and
method in ["update", "update_into"] and
this.calls(object, method)
)
this =
cipherInstance(algorithmName)
.getMember(["decryptor", "encryptor"])
.getReturn()
.getMember(["update", "update_into"])
.getACall()
}
override Cryptography::CryptographicAlgorithm getAlgorithm() {
@@ -298,9 +226,8 @@ private module CryptographyModel {
}
/** Gets a reference to a Hash instance using algorithm with `algorithmName`. */
private DataFlow::TypeTrackingNode hashInstance(DataFlow::TypeTracker t, string algorithmName) {
t.start() and
exists(DataFlow::CallCfgNode call | result = call |
private API::Node hashInstance(string algorithmName) {
exists(API::CallNode call | result = call.getReturn() |
call =
API::moduleImport("cryptography")
.getMember("hazmat")
@@ -312,13 +239,6 @@ private module CryptographyModel {
call.getArg(0), call.getArgByName("algorithm")
]
)
or
exists(DataFlow::TypeTracker t2 | result = hashInstance(t2, algorithmName).track(t2, t))
}
/** Gets a reference to a Hash instance using algorithm with `algorithmName`. */
DataFlow::Node hashInstance(string algorithmName) {
hashInstance(DataFlow::TypeTracker::end(), algorithmName).flowsTo(result)
}
/**
@@ -328,7 +248,9 @@ private module CryptographyModel {
DataFlow::MethodCallNode {
string algorithmName;
CryptographyGenericHashOperation() { this.calls(hashInstance(algorithmName), "update") }
CryptographyGenericHashOperation() {
this = hashInstance(algorithmName).getMember("update").getACall()
}
override Cryptography::CryptographicAlgorithm getAlgorithm() {
result.matchesName(algorithmName)

688
python/ql/lib/semmle/python/frameworks/Django.qll
View File

File diff suppressed because it is too large Load Diff

30
python/ql/lib/semmle/python/frameworks/Fabric.qll
View File

@@ -24,7 +24,7 @@ private module FabricV1 {
API::Node fabric() { result = API::moduleImport("fabric") }
/** Provides models for the `fabric` module. */
module fabric {
module Fabric {
// -------------------------------------------------------------------------
// fabric.api
// -------------------------------------------------------------------------
@@ -32,7 +32,7 @@ private module FabricV1 {
API::Node api() { result = fabric().getMember("api") }
/** Provides models for the `fabric.api` module */
module api {
module Api {
/**
* A call to either
* - `fabric.api.local`
@@ -66,7 +66,7 @@ private module FabricV2 {
API::Node fabric() { result = API::moduleImport("fabric") }
/** Provides models for the `fabric` module. */
module fabric {
module Fabric {
// -------------------------------------------------------------------------
// fabric.connection
// -------------------------------------------------------------------------
@@ -74,13 +74,13 @@ private module FabricV2 {
API::Node connection() { result = fabric().getMember("connection") }
/** Provides models for the `fabric.connection` module */
module connection {
module Connection {
/**
* Provides models for the `fabric.connection.Connection` class
*
* See https://docs.fabfile.org/en/2.5/api/connection.html#fabric.connection.Connection.
*/
module Connection {
module ConnectionClass {
/** Gets a reference to the `fabric.connection.Connection` class. */
API::Node classRef() {
result = fabric().getMember("Connection")
@@ -155,7 +155,7 @@ private module FabricV2 {
private class FabricConnectionRunSudoLocalCall extends SystemCommandExecution::Range,
DataFlow::CallCfgNode {
FabricConnectionRunSudoLocalCall() {
this.getFunction() = fabric::connection::Connection::instanceRunMethods()
this.getFunction() = Fabric::Connection::ConnectionClass::instanceRunMethods()
}
override DataFlow::Node getCommand() {
@@ -170,16 +170,16 @@ private module FabricV2 {
API::Node tasks() { result = fabric().getMember("tasks") }
/** Provides models for the `fabric.tasks` module */
module tasks {
module Tasks {
/** Gets a reference to the `fabric.tasks.task` decorator. */
API::Node task() { result in [tasks().getMember("task"), fabric().getMember("task")] }
}
class FabricTaskFirstParamConnectionInstance extends fabric::connection::Connection::InstanceSource,
class FabricTaskFirstParamConnectionInstance extends Fabric::Connection::ConnectionClass::InstanceSource,
DataFlow::ParameterNode {
FabricTaskFirstParamConnectionInstance() {
exists(Function func |
func.getADecorator() = fabric::tasks::task().getAUse().asExpr() and
func.getADecorator() = Fabric::Tasks::task().getAUse().asExpr() and
this.getParameter() = func.getArg(0)
)
}
@@ -192,7 +192,7 @@ private module FabricV2 {
API::Node group() { result = fabric().getMember("group") }
/** Provides models for the `fabric.group` module */
module group {
module Group {
/**
* Provides models for the `fabric.group.Group` class and its subclasses.
*
@@ -204,7 +204,7 @@ private module FabricV2 {
* - https://docs.fabfile.org/en/2.5/api/group.html#fabric.group.SerialGroup
* - https://docs.fabfile.org/en/2.5/api/group.html#fabric.group.ThreadingGroup
*/
module Group {
module GroupClass {
/**
* A source of instances of a subclass of `fabric.group, extend this class to model new instances.Group`
*
@@ -236,7 +236,9 @@ private module FabricV2 {
* See https://docs.fabfile.org/en/2.5/api/group.html#fabric.group.Group.run
*/
private class FabricGroupRunCall extends SystemCommandExecution::Range, DataFlow::CallCfgNode {
FabricGroupRunCall() { this = fabric::group::Group::subclassInstanceRunMethod().getACall() }
FabricGroupRunCall() {
this = Fabric::Group::GroupClass::subclassInstanceRunMethod().getACall()
}
override DataFlow::Node getCommand() {
result = [this.getArg(0), this.getArgByName("command")]
@@ -249,7 +251,7 @@ private module FabricV2 {
* See https://docs.fabfile.org/en/2.5/api/group.html#fabric.group.SerialGroup.
*/
module SerialGroup {
private class ClassInstantiation extends Group::ModeledSubclass {
private class ClassInstantiation extends GroupClass::ModeledSubclass {
ClassInstantiation() {
this = group().getMember("SerialGroup")
or
@@ -264,7 +266,7 @@ private module FabricV2 {
* See https://docs.fabfile.org/en/2.5/api/group.html#fabric.group.ThreadingGroup.
*/
module ThreadingGroup {
private class ClassInstantiation extends Group::ModeledSubclass {
private class ClassInstantiation extends GroupClass::ModeledSubclass {
ClassInstantiation() {
this = group().getMember("ThreadingGroup")
or

Some files were not shown because too many files have changed in this diff Show More