hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2025-12-22 03:36:30 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/github/codeql into python/captured-variables-for-typetracking

Browse Source
This commit is contained in:
Rasmus Lerchedahl Petersen
2023-04-24 11:50:32 +02:00
parent 9e3d57d442 bfbbb5277d
commit a25c7f7549
3246 changed files with 275867 additions and 170048 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
python/ql/examples/queries.xml
View File

@@ -1 +0,0 @@
<queries language="python"/>

40
python/ql/lib/CHANGELOG.md
View File

@@ -1,3 +1,43 @@
## 0.9.0
### Deprecated APIs
* The recently introduced new data flow and taint tracking APIs have had a
number of module and predicate renamings. The old APIs remain in place for
now.
### Minor Analysis Improvements
* Added modeling of SQL execution in the packages `sqlite3.dbapi2`, `cassandra-driver`, `aiosqlite`, and the functions `sqlite3.Connection.executescript`/`sqlite3.Cursor.executescript` and `asyncpg.connection.connect()`.
* Fixed module resolution so we allow imports of definitions that have had an attribute assigned to it, such as `class Foo; Foo.bar = 42`.
### Bug Fixes
* Fixed some accidental predicate visibility in the backwards-compatible wrapper for data flow configurations. In particular, `DataFlow::hasFlowPath`, `DataFlow::hasFlow`, `DataFlow::hasFlowTo`, and `DataFlow::hasFlowToExpr` were accidentally exposed in a single version.
## 0.8.3
No user-facing changes.
## 0.8.2
### New Features
* Added support for merging two `PathGraph`s via disjoint union to allow results from multiple data flow computations in a single `path-problem` query.
### Major Analysis Improvements
* The main data flow and taint tracking APIs have been changed. The old APIs
remain in place for now and translate to the new through a
backwards-compatible wrapper. If multiple configurations are in scope
simultaneously, then this may affect results slightly. The new API is quite
similar to the old, but makes use of a configuration module instead of a
configuration class.
### Minor Analysis Improvements
* Deleted the deprecated `getPath` and `getFolder` predicates from the `XmlFile` class.
## 0.8.1
### Major Analysis Improvements

4
python/ql/lib/change-notes/2023-03-03-delete-deps.md
View File

@@ -1,4 +0,0 @@
---
category: minorAnalysis
---
* Deleted the deprecated `getPath` and `getFolder` predicates from the `XmlFile` class.

4
python/ql/lib/change-notes/2023-03-13-mergepathgraph.md
View File

@@ -1,4 +0,0 @@
---
category: feature
---
* Added support for merging two `PathGraph`s via disjoint union to allow results from multiple data flow computations in a single `path-problem` query.

15
python/ql/lib/change-notes/2023-03-02-dataflow-conf-module.md → python/ql/lib/change-notes/released/0.8.2.md
View File

@@ -1,9 +1,18 @@
---
category: majorAnalysis
---
## 0.8.2
### New Features
* Added support for merging two `PathGraph`s via disjoint union to allow results from multiple data flow computations in a single `path-problem` query.
### Major Analysis Improvements
* The main data flow and taint tracking APIs have been changed. The old APIs
remain in place for now and translate to the new through a
backwards-compatible wrapper. If multiple configurations are in scope
simultaneously, then this may affect results slightly. The new API is quite
similar to the old, but makes use of a configuration module instead of a
configuration class.
### Minor Analysis Improvements
* Deleted the deprecated `getPath` and `getFolder` predicates from the `XmlFile` class.

3
python/ql/lib/change-notes/released/0.8.3.md Normal file
View File

@@ -0,0 +1,3 @@
## 0.8.3
No user-facing changes.

16
python/ql/lib/change-notes/released/0.9.0.md Normal file
View File

@@ -0,0 +1,16 @@
## 0.9.0
### Deprecated APIs
* The recently introduced new data flow and taint tracking APIs have had a
number of module and predicate renamings. The old APIs remain in place for
now.
### Minor Analysis Improvements
* Added modeling of SQL execution in the packages `sqlite3.dbapi2`, `cassandra-driver`, `aiosqlite`, and the functions `sqlite3.Connection.executescript`/`sqlite3.Cursor.executescript` and `asyncpg.connection.connect()`.
* Fixed module resolution so we allow imports of definitions that have had an attribute assigned to it, such as `class Foo; Foo.bar = 42`.
### Bug Fixes
* Fixed some accidental predicate visibility in the backwards-compatible wrapper for data flow configurations. In particular, `DataFlow::hasFlowPath`, `DataFlow::hasFlow`, `DataFlow::hasFlowTo`, and `DataFlow::hasFlowToExpr` were accidentally exposed in a single version.

2
python/ql/lib/codeql-pack.release.yml
View File

@@ -1,2 +1,2 @@
---
lastReleaseVersion: 0.8.1
lastReleaseVersion: 0.9.0

3
python/ql/lib/qlpack.yml
View File

@@ -1,5 +1,5 @@
name: codeql/python-all
version: 0.8.2-dev
version: 0.9.1-dev
groups: python
dbscheme: semmlecode.python.dbscheme
extractor: python
@@ -8,5 +8,6 @@ upgrades: upgrades
dependencies:
codeql/regex: ${workspace}
codeql/tutorial: ${workspace}
codeql/util: ${workspace}
dataExtensions:
- semmle/python/frameworks/**/model.yml

4
python/ql/lib/semmle/python/Frameworks.qll
View File

@@ -3,12 +3,14 @@
*/
// If you add modeling of a new framework/library, remember to add it to the docs in
// `docs/codeql/support/reusables/frameworks.rst`
// `docs/codeql/reusables/supported-frameworks.rst`
private import semmle.python.frameworks.Aioch
private import semmle.python.frameworks.Aiohttp
private import semmle.python.frameworks.Aiomysql
private import semmle.python.frameworks.Aiosqlite
private import semmle.python.frameworks.Aiopg
private import semmle.python.frameworks.Asyncpg
private import semmle.python.frameworks.CassandraDriver
private import semmle.python.frameworks.ClickhouseDriver
private import semmle.python.frameworks.Cryptodome
private import semmle.python.frameworks.Cryptography

38
python/ql/lib/semmle/python/RegexTreeView.qll
View File

@@ -468,6 +468,8 @@ module Impl implements RegexTreeViewSig {
*/
class RegExpCharEscape = RegExpEscape;
private import codeql.util.Numbers as Numbers
/**
* An escaped regular expression term, that is, a regular expression
* term starting with a backslash, which is not a backreference.
@@ -528,42 +530,8 @@ module Impl implements RegexTreeViewSig {
* E.g. for `\u0061` this returns "a".
*/
private string getUnicode() {
exists(int codepoint | codepoint = sum(this.getHexValueFromUnicode(_)) |
result = codepoint.toUnicode()
)
result = Numbers::parseHexInt(this.getText().suffix(2)).toUnicode()
}
/**
* Gets int value for the `index`th char in the hex number of the unicode escape.
* E.g. for `\u0061` and `index = 2` this returns 96 (the number `6` interpreted as hex).
*/
private int getHexValueFromUnicode(int index) {
this.isUnicode() and
exists(string hex, string char | hex = this.getText().suffix(2) |
char = hex.charAt(index) and
result = 16.pow(hex.length() - index - 1) * toHex(char)
)
}
}
/**
* Gets the hex number for the `hex` char.
*/
private int toHex(string hex) {
hex = [0 .. 9].toString() and
result = hex.toInt()
or
result = 10 and hex = ["a", "A"]
or
result = 11 and hex = ["b", "B"]
or
result = 12 and hex = ["c", "C"]
or
result = 13 and hex = ["d", "D"]
or
result = 14 and hex = ["e", "E"]
or
result = 15 and hex = ["f", "F"]
}
/**

9
python/ql/lib/semmle/python/Unit.qll
View File

@@ -1,10 +1,3 @@
/** Provides the `Unit` class. */
/** The unit type. */
private newtype TUnit = TMkUnit()
/** The trivial type with a single element. */
class Unit extends TUnit {
/** Gets a textual representation of this element. */
string toString() { result = "unit" }
}
import codeql.util.Unit

42
python/ql/lib/semmle/python/dataflow/new/internal/DataFlow.qll
View File

@@ -2,7 +2,7 @@
* Provides an implementation of global (interprocedural) data flow. This file
* re-exports the local (intraprocedural) data flow analysis from
* `DataFlowImplSpecific::Public` and adds a global analysis, mainly exposed
* through the `Make` and `MakeWithState` modules.
* through the `Global` and `GlobalWithState` modules.
*/
private import DataFlowImplCommon
@@ -73,10 +73,10 @@ signature module ConfigSig {
*/
default FlowFeature getAFeature() { none() }
/** Holds if sources should be grouped in the result of `hasFlowPath`. */
/** Holds if sources should be grouped in the result of `flowPath`. */
default predicate sourceGrouping(Node source, string sourceGroup) { none() }
/** Holds if sinks should be grouped in the result of `hasFlowPath`. */
/** Holds if sinks should be grouped in the result of `flowPath`. */
default predicate sinkGrouping(Node sink, string sinkGroup) { none() }
/**
@@ -166,10 +166,10 @@ signature module StateConfigSig {
*/
default FlowFeature getAFeature() { none() }
/** Holds if sources should be grouped in the result of `hasFlowPath`. */
/** Holds if sources should be grouped in the result of `flowPath`. */
default predicate sourceGrouping(Node source, string sourceGroup) { none() }
/** Holds if sinks should be grouped in the result of `hasFlowPath`. */
/** Holds if sinks should be grouped in the result of `flowPath`. */
default predicate sinkGrouping(Node sink, string sinkGroup) { none() }
/**
@@ -182,15 +182,15 @@ signature module StateConfigSig {
}
/**
* Gets the exploration limit for `hasPartialFlow` and `hasPartialFlowRev`
* Gets the exploration limit for `partialFlow` and `partialFlowRev`
* measured in approximate number of interprocedural steps.
*/
signature int explorationLimitSig();
/**
* The output of a data flow computation.
* The output of a global data flow computation.
*/
signature module DataFlowSig {
signature module GlobalFlowSig {
/**
* A `Node` augmented with a call context (except for sinks) and an access path.
* Only those `PathNode`s that are reachable from a source, and which can reach a sink, are generated.
@@ -203,28 +203,28 @@ signature module DataFlowSig {
* The corresponding paths are generated from the end-points and the graph
* included in the module `PathGraph`.
*/
predicate hasFlowPath(PathNode source, PathNode sink);
predicate flowPath(PathNode source, PathNode sink);
/**
* Holds if data can flow from `source` to `sink`.
*/
predicate hasFlow(Node source, Node sink);
predicate flow(Node source, Node sink);
/**
* Holds if data can flow from some source to `sink`.
*/
predicate hasFlowTo(Node sink);
predicate flowTo(Node sink);
/**
* Holds if data can flow from some source to `sink`.
*/
predicate hasFlowToExpr(DataFlowExpr sink);
predicate flowToExpr(DataFlowExpr sink);
}
/**
* Constructs a standard data flow computation.
* Constructs a global data flow computation.
*/
module Make<ConfigSig Config> implements DataFlowSig {
module Global<ConfigSig Config> implements GlobalFlowSig {
private module C implements FullStateConfigSig {
import DefaultState<Config>
import Config
@@ -233,10 +233,15 @@ module Make<ConfigSig Config> implements DataFlowSig {
import Impl<C>
}
/** DEPRECATED: Use `Global` instead. */
deprecated module Make<ConfigSig Config> implements GlobalFlowSig {
import Global<Config>
}
/**
* Constructs a data flow computation using flow state.
* Constructs a global data flow computation using flow state.
*/
module MakeWithState<StateConfigSig Config> implements DataFlowSig {
module GlobalWithState<StateConfigSig Config> implements GlobalFlowSig {
private module C implements FullStateConfigSig {
import Config
}
@@ -244,6 +249,11 @@ module MakeWithState<StateConfigSig Config> implements DataFlowSig {
import Impl<C>
}
/** DEPRECATED: Use `GlobalWithState` instead. */
deprecated module MakeWithState<StateConfigSig Config> implements GlobalFlowSig {
import GlobalWithState<Config>
}
signature class PathNodeSig {
/** Gets a textual representation of this element. */
string toString();

10
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatch.qll
View File

@@ -1614,3 +1614,13 @@ private module OutNodes {
* `kind`.
*/
OutNode getAnOutNode(DataFlowCall call, ReturnKind kind) { call = result.getCall(kind) }
/**
* Holds if flow from `call`'s argument `arg` to parameter `p` is permissible.
*
* This is a temporary hook to support technical debt in the Go language; do not use.
*/
pragma[inline]
predicate golangSpecificParamArgFilter(DataFlowCall call, ParameterNode p, ArgumentNode arg) {
any()
}

91
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl.qll
View File

@@ -8,6 +8,7 @@ private import DataFlowImplCommon
private import DataFlowImplSpecific::Private
private import DataFlowImplSpecific::Public
private import DataFlowImplCommonPublic
private import codeql.util.Unit
import DataFlow
/**
@@ -91,10 +92,10 @@ signature module FullStateConfigSig {
*/
FlowFeature getAFeature();
/** Holds if sources should be grouped in the result of `hasFlowPath`. */
/** Holds if sources should be grouped in the result of `flowPath`. */
predicate sourceGrouping(Node source, string sourceGroup);
/** Holds if sinks should be grouped in the result of `hasFlowPath`. */
/** Holds if sinks should be grouped in the result of `flowPath`. */
predicate sinkGrouping(Node sink, string sinkGroup);
/**
@@ -418,6 +419,10 @@ module Impl<FullStateConfigSig Config> {
)
}
private predicate sourceCallCtx(CallContext cc) {
if hasSourceCallCtx() then cc instanceof CallContextSomeCall else cc instanceof CallContextAny
}
private predicate hasSinkCallCtx() {
exists(FlowFeature feature | feature = Config::getAFeature() |
feature instanceof FeatureHasSinkCallContext or
@@ -441,11 +446,7 @@ module Impl<FullStateConfigSig Config> {
}
private module Stage1 implements StageSig {
class Ap extends int {
// workaround for bad functionality-induced joins (happens when using `Unit`)
pragma[nomagic]
Ap() { this in [0 .. 1] and this < 1 }
}
class Ap = Unit;
private class Cc = boolean;
@@ -1141,19 +1142,13 @@ module Impl<FullStateConfigSig Config> {
import Param
/* Begin: Stage logic. */
// use an alias as a workaround for bad functionality-induced joins
pragma[nomagic]
private predicate revFlowApAlias(NodeEx node, ApApprox apa) {
PrevStage::revFlowAp(node, apa)
}
pragma[nomagic]
private predicate flowIntoCallApa(
DataFlowCall call, ArgNodeEx arg, ParamNodeEx p, boolean allowsFieldFlow, ApApprox apa
) {
flowIntoCall(call, arg, p, allowsFieldFlow) and
PrevStage::revFlowAp(p, pragma[only_bind_into](apa)) and
revFlowApAlias(arg, pragma[only_bind_into](apa))
PrevStage::revFlowAp(arg, pragma[only_bind_into](apa))
}
pragma[nomagic]
@@ -1163,7 +1158,7 @@ module Impl<FullStateConfigSig Config> {
) {
flowOutOfCall(call, ret, kind, out, allowsFieldFlow) and
PrevStage::revFlowAp(out, pragma[only_bind_into](apa)) and
revFlowApAlias(ret, pragma[only_bind_into](apa))
PrevStage::revFlowAp(ret, pragma[only_bind_into](apa))
}
pragma[nomagic]
@@ -1691,16 +1686,6 @@ module Impl<FullStateConfigSig Config> {
pragma[nomagic]
predicate revFlowAp(NodeEx node, Ap ap) { revFlow(node, _, _, _, ap) }
// use an alias as a workaround for bad functionality-induced joins
pragma[nomagic]
additional predicate revFlowAlias(NodeEx node) { revFlow(node, _, _, _, _) }
// use an alias as a workaround for bad functionality-induced joins
pragma[nomagic]
additional predicate revFlowAlias(NodeEx node, FlowState state, Ap ap) {
revFlow(node, state, ap)
}
private predicate fwdConsCand(TypedContent tc, Ap ap) { storeStepFwd(_, ap, tc, _, _) }
private predicate revConsCand(TypedContent tc, Ap ap) { storeStepCand(_, ap, tc, _, _) }
@@ -1974,7 +1959,7 @@ module Impl<FullStateConfigSig Config> {
) {
flowOutOfCallNodeCand1(call, node1, kind, node2, allowsFieldFlow) and
Stage2::revFlow(node2) and
Stage2::revFlowAlias(node1)
Stage2::revFlow(node1)
}
pragma[nomagic]
@@ -1983,7 +1968,7 @@ module Impl<FullStateConfigSig Config> {
) {
flowIntoCallNodeCand1(call, node1, node2, allowsFieldFlow) and
Stage2::revFlow(node2) and
Stage2::revFlowAlias(node1)
Stage2::revFlow(node1)
}
private module LocalFlowBigStep {
@@ -2065,11 +2050,11 @@ module Impl<FullStateConfigSig Config> {
additionalLocalFlowStepNodeCand1(node1, node2) and
state1 = state2 and
Stage2::revFlow(node1, pragma[only_bind_into](state1), false) and
Stage2::revFlowAlias(node2, pragma[only_bind_into](state2), false)
Stage2::revFlow(node2, pragma[only_bind_into](state2), false)
or
additionalLocalStateStep(node1, state1, node2, state2) and
Stage2::revFlow(node1, state1, false) and
Stage2::revFlowAlias(node2, state2, false)
Stage2::revFlow(node2, state2, false)
}
/**
@@ -2262,7 +2247,7 @@ module Impl<FullStateConfigSig Config> {
) {
localFlowBigStep(node1, state1, node2, state2, preservesValue, ap.getType(), _) and
PrevStage::revFlow(node1, pragma[only_bind_into](state1), _) and
PrevStage::revFlowAlias(node2, pragma[only_bind_into](state2), _) and
PrevStage::revFlow(node2, pragma[only_bind_into](state2), _) and
exists(lcc)
}
@@ -2273,7 +2258,7 @@ module Impl<FullStateConfigSig Config> {
exists(FlowState state |
flowOutOfCallNodeCand2(call, node1, kind, node2, allowsFieldFlow) and
PrevStage::revFlow(node2, pragma[only_bind_into](state), _) and
PrevStage::revFlowAlias(node1, pragma[only_bind_into](state), _)
PrevStage::revFlow(node1, pragma[only_bind_into](state), _)
)
}
@@ -2284,7 +2269,7 @@ module Impl<FullStateConfigSig Config> {
exists(FlowState state |
flowIntoCallNodeCand2(call, node1, node2, allowsFieldFlow) and
PrevStage::revFlow(node2, pragma[only_bind_into](state), _) and
PrevStage::revFlowAlias(node1, pragma[only_bind_into](state), _)
PrevStage::revFlow(node1, pragma[only_bind_into](state), _)
)
}
@@ -2586,7 +2571,7 @@ module Impl<FullStateConfigSig Config> {
) {
localFlowBigStep(node1, state1, node2, state2, preservesValue, ap.getType(), lcc) and
PrevStage::revFlow(node1, pragma[only_bind_into](state1), _) and
PrevStage::revFlowAlias(node2, pragma[only_bind_into](state2), _)
PrevStage::revFlow(node2, pragma[only_bind_into](state2), _)
}
pragma[nomagic]
@@ -2596,7 +2581,7 @@ module Impl<FullStateConfigSig Config> {
exists(FlowState state |
flowOutOfCallNodeCand2(call, node1, kind, node2, allowsFieldFlow) and
PrevStage::revFlow(node2, pragma[only_bind_into](state), _) and
PrevStage::revFlowAlias(node1, pragma[only_bind_into](state), _)
PrevStage::revFlow(node1, pragma[only_bind_into](state), _)
)
}
@@ -2607,7 +2592,7 @@ module Impl<FullStateConfigSig Config> {
exists(FlowState state |
flowIntoCallNodeCand2(call, node1, node2, allowsFieldFlow) and
PrevStage::revFlow(node2, pragma[only_bind_into](state), _) and
PrevStage::revFlowAlias(node1, pragma[only_bind_into](state), _)
PrevStage::revFlow(node1, pragma[only_bind_into](state), _)
)
}
@@ -2804,11 +2789,7 @@ module Impl<FullStateConfigSig Config> {
// A PathNode is introduced by a source ...
Stage5::revFlow(node, state) and
sourceNode(node, state) and
(
if hasSourceCallCtx()
then cc instanceof CallContextSomeCall
else cc instanceof CallContextAny
) and
sourceCallCtx(cc) and
sc instanceof SummaryCtxNone and
ap = TAccessPathNil(node.getDataFlowType())
or
@@ -3214,11 +3195,7 @@ module Impl<FullStateConfigSig Config> {
override predicate isSource() {
sourceNode(node, state) and
(
if hasSourceCallCtx()
then cc instanceof CallContextSomeCall
else cc instanceof CallContextAny
) and
sourceCallCtx(cc) and
sc instanceof SummaryCtxNone and
ap = TAccessPathNil(node.getDataFlowType())
}
@@ -3653,7 +3630,7 @@ module Impl<FullStateConfigSig Config> {
* The corresponding paths are generated from the end-points and the graph
* included in the module `PathGraph`.
*/
predicate hasFlowPath(PathNode source, PathNode sink) {
predicate flowPath(PathNode source, PathNode sink) {
exists(PathNodeImpl flowsource, PathNodeImpl flowsink |
source = flowsource and sink = flowsink
|
@@ -3663,6 +3640,9 @@ module Impl<FullStateConfigSig Config> {
)
}
/** DEPRECATED: Use `flowPath` instead. */
deprecated predicate hasFlowPath = flowPath/2;
private predicate flowsTo(PathNodeImpl flowsource, PathNodeSink flowsink, Node source, Node sink) {
flowsource.isSource() and
flowsource.getNodeEx().asNode() = source and
@@ -3673,17 +3653,26 @@ module Impl<FullStateConfigSig Config> {
/**
* Holds if data can flow from `source` to `sink`.
*/
predicate hasFlow(Node source, Node sink) { flowsTo(_, _, source, sink) }
predicate flow(Node source, Node sink) { flowsTo(_, _, source, sink) }
/** DEPRECATED: Use `flow` instead. */
deprecated predicate hasFlow = flow/2;
/**
* Holds if data can flow from some source to `sink`.
*/
predicate hasFlowTo(Node sink) { sink = any(PathNodeSink n).getNodeEx().asNode() }
predicate flowTo(Node sink) { sink = any(PathNodeSink n).getNodeEx().asNode() }
/** DEPRECATED: Use `flowTo` instead. */
deprecated predicate hasFlowTo = flowTo/1;
/**
* Holds if data can flow from some source to `sink`.
*/
predicate hasFlowToExpr(DataFlowExpr sink) { hasFlowTo(exprNode(sink)) }
predicate flowToExpr(DataFlowExpr sink) { flowTo(exprNode(sink)) }
/** DEPRECATED: Use `flowToExpr` instead. */
deprecated predicate hasFlowToExpr = flowToExpr/1;
private predicate finalStats(
boolean fwd, int nodes, int fields, int conscand, int states, int tuples
@@ -4594,7 +4583,7 @@ module Impl<FullStateConfigSig Config> {
*
* To use this in a `path-problem` query, import the module `PartialPathGraph`.
*/
predicate hasPartialFlow(PartialPathNode source, PartialPathNode node, int dist) {
predicate partialFlow(PartialPathNode source, PartialPathNode node, int dist) {
partialFlow(source, node) and
dist = node.getSourceDistance()
}
@@ -4614,7 +4603,7 @@ module Impl<FullStateConfigSig Config> {
* Note that reverse flow has slightly lower precision than the corresponding
* forward flow, as reverse flow disregards type pruning among other features.
*/
predicate hasPartialFlowRev(PartialPathNode node, PartialPathNode sink, int dist) {
predicate partialFlowRev(PartialPathNode node, PartialPathNode sink, int dist) {
revPartialFlow(node, sink) and
dist = node.getSinkDistance()
}

8
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl1.qll
View File

@@ -1,5 +1,5 @@
/**
* DEPRECATED: Use `Make` and `MakeWithState` instead.
* DEPRECATED: Use `Global` and `GlobalWithState` instead.
*
* Provides a `Configuration` class backwards-compatible interface to the data
* flow library.
@@ -11,6 +11,7 @@ import DataFlowImplSpecific::Public
private import DataFlowImpl
import DataFlowImplCommonPublic
import FlowStateString
private import codeql.util.Unit
/**
* A configuration of interprocedural data flow analysis. This defines
@@ -328,7 +329,6 @@ private module Config implements FullStateConfigSig {
}
private import Impl<Config> as I
import I
/**
* A `Node` augmented with a call context (except for sinks), an access path, and a configuration.
@@ -379,6 +379,8 @@ class PathNode instanceof I::PathNode {
final predicate isSinkGroup(string group) { super.isSinkGroup(group) }
}
module PathGraph = I::PathGraph;
private predicate hasFlow(Node source, Node sink, Configuration config) {
exists(PathNode source0, PathNode sink0 |
hasFlowPath(source0, sink0, config) and
@@ -388,7 +390,7 @@ private predicate hasFlow(Node source, Node sink, Configuration config) {
}
private predicate hasFlowPath(PathNode source, PathNode sink, Configuration config) {
hasFlowPath(source, sink) and source.getConfiguration() = config
I::flowPath(source, sink) and source.getConfiguration() = config
}
private predicate hasFlowTo(Node sink, Configuration config) { hasFlow(_, sink, config) }

8
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl2.qll
View File

@@ -1,5 +1,5 @@
/**
* DEPRECATED: Use `Make` and `MakeWithState` instead.
* DEPRECATED: Use `Global` and `GlobalWithState` instead.
*
* Provides a `Configuration` class backwards-compatible interface to the data
* flow library.
@@ -11,6 +11,7 @@ import DataFlowImplSpecific::Public
private import DataFlowImpl
import DataFlowImplCommonPublic
import FlowStateString
private import codeql.util.Unit
/**
* A configuration of interprocedural data flow analysis. This defines
@@ -328,7 +329,6 @@ private module Config implements FullStateConfigSig {
}
private import Impl<Config> as I
import I
/**
* A `Node` augmented with a call context (except for sinks), an access path, and a configuration.
@@ -379,6 +379,8 @@ class PathNode instanceof I::PathNode {
final predicate isSinkGroup(string group) { super.isSinkGroup(group) }
}
module PathGraph = I::PathGraph;
private predicate hasFlow(Node source, Node sink, Configuration config) {
exists(PathNode source0, PathNode sink0 |
hasFlowPath(source0, sink0, config) and
@@ -388,7 +390,7 @@ private predicate hasFlow(Node source, Node sink, Configuration config) {
}
private predicate hasFlowPath(PathNode source, PathNode sink, Configuration config) {
hasFlowPath(source, sink) and source.getConfiguration() = config
I::flowPath(source, sink) and source.getConfiguration() = config
}
private predicate hasFlowTo(Node sink, Configuration config) { hasFlow(_, sink, config) }

8
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl3.qll
View File

@@ -1,5 +1,5 @@
/**
* DEPRECATED: Use `Make` and `MakeWithState` instead.
* DEPRECATED: Use `Global` and `GlobalWithState` instead.
*
* Provides a `Configuration` class backwards-compatible interface to the data
* flow library.
@@ -11,6 +11,7 @@ import DataFlowImplSpecific::Public
private import DataFlowImpl
import DataFlowImplCommonPublic
import FlowStateString
private import codeql.util.Unit
/**
* A configuration of interprocedural data flow analysis. This defines
@@ -328,7 +329,6 @@ private module Config implements FullStateConfigSig {
}
private import Impl<Config> as I
import I
/**
* A `Node` augmented with a call context (except for sinks), an access path, and a configuration.
@@ -379,6 +379,8 @@ class PathNode instanceof I::PathNode {
final predicate isSinkGroup(string group) { super.isSinkGroup(group) }
}
module PathGraph = I::PathGraph;
private predicate hasFlow(Node source, Node sink, Configuration config) {
exists(PathNode source0, PathNode sink0 |
hasFlowPath(source0, sink0, config) and
@@ -388,7 +390,7 @@ private predicate hasFlow(Node source, Node sink, Configuration config) {
}
private predicate hasFlowPath(PathNode source, PathNode sink, Configuration config) {
hasFlowPath(source, sink) and source.getConfiguration() = config
I::flowPath(source, sink) and source.getConfiguration() = config
}
private predicate hasFlowTo(Node sink, Configuration config) { hasFlow(_, sink, config) }

8
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl4.qll
View File

@@ -1,5 +1,5 @@
/**
* DEPRECATED: Use `Make` and `MakeWithState` instead.
* DEPRECATED: Use `Global` and `GlobalWithState` instead.
*
* Provides a `Configuration` class backwards-compatible interface to the data
* flow library.
@@ -11,6 +11,7 @@ import DataFlowImplSpecific::Public
private import DataFlowImpl
import DataFlowImplCommonPublic
import FlowStateString
private import codeql.util.Unit
/**
* A configuration of interprocedural data flow analysis. This defines
@@ -328,7 +329,6 @@ private module Config implements FullStateConfigSig {
}
private import Impl<Config> as I
import I
/**
* A `Node` augmented with a call context (except for sinks), an access path, and a configuration.
@@ -379,6 +379,8 @@ class PathNode instanceof I::PathNode {
final predicate isSinkGroup(string group) { super.isSinkGroup(group) }
}
module PathGraph = I::PathGraph;
private predicate hasFlow(Node source, Node sink, Configuration config) {
exists(PathNode source0, PathNode sink0 |
hasFlowPath(source0, sink0, config) and
@@ -388,7 +390,7 @@ private predicate hasFlow(Node source, Node sink, Configuration config) {
}
private predicate hasFlowPath(PathNode source, PathNode sink, Configuration config) {
hasFlowPath(source, sink) and source.getConfiguration() = config
I::flowPath(source, sink) and source.getConfiguration() = config
}
private predicate hasFlowTo(Node sink, Configuration config) { hasFlow(_, sink, config) }

18
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll
View File

@@ -140,10 +140,8 @@ private module LambdaFlow {
}
pragma[nomagic]
private TReturnPositionSimple viableReturnPosLambda(
DataFlowCall call, DataFlowCallOption lastCall, ReturnKind kind
) {
result = TReturnPositionSimple0(viableCallableLambda(call, lastCall), kind)
private TReturnPositionSimple viableReturnPosLambda(DataFlowCall call, ReturnKind kind) {
result = TReturnPositionSimple0(viableCallableLambda(call, _), kind)
}
private predicate viableReturnPosOutNonLambda(
@@ -155,11 +153,12 @@ private module LambdaFlow {
)
}
pragma[nomagic]
private predicate viableReturnPosOutLambda(
DataFlowCall call, DataFlowCallOption lastCall, TReturnPositionSimple pos, OutNode out
DataFlowCall call, TReturnPositionSimple pos, OutNode out
) {
exists(ReturnKind kind |
pos = viableReturnPosLambda(call, lastCall, kind) and
pos = viableReturnPosLambda(call, kind) and
out = getAnOutNode(call, kind)
)
}
@@ -188,6 +187,7 @@ private module LambdaFlow {
else any()
}
pragma[assume_small_delta]
pragma[nomagic]
predicate revLambdaFlow0(
DataFlowCall lambdaCall, LambdaCallKind kind, Node node, DataFlowType t, boolean toReturn,
@@ -274,6 +274,7 @@ private module LambdaFlow {
)
}
pragma[assume_small_delta]
pragma[nomagic]
predicate revLambdaFlowOut(
DataFlowCall lambdaCall, LambdaCallKind kind, TReturnPositionSimple pos, DataFlowType t,
@@ -285,7 +286,7 @@ private module LambdaFlow {
or
// non-linear recursion
revLambdaFlowOutLambdaCall(lambdaCall, kind, out, t, toJump, call, lastCall) and
viableReturnPosOutLambda(call, _, pos, out)
viableReturnPosOutLambda(call, pos, out)
)
}
@@ -424,7 +425,8 @@ private module Cached {
exists(ParameterPosition ppos |
viableParam(call, ppos, p) and
argumentPositionMatch(call, arg, ppos) and
compatibleTypes(getNodeDataFlowType(arg), getNodeDataFlowType(p))
compatibleTypes(getNodeDataFlowType(arg), getNodeDataFlowType(p)) and
golangSpecificParamArgFilter(call, p, arg)
)
}

8
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplForRegExp.qll
View File

@@ -1,5 +1,5 @@
/**
* DEPRECATED: Use `Make` and `MakeWithState` instead.
* DEPRECATED: Use `Global` and `GlobalWithState` instead.
*
* Provides a `Configuration` class backwards-compatible interface to the data
* flow library.
@@ -11,6 +11,7 @@ import DataFlowImplSpecific::Public
private import DataFlowImpl
import DataFlowImplCommonPublic
import FlowStateString
private import codeql.util.Unit
/**
* A configuration of interprocedural data flow analysis. This defines
@@ -328,7 +329,6 @@ private module Config implements FullStateConfigSig {
}
private import Impl<Config> as I
import I
/**
* A `Node` augmented with a call context (except for sinks), an access path, and a configuration.
@@ -379,6 +379,8 @@ class PathNode instanceof I::PathNode {
final predicate isSinkGroup(string group) { super.isSinkGroup(group) }
}
module PathGraph = I::PathGraph;
private predicate hasFlow(Node source, Node sink, Configuration config) {
exists(PathNode source0, PathNode sink0 |
hasFlowPath(source0, sink0, config) and
@@ -388,7 +390,7 @@ private predicate hasFlow(Node source, Node sink, Configuration config) {
}
private predicate hasFlowPath(PathNode source, PathNode sink, Configuration config) {
hasFlowPath(source, sink) and source.getConfiguration() = config
I::flowPath(source, sink) and source.getConfiguration() = config
}
private predicate hasFlowTo(Node sink, Configuration config) { hasFlow(_, sink, config) }

3
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplSpecific.qll
View File

@@ -7,9 +7,6 @@ private import python as Python
module Private {
import DataFlowPrivate
// import DataFlowDispatch
class Unit = Python::Unit;
}
module Public {

128
python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImpl.qll
View File

@@ -10,6 +10,7 @@ private import FlowSummaryImplSpecific
private import DataFlowImplSpecific::Private
private import DataFlowImplSpecific::Public
private import DataFlowImplCommon
private import codeql.util.Unit
/** Provides classes and predicates for defining flow summaries. */
module Public {
@@ -109,6 +110,7 @@ module Public {
}
/** Gets the stack obtained by dropping the first `i` elements, if any. */
pragma[assume_small_delta]
SummaryComponentStack drop(int i) {
i = 0 and result = this
or
@@ -213,6 +215,54 @@ module Public {
abstract predicate required(SummaryComponent head, SummaryComponentStack tail);
}
/**
* Gets the valid model origin values.
*/
private string getValidModelOrigin() {
result =
[
"ai", // AI (machine learning)
"df", // Dataflow (model generator)
"tb", // Type based (model generator)
"hq", // Heuristic query
]
}
/**
* A class used to represent provenance values for MaD models.
*
* The provenance value is a string of the form `origin-verification`
* (or just `manual`), where `origin` is a value indicating the
* origin of the model, and `verification` is a value indicating, how
* the model was verified.
*
* Examples could be:
* - `df-generated`: A model produced by the model generator, but not verified by a human.
* - `ai-manual`: A model produced by AI, but verified by a human.
*/
class Provenance extends string {
private string verification;
Provenance() {
exists(string origin | origin = getValidModelOrigin() |
this = origin + "-" + verification and
verification = ["manual", "generated"]
)
or
this = verification and verification = "manual"
}
/**
* Holds if this is a valid generated provenance value.
*/
predicate isGenerated() { verification = "generated" }
/**
* Holds if this is a valid manual provenance value.
*/
predicate isManual() { verification = "manual" }
}
/** A callable with a flow summary. */
abstract class SummarizedCallable extends SummarizedCallableBase {
bindingset[this]
@@ -246,41 +296,61 @@ module Public {
}
/**
* Holds if all the summaries that apply to `this` are auto generated and not manually created.
* Holds if there exists a generated summary that applies to this callable.
*/
final predicate isAutoGenerated() {
this.hasProvenance(["generated", "ai-generated"]) and not this.isManual()
final predicate hasGeneratedModel() {
exists(Provenance p | p.isGenerated() and this.hasProvenance(p))
}
/**
* Holds if there exists a manual summary that applies to `this`.
* Holds if all the summaries that apply to this callable are auto generated and not manually created.
* That is, only apply generated models, when there are no manual models.
*/
final predicate isManual() { this.hasProvenance("manual") }
final predicate applyGeneratedModel() {
this.hasGeneratedModel() and
not this.hasManualModel()
}
/**
* Holds if there exists a summary that applies to `this` that has provenance `provenance`.
* Holds if there exists a manual summary that applies to this callable.
*/
predicate hasProvenance(string provenance) { none() }
final predicate hasManualModel() {
exists(Provenance p | p.isManual() and this.hasProvenance(p))
}
/**
* Holds if there exists a manual summary that applies to this callable.
* Always apply manual models if they exist.
*/
final predicate applyManualModel() { this.hasManualModel() }
/**
* Holds if there exists a summary that applies to this callable
* that has provenance `provenance`.
*/
predicate hasProvenance(Provenance provenance) { provenance = "manual" }
}
/** A callable where there is no flow via the callable. */
class NeutralCallable extends SummarizedCallableBase {
NeutralCallable() { neutralElement(this, _) }
private Provenance provenance;
NeutralCallable() { neutralElement(this, provenance) }
/**
* Holds if the neutral is auto generated.
*/
predicate isAutoGenerated() { neutralElement(this, ["generated", "ai-generated"]) }
final predicate hasGeneratedModel() { provenance.isGenerated() }
/**
* Holds if there exists a manual neutral that applies to `this`.
* Holds if there exists a manual neutral that applies to this callable.
*/
final predicate isManual() { this.hasProvenance("manual") }
final predicate hasManualModel() { provenance.isManual() }
/**
* Holds if the neutral has provenance `provenance`.
* Holds if the neutral has provenance `p`.
*/
predicate hasProvenance(string provenance) { neutralElement(this, provenance) }
predicate hasProvenance(Provenance p) { p = provenance }
}
}
@@ -1015,12 +1085,18 @@ module Private {
private predicate relevantSummaryElementGenerated(
AccessPath inSpec, AccessPath outSpec, string kind
) {
summaryElement(this, inSpec, outSpec, kind, ["generated", "ai-generated"]) and
not summaryElement(this, _, _, _, "manual")
exists(Provenance provenance |
provenance.isGenerated() and
summaryElement(this, inSpec, outSpec, kind, provenance)
) and
not this.applyManualModel()
}
private predicate relevantSummaryElement(AccessPath inSpec, AccessPath outSpec, string kind) {
summaryElement(this, inSpec, outSpec, kind, "manual")
exists(Provenance provenance |
provenance.isManual() and
summaryElement(this, inSpec, outSpec, kind, provenance)
)
or
this.relevantSummaryElementGenerated(inSpec, outSpec, kind)
}
@@ -1039,7 +1115,7 @@ module Private {
)
}
override predicate hasProvenance(string provenance) {
override predicate hasProvenance(Provenance provenance) {
summaryElement(this, _, _, _, provenance)
}
}
@@ -1050,6 +1126,20 @@ module Private {
not exists(interpretComponent(c))
}
/** Holds if `provenance` is not a valid provenance value. */
bindingset[provenance]
predicate invalidProvenance(string provenance) { not provenance instanceof Provenance }
/**
* Holds if token `part` of specification `spec` has an invalid index.
* E.g., `Argument[-1]`.
*/
predicate invalidIndexComponent(AccessPath spec, AccessPathToken part) {
part = spec.getToken(_) and
part.getName() = ["Parameter", "Argument"] and
AccessPath::parseInt(part.getArgumentList()) < 0
}
private predicate inputNeedsReference(AccessPathToken c) {
c.getName() = "Argument" or
inputNeedsReferenceSpecific(c)
@@ -1207,11 +1297,11 @@ module Private {
}
private string renderProvenance(SummarizedCallable c) {
if c.isManual() then result = "manual" else c.hasProvenance(result)
if c.applyManualModel() then result = "manual" else c.hasProvenance(result)
}
private string renderProvenanceNeutral(NeutralCallable c) {
if c.isManual() then result = "manual" else c.hasProvenance(result)
if c.hasManualModel() then result = "manual" else c.hasProvenance(result)
}
/**

94
python/ql/lib/semmle/python/dataflow/new/internal/ImportResolution.qll
View File

@@ -65,31 +65,75 @@ private import semmle.python.dataflow.new.internal.DataFlowPrivate
*/
module ImportResolution {
/**
* Holds if the module `m` defines a name `name` by assigning `defn` to it. This is an
* overapproximation, as `name` may not in fact be exported (e.g. by defining an `__all__` that does
* not include `name`).
* Holds if there is an ESSA step from `defFrom` to `defTo`, which should be allowed
* for import resolution.
*/
private predicate allowedEssaImportStep(EssaDefinition defFrom, EssaDefinition defTo) {
// to handle definitions guarded by if-then-else
defFrom = defTo.(PhiFunction).getAnInput()
or
// refined variable
// example: https://github.com/nvbn/thefuck/blob/ceeaeab94b5df5a4fe9d94d61e4f6b0bbea96378/thefuck/utils.py#L25-L45
defFrom = defTo.(EssaNodeRefinement).getInput().getDefinition()
}
/**
* Holds if the module `m` defines a name `name` with the value `val`. The value
* represents the value `name` will have at the end of the module (the last place we
* have def-use flow to).
*
* Note: The handling of re-exporting imports is a bit simplistic. We assume that if
* an import is made, it will be re-exported (which will not be the case if a new
* value is assigned to the name, or it is deleted).
*/
pragma[nomagic]
predicate module_export(Module m, string name, DataFlow::CfgNode defn) {
exists(EssaVariable v, EssaDefinition essaDef |
v.getName() = name and
v.getAUse() = ImportStar::getStarImported*(m).getANormalExit() and
(
essaDef = v.getDefinition()
or
// to handle definitions guarded by if-then-else
essaDef = v.getDefinition().(PhiFunction).getAnInput()
)
predicate module_export(Module m, string name, DataFlow::Node val) {
// Definitions made inside `m` itself
//
// for code such as `foo = ...; foo.bar = ...` there will be TWO
// EssaDefinition/EssaVariable. One for `foo = ...` (AssignmentDefinition) and one
// for `foo.bar = ...`. The one for `foo.bar = ...` (EssaNodeRefinement). The
// EssaNodeRefinement is the one that will reach the end of the module (normal
// exit).
//
// However, we cannot just use the EssaNodeRefinement as the `val`, because the
// normal data-flow depends on use-use flow, and use-use flow targets CFG nodes not
// EssaNodes. So we need to go back from the EssaDefinition/EssaVariable that
// reaches the end of the module, to the first definition of the variable, and then
// track forwards using use-use flow to find a suitable CFG node that has flow into
// it from use-use flow.
exists(EssaVariable lastUseVar, EssaVariable firstDef |
lastUseVar.getName() = name and
// we ignore special variable $ introduced by our analysis (not used for anything)
// we ignore special variable * introduced by `from <pkg> import *` -- TODO: understand why we even have this?
not name in ["$", "*"] and
lastUseVar.getAUse() = m.getANormalExit() and
allowedEssaImportStep*(firstDef, lastUseVar) and
not allowedEssaImportStep(_, firstDef)
|
defn.getNode() = essaDef.(AssignmentDefinition).getValue()
not EssaFlow::defToFirstUse(firstDef, _) and
val.asVar() = firstDef
or
defn.getNode() = essaDef.(ArgumentRefinement).getArgument()
exists(ControlFlowNode mid, ControlFlowNode end |
EssaFlow::defToFirstUse(firstDef, mid) and
EssaFlow::useToNextUse*(mid, end) and
not EssaFlow::useToNextUse(end, _) and
val.asCfgNode() = end
)
)
or
// re-exports from `from <pkg> import *`
exists(Module importedFrom |
importedFrom = ImportStar::getStarImported(m) and
module_export(importedFrom, name, val) and
potential_module_export(importedFrom, name)
)
or
// re-exports from `import <pkg>` or `from <pkg> import <stuff>`
exists(Alias a |
defn.asExpr() = [a.getValue(), a.getValue().(ImportMember).getModule()] and
val.asExpr() = a.getValue() and
a.getAsname().(Name).getId() = name and
defn.getScope() = m
val.getScope() = m
)
}
@@ -263,9 +307,21 @@ module ImportResolution {
module_reexport(reexporter, attr_name, m)
)
or
// Submodules that are implicitly defined with relative imports of the form `from .foo import ...`.
// In practice, we create a definition for each module in a package, even if it is not imported.
// submodules of packages will be available as `<pkg>.<submodule>` after doing
// `import <pkg>.<submodule>` at least once in the program, or can be directly
// imported with `from <pkg> import <submodule>` (even with an empty
// `<pkg>.__init__` file).
//
// Until an import of `<pkg>.<submodule>` is executed, it is technically possible
// that `<pkg>.<submodule>` (or `from <pkg> import <submodule>`) can refer to an
// attribute set in `<pkg>.__init__`.
//
// Therefore, if there is an attribute defined in `<pkg>.__init__` with the same
// name as a submodule, we always consider that this attribute _could_ be a
// reference to the submodule, even if we don't know that the submodule has been
// imported yet.
exists(string submodule, Module package |
submodule = result.asVar().getName() and
SsaSource::init_module_submodule_defn(result.asVar().getSourceVariable(),
package.getEntryNode()) and
m = getModuleFromName(package.getPackageName() + "." + submodule)

18
python/ql/lib/semmle/python/dataflow/new/internal/tainttracking1/TaintTracking.qll
View File

@@ -33,9 +33,9 @@ private module AddTaintDefaults<DataFlowInternal::FullStateConfigSig Config> imp
}
/**
* Constructs a standard taint tracking computation.
* Constructs a global taint tracking computation.
*/
module Make<DataFlow::ConfigSig Config> implements DataFlow::DataFlowSig {
module Global<DataFlow::ConfigSig Config> implements DataFlow::GlobalFlowSig {
private module Config0 implements DataFlowInternal::FullStateConfigSig {
import DataFlowInternal::DefaultState<Config>
import Config
@@ -48,10 +48,15 @@ module Make<DataFlow::ConfigSig Config> implements DataFlow::DataFlowSig {
import DataFlowInternal::Impl<C>
}
/** DEPRECATED: Use `Global` instead. */
deprecated module Make<DataFlow::ConfigSig Config> implements DataFlow::GlobalFlowSig {
import Global<Config>
}
/**
* Constructs a taint tracking computation using flow state.
* Constructs a global taint tracking computation using flow state.
*/
module MakeWithState<DataFlow::StateConfigSig Config> implements DataFlow::DataFlowSig {
module GlobalWithState<DataFlow::StateConfigSig Config> implements DataFlow::GlobalFlowSig {
private module Config0 implements DataFlowInternal::FullStateConfigSig {
import Config
}
@@ -62,3 +67,8 @@ module MakeWithState<DataFlow::StateConfigSig Config> implements DataFlow::DataF
import DataFlowInternal::Impl<C>
}
/** DEPRECATED: Use `GlobalWithState` instead. */
deprecated module MakeWithState<DataFlow::StateConfigSig Config> implements DataFlow::GlobalFlowSig {
import GlobalWithState<Config>
}

53
python/ql/lib/semmle/python/frameworks/Aiomysql.qll
View File

@@ -9,11 +9,10 @@ private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.PEP249
/** Provides models for the `aiomysql` PyPI package. */
private module Aiomysql {
private import semmle.python.internal.Awaited
/**
* Gets a `ConnectionPool` that is created when the result of `aiomysql.create_pool()` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/pool.html
@@ -23,49 +22,29 @@ private module Aiomysql {
}
/**
* Gets a `Connection` that is created when
* A Connection that is created when
* - the result of `aiomysql.connect()` is awaited.
* - the result of calling `acquire` on a `ConnectionPool` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/connection.html#connection
* See
* - https://aiomysql.readthedocs.io/en/stable/connection.html#connection
* - https://aiomysql.readthedocs.io/en/stable/pool.html#Pool.acquire
*/
API::Node connection() {
result = API::moduleImport("aiomysql").getMember("connect").getReturn().getAwaited()
or
result = connectionPool().getMember("acquire").getReturn().getAwaited()
class AiomysqlConnection extends PEP249::AsyncDatabaseConnection {
AiomysqlConnection() {
this = API::moduleImport("aiomysql").getMember("connect").getReturn().getAwaited()
or
this = connectionPool().getMember("acquire").getReturn().getAwaited()
}
}
/**
* Gets a `Cursor` that is created when
* An additional cursor, that is created when
* - the result of calling `cursor` on a `ConnectionPool` is awaited.
* - the result of calling `cursor` on a `Connection` is awaited.
* See https://aiomysql.readthedocs.io/en/stable/cursors.html
* See
* - https://aiomysql.readthedocs.io/en/stable/pool.html##Pool.cursor
*/
API::Node cursor() {
result = connectionPool().getMember("cursor").getReturn().getAwaited()
or
result = connection().getMember("cursor").getReturn().getAwaited()
}
/**
* A query. Calling `execute` on a `Cursor` constructs a query.
* See https://aiomysql.readthedocs.io/en/stable/cursors.html#Cursor.execute
*/
class CursorExecuteCall extends SqlConstruction::Range, API::CallNode {
CursorExecuteCall() { this = cursor().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "operation").asSink() }
}
/**
* An awaited query. Awaiting the result of calling `execute` executes the query.
* See https://aiomysql.readthedocs.io/en/stable/cursors.html#Cursor.execute
*/
class AwaitedCursorExecuteCall extends SqlExecution::Range {
CursorExecuteCall executeCall;
AwaitedCursorExecuteCall() { this = executeCall.getReturn().getAwaited().asSource() }
override DataFlow::Node getSql() { result = executeCall.getSql() }
class AiomysqlCursor extends PEP249::AsyncDatabaseCursor {
AiomysqlCursor() { this = connectionPool().getMember("cursor").getReturn().getAwaited() }
}
/**

53
python/ql/lib/semmle/python/frameworks/Aiopg.qll
View File

@@ -9,11 +9,10 @@ private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.PEP249
/** Provides models for the `aiopg` PyPI package. */
private module Aiopg {
private import semmle.python.internal.Awaited
/**
* Gets a `ConnectionPool` that is created when the result of `aiopg.create_pool()` is awaited.
* See https://aiopg.readthedocs.io/en/stable/core.html#pool
@@ -23,49 +22,29 @@ private module Aiopg {
}
/**
* Gets a `Connection` that is created when
* A Connection that is created when
* - the result of `aiopg.connect()` is awaited.
* - the result of calling `acquire` on a `ConnectionPool` is awaited.
* See https://aiopg.readthedocs.io/en/stable/core.html#connection
* See
* - https://aiopg.readthedocs.io/en/stable/core.html#connection
* - https://aiopg.readthedocs.io/en/stable/core.html#aiopg.Pool.acquire
*/
API::Node connection() {
result = API::moduleImport("aiopg").getMember("connect").getReturn().getAwaited()
or
result = connectionPool().getMember("acquire").getReturn().getAwaited()
class AiopgConnection extends PEP249::AsyncDatabaseConnection {
AiopgConnection() {
this = API::moduleImport("aiopg").getMember("connect").getReturn().getAwaited()
or
this = connectionPool().getMember("acquire").getReturn().getAwaited()
}
}
/**
* Gets a `Cursor` that is created when
* An additional cursor, that is created when
* - the result of calling `cursor` on a `ConnectionPool` is awaited.
* - the result of calling `cursor` on a `Connection` is awaited.
* See https://aiopg.readthedocs.io/en/stable/core.html#cursor
* See
* - https://aiopg.readthedocs.io/en/stable/core.html#aiopg.Pool.cursor
*/
API::Node cursor() {
result = connectionPool().getMember("cursor").getReturn().getAwaited()
or
result = connection().getMember("cursor").getReturn().getAwaited()
}
/**
* A query. Calling `execute` on a `Cursor` constructs a query.
* See https://aiopg.readthedocs.io/en/stable/core.html#aiopg.Cursor.execute
*/
class CursorExecuteCall extends SqlConstruction::Range, API::CallNode {
CursorExecuteCall() { this = cursor().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "operation").asSink() }
}
/**
* An awaited query. Awaiting the result of calling `execute` executes the query.
* See https://aiopg.readthedocs.io/en/stable/core.html#aiopg.Cursor.execute
*/
class AwaitedCursorExecuteCall extends SqlExecution::Range {
CursorExecuteCall execute;
AwaitedCursorExecuteCall() { this = execute.getReturn().getAwaited().asSource() }
override DataFlow::Node getSql() { result = execute.getSql() }
class AiopgCursor extends PEP249::AsyncDatabaseCursor {
AiopgCursor() { this = connectionPool().getMember("cursor").getReturn().getAwaited() }
}
/**

39
python/ql/lib/semmle/python/frameworks/Aiosqlite.qll Normal file
View File

@@ -0,0 +1,39 @@
/**
* Provides classes modeling security-relevant aspects of the `aiosqlite` PyPI package.
* See
* - https://pypi.org/project/aiosqlite/
*/
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.PEP249
/** Provides models for the `aiosqlite` PyPI package. */
private module Aiosqlite {
/**
* A model of `aiosqlite` as a module that implements PEP 249 using asyncio, providing
* ways to execute SQL statements against a database.
*/
class AiosqlitePEP249 extends PEP249::AsyncPEP249ModuleApiNode {
AiosqlitePEP249() { this = API::moduleImport("aiosqlite") }
}
/**
* An additional cursor, that is return from the coroutine Connection.execute,
* see https://aiosqlite.omnilib.dev/en/latest/api.html#aiosqlite.Connection.execute
*/
class AiosqliteCursor extends PEP249::AsyncDatabaseCursor {
AiosqliteCursor() {
this =
API::moduleImport("aiosqlite")
.getMember("connect")
.getReturn()
.getAwaited()
.getMember("execute")
.getReturn()
.getAwaited()
}
}
}

1
python/ql/lib/semmle/python/frameworks/Asyncpg.qll
View File

@@ -22,6 +22,7 @@ private module Asyncpg {
// * - the result of `asyncpg.connect()` is awaited.
// * - the result of calling `acquire` on a `ConnectionPool` is awaited.
"asyncpg.Connection;asyncpg;Member[connect].ReturnValue.Awaited",
"asyncpg.Connection;asyncpg;Member[connection].Member[connect].ReturnValue.Awaited",
"asyncpg.Connection;asyncpg.ConnectionPool;Member[acquire].ReturnValue.Awaited",
// Creating an internal `~Connection` type that contains both `Connection` and `ConnectionPool`.
"asyncpg.~Connection;asyncpg.Connection;", //

61
python/ql/lib/semmle/python/frameworks/CassandraDriver.qll Normal file
View File

@@ -0,0 +1,61 @@
/**
* Provides classes modeling security-relevant aspects of the `cassandra-driver` PyPI package.
* See https://pypi.org/project/cassandra-driver/
*/
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.RemoteFlowSources
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.PEP249
/**
* Provides models for the `cassandra-driver` PyPI package.
* See https://pypi.org/project/cassandra-driver/
*/
private module CassandraDriver {
/**
* A cassandra cluster session.
*
* see
* - https://docs.datastax.com/en/developer/python-driver/3.25/api/cassandra/cluster/#cassandra.cluster.Cluster.connect
* - https://docs.datastax.com/en/developer/python-driver/3.25/api/cassandra/cluster/#cassandra.cluster.Session
*/
API::Node session() {
result =
API::moduleImport("cassandra")
.getMember("cluster")
.getMember("Cluster")
.getReturn()
.getMember("connect")
.getReturn()
}
/**
* see https://docs.datastax.com/en/developer/python-driver/3.25/api/cassandra/cluster/#cassandra.cluster.Session.execute
*/
class CassandraSessionExecuteCall extends SqlExecution::Range, API::CallNode {
CassandraSessionExecuteCall() { this = session().getMember("execute").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").asSink() }
}
/**
* see https://docs.datastax.com/en/developer/python-driver/3.25/api/cassandra/cluster/#cassandra.cluster.Session.execute_async
*/
class CassandraSessionExecuteAsyncCall extends SqlConstruction::Range, API::CallNode {
CassandraSessionExecuteAsyncCall() { this = session().getMember("execute_async").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").asSink() }
}
/**
* see https://docs.datastax.com/en/developer/python-driver/3.25/api/cassandra/cluster/#cassandra.cluster.Session.prepare
*/
class CassandraSessionPrepareCall extends SqlConstruction::Range, API::CallNode {
CassandraSessionPrepareCall() { this = session().getMember("prepare").getACall() }
override DataFlow::Node getSql() { result = this.getParameter(0, "query").asSink() }
}
}

4
python/ql/lib/semmle/python/frameworks/Django.qll
View File

@@ -561,8 +561,8 @@ module PrivateDjango {
API::Node connection() { result = db().getMember("connection") }
/** A `django.db.connection` is a PEP249 compliant DB connection. */
class DjangoDbConnection extends PEP249::Connection::InstanceSource {
DjangoDbConnection() { this = connection().asSource() }
class DjangoDbConnection extends PEP249::DatabaseConnection {
DjangoDbConnection() { this = connection() }
}
// -------------------------------------------------------------------------

156
python/ql/lib/semmle/python/frameworks/PEP249.qll
View File

@@ -22,6 +22,148 @@ module PEP249 {
override string toString() { result = this.(API::Node).toString() }
}
/**
* An API graph node representing a database connection.
*/
abstract class DatabaseConnection extends API::Node {
/** Gets a string representation of this element. */
override string toString() { result = this.(API::Node).toString() }
}
private class DefaultDatabaseConnection extends DatabaseConnection {
DefaultDatabaseConnection() {
this = any(PEP249ModuleApiNode mod).getMember("connect").getReturn()
}
}
/**
* An API graph node representing a database cursor.
*/
abstract class DatabaseCursor extends API::Node {
/** Gets a string representation of this element. */
override string toString() { result = this.(API::Node).toString() }
}
private class DefaultDatabaseCursor extends DatabaseCursor {
DefaultDatabaseCursor() { this = any(DatabaseConnection conn).getMember("cursor").getReturn() }
}
private string getSqlKwargName() {
result in ["sql", "statement", "operation", "query", "query_string", "sql_script"]
}
private string getExecuteMethodName() {
result in ["execute", "executemany", "executescript", "execute_insert", "execute_fetchall"]
}
/**
* A call to an execute method on a database cursor or a connection, such as `execute`
* or `executemany`.
*
* See
* - https://peps.python.org/pep-0249/#execute
* - https://peps.python.org/pep-0249/#executemany
*
* Note: While `execute` method on a connection is not part of PEP249, if it is used, we
* recognize it as an alias for constructing a cursor and calling `execute` on it.
*/
private class ExecuteMethodCall extends SqlExecution::Range, API::CallNode {
ExecuteMethodCall() {
exists(API::Node start |
start instanceof DatabaseCursor or start instanceof DatabaseConnection
|
this = start.getMember(getExecuteMethodName()).getACall()
)
}
override DataFlow::Node getSql() {
result in [this.getArg(0), this.getArgByName(getSqlKwargName()),]
}
}
// ---------------------------------------------------------------------------
// asyncio implementations
// ---------------------------------------------------------------------------
//
// we differentiate between normal and asyncio implementations, since we model the
// `execute` call differently -- as a SqlExecution vs SqlConstruction, since the SQL
// is only executed in asyncio after being awaited (which might happen in something
// like `asyncio.gather`)
/**
* An API graph node representing a module that implements PEP 249 using asyncio.
*/
abstract class AsyncPEP249ModuleApiNode extends API::Node {
/** Gets a string representation of this element. */
override string toString() { result = this.(API::Node).toString() }
}
/**
* An API graph node representing a asyncio database connection (after being awaited).
*/
abstract class AsyncDatabaseConnection extends API::Node {
/** Gets a string representation of this element. */
override string toString() { result = this.(API::Node).toString() }
}
private class DefaultAsyncDatabaseConnection extends AsyncDatabaseConnection {
DefaultAsyncDatabaseConnection() {
this = any(AsyncPEP249ModuleApiNode mod).getMember("connect").getReturn().getAwaited()
}
}
/**
* An API graph node representing a asyncio database cursor (after being awaited).
*/
abstract class AsyncDatabaseCursor extends API::Node {
/** Gets a string representation of this element. */
override string toString() { result = this.(API::Node).toString() }
}
private class DefaultAsyncDatabaseCursor extends AsyncDatabaseCursor {
DefaultAsyncDatabaseCursor() {
this = any(AsyncDatabaseConnection conn).getMember("cursor").getReturn().getAwaited()
}
}
/**
* A call to an execute method on an asyncio database cursor or an asyncio connection,
* such as `execute` or `executemany`.
*
* (This is not an SqlExecution, since that only happens when the coroutine is
* awaited)
*
* See ExecuteMethodCall for more details.
*/
private class AsyncExecuteMethodCall extends SqlConstruction::Range, API::CallNode {
AsyncExecuteMethodCall() {
exists(API::Node start |
start instanceof AsyncDatabaseCursor or start instanceof AsyncDatabaseConnection
|
this = start.getMember(getExecuteMethodName()).getACall()
)
}
override DataFlow::Node getSql() {
result in [this.getArg(0), this.getArgByName(getSqlKwargName()),]
}
}
/** Actual execution of the AsyncExecuteMethodCall coroutine. */
private class AwaitedAsyncExecuteMethodCall extends SqlExecution::Range {
AsyncExecuteMethodCall execute;
AwaitedAsyncExecuteMethodCall() { this = execute.getReturn().getAwaited().asSource() }
override DataFlow::Node getSql() { result = execute.getSql() }
}
// ---------------------------------------------------------------------------
// old impl
// ---------------------------------------------------------------------------
// the goal is to deprecate it in favour of the API graph version, but currently this
// requires a rewrite of the Peewee modeling, which depends on rewriting the
// instance/instance-source stuff to use API graphs instead.
// so is postponed for now.
/** Gets a reference to the `connect` function of a module that implements PEP 249. */
DataFlow::Node connect() {
result = any(PEP249ModuleApiNode a).getMember("connect").getAValueReachableFromSource()
@@ -147,7 +289,10 @@ module PEP249 {
* recognize it as an alias for constructing a cursor and calling `execute` on it.
*/
private class ExecuteCall extends SqlExecution::Range, DataFlow::CallCfgNode {
ExecuteCall() { this.getFunction() = execute() }
ExecuteCall() {
this.getFunction() = execute() and
not this instanceof ExecuteMethodCall
}
override DataFlow::Node getSql() { result in [this.getArg(0), this.getArgByName("sql")] }
}
@@ -170,8 +315,13 @@ module PEP249 {
* recognize it as an alias for constructing a cursor and calling `executemany` on it.
*/
private class ExecutemanyCall extends SqlExecution::Range, DataFlow::CallCfgNode {
ExecutemanyCall() { this.getFunction() = executemany() }
ExecutemanyCall() {
this.getFunction() = executemany() and
not this instanceof ExecuteMethodCall
}
override DataFlow::Node getSql() { result in [this.getArg(0), this.getArgByName("sql")] }
override DataFlow::Node getSql() {
result in [this.getArg(0), this.getArgByName(getSqlKwargName())]
}
}
}

10
python/ql/lib/semmle/python/frameworks/Peewee.qll
View File

@@ -163,11 +163,9 @@ private module Peewee {
* A call to the `connection` method on a `peewee.Database` instance.
* https://docs.peewee-orm.com/en/latest/peewee/api.html#Database.connection.
*/
class PeeweeDatabaseConnectionCall extends PEP249::Connection::InstanceSource,
DataFlow::CallCfgNode
{
class PeeweeDatabaseConnectionCall extends PEP249::DatabaseConnection {
PeeweeDatabaseConnectionCall() {
this = Database::instance().getMember("connection").getACall()
this = Database::instance().getMember("connection").getReturn()
}
}
@@ -175,8 +173,8 @@ private module Peewee {
* A call to the `cursor` method on a `peewee.Database` instance.
* https://docs.peewee-orm.com/en/latest/peewee/api.html#Database.cursor.
*/
class PeeweeDatabaseCursorCall extends PEP249::Cursor::InstanceSource, DataFlow::CallCfgNode {
PeeweeDatabaseCursorCall() { this = Database::instance().getMember("cursor").getACall() }
class PeeweeDatabaseCursorCall extends PEP249::DatabaseCursor {
PeeweeDatabaseCursorCall() { this = Database::instance().getMember("cursor").getReturn() }
}
/**

7
python/ql/lib/semmle/python/frameworks/Stdlib.qll
View File

@@ -2435,9 +2435,14 @@ private module StdlibPrivate {
* against a database.
*
* See https://devdocs.io/python~3.9/library/sqlite3
* https://github.com/python/cpython/blob/3.11/Lib/sqlite3/dbapi2.py
*/
class Sqlite3 extends PEP249::PEP249ModuleApiNode {
Sqlite3() { this = API::moduleImport("sqlite3") }
Sqlite3() {
this = API::moduleImport("sqlite3")
or
this = API::moduleImport("sqlite3").getMember("dbapi2")
}
}
// ---------------------------------------------------------------------------

3
python/ql/lib/semmle/python/frameworks/data/internal/ApiGraphModels.qll
View File

@@ -63,10 +63,9 @@
* the type is not intended to match a static type.
*/
private import codeql.util.Unit
private import ApiGraphModelsSpecific as Specific
private class Unit = Specific::Unit;
private module API = Specific::API;
private module DataFlow = Specific::DataFlow;

3
python/ql/lib/semmle/python/frameworks/data/internal/ApiGraphModelsSpecific.qll
View File

@@ -22,9 +22,6 @@
private import python as PY
private import ApiGraphModels
import semmle.python.ApiGraphs::API as API
class Unit = PY::Unit;
// Re-export libraries needed by ApiGraphModels.qll
import semmle.python.dataflow.new.internal.AccessPathSyntax as AccessPathSyntax
import semmle.python.dataflow.new.DataFlow::DataFlow as DataFlow

6
python/ql/lib/semmle/python/regex.qll
View File

@@ -140,7 +140,11 @@ string mode_from_node(DataFlow::Node node) { node = re_flag_tracker(result) }
/** A StrConst used as a regular expression */
abstract class RegexString extends Expr {
RegexString() { (this instanceof Bytes or this instanceof Unicode) }
RegexString() {
(this instanceof Bytes or this instanceof Unicode) and
// is part of the user code
exists(this.getLocation().getFile().getRelativePath())
}
/**
* Helper predicate for `char_set_start(int start, int end)`.

16
python/ql/src/CHANGELOG.md
View File

@@ -1,3 +1,19 @@
## 0.7.0
### Bug Fixes
* Nonlocal variables are excluded from alerts.
## 0.6.6
No user-facing changes.
## 0.6.5
### New Queries
* Added a new query, `py/shell-command-constructed-from-input`, to detect libraries that unsafely construct shell commands from their inputs.
## 0.6.4
No user-facing changes.

57
python/ql/src/Expressions/IncorrectComparisonUsingIs.ql
View File

@@ -11,16 +11,57 @@
*/
import python
import IsComparisons
from Compare comp, Cmpop op, ClassValue c, string alt
where
invalid_portable_is_comparison(comp, op, c) and
not cpython_interned_constant(comp.getASubExpression()) and
(
op instanceof Is and alt = "=="
/** Holds if the comparison `comp` uses `is` or `is not` (represented as `op`) to compare its `left` and `right` arguments. */
predicate comparison_using_is(Compare comp, ControlFlowNode left, Cmpop op, ControlFlowNode right) {
exists(CompareNode fcomp | fcomp = comp.getAFlowNode() |
fcomp.operands(left, op, right) and
(op instanceof Is or op instanceof IsNot)
)
}
private predicate cpython_interned_value(Expr e) {
exists(string text | text = e.(StrConst).getText() |
text.length() = 0
or
op instanceof IsNot and alt = "!="
text.length() = 1 and text.regexpMatch("[U+0000-U+00ff]")
)
or
exists(int i | i = e.(IntegerLiteral).getN().toInt() | -5 <= i and i <= 256)
or
exists(Tuple t | t = e and not exists(t.getAnElt()))
}
predicate uninterned_literal(Expr e) {
(
e instanceof StrConst
or
e instanceof IntegerLiteral
or
e instanceof FloatLiteral
or
e instanceof Dict
or
e instanceof List
or
e instanceof Tuple
) and
not cpython_interned_value(e)
}
from Compare comp, Cmpop op, string alt
where
exists(ControlFlowNode left, ControlFlowNode right |
comparison_using_is(comp, left, op, right) and
(
op instanceof Is and alt = "=="
or
op instanceof IsNot and alt = "!="
)
|
uninterned_literal(left.getNode())
or
uninterned_literal(right.getNode())
)
select comp,
"Values compared using '" + op.getSymbol() +

34
python/ql/src/Security/CWE-327/FluentApiModel.qll
View File

@@ -4,23 +4,18 @@ import TlsLibraryModel
/**
* Configuration to determine the state of a context being used to create
* a connection. There is one configuration for each pair of `TlsLibrary` and `ProtocolVersion`,
* such that a single configuration only tracks contexts where a specific `ProtocolVersion` is allowed.
* a connection. The configuration uses a flow state to track the `TlsLibrary`
* and the insecure `ProtocolVersion`s that are allowed.
*
* The state is in terms of whether a specific protocol is allowed. This is
* either true or false when the context is created and can then be modified
* later by either restricting or unrestricting the protocol (see the predicates
* `isRestriction` and `isUnrestriction`).
* later by either restricting or unrestricting the protocol (see the predicate
* `isAdditionalFlowStep`).
*
* Since we are interested in the final state, we want the flow to start from
* the last unrestriction, so we disallow flow into unrestrictions. We also
* model the creation as an unrestriction of everything it allows, to account
* for the common case where the creation plays the role of "last unrestriction".
*
* Since we really want "the last unrestriction, not nullified by a restriction",
* we also disallow flow into restrictions.
* The state is represented as a bit vector, where each bit corresponds to a
* protocol version. The bit is set if the protocol is allowed.
*/
module InsecureContextConfiguration2 implements DataFlow::StateConfigSig {
module InsecureContextConfiguration implements DataFlow::StateConfigSig {
private newtype TFlowState =
TMkFlowState(TlsLibrary library, int bits) {
bits in [0 .. max(any(ProtocolVersion v).getBit()) * 2 - 1]
@@ -61,9 +56,14 @@ module InsecureContextConfiguration2 implements DataFlow::StateConfigSig {
}
predicate isSource(DataFlow::Node source, FlowState state) {
exists(ProtocolFamily family |
source = state.getLibrary().unspecific_context_creation(family) and
state.getBits() = family.getBits()
exists(ContextCreation creation | source = creation |
creation = state.getLibrary().unspecific_context_creation() and
state.getBits() =
sum(ProtocolVersion version |
version = creation.getProtocol() and version.isInsecure()
|
version.getBit()
)
)
}
@@ -112,7 +112,7 @@ module InsecureContextConfiguration2 implements DataFlow::StateConfigSig {
}
}
private module InsecureContextFlow = DataFlow::MakeWithState<InsecureContextConfiguration2>;
private module InsecureContextFlow = DataFlow::GlobalWithState<InsecureContextConfiguration>;
/**
* Holds if `conectionCreation` marks the creation of a connection based on the contex
@@ -127,7 +127,7 @@ predicate unsafe_connection_creation_with_context(
) {
// Connection created from a context allowing `insecure_version`.
exists(InsecureContextFlow::PathNode src, InsecureContextFlow::PathNode sink |
InsecureContextFlow::hasFlowPath(src, sink) and
InsecureContextFlow::flowPath(src, sink) and
src.getNode() = contextOrigin and
sink.getNode() = connectionCreation and
sink.getState().allowsInsecureVersion(insecure_version) and

34
python/ql/src/Security/CWE-327/PyOpenSSL.qll
View File

@@ -12,14 +12,14 @@ class PyOpenSslContextCreation extends ContextCreation, DataFlow::CallCfgNode {
this = API::moduleImport("OpenSSL").getMember("SSL").getMember("Context").getACall()
}
override string getProtocol() {
override ProtocolVersion getProtocol() {
exists(DataFlow::Node protocolArg, PyOpenSsl pyo |
protocolArg in [this.getArg(0), this.getArgByName("method")]
|
protocolArg in [
pyo.specific_version(result).getAValueReachableFromSource(),
pyo.unspecific_version(result).getAValueReachableFromSource()
]
protocolArg = pyo.specific_version(result).getAValueReachableFromSource()
or
protocolArg = pyo.unspecific_version().getAValueReachableFromSource() and
result = any(ProtocolVersion pv)
)
}
}
@@ -51,19 +51,23 @@ class SetOptionsCall extends ProtocolRestriction, DataFlow::CallCfgNode {
}
}
class UnspecificPyOpenSslContextCreation extends PyOpenSslContextCreation, UnspecificContextCreation
{
// UnspecificPyOpenSslContextCreation() { library instanceof PyOpenSsl }
}
class PyOpenSsl extends TlsLibrary {
PyOpenSsl() { this = "pyOpenSSL" }
override string specific_version_name(ProtocolVersion version) { result = version + "_METHOD" }
override string unspecific_version_name(ProtocolFamily family) {
// `"TLS_METHOD"` is not actually available in pyOpenSSL yet, but should be coming soon..
result = family + "_METHOD"
override string unspecific_version_name() {
// See
// - https://www.pyopenssl.org/en/23.0.0/api/ssl.html#module-OpenSSL.SSL
// - https://www.openssl.org/docs/manmaster/man3/DTLS_server_method.html#NOTES
//
// PyOpenSSL also allows DTLS
// see https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context
// although they are not mentioned here:
// https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.TLS_METHOD
result = ["TLS", "SSLv23"] + "_METHOD"
or
result = "TLS_" + ["CLIENT", "SERVER"] + "_METHOD"
}
override API::Node version_constants() { result = API::moduleImport("OpenSSL").getMember("SSL") }
@@ -80,7 +84,5 @@ class PyOpenSsl extends TlsLibrary {
override ProtocolRestriction protocol_restriction() { result instanceof SetOptionsCall }
override ProtocolUnrestriction protocol_unrestriction() {
result instanceof UnspecificPyOpenSslContextCreation
}
override ProtocolUnrestriction protocol_unrestriction() { none() }
}

59
python/ql/src/Security/CWE-327/Ssl.qll
View File

@@ -10,20 +10,21 @@ import TlsLibraryModel
class SslContextCreation extends ContextCreation, DataFlow::CallCfgNode {
SslContextCreation() { this = API::moduleImport("ssl").getMember("SSLContext").getACall() }
override string getProtocol() {
override ProtocolVersion getProtocol() {
exists(DataFlow::Node protocolArg, Ssl ssl |
protocolArg in [this.getArg(0), this.getArgByName("protocol")]
|
protocolArg =
[
ssl.specific_version(result).getAValueReachableFromSource(),
ssl.unspecific_version(result).getAValueReachableFromSource()
]
protocolArg = ssl.specific_version(result).getAValueReachableFromSource()
or
protocolArg = ssl.unspecific_version().getAValueReachableFromSource() and
// see https://docs.python.org/3/library/ssl.html#id7
result in ["TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"]
)
or
not exists(this.getArg(_)) and
not exists(this.getArgByName(_)) and
result = "TLS"
// see https://docs.python.org/3/library/ssl.html#id7
result in ["TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"]
}
}
@@ -34,7 +35,7 @@ class SslDefaultContextCreation extends ContextCreation {
// Allowed insecure versions are "TLSv1" and "TLSv1_1"
// see https://docs.python.org/3/library/ssl.html#context-creation
override string getProtocol() { result = "TLS" }
override ProtocolVersion getProtocol() { result in ["TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"] }
}
/** Gets a reference to an `ssl.Context` instance. */
@@ -161,33 +162,29 @@ class ContextSetVersion extends ProtocolRestriction, ProtocolUnrestriction, Data
}
}
class UnspecificSslContextCreation extends SslContextCreation, UnspecificContextCreation {
// UnspecificSslContextCreation() { library instanceof Ssl }
// override ProtocolVersion getUnrestriction() {
// result = UnspecificContextCreation.super.getUnrestriction() and
// // These are turned off by default since Python 3.6
// // see https://docs.python.org/3.6/library/ssl.html#ssl.SSLContext
// not result in ["SSLv2", "SSLv3"]
// }
}
class UnspecificSslDefaultContextCreation extends SslDefaultContextCreation {
// override DataFlow::Node getContext() { result = this }
// // see https://docs.python.org/3/library/ssl.html#ssl.create_default_context
// override ProtocolVersion getUnrestriction() {
// result in ["TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"]
// }
}
// class UnspecificSslContextCreation extends SslContextCreation, UnspecificContextCreation {
// // UnspecificSslContextCreation() { library instanceof Ssl }
// override ProtocolVersion getProtocol() {
// result = UnspecificContextCreation.super.getProtocol() and
// // These are turned off by default since Python 3.6
// // see https://docs.python.org/3.6/library/ssl.html#ssl.SSLContext
// not result in ["SSLv2", "SSLv3"]
// }
// }
// class UnspecificSslDefaultContextCreation extends SslDefaultContextCreation {
// // override DataFlow::Node getContext() { result = this }
// // see https://docs.python.org/3/library/ssl.html#ssl.create_default_context
// override ProtocolVersion getProtocol() { result in ["TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"] }
// }
class Ssl extends TlsLibrary {
Ssl() { this = "ssl" }
override string specific_version_name(ProtocolVersion version) { result = "PROTOCOL_" + version }
override string unspecific_version_name(ProtocolFamily family) {
family = "SSLv23" and result = "PROTOCOL_" + family
override string unspecific_version_name() {
result = "PROTOCOL_SSLv23"
or
family = "TLS" and result = "PROTOCOL_" + family + ["", "_CLIENT", "_SERVER"]
result = "PROTOCOL_TLS" + ["", "_CLIENT", "_SERVER"]
}
override API::Node version_constants() { result = API::moduleImport("ssl") }
@@ -217,9 +214,5 @@ class Ssl extends TlsLibrary {
result instanceof OptionsAugAndNot
or
result instanceof ContextSetVersion
or
result instanceof UnspecificSslContextCreation
or
result instanceof UnspecificSslDefaultContextCreation
}
}

60
python/ql/src/Security/CWE-327/TlsLibraryModel.qll
View File

@@ -37,29 +37,23 @@ class ProtocolVersion extends string {
or
this = "TLSv1_3" and result = 32
}
/** Gets the protocol family for this protocol version. */
ProtocolFamily getFamily() {
result = "SSLv23" and this in ["SSLv2", "SSLv3"]
or
result = "TLS" and this in ["TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"]
}
}
/** An unspecific protocol version */
class ProtocolFamily extends string {
ProtocolFamily() { this in ["SSLv23", "TLS"] }
/** Gets the bit mask for this protocol family. */
int getBits() {
result = sum(ProtocolVersion version | version.getFamily() = this | version.getBit())
}
}
/** The creation of a context. */
abstract class ContextCreation extends DataFlow::Node {
/** Gets the protocol version or family for this context. */
abstract string getProtocol();
/**
* Gets the protocol version for this context.
* There can be multiple values if the context was created
* using a non-specific version such as `TLS`.
*/
abstract ProtocolVersion getProtocol();
/**
* Holds if the context was created with a specific version
* rather than with a version flexible method, see:
* https://www.openssl.org/docs/manmaster/man3/DTLS_server_method.html#NOTES
*/
predicate specificVersion() { count(this.getProtocol()) = 1 }
}
/** The creation of a connection from a context. */
@@ -91,13 +85,12 @@ abstract class ProtocolUnrestriction extends DataFlow::Node {
* This also serves as unrestricting these protocols.
*/
abstract class UnspecificContextCreation extends ContextCreation {
// override ProtocolVersion getUnrestriction() {
// // There is only one family, the two names are aliases in OpenSSL.
// // see https://github.com/openssl/openssl/blob/13888e797c5a3193e91d71e5f5a196a2d68d266f/include/openssl/ssl.h.in#L1953-L1955
// family in ["SSLv23", "TLS"] and
// // see https://docs.python.org/3/library/ssl.html#ssl-contexts
// result in ["SSLv2", "SSLv3", "TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"]
// }
override ProtocolVersion getProtocol() {
// There is only one family, the two names are aliases in OpenSSL.
// see https://github.com/openssl/openssl/blob/13888e797c5a3193e91d71e5f5a196a2d68d266f/include/openssl/ssl.h.in#L1953-L1955
// see https://docs.python.org/3/library/ssl.html#ssl-contexts
result in ["SSLv2", "SSLv3", "TLSv1", "TLSv1_1", "TLSv1_2", "TLSv1_3"]
}
}
/** A model of a SSL/TLS library. */
@@ -108,8 +101,8 @@ abstract class TlsLibrary extends string {
/** Gets the name of a specific protocol version. */
abstract string specific_version_name(ProtocolVersion version);
/** Gets a name, which is a member of `version_constants`, that can be used to specify the protocol family `family`. */
abstract string unspecific_version_name(ProtocolFamily family);
/** Gets a name, which is a member of `version_constants`, that can be used to specify the entire protocol family. */
abstract string unspecific_version_name();
/** Gets an API node representing the module or class holding the version constants. */
abstract API::Node version_constants();
@@ -119,9 +112,9 @@ abstract class TlsLibrary extends string {
result = this.version_constants().getMember(this.specific_version_name(version))
}
/** Gets an API node representing the protocol family `family`. */
API::Node unspecific_version(ProtocolFamily family) {
result = this.version_constants().getMember(this.unspecific_version_name(family))
/** Gets an API node representing the protocol entire family. */
API::Node unspecific_version() {
result = this.version_constants().getMember(this.unspecific_version_name())
}
/** Gets a creation of a context with a default protocol. */
@@ -133,14 +126,15 @@ abstract class TlsLibrary extends string {
/** Gets a creation of a context with a specific protocol version, known to be insecure. */
ContextCreation insecure_context_creation(ProtocolVersion version) {
result in [this.specific_context_creation(), this.default_context_creation()] and
result.specificVersion() and
result.getProtocol() = version and
version.isInsecure()
}
/** Gets a context that was created using `family`, known to have insecure instances. */
ContextCreation unspecific_context_creation(ProtocolFamily family) {
ContextCreation unspecific_context_creation() {
result in [this.specific_context_creation(), this.default_context_creation()] and
result.getProtocol() = family
not result.specificVersion()
}
/** Gets a dataflow node representing a connection being created in an insecure manner, not from a context. */

4
python/ql/src/Variables/UninitializedLocal.ql
View File

@@ -15,7 +15,9 @@ import Undefined
import semmle.python.pointsto.PointsTo
predicate uninitialized_local(NameNode use) {
exists(FastLocalVariable local | use.uses(local) or use.deletes(local) | not local.escapes()) and
exists(FastLocalVariable local | use.uses(local) or use.deletes(local) |
not local.escapes() and not local = any(Nonlocal nl).getAVariable()
) and
(
any(Uninitialized uninit).taints(use) and
PointsToInternal::reachableBlock(use.getBasicBlock(), _)

4
python/ql/src/analysis/Summary.ql
View File

@@ -13,8 +13,8 @@ where
or
key = "Interpreter version" and
exists(string major, string minor |
py_flags_versioned("version.major", major, _) and
py_flags_versioned("version.minor", minor, _) and
py_flags_versioned("extractor_python_version.major", major, _) and
py_flags_versioned("extractor_python_version.minor", minor, _) and
value = major + "." + minor
)
or

7
python/ql/src/change-notes/2023-02-03-unsafe-shell-command-construction.md → python/ql/src/change-notes/released/0.6.5.md
View File

@@ -1,4 +1,5 @@
---
category: newQuery
---
## 0.6.5
### New Queries
* Added a new query, `py/shell-command-constructed-from-input`, to detect libraries that unsafely construct shell commands from their inputs.

3
python/ql/src/change-notes/released/0.6.6.md Normal file
View File

@@ -0,0 +1,3 @@
## 0.6.6
No user-facing changes.

5
python/ql/src/change-notes/released/0.7.0.md Normal file
View File

@@ -0,0 +1,5 @@
## 0.7.0
### Bug Fixes
* Nonlocal variables are excluded from alerts.

2
python/ql/src/codeql-pack.release.yml
View File

@@ -1,2 +1,2 @@
---
lastReleaseVersion: 0.6.4
lastReleaseVersion: 0.7.0

4
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.qhelp Normal file
View File

@@ -0,0 +1,4 @@
<!DOCTYPE qhelp PUBLIC "-//Semmle//qhelp//EN" "qhelp.dtd">
<qhelp>
<include src="TimingAttackAgainstHash.qhelp" />
</qhelp>

37
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.ql Normal file
View File

@@ -0,0 +1,37 @@
/**
* @name Timing attack against Hash
* @description When checking a Hash over a message, a constant-time algorithm should be used.
* Otherwise, an attacker may be able to forge a valid Hash for an arbitrary message
* by running a timing attack if they can send to the validation procedure.
* A successful attack can result in authentication bypass.
* @kind path-problem
* @problem.severity error
* @precision low
* @id py/possible-timing-attack-against-hash
* @tags security
* external/cwe/cwe-208
* experimental
*/
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.TaintTracking
import experimental.semmle.python.security.TimingAttack
import DataFlow::PathGraph
/**
* A configuration that tracks data flow from cryptographic operations
* to equality test
*/
class PossibleTimingAttackAgainstHash extends TaintTracking::Configuration {
PossibleTimingAttackAgainstHash() { this = "PossibleTimingAttackAgainstHash" }
override predicate isSource(DataFlow::Node source) { source instanceof ProduceCryptoCall }
override predicate isSink(DataFlow::Node sink) { sink instanceof NonConstantTimeComparisonSink }
}
from PossibleTimingAttackAgainstHash config, DataFlow::PathNode source, DataFlow::PathNode sink
where config.hasFlowPath(source, sink)
select sink.getNode(), source, sink, "Possible Timing attack against $@ validation.",
source.getNode().(ProduceCryptoCall).getResultType(), "message"

17
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/SafeComparisonOfHash.py Normal file
View File

@@ -0,0 +1,17 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：preventing timing attack Against Hash
"""
import hmac
import hashlib
key = "e179017a-62b0-4996-8a38-e91aa9f1"
msg = "Test"
def sign(pre_key, imsg, alg):
return hmac.new(pre_key, imsg, alg).digest()
def verify(msg, sig):
return hmac.compare_digest(sig, sign(key, msg, hashlib.sha256)) #good

55
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/TimingAttackAgainstHash.qhelp Normal file
View File

@@ -0,0 +1,55 @@
<!DOCTYPE qhelp PUBLIC "-//Semmle//qhelp//EN" "qhelp.dtd">
<qhelp>
<overview>
<p>
Timing Attack is based on the leakage of information by studying how long it takes the system to respond to different inputs.
it can be circumvented by using a constant-time algorithm for checking the value of Hash,
more precisely, the comparison time should not depend on the content of the input. Otherwise the attacker gains
information that is indirectly leaked by the application. This information may then be used for malicious purposes.
</p>
</overview>
<recommendation>
<p>
Two types of countermeasures can be applied against timing attacks. The first one consists
in eliminating timing variations whereas the second renders these variations useless for an attacker.
The only absolute way to prevent timing attacks is to make the computation strictly constant time,
independent of the input.
Use <code>hmac.compare_digest()</code> method to securely check the value of Hash.
If this method is used, then the calculation time depends only on the length of input byte arrays,
and does not depend on the contents of the arrays.
Unlike <code>==</code> is a fail fast check, If the first byte is not equal, it will return immediately.
</p>
</recommendation>
<example>
<p>
The following example uses <code>==</code> which is a fail fast check for validating a Hash.
</p>
<sample src="UnSafeComparisonOfHash.py" />
<p>
The next example use a safe constant-time algorithm for validating a Hash:
</p>
<sample src="SafeComparisonOfHash.py" />
</example>
<references>
<li>
Wikipedia:
<a href="https://en.wikipedia.org/wiki/Timing_attack">Timing attack</a>.
</li>
<li>
<a href="https://docs.python.org/3/library/hmac.html#hmac.compare_digest">hmac.compare_digest() method</a>
</li>
<li>
HMAC:
<a href="https://datatracker.ietf.org/doc/html/rfc2104.html">RFC 2104</a>
</li>
</references>
</qhelp>

38
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/TimingAttackAgainstHash.ql Normal file
View File

@@ -0,0 +1,38 @@
/**
* @name Timing attack against Hash
* @description When checking a Hash over a message, a constant-time algorithm should be used.
* Otherwise, an attacker may be able to forge a valid Hash for an arbitrary message
* by running a timing attack if they can send to the validation procedure.
* A successful attack can result in authentication bypass.
* @kind path-problem
* @problem.severity error
* @precision low
* @id py/timing-attack-against-hash
* @tags security
* external/cwe/cwe-208
*/
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.TaintTracking
import experimental.semmle.python.security.TimingAttack
import DataFlow::PathGraph
/**
* A configuration that tracks data flow from cryptographic operations
* to Equality test.
*/
class TimingAttackAgainsthash extends TaintTracking::Configuration {
TimingAttackAgainsthash() { this = "TimingAttackAgainsthash" }
override predicate isSource(DataFlow::Node source) { source instanceof ProduceCryptoCall }
override predicate isSink(DataFlow::Node sink) { sink instanceof NonConstantTimeComparisonSink }
}
from TimingAttackAgainsthash config, DataFlow::PathNode source, DataFlow::PathNode sink
where
config.hasFlowPath(source, sink) and
sink.getNode().(NonConstantTimeComparisonSink).includesUserInput()
select sink.getNode(), source, sink, "Timing attack against $@ validation.",
source.getNode().(ProduceCryptoCall).getResultType(), "message"

17
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHash/UnSafeComparisonOfHash.py Normal file
View File

@@ -0,0 +1,17 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：timing attack Against Hash
"""
import hmac
import hashlib
key = "e179017a-62b0-4996-8a38-e91aa9f1"
msg = "Test"
def sign(pre_key, imsg, alg):
return hmac.new(pre_key, imsg, alg).digest()
def verify(msg, sig):
return sig == sign(key, msg, hashlib.sha256) #bad

21
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHeaderValue/SafeComparisonOfHeaderValue.py Normal file
View File

@@ -0,0 +1,21 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：preventing timing attack against header value
"""
from flask import Flask
from flask import request
import hmac
@app.route('/good')
def good():
secret = request.headers.get('X-Auth-Token')
if not hmac.compare_digest(secret, "token"):
raise Exception('bad token')
return 'good'
if __name__ == '__main__':
app.debug = True
app.run()

50
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHeaderValue/TimingAttackAgainstHeaderValue.qhelp Normal file
View File

@@ -0,0 +1,50 @@
<!DOCTYPE qhelp PUBLIC "-//Semmle//qhelp//EN" "qhelp.dtd">
<qhelp>
<overview>
<p>
A constant-time algorithm should be used for checking the value of sensitive headers.
In other words, the comparison time should not depend on the content of the input.
Otherwise timing information could be used to infer the header's expected, secret value.
</p>
</overview>
<recommendation>
<p>
Two types of countermeasures can be applied against timing attacks. The first one consists
in eliminating timing variations whereas the second renders these variations useless for an attacker.
The only absolute way to prevent timing attacks is to make the computation strictly constant time,
independent of the input.
Use <code>hmac.compare_digest()</code> method to securely check the secret value.
If this method is used, then the calculation time depends only on the length of input byte arrays,
and does not depend on the contents of the arrays.
Unlike <code>==</code> is a fail fast check, If the first byte is not equal, it will return immediately.
</p>
</recommendation>
<example>
<p>
The following example uses <code>==</code> which is a fail fast check for validating the value of sensitive headers.
</p>
<sample src="UnsafeComparisonOfHeaderValue.py" />
<p>
The next example use a safe constant-time algorithm for validating the value of sensitive headers:
</p>
<sample src="SafeComparisonOfHeaderValue.py" />
</example>
<references>
<li>
Wikipedia:
<a href="https://en.wikipedia.org/wiki/Timing_attack">Timing attack</a>.
</li>
<li>
<a href="https://docs.python.org/3/library/hmac.html#hmac.compare_digest">hmac.compare_digest() method</a>
</li>
</references>
</qhelp>

34
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHeaderValue/TimingAttackAgainstHeaderValue.ql Normal file
View File

@@ -0,0 +1,34 @@
/**
* @name Timing attack against header value
* @description Use of a non-constant-time verification routine to check the value of an HTTP header,
* possibly allowing a timing attack to infer the header's expected value.
* @kind path-problem
* @problem.severity error
* @precision high
* @id py/timing-attack-against-header-value
* @tags security
* external/cwe/cwe-208
* experimental
*/
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.TaintTracking
import experimental.semmle.python.security.TimingAttack
import DataFlow::PathGraph
/**
* A configuration tracing flow from a client Secret obtained by an HTTP header to a unsafe Comparison.
*/
class ClientSuppliedSecretConfig extends TaintTracking::Configuration {
ClientSuppliedSecretConfig() { this = "ClientSuppliedSecretConfig" }
override predicate isSource(DataFlow::Node source) { source instanceof ClientSuppliedSecret }
override predicate isSink(DataFlow::Node sink) { sink instanceof CompareSink }
}
from ClientSuppliedSecretConfig config, DataFlow::PathNode source, DataFlow::PathNode sink
where config.hasFlowPath(source, sink) and not sink.getNode().(CompareSink).flowtolen()
select sink.getNode(), source, sink, "Timing attack against $@ validation.", source.getNode(),
"client-supplied token"

20
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstHeaderValue/UnsafeComparisonOfHeaderValue.py Normal file
View File

@@ -0,0 +1,20 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：preventing timing attack against header value
"""
from flask import Flask
from flask import request
@app.route('/bad')
def bad():
secret = request.headers.get('X-Auth-Token')
if secret == "token":
raise Exception('bad token')
return 'bad'
if __name__ == '__main__':
app.debug = True
app.run()

4
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstSensitiveInfo/PossibleTimingAttackAgainstSensitiveInfo.qhelp Normal file
View File

@@ -0,0 +1,4 @@
<!DOCTYPE qhelp PUBLIC "-//Semmle//qhelp//EN" "qhelp.dtd">
<qhelp>
<include src="TimingAttackAgainstSensitiveInfo.qhelp" />
</qhelp>

34
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstSensitiveInfo/PossibleTimingAttackAgainstSensitiveInfo.ql Normal file
View File

@@ -0,0 +1,34 @@
/**
* @name Timing attack against secret
* @description Use of a non-constant-time verification routine to check the value of an secret,
* possibly allowing a timing attack to retrieve sensitive information.
* @kind path-problem
* @problem.severity error
* @precision low
* @id py/possible-timing-attack-sensitive-info
* @tags security
* external/cwe/cwe-208
* experimental
*/
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.TaintTracking
import experimental.semmle.python.security.TimingAttack
import DataFlow::PathGraph
/**
* A configuration tracing flow from obtaining a client Secret to a unsafe Comparison.
*/
class ClientSuppliedSecretConfig extends TaintTracking::Configuration {
ClientSuppliedSecretConfig() { this = "ClientSuppliedSecretConfig" }
override predicate isSource(DataFlow::Node source) { source instanceof SecretSource }
override predicate isSink(DataFlow::Node sink) { sink instanceof NonConstantTimeComparisonSink }
}
from ClientSuppliedSecretConfig config, DataFlow::PathNode source, DataFlow::PathNode sink
where config.hasFlowPath(source, sink)
select sink.getNode(), source, sink, "Timing attack against $@ validation.", source.getNode(),
"client-supplied token"

21
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstSensitiveInfo/SafeComparisonOfSensitiveInfo.py Normal file
View File

@@ -0,0 +1,21 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：preventing timing attack sensitive info
"""
from flask import Flask
from flask import request
import hmac
app = Flask(__name__)
@app.route('/bad', methods = ['POST', 'GET'])
def bad():
if request.method == 'POST':
password = request.form['pwd']
return hmac.compare_digest(password, "1234")
if __name__ == '__main__':
app.debug = True
app.run()

52
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstSensitiveInfo/TimingAttackAgainstSensitiveInfo.qhelp Normal file
View File

@@ -0,0 +1,52 @@
<!DOCTYPE qhelp PUBLIC "-//Semmle//qhelp//EN" "qhelp.dtd">
<qhelp>
<overview>
<p>
Timing Attack is based on the leakage of information of secret parameters by studying
how long it takes the system to respond to different inputs.
it can be circumvented by using a constant-time algorithm for checking the value of sensitive info,
more precisely, the comparison time should not depend on the content of the input. Otherwise the attacker gains
information that is indirectly leaked by the application. This information is then used for malicious purposes.
</p>
</overview>
<recommendation>
<p>
Two types of countermeasures can be applied against timing attacks. The first one consists
in eliminating timing variations whereas the second renders these variations useless for an attacker.
The only absolute way to prevent timing attacks is to make the computation strictly constant time,
independent of the input.
Use <code>hmac.compare_digest()</code> method to securely check the value of sensitive info.
If this method is used, then the calculation time depends only on the length of input byte arrays,
and does not depend on the contents of the arrays.
Unlike <code>==</code> is a fail fast check, If the first byte is not equal, it will return immediately.
</p>
</recommendation>
<example>
<p>
The following example uses <code>==</code> which is a fail fast check for validating a secret.
</p>
<sample src="UnSafeComparisonOfSensitiveInfo.py" />
<p>
The next example use a safe constant-time algorithm for validating a secret:
</p>
<sample src="SafeComparisonOfSensitiveInfo.py" />
</example>
<references>
<li>
Wikipedia:
<a href="https://en.wikipedia.org/wiki/Timing_attack">Timing attack</a>.
</li>
<li>
<a href="https://docs.python.org/3/library/hmac.html#hmac.compare_digest">hmac.compare_digest() method</a>
</li>
</references>
</qhelp>

39
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstSensitiveInfo/TimingAttackAgainstSensitiveInfo.ql Normal file
View File

@@ -0,0 +1,39 @@
/**
* @name Timing attack against secret
* @description Use of a non-constant-time verification routine to check the value of an secret,
* possibly allowing a timing attack to retrieve sensitive information.
* @kind path-problem
* @problem.severity error
* @precision low
* @id py/timing-attack-sensitive-info
* @tags security
* external/cwe/cwe-208
* experimental
*/
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.TaintTracking
import experimental.semmle.python.security.TimingAttack
import DataFlow::PathGraph
/**
* A configuration tracing flow from obtaining a client Secret to a unsafe Comparison.
*/
class ClientSuppliedSecretConfig extends TaintTracking::Configuration {
ClientSuppliedSecretConfig() { this = "ClientSuppliedSecretConfig" }
override predicate isSource(DataFlow::Node source) { source instanceof SecretSource }
override predicate isSink(DataFlow::Node sink) { sink instanceof NonConstantTimeComparisonSink }
}
from ClientSuppliedSecretConfig config, DataFlow::PathNode source, DataFlow::PathNode sink
where
config.hasFlowPath(source, sink) and
(
source.getNode().(SecretSource).includesUserInput() or
sink.getNode().(NonConstantTimeComparisonSink).includesUserInput()
)
select sink.getNode(), source, sink, "Timing attack against $@ validation.", source.getNode(),
"client-supplied token"

18
python/ql/src/experimental/Security/CWE-208/TimingAttackAgainstSensitiveInfo/UnSafeComparisonOfSensitiveInfo.py Normal file
View File

@@ -0,0 +1,18 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：timing attack against sensitive info
"""
from flask import Flask
from flask import request
@app.route('/bad', methods = ['POST', 'GET'])
def bad():
if request.method == 'POST':
password = request.form['pwd']
return password == "test"
if __name__ == '__main__':
app.debug = True
app.run()

209
python/ql/src/experimental/Security/CWE-327/Azure/UnsafeUsageOfClientSideEncryptionVersion.ql
View File

@@ -1,7 +1,7 @@
/**
* @name Unsafe usage of v1 version of Azure Storage client-side encryption.
* @description Using version v1 of Azure Storage client-side encryption is insecure, and may enable an attacker to decrypt encrypted data
* @kind problem
* @kind path-problem
* @tags security
* experimental
* cryptography
@@ -12,80 +12,145 @@
*/
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.ApiGraphs
predicate isUnsafeClientSideAzureStorageEncryptionViaAttributes(Call call, AttrNode node) {
exists(
API::Node n, API::Node n2, Attribute a, AssignStmt astmt, API::Node uploadBlob,
ControlFlowNode ctrlFlowNode, string s
|
s in ["key_encryption_key", "key_resolver_function"] and
n =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobClient")
.getReturn()
.getMember(s) and
n2 =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobClient")
.getReturn()
.getMember("upload_blob") and
n.getAValueReachableFromSource().asExpr() = a and
astmt.getATarget() = a and
a.getAFlowNode() = node and
uploadBlob =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobClient")
.getReturn()
.getMember("upload_blob") and
uploadBlob.getACall().asExpr() = call and
ctrlFlowNode = call.getAFlowNode() and
node.strictlyReaches(ctrlFlowNode) and
node != ctrlFlowNode and
not exists(
AssignStmt astmt2, Attribute a2, AttrNode encryptionVersionSet, StrConst uc,
API::Node encryptionVersion
|
uc = astmt2.getValue() and
uc.getText() in ["'2.0'", "2.0"] and
encryptionVersion =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobClient")
.getReturn()
.getMember("encryption_version") and
encryptionVersion.getAValueReachableFromSource().asExpr() = a2 and
astmt2.getATarget() = a2 and
a2.getAFlowNode() = encryptionVersionSet and
encryptionVersionSet.strictlyReaches(ctrlFlowNode)
)
)
API::Node getBlobServiceClient(boolean isSource) {
isSource = true and
result =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobServiceClient")
.getReturn()
or
isSource = true and
result =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobServiceClient")
.getMember("from_connection_string")
.getReturn()
}
predicate isUnsafeClientSideAzureStorageEncryptionViaObjectCreation(Call call, ControlFlowNode node) {
exists(API::Node c, string s, Keyword k | k.getAFlowNode() = node |
c.getACall().asExpr() = call and
c = API::moduleImport("azure").getMember("storage").getMember("blob").getMember(s) and
s in ["ContainerClient", "BlobClient", "BlobServiceClient"] and
k.getArg() = "key_encryption_key" and
k = call.getANamedArg() and
not k.getValue() instanceof None and
not exists(Keyword k2 | k2 = call.getANamedArg() |
k2.getArg() = "encryption_version" and
k2.getValue().(StrConst).getText() in ["'2.0'", "2.0"]
)
)
API::CallNode getTransitionToContainerClient() {
result = getBlobServiceClient(_).getMember("get_container_client").getACall()
or
result = getBlobClient(_).getMember("_get_container_client").getACall()
}
from Call call, ControlFlowNode node
where
isUnsafeClientSideAzureStorageEncryptionViaAttributes(call, node) or
isUnsafeClientSideAzureStorageEncryptionViaObjectCreation(call, node)
select node, "Unsafe usage of v1 version of Azure Storage client-side encryption."
API::Node getContainerClient(boolean isSource) {
isSource = false and
result = getTransitionToContainerClient().getReturn()
or
isSource = true and
result =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("ContainerClient")
.getReturn()
or
isSource = true and
result =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("ContainerClient")
.getMember(["from_connection_string", "from_container_url"])
.getReturn()
}
API::CallNode getTransitionToBlobClient() {
result = [getBlobServiceClient(_), getContainerClient(_)].getMember("get_blob_client").getACall()
}
API::Node getBlobClient(boolean isSource) {
isSource = false and
result = getTransitionToBlobClient().getReturn()
or
isSource = true and
result =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobClient")
.getReturn()
or
isSource = true and
result =
API::moduleImport("azure")
.getMember("storage")
.getMember("blob")
.getMember("BlobClient")
.getMember(["from_connection_string", "from_blob_url"])
.getReturn()
}
API::Node anyClient(boolean isSource) {
result in [getBlobServiceClient(isSource), getContainerClient(isSource), getBlobClient(isSource)]
}
newtype TAzureFlowState =
MkUsesV1Encryption() or
MkUsesNoEncryption()
module AzureBlobClientConfig implements DataFlow::StateConfigSig {
class FlowState = TAzureFlowState;
predicate isSource(DataFlow::Node node, FlowState state) {
state = MkUsesNoEncryption() and
node = anyClient(true).asSource()
}
predicate isBarrier(DataFlow::Node node, FlowState state) {
exists(state) and
exists(DataFlow::AttrWrite attr |
node = anyClient(_).getAValueReachableFromSource() and
attr.accesses(node, "encryption_version") and
attr.getValue().asExpr().(StrConst).getText() in ["'2.0'", "2.0"]
)
or
// small optimization to block flow with no encryption out of the post-update node
// for the attribute assignment.
isAdditionalFlowStep(_, MkUsesNoEncryption(), node, MkUsesV1Encryption()) and
state = MkUsesNoEncryption()
}
predicate isAdditionalFlowStep(DataFlow::Node node1, DataFlow::Node node2) {
exists(DataFlow::MethodCallNode call |
call in [getTransitionToContainerClient(), getTransitionToBlobClient()] and
node1 = call.getObject() and
node2 = call
)
}
predicate isAdditionalFlowStep(
DataFlow::Node node1, FlowState state1, DataFlow::Node node2, FlowState state2
) {
node1 = node2.(DataFlow::PostUpdateNode).getPreUpdateNode() and
state1 = MkUsesNoEncryption() and
state2 = MkUsesV1Encryption() and
exists(DataFlow::AttrWrite attr |
node1 = anyClient(_).getAValueReachableFromSource() and
attr.accesses(node1, ["key_encryption_key", "key_resolver_function"])
)
}
predicate isSink(DataFlow::Node node, FlowState state) {
state = MkUsesV1Encryption() and
exists(DataFlow::MethodCallNode call |
call = getBlobClient(_).getMember("upload_blob").getACall() and
node = call.getObject()
)
}
}
module AzureBlobClient = DataFlow::GlobalWithState<AzureBlobClientConfig>;
import AzureBlobClient::PathGraph
from AzureBlobClient::PathNode source, AzureBlobClient::PathNode sink
where AzureBlobClient::flowPath(source, sink)
select sink, source, sink, "Unsafe usage of v1 version of Azure Storage client-side encryption"

4
python/ql/src/experimental/Security/CWE-611/SimpleXmlRpcServer.ql
View File

@@ -1,6 +1,6 @@
/**
* @name SimpleXMLRPCServer DoS vulnerability
* @description SimpleXMLRPCServer is vulnerable to DoS attacks from untrusted user input
* @name SimpleXMLRPCServer denial of service
* @description SimpleXMLRPCServer is vulnerable to denial of service attacks from untrusted user input
* @kind problem
* @problem.severity warning
* @precision high

350
python/ql/src/experimental/semmle/python/security/TimingAttack.qll Normal file
View File

@@ -0,0 +1,350 @@
private import python
private import semmle.python.dataflow.new.TaintTracking2
private import semmle.python.dataflow.new.TaintTracking
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.DataFlow2
private import semmle.python.ApiGraphs
private import semmle.python.dataflow.new.RemoteFlowSources
private import semmle.python.frameworks.Flask
private import semmle.python.frameworks.Django
/** A method call that produces cryptographic result. */
abstract class ProduceCryptoCall extends API::CallNode {
/** Gets a type of cryptographic operation such as MAC, signature, Hash or ciphertext. */
abstract string getResultType();
}
/** Gets a reference to the `cryptography.hazmat.primitives` module. */
API::Node cryptographylib() {
result = API::moduleImport("cryptography").getMember("hazmat").getMember("primitives")
}
/** Gets a reference to the `Crypto` module. */
API::Node cryptodome() { result = API::moduleImport(["Crypto", "Cryptodome"]) }
/** A method call that produces a MAC. */
class ProduceMacCall extends ProduceCryptoCall {
ProduceMacCall() {
this = API::moduleImport("hmac").getMember("digest").getACall() or
this =
API::moduleImport("hmac")
.getMember("new")
.getReturn()
.getMember(["digest", "hexdigest"])
.getACall() or
this =
cryptodome()
.getMember("Hash")
.getMember("HMAC")
.getMember(["new", "HMAC"])
.getMember(["digest", "hexdigest"])
.getACall() or
this =
cryptographylib()
.getMember("hmac")
.getMember("HMAC")
.getReturn()
.getMember("finalize")
.getACall() or
this =
cryptographylib()
.getMember("cmac")
.getMember("CMAC")
.getReturn()
.getMember("finalize")
.getACall() or
this =
cryptodome()
.getMember("Hash")
.getMember("CMAC")
.getMember(["new", "CMAC"])
.getMember(["digest", "hexdigest"])
.getACall()
}
override string getResultType() { result = "MAC" }
}
/** A method call that produces a signature. */
private class ProduceSignatureCall extends ProduceCryptoCall {
ProduceSignatureCall() {
this =
cryptodome()
.getMember("Signature")
.getMember(["DSS", "pkcs1_15", "pss", "eddsa"])
.getMember("new")
.getReturn()
.getMember("sign")
.getACall()
}
override string getResultType() { result = "signature" }
}
private string hashalgo() {
result = ["sha1", "sha224", "sha256", "sha384", "sha512", "blake2b", "blake2s", "md5"]
}
/** A method call that produces a Hash. */
private class ProduceHashCall extends ProduceCryptoCall {
ProduceHashCall() {
this =
cryptographylib()
.getMember("hashes")
.getMember("Hash")
.getReturn()
.getMember("finalize")
.getACall() or
this =
API::moduleImport("hashlib")
.getMember(["new", hashalgo()])
.getReturn()
.getMember(["digest", "hexdigest"])
.getACall() or
this =
cryptodome()
.getMember(hashalgo())
.getMember("new")
.getReturn()
.getMember(["digest", "hexdigest"])
.getACall()
}
override string getResultType() { result = "Hash" }
}
/** A method call that produces a ciphertext. */
private class ProduceCiphertextCall extends ProduceCryptoCall {
ProduceCiphertextCall() {
this =
cryptodome()
.getMember("Cipher")
.getMember(["DES", "DES3", "ARC2", "ARC4", "Blowfish", "PKCS1_v1_5"])
.getMember(["ARC4Cipher", "new", "PKCS115_Cipher"])
.getMember("encrypt")
.getACall() or
this =
cryptographylib()
.getMember("ciphers")
.getMember("Cipher")
.getReturn()
.getMember("finalize")
.getACall()
}
override string getResultType() { result = "ciphertext" }
}
/** A data flow sink for comparison. */
private predicate existsFailFastCheck(Expr firstInput, Expr secondInput) {
exists(Compare compare |
(
compare.getOp(0) instanceof Eq or
compare.getOp(0) instanceof NotEq or
compare.getOp(0) instanceof In or
compare.getOp(0) instanceof NotIn
) and
(
compare.getLeft() = firstInput and
compare.getComparator(0) = secondInput and
not compare.getAComparator() instanceof None
or
compare.getLeft() = secondInput and
compare.getComparator(0) = firstInput and
not compare.getAComparator() instanceof None
)
)
}
/** A sink that compares input using fail fast check. */
class NonConstantTimeComparisonSink extends DataFlow::Node {
Expr anotherParameter;
NonConstantTimeComparisonSink() { existsFailFastCheck(this.asExpr(), anotherParameter) }
/** Holds if remote user input was used in the comparison. */
predicate includesUserInput() {
exists(UserInputInComparisonConfig config |
config.hasFlowTo(DataFlow2::exprNode(anotherParameter))
)
}
}
/** A data flow source of the secret obtained. */
class SecretSource extends DataFlow::Node {
CredentialExpr secret;
SecretSource() { secret = this.asExpr() }
/** Holds if the secret was deliverd by remote user. */
predicate includesUserInput() {
exists(UserInputSecretConfig config | config.hasFlowTo(DataFlow2::exprNode(secret)))
}
}
/** A string for `match` that identifies strings that look like they represent secret data. */
private string suspicious() {
result =
[
"%password%", "%passwd%", "%pwd%", "%refresh%token%", "%secret%token", "%secret%key",
"%passcode%", "%passphrase%", "%token%", "%secret%", "%credential%", "%userpass%", "%digest%",
"%signature%", "%mac%"
]
}
/** A variable that may hold sensitive information, judging by its name. * */
class CredentialExpr extends Expr {
CredentialExpr() {
exists(Variable v | this = v.getAnAccess() | v.getId().toLowerCase().matches(suspicious()))
}
}
/**
* A data flow source of the client Secret obtained according to the remote endpoint identifier specified
* (`X-auth-token`, `proxy-authorization`, `X-Csrf-Header`, etc.) in the header.
*
* For example: `request.headers.get("X-Auth-Token")`.
*/
abstract class ClientSuppliedSecret extends DataFlow::CallCfgNode { }
private class FlaskClientSuppliedSecret extends ClientSuppliedSecret {
FlaskClientSuppliedSecret() {
this = Flask::request().getMember("headers").getMember(["get", "get_all", "getlist"]).getACall() and
[this.getArg(0), this.getArgByName(["key", "name"])].asExpr().(StrConst).getText().toLowerCase() =
sensitiveheaders()
}
}
private class DjangoClientSuppliedSecret extends ClientSuppliedSecret {
DjangoClientSuppliedSecret() {
this =
PrivateDjango::DjangoImpl::DjangoHttp::Request::HttpRequest::classRef()
.getMember(["headers", "META"])
.getMember("get")
.getACall() and
[this.getArg(0), this.getArgByName("key")].asExpr().(StrConst).getText().toLowerCase() =
sensitiveheaders()
}
}
/** Gets a reference to the `tornado.web.RequestHandler` module. */
API::Node requesthandler() {
result = API::moduleImport("tornado").getMember("web").getMember("RequestHandler")
}
private class TornadoClientSuppliedSecret extends ClientSuppliedSecret {
TornadoClientSuppliedSecret() {
this = requesthandler().getMember(["headers", "META"]).getMember("get").getACall() and
[this.getArg(0), this.getArgByName("key")].asExpr().(StrConst).getText().toLowerCase() =
sensitiveheaders()
}
}
/** Gets a reference to the `werkzeug.datastructures.Headers` module. */
API::Node headers() {
result = API::moduleImport("werkzeug").getMember("datastructures").getMember("Headers")
}
private class WerkzeugClientSuppliedSecret extends ClientSuppliedSecret {
WerkzeugClientSuppliedSecret() {
this =
headers().getMember(["headers", "META"]).getMember(["get", "get_all", "getlist"]).getACall() and
[this.getArg(0), this.getArgByName(["key", "name"])].asExpr().(StrConst).getText().toLowerCase() =
sensitiveheaders()
}
}
/** A string for `match` that identifies strings that look like they represent Sensitive Headers. */
private string sensitiveheaders() {
result =
[
"x-auth-token", "x-csrf-token", "http_x_csrf_token", "x-csrf-param", "x-csrf-header",
"http_x_csrf_token", "x-api-key", "authorization", "proxy-authorization", "x-gitlab-token",
"www-authenticate"
]
}
/**
* A config that tracks data flow from remote user input to Variable that hold sensitive info
*/
class UserInputSecretConfig extends TaintTracking::Configuration {
UserInputSecretConfig() { this = "UserInputSecretConfig" }
override predicate isSource(DataFlow::Node source) { source instanceof RemoteFlowSource }
override predicate isSink(DataFlow::Node sink) { sink.asExpr() instanceof CredentialExpr }
}
/**
* A config that tracks data flow from remote user input to Equality test
*/
class UserInputInComparisonConfig extends TaintTracking2::Configuration {
UserInputInComparisonConfig() { this = "UserInputInComparisonConfig" }
override predicate isSource(DataFlow::Node source) { source instanceof RemoteFlowSource }
override predicate isSink(DataFlow::Node sink) {
exists(Compare cmp, Expr left, Expr right, Cmpop cmpop |
cmpop.getSymbol() = ["==", "in", "is not", "!="] and
cmp.compares(left, cmpop, right) and
sink.asExpr() = [left, right]
)
}
}
/**
* A configuration tracing flow from a client Secret obtained by an HTTP header to a len() function.
*/
private class ExcludeLenFunc extends TaintTracking2::Configuration {
ExcludeLenFunc() { this = "ExcludeLenFunc" }
override predicate isSource(DataFlow::Node source) { source instanceof ClientSuppliedSecret }
override predicate isSink(DataFlow::Node sink) {
exists(Call call |
call.getFunc().(Name).getId() = "len" and
sink.asExpr() = call.getArg(0)
)
}
}
/**
* Holds if there is a fast-fail check.
*/
class CompareSink extends DataFlow::Node {
CompareSink() {
exists(Compare compare |
(
compare.getOp(0) instanceof Eq or
compare.getOp(0) instanceof NotEq
) and
(
compare.getLeft() = this.asExpr() and
not compare.getComparator(0).(StrConst).getText() = "bearer"
or
compare.getComparator(0) = this.asExpr() and
not compare.getLeft().(StrConst).getText() = "bearer"
)
)
or
exists(Compare compare |
compare.getOp(0) instanceof IsNot and
(
compare.getLeft() = this.asExpr() and
not compare.getComparator(0) instanceof None
or
compare.getComparator(0) = this.asExpr() and
not compare.getLeft() instanceof None
)
)
}
/**
* Holds if there is a flow to len().
*/
predicate flowtolen() {
exists(ExcludeLenFunc config, DataFlow2::PathNode source, DataFlow2::PathNode sink |
config.hasFlowPath(source, sink)
)
}
}

3
python/ql/src/qlpack.yml
View File

@@ -1,12 +1,11 @@
name: codeql/python-queries
version: 0.6.5-dev
version: 0.7.1-dev
groups:
- python
- queries
dependencies:
codeql/python-all: ${workspace}
codeql/suite-helpers: ${workspace}
codeql/util: ${workspace}
suites: codeql-suites
extractor: python
defaultSuiteFile: codeql-suites/python-code-scanning.qls

1
python/ql/src/queries.xml
View File

@@ -1 +0,0 @@
<queries language="python"/>

0
python/ql/test/experimental/dataflow/coverage-py2/argumentRoutingTest.expected Normal file
View File

1
python/ql/test/experimental/dataflow/coverage-py2/argumentRoutingTest.qlref Normal file
View File

@@ -0,0 +1 @@
../coverage/argumentRoutingTest.ql

54
python/ql/test/experimental/dataflow/coverage-py2/classes.py Normal file
View File

@@ -0,0 +1,54 @@
# Python 2 specific tests, like the one in coverage/classes.py
#
# User-defined methods, both instance methods and class methods, can be called in many non-standard ways
# i.e. differently from simply `c.f()` or `C.f()`. For example, a user-defined `__await__` method on a
# class `C` will be called by the syntactic construct `await c` when `c` is an instance of `C`.
#
# These tests should cover all the class calls that we hope to support.
# It is based on https://docs.python.org/3/reference/datamodel.html, and headings refer there.
#
# All functions starting with "test_" should run and execute `print("OK")` exactly once.
# This can be checked by running validTest.py.
import sys
import os
sys.path.append(os.path.dirname(os.path.dirname((__file__))))
from testlib import expects
def SINK1(x):
pass
def SINK2(x):
pass
def SINK3(x):
pass
def SINK4(x):
pass
def OK():
print("OK")
# 3.3.8. Emulating numeric types
# object.__index__(self)
class With_index:
def __index__(self):
SINK1(self)
OK() # Call not found
return 0
def test_index():
import operator
with_index = With_index() #$ MISSING: arg1="SSA variable with_index" func=With_index.__index__
operator.index(with_index)

1
python/ql/test/experimental/dataflow/coverage-py2/options Normal file
View File

@@ -0,0 +1 @@
semmle-extractor-options: --max-import-depth=1 --lang=2

0
python/ql/test/experimental/dataflow/coverage-py3/argumentRoutingTest.expected Normal file
View File

1
python/ql/test/experimental/dataflow/coverage-py3/argumentRoutingTest.qlref Normal file
View File

@@ -0,0 +1 @@
../coverage/argumentRoutingTest.ql

72
python/ql/test/experimental/dataflow/coverage-py3/classes.py Normal file
View File

@@ -0,0 +1,72 @@
# Python 3 specific tests, like the one in coverage/classes.py
#
# User-defined methods, both instance methods and class methods, can be called in many non-standard ways
# i.e. differently from simply `c.f()` or `C.f()`. For example, a user-defined `__await__` method on a
# class `C` will be called by the syntactic construct `await c` when `c` is an instance of `C`.
#
# These tests should cover all the class calls that we hope to support.
# It is based on https://docs.python.org/3/reference/datamodel.html, and headings refer there.
#
# All functions starting with "test_" should run and execute `print("OK")` exactly once.
# This can be checked by running validTest.py.
import sys
import os
sys.path.append(os.path.dirname(os.path.dirname((__file__))))
from testlib import expects
def SINK1(x):
pass
def SINK2(x):
pass
def SINK3(x):
pass
def SINK4(x):
pass
def OK():
print("OK")
# 3.3.7. Emulating container types
# object.__length_hint__(self)
class With_length_hint:
def __length_hint__(self):
SINK1(self)
OK()
return 0
def test_length_hint():
import operator
with_length_hint = With_length_hint() #$ arg1="SSA variable with_length_hint" func=With_length_hint.__length_hint__
operator.length_hint(with_length_hint)
# 3.3.8. Emulating numeric types
# object.__index__(self)
class With_index:
def __index__(self):
SINK1(self)
OK() # Call not found
return 0
def test_index():
import operator
with_index = With_index() #$ arg1="SSA variable with_index" func=With_index.__index__
operator.index(with_index)

1
python/ql/test/experimental/dataflow/coverage-py3/options Normal file
View File

@@ -0,0 +1 @@
semmle-extractor-options: --max-import-depth=1 --lang=3

22
python/ql/test/experimental/dataflow/coverage/classes.py
View File

@@ -535,21 +535,6 @@ def test_len_if():
pass
# object.__length_hint__(self)
class With_length_hint:
def __length_hint__(self):
SINK1(self)
OK() # Call not found
return 0
def test_length_hint():
import operator
with_length_hint = With_length_hint() #$ MISSING: arg1="SSA variable with_length_hint" func=With_length_hint.__length_hint__
operator.length_hint(with_length_hint)
# object.__getitem__(self, key)
class With_getitem:
def __getitem__(self, key):
@@ -1378,13 +1363,6 @@ class With_index:
return 0
def test_index():
import operator
with_index = With_index() #$ MISSING: arg1="SSA variable with_index" func=With_index.__index__
operator.index(with_index)
def test_index_slicing():
with_index = With_index() #$ MISSING: arg1="SSA variable with_index" func=With_index.__index__
[0][with_index:1]

3
python/ql/test/experimental/dataflow/validTest.py
View File

@@ -64,9 +64,12 @@ if __name__ == "__main__":
check_tests_valid("coverage.test")
check_tests_valid("coverage.argumentPassing")
check_tests_valid("coverage.datamodel")
check_tests_valid("coverage-py2.classes")
check_tests_valid("coverage-py3.classes")
check_tests_valid("variable-capture.in")
check_tests_valid("variable-capture.nonlocal")
check_tests_valid("variable-capture.dict")
check_tests_valid("variable-capture.collections")
check_tests_valid("module-initialization.multiphase")
check_tests_valid("fieldflow.test")
check_tests_valid_after_version("match.test", (3, 10))

64
python/ql/test/experimental/dataflow/variable-capture/collections.py Normal file
View File

@@ -0,0 +1,64 @@
# Here we test the case where a captured variable is being read.
# All functions starting with "test_" should run and execute `print("OK")` exactly once.
# This can be checked by running validTest.py.
import sys
import os
sys.path.append(os.path.dirname(os.path.dirname((__file__))))
from testlib import expects
# These are defined so that we can evaluate the test code.
NONSOURCE = "not a source"
SOURCE = "source"
def is_source(x):
return x == "source" or x == b"source" or x == 42 or x == 42.0 or x == 42j
def SINK(x):
if is_source(x):
print("OK")
else:
print("Unexpected flow", x)
def SINK_F(x):
if is_source(x):
print("Unexpected flow", x)
else:
print("OK")
l = [NONSOURCE]
SINK_F(l[0])
l_mod = [SOURCE for x in l]
SINK(l_mod[0]) #$ captured
l_mod_lambda = [(lambda a : SOURCE)(x) for x in l]
SINK(l_mod_lambda[0]) #$ captured
def mod(x):
return SOURCE
l_mod_function = [mod(x) for x in l]
SINK(l_mod_function[0]) #$ captured
def mod_list(l):
def mod_local(x):
return SOURCE
return [mod_local(x) for x in l]
l_modded = mod_list(l)
SINK(l_modded[0]) #$ MISSING: captured
def mod_list_first(l):
def mod_local(x):
return SOURCE
return [mod_local(l[0])]
l_modded_first = mod_list_first(l)
SINK(l_modded_first[0]) #$ captured

5
python/ql/test/experimental/dataflow/variable-capture/dataflow-consistency.expected
View File

@@ -1,5 +1,10 @@
uniqueEnclosingCallable
uniqueCallEnclosingCallable
| collections.py:39:17:39:38 | Lambda() | Call should have one enclosing callable but has 0. |
| collections.py:39:17:39:38 | Lambda() | Call should have one enclosing callable but has 0. |
| collections.py:45:19:45:24 | mod() | Call should have one enclosing callable but has 0. |
| collections.py:45:19:45:24 | mod() | Call should have one enclosing callable but has 0. |
| collections.py:52:13:52:24 | mod_local() | Call should have one enclosing callable but has 0. |
uniqueType
uniqueNodeLocation
missingLocation

160
python/ql/test/experimental/import-resolution/ModuleExport.expected Normal file
View File

@@ -0,0 +1,160 @@
| attr_clash.__init__ | __file__ | attr_clash/__init__.py:6:6:6:13 | ControlFlowNode for __file__ |
| attr_clash.__init__ | __name__ | attr_clash/__init__.py:0:0:0:0 | GSSA Variable __name__ |
| attr_clash.__init__ | __package__ | attr_clash/__init__.py:0:0:0:0 | GSSA Variable __package__ |
| attr_clash.__init__ | clashing_attr | attr_clash/__init__.py:4:1:4:13 | GSSA Variable clashing_attr |
| attr_clash.__init__ | enter | attr_clash/__init__.py:2:1:2:5 | ControlFlowNode for enter |
| attr_clash.__init__ | exit | attr_clash/__init__.py:6:1:6:4 | ControlFlowNode for exit |
| attr_clash.clashing_attr | __file__ | attr_clash/clashing_attr.py:4:6:4:13 | ControlFlowNode for __file__ |
| attr_clash.clashing_attr | __name__ | attr_clash/clashing_attr.py:0:0:0:0 | GSSA Variable __name__ |
| attr_clash.clashing_attr | __package__ | attr_clash/clashing_attr.py:0:0:0:0 | GSSA Variable __package__ |
| attr_clash.clashing_attr | enter | attr_clash/clashing_attr.py:2:1:2:5 | ControlFlowNode for enter |
| attr_clash.clashing_attr | exit | attr_clash/clashing_attr.py:4:1:4:4 | ControlFlowNode for exit |
| attr_clash.non_clashing_submodule | __file__ | attr_clash/non_clashing_submodule.py:4:6:4:13 | ControlFlowNode for __file__ |
| attr_clash.non_clashing_submodule | __name__ | attr_clash/non_clashing_submodule.py:0:0:0:0 | GSSA Variable __name__ |
| attr_clash.non_clashing_submodule | __package__ | attr_clash/non_clashing_submodule.py:0:0:0:0 | GSSA Variable __package__ |
| attr_clash.non_clashing_submodule | enter | attr_clash/non_clashing_submodule.py:2:1:2:5 | ControlFlowNode for enter |
| attr_clash.non_clashing_submodule | exit | attr_clash/non_clashing_submodule.py:4:1:4:4 | ControlFlowNode for exit |
| bar | __file__ | bar.py:6:6:6:13 | ControlFlowNode for __file__ |
| bar | __name__ | bar.py:0:0:0:0 | GSSA Variable __name__ |
| bar | __package__ | bar.py:0:0:0:0 | GSSA Variable __package__ |
| bar | bar_attr | bar.py:4:1:4:8 | GSSA Variable bar_attr |
| bar | enter | bar.py:2:1:2:5 | ControlFlowNode for enter |
| bar | exit | bar.py:6:1:6:4 | ControlFlowNode for exit |
| baz | __file__ | baz.py:6:6:6:13 | ControlFlowNode for __file__ |
| baz | __name__ | baz.py:0:0:0:0 | GSSA Variable __name__ |
| baz | __package__ | baz.py:0:0:0:0 | GSSA Variable __package__ |
| baz | baz_attr | baz.py:4:1:4:8 | GSSA Variable baz_attr |
| baz | enter | baz.py:2:1:2:5 | ControlFlowNode for enter |
| baz | exit | baz.py:6:1:6:4 | ControlFlowNode for exit |
| block_flow_check | SOURCE | block_flow_check.py:12:25:12:30 | ControlFlowNode for SOURCE |
| block_flow_check | __file__ | block_flow_check.py:14:6:14:13 | ControlFlowNode for __file__ |
| block_flow_check | __name__ | block_flow_check.py:0:0:0:0 | GSSA Variable __name__ |
| block_flow_check | __package__ | block_flow_check.py:0:0:0:0 | GSSA Variable __package__ |
| block_flow_check | check | block_flow_check.py:12:1:12:5 | ControlFlowNode for check |
| block_flow_check | enter | block_flow_check.py:2:1:2:5 | ControlFlowNode for enter |
| block_flow_check | exit | block_flow_check.py:14:1:14:4 | ControlFlowNode for exit |
| block_flow_check | globals | block_flow_check.py:12:33:12:39 | ControlFlowNode for globals |
| block_flow_check | object | block_flow_check.py:4:14:4:19 | ControlFlowNode for object |
| block_flow_check | staticmethod | block_flow_check.py:0:0:0:0 | GSSA Variable staticmethod |
| foo | __file__ | foo.py:14:6:14:13 | ControlFlowNode for __file__ |
| foo | __name__ | foo.py:0:0:0:0 | GSSA Variable __name__ |
| foo | __package__ | foo.py:0:0:0:0 | GSSA Variable __package__ |
| foo | __private_foo_attr | foo.py:8:1:8:18 | GSSA Variable __private_foo_attr |
| foo | bar_reexported | foo.py:11:8:11:10 | ControlFlowNode for ImportExpr |
| foo | bar_reexported | foo.py:12:34:12:47 | ControlFlowNode for bar_reexported |
| foo | check | foo.py:12:1:12:5 | ControlFlowNode for check |
| foo | enter | foo.py:2:1:2:5 | ControlFlowNode for enter |
| foo | exit | foo.py:14:1:14:4 | ControlFlowNode for exit |
| foo | foo_attr | foo.py:5:1:5:8 | GSSA Variable foo_attr |
| foo | globals | foo.py:12:71:12:77 | ControlFlowNode for globals |
| generous_export | Exception | generous_export.py:16:11:16:19 | ControlFlowNode for Exception |
| generous_export | SOURCE | generous_export.py:15:11:15:16 | ControlFlowNode for SOURCE |
| generous_export | SOURCE | generous_export.py:20:25:20:30 | ControlFlowNode for SOURCE |
| generous_export | __file__ | generous_export.py:22:6:22:13 | ControlFlowNode for __file__ |
| generous_export | __name__ | generous_export.py:0:0:0:0 | GSSA Variable __name__ |
| generous_export | __package__ | generous_export.py:0:0:0:0 | GSSA Variable __package__ |
| generous_export | check | generous_export.py:20:1:20:5 | ControlFlowNode for check |
| generous_export | enter | generous_export.py:2:1:2:5 | ControlFlowNode for enter |
| generous_export | eval | generous_export.py:10:4:10:7 | ControlFlowNode for eval |
| generous_export | exit | generous_export.py:22:1:22:4 | ControlFlowNode for exit |
| generous_export | globals | generous_export.py:20:33:20:39 | ControlFlowNode for globals |
| generous_export | object | generous_export.py:4:14:4:19 | ControlFlowNode for object |
| generous_export | print | generous_export.py:15:5:15:9 | ControlFlowNode for print |
| generous_export | staticmethod | generous_export.py:0:0:0:0 | GSSA Variable staticmethod |
| has_defined_all | __all__ | has_defined_all.py:7:1:7:7 | GSSA Variable __all__ |
| has_defined_all | __file__ | has_defined_all.py:9:6:9:13 | ControlFlowNode for __file__ |
| has_defined_all | __name__ | has_defined_all.py:0:0:0:0 | GSSA Variable __name__ |
| has_defined_all | __package__ | has_defined_all.py:0:0:0:0 | GSSA Variable __package__ |
| has_defined_all | all_defined_bar | has_defined_all.py:5:1:5:15 | GSSA Variable all_defined_bar |
| has_defined_all | all_defined_foo | has_defined_all.py:4:1:4:15 | GSSA Variable all_defined_foo |
| has_defined_all | enter | has_defined_all.py:2:1:2:5 | ControlFlowNode for enter |
| has_defined_all | exit | has_defined_all.py:9:1:9:4 | ControlFlowNode for exit |
| has_defined_all_copy | __all__ | has_defined_all_copy.py:9:1:9:7 | GSSA Variable __all__ |
| has_defined_all_copy | __file__ | has_defined_all_copy.py:11:6:11:13 | ControlFlowNode for __file__ |
| has_defined_all_copy | __name__ | has_defined_all_copy.py:0:0:0:0 | GSSA Variable __name__ |
| has_defined_all_copy | __package__ | has_defined_all_copy.py:0:0:0:0 | GSSA Variable __package__ |
| has_defined_all_copy | all_defined_bar_copy | has_defined_all_copy.py:7:1:7:20 | GSSA Variable all_defined_bar_copy |
| has_defined_all_copy | all_defined_foo_copy | has_defined_all_copy.py:6:1:6:20 | GSSA Variable all_defined_foo_copy |
| has_defined_all_copy | enter | has_defined_all_copy.py:4:1:4:5 | ControlFlowNode for enter |
| has_defined_all_copy | exit | has_defined_all_copy.py:11:1:11:4 | ControlFlowNode for exit |
| has_defined_all_indirection | __file__ | has_defined_all_indirection.py:6:6:6:13 | ControlFlowNode for __file__ |
| has_defined_all_indirection | __name__ | has_defined_all_indirection.py:0:0:0:0 | GSSA Variable __name__ |
| has_defined_all_indirection | __package__ | has_defined_all_indirection.py:0:0:0:0 | GSSA Variable __package__ |
| has_defined_all_indirection | all_defined_foo_copy | has_defined_all_copy.py:6:1:6:20 | GSSA Variable all_defined_foo_copy |
| has_defined_all_indirection | enter | has_defined_all_indirection.py:2:1:2:5 | ControlFlowNode for enter |
| has_defined_all_indirection | exit | has_defined_all_indirection.py:6:1:6:4 | ControlFlowNode for exit |
| if_then_else | __file__ | if_then_else.py:16:6:16:13 | ControlFlowNode for __file__ |
| if_then_else | __name__ | if_then_else.py:0:0:0:0 | GSSA Variable __name__ |
| if_then_else | __package__ | if_then_else.py:0:0:0:0 | GSSA Variable __package__ |
| if_then_else | enter | if_then_else.py:2:1:2:5 | ControlFlowNode for enter |
| if_then_else | eval | if_then_else.py:11:8:11:11 | ControlFlowNode for eval |
| if_then_else | exit | if_then_else.py:16:1:16:4 | ControlFlowNode for exit |
| if_then_else | if_then_else_defined | if_then_else.py:7:5:7:24 | GSSA Variable if_then_else_defined |
| if_then_else | if_then_else_defined | if_then_else.py:12:9:12:28 | GSSA Variable if_then_else_defined |
| if_then_else | if_then_else_defined | if_then_else.py:14:9:14:28 | GSSA Variable if_then_else_defined |
| if_then_else_refined | SOURCE | if_then_else_refined.py:11:11:11:16 | ControlFlowNode for SOURCE |
| if_then_else_refined | SOURCE | if_then_else_refined.py:13:11:13:16 | ControlFlowNode for SOURCE |
| if_then_else_refined | __file__ | if_then_else_refined.py:19:6:19:13 | ControlFlowNode for __file__ |
| if_then_else_refined | __name__ | if_then_else_refined.py:0:0:0:0 | GSSA Variable __name__ |
| if_then_else_refined | __package__ | if_then_else_refined.py:0:0:0:0 | GSSA Variable __package__ |
| if_then_else_refined | check | if_then_else_refined.py:17:1:17:5 | ControlFlowNode for check |
| if_then_else_refined | enter | if_then_else_refined.py:4:1:4:5 | ControlFlowNode for enter |
| if_then_else_refined | eval | if_then_else_refined.py:10:4:10:7 | ControlFlowNode for eval |
| if_then_else_refined | exit | if_then_else_refined.py:19:1:19:4 | ControlFlowNode for exit |
| if_then_else_refined | globals | if_then_else_refined.py:17:24:17:30 | ControlFlowNode for globals |
| if_then_else_refined | src | if_then_else_refined.py:17:19:17:21 | ControlFlowNode for src |
| package.__init__ | __file__ | package/__init__.py:7:6:7:13 | ControlFlowNode for __file__ |
| package.__init__ | __name__ | package/__init__.py:0:0:0:0 | GSSA Variable __name__ |
| package.__init__ | __package__ | package/__init__.py:0:0:0:0 | GSSA Variable __package__ |
| package.__init__ | attr_used_in_subpackage | package/__init__.py:4:1:4:23 | GSSA Variable attr_used_in_subpackage |
| package.__init__ | enter | package/__init__.py:2:1:2:5 | ControlFlowNode for enter |
| package.__init__ | exit | package/__init__.py:7:1:7:4 | ControlFlowNode for exit |
| package.__init__ | package_attr | package/__init__.py:5:1:5:12 | GSSA Variable package_attr |
| package.subpackage2.__init__ | __file__ | package/subpackage2/__init__.py:6:6:6:13 | ControlFlowNode for __file__ |
| package.subpackage2.__init__ | __name__ | package/subpackage2/__init__.py:0:0:0:0 | GSSA Variable __name__ |
| package.subpackage2.__init__ | __package__ | package/subpackage2/__init__.py:0:0:0:0 | GSSA Variable __package__ |
| package.subpackage2.__init__ | enter | package/subpackage2/__init__.py:2:1:2:5 | ControlFlowNode for enter |
| package.subpackage2.__init__ | exit | package/subpackage2/__init__.py:6:1:6:4 | ControlFlowNode for exit |
| package.subpackage2.__init__ | subpackage2_attr | package/subpackage2/__init__.py:4:1:4:16 | GSSA Variable subpackage2_attr |
| package.subpackage.__init__ | __file__ | package/subpackage/__init__.py:14:6:14:13 | ControlFlowNode for __file__ |
| package.subpackage.__init__ | __name__ | package/subpackage/__init__.py:0:0:0:0 | GSSA Variable __name__ |
| package.subpackage.__init__ | __package__ | package/subpackage/__init__.py:0:0:0:0 | GSSA Variable __package__ |
| package.subpackage.__init__ | check | package/subpackage/__init__.py:12:1:12:5 | ControlFlowNode for check |
| package.subpackage.__init__ | enter | package/subpackage/__init__.py:2:1:2:5 | ControlFlowNode for enter |
| package.subpackage.__init__ | exit | package/subpackage/__init__.py:14:1:14:4 | ControlFlowNode for exit |
| package.subpackage.__init__ | globals | package/subpackage/__init__.py:12:79:12:85 | ControlFlowNode for globals |
| package.subpackage.__init__ | imported_attr | package/subpackage/__init__.py:7:16:7:55 | ControlFlowNode for ImportMember |
| package.subpackage.__init__ | imported_attr | package/subpackage/__init__.py:8:24:8:36 | ControlFlowNode for imported_attr |
| package.subpackage.__init__ | irrelevant_attr | package/subpackage/__init__.py:11:24:11:38 | ControlFlowNode for ImportMember |
| package.subpackage.__init__ | irrelevant_attr | package/subpackage/__init__.py:11:24:11:38 | GSSA Variable irrelevant_attr |
| package.subpackage.__init__ | submodule | package/subpackage/__init__.py:12:35:12:43 | ControlFlowNode for submodule |
| package.subpackage.__init__ | subpackage_attr | package/subpackage/__init__.py:4:1:4:15 | GSSA Variable subpackage_attr |
| package.subpackage.submodule | __file__ | package/subpackage/submodule.py:7:6:7:13 | ControlFlowNode for __file__ |
| package.subpackage.submodule | __name__ | package/subpackage/submodule.py:0:0:0:0 | GSSA Variable __name__ |
| package.subpackage.submodule | __package__ | package/subpackage/submodule.py:0:0:0:0 | GSSA Variable __package__ |
| package.subpackage.submodule | enter | package/subpackage/submodule.py:2:1:2:5 | ControlFlowNode for enter |
| package.subpackage.submodule | exit | package/subpackage/submodule.py:7:1:7:4 | ControlFlowNode for exit |
| package.subpackage.submodule | irrelevant_attr | package/subpackage/submodule.py:5:1:5:15 | GSSA Variable irrelevant_attr |
| package.subpackage.submodule | submodule_attr | package/subpackage/submodule.py:4:1:4:14 | GSSA Variable submodule_attr |
| refined | SOURCE | refined.py:12:25:12:30 | ControlFlowNode for SOURCE |
| refined | __file__ | refined.py:14:6:14:13 | ControlFlowNode for __file__ |
| refined | __name__ | refined.py:0:0:0:0 | GSSA Variable __name__ |
| refined | __package__ | refined.py:0:0:0:0 | GSSA Variable __package__ |
| refined | check | refined.py:12:1:12:5 | ControlFlowNode for check |
| refined | enter | refined.py:2:1:2:5 | ControlFlowNode for enter |
| refined | exit | refined.py:14:1:14:4 | ControlFlowNode for exit |
| refined | globals | refined.py:12:33:12:39 | ControlFlowNode for globals |
| refined | object | refined.py:4:14:4:19 | ControlFlowNode for object |
| simplistic_reexport | __file__ | simplistic_reexport.py:19:6:19:13 | ControlFlowNode for __file__ |
| simplistic_reexport | __name__ | simplistic_reexport.py:0:0:0:0 | GSSA Variable __name__ |
| simplistic_reexport | __package__ | simplistic_reexport.py:0:0:0:0 | GSSA Variable __package__ |
| simplistic_reexport | bar_attr | simplistic_reexport.py:6:17:6:24 | ControlFlowNode for ImportMember |
| simplistic_reexport | bar_attr | simplistic_reexport.py:10:19:10:26 | ControlFlowNode for bar_attr |
| simplistic_reexport | baz_attr | baz.py:4:1:4:8 | GSSA Variable baz_attr |
| simplistic_reexport | baz_attr | simplistic_reexport.py:17:19:17:26 | ControlFlowNode for baz_attr |
| simplistic_reexport | check | simplistic_reexport.py:17:1:17:5 | ControlFlowNode for check |
| simplistic_reexport | enter | baz.py:2:1:2:5 | ControlFlowNode for enter |
| simplistic_reexport | enter | simplistic_reexport.py:4:1:4:5 | ControlFlowNode for enter |
| simplistic_reexport | exit | baz.py:6:1:6:4 | ControlFlowNode for exit |
| simplistic_reexport | exit | simplistic_reexport.py:19:1:19:4 | ControlFlowNode for exit |
| simplistic_reexport | globals | simplistic_reexport.py:17:44:17:50 | ControlFlowNode for globals |

17
python/ql/test/experimental/import-resolution/ModuleExport.ql Normal file
View File

@@ -0,0 +1,17 @@
import python
import semmle.python.dataflow.new.DataFlow
import semmle.python.dataflow.new.internal.ImportResolution
from Module m, string name, DataFlow::Node defn
where
ImportResolution::module_export(m, name, defn) and
exists(m.getLocation().getFile().getRelativePath()) and
not defn.getScope() = any(Module trace | trace.getName() = "trace") and
not m.getName() = "main" and
// Since we test on both Python 2 and Python 3, but `namespace_package` is not allowed
// on Python 2 because of the missing `__init__.py` files, we remove those results
// from Python 3 tests as well. One alternative is to only run these tests under
// Python 3, but that does not seems like a good solution -- we could easily miss a
// Python 2 only regression then :O
not m.getName() = "namespace_package.namespace_module"
select m.getName(), name, defn

6
python/ql/test/experimental/import-resolution/baz.py Normal file
View File

@@ -0,0 +1,6 @@
from trace import *
enter(__file__)
baz_attr = "baz_attr"
exit(__file__)

14
python/ql/test/experimental/import-resolution/block_flow_check.py Normal file
View File

@@ -0,0 +1,14 @@
from trace import *
enter(__file__)
class SOURCE(object):
@staticmethod
def block_flow(): pass
check("SOURCE", SOURCE, SOURCE, globals()) #$ prints=SOURCE
SOURCE.block_flow()
check("SOURCE", SOURCE, SOURCE, globals())
exit(__file__)

22
python/ql/test/experimental/import-resolution/generous_export.py Normal file
View File

@@ -0,0 +1,22 @@
from trace import *
enter(__file__)
class SOURCE(object):
@staticmethod
def block_flow(): pass
check("SOURCE", SOURCE, SOURCE, globals()) #$ prints=SOURCE
if eval("False"):
# With our current import resolution, this value for SOURCE will be considered to be
# a valid value at the end of this module, because it's the end of a use-use flow.
# This is clearly wrong, so our import resolution is a bit too generous on what is
# exported
print(SOURCE)
raise Exception()
SOURCE.block_flow()
check("SOURCE", SOURCE, SOURCE, globals())
exit(__file__)

9
python/ql/test/experimental/import-resolution/has_defined_all.py Normal file
View File

@@ -0,0 +1,9 @@
from trace import *
enter(__file__)
all_defined_foo = "all_defined_foo"
all_defined_bar = "all_defined_bar"
__all__ = ["all_defined_foo"]
exit(__file__)

11
python/ql/test/experimental/import-resolution/has_defined_all_copy.py Normal file
View File

@@ -0,0 +1,11 @@
# a copy of `has_defined_all.py` that is imported by `has_defined_all_indirection.py`
# with its' own names such that we can check both `import *` without any cross-talk
from trace import *
enter(__file__)
all_defined_foo_copy = "all_defined_foo_copy"
all_defined_bar_copy = "all_defined_bar_copy"
__all__ = ["all_defined_foo_copy"]
exit(__file__)

6
python/ql/test/experimental/import-resolution/has_defined_all_indirection.py Normal file
View File

@@ -0,0 +1,6 @@
from trace import *
enter(__file__)
from has_defined_all_copy import *
exit(__file__)

19
python/ql/test/experimental/import-resolution/if_then_else_refined.py Normal file
View File

@@ -0,0 +1,19 @@
# combination of refined and if_then_else
from trace import *
enter(__file__)
class SOURCE(): pass
# definition based on "random" choice in this case it will always go the the if-branch,
# but our analysis is not able to figure this out
if eval("True"):
src = SOURCE
else:
src = SOURCE
src.foo = 42
check("src", src, src, globals()) #$ prints=SOURCE
exit(__file__)

7
python/ql/test/experimental/import-resolution/importflow.ql
View File

@@ -11,6 +11,9 @@ private class SourceString extends DataFlow::Node {
SourceString() {
this.asExpr().(StrConst).getText() = contents and
this.asExpr().getParent() instanceof Assign
or
this.asExpr().(ClassExpr).getInnerScope().getName() = "SOURCE" and
contents = "SOURCE"
}
string getContents() { result = contents }
@@ -63,6 +66,10 @@ private class ImportConfiguration extends DataFlow::Configuration {
override predicate isSink(DataFlow::Node sink) {
sink = API::moduleImport("trace").getMember("check").getACall().getArg(1)
}
override predicate isBarrier(DataFlow::Node node) {
exists(DataFlow::MethodCallNode call | call.calls(node, "block_flow"))
}
}
class ResolutionTest extends InlineExpectationsTest {

65
python/ql/test/experimental/import-resolution/main.py
View File

@@ -84,6 +84,52 @@ from attr_clash import clashing_attr, non_clashing_submodule #$ imports=attr_cla
check("clashing_attr", clashing_attr, "clashing_attr", globals()) #$ prints=clashing_attr SPURIOUS: prints="<module attr_clash.clashing_attr>"
check("non_clashing_submodule", non_clashing_submodule, "<module attr_clash.non_clashing_submodule>", globals()) #$ prints="<module attr_clash.non_clashing_submodule>"
import attr_clash.clashing_attr as _doesnt_matter #$ imports=attr_clash.clashing_attr as=_doesnt_matter
from attr_clash import clashing_attr, non_clashing_submodule #$ imports=attr_clash.clashing_attr as=clashing_attr imports=attr_clash.non_clashing_submodule as=non_clashing_submodule
check("clashing_attr", clashing_attr, "<module attr_clash.clashing_attr>", globals()) #$ prints="<module attr_clash.clashing_attr>" SPURIOUS: prints=clashing_attr
# check that import * only imports the __all__ attributes
from has_defined_all import *
check("all_defined_foo", all_defined_foo, "all_defined_foo", globals()) #$ prints=all_defined_foo
try:
check("all_defined_bar", all_defined_bar, "all_defined_bar", globals()) #$ SPURIOUS: prints=all_defined_bar
raise Exception("Did not get expected NameError")
except NameError as e:
if "all_defined_bar" in str(e):
print("Got expected NameError:", e)
else:
raise
import has_defined_all # $ imports=has_defined_all as=has_defined_all
check("has_defined_all.all_defined_foo", has_defined_all.all_defined_foo, "all_defined_foo", globals()) #$ prints=all_defined_foo
check("has_defined_all.all_defined_bar", has_defined_all.all_defined_bar, "all_defined_bar", globals()) #$ prints=all_defined_bar
# same check as above, but going through one level of indirection (which can make a difference)
from has_defined_all_indirection import *
check("all_defined_foo_copy", all_defined_foo_copy, "all_defined_foo_copy", globals()) #$ prints=all_defined_foo_copy
try:
check("all_defined_bar_copy", all_defined_bar_copy, "all_defined_bar_copy", globals()) #$ SPURIOUS: prints=all_defined_bar_copy
raise Exception("Did not get expected NameError")
except NameError as e:
if "all_defined_bar_copy" in str(e):
print("Got expected NameError:", e)
else:
raise
# same check as above, but going through one level of indirection (which can make a difference)
import has_defined_all_indirection # $ imports=has_defined_all_indirection as=has_defined_all_indirection
check("has_defined_all_indirection.all_defined_foo_copy", has_defined_all_indirection.all_defined_foo_copy, "all_defined_foo_copy", globals()) #$ prints=all_defined_foo_copy
try:
check("has_defined_all_indirection.all_defined_bar_copy", has_defined_all_indirection.all_defined_bar_copy, "all_defined_bar_copy", globals())
raise Exception("Did not get expected AttributeError")
except AttributeError as e:
if "all_defined_bar_copy" in str(e):
print("Got expected AttributeError:", e)
else:
raise
# check that import * from an __init__ file works
from package.subpackage2 import *
@@ -93,6 +139,25 @@ check("subpackage2_attr", subpackage2_attr, "subpackage2_attr", globals()) #$ pr
from if_then_else import if_then_else_defined
check("if_then_else_defined", if_then_else_defined, "if_defined", globals()) #$ prints=if_defined prints=else_defined_1 prints=else_defined_2
# check that refined definitions are handled correctly
import refined # $ imports=refined as=refined
check("refined.SOURCE", refined.SOURCE, refined.SOURCE, globals()) #$ prints=SOURCE
import if_then_else_refined # $ imports=if_then_else_refined as=if_then_else_refined
check("if_then_else_refined.src", if_then_else_refined.src, if_then_else_refined.src, globals()) #$ prints=SOURCE
import simplistic_reexport # $ imports=simplistic_reexport as=simplistic_reexport
check("simplistic_reexport.bar_attr", simplistic_reexport.bar_attr, "overwritten", globals()) #$ prints=overwritten SPURIOUS: prints=bar_attr
check("simplistic_reexport.baz_attr", simplistic_reexport.baz_attr, "overwritten", globals()) #$ prints=overwritten SPURIOUS: prints=baz_attr
# check that we don't treat all assignments as being exports
import block_flow_check #$ imports=block_flow_check as=block_flow_check
check("block_flow_check.SOURCE", block_flow_check.SOURCE, block_flow_check.SOURCE, globals())
# show that import resolution is a bit too generous with definitions
import generous_export #$ imports=generous_export as=generous_export
check("generous_export.SOURCE", generous_export.SOURCE, generous_export.SOURCE, globals()) #$ SPURIOUS: prints=SOURCE
exit(__file__)
print()

14
python/ql/test/experimental/import-resolution/refined.py Normal file
View File

@@ -0,0 +1,14 @@
from trace import *
enter(__file__)
class SOURCE(object): pass
check("SOURCE", SOURCE, SOURCE, globals()) #$ prints=SOURCE
SOURCE.foo = 42
SOURCE.bar = 43
SOURCE.baz = 44
check("SOURCE", SOURCE, SOURCE, globals()) #$ prints=SOURCE
exit(__file__)

19
python/ql/test/experimental/import-resolution/simplistic_reexport.py Normal file
View File

@@ -0,0 +1,19 @@
# we might consider anything imported to also be exported, but this is not the case
from trace import *
enter(__file__)
from bar import bar_attr
check("bar_attr", bar_attr, "bar_attr", globals()) #$ prints=bar_attr
bar_attr = "overwritten"
check("bar_attr", bar_attr, "overwritten", globals()) #$ prints=overwritten
from baz import *
check("baz_attr", baz_attr, "baz_attr", globals()) #$ MISSING: prints=baz_attr
baz_attr = "overwritten"
check("baz_attr", baz_attr, "overwritten", globals()) #$ prints=overwritten
exit(__file__)

12
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.expected Normal file
View File

@@ -0,0 +1,12 @@
edges
| TimingAttackAgainstHash.py:26:17:26:41 | ControlFlowNode for Attribute() | TimingAttackAgainstHash.py:27:24:27:32 | ControlFlowNode for signature |
| TimingAttackAgainstHash.py:30:12:30:47 | ControlFlowNode for Attribute() | TimingAttackAgainstHash.py:37:19:37:48 | ControlFlowNode for sign() |
nodes
| TimingAttackAgainstHash.py:26:17:26:41 | ControlFlowNode for Attribute() | semmle.label | ControlFlowNode for Attribute() |
| TimingAttackAgainstHash.py:27:24:27:32 | ControlFlowNode for signature | semmle.label | ControlFlowNode for signature |
| TimingAttackAgainstHash.py:30:12:30:47 | ControlFlowNode for Attribute() | semmle.label | ControlFlowNode for Attribute() |
| TimingAttackAgainstHash.py:37:19:37:48 | ControlFlowNode for sign() | semmle.label | ControlFlowNode for sign() |
subpaths
#select
| TimingAttackAgainstHash.py:27:24:27:32 | ControlFlowNode for signature | TimingAttackAgainstHash.py:26:17:26:41 | ControlFlowNode for Attribute() | TimingAttackAgainstHash.py:27:24:27:32 | ControlFlowNode for signature | Possible Timing attack against $@ validation. | signature | message |
| TimingAttackAgainstHash.py:37:19:37:48 | ControlFlowNode for sign() | TimingAttackAgainstHash.py:30:12:30:47 | ControlFlowNode for Attribute() | TimingAttackAgainstHash.py:37:19:37:48 | ControlFlowNode for sign() | Possible Timing attack against $@ validation. | MAC | message |

1
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.qlref Normal file
View File

@@ -0,0 +1 @@
experimental/Security/CWE-208/TimingAttackAgainstHash/PossibleTimingAttackAgainstHash.ql

48
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHash/TimingAttackAgainstHash.py Normal file
View File

@@ -0,0 +1,48 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：Timing Attack Against Hash
"""
import hashlib
import hmac
from cryptography.hazmat.primitives import cmac
from cryptography.hazmat.primitives.ciphers import algorithms
from Crypto.Signature import pkcs1_15
from Crypto.Hash import SHA256
from Crypto.PublicKey import RSA
def UnsafeCmacCheck(actualCmac):
expected = cmac.CMAC(algorithms.AES(key))
expected.update(b"message to authenticate")
expected.finalize()
return actualCmac == expected
def UnsafeCheckSignature(expected):
message = b'To be signed'
key = RSA.import_key(open('private_key.der').read())
h = SHA256.new(message)
signature = pkcs1_15.new(key).sign(h)
return expected == signature
def sign(pre_key, msg, alg):
return hmac.new(pre_key, msg, alg).digest()
def verifyGood(msg, sig):
return constant_time_string_compare(sig, sign(key, msg, hashlib.sha256)) #good
def verifyBad(msg, sig):
key = "e179017a-62b0-4996-8a38-e91aa9f1"
return sig == sign(key, msg, hashlib.sha256) #bad
def constant_time_string_compare(a, b):
if len(a) != len(b):
return False
result = 0
for x, y in zip(a, b):
result |= ord(x) ^ ord(y)
return result == 0

27
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHeaderValue/TimingAttackAgainstHeader.py Normal file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
"""
@Desc ：Timing Attack Against Header
"""
from flask import Flask
from flask import request
import hmac
app = Flask(__name__)
@app.route('/bad')
def bad():
if not request.headers.get('X-Auth-Token') == "token":
raise Exception('bad token')
return 'bad'
@app.route('/good')
def good():
tok = request.headers.get('X-Auth-Token')
if not hmac.compare_digest(tok, "token"):
raise Exception('bad token')
return 'good'
if __name__ == '__main__':
app.debug = True
app.run()

6
python/ql/test/experimental/query-tests/Security/CWE-208/TimingAttackAgainstHeaderValue/TimingAttackAgainstHeaderValue.expected Normal file
View File

@@ -0,0 +1,6 @@
edges
nodes
| TimingAttackAgainstHeader.py:14:12:14:46 | ControlFlowNode for Attribute() | semmle.label | ControlFlowNode for Attribute() |
subpaths
#select
| TimingAttackAgainstHeader.py:14:12:14:46 | ControlFlowNode for Attribute() | TimingAttackAgainstHeader.py:14:12:14:46 | ControlFlowNode for Attribute() | TimingAttackAgainstHeader.py:14:12:14:46 | ControlFlowNode for Attribute() | Timing attack against $@ validation. | TimingAttackAgainstHeader.py:14:12:14:46 | ControlFlowNode for Attribute() | client-supplied token |

Some files were not shown because too many files have changed in this diff Show More