hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2025-12-24 04:36:35 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into amammad-python-WebAppsConstatntSecretKeys

Browse Source
This commit is contained in:
Rasmus Wriedt Larsen
2023-07-14 14:32:43 +02:00
parent 2ba83022c7 9193de6898
commit 8279cf7c9c
677 changed files with 15882 additions and 4544 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
python/ql/lib/CHANGELOG.md
View File

@@ -1,3 +1,26 @@
## 0.10.0
### New Features
* It is now possible to specify flow summaries in the format "MyPkg;Member[list_map];Argument[1].ListElement;Argument[0].Parameter[0];value"
### Minor Analysis Improvements
* Deleted many models that used the old dataflow library, the new models can be found in the `python/ql/lib/semmle/python/frameworks` folder.
* More precise modeling of several container functions (such as `sorted`, `reversed`) and methods (such as `set.add`, `list.append`).
* Added modeling of taint flow through the template argument of `flask.render_template_string` and `flask.stream_template_string`.
* Deleted many deprecated predicates and classes with uppercase `API`, `HTTP`, `XSS`, `SQL`, etc. in their names. Use the PascalCased versions instead.
* Deleted the deprecated `getName()` predicate from the `Container` class, use `getAbsolutePath()` instead.
* Deleted many deprecated module names that started with a lowercase letter, use the versions that start with an uppercase letter instead.
* Deleted many deprecated predicates in `PointsTo.qll`.
* Deleted many deprecated files from the `semmle.python.security` package.
* Deleted the deprecated `BottleRoutePointToExtension` class from `Extensions.qll`.
* Type tracking is now aware of flow summaries. This leads to a richer API graph, and may lead to more results in some queries.
## 0.9.4
No user-facing changes.
## 0.9.3
No user-facing changes.

4
python/ql/lib/change-notes/2023-05-30-typetracking-via-flow-summaries.md
View File

@@ -1,4 +0,0 @@
---
category: minorAnalysis
---
* Type tracking is now aware of flow summaries. This leads to a richer API graph, and may lead to more results in some queries.

9
python/ql/lib/change-notes/2023-06-09-delete-deps.md
View File

@@ -1,9 +0,0 @@
---
category: minorAnalysis
---
* Deleted many deprecated predicates and classes with uppercase `API`, `HTTP`, `XSS`, `SQL`, etc. in their names. Use the PascalCased versions instead.
* Deleted the deprecated `getName()` predicate from the `Container` class, use `getAbsolutePath()` instead.
* Deleted many deprecated module names that started with a lowercase letter, use the versions that start with an uppercase letter instead.
* Deleted many deprecated predicates in `PointsTo.qll`.
* Deleted many deprecated files from the `semmle.python.security` package.
* Deleted the deprecated `BottleRoutePointToExtension` class from `Extensions.qll`.

4
python/ql/lib/change-notes/2023-06-12-flask-render-template-string.md
View File

@@ -1,4 +0,0 @@
---
category: minorAnalysis
---
* Added modeling of taint flow through the template argument of `flask.render_template_string` and `flask.stream_template_string`.

4
python/ql/lib/change-notes/2023-06-13-container-store-steps.md
View File

@@ -1,4 +0,0 @@
---
category: minorAnalysis
---
* More precise modelling of several container functions (such as `sorted`, `reversed`) and methods (such as `set.add`, `list.append`).

4
python/ql/lib/change-notes/2023-06-14-delete-deps.md
View File

@@ -1,4 +0,0 @@
---
category: minorAnalysis
---
* Deleted many models that used the old dataflow library, the new models can be found in the `python/ql/lib/semmle/python/frameworks` folder.

4
python/ql/lib/change-notes/2023-06-20-summaries-from-models.md
View File

@@ -1,4 +0,0 @@
---
category: feature
---
* It is now possible to specify flow summaries in the format "MyPkg;Member[list_map];Argument[1].ListElement;Argument[0].Parameter[0];value"

6
python/ql/lib/change-notes/2023-07-12-default-stateconfigsig-predicates.md Normal file
View File

@@ -0,0 +1,6 @@
---
category: feature
---
* The `DataFlow::StateConfigSig` signature module has gained default implementations for `isBarrier/2` and `isAdditionalFlowStep/4`.
Hence it is no longer needed to provide `none()` implementations of these predicates if they are not needed.

18
python/ql/lib/change-notes/released/0.10.0.md Normal file
View File

@@ -0,0 +1,18 @@
## 0.10.0
### New Features
* It is now possible to specify flow summaries in the format "MyPkg;Member[list_map];Argument[1].ListElement;Argument[0].Parameter[0];value"
### Minor Analysis Improvements
* Deleted many models that used the old dataflow library, the new models can be found in the `python/ql/lib/semmle/python/frameworks` folder.
* More precise modeling of several container functions (such as `sorted`, `reversed`) and methods (such as `set.add`, `list.append`).
* Added modeling of taint flow through the template argument of `flask.render_template_string` and `flask.stream_template_string`.
* Deleted many deprecated predicates and classes with uppercase `API`, `HTTP`, `XSS`, `SQL`, etc. in their names. Use the PascalCased versions instead.
* Deleted the deprecated `getName()` predicate from the `Container` class, use `getAbsolutePath()` instead.
* Deleted many deprecated module names that started with a lowercase letter, use the versions that start with an uppercase letter instead.
* Deleted many deprecated predicates in `PointsTo.qll`.
* Deleted many deprecated files from the `semmle.python.security` package.
* Deleted the deprecated `BottleRoutePointToExtension` class from `Extensions.qll`.
* Type tracking is now aware of flow summaries. This leads to a richer API graph, and may lead to more results in some queries.

3
python/ql/lib/change-notes/released/0.9.4.md Normal file
View File

@@ -0,0 +1,3 @@
## 0.9.4
No user-facing changes.

2
python/ql/lib/codeql-pack.release.yml
View File

@@ -1,2 +1,2 @@
---
lastReleaseVersion: 0.9.3
lastReleaseVersion: 0.10.0

2
python/ql/lib/qlpack.yml
View File

@@ -1,5 +1,5 @@
name: codeql/python-all
version: 0.9.4-dev
version: 0.10.1-dev
groups: python
dbscheme: semmlecode.python.dbscheme
extractor: python

12
python/ql/lib/semmle/python/Concepts.qll
View File

@@ -680,6 +680,9 @@ module Escaping {
/** Gets the escape-kind for escaping a string so it can safely be included in HTML. */
string getHtmlKind() { result = "html" }
/** Gets the escape-kind for escaping a string so it can safely be included in XML. */
string getXmlKind() { result = "xml" }
/** Gets the escape-kind for escaping a string so it can safely be included in a regular expression. */
string getRegexKind() { result = "regex" }
@@ -710,6 +713,15 @@ class HtmlEscaping extends Escaping {
HtmlEscaping() { super.getKind() = Escaping::getHtmlKind() }
}
/**
* An escape of a string so it can be safely included in
* the body of an XML element, for example, replacing `&` and `<>` in
* `<foo>&xxe;<foo>`.
*/
class XmlEscaping extends Escaping {
XmlEscaping() { super.getKind() = Escaping::getXmlKind() }
}
/**
* An escape of a string so it can be safely included in
* the body of a regex.

6
python/ql/lib/semmle/python/dataflow/new/internal/DataFlow.qll
View File

@@ -114,7 +114,7 @@ signature module StateConfigSig {
* Holds if data flow through `node` is prohibited when the flow state is
* `state`.
*/
predicate isBarrier(Node node, FlowState state);
default predicate isBarrier(Node node, FlowState state) { none() }
/** Holds if data flow into `node` is prohibited. */
default predicate isBarrierIn(Node node) { none() }
@@ -131,7 +131,9 @@ signature module StateConfigSig {
* Holds if data may flow from `node1` to `node2` in addition to the normal data-flow steps.
* This step is only applicable in `state1` and updates the flow state to `state2`.
*/
predicate isAdditionalFlowStep(Node node1, FlowState state1, Node node2, FlowState state2);
default predicate isAdditionalFlowStep(Node node1, FlowState state1, Node node2, FlowState state2) {
none()
}
/**
* Holds if an arbitrary number of implicit read steps of content `c` may be

1
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowDispatch.qll
View File

@@ -1227,7 +1227,6 @@ predicate normalCallArg(CallNode call, Node arg, ArgumentPosition apos) {
* time the bound method is used, such that the `clear()` call would essentially be
* translated into `l.clear()`, and we can still have use-use flow.
*/
pragma[assume_small_delta]
cached
predicate getCallArg(CallNode call, Function target, CallType type, Node arg, ArgumentPosition apos) {
Stages::DataFlow::ref() and

48
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl.qll
View File

@@ -254,6 +254,11 @@ module Impl<FullStateConfigSig Config> {
not fullBarrier(node2)
}
pragma[nomagic]
private predicate isUnreachableInCall1(NodeEx n, LocalCallContextSpecificCall cc) {
isUnreachableInCallCached(n.asNode(), cc.getCall())
}
/**
* Holds if data can flow in one local step from `node1` to `node2`.
*/
@@ -460,7 +465,6 @@ module Impl<FullStateConfigSig Config> {
* The Boolean `cc` records whether the node is reached through an
* argument in a call.
*/
pragma[assume_small_delta]
private predicate fwdFlow(NodeEx node, Cc cc) {
sourceNode(node, _) and
if hasSourceCallCtx() then cc = true else cc = false
@@ -570,7 +574,6 @@ module Impl<FullStateConfigSig Config> {
/**
* Holds if `c` is the target of a store in the flow covered by `fwdFlow`.
*/
pragma[assume_small_delta]
pragma[nomagic]
private predicate fwdFlowConsCand(Content c) {
exists(NodeEx mid, NodeEx node |
@@ -1216,7 +1219,6 @@ module Impl<FullStateConfigSig Config> {
fwdFlow1(_, _, _, _, _, _, t0, t, ap, _) and t0 != t
}
pragma[assume_small_delta]
pragma[nomagic]
private predicate fwdFlow0(
NodeEx node, FlowState state, Cc cc, ParamNodeOption summaryCtx, TypOption argT,
@@ -2111,7 +2113,7 @@ module Impl<FullStateConfigSig Config> {
NodeEx node1, FlowState state, NodeEx node2, boolean preservesValue, DataFlowType t,
LocalCallContext cc
) {
not isUnreachableInCallCached(node2.asNode(), cc.(LocalCallContextSpecificCall).getCall()) and
not isUnreachableInCall1(node2, cc) and
(
localFlowEntry(node1, pragma[only_bind_into](state)) and
(
@@ -2126,7 +2128,7 @@ module Impl<FullStateConfigSig Config> {
) and
node1 != node2 and
cc.relevantFor(node1.getEnclosingCallable()) and
not isUnreachableInCallCached(node1.asNode(), cc.(LocalCallContextSpecificCall).getCall())
not isUnreachableInCall1(node1, cc)
or
exists(NodeEx mid |
localFlowStepPlus(node1, pragma[only_bind_into](state), mid, preservesValue, t, cc) and
@@ -2163,10 +2165,8 @@ module Impl<FullStateConfigSig Config> {
preservesValue = false and
t = node2.getDataFlowType() and
callContext.relevantFor(node1.getEnclosingCallable()) and
not exists(DataFlowCall call | call = callContext.(LocalCallContextSpecificCall).getCall() |
isUnreachableInCallCached(node1.asNode(), call) or
isUnreachableInCallCached(node2.asNode(), call)
)
not isUnreachableInCall1(node1, callContext) and
not isUnreachableInCall1(node2, callContext)
}
}
@@ -2706,7 +2706,7 @@ module Impl<FullStateConfigSig Config> {
ParamNodeEx getParamNode() { result = p }
override string toString() { result = p + ": " + ap }
override string toString() { result = p + concat(" : " + ppReprType(t)) + " " + ap }
predicate hasLocationInfo(
string filepath, int startline, int startcolumn, int endline, int endcolumn
@@ -2758,12 +2758,21 @@ module Impl<FullStateConfigSig Config> {
)
}
private predicate forceUnfold(AccessPathApprox apa) {
forceHighPrecision(apa.getHead())
or
exists(Content c2 |
apa = TConsCons(_, _, c2, _) and
forceHighPrecision(c2)
)
}
/**
* Holds with `unfold = false` if a precise head-tail representation of `apa` is
* expected to be expensive. Holds with `unfold = true` otherwise.
*/
private predicate evalUnfold(AccessPathApprox apa, boolean unfold) {
if forceHighPrecision(apa.getHead())
if forceUnfold(apa)
then unfold = true
else
exists(int aps, int nodes, int apLimit, int tupleLimit |
@@ -2777,7 +2786,6 @@ module Impl<FullStateConfigSig Config> {
/**
* Gets the number of `AccessPath`s that correspond to `apa`.
*/
pragma[assume_small_delta]
private int countAps(AccessPathApprox apa) {
evalUnfold(apa, false) and
result = 1 and
@@ -2796,7 +2804,6 @@ module Impl<FullStateConfigSig Config> {
* that it is expanded to a precise head-tail representation.
*/
language[monotonicAggregates]
pragma[assume_small_delta]
private int countPotentialAps(AccessPathApprox apa) {
apa instanceof AccessPathApproxNil and result = 1
or
@@ -2833,7 +2840,6 @@ module Impl<FullStateConfigSig Config> {
}
private newtype TPathNode =
pragma[assume_small_delta]
TPathNodeMid(
NodeEx node, FlowState state, CallContext cc, SummaryCtx sc, DataFlowType t, AccessPath ap
) {
@@ -2918,7 +2924,6 @@ module Impl<FullStateConfigSig Config> {
override AccessPathFrontHead getFront() { result = TFrontHead(head_) }
pragma[assume_small_delta]
override AccessPathApproxCons getApprox() {
result = TConsNil(head_, t) and tail_ = TAccessPathNil()
or
@@ -2927,7 +2932,6 @@ module Impl<FullStateConfigSig Config> {
result = TCons1(head_, this.length())
}
pragma[assume_small_delta]
override int length() { result = 1 + tail_.length() }
private string toStringImpl(boolean needsSuffix) {
@@ -3097,6 +3101,12 @@ module Impl<FullStateConfigSig Config> {
result = " <" + this.(PathNodeMid).getCallContext().toString() + ">"
}
private string ppSummaryCtx() {
this instanceof PathNodeSink and result = ""
or
result = " <" + this.(PathNodeMid).getSummaryCtx().toString() + ">"
}
/** Gets a textual representation of this element. */
string toString() { result = this.getNodeEx().toString() + this.ppType() + this.ppAp() }
@@ -3105,7 +3115,9 @@ module Impl<FullStateConfigSig Config> {
* representation of the call context.
*/
string toStringWithContext() {
result = this.getNodeEx().toString() + this.ppType() + this.ppAp() + this.ppCtx()
result =
this.getNodeEx().toString() + this.ppType() + this.ppAp() + this.ppCtx() +
this.ppSummaryCtx()
}
/**
@@ -3379,7 +3391,6 @@ module Impl<FullStateConfigSig Config> {
* Holds if data may flow from `mid` to `node`. The last step in or out of
* a callable is recorded by `cc`.
*/
pragma[assume_small_delta]
pragma[nomagic]
private predicate pathStep0(
PathNodeMid mid, NodeEx node, FlowState state, CallContext cc, SummaryCtx sc, DataFlowType t,
@@ -3592,7 +3603,6 @@ module Impl<FullStateConfigSig Config> {
)
}
pragma[assume_small_delta]
pragma[nomagic]
private predicate pathThroughCallable0(
DataFlowCall call, PathNodeMid mid, ReturnKindExt kind, FlowState state, CallContext cc,

2
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll
View File

@@ -187,7 +187,6 @@ private module LambdaFlow {
else any()
}
pragma[assume_small_delta]
pragma[nomagic]
predicate revLambdaFlow0(
DataFlowCall lambdaCall, LambdaCallKind kind, Node node, DataFlowType t, boolean toReturn,
@@ -274,7 +273,6 @@ private module LambdaFlow {
)
}
pragma[assume_small_delta]
pragma[nomagic]
predicate revLambdaFlowOut(
DataFlowCall lambdaCall, LambdaCallKind kind, TReturnPositionSimple pos, DataFlowType t,

12
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll
View File

@@ -304,8 +304,12 @@ module EssaFlow {
// see `with_flow` in `python/ql/src/semmle/python/dataflow/Implementation.qll`
with.getContextExpr() = contextManager.getNode() and
with.getOptionalVars() = var.getNode() and
not with.isAsync() and
contextManager.strictlyDominates(var)
// note: we allow this for both `with` and `async with`, since some
// implementations do `async def __aenter__(self): return self`, so you can do
// both:
// * `foo = x.foo(); await foo.async_method(); foo.close()` and
// * `async with x.foo() as foo: await foo.async_method()`.
)
or
// Async with var definition
@@ -314,6 +318,12 @@ module EssaFlow {
// nodeTo is `x`, essa var
//
// This makes the cfg node the local source of the awaited value.
//
// We have this step in addition to the step above, to handle cases where the QL
// modeling of `f(42)` requires a `.getAwaited()` step (in API graphs) when not
// using `async with`, so you can do both:
// * `foo = await x.foo(); await foo.async_method(); foo.close()` and
// * `async with x.foo() as foo: await foo.async_method()`.
exists(With with, ControlFlowNode var |
nodeFrom.(CfgNode).getNode() = var and
nodeTo.(EssaNode).getVar().getDefinition().(WithDefinition).getDefiningNode() = var and

81
python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImpl.qll
View File

@@ -23,29 +23,30 @@ module Public {
* content type, or a return kind.
*/
class SummaryComponent extends TSummaryComponent {
/** Gets a textual representation of this summary component. */
string toString() {
exists(ContentSet c | this = TContentSummaryComponent(c) and result = c.toString())
or
exists(ContentSet c | this = TWithoutContentSummaryComponent(c) and result = "without " + c)
or
exists(ContentSet c | this = TWithContentSummaryComponent(c) and result = "with " + c)
/** Gets a textual representation of this component used for MaD models. */
string getMadRepresentation() {
result = getMadRepresentationSpecific(this)
or
exists(ArgumentPosition pos |
this = TParameterSummaryComponent(pos) and result = "parameter " + pos
this = TParameterSummaryComponent(pos) and
result = "Parameter[" + getArgumentPosition(pos) + "]"
)
or
exists(ParameterPosition pos |
this = TArgumentSummaryComponent(pos) and result = "argument " + pos
this = TArgumentSummaryComponent(pos) and
result = "Argument[" + getParameterPosition(pos) + "]"
)
or
exists(ReturnKind rk | this = TReturnSummaryComponent(rk) and result = "return (" + rk + ")")
or
exists(SummaryComponent::SyntheticGlobal sg |
this = TSyntheticGlobalSummaryComponent(sg) and
result = "synthetic global (" + sg + ")"
exists(string synthetic |
this = TSyntheticGlobalSummaryComponent(synthetic) and
result = "SyntheticGlobal[" + synthetic + "]"
)
or
this = TReturnSummaryComponent(getReturnValueKind()) and result = "ReturnValue"
}
/** Gets a textual representation of this summary component. */
string toString() { result = this.getMadRepresentation() }
}
/** Provides predicates for constructing summary components. */
@@ -110,7 +111,6 @@ module Public {
}
/** Gets the stack obtained by dropping the first `i` elements, if any. */
pragma[assume_small_delta]
SummaryComponentStack drop(int i) {
i = 0 and result = this
or
@@ -125,19 +125,22 @@ module Public {
this = TSingletonSummaryComponentStack(result) or result = this.tail().bottom()
}
/** Gets a textual representation of this stack. */
string toString() {
/** Gets a textual representation of this stack used for MaD models. */
string getMadRepresentation() {
exists(SummaryComponent head, SummaryComponentStack tail |
head = this.head() and
tail = this.tail() and
result = tail + "." + head
result = tail.getMadRepresentation() + "." + head.getMadRepresentation()
)
or
exists(SummaryComponent c |
this = TSingletonSummaryComponentStack(c) and
result = c.toString()
result = c.getMadRepresentation()
)
}
/** Gets a textual representation of this stack. */
string toString() { result = this.getMadRepresentation() }
}
/** Provides predicates for constructing stacks of summary components. */
@@ -166,42 +169,6 @@ module Public {
SummaryComponentStack return(ReturnKind rk) { result = singleton(SummaryComponent::return(rk)) }
}
/** Gets a textual representation of this component used for flow summaries. */
private string getComponent(SummaryComponent sc) {
result = getComponentSpecific(sc)
or
exists(ArgumentPosition pos |
sc = TParameterSummaryComponent(pos) and
result = "Parameter[" + getArgumentPosition(pos) + "]"
)
or
exists(ParameterPosition pos |
sc = TArgumentSummaryComponent(pos) and
result = "Argument[" + getParameterPosition(pos) + "]"
)
or
exists(string synthetic |
sc = TSyntheticGlobalSummaryComponent(synthetic) and
result = "SyntheticGlobal[" + synthetic + "]"
)
or
sc = TReturnSummaryComponent(getReturnValueKind()) and result = "ReturnValue"
}
/** Gets a textual representation of this stack used for flow summaries. */
string getComponentStack(SummaryComponentStack stack) {
exists(SummaryComponent head, SummaryComponentStack tail |
head = stack.head() and
tail = stack.tail() and
result = getComponentStack(tail) + "." + getComponent(head)
)
or
exists(SummaryComponent c |
stack = TSingletonSummaryComponentStack(c) and
result = getComponent(c)
)
}
/**
* A class that exists for QL technical reasons only (the IPA type used
* to represent component stacks needs to be bounded).
@@ -1382,8 +1349,8 @@ module Private {
c.relevantSummary(input, output, preservesValue) and
csv =
c.getCallableCsv() // Callable information
+ getComponentStack(input) + ";" // input
+ getComponentStack(output) + ";" // output
+ input.getMadRepresentation() + ";" // input
+ output.getMadRepresentation() + ";" // output
+ renderKind(preservesValue) + ";" // kind
+ renderProvenance(c) // provenance
)

28
python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImplSpecific.qll
View File

@@ -128,10 +128,30 @@ SummaryComponent interpretComponentSpecific(AccessPathToken c) {
)
}
/** Gets the textual representation of a summary component in the format used for flow summaries. */
string getComponentSpecific(SummaryComponent sc) {
sc = TContentSummaryComponent(any(ListElementContent c)) and
result = "ListElement"
private string getContentSpecific(Content cs) {
cs = TListElementContent() and result = "ListElement"
or
cs = TSetElementContent() and result = "SetElement"
or
exists(int index |
cs = TTupleElementContent(index) and result = "TupleElement[" + index.toString() + "]"
)
or
exists(string key |
cs = TDictionaryElementContent(key) and result = "DictionaryElement[" + key + "]"
)
or
cs = TDictionaryElementAnyContent() and result = "DictionaryElementAny"
or
exists(string attr | cs = TAttributeContent(attr) and result = "Attribute[" + attr + "]")
}
/** Gets the textual representation of a summary component in the format used for MaD models. */
string getMadRepresentationSpecific(SummaryComponent sc) {
exists(Content c |
sc = TContentSummaryComponent(c) and
result = getContentSpecific(c)
)
}
/** Gets the textual representation of a parameter position in the format used for flow summaries. */

45
python/ql/lib/semmle/python/dataflow/new/internal/TypeTracker.qll
View File

@@ -55,10 +55,9 @@ private module Cached {
)
}
pragma[nomagic]
private TypeTracker noContentTypeTracker(boolean hasCall) {
result = MkTypeTracker(hasCall, noContent())
}
/** Gets a type tracker with no content and the call bit set to the given value. */
cached
TypeTracker noContentTypeTracker(boolean hasCall) { result = MkTypeTracker(hasCall, noContent()) }
/** Gets the summary resulting from appending `step` to type-tracking summary `tt`. */
cached
@@ -318,6 +317,8 @@ class StepSummary extends TStepSummary {
/** Provides predicates for updating step summaries (`StepSummary`s). */
module StepSummary {
predicate append = Cached::append/2;
/**
* Gets the summary that corresponds to having taken a forwards
* inter-procedural step from `nodeFrom` to `nodeTo`.
@@ -378,6 +379,35 @@ module StepSummary {
}
deprecated predicate localSourceStoreStep = flowsToStoreStep/3;
/** Gets the step summary for a level step. */
StepSummary levelStep() { result = LevelStep() }
/** Gets the step summary for a call step. */
StepSummary callStep() { result = CallStep() }
/** Gets the step summary for a return step. */
StepSummary returnStep() { result = ReturnStep() }
/** Gets the step summary for storing into `content`. */
StepSummary storeStep(TypeTrackerContent content) { result = StoreStep(content) }
/** Gets the step summary for loading from `content`. */
StepSummary loadStep(TypeTrackerContent content) { result = LoadStep(content) }
/** Gets the step summary for loading from `load` and then storing into `store`. */
StepSummary loadStoreStep(TypeTrackerContent load, TypeTrackerContent store) {
result = LoadStoreStep(load, store)
}
/** Gets the step summary for a step that only permits contents matched by `filter`. */
StepSummary withContent(ContentFilter filter) { result = WithContent(filter) }
/** Gets the step summary for a step that blocks contents matched by `filter`. */
StepSummary withoutContent(ContentFilter filter) { result = WithoutContent(filter) }
/** Gets the step summary for a jump step. */
StepSummary jumpStep() { result = JumpStep() }
}
/**
@@ -540,6 +570,13 @@ module TypeTracker {
* Gets a valid end point of type tracking.
*/
TypeTracker end() { result.end() }
/**
* INTERNAL USE ONLY.
*
* Gets a valid end point of type tracking with the call bit set to the given value.
*/
predicate end = Cached::noContentTypeTracker/1;
}
pragma[nomagic]

8
python/ql/lib/semmle/python/frameworks/MarkupSafe.qll
View File

@@ -83,7 +83,7 @@ private module MarkupSafeModel {
}
/** Taint propagation for `markupsafe.Markup`. */
private class AddtionalTaintStep extends TaintTracking::AdditionalTaintStep {
private class AdditionalTaintStep extends TaintTracking::AdditionalTaintStep {
override predicate step(DataFlow::Node nodeFrom, DataFlow::Node nodeTo) {
nodeTo.(ClassInstantiation).getArg(0) = nodeFrom
}
@@ -92,11 +92,7 @@ private module MarkupSafeModel {
/** Any escaping performed via the `markupsafe` package. */
abstract private class MarkupSafeEscape extends Escaping::Range {
override string getKind() {
// TODO: this package claims to escape for both HTML and XML, but for now we don't
// model XML.
result = Escaping::getHtmlKind()
}
override string getKind() { result in [Escaping::getHtmlKind(), Escaping::getXmlKind()] }
}
/** A call to any of the escaping functions in `markupsafe` */

7
python/ql/lib/semmle/python/security/dataflow/XxeCustomizations.qll
View File

@@ -44,4 +44,11 @@ module Xxe {
)
}
}
/**
* An XML escaping, considered as a sanitizer.
*/
class XmlEscapingAsSanitizer extends Sanitizer {
XmlEscapingAsSanitizer() { this = any(XmlEscaping esc).getOutput() }
}
}

10
python/ql/src/CHANGELOG.md
View File

@@ -1,3 +1,13 @@
## 0.8.0
### Bug Fixes
* The query "Arbitrary file write during archive extraction ("Zip Slip")" (`py/zipslip`) has been renamed to "Arbitrary file access during archive extraction ("Zip Slip")."
## 0.7.4
No user-facing changes.
## 0.7.3
### Bug Fixes

4
python/ql/src/change-notes/2023-07-06-aiohttp-clientsession-modeling.md Normal file
View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* Fixed modeling of `aiohttp.ClientSession` so we properly handle `async with` uses. This can impact results of server-side request forgery queries (`py/full-ssrf`, `py/partial-ssrf`).

3
python/ql/src/change-notes/released/0.7.4.md Normal file
View File

@@ -0,0 +1,3 @@
## 0.7.4
No user-facing changes.

7
python/ql/src/change-notes/2023-06-16-zipslip-rename.md → python/ql/src/change-notes/released/0.8.0.md
View File

@@ -1,4 +1,5 @@
---
category: fix
---
## 0.8.0
### Bug Fixes
* The query "Arbitrary file write during archive extraction ("Zip Slip")" (`py/zipslip`) has been renamed to "Arbitrary file access during archive extraction ("Zip Slip")."

2
python/ql/src/codeql-pack.release.yml
View File

@@ -1,2 +1,2 @@
---
lastReleaseVersion: 0.7.3
lastReleaseVersion: 0.8.0

2
python/ql/src/qlpack.yml
View File

@@ -1,5 +1,5 @@
name: codeql/python-queries
version: 0.7.4-dev
version: 0.8.1-dev
groups:
- python
- queries

10
python/ql/test/experimental/dataflow/TestUtil/FlowTest.qll
View File

@@ -9,7 +9,7 @@ signature module FlowTestSig {
predicate relevantFlow(DataFlow::Node fromNode, DataFlow::Node toNode);
}
private module FlowTest<FlowTestSig Impl> implements TestSig {
module MakeTestSig<FlowTestSig Impl> implements TestSig {
string getARelevantTag() { result = Impl::flowTag() }
predicate hasActualResult(Location location, string element, string tag, string value) {
@@ -37,11 +37,3 @@ private module FlowTest<FlowTestSig Impl> implements TestSig {
)
}
}
module MakeFlowTest<FlowTestSig Impl> {
import MakeTest<FlowTest<Impl>>
}
module MakeFlowTest2<FlowTestSig Impl1, FlowTestSig Impl2> {
import MakeTest<MergeTests<FlowTest<Impl1>, FlowTest<Impl2>>>
}

2
python/ql/test/experimental/dataflow/TestUtil/LocalFlowStepTest.qll
View File

@@ -10,4 +10,4 @@ module LocalFlowStepTest implements FlowTestSig {
}
}
import MakeFlowTest<LocalFlowStepTest>
import MakeTest<MakeTestSig<LocalFlowStepTest>>

2
python/ql/test/experimental/dataflow/TestUtil/MaximalFlowTest.qll
View File

@@ -12,7 +12,7 @@ module MaximalFlowTest implements FlowTestSig {
}
}
import MakeFlowTest<MaximalFlowTest>
import MakeTest<MakeTestSig<MaximalFlowTest>>
/**
* A configuration to find all "maximal" flows.

2
python/ql/test/experimental/dataflow/TestUtil/NormalDataflowTest.qll
View File

@@ -11,7 +11,7 @@ module DataFlowTest implements FlowTestSig {
}
}
import MakeFlowTest<DataFlowTest>
import MakeTest<MakeTestSig<DataFlowTest>>
query predicate missingAnnotationOnSink(Location location, string error, string element) {
error = "ERROR, you should add `# $ MISSING: flow` annotation" and

2
python/ql/test/experimental/dataflow/TestUtil/NormalTaintTrackingTest.qll
View File

@@ -11,7 +11,7 @@ module DataFlowTest implements FlowTestSig {
}
}
import MakeFlowTest<DataFlowTest>
import MakeTest<MakeTestSig<DataFlowTest>>
query predicate missingAnnotationOnSink(Location location, string error, string element) {
error = "ERROR, you should add `# $ MISSING: flow` annotation" and

21
python/ql/test/experimental/dataflow/TestUtil/RoutingTest.qll
View File

@@ -10,22 +10,25 @@ private import semmle.python.dataflow.new.internal.DataFlowPrivate as DataFlowPr
* the functions tested sink their arguments sequentially, that is
* `SINK1(arg1)`, etc.
*/
abstract class RoutingTest extends InlineExpectationsTest {
bindingset[this]
RoutingTest() { any() }
signature module RoutingTestSig {
class Argument;
abstract string flowTag();
string flowTag(Argument arg);
abstract predicate relevantFlow(DataFlow::Node fromNode, DataFlow::Node toNode);
predicate relevantFlow(DataFlow::Node fromNode, DataFlow::Node toNode, Argument arg);
}
override string getARelevantTag() { result in ["func", this.flowTag()] }
module MakeTestSig<RoutingTestSig Impl> implements TestSig {
string getARelevantTag() { result in ["func", Impl::flowTag(_)] }
override predicate hasActualResult(Location location, string element, string tag, string value) {
exists(DataFlow::Node fromNode, DataFlow::Node toNode | this.relevantFlow(fromNode, toNode) |
predicate hasActualResult(Location location, string element, string tag, string value) {
exists(DataFlow::Node fromNode, DataFlow::Node toNode, Impl::Argument arg |
Impl::relevantFlow(fromNode, toNode, arg)
|
location = fromNode.getLocation() and
element = fromNode.toString() and
(
tag = this.flowTag() and
tag = Impl::flowTag(arg) and
if "\"" + tag + "\"" = fromValue(fromNode) then value = "" else value = fromValue(fromNode)
or
// only have result for `func` tag if the function where `arg<n>` is used, is

2
python/ql/test/experimental/dataflow/coverage-py2/argumentRoutingTest.expected
View File

@@ -0,0 +1,2 @@
failures
testFailures

2
python/ql/test/experimental/dataflow/coverage-py3/argumentRoutingTest.expected
View File

@@ -0,0 +1,2 @@
failures
testFailures

2
python/ql/test/experimental/dataflow/coverage/argumentRoutingTest.expected
View File

@@ -0,0 +1,2 @@
failures
testFailures

78
python/ql/test/experimental/dataflow/coverage/argumentRoutingTest.ql
View File

@@ -3,19 +3,22 @@ import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.internal.DataFlowPrivate as DataFlowPrivate
import experimental.dataflow.TestUtil.RoutingTest
class Argument1RoutingTest extends RoutingTest {
Argument1RoutingTest() { this = "Argument1RoutingTest" }
module Argument1RoutingTest implements RoutingTestSig {
class Argument = Unit;
override string flowTag() { result = "arg1" }
string flowTag(Argument arg) { result = "arg1" and exists(arg) }
override predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink) {
exists(Argument1ExtraRoutingConfig cfg | cfg.hasFlow(source, sink))
or
exists(ArgumentRoutingConfig cfg |
cfg.hasFlow(source, sink) and
cfg.isArgSource(source, 1) and
cfg.isGoodSink(sink, 1)
)
predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink, Argument arg) {
(
exists(Argument1ExtraRoutingConfig cfg | cfg.hasFlow(source, sink))
or
exists(ArgumentRoutingConfig cfg |
cfg.hasFlow(source, sink) and
cfg.isArgSource(source, 1) and
cfg.isGoodSink(sink, 1)
)
) and
exists(arg)
}
}
@@ -87,59 +90,54 @@ class Argument1ExtraRoutingConfig extends DataFlow::Configuration {
override predicate isBarrierIn(DataFlow::Node node) { this.isSource(node) }
}
class RestArgumentRoutingTest extends RoutingTest {
ArgNumber argNumber;
module RestArgumentRoutingTest implements RoutingTestSig {
class Argument = ArgNumber;
RestArgumentRoutingTest() {
argNumber > 1 and
this = "Argument" + argNumber + "RoutingTest"
}
string flowTag(Argument arg) { result = "arg" + arg }
override string flowTag() { result = "arg" + argNumber }
override predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink) {
predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink, Argument arg) {
exists(ArgumentRoutingConfig cfg |
cfg.hasFlow(source, sink) and
cfg.isArgSource(source, argNumber) and
cfg.isGoodSink(sink, argNumber)
)
cfg.isArgSource(source, arg) and
cfg.isGoodSink(sink, arg)
) and
arg > 1
}
}
/** Bad flow from `arg<n>` to `SINK<N>_F` */
class BadArgumentRoutingTestSinkF extends RoutingTest {
ArgNumber argNumber;
module BadArgumentRoutingTestSinkF implements RoutingTestSig {
class Argument = ArgNumber;
BadArgumentRoutingTestSinkF() { this = "BadArgumentRoutingTestSinkF" + argNumber }
string flowTag(Argument arg) { result = "bad" + arg }
override string flowTag() { result = "bad" + argNumber }
override predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink) {
predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink, Argument arg) {
exists(ArgumentRoutingConfig cfg |
cfg.hasFlow(source, sink) and
cfg.isArgSource(source, argNumber) and
cfg.isBadSink(sink, argNumber)
cfg.isArgSource(source, arg) and
cfg.isBadSink(sink, arg)
)
}
}
/** Bad flow from `arg<n>` to `SINK<M>` or `SINK<M>_F`, where `n != m`. */
class BadArgumentRoutingTestWrongSink extends RoutingTest {
ArgNumber argNumber;
module BadArgumentRoutingTestWrongSink implements RoutingTestSig {
class Argument = ArgNumber;
BadArgumentRoutingTestWrongSink() { this = "BadArgumentRoutingTestWrongSink" + argNumber }
string flowTag(Argument arg) { result = "bad" + arg }
override string flowTag() { result = "bad" + argNumber }
override predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink) {
predicate relevantFlow(DataFlow::Node source, DataFlow::Node sink, Argument arg) {
exists(ArgumentRoutingConfig cfg |
cfg.hasFlow(source, sink) and
cfg.isArgSource(source, any(ArgNumber i | not i = argNumber)) and
cfg.isArgSource(source, any(ArgNumber i | not i = arg)) and
(
cfg.isGoodSink(sink, argNumber)
cfg.isGoodSink(sink, arg)
or
cfg.isBadSink(sink, argNumber)
cfg.isBadSink(sink, arg)
)
)
}
}
import MakeTest<MergeTests4<MakeTestSig<Argument1RoutingTest>, MakeTestSig<RestArgumentRoutingTest>,
MakeTestSig<BadArgumentRoutingTestSinkF>, MakeTestSig<BadArgumentRoutingTestWrongSink>>>

2
python/ql/test/experimental/dataflow/module-initialization/localFlow.ql
View File

@@ -31,4 +31,4 @@ module RuntimeLocalFlowTest implements FlowTestSig {
}
}
import MakeFlowTest2<ImportTimeLocalFlowTest, RuntimeLocalFlowTest>
import MakeTest<MergeTests<MakeTestSig<ImportTimeLocalFlowTest>, MakeTestSig<RuntimeLocalFlowTest>>>

2
python/ql/test/experimental/dataflow/typetracking-summaries/tracked.expected
View File

@@ -0,0 +1,2 @@
failures
testFailures

10
python/ql/test/experimental/dataflow/typetracking-summaries/tracked.ql
View File

@@ -15,12 +15,10 @@ private DataFlow::TypeTrackingNode tracked(TypeTracker t) {
exists(TypeTracker t2 | result = tracked(t2).track(t2, t))
}
class TrackedTest extends InlineExpectationsTest {
TrackedTest() { this = "TrackedTest" }
module TrackedTest implements TestSig {
string getARelevantTag() { result = "tracked" }
override string getARelevantTag() { result = "tracked" }
override predicate hasActualResult(Location location, string element, string tag, string value) {
predicate hasActualResult(Location location, string element, string tag, string value) {
exists(DataFlow::Node e, TypeTracker t |
exists(e.getLocation().getFile().getRelativePath()) and
e.getLocation().getStartLine() > 0 and
@@ -34,3 +32,5 @@ class TrackedTest extends InlineExpectationsTest {
)
}
}
import MakeTest<TrackedTest>

4
python/ql/test/experimental/library-tests/CallGraph/InlineCallGraphTest.ql
View File

@@ -108,7 +108,7 @@ query predicate pointsTo_found_typeTracker_notFound(CallNode call, string qualna
not typeTrackerCallEdge(call, target) and
qualname = getCallEdgeValue(call, target) and
// ignore SPURIOUS call edges
not exists(FalsePositiveExpectation spuriousResult |
not exists(FalsePositiveTestExpectation spuriousResult |
spuriousResult.getTag() = "pt" and
spuriousResult.getValue() = getCallEdgeValue(call, target) and
spuriousResult.getLocation().getFile() = call.getLocation().getFile() and
@@ -127,7 +127,7 @@ query predicate typeTracker_found_pointsTo_notFound(CallNode call, string qualna
// between the two).
not typeTrackerClassCall(call, target) and
// ignore SPURIOUS call edges
not exists(FalsePositiveExpectation spuriousResult |
not exists(FalsePositiveTestExpectation spuriousResult |
spuriousResult.getTag() = "tt" and
spuriousResult.getValue() = getCallEdgeValue(call, target) and
spuriousResult.getLocation().getFile() = call.getLocation().getFile() and

88
python/ql/test/experimental/meta/ConceptsTest.qll
View File

@@ -255,52 +255,64 @@ module HttpServerRequestHandlerTest implements TestSig {
}
}
class HttpServerHttpResponseTest extends InlineExpectationsTest {
File file;
abstract class DedicatedResponseTest extends string {
bindingset[this]
DedicatedResponseTest() { any() }
HttpServerHttpResponseTest() {
file.getExtension() = "py" and
this = "HttpServerHttpResponseTest: " + file
}
string toString() { result = this }
override string getARelevantTag() { result in ["HttpResponse", "responseBody", "mimetype"] }
abstract predicate isDedicatedFile(File file);
}
override predicate hasActualResult(Location location, string element, string tag, string value) {
module HttpServerHttpResponseTest implements TestSig {
string getARelevantTag() { result in ["HttpResponse", "responseBody", "mimetype"] }
predicate hasActualResult(Location location, string element, string tag, string value) {
// By adding `file` as a class field, and these two restrictions, it's possible to
// say that we only want to check _some_ tags for certain files. This helped make
// flask tests more readable since adding full annotations for HttpResponses in the
// the tests for routing setup is both annoying and not very useful.
location.getFile() = file and
exists(file.getRelativePath()) and
// we need to do this step since we expect subclasses could override getARelevantTag
tag = this.getARelevantTag() and
(
exists(Http::Server::HttpResponse response |
location = response.getLocation() and
element = response.toString() and
value = "" and
tag = "HttpResponse"
)
or
exists(Http::Server::HttpResponse response |
location = response.getLocation() and
element = response.toString() and
value = prettyNodeForInlineTest(response.getBody()) and
tag = "responseBody"
)
or
exists(Http::Server::HttpResponse response |
location = response.getLocation() and
element = response.toString() and
// Ensure that an expectation value such as "mimetype=text/html; charset=utf-8" is parsed as a
// single expectation with tag mimetype, and not as two expectations with tags mimetype and
// charset.
exists(File file |
location.getFile() = file and
file.getExtension() = "py" and
exists(file.getRelativePath()) and
// we need to do this step since we expect subclasses could override getARelevantTag
tag = getARelevantTag() and
(
exists(Http::Server::HttpResponse response |
location = response.getLocation() and
element = response.toString() and
value = "" and
tag = "HttpResponse"
)
or
(
if exists(response.getMimetype().indexOf(" "))
then value = "\"" + response.getMimetype() + "\""
else value = response.getMimetype()
not exists(DedicatedResponseTest d)
or
exists(DedicatedResponseTest d | d.isDedicatedFile(file))
) and
tag = "mimetype"
(
exists(Http::Server::HttpResponse response |
location = response.getLocation() and
element = response.toString() and
value = prettyNodeForInlineTest(response.getBody()) and
tag = "responseBody"
)
or
exists(Http::Server::HttpResponse response |
location = response.getLocation() and
element = response.toString() and
// Ensure that an expectation value such as "mimetype=text/html; charset=utf-8" is parsed as a
// single expectation with tag mimetype, and not as two expectations with tags mimetype and
// charset.
(
if exists(response.getMimetype().indexOf(" "))
then value = "\"" + response.getMimetype() + "\""
else value = response.getMimetype()
) and
tag = "mimetype"
)
)
)
)
}
@@ -545,7 +557,7 @@ import MakeTest<MergeTests5<MergeTests5<SystemCommandExecutionTest, DecodingTest
CodeExecutionTest>,
MergeTests5<SqlConstructionTest, SqlExecutionTest, XPathConstructionTest, XPathExecutionTest,
EscapingTest>,
MergeTests4<HttpServerRouteSetupTest, HttpServerRequestHandlerTest,
MergeTests5<HttpServerRouteSetupTest, HttpServerRequestHandlerTest, HttpServerHttpResponseTest,
HttpServerHttpRedirectResponseTest, HttpServerCookieWriteTest>,
MergeTests5<FileSystemAccessTest, FileSystemWriteAccessTest, PathNormalizationTest,
SafeAccessCheckTest, PublicKeyGenerationTest>,

10
python/ql/test/library-tests/frameworks/aiohttp/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

52
python/ql/test/library-tests/frameworks/aiohttp/client_request.py
View File

@@ -1,35 +1,35 @@
import aiohttp
import asyncio
import ssl
s = aiohttp.ClientSession()
resp = s.request("method", "url") # $ clientRequestUrlPart="url"
resp = s.request("method", url="url") # $ clientRequestUrlPart="url"
async def test():
s = aiohttp.ClientSession()
resp = await s.request("method", "url") # $ clientRequestUrlPart="url"
resp = await s.request("method", url="url") # $ clientRequestUrlPart="url"
with aiohttp.ClientSession() as session:
resp = session.get("url") # $ clientRequestUrlPart="url"
resp = session.request(method="GET", url="url") # $ clientRequestUrlPart="url"
async with aiohttp.ClientSession() as session:
resp = await session.get("url") # $ clientRequestUrlPart="url"
resp = await session.request(method="GET", url="url") # $ clientRequestUrlPart="url"
# other methods than GET
s = aiohttp.ClientSession()
resp = s.post("url") # $ clientRequestUrlPart="url"
resp = s.patch("url") # $ clientRequestUrlPart="url"
resp = s.options("url") # $ clientRequestUrlPart="url"
# other methods than GET
s = aiohttp.ClientSession()
resp = await s.post("url") # $ clientRequestUrlPart="url"
resp = await s.patch("url") # $ clientRequestUrlPart="url"
resp = await s.options("url") # $ clientRequestUrlPart="url"
# disabling of SSL validation
# see https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request
s.get("url", ssl=False) # $ clientRequestUrlPart="url" clientRequestCertValidationDisabled
s.get("url", ssl=0) # $ clientRequestUrlPart="url" clientRequestCertValidationDisabled
# None is treated as default and so does _not_ disable the check
s.get("url", ssl=None) # $ clientRequestUrlPart="url"
# disabling of SSL validation
# see https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.request
s.get("url", ssl=False) # $ clientRequestUrlPart="url" clientRequestCertValidationDisabled
s.get("url", ssl=0) # $ clientRequestUrlPart="url" clientRequestCertValidationDisabled
# None is treated as default and so does _not_ disable the check
s.get("url", ssl=None) # $ clientRequestUrlPart="url"
# deprecated since 3.0, but still supported
s.get("url", verify_ssl=False) # $ clientRequestUrlPart="url" clientRequestCertValidationDisabled
# deprecated since 3.0, but still supported
s.get("url", verify_ssl=False) # $ clientRequestUrlPart="url" clientRequestCertValidationDisabled
# A manually constructed SSLContext does not have safe defaults, so is effectively the
# same as turning off SSL validation
context = ssl.SSLContext()
assert context.check_hostname == False
assert context.verify_mode == ssl.VerifyMode.CERT_NONE
# A manually constructed SSLContext does not have safe defaults, so is effectively the
# same as turning off SSL validation
context = ssl.SSLContext()
assert context.check_hostname == False
assert context.verify_mode == ssl.VerifyMode.CERT_NONE
s.get("url", ssl=context) # $ clientRequestUrlPart="url" MISSING: clientRequestCertValidationDisabled
s.get("url", ssl=context) # $ clientRequestUrlPart="url" MISSING: clientRequestCertValidationDisabled

10
python/ql/test/library-tests/frameworks/django-v1/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

10
python/ql/test/library-tests/frameworks/django-v2-v3/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

10
python/ql/test/library-tests/frameworks/fastapi/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

10
python/ql/test/library-tests/frameworks/flask/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

10
python/ql/test/library-tests/frameworks/flask_admin/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

40
python/ql/test/library-tests/frameworks/markupsafe/taint_test.py
View File

@@ -27,40 +27,40 @@ def test():
# as tainted even after it has been escaped in some place. This _might_ not be the
# case since data-flow library has taint-steps from adjacent uses...
ensure_tainted(ts) # $ tainted
ensure_not_tainted(escape(ts)) # $ escapeInput=ts escapeKind=html escapeOutput=escape(..)
ensure_not_tainted(escape(ts)) # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=escape(..)
ensure_tainted(ts) # $ tainted
ensure_tainted(
ts, # $ tainted
m_unsafe, # $ tainted
m_unsafe + SAFE, # $ escapeInput=SAFE escapeKind=html escapeOutput=BinaryExpr MISSING: tainted
SAFE + m_unsafe, # $ escapeInput=SAFE escapeKind=html escapeOutput=BinaryExpr MISSING: tainted
m_unsafe.format(SAFE), # $ escapeInput=SAFE escapeKind=html escapeOutput=m_unsafe.format(..) MISSING: tainted
m_unsafe % SAFE, # $ escapeInput=SAFE escapeKind=html escapeOutput=BinaryExpr MISSING: tainted
m_unsafe + ts, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr MISSING: tainted
m_unsafe + SAFE, # $ escapeInput=SAFE escapeKind=html escapeKind=xml escapeOutput=BinaryExpr MISSING: tainted
SAFE + m_unsafe, # $ escapeInput=SAFE escapeKind=html escapeKind=xml escapeOutput=BinaryExpr MISSING: tainted
m_unsafe.format(SAFE), # $ escapeInput=SAFE escapeKind=html escapeKind=xml escapeOutput=m_unsafe.format(..) MISSING: tainted
m_unsafe % SAFE, # $ escapeInput=SAFE escapeKind=html escapeKind=xml escapeOutput=BinaryExpr MISSING: tainted
m_unsafe + ts, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr MISSING: tainted
m_safe.format(m_unsafe), # $ tainted
m_safe % m_unsafe, # $ tainted
escape(ts).unescape(), # $ escapeInput=ts escapeKind=html escapeOutput=escape(..) MISSING: tainted
escape_silent(ts).unescape(), # $ escapeInput=ts escapeKind=html escapeOutput=escape_silent(..) MISSING: tainted
escape(ts).unescape(), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=escape(..) MISSING: tainted
escape_silent(ts).unescape(), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=escape_silent(..) MISSING: tainted
)
ensure_not_tainted(
escape(ts), # $ escapeInput=ts escapeKind=html escapeOutput=escape(..)
escape_silent(ts), # $ escapeInput=ts escapeKind=html escapeOutput=escape_silent(..)
escape(ts), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=escape(..)
escape_silent(ts), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=escape_silent(..)
Markup.escape(ts), # $ escapeInput=ts escapeKind=html escapeOutput=Markup.escape(..)
Markup.escape(ts), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=Markup.escape(..)
m_safe,
m_safe + ts, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr
ts + m_safe, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr
m_safe.format(ts), # $ escapeInput=ts escapeKind=html escapeOutput=m_safe.format(..)
m_safe % ts, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr
m_safe + ts, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr
ts + m_safe, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr
m_safe.format(ts), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=m_safe.format(..)
m_safe % ts, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr
escape(ts) + ts, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr escapeOutput=escape(..)
escape_silent(ts) + ts, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr escapeOutput=escape_silent(..)
Markup.escape(ts) + ts, # $ escapeInput=ts escapeKind=html escapeOutput=BinaryExpr escapeOutput=Markup.escape(..)
escape(ts) + ts, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr escapeOutput=escape(..)
escape_silent(ts) + ts, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr escapeOutput=escape_silent(..)
Markup.escape(ts) + ts, # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=BinaryExpr escapeOutput=Markup.escape(..)
)
# flask re-exports these, as:
@@ -73,8 +73,8 @@ def test():
)
ensure_not_tainted(
flask.escape(ts), # $ escapeInput=ts escapeKind=html escapeOutput=flask.escape(..)
flask.Markup.escape(ts), # $ escapeInput=ts escapeKind=html escapeOutput=flask.Markup.escape(..)
flask.escape(ts), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=flask.escape(..)
flask.Markup.escape(ts), # $ escapeInput=ts escapeKind=html escapeKind=xml escapeOutput=flask.Markup.escape(..)
)

10
python/ql/test/library-tests/frameworks/tornado/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

10
python/ql/test/library-tests/frameworks/twisted/ConceptsTest.ql
View File

@@ -1,12 +1,8 @@
import python
import experimental.meta.ConceptsTest
class DedicatedResponseTest extends HttpServerHttpResponseTest {
DedicatedResponseTest() { file.getShortName() = "response_test.py" }
}
class DedicatedTest extends DedicatedResponseTest {
DedicatedTest() { this = "response_test.py" }
class OtherResponseTest extends HttpServerHttpResponseTest {
OtherResponseTest() { not this instanceof DedicatedResponseTest }
override string getARelevantTag() { result = "HttpResponse" }
override predicate isDedicatedFile(File file) { file.getShortName() = this }
}

36
python/ql/test/query-tests/Security/CWE-611-Xxe/Xxe.expected
View File

@@ -1,25 +1,25 @@
edges
| test.py:1:26:1:32 | ControlFlowNode for ImportMember | test.py:1:26:1:32 | GSSA Variable request |
| test.py:1:26:1:32 | GSSA Variable request | test.py:8:19:8:25 | ControlFlowNode for request |
| test.py:1:26:1:32 | GSSA Variable request | test.py:19:19:19:25 | ControlFlowNode for request |
| test.py:8:19:8:25 | ControlFlowNode for request | test.py:8:19:8:30 | ControlFlowNode for Attribute |
| test.py:8:19:8:30 | ControlFlowNode for Attribute | test.py:8:19:8:45 | ControlFlowNode for Subscript |
| test.py:8:19:8:45 | ControlFlowNode for Subscript | test.py:9:34:9:44 | ControlFlowNode for xml_content |
| test.py:19:19:19:25 | ControlFlowNode for request | test.py:19:19:19:30 | ControlFlowNode for Attribute |
| test.py:19:19:19:30 | ControlFlowNode for Attribute | test.py:19:19:19:45 | ControlFlowNode for Subscript |
| test.py:19:19:19:45 | ControlFlowNode for Subscript | test.py:30:34:30:44 | ControlFlowNode for xml_content |
| test.py:1:26:1:32 | GSSA Variable request | test.py:9:19:9:25 | ControlFlowNode for request |
| test.py:1:26:1:32 | GSSA Variable request | test.py:20:19:20:25 | ControlFlowNode for request |
| test.py:9:19:9:25 | ControlFlowNode for request | test.py:9:19:9:30 | ControlFlowNode for Attribute |
| test.py:9:19:9:30 | ControlFlowNode for Attribute | test.py:9:19:9:45 | ControlFlowNode for Subscript |
| test.py:9:19:9:45 | ControlFlowNode for Subscript | test.py:10:34:10:44 | ControlFlowNode for xml_content |
| test.py:20:19:20:25 | ControlFlowNode for request | test.py:20:19:20:30 | ControlFlowNode for Attribute |
| test.py:20:19:20:30 | ControlFlowNode for Attribute | test.py:20:19:20:45 | ControlFlowNode for Subscript |
| test.py:20:19:20:45 | ControlFlowNode for Subscript | test.py:31:34:31:44 | ControlFlowNode for xml_content |
nodes
| test.py:1:26:1:32 | ControlFlowNode for ImportMember | semmle.label | ControlFlowNode for ImportMember |
| test.py:1:26:1:32 | GSSA Variable request | semmle.label | GSSA Variable request |
| test.py:8:19:8:25 | ControlFlowNode for request | semmle.label | ControlFlowNode for request |
| test.py:8:19:8:30 | ControlFlowNode for Attribute | semmle.label | ControlFlowNode for Attribute |
| test.py:8:19:8:45 | ControlFlowNode for Subscript | semmle.label | ControlFlowNode for Subscript |
| test.py:9:34:9:44 | ControlFlowNode for xml_content | semmle.label | ControlFlowNode for xml_content |
| test.py:19:19:19:25 | ControlFlowNode for request | semmle.label | ControlFlowNode for request |
| test.py:19:19:19:30 | ControlFlowNode for Attribute | semmle.label | ControlFlowNode for Attribute |
| test.py:19:19:19:45 | ControlFlowNode for Subscript | semmle.label | ControlFlowNode for Subscript |
| test.py:30:34:30:44 | ControlFlowNode for xml_content | semmle.label | ControlFlowNode for xml_content |
| test.py:9:19:9:25 | ControlFlowNode for request | semmle.label | ControlFlowNode for request |
| test.py:9:19:9:30 | ControlFlowNode for Attribute | semmle.label | ControlFlowNode for Attribute |
| test.py:9:19:9:45 | ControlFlowNode for Subscript | semmle.label | ControlFlowNode for Subscript |
| test.py:10:34:10:44 | ControlFlowNode for xml_content | semmle.label | ControlFlowNode for xml_content |
| test.py:20:19:20:25 | ControlFlowNode for request | semmle.label | ControlFlowNode for request |
| test.py:20:19:20:30 | ControlFlowNode for Attribute | semmle.label | ControlFlowNode for Attribute |
| test.py:20:19:20:45 | ControlFlowNode for Subscript | semmle.label | ControlFlowNode for Subscript |
| test.py:31:34:31:44 | ControlFlowNode for xml_content | semmle.label | ControlFlowNode for xml_content |
subpaths
#select
| test.py:9:34:9:44 | ControlFlowNode for xml_content | test.py:1:26:1:32 | ControlFlowNode for ImportMember | test.py:9:34:9:44 | ControlFlowNode for xml_content | XML parsing depends on a $@ without guarding against external entity expansion. | test.py:1:26:1:32 | ControlFlowNode for ImportMember | user-provided value |
| test.py:30:34:30:44 | ControlFlowNode for xml_content | test.py:1:26:1:32 | ControlFlowNode for ImportMember | test.py:30:34:30:44 | ControlFlowNode for xml_content | XML parsing depends on a $@ without guarding against external entity expansion. | test.py:1:26:1:32 | ControlFlowNode for ImportMember | user-provided value |
| test.py:10:34:10:44 | ControlFlowNode for xml_content | test.py:1:26:1:32 | ControlFlowNode for ImportMember | test.py:10:34:10:44 | ControlFlowNode for xml_content | XML parsing depends on a $@ without guarding against external entity expansion. | test.py:1:26:1:32 | ControlFlowNode for ImportMember | user-provided value |
| test.py:31:34:31:44 | ControlFlowNode for xml_content | test.py:1:26:1:32 | ControlFlowNode for ImportMember | test.py:31:34:31:44 | ControlFlowNode for xml_content | XML parsing depends on a $@ without guarding against external entity expansion. | test.py:1:26:1:32 | ControlFlowNode for ImportMember | user-provided value |

7
python/ql/test/query-tests/Security/CWE-611-Xxe/test.py
View File

@@ -1,5 +1,6 @@
from flask import Flask, request
import lxml.etree
import markupsafe
app = Flask(__name__)
@@ -28,3 +29,9 @@ def super_vuln_handler():
huge_tree=True,
)
return lxml.etree.fromstring(xml_content, parser=parser).text
@app.route("/sanitized-handler")
def sanitized_handler():
xml_content = request.args['xml_content']
xml_content = markupsafe.escape(xml_content)
return lxml.etree.fromstring(xml_content).text