hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2025-12-18 09:43:15 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into change/adjust-extracted-files-diagnostics

Browse Source
This commit is contained in:
Sid Shankar
2024-01-16 21:51:41 -05:00
committed by GitHub
parent b26fef816a 1fba345fb3
commit 59098be8c4
598 changed files with 207550 additions and 24866 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
python/downgrades/503c0516fba2e5da9570f00eb34ef43025ecb8fb/upgrade.properties
View File

@@ -3,4 +3,4 @@ compatibility: backwards
py_exprs.rel: run py_exprs.qlo
py_stmts.rel: run py_stmts.qlo
py_patterns.rel: delete
py_patterns_lists.rel: delete
py_pattern_lists.rel: delete

16
python/ql/lib/CHANGELOG.md
View File

@@ -1,3 +1,19 @@
## 0.11.6
### Major Analysis Improvements
* Added support for global data-flow through captured variables.
### Minor Analysis Improvements
* Captured subclass relationships ahead-of-time for most popular PyPI packages so we are able to resolve subclass relationships even without having the packages installed. For example we have captured that `flask_restful.Resource` is a subclass of `flask.views.MethodView`, so our Flask modeling will still consider a function named `post` on a `class Foo(flask_restful.Resource):` as a HTTP request handler.
* Python now makes use of the shared type tracking library, exposed as `semmle.python.dataflow.new.TypeTracking`. The existing type tracking library, `semmle.python.dataflow.new.TypeTracker`, has consequently been deprecated.
### Bug Fixes
- We would previously confuse all captured variables into a single scope entry node. Now they each get their own node so they can be tracked properly.
- The dataflow graph no longer contains SSA variables. Instead, flow is directed via the corresponding controlflow nodes. This should make the graph and the flow simpler to understand. Minor improvements in flow computation has been observed, but in general negligible changes to alerts are expected.
## 0.11.5
No user-facing changes.

5
python/ql/lib/change-notes/2023-11-20-remove-essa-nodes-from-dataflow-graph.md
View File

@@ -1,5 +0,0 @@
---
category: fix
---
- The dataflow graph no longer contains SSA variables. Instead, flow is directed via the corresponding controlflow nodes. This should make the graph and the flow simpler to understand. Minor improvements in flow computation has been observed, but in general negligible changes to alerts are expected.

4
python/ql/lib/change-notes/2023-11-21-new-type-tracking-lib.md
View File

@@ -1,4 +0,0 @@
---
category: minorAnalysis
---
* Python now makes use of the shared type tracking library, exposed as `semmle.python.dataflow.new.TypeTracking`. The existing type tracking library, `semmle.python.dataflow.new.TypeTracker`, has consequently been deprecated.

4
python/ql/lib/change-notes/2023-12-18-support-variable-capture.md
View File

@@ -1,4 +0,0 @@
---
category: majorAnalysis
---
* Added support for global data-flow through captured variables.

5
python/ql/lib/change-notes/2023-12-20-add-scope-entry-definition-nodes.md
View File

@@ -1,5 +0,0 @@
---
category: fix
---
- We would previously confuse all captured variables into a single scope entry node. Now they each get their own node so they can be tracked properly.

15
python/ql/lib/change-notes/released/0.11.6.md Normal file
View File

@@ -0,0 +1,15 @@
## 0.11.6
### Major Analysis Improvements
* Added support for global data-flow through captured variables.
### Minor Analysis Improvements
* Captured subclass relationships ahead-of-time for most popular PyPI packages so we are able to resolve subclass relationships even without having the packages installed. For example we have captured that `flask_restful.Resource` is a subclass of `flask.views.MethodView`, so our Flask modeling will still consider a function named `post` on a `class Foo(flask_restful.Resource):` as a HTTP request handler.
* Python now makes use of the shared type tracking library, exposed as `semmle.python.dataflow.new.TypeTracking`. The existing type tracking library, `semmle.python.dataflow.new.TypeTracker`, has consequently been deprecated.
### Bug Fixes
- We would previously confuse all captured variables into a single scope entry node. Now they each get their own node so they can be tracked properly.
- The dataflow graph no longer contains SSA variables. Instead, flow is directed via the corresponding controlflow nodes. This should make the graph and the flow simpler to understand. Minor improvements in flow computation has been observed, but in general negligible changes to alerts are expected.

2
python/ql/lib/codeql-pack.release.yml
View File

@@ -1,2 +1,2 @@
---
lastReleaseVersion: 0.11.5
lastReleaseVersion: 0.11.6

2
python/ql/lib/qlpack.yml
View File

@@ -1,5 +1,5 @@
name: codeql/python-all
version: 0.11.6-dev
version: 0.11.7-dev
groups: python
dbscheme: semmlecode.python.dbscheme
extractor: python

2
python/ql/lib/semmle/python/Module.qll
View File

@@ -177,7 +177,7 @@ private predicate legalDottedName(string name) {
}
bindingset[name]
private predicate legalShortName(string name) { name.regexpMatch("(\\p{L}|_)(\\p{L}|\\d|_)*") }
predicate legalShortName(string name) { name.regexpMatch("(\\p{L}|_)(\\p{L}|\\d|_)*") }
private string moduleNameFromBase(Container file) {
// We used to also require `isPotentialPackage(f)` to hold in this case,

16
python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll
View File

@@ -1009,22 +1009,6 @@ predicate attributeClearStep(Node n, AttributeContent c) {
*/
predicate isUnreachableInCall(Node n, DataFlowCall call) { none() }
//--------
// Virtual dispatch with call context
//--------
/**
* Gets a viable dispatch target of `call` in the context `ctx`. This is
* restricted to those `call`s for which a context might make a difference.
*/
DataFlowCallable viableImplInCallContext(DataFlowCall call, DataFlowCall ctx) { none() }
/**
* Holds if the set of viable implementations that can be called by `call`
* might be improved by knowing the call context. This is the case if the qualifier accesses a parameter of
* the enclosing callable `c` (including the implicit `this` parameter).
*/
predicate mayBenefitFromCallContext(DataFlowCall call, DataFlowCallable c) { none() }
/**
* Holds if access paths with `c` at their head always should be tracked at high
* precision. This disables adaptive access path precision for such access paths.

3
python/ql/lib/semmle/python/frameworks/Aioch.qll
View File

@@ -9,6 +9,7 @@ private import python
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.ClickhouseDriver
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -24,6 +25,8 @@ module Aioch {
/** Gets a reference to the `aioch.Client` class or any subclass. */
API::Node subclassRef() {
result = API::moduleImport("aioch").getMember("Client").getASubclass*()
or
result = ModelOutput::getATypeNode("aioch.Client~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `clickhouse_driver.Client` or any subclass. */

11
python/ql/lib/semmle/python/frameworks/Aiohttp.qll
View File

@@ -14,6 +14,7 @@ private import semmle.python.frameworks.internal.SelfRefMixin
private import semmle.python.frameworks.Multidict
private import semmle.python.frameworks.Yarl
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -31,6 +32,8 @@ module AiohttpWebModel {
/** Gets a reference to the `aiohttp.web.View` class or any subclass. */
API::Node subclassRef() {
result = API::moduleImport("aiohttp").getMember("web").getMember("View").getASubclass*()
or
result = ModelOutput::getATypeNode("aiohttp.web.View~Subclass").getASubclass*()
}
}
@@ -706,10 +709,12 @@ module AiohttpWebModel {
}
/**
* INTERNAL: Do not use.
*
* Provides models for the web server part (`aiohttp.client`) of the `aiohttp` PyPI package.
* See https://docs.aiohttp.org/en/stable/client.html
*/
private module AiohttpClientModel {
module AiohttpClientModel {
/**
* Provides models for the `aiohttp.ClientSession` class
*
@@ -717,8 +722,10 @@ private module AiohttpClientModel {
*/
module ClientSession {
/** Gets a reference to the `aiohttp.ClientSession` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("aiohttp").getMember("ClientSession")
or
result = ModelOutput::getATypeNode("aiohttp.ClientSession~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `aiohttp.ClientSession`. */

4
python/ql/lib/semmle/python/frameworks/ClickhouseDriver.qll
View File

@@ -9,6 +9,7 @@ private import python
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.PEP249
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -37,6 +38,9 @@ module ClickhouseDriver {
or
// commonly used alias
classRef = API::moduleImport("clickhouse_driver").getMember("Client")
or
// Models-as-Data subclass
classRef = ModelOutput::getATypeNode("clickhouse_driver.client.Client~Subclass")
|
result = classRef.getASubclass*()
)

117
python/ql/lib/semmle/python/frameworks/Django.qll
View File

@@ -16,6 +16,7 @@ private import semmle.python.frameworks.internal.PoorMansFunctionResolution
private import semmle.python.frameworks.internal.SelfRefMixin
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.security.dataflow.UrlRedirectCustomizations
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -85,6 +86,10 @@ module Django {
}
}
private class MaDSubclass extends ModeledSubclass {
MaDSubclass() { this = ModelOutput::getATypeNode("Django.Views.View~Subclass") }
}
/** Gets a reference to the `django.views.generic.View` class or any subclass. */
API::Node subclassRef() { result = any(ModeledSubclass subclass).getASubclass*() }
}
@@ -185,6 +190,10 @@ module Django {
}
}
private class MaDSubclass extends ModeledSubclass {
MaDSubclass() { this = ModelOutput::getATypeNode("django.forms.BaseForm~Subclass") }
}
/** Gets a reference to the `django.forms.forms.BaseForm` class or any subclass. */
API::Node subclassRef() { result = any(ModeledSubclass subclass).getASubclass*() }
}
@@ -290,6 +299,10 @@ module Django {
}
}
private class MaDSubclass extends ModeledSubclass {
MaDSubclass() { this = ModelOutput::getATypeNode("Django.Forms.Field~Subclass") }
}
/** Gets a reference to the `django.forms.fields.Field` class or any subclass. */
API::Node subclassRef() { result = any(ModeledSubclass subclass).getASubclass*() }
}
@@ -596,6 +609,8 @@ module PrivateDjango {
.getMember("models")
.getMember("PolymorphicModel")
.getASubclass*()
or
result = ModelOutput::getATypeNode("Django.db.models.Model~Subclass").getASubclass*()
}
/**
@@ -766,6 +781,9 @@ module PrivateDjango {
.getMember(className)
.getASubclass*()
)
or
result =
ModelOutput::getATypeNode("django.db.models.FileField~Subclass").getASubclass*()
}
}
@@ -823,6 +841,10 @@ module PrivateDjango {
or
// Commonly used alias
result = models().getMember("RawSQL")
or
result =
ModelOutput::getATypeNode("django.db.models.expressions.RawSQL~Subclass")
.getASubclass*()
}
/**
@@ -1157,6 +1179,9 @@ module PrivateDjango {
or
// handle django.http.HttpRequest alias
result = http().getMember("HttpRequest")
or
result =
ModelOutput::getATypeNode("django.http.request.HttpRequest~Subclass").getASubclass*()
}
/**
@@ -1322,7 +1347,13 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponse` class or any subclass. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*()
or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponse~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponse`, extend this class to model new instances.
@@ -1383,7 +1414,12 @@ module PrivateDjango {
}
/** Gets a reference to a subclass of the `django.http.response.HttpResponseRedirect` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseRedirect~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseRedirect`, extend this class to model new instances.
@@ -1446,7 +1482,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponsePermanentRedirect` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponsePermanentRedirect~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponsePermanentRedirect`, extend this class to model new instances.
@@ -1510,7 +1551,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseNotModified` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseNotModified~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseNotModified`, extend this class to model new instances.
@@ -1562,7 +1608,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseBadRequest` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseBadRequest~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseBadRequest`, extend this class to model new instances.
@@ -1616,7 +1667,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseNotFound` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseNotFound~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseNotFound`, extend this class to model new instances.
@@ -1670,7 +1726,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseForbidden` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseForbidden~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseForbidden`, extend this class to model new instances.
@@ -1724,7 +1785,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseNotAllowed` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseNotAllowed~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseNotAllowed`, extend this class to model new instances.
@@ -1779,7 +1845,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseGone` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseGone~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseGone`, extend this class to model new instances.
@@ -1833,7 +1904,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.HttpResponseServerError` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.HttpResponseServerError~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.HttpResponseServerError`, extend this class to model new instances.
@@ -1887,7 +1963,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.JsonResponse` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.JsonResponse~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.JsonResponse`, extend this class to model new instances.
@@ -1944,7 +2025,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.StreamingHttpResponse` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.StreamingHttpResponse~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.StreamingHttpResponse`, extend this class to model new instances.
@@ -1998,7 +2084,12 @@ module PrivateDjango {
}
/** Gets a reference to the `django.http.response.FileResponse` class. */
API::Node classRef() { result = baseClassRef().getASubclass*() }
API::Node classRef() {
result = baseClassRef().getASubclass*() or
result =
ModelOutput::getATypeNode("django.http.response.FileResponse~Subclass")
.getASubclass*()
}
/**
* A source of instances of `django.http.response.FileResponse`, extend this class to model new instances.

8
python/ql/lib/semmle/python/frameworks/Fabric.qll
View File

@@ -12,6 +12,7 @@ private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.RemoteFlowSources
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* Provides classes modeling security-relevant aspects of the `fabric` PyPI package, for
@@ -65,12 +66,14 @@ private module FabricV1 {
}
/**
* INTERNAL: Do not use.
*
* Provides classes modeling security-relevant aspects of the `fabric` PyPI package, for
* version 2.x.
*
* See http://docs.fabfile.org/en/2.5/getting-st arted.html.
*/
private module FabricV2 {
module FabricV2 {
/** Gets a reference to the `fabric` module. */
API::Node fabric() { result = API::moduleImport("fabric") }
@@ -95,6 +98,9 @@ private module FabricV2 {
result = fabric().getMember("Connection")
or
result = connection().getMember("Connection")
or
result =
ModelOutput::getATypeNode("fabric.connection.Connection~Subclass").getASubclass*()
}
/**

19
python/ql/lib/semmle/python/frameworks/FastApi.qll
View File

@@ -11,18 +11,23 @@ private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.Pydantic
private import semmle.python.frameworks.Starlette
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `fastapi` PyPI package.
* See https://fastapi.tiangolo.com/.
*/
private module FastApi {
module FastApi {
/**
* Provides models for FastAPI applications (an instance of `fastapi.FastAPI`).
*/
module App {
API::Node cls() { result = API::moduleImport("fastapi").getMember("FastAPI") }
/** Gets a reference to a FastAPI application (an instance of `fastapi.FastAPI`). */
API::Node instance() { result = API::moduleImport("fastapi").getMember("FastAPI").getReturn() }
API::Node instance() { result = cls().getReturn() }
}
/**
@@ -31,10 +36,14 @@ private module FastApi {
* See https://fastapi.tiangolo.com/tutorial/bigger-applications/.
*/
module ApiRouter {
/** Gets a reference to an instance of `fastapi.ApiRouter`. */
API::Node instance() {
result = API::moduleImport("fastapi").getMember("APIRouter").getASubclass*().getReturn()
API::Node cls() {
result = API::moduleImport("fastapi").getMember("APIRouter").getASubclass*()
or
result = ModelOutput::getATypeNode("fastapi.APIRouter~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `fastapi.ApiRouter`. */
API::Node instance() { result = cls().getReturn() }
}
// ---------------------------------------------------------------------------

20
python/ql/lib/semmle/python/frameworks/Flask.qll
View File

@@ -14,6 +14,7 @@ private import semmle.python.ApiGraphs
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.security.dataflow.PathInjectionCustomizations
private import semmle.python.dataflow.new.FlowSummary
private import semmle.python.frameworks.data.ModelsAsData
/**
* Provides models for the `flask` PyPI package.
@@ -39,6 +40,10 @@ module Flask {
"MethodView"
])
.getASubclass*()
or
result = ModelOutput::getATypeNode("flask.View~Subclass").getASubclass*()
or
result = ModelOutput::getATypeNode("flask.MethodView~Subclass").getASubclass*()
}
}
@@ -52,6 +57,8 @@ module Flask {
API::Node subclassRef() {
result =
API::moduleImport("flask").getMember("views").getMember("MethodView").getASubclass*()
or
result = ModelOutput::getATypeNode("flask.MethodView~Subclass").getASubclass*()
}
}
}
@@ -63,7 +70,10 @@ module Flask {
*/
module FlaskApp {
/** Gets a reference to the `flask.Flask` class. */
API::Node classRef() { result = API::moduleImport("flask").getMember("Flask") }
API::Node classRef() {
result = API::moduleImport("flask").getMember("Flask") or
result = ModelOutput::getATypeNode("flask.Flask~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `flask.Flask` (a flask application). */
API::Node instance() { result = classRef().getReturn() }
@@ -80,6 +90,8 @@ module Flask {
result = API::moduleImport("flask").getMember("Blueprint")
or
result = API::moduleImport("flask").getMember("blueprints").getMember("Blueprint")
or
result = ModelOutput::getATypeNode("flask.Blueprint~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `flask.Blueprint`. */
@@ -87,7 +99,9 @@ module Flask {
}
/** Gets a reference to the `flask.request` object. */
API::Node request() { result = API::moduleImport("flask").getMember("request") }
API::Node request() {
result = API::moduleImport(["flask", "flask_restful"]).getMember("request")
}
/**
* Provides models for the `flask.Response` class
@@ -104,6 +118,8 @@ module Flask {
result = API::moduleImport("flask").getMember("Response")
or
result = [FlaskApp::classRef(), FlaskApp::instance()].getMember("response_class")
or
result = ModelOutput::getATypeNode("flask.Response~Subclass").getASubclass*()
}
/**

9
python/ql/lib/semmle/python/frameworks/Httpx.qll
View File

@@ -9,15 +9,18 @@
private import python
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `httpx` PyPI package.
*
* See
* - https://pypi.org/project/httpx/
* - https://www.python-httpx.org/
*/
private module HttpxModel {
module HttpxModel {
/**
* An outgoing HTTP request, from the `httpx` library.
*
@@ -59,8 +62,10 @@ private module HttpxModel {
*/
module Client {
/** Get a reference to the `httpx.Client` or `httpx.AsyncClient` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("httpx").getMember(["Client", "AsyncClient"])
or
result = ModelOutput::getATypeNode("httpx.Client~Subclass").getASubclass*()
}
/** A method call on a Client that sends off a request */

7
python/ql/lib/semmle/python/frameworks/Invoke.qll
View File

@@ -7,12 +7,15 @@ private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `invoke` PyPI package.
* See https://www.pyinvoke.org/.
*/
private module Invoke {
module Invoke {
// ---------------------------------------------------------------------------
// invoke
// ---------------------------------------------------------------------------
@@ -30,6 +33,8 @@ private module Invoke {
result = API::moduleImport("invoke").getMember("context").getMember("Context")
or
result = API::moduleImport("invoke").getMember("Context")
or
result = ModelOutput::getATypeNode("invoke.context.Context~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `invoke.context.Context`. */

53
python/ql/lib/semmle/python/frameworks/Lxml.qll
View File

@@ -10,15 +10,25 @@ private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides classes modeling security-relevant aspects of the `lxml` PyPI package
*
* See
* - https://pypi.org/project/lxml/
* - https://lxml.de/tutorial.html
*/
private module Lxml {
module Lxml {
/** Gets a reference to the `lxml.etree` module */
API::Node etreeRef() {
result = API::moduleImport("lxml").getMember("etree")
or
result = ModelOutput::getATypeNode("lxml.etree~Alias")
}
// ---------------------------------------------------------------------------
// XPath
// ---------------------------------------------------------------------------
@@ -34,9 +44,7 @@ private module Lxml {
* - https://lxml.de/apidoc/lxml.etree.html#lxml.etree.ETXPath
*/
private class XPathClassCall extends XML::XPathConstruction::Range, DataFlow::CallCfgNode {
XPathClassCall() {
this = API::moduleImport("lxml").getMember("etree").getMember(["XPath", "ETXPath"]).getACall()
}
XPathClassCall() { this = etreeRef().getMember(["XPath", "ETXPath"]).getACall() }
override DataFlow::Node getXPath() { result in [this.getArg(0), this.getArgByName("path")] }
@@ -62,20 +70,11 @@ private module Lxml {
XPathCall() {
exists(API::Node parseResult |
parseResult =
API::moduleImport("lxml")
.getMember("etree")
.getMember(["parse", "fromstring", "fromstringlist", "HTML", "XML"])
.getReturn()
etreeRef().getMember(["parse", "fromstring", "fromstringlist", "HTML", "XML"]).getReturn()
or
// TODO: lxml.etree.parseid(<text>)[0] will contain the root element from parsing <text>
// but we don't really have a way to model that nicely.
parseResult =
API::moduleImport("lxml")
.getMember("etree")
.getMember("XMLParser")
.getReturn()
.getMember("close")
.getReturn()
parseResult = etreeRef().getMember("XMLParser").getReturn().getMember("close").getReturn()
|
this = parseResult.getMember("xpath").getACall()
)
@@ -87,14 +86,7 @@ private module Lxml {
}
class XPathEvaluatorCall extends XML::XPathExecution::Range, DataFlow::CallCfgNode {
XPathEvaluatorCall() {
this =
API::moduleImport("lxml")
.getMember("etree")
.getMember("XPathEvaluator")
.getReturn()
.getACall()
}
XPathEvaluatorCall() { this = etreeRef().getMember("XPathEvaluator").getReturn().getACall() }
override DataFlow::Node getXPath() { result = this.getArg(0) }
@@ -130,9 +122,7 @@ private module Lxml {
* See https://lxml.de/apidoc/lxml.etree.html?highlight=xmlparser#lxml.etree.XMLParser
*/
private class LxmlParser extends InstanceSource, API::CallNode {
LxmlParser() {
this = API::moduleImport("lxml").getMember("etree").getMember("XMLParser").getACall()
}
LxmlParser() { this = etreeRef().getMember("XMLParser").getACall() }
// NOTE: it's not possible to change settings of a parser after constructing it
override predicate vulnerableTo(XML::XmlParsingVulnerabilityKind kind) {
@@ -162,10 +152,7 @@ private module Lxml {
* See https://lxml.de/apidoc/lxml.etree.html?highlight=xmlparser#lxml.etree.get_default_parser
*/
private class LxmlDefaultParser extends InstanceSource, DataFlow::CallCfgNode {
LxmlDefaultParser() {
this =
API::moduleImport("lxml").getMember("etree").getMember("get_default_parser").getACall()
}
LxmlDefaultParser() { this = etreeRef().getMember("get_default_parser").getACall() }
override predicate vulnerableTo(XML::XmlParsingVulnerabilityKind kind) {
// as highlighted by
@@ -240,7 +227,7 @@ private module Lxml {
LxmlParsing() {
functionName in ["fromstring", "fromstringlist", "XML", "XMLID", "parse", "parseid"] and
this = API::moduleImport("lxml").getMember("etree").getMember(functionName).getACall()
this = etreeRef().getMember(functionName).getACall()
}
override DataFlow::Node getAnInput() {
@@ -309,9 +296,7 @@ private module Lxml {
private class LxmlIterparseCall extends API::CallNode, XML::XmlParsing::Range,
FileSystemAccess::Range
{
LxmlIterparseCall() {
this = API::moduleImport("lxml").getMember("etree").getMember("iterparse").getACall()
}
LxmlIterparseCall() { this = etreeRef().getMember("iterparse").getACall() }
override DataFlow::Node getAnInput() { result in [this.getArg(0), this.getArgByName("source")] }

7
python/ql/lib/semmle/python/frameworks/MarkupSafe.qll
View File

@@ -9,12 +9,15 @@ private import semmle.python.dataflow.new.TaintTracking
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `MarkupSafe` PyPI package.
* See https://markupsafe.palletsprojects.com/en/2.0.x/.
*/
private module MarkupSafeModel {
module MarkupSafeModel {
/**
* Provides models for the `markupsafe.Markup` class
*
@@ -26,6 +29,8 @@ private module MarkupSafeModel {
result = API::moduleImport("markupsafe").getMember("Markup")
or
result = API::moduleImport("flask").getMember("Markup")
or
result = ModelOutput::getATypeNode("markupsafe.Markup~Subclass").getASubclass*()
}
/**

3
python/ql/lib/semmle/python/frameworks/Multidict.qll
View File

@@ -9,6 +9,7 @@ private import semmle.python.dataflow.new.TaintTracking
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -28,6 +29,8 @@ module Multidict {
/** Gets a reference to a `MultiDictProxy` class. */
API::Node classRef() {
result = API::moduleImport("multidict").getMember(["MultiDictProxy", "CIMultiDictProxy"])
or
result = ModelOutput::getATypeNode("multidict.MultiDictProxy~Subclass").getASubclass*()
}
/**

9
python/ql/lib/semmle/python/frameworks/Peewee.qll
View File

@@ -11,14 +11,17 @@ private import semmle.python.dataflow.new.TaintTracking
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.PEP249
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `peewee` PyPI package.
* See
* - https://pypi.org/project/peewee/
* - https://docs.peewee-orm.com/en/latest/index.html
*/
private module Peewee {
module Peewee {
/** Provides models for the `peewee.Database` class and subclasses. */
module Database {
/** Gets a reference to the `peewee.Database` class or any subclass. */
@@ -31,7 +34,7 @@ private module Peewee {
.getMember(["SqliteDatabase", "MySQLDatabase", "PostgresqlDatabase"])
.getASubclass*()
or
// Ohter known subclasses, semi auto generated by using
// Other known subclasses, semi auto generated by using
// ```ql
// class DBClass extends Class, SelfRefMixin {
// DBClass() {
@@ -153,6 +156,8 @@ private module Peewee {
.getMember("sqliteq")
.getMember("SqliteQueueDatabase")
.getASubclass*()
or
result = ModelOutput::getATypeNode("peewee.Database~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `peewee.Database` or any subclass. */

11
python/ql/lib/semmle/python/frameworks/Pycurl.qll
View File

@@ -9,15 +9,18 @@
private import python
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `pycurl` PyPI package.
*
* See
* - https://pypi.org/project/pycurl/
* - https://pycurl.io/docs/latest/
*/
private module Pycurl {
module Pycurl {
/**
* Provides models for the `pycurl.Curl` class
*
@@ -25,7 +28,11 @@ private module Pycurl {
*/
module Curl {
/** Gets a reference to the `pycurl.Curl` class. */
private API::Node classRef() { result = API::moduleImport("pycurl").getMember("Curl") }
API::Node classRef() {
result = API::moduleImport("pycurl").getMember("Curl")
or
result = ModelOutput::getATypeNode("pycurl.Curl~Subclass").getASubclass*()
}
/** Gets a reference to an instance of `pycurl.Curl`. */
private API::Node instance() { result = classRef().getReturn() }

3
python/ql/lib/semmle/python/frameworks/Pydantic.qll
View File

@@ -11,6 +11,7 @@ private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.TaintTracking
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -31,6 +32,8 @@ module Pydantic {
/** Gets a reference to a `pydantic.BaseModel` subclass (a pydantic model). */
API::Node subclassRef() {
result = API::moduleImport("pydantic").getMember("BaseModel").getASubclass+()
or
result = ModelOutput::getATypeNode("pydantic.BaseModel~Subclass").getASubclass*()
}
/**

7
python/ql/lib/semmle/python/frameworks/Requests.qll
View File

@@ -12,6 +12,7 @@ private import semmle.python.ApiGraphs
private import semmle.python.dataflow.new.TaintTracking
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.Stdlib
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -22,7 +23,7 @@ private import semmle.python.frameworks.Stdlib
* - https://pypi.org/project/requests/
* - https://requests.readthedocs.io/en/latest/
*/
private module Requests {
module Requests {
/**
* An outgoing HTTP request, from the `requests` library.
*
@@ -91,10 +92,12 @@ private module Requests {
*/
module Response {
/** Gets a reference to the `requests.models.Response` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("requests").getMember("models").getMember("Response")
or
result = API::moduleImport("requests").getMember("Response")
or
result = ModelOutput::getATypeNode("requests.models.Response~Subclass").getASubclass*()
}
/**

31
python/ql/lib/semmle/python/frameworks/RestFramework.qll
View File

@@ -16,6 +16,7 @@ private import semmle.python.ApiGraphs
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.Django
private import semmle.python.frameworks.Stdlib
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -27,7 +28,7 @@ private import semmle.python.frameworks.Stdlib
* - https://www.django-rest-framework.org/
* - https://pypi.org/project/djangorestframework/
*/
private module RestFramework {
module RestFramework {
// ---------------------------------------------------------------------------
// rest_framework.views.APIView handling
// ---------------------------------------------------------------------------
@@ -215,8 +216,10 @@ private module RestFramework {
*/
module Request {
/** Gets a reference to the `rest_framework.request.Request` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("rest_framework").getMember("request").getMember("Request")
or
result = ModelOutput::getATypeNode("rest_framework.request.Request~Subclass").getASubclass*()
}
/**
@@ -299,8 +302,11 @@ private module RestFramework {
*/
module Response {
/** Gets a reference to the `rest_framework.response.Response` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("rest_framework").getMember("response").getMember("Response")
or
result =
ModelOutput::getATypeNode("rest_framework.response.Response~Subclass").getASubclass*()
}
/** A direct instantiation of `rest_framework.response.Response`. */
@@ -328,6 +334,23 @@ private module RestFramework {
* See https://www.django-rest-framework.org/api-guide/exceptions/#api-reference
*/
module ApiException {
API::Node classRef() {
exists(string className |
className in [
"APIException", "ValidationError", "ParseError", "AuthenticationFailed",
"NotAuthenticated", "PermissionDenied", "NotFound", "NotAcceptable"
] and
result =
API::moduleImport("rest_framework")
.getMember("exceptions")
.getMember(className)
.getASubclass*()
)
or
result =
ModelOutput::getATypeNode("rest_framework.exceptions.APIException~Subclass").getASubclass*()
}
/** A direct instantiation of `rest_framework.exceptions.ApiException` or subclass. */
private class ClassInstantiation extends Http::Server::HttpResponse::Range,
DataFlow::CallCfgNode
@@ -345,6 +368,8 @@ private module RestFramework {
.getMember("exceptions")
.getMember(className)
.getACall()
or
this = classRef().getACall() and className = "APIException"
}
override DataFlow::Node getBody() {

13
python/ql/lib/semmle/python/frameworks/SqlAlchemy.qll
View File

@@ -13,6 +13,7 @@ private import semmle.python.Concepts
// This import is done like this to avoid importing the deprecated top-level things that
// would pollute the namespace
private import semmle.python.frameworks.PEP249::PEP249 as PEP249
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -34,10 +35,12 @@ module SqlAlchemy {
*/
module Engine {
/** Gets a reference to a SQLAlchemy Engine class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("sqlalchemy").getMember("engine").getMember("Engine")
or
result = API::moduleImport("sqlalchemy").getMember("future").getMember("Engine")
or
result = ModelOutput::getATypeNode("sqlalchemy.engine.Engine~Subclass").getASubclass*()
}
/**
@@ -87,7 +90,7 @@ module SqlAlchemy {
*/
module Connection {
/** Gets a reference to a SQLAlchemy Connection class. */
private API::Node classRef() {
API::Node classRef() {
result =
API::moduleImport("sqlalchemy")
.getMember("engine")
@@ -95,6 +98,8 @@ module SqlAlchemy {
.getMember("Connection")
or
result = API::moduleImport("sqlalchemy").getMember("future").getMember("Connection")
or
result = ModelOutput::getATypeNode("sqlalchemy.engine.Connection~Subclass").getASubclass*()
}
/**
@@ -178,8 +183,10 @@ module SqlAlchemy {
*/
module Session {
/** Gets a reference to the `sqlalchemy.orm.Session` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("sqlalchemy").getMember("orm").getMember("Session")
or
result = ModelOutput::getATypeNode("sqlalchemy.orm.Session~Subclass").getASubclass*()
}
/**

7
python/ql/lib/semmle/python/frameworks/Starlette.qll
View File

@@ -13,6 +13,7 @@ private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.Stdlib
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -35,6 +36,8 @@ module Starlette {
result = API::moduleImport("starlette").getMember("websockets").getMember("WebSocket")
or
result = API::moduleImport("fastapi").getMember("WebSocket")
or
result = ModelOutput::getATypeNode("starlette.websockets.WebSocket~Subclass").getASubclass*()
}
/**
@@ -100,8 +103,10 @@ module Starlette {
*/
module Url {
/** Gets a reference to the `starlette.requests.URL` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("starlette").getMember("requests").getMember("URL")
or
result = ModelOutput::getATypeNode("starlette.requests.URL~Subclass").getASubclass*()
}
/**

168
python/ql/lib/semmle/python/frameworks/Stdlib.qll
View File

@@ -17,6 +17,7 @@ private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
// modeling split over multiple files to keep this file from becoming too big
private import semmle.python.frameworks.Stdlib.Urllib
private import semmle.python.frameworks.Stdlib.Urllib2
private import semmle.python.frameworks.data.ModelsAsData
/** Provides models for the Python standard library. */
module Stdlib {
@@ -181,8 +182,10 @@ module Stdlib {
*/
module SplitResult {
/** Gets a reference to the `urllib.parse.SplitResult` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("urllib").getMember("parse").getMember("SplitResult")
or
result = ModelOutput::getATypeNode("urllib.parse.SplitResult~Subclass").getASubclass*()
}
/**
@@ -252,8 +255,10 @@ module Stdlib {
*/
module Logger {
/** Gets a reference to the `logging.Logger` class or any subclass. */
private API::Node subclassRef() {
API::Node subclassRef() {
result = API::moduleImport("logging").getMember("Logger").getASubclass*()
or
result = ModelOutput::getATypeNode("logging.Logger~Subclass").getASubclass*()
}
/**
@@ -292,13 +297,15 @@ module Stdlib {
}
/**
* INTERNAL: Do not use.
*
* Provides models for the Python standard library.
*
* This module is marked private as exposing it means committing to 1-year deprecation
* policy, and the code is not in a polished enough state that we want to do so -- at
* least not without having convincing use-cases for it :)
*/
private module StdlibPrivate {
module StdlibPrivate {
// ---------------------------------------------------------------------------
// os
// ---------------------------------------------------------------------------
@@ -1293,14 +1300,36 @@ private module StdlibPrivate {
// pickle
// ---------------------------------------------------------------------------
/** Gets a reference to any of the `pickle` modules. */
API::Node pickle() { result = API::moduleImport(["pickle", "cPickle", "_pickle"]) }
API::Node pickle() {
result = API::moduleImport(["pickle", "cPickle", "_pickle"])
or
result = ModelOutput::getATypeNode("pickle~Alias")
}
/**
* Gets a reference to `pickle.load`
*/
API::Node pickle_load() {
result = pickle().getMember("load")
or
result = ModelOutput::getATypeNode("pickle.load~Alias")
}
/**
* Gets a reference to `pickle.loads`
*/
API::Node pickle_loads() {
result = pickle().getMember("loads")
or
result = ModelOutput::getATypeNode("pickle.loads~Alias")
}
/**
* A call to `pickle.load`
* See https://docs.python.org/3/library/pickle.html#pickle.load
*/
private class PickleLoadCall extends Decoding::Range, DataFlow::CallCfgNode {
PickleLoadCall() { this = pickle().getMember("load").getACall() }
private class PickleLoadCall extends Decoding::Range, API::CallNode {
PickleLoadCall() { this = pickle_load().getACall() }
override predicate mayExecuteInput() { any() }
@@ -1315,8 +1344,8 @@ private module StdlibPrivate {
* A call to `pickle.loads`
* See https://docs.python.org/3/library/pickle.html#pickle.loads
*/
private class PickleLoadsCall extends Decoding::Range, DataFlow::CallCfgNode {
PickleLoadsCall() { this = pickle().getMember("loads").getACall() }
private class PickleLoadsCall extends Decoding::Range, API::CallNode {
PickleLoadsCall() { this = pickle_loads().getACall() }
override predicate mayExecuteInput() { any() }
@@ -1729,8 +1758,21 @@ private module StdlibPrivate {
* See https://docs.python.org/3/library/cgi.html.
*/
module FieldStorage {
/** Gets a reference to the `cgi.FieldStorage` class. */
API::Node classRef() { result = cgi().getMember("FieldStorage") }
/**
* DEPRECATED: Use `subclassRef` predicate instead.
*
* Gets a reference to the `cgi.FieldStorage` class.
*/
deprecated API::Node classRef() {
result = API::moduleImport("cgi").getMember("FieldStorage")
}
/** Gets a reference to the `cgi.FieldStorage` class or any subclass. */
API::Node subclassRef() {
result = API::moduleImport("cgi").getMember("FieldStorage").getASubclass*()
or
result = ModelOutput::getATypeNode("cgi.FieldStorage~Subclass").getASubclass*()
}
/**
* A source of instances of `cgi.FieldStorage`, extend this class to model new instances.
@@ -1753,13 +1795,13 @@ private module StdlibPrivate {
private class ClassInstantiation extends InstanceSource, RemoteFlowSource::Range,
DataFlow::CallCfgNode
{
ClassInstantiation() { this = classRef().getACall() }
ClassInstantiation() { this = subclassRef().getACall() }
override string getSourceType() { result = "cgi.FieldStorage" }
}
/** Gets a reference to an instance of `cgi.FieldStorage`. */
API::Node instance() { result = classRef().getReturn() }
API::Node instance() { result = subclassRef().getReturn() }
/** Gets a reference to the `getvalue` method on a `cgi.FieldStorage` instance. */
API::Node getvalueRef() { result = instance().getMember("getvalue") }
@@ -2013,7 +2055,7 @@ private module StdlibPrivate {
* - https://docs.python.org/3.9/library/http.server.html#http.server.BaseHTTPRequestHandler
* - https://docs.python.org/2.7/library/basehttpserver.html#BaseHTTPServer.BaseHTTPRequestHandler
*/
private module HttpRequestHandler {
module BaseHttpRequestHandler {
/** Gets a reference to the `BaseHttpRequestHandler` class or any subclass. */
API::Node subclassRef() {
result =
@@ -2027,6 +2069,9 @@ private module StdlibPrivate {
API::moduleImport("http").getMember("server").getMember("SimpleHTTPRequestHandler"),
API::moduleImport("http").getMember("server").getMember("CGIHTTPRequestHandler"),
].getASubclass*()
or
result =
ModelOutput::getATypeNode("http.server.BaseHTTPRequestHandler~Subclass").getASubclass*()
}
/** A HttpRequestHandler class definition (most likely in project code). */
@@ -2121,17 +2166,20 @@ private module StdlibPrivate {
// wsgiref.simple_server
// ---------------------------------------------------------------------------
/** Provides models for the `wsgiref.simple_server` module. */
private module WsgirefSimpleServer {
module WsgirefSimpleServer {
API::Node subclassRef() {
result =
API::moduleImport("wsgiref")
.getMember("simple_server")
.getMember("WSGIServer")
.getASubclass*()
or
result =
ModelOutput::getATypeNode("wsgiref.simple_server.WSGIServer~Subclass").getASubclass*()
}
class WsgiServerSubclass extends Class, SelfRefMixin {
WsgiServerSubclass() {
this.getParent() =
API::moduleImport("wsgiref")
.getMember("simple_server")
.getMember("WSGIServer")
.getASubclass*()
.asSource()
.asExpr()
}
WsgiServerSubclass() { this.getParent() = subclassRef().asSource().asExpr() }
}
/**
@@ -2148,13 +2196,7 @@ private module StdlibPrivate {
exists(DataFlow::Node appArg, DataFlow::CallCfgNode setAppCall |
(
setAppCall =
API::moduleImport("wsgiref")
.getMember("simple_server")
.getMember("WSGIServer")
.getASubclass*()
.getReturn()
.getMember("set_app")
.getACall()
WsgirefSimpleServer::subclassRef().getReturn().getMember("set_app").getACall()
or
setAppCall
.(DataFlow::MethodCallNode)
@@ -2292,7 +2334,7 @@ private module StdlibPrivate {
*/
module HttpConnection {
/** Gets a reference to the `http.client.HttpConnection` class. */
private API::Node classRef() {
API::Node classRef() {
exists(string className | className in ["HTTPConnection", "HTTPSConnection"] |
// Python 3
result = API::moduleImport("http").getMember("client").getMember(className)
@@ -2303,6 +2345,8 @@ private module StdlibPrivate {
result =
API::moduleImport("six").getMember("moves").getMember("http_client").getMember(className)
)
or
result = ModelOutput::getATypeNode("http.client.HTTPConnection~Subclass").getASubclass*()
}
/**
@@ -2414,8 +2458,10 @@ private module StdlibPrivate {
*/
module HttpResponse {
/** Gets a reference to the `http.client.HttpResponse` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("http").getMember("client").getMember("HTTPResponse")
or
result = ModelOutput::getATypeNode("http.client.HTTPResponse~Subclass").getASubclass*()
}
/**
@@ -3534,8 +3580,10 @@ private module StdlibPrivate {
*/
module StringIO {
/** Gets a reference to the `io.StringIO` class. */
private API::Node classRef() {
API::Node classRef() {
result = API::moduleImport("io").getMember(["StringIO", "BytesIO"])
or
result = ModelOutput::getATypeNode("io.StringIO~Subclass").getASubclass*()
}
/**
@@ -3585,6 +3633,12 @@ private module StdlibPrivate {
// ---------------------------------------------------------------------------
// xml.etree.ElementTree
// ---------------------------------------------------------------------------
/** Gets a reference to the `xml.etree.ElementTree` class */
API::Node elementTreeClassRef() {
result = API::moduleImport("xml").getMember("etree").getMember("ElementTree").getASubclass*() or
result = ModelOutput::getATypeNode("xml.etree.ElementTree~Subclass").getASubclass*()
}
/**
* An instance of `xml.etree.ElementTree.ElementTree`.
*
@@ -3592,20 +3646,10 @@ private module StdlibPrivate {
*/
private API::Node elementTreeInstance() {
//parse to a tree
result =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
.getMember("parse")
.getReturn()
result = elementTreeClassRef().getMember("parse").getReturn()
or
// construct a tree without parsing
result =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
.getMember("ElementTree")
.getReturn()
result = elementTreeClassRef().getMember("ElementTree").getReturn()
}
/**
@@ -3618,21 +3662,9 @@ private module StdlibPrivate {
result = elementTreeInstance().getMember(["parse", "getroot"]).getReturn()
or
// parse directly to an element
result =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
.getMember(["fromstring", "fromstringlist", "XML"])
.getReturn()
result = elementTreeClassRef().getMember(["fromstring", "fromstringlist", "XML"]).getReturn()
or
result =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
.getMember("XMLParser")
.getReturn()
.getMember("close")
.getReturn()
result = elementTreeClassRef().getMember("XMLParser").getReturn().getMember("close").getReturn()
}
/**
@@ -3677,12 +3709,7 @@ private module StdlibPrivate {
/** A direct instantiation of `xml.etree` parsers. */
private class ClassInstantiation extends InstanceSource, DataFlow::CallCfgNode {
ClassInstantiation() {
this =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
.getMember(["XMLParser", "XMLPullParser"])
.getACall()
this = elementTreeClassRef().getMember(["XMLParser", "XMLPullParser"]).getACall()
}
}
@@ -3739,9 +3766,7 @@ private module StdlibPrivate {
private class XmlEtreeParsing extends DataFlow::CallCfgNode, XML::XmlParsing::Range {
XmlEtreeParsing() {
this =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
elementTreeClassRef()
.getMember(["fromstring", "fromstringlist", "XML", "XMLID", "parse", "iterparse"])
.getACall()
or
@@ -3789,12 +3814,7 @@ private module StdlibPrivate {
*/
private class FileAccessFromXmlEtreeParsing extends XmlEtreeParsing, FileSystemAccess::Range {
FileAccessFromXmlEtreeParsing() {
this =
API::moduleImport("xml")
.getMember("etree")
.getMember("ElementTree")
.getMember(["parse", "iterparse"])
.getACall()
this = elementTreeClassRef().getMember(["parse", "iterparse"]).getACall()
or
this = elementTreeInstance().getMember("parse").getACall()
// I considered whether we should try to reduce FPs from people passing file-like

20
python/ql/lib/semmle/python/frameworks/Tornado.qll
View File

@@ -12,6 +12,7 @@ private import semmle.python.ApiGraphs
private import semmle.python.regex
private import semmle.python.frameworks.Stdlib
private import semmle.python.frameworks.internal.InstanceTaintStepsHelper
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
@@ -87,7 +88,11 @@ module Tornado {
*/
module RequestHandler {
/** Gets a reference to the `tornado.web.RequestHandler` class or any subclass. */
API::Node subclassRef() { result = web().getMember("RequestHandler").getASubclass*() }
API::Node subclassRef() {
result = web().getMember("RequestHandler").getASubclass*()
or
result = ModelOutput::getATypeNode("tornado.web.RequestHandler~Subclass").getASubclass*()
}
/** A RequestHandler class (most likely in project code). */
class RequestHandlerClass extends Class {
@@ -213,7 +218,11 @@ module Tornado {
*/
module Application {
/** Gets a reference to the `tornado.web.Application` class. */
API::Node classRef() { result = web().getMember("Application") }
API::Node classRef() {
result = web().getMember("Application")
or
result = ModelOutput::getATypeNode("tornado.web.Application~Subclass").getASubclass*()
}
/**
* A source of instances of `tornado.web.Application`, extend this class to model new instances.
@@ -270,7 +279,12 @@ module Tornado {
*/
module HttpServerRequest {
/** Gets a reference to the `tornado.httputil.HttpServerRequest` class. */
API::Node classRef() { result = httputil().getMember("HttpServerRequest") }
API::Node classRef() {
result = httputil().getMember("HttpServerRequest")
or
result =
ModelOutput::getATypeNode("tornado.httputil.HttpServerRequest~Subclass").getASubclass*()
}
/**
* A source of instances of `tornado.httputil.HttpServerRequest`, extend this class to model new instances.

9
python/ql/lib/semmle/python/frameworks/Urllib3.qll
View File

@@ -9,15 +9,18 @@
private import python
private import semmle.python.Concepts
private import semmle.python.ApiGraphs
private import semmle.python.frameworks.data.ModelsAsData
/**
* INTERNAL: Do not use.
*
* Provides models for the `urllib3` PyPI package.
*
* See
* - https://pypi.org/project/urllib3/
* - https://urllib3.readthedocs.io/en/stable/reference/
*/
private module Urllib3 {
module Urllib3 {
/**
* Provides models for the `urllib3.request.RequestMethods` class and subclasses, such
* as the `urllib3.PoolManager` class
@@ -30,7 +33,7 @@ private module Urllib3 {
*/
module PoolManager {
/** Gets a reference to the `urllib3.PoolManager` class. */
private API::Node classRef() {
API::Node classRef() {
result =
API::moduleImport("urllib3")
.getMember(["PoolManager", "ProxyManager", "HTTPConnectionPool", "HTTPSConnectionPool"])
@@ -40,6 +43,8 @@ private module Urllib3 {
.getMember("request")
.getMember("RequestMethods")
.getASubclass+()
or
result = ModelOutput::getATypeNode("urllib3.PoolManager~Subclass").getASubclass*()
}
/**

174831
python/ql/lib/semmle/python/frameworks/data/internal/subclass-capture/ALL.model.yml generated Normal file
View File

File diff suppressed because it is too large Load Diff

15
python/ql/lib/semmle/python/frameworks/internal/SelfRefMixin.qll
View File

@@ -6,6 +6,7 @@
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.internal.DataFlowDispatch
/**
* INTERNAL: Do not use.
@@ -16,13 +17,16 @@ abstract class SelfRefMixin extends Class {
/**
* Gets a reference to instances of this class, originating from a self parameter of
* a method defined on this class.
*
* Note: TODO: This doesn't take MRO into account
* Note: TODO: This doesn't take staticmethod/classmethod into account
*/
private DataFlow::TypeTrackingNode getASelfRef(DataFlow::TypeTracker t) {
t.start() and
result.(DataFlow::ParameterNode).getParameter() = this.getAMethod().getArg(0)
exists(Class cls, Function meth |
cls = getADirectSuperclass*(this) and
meth = cls.getAMethod() and
not isStaticmethod(meth) and
not isClassmethod(meth) and
result.(DataFlow::ParameterNode).getParameter() = meth.getArg(0)
)
or
exists(DataFlow::TypeTracker t2 | result = this.getASelfRef(t2).track(t2, t))
}
@@ -30,9 +34,6 @@ abstract class SelfRefMixin extends Class {
/**
* Gets a reference to instances of this class, originating from a self parameter of
* a method defined on this class.
*
* Note: TODO: This doesn't take MRO into account
* Note: TODO: This doesn't take staticmethod/classmethod into account
*/
DataFlow::Node getASelfRef() { this.getASelfRef(DataFlow::TypeTracker::end()).flowsTo(result) }
}

145
python/ql/lib/semmle/python/frameworks/internal/SubclassFinder.qll
View File

@@ -7,32 +7,23 @@
private import python
private import semmle.python.dataflow.new.DataFlow
private import semmle.python.dataflow.new.internal.ImportResolution
private import semmle.python.ApiGraphs
private import semmle.python.filters.Tests
private import semmle.python.Module
// very much inspired by the draft at https://github.com/github/codeql/pull/5632
private module NotExposed {
module NotExposed {
// Instructions:
// This needs to be automated better, but for this prototype, here are some rough instructions:
// 0) get a database of the library you are about to model
// 1) fill out the `getAlreadyModeledClass` body below
// 2) quick-eval the `quickEvalMe` predicate below, and copy the output to your modeling predicate
class MySpec extends FindSubclassesSpec {
MySpec() { this = "MySpec" }
override API::Node getAlreadyModeledClass() {
// FILL ME OUT ! (but don't commit with any changes)
none()
// for example
// result = API::moduleImport("rest_framework").getMember("views").getMember("APIView")
}
}
predicate quickEvalMe(string newImport) {
newImport =
"// imports generated by python/frameworks/internal/SubclassFinder.qll\n" + "this = API::" +
concat(string newModelFullyQualified |
newModel(any(MySpec spec), newModelFullyQualified, _, _, _)
newModel(any(FindSubclassesSpec spec), newModelFullyQualified, _, _, _)
|
fullyQualifiedToApiGraphPath(newModelFullyQualified), " or this = API::"
)
@@ -75,14 +66,31 @@ private module NotExposed {
bindingset[this]
abstract class FindSubclassesSpec extends string {
/**
* Gets an API node for a class that has already been modeled. You can include
* `.getASubclass*()` without causing problems, but it is not needed.
*/
abstract API::Node getAlreadyModeledClass();
/**
* Gets the fully qualified name that this spec represents.
*
* This should be implemented by all classes for which `getSuperClass` is
* implemented, at least if they are defined in a different module than what they
* subclass.
*/
string getFullyQualifiedName() { none() }
FindSubclassesSpec getSuperClass() { none() }
final FindSubclassesSpec getSubClass() { result.getSuperClass() = this }
}
/**
* Holds if `newModelFullyQualified` describes either a new subclass, or a new alias, belonging to `spec` that we should include in our automated modeling.
* This new element is defined by `ast`, which is defined at `loc` in the module `mod`.
*/
query predicate newModel(
predicate newModel(
FindSubclassesSpec spec, string newModelFullyQualified, AstNode ast, Module mod, Location loc
) {
(
@@ -90,7 +98,8 @@ private module NotExposed {
or
newDirectAlias(spec, newModelFullyQualified, ast, mod, loc)
or
newImportStar(spec, newModelFullyQualified, ast, mod, _, _, loc)
newImportAlias(spec, newModelFullyQualified, mod, _, _, loc) and
ast = mod
)
}
@@ -103,15 +112,34 @@ private module NotExposed {
)
}
/**
* Holds if `fullyQualifiedName` is already explicitly modeled in the `spec`.
*
* For specs that do `.getASubclass*()`, items found by following a `.getASubclass`
* edge will not be considered explicitly modeled.
*/
bindingset[fullyQualifiedName]
predicate alreadyModeled(FindSubclassesSpec spec, string fullyQualifiedName) {
predicate alreadyExplicitlyModeled(FindSubclassesSpec spec, string fullyQualifiedName) {
fullyQualifiedToApiGraphPath(fullyQualifiedName) = spec.getAlreadyModeledClass().getPath()
}
predicate isNonTestProjectCode(AstNode ast) {
not ast.getScope*() instanceof TestScope and
not ast.getLocation().getFile().getRelativePath().matches("tests/%") and
exists(ast.getLocation().getFile().getRelativePath())
predicate isAllowedModule(Module mod) {
// for tests
mod.getName() = "find_subclass_test"
or
// don't include anything found in site-packages
exists(mod.getFile().getRelativePath()) and
not mod.getFile().getRelativePath().regexpMatch("(?i)((^|/)examples?|^docs)/.*") and
// to counter things like `my-example/app/foo.py` being allowed under `app.foo`
forall(string part | part = mod.getFile().getParent().getRelativePath().splitAt("/") |
legalShortName(part)
)
}
predicate isTestCode(AstNode ast) {
ast.getScope*() instanceof TestScope
or
ast.getLocation().getFile().getRelativePath().matches("tests/%")
}
predicate hasAllStatement(Module mod) {
@@ -144,40 +172,47 @@ private module NotExposed {
* ```
*/
predicate newDirectAlias(
FindSubclassesSpec spec, string newAliasFullyQualified, ImportMember importMember, Module mod,
Location loc
FindSubclassesSpec spec, string newAliasFullyQualified, Expr value, Module mod, Location loc
) {
importMember = newOrExistingModeling(spec).getAValueReachableFromSource().asExpr() and
importMember.getScope() = mod and
loc = importMember.getLocation() and
(
mod.isPackageInit() and
newAliasFullyQualified = mod.getPackageName() + "." + importMember.getName()
or
not mod.isPackageInit() and
newAliasFullyQualified = mod.getName() + "." + importMember.getName()
) and
(
not hasAllStatement(mod)
or
mod.declaredInAll(importMember.getName())
) and
not alreadyModeled(spec, newAliasFullyQualified) and
isNonTestProjectCode(importMember)
exists(Alias alias | value = alias.getValue() |
value = newOrExistingModeling(spec).getASubclass*().getAValueReachableFromSource().asExpr() and
value.getScope() = mod and
loc = value.getLocation() and
exists(string base |
mod.isPackageInit() and base = mod.getPackageName()
or
not mod.isPackageInit() and base = mod.getName()
|
newAliasFullyQualified = base + "." + alias.getAsname().(Name).getId()
) and
(
not hasAllStatement(mod)
or
mod.declaredInAll(alias.getAsname().(Name).getId())
) and
not alreadyExplicitlyModeled(spec, newAliasFullyQualified) and
not isTestCode(value) and
isAllowedModule(mod)
)
}
/** same as `newDirectAlias` predicate, but handling `from <module> import *`, considering all `<member>`, where `<module>.<member>` belongs to `spec`. */
predicate newImportStar(
FindSubclassesSpec spec, string newAliasFullyQualified, ImportStar importStar, Module mod,
API::Node relevantClass, string relevantName, Location loc
/**
* same as `newDirectAlias` predicate, but written in a generic way to handle any import (also import *).
*
* it might be safe to delete `newDirectAlias` with this in place, but have not done the testing yet.
*/
predicate newImportAlias(
FindSubclassesSpec spec, string newAliasFullyQualified, Module mod, DataFlow::Node def,
string relevantName, Location loc
) {
relevantClass = newOrExistingModeling(spec) and
loc = importStar.getLocation() and
importStar.getScope() = mod and
// WHAT A HACK :D :D
relevantClass.getPath() =
relevantClass.getAPredecessor().getPath() + ".getMember(\"" + relevantName + "\")" and
relevantClass.getAPredecessor().getAValueReachableFromSource().asExpr() = importStar.getModule() and
loc = mod.getLocation() and
exists(API::Node relevantClass, ControlFlowNode value |
relevantClass = newOrExistingModeling(spec).getASubclass*() and
ImportResolution::module_export(mod, relevantName, def) and
value = relevantClass.getAValueReachableFromSource().asCfgNode() and
value = def.asCfgNode()
// value could be a ClassExpr if a new class is defined, or a Name if defining an alias
) and
(
mod.isPackageInit() and
newAliasFullyQualified = mod.getPackageName() + "." + relevantName
@@ -190,8 +225,9 @@ private module NotExposed {
or
mod.declaredInAll(relevantName)
) and
not alreadyModeled(spec, newAliasFullyQualified) and
isNonTestProjectCode(importStar)
not alreadyExplicitlyModeled(spec, newAliasFullyQualified) and
not isTestCode(mod) and
isAllowedModule(mod)
}
/** Holds if `classExpr` defines a new subclass that belongs to `spec`, which has the fully qualified name `newSubclassQualified`. */
@@ -203,7 +239,8 @@ private module NotExposed {
classExpr.getScope() = mod and
newSubclassQualified = mod.getName() + "." + classExpr.getName() and
loc = classExpr.getLocation() and
not alreadyModeled(spec, newSubclassQualified) and
isNonTestProjectCode(classExpr)
not alreadyExplicitlyModeled(spec, newSubclassQualified) and
not isTestCode(classExpr) and
isAllowedModule(mod)
}
}

4
python/ql/src/CHANGELOG.md
View File

@@ -1,3 +1,7 @@
## 0.9.6
No user-facing changes.
## 0.9.5
No user-facing changes.

26
python/ql/src/Security/CWE-601/UrlRedirect.qhelp
View File

@@ -16,6 +16,10 @@ To guard against untrusted URL redirection, it is advisable to avoid putting use
directly into a redirect URL. Instead, maintain a list of authorized
redirects on the server; then choose from that list based on the user input provided.
</p>
<p>
If this is not possible, then the user input should be validated in some other way,
for example, by verifying that the target URL does not include an explicit host name.
</p>
</recommendation>
<example>
@@ -27,11 +31,29 @@ without validating the input, which facilitates phishing attacks:
<sample src="examples/redirect_bad.py"/>
<p>
One way to remedy the problem is to validate the user input against a known fixed string
before doing the redirection:
If you know the set of valid redirect targets, you can maintain a list of them on the server
and check that the user input is in that list:
</p>
<sample src="examples/redirect_good.py"/>
<p>
Often this is not possible, so an alternative is to check that the target URL does not
specify an explicit host name. For example, the Django framework provides a
function <code>url_has_allowed_host_and_scheme</code> that can be used to check that a
URL is safe to redirect to, as shown in the following example:
</p>
<sample src="examples/redirect_good2.py"/>
<p>
Note that many browsers accept backslash characters (<code>\</code>) as equivalent to
forward slash characters (<code>/</code>) in URLs, so it is important to account for this
when validating the URL, for example by first replacing all backslashes with forward
slashes. Django's <code>url_has_allowed_host_and_scheme</code> function
does this automatically, but other libraries may not.
</p>
</example>
<references>

3
python/ql/src/Security/CWE-601/examples/redirect_good.py
View File

@@ -10,4 +10,5 @@ def hello():
if target == VALID_REDIRECT:
return redirect(target, code=302)
else:
... # Error
# ignore the target and redirect to the home page
return redirect('/', code=302)

13
python/ql/src/Security/CWE-601/examples/redirect_good2.py Normal file
View File

@@ -0,0 +1,13 @@
from django.http import HttpResponseRedirect
from django.shortcuts import redirect
from django.utils.http import url_has_allowed_host_and_scheme
from django.views import View
class RedirectView(View):
def get(self, request, *args, **kwargs):
target = request.GET.get('target', '')
if url_has_allowed_host_and_scheme(target, allowed_hosts=None):
return HttpResponseRedirect(target)
else:
# ignore the target and redirect to the home page
return redirect('/')

3
python/ql/src/change-notes/released/0.9.6.md Normal file
View File

@@ -0,0 +1,3 @@
## 0.9.6
No user-facing changes.

2
python/ql/src/codeql-pack.release.yml
View File

@@ -1,2 +1,2 @@
---
lastReleaseVersion: 0.9.5
lastReleaseVersion: 0.9.6

589
python/ql/src/meta/ClassHierarchy/Find.ql Normal file
View File

@@ -0,0 +1,589 @@
/**
* @name Find new subclasses to model
* @id py/meta/find-subclasses-to-model
* @kind table
*/
import python
import semmle.python.dataflow.new.DataFlow
private import semmle.python.ApiGraphs
import semmle.python.frameworks.internal.SubclassFinder::NotExposed
private import semmle.python.frameworks.Flask
private import semmle.python.frameworks.FastApi
private import semmle.python.frameworks.Django
private import semmle.python.frameworks.Tornado
private import semmle.python.frameworks.Stdlib
private import semmle.python.frameworks.Requests
private import semmle.python.frameworks.Starlette
private import semmle.python.frameworks.ClickhouseDriver
private import semmle.python.frameworks.Aiohttp
private import semmle.python.frameworks.Fabric
private import semmle.python.frameworks.Httpx
private import semmle.python.frameworks.Invoke
private import semmle.python.frameworks.MarkupSafe
private import semmle.python.frameworks.Multidict
private import semmle.python.frameworks.Pycurl
private import semmle.python.frameworks.RestFramework
private import semmle.python.frameworks.SqlAlchemy
private import semmle.python.frameworks.Tornado
private import semmle.python.frameworks.Urllib3
private import semmle.python.frameworks.Pydantic
private import semmle.python.frameworks.Peewee
private import semmle.python.frameworks.Aioch
private import semmle.python.frameworks.Lxml
import semmle.python.frameworks.data.internal.ApiGraphModelsExtensions as Extensions
class FlaskViewClasses extends FindSubclassesSpec {
FlaskViewClasses() { this = "flask.View~Subclass" }
override API::Node getAlreadyModeledClass() { result = Flask::Views::View::subclassRef() }
}
class FlaskMethodViewClasses extends FindSubclassesSpec {
FlaskMethodViewClasses() { this = "flask.MethodView~Subclass" }
override API::Node getAlreadyModeledClass() { result = Flask::Views::MethodView::subclassRef() }
override FindSubclassesSpec getSuperClass() { result instanceof FlaskViewClasses }
override string getFullyQualifiedName() { result = "flask.views.MethodView" }
}
class FastApiRouter extends FindSubclassesSpec {
FastApiRouter() { this = "fastapi.APIRouter~Subclass" }
override API::Node getAlreadyModeledClass() { result = FastApi::ApiRouter::cls() }
}
class DjangoForms extends FindSubclassesSpec {
DjangoForms() { this = "django.forms.BaseForm~Subclass" }
override API::Node getAlreadyModeledClass() {
result = any(Django::Forms::Form::ModeledSubclass subclass)
}
}
class DjangoView extends FindSubclassesSpec {
DjangoView() { this = "Django.Views.View~Subclass" }
override API::Node getAlreadyModeledClass() {
result = any(Django::Views::View::ModeledSubclass subclass)
}
}
class DjangoField extends FindSubclassesSpec {
DjangoField() { this = "Django.Forms.Field~Subclass" }
override API::Node getAlreadyModeledClass() {
result = any(Django::Forms::Field::ModeledSubclass subclass)
}
}
class DjangoModel extends FindSubclassesSpec {
DjangoModel() { this = "Django.db.models.Model~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DB::Models::Model::subclassRef()
}
}
class TornadoRequestHandler extends FindSubclassesSpec {
TornadoRequestHandler() { this = "tornado.web.RequestHandler~Subclass" }
override API::Node getAlreadyModeledClass() {
result = Tornado::TornadoModule::Web::RequestHandler::subclassRef()
}
}
class WSGIServer extends FindSubclassesSpec {
WSGIServer() { this = "wsgiref.simple_server.WSGIServer~Subclass" }
override API::Node getAlreadyModeledClass() {
result = StdlibPrivate::WsgirefSimpleServer::subclassRef()
}
}
class StdlibBaseHttpRequestHandler extends FindSubclassesSpec {
StdlibBaseHttpRequestHandler() { this = "http.server.BaseHTTPRequestHandler~Subclass" }
override API::Node getAlreadyModeledClass() {
result = StdlibPrivate::BaseHttpRequestHandler::subclassRef()
}
}
class StdlibCgiFieldStorage extends FindSubclassesSpec {
StdlibCgiFieldStorage() { this = "cgi.FieldStorage~Subclass" }
override API::Node getAlreadyModeledClass() {
result = StdlibPrivate::Cgi::FieldStorage::subclassRef()
}
}
class DjangoHttpResponse extends FindSubclassesSpec {
DjangoHttpResponse() { this = "django.http.response.HttpResponse~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponse::classRef()
}
}
class DjangoHttpResponseRedirect extends FindSubclassesSpec {
DjangoHttpResponseRedirect() { this = "django.http.response.HttpResponseRedirect~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseRedirect::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.HttpResponseRedirect" }
}
class DjangoHttpResponsePermanentRedirect extends FindSubclassesSpec {
DjangoHttpResponsePermanentRedirect() {
this = "django.http.response.HttpResponsePermanentRedirect~Subclass"
}
override API::Node getAlreadyModeledClass() {
result =
PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponsePermanentRedirect::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() {
result = "django.http.response.HttpResponsePermanentRedirect"
}
}
class DjangoHttpResponseNotModified extends FindSubclassesSpec {
DjangoHttpResponseNotModified() { this = "django.http.response.HttpResponseNotModified~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseNotModified::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() {
result = "django.http.response.HttpResponseNotModified"
}
}
class DjangoHttpResponseBadRequest extends FindSubclassesSpec {
DjangoHttpResponseBadRequest() { this = "django.http.response.HttpResponseBadRequest~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseBadRequest::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.HttpResponseBadRequest" }
}
class DjangoHttpResponseNotFound extends FindSubclassesSpec {
DjangoHttpResponseNotFound() { this = "django.http.response.HttpResponseNotFound~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseNotFound::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.HttpResponseNotFound" }
}
class DjangoHttpResponseForbidden extends FindSubclassesSpec {
DjangoHttpResponseForbidden() { this = "django.http.response.HttpResponseForbidden~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseForbidden::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.HttpResponseForbidden" }
}
class DjangoHttpResponseNotAllowed extends FindSubclassesSpec {
DjangoHttpResponseNotAllowed() { this = "django.http.response.HttpResponseNotAllowed~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseNotAllowed::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.HttpResponseNotAllowed" }
}
class DjangoHttpResponseGone extends FindSubclassesSpec {
DjangoHttpResponseGone() { this = "django.http.response.HttpResponseGone~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseGone::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.HttpResponseGone" }
}
class DjangoHttpResponseServerError extends FindSubclassesSpec {
DjangoHttpResponseServerError() { this = "django.http.response.HttpResponseServerError~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::HttpResponseServerError::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() {
result = "django.http.response.HttpResponseServerError"
}
}
class DjangoHttpResponseJsonResponse extends FindSubclassesSpec {
DjangoHttpResponseJsonResponse() { this = "django.http.response.JsonResponse~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::JsonResponse::classRef()
}
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "django.http.response.JsonResponse" }
}
class DjangoHttpResponseStreamingResponse extends FindSubclassesSpec {
DjangoHttpResponseStreamingResponse() {
this = "django.http.response.StreamingHttpResponse~Subclass"
}
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::StreamingHttpResponse::classRef()
}
}
class DjangoHttpResponseFileResponse extends FindSubclassesSpec {
DjangoHttpResponseFileResponse() { this = "django.http.response.FileResponse~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Response::FileResponse::classRef()
}
override FindSubclassesSpec getSuperClass() {
result instanceof DjangoHttpResponseStreamingResponse
}
override string getFullyQualifiedName() { result = "django.http.response.FileResponse" }
}
class FlaskResponse extends FindSubclassesSpec {
FlaskResponse() { this = "flask.Response~Subclass" }
override API::Node getAlreadyModeledClass() { result = Flask::Response::classRef() }
}
class RequestsResponse extends FindSubclassesSpec {
RequestsResponse() { this = "requests.models.Response~Subclass" }
override API::Node getAlreadyModeledClass() { result = Requests::Response::classRef() }
}
class HttpClientHttpResponse extends FindSubclassesSpec {
HttpClientHttpResponse() { this = "http.client.HTTPResponse~Subclass" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::HttpResponse::classRef() }
}
class StarletteWebsocket extends FindSubclassesSpec {
StarletteWebsocket() { this = "starlette.websockets.WebSocket~Subclass" }
override API::Node getAlreadyModeledClass() { result = Starlette::WebSocket::classRef() }
}
class StarletteUrl extends FindSubclassesSpec {
StarletteUrl() { this = "starlette.requests.URL~Subclass" }
override API::Node getAlreadyModeledClass() { result = Starlette::Url::classRef() }
}
class ClickhouseClient extends FindSubclassesSpec {
ClickhouseClient() { this = "clickhouse_driver.client.Client~Subclass" }
override API::Node getAlreadyModeledClass() { result = ClickhouseDriver::Client::subclassRef() }
}
class AiohttpSession extends FindSubclassesSpec {
AiohttpSession() { this = "aiohttp.ClientSession~Subclass" }
override API::Node getAlreadyModeledClass() {
result = AiohttpClientModel::ClientSession::classRef()
}
}
class FabricConnection extends FindSubclassesSpec {
FabricConnection() { this = "fabric.connection.Connection~Subclass" }
override API::Node getAlreadyModeledClass() {
result = FabricV2::Fabric::Connection::ConnectionClass::classRef()
}
}
class DjangoRawSql extends FindSubclassesSpec {
DjangoRawSql() { this = "django.db.models.expressions.RawSQL~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DB::Models::Expressions::RawSql::classRef()
}
}
class DjangoHttpRequest extends FindSubclassesSpec {
DjangoHttpRequest() { this = "django.http.request.HttpRequest~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DjangoHttp::Request::HttpRequest::classRef()
}
}
class FlaskClass extends FindSubclassesSpec {
FlaskClass() { this = "flask.Flask~Subclass" }
override API::Node getAlreadyModeledClass() { result = Flask::FlaskApp::classRef() }
}
class FlaskBlueprint extends FindSubclassesSpec {
FlaskBlueprint() { this = "flask.Blueprint~Subclass" }
override API::Node getAlreadyModeledClass() { result = Flask::Blueprint::classRef() }
}
class HttpxClient extends FindSubclassesSpec {
HttpxClient() { this = "httpx.Client~Subclass" }
override API::Node getAlreadyModeledClass() { result = HttpxModel::Client::classRef() }
}
class InvokeContext extends FindSubclassesSpec {
InvokeContext() { this = "invoke.context.Context~Subclass" }
override API::Node getAlreadyModeledClass() {
result = Invoke::InvokeModule::Context::ContextClass::classRef()
}
}
class MarkupSafe extends FindSubclassesSpec {
MarkupSafe() { this = "markupsafe.Markup~Subclass" }
override API::Node getAlreadyModeledClass() { result = MarkupSafeModel::Markup::classRef() }
}
class Multidict extends FindSubclassesSpec {
Multidict() { this = "multidict.MultiDictProxy~Subclass" }
override API::Node getAlreadyModeledClass() { result = Multidict::MultiDictProxy::classRef() }
}
class PyCurl extends FindSubclassesSpec {
PyCurl() { this = "pycurl.Curl~Subclass" }
override API::Node getAlreadyModeledClass() { result = Pycurl::Curl::classRef() }
}
class RestFrameworkRequest extends FindSubclassesSpec {
RestFrameworkRequest() { this = "rest_framework.request.Request~Subclass" }
override API::Node getAlreadyModeledClass() { result = RestFramework::Request::classRef() }
}
class RestFrameworkResponse extends FindSubclassesSpec {
RestFrameworkResponse() { this = "rest_framework.response.Response~Subclass" }
override API::Node getAlreadyModeledClass() { result = RestFramework::Response::classRef() }
override FindSubclassesSpec getSuperClass() { result instanceof DjangoHttpResponse }
override string getFullyQualifiedName() { result = "rest_framework.response.Response" }
}
class SqlAlchemyEngine extends FindSubclassesSpec {
SqlAlchemyEngine() { this = "sqlalchemy.engine.Engine~Subclass" }
override API::Node getAlreadyModeledClass() { result = SqlAlchemy::Engine::classRef() }
}
class SqlAlchemyConnection extends FindSubclassesSpec {
SqlAlchemyConnection() { this = "sqlalchemy.engine.Connection~Subclass" }
override API::Node getAlreadyModeledClass() { result = SqlAlchemy::Connection::classRef() }
}
class SqlAlchemySession extends FindSubclassesSpec {
SqlAlchemySession() { this = "sqlalchemy.orm.Session~Subclass" }
override API::Node getAlreadyModeledClass() { result = SqlAlchemy::Session::classRef() }
}
class UrlLibParseSplitResult extends FindSubclassesSpec {
UrlLibParseSplitResult() { this = "urllib.parse.SplitResult~Subclass" }
override API::Node getAlreadyModeledClass() { result = Stdlib::SplitResult::classRef() }
}
class StdlibHttpConnection extends FindSubclassesSpec {
StdlibHttpConnection() { this = "http.client.HTTPConnection~Subclass" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::HttpConnection::classRef() }
}
class StringIO extends FindSubclassesSpec {
StringIO() { this = "io.StringIO~Subclass" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::StringIO::classRef() }
}
class TornadoApplication extends FindSubclassesSpec {
TornadoApplication() { this = "tornado.web.Application~Subclass" }
override API::Node getAlreadyModeledClass() {
result = Tornado::TornadoModule::Web::Application::classRef()
}
}
class TornadoRequest extends FindSubclassesSpec {
TornadoRequest() { this = "tornado.httputil.HttpServerRequest~Subclass" }
override API::Node getAlreadyModeledClass() {
result = Tornado::TornadoModule::HttpUtil::HttpServerRequest::classRef()
}
}
class Urllib3PoolManager extends FindSubclassesSpec {
Urllib3PoolManager() { this = "urllib3.PoolManager~Subclass" }
override API::Node getAlreadyModeledClass() { result = Urllib3::PoolManager::classRef() }
}
class StdlibLogger extends FindSubclassesSpec {
StdlibLogger() { this = "logging.Logger~Subclass" }
override API::Node getAlreadyModeledClass() { result = Stdlib::Logger::subclassRef() }
}
class PydanticBaseModel extends FindSubclassesSpec {
PydanticBaseModel() { this = "pydantic.BaseModel~Subclass" }
override API::Node getAlreadyModeledClass() { result = Pydantic::BaseModel::subclassRef() }
}
class PeeweeDatabase extends FindSubclassesSpec {
PeeweeDatabase() { this = "peewee.Database~Subclass" }
override API::Node getAlreadyModeledClass() { result = Peewee::Database::subclassRef() }
}
class AiochClient extends FindSubclassesSpec {
AiochClient() { this = "aioch.Client~Subclass" }
override API::Node getAlreadyModeledClass() { result = Aioch::Client::subclassRef() }
}
class AiohttpView extends FindSubclassesSpec {
AiohttpView() { this = "aiohttp.web.View~Subclass" }
override API::Node getAlreadyModeledClass() { result = AiohttpWebModel::View::subclassRef() }
}
class DjangoFileField extends FindSubclassesSpec {
DjangoFileField() { this = "django.db.models.FileField~Subclass" }
override API::Node getAlreadyModeledClass() {
result = PrivateDjango::DjangoImpl::DB::Models::FileField::subclassRef()
}
}
class RestFrameworkApiException extends FindSubclassesSpec {
RestFrameworkApiException() { this = "rest_framework.exceptions.APIException~Subclass" }
override API::Node getAlreadyModeledClass() { result = RestFramework::ApiException::classRef() }
}
class ElementTree extends FindSubclassesSpec {
ElementTree() { this = "xml.etree.ElementTree~Subclass" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::elementTreeClassRef() }
}
class LxmlETreeAlias extends FindSubclassesSpec {
LxmlETreeAlias() { this = "lxml.etree~Alias" }
override API::Node getAlreadyModeledClass() { result = Lxml::etreeRef() }
}
class PickleAlias extends FindSubclassesSpec {
PickleAlias() { this = "pickle~Alias" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::pickle() }
}
class PickleLoadAlias extends FindSubclassesSpec {
PickleLoadAlias() { this = "pickle.load~Alias" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::pickle_load() }
}
class PickleLoadsAlias extends FindSubclassesSpec {
PickleLoadsAlias() { this = "pickle.loads~Alias" }
override API::Node getAlreadyModeledClass() { result = StdlibPrivate::pickle_loads() }
}
bindingset[fullyQualified]
predicate fullyQualifiedToYamlFormat(string fullyQualified, string type2, string path) {
exists(int firstDot | firstDot = fullyQualified.indexOf(".", 0, 0) |
type2 = fullyQualified.prefix(firstDot) and
path =
("Member[" + fullyQualified.suffix(firstDot + 1).replaceAll(".", "].Member[") + "]")
.replaceAll(".Member[__init__].", "")
.replaceAll("Member[__init__].", "")
)
}
from FindSubclassesSpec spec, string newModelFullyQualified, string type2, string path, Module mod
where
newModel(spec, newModelFullyQualified, _, mod, _) and
not exists(FindSubclassesSpec subclass | subclass.getSuperClass() = spec |
// Since a class C which is a subclass for flask.MethodView is always a subclass of
// flask.View, and we chose to care about this distinction, in a naive approach we
// would always record rows for _both_ specs... that's just wasteful, so instead we
// only record the row for the more specific spec -- this is captured by the
// .getSuperClass() method on a spec, which can links specs together in this way.
// However, if the definition actually depends on some logic, like below, we should
// still record both rows
// ```
// if <cond>:
// class C(flask.View): ...
// else:
// class C(flask.MethodView): ...
// ```
newModel(subclass, newModelFullyQualified, _, mod, _)
or
// When defining specs for both foo.Foo and bar.Bar, and you encounter the class
// definition for Bar as `class Bar(foo.Foo): ...` inside `__init__.py` of the `bar`
// PyPI package, we would normally record this new class as being an unmodeled
// subclass of foo.Foo (since the class definition is not found when using
// API::moduleImport("bar").getMember("Bar")). However, we don't actually want to
// treat this as foo.Foo, since it's actually bar.Bar -- so we use the fully
// qualified name ot ignore cases such as this!
newModelFullyQualified = subclass.getFullyQualifiedName()
) and
fullyQualifiedToYamlFormat(newModelFullyQualified, type2, path) and
not Extensions::typeModel(spec, type2, path) and
(
not newModelFullyQualified.regexpMatch("(?i).*tests?_?.*")
or
type2 = "find_subclass_test"
)
select spec.(string), type2, path

36
python/ql/src/meta/ClassHierarchy/join-yml-files.py Executable file
View File

@@ -0,0 +1,36 @@
#!/usr/bin/env python3
"""Concerns were raised about performance on Windows with having 2.5 k files for modeling, and it was recommended we join them all together when shipping.
This script does that.
Workflow when working on the automatic subclass modeling:
1. split files
2. do your work
3. join files
4. commit your changes
"""
import sys
import glob
import os
from shared_subclass_functions import *
if joined_file.exists():
sys.exit(f"File {joined_file} already exists")
package_data = gather_from_existing()
as_lists = list()
for data in package_data.values():
as_lists.extend(list(t) for t in data)
as_lists.sort()
to_write = wrap_in_template(as_lists)
write_data(to_write, joined_file)
print("Joined all files into", joined_file)
for f in glob.glob(f"{subclass_capture_path}/auto-*.model.yml", recursive=True):
os.unlink(f)

79
python/ql/src/meta/ClassHierarchy/process-mrva-results.py Normal file
View File

@@ -0,0 +1,79 @@
#!/usr/bin/env python3
import sys
import glob
import json
import subprocess
from collections import defaultdict
import shutil
import os
from shared_subclass_functions import *
assert mad_path.exists(), mad_path
# process data
class CodeQL:
def __init__(self):
pass
def __enter__(self):
self.proc = subprocess.Popen(['codeql', 'execute','cli-server'],
executable=shutil.which('codeql'),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=sys.stderr,
env=os.environ.copy(),
)
return self
def __exit__(self, type, value, tb):
self.proc.stdin.write(b'["shutdown"]\0')
self.proc.stdin.close()
try:
self.proc.wait(5)
except:
self.proc.kill()
def command(self, args):
data = json.dumps(args)
data_bytes = data.encode('utf-8')
self.proc.stdin.write(data_bytes)
self.proc.stdin.write(b'\0')
self.proc.stdin.flush()
res = b''
while True:
b = self.proc.stdout.read(1)
if b == b'\0':
return res.decode('utf-8')
res += b
def gather_from_bqrs_results():
package_data = defaultdict(set)
with CodeQL() as codeql:
if os.path.exists(sys.argv[1]) and not os.path.isdir(sys.argv[1]) and sys.argv[1].endswith(".bqrs"):
files = [sys.argv[1]]
else:
files = glob.glob(f"{sys.argv[1]}/**.bqrs", recursive=True)
for f in files:
print(f"Processing {f}")
json_data = codeql.command(["bqrs", "decode", "--format=json", f])
select = json.loads(json_data)
for t in select["#select"]["tuples"]:
pkg = t[1]
package_data[pkg].add(tuple(t))
return package_data
if __name__ == "__main__":
if joined_file.exists():
sys.exit(f"File {joined_file} exists, you should split it up first")
package_data = gather_from_bqrs_results()
write_all_package_data_to_files(package_data)

79
python/ql/src/meta/ClassHierarchy/shared_subclass_functions.py Normal file
View File

@@ -0,0 +1,79 @@
from typing import Dict
import yaml
from pathlib import Path
import glob
from collections import defaultdict
import re
VERSION = "process-mrva-results 0.0.1"
mad_path = Path(__file__).parent.parent.parent.parent / "lib/semmle/python/frameworks/data/internal/"
subclass_capture_path = mad_path / "subclass-capture"
joined_file = subclass_capture_path / "ALL.model.yml"
def parse_from_file(path: Path) -> set:
if not path.exists():
return set()
f = path.open("r")
assert f.readline().startswith(f"# {VERSION}\n"), path
raw_data = yaml.load(f, Loader=yaml.CBaseLoader)
assert len(raw_data["extensions"]) == 1, path
assert raw_data["extensions"][0]["addsTo"]["extensible"] == "typeModel", path
return set(tuple(x) for x in raw_data["extensions"][0]["data"])
def wrap_in_template(data):
return {
"extensions": [
{
"addsTo": {
"pack": "codeql/python-all",
"extensible": "typeModel",
},
"data": data,
}
]
}
def write_data(data, path: Path):
f = path.open("w+")
f.write(f"# {VERSION}\n")
yaml.dump(data, indent=2, stream=f, Dumper=yaml.CDumper)
def gather_from_existing():
package_data = defaultdict(set)
for f in glob.glob(f"{subclass_capture_path}/auto-*.model.yml", recursive=True):
print(f"Processing {f}")
all_data = parse_from_file(Path(f))
pkg = f.split("/")[-1].split(".")[0][5:]
package_data[pkg].update(all_data)
return package_data
def write_all_package_data_to_files(package_data: Dict[str, set]):
for pkg in package_data:
if not re.match(r"[a-zA-Z0-9-_]+", pkg):
print(f"Skipping {repr(pkg)}")
continue
pkg_path = subclass_capture_path / f"auto-{pkg}.model.yml"
print(f"Writing {pkg_path}")
all_data = parse_from_file(pkg_path)
all_data.update(package_data[pkg])
as_lists = [list(t) for t in all_data]
as_lists.sort()
data_for_yaml = wrap_in_template(as_lists)
write_data(data_for_yaml, pkg_path)

28
python/ql/src/meta/ClassHierarchy/split-yml-files.py Executable file
View File

@@ -0,0 +1,28 @@
#!/usr/bin/env python3
"""Concerns were raised about performance on Windows with having 2.5 k files for modeling, and it was recommended we join them all together when shipping.
This script does the opposite, so it's easier to work with locally.
Workflow when working on the automatic subclass modeling:
1. split files
2. do your work
3. join files
4. commit your changes
"""
import sys
from collections import defaultdict
from shared_subclass_functions import *
if not joined_file.exists():
sys.exit(f"File {joined_file} does not exists")
all_data = parse_from_file(joined_file)
package_data = defaultdict(set)
for t in all_data:
package_data[t[1]].add(t)
write_all_package_data_to_files(package_data)
joined_file.unlink()

17
python/ql/src/meta/alerts/InterestingTaintSinks.ql Normal file
View File

@@ -0,0 +1,17 @@
/**
* @name Interesting taint sinks
* @description Interesting sinks from TaintTracking queries.
* @kind problem
* @problem.severity recommendation
* @id py/meta/alerts/interesting-taint-sinks
* @tags meta
* @precision very-low
*/
private import python
private import semmle.python.dataflow.new.DataFlow
private import Sinks
from string kind
where not kind in ["CleartextLogging", "LogInjection"]
select taintSink(kind), kind + " sink"

79
python/ql/src/meta/alerts/Sinks.qll Normal file
View File

@@ -0,0 +1,79 @@
private import python
private import semmle.python.dataflow.new.DataFlow
private import meta.MetaMetrics
import semmle.python.security.dataflow.CleartextLoggingCustomizations
import semmle.python.security.dataflow.CleartextStorageCustomizations
import semmle.python.security.dataflow.CodeInjectionCustomizations
import semmle.python.security.dataflow.CommandInjectionCustomizations
import semmle.python.security.dataflow.LdapInjectionCustomizations
import semmle.python.security.dataflow.LogInjectionCustomizations
import semmle.python.security.dataflow.NoSqlInjectionCustomizations
import semmle.python.security.dataflow.PathInjectionCustomizations
import semmle.python.security.dataflow.PolynomialReDoSCustomizations
import semmle.python.security.dataflow.ReflectedXSSCustomizations
import semmle.python.security.dataflow.RegexInjectionCustomizations
import semmle.python.security.dataflow.ServerSideRequestForgeryCustomizations
import semmle.python.security.dataflow.SqlInjectionCustomizations
import semmle.python.security.dataflow.StackTraceExposureCustomizations
import semmle.python.security.dataflow.TarSlipCustomizations
import semmle.python.security.dataflow.UnsafeDeserializationCustomizations
import semmle.python.security.dataflow.UrlRedirectCustomizations
import semmle.python.security.dataflow.WeakSensitiveDataHashingCustomizations
import semmle.python.security.dataflow.XmlBombCustomizations
import semmle.python.security.dataflow.XpathInjectionCustomizations
import semmle.python.security.dataflow.XxeCustomizations
DataFlow::Node taintSink(string kind) {
not result.getLocation().getFile() instanceof IgnoredFile and
(
kind = "CleartextLogging" and result instanceof CleartextLogging::Sink
or
kind = "CleartextStorage" and result instanceof CleartextStorage::Sink
or
kind = "CodeInjection" and result instanceof CodeInjection::Sink
or
kind = "CommandInjection" and result instanceof CommandInjection::Sink
or
kind = "LdapInjection (DN)" and result instanceof LdapInjection::DnSink
or
kind = "LdapInjection (Filter)" and result instanceof LdapInjection::FilterSink
or
kind = "LogInjection" and result instanceof LogInjection::Sink
or
kind = "PathInjection" and result instanceof PathInjection::Sink
or
kind = "PolynomialReDoS" and result instanceof PolynomialReDoS::Sink
or
kind = "ReflectedXss" and result instanceof ReflectedXss::Sink
or
kind = "RegexInjection" and result instanceof RegexInjection::Sink
or
kind = "NoSqlInjection (string sink)" and result instanceof NoSqlInjection::StringSink
or
kind = "NoSqlInjection (dict sink)" and result instanceof NoSqlInjection::DictSink
or
kind = "ServerSideRequestForgery" and result instanceof ServerSideRequestForgery::Sink
or
kind = "SqlInjection" and result instanceof SqlInjection::Sink
or
kind = "StackTraceExposure" and result instanceof StackTraceExposure::Sink
or
kind = "TarSlip" and result instanceof TarSlip::Sink
or
kind = "UnsafeDeserialization" and result instanceof UnsafeDeserialization::Sink
or
kind = "UrlRedirect" and result instanceof UrlRedirect::Sink
or
kind = "WeakSensitiveDataHashing (NormalHashFunction)" and
result instanceof NormalHashFunction::Sink
or
kind = "WeakSensitiveDataHashing (ComputationallyExpensiveHashFunction)" and
result instanceof ComputationallyExpensiveHashFunction::Sink
or
kind = "XmlBomb" and result instanceof XmlBomb::Sink
or
kind = "XpathInjection" and result instanceof XpathInjection::Sink
or
kind = "Xxe" and result instanceof Xxe::Sink
)
}

80
python/ql/src/meta/alerts/TaintSinks.ql
View File

@@ -10,83 +10,7 @@
private import python
private import semmle.python.dataflow.new.DataFlow
private import meta.MetaMetrics
import semmle.python.security.dataflow.CleartextLoggingCustomizations
import semmle.python.security.dataflow.CleartextStorageCustomizations
import semmle.python.security.dataflow.CodeInjectionCustomizations
import semmle.python.security.dataflow.CommandInjectionCustomizations
import semmle.python.security.dataflow.LdapInjectionCustomizations
import semmle.python.security.dataflow.LogInjectionCustomizations
import semmle.python.security.dataflow.NoSqlInjectionCustomizations
import semmle.python.security.dataflow.PathInjectionCustomizations
import semmle.python.security.dataflow.PolynomialReDoSCustomizations
import semmle.python.security.dataflow.ReflectedXSSCustomizations
import semmle.python.security.dataflow.RegexInjectionCustomizations
import semmle.python.security.dataflow.ServerSideRequestForgeryCustomizations
import semmle.python.security.dataflow.SqlInjectionCustomizations
import semmle.python.security.dataflow.StackTraceExposureCustomizations
import semmle.python.security.dataflow.TarSlipCustomizations
import semmle.python.security.dataflow.UnsafeDeserializationCustomizations
import semmle.python.security.dataflow.UrlRedirectCustomizations
import semmle.python.security.dataflow.WeakSensitiveDataHashingCustomizations
import semmle.python.security.dataflow.XmlBombCustomizations
import semmle.python.security.dataflow.XpathInjectionCustomizations
import semmle.python.security.dataflow.XxeCustomizations
DataFlow::Node relevantTaintSink(string kind) {
not result.getLocation().getFile() instanceof IgnoredFile and
(
kind = "CleartextLogging" and result instanceof CleartextLogging::Sink
or
kind = "CleartextStorage" and result instanceof CleartextStorage::Sink
or
kind = "CodeInjection" and result instanceof CodeInjection::Sink
or
kind = "CommandInjection" and result instanceof CommandInjection::Sink
or
kind = "LdapInjection (DN)" and result instanceof LdapInjection::DnSink
or
kind = "LdapInjection (Filter)" and result instanceof LdapInjection::FilterSink
or
kind = "LogInjection" and result instanceof LogInjection::Sink
or
kind = "PathInjection" and result instanceof PathInjection::Sink
or
kind = "PolynomialReDoS" and result instanceof PolynomialReDoS::Sink
or
kind = "ReflectedXss" and result instanceof ReflectedXss::Sink
or
kind = "RegexInjection" and result instanceof RegexInjection::Sink
or
kind = "NoSqlInjection (string sink)" and result instanceof NoSqlInjection::StringSink
or
kind = "NoSqlInjection (dict sink)" and result instanceof NoSqlInjection::DictSink
or
kind = "ServerSideRequestForgery" and result instanceof ServerSideRequestForgery::Sink
or
kind = "SqlInjection" and result instanceof SqlInjection::Sink
or
kind = "StackTraceExposure" and result instanceof StackTraceExposure::Sink
or
kind = "TarSlip" and result instanceof TarSlip::Sink
or
kind = "UnsafeDeserialization" and result instanceof UnsafeDeserialization::Sink
or
kind = "UrlRedirect" and result instanceof UrlRedirect::Sink
or
kind = "WeakSensitiveDataHashing (NormalHashFunction)" and
result instanceof NormalHashFunction::Sink
or
kind = "WeakSensitiveDataHashing (ComputationallyExpensiveHashFunction)" and
result instanceof ComputationallyExpensiveHashFunction::Sink
or
kind = "XmlBomb" and result instanceof XmlBomb::Sink
or
kind = "XpathInjection" and result instanceof XpathInjection::Sink
or
kind = "Xxe" and result instanceof Xxe::Sink
)
}
private import Sinks
from string kind
select relevantTaintSink(kind), kind + " sink"
select taintSink(kind), kind + " sink"

2
python/ql/src/qlpack.yml
View File

@@ -1,5 +1,5 @@
name: codeql/python-queries
version: 0.9.6-dev
version: 0.9.7-dev
groups:
- python
- queries

17
python/ql/test/experimental/library-tests/FindSubclass/Find.expected Normal file
View File

@@ -0,0 +1,17 @@
| Django.Views.View~Subclass | find_subclass_test | Member[clash2] |
| Django.Views.View~Subclass | find_subclass_test | Member[clash3] |
| Django.Views.View~Subclass | find_subclass_test | Member[clash] |
| flask.MethodView~Subclass | find_subclass_test | Member[C] |
| flask.View~Subclass | find_subclass_test | Member[A] |
| flask.View~Subclass | find_subclass_test | Member[B] |
| flask.View~Subclass | find_subclass_test | Member[ViewAliasInExcept] |
| flask.View~Subclass | find_subclass_test | Member[ViewAliasInTry] |
| flask.View~Subclass | find_subclass_test | Member[ViewAlias] |
| flask.View~Subclass | find_subclass_test | Member[ViewAlias_no_use] |
| flask.View~Subclass | find_subclass_test | Member[View] |
| flask.View~Subclass | find_subclass_test | Member[clash2] |
| flask.View~Subclass | find_subclass_test | Member[clash3] |
| flask.View~Subclass | find_subclass_test | Member[clash] |
| flask.View~Subclass | find_subclass_test | Member[complete_module_alias] |
| flask.View~Subclass | find_subclass_test | Member[complete_module_alias_no_use] |
| rest_framework.response.Response~Subclass | find_subclass_test | Member[MyRestResponse] |

1
python/ql/test/experimental/library-tests/FindSubclass/Find.qlref Normal file
View File

@@ -0,0 +1 @@
meta/ClassHierarchy/Find.ql

57
python/ql/test/experimental/library-tests/FindSubclass/find_subclass_test.py Normal file
View File

@@ -0,0 +1,57 @@
from flask.views import View
import flask.views
class A(View):
pass
class B(A):
pass
class C(flask.views.MethodView):
pass
ViewAlias = View
print(ViewAlias)
ViewAlias_no_use = View
try:
from flask.views import View as ViewAliasInTry
except:
from flask.views import View as ViewAliasInExcept
if cond:
from flask.views import View as clash
else:
from django.views.generic import View as clash
if cond:
from flask.views import View as clash2
else:
from django.views.generic import View as clash2
print(clash2)
if cond:
from flask.views import View as clash3
else:
from django.views.generic import View as clash3
print(clash3)
import flask.views as containing_module_alias # $ MISSING
# now `find_subclass_test.containing_module_alias.View` is an alias of flask.views.View
# NOTE: this is not valid code, since View is not a module... but it could be in some cases, like for xml.etree.ElementTree, which is actually not a class but a module 😕
import flask.views.View as complete_module_alias
print(complete_module_alias)
import flask.views.View as complete_module_alias_no_use
def wrapper():
return View # $ MISSING
import rest_framework
class MyRestResponse(rest_framework.response.Response):
pass

1
python/ql/test/experimental/library-tests/FindSubclass/options Normal file
View File

@@ -0,0 +1 @@
semmle-extractor-options: --max-import-depth=0

2
python/ql/test/library-tests/frameworks/rest_framework/testapp/urls.py
View File

@@ -15,5 +15,5 @@ urlpatterns = [
path("function-based-view/", views.function_based_view), # $ routeSetup="function-based-view/"
path("cookie-test/", views.cookie_test), # $ routeSetup="cookie-test/"
path("exception-test/", views.exception_test), # $ routeSetup="exception-test/"
path("viewset-entrypoints-test/", views.EntrypointViewSet.as_view()) # $ routeSetup="viewset-entrypoints-test/"
path("viewset-entrypoints-test/", views.EntrypointViewSet.as_view({"get": "list"})) # $ routeSetup="viewset-entrypoints-test/"
]