Move older change notes to old-change-notes

Now that change notes are per-package, new change notes should be created in the `change-notes` folder under the affected pack (e.g., `cpp/ql/src/change-notes` for C++ query change notes. I've moved all of the change note files that were added before we started publishing them in packs to an `old-change-notes` directory under each language, to reduce the temptation to add new change notes there.

I'm working on a document to describe how and when to create change notes for packs separately.

python/old-change-notes/2020-11-25-better-open-models.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Modeling of file system access has been improved to recognize `io.open` and `builtins.open`.

python/old-change-notes/2020-12-03-model-realpath-abspath.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of `os.path.abspath` and `os.path.realpath` for Path Injection (py/path-injection).

python/old-change-notes/2020-12-04-django-class-based-view-handlers.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of django class based view handlers (subclasses of `django.views.generic.View`) as sources of remote user input (`RemoteFlowSource`).

python/old-change-notes/2020-12-08-stdlib-http-source-modeling.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Added modeling of HTTP servers created with `BaseHTTPRequestHandler` from standard library as a source of remote user input (`RemoteFlowSource`).
+* Added modeling of HTML form submission with `cgi.FieldStorage` from standard library as a source of remote user input (`RemoteFlowSource`).

python/old-change-notes/2020-12-09-add-sqlite3-model.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added model of `sqlite3` as SQL interface following PEP249, resulting in additional sinks for `py/sql-injection`.

python/old-change-notes/2020-12-14-add-PyMySQL-model.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added model of `PyMySQL` PyPI package as a SQL interface following PEP249, resulting in additional sinks for `py/sql-injection`.

python/old-change-notes/2020-12-21-django-with-unknown-route.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of `django` to recognize request handlers on `View` classes without known route, thereby leading to more sources of remote user input (`RemoteFlowSource`).

python/old-change-notes/2020-12-22-tornado-source-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of sources of remote user input (`RemoteFlowSource`) when using `tornado` to create HTTP servers, to the new data-flow queries.

python/old-change-notes/2021-01-12-flask-class-based-view-handlers.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of flask class based view handlers (subclasses of `flask.views.View` and `flask.views.MethodView`). This means we're now able to detect routed parameters for request handler defined on these classes, as sources of remote user input (`RemoteFlowSource`).

python/old-change-notes/2021-01-19-port-url-redirect-query.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Ported URL redirection (`py/url-redirection`) query to use new data-flow library. This might result in different results, but overall a more robust and accurate analysis.

python/old-change-notes/2021-02-02-port-weak-crypto-key-query.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Updated _Use of weak cryptographic key_ (`py/weak-crypto-key`) query to use the new type-tracking approach instead of points-to analysis. You may see differences in the results found by the query, but overall this change should result in a more robust and accurate analysis.
+* Renamed the query file for _Use of weak cryptographic key_ (`py/weak-crypto-key`) from `WeakCrypto.ql` to `WeakCryptoKey.ql` (in the `python/ql/src/Security/CWE-326/` folder). This will affect any custom query suites that include or exclude this query using its path.

python/old-change-notes/2021-02-03-flask-add-blueprint-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of flask blueprints (`flask.Blueprint`), specifically request handlers defined with such blueprints. This can result in new sources of remote user input (`RemoteFlowSource`) -- since we're now able to detect routed parameters -- and new XSS sinks from the responses of these request handlers.

python/old-change-notes/2021-02-04-api-graphs.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added support for API graphs. Data-flow nodes referring to an external API component such as `flask.views.View` can now be found using `API::moduleImport("flask").getMember("views").getMember("View").getAUse()` when the `semmle.python.ApiGraphs` module has been imported.

python/old-change-notes/2021-02-10-django-improvements.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* Improved modeling of `django` to recognize request handlers functions that are decorated (for example with `django.views.decorators.http.require_GET`). This leads to more sources of remote user input (`RemoteFlowSource`), since we correctly identify the first parameter as being passed a django request.
+* Improved modeling of django View classes. We now consider any class using in a routing setup with `<class>.as_view()` as django view class. This leads to more sources of remote user input (`RemoteFlowSource`), since we correctly identify the first parameter as being passed a django request.
+* Improved modeling of `django`, so for View classes we now model `self.request`, `self.args`, and `self.kwargs` as sources of remote user input (`RemoteFlowSource`).

python/old-change-notes/2021-02-10-yaml-more-loading-functions.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of the `PyYAML` PyPI package (imported as `yaml`) now includes `safe_load`, `unsafe_load`, and `full_load` (as well as the `..._load_all` functions). In the current version of PyYAML (5.4.1), only `safe_load` and `safe_load_all` are known to be safe from code execution exploits. Consequently, calls to the other functions are modeled as sinks of the _Deserializing untrusted input_ (`py/unsafe-deserialization`) query.

python/old-change-notes/2021-02-12-django-get_redirect_url.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of `django` to recognize request redirects from `get_redirect_url` on a `RedirectView` subclass.

python/old-change-notes/2021-02-18-type-backtrackers.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added support for type backtracking. Dual to the `TypeTracker` class, the `TypeBackTracker` class allows the propagation of user-defined type information backwards from a set of data flow nodes.

python/old-change-notes/2021-02-23-port-bind-to-all-interfaces.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Updated _Binding a socket to all network interfaces_ (`py/bind-socket-all-network-interfaces`) query to use the new type-tracking approach instead of points-to analysis. You may see differences in the results found by the query, but overall this change should result in a more robust and accurate analysis.
+* Updated _Binding a socket to all network interfaces_ (`py/bind-socket-all-network-interfaces`) to recognize binding to all interfaces in IPv6 with hostnames `::` and `::0`

python/old-change-notes/2021-02-23-port-insecure-default-protocol.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Updated the query that detects insecure SSL/TLS protocol creation from default values (`py/insecure-default-protocol`) to use the new API graphs. The query also no longer reports use of the default value for constructing `ssl.SSLContext`, since that _can_ still be secure, either through manipulation of the `options` field or the `minimum_version` field. If the usage is not secure, this should be reported by the `py/insecure-protocol` query.

python/old-change-notes/2021-02-24-port-flask-debug.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Updated _Flask app is run in debug mode_ (`py/flask-debug`) query to use the new type-tracking approach instead of points-to analysis. You may see differences in the results found by the query, but overall this change should result in a more robust and accurate analysis.

python/old-change-notes/2021-02-25-port-stactrace-exposure-query.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Updated _Information exposure through an exception_ (`py/stack-trace-exposure`) query to use the new data-flow library and type-tracking approach instead of points-to analysis. You may see differences in the results found by the query, but overall this change should result in a more robust and accurate analysis.

python/old-change-notes/2021-03-01-fluent-interface-data-flow.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The data-flow library now recognises more side-effects of method chaining (e.g. `someObject.setX(clean).setY(tainted).setZ...` having a side-effect on `someObject`), as well as other related circumstances where a function input is directly passed to its output. All queries that use data-flow analysis, including most security queries, may return more results accordingly.

python/old-change-notes/2021-03-11-api-graph-builtins.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* API graphs now contain nodes for built-in functions and classes. For instance, `API::builtin("open")` is the API graph node corresponding to the built-in `open` function.

python/old-change-notes/2021-03-12-small-api-enhancements.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* The class ParameterNode now extends LocalSourceNode, thus making methods like flowsTo available.
+* The new predicate `parameterNode` can now be used to map from a `Parameter` to a data-flow node.

python/old-change-notes/2021-03-15-port-insecure-protocol.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Ported use of insecure SSL/TLS version (`py/insecure-protocol`) query to use new data-flow library. This might result in different results, but overall a more robust and accurate analysis.

python/old-change-notes/2021-03-18-yaml-handle-C-based-loaders.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of the `PyYAML` PyPI package, so we now correctly treat `CSafeLoader` and `CBaseLoader` as being safe loaders that can not lead to code execution.

python/old-change-notes/2021-03-22-django-queryset-chains.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of `django` to recognize QuerySet chains such as `User.objects.using("db-name").exclude(username="admin").extra("some sql")`. This can lead to new results for `py/sql-injection`.

python/old-change-notes/2021-03-22-getacall-callcfgnode.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The `API::Node::getACall` method now has the more specific return type `DataFlow::CallCfgNode`, which improves the ease of use when working with calls to API functions.

python/old-change-notes/2021-03-23-django-forms-fields-classes.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of `django` to recognize sources of remote user input (`RemoteFlowSource`) in Django forms (`django.forms.Form`) and fields (`django.forms.Field`) subclasses.

python/old-change-notes/2021-03-25-remove-legacy.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* The legacy code duplication library has been removed.
+* Legacy filter queries have been removed.

python/old-change-notes/2021-04-09-split-weak-crypto-query.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Updated the _Use of a broken or weak cryptographic algorithm_ (`py/weak-cryptographic-algorithm`) query, so it alerts on any use of a weak cryptographic non-hashing algorithm. Introduced a new query _Use of a broken or weak cryptographic hashing algorithm on sensitive data_ (`py/weak-sensitive-data-hashing`) to handle weak cryptographic hashing algorithms, which only alerts when used on sensitive data.

python/old-change-notes/2021-04-13-pep249-api-graphs.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* Modeling of libraries supporting PEP249 has been changed to use API graphs. When defining new
+  models, the relevant extension point is now `PEP249ModuleApiNode` in the `PEP249` module, instead
+  of `PEP249Module`. The latter class has now been deprecated.

python/old-change-notes/2021-04-13-werkzeug-api-graphs.md

@@ -0,0 +1,5 @@
+lgtm,codescanning
+* The Werkzeug model has been changed to use API graphs. When defining new models for classes based
+  on the `MultiDict` and `FileStorage` classes in `werkzeug.datastructures`, the relevant extension
+  points are now the two `InstanceSourceApiNode` classes in the `semmle.python.frameworks.Werkzeug`
+  module, instead of `InstanceSource`. The latter classes have now been deprecated.

python/old-change-notes/2021-04-15-pathlib-Paths.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of `pathlib` from the standard library to recognize `Path` objects constructed in various ways and resulting file accesses. This can lead to new results for `py/path-injection`.

python/old-change-notes/2021-04-20-stepsummary-localsourcenode.md

@@ -0,0 +1,7 @@
+lgtm,codescanning
+* The predicates `StepSummary::step` and `TypeTracker::step` in `TypeTracker.qll` have been changed
+  to use the more restrictive type `LocalSourceNode` for their second argument. For cases where
+  stepping between non-`LocalSourceNode`s is required, the `StepSummary::smallstep` predicate may be
+  used instead.
+* The methods `Node::track` and `Node::backtrack` have been moved to the class `LocalSourceNode`. If
+  the old behavior is required, one can use `LocalSourceNode::flowsTo` to add back the missing flow.

python/old-change-notes/2021-04-21-django-v3.2.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Modeling of Django has been updated to handle new 3.2 release, by supporting the new `QuerySet.alias()` method, which can be a sink for SQL injection.

python/old-change-notes/2021-05-10-idna-add-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the PyPI package `idna`, for encoding/decoding Internationalised Domain Names in Applications.

python/old-change-notes/2021-05-10-simplejson-add-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the PyPI package `simplejson`.

python/old-change-notes/2021-05-10-ujson-add-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the PyPI package `ujson`.

python/old-change-notes/2021-05-21-api-graph-await.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* API graph nodes now contain a `getAwaited()` member predicate, for getting the result of awaiting an item, such as `await foo`.

python/old-change-notes/2021-05-25-add-ClickHouse-sql-libs.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added model of SQL execution in `clickhouse-driver` and `aioch` PyPI packages, resulting in additional sinks for the SQL Injection query (`py/sql-injection`). This modeling was originally [submitted as a contribution by @japroc](https://github.com/github/codeql/pull/5889).

python/old-change-notes/2021-06-03-aiohttp-webserver-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of sources/sinks when using the `aiohttp.web` web framework.

python/old-change-notes/2021-06-04-sensitive-data-modeling-expanded.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Expanded modeling of sensitive data sources to include: subscripting with a key that indicates sensitive data (`obj["password"]`), parameters whose names indicate sensitive data (`def func(password):`), and assignments to variables whose names indicate sensitive data (`password = ...`).

python/old-change-notes/2021-06-08-twisted-add-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of sources/sinks when using `twisted` to create web servers.

python/old-change-notes/2021-06-09-add-jmespath-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the PyPI package `jmespath`.

python/old-change-notes/2021-06-09-rsa-add-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the PyPI package `rsa`.

python/old-change-notes/2021-06-15-add-method-call-conveniences.md

@@ -0,0 +1,5 @@
+lgtm,codescanning
+* A new class `DataFlow::MethodCallNode` extends `DataFlow::CallCfgNode` with convenient methods for
+  accessing the receiver and method name of a method call.
+* The `LocalSourceNode` class now has a `getAMethodCall` method, with which one can easily access
+  method calls with the given node as a receiver.

python/old-change-notes/2021-06-16-MarkupSafe-add-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the PyPI package `MarkupSafe`.

python/old-change-notes/2021-06-24-add-CookieWrite-concept.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added `HTTP::Server::CookieWrite` concept for statements that sets a cookie in an HTTP response, along with modeling of this in supported web frameworks (aiohttp/flask/django/tornado/twisted).

python/old-change-notes/2021-06-24-dataflow-implicit-reads.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The DataFlow libraries have been augmented with support for `Configuration`-specific in-place read steps at, for example, sinks and custom taint steps. This means that it is now possible to specify sinks that accept flow with non-empty access paths.

python/old-change-notes/2021-06-25-add-peewee-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of raw SQL execution from the PyPI package `peewee`.

python/old-change-notes/2021-07-12-add-typetrackingnode.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The `track` and `backtrack` methods on `LocalSourceNode` are in the process of being deprecated. When using type trackers, the corresponding methods on `TypeTrackingNode` should be used instead.

python/old-change-notes/2021-07-13-path-problem-customization.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Changed the way to provide extra sources/sinks for `@kind path-problem` queries, to avoid a potential performance problem due to re-evaluation of data-flow configurations. Please use the new `<query>Customization.qll` files and extend their classes instead (such as extending the `Sink` class from `python/ql/src/semmle/python/security/dataflow/SqlInjectionCustomizations.qll`). This is relevant for the queries: `py/sql-injection`, `py/code-injection`, `py/command-line-injection`, `py/reflective-xss`, `py/url-redirection`, `py/unsafe-deserialization`, `py/stack-trace-exposure`, `py/path-injection`.

python/old-change-notes/2021-07-16-deprecate-importnode.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* The `importNode` predicate from the data-flow library has been deprecated. In its place, we
+  recommend using the API graphs library, accessible via `import semmle.python.ApiGraphs`.
+  

python/old-change-notes/2021-07-28-port-RoDoS-queries.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Added an experimental _Inefficient regular expression_ (`py/redos`) query, which is already available in JavaScript.
+* Added an experimental _Polynomial regular expression used on uncontrolled data_ (`py/polynomial-redos`), which is already available in JavaScript.

python/old-change-notes/2021-08-26-bad-tag-filter.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* A new query, `py/bad-tag-filter`, has been added to the query suite,
+  highlighting regular expressions that only match a subset of the HTML tags
+  it is supposed to match.

python/old-change-notes/2021-08-30-port-modifying-default-query.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Updated _Modification of parameter with default_ (`py/modification-of-default-value`) query to use the new data flow library instead of the old taint tracking library and to remove the use of points-to analysis. You may see differences in the results found by the query, but overall this change should result in a more robust and accurate analysis.

python/old-change-notes/2021-09-02-add-Flask-SQLAlchemy-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of SQL execution in the `Flask-SQLAlchemy` PyPI package, resulting in additional sinks for the SQL Injection query (`py/sql-injection`).

python/old-change-notes/2021-09-02-add-SQLAlchemy-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of SQL execution in the `SQLAlchemy` PyPI package, resulting in additional sinks for the SQL Injection query (`py/sql-injection`). This modeling was originally [submitted as a contribution by @mrthankyou](https://github.com/github/codeql/pull/5680).

python/old-change-notes/2021-09-02-add-SQLAlchemyTextClauseInjection.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Expanded the query _SQL query built from user-controlled sources_ (`py/sql-injection`) to alert if user-input is added to a TextClause from SQLAlchemy, since that can lead to SQL injection.

python/old-change-notes/2021-09-08-add-flow-from-default-values.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Function parameters with default values will now see flow from those values.

python/old-change-notes/2021-09-14-promote-regex-injection.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The query "Regular expression injection" (`py/regex-injection`) has been promoted from experimental to the main query pack. Its results will now appear by default. This query was originally [submitted as an experimental query by @jorgectf](https://github.com/github/codeql/pull/5442).

python/old-change-notes/2021-09-29-model-asyncpg.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Added modeling of `asyncpg` for sinks executing SQL and/or accessing the file system.
+* Corrected the API graph, such that all awaited values now are referred to via `getAwaited`.

python/old-change-notes/2021-10-08-add-dataflow-for-boolean-expressions.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Added data-flow from both `x` and `y` to `x or y` and `x and y`, as a slight over-approximation of what is described in the
+  [Python Language Reference](https://docs.python.org/3/reference/expressions.html#boolean-operations).

python/old-change-notes/2021-10-08-improve-pickle-dill-shelve-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Improved modeling of decoding through pickle related functions (which can lead to code execution), resulting in additional sinks for the _Deserializing untrusted input_ query (`py/unsafe-deserialization`). Now we fully support `pickle.load`, `pickle.loads`, `pickle.Unpickler`, `marshal.load`, `marshal.loads`, `dill.load`, `dill.loads`, `shelve.open`.

python/old-change-notes/2021-10-11-model-aiomysql.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of `aiomysql` for sinks executing SQL

python/old-change-notes/2021-10-20-extraction-errors-as-warnings.md

@@ -0,0 +1,2 @@
+codescanning
+* Problems with extraction that in most cases won't completely break the analysis are now reported as warnings rather than errors.

python/old-change-notes/2021-10-25-add-FastAPI-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of sources/sinks when using FastAPI to create web servers.

python/old-change-notes/2021-10-26-ruamel.yaml-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the `ruamel.yaml` PyPI package, resulting in additional sinks for the _Deserializing untrusted input_ (`py/unsafe-deserialization`) query (since `ruamel.yaml.load` can lead to code execution).

python/old-change-notes/2021-10-28-flask-send_file.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of the `send_from_directory` and `send_file` functions from the `flask` PyPI package, resulting in additional sinks for the _Uncontrolled data used in path expression_ (`py/path-injection`) query. This addition was originally [submitted as an external contribution by @porcupineyhairs](https://github.com/github/codeql/pull/6330).

python/old-change-notes/2021-10-28-promote-ReDoS-queries.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* The query "Inefficient regular expression" (`py/redos`) has been promoted from experimental to the main query pack. Its results will now appear by default.
+* The query "Polynomial regular expression used on uncontrolled data" (`py/polynomial-redos`) has been promoted from experimental to the main query pack. Its results will now appear by default.

python/old-change-notes/2021-10-29-django-REST-framework-modeling.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Added modeling of HTTP requests and responses when using the Django REST Framework (`djangorestframework` PyPI package), which leads to additional remote flow sources.