Merge pull request #22109 from github/release-prep/2.26.0

Release preparation for version 2.26.0

python/ql/lib/CHANGELOG.md

@@ -70,7 +70,7 @@ No user-facing changes.
 
 ### Minor Analysis Improvements
 
-* Added new full SSRF sanitization barrier from the new AntiSSRF library. 
+* Added new full SSRF sanitization barrier from the new AntiSSRF library.
 * When a guard such as `isSafe(x)` is defined, we now also automatically handle `isSafe(x) == true` and `isSafe(x) != false`.
 
 ## 6.1.1
@@ -169,7 +169,7 @@ No user-facing changes.
 ### Minor Analysis Improvements
 
 - The modelling of Psycopg2 now supports the use of `psycopg2.pool` connection pools for handling database connections.
-* Removed `lxml` as an XML bomb sink. The underlying libxml2 library now includes [entity reference loop detection](https://github.com/lxml/lxml/blob/f33ac2c2f5f9c4c4c1fc47f363be96db308f2fa6/doc/FAQ.txt#L1077) that prevents XML bomb attacks. 
+* Removed `lxml` as an XML bomb sink. The underlying libxml2 library now includes [entity reference loop detection](https://github.com/lxml/lxml/blob/f33ac2c2f5f9c4c4c1fc47f363be96db308f2fa6/doc/FAQ.txt#L1077) that prevents XML bomb attacks.
 
 ## 4.0.13
 
@@ -262,7 +262,7 @@ No user-facing changes.
 ### Minor Analysis Improvements
 
 * The sensitive data library has been improved so that `snake_case` style variable names are recognized more reliably. This may result in more sensitive data being identified, and more results from queries that use the sensitive data library.
-- Additional taint steps through methods of `lxml.etree.Element` and `lxml.etree.ElementTree` objects from the `lxml` PyPI package have been modeled. 
+- Additional taint steps through methods of `lxml.etree.Element` and `lxml.etree.ElementTree` objects from the `lxml` PyPI package have been modeled.
 
 ## 3.1.0
 
@@ -316,7 +316,7 @@ No user-facing changes.
 
 ### Minor Analysis Improvements
 
-* The common sanitizer guard `StringConstCompareBarrier` has been renamed to `ConstCompareBarrier` and expanded to cover comparisons with other constant values such as `None`. This may result in fewer false positive results for several queries. 
+* The common sanitizer guard `StringConstCompareBarrier` has been renamed to `ConstCompareBarrier` and expanded to cover comparisons with other constant values such as `None`. This may result in fewer false positive results for several queries.
 
 ## 2.0.0
 
@@ -545,7 +545,7 @@ No user-facing changes.
 
 ### New Features
 
-* The `DataFlow::StateConfigSig` signature module has gained default implementations for `isBarrier/2` and `isAdditionalFlowStep/4`. 
+* The `DataFlow::StateConfigSig` signature module has gained default implementations for `isBarrier/2` and `isAdditionalFlowStep/4`.
   Hence it is no longer needed to provide `none()` implementations of these predicates if they are not needed.
 
 ### Minor Analysis Improvements
@@ -572,7 +572,7 @@ No user-facing changes.
 * Deleted many deprecated predicates and classes with uppercase `API`, `HTTP`, `XSS`, `SQL`, etc. in their names. Use the PascalCased versions instead.
 * Deleted the deprecated `getName()` predicate from the `Container` class, use `getAbsolutePath()` instead.
 * Deleted many deprecated module names that started with a lowercase letter, use the versions that start with an uppercase letter instead.
-* Deleted many deprecated predicates in `PointsTo.qll`. 
+* Deleted many deprecated predicates in `PointsTo.qll`.
 * Deleted many deprecated files from the `semmle.python.security` package.
 * Deleted the deprecated `BottleRoutePointToExtension` class from `Extensions.qll`.
 * Type tracking is now aware of flow summaries. This leads to a richer API graph, and may lead to more results in some queries.
@@ -729,7 +729,7 @@ No user-facing changes.
 ### Deprecated APIs
 
 * Some unused predicates in `SsaDefinitions.qll`, `TObject.qll`, `protocols.qll`, and the `pointsto/` folder have been deprecated.
-* Some classes/modules with upper-case acronyms in their name have been renamed to follow our style-guide. 
+* Some classes/modules with upper-case acronyms in their name have been renamed to follow our style-guide.
   The old name still exists as a deprecated alias.
 
 ### Minor Analysis Improvements
@@ -748,9 +748,9 @@ No user-facing changes.
 
 ### Deprecated APIs
 
-* Many classes/predicates/modules with upper-case acronyms in their name have been renamed to follow our style-guide. 
+* Many classes/predicates/modules with upper-case acronyms in their name have been renamed to follow our style-guide.
   The old name still exists as a deprecated alias.
-* The utility files previously in the `semmle.python.security.performance` package have been moved to the `semmle.python.security.regexp` package.  
+* The utility files previously in the `semmle.python.security.performance` package have been moved to the `semmle.python.security.regexp` package.
   The previous files still exist as deprecated aliases.
 
 ### Minor Analysis Improvements
@@ -843,9 +843,9 @@ No user-facing changes.
 
 ### Deprecated APIs
 
-* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide. 
+* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide.
   The old name still exists as a deprecated alias.
-* Some modules that started with a lowercase letter have been renamed to follow our style-guide. 
+* Some modules that started with a lowercase letter have been renamed to follow our style-guide.
   The old name still exists as a deprecated alias.
 
 ### New Features

python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll

@@ -1138,7 +1138,9 @@ predicate clearsContent(Node n, ContentSet cs) {
  * Holds if the value that is being tracked is expected to be stored inside content `c`
  * at node `n`.
  */
-predicate expectsContent(Node n, ContentSet c) { none() }
+predicate expectsContent(Node n, ContentSet c) {
+  FlowSummaryImpl::Private::Steps::summaryExpectsContent(n.(FlowSummaryNode).getSummaryNode(), c)
+}
 
 /**
  * Holds if values stored inside attribute `c` are cleared at node `n`.

python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImpl.qll

@@ -91,6 +91,8 @@ module Input implements InputSig<Location, DataFlowImplSpecific::PythonDataFlow>
     cs.isAnyTupleOrDictionaryElement() and result = "AnyTupleOrDictionaryElement" and arg = ""
   }
 
+  string encodeWithContent(ContentSet c, string arg) { result = "With" + encodeContent(c, arg) }
+
   bindingset[token]
   ParameterPosition decodeUnknownParameterPosition(AccessPath::AccessPathTokenBase token) {
     // needed to support `Argument[x..y]` ranges

python/ql/lib/semmle/python/dataflow/new/internal/TypeTrackingImpl.qll

@@ -170,7 +170,13 @@ module TypeTrackingInput implements Shared::TypeTrackingInput<Location> {
 
   /** Holds if there is a level step from `nodeFrom` to `nodeTo`, which may depend on the call graph. */
   predicate levelStepCall(Node nodeFrom, LocalSourceNode nodeTo) {
-    instanceFieldStep(nodeFrom, nodeTo)
+    // HOTFIX: `instanceFieldStep` is temporarily disabled (via `and none()`).
+    // It uses `classInstanceTracker(cls)` -- itself a type-tracker run --
+    // from inside `levelStepCall`, creating a structural mutual recursion
+    // that causes catastrophic query slowdowns on some OOP-heavy Python
+    // codebases (e.g. mypy and dask). The `and none()` should be removed
+    // once that recursion is redesigned.
+    instanceFieldStep(nodeFrom, nodeTo) and none()
     or
     inheritedFieldStep(nodeFrom, nodeTo)
   }

python/ql/lib/semmle/python/frameworks/Stdlib.qll

@@ -4199,11 +4199,9 @@ module StdlibPrivate {
       // The positional argument contains a mapping.
       // TODO: these values can be overwritten by keyword arguments
       //  - dict mapping
-      exists(DataFlow::DictionaryElementContent dc, string key | key = dc.getKey() |
-        input = "Argument[0].DictionaryElement[" + key + "]" and
-        output = "ReturnValue.DictionaryElement[" + key + "]" and
-        preservesValue = true
-      )
+      input = "Argument[0].WithAnyDictionaryElement" and
+      output = "ReturnValue" and
+      preservesValue = true
       or
       //  - list-of-pairs mapping
       input = "Argument[0].ListElement.TupleElement[1]" and
@@ -4240,9 +4238,7 @@ module StdlibPrivate {
         or
         input = "Argument[0].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[0].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[0].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       // Element content is mutated into list element content
@@ -4266,11 +4262,9 @@ module StdlibPrivate {
     }
 
     override predicate propagatesFlow(string input, string output, boolean preservesValue) {
-      exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-        input = "Argument[0].TupleElement[" + i.toString() + "]" and
-        output = "ReturnValue.TupleElement[" + i.toString() + "]" and
-        preservesValue = true
-      )
+      input = "Argument[0].WithAnyTupleElement" and
+      output = "ReturnValue" and
+      preservesValue = true
       or
       input = "Argument[0].ListElement" and
       output = "ReturnValue" and
@@ -4294,9 +4288,7 @@ module StdlibPrivate {
         or
         input = "Argument[0].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[0].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[0].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       output = "ReturnValue.SetElement" and
@@ -4342,9 +4334,7 @@ module StdlibPrivate {
         or
         input = "Argument[0].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[0].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[0].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       output = "ReturnValue.ListElement" and
@@ -4372,9 +4362,7 @@ module StdlibPrivate {
         or
         content = "SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          content = "TupleElement[" + i.toString() + "]"
-        )
+        content = "AnyTupleElement"
       |
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
         input = "Argument[0]." + content and
@@ -4404,9 +4392,7 @@ module StdlibPrivate {
         or
         input = "Argument[0].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[0].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[0].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       output = "ReturnValue.ListElement" and
@@ -4434,9 +4420,7 @@ module StdlibPrivate {
         or
         input = "Argument[0].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[0].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[0].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       output = "ReturnValue" and
@@ -4468,9 +4452,7 @@ module StdlibPrivate {
           // We reduce generality slightly by not tracking tuple contents on list arguments beyond the first, for performance.
           // TODO: Once we have TupleElementAny, this generality can be increased.
           i = 0 and
-          exists(DataFlow::TupleElementContent tc, int j | j = tc.getIndex() |
-            input = "Argument[1].TupleElement[" + j.toString() + "]"
-          )
+          input = "Argument[1].AnyTupleElement"
           // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
         ) and
         output = "Argument[0].Parameter[" + i.toString() + "]" and
@@ -4499,9 +4481,7 @@ module StdlibPrivate {
         or
         input = "Argument[1].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[1].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[1].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       (output = "Argument[0].Parameter[0]" or output = "ReturnValue.ListElement") and
@@ -4525,9 +4505,7 @@ module StdlibPrivate {
         or
         input = "Argument[0].SetElement"
         or
-        exists(DataFlow::TupleElementContent tc, int i | i = tc.getIndex() |
-          input = "Argument[0].TupleElement[" + i.toString() + "]"
-        )
+        input = "Argument[0].AnyTupleElement"
         // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
       ) and
       output = "ReturnValue.ListElement.TupleElement[1]" and
@@ -4552,12 +4530,7 @@ module StdlibPrivate {
           or
           input = "Argument[" + i.toString() + "].SetElement"
           or
-          // We reduce generality slightly by not tracking tuple contents on arguments beyond the first two, for performance.
-          // TODO: Once we have TupleElementAny, this generality can be increased.
-          i in [0 .. 1] and
-          exists(DataFlow::TupleElementContent tc, int j | j = tc.getIndex() |
-            input = "Argument[" + i.toString() + "].TupleElement[" + j.toString() + "]"
-          )
+          input = "Argument[" + i.toString() + "].AnyTupleElement"
           // TODO: Once we have DictKeyContent, we need to transform that into ListElementContent
         ) and
         output = "ReturnValue.ListElement.TupleElement[" + i.toString() + "]" and
@@ -4580,12 +4553,6 @@ module StdlibPrivate {
     override DataFlow::ArgumentNode getACallback() { none() }
 
     override predicate propagatesFlow(string input, string output, boolean preservesValue) {
-      exists(DataFlow::Content c |
-        input = "Argument[self]." + c.getMaDRepresentation() and
-        output = "ReturnValue." + c.getMaDRepresentation() and
-        preservesValue = true
-      )
-      or
       input = "Argument[self]" and
       output = "ReturnValue" and
       preservesValue = true
@@ -4741,12 +4708,10 @@ module StdlibPrivate {
     override DataFlow::ArgumentNode getACallback() { none() }
 
     override predicate propagatesFlow(string input, string output, boolean preservesValue) {
-      exists(DataFlow::DictionaryElementContent dc, string key | key = dc.getKey() |
-        input = "Argument[self].DictionaryElement[" + key + "]" and
-        output = "ReturnValue.TupleElement[1]" and
-        preservesValue = true
-        // TODO: put `key` into "ReturnValue.TupleElement[0]"
-      )
+      input = "Argument[self].AnyDictionaryElement" and
+      output = "ReturnValue.TupleElement[1]" and
+      preservesValue = true
+      // TODO: put `key` into "ReturnValue.TupleElement[0]"
     }
   }
 
@@ -4825,11 +4790,9 @@ module StdlibPrivate {
     }
 
     override predicate propagatesFlow(string input, string output, boolean preservesValue) {
-      exists(DataFlow::DictionaryElementContent dc, string key | key = dc.getKey() |
-        input = "Argument[self].DictionaryElement[" + key + "]" and
-        output = "ReturnValue.ListElement" and
-        preservesValue = true
-      )
+      input = "Argument[self].AnyDictionaryElement" and
+      output = "ReturnValue.ListElement" and
+      preservesValue = true
       or
       input = "Argument[self]" and
       output = "ReturnValue" and
@@ -4876,11 +4839,9 @@ module StdlibPrivate {
     }
 
     override predicate propagatesFlow(string input, string output, boolean preservesValue) {
-      exists(DataFlow::DictionaryElementContent dc, string key | key = dc.getKey() |
-        input = "Argument[self].DictionaryElement[" + key + "]" and
-        output = "ReturnValue.ListElement.TupleElement[1]" and
-        preservesValue = true
-      )
+      input = "Argument[self].AnyDictionaryElement" and
+      output = "ReturnValue.ListElement.TupleElement[1]" and
+      preservesValue = true
       or
       // TODO: Add the keys to output list
       input = "Argument[self]" and

python/ql/test/library-tests/dataflow/coverage/test_builtins.py

@@ -589,11 +589,11 @@ def test_zip_tuple():
 
     SINK(z[0][0])  # $ flow="SOURCE, l:-7 -> z[0][0]"
     SINK(z[0][1])  # $ flow="SOURCE, l:-7 -> z[0][1]"
-    SINK_F(z[0][2])
+    SINK_F(z[0][2]) # $ SPURIOUS: flow="SOURCE, l:-7 -> z[0][2]"
     SINK_F(z[0][3])
     SINK(z[1][0])  # $ flow="SOURCE, l:-11 -> z[1][0]"
     SINK_F(z[1][1]) # $ SPURIOUS: flow="SOURCE, l:-11 -> z[1][1]"
-    SINK(z[1][2]) # $ MISSING: flow="SOURCE, l:-11 -> z[1][2]" # Tuple contents are not tracked beyond the first two arguments for performance.
+    SINK(z[1][2]) # $ flow="SOURCE, l:-11 -> z[1][2]"
     SINK_F(z[1][3])
 
 @expects(4)

python/ql/test/library-tests/dataflow/typetracking/attribute_tests.py

@@ -157,7 +157,7 @@ class MyClass2(object):
         print(self.foo) # $ tracked MISSING: tracked=foo
 
 instance = MyClass2()
-print(instance.foo) # $ tracked MISSING: tracked=foo
+print(instance.foo) # $ MISSING: tracked=foo tracked
 instance.print_foo() # $ MISSING: tracked=foo
 
 
@@ -195,7 +195,7 @@ class Sub1(Base1):
 
 sub1 = Sub1()
 sub1.read_foo()
-print(sub1.foo) # $ tracked MISSING: tracked=foo
+print(sub1.foo) # $ MISSING: tracked=foo tracked
 
 
 # attribute written in a subclass method, read in an inherited base class method
@@ -210,7 +210,7 @@ class Sub2(Base2):
 
 sub2 = Sub2()
 sub2.read_bar()
-print(sub2.bar) # $ tracked MISSING: tracked=bar
+print(sub2.bar) # $ MISSING: tracked=bar tracked
 
 
 # attribute written in a base class method, read on an instance of the subclass
@@ -223,4 +223,4 @@ class Sub3(Base3):
     pass
 
 sub3 = Sub3()
-print(sub3.baz) # $ tracked MISSING: tracked=baz
+print(sub3.baz) # $ MISSING: tracked=baz tracked

python/ql/test/library-tests/frameworks/django-orm/testapp/orm_tests.py

@@ -362,7 +362,7 @@ def test_load_in_bulk():
     # see https://docs.djangoproject.com/en/4.0/ref/models/querysets/#in-bulk
     d = TestLoad.objects.in_bulk([1])
     for val in d.values():
-        SINK(val.text) # $ MISSING: flow
+        SINK(val.text) # $ flow="SOURCE, l:-65 -> val.text"
     SINK(d[1].text) # $ flow="SOURCE, l:-66 -> d[1].text"
 
 

python/ql/test/query-tests/Security/CWE-089-SqlInjection/SqlInjection.expected

@@ -1,7 +1,6 @@
 #select
 | app.py:23:20:23:24 | ControlFlowNode for query | app.py:20:18:20:21 | ControlFlowNode for name | app.py:23:20:23:24 | ControlFlowNode for query | This SQL query depends on a $@. | app.py:20:18:20:21 | ControlFlowNode for name | user-provided value |
 | app.py:30:20:30:24 | ControlFlowNode for query | app.py:27:19:27:22 | ControlFlowNode for name | app.py:30:20:30:24 | ControlFlowNode for query | This SQL query depends on a $@. | app.py:27:19:27:22 | ControlFlowNode for name | user-provided value |
-| app.py:37:20:37:24 | ControlFlowNode for query | app.py:34:19:34:22 | ControlFlowNode for name | app.py:37:20:37:24 | ControlFlowNode for query | This SQL query depends on a $@. | app.py:34:19:34:22 | ControlFlowNode for name | user-provided value |
 | app.py:44:20:44:24 | ControlFlowNode for query | app.py:41:19:41:22 | ControlFlowNode for name | app.py:44:20:44:24 | ControlFlowNode for query | This SQL query depends on a $@. | app.py:41:19:41:22 | ControlFlowNode for name | user-provided value |
 | app.py:51:20:51:24 | ControlFlowNode for query | app.py:48:19:48:22 | ControlFlowNode for name | app.py:51:20:51:24 | ControlFlowNode for query | This SQL query depends on a $@. | app.py:48:19:48:22 | ControlFlowNode for name | user-provided value |
 | sql_injection.py:21:24:21:77 | ControlFlowNode for BinaryExpr | sql_injection.py:14:15:14:22 | ControlFlowNode for username | sql_injection.py:21:24:21:77 | ControlFlowNode for BinaryExpr | This SQL query depends on a $@. | sql_injection.py:14:15:14:22 | ControlFlowNode for username | user-provided value |
@@ -25,8 +24,6 @@ edges
 | app.py:21:5:21:9 | ControlFlowNode for query | app.py:23:20:23:24 | ControlFlowNode for query | provenance |  |
 | app.py:27:19:27:22 | ControlFlowNode for name | app.py:28:5:28:9 | ControlFlowNode for query | provenance |  |
 | app.py:28:5:28:9 | ControlFlowNode for query | app.py:30:20:30:24 | ControlFlowNode for query | provenance |  |
-| app.py:34:19:34:22 | ControlFlowNode for name | app.py:35:5:35:9 | ControlFlowNode for query | provenance |  |
-| app.py:35:5:35:9 | ControlFlowNode for query | app.py:37:20:37:24 | ControlFlowNode for query | provenance |  |
 | app.py:41:19:41:22 | ControlFlowNode for name | app.py:42:5:42:9 | ControlFlowNode for query | provenance |  |
 | app.py:42:5:42:9 | ControlFlowNode for query | app.py:44:20:44:24 | ControlFlowNode for query | provenance |  |
 | app.py:48:19:48:22 | ControlFlowNode for name | app.py:49:5:49:9 | ControlFlowNode for query | provenance |  |
@@ -54,9 +51,6 @@ nodes
 | app.py:27:19:27:22 | ControlFlowNode for name | semmle.label | ControlFlowNode for name |
 | app.py:28:5:28:9 | ControlFlowNode for query | semmle.label | ControlFlowNode for query |
 | app.py:30:20:30:24 | ControlFlowNode for query | semmle.label | ControlFlowNode for query |
-| app.py:34:19:34:22 | ControlFlowNode for name | semmle.label | ControlFlowNode for name |
-| app.py:35:5:35:9 | ControlFlowNode for query | semmle.label | ControlFlowNode for query |
-| app.py:37:20:37:24 | ControlFlowNode for query | semmle.label | ControlFlowNode for query |
 | app.py:41:19:41:22 | ControlFlowNode for name | semmle.label | ControlFlowNode for name |
 | app.py:42:5:42:9 | ControlFlowNode for query | semmle.label | ControlFlowNode for query |
 | app.py:44:20:44:24 | ControlFlowNode for query | semmle.label | ControlFlowNode for query |

python/ql/test/query-tests/Security/CWE-089-SqlInjection/app.py

@@ -31,10 +31,10 @@ async def unsafe2(name: str): # $ Source
     cursor.close()
 
 @app.get("/unsafe3/")
-async def unsafe3(name: str): # $ Source
+async def unsafe3(name: str): # $ MISSING: Source
     query = "select * from users where name=" + name
     cursor = hdb_con3.cursor()
-    cursor.execute(query) # $ Alert
+    cursor.execute(query) # $ MISSING: Alert
     cursor.close()
 
 @app.get("/unsafe4/")