Merge branch 'main' into ts-54

docs/codeql/codeql-for-visual-studio-code/testing-codeql-queries-in-visual-studio-code.rst

@@ -12,27 +12,31 @@ About testing queries in VS Code
 
 To ensure that your CodeQL queries produce the expected results, you can run tests that compare the expected query results with the actual results.
 
-The CodeQL extension automatically prompts VS Code to install the Test Explorer extension as a dependency. The Test Explorer displays any workspace folders with a name ending in ``-tests`` and provides a UI for exploring and running tests in those folders.
+The CodeQL extension automatically registers itself with the **Testing** view. The **Testing** view displays all tests found in your current workspace and provides a UI for exploring and running tests in your workspace.
 
 For more information about how CodeQL tests work, see "`Testing custom queries <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries>`__" in the CLI help.
 
 Testing the results of your queries
 -----------------------------------
 
-1. Open the Test Explorer view in the sidebar.
+1. Open the **Testing** view in the sidebar.
 
-   .. image:: ../images/codeql-for-visual-studio-code/open-test-explorer.png
+   .. image:: ../images/codeql-for-visual-studio-code/open-testing-view.png
       :width: 350
-      :alt: Open the Test Explorer view
+      :alt: Open the Testing view
 
 2. To run a specific test, hover over the file or folder name and click the play button. To run all tests in your workspace, click the play button at the top of the view. If a test takes too long to run, you can click the stop button at the top of the view to cancel the test.
-3. The icons show whether a test passed or failed. If it failed, right-click the failed test and click **CodeQL: Show Test Output Differences** to display the differences between the expected output and the actual output.
+3. The icons show whether a test passed or failed. If it failed, click the test in the **Test Results** panel to display the differences between the expected output and the actual output.
 
-   .. image:: ../images/codeql-for-visual-studio-code/show-test-diff.png
+   .. image:: ../images/codeql-for-visual-studio-code/test-results-panel.png
+      :width: 800
+      :alt: View the Test Results panel
+
+4. Compare the results. If you want to update the test with the actual output, right-click the test in the **Testing** view and click **Accept Test Output**.
+
+   .. image:: ../images/codeql-for-visual-studio-code/accept-test-output.png
       :width: 400
-      :alt: Show test output differences
-
-4. Compare the results. If you want to update the test with the actual output, click **CodeQL: Accept Test Output**.
+      :alt: Acccept test output in the Testing view
 
 Monitoring the performance of your queries
 ------------------------------------------

docs/codeql/codeql-for-visual-studio-code/using-the-codeql-model-editor.rst

@@ -18,9 +18,9 @@ When you open the model editor, it analyzes the currently selected CodeQL databa
 
 The model editor has two different modes:
 
-- Application mode (default view): The editor lists each external framework used by the selected CodeQL database. When you expand a framework, a list of all calls to and from the external API is shown with the options available to model dataflow through each call. This mode is most useful for improving the CodeQL results for a specific codebase.
+- **Application mode (default view):** The editor lists each external framework used by the selected CodeQL database. When you expand a framework, a list of all calls to and from the external API is shown with the options available to model dataflow through each call. This mode is most useful for improving the CodeQL results for a specific codebase.
 
-- Dependency mode: The editor identifies all of the publicly accessible APIs in the selected CodeQL database. This view guides you through modeling each public API that the codebase makes available. When you have finished modeling the entire API, you can save the model and use it to improve the CodeQL analysis for all codebases that use the dependency.
+- **Dependency mode:** The editor identifies all of the publicly accessible APIs in the selected CodeQL database. This view guides you through modeling each public API that the codebase makes available. When you have finished modeling the entire API, you can save the model and use it to improve the CodeQL analysis for all codebases that use the dependency.
 
 Displaying the CodeQL model editor
 ----------------------------------
@@ -32,7 +32,7 @@ Displaying the CodeQL model editor
 #. The CodeQL model editor runs a series of telemetry queries to identify APIs in the code and the editor is displayed in a new tab.
 #. When the telemetry queries are complete, the APIs that have been identified are shown in the editor.
 
-.. tip::
+.. pull-quote:: Tip
 
    The "CodeQL method modeling" section is a view that you can move from the primary sidebar to the secondary sidebar, when you want more space while you are modeling calls or methods. If you close the view, you can reopen it from the "Open Views" option in the **View** menu.
 
@@ -41,6 +41,8 @@ Modeling the calls your codebase makes to external APIs
 
 You typically use this approach when you are looking at a specific codebase where you want to improve the precision of CodeQL results. This is useful when the codebase uses frameworks or libraries that are not supported by CodeQL and if the source code of the framework or library is not included in the analysis.
 
+This section uses an open source Java project called "sofa-jraft" as an example. The experience of modeling calls to external APIs written in other static languages is similar.
+
 #. Select the CodeQL database that you want to improve CodeQL coverage for.
 #. Display the CodeQL model editor. By default the editor runs in application mode, so the list of external APIs used by the selected codebase is shown.
 
@@ -90,6 +92,8 @@ The models are stored in a series of YAML data extension files, one for each ext
 Modeling the public API of a codebase
 -------------------------------------
 
+This section uses an open source Java project called "sofa-jraft" as an example. The experience of modeling the public API written using other static languages is similar.
+
 You typically use this method when you want to model a framework or library that your organization uses in more than one codebase. Once you have finished creating and testing the model, you can publish the CodeQL model pack to the GitHub Container Registry for your whole organization to use.
 
 #. Select the CodeQL database that you want to model.

docs/codeql/codeql-language-guides/analyzing-data-flow-in-java.rst

@@ -27,7 +27,13 @@ Local data flow is data flow within a single method or callable. Local data flow
 Using local data flow
 ~~~~~~~~~~~~~~~~~~~~~
 
-The local data flow library is in the module ``DataFlow``, which defines the class ``Node`` denoting any element that data can flow through. ``Node``\ s are divided into expression nodes (``ExprNode``) and parameter nodes (``ParameterNode``). You can map between data flow nodes and expressions/parameters using the member predicates ``asExpr`` and ``asParameter``:
+To use the data flow library you need the following import:
+
+.. code-block:: ql
+
+   import semmle.code.java.dataflow.DataFlow
+
+The ``DataFlow`` module defines the class ``Node`` denoting any element that data can flow through. ``Node``\ s are divided into expression nodes (``ExprNode``) and parameter nodes (``ParameterNode``). You can map between data flow nodes and expressions/parameters using the member predicates ``asExpr`` and ``asParameter``:
 
 .. code-block:: ql
 
@@ -75,7 +81,14 @@ Local taint tracking extends local data flow by including non-value-preserving f
 
 If ``x`` is a tainted string then ``y`` is also tainted.
 
-The local taint tracking library is in the module ``TaintTracking``. Like local data flow, a predicate ``localTaintStep(DataFlow::Node nodeFrom, DataFlow::Node nodeTo)`` holds if there is an immediate taint propagation edge from the node ``nodeFrom`` to the node ``nodeTo``. You can apply the predicate recursively by using the ``+`` and ``*`` operators, or by using the predefined recursive predicate ``localTaint``, which is equivalent to ``localTaintStep*``.
+
+To use the taint tracking library you need the following import:
+
+.. code-block:: ql
+
+   import semmle.code.java.dataflow.TaintTracking
+
+Like local data flow, a predicate ``localTaintStep(DataFlow::Node nodeFrom, DataFlow::Node nodeTo)`` holds if there is an immediate taint propagation edge from the node ``nodeFrom`` to the node ``nodeTo``. You can apply the predicate recursively by using the ``+`` and ``*`` operators, or by using the predefined recursive predicate ``localTaint``, which is equivalent to ``localTaintStep*``.
 
 For example, you can find taint propagation from a parameter ``source`` to an expression ``sink`` in zero or more local steps:
 

docs/codeql/codeql-language-guides/codeql-for-csharp.rst

@@ -11,6 +11,7 @@ Experiment and learn how to write effective and efficient queries for CodeQL dat
    basic-query-for-csharp-code
    codeql-library-for-csharp
    analyzing-data-flow-in-csharp
+   customizing-library-models-for-csharp
 
 -  :doc:`Basic query for C# code <basic-query-for-csharp-code>`: Learn to write and run a simple CodeQL query.
 
@@ -18,4 +19,5 @@ Experiment and learn how to write effective and efficient queries for CodeQL dat
 
 -  :doc:`Analyzing data flow in C# <analyzing-data-flow-in-csharp>`: You can use CodeQL to track the flow of data through a C# program to its use.
 
+-  :doc:`Customizing library models for C# <customizing-library-models-for-csharp>`: You can model frameworks and libraries that your codebase depends on using data extensions and publish them as CodeQL model packs.
 

docs/codeql/codeql-language-guides/customizing-library-models-for-csharp.rst

@@ -0,0 +1,358 @@
+.. _customizing-library-models-for-csharp:
+
+Customizing library models for C#
+=================================
+
+You can model the methods and callables that control data flow in any framework or library. This is especially useful for custom frameworks or niche libraries, that are not supported by the standard CodeQL libraries.
+
+.. include:: ../reusables/beta-note-customizing-library-models.rst
+
+About this article
+------------------
+
+This article contains reference material about how to define custom models for sources, sinks, and flow summaries for C# dependencies in data extension files.
+
+About data extensions
+---------------------
+
+You can customize analysis by defining models (summaries, sinks, and sources) of your code's C#/.NET dependencies in data extension files. Each model defines the behavior of one or more elements of your library or framework, such as methods, properties, and callables. When you run dataflow analysis, these models expand the potential sources and sinks tracked by dataflow analysis and improve the precision of results.
+
+Most of the security queries search for paths from a source of untrusted input to a sink that represents a vulnerability. This is known as taint tracking. Each source is a starting point for dataflow analysis to track tainted data and each sink is an end point.
+
+Taint tracking queries also need to know how data can flow through elements that are not included in the source code. These are modeled as summaries. A summary model enables queries to synthesize the flow behavior through elements in dependency code that is not stored in your repository.
+
+Syntax used to define an element in an extension file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each model of an element is defined using a data extension where each tuple constitutes a model.
+A data extension file to extend the standard C# queries included with CodeQL is a YAML file with the form:
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: <name of extensible predicate>
+       data:
+         - <tuple1>
+         - <tuple2>
+         - ...
+
+Each YAML file may contain one or more top-level extensions.
+
+- ``addsTo`` defines the CodeQL pack name and extensible predicate that the extension is injected into.
+- ``data`` defines one or more rows of tuples that are injected as values into the extensible predicate. The number of columns and their types must match the definition of the extensible predicate.
+
+Data extensions use union semantics, which means that the tuples of all extensions for a single extensible predicate are combined, duplicates are removed, and all of the remaining tuples are queryable by referencing the extensible predicate.
+
+Publish data extension files in a CodeQL model pack to share
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can group one or more data extension files into a CodeQL model pack and publish it to the GitHub Container Registry. This makes it easy for anyone to download the model pack and use it to extend their analysis. For more information, see `Creating a CodeQL model pack <https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-a-codeql-model-pack>`__ and `Publishing and using CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/publishing-and-using-codeql-packs/>`__ in the CodeQL CLI documentation.
+
+Extensible predicates used to create custom models in C#
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The CodeQL library for C# analysis exposes the following extensible predicates:
+
+- ``sourceModel(namespace, type, subtypes, name, signature, ext, output, kind, provenance)``. This is used to model sources of potentially tainted data. The ``kind`` of the sources defined using this predicate determine which threat model they are associated with. Different threat models can be used to customize the sources used in an analysis. For more information, see ":ref:`Threat models <threat-models-csharp>`."
+- ``sinkModel(namespace, type, subtypes, name, signature, ext, input, kind, provenance)``. This is used to model sinks where tainted data may be used in a way that makes the code vulnerable.
+- ``summaryModel(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance)``. This is used to model flow through elements.
+- ``neutralModel(namespace, type, name, signature, kind, provenance)``. This is similar to a summary model but used to model the flow of values that have only a minor impact on the dataflow analysis. Manual neutral models (those with a provenance such as ``manual`` or ``ai-manual``) can be used to override generated summary models (those with a provenance such as ``df-generated``), so that the summary model will be ignored. Other than that, neutral models have no effect.
+
+The extensible predicates are populated using the models defined in data extension files.
+
+Examples of custom model definitions
+------------------------------------
+
+The examples in this section are taken from the standard CodeQL C# query pack published by GitHub. They demonstrate how to add tuples to extend extensible predicates that are used by the standard queries.
+
+Example: Taint sink in the ``System.Data.SqlClient`` namespace
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This example shows how the C# query pack models the argument of the ``SqlCommand`` constructor as a SQL injection sink.
+This is the constructor of the ``SqlCommand`` class, which is located in the ``System.Data.SqlClient`` namespace.
+
+.. code-block:: csharp
+
+   public static void TaintSink(SqlConnection conn, string query) {
+        SqlCommand command = new SqlCommand(query, connection) // The argument to this method is a SQL injection sink.
+        ...
+   }
+
+We need to add a tuple to the ``sinkModel``\(namespace, type, subtypes, name, signature, ext, input, kind, provenance) extensible predicate by updating a data extension file.
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: sinkModel
+       data:
+         - ["System.Data.SqlClient", "SqlCommand", False, "SqlCommand", "(System.String,System.Data.SqlClient.SqlConnection)", "", "Argument[0]", "sql-injection", "manual"]
+
+Since we want to add a new sink, we need to add a tuple to the ``sinkModel`` extensible predicate.
+The first five values identify the callable (in this case a method) to be modeled as a sink.
+
+- The first value ``System.Data.SqlClient`` is the namespace name.
+- The second value ``SqlCommand`` is the name of the class (type) that contains the method.
+- The third value ``False`` is a flag that indicates whether or not the sink also applies to all overrides of the method.
+- The fourth value ``SqlCommand`` is the method name. Constructors are named after the class.
+- The fifth value ``(System.String,System.Data.SqlClient.SqlConnection)`` is the method input type signature. The type names must be fully qualified.
+
+The sixth value should be left empty and is out of scope for this documentation.
+The remaining values are used to define the ``access path``, the ``kind``, and the ``provenance`` (origin) of the sink.
+
+- The seventh value ``Argument[0]`` is the ``access path`` to the first argument passed to the method, which means that this is the location of the sink.
+- The eighth value ``sql-injection`` is the kind of the sink. The sink kind is used to define the queries where the sink is in scope. In this case - the SQL injection queries.
+- The ninth value ``manual`` is the provenance of the sink, which is used to identify the origin of the sink.
+
+Example: Taint source from the ``System.Net.Sockets`` namespace
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This example shows how the C# query pack models the return value from the ``GetStream`` method as a ``remote`` source.
+This is the ``GetStream`` method in the ``TcpClient`` class, which is located in the ``System.Net.Sockets`` namespace.
+
+.. code-block:: csharp
+
+   public static void Tainted(TcpClient client) {
+       NetworkStream stream = client.GetStream(); // The return value of this method is a remote source of taint.
+       ...
+   }
+
+We need to add a tuple to the ``sourceModel``\(namespace, type, subtypes, name, signature, ext, output, kind, provenance) extensible predicate by updating a data extension file.
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: sourceModel
+       data:
+         - ["System.Net.Sockets", "TcpClient", False, "GetStream", "()", "", "ReturnValue", "remote", "manual"]
+
+
+Since we are adding a new source, we need to add a tuple to the ``sourceModel`` extensible predicate.
+The first five values identify the callable (in this case a method) to be modeled as a source.
+
+- The first value ``System.Net.Sockets`` is the namespace name.
+- The second value ``TcpClient`` is the name of the class (type) that contains the source.
+- The third value ``False`` is a flag that indicates whether or not the source also applies to all overrides of the method.
+- The fourth value ``GetStream`` is the method name.
+- The fifth value ``()`` is the method input type signature.
+
+The sixth value should be left empty and is out of scope for this documentation.
+The remaining values are used to define the ``access path``, the ``kind``, and the ``provenance`` (origin) of the source.
+
+- The seventh value ``ReturnValue`` is the access path to the return of the method, which means that it is the return value that should be considered a source of tainted input.
+- The eighth value ``remote`` is the kind of the source. The source kind is used to define the threat model where the source is in scope. ``remote`` applies to many of the security related queries as it means a remote source of untrusted data. As an example the SQL injection query uses ``remote`` sources. For more information, see ":ref:`Threat models <threat-models-csharp>`."
+- The ninth value ``manual`` is the provenance of the source, which is used to identify the origin of the source.
+
+Example: Add flow through the ``Concat`` method
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This example shows how the C# query pack models flow through a method for a simple case.
+This pattern covers many of the cases where we need to summarize flow through a method that is stored in a library or framework outside the repository.
+
+.. code-block:: csharp
+
+   public static void TaintFlow(string s1, string s2) {
+       string t = String.Concat(s1, s2); // There is taint flow from s1 and s2 to t.
+       ...
+   }
+
+We need to add tuples to the ``summaryModel``\(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance) extensible predicate by updating a data extension file:
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: summaryModel
+       data:
+         - ["System", "String", False, "Concat", "(System.Object,System.Object)", "", "Argument[0]", "ReturnValue", "taint", "manual"]
+         - ["System", "String", False, "Concat", "(System.Object,System.Object)", "", "Argument[1]", "ReturnValue", "taint", "manual"]
+
+Since we are adding flow through a method, we need to add tuples to the ``summaryModel`` extensible predicate.
+Each tuple defines flow from one argument to the return value.
+The first row defines flow from the first argument (``s1`` in the example) to the return value (``t`` in the example) and the second row defines flow from the second argument (``s2`` in the example) to the return value (``t`` in the example).
+
+The first five values identify the callable (in this case a method) to be modeled as a summary.
+These are the same for both of the rows above as we are adding two summaries for the same method.
+
+- The first value ``System`` is the namespace name.
+- The second value ``String`` is the class (type) name.
+- The third value ``False`` is a flag that indicates whether or not the summary also applies to all overrides of the method.
+- The fourth value ``Concat`` is the method name.
+- The fifth value ``(System.Object,System.Object)`` is the method input type signature.
+
+The sixth value should be left empty and is out of scope for this documentation.
+The remaining values are used to define the ``access path``, the ``kind``, and the ``provenance`` (origin) of the summary.
+
+- The seventh value is the access path to the input (where data flows from). ``Argument[0]`` is the access path to the first argument (``s1`` in the example) and ``Argument[1]`` is the access path to the second argument (``s2`` in the example).
+- The eighth value ``ReturnValue`` is the access path to the output (where data flows to), in this case ``ReturnValue``, which means that the input flows to the return value.
+- The ninth value ``taint`` is the kind of the flow. ``taint`` means that taint is propagated through the call.
+- The tenth value ``manual`` is the provenance of the summary, which is used to identify the origin of the summary.
+
+It would also be possible to merge the two rows into one by using a comma-separated list in the seventh value. This would be useful if the method has many arguments and the flow is the same for all of them.
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: summaryModel
+       data:
+         - ["System", "String", False, "Concat", "(System.Object,System.Object)", "", "Argument[0,1]", "ReturnValue", "taint", "manual"]
+
+This row defines flow from both the first and the second argument to the return value. The seventh value ``Argument[0,1]`` is shorthand for specifying an access path to both ``Argument[0]`` and ``Argument[1]``.
+
+Example: Add flow through the ``Trim`` method
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This example shows how the C# query pack models flow through a method for a simple case.
+
+.. code-block:: csharp
+
+   public static void TaintFlow(string s) {
+       string t = s.Trim(); // There is taint flow from s to t.
+       ...
+   }
+
+We need to add a tuple to the ``summaryModel``\(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance) extensible predicate by updating a data extension file:
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: summaryModel
+       data:
+         - ["System", "String", False, "Trim", "()", "", "Argument[this]", "ReturnValue", "taint", "manual"]
+
+Since we are adding flow through a method, we need to add tuples to the ``summaryModel`` extensible predicate.
+Each tuple defines flow from one argument to the return value.
+The first row defines flow from the qualifier of the method call (``s1`` in the example) to the return value (``t`` in the example).
+
+The first five values identify the callable (in this case a method) to be modeled as a summary.
+These are the same for both of the rows above as we are adding two summaries for the same method.
+
+- The first value ``System`` is the namespace name.
+- The second value ``String`` is the class (type) name.
+- The third value ``False`` is a flag that indicates whether or not the summary also applies to all overrides of the method.
+- The fourth value ``Trim`` is the method name.
+- The fifth value ``()`` is the method input type signature.
+
+The sixth value should be left empty and is out of scope for this documentation.
+The remaining values are used to define the ``access path``, the ``kind``, and the ``provenance`` (origin) of the summary.
+
+- The seventh value is the access path to the input (where data flows from). ``Argument[this]`` is the access path to the qualifier (``s`` in the example).
+- The eighth value ``ReturnValue`` is the access path to the output (where data flows to), in this case ``ReturnValue``, which means that the input flows to the return value.
+- The ninth value ``taint`` is the kind of the flow. ``taint`` means that taint is propagated through the call.
+- The tenth value ``manual`` is the provenance of the summary, which is used to identify the origin of the summary.
+
+Example: Add flow through the ``Select`` method
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This example shows how the C# query pack models a more complex flow through a method.
+Here we model flow through higher order methods and collection types, as well as how to handle extension methods and generics.
+
+.. code-block:: csharp
+
+   public static void TaintFlow(IEnumerable<string> stream) {
+     IEnumerable<string> lines = stream.Select(item => item + "\n");
+     ...
+   }
+
+We need to add tuples to the ``summaryModel``\(namespace, type, subtypes, name, signature, ext, input, output, kind, provenance) extensible predicate by updating a data extension file:
+
+.. code-block:: yaml
+
+   extensions:
+     - addsTo:
+         pack: codeql/csharp-all
+         extensible: summaryModel
+       data:
+         - ["System.Linq", "Enumerable", False, "Select<TSource,TResult>", "(System.Collections.Generic.IEnumerable<TSource>,System.Func<TSource,TResult>)", "", "Argument[0].Element", "Argument[1].Parameter[0]", "value", "manual"]
+         - ["System.Linq", "Enumerable", False, "Select<TSource,TResult>", "(System.Collections.Generic.IEnumerable<TSource>,System.Func<TSource,TResult>)", "", "Argument[1].ReturnValue", "ReturnValue.Element", "value", "manual"]
+
+
+Since we are adding flow through a method, we need to add tuples to the ``summaryModel`` extensible predicate.
+Each tuple defines part of the flow that comprises the total flow through the ``Select`` method.
+The first five values identify the callable (in this case a method) to be modeled as a summary.
+These are the same for both of the rows above as we are adding two summaries for the same method.
+
+- The first value ``System.Linq`` is the namespace name.
+- The second value ``Enumerable`` is the class (type) name.
+- The third value ``False`` is a flag that indicates whether or not the summary also applies to all overrides of the method.
+- The fourth value ``Select<TSource,TResult>`` is the method name, along with the type parameters for the method. The names of the generic type parameters provided in the model must match the names of the generic type parameters in the method signature in the source code.
+- The fifth value ``(System.Collections.Generic.IEnumerable<TSource>,System.Func<TSource,TResult>)`` is the method input type signature. The generics in the signature must match the generics in the method signature in the source code. 
+
+The sixth value should be left empty and is out of scope for this documentation.
+The remaining values are used to define the ``access path``, the ``kind``, and the ``provenance`` (origin) of the summary definition.
+
+- The seventh value is the access path to the ``input`` (where data flows from).
+- The eighth value is the access path to the ``output`` (where data flows to).
+
+For the first row:
+
+- The seventh value is ``Argument[0].Element``, which is the access path to the elements of the qualifier (the elements of the enumerable ``stream`` in the example).
+- The eight value is ``Argument[1].Parameter[0]``, which is the access path to the first parameter of the ``System.Func<TSource,TResult>`` argument of ``Select`` (the lambda parameter ``item`` in the example).
+
+For the second row:
+
+- The seventh value is ``Argument[1].ReturnValue``, which is the access path to the return value of the ``System.Func<TSource,TResult>`` argument of ``Select`` (the return value of the lambda in the example).
+- The eighth value is ``ReturnValue.Element``, which is the access path to the elements of the return value of ``Select`` (the elements of the enumerable ``lines`` in the example).
+
+For the remaining values for both rows:
+
+- The ninth value ``value`` is the kind of the flow. ``value`` means that the value is preserved.
+- The tenth value ``manual`` is the provenance of the summary, which is used to identify the origin of the summary.
+
+That is, the first row specifies that values can flow from the elements of the qualifier enumerable into the first argument of the function provided to ``Select``.  The second row specifies that values can flow from the return value of the function to the elements of the enumerable returned from ``Select``.
+
+Example: Add a ``neutral`` method
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This example shows how we can model a method as being neutral with respect to flow. We will also cover how to model a property by modeling the getter of the ``Now`` property of the ``DateTime`` class as neutral. 
+A neutral model is used to define that there is no flow through a method.
+
+.. code-block:: csharp
+
+   public static void TaintFlow() {
+       System.DateTime t = System.DateTime.Now; // There is no flow from Now to t.
+       ...
+   }
+
+We need to add a tuple to the ``neutralModel``\(namespace, type, name, signature, kind, provenance) extensible predicate by updating a data extension file.
+
+.. code-block:: yaml
+
+   extensions:
+   - addsTo:
+       pack: codeql/csharp-all
+       extensible: neutralModel
+     data:
+       - ["System", "DateTime", "get_Now", "()", "summary", "manual"]
+
+
+Since we are adding a neutral model, we need to add tuples to the ``neutralModel`` extensible predicate.
+The first four values identify the callable (in this case the getter of the ``Now`` property) to be modeled as a neutral, the fifth value is the kind, and the sixth value is the provenance (origin) of the neutral.
+
+- The first value ``System`` is the namespace name.
+- The second value ``DateTime`` is the class (type) name.
+- The third value ``get_Now`` is the method name. Getter and setter methods are named ``get_<name>`` and ``set_<name>`` respectively.
+- The fourth value ``()`` is the method input type signature.
+- The fifth value ``summary`` is the kind of the neutral.
+- The sixth value ``manual`` is the provenance of the neutral.
+
+.. _threat-models-csharp:
+
+Threat models
+-------------
+
+.. include:: ../reusables/beta-note-threat-models.rst
+
+A threat model is a named class of dataflow sources that can be enabled or disabled independently. Threat models allow you to control the set of dataflow sources that you want to consider unsafe. For example, one codebase may only consider remote HTTP requests to be tainted, whereas another may also consider data from local files to be unsafe. You can use threat models to ensure that the relevant taint sources are used in a CodeQL analysis.
+
+The ``kind`` property of ``sourceModel`` determines which threat model a source is associated with. There are two main categories:
+
+- ``remote`` which represents requests and responses from the network. 
+- ``local`` which represents data from local files (``file``), command-line arguments (``commandargs``), database reads (``database``), and environment variables(``environment``).
+ 
+When running a CodeQL analysis, the ``remote`` threat model is included by default. You can optionally include other threat models as appropriate when using the CodeQL CLI and in GitHub code scanning. For more information, see `Analyzing your code with CodeQL queries <https://docs.github.com/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries#including-model-packs-to-add-potential-sources-of-tainted-data>`__ and `Customizing your advanced setup for code scanning <https://docs.github.com/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#extending-codeql-coverage-with-threat-models>`__.

docs/codeql/codeql-language-guides/customizing-library-models-for-java-and-kotlin.rst

@@ -7,7 +7,7 @@ You can model the methods and callables that control data flow in any framework
 
 .. include:: ../reusables/kotlin-beta-note.rst
 
-.. include:: ../reusables/beta-note-model-packs-java.rst
+.. include:: ../reusables/beta-note-customizing-library-models.rst
 
 About this article
 ------------------
@@ -61,10 +61,10 @@ Extensible predicates used to create custom models in Java and Kotlin
 
 The CodeQL library for Java and Kotlin analysis exposes the following extensible predicates:
 
-- ``sourceModel(package, type, subtypes, name, signature, ext, output, kind, provenance)``. This is used to model sources of potentially tainted data. The ``kind`` of the sources defined using this predicate determine which threat model they are associated with. Different threat models can be used to customize the sources used in an analysis. For more information, see ":ref:`Threat models <threat-models>`."
+- ``sourceModel(package, type, subtypes, name, signature, ext, output, kind, provenance)``. This is used to model sources of potentially tainted data. The ``kind`` of the sources defined using this predicate determine which threat model they are associated with. Different threat models can be used to customize the sources used in an analysis. For more information, see ":ref:`Threat models <threat-models-java>`."
 - ``sinkModel(package, type, subtypes, name, signature, ext, input, kind, provenance)``. This is used to model sinks where tainted data maybe used in a way that makes the code vulnerable.
 - ``summaryModel(package, type, subtypes, name, signature, ext, input, output, kind, provenance)``. This is used to model flow through elements.
-- ``neutralModel(package, type, name, signature, kind, provenance)``. This is similar to a summary model but used to model the flow of values that have only a minor impact on the dataflow analysis.
+- ``neutralModel(package, type, name, signature, kind, provenance)``. This is similar to a summary model but used to model the flow of values that have only a minor impact on the dataflow analysis. Manual neutral models (those with a provenance such as ``manual`` or ``ai-manual``) override generated summary models (those with a provenance such as ``df-generated``) so that the summary will be ignored. Other than that, neutral models have a slight impact on the dataflow dispatch logic, which is out of scope for this documentation.
 
 The extensible predicates are populated using the models defined in data extension files.
 
@@ -151,7 +151,7 @@ The sixth value should be left empty and is out of scope for this documentation.
 The remaining values are used to define the ``access path``, the ``kind``, and the ``provenance`` (origin) of the source.
 
 - The seventh value ``ReturnValue`` is the access path to the return of the method, which means that it is the return value that should be considered a source of tainted input.
-- The eighth value ``remote`` is the kind of the source. The source kind is used to define the threat model where the source is in scope. ``remote`` applies to many of the security related queries as it means a remote source of untrusted data. As an example the SQL injection query uses ``remote`` sources. For more information, see ":ref:`Threat models <threat-models>`."
+- The eighth value ``remote`` is the kind of the source. The source kind is used to define the threat model where the source is in scope. ``remote`` applies to many of the security related queries as it means a remote source of untrusted data. As an example the SQL injection query uses ``remote`` sources. For more information, see ":ref:`Threat models <threat-models-java>`."
 - The ninth value ``manual`` is the provenance of the source, which is used to identify the origin of the source.
 
 Example: Add flow through the ``concat`` method
@@ -292,12 +292,12 @@ The first four values identify the callable (in this case a method) to be modele
 - The fifth value ``summary`` is the kind of the neutral.
 - The sixth value ``manual`` is the provenance of the neutral.
 
-.. _threat-models:
+.. _threat-models-java:
 
 Threat models
 -------------
 
-.. include:: ../reusables/beta-note-threat-models-java.rst
+.. include:: ../reusables/beta-note-threat-models.rst
 
 A threat model is a named class of dataflow sources that can be enabled or disabled independently. Threat models allow you to control the set of dataflow sources that you want to consider unsafe. For example, one codebase may only consider remote HTTP requests to be tainted, whereas another may also consider data from local files to be unsafe. You can use threat models to ensure that the relevant taint sources are used in a CodeQL analysis.
 

docs/codeql/codeql-language-guides/extensible-predicates.rst

@@ -8,7 +8,7 @@ Extensible predicates and their interaction with data extensions
 
 You can use data extensions to model the methods and callables that control dataflow in any framework or library. This is especially useful for custom frameworks or niche libraries, that are not supported by the standard CodeQL libraries.
 
-.. include:: ../reusables/beta-note-model-packs-java.rst
+.. include:: ../reusables/beta-note-customizing-library-models.rst
 
 About this article
 ------------------

docs/codeql/codeql-overview/codeql-changelog/codeql-cli-2.16.3.rst

@@ -0,0 +1,194 @@
+.. _codeql-cli-2.16.3:
+
+==========================
+CodeQL 2.16.3 (2024-02-22)
+==========================
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: none
+
+This is an overview of changes in the CodeQL CLI and relevant CodeQL query and library packs. For additional updates on changes to the CodeQL code scanning experience, check out the `code scanning section on the GitHub blog <https://github.blog/tag/code-scanning/>`__, `relevant GitHub Changelog updates <https://github.blog/changelog/label/code-scanning/>`__, `changes in the CodeQL extension for Visual Studio Code <https://marketplace.visualstudio.com/items/GitHub.vscode-codeql/changelog>`__, and the `CodeQL Action changelog <https://github.com/github/codeql-action/blob/main/CHANGELOG.md>`__.
+
+Security Coverage
+-----------------
+
+CodeQL 2.16.3 runs a total of 408 security queries when configured with the Default suite (covering 160 CWE). The Extended suite enables an additional 131 queries (covering 34 more CWE). 2 security queries have been added with this release.
+
+CodeQL CLI
+----------
+
+Bug Fixes
+~~~~~~~~~
+
+*   Fixed a bug where CodeQL may produce an invalid database when it exhausts all available ID numbers. Now it detects the condition and reports an error instead.
+
+New Features
+~~~~~~~~~~~~
+
+*   A new extractor option has been added to the Python extractor: :code:`python_executable_name`.
+    You can use this option to override the default process the extractor uses to find and select a Python executable.
+    Pass one of :code:`--extractor-option python_executable_name=py` or :code:`--extractor-option python_executable_name=python` or :code:`--extractor-option python_executable_name=python3` to commands that run the extractor, for example: :code:`codeql database create`. 
+    
+    On Windows machines, the Python extractor will expect to find :code:`py.exe` on the system :code:`PATH` by default.
+    If the Python executable has a different name, you can set the new extractor option to override this value and look for :code:`python.exe` or :code:`python3.exe`.
+    
+    For more information about using the extractor option with the CodeQL CLI, see `Extractor options <https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options>`__.
+
+Security Updates
+~~~~~~~~~~~~~~~~
+
+*   Fixes CVE-2024-25129, a limited data exfiltration vulnerability that could be triggered by untrusted databases or QL packs.  See the
+    \ `security advisory <https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gf8p-v3g3-3wph>`__ for more information.
+
+Query Packs
+-----------
+
+Bug Fixes
+~~~~~~~~~
+
+JavaScript/TypeScript
+"""""""""""""""""""""
+
+*   The left operand of the :code:`&&` operator no longer propagates data flow by default.
+
+Major Analysis Improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Golang
+""""""
+
+*   The query "Use of a hardcoded key for signing JWT" (:code:`go/hardcoded-key`) has been promoted from experimental to the main query pack. Its results will now appear by default as part of :code:`go/hardcoded-credentials`. This query was originally `submitted as an experimental query by @porcupineyhairs <https://github.com/github/codeql/pull/9378>`__.
+
+Java
+""""
+
+*   The sinks of the queries :code:`java/path-injection` and :code:`java/path-injection-local` have been reworked. Path creation sinks have been converted to summaries instead, while sinks now are actual file read/write operations only. This has reduced the false positive ratio of both queries.
+
+Minor Analysis Improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+C/C++
+"""""
+
+*   The "non-constant format string" query (:code:`cpp/non-constant-format`) has been updated to produce fewer false positives.
+*   Added dataflow models for the :code:`gettext` function variants.
+
+C#
+""
+
+*   Added sanitizers for relative URLs, :code:`List.Contains()`, and checking the :code:`.Host` property on an URI to the :code:`cs/web/unvalidated-url-redirection` query.
+
+Java
+""""
+
+*   The sanitizer for the path injection queries has been improved to handle more cases where :code:`equals` is used to check an exact path match.
+*   The query :code:`java/unvalidated-url-redirection` now sanitizes results following the same logic as the query :code:`java/ssrf`. URLs where the destination cannot be controlled externally are no longer reported.
+
+New Queries
+~~~~~~~~~~~
+
+Golang
+""""""
+
+*   The query "Missing JWT signature check" (:code:`go/missing-jwt-signature-check`) has been promoted from experimental to the main query pack. Its results will now appear by default. This query was originally `submitted as an experimental query by @am0o0 <https://github.com/github/codeql/pull/14075>`__.
+
+Java
+""""
+
+*   Added a new query :code:`java/android/insecure-local-authentication` for finding uses of biometric authentication APIs that do not make use of a :code:`KeyStore`\ -backed key and thus may be bypassed.
+
+Swift
+"""""
+
+*   Added a new experimental query, :code:`swift/unsafe-unpacking`, that detects unpacking user controlled zips without validating the destination file path is within the destination directory.
+
+Query Metadata Changes
+~~~~~~~~~~~~~~~~~~~~~~
+
+Java
+""""
+
+*   The :code:`security-severity` score of the query :code:`java/relative-path-command` has been reduced to better adjust it to the specific conditions needed for exploitation.
+
+Language Libraries
+------------------
+
+Minor Analysis Improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+C#
+""
+
+*   C# 12: The QL and data flow library now support primary constructors.
+*   Added a new database relation to store key-value pairs corresponding to compilations. The new relation is used in buildless mode to surface information related to dependency fetching.
+
+Java
+""""
+
+*   An extension point for sanitizers of the query :code:`java/unvalidated-url-redirection` has been added.
+    
+*   Added models for the following packages:
+
+    *   java.io
+    *   java.lang
+    *   java.net
+    *   java.net.http
+    *   java.nio.file
+    *   java.util.zip
+    *   javax.servlet
+    *   org.apache.commons.io
+    *   org.apache.hadoop.fs
+    *   org.apache.hadoop.fs.s3a
+    *   org.eclipse.jetty.client
+    *   org.gradle.api.file
+
+JavaScript/TypeScript
+"""""""""""""""""""""
+
+*   The name "certification" is no longer seen as possibly being a certificate, and will therefore no longer be flagged in queries like "clear-text-logging" which look for sensitive data.
+
+Python
+""""""
+
+*   The name "certification" is no longer seen as possibly being a certificate, and will therefore no longer be flagged in queries like "clear-text-logging" which look for sensitive data.
+*   Added modeling of the :code:`psycopg` PyPI package as a SQL database library.
+
+Ruby
+""""
+
+*   Raw output ERB tags of the form :code:`<%== ... %>` are now recognised as cross-site scripting sinks.
+*   The name "certification" is no longer seen as possibly being a certificate, and will therefore no longer be flagged in queries like "clear-text-logging" which look for sensitive data.
+
+Swift
+"""""
+
+*   The name "certification" is no longer seen as possibly being a certificate, and will therefore no longer be flagged in queries like "clear-text-logging" which look for sensitive data.
+
+Deprecated APIs
+~~~~~~~~~~~~~~~
+
+Java
+""""
+
+*   The :code:`PathCreation` class in :code:`PathCreation.qll` has been deprecated.
+
+New Features
+~~~~~~~~~~~~
+
+C/C++
+"""""
+
+*   A :code:`getInitialization` predicate was added to the :code:`RangeBasedForStmt` class that yields the C++20-style initializer of the range-based :code:`for` statement when it exists.
+
+Shared Libraries
+----------------
+
+Breaking Changes
+~~~~~~~~~~~~~~~~
+
+Dataflow Analysis
+"""""""""""""""""
+
+*   The :code:`edges` predicate contained in :code:`PathGraph` now contains two additional columns for propagating model provenance information. This is primarily an internal change without any impact on any APIs, except for specialised queries making use of :code:`MergePathGraph` in conjunction with custom :code:`PathGraph` implementations. Such queries will need to be updated to reference the two new columns. This is expected to be very rare, as :code:`MergePathGraph` is an advanced feature, but it is a breaking change for any such affected queries.

docs/codeql/codeql-overview/codeql-changelog/index.rst

@@ -11,6 +11,7 @@ A list of queries for each suite and language `is available here <https://docs.g
 .. toctree::
    :maxdepth: 1
 
+   codeql-cli-2.16.3
    codeql-cli-2.16.2
    codeql-cli-2.16.1
    codeql-cli-2.16.0

docs/codeql/reusables/beta-note-model-pack-editor-vsc.rst

@@ -2,4 +2,4 @@
 
     Note
 
-    The CodeQL model editor and CodeQL model packs are currently in beta and subject to change. During the beta, model packs are supported only by Java/Kotlin analysis. To use this beta functionality, install the latest version of the CodeQL extension for Visual Studio Code.
+    The CodeQL model editor and CodeQL model packs are currently in beta and subject to change. During the beta, model packs are supported only by Java/Kotlin and C# analysis. To use this beta functionality, install the latest version of the CodeQL extension for Visual Studio Code.

docs/codeql/reusables/beta-note-model-packs-java.rst

@@ -1,5 +0,0 @@
-.. pull-quote::
-
-    Note
-
-    CodeQL model packs are currently in beta and subject to change. During the beta, model packs are supported only by Java/Kotlin analysis. To use this beta functionality, install the latest version of the CodeQL CLI bundle from: https://github.com/github/codeql-action/releases.

docs/codeql/reusables/beta-note-threat-models.rst

@@ -2,4 +2,4 @@
 
     Note
 
-    Threat models are currently in beta and subject to change. During the beta, threat models are supported only by Java analysis.
+    Threat models are currently in beta and subject to change. During the beta, threat models are supported only by Java and C# analysis.

docs/codeql/reusables/supported-versions-compilers.rst

@@ -11,16 +11,16 @@
    Microsoft extensions (up to VS 2019),
 
    Arm Compiler 5 [4]_","``.cpp``, ``.c++``, ``.cxx``, ``.hpp``, ``.hh``, ``.h++``, ``.hxx``, ``.c``, ``.cc``, ``.h``"
-   C#,C# up to 11,"Microsoft Visual Studio up to 2019 with .NET up to 4.8,
+   C#,C# up to 12,"Microsoft Visual Studio up to 2019 with .NET up to 4.8,
 
    .NET Core up to 3.1
 
-   .NET 5, .NET 6, .NET 7","``.sln``, ``.csproj``, ``.cs``, ``.cshtml``, ``.xaml``"
+   .NET 5, .NET 6, .NET 7, .NET 8","``.sln``, ``.csproj``, ``.cs``, ``.cshtml``, ``.xaml``"
    Go (aka Golang), "Go up to 1.21", "Go 1.11 or more recent", ``.go``
    Java,"Java 7 to 21 [5]_","javac (OpenJDK and Oracle JDK),
 
    Eclipse compiler for Java (ECJ) [6]_",``.java``
-   Kotlin [7]_,"Kotlin 1.5.0 to 1.9.20","kotlinc",``.kt``
+   Kotlin [7]_,"Kotlin 1.5.0 to 1.9.2\ *x*","kotlinc",``.kt``
    JavaScript,ECMAScript 2022 or lower,Not applicable,"``.js``, ``.jsx``, ``.mjs``, ``.es``, ``.es6``, ``.htm``, ``.html``, ``.xhtm``, ``.xhtml``, ``.vue``, ``.hbs``, ``.ejs``, ``.njk``, ``.json``, ``.yaml``, ``.yml``, ``.raml``, ``.xml`` [8]_"
    Python [9]_,"2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12",Not applicable,``.py``
    Ruby [10]_,"up to 3.3",Not applicable,"``.rb``, ``.erb``, ``.gemspec``, ``Gemfile``"