Merge branch 'main' into ts53-ts

docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst

@@ -42,14 +42,47 @@ Downloading a database from GitHub
 
 .. include:: ../reusables/download-github-database.rst
 
+.. _filtering-databases-and-queries-by-language:
+
+Filtering databases and queries by language
+-------------------------------------------
+
+Optionally, to see databases containing a specific language and queries written for that language, you can apply a language filter using the language selector.
+
+#. To see available language filters, in the sidebar, click the **Language** title bar.
+#. Hover over the language filter you would like to apply, then click **Select**.
+
+   .. image:: ../images/codeql-for-visual-studio-code/choose-language-filter.png
+      :width: 350
+      :alt: Screenshot of the language selector. The "Select" button for a language filter is outlined in dark orange.
+
+Creating a custom query
+------------------------
+
+You can generate a query template for a specific language from the queries panel, then write your own code to quickly create a custom query.
+
+#. Optionally, to create a custom query in an existing directory, in the sidebar, click the **Queries** title bar to expand the queries panel, then select the desired directory.
+#. In the sidebar, hover over the **Queries** title bar, then click the **Create query** icon.
+
+   .. image:: ../images/codeql-for-visual-studio-code/create-query-icon.png
+      :width: 350
+      :alt: Screenshot of the queries panel. The "Create query" icon is outlined in dark orange.
+
+#. In the Command Palette, select the target language for your query. If you've chosen not to create your custom query in an existing directory, selecting a language will autogenerate a directory labeled ``codeql-custom-queries-<language>``, where ``<language>`` is the name of the selected language. A query template labeled ``example.ql`` will then be added to the existing or autogenerated directory.
+#. In the template, write your custom query, then save the file. Once your query is finished, you can run it from the queries panel.
+
 Running a query
 ------------------------
 
 The `CodeQL repository <https://github.com/github/codeql>`__ on GitHub contains lots of example queries.
-If you have that folder (or a different CodeQL pack) available in your workspace, you can access existing queries under ``<language>/ql/src/<category>``, for example ``java/ql/src/Likely Bugs``.
+You can access any existing queries in your workspace through the queries panel.
 
-#. Open a query (``.ql``) file. It is displayed in the editor, with IntelliSense features such as syntax highlighting and autocomplete suggestions.
-#. Right-click in the query window and select **CodeQL: Run Query on Selected Database**. (Alternatively, run the command from the Command Palette.)
+#. In the sidebar, to expand the queries panel, click the **Queries** title bar.
+#. To run a query against the selected database, hover over the desired query, then click the **Run local query** icon.
+
+   .. image:: ../images/codeql-for-visual-studio-code/run-local-query-icon.png
+      :width: 350
+      :alt: Screenshot of the mouse pointer hovering over a query in the queries panel. The "Run local query" icon is outlined in dark orange.
 
 The CodeQL extension runs the query on the current database and reports progress in the bottom right corner of the application.
 When the results are ready, they're displayed in the Results view.
@@ -61,6 +94,23 @@ For more information, see ":doc:`Troubleshooting CodeQL for Visual Studio Code <
 Running multiple queries
 --------------------------
 
+You can quickly run multiple queries against the database you've selected using the queries panel or a single command.
+
+Running all queries in a directory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can easily run every query in a directory using the queries panel.
+
+#. In the sidebar, to expand the queries panel, click the **Queries** title bar.
+#. Hover over the desired directory of queries, then click the **Run local queries** icon.
+
+   .. image:: ../images/codeql-for-visual-studio-code/run-local-queries-icon.png
+      :width: 350
+      :alt: Screenshot of the mouse pointer hovering over a directory of queries in the queries panel. The "Run local queries" icon is outlined in dark orange.
+
+Running a selection of queries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 You can run multiple queries with a single command.
 
 #. Go to the File Explorer.
@@ -122,6 +172,7 @@ To see the queries that you have run in the current session, open the Query Hist
 
 The Query History contains information including the date and time when the query was run, the name of the query, the database on which it was run, and how long it took to run the query.
 To customize the information that is displayed, right-click an entry and select **Rename**.
+You can also filter the Query History view by language using the language selector. For more information, see ":ref:`Filtering databases and queries by language <filtering-databases-and-queries-by-language>`."
 
 Click an entry to display the corresponding results in the Query Results view, and double-click
 to display the query itself in the editor (or right-click and select **View Query**).

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

@@ -39,7 +39,7 @@ Changing the labels of query history items
 
 The query history **Format** setting controls how the extension lists queries in the query history. By default, each item has a label with the following format::
     
-    %q on %d - %s, %r result count [%t]
+    %q on %d - %s %r [%t]
 
 - ``%q`` is the query name
 - ``%d`` is the database name
@@ -107,6 +107,19 @@ You can also edit the items shown in the Variant Analysis Repositories panel by
 
 You can change the items shown in the panel or add new items by directly editing this file.
 
+Configuring settings for adding databases
+------------------------------------------------
+
+To automatically add database source folders to your workspace, you can enable the **Adding Databases > Add Database Source to Workspace** setting.
+
+This setting is disabled by default. You may want to enable the setting if you regularly browse the source code of databases, for example to view the abstract syntax tree of the code. For more information, see ":ref:`Exploring the structure of your source code <exploring-the-structure-of-your-source-code>`."
+
+.. pull-quote:: Note
+
+   If you are in a single-folder workspace, adding database source folders will cause the workspace to reload as a multi-root workspace. This may cause query history and database lists to be reset.
+
+   Before enabling this setting, we recommend that you save your workspace as a multi-root workspace. For more information, see "`Multi-root Workspaces <https://code.visualstudio.com/docs/editor/multi-root-workspaces>`__" in the Visual Studio Code help.
+
 Configuring settings for testing queries locally
 ------------------------------------------------
 

docs/codeql/codeql-for-visual-studio-code/exploring-the-structure-of-your-source-code.rst

@@ -18,20 +18,26 @@ CodeQL for Visual Studio Code contains an AST viewer. The viewer consists of a g
 Viewing the abstract syntax tree of a source file
 --------------------------------------------------
 
-1. Open a source file from a CodeQL database. For example, you can navigate to a source file in the File Explorer.
+1. Open the CodeQL Databases view and right-click the database that you want to explore. Click **Add Database Source to Workspace**.
+
+   .. image:: ../images/codeql-for-visual-studio-code/add-database-source-to-workspace.png
+      :width: 350
+      :alt: Add database source to workspace
+
+2. Navigate to a CodeQL database's source file in the File Explorer.
 
    .. image:: ../images/codeql-for-visual-studio-code/open-source-file.png
       :width: 350
       :alt: Open a source file
 
-2. Run **CodeQL: View AST** from the Command Palette. This runs a CodeQL query (usually called ``printAST.ql``) over the active file, which may take a few seconds.
+3. Run **CodeQL: View AST** from the Command Palette. This runs a CodeQL query (usually called ``printAST.ql``) over the active file, which may take a few seconds.
    
    .. pull-quote:: Note
 
       If you don't have an appropriate ``printAST.ql`` query in your workspace, the **CodeQL: View AST** command won't work. To fix this, you can update your copy of the `CodeQL <https://github.com/github/codeql>`__ repository from ``main``. If you do this, you may need to upgrade your databases. Also, query caches may be discarded and your next query runs could be slower.
 
-3. Once the query has run, the AST viewer displays the structure of the source file.
-4. To see the nested structure, click the arrows and expand the nodes.
+4. Once the query has run, the AST viewer displays the structure of the source file.
+5. To see the nested structure, click the arrows and expand the nodes.
 
    .. image:: ../images/codeql-for-visual-studio-code/explore-ast.png
       :alt: Explore the AST

docs/codeql/codeql-for-visual-studio-code/using-the-codeql-model-editor.rst

@@ -119,8 +119,8 @@ Testing CodeQL model packs
 
 You can test any CodeQL model packs you create in VS Code by toggling the "use model packs" setting on and off. This method works for both databases and for variant analysis repositories.
 
-- To run queries on a CodeQL database with any model packs that are stored within the ``.github/codeql/extensions`` directory of the workspace, update your ``settings.json`` file with: ``"codeQL.runningQueries.useModelPacks": all,``
-- To run queries on a CodeQL database without using model packs, update your ``settings.json`` file with: ``"codeQL.runningQueries.useModelPacks": none,``
+- To run queries on a CodeQL database with any model packs that are stored within the ``.github/codeql/extensions`` directory of the workspace, update your ``settings.json`` file with: ``"codeQL.runningQueries.useExtensionPacks": "all",``
+- To run queries on a CodeQL database without using model packs, update your ``settings.json`` file with: ``"codeQL.runningQueries.useExtensionPacks": "none",``
 
 If your model is working well, you should see a difference in the results of the two different runs. If you don't see any differences in results, you may need to introduce a known bug to verify that the model behaves as expected.
 

docs/codeql/codeql-language-guides/extensible-predicates.rst

@@ -22,7 +22,7 @@ About extensible predicates
 
 At a high level, there are two main components to using data extensions. The query writer defines one or more extensible predicates in their query libraries. CLI and code scanning users who want to augment these predicates supply one or more extension files whose data gets injected into the extensible predicate during evaluation. The extension files are either stored directly in the repository where the codebase to be analyzed is hosted, or downloaded as CodeQL model packs.
 
-This example of an extensible predicate for a source is taken from the core Java libraries https://github.com/github/codeql/blob/main/java/ql/lib/semmle/code/java/dataflow/ExternalFlowExtensions.qll#L8-L11
+This example of an extensible predicate for a source is taken from the core Java libraries https://github.com/github/codeql/blob/main/java/ql/lib/semmle/code/java/dataflow/internal/ExternalFlowExtensions.qll#L8-L11
 
 .. code-block:: ql
 

docs/codeql/reusables/supported-frameworks.rst

@@ -11,8 +11,8 @@ and the CodeQL library pack ``codeql/cpp-all`` (`changelog <https://github.com/g
    :widths: auto
 
    Name, Category
-   `Bloomberg Standard Library <https://github.com/bloomberg/bde>`__, Utility library
    `Berkeley socket API library <https://en.wikipedia.org/wiki/Berkeley_sockets#Socket_API_functions>`__, Network communicator
+   `Bloomberg Standard Library <https://github.com/bloomberg/bde>`__, Utility library
    string.h, String library
 
 C# built-in support
@@ -82,8 +82,8 @@ and the CodeQL library pack ``codeql/go-all`` (`changelog <https://github.com/gi
    `nhooyr.io/websocket <http://nhooyr.io/websocket>`_, Network communicator
    `protobuf <https://pkg.go.dev/google.golang.org/protobuf>`_, Serialization
    `Revel <http://revel.github.io/>`_, Web framework
-   `sqlx <http://jmoiron.github.io/sqlx/>`_, Database
    `SendGrid <https://github.com/sendgrid/sendgrid-go>`_, Email library
+   `sqlx <http://jmoiron.github.io/sqlx/>`_, Database
    `Squirrel <https://github.com/Masterminds/squirrel>`_, Database
    `ws <https://github.com/gobwas/ws>`_, Network communicator
    `xmlpath <https://gopkg.in/xmlpath.v2>`_, XPath library
@@ -108,20 +108,20 @@ and the CodeQL library pack ``codeql/java-all`` (`changelog <https://github.com/
    :widths: auto
 
    Name, Category
-   Apache Commons Lang, Utility library
    Apache Commons Collections, Data structure utility library
+   Apache Commons Lang, Utility library
    Apache HTTP components, Network communicator
    Guava, Utility and collections library
    Hibernate, Database
    iBatis / MyBatis, Database
    Jackson, Serialization
-   JSON-java, Serialization
    Java Persistence API (JPA), Database
    JaxRS, Jakarta EE API specification
    JDBC, Database
-   Protobuf, Serialization
+   JSON-java, Serialization
    Kryo deserialization, Serialization
    Project Lombok, Utility library
+   Protobuf, Serialization
    SnakeYaml, Serialization
    Spring JDBC, Database
    Spring MVC, Web application framework
@@ -160,8 +160,8 @@ and the CodeQL library pack ``codeql/javascript-all`` (`changelog <https://githu
    mssql, Database
    mustache, templating language
    mysql, Database
-   node, Runtime environment
    nest.js, Server
+   node, Runtime environment
    nunjucks, templating language
    postgres, Database
    ramda, Utility library
@@ -197,12 +197,12 @@ and the CodeQL library pack ``codeql/python-all`` (`changelog <https://github.co
    djangorestframework, Web framework
    FastAPI, Web framework
    Flask, Web framework
+   Flask-Admin, Web framework
    Tornado, Web framework
    Twisted, Web framework
-   Flask-Admin, Web framework
    starlette, Asynchronous Server Gateway Interface (ASGI)
-   python-ldap, Lightweight Directory Access Protocol (LDAP)
    ldap3, Lightweight Directory Access Protocol (LDAP)
+   python-ldap, Lightweight Directory Access Protocol (LDAP)
    httpx, HTTP client
    pycurl, HTTP client
    requests, HTTP client
@@ -230,8 +230,8 @@ and the CodeQL library pack ``codeql/python-all`` (`changelog <https://github.co
    cassandra-driver, Database
    clickhouse-driver, Database
    cx_Oracle, Database
-   mysql-connector-python, Database
    mysql-connector, Database
+   mysql-connector-python, Database
    MySQL-python, Database
    mysqlclient, Database
    oracledb, Database
@@ -310,5 +310,5 @@ and the CodeQL library pack ``codeql/swift-all`` (`changelog <https://github.com
    `RNCryptor <https://github.com/RNCryptor/RNCryptor>`__, Cryptography library
    `SQLite3 <https://sqlite.org/index.html>`__, Database
    `SQLite.swift <https://github.com/stephencelis/SQLite.swift>`__, Database
-   `WebKit <https://developer.apple.com/documentation/webkit>`__, User interface library
    `UIKit <https://developer.apple.com/documentation/uikit>`__, User interface library
+   `WebKit <https://developer.apple.com/documentation/webkit>`__, User interface library

docs/codeql/writing-codeql-queries/metadata-for-codeql-queries.rst

@@ -49,7 +49,7 @@ The following properties are supported by all query files:
 |                       | | ``warning``             |                                                                                                                                                                                                                                                                                                                                                                       |
 |                       | | ``recommendation``      |                                                                                                                                                                                                                                                                                                                                                                       |
 +-----------------------+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``@security-severity``| ``<score>``               | Defines the level of severity, between 0.0 and 10.0, for queries with ``@tags security``. For more information about calculating ``@security-severity``, see the `GitHub changelog <https://github.blog/changelog/2021-07-19-codeql-code-scanning-new-severity-levels-for-security-alerts/>`__.                                                                       |
+| ``@security-severity``| ``<score>``               | Defines the level of severity, between 0.0 and 10.0, for queries with ``@tags security``. For more information about calculating ``@security-severity``, see the `GitHub changelog <https://gh.io/2021-07-19-codeql-security-severity>`__.                                                                                                                            |
 +-----------------------+---------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
 Example

docs/ql-libraries/dataflow/dataflow.md

@@ -618,12 +618,31 @@ impact on performance for large databases.
 
 ### Hidden nodes
 
-Certain synthetic nodes can be hidden to exclude them from occurring in path
-explanations. This is done through the following predicate:
+Certain synthetic nodes are hidden by default to prevent them from occurring in
+path explanations. Which nodes are included in this is determined by the
+following predicate:
+
 ```ql
 predicate nodeIsHidden(Node n)
 ```
 
+To include these nodes in path explanations, add the following to a
+configuration:
+
+```ql
+predicate includeHiddenNodes() { any() }
+```
+
+### Show more nodes in path explanations
+
+Path explanations skip over most nodes and only show the important nodes in a
+path. To ensure that particular nodes are shown, define the following predicate
+in a configuration:
+
+```ql
+predicate neverSkip(Node n)
+```
+
 ### Unreachable nodes
 
 Consider: