Merge branch 'main' into aeisenberg/update-analyzing-databases

docs/codeql/codeql-cli/about-codeql-workspaces.rst

@@ -1,6 +1,6 @@
 .. _about-codeql-workspaces:
 
-About CodeQL Workspaces
+About CodeQL workspaces
 =======================
 
 .. include:: ../reusables/beta-note-package-management.rst
@@ -12,16 +12,17 @@ The main benefit of a CodeQL workspace is that it makes it easier for you to dev
 In most cases, you should store the CodeQL workspace and the CodeQL packs contained in it in one git repository. This makes it easier to share your CodeQL development environment.
 
 The ``codeql-workspace.yml`` file
-----------------------------------
+---------------------------------
 
-A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block.
+A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally ``ignore`` and ``registries`` blocks.
 
 * The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace.
 * The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace.
+* The ``registries`` block contains a list of GHES URLs and package patterns that control which container registry is used for publishing CodeQL packs. For more information, see :ref:`Working with CodeQL packs on GitHub Enterprise Server <working-with-codeql-packs-on-ghes>`.
 
 Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see "`@actions/glob <https://github.com/actions/toolkit/tree/main/packages/glob#patterns>`__ ."
 
-For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all the CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory:
+For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all the CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory. The ``registries`` block specifies that ``codeql/*`` packs should be downloaded from https://ghcr.io/v2/, which is GitHub's default container registry. All other packs should be downloaded from and published to the regsitry at ``GHE_HOSTNAME``.
 
 .. code-block:: yaml
 
@@ -30,6 +31,13 @@ For example, the following ``codeql-workspace.yml`` file defines a workspace tha
     ignore:
       - "*/codeql-packs/**/experimental/**/qlpack.yml"
 
+    registries:
+     - packages: 'codeql/*'
+       url: https://ghcr.io/v2/
+
+     - packages: '*'
+       url: https://containers.GHE_HOSTNAME/v2/
+
 To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that you expect, run the ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace.
 
 .. _source-dependencies:

docs/codeql/codeql-cli/about-ql-packs.rst

@@ -0,0 +1,12 @@
+.. _about-ql-packs:
+
+About QL packs
+==============
+
+This page has been moved to ":doc:`About CodeQL packs <about-codeql-packs>`."
+
+
+.. toctree::
+   :hidden:
+
+   about-ql-packs

docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst

@@ -219,6 +219,56 @@ When the analysis has finished, a SARIF results file is generated. Specifying ``
 that the results are formatted according to the most recent SARIF specification
 supported by CodeQL.
 
+.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
+
+Running a subset of queries in a CodeQL pack
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are using CodeQL CLI v2.8.1 or later, you can include a path at the end of a pack specification to run a subset of queries inside the pack. This applies to any command that locates or runs queries within a pack.
+
+The complete way to specify a set of queries is in the form ``scope/name@range:path``, where:
+
+- ``scope/name`` is the qualified name of a CodeQL pack.
+- ``range`` is a `semver range <https://docs.npmjs.com/cli/v6/using-npm/semver#ranges>`_.
+- ``path`` is a file system path to a single query, a directory containing queries, or a query suite file.
+
+When you specify a ``scope/name``, the ``range`` and ``path`` are
+optional. If you omit a ``range`` then the latest version of the
+specified pack is used. If you omit a ``path`` then the default query suite
+of the specified pack is used.
+
+The ``path`` can be one of a ``*.ql`` query file, a directory
+containing one or more queries, or a ``.qls`` query suite file. If
+you omit a pack name, then you must provide a ``path``,
+which will be interpreted relative to the working directory
+of the current process.
+
+If you specify a ``scope/name`` and ``path``, then the ``path`` cannot
+be absolute. It is considered relative to the root of the CodeQL
+pack.
+
+To analyze a database using all queries in the `experimental/Security` folder within the `codeql/cpp-queries` CodeQL pack you can use::
+
+    codeql database analyze --format=sarif-latest --output=results <db> \
+        codeql/cpp-queries:experimental/Security
+
+To run the `RedundantNullCheckParam.ql` query in the `codeql/cpp-queries` CodeQL pack use::
+
+    codeql database analyze --format=sarif-latest --output=results <db> \
+        'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql'
+
+To analyze your database using the `cpp-security-and-quality.qls` query suite from a version of the `codeql/cpp-queries` CodeQL pack that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen) you can use::
+
+    codeql database analyze --format=sarif-latest --output=results <db> \
+       'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls'
+
+If you need to reference a query file, directory, or suite whose path contains a literal `@` or `:`, you can prefix the query specification with `path:` like so::
+
+    codeql database analyze --format=sarif-latest --output=results <db> \
+        path:C:/Users/ci/workspace@2/security/query.ql
+
+For more information about CodeQL packs, see :doc:`About CodeQL Packs <about-codeql-packs>`.
+
 Running query suites
 ~~~~~~~~~~~~~~~~~~~~
 

docs/codeql/codeql-cli/creating-codeql-databases.rst

@@ -11,11 +11,11 @@ or download them from GitHub.com.
 CodeQL analysis relies on extracting relational data from your code, and
 using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL
 databases contain all of the important information about a codebase, which can
-be analyzed by executing CodeQL queries against it. GitHub creates and 
+be analyzed by executing CodeQL queries against it. GitHub creates and
 stores CodeQL databases for a large number of open-source projects. For more information,
 see ":ref:`Downloading CodeQL databases from GitHub.com <downloading-codeql-databases-from-github-com>`."
 
-You can also create CodeQL databases yourself using the CodeQL CLI. 
+You can also create CodeQL databases yourself using the CodeQL CLI.
 Before you generate a CodeQL database, you need to:
 
 - Install and set up the CodeQL CLI. For more information, see
@@ -214,12 +214,12 @@ commands that you can specify for compiled languages.
 
      codeql database create cpp-database --language=cpp --command=make
 
-- C# project built using ``dotnet build``::
+- C# project built using ``dotnet build``:
 
-     It is a good idea to add `/t:rebuild` to ensure that all code will be built, or do a
-     prior `dotnet clean` (code that is not built will not be included in the CodeQL database):
+  It is a good idea to add `/t:rebuild` to ensure that all code will be built, or do a
+  prior `dotnet clean` (code that is not built will not be included in the CodeQL database)::
 
-     codeql database create csharp-database --language=csharp --command='dotnet build /t:rebuild' 
+     codeql database create csharp-database --language=csharp --command='dotnet build /t:rebuild'
 
 - Go project built using the ``CODEQL_EXTRACTOR_GO_BUILD_TRACING=on`` environment variable::
 
@@ -391,9 +391,9 @@ Downloading databases from GitHub.com
 -------------------------------------
 
 .. include:: ../reusables/download-github-database.rst
-   
+
 Before running an analysis with the CodeQL CLI, you must unzip the databases.
-   
+
 Further reading
 ---------------
 

docs/codeql/codeql-cli/creating-codeql-query-suites.rst

@@ -118,8 +118,10 @@ typically a query metadata property. The value can be:
 
 To match a constraint, a metadata value must match one of the strings or
 regular expressions. When there is more than one metadata key, each key must be matched.
-For more information about query metadata properties, see ":ref:`Metadata for CodeQL queries
-<metadata-for-codeql-queries>`."
+The standard metadata keys available to match on are: ``description``, ``id``, ``kind``,
+``name``, ``tags``, ``precision``, and ``problem.severity``.
+For more information about query metadata properties, see
+":ref:`Metadata for CodeQL queries <metadata-for-codeql-queries>`."
 
 In addition to metadata tags, the keys in the constraint block can also be:
 
@@ -131,8 +133,37 @@ In addition to metadata tags, the keys in the constraint block can also be:
 - ``tags contain all``---each of the given match strings must match one of the
   components of the ``@tags`` metadata property.
 
-Examples
-~~~~~~~~
+Examples of filtering which queries are run
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A common use case is to create a query suite that runs all queries in a CodeQL pack,
+except for a few specific queries that the user does not want to run. In general, we
+recommend filtering on the query ``id``, which is a unique and stable identifier for
+each query. The following three query suite definitions are semantically identical and
+filter by the query ``id``:
+
+This filter matches all the queries in the default suite of ``codeql/cpp-queries``, except for the two queries with the excluded identifiers::
+
+   - qlpack: codeql/cpp-queries
+   - exclude:
+       id:
+         - cpp/cleartext-transmission
+         - cpp/cleartext-storage-file
+
+In this example, a separate ``exclude`` instruction is used for each query::
+
+   - qlpack: codeql/cpp-queries
+   - exclude:
+       id: cpp/cleartext-transmission
+   - exclude:
+       id: cpp/cleartext-storage-file
+
+In this example, a regular expression excludes the same two queries. It would also exclude any future queries added to the suite with identifiers that begin: ``cpp/cleartext-``::
+
+   - qlpack: codeql/cpp-queries
+   - exclude:
+       id:
+         - /^cpp\/cleartext-.*/
 
 To define a suite that selects all queries in the default suite of the
 ``codeql/cpp-queries`` CodeQL pack, and then refines them to only include
@@ -150,6 +181,15 @@ and ``@precision high`` from the ``my-custom-queries`` directory, use::
        kind: problem
        precision: very-high
 
+Note that the following query suite definition behaves differently from the definition above. This definition selects queries that are ``@kind problem`` *or*
+are ``@precision very-high``::
+
+   - queries: my-custom-queries
+   - include:
+       kind: problem
+   - include:
+       precision: very-high
+
 To create a suite that selects all queries with ``@kind problem`` from the
 ``my-custom-queries`` directory except those with ``@problem.severity
 recommendation``, use::
@@ -172,6 +212,15 @@ use::
        - high
        - very-high
 
+.. pull-quote::
+
+    Tip
+
+    You can use the ``codeql resolve queries /path/to/suite.qls`` command to see
+    which queries are selected by a query suite definition. For more information,
+    see the `resolve queries <../../codeql-cli/manual/resolve-queries>`__
+    reference documentation.
+
 Reusing existing query suite definitions
 -----------------------------------------
 
@@ -208,14 +257,8 @@ Existing query suite definitions can be reused by specifying:
   conditions, saved in a ``.yml`` file, to multiple query definitions. For more
   information, see the `example <#example>`__ below.
 
-- An ``eval`` instruction---performs the same function as an ``import``
-  instruction, but takes a full suite definition as the argument, rather than the
-  path to a ``.qls`` file on disk.
-
-To see what queries are included in a query suite, you can run the ``codeql resolve queries my-suite.qls`` command.
-
-Example
-~~~~~~~
+Reusability Examples
+~~~~~~~~~~~~~~~~~~~~
 
 To use the same conditions in multiple query suite definitions, create a
 separate ``.yml`` file containing your instructions. For example, save the
@@ -252,6 +295,30 @@ instruction::
     from: my-org/my-custom-instructions
     version: ^1.2.3 # optional
 
+A common use case for an ``import`` instruction is to apply a further filter to queries from another
+query suite. For example, this suite will further filter the ``cpp-security-and-quality`` suite
+and exclude ``low`` and ``medium`` precision queries::
+
+    - import: codeql-suites/cpp-security-and-quality.qls
+      from: codeql/cpp-queries
+    - exclude:
+        precision:
+          - low
+          - medium
+
+If you want to ``include`` queries imported from another suite, the syntax is a little different::
+
+    - import: codeql-suites/cpp-security-and-quality.qls
+      from: codeql/cpp-queries
+    - exclude: {}
+    - include:
+        precision:
+          - very-high
+          - high
+
+Notice the empty ``exclude`` instruction. This is required to ensure that the subsequent ``include``
+instruction is able to filter queries from the imported suite.
+
 Naming a query suite
 --------------------
 

docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst

@@ -74,6 +74,8 @@ The ``analyze`` command will run the default suite of any specified CodeQL packs
 
    codeql <database> analyze <scope>/<pack> <scope>/<other-pack>
 
+.. _working-with-codeql-packs-on-ghes:
+
 Working with CodeQL packs on GitHub Enterprise Server
 -----------------------------------------------------
 
@@ -91,13 +93,17 @@ For example, the following ``qlconfig.yml`` file associates all packs with the C
 .. code-block:: yaml
 
    registries:
-   - packages: 'codeql/*'
+   - packages:
+     - 'codeql/*'
+     - 'other-org/*'
      url: https://ghcr.io/v2/
    - packages: '*'
      url: https://containers.GHE_HOSTNAME/v2/
 
 The CodeQL CLI will determine which registry to use for a given package name by finding the first item in the ``registries`` list with a ``packages`` property that matches that package name.
-This means that you'll generally want to define the most specific package name patterns first.
+This means that you'll generally want to define the most specific package name patterns first. The ``packages`` property may be a single package name, a glob pattern, or a YAML list of package names and glob patterns.
+
+The ``registries`` list can also be placed inside of a ``codeql-workspace.yml`` file. Doing so will allow you to define the registries to be used within a specific workspace, so that it can be shared amongst other CodeQL users of the workspace. The ``registries`` list in the ``codeql-workspace.yml`` will be merged with and take precedence over the list in the global ``qlconfig.yml``. For more information about ``codeql-workspace.yml``, see :ref:`About CodeQL workspaces <about-codeql-workspaces>`.
 
 You can now use ``codeql pack publish``, ``codeql pack download``, and ``codeql database analyze`` to manage packs on GitHub Enterprise Server.