From 38909076ea8bdbacd3134f6b09f824293adbaead Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 19 Aug 2022 11:47:22 -0700 Subject: [PATCH] Update other articles to with details about packaging In this commit: - Replace _QL pack_ with _CodeQL pack_ - Replace `about-ql-pack` references with `about-codeql-packs` - Replace examples using `libraryPathDependencies with `dependencies` - Update some examples to specify the optional `version` field - Update description of query specifiers to note that a path within a pack is valid. --- docs/codeql/codeql-cli/about-codeql-packs.rst | 109 ++++++++++++----- ...nalyzing-databases-with-the-codeql-cli.rst | 26 ++-- .../codeql-cli/codeql-cli-reference.rst | 7 +- ...creating-and-working-with-codeql-packs.rst | 13 +- .../creating-codeql-query-suites.rst | 75 +++++++----- .../getting-started-with-the-codeql-cli.rst | 20 ++-- .../publishing-and-using-codeql-packs.rst | 3 +- .../codeql-cli/query-reference-files.rst | 22 ++-- ...options-in-a-codeql-configuration-file.rst | 10 +- .../codeql-cli/testing-custom-queries.rst | 111 +++++++++--------- .../codeql-cli/testing-query-help-files.rst | 42 +++---- ...ing-custom-queries-with-the-codeql-cli.rst | 19 ++- .../analyzing-your-projects.rst | 14 +-- ...etting-up-codeql-in-visual-studio-code.rst | 2 +- .../codeql-overview/codeql-glossary.rst | 52 ++++---- .../ql-language-specification.rst | 14 +-- .../about-codeql-queries.rst | 4 +- 17 files changed, 303 insertions(+), 240 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-packs.rst b/docs/codeql/codeql-cli/about-codeql-packs.rst index 4731f32d1e6..f8480bb92f1 100644 --- a/docs/codeql/codeql-cli/about-codeql-packs.rst +++ b/docs/codeql/codeql-cli/about-codeql-packs.rst @@ -3,6 +3,8 @@ About CodeQL packs ================== +.. include:: ../reusables/beta-note-package-management.rst + CodeQL packs are used to create, share, depend on, and run CodeQL queries and libraries. You can publish your own CodeQL packs and download packs created by others. CodeQL packs contain queries, library files, query suites, and metadata. There are two types of CodeQL packs: query packs and library packs. @@ -60,71 +62,114 @@ The following properties are supported in ``qlpack.yml`` files. - Required - Purpose * - ``name`` - - ``name: octo-org/security-queries`` + - .. code-block:: yaml + + name: octo-org/security-queries + - All packs - - The scope, where the CodeQL pack is published, and the name of the pack defined using alphanumeric characters and hyphens. It must be unique as CodeQL cannot differentiate between CodeQL packs with identical names. Use the pack name to specify queries to run using ``database analyze`` and to define dependencies between QL packs (see examples below). + - The scope, where the CodeQL pack is published, and the name of the pack defined using alphanumeric characters and hyphens. It must be unique as CodeQL cannot differentiate between CodeQL packs with identical names. Use the pack name to specify queries to run using ``database analyze`` and to define dependencies between CodeQL packs (see examples below). * - ``version`` - - ``version: 0.0.0`` + - .. code-block:: yaml + + version: 0.0.0 + - All packs - A semantic version for this CodeQL pack that must adhere to the `SemVer v2.0.0 specification `__. * - ``dependencies`` - .. code-block:: yaml - dependencies: - codeql/cpp-all: ^0.0.2 + dependencies: + codeql/cpp-all: ^0.0.2 - Packs that define CodeQL package dependencies on other packs - A map from pack references to the semantic version range that is compatible with this pack. Supported from CLI version v2.6.0 and onwards. * - ``defaultSuiteFile`` - - ``defaultSuiteFile: cpp-code-scanning.qls`` + - .. code-block:: yaml + + defaultSuiteFile: cpp-code-scanning.qls + - Packs that export a set of default queries to run - The path to a query suite file containing all of the queries that are run by default when this pack is passed to the ``codeql database analyze`` command. Supported from CLI version v2.6.0 and onwards. * - ``defaultSuite`` - .. code-block:: yaml - defaultSuite: - queries: . - exclude: - precision: medium + defaultSuite: + queries: . + exclude: + precision: medium - Packs that export a set of default queries to run - An inlined query suite containing all of the queries that are run by default when this pack is passed to the ``codeql database analyze`` command. Supported from CLI version v2.6.0 and onwards. * - ``library`` - - ``library: true`` + - .. code-block:: yaml + + library: true + - Library packs - A boolean value that indicates whether this pack is a library pack. Library packs do not contain queries and are not compiled. Query packs can ignore this field or explicitly set it to ``false``. * - ``suites`` - - ``suites: octo-org-query-suites`` + - .. code-block:: yaml + + suites: octo-org-query-suites + - Optional - - The path to a directory in the pack that contains the query suites you want to make known to the CLI, defined relative to the pack directory. QL pack users can run "well-known" suites stored in this directory by specifying the pack name, without providing their full path. This is not supported for CodeQL packs downloaded from a package registry. For more information about query suites, see ":doc:`Creating CodeQL query suites `." + - The path to a directory in the pack that contains the query suites you want to make known to the CLI, defined relative to the pack directory. CodeQL pack users can run "well-known" suites stored in this directory by specifying the pack name, without providing their full path. This is not supported for CodeQL packs downloaded from a package registry. For more information about query suites, see ":doc:`Creating CodeQL query suites `." * - ``extractor`` - - ``javascript`` + - .. code-block:: yaml + + extractor: javascript + - All test packs - The CodeQL language extractor to use when the CLI creates a database in the pack. For more information about testing queries, see ":doc:`Testing custom queries `." * - ``tests`` - - ``.`` - - Optional for test packs - - The path to a directory within the pack that contains tests, defined relative to the pack directory. Use ``.`` to specify the whole pack. Any queries in this directory are run as tests when ``test run`` is run with the ``--strict-test-discovery`` option. These queries are ignored by query suite definitions that use ``queries`` or ``qlpack`` instructions to ask for all queries in a particular pack. - * - ``dbscheme`` TODO: Internal only. Delete? - - ``semmlecode.python.dbscheme`` + - .. code-block:: yaml + + tests: . + + - Optional for test packs. Ignored for non-test packs. + - The path to a directory within the pack that contains tests, defined relative to the pack directory. Use ``.`` to specify the whole pack. Any queries in this directory are run as tests when ``test run`` is run with the ``--strict-test-discovery`` option. These queries are ignored by query suite definitions that use ``queries`` or ``qlpack`` instructions to ask for all queries in a particular pack. If this property is missing, then ``.`` is assumed. + * - ``dbscheme`` + - .. code-block:: yaml + + dbscheme: semmlecode.python.dbscheme + - Core language packs only - The path to the :ref:`database schema ` for all libraries and queries written for this CodeQL language (see example below). - * - ``upgrades`` TODO: Internal only. Delete? - - ``.`` + * - ``upgrades`` + - .. code-block:: yaml + + upgrades: . + - Core language packs only - The path to a directory within the pack that contains upgrade scripts, defined relative to the pack directory. The ``database upgrade`` action uses these scripts to upgrade databases that were created by an older version of an extractor or downgrade databases that were created by a newer version of the extractor so they're compatible with the current extractor (see `Upgrade scripts for a language <#upgrade-scripts-for-a-language>`__ below.) * - ``authors`` - - ``example@github.com`` + - .. code-block:: yaml + + authors: author1@github.com,author2@github.com + - All packs - Metadata that will be displayed on the packaging search page in the packages section of the account that the CodeQL pack is published to. - * - ``licenses`` - - ``(LGPL-2.1 AND MIT)`` + * - ``license`` + - .. code-block:: yaml + + license: MIT + - All packs - Metadata that will be displayed on the packaging search page in the packages section of the account that the CodeQL pack is published to. For a list of allowed licenses, see `SPDX License List `__ in the SPDX Specification. * - ``description`` - - ``Human-readable description of the contents of the CodeQL pack.`` + - .. code-block:: yaml + + description: Human-readable description of the contents of the CodeQL pack. + - All packs - Metadata that will be displayed on the packaging search page in the packages section of the account that the CodeQL pack is published to. + * - ``libraryPathDependencies`` + - .. code-block:: yaml + + libraryPathDependencies: codeql/javascript-all + + - Optional, deprecated + - Use the ``dependencies`` property instead. The names of any CodeQL packs that this CodeQL pack depends on, as a sequence. This gives the pack access to any libraries, database schema, and query suites defined in the dependency. .. _about-codeql-pack-lock: @@ -166,9 +211,9 @@ When you write custom queries or tests, you should save them in custom CodeQL pa CodeQL packs for custom queries ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A custom QL pack for queries must include a ``qlpack.yml`` file at the pack root, containing ``name``, ``version``, and ``dependencies`` properties. If the pack contains query suites, you can use the ``suites`` property to define their location. Query suites defined here are called "well-known" suites, and can be used on the command line by referring to their name only, rather than their full path. For more information about query suites, see ":doc:`Creating CodeQL query suites `." +A custom CodeQL pack for queries must include a ``qlpack.yml`` file at the pack root, containing ``name``, ``version``, and ``dependencies`` properties. If the pack contains query suites, you can use the ``suites`` property to define their location. Query suites defined here are called "well-known" suites, and can be used on the command line by referring to their name only, rather than their full path. For more information about query suites, see ":doc:`Creating CodeQL query suites `." -For example, a ``qlpack.yml`` file for a QL pack featuring custom C++ queries +For example, a ``qlpack.yml`` file for a CodeQL pack featuring custom C++ queries and libraries may contain: .. code-block:: yaml @@ -179,7 +224,7 @@ and libraries may contain: codeql/cpp-all: ^0.1.2 suites: my-custom-suites -where ``codeql/cpp-all`` is the name of the QL pack for C/C++ analysis included in the CodeQL repository. The version range ``^0.1.2`` indicates that this pack is compatible with all versions of ``codeql/cpp-all`` that are greater than or equal to ``0.1.2`` and less than ``0.2.0``. +where ``codeql/cpp-all`` is the name of the CodeQL pack for C/C++ analysis included in the CodeQL repository. The version range ``^0.1.2`` indicates that this pack is compatible with all versions of ``codeql/cpp-all`` that are greater than or equal to ``0.1.2`` and less than ``0.2.0``. CodeQL packs for custom test files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -249,10 +294,10 @@ contains: This query pack depends on ``codeql/cpp-all`` and ``codeql/suite-helpers``. Since these dependencies are resolved from source, it does not matter what version of the CodeQL pack they are compatible with. -Tests for the core QL pack -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Tests for the core CodeQL pack +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``qlpack.yml`` file for the tests for the core QL packs use the following properties: ``name``, ``dependencies``, ``extractor``, ``tests``. +The ``qlpack.yml`` file for the tests for the core CodeQL packs use the following properties: ``name``, ``dependencies``, ``extractor``, ``tests``. For example, the ``qlpack.yml`` file for `C/C++ analysis tests `__ @@ -267,4 +312,4 @@ contains: extractor: cpp tests: . -The ``dependencies`` property always specifies the core QL pack. The ``extractor`` property specifies that all the tests will use the same C++ extractor to create the database for the tests. The ``tests`` property specifies the location of the tests. In this case, the tests are in the root folder (and all sub-folders) of the pack. Note that there is no ``version`` property for the tests pack since this pack is not meant to be published. +The ``dependencies`` property always specifies the core CodeQL pack. The ``extractor`` property specifies that all the tests will use the same C++ extractor to create the database for the tests. The ``tests`` property specifies the location of the tests. In this case, the tests are in the root folder (and all sub-folders) of the pack. Note that there is no ``version`` property for the tests pack since this pack is not meant to be published. diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index bb428f2c00d..7ff8a393017 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -58,9 +58,10 @@ You can also specify: - a path to a directory containing query files - a path to a query suite file - the name of a CodeQL query pack - If omitted, the default query suite for the language - of the database being analyzed will be used. For more information, see the - :ref:`examples ` below. + - with an optional a version range + - with an optional path to a query, directory, or query suite inside the pack + + If omitted, the default query suite for the language of the database being analyzed will be used. For more information, see the :ref:`examples ` below. - ``--sarif-category``: an identifying category for the results. Used when you want to upload more than one set of results for a commit. @@ -192,10 +193,10 @@ For more information, see `Analyzing a CodeQL database `__ in the GitHub documentation. CodeQL query suites are ``.qls`` files that use directives to select queries to run -based on certain metadata properties. The standard QL packs have metadata that specify +based on certain metadata properties. The standard CodeQL packs have metadata that specify the location of the query suites used by code scanning, so the CodeQL CLI knows where to find these suite files automatically, and you don't have to specify the full path on the command line. -For more information, see ":ref:`About QL packs `." +For more information, see ":ref:`About CodeQL packs `." The standard query suites are stored at the following paths in the CodeQL repository:: @@ -239,18 +240,23 @@ recursively, so any queries contained in subfolders will also be executed. Important - You shouldn't specify the root of a :doc:`QL pack - ` when executing ``database analyze`` + You shouldn't specify the root of a :doc:`CodeQL pack + ` when executing ``database analyze`` as it contains some special queries that aren't designed to be used with the command. Rather, to run a wide range of useful queries, run one of the LGTM.com query suites. -For example, to execute all Python queries contained in the ``Functions`` -directory you would run:: +For example, to execute all Python queries contained in the ``Functions`` directory you would run:: codeql database analyze ../ql/python/ql/src/Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif -A SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures +If you do not have the codeql repository checked out, you can execute the same queries by specifying the query pack name and the path to the queries: + + codeql database analyze --download codeql/python-queries:Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif + +Use the ``--download`` flag to download the query pack if it isn't yet available locally. + +After evaluating, a SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures that the results are formatted according to the most recent SARIF specification supported by CodeQL. diff --git a/docs/codeql/codeql-cli/codeql-cli-reference.rst b/docs/codeql/codeql-cli/codeql-cli-reference.rst index e0ffdeaaa36..fadf5ada79f 100644 --- a/docs/codeql/codeql-cli/codeql-cli-reference.rst +++ b/docs/codeql/codeql-cli/codeql-cli-reference.rst @@ -3,22 +3,19 @@ CodeQL CLI reference ==================== -Learn more about the files you can use when running CodeQL processes and the results format and exit codes that CodeQL generates. +Learn more about the files you can use when running CodeQL processes and the results format and exit codes that CodeQL generates. .. toctree:: :titlesonly: :hidden: about-codeql-packs - about-ql-packs query-reference-files sarif-output exit-codes extractor-options -- :doc:`About CodeQL packs `: CodeQL packs are created with the CodeQL CLI and are used to create, depend on, publish, and run CodeQL queries and libraries. -- :doc:`About QL packs `: QL packs are used to organize the files used in CodeQL analysis. They - contain queries, library files, query suites, and important metadata. +- :doc:`About CodeQL packs `: CodeQL packs are created with the CodeQL CLI and are used to create, depend on, publish, and run CodeQL queries, libraries, and query suites. - :doc:`Query reference files `: A query reference file is text file that defines the location of one query to test. - :doc:`SARIF output `: CodeQL supports SARIF as an output format for sharing static analysis results. - :doc:`Exit codes `: The CodeQL CLI reports the status of each command it runs as an exit code. diff --git a/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst b/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst index da7a0872803..a5f268c2c74 100644 --- a/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst +++ b/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst @@ -34,24 +34,26 @@ You must specify: The ``codeql pack init`` command creates the directory structure and configuration files for a CodeQL pack. By default, the command creates a query pack. If you want to create a library pack, you must edit the ``qlpack.yml`` file to explicitly declare the file as a library pack by including the ``library:true`` property. -Modifying an existing QL pack to create a CodeQL pack ------------------------------------------------------ +Modifying an existing legacy CodeQL pack to create a CodeQL pack +---------------------------------------------------------------- + If you already have a ``qlpack.yml`` file, you can edit it manually to convert it into a CodeQL pack. #. Edit the ``name`` property so that it matches the format ``/``, where ```` is the name of the GitHub organization or user account that you will publish to. #. In the ``qlpack.yml`` file, include a ``version`` property with a semver identifier, as well as an optional ``dependencies`` block. +#. Migrate the list of dependencies in ``libraryPathDependencies`` to the ``dependencies`` block. Specify the version range for each dependency. If the range is unimportant, or you are unsure of compatibility, you can specify ``"*"``, which indicates that any version is acceptable, with will default to the latest version when ``codeql pack install`` is run. For more information about the properties, see ":ref:`About CodeQL packs `." Adding and installing dependencies to a CodeQL pack --------------------------------------------------- -You can add dependencies on CodeQL packs using the command ``codeql pack add``. You must specify the scope, name, and version range. +You can add dependencies on CodeQL packs using the command ``codeql pack add``. You must specify the scope, name, and (optionally) a compatible version range. :: codeql pack add /@x.x.x / -The version range is optional. If you leave off the version range, the latest version will be added. Otherwise, the latest version that satisfies the requested range will be added. +If you leave off the version range, the latest version will be added. Otherwise, the latest version that satisfies the requested range will be added. This command updates the ``qlpack.yml`` file with the requested dependencies and downloads them into the package cache. Please note that this command will reformat the file and remove all comments. @@ -67,7 +69,8 @@ This command downloads all dependencies to the shared cache on the local disk. Note - Running the ``codeql pack add`` and ``codeql pack install`` commands will generate or update the ``qlpack.lock.yml`` file. This file should be checked-in to version control. The ``qlpack.lock.yml`` file contains the precise version numbers used by the pack. + Running the ``codeql pack add`` and ``codeql pack install`` commands will generate or update the ``codeql-pack.lock.yml`` file. This file should be checked-in to version control. The ``codeql-pack.lock.yml`` file contains the precise version numbers used by the pack. + For more information, see ":ref:`About codeql-pack.lock.yml files `." .. pull-quote:: diff --git a/docs/codeql/codeql-cli/creating-codeql-query-suites.rst b/docs/codeql/codeql-cli/creating-codeql-query-suites.rst index ceaa104f4d0..a4e291ce168 100644 --- a/docs/codeql/codeql-cli/creating-codeql-query-suites.rst +++ b/docs/codeql/codeql-cli/creating-codeql-query-suites.rst @@ -4,7 +4,7 @@ Creating CodeQL query suites ============================ CodeQL query suites provide a way of selecting queries, based on their -filename, location on disk or in a QL pack, or metadata properties. +filename, location on disk or in a CodeQL pack, or metadata properties. Create query suites for the queries that you want to frequently use in your CodeQL analyses. @@ -19,7 +19,7 @@ suite definition have been executed, the result is a set of selected queries. .. pull-quote:: Note Any custom queries that you want to add to a query suite must be in a :doc:`QL - pack ` and contain the correct query metadata. + pack ` and contain the correct query metadata. For more information, see ":doc:`Using custom queries with the CodeQL CLI `." @@ -35,7 +35,7 @@ queries using: - query: - The argument must be one or more file paths, relative to the QL pack containing + The argument must be one or more file paths, relative to the CodeQL pack containing the suite definition. - A ``queries`` instruction---tells CodeQL to recursively scan a directory @@ -43,22 +43,30 @@ queries using: - queries: - The path of the directory must be relative to the root of the QL pack that + The path of the directory must be relative to the root of the CodeQL pack that contains the suite definition file. To find the queries relative to a - different QL pack, add a ``from`` field:: + different CodeQL pack, add a ``from`` field:: - queries: from: + version: ^x.y.z + + The ``version`` field is optional and specifies a range of compatible versions of this CodeQL pack. + If the version is excluded, then the most recent version of the pack is used. - A ``qlpack`` instruction---tells CodeQL to resolve queries in the default suite of the - named QL pack:: + named CodeQL pack:: - qlpack: + version: ^x.y.z The default suite of a query pack includes a recommended set of queries inside of that query pack. Not all query packs have a default suite. If the given query pack does not define a default suite, the `qlpack` instruction will resolve to all of the queries within the pack. + The ``version`` field is optional and specifies a range of compatible versions of this CodeQL pack. + If the version is excluded, then the most recent version of the pack is used. + .. pull-quote:: Note When pathnames appear in query suite definitions, they must always @@ -68,7 +76,7 @@ queries using: You must add at least one ``query``, ``queries``, or ``qlpack`` instruction to your suite definition, otherwise no queries will be selected. If the suite contains no further instructions, all the queries found from the list of files, -in the given directory, or in the named QL pack are selected. If there are further +in the given directory, or in the named CodeQL pack are selected. If there are further filtering instructions, only queries that match the constraints imposed by those instructions will be selected. @@ -117,7 +125,7 @@ In addition to metadata tags, the keys in the constraint block can also be: - ``query filename``---matches on the last path component of the query file name. - ``query path``---matches on the path to the query file relative to its - enclosing QL pack. + enclosing CodeQL pack. - ``tags contain``---one of the given match strings must match one of the space-separated components of the value of the ``@tags`` metadata property. - ``tags contain all``---each of the given match strings must match one of the @@ -127,7 +135,7 @@ Examples ~~~~~~~~ To define a suite that selects all queries in the default suite of the -``codeql/cpp-queries`` QL pack, and then refines them to only include +``codeql/cpp-queries`` CodeQL pack, and then refines them to only include security queries, use:: - qlpack: codeql/cpp-queries @@ -153,7 +161,7 @@ recommendation``, use:: problem.severity: recommendation To create a suite that selects all queries with ``@tag security`` and -``@problem.severity high`` or ``very-high`` from the ``codeql/cpp-queries`` QL pack, +``@problem.severity high`` or ``very-high`` from the ``codeql/cpp-queries`` CodeQL pack, use:: - queries: . @@ -174,12 +182,16 @@ Existing query suite definitions can be reused by specifying: - import: - The path to the imported suite must be relative to the QL pack containing the + The path to the imported suite must be relative to the CodeQL pack containing the current suite definition. If the imported query suite is in a different QL pack you can use:: - import: from: + version: ^x.y.z + + The ``version`` field is optional and specifies a range of compatible versions of this CodeQL pack. + If the version is excluded, then the most recent version of the pack is used. Queries added using an ``import`` instruction can be filtered using subsequent ``exclude`` instructions. @@ -200,6 +212,8 @@ Existing query suite definitions can be reused by specifying: instruction, but takes a full suite definition as the argument, rather than the path to a ``.qls`` file on disk. +To see what queries are included in a query suite, you can run the ``codeql resolve queries my-suite.qls`` command. + Example ~~~~~~~ @@ -216,28 +230,27 @@ following in a file called ``reusable-instructions.yml``:: - high - very-high -Add ``reusable-instructions.yml`` to the same QL pack as your current query -suite (for example, ``my-custom-queries``). Apply the reusable instructions -to the queries in your current suite using:: - - - qlpack: my-custom-queries - - apply: reusable-instructions.yml - -To apply the same conditions to a different suite or directory within the same -QL pack, create a new definition and change (or replace) the ``qlpack`` -instruction. For example:: +Add ``reusable-instructions.yml`` to the same CodeQL pack as your current query +suite. Then, in one or more query suites, use the ``apply`` instruction to apply +the reusable instructions to the current suite. For example:: - queries: queries/cpp/custom - apply: reusable-instructions.yml +This will filter the queries in ``queries/cpp/custom`` to only include those that match the reusable conditions. + You can also create a suite definition using ``reusable-instructions.yml`` on -queries in a different QL pack. If the ``.qls`` file is in the same QL pack as +queries in a different codeQL pack. If the ``.qls`` file is in the same CodeQL pack as the queries, you can add a ``from`` field immediately after the ``apply`` instruction:: - - qlpack: my-other-custom-queries + # load queries from the default suite of my-org/my-other-custom-queries + - qlpack: my-org/my-other-custom-queries + + # apply the reusable instructions from the my-org/my-custom-instructions CodeQL pack - apply: reusable-instructions.yml - from: + from: my-org/my-custom-instructions + version: ^1.2.3 # optional Naming a query suite -------------------- @@ -255,20 +268,20 @@ directory. For more information, see "`Specifying well-known query suites Saving a query suite -------------------- -Save your query suite in a file with a ``.qls`` extension and add it to a QL -pack. For more information, see ":ref:`About QL packs `." +Save your query suite in a file with a ``.qls`` extension and add it to a CodeQL +pack. For more information, see ":ref:`About CodeQL packs `." Specifying well-known query suites ---------------------------------- -You can use QL packs to declare directories that contain "well-known" query +You can use CodeQL packs to declare directories that contain "well-known" query suites. You can use "well-known" query suites on the command line by referring to their file name, without providing their full path. This gives you a simple way of specifying a -set of queries, without needing to search inside QL packs and distributions. +set of queries, without needing to search inside CodeQL packs and distributions. To declare a directory that contains "well-known" query suites, add the directory -to the ``suites`` property in the ``qlpack.yml`` file at the root of your QL pack. -For more information, see ":ref:`About QL packs `." +to the ``suites`` property in the ``qlpack.yml`` file at the root of your CodeQL pack. +For more information, see ":ref:`About CodeQL packs `." Using query suites with CodeQL ------------------------------ @@ -288,7 +301,7 @@ JavaScript, visit https://github.com/github/codeql/tree/main/javascript/ql/src/codeql-suites. These suite definitions apply reusable filter patterns to the queries -located in the standard QL packs for each supported language. For more +located in the standard CodeQL packs for each supported language. For more information, see the `suite-helpers `__ in the CodeQL repository. diff --git a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst index f859f07cb7c..b95367eeed7 100644 --- a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst @@ -23,11 +23,11 @@ and 4 are slightly different---for further details, see the sections labeled **Information for macOS "Catalina" (or newer) users**. If you are using macOS on Apple Silicon (e.g. Apple M1), ensure that the `Xcode command-line developer tools `__ and `Rosetta 2 -`__ are installed. +`__ are installed. .. pull-quote:: Note - The CodeQL CLI is currently not compatible with non-glibc Linux + The CodeQL CLI is currently not compatible with non-glibc Linux distributions such as (muslc-based) Alpine Linux. For information about installing the CodeQL CLI in a CI system to create results @@ -47,7 +47,7 @@ Conditions `__. .. pull-quote:: Important - There are several different versions of the CLI available to download, depending + There are different versions of the CLI available to download, depending on your use case: - If you want to use the most up to date CodeQL tools and features, download the @@ -91,7 +91,7 @@ Alternatively, you can download ``codeql.zip``, which contains the CLI for all s iv. Ensure the check-box labeled **Open "safe" files after downloading**. is unchecked. -2. Extract the zip archive +1. Extract the zip archive ~~~~~~~~~~~~~~~~~~~~~~~~~~ For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) @@ -202,8 +202,6 @@ CLI that you will extract in step 4. If you use git on the command line, you can clone and rename the repository in a single step by running ``git clone git@github.com:github/codeql.git codeql-repo`` in the ``codeql-home`` folder. -For Go analysis, run ``codeql-repo/go/scripts/install-deps.sh`` to install its dependencies. - .. pull-quote:: Note The CodeQL libraries and queries for Go analysis used to live in a @@ -213,10 +211,10 @@ For Go analysis, run ``codeql-repo/go/scripts/install-deps.sh`` to install its d For more information, see the `Relocation announcement `__. -Within this repository, the queries and libraries are organized into QL -packs. Along with the queries themselves, QL packs contain important metadata +Within this repository, the queries and libraries are organized into CodeQL +packs. Along with the queries themselves, CodeQL packs contain important metadata that tells the CodeQL CLI how to process the query files. For more information, -see ":doc:`About QL packs `." +see ":doc:`About CodeQL packs `." .. pull-quote:: Important @@ -271,8 +269,8 @@ up to create and analyze databases: - Run ``codeql resolve languages`` to show which languages are available for database creation. This will list the languages supported by default in your CodeQL CLI package. -- Run ``codeql resolve qlpacks`` to show which QL packs the CLI can find. This - will display the names of all the QL packs directly available to the CodeQL CLI. +- Run ``codeql resolve qlpacks`` to show which CodeQL packs the CLI can find. This + will display the names of all the CodeQL packs directly available to the CodeQL CLI. This should include: - Query packs for each supported language, for example, ``codeql/{language}-queries``. diff --git a/docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst b/docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst index 985ca2659f3..3aa61e2b7fe 100644 --- a/docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst +++ b/docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst @@ -60,12 +60,13 @@ To analyze a CodeQL database with a CodeQL pack, run the following command: :: - codeql database analyze /@x.x.x + codeql database analyze /@x.x.x: - ````: the CodeQL database to be analyzed. - ````: the name of the GitHub organization that the pack is published to. - ````: the name for the pack that you are using. - ``@x.x.x``: an optional version number. If omitted, the latest version will be used. +- ``:``: an optional path to a query, directory, or query suite. If omitted, the pack's default query suite will be used. The ``analyze`` command will run the default suite of any specified CodeQL packs. You can specify multiple CodeQL packs to be used for analyzing a CodeQL database. For example: diff --git a/docs/codeql/codeql-cli/query-reference-files.rst b/docs/codeql/codeql-cli/query-reference-files.rst index f19996aae29..549c47025c0 100644 --- a/docs/codeql/codeql-cli/query-reference-files.rst +++ b/docs/codeql/codeql-cli/query-reference-files.rst @@ -10,10 +10,10 @@ to run a query that's not part of a test directory. There are two ways to specify queries that you want to run as tests: #. Use a query reference file to specify the location of a query to test. - This is useful when you create tests for alert and path queries that - are intended to identify problems in real codebases. You might create + This is useful when you create tests for alert and path queries that + are intended to identify problems in real codebases. You might create several directories of test code, each focusing on different - aspects of the query. Then you would add a query reference file to + aspects of the query. Then you would add a query reference file to each directory of test code, to specify the query to test. #. Add the query directly to a directory of tests. These is typically useful when you're writing queries explicitly to test the behavior @@ -25,22 +25,24 @@ Defining a query reference file Each query reference file, ``.qlref``, contains a single line that defines where to find one query. The location must be defined relative -to the root of the QL pack that contains the query. -Usually, this is a QL pack specified by the ``libraryPathDependencies`` for the test pack. +to the root of the CodeQL pack that contains the query. +Usually, this is either the CodeQL pack that contains the ``.qlref``, a CodeQL pack +specified in the ``dependencies`` block for the test pack, or a transitive dependency of +the CodeQL pack. You should use forward slashes in the path on all operating -systems to ensure compatibility between systems. +systems to ensure compatibility between systems. Example -^^^^^^^^ +^^^^^^^ A query reference file to test a JavaScript alert query: `DeadAngularJSEventListener.qlref `__ -The `QL pack `__ -for the ``javascript/ql/test`` directory defines the ``codeql-javascript`` queries as +The `CodeQL pack `__ +for the CodeQL pack at ``javascript/ql/test`` defines ``codeql/javascript-queries`` as a dependency. So the query reference file defines the location of the query relative -to the ``codeql-javascript`` QL pack:: +to the ``codeql/javascript-queries`` CodeQL pack:: AngularJS/DeadAngularJSEventListener.ql diff --git a/docs/codeql/codeql-cli/specifying-command-options-in-a-codeql-configuration-file.rst b/docs/codeql/codeql-cli/specifying-command-options-in-a-codeql-configuration-file.rst index f0b3571feb8..1aef971bea3 100644 --- a/docs/codeql/codeql-cli/specifying-command-options-in-a-codeql-configuration-file.rst +++ b/docs/codeql/codeql-cli/specifying-command-options-in-a-codeql-configuration-file.rst @@ -8,15 +8,15 @@ configuration file. You can specify CodeQL CLI command options in two ways: -- Directly in the command line, using the appropriate flag. +- Directly in the command line, using the appropriate flag. - In a configuration (or ``config``) file that CodeQL scans for relevant options each time a command is executed. For options that are likely to change each time you execute a command, specifying the value on the command line is the most convenient way of passing the information to CodeQL. Saving options in a ``config`` file is a good way to -specify options you use frequently. -It's also a good way to add custom QL packs that you use regularly to your search path. +specify options you use frequently. +It's also a good way to add custom CodeQL packs that you use regularly to your search path. Using a CodeQL configuration file --------------------------------- @@ -39,7 +39,7 @@ To apply the same options to more than one command you can: .. pull-quote:: Note - + - ``config`` files only accept spaces between between option flags and values---CodeQL will throw an error if you use ``=`` to specify an option value. - If you specify an option in the command line, this overrides the ``config`` @@ -66,7 +66,7 @@ Examples --ram 4096 --threads 4 -- To globally specify a directory for CodeQL to scan for QL packs (which is not a +- To globally specify a directory for CodeQL to scan for CodeQL packs (which is not a sibling of the installation directory), use:: --search-path diff --git a/docs/codeql/codeql-cli/testing-custom-queries.rst b/docs/codeql/codeql-cli/testing-custom-queries.rst index 7f347de8b1c..f6c7f2d32f7 100644 --- a/docs/codeql/codeql-cli/testing-custom-queries.rst +++ b/docs/codeql/codeql-cli/testing-custom-queries.rst @@ -13,10 +13,10 @@ on the query and the expected results until the actual results and the expected results exactly match. This topic shows you how to create test files and execute tests on them using the ``test run`` subcommand. -Setting up a test QL pack for custom queries --------------------------------------------- +Setting up a test CodeQL pack for custom queries +------------------------------------------------ -All CodeQL tests must be stored in a special "test" QL pack. +All CodeQL tests must be stored in a special "test" CodeQL pack. That is, a directory for test files with a ``qlpack.yml`` file that defines: @@ -24,22 +24,25 @@ file that defines: name: version: 0.0.0 - libraryPathDependencies: + dependencies: + : "*" extractor: -The ``libraryPathDependencies`` value specifies the CodeQL queries to test. -The ``extractor`` defines which language the CLI will use -to create test databases from the code files stored in this QL pack. -For more information, see ":doc:`About QL packs `." +The ``dependencies`` value specifies the CodeQL packs containing queries to test. +Typically, these packs will be resolved from source, and so it is not necessary +to specify a fixed version of the pack. +The ``extractor`` defines which language the CLI will use +to create test databases from the code files stored in this CodeQL pack. +For more information, see ":doc:`About CodeQL packs `." -You may find it useful to look at the way query tests are organized in the -`CodeQL repository `__. -Each language has a ``src`` directory, ``ql//ql/src``, -that contains libraries and queries for analyzing codebases. -Alongside the ``src`` directory, -there's a ``test`` directory with tests for these libraries and queries. +You may find it useful to look at the way query tests are organized in the +`CodeQL repository `__. +Each language has a ``src`` directory, ``ql//ql/src``, +that contains libraries and queries for analyzing codebases. +Alongside the ``src`` directory, there is a ``test`` directory with tests for +these libraries and queries. -Each ``test`` directory is configured as a test QL pack with two subdirectories: +Each ``test`` directory is configured as a test CodeQL pack with two subdirectories: - ``query-tests`` a series of subdirectories with tests for queries stored in the ``src`` directory. Each subdirectory contains test code and a QL reference file that specifies the query to test. @@ -49,18 +52,18 @@ Each ``test`` directory is configured as a test QL pack with two subdirectories: Setting up the test files for a query ------------------------------------- -For each query you want to test, you should create a sub-directory in the test QL pack. +For each query you want to test, you should create a sub-directory in the test CodeQL pack. Then add the following files to the subdirectory before you run the test command: - A query reference file (``.qlref`` file) defining the location of the query to test. - The location is defined relative to the root of the QL pack that contains the - query. Usually, this is a QL pack specified by the - ``libraryPathDependencies`` for the test pack. + The location is defined relative to the root of the CodeQL pack that contains the + query. Usually, this is a CodeQL pack specified in the + ``dependencies`` block of the test pack. For more information, see ":doc:`Query reference files `." - You don't need to add a query reference file if the query you want to + You do not need to add a query reference file if the query you want to test is stored in the test directory, - but it's generally good practice to store queries separately from tests. + but it is generally good practice to store queries separately from tests. The only exception is unit tests for QL libraries, which tend to be stored in test packs, separate from queries that generate alerts or paths. @@ -68,15 +71,13 @@ Then add the following files to the subdirectory before you run the test command should consist of one or more files containing examples of the code the query is designed to identify. -You can also define the results you expect to see when you run the query against +You can also define the results you expect to see when you run the query against the example code, by creating a file with the extension ``.expected``. Alternatively, you can leave the test command to create the ``.expected`` file -for you. (If you're using CodeQL CLI 2.0.2--2.0.6, you need to create an -``.expected`` file otherwise the command will fail to find your test query. -You can create an empty ``.expected`` file to workaround this limitation.) +for you. For an example showing how to create and test a query, see the `example -<#example>`__ below. +<#example>`__ below. .. pull-quote:: Important @@ -86,14 +87,14 @@ For an example showing how to create and test a query, see the `example it must have the same base name as the corresponding ``.expected`` file. For example, if the query is ``MyJavaQuery.ql``, the expected results file must be ``MyJavaQuery.expected``. - + If you want to specify a ``.qlref`` file in the command, it must have the same base name as the corresponding ``.expected`` file, but the query itself - may have a different name. + may have a different name. The names of the example code files don't have to be consistent with the other test files. All example code files found next to the ``.qlref`` (or ``.ql``) - file and in any subdirectories will be used to create a test database. + file and in any subdirectories will be used to create a test database. Therefore, for simplicity, we recommend you don't save test files in directories that are ancestors of each other. @@ -108,10 +109,10 @@ The ```` argument can be one or more of the following: - Path to a ``.ql`` file. - Path to a ``.qlref`` file that references a ``.ql`` file. -- Path to a directory that will be searched recursively for ``.ql`` and +- Path to a directory that will be searched recursively for ``.ql`` and ``.qlref`` files. -You can also specify: +You can also specify: - .. include:: ../reusables/threads-query-execution.rst @@ -123,7 +124,7 @@ Example The following example shows you how to set up a test for a query that searches Java code for ``if`` statements that have empty ``then`` blocks. It includes -steps to add the custom query and corresponding test files to separate QL packs +steps to add the custom query and corresponding test files to separate CodeQL packs outside your checkout of the CodeQL repository. This ensures when you update the CodeQL libraries, or check out a different branch, you won't overwrite your custom queries and tests. @@ -144,36 +145,36 @@ Prepare a query and test files #. Save the query to a file named ``EmptyThen.ql`` in a directory with your other custom queries. For example, - ``custom-queries/java/queries/EmptyThen.ql``. - -#. If you haven't already added your custom queries to a QL pack, - create a QL pack now. For example, if your custom Java queries + ``custom-queries/java/queries/EmptyThen.ql``. + +#. If you haven't already added your custom queries to a CodeQL pack, + create a CodeQL pack now. For example, if your custom Java queries are stored in ``custom-queries/java/queries``, add a ``qlpack.yml`` file with the following contents to ``custom-queries/java/queries``: .. code-block:: yaml name: my-custom-queries - version: 0.0.0 - libraryPathDependencies: codeql-java + dependencies: + codeql/java-queries: "*" - For more information about QL packs, see ":doc:`About QL packs - `." + For more information about CodeQL packs, see ":doc:`About CodeQL packs + `." -#. Create a QL pack for your Java tests by adding a ``qlpack.yml`` file +#. Create a CodeQL pack for your Java tests by adding a ``qlpack.yml`` file with the following contents to ``custom-queries/java/tests``, - updating ``libraryPathDependencies`` to match the name of your QL pack of custom queries: + updating the ``dependencies`` to match the name of your CodeQL pack of custom queries: .. include:: ../reusables/test-qlpack.rst #. Within the Java test pack, create a directory to contain the test files associated with ``EmptyThen.ql``. - For example, ``custom-queries/java/tests/EmptyThen``. + For example, ``custom-queries/java/tests/EmptyThen``. -#. In the new directory, create ``EmptyThen.qlref`` to define the location of ``EmptyThen.ql``. +#. In the new directory, create ``EmptyThen.qlref`` to define the location of ``EmptyThen.ql``. The path to the query must be specified relative to the root of - the QL pack that contains the query. In this case, the query is in the - top level directory of the QL pack named ``my-custom-queries``, + the CodeQL pack that contains the query. In this case, the query is in the + top level directory of the CodeQL pack named ``my-custom-queries``, which is declared as a dependency for ``my-query-tests``. Therefore, ``EmptyThen.qlref`` should simply contain ``EmptyThen.ql``. @@ -191,7 +192,7 @@ Prepare a query and test files System.out.println("Empty argument"); } } - + public void good(String arg) { if (arg.isEmpty()) { System.out.println("Empty argument"); @@ -211,8 +212,8 @@ When the test runs it: #. Extracts a CodeQL database from the ``.java`` files stored in the ``EmptyThen`` directory. #. Compiles the query referenced by the ``EmptyThen.qlref`` file. - If this step fails, it's because the CLI can't find your custom QL pack. - Re-run the command and specify the location of your custom QL pack, for example: + If this step fails, it's because the CLI can't find your custom CodeQL pack. + Re-run the command and specify the location of your custom CodeQL pack, for example: ``codeql test run --search-path=java java/tests/EmptyThen`` @@ -234,14 +235,14 @@ CodeQL generates the following files in the ``EmptyThen`` directory: When tests complete successfully, this database is deleted in a housekeeping step. You can override this step by running ``test run`` with the ``--keep-databases`` option. -In this case, the failure was expected and is easy to fix. +In this case, the failure was expected and is easy to fix. If you open the ``EmptyThen.actual`` file, you can see the results of the test: .. code-block:: none - + | Test.java:3:5:3:22 | stmt | This if statement has an empty then. | -This file contains a table, with a column for the location of the result, +This file contains a table, with a column for the location of the result, along with separate columns for each part of the ``select`` clause the query outputs. Since the results are what we expected, we can update the file extension to define this as the expected result for this test (``EmptyThen.expected``). @@ -252,17 +253,17 @@ If you rerun the test now, the output will be similar but it will finish by repo If the results of the query change, for example, if you revise the ``select`` statement for the query, the test will fail. For failed results, the CLI output includes a unified diff of the ``EmptyThen.expected`` and ``EmptyThen.actual`` files. -This information may be sufficient to debug trivial test failures. +This information may be sufficient to debug trivial test failures. For failures that are harder to debug, you can import ``EmptyThen.testproj`` into CodeQL for VS Code, execute ``EmptyThen.ql``, and view the results in the ``Test.java`` example code. For more information, see ":ref:`Analyzing your projects `" in the CodeQL for VS Code -help. +help. Further reading --------------- - ":ref:`CodeQL queries `" -- ":ref:`Testing CodeQL queries in Visual Studio Code `" \ No newline at end of file +- ":ref:`Testing CodeQL queries in Visual Studio Code `" diff --git a/docs/codeql/codeql-cli/testing-query-help-files.rst b/docs/codeql/codeql-cli/testing-query-help-files.rst index 26cea264043..d00420d71e9 100644 --- a/docs/codeql/codeql-cli/testing-query-help-files.rst +++ b/docs/codeql/codeql-cli/testing-query-help-files.rst @@ -3,7 +3,7 @@ Testing query help files ======================== -Test query help files by rendering them as markdown to ensure they are valid +Test query help files by rendering them as markdown to ensure they are valid before uploading them to the CodeQL repository or using them in code scanning. Query help is documentation that accompanies a query to explain how the query works, @@ -12,46 +12,46 @@ It is good practice to write query help for all new queries. For more informatio see `Contributing to CodeQL `__ in the CodeQL repository. -The CodeQL CLI includes a command to test query help and render the content as -markdown, so that you can easily preview the content in your IDE. Use the command to validate -query help files before uploading them to the CodeQL repository or sharing them with other users. -From CodeQL CLI 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files -generated during CodeQL analyses so that the query help can be displayed in the code scanning UI. -For more information, see +The CodeQL CLI includes a command to test query help and render the content as +markdown, so that you can easily preview the content in your IDE. Use the command to validate +query help files before uploading them to the CodeQL repository or sharing them with other users. +From CodeQL CLI 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files +generated during CodeQL analyses so that the query help can be displayed in the code scanning UI. +For more information, see ":ref:`Analyzing databases with the CodeQL CLI `." Prerequisites ------------- -- The query help (``.qhelp``) file must have an accompanying query (``.ql``) file with +- The query help (``.qhelp``) file must have an accompanying query (``.ql``) file with an identical base name. - The query help file should follow the standard structure and style for query help documentation. - For more information, see the `Query help style guide `__ - in the CodeQL repository. + For more information, see the `Query help style guide `__ + in the CodeQL repository. Running ``codeql generate query-help`` -------------------------------------- You can test query help files by running the following command:: - codeql generate query-help --format= [--output=] + codeql generate query-help --format= [--output=] where ```` is one of: - the path to a ``.qhelp`` file. - the path to a ``.ql`` file. - the path to a directory containing queries and query help files. -- the path to a query suite, or the name of a well-known query suite for a QL pack. +- the path to a query suite, or the name of a well-known query suite for a CodeQL pack. For more information, see "`Creating CodeQL query suites `__." -You must specify a ``--format`` option, which defines how the query help is rendered. -Currently, you must specify ``markdown`` to render the query help as markdown. +You must specify a ``--format`` option, which defines how the query help is rendered. +Currently, you must specify ``markdown`` to render the query help as markdown. The ``--output`` option defines a file path where the rendered query help will be saved. -- For directories containing ``.qhelp`` files or a query suites - defining one or more ``.qhelp`` files, you must specify an ``--output`` directory. - Filenames within the output directory will be derived from the ``.qhelp`` file names. +- For directories containing ``.qhelp`` files or a query suites + defining one or more ``.qhelp`` files, you must specify an ``--output`` directory. + Filenames within the output directory will be derived from the ``.qhelp`` file names. - For single ``.qhelp`` or ``.ql`` files, you may specify an ``--output`` option. If you don't specify an output path, the rendered query help is written to ``stdout``. @@ -63,15 +63,15 @@ see the `generate query-help reference documentation Results ------- -When you run the command, CodeQL attempts to render +When you run the command, CodeQL attempts to render each ``.qhelp`` file that has an accompanying ``.ql`` file. For single files, the rendered -content will be printed to ``stdout`` if you don't specify an ``--output`` option. For all other -use cases, the rendered content is saved to the specified output path. +content will be printed to ``stdout`` if you don't specify an ``--output`` option. For all other +use cases, the rendered content is saved to the specified output path. By default, the CodeQL CLI will print a warning message if: - Any of the query help is invalid, along with a description of the invalid query help elements -- Any ``.qhelp`` files specified in the command don't have the same base name +- Any ``.qhelp`` files specified in the command don't have the same base name as an accompanying ``.ql`` file - Any ``.ql`` files specified in the command don't have the same base name as an accompanying ``.qhelp`` file diff --git a/docs/codeql/codeql-cli/using-custom-queries-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/using-custom-queries-with-the-codeql-cli.rst index 8208cd59a56..743f2c57586 100644 --- a/docs/codeql/codeql-cli/using-custom-queries-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/using-custom-queries-with-the-codeql-cli.rst @@ -54,23 +54,20 @@ Packaging custom QL queries .. include:: ../reusables/beta-note-package-management.rst -When you write your own queries, you should save them in a custom QL pack -directory. When you are ready to share your queries with other users, you can publish the pack as a CodeQL pack to GitHub Packages - the GitHub Container registry. +When you write your own queries with the intention to share them with others, you should +save them in a custom CodeQL pack. You can publish the pack as a CodeQL pack to GitHub +Packages - the GitHub Container registry. For further information see +":ref:`About CodeQL packs `." -QL packs organize the files used in CodeQL analysis and can store queries, +CodeQL packs organize the files used in CodeQL analysis and can store queries, library files, query suites, and important metadata. Their root directory must contain a file named ``qlpack.yml``. Your custom queries should be saved in the -QL pack root, or its subdirectories. +CodeQL pack root, or its subdirectories. -For each QL pack, the ``qlpack.yml`` file includes information that tells CodeQL +For each CodeQL pack, the ``qlpack.yml`` file includes information that tells CodeQL how to compile the queries, which other CodeQL packs and libraries the pack depends on, and where to find query suite definitions. For more information -about what to include in this file, see ":ref:`About QL packs `." - -CodeQL packages are used to create, share, depend on, and run CodeQL queries and -libraries. You can publish your own CodeQL packages and download ones created by -others via the the Container registry. For further information see -":ref:`About CodeQL packs `." +about what to include in this file, see ":ref:`About CodeQL packs `." Contributing to the CodeQL repository ------------------------------------- diff --git a/docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst b/docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst index 71f72223620..b23ddacf157 100644 --- a/docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst +++ b/docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst @@ -19,7 +19,7 @@ To analyze a project, you need to add a :ref:`CodeQL database ` .. image:: ../images/codeql-for-visual-studio-code/choose-database.png :width: 350 :alt: Choose a database to analyze - + For more information about obtaining a local database, see below. #. Once you've chosen a database, it is displayed in the Databases view. To see the menu options for interacting with a database, right-click an entry in the list. You can select multiple databases using **Ctrl/Cmd+click**. @@ -27,7 +27,7 @@ To analyze a project, you need to add a :ref:`CodeQL database ` Obtaining a local database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP file, you can add it to Visual Studio Code. There are several ways to obtain a local CodeQL database. +If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP file, you can add it to Visual Studio Code. There are several ways to obtain a local CodeQL database. - To create a database with the CodeQL CLI, see ":ref:`Creating CodeQL databases `." @@ -36,14 +36,14 @@ If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP - To analyze a test database, add a ``.testproj`` folder to the Databases view. Test databases (that is, folders with a ``.testproj`` extension) are generated when you run regression tests on custom queries using the :ref:`CodeQL CLI `. If a query fails a regression test, you may want to analyze the test database in Visual Studio Code to debug the failure. - - For more information about running query tests, see ":ref:`Testing custom queries `" in the CodeQL CLI help. + + For more information about running query tests, see ":ref:`Testing custom queries `" in the CodeQL CLI help. Running a query ------------------------ The `CodeQL repository `__ on GitHub contains lots of example queries. -If you have that folder (or a different QL pack) available in your workspace, you can access existing queries under ``/ql/src/``, for example ``java/ql/src/Likely Bugs``. +If you have that folder (or a different CodeQL pack) available in your workspace, you can access existing queries under ``/ql/src/``, for example ``java/ql/src/Likely Bugs``. #. Open a query (``.ql``) file. It is displayed in the editor, with IntelliSense features such as syntax highlighting and autocomplete suggestions. #. Right-click in the query window and select **CodeQL: Run Query**. (Alternatively, run the command from the Command Palette.) @@ -77,7 +77,7 @@ To run the query, use **CodeQL: Run Query**. You can see all quick queries that you've run in the current session in the Query History view. Click an entry to see the exact text of the quick query that produced the results. -Once you're happy with your quick query, you should save it in a QL pack so you can access it later. For more information, see ":ref:`About QL packs `." +Once you're happy with your quick query, you should save it in a CodeQL pack so you can access it later. For more information, see ":ref:`About CodeQL packs `." .. _running-a-specific-part-of-a-query-or-library: @@ -87,7 +87,7 @@ Running a specific part of a query or library This is helpful if you're debugging a query or library and you want to locate the part that is wrong. Instead of using **CodeQL: Run Query** to run the whole query (the :ref:`select clause ` and any :ref:`query predicates `), you can use **CodeQL: Quick Evaluation** to run a specific part of a ``.ql`` or ``.qll`` file. -**CodeQL: Quick Evaluation** evaluates a code snippet (instead of the whole query) and displays results of that selection in the Results view. +**CodeQL: Quick Evaluation** evaluates a code snippet (instead of the whole query) and displays results of that selection in the Results view. Possible targets for quick evaluation include: - Selecting the name of a CodeQL entity (such as a :ref:`class ` or :ref:`predicate `) to evaluate that entity. diff --git a/docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst b/docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst index b4969ce785d..1c762348a9d 100644 --- a/docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst +++ b/docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst @@ -110,7 +110,7 @@ For example, to make a custom CodeQL folder called ``my-custom-cpp-pack`` depend version: 0.0.0 libraryPathDependencies: codeql/cpp-all -For more information about why you need to add a ``qlpack.yml`` file, see ":ref:`About QL packs `." +For more information about why you need to add a ``qlpack.yml`` file, see ":ref:`About CodeQL packs `." Further reading ---------------- diff --git a/docs/codeql/codeql-overview/codeql-glossary.rst b/docs/codeql/codeql-overview/codeql-glossary.rst index 0acc7bd25f1..3770cab7921 100644 --- a/docs/codeql/codeql-overview/codeql-glossary.rst +++ b/docs/codeql/codeql-overview/codeql-glossary.rst @@ -5,7 +5,7 @@ CodeQL glossary =============== -An overview of the technical terms and concepts in CodeQL. +An overview of the technical terms and concepts in CodeQL. .. _bqrs-file: @@ -26,11 +26,11 @@ exported as a variety of different formats, including SARIF. CodeQL database --------------- -A database (or CodeQL database) is a directory containing: +A database (or CodeQL database) is a directory containing: - queryable data, extracted from the code. - a source reference, for displaying query results directly in the code. -- query results. +- query results. - log files generated during database creation, query execution, and other operations. @@ -39,15 +39,15 @@ A database (or CodeQL database) is a directory containing: DIL --- -DIL stands for Datalog Intermediary Language. It is an intermediate -representation between QL and relation algebra (RA) that is generated -during query compilation. DIL is useful for advanced users as an aid -for debugging query performance. +DIL stands for Datalog Intermediary Language. It is an intermediate +representation between QL and relation algebra (RA) that is generated +during query compilation. DIL is useful for advanced users as an aid +for debugging query performance. The DIL format may change without warning between CLI releases. -When you specify the ``--dump-dil`` option for ``codeql query compile``, CodeQL -prints DIL to standard output for the queries it compiles. You can also -view results in DIL format when you run queries in VS Code. +When you specify the ``--dump-dil`` option for ``codeql query compile``, CodeQL +prints DIL to standard output for the queries it compiles. You can also +view results in DIL format when you run queries in VS Code. For more information, see ":ref:`Analyzing your projects `" in the CodeQL for VS Code help. .. _extractor: @@ -56,7 +56,7 @@ Extractor --------- An extractor is a tool that produces the relational data and source -reference for each input file, from which a CodeQL database can be built. +reference for each input file, from which a CodeQL database can be built. .. _codeql-database-schema: @@ -67,7 +67,7 @@ A QL database schema is a file describing the column types and extensional relations that make up a raw QL dataset. It is a text file with the ``.dbscheme`` extension. -The extractor and core QL pack for a language each declare the database +The extractor and core CodeQL pack for a language each declare the database schema that they use. This defines the database layout they create or expect. When you create a CodeQL database, the extractor copies its schema into the database. The CLI uses this to check whether the @@ -83,16 +83,16 @@ There is currently no public-facing specification for the syntax of schemas. -------------- ``.qlo`` files are optionally generated during query compilation. -If you specify the ``--dump-qlo`` option for ``codeql query compile``, +If you specify the ``--dump-qlo`` option for ``codeql query compile``, CodeQL writes ``.qlo`` files for the queries it compiles. They can be used -as an aid for debugging and performance tuning for advanced users. +as an aid for debugging and performance tuning for advanced users. -``.qlo`` is a binary format that represents a compiled -and optimized query in terms of relational algebra (RA) or the -intermediate :ref:`DIL ` format. ``.qlo`` files can be expanded to -readable text using ``codeql query decompile``. +``.qlo`` is a binary format that represents a compiled +and optimized query in terms of relational algebra (RA) or the +intermediate :ref:`DIL ` format. ``.qlo`` files can be expanded to +readable text using ``codeql query decompile``. -The exact details of the ``.qlo`` format may change without warning between CLI releases. +The exact details of the ``.qlo`` format may change without warning between CLI releases. .. _sarif-file: @@ -101,22 +101,22 @@ SARIF file Static analysis results interchange format (SARIF) is an output format used for sharing static analysis results. For more information, see ":ref:`SARIF -output `." +output `." .. _source-reference: Source reference ---------------- -A source reference is a mechanism that allows the retrieval of the -contents of a source file, given an absolute filename at which that file +A source reference is a mechanism that allows the retrieval of the +contents of a source file, given an absolute filename at which that file resided during extraction. Specific examples include: -- A source archive directory, within which the requested absolute +- A source archive directory, within which the requested absolute filename maps to a UTF8-encoded file. -- A source archive, typically in ZIP format, which contains the UTF8-encoded +- A source archive, typically in ZIP format, which contains the UTF8-encoded content of all source files. -- A source archive repository, typically in ``git`` format, typically bare, +- A source archive repository, typically in ``git`` format, typically bare, which contains the UTF8-encoded content of all source files. Source references are typically included in CodeQL databases. @@ -126,7 +126,7 @@ Source references are typically included in CodeQL databases. TRAP file --------- -A TRAP file is a UTF-8 encoded file generated by a CodeQL extractor +A TRAP file is a UTF-8 encoded file generated by a CodeQL extractor with the extension ``.trap``. To save space, they are usually archived. They contain the information that, when interpreted relative to a QL database schema, is used to create a QL dataset. diff --git a/docs/codeql/ql-language-reference/ql-language-specification.rst b/docs/codeql/ql-language-reference/ql-language-specification.rst index 46d0123d6d4..ea90ebbf843 100644 --- a/docs/codeql/ql-language-reference/ql-language-specification.rst +++ b/docs/codeql/ql-language-reference/ql-language-specification.rst @@ -68,7 +68,7 @@ contain a single top-level tag named of the active database schema (for example, ````). -A ``qlpack.yml`` file defines a :ref:`QL pack `. +A ``qlpack.yml`` file defines a :ref:`CodeQL pack `. The content of a ``qlpack.yml`` file is described in the CodeQL CLI documentation. If both a ``queries.xml`` and a ``qlpack.yml`` exist in the same @@ -77,16 +77,16 @@ exist for compatibility with older tooling). The CodeQL CLI and newer tools based on it (such as, GitHub code scanning and the CodeQL extension for Visual Studio Code) -construct a library path using QL packs. For each QL pack -added to the library path, the QL packs named in its +construct a library path using CodeQL packs. For each CodeQL pack +added to the library path, the CodeQL packs named in its ``libraryPathDependencies`` will be subsequently added to the library path, and the process continues until all packs have been resolved. The actual library path consists of the root directories of -the selected QL packs. This process depends on a mechanism for finding -QL packs by pack name, as described in the :ref:`CodeQL CLI documentation `. +the selected CodeQL packs. This process depends on a mechanism for finding +CodeQL packs by pack name, as described in the :ref:`CodeQL CLI documentation `. When the query directory contains a ``queries.xml`` file but no -``qlpack.yml``, the QL pack resolution behaves as if it defines a QL +``qlpack.yml``, the CodeQL pack resolution behaves as if it defines a QL pack with no name and a single library path dependency named ``legacy-libraries-LANGUAGE`` where ``LANGUAGE`` is taken from ``queries.xml``. The ``github/codeql`` repository provides packs with @@ -94,7 +94,7 @@ names following this pattern, which themselves depend on the actual CodeQL libraries for each language. When the query directory contains neither a ``queries.xml`` nor -``qlpack.yml`` file, it is considered to be a QL pack with no name and +``qlpack.yml`` file, it is considered to be a CodeQL pack with no name and no library dependencies. This causes the library path to consist of *only* the query directory itself. This is not generally useful, but it suffices for running toy examples of QL code that don't diff --git a/docs/codeql/writing-codeql-queries/about-codeql-queries.rst b/docs/codeql/writing-codeql-queries/about-codeql-queries.rst index be5bca75622..61d81bf6f3e 100644 --- a/docs/codeql/writing-codeql-queries/about-codeql-queries.rst +++ b/docs/codeql/writing-codeql-queries/about-codeql-queries.rst @@ -3,7 +3,7 @@ About CodeQL queries #################### -CodeQL queries are used to analyze code for issues related to security, correctness, maintainability, and readability. +CodeQL queries are used to analyze code for issues related to security, correctness, maintainability, and readability. Overview ******** @@ -13,7 +13,7 @@ CodeQL includes queries to find the most relevant and interesting problems for e - **Alert queries**: queries that highlight issues in specific locations in your code. - **Path queries**: queries that describe the flow of information between a source and a sink in your code. -You can add custom queries to :doc:`QL packs <../codeql-cli/about-ql-packs>` to analyze your projects with "`Code scanning `__", use them to analyze a database with the ":ref:`CodeQL CLI `," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub `__. +You can add custom queries to :doc:`CodeQL packs <../codeql-cli/about-codeql-packs>` to analyze your projects with "`Code scanning `__", use them to analyze a database with the ":ref:`CodeQL CLI `," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub `__. This topic is a basic introduction to query files. You can find more information on writing queries for specific programming languages in the ":ref:`CodeQL language guides `," and detailed technical information about QL in the ":ref:`QL language reference `." For more information on how to format your code when contributing queries to the GitHub repository, see the `CodeQL style guide `__.