hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2025-12-16 00:33:11 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update CodeQL CLI articles with migration message

Browse Source
This commit is contained in:
Sarita Iyer
2023-02-02 14:11:02 -05:00
parent d671cc6e43
commit 2772cf79f4
36 changed files with 105 additions and 3337 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/codeql/CONTRIBUTING.md
View File

@@ -8,11 +8,9 @@ To make changes to the documentation on [codeql.github.com](https://codeql.githu
## Contributing to CodeQL CLI docs on `docs.github.com`
We are in the process of moving all documentation about the CodeQL CLI from [github/codeql](docs/codeql) to the public [github/docs](https://github.com/github/docs) repository so that this documentation is published on the [GitHub Docs](https://docs.github.com/en/code-security/code-scanning) site. This includes all articles that are currently published under "[Using the CodeQL CLI](https://codeql.github.com/docs/codeql-cli/using-the-codeql-cli/)" and "[CodeQL CLI reference](https://codeql.github.com/docs/codeql-cli/codeql-cli-reference/)" categories on the CodeQL microsite. This will make it easier for code scanning users to find information about using CodeQL to query their codebases.
We have moved all documentation about the CodeQL CLI from [github/codeql](docs/codeql) to the public [github/docs](https://github.com/github/docs) repository so that this documentation is published on the [GitHub Docs](https://docs.github.com/en/code-security/codeql-cli) site. This includes all articles that are currently published under "[Using the CodeQL CLI](https://codeql.github.com/docs/codeql-cli/using-the-codeql-cli/)" and "[CodeQL CLI reference](https://codeql.github.com/docs/codeql-cli/codeql-cli-reference/)" categories on the CodeQL microsite. This will make it easier for code scanning users to find information about using CodeQL to query their codebases.
**Note**: For a brief time, we will have source files for CodeQL CLI documentation in two locations. During this period we will not accept changes to the old files in the `codeql` repository, only to the new files in the `docs` repository.
To contribute to these docs, which are located in the [`code-scanning`](https://github.com/github/docs/tree/main/content/code-security/code-scanning) directory, please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file in the `docs` repository.
To contribute to these docs, which are located in the [`codeql-cli`](https://github.com/github/docs/tree/main/content/code-security/codeql-cli) directory, please refer to the [CONTRIBUTING.md](https://github.com/github/docs/blob/main/CONTRIBUTING.md) file in the `docs` repository.

392
docs/codeql/codeql-cli/about-codeql-packs.rst
View File

@@ -3,391 +3,7 @@
About CodeQL packs
==================
.. include:: ../reusables/beta-note-package-management.rst
CodeQL packs are used to create, share, depend on, and run CodeQL queries and libraries. You can publish your own CodeQL packs and download packs created by others. CodeQL packs contain queries, library files, query suites, and metadata.
There are two types of CodeQL packs: query packs and library packs.
* Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and pre-compiled representations of each query, in addition to the query sources. This ensures consistent and efficient execution of the queries in the pack.
* Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled separately.
You can use the package management commands in the CodeQL CLI to create CodeQL packs, add dependencies to packs, and install or update dependencies. For more information, see ":ref:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`." You can also publish and download CodeQL packs using the CodeQL CLI. For more information, see ":doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
The standard CodeQL packages for all supported languages are published in the `GitHub Container registry <https://github.com/orgs/codeql/packages>`__.
The `CodeQL repository <https://github.com/github/codeql>`__ contains source files for the standard CodeQL packs for all supported languages.
.. _codeql-pack-structure:
CodeQL pack structure
---------------------
A CodeQL pack must contain a file called ``qlpack.yml`` in its root directory. In the ``qlpack.yml`` file, the ``name:`` field must have a value that follows the format of ``<scope>/<pack>``, where ``<scope>`` is the GitHub organization or user account that the pack will be published to and ``<pack>`` is the name of the pack. Additionally, query packs and library packs with CodeQL tests contain a ``codeql-pack.lock.yml`` file that contains the resolved dependencies of the pack. This file is generated during a call to the ``codeql pack install`` command, is not meant to be edited by hand, and should be added to your version control system.
The other files and directories within the pack should be logically organized. For example, typically:
- Queries are organized into directories for specific categories.
- Queries for specific products, libraries, and frameworks are organized into
their own top-level directories.
About published packs
~~~~~~~~~~~~~~~~~~~~~~~~~
When a pack is published for use in analyses, the ``codeql pack create`` or ``codeql pack publish`` command verifies that the content is complete and also adds some additional pieces of content to it:
* For query packs, a copy of each of the library packs it depends on, in the precise versions it has been developed with. Users of the query pack won't need to download these library packs separately.
* For query packs, precompiled representations of each of the queries. These are faster to execute than it would be to compile the QL source for the query at each analysis.
Most of this data is located in a directory named ``.codeql`` in the published pack, but precompiled queries are in files with a ``.qlx`` suffix next to the ``.ql`` source for each query. When analyzing a database with a query from a published pack, CodeQL will load these files instead of the ``.ql`` source. If you need to modify the content of a *published* pack, be sure to remove all of the ``.qlx`` files, since they may prevent modifications in the ``.ql`` files from taking effect.
About ``qlpack.yml`` files
--------------------------
When executing query-related commands, CodeQL first looks in siblings of the installation directory (and their subdirectories) for ``qlpack.yml`` files.
Then it checks the package cache for CodeQL packs which have been downloaded. This means that when you are developing queries locally, the local packages
in the installation directory override packages of the same name in the package cache, so that you can test your local changes.
The metadata in each ``qlpack.yml`` file tells
CodeQL how to compile any queries in the pack, what libraries the pack depends on, and where to
find query suite definitions.
The contents of the CodeQL pack (queries or libraries used in CodeQL analysis) is included in the same directory as ``qlpack.yml``, or its subdirectories.
The directory containing the ``qlpack.yml`` file serves as the root directory for the content of the CodeQL pack. That is, for all ``.ql`` and ``.qll`` files in the pack, CodeQL will resolve all import statements relative to the directory containing the ``qlpack.yml`` file at the pack's root.
.. _codeqlpack-yml-properties:
``qlpack.yml`` properties
~~~~~~~~~~~~~~~~~~~~~~~~~
The following properties are supported in ``qlpack.yml`` files.
.. list-table::
:header-rows: 1
:widths: auto
* - Property
- When to use
- Explanation
* - ``name``
- Required by all packs
- The scope, where the CodeQL pack is published, and the name of the pack defined using alphanumeric characters and hyphens. It must be unique as CodeQL cannot differentiate between CodeQL packs with identical names. Use the pack name to specify queries to run using ``database analyze`` and to define dependencies between CodeQL packs (see examples below).
Example:
.. code-block:: yaml
name: octo-org/security-queries
* - ``version``
- Required by all packs that are published
- A semantic version for this CodeQL pack that must adhere to the `SemVer v2.0.0 specification <https://semver.org/spec/v2.0.0.html>`__.
Example:
.. code-block:: yaml
version: 0.0.0
* - ``dependencies``
- Required by packs that define CodeQL package dependencies on other packs.
- A map from pack references to the semantic version range that is compatible with this pack. Supported for CLI versions v2.6.0 and later.
Example:
.. code-block:: yaml
dependencies:
codeql/cpp-all: ^0.0.2
* - ``defaultSuiteFile``
- Required by packs that export a set of default queries to run
- The path to a query suite file relative to the package root, containing all of the queries that are run by default when this pack is passed to the ``codeql database analyze`` command. Supported from CLI version v2.6.0 and onwards. Only one of ``defaultSuiteFile`` or ``defaultSuite`` can be defined.
Example:
.. code-block:: yaml
defaultSuiteFile: cpp-code-scanning.qls
* - ``defaultSuite``
- Required by packs that export a set of default queries to run.
- An inlined query suite containing all of the queries that are run by default when this pack is passed to the ``codeql database analyze`` command. Supported from CLI version v2.6.0 and onwards. Only one of ``defaultSuiteFile`` or ``defaultSuite`` can be defined.
Example:
.. code-block:: yaml
defaultSuite:
queries: .
exclude:
precision: medium
* - ``library``
- Required by library packs
- A boolean value that indicates whether this pack is a library pack. Library packs do not contain queries and are not compiled. Query packs can ignore this field or explicitly set it to ``false``.
Example:
.. code-block:: yaml
library: true
* - ``suites``
- Optional for packs that define query suites
- The path to a directory in the pack that contains the query suites you want to make known to the CLI, defined relative to the pack directory. CodeQL pack users can run "well-known" suites stored in this directory by specifying the pack name, without providing their full path. This is not supported for CodeQL packs downloaded from the Container registry. For more information about query suites, see ":doc:`Creating CodeQL query suites <creating-codeql-query-suites>`."
Example:
.. code-block:: yaml
suites: octo-org-query-suites
* - ``tests``
- Optional for packs containing CodeQL tests. Ignored for packs without tests.
- The path to a directory within the pack that contains tests, defined relative to the pack directory. Use ``.`` to specify the whole pack. Any queries in this directory are run as tests when ``test run`` is run with the ``--strict-test-discovery`` option. These queries are ignored by query suite definitions that use ``queries`` or ``qlpack`` instructions to ask for all queries in a particular pack. If this property is missing, then ``.`` is assumed.
Example:
.. code-block:: yaml
tests: .
* - ``extractor``
- All packs containing CodeQL tests
- The CodeQL language extractor to use when running the CodeQL tests in the pack. For more information about testing queries, see ":doc:`Testing custom queries <testing-custom-queries>`."
Example:
.. code-block:: yaml
extractor: javascript
* - ``authors``
- Optional
- Metadata that will be displayed on the packaging search page in the packages section of the account that the CodeQL pack is published to.
Example:
.. code-block:: yaml
authors: author1@github.com,author2@github.com
* - ``license``
- Optional
- Metadata that will be displayed on the packaging search page in the packages section of the account that the CodeQL pack is published to. For a list of allowed licenses, see `SPDX License List <https://spdx.org/licenses/>`__ in the SPDX Specification.
Example:
.. code-block:: yaml
license: MIT
* - ``description``
- Optional
- Metadata that will be displayed on the packaging search page in the packages section of the account that the CodeQL pack is published to.
Example:
.. code-block:: yaml
description: Human-readable description of the contents of the CodeQL pack.
* - ``libraryPathDependencies``
- Optional, deprecated
- Use the ``dependencies`` property instead. The names of any CodeQL packs that this CodeQL pack depends on, as an array. This gives the pack access to any libraries, database schema, and query suites defined in the dependency.
Example:
.. code-block:: yaml
libraryPathDependencies: codeql/javascript-all
* - ``dbscheme``
- Required by core language packs only
- The path to the :ref:`database schema <codeql-database-schema>` for all libraries and queries written for this CodeQL language (see example below).
Example:
.. code-block:: yaml
dbscheme: semmlecode.python.dbscheme
* - ``upgrades``
- Required by core language packs only
- The path to a directory within the pack that contains database upgrade scripts, defined relative to the pack directory. Database upgrades are used internally to ensure that a database created with a different version of the CodeQL CLI is compatible with the current version of the CLI.
Example:
.. code-block:: yaml
upgrades: .
.. _about-codeql-pack-lock:
About ``codeql-pack.lock.yml`` files
------------------------------------
``codeql-pack.lock.yml`` files store the versions of the resolved transitive dependencies of a CodeQL pack. This file is created by the ``codeql pack install`` command if it does not already exist and should be added to your version control system. The ``dependencies`` section of the ``qlpack.yml`` file contains version ranges that are compatible with the pack. The ``codeql-pack.lock.yml`` file locks the versions to precise dependencies. This ensures that running ``codeql pack install`` on this the pack will always retrieve the same versions of dependencies even if newer compatible versions exist.
For example, if a ``qlpack.yml`` file contains the following dependencies:
.. code-block:: yaml
dependencies:
codeql/cpp-all: ^0.1.2
my-user/my-lib: ^0.2.3
other-dependency/from-source: "*"
The ``codeql-pack.lock.yml`` file will contain something like the following:
.. code-block:: yaml
dependencies:
codeql/cpp-all:
version: 0.1.4
my-user/my-lib:
version: 0.2.4
my-user/transitive-dependency:
version: 1.2.4
The ``codeql/cpp-all`` dependency is locked to version 0.1.4. The ``my-user/my-lib`` dependency is locked to version 0.2.4. The ``my-user/transitive-dependency``, which is a transitive dependency and is not specified in the ``qlpack.yml`` file, is locked to version 1.2.4. The ``other-dependency/from-source`` is absent from the lock file since it is resolved from source. This dependency must be available in the same CodeQL workspace as the pack. For more information about CodeQL workspaces and resolving dependencies from source, see ":doc:`About CodeQL Workspaces <about-codeql-workspaces>`."
In most cases, the ``codeql-pack.lock.yml`` file is only relevant for query packs since library packs are non-executable and usually do not need their transitive dependencies to be fixed. The exception to this is for library packs that contain tests. In this case, the ``codeql-pack.lock.yml`` file is used to ensure that the tests are always run with the same versions of dependencies to avoid spurious failures when there are mismatched dependencies.
.. _custom-codeql-packs:
Examples of custom CodeQL packs
-------------------------------
When you write custom queries or tests, you should save them in custom CodeQL packs. For simplicity, try to organize each pack logically. For more information, see "`CodeQL pack structure <#codeql-pack-structure>`__." Save files for queries and tests in separate packs and, where possible, organize custom packs into specific folders for each target language. This is particuarly useful if you intend to publish your CodeQL packs so they can be shared with others or used in GitHub `Code scanning <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning>`__.
CodeQL packs for custom libraries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A custom CodeQL pack containing custom C++ libraries, with no queries or tests, may have a ``qlpack.yml`` file containing:
.. code-block:: yaml
name: my-github-user/my-custom-libraries
version: 1.2.3
library: true
dependencies:
codeql/cpp-all: ^0.1.2
where ``codeql/cpp-all`` is the name of the CodeQL pack for C/C++ analysis included in the CodeQL repository. The version range ``^0.1.2`` indicates that this pack is compatible with all versions of ``codeql/cpp-all`` that are greater than or equal to ``0.1.2`` and less than ``0.2.0``. Any CodeQL library file (a file with a ``.qll`` extension) defined in this pack will be available to queries defined in any query pack that includes this pack in its dependencies block.
The ``library`` property indicates that this pack is a library pack and does not contain any queries.
CodeQL packs for custom queries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A custom CodeQL pack containing custom C++ queries and libraries may have a ``qlpack.yml`` file containing:
.. code-block:: yaml
name: my-github-user/my-custom-queries
version: 1.2.3
dependencies:
codeql/cpp-all: ^0.1.2
my-github-user/my-custom-libraries: ^1.2.3
suites: my-custom-suites
where ``codeql/cpp-all`` is the name of the CodeQL pack for C/C++ analysis included in the CodeQL repository. The version range ``^0.1.2`` indicates that this pack is compatible with all versions of ``codeql/cpp-all`` that are greater than or equal to ``0.1.2`` and less than ``0.2.0``. ``my-github-user/my-custom-libraries`` is the name of a CodeQL pack containing custom CodeQL libraries for C++. Any CodeQL library file (a file with a ``.qll`` extension) defined in this pack will be available to queries in the ``my-github-user/my-custom-queries`` pack.
The ``suites`` property indicates a directory where "well-known" query suites can be found. These suites can be used on the command line by referring to their name only, rather than their full path. For more information about query suites, see ":doc:`Creating CodeQL query suites <creating-codeql-query-suites>`."
CodeQL packs for custom tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For custom CodeQL packs containing test files, you also need to include an
``extractor`` property so that the ``test run`` command knows how to create test
databases. You may also wish to specify the ``tests`` property.
.. include:: ../reusables/test-qlpack.rst
For more information about running tests, see ":doc:`Testing custom queries
<testing-custom-queries>`."
.. _standard-codeql-packs:
Examples of CodeQL packs in the CodeQL repository
-------------------------------------------------
Each of the languages in the CodeQL repository has four main CodeQL packs:
- Core library pack for the language, with the :ref:`database schema <codeql-database-schema>`
used by the language, and CodeQL libraries, and queries at ``<language>/ql/lib``
- Core query pack for the language that includes the default queries for the language, along
with their query suites at ``<language>/ql/src``
- Tests for the core language libraries and queries at ``<language>/ql/test``
- Example queries for the language at ``<language>/ql/examples``
Core library pack
~~~~~~~~~~~~~~~~~
Here is an example ``qlpack.yml`` file for the `C/C++ analysis libraries
<https://github.com/github/codeql/blob/main/cpp/ql/lib/qlpack.yml>`__
core language pack:
.. code-block:: yaml
name: codeql/cpp-all
version: x.y.z-dev
dbscheme: semmlecode.cpp.dbscheme
library: true
upgrades: upgrades
Some extra notes on the following properties:
- ``library``: Indicates that this is a library pack with no executable queries. It is only meant to be used as a dependency for other packs.
- ``dbscheme`` and ``upgrades``: These properties are internal to the CodeQL CLI and should only be defined in the core QL pack for a language.
.. _standard-codeql-query-packs:
Core query pack
~~~~~~~~~~~~~~~
Here is an example ``qlpack.yml`` file for `C/C++ analysis queries
<https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml>`__
core query pack:
.. code-block:: yaml
name: codeql/cpp-queries
version: x.y.z-dev
dependencies:
codeql/cpp-all: "*"
codeql/suite-helpers: "*"
suites: codeql-suites
defaultSuiteFile: codeql-suites/cpp-code-scanning.qls
Some extra notes on the following properties:
- ``dependencies``: This query pack depends on ``codeql/cpp-all`` and ``codeql/suite-helpers``. Since these dependencies are resolved from source, it does not matter what version of the CodeQL pack they are compatible with. For more information about resolving dependencies from source, see ":ref:`Source Dependencies <source-dependencies>`."
- ``suites``: Indicates the directory containing "well-known" query suites.
- ``defaultSuiteFile``: The name of the default query suite file that is used when no query suite is specified.
Tests for the core CodeQL pack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here is an example ``qlpack.yml`` file for `C/C++ analysis tests
<https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml>`__
core test pack:
.. code-block:: yaml
name: codeql/cpp-tests
dependencies:
codeql/cpp-all: "*"
codeql/cpp-queries: "*"
extractor: cpp
tests: .
Some extra notes on the following properties:
- ``dependencies``: This pack depends on the core CodeQL query and library packs for C++.
- ``extractor``: This specifies that all the tests will use the same C++ extractor to create the database for the tests.
- ``tests``: This specifies the location of the tests. In this case, the tests are in the root folder (and all sub-folders) of the pack.
- ``version``: There is no ``version`` property for the tests pack. This prevents test packs from accidentally being published.
.. pull-quote::
This article was moved to "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

102
docs/codeql/codeql-cli/about-codeql-workspaces.rst
View File

@@ -3,100 +3,8 @@
About CodeQL workspaces
=======================
.. include:: ../reusables/beta-note-package-management.rst
CodeQL workspaces are used to group multiple CodeQL packs together. A typical use case for a CodeQL workspace is to develop a set of CodeQL library and query packs that are mutually dependent. For more information on CodeQL packs, see ":doc:`About CodeQL packs <about-codeql-packs>`."
The main benefit of a CodeQL workspace is that it makes it easier for you to develop and maintain multiple CodeQL packs. When you use a CodeQL workspace, all the CodeQL packs in the workspace are available as *source dependencies* for each other when you run a CodeQL command that resolves queries. This makes it easier to develop, maintain, and publish multiple, related CodeQL packs.
In most cases, you should store the CodeQL workspace and the CodeQL packs contained in it in one git repository. This makes it easier to share your CodeQL development environment.
The ``codeql-workspace.yml`` file
---------------------------------
A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally ``ignore`` and ``registries`` blocks.
* The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace.
* The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace.
* The ``registries`` block contains a list of GHES URLs and package patterns that control which container registry is used for publishing CodeQL packs. For more information, see :ref:`Working with CodeQL packs on GitHub Enterprise Server <working-with-codeql-packs-on-ghes>`.
Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see "`@actions/glob <https://github.com/actions/toolkit/tree/main/packages/glob#patterns>`__ ."
For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all the CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory. The ``registries`` block specifies that ``codeql/*`` packs should be downloaded from https://ghcr.io/v2/, which is GitHub's default container registry. All other packs should be downloaded from and published to the regsitry at ``GHE_HOSTNAME``.
.. code-block:: yaml
provide:
- "*/codeql-packs/**/qlpack.yml"
ignore:
- "*/codeql-packs/**/experimental/**/qlpack.yml"
registries:
- packages: 'codeql/*'
url: https://ghcr.io/v2/
- packages: '*'
url: https://containers.GHE_HOSTNAME/v2/
To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that you expect, run the ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace.
.. _source-dependencies:
Source Dependencies
-------------------
Source dependencies are CodeQL packs that are resolved from the local file system outside of the CodeQL package cache. These dependencies can be in the same CodeQL workspace, or specified as a path option using the ``--additional-packs`` argument. When you compile and run queries locally, source dependencies override any dependencies found in the CodeQL package cache as well as version constraints defined in the ``qlpack.yml``. All references to CodeQL packs in the same workspace are resolved as source dependencies.
This is particularly useful in the following situations:
- One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to reference that pack.
- You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it.
CodeQL workspaces and query resolution
--------------------------------------
All CodeQL packs in a workspace are available as source dependencies for each other when you run any CodeQL command that resolves queries or packs. For example, when you run ``codeql pack install`` in a pack directory in a workspace, any dependency that can be found in the workspace will be used instead of downloading that dependency to the package cache and adding it to the ``codeql-pack.lock.yml`` file. For more information, see ":ref:`Adding and Installing Dependencies <adding-and-installing-dependencies>`."
Similarly, when you publish a CodeQL query pack to the GitHub container registry using ``codeql pack publish`` the command will always use the dependencies from the workspace instead of using dependencies found in the local package cache.
This ensures that any local changes you make to a query library in a dependency are automatically reflected in any query packs you publish from that workspace.
Example
~~~~~~~
Consider the following ``codeql-workspace.yml`` file:
.. code-block:: yaml
provide:
- "**/qlpack.yml"
And the following CodeQL library pack ``qlpack.yml`` file in the workspace:
.. code-block:: yaml
name: my-company/my-library
library: true
version: 1.0.0
And the following CodeQL query pack ``qlpack.yml`` file in the workspace:
.. code-block:: yaml
name: my-company/my-queries
version: 1.0.0
dependencies:
my-company/my-library: "*"
codeql/cpp-all: ~0.2.0
Notice that the ``dependencies`` block for the CodeQL query pack, ``my-company/my-queries``, specifies ``"*"`` as the version of the library pack. Since the library pack is already defined as a source dependency in ``codeql-workspace.yml``, the library pack's content is always resolved from inside the workspace. Any version constraint you define will be ignored in this case. We recommend that you use ``"*"`` for source dependencies to make it clear that the version is inherited from the workspace.
When you execute ``codeql pack install`` from the query pack directory, an appropriate version of ``codeql/cpp-all`` is downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file is created that contains the resolved version of ``codeql/cpp-all``. The lock file won't contain an entry for ``my-company/my-library`` since it is resolved from source dependencies. The ``codeql-pack.lock.yml`` file will look something like this:
.. code-block:: yaml
dependencies:
codeql/cpp-all:
version: 0.2.2
When you execute ``codeql pack publish`` from the query pack directory, the ``codeql/cpp-all`` dependency from the package cache and the ``my-company/my-library`` from the workspace are bundled with ``my-company/my-queries`` and published to the GitHub container registry.
.. pull-quote::
This article was moved to "`About CodeQL workspaces <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-workspaces>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

11
docs/codeql/codeql-cli/about-ql-packs.rst
View File

@@ -3,10 +3,7 @@
About QL packs
==============
This page has been moved to ":doc:`About CodeQL packs <about-codeql-packs>`."
.. toctree::
:hidden:
about-ql-packs
.. pull-quote::
This article was moved to "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

28
docs/codeql/codeql-cli/about-the-codeql-cli.rst
View File

@@ -3,27 +3,7 @@
About the CodeQL CLI
====================
Software developers and security researchers can secure their code
using the CodeQL CLI.
The CodeQL CLI is a command-line tool used to run CodeQL processes locally on
open source software projects. You can use the CodeQL CLI to:
- Run CodeQL analyses using queries provided by GitHub engineers and the open
source community
- Create CodeQL databases to use in the CodeQL for Visual Studio Code
- Develop and test custom CodeQL queries to use in your own analyses
For information about using the CodeQL CLI, see
":ref:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`."
CodeQL CLI commands
-------------------
The CodeQL CLI includes commands to create and analyze CodeQL databases from the
command line. To run a command, use::
codeql [command] [subcommand]
To view the reference documentation for a command, add the ``--help`` flag, or visit the
"`CodeQL CLI manual <../manual>`__."
.. pull-quote::
This article was moved to "`About the CodeQL CLI <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/about-the-codeql-cli>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

394
docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst
View File

@@ -3,393 +3,7 @@
Analyzing databases with the CodeQL CLI
=======================================
To analyze a codebase, you run queries against a CodeQL
database extracted from the code.
CodeQL analyses produce :ref:`interpreted results
<interpret-query-results>` that can be displayed as alerts or paths in source code.
For information about writing queries to run with ``database analyze``, see
":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
.. include:: ../reusables/advanced-query-execution.rst
Before starting an analysis you must:
- :doc:`Set up the CodeQL CLI <getting-started-with-the-codeql-cli>` to run commands locally.
- :doc:`Create a CodeQL database <creating-codeql-databases>` for the source
code you want to analyze.
The simplest way to run ``codeql database analyze`` is using CodeQL packs. You can also
run the command using queries from a local checkout of the CodeQL repository,
which you may want to do if you want to customize the CodeQL core queries.
Running ``codeql database analyze``
------------------------------------
When you run ``database analyze``, it:
#. Optionally downloads any referenced CodeQL packages that are not available locally.
#. Executes one or more query files, by running them over a CodeQL database.
#. Interprets the results, based on certain query metadata, so that alerts can be
displayed in the correct location in the source code.
#. Reports the results of any diagnostic and summary queries to standard output.
You can analyze a database by running the following command::
codeql database analyze <database> --format=<format> --output=<output> <query-specifiers>...
You must specify:
- ``<database>``: the path to the CodeQL database you want to analyze.
- ``--format``: the format of the results file generated during analysis. A
number of different formats are supported, including CSV, :ref:`SARIF
<sarif-file>`, and graph formats. For more information about CSV and SARIF,
see `Results <#results>`__. To find out which other results formats are
supported, see the `database analyze reference
<../manual/database-analyze>`__.
- ``--output``: the output path of the results file generated during analysis.
You can also specify:
- ``<query-specifiers>...``: a space-separated list of queries to run over your database. This
is a list of arguments, where each argument can be:
- a path to a query file
- a path to a directory containing query files
- a path to a query suite file
- the name of a CodeQL query pack
- with an optional version range
- with an optional path to a query, directory, or query suite inside the pack
If omitted, the default query suite for the language of the analyzed database will be used. For the complete syntax of query specifiers, see :ref:`"Specifying which queries to run in a CodeQL pack"<specifying-which-queries>`.
- ``--sarif-category``: an identifying category for the results. Used when
you want to upload more than one set of results for a commit.
For example, when you use ``github upload-results`` to send results for more than one
language to the GitHub code scanning API. For more information about this use case,
see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__ in the GitHub documentation.
- ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written
in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be
converted to ``.md`` before running the analysis. For further information,
see ":ref:`Including query help for custom CodeQL queries in SARIF files <including-query-help-for-custom-codeql-queries-in-sarif-files>`."
- ``--download``: a boolean flag that will allow the CLI to download any referenced CodeQL packages that are not available locally.
If this flag is missing and a referenced CodeQL package is not available locally, the command will fail.
- .. include:: ../reusables/threads-query-execution.rst
.. pull-quote::
Upgrading databases
For databases that were created by CodeQL CLI v2.3.3 or earlier, you will need
to explicitly upgrade the database before you can run an analysis with a newer
version of the CodeQL CLI. If this step is necessary, then you will see a message telling you
that your database needs to be upgraded when you run ``database analyze``.
For databases that were created by CodeQL CLI v2.3.4 or later, the CLI will implicitly run any
required upgrades. Explicitly running the upgrade command is not necessary.
For full details of all the options you can use when analyzing databases, see
the `database analyze reference documentation <../manual/database-analyze>`__.
.. _specifying-which-queries:
Specifying which queries to run in a CodeQL pack
------------------------------------------------
Query specifiers are used by ``codeql database analyze`` and other commands that operate on a set of queries.
The complete form of a query specifier is``scope/name@range:path``, where:
- ``scope/name`` is the qualified name of a CodeQL pack.
- ``range`` is a `semver range <https://docs.npmjs.com/cli/v6/using-npm/semver#ranges>`_.
- ``path`` is a file system path to a single query, a directory containing queries, or a query suite file.
When you specify a ``scope/name``, the ``range`` and ``path`` are
optional. If you omit a ``range`` then the latest version of the
specified pack is used. If you omit a ``path`` then the default query suite
of the specified pack is used.
The ``path`` can be one of: a ``.ql`` query file, a directory
containing one or more queries, or a ``.qls`` query suite file. If
you omit a pack name, then you must provide a ``path``,
which will be interpreted relative to the working directory
of the current process. Glob patterns are not supported.
If you specify both a ``scope/name`` and ``path``, then the ``path`` cannot
be absolute. It is considered relative to the root of the CodeQL
pack.
Example query specifiers
~~~~~~~~~~~~~~~~~~~~~~~~
* ``codeql/python-queries`` - All the queries in the default query suite of the latest version of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@1.2.3`` - All the queries in the default query suite of version ``1.2.3`` of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@~1.2.3`` - All the queries in the default query suite of the latest version of the ``codeql/python-queries`` pack that is >= ``1.2.3`` and < ``1.3.0``.
* ``codeql/python-queries:Functions`` - All queries in the ``Functions`` directory in the latest version of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@1.2.3:Functions`` - All queries in the ``Functions`` directory in version 1.2.3 of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@1.2.3:codeql-suites/python-code-scanning.qls`` - All queries in the ``codeql-suites/python-code-scanning.qls`` directory in version 1.2.3 of the ``codeql/python-queries`` pack.
* ``suites/my-suite.qls`` - All queries in the ``suites/my-suite.qls`` file relative to the current working directory.
.. pull-quote::
Tip
The default query suite of the standard CodeQL query packs are ``codeql-suites/<lang>-code-scanning.qls``. Several other useful query suites can also be found in the ``codeql-suites`` directory of each pack. For example, the ``codeql/cpp-queries`` pack contains the following query suites:
* ``cpp-code-scanning.qls`` - Standard Code Scanning queries for C++. The default query suite for this pack.
* ``cpp-security-extended.qls`` - Queries from the default ``cpp-code-scanning.qls`` suite for C++, plus lower severity and precision queries.
* ``cpp-security-and-quality.qls`` - Queries from ``cpp-security-extended.qls``, plus maintainability and reliability queries.
You can see the sources for these query suites in the `CodeQL repository <https://github.com/github/codeql/tree/main/cpp/ql/src/codeql-suites>`__. Query suites for other languages are similar.
Examples of running database analyses
---------------------------------------------
The following examples show how to run ``database analyze`` using CodeQL packs, and how to use a local checkout of the CodeQL repository. These examples assume your CodeQL databases have been created in a directory that is a sibling of your local copies of the CodeQL repository.
.. _run-query-pack:
Running a CodeQL query pack
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: ../reusables/beta-note-package-management.rst
To run an existing CodeQL query pack from the GitHub Container registry, you can specify one or more
pack names::
codeql database analyze <database> microsoft/coding-standards@1.0.0 github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
This command runs the default query suite of two CodeQL query packs: ``microsoft/coding-standards`` version 1.0.0 and the latest version of ``github/security-queries`` on the specified database. For further information about default suites, see ":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`".
The ``--download`` flag is optional. Using it will ensure the query pack is downloaded if it isn't yet available locally.
Running a single query
~~~~~~~~~~~~~~~~~~~~~~
To run a single query over a CodeQL database for a JavaScript codebase,
you could use the following command from the directory containing your database::
codeql database analyze --download <javascript-database> codeql/javascript-queries:Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
This command runs a simple query that finds potential bugs related to unused
variables, imports, functions, or classes---it is one of the JavaScript
queries included in the CodeQL repository. You could run more than one query by
specifying a space-separated list of similar paths.
The analysis generates a CSV file (``js-results.csv``) in a new directory (``js-analysis``).
Alternatively, if you have the CodeQL repository checked out, you can execute the same queries by specifying the path to the query directly::
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
You can also run your own custom queries with the ``database analyze`` command.
For more information about preparing your queries to use with the CodeQL CLI,
see ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
Running all queries in a directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can run all the queries located in a directory by providing the directory
path, rather than listing all the individual query files. Paths are searched
recursively, so any queries contained in subfolders will also be executed.
.. pull-quote::
Important
You should avoid specifying the root of a :ref:`core CodeQL query pack
<standard-codeql-query-packs>` when executing ``database analyze``
as it might contain some special queries that aren't designed to be used with
the command. Rather, :ref:`run the query pack <run-query-pack>` to include the
pack's default queries in the analysis, or run one of the
code scanning query suites.
For example, to execute all Python queries contained in the ``Functions`` directory in the
``codeql/python-queries`` query pack you would run::
codeql database analyze <python-database> codeql/python-queries:Functions --format=sarif-latest --output=python-analysis/python-results.sarif --download
Alternatively, if you have the CodeQL repository checked out, you can execute the
same queries by specifying the path to the directory directly::
codeql database analyze <python-database> ../ql/python/ql/src/Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif
When the analysis has finished, a SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures
that the results are formatted according to the most recent SARIF specification
supported by CodeQL.
Running a subset of queries in a CodeQL pack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are using CodeQL CLI v2.8.1 or later, you can include a path at the end of a pack specification to run a subset of queries inside the pack. This applies to any command that locates or runs queries within a pack.
The complete way to specify a set of queries is in the form ``scope/name@range:path``, where:
- ``scope/name`` is the qualified name of a CodeQL pack.
- ``range`` is a `semver range <https://docs.npmjs.com/cli/v6/using-npm/semver#ranges>`_.
- ``path`` is a file system path to a single query, a directory containing queries, or a query suite file.
When you specify a ``scope/name``, the ``range`` and ``path`` are
optional. If you omit a ``range`` then the latest version of the
specified pack is used. If you omit a ``path`` then the default query suite
of the specified pack is used.
The ``path`` can be one of a ``*.ql`` query file, a directory
containing one or more queries, or a ``.qls`` query suite file. If
you omit a pack name, then you must provide a ``path``,
which will be interpreted relative to the working directory
of the current process.
If you specify a ``scope/name`` and ``path``, then the ``path`` cannot
be absolute. It is considered relative to the root of the CodeQL
pack.
To analyze a database using all queries in the `experimental/Security` folder within the `codeql/cpp-queries` CodeQL pack you can use::
codeql database analyze --format=sarif-latest --output=results <db> \
codeql/cpp-queries:experimental/Security
To run the `RedundantNullCheckParam.ql` query in the `codeql/cpp-queries` CodeQL pack use::
codeql database analyze --format=sarif-latest --output=results <db> \
'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql'
To analyze your database using the `cpp-security-and-quality.qls` query suite from a version of the `codeql/cpp-queries` CodeQL pack that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen) you can use::
codeql database analyze --format=sarif-latest --output=results <db> \
'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls'
If you need to reference a query file, directory, or suite whose path contains a literal `@` or `:`, you can prefix the query specification with `path:` like so::
codeql database analyze --format=sarif-latest --output=results <db> \
path:C:/Users/ci/workspace@2/security/query.ql
For more information about CodeQL packs, see :doc:`About CodeQL Packs <about-codeql-packs>`.
Running query suites
~~~~~~~~~~~~~~~~~~~~
To run a query suite on a CodeQL database for a C/C++ codebase,
you could use the following command from the directory containing your database::
codeql database analyze <cpp-database> codeql/cpp-queries:codeql-suites/cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif --download
This command downloads the ``codeql/cpp-queries`` CodeQL query pack, runs the analysis, and generates a file in the SARIF version 2.1.0 format that is supported by all versions of GitHub. This file can be uploaded to GitHub by executing ``codeql github upload-results`` or the code scanning API.
For more information, see `Analyzing a CodeQL database <https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__
or `Code scanning API <https://docs.github.com/en/rest/reference/code-scanning>`__ in the GitHub documentation.
CodeQL query suites are ``.qls`` files that use directives to select queries to run
based on certain metadata properties. The standard CodeQL packs have metadata that specify
the location of the query suites used by code scanning, so the CodeQL CLI knows where to find these
suite files automatically, and you don't have to specify the full path on the command line.
For more information, see ":ref:`Creating CodeQL query suites <creating-codeql-query-suites>`."
For information about creating custom query suites, see ":doc:`Creating
CodeQL query suites <creating-codeql-query-suites>`."
Diagnostic and summary information
..................................
When you create a CodeQL database, the extractor stores diagnostic data in the database. The code scanning query suites include additional queries to report on this diagnostic data and calculate summary metrics. When the ``database analyze`` command completes, the CLI generates the results file and reports any diagnostic and summary data to standard output. If you choose to generate SARIF output, the additional data is also included in the SARIF file.
If the analysis found fewer results for standard queries than you expected, review the results of the diagnostic and summary queries to check whether the CodeQL database is likely to be a good representation of the codebase that you want to analyze.
Integrating a CodeQL pack into a code scanning workflow in GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use CodeQL query packs in your code scanning setup. This allows you to select query packs published by various sources and use them to analyze your code.
For more information, see "`Using CodeQL query packs in the CodeQL action <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-codeql-query-packs/>`_" or "`Downloading and using CodeQL query packs in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#downloading-and-using-codeql-query-packs>`_."
.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
Including query help for custom CodeQL queries in SARIF files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you use the CodeQL CLI to run code scanning analyses on third party CI/CD systems,
you can include the query help for your custom queries in SARIF files generated during an analysis.
After uploading the SARIF file to GitHub, the query help is shown in the code scanning UI for any
alerts generated by the custom queries.
From CodeQL CLI v2.7.1 onwards, you can include markdown-rendered query help in SARIF files
by providing the ``--sarif-add-query-help`` option when running
``codeql database analyze``.
For more information, see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__
in the GitHub documentation.
You can write query help for custom queries directly in a markdown file and save it alongside the
corresponding query. Alternatively, for consistency with the standard CodeQL queries,
you can write query help in the ``.qhelp`` format. Query help written in ``.qhelp``
files can't be included in SARIF files, and they can't be processed by code
scanning so must be converted to markdown before running
the analysis. For more information, see ":ref:`Query help files <query-help-files>`"
and ":doc:`Testing query help files <testing-query-help-files>`."
Results
-------
You can save analysis results in a number of different formats, including SARIF
and CSV.
The SARIF format is designed to represent the output of a broad range of static
analysis tools. For more information, see :doc:`SARIF output <sarif-output>`.
If you choose to generate results in CSV format, then each line in the output file
corresponds to an alert. Each line is a comma-separated list with the following information:
.. list-table::
:header-rows: 1
:widths: 20 40 40
* - Property
- Description
- Example
* - Name
- Name of the query that identified the result.
- ``Inefficient regular expression``
* - Description
- Description of the query.
- ``A regular expression that requires exponential time to match certain
inputs can be a performance bottleneck, and may be vulnerable to
denial-of-service attacks.``
* - Severity
- Severity of the query.
- ``error``
* - Message
- Alert message.
- ``This part of the regular expression may cause exponential backtracking
on strings containing many repetitions of '\\\\'.``
* - Path
- Path of the file containing the alert.
- ``/vendor/codemirror/markdown.js``
* - Start line
- Line of the file where the code that triggered the alert begins.
- ``617``
* - Start column
- Column of the start line that marks the start of the alert code. Not
included when equal to 1.
- ``32``
* - End line
- Line of the file where the code that triggered the alert ends. Not
included when the same value as the start line.
- ``64``
* - End column
- Where available, the column of the end line that marks the end of the
alert code. Otherwise the end line is repeated.
- ``617``
Results files can be integrated into your own code-review or debugging
infrastructure. For example, SARIF file output can be used to highlight alerts
in the correct location in your source code using a SARIF viewer plugin for your
IDE.
Further reading
---------------
- ":ref:`Analyzing your projects in CodeQL for VS Code <analyzing-your-projects>`"
.. pull-quote::
This article was moved to "`Analyzing databases with the CodeQL CLI <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

24
docs/codeql/codeql-cli/codeql-cli-reference.rst
View File

@@ -3,23 +3,7 @@
CodeQL CLI reference
====================
Learn more about the files you can use when running CodeQL processes and the results format and exit codes that CodeQL generates.
.. toctree::
:titlesonly:
:hidden:
about-codeql-packs
about-codeql-workspaces
query-reference-files
sarif-output
exit-codes
extractor-options
- :doc:`About CodeQL packs <about-codeql-packs>`: CodeQL packs are created with the CodeQL CLI and are used to create, depend on, publish, and run CodeQL queries, libraries, and query suites.
- :doc:`About CodeQL workspaces <about-codeql-workspaces>`: CodeQL workspaces are used to group multiple CodeQL packs together.
- :doc:`Query reference files <query-reference-files>`: A query reference file is text file that defines the location of one query to test.
- :doc:`SARIF output <sarif-output>`: CodeQL supports SARIF as an output format for sharing static analysis results.
- :doc:`Exit codes <exit-codes>`: The CodeQL CLI reports the status of each command it runs as an exit code.
This exit code provides information for subsequent commands or for other tools that rely on the CodeQL CLI.
- :doc:`Extractor options <extractor-options>`: You can customize the behavior of extractors by setting options through the CodeQL CLI.
.. pull-quote::
This category was moved to "`CodeQL CLI reference <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

94
docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst
View File

@@ -3,93 +3,7 @@
Creating and working with CodeQL packs
======================================
You can use CodeQL packs to create, share, depend on, and run CodeQL queries and libraries.
.. include:: ../reusables/beta-note-package-management.rst
About CodeQL packs and the CodeQL CLI
-------------------------------------
With CodeQL packs and the package management commands in the CodeQL CLI, you can publish your custom queries and integrate them into your codebase analysis.
There are two types of CodeQL packs: query packs and library packs.
* Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and pre-compiled representations of each query, in addition to the query sources. This ensures consistent and efficient execution of the queries in the pack.
* Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled separately.
You can use the ``pack`` command in the CodeQL CLI to create CodeQL packs, add dependencies to packs, and install or update dependencies. You can also publish and download CodeQL packs using the ``pack`` command. For more information, see ":doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
For more information about compatibility between published query packs and different CodeQL releases, see ":ref:`About CodeQL pack compatibility <about-codeql-pack-compatibility>`."
Creating a CodeQL pack
----------------------
You can create a CodeQL pack by running the following command from the checkout root of your project:
::
codeql pack init <scope>/<pack>
You must specify:
- ``<scope>``: the name of the GitHub organization or user account that you will publish to.
- ``<pack>``: the name for the pack that you are creating.
The ``codeql pack init`` command creates the directory structure and configuration files for a CodeQL pack. By default, the command creates a query pack. If you want to create a library pack, you must edit the ``qlpack.yml`` file to explicitly declare the file as a library pack by including the ``library:true`` property.
Modifying an existing legacy QL pack to create a CodeQL pack
------------------------------------------------------------
If you already have a ``qlpack.yml`` file, you can edit it manually to convert it into a CodeQL pack.
#. Edit the ``name`` property so that it matches the format ``<scope>/<name>``, where ``<scope>`` is the name of the GitHub organization or user account that you will publish to.
#. In the ``qlpack.yml`` file, include a ``version`` property with a semver identifier, as well as an optional ``dependencies`` block.
#. Migrate the list of dependencies in ``libraryPathDependencies`` to the ``dependencies`` block. Specify the version range for each dependency. If the range is unimportant, or you are unsure of compatibility, you can specify ``"*"``, which indicates that any version is acceptable and will default to the latest version when you run ``codeql pack install``.
For more information about the properties, see ":ref:`About CodeQL packs <about-codeql-packs>`."
.. _adding-and-installing-dependencies:
Adding and installing dependencies to a CodeQL pack
---------------------------------------------------
You can add dependencies on CodeQL packs using the command ``codeql pack add``. You must specify the scope, name, and (optionally) a compatible version range.
::
codeql pack add <scope>/<name>@x.x.x <scope>/<other-name>
If you don't specify a version range, the latest version will be added. Otherwise, the latest version that satisfies the requested range will be added.
This command updates the ``qlpack.yml`` file with the requested dependencies and downloads them into the package cache. Please note that this command will reformat the file and remove all comments.
You can also manually edit the ``qlpack.yml`` file to include dependencies and install the dependencies with the command:
::
codeql pack install
This command downloads all dependencies to the shared cache on the local disk.
.. pull-quote::
Note
Running the ``codeql pack add`` and ``codeql pack install`` commands will generate or update the ``codeql-pack.lock.yml`` file. This file should be checked-in to version control. The ``codeql-pack.lock.yml`` file contains the precise version numbers used by the pack.
For more information, see ":ref:`About codeql-pack.lock.yml files <about-codeql-pack-lock>`."
.. pull-quote::
Note
By default ``codeql pack install`` will install dependencies from the Container registry on GitHub.com.
You can install dependencies from a GitHub Enterprise Server Container registry by creating a ``qlconfig.yml`` file.
For more information, see ":doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
Customizing a downloaded CodeQL pack
---------------------------------------------------
The recommended way to experiment with changes to a pack is to clone the repository containing its source code.
If no source respository is available and you need to base modifications on a pack downloaded from the Container registry, be aware that these packs are not intended to be modified or customized after downloading, and their format may change in the future without much notice. We recommend taking the following steps after downloading a pack if you need to modify the content:
- Change the pack *name* in ``qlpack.yml`` so you avoid confusion with results from the unmodified pack.
- Remove all files named ``*.qlx`` anywhere in the unpacked directory structure. These files contain precompiled versions of the queries, and in some situations CodeQL will use them in preference to the QL source you have modified.
.. pull-quote::
This article was moved to "`Creating and working with CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-and-working-with-codeql-packs>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

402
docs/codeql/codeql-cli/creating-codeql-databases.rst
View File

@@ -3,401 +3,7 @@
Creating CodeQL databases
=========================
Before you analyze your code using CodeQL, you need to create a CodeQL
database containing all the data required to run queries on your code.
You can create CodeQL databases yourself using the CodeQL CLI,
or download them from GitHub.com.
CodeQL analysis relies on extracting relational data from your code, and
using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL
databases contain all of the important information about a codebase, which can
be analyzed by executing CodeQL queries against it. GitHub creates and
stores CodeQL databases for a large number of open-source projects. For more information,
see ":ref:`Downloading CodeQL databases from GitHub.com <downloading-codeql-databases-from-github-com>`."
You can also create CodeQL databases yourself using the CodeQL CLI.
Before you generate a CodeQL database, you need to:
- Install and set up the CodeQL CLI. For more information, see
":doc:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`."
- Check out the version of your codebase you want to analyze. The directory
should be ready to build, with all dependencies already installed.
For information about using the CodeQL CLI in a third-party CI system to create results
to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__
in the GitHub documentation. For information about enabling CodeQL code scanning using GitHub Actions,
see `Setting up code scanning for a repository <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository>`__
in the GitHub documentation.
Running ``codeql database create``
----------------------------------
CodeQL databases are created by running the following command from the checkout root
of your project:
::
codeql database create <database> --language=<language-identifier>
You must specify:
- ``<database>``: a path to the new database to be created. This directory will
be created when you execute the command---you cannot specify an existing
directory.
- ``--language``: the identifier for the language to create a database for.
When used with ``--db-cluster``, the option accepts a comma-separated list,
or can be specified more than once.
CodeQL supports creating databases for the following languages:
.. include:: ../reusables/extractors.rst
You can specify additional options depending on the location of your source file,
if the code needs to be compiled, and if you want to create CodeQL databases for
more than one language:
- ``--source-root``: the root folder for the primary source files used in
database creation. By default, the command assumes that the current
directory is the source root---use this option to specify a different location.
- ``--db-cluster``: use for multi-language codebases when you want to create
databases for more than one language.
- ``--command``: used when you create a database for one or more compiled languages,
omit if the only languages requested are Python and JavaScript.
This specifies the build commands needed to invoke the compiler.
Commands are run from the current folder, or ``--source-root``
if specified. If you don't include a ``--command``, CodeQL will attempt to
detect the build system automatically, using a built-in autobuilder.
- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build
command for languages where the CodeQL CLI does not need to monitor the build
(for example, Python and JavaScript/TypeScript).
You can specify extractor options to customize the behavior of extractors that create CodeQL databases. For more information, see
":doc:`Extractor options <extractor-options>`."
For full details of all the options you can use when creating databases,
see the `database create reference documentation <../manual/database-create>`__.
Progress and results
--------------------
Errors are reported if there are any problems with the options you have
specified. For interpreted languages, the extraction progress is displayed in
the console---for each source file, it reports if extraction was successful or if
it failed. For compiled languages, the console will display the output of the
build system.
When the database is successfully created, you'll find a new directory at the
path specified in the command. If you used the ``--db-cluster`` option to create
more than one database, a subdirectory is created for each language.
Each CodeQL database directory contains a number of
subdirectories, including the relational data (required for analysis) and a
source archive---a copy of the source files made at the time the database was
created---which is used for displaying analysis results.
Creating databases for non-compiled languages
---------------------------------------------
The CodeQL CLI includes extractors to create databases for non-compiled
languages---specifically, JavaScript (and TypeScript), Python, and Ruby. These
extractors are automatically invoked when you specify JavaScript, Python, or Ruby as
the ``--language`` option when executing ``database create``. When creating
databases for these languages you must ensure that all additional dependencies
are available.
.. pull-quote:: Important
When you run ``database create`` for JavaScript, TypeScript, Python, and Ruby, you should not
specify a ``--command`` option. Otherwise this overrides the normal
extractor invocation, which will create an empty database. If you create
databases for multiple languages and one of them is a compiled language,
use the ``--no-run-unnecessary-builds`` option to skip the command for the languages that don't need to be compiled.
JavaScript and TypeScript
~~~~~~~~~~~~~~~~~~~~~~~~~
Creating databases for JavaScript requires no additional dependencies, but if
the project includes TypeScript files, you must install Node.js 6.x
or later. In the command line you can specify ``--language=javascript`` to
extract both JavaScript and TypeScript files::
codeql database create --language=javascript --source-root <folder-to-extract> <output-folder>/javascript-database
Here, we have specified a ``--source-root`` path, which is the location where
database creation is executed, but is not necessarily the checkout root of the
codebase.
By default, files in ``node_modules`` and ``bower_components`` directories are not extracted.
Python
~~~~~~
When creating databases for Python you must ensure:
- You have Python 3 installed and available to the CodeQL extractor.
- You have the version of Python used by your code installed.
- You have access to the `pip <https://pypi.org/project/pip/>`__
packaging management system and can install any
packages that the codebase depends on.
- You have installed the `virtualenv <https://pypi.org/project/virtualenv/>`__ pip module.
In the command line you must specify ``--language=python``. For example::
::
codeql database create --language=python <output-folder>/python-database
This executes the ``database create`` subcommand from the code's checkout root,
generating a new Python database at ``<output-folder>/python-database``.
Ruby
~~~~
Creating databases for Ruby requires no additional dependencies.
In the command line you must specify ``--language=ruby``. For example::
codeql database create --language=ruby --source-root <folder-to-extract> <output-folder>/ruby-database
Here, we have specified a ``--source-root`` path, which is the location where
database creation is executed, but is not necessarily the checkout root of the
codebase.
Creating databases for compiled languages
-----------------------------------------
For compiled languages, CodeQL needs to invoke the required build system to
generate a database, therefore the build method must be available to the CLI.
Detecting the build system
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: ../reusables/kotlin-beta-note.rst
The CodeQL CLI includes autobuilders for C/C++, C#, Go, Java and Kotlin code. CodeQL
autobuilders allow you to build projects for compiled languages without
specifying any build commands. When an autobuilder is invoked, CodeQL examines
the source for evidence of a build system and attempts to run the optimal set of
commands required to extract a database.
An autobuilder is invoked automatically when you execute ``codeql database
create`` for a compiled ``--language`` if don't include a
``--command`` option. For example, for a Java codebase, you would simply run::
codeql database create --language=java <output-folder>/java-database
If a codebase uses a standard build system, relying on an autobuilder is often
the simplest way to create a database. For sources that require non-standard
build steps, you may need to explicitly define each step in the command line.
.. pull-quote:: Creating databases for Go
For Go, install the Go toolchain (version 1.11 or later) and, if there
are dependencies, the appropriate dependency manager (such as `dep
<https://golang.github.io/dep/>`__).
The Go autobuilder attempts to automatically detect code written in Go in a repository,
and only runs build scripts in an attempt to fetch dependencies. To force
CodeQL to limit extraction to the files compiled by your build script, set the environment variable
`CODEQL_EXTRACTOR_GO_BUILD_TRACING=on` or use the ``--command`` option to specify a
build command.
Specifying build commands
~~~~~~~~~~~~~~~~~~~~~~~~~
The following examples are designed to give you an idea of some of the build
commands that you can specify for compiled languages.
.. pull-quote:: Important
The ``--command`` option accepts a single argument---if you need to
use more than one command, specify ``--command`` multiple times.
If you need to pass subcommands and options, the whole argument needs to be
quoted to be interpreted correctly.
- C/C++ project built using ``make``::
codeql database create cpp-database --language=cpp --command=make
- C# project built using ``dotnet build``:
It is a good idea to add `/t:rebuild` to ensure that all code will be built, or do a
prior `dotnet clean` (code that is not built will not be included in the CodeQL database)::
codeql database create csharp-database --language=csharp --command='dotnet build /t:rebuild'
- Go project built using the ``CODEQL_EXTRACTOR_GO_BUILD_TRACING=on`` environment variable::
CODEQL_EXTRACTOR_GO_BUILD_TRACING=on codeql database create go-database --language=go
- Go project built using a custom build script::
codeql database create go-database --language=go --command='./scripts/build.sh'
- Java project built using Gradle::
# Use `--no-daemon` because a build delegated to an existing daemon cannot be detected by CodeQL:
codeql database create java-database --language=java --command='gradle --no-daemon clean test'
- Java project built using Maven::
codeql database create java-database --language=java --command='mvn clean install'
- Java project built using Ant::
codeql database create java-database --language=java --command='ant -f build.xml'
- Project built using Bazel::
# Navigate to the Bazel workspace.
# Before building, remove cached objects
# and stop all running Bazel server processes.
bazel clean --expunge
# Build using the following Bazel flags, to help CodeQL detect the build:
# `--spawn_strategy=local`: build locally, instead of using a distributed build
# `--nouse_action_cache`: turn off build caching, which might prevent recompilation of source code
# `--noremote_accept_cached`, `--noremote_upload_local_results`: avoid using a remote cache
codeql database create new-database --language=<language> \
--command='bazel build --spawn_strategy=local --nouse_action_cache --noremote_accept_cached --noremote_upload_local_results //path/to/package:target'
# After building, stop all running Bazel server processes.
# This ensures future build commands start in a clean Bazel server process
# without CodeQL attached.
bazel shutdown
- Project built using a custom build script::
codeql database create new-database --language=<language> --command='./scripts/build.sh'
This command runs a custom script that contains all of the commands required
to build the project.
Using indirect build tracing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with ``codeql database trace-command``, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
To create a CodeQL database with indirect build tracing, run the following command from the checkout root of your project:
::
codeql database init ... --begin-tracing <database>
You must specify:
- ``<database>``: a path to the new database to be created. This directory will
be created when you execute the command---you cannot specify an existing
directory.
- ``--begin-tracing``: creates scripts that can be used to set up an environment in which build commands will be traced.
You may specify other options for the ``codeql database init`` command as normal.
.. pull-quote:: Note
If the build runs on Windows, you must set either ``--trace-process-level <number>`` or ``--trace-process-name <parent process name>`` so that the option points to a parent CI process that will observe all build steps for the code being analyzed.
The ``codeql database init`` command will output a message::
Created skeleton <database>. This in-progress database is ready to be populated by an extractor.
In order to initialise tracing, some environment variables need to be set in the shell your build will run in.
A number of scripts to do this have been created in <database>/temp/tracingEnvironment.
Please run one of these scripts before invoking your build command.
Based on your operating system, we recommend you run: ...
The ``codeql database init`` command creates ``<database>/temp/tracingEnvironment`` with files that contain environment variables and values that will enable CodeQL to trace a sequence of build steps. These files are named ``start-tracing.{json,sh,bat,ps1}``. Use one of these files with your CI system's mechanism for setting environment variables for future steps. You can:
* Read the JSON file, process it, and print out environment variables in the format expected by your CI system. For example, Azure DevOps expects ``echo "##vso[task.setvariable variable=NAME]VALUE"``.
* Or, if your CI system persists the environment, source the appropriate ``start-tracing`` script to set the CodeQL variables in the shell environment of the CI system.
Build your code; optionally, unset the environment variables using an ``end-tracing.{json,sh,bat,ps1}`` script from the directory where the ``start-tracing`` scripts are stored; and then run the command ``codeql database finalize <database>``.
Once you have created a CodeQL database using indirect build tracing, you can work with it like any other CodeQL database. For example, analyze the database, and upload the results to GitHub if you use code scanning.
Example of creating a CodeQL database using indirect build tracing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following example shows how you could use indirect build tracing in an Azure DevOps pipeline to create a CodeQL database::
steps:
# Download the CodeQL CLI and query packs...
# Check out the repository ...
# Run any pre-build tasks, for example, restore NuGet dependencies...
# Initialize the CodeQL database.
# In this example, the CodeQL CLI has been downloaded and placed on the PATH.
- task: CmdLine@1
displayName: Initialize CodeQL database
inputs:
# Assumes the source code is checked out to the current working directory.
# Creates a database at `<current working directory>/db`.
# Running on Windows, so specifies a trace process level.
script: "codeql database init --language csharp --trace-process-name Agent.Worker.exe --source-root . --begin-tracing db"
# Read the generated environment variables and values,
# and set them so they are available for subsequent commands
# in the build pipeline. This is done in PowerShell in this example.
- task: PowerShell@1
displayName: Set CodeQL environment variables
inputs:
targetType: inline
script: >
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/start-tracing.json | ConvertFrom-Json
$json.PSObject.Properties | ForEach-Object {
$template = "##vso[task.setvariable variable="
$template += $_.Name
$template += "]"
$template += $_.Value
echo "$template"
}
# Execute the pre-defined build step. Note the `msbuildArgs` variable.
- task: VSBuild@1
inputs:
solution: '**/*.sln'
msbuildArgs: /p:OutDir=$(Build.ArtifactStagingDirectory)
platform: Any CPU
configuration: Release
# Execute a clean build, in order to remove any existing build artifacts prior to the build.
clean: True
displayName: Visual Studio Build
# Read and set the generated environment variables to end build tracing. This is done in PowerShell in this example.
- task: PowerShell@1
displayName: Clear CodeQL environment variables
inputs:
targetType: inline
script: >
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/end-tracing.json | ConvertFrom-Json
$json.PSObject.Properties | ForEach-Object {
$template = "##vso[task.setvariable variable="
$template += $_.Name
$template += "]"
$template += $_.Value
echo "$template"
}
- task: CmdLine@2
displayName: Finalize CodeQL database
inputs:
script: 'codeql database finalize db'
# Other tasks go here, for example:
# `codeql database analyze`
# then `codeql github upload-results` ...
.. _downloading-codeql-databases-from-github-com:
Downloading databases from GitHub.com
-------------------------------------
.. include:: ../reusables/download-github-database.rst
Before running an analysis with the CodeQL CLI, you must unzip the databases.
Further reading
---------------
- ":ref:`Analyzing your projects in CodeQL for VS Code <analyzing-your-projects>`"
.. pull-quote::
This article was moved to "`Creating CodeQL databases <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-databases>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

364
docs/codeql/codeql-cli/creating-codeql-query-suites.rst
View File

@@ -3,363 +3,7 @@
Creating CodeQL query suites
============================
CodeQL query suites provide a way of selecting queries, based on their
filename, location on disk or in a CodeQL pack, or metadata properties.
Create query suites for the queries that you want to frequently use in
your CodeQL analyses.
Query suites allow you to pass multiple queries to
CodeQL without having to specify the path to each query file individually.
Query suite definitions are stored in YAML files with the extension ``.qls``. A
suite definition is a sequence of instructions, where each instruction is a YAML
mapping with (usually) a single key. The instructions are executed in the order
they appear in the query suite definition. After all the instructions in the
suite definition have been executed, the result is a set of selected queries.
.. pull-quote:: Note
Any custom queries that you want to add to a query suite must be in a :doc:`CodeQL
pack <about-codeql-packs>` and contain the correct query metadata.
For more information, see
":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
Locating queries to add to a query suite
----------------------------------------
When creating a query suite, you first need to specify the locations of the
queries that you want to select. You can define the location of one or more
queries using:
- A ``query`` instruction---tells CodeQL to look for one or more specified ``.ql``
files::
- query: <path-to-query>
The argument must be one or more file paths, relative to the CodeQL pack containing
the suite definition.
- A ``queries`` instruction---tells CodeQL to recursively scan a directory
for ``.ql`` files::
- queries: <path-to-subdirectory>
The path of the directory must be relative to the root of the CodeQL pack that
contains the suite definition file. To find the queries relative to a
different CodeQL pack, add a ``from`` field::
- queries: <path-to-subdirectory>
from: <ql-pack-name>
version: ^x.y.z
The ``version`` field is optional and specifies a range of compatible versions of this CodeQL pack.
If you don't specify a version, then the most recent version of the pack is used.
- A ``qlpack`` instruction---tells CodeQL to resolve queries in the default suite of the
named CodeQL pack::
- qlpack: <qlpack-name>
version: ^x.y.z
The default suite of a query pack includes a recommended set of queries
inside of that query pack. Not all query packs have a default suite. If the given query pack does not
define a default suite, the `qlpack` instruction will resolve to all of the queries within the pack.
The ``version`` field is optional and specifies a range of compatible versions of this CodeQL pack.
If you don't specify a version, then the most recent version of the pack is used.
.. pull-quote:: Note
When pathnames appear in query suite definitions, they must always
be given with a forward slash, ``/``, as a directory separator.
This ensures that query suite definitions work on all operating systems.
You must add at least one ``query``, ``queries``, or ``qlpack`` instruction to
your suite definition, otherwise no queries will be selected. If the suite
contains no further instructions, all the queries found from the list of files,
in the given directory, or in the named CodeQL pack are selected. If there are further
filtering instructions, only queries that match the constraints imposed by those
instructions will be selected.
Filtering the queries in a query suite
----------------------------------------
After you have defined the initial set of queries to add to your suite by
specifying ``query``, ``queries``, or ``qlpack`` instructions, you can add
``include`` and ``exclude`` instructions. These instructions define selection
criteria based on specific properties:
- When you execute an ``include`` instruction on a set of queries, any
queries that match your conditions are retained in the selection, and queries
that don't match are removed.
- When you execute an ``exclude`` instructions on a set of queries,
any queries that match your conditions are removed from the selection, and queries
that don't match are retained.
The order of your filter instructions is important. The first filter instruction
that appears after the locating instructions determines whether the queries are
included or excluded by default. If the first filter is an ``include``, the
initially located queries will only be part of the suite if they match an
explicit ``include`` filter. If the first filter is an ``exclude``, the initially
located queries are part of the suite unless they are explicitly excluded.
Subsequent instructions are executed in order and the instructions that appear
later in the file take precedence over the earlier instructions. So, ``include``
instructions can be overridden by a later ``exclude`` instructions that match
the same query. Similarly, ``exclude``\ s can be overridden by a later
``include``.
For both instructions, the argument is a constraint block---that is, a YAML map
representing the constraints. Each constraint is a map entry, where the key is
typically a query metadata property. The value can be:
- A single string.
- A ``/``\ -enclosed `regular expression <https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html>`__.
- A list containing strings, regular expressions, or both.
To match a constraint, a metadata value must match one of the strings or
regular expressions. When there is more than one metadata key, each key must be matched.
The standard metadata keys available to match on are: ``description``, ``id``, ``kind``,
``name``, ``tags``, ``precision``, and ``problem.severity``.
For more information about query metadata properties, see
":ref:`Metadata for CodeQL queries <metadata-for-codeql-queries>`."
In addition to metadata tags, the keys in the constraint block can also be:
- ``query filename``---matches on the last path component of the query file name.
- ``query path``---matches on the path to the query file relative to its
enclosing CodeQL pack.
- ``tags contain``---one of the given match strings must match
one of the space-separated components of the value of the ``@tags`` metadata property.
- ``tags contain all``---each of the given match strings must match one of the
components of the ``@tags`` metadata property.
Examples of filtering which queries are run
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A common use case is to create a query suite that runs all queries in a CodeQL pack,
except for a few specific queries that the user does not want to run. In general, we
recommend filtering on the query ``id``, which is a unique and stable identifier for
each query. The following three query suite definitions are semantically identical and
filter by the query ``id``:
This filter matches all the queries in the default suite of ``codeql/cpp-queries``, except for the two queries with the excluded identifiers::
- qlpack: codeql/cpp-queries
- exclude:
id:
- cpp/cleartext-transmission
- cpp/cleartext-storage-file
In this example, a separate ``exclude`` instruction is used for each query::
- qlpack: codeql/cpp-queries
- exclude:
id: cpp/cleartext-transmission
- exclude:
id: cpp/cleartext-storage-file
In this example, a regular expression excludes the same two queries. It would also exclude any future queries added to the suite with identifiers that begin: ``cpp/cleartext-``::
- qlpack: codeql/cpp-queries
- exclude:
id:
- /^cpp\/cleartext-.*/
To define a suite that selects all queries in the default suite of the
``codeql/cpp-queries`` CodeQL pack, and then refines them to only include
security queries, use::
- qlpack: codeql/cpp-queries
- include:
tags contain: security
To define a suite that selects all queries with ``@kind problem``
and ``@precision high`` from the ``my-custom-queries`` directory, use::
- queries: my-custom-queries
- include:
kind: problem
precision: very-high
Note that the following query suite definition behaves differently from the definition above. This definition selects queries that are ``@kind problem`` *or*
are ``@precision very-high``::
- queries: my-custom-queries
- include:
kind: problem
- include:
precision: very-high
To create a suite that selects all queries with ``@kind problem`` from the
``my-custom-queries`` directory except those with ``@problem.severity
recommendation``, use::
- queries: my-custom-queries
- include:
kind: problem
- exclude:
problem.severity: recommendation
To create a suite that selects all queries with ``@tag security`` and
``@problem.severity high`` or ``very-high`` from the ``codeql/cpp-queries`` CodeQL pack,
use::
- queries: .
from: codeql/cpp-queries
- include:
tags contain: security
problem.severity:
- high
- very-high
.. pull-quote::
Tip
You can use the ``codeql resolve queries /path/to/suite.qls`` command to see
which queries are selected by a query suite definition. For more information,
see the `resolve queries <../../codeql-cli/manual/resolve-queries>`__
reference documentation.
Reusing existing query suite definitions
-----------------------------------------
Existing query suite definitions can be reused by specifying:
- An ``import`` instruction---adds the queries selected by a
previously defined ``.qls`` file to the current suite::
- import: <path-to-query-suite>
The path to the imported suite must be relative to the CodeQL pack containing the
current suite definition. If the imported query suite is in a different QL
pack you can use::
- import: <path-to-query-suite>
from: <ql-pack>
version: ^x.y.z
The ``version`` field is optional and specifies a range of compatible versions of this CodeQL pack.
If you don't specify a version, then the most recent version of the pack is used.
Queries added using an ``import`` instruction can be filtered using subsequent
``exclude`` instructions.
- An ``apply`` instruction---adds all of the instructions from a
previously defined ``.qls`` file to the current suite. The instructions in the
applied ``.qls`` file are executed as if they appear in place of ``apply``.
Any ``include`` and ``exclude`` instructions from the applied suite also act on
queries added by any earlier instructions::
- apply: <path-to-query-suite>
The ``apply`` instruction can also be used to apply a set of reusable
conditions, saved in a ``.yml`` file, to multiple query definitions. For more
information, see the `example <#example>`__ below.
Reusability Examples
~~~~~~~~~~~~~~~~~~~~
To use the same conditions in multiple query suite definitions, create a
separate ``.yml`` file containing your instructions. For example, save the
following in a file called ``reusable-instructions.yml``::
- include:
kind:
- problem
- path-problem
tags contain: security
precision:
- high
- very-high
Add ``reusable-instructions.yml`` to the same CodeQL pack as your current query
suite. Then, in one or more query suites, use the ``apply`` instruction to apply
the reusable instructions to the current suite. For example::
- queries: queries/cpp/custom
- apply: reusable-instructions.yml
This will filter the queries in ``queries/cpp/custom`` to only include those that match the reusable conditions.
You can also create a suite definition using ``reusable-instructions.yml`` on
queries in a different CodeQL pack. If the ``.qls`` file is in the same CodeQL pack as
the queries, you can add a ``from`` field immediately after the ``apply``
instruction::
# load queries from the default suite of my-org/my-other-custom-queries
- qlpack: my-org/my-other-custom-queries
# apply the reusable instructions from the my-org/my-custom-instructions CodeQL pack
- apply: reusable-instructions.yml
from: my-org/my-custom-instructions
version: ^1.2.3 # optional
A common use case for an ``import`` instruction is to apply a further filter to queries from another
query suite. For example, this suite will further filter the ``cpp-security-and-quality`` suite
and exclude ``low`` and ``medium`` precision queries::
- import: codeql-suites/cpp-security-and-quality.qls
from: codeql/cpp-queries
- exclude:
precision:
- low
- medium
If you want to ``include`` queries imported from another suite, the syntax is a little different::
- import: codeql-suites/cpp-security-and-quality.qls
from: codeql/cpp-queries
- exclude: {}
- include:
precision:
- very-high
- high
Notice the empty ``exclude`` instruction. This is required to ensure that the subsequent ``include``
instruction is able to filter queries from the imported suite.
Naming a query suite
--------------------
You can provide a name for your query suite by specifying a ``description``
instruction::
- description: <name-of-query-suite>
This value is displayed when you run `codeql resolve queries
<../manual/resolve-queries>`__, if the suite is added to a "well-known"
directory. For more information, see "`Specifying well-known query suites
<#specifying-well-known-query-suites>`__."
Saving a query suite
--------------------
Save your query suite in a file with a ``.qls`` extension and add it to a CodeQL
pack. For more information, see ":ref:`About CodeQL packs <custom-codeql-packs>`."
Specifying well-known query suites
----------------------------------
You can use CodeQL packs to declare directories that contain "well-known" query
suites. You can use "well-known" query suites on the command line by referring
to their file name,
without providing their full path. This gives you a simple way of specifying a
set of queries, without needing to search inside CodeQL packs and distributions.
To declare a directory that contains "well-known" query suites, add the directory
to the ``suites`` property in the ``qlpack.yml`` file at the root of your CodeQL pack.
For more information, see ":ref:`About CodeQL packs <codeqlpack-yml-properties>`."
Using query suites with CodeQL
------------------------------
You can specify query suites on the command line for any command that accepts
``.qls`` files. For example, you can compile the queries selected by a suite
definition using ``query compile``, or use the queries in an analysis using
``database analyze``. For more information about analyzing CodeQL databases, see
":doc:`Analyzing databases with the CodeQL CLI <analyzing-databases-with-the-codeql-cli>`."
Further reading
---------------
- ":ref:`CodeQL queries <codeql-queries>`"
.. pull-quote::
This article was moved to "`Creating CodeQL query suites <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-query-suites>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

73
docs/codeql/codeql-cli/exit-codes.rst
View File

@@ -3,72 +3,7 @@
Exit codes
==========
The CodeQL CLI reports the status of each command it runs as an exit code.
This exit code provides information for subsequent commands or for other tools that rely on the CodeQL CLI.
0
---
Success, normal termination.
1
---
The command successfully determined that the answer to your question is "no".
This exit code is only used by a few commands, such as `codeql test run <../manual/test-run>`__, `codeql database check <../manual/dataset-check>`__, `codeql query format <../manual/query-format>`__,and `codeql resolve extractor <../manual/resolve-extractor>`__.
For more details, see the documentation for those commands.
2
---
Something went wrong.
The CLI writes a human-readable error message to stderr.
This includes cases where an extractor fails with an internal error, because the ``codeql`` driver can't distinguish between internal and user-facing errors in extractor behavior.
3
---
The launcher was unable to find the CodeQL installation directory.
In this case, the launcher can't start the Java code for the CodeQL CLI at all. This should only happen when something is severely wrong with the CodeQL installation.
32
---
The extractor didn't find any code to analyze when running `codeql database create <../manual/database-create>`__ or `codeql database finalize <../manual/database-finalize>`__.
33
---
One or more query evaluations timed out.
It's possible that some queries that were evaluated in parallel didn't time out. The results for those queries are produced as usual.
98
---
Evaluation was explicitly canceled.
99
---
The CodeQL CLI ran out of memory.
This doesn't necessarily mean that all the machine's physical RAM has been used.
If you don't use the ``--ram`` option to set a limit explicitly, the JVM decides on a default limit at startup.
100
---
A fatal internal error occurred.
This should be considered a bug. The CLI usually writes an abbreviated error description to stderr.
If you can reproduce the bug, it's helpful to use ``--logdir`` and send the log files to GitHub in a bug report.
Other
-----
In the case of really severe problems within the JVM that runs ``codeql``, it might return a nonzero exit code of its own choosing.
This should only happen if something is severely wrong with the CodeQL installation, or if there is a memory issue with the host system running the CodeQL process. For example, Unix systems may return `Exit Code 137` to indicate that the kernel has killed a process that CodeQL has started. One way to troubleshoot this is to modify your `--ram=` flag for the `codeql database analyze` step and re-run your workflow.
.. pull-quote::
This article was moved to "`Exit codes <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/exit-codes>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

147
docs/codeql/codeql-cli/extractor-options.rst
View File

@@ -3,146 +3,7 @@
Extractor options
=================
The CodeQL CLI uses special programs, called extractors, to extract information from the source code of a
software system into a database that can be queried. You can customize the behavior of extractors by
setting extractor configuration options through the CodeQL CLI.
About extractor options
-----------------------
Each extractor defines its own set of configuration options. To find out which options are available for a particular extractor, you can run ``codeql resolve languages`` or ``codeql resolve extractor`` with the ``--format=betterjson`` option. The ``betterjson`` output format provides the root paths of extractors and additional information. The output of ``codeql resolve extractor --format=betterjson`` will often be formatted like the following example:
.. code-block:: json
{
"extractor_root" : "/home/user/codeql/java",
"extractor_options" : {
"option1" : {
"title" : "Java extractor option 1",
"description" : "An example string option for the Java extractor.",
"type" : "string",
"pattern" : "[a-z]+"
},
"group1" : {
"title" : "Java extractor group 1",
"description" : "An example option group for the Java extractor.",
"type" : "object",
"properties" : {
"option2" : {
"title" : "Java extractor option 2",
"description" : "An example array option for the Java extractor",
"type" : "array",
"pattern" : "[1-9][0-9]*"
}
}
}
}
}
The extractor option names and descriptions are listed under ``extractor_options``. Each option may contain the following fields:
* ``title`` (required): The title of the option
* ``description`` (required): The description of the option
* ``type`` (required): The type of the option, which can be
* ``string``: indicating that the option can have a single string value
* ``array``: indicating that the option can have a sequence of string values
* ``object``: indicating that it is not an option itself, but a grouping that may contain other options and option groups
* ``pattern`` (optional): The regular expression patterns that all values of the option should match. Note that the extractor may impose additional constraints on option values that are not or cannot be expressed in this regular expression pattern. Such constraints, if they exist, would be explained under the description field.
* ``properties`` (optional): A map from extractor option names in the option group to the corresponding extractor option descriptions. This field can only be present for option groups. For example, options of ``object`` type.
In the example above, the extractor declares two options:
* ``option1`` is a ``string`` option with value matching ``[a-z]+``
* ``group1.option2`` is an ``array`` option with values matching ``[1-9][0-9]*``
Setting extractor options with the CodeQL CLI
---------------------------------------------
The CodeQL CLI supports setting extractor options in subcommands that directly or indirectly invoke extractors. These commands are:
* ``codeql database create``
* ``codeql database start-tracing``
* ``codeql database trace-command``
* ``codeql database index-files``
When running these subcommands, you can set extractor options with the ``--extractor-option`` CLI option. For example:
* ``codeql database create --extractor-option java.option1=abc ...``
* ``codeql database start-tracing --extractor-option java.group1.option2=102 ...``
``--extractor-option`` requires exactly one argument of the form ``extractor_option_name=extractor_option_value``. ``extractor_option_name`` is the name of the extractor (in this example, ``java``) followed by a period and then the name of the extractor option (in this example, either ``option1`` or ``group1.option2``). ``extractor_option_value`` is the value being assigned to the extractor option. The value must match the regular expression pattern of the extractor option (if it exists), and it must not contain newline characters.
Using ``--extractor-option`` to assign an extractor option that does not exist is an error.
The CodeQL CLI accepts multiple ``--extractor-option`` options in the same invocation. If you set a ``string`` extractor option multiple times, the last option value overwrites all previous ones. If you set an `array` extractor option multiple times, all option values are concatenated in order.
You can also specify extractor option names without the extractor name. For example:
* ``codeql database create --extractor-option option1=abc ...``
* ``codeql database start-tracing --extractor-option group1.option2=102 ...``
If you do not specify an extractor name, the extractor option settings will apply to all extractors that declare an option with the given name. In the above example, the first command would set the extractor option ``option1`` to ``abc`` for the ``java`` extractor and every extractor that has an option of ``option1``, for example the ``cpp`` extractor, if the ``option1`` extractor option exists for that extractor.
Setting extractor options from files
------------------------------------
You can also set extractor options through a file. The CodeQL CLI subcommands that accept ``--extractor-option`` also accept ``--extractor-options-file``, which has a required argument of the path to a YAML file (with extension ``.yaml`` or ``.yml``) or a JSON file (with extension ``.json``). For example:
* ``codeql database create --extractor-options-file options.yml ...``
* ``codeql database start-tracing --extractor-options-file options.json ...``
Each option file contains a tree structure of nested maps. At the root is an extractor map key, and beneath it are map keys that correspond to extractor names. Starting at the third level, there are extractor options and option groups.
In JSON:
.. code-block:: json
{
"extractor" : {
"java": {
"option1" : "abc",
"group1" : {
"option2" : [ 102 ]
}
}
}
}
In YAML:
.. code-block:: yaml
extractor:
java:
option1: "abc"
group1:
option2: [ 102 ]
The value for a ``string`` extractor option must be a string or a number (which will be converted to a string before further processing).
The value for an ``array`` extractor option must be an array of strings or numbers.
The value for an option group (of type ``object``) must be a map, which may contain nested extractor options and option groups.
Each extractor option value must match the regular expression pattern of the extractor option (if it exists), and it must not contain newline characters.
Assigning an extractor option that does not exist is an error. You can make the CodeQL CLI ignore unknown extractor options by using a special ``__allow_unknown_properties`` Boolean field. For example, the following option file asks the CodeQL CLI to ignore all unknown extractor options and option groups under ``group1``:
.. code-block:: yaml
extractor:
java:
option1: "abc"
group1:
__allow_unknown_properties: true
option2: [ 102 ]
You can specify ``--extractor-options-file`` multiple times. The extractor option assignments are processed in the following order:
1. All extractor option files specified by ``--extractor-options-file`` are processed in the order they appear on the command line, then
2. All extractor option assignments specified by ``--extractor-option`` are processed in the order they appear on the command line
The same rules govern what happens when the same extractor option is set multiple times, regardless of whether the assignments are done using ``--extractor-option``, using ``--extractor-options-file``, or some combination of the two. If you set a ``string`` extractor option multiple times, the last option value overwrites all previous values. If you set an ``array`` extractor option multiple times, all option values are concatenated in order.
.. pull-quote::
This article was moved to "`Extractor options <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/extractor-options>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

278
docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst
View File

@@ -3,277 +3,7 @@
Getting started with the CodeQL CLI
===================================
To run CodeQL commands, you need to set up the CLI so that it can access
the tools, queries, and libraries required to create and analyze databases.
.. include:: ../reusables/license-note.rst
.. _setting-up-cli:
Setting up the CodeQL CLI
-------------------------
The CodeQL CLI can be set up to support many different use cases and directory
structures. To get started quickly, we recommend adopting a relatively simple
setup, as outlined in the steps below.
If you use Linux, Windows, or macOS version 10.14 ("Mojave") or earlier, simply
follow the steps below. For macOS version 10.15 ("Catalina") or newer, steps 1
and 4 are slightly different---for further details, see the sections labeled
**Information for macOS "Catalina" (or newer) users**. If you are using macOS
on Apple Silicon (e.g. Apple M1), ensure that the `Xcode command-line developer
tools <https://developer.apple.com/downloads/index.action>`__ and `Rosetta 2
<https://support.apple.com/en-us/HT211861>`__ are installed.
.. pull-quote:: Note
The CodeQL CLI is currently not compatible with non-glibc Linux
distributions such as (muslc-based) Alpine Linux.
For information about installing the CodeQL CLI in a CI system to create results
to display in GitHub as code scanning alerts, see
`Installing CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system>`__
in the GitHub documentation.
.. _download-cli:
1. Download the CodeQL CLI zip package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The CodeQL CLI download package is a zip archive containing tools, scripts, and
various CodeQL-specific files. If you don't have an Enterprise license then, by
downloading this archive, you are agreeing to the `GitHub CodeQL Terms and
Conditions <https://securitylab.github.com/tools/codeql/license>`__.
.. pull-quote:: Important
There are several versions of the CLI available to download, depending
on your use case:
* If you want to use the most up to date CodeQL tools and features, download the version tagged ``latest``.
* If you want to generate code scanning data to upload to GitHub Enterprise server, then download the version that is compatible with the CodeQL CLI used in your CI system. For more information, see "`Installing CodeQL CLI in your CI system <https://docs.github.com/en/enterprise-server/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system#downloading-the-codeql-cli>`__" in the GitHub documentation.
If you use Linux, Windows, or macOS version 10.14 ("Mojave") or earlier, simply
`download the zip archive
<https://github.com/github/codeql-cli-binaries/releases>`__
for the version you require.
If you want the CLI for a specific platform, download the appropriate ``codeql-PLATFORM.zip`` file.
Alternatively, you can download ``codeql.zip``, which contains the CLI for all supported platforms.
.. container:: toggle
.. container:: name
**Information for macOS "Catalina" (or newer) users**
.. pull-quote:: macOS "Catalina" (or newer)
If you use macOS version 10.15 ("Catalina"), version 11 ("Big Sur"), or the upcoming
version 12 ("Monterey"), you need to ensure that your web browser does not automatically
extract zip files. If you use Safari, complete the following steps before downloading
the CodeQL CLI zip archive:
i. Open Safari.
ii. From the Safari menu, select **Preferences...**.
iii. Click the **General** Tab.
iv. Ensure the check-box labeled **Open "safe" files after downloading**.
is unchecked.
2. Extract the zip archive
~~~~~~~~~~~~~~~~~~~~~~~~~~
For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier)
simply extract the zip archive.
.. container:: toggle
.. container:: name
**Information for macOS "Catalina" (or newer) users**
.. pull-quote:: macOS "Catalina"
macOS "Catalina", "Big Sur", or "Monterey" users should run the following
commands in the Terminal, where ``${extraction-root}`` is the path to the
directory where you will extract the CodeQL CLI zip archive:
i. ``mv ~/Downloads/codeql*.zip ${extraction-root}``
ii. ``cd ${extraction-root}``
iii. ``/usr/bin/xattr -c codeql*.zip``
iv. ``unzip codeql*.zip``
.. _launch-codeql-cli:
3. Launch ``codeql``
~~~~~~~~~~~~~~~~~~~~
Once extracted, you can run CodeQL processes by running the ``codeql``
executable in a couple of ways:
- By executing ``<extraction-root>/codeql/codeql``, where
``<extraction-root>`` is the folder where you extracted the CodeQL CLI
package.
- By adding ``<extraction-root>/codeql`` to your ``PATH``, so that you
can run the executable as just ``codeql``.
At this point, you can execute CodeQL commands. For a full list of the CodeQL
CLI commands, see the "`CodeQL CLI manual <../manual>`__."
.. pull-quote:: Note
If you add ``codeql`` to your ``PATH``, it can be accessed by CodeQL
for Visual Studio Code to compile and run queries.
For more information about configuring VS Code to access the CodeQL CLI, see
":ref:`Setting up CodeQL in Visual Studio Code <setting-up-codeql-in-visual-studio-code>`."
4. Verify your CodeQL CLI setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CodeQL CLI has subcommands you can execute to verify that you are correctly set
up to create and analyze databases:
- Run ``codeql resolve languages`` to show which languages are
available for database creation. This will list the languages supported by
default in your CodeQL CLI package.
- (Optional) You can download some ":ref:`CodeQL packs <about-codeql-packs>`" containing pre-compiled queries you would like to run.
To do this, run ``codeql pack download <pack-name> [...pack-name]``, where ``pack-name`` is the name of
the pack you want to download. The core query packs are a good place to start. They are:
- ``codeql/cpp-queries``
- ``codeql/csharp-queries``
- ``codeql/go-queries``
- ``codeql/java-queries``
- ``codeql/javascript-queries``
- ``codeql/python-queries``
- ``codeql/ruby-queries``
Alternatively, you can download query packs during the analysis by using the ``--download`` flag of the ``codeql database analyze``
command.
Checking out the CodeQL source code directly
--------------------------------------------
Some users prefer working with CodeQL query sources directly in order to work on or contribute to the Open Source shared queries. In
order to do this, the following steps are recommended. Note that the following instructions are a slightly more complicated alternative
to working with CodeQL packages as explained above.
1. Download the CodeQL CLI zip
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Follow :ref:`step 1 from the previous section<download-cli>`.
2. Create a new CodeQL directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create a new directory where you can place the CLI and any queries and libraries
you want to use. For example, ``$HOME/codeql-home``.
The CLI's built-in search operations automatically look in all of its sibling
directories for the files used in database creation and analysis. Keeping these
components in their own directory prevents the CLI searching unrelated sibling
directories while ensuring all files are available without specifying any
further options on the command line.
.. _local-copy-codeql-queries:
3. Obtain a local copy of the CodeQL queries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `CodeQL repository <https://github.com/github/codeql>`__ contains
the queries and libraries required for CodeQL analysis of all supported languages.
Clone a copy of this repository into ``codeql-home``.
By default, the root of the cloned repository will be called ``codeql``.
Rename this folder ``codeql-repo`` to avoid conflicting with the CodeQL
CLI that you will extract in step 4. If you use git on the command line, you can
clone and rename the repository in a single step by running
``git clone git@github.com:github/codeql.git codeql-repo`` in the ``codeql-home`` folder.
.. pull-quote:: Note
The CodeQL libraries and queries for Go analysis used to live in a
separate `CodeQL for Go repository <https://github.com/github/codeql-go/>`__.
These have been moved to the ``github/codeql`` repository.
It is no longer necessary to clone the ``github/codeql-go`` into a separate ``codeql-home/codeql-go`` folder.
For more information, see the `Relocation announcement <https://github.com/github/codeql-go/issues/741>`__.
Within this repository, the queries and libraries are organized into CodeQL
packs. Along with the queries themselves, CodeQL packs contain important metadata
that tells the CodeQL CLI how to process the query files. For more information,
see ":doc:`About CodeQL packs <about-codeql-packs>`."
.. pull-quote:: Important
There are different versions of the CodeQL queries available for different
users. Check out the correct version for your use case:
- For the queries that are intended to be used with the latest CodeQL CLI release, check out the
branch tagged ``codeql-cli/latest``. You should use this branch for databases you've built
using the CodeQL CLI, fetched from code scanning on GitHub, or recently downloaded from GitHub.com.
- For the most up to date CodeQL queries, check out the ``main`` branch.
This branch represents the very latest version of CodeQL's analysis.
4. Extract the zip archive
~~~~~~~~~~~~~~~~~~~~~~~~~~
For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier)
simply
extract the zip archive into the directory you created in step 2.
For example, if the path to your copy of the CodeQL repository is
``$HOME/codeql-home/codeql-repo``, then extract the CLI into
``$HOME/codeql-home/``.
5. Launch ``codeql``
~~~~~~~~~~~~~~~~~~~~
See :ref:`step 3 from the previous section<launch-codeql-cli>`.
6. Verify your CodeQL CLI setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CodeQL CLI has subcommands you can execute to verify that you are correctly set
up to create and analyze databases:
- Run ``codeql resolve languages`` to show which languages are
available for database creation. This will list the languages supported by
default in your CodeQL CLI package.
- Run ``codeql resolve qlpacks`` to show which CodeQL packs the CLI can find. This
will display the names of all the CodeQL packs directly available to the CodeQL CLI.
This should include:
- Query packs for each supported language, for example, ``codeql/{language}-queries``.
These packs contain the standard queries that will be run for each analysis.
- Library packs for each supported language, for example, ``codeql/{language}-all``. These
packs contain query libraries, such as control flow and data flow libraries, that
may be useful to query writers.
- Example packs for each supported language, for example, ``codeql/{language}-examples``.
These packs contain useful snippets of CodeQL that query writers may find useful.
- Legacy packs that ensure custom queries and libraries created using older products are
compatible with your version of CodeQL.
.. _using-two-versions-of-the-codeql-cli:
Using two versions of the CodeQL CLI
------------------------------------
If you want to use the latest CodeQL features to execute queries or CodeQL tests,
but also want to prepare databases that are compatible with a specific version of
CodeQL code scanning on GitHub Enterprise Server, you may need to install two versions of the CLI. The
recommended directory setup depends on which versions you want to install:
- If both versions are 2.0.2 (or newer), you can unpack both CLI archives in the
same parent directory.
- If at least one of the versions is 2.0.1 (or older), the unpacked CLI archives cannot
be in the same parent directory, but they can share the same grandparent
directory. For example, if you unpack version 2.0.2 into
``$HOME/codeql-home/codeql-cli``, the older version should be
unpacked into ``$HOME/codeql-older-version/old-codeql-cli``. Here, the common
grandparent is the ``$HOME`` directory.
.. pull-quote::
This article was moved to "`Getting started with the CodeQL CLI <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

172
docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst
View File

@@ -3,171 +3,7 @@
Publishing and using CodeQL packs
=================================
You can publish your own CodeQL packs and use packs published by other people.
.. include:: ../reusables/beta-note-package-management.rst
Configuring the ``qlpack.yml`` file before publishing
-----------------------------------------------------
You can check and modify the configuration details of your CodeQL pack prior to publishing. Open the ``qlpack.yml`` file in your preferred text editor.
.. code-block:: none
library: # set to true if the pack is a library. Set to false or omit for a query pack
name: <scope>/<pack>
version: <x.x.x>
description: <Description to publish with the package>
default-suite: # optional, one or more queries in the pack to run by default
- query: <relative-path>/query-file>.ql
default-suite-file: default-queries.qls # optional, a pointer to a query-suite in this pack
license: # optional, the license under which the pack is published
dependencies: # map from CodeQL pack name to version range
- ``name:`` must follow the <scope>/<pack> format, where <scope> is the GitHub organization that you will publish to and <pack> is the name for the pack.
- A maximum of one of ``default-suite`` or ``default-suite-file`` is allowed. These are two different ways to define a default query suite to be run, the first by specifying queries directly in the `qlpack.yml` file and the second by specifying a query suite in the pack.
Running ``codeql pack publish``
-------------------------------
When you are ready to publish a pack to the GitHub Container registry, you can run the following command in the root of the pack directory:
::
codeql pack publish
The published package will be displayed in the packages section of GitHub organization specified by the scope in the ``qlpack.yml`` file.
Running ``codeql pack download <scope>/<pack>``
-----------------------------------------------
To run a pack that someone else has created, you must first download it by running the following command:
::
codeql pack download <scope>/<pack>@x.x.x
- ``<scope>``: the name of the GitHub organization that you will download from.
- ``<pack>``: the name for the pack that you want to download.
- ``@x.x.x``: an optional version number. If omitted, the latest version will be downloaded.
This command accepts arguments for multiple packs.
If you write scripts that specify a particular version number of a query pack to download, keep in mind that when you update your version of CodeQL to a newer one, you may also need to switch to a newer version of the query pack. Newer versions of CodeQL *may* provide
degraded performance when used with query packs that have been pinned to a very old version. For more information, see ":ref:`About CodeQL pack compatibility <about-codeql-pack-compatibility>`."
Using a CodeQL pack to analyze a CodeQL database
------------------------------------------------
To analyze a CodeQL database with a CodeQL pack, run the following command:
::
codeql database analyze <database> <scope>/<pack>@x.x.x:<path>
- ``<database>``: the CodeQL database to be analyzed.
- ``<scope>``: the name of the GitHub organization that the pack is published to.
- ``<pack>``: the name for the pack that you are using.
- ``@x.x.x``: an optional version number. If omitted, the latest version will be used.
- ``:<path>``: an optional path to a query, directory, or query suite. If omitted, the pack's default query suite will be used.
The ``analyze`` command will run the default suite of any specified CodeQL packs. You can specify multiple CodeQL packs to be used for analyzing a CodeQL database. For example:
::
codeql <database> analyze <scope>/<pack> <scope>/<other-pack>
.. pull-quote::
Note
The ``codeql pack download`` command stores the pack it downloads in an internal location that is not intended for local modification. Unexpected (and hard to troubleshoot) behavior may result if the pack is modified after downloading. For more information about customizing packs, see ":ref:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`."
.. _about-codeql-pack-compatibility:
About CodeQL pack compatibility
-------------------------------
When a query pack is published, it includes pre-compiled representations of all the queries in it. These pre-compiled queries are generally much faster to execute than it is to compile the QL source from scratch during the analysis. However, the pre-compiled queries also depend on certain internals of the QL evaluator, so if the version of CodeQL that performs the analysis is too different from the version that ran ``codeql pack publish``, it may be necessary to compile the queries from source instead during analysis. The recompilation happens automatically and will not affect the *results* of the analysis, but it can make the
analysis significantly slower.
It can generally be assumed that if a pack is published with one release of CodeQL, the precompiled queries in it can be used directly by *later* releases of CodeQL, as long as there is no more than 6 months between the release dates. We will make reasonable efforts to keep new releases compatible for longer than that, but make no promises.
It can also be assumed that a pack published by the *latest* public release of CodeQL will be useable by the version of CodeQL that is used by code scanning and GitHub Actions, even though that is often a slightly older release.
As an exception to the above, packs published with versions of CodeQL *earlier than 2.12.0* are not compatible with any earlier or later versions. These old versions did not write pre-compiled queries in a format that supported compatibility between releases. Packs published by these versions can still be *used* by newer versions, but the analysis will be slower because the queries have to be recompiled first.
As a user of a published query pack, you can check that the CodeQL makes use of the precompiled queries in it by inspecting the terminal output from an analysis runs that uses the query pack. If it contains lines looking like the following, then the precompiled queries were used successfully:
::
[42/108] Loaded /long/path/to/query/Filename.qlx.
However, if they instead look like the following, then usage of the precompiled queries failed:
::
Compiling query plan for /long/path/to/query/Filename.ql.
[42/108 comp 25s] Compiled /long/path/to/query/Filename.ql.
The results of the analysis will still be good in this case, but to get optimal performance you may need to upgrade to a newer version of the CodeQL CLI and/or of the query pack.
If you publish query packs on the Container registry on GitHub.com for others to use, we recommend that you use a recent release of CodeQL to run ``codeql pack publish``, and that you publish a fresh version of your pack with an updated CodeQL version before the version you used turns 6 months old. That way you can ensure that users of your pack who keep *their* CodeQL up to date will benefit from the pre-compiled queries in your pack.
If you publish query packs with the intention of using them on a GitHub Enterprise Server installation that uses its bundled CodeQL binaries, use the same CodeQL version to run ``codeql pack publish``. Newer versions might produce pre-compiled queries that the one in GitHub Enterprise Server may not recognize. Your GitHub Enterprise Server administrator may choose to upgrade to a newer version of CodeQL periodically. If so, follow their lead.
.. _working-with-codeql-packs-on-ghes:
Working with CodeQL packs on GitHub Enterprise Server
-----------------------------------------------------
.. pull-quote::
Note
The Container registry for GitHub Enterprise Server supports CodeQL query packs from GitHub Enterprise Server 3.6 onward.
By default, the CodeQL CLI expects to download CodeQL packs from and publish packs to the Container registry on GitHub.com. However, you can also work with CodeQL packs in a Container registry on GitHub Enterprise Server 3.6, and later, by creating a ``qlconfig.yml`` file to tell the CLI which Container registry to use for each pack.
Create a ``~/.codeql/qlconfig.yml`` file using your preferred text editor, and add entries to specify which registry to use for one or more package name patterns.
For example, the following ``qlconfig.yml`` file associates all packs with the Container registry for the GitHub Enterprise Server at ``GHE_HOSTNAME``, except packs matching ``codeql/*``, which are associated with the Container registry on GitHub.com:
.. code-block:: yaml
registries:
- packages:
- 'codeql/*'
- 'other-org/*'
url: https://ghcr.io/v2/
- packages: '*'
url: https://containers.GHE_HOSTNAME/v2/
The CodeQL CLI will determine which registry to use for a given package name by finding the first item in the ``registries`` list with a ``packages`` property that matches that package name.
This means that you'll generally want to define the most specific package name patterns first. The ``packages`` property may be a single package name, a glob pattern, or a YAML list of package names and glob patterns.
The ``registries`` list can also be placed inside of a ``codeql-workspace.yml`` file. Doing so will allow you to define the registries to be used within a specific workspace, so that it can be shared amongst other CodeQL users of the workspace. The ``registries`` list in the ``codeql-workspace.yml`` will be merged with and take precedence over the list in the global ``qlconfig.yml``. For more information about ``codeql-workspace.yml``, see :ref:`About CodeQL workspaces <about-codeql-workspaces>`.
You can now use ``codeql pack publish``, ``codeql pack download``, and ``codeql database analyze`` to manage packs on GitHub Enterprise Server.
Authenticating to GitHub Container registries
---------------------------------------------
You can publish packs and download private packs by authenticating to the appropriate GitHub Container registry.
You can authenticate to the Container registry on GitHub.com in two ways:
1. Pass the ``--github-auth-stdin`` option to the CodeQL CLI, then supply a GitHub Apps token or personal access token via standard input.
2. Set the ``GITHUB_TOKEN`` environment variable to a GitHub Apps token or personal access token.
Similarly, you can authenticate to a GHES Container registry, or authenticate to multiple registries simultaneously (for example, to download or run private packs from multiple registries) in two ways:
1. Pass the ``--registries-auth-stdin`` option to the CodeQL CLI, then supply a registry authentication string via standard input.
2. Set the ``CODEQL_REGISTRIES_AUTH`` environment variable to a registry authentication string.
A registry authentication string is a comma-separated list of ``<registry-url>=<token>`` pairs, where ``registry-url`` is a GitHub Container registry URL, such as ``https://containers.GHE_HOSTNAME/v2/``, and ``token`` is a GitHub Apps token or personal access token for that GitHub Container registry.
This ensures that each token is only passed to the Container registry you specify.
For instance, the following registry authentication string specifies that the CodeQL CLI should authenticate to the Container registry on GitHub.com using the token ``<token1>`` and to the Container registry for the GHES instance at ``GHE_HOSTNAME`` using the token ``<token2>``:
.. code-block:: none
https://ghcr.io/v2/=<token1>,https://containers.GHE_HOSTNAME/v2/=<token2>
.. pull-quote::
This article was moved to "`Publishing and using CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/publishing-and-using-codeql-packs>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

48
docs/codeql/codeql-cli/query-reference-files.rst
View File

@@ -3,47 +3,7 @@
Query reference files
=====================
A query reference file is text file that defines the location of one query to test.
You use a query reference file when you want to tell the ``test run`` subcommand
to run a query that's not part of a test directory.
There are two ways to specify queries that you want to run as tests:
#. Use a query reference file to specify the location of a query to test.
This is useful when you create tests for alert and path queries that
are intended to identify problems in real codebases. You might create
several directories of test code, each focusing on different
aspects of the query. Then you would add a query reference file to
each directory of test code, to specify the query to test.
#. Add the query directly to a directory of tests.
These is typically useful when you're writing queries explicitly to test the behavior
of QL libraries. Often these queries contain just a few calls to library predicates,
wrapping them in a ``select`` statement so their output can be tested.
Defining a query reference file
-------------------------------
Each query reference file, ``.qlref``, contains a single line that defines
where to find one query. The location must be defined relative
to the root of the CodeQL pack that contains the query.
Usually, this is either the CodeQL pack that contains the ``.qlref``, a CodeQL pack
specified in the ``dependencies`` block for the test pack, or a transitive dependency of
the CodeQL pack.
You should use forward slashes in the path on all operating
systems to ensure compatibility between systems.
Example
^^^^^^^
A query reference file to test a JavaScript alert query:
`DeadAngularJSEventListener.qlref <https://github.com/github/codeql/blob/main/javascript/ql/test/query-tests/AngularJS/DeadAngularJSEventListener/DeadAngularJSEventListener.qlref>`__
The ```qlpack.yml`` file <https://github.com/github/codeql/blob/main/javascript/ql/test/qlpack.yml>`__
for the CodeQL pack at ``javascript/ql/test`` defines ``codeql/javascript-queries`` as
a dependency. So the query reference file defines the location of the query relative
to the ``codeql/javascript-queries`` CodeQL pack::
AngularJS/DeadAngularJSEventListener.ql
For another example, see :doc:`Testing custom queries <testing-custom-queries>`.
.. pull-quote::
This article was moved to "`Query reference files <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/query-reference-files>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

291
docs/codeql/codeql-cli/sarif-output.rst
View File

@@ -3,290 +3,7 @@
SARIF output
============
CodeQL supports SARIF as an output format for sharing static analysis results.
SARIF is designed to represent the output of a broad range of static analysis
tools, and there are many features in the SARIF specification that are
considered "optional". This document details the output produced when using the
format type ``sarifv2.1.0``, which corresponds to the SARIF v2.1.0.csd1
specification.
For more information on selecting a file format for your analysis results, see
the `database analyze reference <../manual/database-analyze>`__.
SARIF specification and schema
------------------------------
This topic is intended to be read alongside the detailed SARIF specification.
For more information on the specification and the SARIF schema, see the `SARIF
specification documentation
<https://github.com/oasis-tcs/sarif-spec/tree/master/Documents/CommitteeSpecificationDrafts/>`__
on GitHub.
Change notes
------------
Changes between versions
~~~~~~~~~~~~~~~~~~~~~~~~
+----------------+-----------------+-----------------------------------------------------------------------------------------------------------------------------+
| CodeQL version | Format type | Changes |
+================+=================+=============================================================================================================================+
| 2.0.0 | ``sarifv2.1.0`` | First version of this format. |
+----------------+-----------------+-----------------------------------------------------------------------------------------------------------------------------+
Future changes to the output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The output produced for a given specific format type (for example
``sarifv2.1.0``) may change in future CodeQL releases. We will endeavor to
maintain backwards compatibility with consumers of the generated SARIF by
ensuring that:
- No field which is marked as “Always” being generated will be removed.
- The circumstances under which “Optional” fields are generated may change.
Consumers of the CodeQL SARIF output should be robust to the presence or absence
of these fields.
New output fields may be added in future releases under the same format
type–these are not considered to break backwards compatibility, and consumers
should be robust to the presence of newly added fields.
New format argument types may be added in future versions of CodeQL---for example,
to support new versions of SARIF. These have no guarantee of backwards
compatibility, unless explicitly documented.
Generated SARIF objects
-----------------------
This details each SARIF component that may be generated, along with any specific
circumstances. We omit any properties that are never generated.
``sarifLog`` object
~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===========================================================================================================================================+
| ``$schema`` | Always | Provides a link to the `SARIF schema <https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json>`__. |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| ``version`` | Always | The version of the SARIF used to generate the output. |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| ``runs`` | Always | An array containing a single run object, for one language. |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
``run`` object
~~~~~~~~~~~~~~
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===========================================================================================================================================================================================================================================================================================+
| ``tool`` | Always | – |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``originalUriBaseIds`` | Always | A dictionary of ``uriBaseIds`` to artifactLocations representing the original locations on the analysis machine. At a minimum, this will contain the ``%SRCROOT%`` ``uriBaseId``, which represents the root location on the analysis machine of the source code for the analyzed project. |
| | | Each ``artifactLocation`` will contain the ``uri`` and ``description`` properties. |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``artifacts`` | Always | An array containing at least one artifact object for every file referenced in a result. |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``results`` | Always | – |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``newLineSequences`` | Always | – |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``columnKind`` | Always | – |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``properties`` | Always | The properties dictionary will contain the ``semmle.formatSpecifier``, which identifies the format specifier passed to the CodeQL CLI. |
+------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
``tool`` object
~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-----------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===========+
| ``driver`` | Always | – |
+------------------------+-----------------------------+-----------+
``toolComponent`` object
~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+==========================================================================================================================================================================================================================================================================================+
| ``name`` | Always | Set to “CodeQL command-line toolchain” for output from the CodeQL CLI tools. Note, if the output was generated using a different tool a different ``name`` is reported, and the format may not be as described here. |
+------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``organization`` | Always | Set to “GitHub”. |
+------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``version`` | Always | Set to the CodeQL release version e.g. “2.0.0”. |
+------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``rules`` | Always | An array of ``reportingDescriptor`` objects that represent rules. This array will contain, at a minimum, all the rules that were run during this analysis, but may contain rules which were available but not run. For more detail about enabling queries, see ``defaultConfiguration``. |
+------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
``reportingDescriptor`` object (for rule)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``reportingDescriptor`` objects may be used in multiple places in the SARIF specification. When a ``reportingDescriptor`` is included in the rules array of a ``toolComponent`` object it has the following properties.
+------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+==================================================================+================================+====================================================================================================================================================================================================================================================================================+
| ``id`` | Always | Will contain the ``@id`` property specified in the query that defines the rule, which is usually of the format ``language/rule-name`` (for example ``cpp/unsafe-format-string``). If your organization defines the ``@opaqueid`` property in the query it will be used instead. |
+------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``name`` | Always | Will contain the ``@id`` property specified in the query. See the ``id`` property above for an example. |
+------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``shortDescription`` | Always | Will contain the ``@name`` property specified in the query that defines the rule. |
+------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``fullDescription`` | Always | Will contain the ``@description`` property specified in the query that defines the rule. |
+------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``defaultConfiguration`` | Always | A ``reportingConfiguration`` object, with the enabled property set to true or false, and a level property set according to the ``@severity`` property specified in the query that defines the rule. Omitted if the ``@severity`` property was not specified. |
+------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
``artifact`` object
~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+=================================================================================================================================================================================================================================================================+
| ``location`` | Always | An ``artifactLocation`` object. |
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``index`` | Always | The index of the ``artifact`` object. |
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``contents`` | Optionally | If results are generated using the ``--sarif-add-file-contents`` flag, and the source code is available at the time the SARIF file is generated, then the ``contents`` property is populated with an ``artifactContent`` object, with the ``text`` property set.|
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
``artifactLocation`` object
~~~~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+======================================================================================================================================+
| ``uri`` | Always | – |
+------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| ``index`` | Always | – |
+------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| ``uriBaseId`` | Optionally | If the file is relative to some known abstract location, such as the root source location on the analysis machine, this will be set. |
+------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
``result`` object
~~~~~~~~~~~~~~~~~
The composition of the results is dependent on the options provided to CodeQL.
By default, the results are grouped by unique message format string and
primary location. Thus, two results that occur at the same location with the
same underlying message, will appear as a single result in the output. This
behavior can be disabled by using the flag ``--ungroup-results``, in which case
no results are grouped.
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+=========================+=============================+=========================================================================================================================================================================================================================================================================================================+
| ``ruleId`` | Always | See the description of the ``id`` property in ``reportingDescriptor`` object (for rule) . |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``ruleIndex`` | Always | – |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``message`` | Always | A message describing the problem(s) occurring at this location. This message may be a SARIF “Message with placeholder”, containing links that refer to locations in the ``relatedLocations`` property. |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``locations`` | Always | An array containing a single ``location`` object. |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``partialFingerprints`` | Always | A dictionary from named fingerprint types to the fingerprint. This will contain, at a minimum, a value for the ``primaryLocationLineHash``, which provides a fingerprint based on the context of the primary location. |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``codeFlows`` | Optionally | This array may be populated with one or more ``codeFlow`` objects if the query that defines the rule for this result is of ``@kind path-problem``. |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``relatedLocations`` | Optionally | This array will be populated if the query that defines the rule for this result has a message with placeholder options. Each unique location is included once. |
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``suppressions`` | Optionally | If the result is suppressed, then this will contain a single ``suppression`` object, with the ``@kind`` property set to ``IN_SOURCE``. If this result is not suppressed, but there is at least one result that has a suppression, then this will be set to an empty array, otherwise it will not be set.|
+-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
``location`` object
~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+============================================================================================================================+
| ``physicalLocation`` | Always | – |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
| ``id`` | Optionally | ``location`` objects that appear in the ``relatedLocations`` array of a ``result`` object may contain the ``id`` property. |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
| ``message`` | Optionally | ``location`` objects may contain the ``message`` property if: |
| | | |
| | | - They appear in the ``relatedLocations`` array of a ``result`` object may contain the ``message`` property. |
| | | - They appear in the ``threadFlowLocation.location`` property. |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
``physicalLocation`` object
~~~~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===================================================================================================================================+
| ``artifactLocation`` | Always | – |
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| ``region`` | Optionally | If the given ``physicalLocation`` exists in a text file, such as a source code file, then the ``region`` property may be present. |
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| ``contextRegion`` | Optionally | May be present if this location has an associated ``snippet``. |
+------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
``region`` object
~~~~~~~~~~~~~~~~~
There are two types of ``region`` object produced by CodeQL:
- Line/column offset regions
- Character offset and length regions
Any region produced by CodeQL may be specified in either format, and consumers
should robustly handle either type.
For line/column offset regions, the following properties will be set:
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+==============================================================================================+
| ``startLine`` | Always | – |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| ``startColumn`` | Optionally | Not included if equal to the default value of 1. |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| ``endLine`` | Optionally | Not included if identical to ``startLine``. |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| ``endColumn`` | Always | – |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| ``snippet`` | Optionally | – |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
For character offset and length regions, the following properties will be set:
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+==============================================================================================+
| ``charOffset`` | Optionally | Provided if ``startLine``, ``startColumn``, ``endLine``, and ``endColumn`` are not populated.|
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| ``charLength`` | Optionally | Provided if ``startLine``, ``startColumn``, ``endLine``, and ``endColumn`` are not populated.|
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
| ``snippet`` | Optionally | – |
+------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
``codeFlow`` object
~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-----------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===========+
| ``threadFlows`` | Always | – |
+------------------------+-----------------------------+-----------+
``threadFlow`` object
~~~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-----------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===========+
| ``locations`` | Always | – |
+------------------------+-----------------------------+-----------+
``threadFlowLocation`` object
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------+-----------------------------+-----------+
| JSON property name | When is this generated? | Notes |
+========================+=============================+===========+
| ``location`` | Always | – |
+------------------------+-----------------------------+-----------+
.. pull-quote::
This article was moved to "`CodeQL CLI SARIF output <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/sarif-output>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

71
docs/codeql/codeql-cli/specifying-command-options-in-a-codeql-configuration-file.rst
View File

@@ -3,70 +3,7 @@
Specifying command options in a CodeQL configuration file
=========================================================
You can save default or frequently used options for your commands in a per-user
configuration file.
You can specify CodeQL CLI command options in two ways:
- Directly in the command line, using the appropriate flag.
- In a configuration (or ``config``) file that CodeQL scans for relevant
options each time a command is executed.
For options that are likely to change each time you execute a command,
specifying the value on the command line is the most convenient way of passing
the information to CodeQL. Saving options in a ``config`` file is a good way to
specify options you use frequently.
It's also a good way to add custom CodeQL packs that you use regularly to your search path.
Using a CodeQL configuration file
---------------------------------
You need to save the ``config`` file under your home (Linux and macOS) or user profile
(Windows) directory in the ``.config/codeql/`` subdirectory.
For example, ``$HOME/.config/codeql/config``.
The syntax for specifying options is as follows::
<command> <subcommand> <option> <value>
To apply the same options to more than one command you can:
- Omit the ``<subcommand>``, which will specify the option for every
``<subcommand>`` to which it's relevant.
- Omit both ``<command>`` and ``<subcommand>``, which will globally specify the
option for every ``<command>`` and ``<subcommand>`` to which it's relevant.
.. pull-quote::
Note
- ``config`` files only accept spaces between between option flags and
values---CodeQL will throw an error if you use ``=`` to specify an option value.
- If you specify an option in the command line, this overrides the ``config``
value defined for that option.
- If you want to specify more than one option for a ``<command>``,
``<subcommand>`` or globally, use one line per option.
Examples
~~~~~~~~
- To output all analysis results generated by ``codeql database analyze`` as
CSV format, you would specify::
database analyze --format csv
Here, you have to specify the command and subcommand to prevent any of the
low-level commands that are executed during ``database analyze`` being passed
the same ``--format`` option.
- To define the RAM (4096 MB) and number of threads (4) to use when running
CodeQL commands, specify the following, on separate lines::
--ram 4096
--threads 4
- To globally specify a directory for CodeQL to scan for CodeQL packs (which is not a
sibling of the installation directory), use::
--search-path <path-to-directory>
.. pull-quote::
This article was moved to "`Specifying command options in a CodeQL configuration file <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/specifying-command-options-in-a-codeql-configuration-file>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

268
docs/codeql/codeql-cli/testing-custom-queries.rst
View File

@@ -3,267 +3,7 @@
Testing custom queries
======================
CodeQL provides a simple test framework for automated regression testing
of queries. Test your queries to ensure that they behave as expected.
During a query test, CodeQL compares the results the user expects
the query to produce with those actually produced. If the expected and
actual results differ, the query test fails. To fix the test, you should iterate
on the query and the expected results until the actual results and the expected
results exactly match. This topic shows you how to create test files and execute
tests on them using the ``test run`` subcommand.
Setting up a test CodeQL pack for custom queries
------------------------------------------------
All CodeQL tests must be stored in a special "test" CodeQL pack.
That is, a directory for test files with a ``qlpack.yml``
file that defines:
.. code-block:: yaml
name: <name-of-test-pack>
version: 0.0.0
dependencies:
<codeql-libraries-and-queries-to-test>: "*"
extractor: <language-of-code-to-test>
The ``dependencies`` value specifies the CodeQL packs containing queries to test.
Typically, these packs will be resolved from source, and so it is not necessary
to specify a fixed version of the pack.
The ``extractor`` defines which language the CLI will use
to create test databases from the code files stored in this CodeQL pack.
For more information, see ":doc:`About CodeQL packs <about-codeql-packs>`."
You may find it useful to look at the way query tests are organized in the
`CodeQL repository <https://github.com/github/codeql>`__.
Each language has a ``src`` directory, ``ql/<language>/ql/src``,
that contains libraries and queries for analyzing codebases.
Alongside the ``src`` directory, there is a ``test`` directory with tests for
these libraries and queries.
Each ``test`` directory is configured as a test CodeQL pack with two subdirectories:
- ``query-tests`` a series of subdirectories with tests for queries stored in the ``src`` directory.
Each subdirectory contains test code and a QL reference file that specifies the query to test.
- ``library-tests`` a series of subdirectories with tests for QL library files.
Each subdirectory contains test code and queries that were written as unit tests for a library.
Setting up the test files for a query
-------------------------------------
For each query you want to test, you should create a sub-directory in the test CodeQL pack.
Then add the following files to the subdirectory before you run the test command:
- A query reference file (``.qlref`` file) defining the location of the query to test.
The location is defined relative to the root of the CodeQL pack that contains the
query. Usually, this is a CodeQL pack specified in the
``dependencies`` block of the test pack.
For more information, see ":doc:`Query reference files <query-reference-files>`."
You do not need to add a query reference file if the query you want to
test is stored in the test directory,
but it is generally good practice to store queries separately from tests.
The only exception is unit tests for QL libraries, which tend to be
stored in test packs, separate from queries that generate alerts or paths.
- The example code you want to run your query against. This
should consist of one or more files containing examples of the code the
query is designed to identify.
You can also define the results you expect to see when you run the query against
the example code, by creating a file with the extension ``.expected``.
Alternatively, you can leave the test command to create the ``.expected`` file
for you.
For an example showing how to create and test a query, see the `example
<#example>`__ below.
.. pull-quote:: Important
Your ``.ql``, ``.qlref``, and ``.expected`` files must have consistent names.
If you want to directly specify the ``.ql`` file itself in the test command,
it must have the same base name as the corresponding ``.expected`` file. For
example, if the query is ``MyJavaQuery.ql``, the expected results file must
be ``MyJavaQuery.expected``.
If you want to specify a ``.qlref`` file in the command, it must have the same base
name as the corresponding ``.expected`` file, but the query itself
may have a different name.
The names of the example code files don't have to be consistent with the
other test files. All example code files found next to the ``.qlref`` (or ``.ql``)
file and in any subdirectories will be used to create a test database.
Therefore, for simplicity, we recommend you don't save test files in
directories that are ancestors of each other.
Running ``codeql test run``
---------------------------
CodeQL query tests are executed by running the following command::
codeql test run <test|dir>
The ``<test|dir>`` argument can be one or more of the following:
- Path to a ``.ql`` file.
- Path to a ``.qlref`` file that references a ``.ql`` file.
- Path to a directory that will be searched recursively for ``.ql`` and
``.qlref`` files.
You can also specify:
- .. include:: ../reusables/threads-query-execution.rst
For full details of all the options you can use when testing queries,
see the `test run reference documentation <../manual/test-run>`__.
Example
-------
The following example shows you how to set up a test for a query that searches
Java code for ``if`` statements that have empty ``then`` blocks. It includes
steps to add the custom query and corresponding test files to separate CodeQL packs
outside your checkout of the CodeQL repository. This ensures when you update the
CodeQL libraries, or check out a different branch, you won't overwrite your
custom queries and tests.
Prepare a query and test files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. Develop the query. For example, the following simple query finds empty ``then``
blocks in Java code:
.. code-block:: ql
import java
from IfStmt ifstmt
where ifstmt.getThen() instanceof EmptyStmt
select ifstmt, "This if statement has an empty then."
#. Save the query to a file named ``EmptyThen.ql`` in a directory with your
other custom queries. For example,
``custom-queries/java/queries/EmptyThen.ql``.
#. If you haven't already added your custom queries to a CodeQL pack,
create a CodeQL pack now. For example, if your custom Java queries
are stored in ``custom-queries/java/queries``, add a ``qlpack.yml`` file with the
following contents to ``custom-queries/java/queries``:
.. code-block:: yaml
name: my-custom-queries
dependencies:
codeql/java-queries: "*"
For more information about CodeQL packs, see ":doc:`About CodeQL packs
<about-codeql-packs>`."
#. Create a CodeQL pack for your Java tests by adding a ``qlpack.yml`` file
with the following contents to ``custom-queries/java/tests``,
updating the ``dependencies`` to match the name of your CodeQL pack of custom queries:
.. include:: ../reusables/test-qlpack.rst
#. Within the Java test pack, create a directory to contain the test files
associated with ``EmptyThen.ql``.
For example, ``custom-queries/java/tests/EmptyThen``.
#. In the new directory, create ``EmptyThen.qlref`` to define the location of ``EmptyThen.ql``.
The path to the query must be specified relative to the root of
the CodeQL pack that contains the query. In this case, the query is in the
top level directory of the CodeQL pack named ``my-custom-queries``,
which is declared as a dependency for ``my-query-tests``.
Therefore, ``EmptyThen.qlref`` should simply contain ``EmptyThen.ql``.
#. Create a code snippet to test. The following Java code contains an
empty ``if`` statement on the third line. Save it in
``custom-queries/java/tests/EmptyThen/Test.java``.
.. code-block:: java
class Test {
public void problem(String arg) {
if (arg.isEmpty())
;
{
System.out.println("Empty argument");
}
}
public void good(String arg) {
if (arg.isEmpty()) {
System.out.println("Empty argument");
}
}
}
Execute the test
^^^^^^^^^^^^^^^^
To execute the test, move into the ``custom-queries`` directory and run ``codeql
test run java/tests/EmptyThen``.
When the test runs it:
#. Finds one test in the ``EmptyThen`` directory.
#. Extracts a CodeQL database from the ``.java`` files stored in the ``EmptyThen`` directory.
#. Compiles the query referenced by the ``EmptyThen.qlref`` file.
If this step fails, it's because the CLI can't find your custom CodeQL pack.
Re-run the command and specify the location of your custom CodeQL pack, for example:
``codeql test run --search-path=java java/tests/EmptyThen``
For information about saving the search path as part of your configuration, see
":ref:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`."
#. Executes the test by running the query and generating an ``EmptyThen.actual`` results file.
#. Checks for an ``EmptyThen.expected`` file to compare with the ``.actual`` results file.
#. Reports the results of the test --- in this case, a failure: ``0 tests passed; 1 tests failed:``.
The test failed because we haven't yet added a file with the expected results of the query.
View the query test output
^^^^^^^^^^^^^^^^^^^^^^^^^^
CodeQL generates the following files in the ``EmptyThen`` directory:
- ``EmptyThen.actual``, a file that contains the actual results generated by the
query.
- ``EmptyThen.testproj``, a test database that you can load into VS Code and use to debug failing tests.
When tests complete successfully, this database is deleted in a housekeeping step.
You can override this step by running ``test run`` with the ``--keep-databases`` option.
In this case, the failure was expected and is easy to fix.
If you open the ``EmptyThen.actual`` file, you can see the results of the test:
.. code-block:: none
| Test.java:3:5:3:22 | stmt | This if statement has an empty then. |
This file contains a table, with a column for the location of the result,
along with separate columns for each part of the ``select`` clause the query outputs.
Since the results are what we expected, we can update the file extension to define
this as the expected result for this test (``EmptyThen.expected``).
If you rerun the test now, the output will be similar but it will finish by reporting:
``All 1 tests passed.``.
If the results of the query change, for example, if you revise the ``select`` statement for the query,
the test will fail. For failed results, the CLI output includes a unified diff of the
``EmptyThen.expected`` and ``EmptyThen.actual`` files.
This information may be sufficient to debug trivial test failures.
For failures that are harder to debug, you can import ``EmptyThen.testproj``
into CodeQL for VS Code, execute ``EmptyThen.ql``, and view the results in the
``Test.java`` example code. For more information, see ":ref:`Analyzing your projects
<analyzing-your-projects>`" in the CodeQL for VS Code
help.
Further reading
---------------
- ":ref:`CodeQL queries
<codeql-queries>`"
- ":ref:`Testing CodeQL queries in Visual Studio Code <testing-codeql-queries-in-visual-studio-code>`"
.. pull-quote::
This article was moved to "`Testing custom queries <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

84
docs/codeql/codeql-cli/testing-query-help-files.rst
View File

@@ -3,83 +3,7 @@
Testing query help files
========================
Test query help files by rendering them as markdown to ensure they are valid
before uploading them to the CodeQL repository or using them in code scanning.
Query help is documentation that accompanies a query to explain how the query works,
as well as providing information about the potential problem that the query identifies.
It is good practice to write query help for all new queries. For more information,
see `Contributing to CodeQL <https://github.com/github/codeql/blob/main/CONTRIBUTING.md>`__
in the CodeQL repository.
The CodeQL CLI includes a command to test query help and render the content as
markdown, so that you can easily preview the content in your IDE. Use the command to validate
query help files before uploading them to the CodeQL repository or sharing them with other users.
From CodeQL CLI 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files
generated during CodeQL analyses so that the query help can be displayed in the code scanning UI.
For more information, see
":ref:`Analyzing databases with the CodeQL CLI <including-query-help-for-custom-codeql-queries-in-sarif-files>`."
Prerequisites
-------------
- The query help (``.qhelp``) file must have an accompanying query (``.ql``) file with
an identical base name.
- The query help file should follow the standard structure and style for query help documentation.
For more information, see the `Query help style guide <https://github.com/github/codeql/blob/main/docs/query-help-style-guide.md>`__
in the CodeQL repository.
Running ``codeql generate query-help``
--------------------------------------
You can test query help files by running the following command::
codeql generate query-help <qhelp|query|dir|suite> --format=<format> [--output=<dir|file>]
where ``<qhelp|query|dir|suite>`` is one of:
- the path to a ``.qhelp`` file.
- the path to a ``.ql`` file.
- the path to a directory containing queries and query help files.
- the path to a query suite, or the name of a well-known query suite for a CodeQL pack.
For more information, see "`Creating CodeQL query suites <https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites#specifying-well-known-query-suites>`__."
You must specify a ``--format`` option, which defines how the query help is rendered.
Currently, you must specify ``markdown`` to render the query help as markdown.
The ``--output`` option defines a file path where the rendered query help will be saved.
- For directories containing ``.qhelp`` files or a query suites
defining one or more ``.qhelp`` files, you must specify an ``--output`` directory.
Filenames within the output directory will be derived from the ``.qhelp`` file names.
- For single ``.qhelp`` or ``.ql`` files, you may specify an ``--output`` option.
If you don't specify an output path, the rendered query help is written to ``stdout``.
For full details of all the options you can use when testing query help files,
see the `generate query-help reference documentation
<../manual/generate-query-help>`__.
Results
-------
When you run the command, CodeQL attempts to render
each ``.qhelp`` file that has an accompanying ``.ql`` file. For single files, the rendered
content will be printed to ``stdout`` if you don't specify an ``--output`` option. For all other
use cases, the rendered content is saved to the specified output path.
By default, the CodeQL CLI will print a warning message if:
- Any of the query help is invalid, along with a description of the invalid query help elements
- Any ``.qhelp`` files specified in the command don't have the same base name
as an accompanying ``.ql`` file
- Any ``.ql`` files specified in the command don't have the same base name
as an accompanying ``.qhelp`` file
You can tell the CodeQL CLI how to handle these warnings by including a ``--warnings`` option in your command.
For more information, see the `generate query-help reference documentation <../manual/generate-query-help#cmdoption-codeql-generate-query-help-warnings>`__.
Further reading
---------------
- ":ref:`Query help files <query-help-files>`"
.. pull-quote::
This article was moved to "`Testing query help files <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-query-help-files>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

83
docs/codeql/codeql-cli/using-custom-queries-with-the-codeql-cli.rst
View File

@@ -3,82 +3,7 @@
Using custom queries with the CodeQL CLI
=========================================
You can customize your CodeQL analyses by writing your own queries to highlight
specific vulnerabilities or errors.
This topic is specifically about writing
queries to use with the `database analyze <../manual/database-analyze>`__
command to produce :ref:`interpreted results <interpret-query-results>`.
.. include:: ../reusables/advanced-query-execution.rst
Writing a valid query
---------------------
Before running a custom analysis you need to write a valid query, and save it in
a file with a ``.ql`` extension. There is extensive documentation available to
help you write queries. For more information, see ":ref:`CodeQL queries
<codeql-queries>`."
.. _including-query-metadata:
Including query metadata
------------------------
Query metadata is included at the top of each query file. It provides users with information about
the query, and tells the CodeQL CLI how to process the query results.
When running queries with the ``database analyze`` command, you must include the
following two properties to ensure that the results are interpreted correctly:
- Query identifier (``@id``): a sequence of words composed of lowercase letters or
digits, delimited by ``/`` or ``-``, identifying and classifying the query.
- Query type (``@kind``): identifies the query as a simple alert (``@kind problem``),
an alert documented by a sequence of code locations (``@kind path-problem``),
for extractor troubleshooting (``@kind diagnostic``), or a summary metric
(``@kind metric`` and ``@tags summary``).
For more information about these metadata properties, see ":ref:`Metadata for CodeQL queries
<metadata-for-codeql-queries>`" and the `Query metadata style guide
<https://github.com/github/codeql/blob/main/docs/query-metadata-style-guide.md>`__.
.. pull-quote:: Note
Metadata requirements may differ if you want to use your query with other
applications. For more information, see ":ref:`Metadata for CodeQL queries
<metadata-for-codeql-queries>`
."
Packaging custom QL queries
---------------------------
.. include:: ../reusables/beta-note-package-management.rst
When you write your own queries with the intention to share them with others, you should
save them in a custom CodeQL pack. You can publish the pack as a CodeQL pack to GitHub
Packages - the GitHub Container registry. For further information see
":ref:`About CodeQL packs <about-codeql-packs>`."
CodeQL packs organize the files used in CodeQL analysis and can store queries,
library files, query suites, and important metadata. Their root directory must
contain a file named ``qlpack.yml``. Your custom queries should be saved in the
CodeQL pack root, or its subdirectories.
For each CodeQL pack, the ``qlpack.yml`` file includes information that tells the CodeQL CLI
how to compile the queries, which other CodeQL packs and libraries the pack
depends on, and where to find query suite definitions. For more information
about what to include in this file, see ":ref:`About CodeQL packs <codeqlpack-yml-properties>`."
Contributing to the CodeQL repository
-------------------------------------
If you would like to share your query with other CodeQL users, you can open a
pull request in the `CodeQL repository <https://github.com/github/codeql>`__. For
further information, see `Contributing to CodeQL
<https://github.com/github/codeql/blob/main/CONTRIBUTING.md>`__.
Further reading
---------------
- ":ref:`CodeQL queries
<codeql-queries>`"
.. pull-quote::
This article was moved to "`Using custom queries with the CodeQL CLI <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/using-custom-queries-with-the-codeql-cli>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst

66
docs/codeql/codeql-cli/using-the-codeql-cli.rst
View File

@@ -3,66 +3,8 @@
Using the CodeQL CLI
====================
.. include:: ../reusables/codeql-cli-overview.rst
.. pull-quote::
This category was moved to "`Using the CodeQL CLI <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli>`__" on the `GitHub Docs <https://docs.github.com/en/code-security/codeql-cli>`__ site as of January 2023.
.. include:: ../reusables/codeql-cli-articles-migration-note.rst
See the following links to learn how to get set up and run CodeQL commands:
- :doc:`About the CodeQL CLI <about-the-codeql-cli>`: Software
developers and security researchers can secure their code using the CodeQL CLI.
- :doc:`Getting started with the CodeQL CLI
<getting-started-with-the-codeql-cli>`: Set up the
CodeQL CLI so that you can run CodeQL processes from your command line.
- :doc:`Creating CodeQL databases
<creating-codeql-databases>`: Create relational
representations of source code that can be queried like any other database.
- :doc:`Extractor options
<extractor-options>`: Set options for the
behavior of extractors that create CodeQL databases.
- :doc:`Analyzing CodeQL databases with the CodeQL CLI
<analyzing-databases-with-the-codeql-cli>`: Analyze your code using queries
written in a specially-designed, object-oriented query language.
- :doc:`Using custom queries with the CodeQL CLI
<using-custom-queries-with-the-codeql-cli>`: Use custom queries to extend your
analysis or highlight errors that are specific to a particular codebase.
- :doc:`Creating CodeQL query suites <creating-codeql-query-suites>`:
Define query suite definitions for groups of frequently used queries.
- :doc:`Testing custom queries <testing-custom-queries>`: Set up
regression testing of custom queries to ensure that they behave as expected in
your analysis.
- :doc:`Testing query help files <testing-query-help-files>`:
Test query help files by rendering them as markdown to ensure they are valid
before adding them to the CodeQL repository or using them in code scanning.
- :doc:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`:
Create, share, depend on, and run CodeQL queries and libraries.
- :doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`:
Publish your own or use others CodeQL packs for code scanning.
- :doc:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`:
You can save default or frequently used options for your commands in a per-user configuration file.
.. toctree::
:titlesonly:
:hidden:
about-the-codeql-cli
getting-started-with-the-codeql-cli
creating-codeql-databases
extractor-options
analyzing-databases-with-the-codeql-cli
using-custom-queries-with-the-codeql-cli
creating-codeql-query-suites
testing-custom-queries
testing-query-help-files
creating-and-working-with-codeql-packs
publishing-and-using-codeql-packs
Specifying command options <specifying-command-options-in-a-codeql-configuration-file>

8
docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst
View File

@@ -29,13 +29,13 @@ Obtaining a local database
If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP file, you can add it to Visual Studio Code. There are several ways to obtain a local CodeQL database.
- To create a database with the CodeQL CLI, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`."
- To create a database with the CodeQL CLI, see "`Creating CodeQL databases <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-databases>`__."
- To analyze a test database, add a ``.testproj`` folder to the Databases view.
Test databases (that is, folders with a ``.testproj`` extension) are generated when you run regression tests on custom queries using the :ref:`CodeQL CLI <codeql-cli>`.
If a query fails a regression test, you may want to analyze the test database in Visual Studio Code to debug the failure.
For more information about running query tests, see ":ref:`Testing custom queries <testing-custom-queries>`" in the CodeQL CLI help.
For more information about running query tests, see "`Testing custom queries <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries>`__" in the CodeQL CLI help.
.. include:: ../reusables/download-github-database.rst
@@ -77,7 +77,7 @@ To run the query, use **CodeQL: Run Query on Selected Database**.
You can see all quick queries that you've run in the current session in the Query History view. Click an entry to see the exact text of the quick query that produced the results.
Once you're happy with your quick query, you should save it in a CodeQL pack so you can access it later. For more information, see ":ref:`About CodeQL packs <about-codeql-packs>`."
Once you're happy with your quick query, you should save it in a CodeQL pack so you can access it later. For more information, see "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__."
.. _running-a-specific-part-of-a-query-or-library:
@@ -135,7 +135,7 @@ Viewing query results
.. pull-quote:: Note
Depending on the query, you can also choose different views such as CSV, :ref:`SARIF <sarif-output>`, or :ref:`DIL format <dil>`. For example, to view the DIL format, right-click a result and select **View DIL**.
Depending on the query, you can also choose different views such as CSV, `SARIF <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/sarif-output>`__, or :ref:`DIL format <dil>`. For example, to view the DIL format, right-click a result and select **View DIL**.
The available output views are determined by the format and the metadata of the query. For more information, see ":ref:`CodeQL queries <codeql-queries>`."
#. Use the dropdown menu in the Results view to choose which results to display, and in what form to display them, such as a formatted alert message or a table of raw results.

2
docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst
View File

@@ -28,7 +28,7 @@ Editing settings
Choosing a version of the CodeQL CLI
--------------------------------------
The CodeQL extension uses the CodeQL CLI to run commands. If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from GitHub.com. Otherwise, the extension automatically manages access to the executable of the CLI for you. For more information about creating databases, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`" in the CLI help.
The CodeQL extension uses the CodeQL CLI to run commands. If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from GitHub.com. Otherwise, the extension automatically manages access to the executable of the CLI for you. For more information about creating databases, see "`Creating CodeQL databases <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-databases>`__" in the CLI help.
To override the default behavior and use a different CLI, you can specify the CodeQL CLI **Executable Path**.

4
docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst
View File

@@ -55,7 +55,7 @@ There are two ways to do this:
.. pull-quote:: Note
For CLI users there is a third option: If you have followed the instructions in ":ref:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`" to create a CodeQL directory (for example ``codeql-home``) containing the CodeQL libraries, you can open this directory in VS Code. This also gives the extension access to the CodeQL libraries.
For CLI users there is a third option: If you have followed the instructions in "`Getting started with the CodeQL CLI <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli>`__" to create a CodeQL directory (for example ``codeql-home``) containing the CodeQL libraries, you can open this directory in VS Code. This also gives the extension access to the CodeQL libraries.
.. _starter-workspace:
@@ -96,7 +96,7 @@ For example, to make a custom CodeQL folder called ``my-custom-cpp-pack`` depend
version: 0.0.0
libraryPathDependencies: codeql/cpp-all
For more information about why you need to add a ``qlpack.yml`` file, see ":ref:`About CodeQL packs <about-codeql-packs>`."
For more information about why you need to add a ``qlpack.yml`` file, see "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__."
Further reading
----------------

4
docs/codeql/codeql-for-visual-studio-code/testing-codeql-queries-in-visual-studio-code.rst
View File

@@ -14,7 +14,7 @@ To ensure that your CodeQL queries produce the expected results, you can run tes
The CodeQL extension automatically prompts VS Code to install the Test Explorer extension as a dependency. The Test Explorer displays any workspace folders with a name ending in ``-tests`` and provides a UI for exploring and running tests in those folders.
For more information about how CodeQL tests work, see ":ref:`Testing custom queries <testing-custom-queries>`" in the CLI help.
For more information about how CodeQL tests work, see "`Testing custom queries <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries>`__" in the CLI help.
Testing the results of your queries
-----------------------------------
@@ -49,4 +49,4 @@ For more information, see ":ref:`Troubleshooting query performance <troubleshoot
Further reading
----------------
* ":ref:`Testing custom queries <testing-custom-queries>`"
* "`Testing custom queries <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries>`__"

6
docs/codeql/codeql-for-visual-studio-code/working-with-codeql-packs-in-visual-studio-code.rst
View File

@@ -11,7 +11,7 @@ You can view CodeQL packs and write and edit queries for them in Visual Studio C
About CodeQL packs
------------------
CodeQL packs are used to create, share, depend on, and run CodeQL queries and libraries. You can publish your own CodeQL packs and download packs created by others. For more information, see ":ref:`About CodeQL packs <about-codeql-packs>`."
CodeQL packs are used to create, share, depend on, and run CodeQL queries and libraries. You can publish your own CodeQL packs and download packs created by others. For more information, see "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__."
Using standard CodeQL packs in Visual Studio Code
--------------------------------------------------------------
@@ -21,11 +21,11 @@ You can write and run query packs that depend on the CodeQL standard libraries,
Creating and editing CodeQL packs in Visual Studio Code
-------------------------------------------------------
To create a new CodeQL pack, you will need to use the CodeQL CLI from a terminal, which you can do within Visual Studio Code or outside of it with the ``codeql pack init`` command. Once you create an empty pack, you can edit the ``qlpack.yml`` file or run the ``codeql pack add`` command to add dependencies or change the name or version. For more information, see ":ref:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`."
To create a new CodeQL pack, you will need to use the CodeQL CLI from a terminal, which you can do within Visual Studio Code or outside of it with the ``codeql pack init`` command. Once you create an empty pack, you can edit the ``qlpack.yml`` file or run the ``codeql pack add`` command to add dependencies or change the name or version. For more information, see "`Creating and working with CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-and-working-with-codeql-packs>`__."
You can create or edit queries in a CodeQL pack in Visual Studio Code as you would with any CodeQL query, using the standard code editing features such as autocomplete suggestions to find elements to use from the pack's dependencies.
You can then use the CodeQL CLI to publish your pack to share with others. For more information, see ":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
You can then use the CodeQL CLI to publish your pack to share with others. For more information, see "`Publishing and using CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/publishing-and-using-codeql-packs>`__."
Viewing CodeQL packs and their dependencies in Visual Studio Code
-----------------------------------------------------------------

3
docs/codeql/codeql-overview/codeql-glossary.rst
View File

@@ -100,8 +100,7 @@ SARIF file
----------
Static analysis results interchange format (SARIF) is an output format used for
sharing static analysis results. For more information, see ":ref:`SARIF
output <sarif-output>`."
sharing static analysis results. For more information, see "`SARIF <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/sarif-output>`__."
.. _source-reference:

2
docs/codeql/codeql-overview/codeql-tools.rst
View File

@@ -39,7 +39,7 @@ maintained by GitHub are:
- ``codeql/ruby-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/src>`__)
- ``codeql/ruby-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/lib>`__)
For more information, see ":ref:`About CodeQL packs <about-codeql-packs>`."
For more information, see "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__."
CodeQL bundle
-----------------------------

2
docs/codeql/ql-language-reference/ql-language-specification.rst
View File

@@ -62,7 +62,7 @@ where such a file is found is the query directory. If there is no such
directory, the directory of the ``.ql`` file itself is the query
directory.
A ``qlpack.yml`` file defines a :ref:`CodeQL pack <about-codeql-packs>`.
A ``qlpack.yml`` file defines a `CodeQL pack <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__.
The content of a ``qlpack.yml`` file is described in the CodeQL CLI documentation.
``codeql-pack.yml`` is an alias for ``qlpack.yml``.

1
docs/codeql/reusables/codeql-cli-articles-migration-note.rst Normal file
View File

@@ -0,0 +1 @@
The source file now exists in the public, open-source `docs <https://github.com/github/docs/tree/main/content/code-security/codeql-cli>`__ repository. If you would like to contribute, you can consult the `GitHub docs contributing guide <https://github.com/github/docs/blob/main/CONTRIBUTING.md>`__.

2
docs/codeql/writing-codeql-queries/about-codeql-queries.rst
View File

@@ -13,7 +13,7 @@ CodeQL includes queries to find the most relevant and interesting problems for e
- **Alert queries**: queries that highlight issues in specific locations in your code.
- **Path queries**: queries that describe the flow of information between a source and a sink in your code.
You can add custom queries to :doc:`CodeQL packs <../codeql-cli/about-codeql-packs>` to analyze your projects with "`Code scanning <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning>`__", use them to analyze a database with the ":ref:`CodeQL CLI <codeql-cli>`," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub <https://github.com/github/codeql>`__.
You can add custom queries to `CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__ to analyze your projects with "`Code scanning <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning>`__", use them to analyze a database with the ":ref:`CodeQL CLI <codeql-cli>`," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub <https://github.com/github/codeql>`__.
This topic is a basic introduction to query files. You can find more information on writing queries for specific programming languages in the ":ref:`CodeQL language guides <codeql-language-guides>`," and detailed technical information about QL in the ":ref:`QL language reference <ql-language-reference>`."
For more information on how to format your code when contributing queries to the GitHub repository, see the `CodeQL style guide <https://github.com/github/codeql/blob/main/docs/ql-style-guide.md>`__.

2
docs/query-help-style-guide.md
View File

@@ -123,7 +123,7 @@ codeql generate query-help ./myCustomQuery.qhelp --format=markdown
Please include the `.qhelp` files (and any associated code snippets) in your pull request, but do not commit the generated Markdown.
More information on how to test your `.qhelp` files can be found [within the documentation](https://codeql.github.com/docs/codeql-cli/testing-query-help-files/)
More information on how to test your `.qhelp` files can be found [within the documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-query-help-files)
## Query help example

2
docs/supported-queries.md
View File

@@ -27,7 +27,7 @@ The process must begin with the first step and must conclude with the final step
Add one or more unit tests for the query (and for any library changes you make) to the `ql/<language>/ql/test/experimental` directory. Tests for library changes go into the `library-tests` subdirectory, and tests for queries go into `query-tests` with their relative path mirroring the query's location under `ql/<language>/ql/src/experimental`.
- See the section on [Testing custom queries](https://codeql.github.com/docs/codeql-cli/testing-custom-queries/) in the [CodeQL documentation](https://codeql.github.com/docs/) for more information.
- See the section on [Testing custom queries](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries) in the [CodeQL CLI documentation](https://docs.github.com/en/code-security/codeql-cli) for more information.
- See [C/C++ CodeQL tests](/cpp/ql/test/README.md) for more information about contributing tests for C/C++ queries in particular.
4. **Test for correctness on real-world code**

2
go/CONTRIBUTING.md
View File

@@ -8,7 +8,7 @@ Please note that this project is released with a [Contributor Code of Conduct](C
## Building and testing
1. Install the CodeQL CLI as described in [Getting started with the CodeQL CLI](https://codeql.github.com/docs/codeql-cli/getting-started-with-the-codeql-cli/).
1. Install the CodeQL CLI as described in [Getting started with the CodeQL CLI](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli).
2. Ensure that `<extraction-root>/codeql` is in your `PATH`.

2
go/README.md
View File

@@ -13,7 +13,7 @@ It contains two major components:
To analyze a Go codebase, either use the [CodeQL command-line
interface](https://codeql.github.com/docs/codeql-cli/) to create a database yourself, or
download a pre-built database from [GitHub.com](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#downloading-databases-from-github-com). You can then run any of the
download a pre-built database from [GitHub.com](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-databases#downloading-databases-from-githubcom). You can then run any of the
queries contained in this repository either on the command line or using the VS Code extension.
## Contributions

2
swift/README.md
View File

@@ -25,7 +25,7 @@ Notice you can run `bazel run :create-extractor-pack` if you already are in the
Using `codeql ... --search-path=swift/extractor-pack` will then pick up the Swift extractor. You can also use
`--search-path=.`, as the extractor pack is mentioned in the root `codeql-workspace.yml`. Alternatively, you can
set up the search path
in [the per-user CodeQL configuration file](https://codeql.github.com/docs/codeql-cli/specifying-command-options-in-a-codeql-configuration-file/#using-a-codeql-configuration-file)
in [the per-user CodeQL configuration file](https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/specifying-command-options-in-a-codeql-configuration-file#using-a-codeql-configuration-file)
.
## Code generation