hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2026-04-28 02:05:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #10459 from github/aeisenberg/update-analyzing-databases

Browse Source 
Update the analyze databases article
This commit is contained in:
Andrew Eisenberg
2022-09-28 10:14:08 -07:00
committed by GitHub
parent 470908f53e e8a0d07217
commit ffd58861f0
1 changed files with 96 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

153
docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst
View File

@@ -15,11 +15,13 @@ For information about writing queries to run with ``database analyze``, see
Before starting an analysis you must:
- :doc:`Set up the CodeQL CLI <getting-started-with-the-codeql-cli>` so that it can find the queries
and libraries included in the CodeQL repository.
- :doc:`Set up the CodeQL CLI <getting-started-with-the-codeql-cli>` to run commands locally.
- :doc:`Create a CodeQL database <creating-codeql-databases>` for the source
code you want to analyze.
The simplest way to run ``codeql database analyze`` is using CodeQL packs. You can also
run the command using queries from a local checkout of the CodeQL repository,
which you may want to do if you want to customize the CodeQL core queries.
Running ``codeql database analyze``
------------------------------------
@@ -34,7 +36,7 @@ When you run ``database analyze``, it:
You can analyze a database by running the following command::
codeql database analyze <database> --format=<format> --output=<output> <queries>
codeql database analyze <database> --format=<format> --output=<output> <query-specifiers>...
You must specify:
@@ -52,8 +54,8 @@ You must specify:
You can also specify:
- ``...<query-specifications>``: a list of queries to run over your database. This
is a list of arguments. Where each argument can be:
- ``<query-specifiers>...``: a space-separated list of queries to run over your database. This
is a list of arguments, where each argument can be:
- a path to a query file
- a path to a directory containing query files
@@ -62,7 +64,7 @@ You can also specify:
- with an optional version range
- with an optional path to a query, directory, or query suite inside the pack
If omitted, the default query suite for the language of the database being analyzed will be used. For more information, see the :ref:`examples <database-analyze-examples>` below.
If omitted, the default query suite for the language of the analyzed database will be used. For the complete syntax of query specifiers, see :ref:`"Specifying which queries to run in a CodeQL pack"<specifying-which-queries>`.
- ``--sarif-category``: an identifying category for the results. Used when
you want to upload more than one set of results for a commit.
@@ -95,14 +97,77 @@ You can also specify:
For full details of all the options you can use when analyzing databases, see
the `database analyze reference documentation <../manual/database-analyze>`__.
.. _database-analyze-examples:
Examples
--------
.. _specifying-which-queries:
The following examples assume your CodeQL databases have been created in a
directory that is a sibling of your local copies of the CodeQL and CodeQL for Go
repositories.
Specifying which queries to run in a CodeQL pack
------------------------------------------------
Query specifiers are used by ``codeql database analyze`` and other commands that operate on a set of queries.
The complete form of a query specifier is``scope/name@range:path``, where:
- ``scope/name`` is the qualified name of a CodeQL pack.
- ``range`` is a `semver range <https://docs.npmjs.com/cli/v6/using-npm/semver#ranges>`_.
- ``path`` is a file system path to a single query, a directory containing queries, or a query suite file.
When you specify a ``scope/name``, the ``range`` and ``path`` are
optional. If you omit a ``range`` then the latest version of the
specified pack is used. If you omit a ``path`` then the default query suite
of the specified pack is used.
The ``path`` can be one of: a ``.ql`` query file, a directory
containing one or more queries, or a ``.qls`` query suite file. If
you omit a pack name, then you must provide a ``path``,
which will be interpreted relative to the working directory
of the current process. Glob patterns are not supported.
If you specify both a ``scope/name`` and ``path``, then the ``path`` cannot
be absolute. It is considered relative to the root of the CodeQL
pack.
Example query specifiers
~~~~~~~~~~~~~~~~~~~~~~~~
* ``codeql/python-queries`` - All the queries in the default query suite of the latest version of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@1.2.3`` - All the queries in the default query suite of version ``1.2.3`` of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@~1.2.3`` - All the queries in the default query suite of the latest version of the ``codeql/python-queries`` pack that is >= ``1.2.3`` and < ``1.3.0``.
* ``codeql/python-queries:Functions`` - All queries in the ``Functions`` directory in the latest version of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@1.2.3:Functions`` - All queries in the ``Functions`` directory in version 1.2.3 of the ``codeql/python-queries`` pack.
* ``codeql/python-queries@1.2.3:codeql-suites/python-code-scanning.qls`` - All queries in the ``codeql-suites/python-code-scanning.qls`` directory in version 1.2.3 of the ``codeql/python-queries`` pack.
* ``suites/my-suite.qls`` - All queries in the ``suites/my-suite.qls`` file relative to the current working directory.
.. pull-quote::
Tip
The default query suite of the standard CodeQL query packs are ``codeql-suites/<lang>-code-scanning.qls``. Several other useful query suites can also be found in the ``codeql-suites`` directory of each pack. For example, the ``codeql/cpp-queries`` pack contains the following query suites:
* ``cpp-code-scanning.qls`` - Standard Code Scanning queries for C++. The default query suite for this pack.
* ``cpp-security-extended.qls`` - Queries from the default ``cpp-code-scanning.qls`` suite for C++, plus lower severity and precision queries.
* ``cpp-security-and-quality.qls`` - Queries from ``cpp-security-extended.qls``, plus maintainability and reliability queries.
You can see the sources for these query suites in the `CodeQL repository <https://github.com/github/codeql/tree/main/cpp/ql/src/codeql-suites>`__. Query suites for other languages are similar.
Examples of running database analyses
---------------------------------------------
The following examples show how to run ``database analyze`` using CodeQL packs, and how to use a local checkout of the CodeQL repository. These examples assume your CodeQL databases have been created in a directory that is a sibling of your local copies of the CodeQL repository.
.. _run-query-pack:
Running a CodeQL query pack
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: ../reusables/beta-note-package-management.rst
To run an existing CodeQL query pack from the GitHub Container registry, you can specify one or more
pack names::
codeql database analyze <database> microsoft/coding-standards@1.0.0 github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
This command runs the default query suite of two CodeQL query packs: ``microsoft/coding-standards`` version 1.0.0 and the latest version of ``github/security-queries`` on the specified database. For further information about default suites, see ":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`".
The ``--download`` flag is optional. Using it will ensure the query pack is downloaded if it isn't yet available locally.
Running a single query
~~~~~~~~~~~~~~~~~~~~~~
@@ -110,41 +175,23 @@ Running a single query
To run a single query over a CodeQL database for a JavaScript codebase,
you could use the following command from the directory containing your database::
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
codeql database analyze --download <javascript-database> codeql/javascript-queries:Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
This command runs a simple query that finds potential bugs related to unused
variables, imports, functions, or classes---it is one of the JavaScript
queries included in the CodeQL repository. You could run more than one query by
specifying a space-separated list of similar paths.
The analysis generates a CSV file (``js-results.csv``) in a new directory
(``js-analysis``).
The analysis generates a CSV file (``js-results.csv``) in a new directory (``js-analysis``).
Alternatively, if you have the CodeQL repository checked out, you can execute the same queries by specifying the path to the query directly::
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
You can also run your own custom queries with the ``database analyze`` command.
For more information about preparing your queries to use with the CodeQL CLI,
see ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
If you do not have the CodeQL repository checked out, you can execute the same queries by specifying the query pack name and the path to the queries::
codeql database analyze --download <javascript-database> codeql/javascript-queries:Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
Use the ``--download`` flag to download the query pack if it isn't yet available locally.
.. _run-query-pack:
Running a CodeQL pack
~~~~~~~~~~~~~~~~~~~~~
.. include:: ../reusables/beta-note-package-management.rst
To run an existing CodeQL query pack from the GitHub Container registry, you can specify one or more
pack names and use the ``--download`` flag::
codeql database analyze <database> microsoft/coding-standards@1.0.0 github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
The ``analyze`` command above runs the default suite from ``microsoft/coding-standards v1.0.0`` and the latest version of ``github/security-queries`` on the specified database.
For further information about default suites, see ":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`".
Running all queries in a directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -163,7 +210,13 @@ recursively, so any queries contained in subfolders will also be executed.
pack's default queries in the analysis, or run one of the
code scanning query suites.
For example, to execute all Python queries contained in the ``Functions`` directory you would run::
For example, to execute all Python queries contained in the ``Functions`` directory in the
``codeql/python-queries`` query pack you would run::
codeql database analyze <python-database> codeql/python-queries:Functions --format=sarif-latest --output=python-analysis/python-results.sarif --download
Alternatively, if you have the CodeQL repository checked out, you can execute the
same queries by specifying the path to the directory directly::
codeql database analyze <python-database> ../ql/python/ql/src/Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif
@@ -171,8 +224,6 @@ When the analysis has finished, a SARIF results file is generated. Specifying ``
that the results are formatted according to the most recent SARIF specification
supported by CodeQL.
.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
Running a subset of queries in a CodeQL pack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -224,13 +275,12 @@ For more information about CodeQL packs, see :doc:`About CodeQL Packs <about-cod
Running query suites
~~~~~~~~~~~~~~~~~~~~
To run a query suite over a CodeQL database for a C/C++ codebase,
To run a query suite on a CodeQL database for a C/C++ codebase,
you could use the following command from the directory containing your database::
codeql database analyze <cpp-database> cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif
codeql database analyze <cpp-database> codeql/cpp-queries:codeql-suites/cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif --download
The analysis generates a file in the v2.1.0 SARIF format that is supported by all versions of GitHub.
This file can be uploaded to GitHub by executing ``codeql github upload-results`` or the code scanning API.
This command downloads the ``codeql/cpp-queries`` CodeQL query pack, runs the analysis, and generates a file in the SARIF version 2.1.0 format that is supported by all versions of GitHub. This file can be uploaded to GitHub by executing ``codeql github upload-results`` or the code scanning API.
For more information, see `Analyzing a CodeQL database <https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__
or `Code scanning API <https://docs.github.com/en/rest/reference/code-scanning>`__ in the GitHub documentation.
@@ -238,19 +288,8 @@ CodeQL query suites are ``.qls`` files that use directives to select queries to
based on certain metadata properties. The standard CodeQL packs have metadata that specify
the location of the query suites used by code scanning, so the CodeQL CLI knows where to find these
suite files automatically, and you don't have to specify the full path on the command line.
For more information, see ":ref:`About CodeQL packs <standard-codeql-packs>`."
For more information, see ":ref:`Creating CodeQL query suites <creating-codeql-query-suites>`."
The standard query suites are stored at the following paths in
the CodeQL repository::
ql/<language>/ql/src/codeql-suites/<language>-code-scanning.qls
and at the following path in the CodeQL for Go repository::
ql/src/codeql-suites/go-code-scanning.qls
The repository also includes the query suites used by `LGTM.com <https://lgtm.com>`__.
These are stored alongside the query suites for code scanning with names of the form: ``<language>-lgtm.qls``.
For information about creating custom query suites, see ":doc:`Creating
CodeQL query suites <creating-codeql-query-suites>`."
@@ -265,11 +304,11 @@ If the analysis found fewer results for standard queries than you expected, revi
Integrating a CodeQL pack into a code scanning workflow in GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: ../reusables/beta-note-package-management.rst
You can use CodeQL query packs in your code scanning setup. This allows you to select query packs published by various sources and use them to analyze your code.
For more information, see "`Using CodeQL query packs in the CodeQL action <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-codeql-query-packs/>`_" or "`Downloading and using CodeQL query packs in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#downloading-and-using-codeql-query-packs>`_."
.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
Including query help for custom CodeQL queries in SARIF files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~