diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 8a5fa1c04d1..6e1a76c8c0e 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -53,13 +53,14 @@ You can also specify: - ``...``: a list of queries to run over your database. This is a list of arguments. Where each argument can be: + - a path to a query file - a path to a directory containing query files - a path to a query suite file - the name of a CodeQL query pack - If omitted, the default query suite for the language - of the database being analyzed will be usedFor more information, see the - :ref:`examples ` below. + If omitted, the default query suite for the language + of the database being analyzed will be usedFor more information, see the + :ref:`examples ` below. - ``--sarif-category``: an identifying category for the results. Used when you want to upload more than one set of results for a commit. @@ -68,12 +69,12 @@ You can also specify: see `Configuring CodeQL CLI in your CI system `__ in the GitHub documentation. - ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written - in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be - converted to ``.md`` before running the analysis. For further information, - see ":ref:`Including query help for custom CodeQL queries in SARIF files `." + in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be + converted to ``.md`` before running the analysis. For further information, + see ":ref:`Including query help for custom CodeQL queries in SARIF files `." - ``--download``: a boolean flag that will allow the CLI to download any referenced CodeQL packages that are not available locally. - If this flag is missing and a referenced CodeQL package is not available locally, the command will fail. + If this flag is missing and a referenced CodeQL package is not available locally, the command will fail. - .. include:: ../reusables/threads-query-execution.rst @@ -129,7 +130,7 @@ Running a CodeQL pack To run an existing CodeQL query pack from the GitHub Container registry, you can specify one or more pack names and use the ``--download`` flag:: - codeql database analyze --download microsoft/coding-standards@1.0.0 github/secutiry-queries --format=sarifv2.1.0 --output=query-results.sarif + codeql database analyze microsoft/coding-standards@1.0.0 github/secutiry-queries --format=sarifv2.1.0 --output=query-results.sarif --download The ``analyze`` command above runs the default suite from ``microsoft/coding-standards v1.0.0`` and the latest version of ``github/secutiry-queries`` on the specified database. For further information about default suites, see ":ref:`Publishing and using CodeQL packs `". diff --git a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst index 0063b6bfc3d..2cc2a4564f3 100644 --- a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst @@ -30,6 +30,8 @@ to display in GitHub as code scanning alerts, see `Installing CodeQL CLI in your CI system `__ in the GitHub documentation. +.. _download-cli: + 1. Download the CodeQL CLI zip package ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -107,6 +109,8 @@ simply extract the zip archive. iii. ``/usr/bin/xattr -c codeql*.zip`` iv. ``unzip codeql*.zip`` +.. _launch-codeql-cli: + 3. Launch ``codeql`` ~~~~~~~~~~~~~~~~~~~~ @@ -139,7 +143,7 @@ up to create and analyze databases: - Run ``codeql resolve languages`` to show which languages are available for database creation. This will list the languages supported by default in your CodeQL CLI package. -- (Optional) You can download some ":ref"`CodeQL packs ` containing pre-compiled queries you would like to run. +- (Optional) You can download some ":ref`CodeQL packs `" containing pre-compiled queries you would like to run. To do this, run ``codeql pack download [...pack-name]``, where ``pack-name`` is the name of the pack you want to download. The core query packs are a good place to start. They are: @@ -150,9 +154,127 @@ up to create and analyze databases: - ``codeql/python-queries`` - ``codeql/ruby-queries`` - Alternatively, you can download query packs during the analysis by using the `--download` flag of the `codeql database analyze` + Alternatively, you can download query packs during the analysis by using the ``--download`` flag of the ``codeql database analyze`` command. + +Checking out the CodeQL source code directly +-------------------------------------------- + +Some users prefer working with CodeQL query sources directly in order to work on or contribute to the Open Source shared queries. In +order to do this, the following steps are recommended. Note that the following instructions are a slightly more complicated alternative +to working with CodeQL packages as explained above. + +1. Download the CodeQL CLI zip +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Follow :ref:`step 1 from the previous section`. + +2. Create a new CodeQL directory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Create a new directory where you can place the CLI and any queries and libraries +you want to use. For example, ``$HOME/codeql-home``. + +The CLI's built-in search operations automatically look in all of its sibling +directories for the files used in database creation and analysis. Keeping these +components in their own directory prevents the CLI searching unrelated sibling +directories while ensuring all files are available without specifying any +further options on the command line. + +.. _local-copy-codeql-queries: + +3. Obtain a local copy of the CodeQL queries +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The `CodeQL repository `__ contains +the queries and libraries required for CodeQL analysis of C/C++, C#, Java, +JavaScript/TypeScript, Python, and Ruby. +Clone a copy of this repository into ``codeql-home``. + +By default, the root of the cloned repository will be called ``codeql``. +Rename this folder ``codeql-repo`` to avoid conflicting with the CodeQL +CLI that you will extract in step 4. If you use git on the command line, you can +clone and rename the repository in a single step by running +``git clone git@github.com:github/codeql.git codeql-repo`` in the ``codeql-home`` folder. + +The CodeQL libraries and queries for Go analysis live in the `CodeQL for Go +repository `__. Clone a copy of this +repository into ``codeql-home``. + +The cloned repositories should have a sibling relationship. +For example, if the root of the cloned CodeQL repository is +``$HOME/codeql-home/codeql-repo``, then the root of the cloned CodeQL for Go +repository should be ``$HOME/codeql-home/codeql-go``. + +Within these repositories, the queries and libraries are organized into QL +packs. Along with the queries themselves, QL packs contain important metadata +that tells the CodeQL CLI how to process the query files. For more information, +see ":doc:`About QL packs `." + +.. pull-quote:: Important + + There are different versions of the CodeQL queries available for different + users. Check out the correct version for your use case: + + - For the queries used on `LGTM.com `__, check out the + ``lgtm.com`` branch. You should use this branch for databases you've built + using the CodeQL CLI, fetched from code scanning on GitHub, or recently downloaded from LGTM.com. + The queries on the ``lgtm.com`` branch are more likely to be compatible + with the ``latest`` CLI, so you'll be less likely to have to upgrade + newly-created databases than if you use the ``main`` branch. Older databases + may need to be upgraded before you can analyze them. + + - For the most up to date CodeQL queries, check out the ``main`` branch. + This branch represents the very latest version of CodeQL's analysis. Even + databases created using the most recent version of the CLI may have to be + upgraded before you can analyze them. For more information, see + ":doc:`Upgrading CodeQL databases `." + + - For the queries used in a particular LGTM Enterprise release, check out the + branch tagged with the relevant release number. For example, the branch + tagged ``v1.27.0`` corresponds to LGTM Enterprise 1.27. You must use this + version if you want to upload data to LGTM Enterprise. For further + information, see `Preparing CodeQL databases to upload to LGTM + `__ + in the LGTM admin help. + +4. Extract the zip archive +-------------------------- + +For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) +simply +extract the zip archive into the directory you created in step 2. + +For example, if the path to your copy of the CodeQL repository is +``$HOME/codeql-home/codeql-repo``, then extract the CLI into +``$HOME/codeql-home/``. + + +5. Launch ``codeql`` +~~~~~~~~~~~~~~~~~~~~ + +See :ref:`step 3 from the previous section`. + +6. Verify your CodeQL CLI setup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +CodeQL CLI has subcommands you can execute to verify that you are correctly set +up to create and analyze databases: + +- Run ``codeql resolve languages`` to show which languages are + available for database creation. This will list the languages supported by + default in your CodeQL CLI package. +- Run ``codeql resolve qlpacks`` to show which QL packs the CLI can find. This + will display the names of the QL packs included in the CodeQL repositories: + ``codeql-cpp``, ``codeql-csharp``, ``codeql-go``, + ``codeql-java``, ``codeql-javascript``, and ``codeql-python``. The CodeQL + repositories also contain 'upgrade' packs and 'legacy' packs. Upgrade packs + are used by the CLI when you want to upgrade a database so that it can be + analyzed with a newer version of the CodeQL toolchain than was used to create + it. Legacy packs ensure that custom queries and libraries created using older + products are compatible with your version of CodeQL. + .. _using-two-versions-of-the-codeql-cli: Using two versions of the CodeQL CLI diff --git a/docs/codeql/codeql-cli/upgrading-codeql-databases.rst b/docs/codeql/codeql-cli/upgrading-codeql-databases.rst index 4d99ced7a2f..334a7501392 100644 --- a/docs/codeql/codeql-cli/upgrading-codeql-databases.rst +++ b/docs/codeql/codeql-cli/upgrading-codeql-databases.rst @@ -14,6 +14,12 @@ Databases become out of date when: - For databases downloaded from LGTM.com, the CodeQL tools used by LGTM.com to create that revision of the code are older than your copy of the CodeQL queries. +If you have a local checkout of the ``github/codeql`` repository, please note that +the ``main`` branch of the CodeQL queries is updated more often than both the +CLI and LGTM.com, so databases are most likely to become out of date if you use +the queries on this branch. For more information about the different versions of +the CodeQL queries, see ":ref:`Getting started with the CodeQL CLI `." + Out-of-date databases must be upgraded before they can be analyzed. This topic shows you how to upgrade a CodeQL database using the ``database upgrade`` subcommand.