From 4a8b71a245782d2330ac22584d350d121f39ffb5 Mon Sep 17 00:00:00 2001 From: Felicity Chapman Date: Thu, 24 Jun 2021 14:44:08 +0100 Subject: [PATCH] Make initial changes for multi-db options --- ...nalyzing-databases-with-the-codeql-cli.rst | 49 ++++++------ .../codeql-cli/creating-codeql-databases.rst | 77 ++++++++++++------- .../getting-started-with-the-codeql-cli.rst | 5 ++ .../about-codeql-queries.rst | 2 +- 4 files changed, 78 insertions(+), 55 deletions(-) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index da87b3d57d5..76fdd296db1 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -56,6 +56,12 @@ You must specify: You can also specify: +- ``--sarif-category``: an identifying category for the results. Used when + you want to upload more than one set of results for a commit. + For example, when you use ``github upload-results`` to send results for more than one + language to the GitHub code scanning API. For more information about this use case, + see `Configuring CodeQL CLI in your CI system `__ in the GitHub documentation. + - .. include:: ../reusables/threads-query-execution.rst @@ -85,8 +91,8 @@ repositories. Running a single query ~~~~~~~~~~~~~~~~~~~~~~ -To run a single query over a JavaScript codebase, you could use the following -command from the directory containing your database:: +To run a single query over a CodeQL database for a JavaScript codebase, +you could use the following command from the directory containing your database:: codeql database analyze ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv @@ -105,13 +111,23 @@ see ":doc:`Using custom queries with the CodeQL CLI `__. The query suites are located at the following paths in + codeql database analyze cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-analysis/cpp-results.sarif + +The analysis generates a file in the v2.1 SARIF format that is supported by all versions of GitHub. +This file can be uploaded to GitHub using ``github upload-results`` or the code scanning API. +For more information, see `Analyzing a CodeQL database `__ +or `Code scanning API `__ in the GitHub documentation. + +CodeQL query suites are ``.qls`` files that use directives to select queries to run +based on certain metadata properties. The standard QL packs have metadata that specify +the location of the code scanning suites, so the CodeQL CLI knows where to find these +suite files automatically, and you don't have to specify the full path on the command line. +For more information, see ":ref:`About QL packs `." + +The standard query suites are stored at the following paths in the CodeQL repository:: ql//ql/src/codeql-suites/-code-scanning.qls @@ -120,23 +136,6 @@ and at the following path in the CodeQL for Go repository:: ql/src/codeql-suites/go-code-scanning.qls -These locations are specified in the metadata included in the standard QL packs. -This means that the CodeQL CLI knows where to find the suite files automatically, and -you don't have to specify the full path on the command line when running an -analysis. For more information, see ":ref:`About QL packs `." - -.. pull-quote:: - - Important - - If you plan to upload the results to GitHub, you must generate SARIF results. - For more information, see `Analyzing a CodeQL database `__ in the GitHub documentation. - -For example, to run the code scanning query suite on a C++ codebase and generate -results in the v2.1 SARIF format supported by all versions of GitHub, you would run:: - - codeql database analyze cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-analysis/cpp-results.sarif - The repository also includes the query suites used by `LGTM.com `__. These are stored alongside the code scanning suites with names of the form: ``-lgtm.qls``. diff --git a/docs/codeql/codeql-cli/creating-codeql-databases.rst b/docs/codeql/codeql-cli/creating-codeql-databases.rst index 637df58555b..2eba1005281 100644 --- a/docs/codeql/codeql-cli/creating-codeql-databases.rst +++ b/docs/codeql/codeql-cli/creating-codeql-databases.rst @@ -17,6 +17,10 @@ Before you generate a CodeQL database, you need to: - Check out the version of your codebase you want to analyze. The directory should be ready to build, with all dependencies already installed. +For information about using the CodeQL CLI in a CI system to create results +to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system `__ +in the GitHub documentation. + Running ``codeql database create`` ---------------------------------- @@ -33,21 +37,30 @@ You must specify: be created when you execute the command---you cannot specify an existing directory. - ``--language``: the identifier for the language to create a database for. + When used with ``--db-cluster``, the option accepts a comma-separated list, + or can be specified more than once. CodeQL supports creating databases for the following languages: .. include:: ../reusables/extractors.rst -Other options may be specified depending on the location of your source file and -the language you want to analyze: +You can specify additional options depending on the location of your source file, +if the code needs to be compiled, and if you want to create CodeQL databases for +more than one language: - ``--source-root``: the root folder for the primary source files used in database creation. By default, the command assumes that the current directory is the source root---use this option to specify a different location. -- ``--command``: for compiled languages only, the build commands that invoke the - compiler. Do not specify ``--command`` options for Python and - JavaScript. Commands will be run from the current folder, or ``--source-root`` +- ``--db-cluster``: use for multi-language codebases when you want to create + databases for more than one language. +- ``--command``: used when you create a database for one or more compiled languages, + omit if the only langauges requested are Python and JavaScript. + This specifies the build commands needed to invoke the compiler. + Commands are run from the current folder, or ``--source-root`` if specified. If you don't include a ``--command``, CodeQL will attempt to detect the build system automatically, using a built-in autobuilder. +- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build + command for languages where the CodeQL CLI does not need to monitor the build + (for example, Python and JavaScript/TypeScript). For full details of all the options you can use when creating databases, see the `database create reference documentation <../manual/database-create>`__. @@ -62,31 +75,13 @@ it failed. For compiled languages, the console will display the output of the build system. When the database is successfully created, you'll find a new directory at the -path specified in the command. This directory contains a number of +path specified in the command. If you used the ``--db-cluster`` option to create +more than one database, a subdirectory is created for each langauge. +Each CodeQL database directory contains a number of subdirectories, including the relational data (required for analysis) and a source archive---a copy of the source files made at the time the database was created---which is used for displaying analysis results. -Obtaining databases from LGTM.com ---------------------------------- - -`LGTM.com `__ analyzes thousands of open-source projects using -CodeQL. For each project on LGTM.com, you can download an archived CodeQL -database corresponding to the most recently analyzed revision of the code. These -databases can also be analyzed using the CodeQL CLI. - -.. include:: ../reusables/download-lgtm-database.rst - -Before running an analysis, unzip the databases and try :doc:`upgrading ` the -unzipped databases to ensure they are compatible with your local copy of the -CodeQL queries and libraries. - -.. pull-quote:: - - Note - - .. include:: ../reusables/index-files-note.rst - Creating databases for non-compiled languages --------------------------------------------- @@ -99,9 +94,12 @@ are available. .. pull-quote:: Important - When running ``database create`` for JavaScript, TypeScript, and Python, you must not - specify a ``--command`` option. If you do, you will override the normal - extractor invocation, which will create an empty database. + When you run ``database create`` for JavaScript, TypeScript, and Python, you should not + specify a ``--command`` option. Otherwise this overrides the normal + extractor invocation, which will create an empty database. If you create + databases for multiple languages and one of them is a compiled language, + use the ``--no-run-unnecessary-builds`` option to skip the command for these + languages. JavaScript and TypeScript ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -229,6 +227,27 @@ commands that you can specify for compiled languages. This command runs a custom script that contains all of the commands required to build the project. +Obtaining databases from LGTM.com +--------------------------------- + +`LGTM.com `__ analyzes thousands of open-source projects using +CodeQL. For each project on LGTM.com, you can download an archived CodeQL +database corresponding to the most recently analyzed revision of the code. These +databases can also be analyzed using the CodeQL CLI or used with the CodeQL +extension for Visual Studio Code. + +.. include:: ../reusables/download-lgtm-database.rst + +Before running an analysis, unzip the databases and try :doc:`upgrading ` the +unzipped databases to ensure they are compatible with your local copy of the +CodeQL queries and libraries. + +.. pull-quote:: + + Note + + .. include:: ../reusables/index-files-note.rst + Further reading --------------- diff --git a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst index 53d08e41d72..c9bcbe5d6c5 100644 --- a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst @@ -22,6 +22,11 @@ follow the steps below. For macOS version 10.15 ("Catalina"), steps 1 and 4 are slightly different---for further details, see the sections labeled **Information for macOS "Catalina" users**. +For information about installing the CodeQL CLI in a CI system to create results +to display in GitHub as code scanning alerts, see +`Installing CodeQL CLI in your CI system `__ +in the GitHub documentation. + 1. Download the CodeQL CLI zip package ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/codeql/writing-codeql-queries/about-codeql-queries.rst b/docs/codeql/writing-codeql-queries/about-codeql-queries.rst index cd36acd2ac8..fc7da3b67bb 100644 --- a/docs/codeql/writing-codeql-queries/about-codeql-queries.rst +++ b/docs/codeql/writing-codeql-queries/about-codeql-queries.rst @@ -13,7 +13,7 @@ CodeQL includes queries to find the most relevant and interesting problems for e - **Alert queries**: queries that highlight issues in specific locations in your code. - **Path queries**: queries that describe the flow of information between a source and a sink in your code. -You can add custom queries to :doc:`QL packs <../codeql-cli/about-ql-packs>` to analyze your projects with "`Code scanning `__", use them to analyze a database with the ":ref:`CodeQL CLI `," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub `__. +You can add custom queries to :doc:`QL packs <../codeql-cli/about-ql-packs>` to analyze your projects with "`Code scanning `__", use them to analyze a database with the ":ref:`CodeQL CLI `," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub `__. This topic is a basic introduction to query files. You can find more information on writing queries for specific programming languages in the ":ref:`CodeQL language guides `," and detailed technical information about QL in the ":ref:`QL language reference `." For more information on how to format your code when contributing queries to the GitHub repository, see the `CodeQL style guide `__.