From b78faa4b717ab9a00039d03988e29fdaefb97824 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 23 Aug 2022 09:35:31 -0700 Subject: [PATCH] Remove upgrade database docs Database upgrades happens implicitly now as part of analysis. There is no need to call this out in separate steps. --- ...nalyzing-databases-with-the-codeql-cli.rst | 11 ---- .../codeql-cli/creating-codeql-databases.rst | 56 +++++++++---------- .../getting-started-with-the-codeql-cli.rst | 7 +-- .../codeql-cli/upgrading-codeql-databases.rst | 55 ------------------ .../codeql-cli/using-the-codeql-cli.rst | 15 ++--- 5 files changed, 33 insertions(+), 111 deletions(-) delete mode 100644 docs/codeql/codeql-cli/upgrading-codeql-databases.rst diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 0931a3476b1..209b8877098 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -80,17 +80,6 @@ You can also specify: - .. include:: ../reusables/threads-query-execution.rst -.. pull-quote:: - - Upgrading databases - - If the CodeQL queries you want to use are newer than the - extractor used to create the database, then you may see a message telling you - that your database needs to be upgraded when you run ``database analyze``. - You can quickly upgrade a database by running the ``database upgrade`` - command. For more information, see ":doc:`Upgrading CodeQL databases - `." - For full details of all the options you can use when analyzing databases, see the `database analyze reference documentation <../manual/database-analyze>`__. diff --git a/docs/codeql/codeql-cli/creating-codeql-databases.rst b/docs/codeql/codeql-cli/creating-codeql-databases.rst index aa60cb24e79..f76b7c811ca 100644 --- a/docs/codeql/codeql-cli/creating-codeql-databases.rst +++ b/docs/codeql/codeql-cli/creating-codeql-databases.rst @@ -7,8 +7,8 @@ Before you analyze your code using CodeQL, you need to create a CodeQL database containing all the data required to run queries on your code. CodeQL analysis relies on extracting relational data from your code, and -using it to build a :ref:`CodeQL database `. CodeQL -databases contain all of the important information about a codebase, which can +using it to build a :ref:`CodeQL database `. CodeQL +databases contain all of the important information about a codebase, which can be analyzed by executing CodeQL queries against it. Before you generate a CodeQL database, you need to: @@ -18,9 +18,9 @@ Before you generate a CodeQL database, you need to: should be ready to build, with all dependencies already installed. For information about using the CodeQL CLI in a third-party CI system to create results -to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system `__ +to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system `__ in the GitHub documentation. For information about enabling CodeQL code scanning using GitHub Actions, -see `Setting up code scanning for a repository `__ +see `Setting up code scanning for a repository `__ in the GitHub documentation. Running ``codeql database create`` @@ -37,38 +37,38 @@ You must specify: - ````: a path to the new database to be created. This directory will be created when you execute the command---you cannot specify an existing - directory. + directory. - ``--language``: the identifier for the language to create a database for. - When used with ``--db-cluster``, the option accepts a comma-separated list, + When used with ``--db-cluster``, the option accepts a comma-separated list, or can be specified more than once. CodeQL supports creating databases for the following languages: .. include:: ../reusables/extractors.rst -You can specify additional options depending on the location of your source file, -if the code needs to be compiled, and if you want to create CodeQL databases for +You can specify additional options depending on the location of your source file, +if the code needs to be compiled, and if you want to create CodeQL databases for more than one language: - ``--source-root``: the root folder for the primary source files used in database creation. By default, the command assumes that the current directory is the source root---use this option to specify a different location. - ``--db-cluster``: use for multi-language codebases when you want to create - databases for more than one language. + databases for more than one language. - ``--command``: used when you create a database for one or more compiled languages, - omit if the only languages requested are Python and JavaScript. - This specifies the build commands needed to invoke the compiler. + omit if the only languages requested are Python and JavaScript. + This specifies the build commands needed to invoke the compiler. Commands are run from the current folder, or ``--source-root`` if specified. If you don't include a ``--command``, CodeQL will attempt to - detect the build system automatically, using a built-in autobuilder. -- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build - command for languages where the CodeQL CLI does not need to monitor the build + detect the build system automatically, using a built-in autobuilder. +- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build + command for languages where the CodeQL CLI does not need to monitor the build (for example, Python and JavaScript/TypeScript). You can specify extractor options to customize the behavior of extractors that create CodeQL databases. For more information, see ":doc:`Extractor options `." For full details of all the options you can use when creating databases, -see the `database create reference documentation <../manual/database-create>`__. +see the `database create reference documentation <../manual/database-create>`__. Progress and results -------------------- @@ -117,7 +117,7 @@ extract both JavaScript and TypeScript files:: Here, we have specified a ``--source-root`` path, which is the location where database creation is executed, but is not necessarily the checkout root of the -codebase. +codebase. By default, files in ``node_modules`` and ``bower_components`` directories are not extracted. @@ -127,7 +127,7 @@ Python When creating databases for Python you must ensure: - You have the all of the required versions of Python installed. -- You have access to the `pip `__ +- You have access to the `pip `__ packaging management system and can install any packages that the codebase depends on. - You have installed the `virtualenv `__ pip module. @@ -143,14 +143,14 @@ generating a new Python database at ``/python-database``. Ruby ~~~~ -Creating databases for Ruby requires no additional dependencies. +Creating databases for Ruby requires no additional dependencies. In the command line you must specify ``--language=ruby``. For example:: codeql database create --language=ruby --source-root /ruby-database Here, we have specified a ``--source-root`` path, which is the location where database creation is executed, but is not necessarily the checkout root of the -codebase. +codebase. Creating databases for compiled languages ----------------------------------------- @@ -179,11 +179,11 @@ build steps, you may need to explicitly define each step in the command line. .. pull-quote:: Creating databases for Go - + For Go, install the Go toolchain (version 1.11 or later) and, if there are dependencies, the appropriate dependency manager (such as `dep `__). - + The Go autobuilder attempts to automatically detect code written in Go in a repository, and only runs build scripts in an attempt to fetch dependencies. To force CodeQL to limit extraction to the files compiled by your build script, set the environment variable @@ -194,7 +194,7 @@ Specifying build commands ~~~~~~~~~~~~~~~~~~~~~~~~~ The following examples are designed to give you an idea of some of the build -commands that you can specify for compiled languages. +commands that you can specify for compiled languages. .. pull-quote:: Important @@ -214,7 +214,7 @@ commands that you can specify for compiled languages. in the build command. It is also a good idea to add `/t:rebuild` to ensure that all code will be built (code that is not built will not be included in the CodeQL database): - codeql database create csharp-database --language=csharp --command='dotnet build /p:UseSharedCompilation=false /t:rebuild' + codeql database create csharp-database --language=csharp --command='dotnet build /p:UseSharedCompilation=false /t:rebuild' - Go project built using the ``CODEQL_EXTRACTOR_GO_BUILD_TRACING=on`` environment variable:: @@ -260,7 +260,7 @@ commands that you can specify for compiled languages. - Project built using a custom build script:: codeql database create new-database --language= --command='./scripts/build.sh' - + This command runs a custom script that contains all of the commands required to build the project. @@ -279,7 +279,7 @@ You must specify: - ````: a path to the new database to be created. This directory will be created when you execute the command---you cannot specify an existing - directory. + directory. - ``--begin-tracing``: creates scripts that can be used to set up an environment in which build commands will be traced. You may specify other options for the ``codeql database init`` command as normal. @@ -388,14 +388,10 @@ Obtaining databases from LGTM.com CodeQL. For each project on LGTM.com, you can download an archived CodeQL database corresponding to the most recently analyzed revision of the code. These databases can also be analyzed using the CodeQL CLI or used with the CodeQL -extension for Visual Studio Code. +extension for Visual Studio Code. .. include:: ../reusables/download-lgtm-database.rst -Before running an analysis, unzip the databases and try :doc:`upgrading ` the -unzipped databases to ensure they are compatible with your local copy of the -CodeQL queries and libraries. - .. pull-quote:: Note diff --git a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst index 75fc118adeb..0fd8e6b8fa0 100644 --- a/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst @@ -230,10 +230,7 @@ see ":doc:`About CodeQL packs `." may need to be upgraded before you can analyze them. - For the most up to date CodeQL queries, check out the ``main`` branch. - This branch represents the very latest version of CodeQL's analysis. Even - databases created using the most recent version of the CLI may have to be - upgraded before you can analyze them. For more information, see - ":doc:`Upgrading CodeQL databases `." + This branch represents the very latest version of CodeQL's analysis. - For the queries used in a particular LGTM Enterprise release, check out the branch tagged with the relevant release number. For example, the branch @@ -243,7 +240,7 @@ see ":doc:`About CodeQL packs `." `__ in the LGTM admin help. -4. Extract the zip archive +1. Extract the zip archive ~~~~~~~~~~~~~~~~~~~~~~~~~~ For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) diff --git a/docs/codeql/codeql-cli/upgrading-codeql-databases.rst b/docs/codeql/codeql-cli/upgrading-codeql-databases.rst deleted file mode 100644 index 334a7501392..00000000000 --- a/docs/codeql/codeql-cli/upgrading-codeql-databases.rst +++ /dev/null @@ -1,55 +0,0 @@ -.. _upgrading-codeql-databases: - -Upgrading CodeQL databases -========================== - -As the CodeQL CLI tools and queries evolve, you may find that some of your -CodeQL databases become out of date. You must upgrade out-of-date databases -before you can analyze them. - -Databases become out of date when: - -- For databases created using the CodeQL CLI, the version of CLI tools used to - create them is older than your copy of the CodeQL queries. -- For databases downloaded from LGTM.com, the CodeQL tools used by LGTM.com to create - that revision of the code are older than your copy of the CodeQL queries. - -If you have a local checkout of the ``github/codeql`` repository, please note that -the ``main`` branch of the CodeQL queries is updated more often than both the -CLI and LGTM.com, so databases are most likely to become out of date if you use -the queries on this branch. For more information about the different versions of -the CodeQL queries, see ":ref:`Getting started with the CodeQL CLI `." - -Out-of-date databases must be upgraded before they can be analyzed. This topic -shows you how to upgrade a CodeQL database using the ``database upgrade`` -subcommand. - -Prerequisites -------------- - -Archived databases downloaded from LGTM.com must be unzipped before they are -upgraded. - -Running ``codeql database upgrade`` ------------------------------------ - -CodeQL databases are upgraded by running the following command:: - - codeql database upgrade - -where ````, the path to the CodeQL database you -want to upgrade, must be specified. - -For full details of all the options you can use when upgrading databases, -see the "`database upgrade <../manual/database-upgrade>`__" reference documentation. - -Progress and results --------------------- - -When you execute the ``database upgrade`` command, CodeQL identifies the version -of the :ref:`schema ` associated with the database. From -there, it works out what (if anything) is required to make the database work -with your queries and libraries. It will rewrite the database, if necessary, or -make no changes if the database is already compatible (or if it finds no -information about how to perform an upgrade). Once a database has been upgraded -it cannot be downgraded for use with older versions of the CodeQL products. diff --git a/docs/codeql/codeql-cli/using-the-codeql-cli.rst b/docs/codeql/codeql-cli/using-the-codeql-cli.rst index 7b67af59c6a..4d5ea6be7a0 100644 --- a/docs/codeql/codeql-cli/using-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/using-the-codeql-cli.rst @@ -7,7 +7,7 @@ Using the CodeQL CLI See the following links to learn how to get set up and run CodeQL commands: -- :doc:`About the CodeQL CLI `: Software +- :doc:`About the CodeQL CLI `: Software developers and security researchers can secure their code using the CodeQL CLI. - :doc:`Getting started with the CodeQL CLI @@ -17,19 +17,15 @@ See the following links to learn how to get set up and run CodeQL commands: - :doc:`Creating CodeQL databases `: Create relational representations of source code that can be queried like any other database. - + - :doc:`Extractor options - `: Set options for the + `: Set options for the behavior of extractors that create CodeQL databases. - :doc:`Analyzing CodeQL databases with the CodeQL CLI `: Analyze your code using queries written in a specially-designed, object-oriented query language. -- :doc:`Upgrading CodeQL databases - `: Upgrade your databases so - that they can be analyzed using the most up to date CodeQL products. - - :doc:`Using custom queries with the CodeQL CLI `: Use custom queries to extend your analysis or highlight errors that are specific to a particular codebase. @@ -42,7 +38,7 @@ See the following links to learn how to get set up and run CodeQL commands: your analysis. - :doc:`Testing query help files `: - Test query help files by rendering them as markdown to ensure they are valid + Test query help files by rendering them as markdown to ensure they are valid before adding them to the CodeQL repository or using them in code scanning. - :doc:`Creating and working with CodeQL packs `: @@ -51,7 +47,7 @@ See the following links to learn how to get set up and run CodeQL commands: - :doc:`Publishing and using CodeQL packs `: Publish your own or use others CodeQL packs for code scanning. -- :doc:`Specifying command options in a CodeQL configuration file `: +- :doc:`Specifying command options in a CodeQL configuration file `: You can save default or frequently used options for your commands in a per-user configuration file. .. toctree:: @@ -63,7 +59,6 @@ See the following links to learn how to get set up and run CodeQL commands: creating-codeql-databases extractor-options analyzing-databases-with-the-codeql-cli - upgrading-codeql-databases using-custom-queries-with-the-codeql-cli creating-codeql-query-suites testing-custom-queries