mirror of
https://github.com/github/codeql.git
synced 2026-02-01 07:42:57 +01:00
Remove upgrade database docs
Database upgrades happens implicitly now as part of analysis. There is no need to call this out in separate steps.
This commit is contained in:
Andrew Eisenberg
@@ -80,17 +80,6 @@ You can also specify:
|
||||
- .. include:: ../reusables/threads-query-execution.rst
|
||||
|
||||
|
||||
.. pull-quote::
|
||||
|
||||
Upgrading databases
|
||||
|
||||
If the CodeQL queries you want to use are newer than the
|
||||
extractor used to create the database, then you may see a message telling you
|
||||
that your database needs to be upgraded when you run ``database analyze``.
|
||||
You can quickly upgrade a database by running the ``database upgrade``
|
||||
command. For more information, see ":doc:`Upgrading CodeQL databases
|
||||
<upgrading-codeql-databases>`."
|
||||
|
||||
For full details of all the options you can use when analyzing databases, see
|
||||
the `database analyze reference documentation <../manual/database-analyze>`__.
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ Before you analyze your code using CodeQL, you need to create a CodeQL
|
||||
database containing all the data required to run queries on your code.
|
||||
|
||||
CodeQL analysis relies on extracting relational data from your code, and
|
||||
using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL
|
||||
databases contain all of the important information about a codebase, which can
|
||||
using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL
|
||||
databases contain all of the important information about a codebase, which can
|
||||
be analyzed by executing CodeQL queries against it.
|
||||
Before you generate a CodeQL database, you need to:
|
||||
|
||||
@@ -18,9 +18,9 @@ Before you generate a CodeQL database, you need to:
|
||||
should be ready to build, with all dependencies already installed.
|
||||
|
||||
For information about using the CodeQL CLI in a third-party CI system to create results
|
||||
to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__
|
||||
to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__
|
||||
in the GitHub documentation. For information about enabling CodeQL code scanning using GitHub Actions,
|
||||
see `Setting up code scanning for a repository <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository>`__
|
||||
see `Setting up code scanning for a repository <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository>`__
|
||||
in the GitHub documentation.
|
||||
|
||||
Running ``codeql database create``
|
||||
@@ -37,38 +37,38 @@ You must specify:
|
||||
|
||||
- ``<database>``: a path to the new database to be created. This directory will
|
||||
be created when you execute the command---you cannot specify an existing
|
||||
directory.
|
||||
directory.
|
||||
- ``--language``: the identifier for the language to create a database for.
|
||||
When used with ``--db-cluster``, the option accepts a comma-separated list,
|
||||
When used with ``--db-cluster``, the option accepts a comma-separated list,
|
||||
or can be specified more than once.
|
||||
CodeQL supports creating databases for the following languages:
|
||||
|
||||
.. include:: ../reusables/extractors.rst
|
||||
|
||||
You can specify additional options depending on the location of your source file,
|
||||
if the code needs to be compiled, and if you want to create CodeQL databases for
|
||||
You can specify additional options depending on the location of your source file,
|
||||
if the code needs to be compiled, and if you want to create CodeQL databases for
|
||||
more than one language:
|
||||
|
||||
- ``--source-root``: the root folder for the primary source files used in
|
||||
database creation. By default, the command assumes that the current
|
||||
directory is the source root---use this option to specify a different location.
|
||||
- ``--db-cluster``: use for multi-language codebases when you want to create
|
||||
databases for more than one language.
|
||||
databases for more than one language.
|
||||
- ``--command``: used when you create a database for one or more compiled languages,
|
||||
omit if the only languages requested are Python and JavaScript.
|
||||
This specifies the build commands needed to invoke the compiler.
|
||||
omit if the only languages requested are Python and JavaScript.
|
||||
This specifies the build commands needed to invoke the compiler.
|
||||
Commands are run from the current folder, or ``--source-root``
|
||||
if specified. If you don't include a ``--command``, CodeQL will attempt to
|
||||
detect the build system automatically, using a built-in autobuilder.
|
||||
- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build
|
||||
command for languages where the CodeQL CLI does not need to monitor the build
|
||||
detect the build system automatically, using a built-in autobuilder.
|
||||
- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build
|
||||
command for languages where the CodeQL CLI does not need to monitor the build
|
||||
(for example, Python and JavaScript/TypeScript).
|
||||
|
||||
You can specify extractor options to customize the behavior of extractors that create CodeQL databases. For more information, see
|
||||
":doc:`Extractor options <extractor-options>`."
|
||||
|
||||
For full details of all the options you can use when creating databases,
|
||||
see the `database create reference documentation <../manual/database-create>`__.
|
||||
see the `database create reference documentation <../manual/database-create>`__.
|
||||
|
||||
Progress and results
|
||||
--------------------
|
||||
@@ -117,7 +117,7 @@ extract both JavaScript and TypeScript files::
|
||||
|
||||
Here, we have specified a ``--source-root`` path, which is the location where
|
||||
database creation is executed, but is not necessarily the checkout root of the
|
||||
codebase.
|
||||
codebase.
|
||||
|
||||
By default, files in ``node_modules`` and ``bower_components`` directories are not extracted.
|
||||
|
||||
@@ -127,7 +127,7 @@ Python
|
||||
When creating databases for Python you must ensure:
|
||||
|
||||
- You have the all of the required versions of Python installed.
|
||||
- You have access to the `pip <https://pypi.org/project/pip/>`__
|
||||
- You have access to the `pip <https://pypi.org/project/pip/>`__
|
||||
packaging management system and can install any
|
||||
packages that the codebase depends on.
|
||||
- You have installed the `virtualenv <https://pypi.org/project/virtualenv/>`__ pip module.
|
||||
@@ -143,14 +143,14 @@ generating a new Python database at ``<output-folder>/python-database``.
|
||||
Ruby
|
||||
~~~~
|
||||
|
||||
Creating databases for Ruby requires no additional dependencies.
|
||||
Creating databases for Ruby requires no additional dependencies.
|
||||
In the command line you must specify ``--language=ruby``. For example::
|
||||
|
||||
codeql database create --language=ruby --source-root <folder-to-extract> <output-folder>/ruby-database
|
||||
|
||||
Here, we have specified a ``--source-root`` path, which is the location where
|
||||
database creation is executed, but is not necessarily the checkout root of the
|
||||
codebase.
|
||||
codebase.
|
||||
|
||||
Creating databases for compiled languages
|
||||
-----------------------------------------
|
||||
@@ -179,11 +179,11 @@ build steps, you may need to explicitly define each step in the command line.
|
||||
|
||||
|
||||
.. pull-quote:: Creating databases for Go
|
||||
|
||||
|
||||
For Go, install the Go toolchain (version 1.11 or later) and, if there
|
||||
are dependencies, the appropriate dependency manager (such as `dep
|
||||
<https://golang.github.io/dep/>`__).
|
||||
|
||||
|
||||
The Go autobuilder attempts to automatically detect code written in Go in a repository,
|
||||
and only runs build scripts in an attempt to fetch dependencies. To force
|
||||
CodeQL to limit extraction to the files compiled by your build script, set the environment variable
|
||||
@@ -194,7 +194,7 @@ Specifying build commands
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following examples are designed to give you an idea of some of the build
|
||||
commands that you can specify for compiled languages.
|
||||
commands that you can specify for compiled languages.
|
||||
|
||||
.. pull-quote:: Important
|
||||
|
||||
@@ -214,7 +214,7 @@ commands that you can specify for compiled languages.
|
||||
in the build command. It is also a good idea to add `/t:rebuild` to ensure that all code will be built (code
|
||||
that is not built will not be included in the CodeQL database):
|
||||
|
||||
codeql database create csharp-database --language=csharp --command='dotnet build /p:UseSharedCompilation=false /t:rebuild'
|
||||
codeql database create csharp-database --language=csharp --command='dotnet build /p:UseSharedCompilation=false /t:rebuild'
|
||||
|
||||
- Go project built using the ``CODEQL_EXTRACTOR_GO_BUILD_TRACING=on`` environment variable::
|
||||
|
||||
@@ -260,7 +260,7 @@ commands that you can specify for compiled languages.
|
||||
- Project built using a custom build script::
|
||||
|
||||
codeql database create new-database --language=<language> --command='./scripts/build.sh'
|
||||
|
||||
|
||||
This command runs a custom script that contains all of the commands required
|
||||
to build the project.
|
||||
|
||||
@@ -279,7 +279,7 @@ You must specify:
|
||||
|
||||
- ``<database>``: a path to the new database to be created. This directory will
|
||||
be created when you execute the command---you cannot specify an existing
|
||||
directory.
|
||||
directory.
|
||||
- ``--begin-tracing``: creates scripts that can be used to set up an environment in which build commands will be traced.
|
||||
|
||||
You may specify other options for the ``codeql database init`` command as normal.
|
||||
@@ -388,14 +388,10 @@ Obtaining databases from LGTM.com
|
||||
CodeQL. For each project on LGTM.com, you can download an archived CodeQL
|
||||
database corresponding to the most recently analyzed revision of the code. These
|
||||
databases can also be analyzed using the CodeQL CLI or used with the CodeQL
|
||||
extension for Visual Studio Code.
|
||||
extension for Visual Studio Code.
|
||||
|
||||
.. include:: ../reusables/download-lgtm-database.rst
|
||||
|
||||
Before running an analysis, unzip the databases and try :doc:`upgrading <upgrading-codeql-databases>` the
|
||||
unzipped databases to ensure they are compatible with your local copy of the
|
||||
CodeQL queries and libraries.
|
||||
|
||||
.. pull-quote::
|
||||
|
||||
Note
|
||||
|
||||
@@ -230,10 +230,7 @@ see ":doc:`About CodeQL packs <about-codeql-packs>`."
|
||||
may need to be upgraded before you can analyze them.
|
||||
|
||||
- For the most up to date CodeQL queries, check out the ``main`` branch.
|
||||
This branch represents the very latest version of CodeQL's analysis. Even
|
||||
databases created using the most recent version of the CLI may have to be
|
||||
upgraded before you can analyze them. For more information, see
|
||||
":doc:`Upgrading CodeQL databases <upgrading-codeql-databases>`."
|
||||
This branch represents the very latest version of CodeQL's analysis.
|
||||
|
||||
- For the queries used in a particular LGTM Enterprise release, check out the
|
||||
branch tagged with the relevant release number. For example, the branch
|
||||
@@ -243,7 +240,7 @@ see ":doc:`About CodeQL packs <about-codeql-packs>`."
|
||||
<https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html>`__
|
||||
in the LGTM admin help.
|
||||
|
||||
4. Extract the zip archive
|
||||
1. Extract the zip archive
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier)
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
.. _upgrading-codeql-databases:
|
||||
|
||||
Upgrading CodeQL databases
|
||||
==========================
|
||||
|
||||
As the CodeQL CLI tools and queries evolve, you may find that some of your
|
||||
CodeQL databases become out of date. You must upgrade out-of-date databases
|
||||
before you can analyze them.
|
||||
|
||||
Databases become out of date when:
|
||||
|
||||
- For databases created using the CodeQL CLI, the version of CLI tools used to
|
||||
create them is older than your copy of the CodeQL queries.
|
||||
- For databases downloaded from LGTM.com, the CodeQL tools used by LGTM.com to create
|
||||
that revision of the code are older than your copy of the CodeQL queries.
|
||||
|
||||
If you have a local checkout of the ``github/codeql`` repository, please note that
|
||||
the ``main`` branch of the CodeQL queries is updated more often than both the
|
||||
CLI and LGTM.com, so databases are most likely to become out of date if you use
|
||||
the queries on this branch. For more information about the different versions of
|
||||
the CodeQL queries, see ":ref:`Getting started with the CodeQL CLI <local-copy-codeql-queries>`."
|
||||
|
||||
Out-of-date databases must be upgraded before they can be analyzed. This topic
|
||||
shows you how to upgrade a CodeQL database using the ``database upgrade``
|
||||
subcommand.
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
Archived databases downloaded from LGTM.com must be unzipped before they are
|
||||
upgraded.
|
||||
|
||||
Running ``codeql database upgrade``
|
||||
-----------------------------------
|
||||
|
||||
CodeQL databases are upgraded by running the following command::
|
||||
|
||||
codeql database upgrade <database>
|
||||
|
||||
where ``<database>``, the path to the CodeQL database you
|
||||
want to upgrade, must be specified.
|
||||
|
||||
For full details of all the options you can use when upgrading databases,
|
||||
see the "`database upgrade <../manual/database-upgrade>`__" reference documentation.
|
||||
|
||||
Progress and results
|
||||
--------------------
|
||||
|
||||
When you execute the ``database upgrade`` command, CodeQL identifies the version
|
||||
of the :ref:`schema <codeql-database-schema>` associated with the database. From
|
||||
there, it works out what (if anything) is required to make the database work
|
||||
with your queries and libraries. It will rewrite the database, if necessary, or
|
||||
make no changes if the database is already compatible (or if it finds no
|
||||
information about how to perform an upgrade). Once a database has been upgraded
|
||||
it cannot be downgraded for use with older versions of the CodeQL products.
|
||||
@@ -7,7 +7,7 @@ Using the CodeQL CLI
|
||||
|
||||
See the following links to learn how to get set up and run CodeQL commands:
|
||||
|
||||
- :doc:`About the CodeQL CLI <about-the-codeql-cli>`: Software
|
||||
- :doc:`About the CodeQL CLI <about-the-codeql-cli>`: Software
|
||||
developers and security researchers can secure their code using the CodeQL CLI.
|
||||
|
||||
- :doc:`Getting started with the CodeQL CLI
|
||||
@@ -17,19 +17,15 @@ See the following links to learn how to get set up and run CodeQL commands:
|
||||
- :doc:`Creating CodeQL databases
|
||||
<creating-codeql-databases>`: Create relational
|
||||
representations of source code that can be queried like any other database.
|
||||
|
||||
|
||||
- :doc:`Extractor options
|
||||
<extractor-options>`: Set options for the
|
||||
<extractor-options>`: Set options for the
|
||||
behavior of extractors that create CodeQL databases.
|
||||
|
||||
- :doc:`Analyzing CodeQL databases with the CodeQL CLI
|
||||
<analyzing-databases-with-the-codeql-cli>`: Analyze your code using queries
|
||||
written in a specially-designed, object-oriented query language.
|
||||
|
||||
- :doc:`Upgrading CodeQL databases
|
||||
<upgrading-codeql-databases>`: Upgrade your databases so
|
||||
that they can be analyzed using the most up to date CodeQL products.
|
||||
|
||||
- :doc:`Using custom queries with the CodeQL CLI
|
||||
<using-custom-queries-with-the-codeql-cli>`: Use custom queries to extend your
|
||||
analysis or highlight errors that are specific to a particular codebase.
|
||||
@@ -42,7 +38,7 @@ See the following links to learn how to get set up and run CodeQL commands:
|
||||
your analysis.
|
||||
|
||||
- :doc:`Testing query help files <testing-query-help-files>`:
|
||||
Test query help files by rendering them as markdown to ensure they are valid
|
||||
Test query help files by rendering them as markdown to ensure they are valid
|
||||
before adding them to the CodeQL repository or using them in code scanning.
|
||||
|
||||
- :doc:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`:
|
||||
@@ -51,7 +47,7 @@ See the following links to learn how to get set up and run CodeQL commands:
|
||||
- :doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`:
|
||||
Publish your own or use others CodeQL packs for code scanning.
|
||||
|
||||
- :doc:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`:
|
||||
- :doc:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`:
|
||||
You can save default or frequently used options for your commands in a per-user configuration file.
|
||||
|
||||
.. toctree::
|
||||
@@ -63,7 +59,6 @@ See the following links to learn how to get set up and run CodeQL commands:
|
||||
creating-codeql-databases
|
||||
extractor-options
|
||||
analyzing-databases-with-the-codeql-cli
|
||||
upgrading-codeql-databases
|
||||
using-custom-queries-with-the-codeql-cli
|
||||
creating-codeql-query-suites
|
||||
testing-custom-queries
|
||||
|
||||
Reference in New Issue
Block a user