address review comments

Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>

.github/workflows/generate-query-help-docs.yml

@@ -0,0 +1,59 @@
+name: Generate CodeQL query help documentation using Sphinx
+
+on:
+  workflow_dispatch:
+      inputs:
+        description:
+          description: A description of the purpose of this job. For human consumption.
+          required: false
+  push:
+    branches:
+     - 'lgtm.com'
+  pull_request:
+    paths:
+      - 'docs/codeql/query-help/**'
+      
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Clone github/codeql
+      uses: actions/checkout@v2
+      with:
+        path: codeql
+    - name: Clone github/codeql-go
+      uses: actions/checkout@v2
+      with:
+        repository: 'github/codeql-go'
+        path: codeql-go
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+    - name: Download CodeQL CLI
+      uses: dsaltares/fetch-gh-release-asset@aa37ae5c44d3c9820bc12fe675e8670ecd93bd1c
+      with:
+        repo: "github/codeql-cli-binaries"
+        version: "latest"
+        file: "codeql-linux64.zip"
+        token: ${{ secrets.GITHUB_TOKEN }}
+    - name: Unzip CodeQL CLI
+      run: unzip -d codeql-cli codeql-linux64.zip
+    - name: Set up query help docs folder  
+      run: |
+        cp -r codeql/docs/codeql/** .
+    - name: Query help to markdown
+      run: |
+        PATH="$PATH:codeql-cli/codeql" python codeql/docs/codeql/query-help-markdown.py
+    - name: Run Sphinx for query help
+      uses: ammaraskar/sphinx-action@master
+      with:
+        docs-folder: "query-help/"
+        pre-build-command: "python -m pip install --upgrade recommonmark"
+        build-command: "sphinx-build -b dirhtml . _build" 
+    - name: Upload HTML artifacts
+      uses: actions/upload-artifact@v2
+      with:
+        name: query-help-html
+        path: query-help/_build
+    

docs/codeql/README.rst

@@ -5,7 +5,7 @@ Overview
 ********
 
 The CodeQL documentation in this repository is written in reStructuredText and converted to
-HTML for manual publication on `help.semmle.com <https://help.semmle.com>`__ using Sphinx. 
+HTML using Sphinx. 
 
 For more information on writing in reStructuredText, 
 see http://docutils.sourceforge.net/rst.html.
@@ -15,25 +15,21 @@ For more information on Sphinx, see https://www.sphinx-doc.org.
 Project structure
 *****************
 
-The documentation currently consists of the following Sphinx projects:
-
-- ``learn-ql``–help topics to help you learn CodeQL and write queries
-- ``ql-language-reference``–an overview of important concepts in QL, the language that underlies CodeQL analysis
-- ``support``–the languages and frameworks currently supported in CodeQL analysis
-- ``ql-training``–source files for the CodeQL training and variant analysis examples slide decks
-
-Each project contains:
+The project contains:
 
 - an ``index.html`` file, the project's 
   `master document <https://www.sphinx-doc.org/en/master/glossary.html#term-master-document>`__.
 - a ``conf.py`` file that defines some project-specific configuration values
-- one or more reStructuredText source files
+- the reStructuredText source files
 
-Shared configuration values are specified in ``global-conf.py``, which is found 
-in the ``global-sphinx-files`` directory.
-This directory also contains any other files, such as templates and stylesheets, 
-that are used by multiple projects.
-Images used in the documentation are located in the ``images`` directory.
+The documentation consists of the following categories:
+
+- CodeQL overview
+- Writing CodeQL queries
+- CodeQL language guides
+- QL language reference
+- CodeQL CLI
+- CodeQL for Visual Studio Code
 
 The ``ql-training`` project contains the source files, themes, and static files 
 used to generate the CodeQL training and variant analysis presentations. 
@@ -110,5 +106,5 @@ Viewing the current version of the CodeQL documentation
 
 The documentation for the most recent release is 
 published to `help.semmle.com <https://help.semmle.com>`__. 
-There, you can also find the documentation for the CodeQL CLI,
-the CodeQL extension for Visual Studio Code, and LGTM Enterprise. 
+There, you can find the documentation for the CodeQL queries,
+the CodeQL standard libraries, and LGTM Enterprise. 

docs/codeql/_templates/layout.html

@@ -97,7 +97,7 @@
 <main class="bg-gray-light clearfix">
 <nav class="SideNav position-sticky top-0 col-lg-3 col-md-3 float-left p-4 hide-sm hide-md overflow-y-auto">
 
-    {{ toctree(includehidden=true, maxdepth=2, collapse=true) }}
+    {{ toctree(includehidden=true, maxdepth=3, collapse=true) }}
 
 </nav>
 

docs/codeql/codeql-cli/codeql-cli-reference/about-ql-packs.rst

@@ -0,0 +1,231 @@
+.. _about-ql-packs:
+
+About QL packs
+==============
+
+QL packs are used to organize the files used in CodeQL analysis. They
+contain queries, library files, query suites, and important metadata. 
+
+The `CodeQL repository <https://github.com/github/codeql>`__ contains QL packs for
+C/C++, C#, Java, JavaScript, and Python. The `CodeQL for Go
+<https://github.com/github/codeql-go/>`__ repository contains a QL pack for Go
+analysis. You can also make custom QL packs to contain your own queries and
+libraries.
+
+QL pack structure
+-----------------
+
+A QL pack must contain a file called ``qlpack.yml`` in its root directory. The other 
+files and directories within the pack should be logically organized. For example, typically:  
+
+- Queries are organized into directories for specific categories.
+- Queries for specific products, libraries, and frameworks are organized into 
+  their own top-level directories.
+- There is a top-level directory named ``<owner>/<ql-language-specification>`` for query library 
+  (``.qll``) files. Within this directory, ``.qll`` files should be organized into 
+  subdirectories for specific categories. 
+
+About ``qlpack.yml`` files
+--------------------------
+
+When executing commands, CodeQL scans siblings of the installation directory (and
+their subdirectories) for ``qlpack.yml`` files. The metadata in the file tells
+CodeQL how to compile queries, what libraries the pack depends on, and where to
+find query suite definitions. 
+
+The content of the QL pack (queries and libraries used in CodeQL analysis) is 
+included in the same directory as ``qlpack.yml``, or its subdirectories.
+
+The location of ``qlpack.yml`` defines the library path for the content 
+of the QL pack. That is, for all ``.ql`` and ``.qll`` files in the QL pack, 
+CodeQL will resolve all import statements relative to the ``qlpack.yml`` at the 
+pack's root.
+
+For example, in a QL pack with the following contents, you can import ``CustomSinks.qll``
+from any location in the pack by declaring ``import mycompany.java.CustomSinks``. 
+
+.. code-block:: none
+
+   qlpack.yml
+   mycompany/
+     java/
+       security/
+         CustomSinks.qll
+   Security/
+   CustomQuery.ql
+
+For more information, see ":ref:`Importing modules <importing-modules>`" 
+in the QL language reference.
+
+``qlpack.yml`` properties
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following properties are supported in ``qlpack.yml`` files.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Property
+     - Example
+     - Required
+     - Purpose
+   * - ``name``
+     - ``org-queries``
+     - All packs
+     - The name of the QL pack defined using alphanumeric characters, hyphens, and periods. It must be unique as CodeQL cannot differentiate between QL packs with identical names. If you intend to    distribute the pack, prefix the name with your (or your organization's) name followed by a hyphen. Use the pack name to specify queries to run using ``database analyze`` and to define    dependencies between QL packs (see examples below).- '
+   * - ``version``
+     - ``0.0.0``
+     - All packs
+     - A version number for this QL pack. This field is not currently used by any commands, but may be required by future releases of CodeQL.
+   * - ``libraryPathDependencies``
+     - ``codeql-javascript``
+     - Optional
+     - The names of any QL packs that this QL pack depends on, as a sequence. This gives the pack access to any libraries, database schema, and query suites defined in the dependency.
+   * - ``suites``
+     - ``suites``
+     - Optional
+     - The path to a directory that contains the "well-known" query suites in the pack, defined relative to the pack directory. You can run "well-known" suites stored in this directory by specifying the pack name, without providing their full path. To use query suites stored in other directories in the pack, you must provide their full path. For more information about query suites, see ":doc:`Creating CodeQL query suites <../using-the-codeql-cli/creating-codeql-query-suites>`."
+   * - ``extractor``
+     - ``javascript``
+     - All test packs
+     - The CodeQL language extractor to use when the CLI creates a database from test files in the pack. For more information about testing queries, see ":doc:`Testing custom queries <../using-the-codeql-cli/testing-custom-queries>`."
+   * - ``tests``
+     - ``.``
+     - Optional for test packs
+     - Supported from release 2.1.0 onwards. The path to a directory within the pack that contains tests, defined relative to the pack directory. Use ``.`` to specify the whole pack. Any queries in this directory are run as tests when ``test run`` is run with the ``--strict-test-discovery`` option. These queries are ignored by query suite definitions that use ``queries`` or ``qlpack``    instructions to ask for all queries in a particular pack.
+   * - ``dbscheme``
+     - ``semmlecode.python.dbscheme``
+     - Core language pack only
+     - The path to the :ref:`database schema <codeql-database-schema>` for all libraries and queries written for this CodeQL language (see example below).
+   * - ``upgrades``
+     - ``.``
+     - Packs with upgrades
+     - The path to a directory within the pack that contains upgrade scripts, defined relative to the pack directory. The ``database upgrade`` action uses these scripts to update databases that were created by an older version of an extractor so they're compatible with the current extractor (see `Upgrade scripts for a language <upgrade-scripts-for-a-language>`__ below.)
+
+
+.. _custom-ql-packs:
+
+Examples of custom QL packs
+---------------------------
+
+When you write custom queries or tests, you should save them in
+custom QL packs. For simplicity, try to organize each pack logically. For more
+information, see `QL pack structure <#ql-pack-structure>`__. Save files for queries 
+and tests in separate packs and, where possible, organize custom packs into specific 
+folders for each target language. 
+
+QL packs for custom queries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A custom QL pack for queries must include a ``qlpack.yml`` file at
+the pack root, containing ``name``, ``version``,
+and ``libraryPathDependencies`` properties. If the pack contains query suites, you can
+use the ``suites`` property to define their location. Query suites defined 
+here are called "well-known" suites, and can be used on the command line by referring to 
+their name only, rather than their full path.
+For more information about query suites, see ":doc:`Creating CodeQL query suites <../using-the-codeql-cli/creating-codeql-query-suites>`."
+
+For example, a ``qlpack.yml`` file for a QL pack featuring custom C++ queries
+and libraries may contain:
+
+.. code-block:: yaml
+
+   name: my-custom-queries
+   version: 0.0.0
+   libraryPathDependencies: codeql-cpp
+   suites: my-custom-suites
+
+where ``codeql-cpp`` is the name of the QL pack for C/C++ analysis included in
+the CodeQL repository. 
+
+.. pull-quote::
+
+   Note
+
+   When you create a custom QL pack, it's usually a good idea to add it to the search path in your CodeQL configuration.
+   This will ensure that any libraries the pack contains are available to the CodeQL CLI.
+   For more information, see ":ref:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`."
+
+QL packs for custom test files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For custom QL packs containing test files, you also need to include an
+``extractor`` property so that the ``test run`` command knows how to create test
+databases. You may also wish to specify the ``tests`` property.
+
+.. include:: ../../reusables/test-qlpack.rst
+
+For more information about running tests, see ":doc:`Testing custom queries
+<../using-the-codeql-cli/testing-custom-queries>`."
+
+.. _standard-ql-packs:
+
+Examples of QL packs in the CodeQL repository
+---------------------------------------------
+
+Each of the languages in the CodeQL repository has three main QL packs:
+
+- Core QL pack for the language, with the :ref:`database schema <codeql-database-schema>`
+  used by the language, CodeQL libraries, and queries at ``ql/<language>/ql/src``
+- Tests for the core language libraries and queries pack at ``ql/<language>/ql/test``
+- Upgrade scripts for the language at ``ql/<language>/upgrades``
+
+Core QL pack
+~~~~~~~~~~~~
+
+The ``qlpack.yml`` file for a core QL pack uses the following properties:
+``name``, ``version``, ``dbscheme``, and ``suites``.
+The ``dbscheme`` property should only be defined in the core QL
+pack for a language.
+
+For example, the ``qlpack.yml`` file for `C/C++ analysis
+<https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml>`__
+contains:
+
+.. code-block:: yaml
+
+   name: codeql-cpp
+   version: 0.0.0
+   dbscheme: semmlecode.cpp.dbscheme
+   suites: codeql-suites
+
+Tests for the core QL pack
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``qlpack.yml`` file for the tests for the core QL packs use the following
+properties: ``name``, ``version``, and ``libraryPathDependencies``.
+The ``libraryPathDependencies`` always specifies the core QL pack.
+
+For example, the ``qlpack.yml`` file for `C/C++ analysis tests
+<https://github.com/github/codeql/blob/main/cpp/ql/test/qlpack.yml>`__
+contains:
+
+.. code-block:: yaml
+
+   name: codeql-cpp-tests
+   version: 0.0.0
+   libraryPathDependencies: codeql-cpp
+
+Notice that, unlike the example QL pack for custom tests, this file does not define
+an ``extractor`` or ``tests`` property. These properties have been added to
+the QL pack file since the release of CodeQL CLI 2.0.1. 
+They haven't been added yet to ensure compatibility for LGTM Enterprise users.
+After the next release of LGTM Enterprise, these files can be updated. 
+
+.. _upgrade-ql-packs:
+
+Upgrade scripts for a language
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``qlpack.yml`` file for a QL pack that contains only upgrade scripts
+uses the following properties: ``name`` and ``upgrades``.
+
+For example, the ``qlpack.yml`` file for `C/C++ upgrades
+<https://github.com/github/codeql/blob/main/cpp/upgrades/qlpack.yml>`__
+contains:
+
+.. code-block:: yaml
+
+   name: codeql-cpp-upgrades
+   upgrades: .

docs/codeql/codeql-cli/codeql-cli-reference/exit-codes.rst

@@ -0,0 +1,74 @@
+.. _exit-codes:
+
+Exit codes
+==========
+
+The CodeQL CLI reports the status of each command it runs as an exit code.
+This exit code provides information for subsequent commands or for other tools that rely on the CodeQL CLI.
+
+0
+---
+
+Success, normal termination.
+
+1
+---
+
+The command successfully determined that the answer to your question is "no".
+
+This exit code is only used by a few commands, such as `codeql test run <../codeql-cli-manual/test-run.html>`__, `codeql database check <../codeql-cli-manual/dataset-check.html>`__, `codeql query format <../codeql-cli-manual/query-format.html>`__,and `codeql resolve extractor <../codeql-cli-manual/resolve-extractor.html>`__.
+For more details, see the documentation for those commands.
+
+2
+---
+
+Something went wrong.
+
+The CLI writes a human-readable error message to stderr.
+This includes cases where an extractor fails with an internal error, because the ``codeql`` driver can't distinguish between internal and user-facing errors in extractor behavior.
+
+3
+---
+
+The launcher was unable to find the CodeQL installation directory.
+
+In this case, the launcher can't start the Java code for the CodeQL CLI at all. This should only happen when something is severely wrong with the CodeQL installation.
+
+32
+---
+
+The extractor didn't find any code to analyze when running `codeql database create <../codeql-cli-manual/database-create.html>`__ or `codeql database finalize <../codeql-cli-manual/database-finalize.html>`__.
+
+33
+---
+
+One or more query evaluations timed out.
+
+It's possible that some queries that were evaluated in parallel didn't time out. The results for those queries are produced as usual.
+
+98
+---
+
+Evaluation was explicitly canceled.
+
+99
+---
+
+The CodeQL CLI ran out of memory. 
+
+This doesn't necessarily mean that all the machine's physical RAM has been used.
+If you don't use the ``--ram`` option to set a limit explicitly, the JVM decides on a default limit at startup.
+
+100
+---
+
+A fatal internal error occurred.
+
+This should be considered a bug. The CLI usually writes an abbreviated error description to stderr.
+If you can reproduce the bug, it's helpful to use ``--logdir`` and send the log files to GitHub in a bug report.
+
+Other
+-----
+
+In the case of really severe problems within the JVM that runs ``codeql``, it might return a nonzero exit code of its own choosing.
+This should only happen if something is severely wrong with the CodeQL installation.

docs/codeql/codeql-cli/codeql-cli-reference/index.rst

@@ -0,0 +1,31 @@
+.. _codeql-cli-reference:
+
+CodeQL CLI reference
+====================
+
+Learn more about the files you can use when running CodeQL processes and the results format and exit codes that CodeQL generates. 
+
+.. toctree::
+   :titlesonly:
+   :hidden:
+
+   about-ql-packs
+   query-reference-files
+   sarif-output
+   exit-codes
+
+
+- :doc:`About QL packs <about-ql-packs>`: QL packs are used to organize the files used in CodeQL analysis. They
+  contain queries, library files, query suites, and important metadata. 
+- :doc:`Query reference files <query-reference-files>`: A query reference file is text file that defines the location of one query to test.
+- :doc:`SARIF output <sarif-output>`: CodeQL supports SARIF as an output format for sharing static analysis results.
+- :doc:`Exit codes <exit-codes>`: The CodeQL CLI reports the status of each command it runs as an exit code.
+  This exit code provides information for subsequent commands or for other tools that rely on the CodeQL CLI.
+
+.. _cli-commands:
+
+CodeQL CLI manual
+-----------------
+
+To view detailed information about each CodeQL CLI command,
+including its usage and options, add the ``--help`` flag or visit the "`CodeQL CLI manual <../codeql-cli-manual>`__."

docs/codeql/codeql-cli/codeql-cli-reference/query-reference-files.rst

@@ -0,0 +1,47 @@
+.. _query-reference-files:
+
+Query reference files
+=====================
+
+A query reference file is text file that defines the location of one query to test.
+
+You use a query reference file when you want to tell the ``test run`` subcommand
+to run a query that's not part of a test directory.
+There are two ways to specify queries that you want to run as tests:
+
+#. Use a query reference file to specify the location of a query to test.
+   This is useful when you create tests for alert and path queries that 
+   are intended to identify problems in real codebases. You might create 
+   several directories of test code, each focusing on different
+   aspects of the query. Then you would add a query reference file to 
+   each directory of test code, to specify the query to test.
+#. Add the query directly to a directory of tests.
+   These is typically useful when you're writing queries explicitly to test the behavior
+   of QL libraries. Often these queries contain just a few calls to library predicates,
+   wrapping them in a ``select`` statement so their output can be tested.
+
+Defining a query reference file
+-------------------------------
+
+Each query reference file, ``.qlref``, contains a single line that defines
+where to find one query. The location must be defined relative
+to the root of the QL pack that contains the query. 
+Usually, this is a QL pack specified by the ``libraryPathDependencies`` for the test pack.
+
+You should use forward slashes in the path on all operating
+systems to ensure compatibility between systems. 
+
+Example
+^^^^^^^^
+
+A query reference file to test a JavaScript alert query:
+`DeadAngularJSEventListener.qlref <https://github.com/github/codeql/blob/main/javascript/ql/test/query-tests/AngularJS/DeadAngularJSEventListener/DeadAngularJSEventListener.qlref>`__
+
+The `QL pack <https://github.com/github/codeql/blob/main/javascript/ql/test/qlpack.yml>`__ 
+for the ``javascript/ql/test`` directory defines the ``codeql-javascript`` queries as
+a dependency. So the query reference file defines the location of the query relative
+to the ``codeql-javascript`` QL pack::
+
+    AngularJS/DeadAngularJSEventListener.ql
+
+For another example, see `Testing custom queries <../using-the-codeql-cli/test-queries.html#example>`__.

docs/codeql/codeql-cli/codeql-cli-reference/sarif-output.rst

@@ -0,0 +1,292 @@
+.. _sarif-output:
+
+SARIF output
+============
+
+CodeQL supports SARIF as an output format for sharing static analysis results.
+
+SARIF is designed to represent the output of a broad range of static analysis
+tools, and there are many features in the SARIF specification that are
+considered "optional". This document details the output produced when using the
+format type ``sarifv2.1.0``, which corresponds to the SARIF v2.1.0.csd1
+specification.
+For more information on selecting a file format for your analysis results, see
+the `database analyze reference <../codeql-cli-manual/database-analyze.html>`__.
+
+SARIF specification and schema
+------------------------------
+
+This topic is intended to be read alongside the detailed SARIF specification.
+For more information on the specification and the SARIF schema, see the `SARIF
+specification documentation
+<https://github.com/oasis-tcs/sarif-spec/tree/master/Documents/CommitteeSpecificationDrafts/>`__
+on GitHub.
+
+Change notes
+------------
+
+Changes between versions
+~~~~~~~~~~~~~~~~~~~~~~~~
+
++----------------+-----------------+-----------------------------------------------------------------------------------------------------------------------------+
+| CodeQL version | Format type     | Changes                                                                                                                     | 
++================+=================+=============================================================================================================================+
+| 2.0.0          | ``sarifv2.1.0`` | First version of this format.                                                                                               |
++----------------+-----------------+-----------------------------------------------------------------------------------------------------------------------------+
+
+Future changes to the output
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The output produced for a given specific format type (for example
+``sarifv2.1.0``) may change in future CodeQL releases. We will endeavor to
+maintain backwards compatibility with consumers of the generated SARIF by
+ensuring that:
+
+-  No field which is marked as “Always” being generated will be removed.
+
+-  The circumstances under which “Optional” fields are generated may change.
+   Consumers of the CodeQL SARIF output should be robust to the presence or absence
+   of these fields.
+
+New output fields may be added in future releases under the same format
+type–these are not considered to break backwards compatibility, and consumers
+should be robust to the presence of newly added fields.
+
+New format argument types may be added in future versions of CodeQL---for example,
+to support new versions of SARIF. These have no guarantee of backwards
+compatibility, unless explicitly documented.
+
+Generated SARIF objects
+-----------------------
+
+This details each SARIF component that may be generated, along with any specific
+circumstances. We omit any properties that are never generated.
+
+``sarifLog`` object
+~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                                   |
++========================+=============================+===========================================================================================================================================+
+| ``$schema``            | Always                      | Provides a link to the `SARIF schema <https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json>`__. |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+| ``version``            | Always                      | The version of the SARIF used to generate the output.                                                                                     |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+| ``runs``               | Always                      | An array containing a single run object, for one language.                                                                                |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
+
+``run`` object
+~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                                                                                                                                                                                   |
++========================+=============================+===========================================================================================================================================================================================================================================================================================+
+| ``tool``               | Always                      | –                                                                                                                                                                                                                                                                                         |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``originalUriBaseIds`` | Always                      | A dictionary of ``uriBaseIds`` to artifactLocations representing the original locations on the analysis machine. At a minimum, this will contain the ``%SRCROOT%`` ``uriBaseId``, which represents the root location on the analysis machine of the source code for the analyzed project. |
+|                        |                             | Each ``artifactLocation`` will contain the ``uri`` and ``description`` properties.                                                                                                                                                                                                        |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``artifacts``          | Always                      | An array containing at least one artifact object for every file referenced in a result.                                                                                                                                                                                                   |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``results``            | Always                      | –                                                                                                                                                                                                                                                                                         |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``newLineSequences``   | Always                      | –                                                                                                                                                                                                                                                                                         |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``columnKind``         | Always                      | –                                                                                                                                                                                                                                                                                         |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``properties``         | Always                      | The properties dictionary will contain the ``semmle.formatSpecifier``, which identifies the format specifier passed to the CodeQL CLI.                                                                                                                                                    |
++------------------------+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+``tool`` object
+~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-----------+
+|   JSON property name   |   When is this generated?   |   Notes   |
++========================+=============================+===========+
+| ``driver``             | Always                      | –         |
++------------------------+-----------------------------+-----------+
+
+``toolComponent`` object
+~~~~~~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                                                                                                                                                                                  |
++========================+=============================+==========================================================================================================================================================================================================================================================================================+
+| ``name``               | Always                      | Set to “CodeQL command-line toolchain” for output from the CodeQL CLI tools. Note, if the output was generated using a different tool a different ``name`` is reported, and the format may not be as described here.                                                                     |
++------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``organization``       | Always                      | Set to “GitHub”.                                                                                                                                                                                                                                                                         |
++------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``version``            | Always                      | Set to the CodeQL release version e.g. “2.0.0”.                                                                                                                                                                                                                                          |                                   
++------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``rules``              | Always                      | An array of ``reportingDescriptor`` objects that represent rules. This array will contain, at a minimum, all the rules that were run during this analysis, but may contain rules which were available but not run. For more detail about enabling queries, see ``defaultConfiguration``. |
++------------------------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+``reportingDescriptor`` object (for rule)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``reportingDescriptor`` objects may be used in multiple places in the SARIF specification. When a ``reportingDescriptor`` is included in the rules array of a ``toolComponent`` object it has the following properties.
+
++------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name                                             |   When is this generated?      |   Notes                                                                                                                                                                                                                                                                            |
++==================================================================+================================+====================================================================================================================================================================================================================================================================================+
+| ``id``                                                           | Always                         | Will contain the ``@id`` property specified in the query that defines the rule, which is usually of the format ``language/rule-name`` (for example ``cpp/unsafe-format-string``). If your organization defines the ``@opaqueid`` property in the query it will be used instead.    |
++------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``name``                                                         | Always                         | Will contain the ``@id`` property specified in the query. See the ``id`` property above for an example.                                                                                                                                                                            |
++------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``shortDescription``                                             | Always                         | Will contain the ``@name`` property specified in the query that defines the rule.                                                                                                                                                                                                  |
++------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``fullDescription``                                              | Always                         | Will contain the ``@description`` property specified in the query that defines the rule.                                                                                                                                                                                           |
++------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``defaultConfiguration``                                         | Always                         | A ``reportingConfiguration`` object, with the enabled property set to true or false, and a level property set according to the ``@severity`` property specified in the query that defines the rule. Omitted if the ``@severity`` property was not specified.                       |
++------------------------------------------------------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+``artifact`` object
+~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                                                                                                                                                         |
++========================+=============================+=================================================================================================================================================================================================================================================================+
+| ``location``           | Always                      | An ``artifactLocation`` object.                                                                                                                                                                                                                                 |
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``index``              | Always                      | The index of the ``artifact`` object.                                                                                                                                                                                                                           |
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``contents``           | Optionally                  | If results are generated using the ``--sarif-add-file-contents`` flag, and the source code is available at the time the SARIF file is generated, then the ``contents`` property is populated with an ``artifactContent`` object, with the ``text`` property set.|
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+``artifactLocation`` object
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                              |
++========================+=============================+======================================================================================================================================+
+| ``uri``                | Always                      | –                                                                                                                                    |
++------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
+| ``index``              | Always                      | –                                                                                                                                    |
++------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
+| ``uriBaseId``          | Optionally                  | If the file is relative to some known abstract location, such as the root source location on the analysis machine, this will be set. |
++------------------------+-----------------------------+--------------------------------------------------------------------------------------------------------------------------------------+
+
+``result`` object
+~~~~~~~~~~~~~~~~~
+
+The composition of the results is dependent on the options provided to CodeQL. 
+By default, the results are grouped by unique message format string and
+primary location. Thus, two results that occur at the same location with the
+same underlying message, will appear as a single result in the output. This
+behavior can be disabled by using the flag ``--ungroup-results``, in which case
+no results are grouped.
+
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name    |   When is this generated?   |   Notes                                                                                                                                                                                                                                                                                                 |
++=========================+=============================+=========================================================================================================================================================================================================================================================================================================+
+| ``ruleId``              | Always                      | See the description of the ``id`` property in ``reportingDescriptor`` object (for rule) .                                                                                                                                                                                                               |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``ruleIndex``           | Always                      | –                                                                                                                                                                                                                                                                                                       |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``message``             | Always                      | A message describing the problem(s) occurring at this location. This message may be a SARIF “Message with placeholder”, containing links that refer to locations in the ``relatedLocations`` property.                                                                                                  |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``locations``           | Always                      | An array containing a single ``location`` object.                                                                                                                                                                                                                                                       |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``partialFingerprints`` | Always                      | A dictionary from named fingerprint types to the fingerprint. This will contain, at a minimum, a value for the ``primaryLocationLineHash``, which provides a fingerprint based on the context of the primary location.                                                                                  |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``codeFlows``           | Optionally                  | This array may be populated with one or more ``codeFlow`` objects if the query that defines the rule for this result is of ``@kind path-problem``.                                                                                                                                                      |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``relatedLocations``    | Optionally                  | This array will be populated if the query that defines the rule for this result has a message with placeholder options. Each unique location is included once.                                                                                                                                          |
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``suppressions``        | Optionally                  | If the result is suppressed, then this will contain a single ``suppression`` object, with the ``@kind`` property set to ``IN_SOURCE``. If this result is not suppressed, but there is at least one result that has a suppression, then this will be set to an empty array, otherwise it will not be set.|
++-------------------------+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+``location`` object
+~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                    |
++========================+=============================+============================================================================================================================+
+| ``physicalLocation``   | Always                      | –                                                                                                                          |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
+| ``id``                 | Optionally                  | ``location`` objects that appear in the ``relatedLocations`` array of a ``result`` object may contain the ``id`` property. |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
+| ``message``            | Optionally                  | ``location`` objects may contain the ``message`` property if:                                                              |
+|                        |                             |                                                                                                                            |
+|                        |                             | -  They appear in the ``relatedLocations`` array of a ``result`` object may contain the ``message`` property.              |
+|                        |                             | -  They appear in the ``threadFlowLocation.location`` property.                                                            |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------------------------+
+
+``physicalLocation`` object
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                                                           |
++========================+=============================+===================================================================================================================================+
+| ``artifactLocation``   | Always                      | –                                                                                                                                 |
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+| ``region``             | Optionally                  | If the given ``physicalLocation`` exists in a text file, such as a source code file, then the ``region`` property may be present. |
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+| ``contextRegion``      | Optionally                  | May be present if this location has an associated ``snippet``.                                                                    |
++------------------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
+
+``region`` object
+~~~~~~~~~~~~~~~~~
+
+There are two types of ``region`` object produced by CodeQL:
+
+-  Line/column offset regions
+-  Character offset and length regions
+
+Any region produced by CodeQL may be specified in either format, and consumers
+should robustly handle either type.
+
+For line/column offset regions, the following properties will be set:
+
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                      |
++========================+=============================+==============================================================================================+
+| ``startLine``          | Always                      | –                                                                                            |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+| ``startColumn``        | Optionally                  | Not included if equal to the default value of 1.                                             |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+| ``endLine``            | Optionally                  | Not included if identical to ``startLine``.                                                  |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+| ``endColumn``          | Always                      | –                                                                                            |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+| ``snippet``            | Optionally                  | –                                                                                            |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+
+For character offset and length regions, the following properties will be set:
+
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+|   JSON property name   |   When is this generated?   |   Notes                                                                                      |
++========================+=============================+==============================================================================================+
+| ``charOffset``         | Optionally                  | Provided if ``startLine``, ``startColumn``, ``endLine``, and ``endColumn`` are not populated.|
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+| ``charLength``         | Optionally                  | Provided if ``startLine``, ``startColumn``, ``endLine``, and ``endColumn`` are not populated.|
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+| ``snippet``            | Optionally                  | –                                                                                            |
++------------------------+-----------------------------+----------------------------------------------------------------------------------------------+
+
+``codeFlow`` object
+~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-----------+
+|   JSON property name   |   When is this generated?   |   Notes   |
++========================+=============================+===========+
+| ``threadFlows``        | Always                      | –         |
++------------------------+-----------------------------+-----------+
+
+``threadFlow`` object
+~~~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-----------+
+|   JSON property name   |   When is this generated?   |   Notes   |
++========================+=============================+===========+
+| ``locations``          | Always                      | –         |
++------------------------+-----------------------------+-----------+
+
+``threadFlowLocation`` object
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++------------------------+-----------------------------+-----------+
+|   JSON property name   |   When is this generated?   |   Notes   |
++========================+=============================+===========+
+| ``location``           | Always                      | –         |
++------------------------+-----------------------------+-----------+

docs/codeql/codeql-cli/index.rst

@@ -0,0 +1,23 @@
+.. _codeql-cli:
+
+CodeQL CLI
+==========
+
+.. include:: ../reusables/codeql-cli-overview.rst
+
+See the following links to learn how to get set up and run CodeQL commands:
+
+- :ref:`Using the CodeQL CLI <using-the-codeql-cli>`: Software 
+  developers and security researchers can secure their code using the CodeQL CLI.
+
+- :ref:`CodeQL CLI reference <codeql-cli-reference>`: Learn more about the files you can use when running CodeQL processes and the results format and exit codes that CodeQL generates.
+
+- `CodeQL CLI manual <codeql-cli-manual>`__: Detailed information about all the commands available with the CodeQL CLI.
+
+.. toctree::
+   :titlesonly:
+   :hidden:
+
+   using-the-codeql-cli/index
+   codeql-cli-reference/index
+   

docs/codeql/codeql-cli/using-the-codeql-cli/about-the-codeql-cli.rst

@@ -0,0 +1,29 @@
+.. _about-the-codeql-cli:
+
+About the CodeQL CLI
+====================
+
+Software developers and security researchers can secure their code
+using the CodeQL CLI.
+
+The CodeQL CLI is a command-line tool used to run CodeQL processes locally on
+open source software projects. You can use the CodeQL CLI to:
+
+- Run CodeQL analyses using queries provided by GitHub engineers and the open
+  source community
+- Create CodeQL databases to use in the CodeQL for Visual Studio Code
+- Develop and test custom CodeQL queries to use in your own analyses
+
+For information about using the CodeQL CLI, see 
+":ref:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`."
+
+CodeQL CLI commands
+-------------------
+
+The CodeQL CLI includes commands to create and analyze CodeQL databases from the
+command line. To run a command, use::
+
+   codeql [command] [subcommand]
+
+To view the reference documentation for a command, add the ``--help`` flag, or visit the 
+"`CodeQL CLI manual <../../codeql-cli-manual>`__."

docs/codeql/codeql-cli/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli.rst

@@ -0,0 +1,228 @@
+.. _analyzing-databases-with-the-codeql-cli:
+
+Analyzing databases with the CodeQL CLI
+=======================================
+
+To analyze a codebase, you run queries against a CodeQL
+database extracted from the code.
+
+CodeQL analyses produce :ref:`interpreted results
+<interpret-query-results>` that can be displayed as alerts or paths in source code.
+For information about writing queries to run with ``database analyze``, see
+":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
+
+.. include:: ../../reusables/advanced-query-execution.rst
+
+Before starting an analysis you must:
+
+- :doc:`Set up the CodeQL CLI <getting-started-with-the-codeql-cli>` so that it can find the queries
+  and libraries included in the CodeQL repository. 
+- :doc:`Create a CodeQL database <creating-codeql-databases>` for the source 
+  code you want to analyze. 
+
+
+Running ``codeql database analyze``
+------------------------------------
+
+When you run ``database analyze``, it does two things:
+
+#. Executes one or more query files, by running them over a CodeQL database.
+#. Interprets the results, based on certain query metadata, so that alerts can be
+   displayed in the correct location in the source code.
+
+You can analyze a database by running the following command::
+
+   codeql database analyze <database> <queries> --format=<format> --output=<output>
+
+You must specify:
+
+- ``<database>``: the path to the CodeQL database you want to analyze.
+
+- ``<queries>``: the queries to run over your database. You can
+  list one or more individual query files, specify a directory that will be
+  searched recursively for query files, or name a query suite that defines a
+  particular set of queries. For more information, see the :ref:`examples
+  <database-analyze-examples>` below.
+
+- ``--format``: the format of the results file generated during analysis. A
+  number of different formats are supported, including CSV, :ref:`SARIF
+  <sarif-file>`, and graph formats. For more information about CSV and SARIF,
+  see `Results <#results>`__. To find out which other results formats are
+  supported, see the `database analyze reference
+  <../codeql-cli-manual/database-analyze.html>`__.
+
+- ``--output``: the output path of the results file generated during analysis.
+
+You can also specify:
+
+- .. include:: ../../reusables/threads-query-execution.rst
+
+
+.. pull-quote:: 
+
+   Upgrading databases
+
+   If the CodeQL queries you want to use are newer than the
+   extractor used to create the database, then you may see a message telling you
+   that your database needs to be upgraded when you run ``database analyze``.
+   You can quickly upgrade a database by running the ``database upgrade``
+   command. For more information, see ":doc:`Upgrading CodeQL databases
+   <upgrading-codeql-databases>`."
+
+For full details of all the options you can use when analyzing databases, see
+the `database analyze reference documentation <../codeql-cli-manual/database-analyze.html>`__.
+
+.. _database-analyze-examples:
+
+Examples
+--------
+
+The following examples assume your CodeQL databases have been created in a
+directory that is a sibling of your local copies of the CodeQL and CodeQL for Go
+repositories.
+
+Running a single query
+~~~~~~~~~~~~~~~~~~~~~~
+
+To run a single query over a JavaScript codebase, you could use the following
+command from the directory containing your database::
+
+   codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv 
+
+This command runs a simple query that finds potential bugs related to unused
+variables, imports, functions, or classes---it is one of the JavaScript
+queries included in the CodeQL repository. You could run more than one query by
+specifying a space-separated list of similar paths.
+
+The analysis generates a CSV file (``js-results.csv``) in a new directory
+(``js-analysis``).  
+
+You can also run your own custom queries with the ``database analyze`` command.
+For more information about preparing your queries to use with the CodeQL CLI,
+see ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
+
+
+Running LGTM.com query suites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The CodeQL repository also includes query suites, which can be run over your
+code as part of a broader code review. CodeQL query suites are ``.qls`` files
+that use directives to select queries to run based on certain metadata
+properties.
+
+The query suites included in the CodeQL repository select the same set of
+queries that are run by default on `LGTM.com <https://lgtm.com>`__. The queries
+are selected to highlight the most relevant and useful results for each
+language. 
+
+The language-specific LGTM query suites are located at the following paths in
+the CodeQL repository::
+
+   ql/<language>/ql/src/codeql-suites/<language>-lgtm.qls
+
+and at the following path in the CodeQL for Go repository::
+
+   ql/src/codeql-suites/go-lgtm.qls
+
+These locations are specified in the metadata included in the standard QL packs.
+This means that CodeQL knows where to find the suite files automatically, and
+you don't have to specify the full path on the command line when running an
+analysis. For more information, see ":ref:`About QL packs <standard-ql-packs>`."
+
+For example, to run the LGTM.com query suite on a C++ codebase (generating
+results in the latest SARIF format), you would run::
+
+   codeql database analyze <cpp-database> cpp-lgtm.qls --format=sarif-latest --output=cpp-analysis/cpp-results.sarif
+   
+For information about creating custom query suites, see ":doc:`Creating
+CodeQL query suites <creating-codeql-query-suites>`."
+
+Running all queries in a directory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can run all the queries located in a directory by providing the directory
+path, rather than listing all the individual query files. Paths are searched
+recursively, so any queries contained in subfolders will also be executed. 
+
+.. pull-quote::
+
+   Important
+
+   You shouldn't specify the root of a :doc:`QL pack
+   <../codeql-cli-reference/about-ql-packs>` when executing ``database analyze``
+   as it contains some special queries that aren't designed to be used with
+   the command. Rather, to run a wide range of useful queries, run one of the
+   LGTM.com query suites. 
+   
+For example, to execute all Python queries contained in the ``Functions``
+directory you would run::
+
+   codeql database analyze <python-database> ../ql/python/ql/src/Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif 
+
+A SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures
+that the results are formatted according to the most recent SARIF specification
+supported by CodeQL.
+
+
+Results
+-------
+
+You can save analysis results in a number of different formats, including SARIF
+and CSV.
+
+The SARIF format is designed to represent the output of a broad range of static
+analysis tools. For more information, see :doc:`SARIF output <../codeql-cli-reference/sarif-output>`.
+
+If you choose to generate results in CSV format, then each line in the output file
+corresponds to an alert. Each line is a comma-separated list with the following information:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Property
+     - Description
+     - Example
+   * - Name
+     - Name of the query that identified the result. 
+     - ``Inefficient regular expression``
+   * - Description
+     - Description of the query. 
+     - ``A regular expression that requires exponential time to match certain 
+       inputs can be a performance bottleneck, and may be vulnerable to 
+       denial-of-service attacks.``
+   * - Severity
+     - Severity of the query.
+     - ``error``
+   * - Message
+     - Alert message. 
+     - ``This part of the regular expression may cause exponential backtracking 
+       on strings containing many repetitions of '\\\\'.``
+   * - Path
+     - Path of the file containing the alert.
+     - ``/vendor/codemirror/markdown.js``
+   * - Start line
+     - Line of the file where the code that triggered the alert begins.
+     - ``617``
+   * - Start column
+     - Column of the start line that marks the start of the alert code. Not
+       included when equal to 1.
+     - ``32``
+   * - End line
+     - Line of the file where the code that triggered the alert ends. Not
+       included when the same value as the start line.
+     - ``64``
+   * - End column
+     - Where available, the column of the end line that marks the end of the 
+       alert code. Otherwise the end line is repeated.
+     - ``617``
+
+Results files can be integrated into your own code-review or debugging
+infrastructure. For example, SARIF file output can be used to highlight alerts
+in the correct location in your source code using a SARIF viewer plugin for your
+IDE. 
+
+Further reading
+---------------
+
+- ":ref:`Analyzing your projects in CodeQL for VS Code <analyzing-your-projects>`"

docs/codeql/codeql-cli/using-the-codeql-cli/creating-codeql-databases.rst

@@ -0,0 +1,226 @@
+.. _creating-codeql-databases:
+
+Creating CodeQL databases
+=========================
+
+Before you analyze your code using CodeQL, you need to create a CodeQL
+database containing all the data required to run queries on your code.
+
+CodeQL analysis relies on extracting relational data from your code, and
+using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL 
+databases contain all of the important information about a codebase, which can 
+be analyzed by executing CodeQL queries against it.
+Before you generate a CodeQL database, you need to:
+
+- Install and set up the CodeQL CLI. For more information, see
+  ":doc:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`."
+- Check out the version of your codebase you want to analyze. The directory
+  should be ready to build, with all dependencies already installed.
+
+Running ``codeql database create``
+----------------------------------
+
+CodeQL databases are created by running the following command from the checkout root
+of your project:
+
+::
+
+  codeql database create <database> --language=<language-identifier>
+
+You must specify:
+
+- ``<database>``: a path to the new database to be created. This directory will
+  be created when you execute the command---you cannot specify an existing
+  directory. 
+- ``--language``: the identifier for the language to create a database for.
+  CodeQL supports creating databases for the following languages:
+
+  .. include:: ../../reusables/extractors.rst
+
+Other options may be specified depending on the location of your source file and
+the language you want to analyze:
+
+- ``--source-root``: the root folder for the primary source files used in
+  database creation. By default, the command assumes that the current
+  directory is the source root---use this option to specify a different location.
+- ``--command``: for compiled languages only, the build commands that invoke the
+  compiler. Do not specify ``--command`` options for Python and
+  JavaScript. Commands will be run from the current folder, or ``--source-root``
+  if specified. If you don't include a ``--command``, CodeQL will attempt to
+  detect the build system automatically, using a built-in autobuilder. 
+   
+For full details of all the options you can use when creating databases,
+see the `database create reference documentation <../codeql-cli-manual/database-create.html>`__.  
+
+Progress and results
+--------------------
+
+Errors are reported if there are any problems with the options you have
+specified. For interpreted languages, the extraction progress is displayed in
+the console---for each source file, it reports if extraction was successful or if
+it failed. For compiled languages, the console will display the output of the
+build system.
+
+When the database is successfully created, you'll find a new directory at the
+path specified in the command. This directory contains a number of
+subdirectories, including the relational data (required for analysis) and a
+source archive---a copy of the source files made at the time the database was
+created---which is used for displaying analysis results.
+
+Obtaining databases from LGTM.com
+---------------------------------
+
+`LGTM.com <https://lgtm.com>`__ analyzes thousands of open-source projects using
+CodeQL. For each project on LGTM.com, you can download an archived CodeQL
+database corresponding to the most recently analyzed revision of the code. These
+databases can also be analyzed using the CodeQL CLI. 
+
+.. include:: ../../reusables/download-lgtm-database.rst
+
+Before running an analysis, unzip the databases and try :doc:`upgrading <upgrading-codeql-databases>` the
+unzipped databases to ensure they are compatible with your local copy of the
+CodeQL queries and libraries.
+   
+.. pull-quote::
+
+   Note
+
+   .. include:: ../../reusables/index-files-note.rst
+
+Creating databases for non-compiled languages
+---------------------------------------------
+
+The CodeQL CLI includes extractors to create databases for non-compiled
+languages---specifically, JavaScript (and TypeScript) and Python. These
+extractors are automatically invoked when you specify JavaScript or Python as
+the ``--language`` option when executing ``database create``. When creating
+databases for these languages you must ensure that all additional dependencies
+are available.
+
+.. pull-quote:: Important
+
+   When running ``database create`` for JavaScript, TypeScript, and Python, you must not
+   specify a ``--command`` option. If you do, you will override the normal
+   extractor invocation, which will create an empty database.
+
+JavaScript and TypeScript
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Creating databases for JavaScript requires no additional dependencies, but if
+the project includes TypeScript files, you must install Node.js 6.x
+or later. In the command line you can specify ``--language=javascript`` to
+extract both JavaScript and TypeScript files::
+
+   codeql database create --language=javascript --source-root <folder-to-extract> <output-folder>/javascript-database
+
+Here, we have specified a ``--source-root`` path, which is the location where
+database creation is executed, but is not necessarily the checkout root of the
+codebase. 
+
+Python
+~~~~~~
+
+When creating databases for Python you must ensure:
+
+- You have the all of the required versions of Python installed.
+- You have access to the `pip <https://pypi.org/project/pip/>`__ 
+  packaging management system and can install any
+  packages that the codebase depends on.
+- You have installed the `virtualenv <https://pypi.org/project/virtualenv/>`__ pip module.
+
+In the command line you must specify ``--language=python``. For example
+::
+
+   codeql database create --language=python <output-folder>/python-database
+
+executes the ``database create`` subcommand from the code's checkout root,
+generating a new Python database at ``<output-folder>/python-database``.
+
+
+Creating databases for compiled languages
+-----------------------------------------
+
+For compiled languages, CodeQL needs to invoke the required build system to
+generate a database, therefore the build method must be available to the CLI.
+
+Detecting the build system
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The CodeQL CLI includes autobuilders for C/C++, C#, Go, and Java code. CodeQL
+autobuilders allow you to build projects for compiled languages without
+specifying any build commands. When an autobuilder is invoked, CodeQL examines
+the source for evidence of a build system and attempts to run the optimal set of
+commands required to extract a database.
+
+An autobuilder is invoked automatically when you execute ``codeql database
+create`` for a compiled ``--language`` if don't include a
+``--command`` option. For example, for a Java codebase, you would simply run::
+
+   codeql database create --language=java <output-folder>/java-database
+
+If a codebase uses a standard build system, relying on an autobuilder is often
+the simplest way to create a database. For sources that require non-standard
+build steps, you may need to explicitly define each step in the command line.
+
+
+.. pull-quote:: Creating databases for Go
+  
+   For Go, you should always use the CodeQL autobuilder. Install the Go
+   toolchain (version 1.11 or later) and, if there are dependencies, the
+   appropriate dependency manager (such as `dep
+   <https://golang.github.io/dep/>`__ or `Glide <http://glide.sh/>`__).
+   
+   Do not specify any build commands, as you will override the autobuilder
+   invocation, which will create an empty database.  
+
+Specifying build commands
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following examples are designed to give you an idea of some of the build
+commands that you can specify for compiled languages. 
+
+.. pull-quote:: Important
+
+   The ``--command`` option accepts a single argument---if you need to
+   use more than one command, specify ``--command`` multiple times.
+
+   If you need to pass subcommands and options, the whole argument needs to be
+   quoted to be interpreted correctly.
+
+- C/C++ project built using ``make``::
+
+     codeql database create cpp-database --language=cpp --command=make
+
+- C# project built using ``dotnet build`` (.NET Core 3.0 or later)::
+
+     codeql database create csharp-database --language=csharp --command='dotnet build /t:rebuild' 
+
+  On Linux and macOS (but not Windows), you need to disable shared compilation when building C# projects
+  with .NET Core 2 or earlier, so expand the command to::
+
+     codeql database create csharp-database --language=csharp --command='dotnet build /p:UseSharedCompilation=false /t:rebuild'
+
+- Java project built using Gradle::
+
+     codeql database create java-database --language=java --command='gradle clean test'
+
+- Java project built using Maven::
+
+     codeql database create java-database --language=java --command='mvn clean install'
+
+- Java project built using Ant::
+
+     codeql database create java-database --language=java --command='ant -f build.xml'
+
+- Project built using a custom build script::
+
+     codeql database create new-database --language=<language> --command='./scripts/build.sh'
+   
+  This command runs a custom script that contains all of the commands required
+  to build the project.
+
+
+Further reading
+---------------
+
+- ":ref:`Analyzing your projects in CodeQL for VS Code <analyzing-your-projects>`"

docs/codeql/codeql-cli/using-the-codeql-cli/creating-codeql-query-suites.rst

@@ -0,0 +1,285 @@
+.. _creating-codeql-query-suites:
+
+Creating CodeQL query suites
+============================
+
+CodeQL query suites provide a way of selecting queries, based on their
+filename, location on disk or in a QL pack, or metadata properties. 
+Create query suites for the queries that you want to frequently use in 
+your CodeQL analyses.
+
+Query suites allow you to pass multiple queries to
+CodeQL without having to specify the path to each query file individually.
+Query suite definitions are stored in YAML files with the extension ``.qls``. A
+suite definition is a sequence of instructions, where each instruction is a YAML
+mapping with (usually) a single key. The instructions are executed in the order
+they appear in the query suite definition. After all the instructions in the
+suite definition have been executed, the result is a set of selected queries.
+
+.. note::
+
+   Any custom queries that you want to add to a query suite must be in a :doc:`QL
+   pack <../codeql-cli-reference/about-ql-packs>` and contain the correct query metadata. 
+   For more information, see
+   ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
+
+Locating queries to add to a query suite
+----------------------------------------
+
+When creating a query suite, you first need to specify the locations of the
+queries that you want to select. You can define the location of one or more
+queries using:
+
+- A ``query`` instruction---tells CodeQL to look for one or more specified ``.ql``
+  files::
+
+     - query: <path-to-query>
+
+  The argument must be one or more file paths, relative to the QL pack containing
+  the suite definition.
+
+- A ``queries`` instruction---tells CodeQL to recursively scan a directory
+  for ``.ql`` files::
+
+   - queries: <path-to-subdirectory>
+
+  The path of the directory must be relative to the root of the QL pack that
+  contains the suite definition file. To find the queries relative to a
+  different QL pack, add a ``from`` field::
+
+   - queries: <path-to-subdirectory>
+     from: <ql-pack-name>
+
+- A ``qlpack`` instruction---tells CodeQL to look for queries in a named QL pack::
+
+   - qlpack: <qlpack-name>
+
+.. note::
+
+   When pathnames appear in query suite definitions, they must always
+   be given with a forward slash, ``/``, as a directory separator.
+   This ensures that query suite definitions work on all operating systems.
+
+You must add at least one ``query``, ``queries``, or ``qlpack`` instruction to
+your suite definition, otherwise no queries will be selected. If the suite
+contains no further instructions, all the queries found from the list of files,
+in the given directory, or in the named QL pack are selected. If there are further
+filtering instructions, only queries that match the constraints imposed by those
+instructions will be selected.
+
+Filtering the queries in a query suite
+----------------------------------------
+
+After you have defined the initial set of queries to add to your suite by
+specifying ``query``, ``queries``, or ``qlpack`` instructions, you can add
+``include`` and ``exclude`` instructions. These instructions define selection
+criteria based on specific properties: 
+
+- When you execute an ``include`` instruction on a set of queries, any
+  queries that match your conditions are retained in the selection, and queries
+  that don't match are removed.
+- When you execute an ``exclude`` instructions on a set of queries,
+  any queries that match your conditions are removed from the selection, and queries
+  that don't match are retained.
+
+The order of your filter instructions is important. The first filter instruction
+that appears after the locating instructions determines whether the queries are
+included or excluded by default. If the first filter is an ``include``, the
+initially located queries will only be part of the suite if they match an
+explicit ``include`` filter. If the first filter is an ``exclude``, the initially
+located queries are part of the suite unless they are explicitly excluded.
+
+Subsequent instructions are executed in order and the instructions that appear
+later in the file take precedence over the earlier instructions. So, ``include``
+instructions can be overridden by a later ``exclude`` instructions that match
+the same query. Similarly, ``exclude``\ s can be overridden by a later
+``include``.
+
+For both instructions, the argument is a constraint block---that is, a YAML map
+representing the constraints. Each constraint is a map entry, where the key is
+typically a query metadata property. The value can be:
+
+- A single string. 
+- A ``/``\ -enclosed `regular expression <https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html>`__.
+- A list containing strings, regular expressions, or both.
+
+To match a constraint, a metadata value must match one of the strings or
+regular expressions. When there is more than one metadata key, each key must be matched. 
+For more information about query metadata properties, see ":ref:`Metadata for CodeQL queries
+<metadata-for-codeql-queries>`."
+
+In addition to metadata tags, the keys in the constraint block can also be:
+
+- ``query filename``---matches on the last path component of the query file name.
+- ``query path``---matches on the path to the query file relative to its
+  enclosing QL pack. 
+- ``tags contain``---one of the given match strings must match
+  one of the space-separated components of the value of the ``@tags`` metadata property.
+- ``tags contain all``---each of the given match strings must match one of the
+  components of the ``@tags`` metadata property.
+
+Examples
+~~~~~~~~
+
+To define a suite that selects all queries with ``@kind problem``
+and ``@precision high`` from the ``my-custom-queries`` directory, use::
+
+   - queries: my-custom-queries
+   - include:
+       kind: problem
+       precision: very-high
+
+To create a suite that selects all queries with ``@kind problem`` from the
+``my-custom-queries`` directory except those with ``@problem.severity
+recommendation``, use::
+
+   - queries: my-custom-queries
+   - include:
+       kind: problem
+   - exclude:    
+       problem.severity: recommendation
+
+To create a suite that selects all queries with ``@tag security`` and
+``@problem.severity high`` or ``very-high`` from the ``codeql-cpp`` QL pack,
+use::
+
+   - qlpack: codeql-cpp
+   - include: 
+       tags contain: security
+       problem.severity: 
+       - high
+       - very-high
+
+Reusing existing query suite definitions
+-----------------------------------------
+
+Existing query suite definitions can be reused by specifying: 
+
+- An ``import`` instruction---adds the queries selected by a
+  previously defined ``.qls`` file to the current suite::
+    
+    - import: <path-to-query-suite>
+
+  The path to the imported suite must be relative to the QL pack containing the
+  current suite definition. If the imported query suite is in a different QL
+  pack you can use::
+  
+    - import: <path-to-query-suite>
+      from: <ql-pack>
+
+  Queries added using an ``import`` instruction can be filtered using subsequent
+  ``exclude`` instructions.
+
+- An ``apply`` instruction---adds all of the instructions from a
+  previously defined ``.qls`` file to the current suite. The instructions in the
+  applied ``.qls`` file are executed as if they appear in place of ``apply``.
+  Any ``include`` and ``exclude`` instructions from the applied suite also act on
+  queries added by any earlier instructions::
+    
+    - apply: <path-to-query-suite>
+
+  The ``apply`` instruction can also be used to apply a set of reusable
+  conditions, saved in a ``.yml`` file, to multiple query definitions. For more
+  information, see the `example <#example>`__ below. 
+
+- An ``eval`` instruction---performs the same function as an ``import``
+  instruction, but takes a full suite definition as the argument, rather than the
+  path to a ``.qls`` file on disk.
+
+Example
+~~~~~~~
+
+To use the same conditions in multiple query suite definitions, create a
+separate ``.yml`` file containing your instructions. For example, save the
+following in a file called ``reusable-instructions.yml``::
+
+   - include:
+       kind:
+       - problem
+       - path-problem
+       tags contain: security
+       precision:
+       - high
+       - very-high
+
+Add ``reusable-instructions.yml`` to the same QL pack as your current query
+suite (for example, ``my-custom-queries``). Apply the reusable instructions
+to the queries in your current suite using::
+
+  - qlpack: my-custom-queries
+  - apply: reusable-instructions.yml
+
+To apply the same conditions to a different suite or directory within the same
+QL pack, create a new definition and change (or replace) the ``qlpack``
+instruction. For example::
+
+  - queries: queries/cpp/custom
+  - apply: reusable-instructions.yml
+
+You can also create a suite definition using ``reusable-instructions.yml`` on
+queries in a different QL pack. If the ``.qls`` file is in the same QL pack as
+the queries, you can add a ``from`` field immediately after the ``apply``
+instruction::
+
+  - qlpack: my-other-custom-queries
+  - apply: reusable-instructions.yml
+    from: <name-of-ql-pack>
+
+Naming a query suite
+--------------------
+
+You can provide a name for your query suite by specifying a ``description``
+instruction::
+
+   - description: <name-of-query-suite>
+
+This value is displayed when you run `codeql resolve queries
+<../codeql-cli-manual/resolve-queries.html>`__, if the suite is added to a "well-known"
+directory. For more information, see "`Specifying well-known query suites
+<#specifying-well-known-query-suites>`__." 
+
+Saving a query suite
+--------------------
+
+Save your query suite in a file with a ``.qls`` extension and add it to a QL
+pack. For more information, see ":ref:`About QL packs <custom-ql-packs>`."
+
+Specifying well-known query suites
+----------------------------------
+
+You can use QL packs to declare directories that contain "well-known" query
+suites. You can use "well-known" query suites on the command line by referring 
+to their file name, 
+without providing their full path. This gives you a simple way of specifying a
+set of queries, without needing to search inside QL packs and distributions.
+To declare a directory that contains "well-known" query suites, add the directory
+to the ``suites`` property in the ``qlpack.yml`` file at the root of your QL pack.
+For more information, see "`About QL packs <../codeql-cli-reference/qlpack-overview.html#qlpack-yml-properties>`__."
+
+Using query suites with CodeQL
+------------------------------
+
+You can specify query suites on the command line for any command that accepts
+``.qls`` files. For example, you can compile the queries selected by a suite
+definition using ``query compile``, or use the queries in an analysis using
+``database analyze``. For more information about analyzing CodeQL databases, see
+":doc:`Analyzing databases with the CodeQL CLI <analyzing-databases-with-the-codeql-cli>`." 
+
+Viewing the query suites used on LGTM.com
+-----------------------------------------
+
+The query suite definitions used to select queries to run on LGTM.com can be
+found in the CodeQL repository. For example, to view the CodeQL queries for
+JavaScript, visit
+https://github.com/github/codeql/tree/main/javascript/ql/src/codeql-suites.
+
+These suite definitions apply reusable filter patterns to the queries
+located in the standard QL packs for each supported language. For more
+information, see the `suite-helpers
+<https://github.com/github/codeql/tree/main/misc/suite-helpers>`__ in the CodeQL
+repository.
+
+Further reading
+---------------
+
+- ":ref:`CodeQL queries <codeql-queries>`"

docs/codeql/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli.rst

@@ -0,0 +1,235 @@
+.. _getting-started-with-the-codeql-cli:
+
+Getting started with the CodeQL CLI
+===================================
+
+To run CodeQL commands, you need to set up the CLI so that it can access
+the tools, queries, and libraries required to create and analyze databases. 
+
+.. include:: ../../reusables/license-note.rst
+
+.. _setting-up-cli:
+
+Setting up the CodeQL CLI
+-------------------------
+
+The CodeQL CLI can be set up to support many different use cases and directory
+structures. To get started quickly, we recommend adopting a relatively simple
+setup, as outlined in the steps below.
+
+If you use Linux, Windows, or macOS version 10.14 ("Mojave") or earlier, simply
+follow the steps below. For macOS version 10.15 ("Catalina"), steps 1 and 4 are
+slightly different---for further details, see the sections labeled **Information
+for macOS "Catalina" users**.
+
+1. Download the CodeQL CLI zip package
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The CodeQL CLI download package is a zip archive containing tools, scripts, and
+various CodeQL-specific files. If you don't have an Enterprise license then, by
+downloading this archive, you are agreeing to the `GitHub CodeQL Terms and
+Conditions <https://securitylab.github.com/tools/codeql/license>`__.
+
+.. pull-quote:: Important
+
+   There are several different versions of the CLI available to download, depending
+   on your use case:
+   
+   - If you want to use the most up to date CodeQL tools and features, download the
+     version tagged ``latest``. 
+
+   - If you want to create CodeQL databases to upload to LGTM Enterprise, download
+     the version that is compatible with the relevant LGTM Enterprise version
+     number. Compatibility information is included in the description for each
+     release on the `CodeQL CLI releases page
+     <https://github.com/github/codeql-cli-binaries/releases>`__ on GitHub. Using the
+     correct version of the CLI ensures that your CodeQL databases are
+     compatible with your version of LGTM Enterprise. For more information,
+     see `Preparing CodeQL databases to upload to LGTM 
+     <https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html>`__
+     in the LGTM admin help. 
+
+If you use Linux, Windows, or macOS version 10.14 ("Mojave") or earlier, simply
+`download the zip archive
+<https://github.com/github/codeql-cli-binaries/releases>`__
+for the version you require.
+
+If you want the CLI for a specific platform, download the appropriate ``codeql-PLATFORM.zip`` file.
+Alternatively, you can download ``codeql.zip``, which contains the CLI for all supported platforms.
+
+.. container:: toggle
+
+   .. container:: name
+
+      **Information for macOS "Catalina" users**
+
+   .. pull-quote:: macOS "Catalina"
+      
+      If you use macOS version 10.15 ("Catalina"), you need to ensure that your web
+      browser does not automatically extract zip files. If you use Safari,
+      complete the following steps before downloading the CodeQL CLI zip archive:
+          
+      i. Open Safari.
+      ii. From the Safari menu, select **Preferences...**.
+      iii. Click the **General** Tab.
+      iv. Ensure the check-box labeled **Open "safe" files after downloading**.
+          is unchecked.
+
+2. Create a new CodeQL directory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create a new directory where you can place the CLI and any queries and libraries
+you want to use. For example, ``$HOME/codeql-home``.
+
+The CLI's built-in search operations automatically look in all of its sibling
+directories for the files used in database creation and analysis. Keeping these
+components in their own directory prevents the CLI searching unrelated sibling
+directories while ensuring all files are available without specifying any
+further options on the command line.
+
+.. _local-copy-codeql-queries:
+
+3. Obtain a local copy of the CodeQL queries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `CodeQL repository <https://github.com/github/codeql>`__ contains 
+the queries and libraries required for CodeQL analysis of C/C++, C#, Java,
+JavaScript/TypeScript, and Python.
+Clone a copy of this repository into ``codeql-home``. 
+ 
+By default, the root of the cloned repository will be called ``codeql``. 
+Rename this folder ``codeql-repo`` to avoid conflicting with the CodeQL 
+CLI that you will extract in step 4. If you use git on the command line, you can 
+clone and rename the repository in a single step by running 
+``git clone git@github.com:github/codeql.git codeql-repo`` in the ``codeql-home`` folder.
+
+The CodeQL libraries and queries for Go analysis live in the `CodeQL for Go
+repository <https://github.com/github/codeql-go/>`__. Clone a copy of this
+repository into ``codeql-home``. 
+
+The cloned repositories should have a sibling relationship. 
+For example, if the root of the cloned CodeQL repository is
+``$HOME/codeql-home/codeql-repo``, then the root of the cloned CodeQL for Go
+repository should be ``$HOME/codeql-home/codeql-go``.
+
+Within these repositories, the queries and libraries are organized into QL
+packs. Along with the queries themselves, QL packs contain important metadata
+that tells the CodeQL CLI how to process the query files. For more information,
+see ":doc:`About QL packs <../codeql-cli-reference/about-ql-packs>`."
+
+.. pull-quote:: Important
+
+   There are different versions of the CodeQL queries available for different
+   users. Check out the correct version for your use case:
+   
+   - For the most up to date CodeQL queries, check out the ``main`` branch. 
+     This branch represents the very latest version of CodeQL's analysis. Even
+     databases created using the most recent version of the CLI may have to be
+     upgraded before you can analyze them. For more information, see
+     ":doc:`Upgrading CodeQL databases <upgrading-codeql-databases>`."
+    
+   - For the queries used on `LGTM.com <https://lgtm.com>`__, check out the
+     ``lgtm.com`` branch. You can run these queries on databases you've recently
+     downloaded from LGTM.com. Older databases may need to be upgraded before
+     you can analyze them. The queries on the ``lgtm.com`` branch are also more
+     likely to be compatible with the ``latest`` CLI, so you'll be less likely
+     to have to upgrade newly-created databases than if you use the ``main``
+     branch.
+           
+   - For the queries used in a particular LGTM Enterprise release, check out the
+     branch tagged with the relevant release number. For example, the branch
+     tagged ``v1.23.0`` corresponds to LGTM Enterprise 1.23. You must use this
+     version if you want to upload data to LGTM Enterprise. For further
+     information, see `Preparing CodeQL databases to upload to LGTM 
+     <https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html>`__
+     in the LGTM admin help.
+
+4. Extract the zip archive
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) 
+simply
+extract the zip archive into the directory you created in step 2.
+
+For example, if the path to your copy of the CodeQL repository is
+``$HOME/codeql-home/codeql-repo``, then extract the CLI into
+``$HOME/codeql-home/``.
+
+.. container:: toggle
+
+   .. container:: name
+
+      **Information for macOS "Catalina" users**
+
+   .. pull-quote:: macOS "Catalina"
+   
+      macOS "Catalina" users should run the following commands in the Terminal,
+      where ``${install_loc}`` is the path to the directory you created in step 2:
+   
+      i. ``mv ~/Downloads/codeql*.zip ${install_loc}``
+      ii. ``cd ${install_loc}``
+      iii. ``xattr -c codeql*.zip``
+      iv. ``unzip codeql*.zip``
+   
+5. Launch ``codeql``
+~~~~~~~~~~~~~~~~~~~~
+
+Once extracted, you can run CodeQL processes by running the ``codeql``
+executable in a couple of ways:
+
+- By executing ``<extraction-root>/codeql/codeql``, where
+  ``<extraction-root>`` is the folder where you extracted the CodeQL CLI
+  package.
+- By adding ``<extraction-root>/codeql`` to your ``PATH``, so that you
+  can run the executable as just ``codeql``. 
+
+At this point, you can execute CodeQL commands. For a full list of the CodeQL
+CLI commands, see the "`CodeQL CLI manual <../../codeql-cli-manual>`__."
+
+.. pull-quote:: Note
+
+   If you add ``codeql`` to your ``PATH``, it can be accessed by CodeQL
+   for Visual Studio Code to compile and run queries.
+   For more information about configuring VS Code to access the CodeQL CLI, see
+   ":ref:`Setting up CodeQL in Visual Studio Code <setting-up-codeql-in-visual-studio-code>`."
+
+
+6. Verify your CodeQL CLI setup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CodeQL CLI has subcommands you can execute to verify that you are correctly set
+up to create and analyze databases:
+
+- Run ``codeql resolve languages`` to show which languages are
+  available for database creation. This will list the languages supported by
+  default in your CodeQL CLI package. 
+- Run ``codeql resolve qlpacks`` to show which QL packs the CLI can find. This
+  will display the names of the QL packs included in the CodeQL repositories:
+  ``codeql-cpp``, ``codeql-csharp``, ``codeql-go``,
+  ``codeql-java``, ``codeql-javascript``, and ``codeql-python``. The CodeQL
+  repositories also contain 'upgrade' packs and 'legacy' packs. Upgrade packs
+  are used by the CLI when you want to upgrade a database so that it can be
+  analyzed with a newer version of the CodeQL toolchain than was used to create
+  it. Legacy packs ensure that custom queries and libraries created using older
+  products are compatible with your version of CodeQL.
+
+
+.. _using-two-versions-of-the-codeql-cli:
+
+Using two versions of the CodeQL CLI
+------------------------------------
+
+If you want to use the latest CodeQL features to execute queries or CodeQL tests,
+but also want to prepare databases that are compatible with a specific version of
+LGTM Enterprise, you may need to install two versions of the CLI. The
+recommended directory setup depends on which versions you want to install:
+
+- If both versions are 2.0.2 (or newer), you can unpack both CLI archives in the
+  same parent directory.
+
+- If at least one of the versions is 2.0.1 (or older), the unpacked CLI archives cannot
+  be in the same parent directory, but they can share the same grandparent
+  directory. For example, if you unpack version 2.0.2 into
+  ``$HOME/codeql-home/codeql-cli``, the older version should be
+  unpacked into ``$HOME/codeql-older-version/old-codeql-cli``. Here, the common
+  grandparent is the ``$HOME`` directory. 

docs/codeql/codeql-cli/using-the-codeql-cli/index.rst

@@ -0,0 +1,60 @@
+.. _using-the-codeql-cli:
+
+Using the CodeQL CLI
+====================
+
+.. include:: ../../reusables/codeql-cli-overview.rst
+
+See the following links to learn how to get set up and run CodeQL commands:
+
+- :doc:`About the CodeQL CLI <about-the-codeql-cli>`: Software 
+  developers and security researchers can secure their code using the CodeQL CLI.
+
+- :doc:`Getting started with the CodeQL CLI
+  <getting-started-with-the-codeql-cli>`: Set up the
+  CodeQL CLI so that you can run CodeQL processes from your command line.
+
+- :doc:`Creating CodeQL databases
+  <creating-codeql-databases>`: Create relational
+  representations of source code that can be queried like any other database.
+  
+- :doc:`Analyzing CodeQL databases with the CodeQL CLI
+  <analyzing-databases-with-the-codeql-cli>`: Analyze your code using queries
+  written in a specially-designed, object-oriented query language.
+
+- :doc:`Upgrading CodeQL databases
+  <upgrading-codeql-databases>`: Upgrade your databases so
+  that they can be analyzed using the most up to date CodeQL products.
+
+- :doc:`Using custom queries with the CodeQL CLI
+  <using-custom-queries-with-the-codeql-cli>`: Use custom queries to extend your
+  analysis or highlight errors that are specific to a particular codebase.
+
+- :doc:`Creating CodeQL query suites <creating-codeql-query-suites>`:
+  Define query suite definitions for groups of frequently used queries.
+
+- :doc:`Testing custom queries <testing-custom-queries>`: Set up
+  regression testing of custom queries to ensure that they behave as expected in
+  your analysis.
+
+- :doc:`Testing query help files <testing-query-help-files>`:
+  Test query help files by rendering them as markdown to ensure they are valid 
+  before adding them to the CodeQL repository or using them in code scanning.
+
+- :doc:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`: 
+  You can save default or frequently used options for your commands in a per-user configuration file.
+
+.. toctree::
+   :titlesonly:
+   :hidden:
+
+   about-the-codeql-cli
+   getting-started-with-the-codeql-cli
+   creating-codeql-databases
+   analyzing-databases-with-the-codeql-cli
+   upgrading-codeql-databases
+   using-custom-queries-with-the-codeql-cli
+   creating-codeql-query-suites
+   testing-custom-queries
+   testing-query-help-files
+   Specifying command options <specifying-command-options-in-a-codeql-configuration-file>

docs/codeql/codeql-cli/using-the-codeql-cli/specifying-command-options-in-a-codeql-configuration-file.rst

@@ -0,0 +1,72 @@
+.. _specifying-command-options-in-a-codeql-configuration-file:
+
+Specifying command options in a CodeQL configuration file
+=========================================================
+
+You can save default or frequently used options for your commands in a per-user
+configuration file.
+
+You can specify CodeQL CLI command options in two ways:
+
+- Directly in the command line, using the appropriate flag. 
+- In a configuration (or ``config``) file that CodeQL scans for relevant
+  options each time a command is executed.
+
+For options that are likely to change each time you execute a command,
+specifying the value on the command line is the most convenient way of passing
+the information to CodeQL. Saving options in a ``config`` file is a good way to
+specify options you use frequently. 
+It's also a good way to add custom QL packs that you use regularly to your search path.
+
+Using a CodeQL configuration file
+---------------------------------
+
+You need to save the ``config`` file under your home (Linux and macOS) or user profile
+(Windows) directory in the ``.config/codeql/`` subdirectory.
+For example, ``$HOME/.config/codeql/config``.
+
+The syntax for specifying options is as follows::
+
+   <command> <subcommand> <option> <value>
+
+To apply the same options to more than one command you can:
+
+- Omit the ``<subcommand>``, which will specify the option for every
+  ``<subcommand>`` to which it's relevant.
+- Omit both ``<command>`` and ``<subcommand>``, which will globally specify the
+  option for every ``<command>`` and ``<subcommand>`` to which it's relevant.
+
+.. pull-quote::
+
+   Note
+  
+   - ``config`` files only accept spaces between between option flags and
+     values---CodeQL will throw an error if you use ``=`` to specify an option value.
+   - If you specify an option in the command line, this overrides the ``config``
+     value defined for that option.
+   - If you want to specify more than one option for a ``<command>``,
+     ``<subcommand>`` or globally, use one line per option.
+
+
+Examples
+~~~~~~~~
+
+- To output all analysis results generated by ``codeql database analyze`` as
+  CSV format, you would specify::
+
+     database analyze --format csv
+
+  Here, you have to specify the command and subcommand to prevent any of the
+  low-level commands that are executed during ``database analyze`` being passed
+  the same ``--format`` option.
+
+- To define the RAM (4096 MB) and number of threads (4) to use when running
+  CodeQL commands, specify the following, on separate lines::
+
+      --ram 4096
+      --threads 4
+
+- To globally specify a directory for CodeQL to scan for QL packs (which is not a
+  sibling of the installation directory), use::
+
+     --search-path <path-to-directory>

docs/codeql/codeql-cli/using-the-codeql-cli/testing-custom-queries.rst

@@ -0,0 +1,282 @@
+.. _testing-custom-queries:
+
+Testing custom queries
+======================
+
+CodeQL provides a simple test framework for automated regression testing
+of queries. Test your queries to ensure that they behave as expected.
+
+During a query test, CodeQL compares the results the user expects
+the query to produce with those actually produced. If the expected and
+actual results differ, the query test fails. To fix the test, you should iterate
+on the query and the expected results until the actual results and the expected
+results exactly match. This topic shows you how to create test files and execute
+tests on them using the ``test run`` subcommand.
+
+.. container:: toggle
+
+   .. container:: name
+
+      **Information for LGTM Enterprise 1.23 users**
+
+   CodeQL tests are a new feature in CodeQL CLI version 2.0.2.
+   
+   If you are an LGTM Enterprise 1.23 user, you should still use CodeQL CLI
+   version 2.0.1 to prepare databases to upload to your instance of LGTM. You
+   can use version 2.0.2 (or newer) to test your custom queries, but databases
+   created with versions newer than 2.0.1 are not compatible with LGTM
+   Enterprise 1.23. For more information, see ":ref:`Getting started with the CodeQL CLI <using-two-versions-of-the-codeql-cli>`."
+
+Setting up a test QL pack for custom queries
+--------------------------------------------
+
+All CodeQL tests must be stored in a special "test" QL pack. 
+That is, a directory for test files with a ``qlpack.yml``
+file that defines:
+
+.. code-block:: yaml
+
+   name: <name-of-test-pack>
+   version: 0.0.0
+   libraryPathDependencies: <codeql-libraries-and-queries-to-test>
+   extractor: <language-of-code-to-test>
+
+The ``libraryPathDependencies`` value specifies the CodeQL queries to test.
+The ``extractor`` defines which language the CLI will use 
+to create test databases from the code files stored in this QL pack.
+For more information, see ":doc:`About QL packs <../codeql-cli-reference/about-ql-packs>`."
+
+You may find it useful to look at the way query tests are organized in the 
+`CodeQL repository <https://github.com/github/codeql>`__. 
+Each language has a ``src`` directory, ``ql/<language>/ql/src``, 
+that contains libraries and queries for analyzing codebases. 
+Alongside the ``src`` directory,
+there's a ``test`` directory with tests for these libraries and queries.
+
+Each ``test`` directory is configured as a test QL pack with two subdirectories:
+
+- ``query-tests`` a series of subdirectories with tests for queries stored in the ``src`` directory.
+  Each subdirectory contains test code and a QL reference file that specifies the query to test.
+- ``library-tests`` a series of subdirectories with tests for QL library files.
+  Each subdirectory contains test code and queries that were written as unit tests for a library.
+
+Setting up the test files for a query
+-------------------------------------
+
+For each query you want to test, you should create a sub-directory in the test QL pack.
+Then add the following files to the subdirectory before you run the test command:
+
+- A query reference file (``.qlref`` file) defining the location of the query to test.
+  The location is defined relative to the root of the QL pack that contains the
+  query. Usually, this is a QL pack specified by the
+  ``libraryPathDependencies`` for the test pack.
+  For more information, see ":doc:`Query reference files <../codeql-cli-reference/query-reference-files>`."
+
+  You don't need to add a query reference file if the query you want to
+  test is stored in the test directory,
+  but it's generally good practice to store queries separately from tests.
+  The only exception is unit tests for QL libraries, which tend to be
+  stored in test packs, separate from queries that generate alerts or paths.
+
+- The example code you want to run your query against. This
+  should consist of one or more files containing examples of the code the
+  query is designed to identify.
+
+You can also define the results you expect to see when you run the query against 
+the example code, by creating a file with the extension ``.expected``.
+Alternatively, you can leave the test command to create the ``.expected`` file
+for you. (If you're using CodeQL CLI 2.0.2--2.0.6, you need to create an
+``.expected`` file otherwise the command will fail to find your test query.
+You can create an empty ``.expected`` file to workaround this limitation.)
+
+For an example showing how to create and test a query, see the `example
+<#example>`__ below. 
+
+.. pull-quote:: Important
+
+   Your ``.ql``, ``.qlref``, and ``.expected`` files must have consistent names.
+
+   If you want to directly specify the ``.ql`` file itself in the test command,
+   it must have the same base name as the corresponding ``.expected`` file. For
+   example, if the query is ``MyJavaQuery.ql``, the expected results file must
+   be ``MyJavaQuery.expected``.
+   
+   If you want to specify a ``.qlref`` file in the command, it must have the same base
+   name as the corresponding ``.expected`` file, but the query itself
+   may have a different name. 
+
+   The names of the example code files don't have to be consistent with the
+   other test files. All example code files found next to the ``.qlref`` (or ``.ql``)
+   file and in any subdirectories will be used to create a test database. 
+   Therefore, for simplicity, we recommend you don't save test files in
+   directories that are ancestors of each other.
+
+Running ``codeql test run``
+---------------------------
+
+CodeQL query tests are executed by running the following command::
+
+   codeql test run <test|dir>
+
+The ``<test|dir>`` argument can be one or more of the following:
+
+- Path to a ``.ql`` file.
+- Path to a ``.qlref`` file that references a ``.ql`` file.
+- Path to a directory that will be searched recursively for ``.ql`` and  
+  ``.qlref`` files.
+
+You can also specify: 
+
+- .. include:: ../../reusables/threads-query-execution.rst
+
+For full details of all the options you can use when testing queries,
+see the `test run reference documentation <../codeql-cli-manual/test-run.html>`__.
+
+Example
+-------
+
+The following example shows you how to set up a test for a query that searches
+Java code for ``if`` statements that have empty ``then`` blocks. It includes
+steps to add the custom query and corresponding test files to separate QL packs
+outside your checkout of the CodeQL repository. This ensures when you update the
+CodeQL libraries, or check out a different branch, you won't overwrite your
+custom queries and tests.
+
+Prepare a query and test files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Develop the query. For example, the following simple query finds empty ``then``
+   blocks in Java code:
+
+   .. code-block:: ql
+
+      import java
+
+      from IfStmt ifstmt
+      where ifstmt.getThen() instanceof EmptyStmt
+      select ifstmt, "This if statement has an empty then."
+
+#. Save the query to a file named ``EmptyThen.ql`` in a directory with your
+   other custom queries. For example,
+   ``custom-queries/java/queries/EmptyThen.ql``. 
+   
+#. If you haven't already added your custom queries to a QL pack,
+   create a QL pack now. For example, if your custom Java queries
+   are stored in ``custom-queries/java/queries``, add a ``qlpack.yml`` file with the
+   following contents to ``custom-queries/java/queries``:
+
+   .. code-block:: yaml
+
+      name: my-custom-queries
+      version: 0.0.0
+      libraryPathDependencies: codeql-java
+
+   For more information about QL packs, see ":doc:`About QL packs
+   <../codeql-cli-reference/about-ql-packs>`."
+
+#. Create a QL pack for your Java tests by adding a ``qlpack.yml`` file
+   with the following contents to ``custom-queries/java/tests``,
+   updating ``libraryPathDependencies`` to match the name of your QL pack of custom queries:
+
+   .. include:: ../../reusables/test-qlpack.rst
+
+#. Within the Java test pack, create a directory to contain the test files
+   associated with ``EmptyThen.ql``.
+   For example, ``custom-queries/java/tests/EmptyThen``. 
+
+#. In the new directory, create ``EmptyThen.qlref`` to define the location of ``EmptyThen.ql``. 
+   The path to the query must be specified relative to the root of
+   the QL pack that contains the query. In this case, the query is in the 
+   top level directory of the QL pack named ``my-custom-queries``, 
+   which is declared as a dependency for ``my-query-tests``.
+   Therefore, ``EmptyThen.qlref`` should simply contain ``EmptyThen.ql``.
+
+#. Create a code snippet to test. The following Java code contains an
+   empty ``if`` statement on the third line. Save it in
+   ``custom-queries/java/tests/EmptyThen/Test.java``.
+
+   .. code-block:: java
+
+      class Test {
+        public void problem(String arg) {
+          if (arg.isEmpty())
+          ;
+          {
+              System.out.println("Empty argument");
+          }
+        }
+        
+        public void good(String arg) {
+          if (arg.isEmpty()) {
+              System.out.println("Empty argument");
+          }
+        }
+      }
+
+Execute the test
+^^^^^^^^^^^^^^^^
+
+To execute the test, move into the ``custom-queries`` directory and run ``codeql
+test run java/tests/EmptyThen``.
+
+When the test runs it:
+
+#. Finds one test in the ``EmptyThen`` directory.
+#. Extracts a CodeQL database from the ``.java`` files stored in the ``EmptyThen`` directory.
+#. Compiles the query referenced by the ``EmptyThen.qlref`` file.
+
+   If this step fails, it's because the CLI can't find your custom QL pack.
+   Re-run the command and specify the location of your custom QL pack, for example:
+
+   ``codeql test run --search-path=java java/tests/EmptyThen``
+
+   For information about saving the search path as part of your configuration, see
+   ":ref:`Specifying command options in a CodeQL configuration file <specifying-command-options-in-a-codeql-configuration-file>`."
+#. Executes the test by running the query and generating an ``EmptyThen.actual`` results file.
+#. Checks for an ``EmptyThen.expected`` file to compare with the ``.actual`` results file.
+#. Reports the results of the test --- in this case, a failure: ``0 tests passed; 1 tests failed:``.
+   The test failed because we haven't yet added a file with the expected results of the query.
+
+View the query test output
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+CodeQL generates the following files in the ``EmptyThen`` directory:
+
+- ``EmptyThen.actual``, a file that contains the actual results generated by the
+  query.
+- ``EmptyThen.testproj``, a test database that you can load into VS Code and use to debug failing tests.
+  When tests complete successfully, this database is deleted in a housekeeping step.
+  You can override this step by running ``test run`` with the ``--keep-databases`` option.
+
+In this case, the failure was expected and is easy to fix. 
+If you open the ``EmptyThen.actual`` file, you can see the results of the test:
+
+   .. code-block:: none
+      
+      | Test.java:3:5:3:22 | stmt | This if statement has an empty then. |
+
+This file contains a table, with a column for the location of the result, 
+along with separate columns for each part of the ``select`` clause the query outputs.
+Since the results are what we expected, we can update the file extension to define
+this as the expected result for this test (``EmptyThen.expected``).
+
+If you rerun the test now, the output will be similar but it will finish by reporting:
+``All 1 tests passed.``.
+
+If the results of the query change, for example, if you revise the ``select`` statement for the query,
+the test will fail. For failed results, the CLI output includes a unified diff of the
+``EmptyThen.expected`` and ``EmptyThen.actual`` files.
+This information may be sufficient to debug trivial test failures. 
+
+For failures that are harder to debug, you can import ``EmptyThen.testproj``
+into CodeQL for VS Code, execute ``EmptyThen.ql``, and view the results in the
+``Test.java`` example code. For more information, see ":ref:`Analyzing your projects
+<analyzing-your-projects>`" in the CodeQL for VS Code
+help.  
+
+Further reading
+---------------
+
+- ":ref:`CodeQL queries
+  <codeql-queries>`"
+- ":ref:`Testing CodeQL queries in Visual Studio Code <testing-codeql-queries-in-visual-studio-code>`"

docs/codeql/codeql-cli/using-the-codeql-cli/testing-query-help-files.rst

@@ -0,0 +1,81 @@
+.. _testing-query-help-files:
+
+Testing query help files
+========================
+
+Test query help files by rendering them as markdown to ensure they are valid 
+before uploading them to the CodeQL repository or using them in code scanning.
+
+Query help is documentation that accompanies a query to explain how the query works,
+as well as providing information about the potential problem that the query identifies.
+It is good practice to write query help for all new queries. For more information,
+see `Contributing to CodeQL <https://github.com/github/codeql/blob/main/CONTRIBUTING.md>`__
+in the CodeQL repository.
+
+The CodeQL CLI includes a command to test query help and render the content as 
+markdown, so that you can easily preview the content in your IDE. Use the command to validate 
+query help files before uploading them to the CodeQL repository or sharing them with other users. 
+
+Prerequisites
+-------------
+
+- The query help (``.qhelp``) file must have an accompanying query (``.ql``) file with 
+  an identical base name.
+- The query help file should follow the standard structure and style for query help documentation.
+  For more information, see the `Query help style guide <https://github.com/github/codeql/blob/main/docs/query-help-style-guide.md>`__ 
+  in the CodeQL repository. 
+
+Running ``codeql generate query-help``
+--------------------------------------
+
+You can test query help files by running the following command::
+
+   codeql generate query-help <qhelp|query|dir|suite> --format=<format> [--output=<dir|file>] 
+
+where ``<qhelp|query|dir|suite>`` is one of:
+
+- the path to a ``.qhelp`` file.
+- the path to a ``.ql`` file.
+- the path to a directory containing queries and query help files.
+- the path to a query suite, or the name of a well-known query suite for a QL pack. 
+  For more information, see "`Creating CodeQL query suites <creating-codeql-query-suites.html#specifying-well-known-query-suites>`__."
+
+You must specify a ``--format`` option, which defines how the query help is rendered. 
+Currently, you must specify ``markdown`` to render the query help as markdown. 
+
+The ``--output`` option defines a file path where the rendered query help will be saved.
+
+- For directories containing ``.qhelp`` files or a query suites 
+  defining one or more ``.qhelp`` files, you must specify an ``--output`` directory. 
+  Filenames within the output directory will be derived from the ``.qhelp`` file names. 
+
+- For single ``.qhelp`` or ``.ql`` files, you may specify an ``--output`` option.
+  If you don't specify an output path, the rendered query help is written to ``stdout``.
+
+For full details of all the options you can use when testing query help files,
+see the `generate query-help reference documentation
+<codeql-cli-manual:generate-query-help>`__.
+
+Results
+-------
+
+When you run the command, CodeQL attempts to render  
+each ``.qhelp`` file that has an accompanying ``.ql`` file. For single files, the rendered
+content will be printed to ``stdout`` if you don't specify an ``--output`` option. For all other 
+use cases, the rendered content is saved to the specified output path. 
+
+By default, the CodeQL CLI will print a warning message if:
+
+- Any of the query help is invalid, along with a description of the invalid query help elements
+- Any ``.qhelp`` files specified in the command don't have the same base name 
+  as an accompanying ``.ql`` file
+- Any ``.ql`` files specified in the command don't have the same base name
+  as an accompanying ``.qhelp`` file
+
+You can tell the CodeQL CLI how to handle these warnings by including a ``--warnings`` option in your command.
+For more information, see the `generate query-help reference documentation <../codeql-cli-manual/generate-query-help.html#cmdoption-codeql-generate-query-help-warnings>`__.
+
+Further reading
+---------------
+
+- ":ref:`Query help files <query-help-files>`"

docs/codeql/codeql-cli/using-the-codeql-cli/upgrading-codeql-databases.rst

@@ -0,0 +1,54 @@
+.. _upgrading-codeql-databases:
+
+Upgrading CodeQL databases
+==========================
+
+As the CodeQL CLI tools and queries evolve, you may find that some of your
+CodeQL databases become out of date. You must upgrade out-of-date databases 
+before you can analyze them.
+
+Databases become out of date when:
+
+- For databases created using the CodeQL CLI, the version of CLI tools used to
+  create them is older than your copy of the CodeQL queries. 
+- For databases downloaded from LGTM.com, the CodeQL tools used by LGTM.com to create
+  that revision of the code are older than your copy of the CodeQL queries.
+
+The ``main`` branch of the CodeQL queries is updated more often than both the
+CLI and LGTM.com, so databases are most likely to become out of date if you use
+the queries on this branch. For more information about the different versions of
+the CodeQL queries, see ":ref:`Getting started with the CodeQL CLI <local-copy-codeql-queries>`." 
+
+Out-of-date databases must be upgraded before they can be analyzed. This topic
+shows you how to upgrade a CodeQL database using the ``database upgrade``
+subcommand.
+
+Prerequisites
+-------------
+
+Archived databases downloaded from LGTM.com must be unzipped before they are
+upgraded.
+
+Running ``codeql database upgrade``
+-----------------------------------
+
+CodeQL databases are upgraded by running the following command::
+
+   codeql database upgrade <database>
+
+where ``<database>``, the path to the CodeQL database you
+want to upgrade, must be specified.
+
+For full details of all the options you can use when upgrading databases,
+see the "`database upgrade <../codeql-cli-manual/database-upgrade.html>`__"  reference documentation.
+
+Progress and results
+--------------------
+
+When you execute the ``database upgrade`` command, CodeQL identifies the version
+of the :ref:`schema <codeql-database-schema>` associated with the database. From
+there, it works out what (if anything) is required to make the database work
+with your queries and libraries. It will rewrite the database, if necessary, or
+make no changes if the database is already compatible (or if it finds no
+information about how to perform an upgrade). Once a database has been upgraded
+it cannot be downgraded for use with older versions of the CodeQL products.

docs/codeql/codeql-cli/using-the-codeql-cli/using-custom-queries-with-the-codeql-cli.rst

@@ -0,0 +1,77 @@
+.. _using-custom-queries-with-the-codeql-cli:
+
+Using custom queries with the CodeQL CLI
+=========================================
+
+You can customize your CodeQL analyses by writing your own queries to highlight
+specific vulnerabilities or errors.
+
+This topic is specifically about writing
+queries to use with the `database analyze <../codeql-cli-manual/database-analyze.html>`__
+command to produce :ref:`interpreted results <interpret-query-results>`. 
+
+.. include:: ../../reusables/advanced-query-execution.rst
+
+Writing a valid query
+---------------------
+
+Before running a custom analysis you need to write a valid query, and save it in
+a file with a ``.ql`` extension. There is extensive documentation available to
+help you write queries. For more information, see ":ref:`CodeQL queries
+<codeql-queries>`."
+
+.. _including-query-metadata:
+
+Including query metadata
+------------------------
+
+Query metadata is included at the top of each query file. It provides users with information about
+the query, and tells the CodeQL CLI how to process the query results. 
+
+When running queries with the ``database analyze`` command, you must include the
+following two properties to ensure that the results are interpreted correctly:
+
+- Query identifier (``@id``): a sequence of words composed of lowercase letters or
+  digits, delimited by ``/`` or ``-``, identifying and classifying the query. 
+- Query type (``@kind``): identifies the query is an alert (``@kind problem``)
+  or a path (``@kind path-problem``).
+
+For more information about these metadata properties, see ":ref:`Metadata for CodeQL queries
+<metadata-for-codeql-queries>`" and the `Query metadata style guide
+<https://github.com/github/codeql/blob/main/docs/query-metadata-style-guide.md>`__.
+
+.. pull-quote:: Note
+
+   Metadata requirements may differ if you want to use your query with other
+   applications. For more information, see ":ref:`Metadata for CodeQL queries
+   <metadata-for-codeql-queries>`
+   ."  
+
+
+Creating a custom QL pack
+-------------------------
+
+When writing your own queries, you should save them in a custom QL pack
+directory. QL packs provide a way of organizing
+the files used in CodeQL analysis. This directory must contain a file
+named ``qlpack.yml`` at the root. Your custom queries should be saved in the QL
+pack root, or its subdirectories.
+
+For each QL pack, the ``qlpack.yml`` file includes information that tells CodeQL
+how to compile the queries, what libraries the pack depends on, and where to find
+query suite definitions. For more information about what to include in this
+file, see ":ref:`About QL packs <custom-ql-packs>`."
+
+Contributing to the CodeQL repository
+-------------------------------------
+
+If you would like to share your query with other CodeQL users, you can open a
+pull request in the `CodeQL repository <https://github.com/github/codeql>`__. For
+further information, see `Contributing to CodeQL
+<https://github.com/github/codeql/blob/main/CONTRIBUTING.md>`__.
+
+Further reading
+---------------
+
+- ":ref:`CodeQL queries
+  <codeql-queries>`"

docs/codeql/codeql-for-visual-studio-code/.gitignore

@@ -0,0 +1,5 @@
+# VSCode rst extension build directory
+_build/
+
+# VSCode rst extension settings
+.vscode

docs/codeql/codeql-for-visual-studio-code/about-codeql-for-visual-studio-code.rst

@@ -0,0 +1,31 @@
+.. _about-codeql-for-visual-studio-code:
+
+About CodeQL for Visual Studio Code
+=================================================
+
+CodeQL for Visual Studio Code is an extension that lets you write, run, and test CodeQL queries in Visual Studio Code.
+
+Features
+----------
+
+CodeQL for Visual Studio Code provides an easy way to run queries from the large, open source repository of `CodeQL security queries <https://github.com/github/codeql>`__.
+With these queries, or your own custom queries, you can analyze databases generated from source code to find errors and security vulnerabilities.
+The Results view shows the flow of data through the results of path queries, which is essential for triaging security results.
+
+The CodeQL extension also adds a **CodeQL** sidebar view to VS Code. This contains a list of databases, and an overview of the queries that you have run in the current session.
+
+The extension provides standard `IntelliSense <https://code.visualstudio.com/docs/editor/intellisense>`__
+features for query files (extension ``.ql``) and library files (extension ``.qll``) that you open in the Visual Studio Code editor.
+
+- Syntax highlighting
+- Right-click options (such as **Go To Definition**)
+- Autocomplete suggestions
+- Hover information
+
+You can also use the VS Code **Format Document** command to format your code according to the `CodeQL style guide <https://github.com/github/codeql/blob/main/docs/ql-style-guide.md>`__.
+
+Further reading
+-------------------
+
+- ":doc:`Setting up CodeQL in Visual Studio Code <setting-up-codeql-in-visual-studio-code>`"
+- ":doc:`Analyzing your projects <analyzing-your-projects>`"

docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst

@@ -0,0 +1,155 @@
+.. _analyzing-your-projects:
+
+Analyzing your projects
+=================================================
+
+You can run queries on CodeQL databases and view the results in Visual Studio Code.
+
+Choosing a database
+------------------------
+
+To analyze a project, you need to add a :ref:`CodeQL database <codeql-database>` for that project.
+
+#. Open the CodeQL Databases view in the sidebar.
+
+#. Hover over the **Databases** title bar and click the appropriate icon to add your database. You can add a database from a local ZIP archive or folder, from a public URL, or from a project URL on LGTM.com.
+
+   .. image:: ../images/codeql-for-visual-studio-code/choose-database.png
+      :width: 350
+      :alt: Choose a database to analyze
+   
+   For more information about obtaining a local database, see below.
+
+#. Once you've chosen a database, it is displayed in the Databases view. To see the menu options for interacting with a database, right-click an entry in the list. You can select multiple databases using **Ctrl/Cmd+click**.
+
+Obtaining a local database
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP file, you can add it to Visual Studio Code. There are several ways to obtain a local CodeQL database. 
+
+- To create a database with the CodeQL CLI, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`."
+
+- .. include:: ../reusables/download-lgtm-database.rst
+
+- To analyze a test database, add a ``.testproj`` folder to the Databases view.
+  Test databases (that is, folders with a ``.testproj`` extension) are generated when you run regression tests on custom queries using the :ref:`CodeQL CLI <codeql-cli>`.
+  If a query fails a regression test, you may want to analyze the test database in Visual Studio Code to debug the failure.
+   
+  For more information about running query tests, see ":ref:`Testing custom queries <testing-custom-queries>`" in the CodeQL CLI help.  
+
+Running a query
+------------------------
+
+The `CodeQL repository <https://github.com/github/codeql>`__ on GitHub contains lots of example queries.
+If you have that folder (or a different QL pack) available in your workspace, you can access existing queries under ``<language>/ql/src/<category>``, for example ``java/ql/src/Likely Bugs``.
+
+#. Open a query (``.ql``) file. It is displayed in the editor, with IntelliSense features such as syntax highlighting and autocomplete suggestions.
+#. Right-click in the query window and select **CodeQL: Run Query**. (Alternatively, run the command from the Command Palette.)
+
+The CodeQL extension runs the query on the current database and reports progress in the bottom right corner of the application.
+When the results are ready, they're displayed in the Results view.
+
+If there are any problems running a query, a notification is displayed in the bottom right corner of the application.
+In addition to the error message, the notification includes details of how to fix the problem.
+For more information, see ":doc:`Troubleshooting CodeQL for Visual Studio Code <troubleshooting-codeql-for-visual-studio-code>`."
+
+Running multiple queries
+--------------------------
+
+You can run multiple queries with a single command.
+
+#. Go to the File Explorer.
+#. Select multiple files or folders that contain queries.
+#. Right-click and select **CodeQL: Run Queries in Selected Files**.
+
+   .. image:: ../images/codeql-for-visual-studio-code/run-multiple-queries.png
+      :width: 350
+      :alt: Run multiple queries from the File Explorer
+
+Running a quick query
+------------------------
+
+When working on a new query, you can open a "quick query" tab to easily execute your code and view the results, without having to save a ``.ql`` file in your workspace.
+Open a quick query editing tab by selecting **CodeQL: Quick Query** from the Command Palette.
+To run the query, use **CodeQL: Run Query**.
+
+You can see all quick queries that you've run in the current session in the Query History view. Click an entry to see the exact text of the quick query that produced the results.
+
+Once you're happy with your quick query, you should save it in a QL pack so you can access it later. For more information, see ":ref:`About QL packs <about-ql-packs>`."
+
+Running a specific part of a query or library
+----------------------------------------------
+
+This is helpful if you're debugging a query or library and you want to locate the part that is wrong.
+Instead of using **CodeQL: Run Query** to run the whole query (the :ref:`select clause <select-clauses>` and any :ref:`query predicates <query-predicates>`), you can use **CodeQL: Quick Evaluation** to run a specific part of a ``.ql`` or ``.qll`` file.
+
+**CodeQL: Quick Evaluation** evaluates a code snippet (instead of the whole query) and displays results of that selection in the Results view. 
+Possible targets for quick evaluation include:
+
+- Selecting the name of a CodeQL entity (such as a :ref:`class <classes>` or :ref:`predicate <predicates>`) to evaluate that entity.
+- Selecting a :ref:`formula <formulas>` or :ref:`expression <expressions>` with free variables to evaluate that formula or expression.
+
+For example, in the following snippet, you could select the predicate name ``foo`` or the formula ``s = "bar"`` for quick evaluation.
+
+.. code-block:: ql
+
+   predicate foo(string s) { s = "bar" }
+
+Viewing previous queries
+--------------------------
+
+To see the queries that you have run in the current session, open the Query History view.
+
+   .. image:: ../images/codeql-for-visual-studio-code/query-history.png
+      :width: 350
+      :alt: See a list of previous queries
+
+The Query History contains information including the date and time when the query was run, the name of the query, the database on which it was run, and how long it took to run the query.
+To customize the information that is displayed, right-click an entry and select **Set Label**.
+
+Click an entry to display the corresponding results in the Query Results view, and double-click
+to display the query itself in the editor (or right-click and select **Open Query**).
+To display the exact text that produced the results for a particular entry, right-click it and select **Show Query Text**. This can differ from **Open Query** as the query file may have been modified since you last ran it.
+
+To remove queries from the Query History view, select all the queries you want to remove, then right-click and select **Remove History Item**.
+
+.. _viewing-query-results:
+
+Viewing query results
+-----------------------
+
+#. Click a query in the Query History view to display its results in the Results view. Alternatively, right-click the result to select a different view:
+
+   - To view the results in :ref:`SARIF format <sarif-output>`, right-click and select **View SARIF**.
+   - To view the results in :ref:`DIL format <dil>`, right-click and select **View DIL**.
+
+#. Use the dropdown menu in the Results view to choose which results to display, and in what form to display them, such as a formatted alert message or a table of raw results. The available output forms are specified by the format of the query and the metadata. For more information, see ":ref:`CodeQL queries <codeql-queries>`."
+
+#. To sort the results by the entries in a particular column, click the column header.
+
+If a result links to a source code element, you can click it to display it in the source.
+
+To use standard code navigation features in the source code, you can right-click an element and use the commands **Go to Definition** or **Go to References**. This runs a CodeQL query over the active file, which may take a few seconds. This query needs to run once for every file, so any additional references from the same file will be fast.
+
+  .. pull-quote:: Note
+
+     If you're using an older database, code navigation commands such as **Go to Definition** and **Go to References** may not work.
+     To use code navigation, try unzipping the database and running ``codeql database cleanup <database>`` on the unzipped database using the CodeQL CLI. Then, re-add the database to Visual Studio Code.
+     For more information, see the `database cleanup <../codeql-cli/codeql-cli-manual/database-cleanup.html>`__ reference documentation.
+
+Comparing query results
+------------------------
+
+When you're writing or debugging a query, it's useful to see how your changes affect the results.
+You can compare two sets of results to see exactly what has changed.
+To compare results, the two queries must be run on the same database.
+
+#. Right-click a query in the Query History view and select **Compare Results**.
+#. A Quick Pick menu shows all valid queries to compare with. Select a query.
+#. The Compare view shows the differences in the results of the two queries.
+
+Further reading
+------------------------
+
+- ":ref:`CodeQL queries <codeql-queries>`"
+- ":doc:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

@@ -0,0 +1,63 @@
+.. _customizing-settings:
+
+Customizing settings
+====================
+
+You can edit the settings for the CodeQL extension to suit your needs.
+
+About CodeQL extension settings
+---------------------------------
+
+The CodeQL extension comes with a number of settings that you can edit. These determine how the extension behaves, including: which version of the CodeQL CLI the extension uses, how the extension displays previous queries, and how it runs queries.
+
+Editing settings
+-----------------
+
+1. Open the Extensions view and right click **CodeQL**.
+
+2. Click **Extension Settings**.
+
+   .. image:: ../images/codeql-for-visual-studio-code/open-extension-settings.png
+      :width: 400
+      :alt: Open the CodeQL extension settings
+
+3. Edit a setting. The new settings are saved automatically.
+
+Choosing a version of the CodeQL CLI
+--------------------------------------
+
+The CodeQL extension uses the CodeQL CLI to run commands. If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from LGTM.com. Otherwise, the extension automatically manages access to the executable of the CLI for you. For more information about creating databases, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`" in the CLI help.
+
+To override the default behavior and use a different CLI, you can specify the CodeQL CLI **Executable Path**. Note that this is only available as a user setting, not as a workspace setting.
+
+Changing the labels of query history items
+--------------------------------------------
+
+The query history **Format** setting controls how the extension lists queries in the query history. By default, each item has a label with the following format::
+    
+    [%t] %q on %d - %s
+  
+- ``%t`` is the time the query was run
+- ``%q`` is the query name
+- ``%d`` is the database name
+- ``%s`` is a status string
+
+To override the default label, you can specify a different format for the query history items.
+
+Configuring settings for running queries
+-----------------------------------------
+
+There are a number of settings for **Running Queries**. If your queries run too slowly and time out frequently, you may want to increase the memory. 
+
+.. include:: ../reusables/running-queries-debug.rst
+
+Configuring settings for testing queries
+-----------------------------------------
+
+To increase the number of threads used for testing queries, you can update the **Running Tests > Number Of Threads** setting.
+
+Further reading
+----------------
+
+- `User and workspace settings <https://code.visualstudio.com/docs/getstarted/settings>`__ in the Visual Studio Code help
+- ":ref:`CodeQL CLI <codeql-cli>`"

docs/codeql/codeql-for-visual-studio-code/exploring-data-flow-with-path-queries.rst

@@ -0,0 +1,35 @@
+.. _exploring-data-flow-with-path-queries:
+
+Exploring data flow with path queries
+=================================================
+
+You can run CodeQL queries in VS Code to help you track the flow of data through a program, highlighting areas that are potential security vulnerabilities.
+
+About path queries
+--------------------
+
+A path query is a CodeQL query with the property ``@kind path-problem``. 
+You can find a number of these in the standard CodeQL libraries, for example, a security query that finds cross-site scripting vulnerabilities in Java projects:
+`Cross-site scripting <https://github.com/github/codeql/blob/main/java/ql/src/Security/CWE/CWE-079/XSS.ql>`__.
+
+You can run the standard CodeQL path queries to identify security vulnerabilities and manually look through the results.
+You can also modify the existing queries to model data flow more precisely for the specific framework of your project, or write completely new path queries to find a different vulnerability.
+
+To ensure that your path query uses the correct format and metadata, follow the instructions in ":ref:`Creating path queries <creating-path-queries>`."
+This topic also contains detailed information about how to define new sources and sinks, as well as templates and examples of how to extend the CodeQL libraries to suit your analysis.
+
+Running path queries in VS Code
+-----------------------------------
+
+#. Open a path query in the editor.
+#. Right-click in the query window and select **CodeQL: Run Query**. (Alternatively, run the command from the Command Palette.)
+#. Once the query has finished running, you can see the results in the Results view as usual (under ``alerts`` in the dropdown menu). Each query result describes the flow of information between a source and a sink.
+#. Expand the result to see the individual steps that the data follows. 
+#. Click each step to jump to it in the source code and investigate the problem further.
+#. To navigate the path from your keyboard, you can bind shortcuts to the **CodeQL: Show Previous Step on Path** and **CodeQL: Show Next Step on Path** commands.
+
+Further reading
+-----------------
+
+- ":ref:`About data flow analysis <about-data-flow-analysis>`"
+- ":ref:`Creating path queries <creating-path-queries>`"

docs/codeql/codeql-for-visual-studio-code/exploring-the-structure-of-your-source-code.rst

@@ -0,0 +1,37 @@
+.. _exploring-the-structure-of-your-source-code:
+
+Exploring the structure of your source code
+=================================================
+
+You can use the AST viewer to display the abstract syntax tree of a CodeQL database.
+
+About the abstract syntax tree
+-------------------------------
+
+The abstract syntax tree (AST) of a program represents the program's syntactic structure. Nodes on the AST represent elements such as statements and expressions.
+A CodeQL database encodes these program elements and the relationships between them through a :ref:`database schema <codeql-database-schema>`.
+
+CodeQL for Visual Studio Code contains an AST viewer. The viewer consists of a graph visualization view that lets you explore the AST of a file in a CodeQL database. This helps you see which CodeQL classes correspond to which parts of your source files.
+
+Viewing the abstract syntax tree of a source file
+--------------------------------------------------
+
+1. Open a source file from a CodeQL database. For example, you can navigate to a source file in the File Explorer.
+
+   .. image:: ../images/codeql-for-visual-studio-code/open-source-file.png
+      :width: 350
+      :alt: Open a source file
+
+2. Run **CodeQL: View AST** from the Command Palette. This runs a CodeQL query (usually called ``printAST.ql``) over the active file, which may take a few seconds.
+   
+   .. pull-quote:: Note
+
+      If you don't have an appropriate ``printAST.ql`` query in your workspace, the **CodeQL: View AST** command won't work. To fix this, you can update your copy of the `CodeQL <https://github.com/github/codeql>`__ repository (or `CodeQL for Go <https://github.com/github/codeql-go>`__ repository) from ``main``. If you do this, you may need to upgrade your databases. Also, query caches may be discarded and your next query runs could be slower.
+
+3. Once the query has run, the AST viewer displays the structure of the source file.
+4. To see the nested structure, click the arrows and expand the nodes.
+
+   .. image:: ../images/codeql-for-visual-studio-code/explore-ast.png
+      :alt: Explore the AST
+
+You can click a node in the AST viewer to jump to it in the source code. Conversely, if you click a section of the source code, the AST viewer displays the corresponding node.

docs/codeql/codeql-for-visual-studio-code/index.rst

@@ -0,0 +1,55 @@
+.. _codeql-for-visual-studio-code:
+
+CodeQL for Visual Studio Code
+=============================
+
+The CodeQL extension for Visual Studio Code adds rich language support for CodeQL and allows you to easily find problems in codebases.
+
+See the following help topics for more information:
+
+- :doc:`About CodeQL for Visual Studio Code
+  <about-codeql-for-visual-studio-code>`: CodeQL for Visual Studio
+  Code is an extension that lets you write, run, and test CodeQL queries in Visual
+  Studio Code.
+
+- :doc:`Setting up CodeQL in Visual Studio Code
+  <setting-up-codeql-in-visual-studio-code>`: You can install and configure 
+  the CodeQL extension in Visual Studio Code.
+
+- :doc:`Analyzing your projects
+  <analyzing-your-projects>`: You can run queries on CodeQL
+  databases and view the results in Visual Studio Code.
+
+- :doc:`Exploring the structure of your source code
+  <exploring-the-structure-of-your-source-code>`: 
+  You can use the AST viewer to display the abstract syntax tree of a CodeQL database.
+
+- :doc:`Exploring data flow with path queries
+  <exploring-data-flow-with-path-queries>`: You can run CodeQL queries in
+  VS Code to help you track the flow of data through a program, highlighting
+  areas that are potential security vulnerabilities.  
+
+- :doc:`Testing CodeQL queries in Visual Studio Code
+  <testing-codeql-queries-in-visual-studio-code>`: You can run unit tests for
+  CodeQL queries using the Visual Studio Code extension.
+
+- :doc:`Customizing settings
+  <customizing-settings>`: You can edit the settings for the 
+  CodeQL extension to suit your needs.
+
+- :doc:`Troubleshooting CodeQL for Visual Studio Code
+  <troubleshooting-codeql-for-visual-studio-code>`: You can use the detailed 
+  information written to the extension's log files if you need to troubleshoot problems.
+
+.. toctree::
+   :hidden:
+   :titlesonly:
+
+   about-codeql-for-visual-studio-code
+   setting-up-codeql-in-visual-studio-code
+   analyzing-your-projects
+   exploring-the-structure-of-your-source-code
+   exploring-data-flow-with-path-queries
+   testing-codeql-queries-in-visual-studio-code
+   customizing-settings
+   troubleshooting-codeql-for-visual-studio-code

docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst

@@ -0,0 +1,123 @@
+.. _setting-up-codeql-in-visual-studio-code:
+
+Setting up CodeQL in Visual Studio Code
+=================================================
+
+You can install and configure the CodeQL extension in Visual Studio Code.
+
+.. include:: ../reusables/license-note.rst
+
+Prerequisites
+-----------------
+
+The CodeQL extension requires a minimum of Visual Studio Code 1.39. Older versions are not supported.
+
+Installing the extension
+------------------------
+
+You can install the CodeQL extension using any of the normal methods for installing a VS Code extension:
+
+* Go to the `Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-codeql>`__ in your browser and click **Install**.
+* In the Extensions view (**Ctrl+Shift+X** or **Cmd+Shift+X**), search for ``CodeQL``, then select **Install**.
+* Download the `CodeQL VSIX file <https://github.com/github/vscode-codeql/releases>`__. Then, in the Extensions view, click **More actions** > **Install from VSIX**, and select the CodeQL VSIX file.
+
+Configuring access to the CodeQL CLI
+------------------------------------
+
+The extension uses the CodeQL CLI to compile and run queries.
+
+If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from LGTM.com. For more information, see ":ref:`CodeQL CLI <codeql-cli>`."
+
+Otherwise, the extension automatically manages access to the executable of the CLI for you. This ensures that the CLI is compatible with the CodeQL extension. You can also check for updates with the **CodeQL: Check for CLI Updates** command.
+
+.. pull-quote:: Note
+
+   The extension-managed CLI is not accessible from the terminal.
+   If you intend to use the CLI outside of the extension (for example to create databases), we recommend that you install your own copy of the CLI.
+   To avoid having two copies of the CLI on your machine, you can point the CodeQL CLI **Executable Path** setting to your existing copy of the CLI.
+
+If you want the extension to use a specific version of the CLI, set the CodeQL CLI **Executable Path** to the location of the executable file for the CLI.
+That is, the file named ``codeql`` (Linux/Mac) or ``codeql.exe`` (Windows). For more information, see ":ref:`Customizing settings <customizing-settings>`."
+
+If you have any difficulty setting up access to the CodeQL CLI, check the CodeQL Extension Log for error messages. For more information, see ":doc:`Troubleshooting CodeQL for Visual Studio Code <troubleshooting-codeql-for-visual-studio-code>`."
+
+Setting up a CodeQL workspace
+-----------------------------
+
+When you're working with CodeQL, you need access to the standard CodeQL libraries. This also makes a wide variety of queries available to explore.
+
+There are two ways to do this:
+
+* Recommended, use the "starter" workspace. This is maintained as a Git repository which makes it easy to keep up to date with changes to the libraries. For more information, see ":ref:`Using the starter workspace <starter-workspace>`" below.
+* More advanced, add the CodeQL libraries and queries to an existing workspace. For more information, see ":ref:`Updating an existing workspace for CodeQL <existing-workspace>`" below.
+
+.. pull-quote:: Note
+   
+   For CLI users there is a third option: If you have followed the instructions in ":ref:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`" to create a CodeQL directory (for example ``codeql-home``) containing the CodeQL libraries, you can open this directory in VS Code. This also gives the extension access to the CodeQL libraries.
+
+.. container:: toggle
+
+   .. container:: name
+
+      **Click to show information for LGTM Enterprise users**
+
+   Your local version of the CodeQL queries and libraries should match your version of LGTM Enterprise. For example, if you
+   use LGTM Enterprise 1.23, then you should clone the ``1.23.0`` branch of the `starter workspace <https://github.com/github/vscode-codeql-starter/>`__ (or the appropriate ``1.23.x`` branch, corresponding to each maintenance release).
+   
+   This ensures that the queries and libraries you write in VS Code also work in the query console on LGTM Enterprise.
+
+   If you prefer to add the CodeQL queries and libraries to an :ref:`existing workspace <existing-workspace>` instead of the starter workspace, then you should
+   clone the appropriate branch of the `general CodeQL repository <https://github.com/github/codeql>`__ and the 
+   `CodeQL repository for Go <https://github.com/github/codeql-go>`__ and add them to your workspace.
+
+.. _starter-workspace:
+
+Using the starter workspace
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The starter workspace is a Git repository. It contains:
+
+* The `repository of CodeQL libraries and queries <https://github.com/github/codeql>`__ for C/C++, C#, Java, JavaScript, and Python. This is included as a submodule, so it can be updated without affecting your custom queries.
+* The `repository of CodeQL libraries and queries <https://github.com/github/codeql-go>`__ for Go. This is also included as a submodule.
+* A series of folders named ``codeql-custom-queries-<ql-language-specification>``. These are ready for you to start developing your own custom queries for each language, using the standard libraries. There are some example queries to get you started.
+
+To use the starter workspace:
+
+#. Clone the https://github.com/github/vscode-codeql-starter/ repository to your computer:
+    * Make sure you include the submodules, either by using ``git clone --recursive``, or using by ``git submodule update --init --remote`` after cloning.
+    * Use ``git submodule update --remote`` regularly to keep the submodules up to date.
+
+#. In VS Code, use the **File** > **Open Workspace** option to open the ``vscode-codeql-starter.code-workspace`` file from your checkout of the workspace repository.
+
+.. _existing-workspace:
+
+Updating an existing workspace for CodeQL
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+You can add the CodeQL libraries to an existing workspace by making a local clone of the CodeQL repository directly: https://github.com/github/codeql. 
+
+To make the standard libraries available in your workspace:
+
+#. Select **File** > **Add Folder to Workspace**, and choose your local checkout of the ``github/codeql`` repository.
+
+#. Create one new folder per target language, using either the **New Folder** or **Add Folder to Workspace** options, to hold custom queries and libraries.
+
+#. Create a ``qlpack.yml`` file in each target language folder. This tells the CodeQL CLI the target language for that folder and what its dependencies are. (The ``main`` branch of ``github/codeql`` already has these files.) CodeQL will look for the dependencies in all the open workspace folders, or on the user's search path.
+
+For example, to make a custom CodeQL folder called ``my-custom-cpp-pack`` depend on the CodeQL standard library for C++, create a ``qlpack.yml`` file with the following contents:
+
+.. code-block:: yaml
+
+    name: my-custom-cpp-pack
+    version: 0.0.0
+    libraryPathDependencies: codeql-cpp
+
+For more information about why you need to add a ``qlpack.yml`` file, see ":ref:`About QL packs <about-ql-packs>`."
+
+.. pull-quote:: Note
+
+   The CodeQL libraries for Go are not included in the ``github/codeql`` repository, but are stored separately. To analyze Go projects, clone the repository at https://github.com/github/codeql-go and add it to your workspace as above.
+
+Further reading
+----------------
+
+* ":doc:`Analyzing your projects <analyzing-your-projects>`"
+* ":ref:`CodeQL CLI <codeql-cli>`"

docs/codeql/codeql-for-visual-studio-code/testing-codeql-queries-in-visual-studio-code.rst

@@ -0,0 +1,50 @@
+.. _testing-codeql-queries-in-visual-studio-code:
+
+Testing CodeQL queries in Visual Studio Code
+============================================
+
+You can run unit tests for CodeQL queries using the Visual Studio Code extension.
+
+About testing queries in VS Code
+---------------------------------
+
+To ensure that your CodeQL queries produce the expected results, you can run tests that compare the expected query results with the actual results.
+
+The CodeQL extension automatically prompts VS Code to install the Test Explorer extension as a dependency. The Test Explorer displays any workspace folders with a name ending in ``-tests`` and provides a UI for exploring and running tests in those folders.
+
+For more information about how CodeQL tests work, see ":ref:`Testing custom queries <testing-custom-queries>`" in the CLI help.
+
+Testing the results of your queries
+-----------------------------------
+
+1. Open the Test Explorer view in the sidebar.
+
+   .. image:: ../images/codeql-for-visual-studio-code/open-test-explorer.png
+      :width: 350
+      :alt: Open the Test Explorer view
+
+2. To run a specific test, hover over the file or folder name and click the play button. To run all tests in your workspace, click the play button at the top of the view. If a test takes too long to run, you can click the stop button at the top of the view to cancel the test.
+3. The icons show whether a test passed or failed. If it failed, right-click the failed test and click **CodeQL: Show Test Output Differences** to display the differences between the expected output and the actual output.
+
+   .. image:: ../images/codeql-for-visual-studio-code/show-test-diff.png
+      :width: 400
+      :alt: Show test output differences
+
+4. Compare the results. If you want to update the test with the actual output, click **CodeQL: Accept Test Output**.
+
+Monitoring the performance of your queries
+------------------------------------------
+
+Query performance is important when you want to run a query on large databases, or as part of your continuous integration system. 
+
+.. include:: ../reusables/running-queries-debug.rst
+
+When a query is evaluated, the query server caches the predicates that it calculates. So when you want to compare the performance of two evaluations, you should clear the query server's cache before each run (**CodeQL: Clear Cache** command). This ensures that you're comparing equivalent data.
+
+For more information, see ":ref:`Troubleshooting query performance <troubleshooting-query-performance>`" and ":ref:`Evaluation of QL programs <evaluation-of-ql-programs>`."
+
+
+Further reading
+----------------
+
+* ":ref:`Testing custom queries <testing-custom-queries>`"

docs/codeql/codeql-for-visual-studio-code/troubleshooting-codeql-for-visual-studio-code.rst

@@ -0,0 +1,48 @@
+.. _troubleshooting-codeql-for-visual-studio-code:
+
+Troubleshooting CodeQL for Visual Studio Code
+=============================================
+
+You can use the detailed information written to the extension's log files if you need to troubleshoot problems.
+
+About the log files
+--------------------
+
+Progress and error messages are displayed as notifications in the bottom right corner of the workspace.
+These link to more detailed logs and error messages in the Output panel.
+You can use the dropdown list to select the logs you need.
+
+   .. image:: ../images/codeql-for-visual-studio-code/select-logs.png
+      :alt: Select the logs in the Output view
+
+Troubleshooting installation and configuration problems
+------------------------------------------------------------
+
+If you encounter any problems when installing and configuring the extension, check the CodeQL Extension Log to see general extension logging messages, including details about the CodeQL CLI and the commands invoked by the extension.
+
+In particular, you can see the location of the CLI that is being used. This is useful if you want to see whether this is an extension-managed CLI or an external one.
+
+If you use the extension-managed CLI, the extension checks for updates automatically (or with the **CodeQL: Check for CLI Updates** command) and prompts you to accept the updated version.
+If you use an external CLI, you need to update it manually (when updates are necessary).
+
+Exploring problems with queries and databases
+----------------------------------------------
+
+For details about compiling and running queries, as well as information about database upgrades, check the CodeQL Query Server log.
+
+If you see behavior or errors that suggest problems, you can use the **CodeQL: Restart Query Server** command to restart the query server. This restarts the server without affecting your CodeQL session history.
+You are most likely to need to restart the query server if you make external changes to files that the extension is using. For example, regenerating a CodeQL database that's open in VS Code. In addition to problems in the log, you might also see: errors in code highlighting, incorrect results totals, or duplicate notifications that a query is running.
+
+To see the logs from running a particular query, right-click the query in the Query History and select **Show Query Log**.
+If the log file is too large for the extension to open in the VS Code editor, the file will be displayed in your file explorer so you can open it with an external program.
+
+Exploring problems with running tests
+----------------------------------------------
+
+To see more detailed output from running unit tests, open the CodeQL Tests log.
+For more information about tests, see ":doc:`Testing CodeQL queries in Visual Studio Code <testing-codeql-queries-in-visual-studio-code>`."
+
+Generating a bug report for GitHub
+--------------------------------------
+
+The CodeQL Language Server contains more advanced debug logs for CodeQL language maintainers. You should only need these to provide details in a bug report.

docs/codeql/codeql-language-guides/codeql-for-cpp/analyzing-data-flow-in-cpp.rst

@@ -1,3 +1,5 @@
+.. _analyzing-data-flow-in-cpp:
+
 Analyzing data flow in C and C++
 ================================
 
@@ -6,7 +8,7 @@ You can use data flow analysis to track the flow of potentially malicious or ins
 About data flow
 ---------------
 
-Data flow analysis computes the possible values that a variable can hold at various points in a program, determining how those values propagate through the program, and where they are used. In CodeQL, you can model both local data flow and global data flow. For a more general introduction to modeling data flow, see ":doc:`About data flow analysis <../about-data-flow-analysis>`."
+Data flow analysis computes the possible values that a variable can hold at various points in a program, determining how those values propagate through the program, and where they are used. In CodeQL, you can model both local data flow and global data flow. For a more general introduction to modeling data flow, see ":ref:`About data flow analysis <about-data-flow-analysis>`."
 
 Local data flow
 ---------------
@@ -390,7 +392,7 @@ Exercise 4
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/cpp-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-cpp/basic-query-for-cpp-code.rst

@@ -1,3 +1,5 @@
+.. _basic-query-for-cpp-code:
+
 Basic query for C and C++ code
 ==============================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/codeql-library-for-cpp.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-cpp:
+
 CodeQL library for C and C++
 ============================
 
@@ -49,32 +51,32 @@ This table lists `Declaration <https://help.semmle.com/qldoc/cpp/semmle/code/cpp
 | | ``template < typename`` *T* ``>``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `FunctionTemplateSpecialization <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$FunctionTemplateSpecialization.html>`__                 |                                                                                                                                                                                                    |
 | | *func* ``< int,`` *T* ``> (`` ... ``) {`` ... ``}``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                                                                                                                                                                       |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| | ``class`` *C* ``{`` ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `MemberFunction <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$MemberFunction.html>`__                                                 |                                                                                                                                                                                                    |
+| | ``class`` *C* ``{`` ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `MemberFunction <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$MemberFunction.html>`__                                     |                                                                                                                                                                                                    |
 | |  ``int`` *func* ``( float`` *param* ``);`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                                                                                                                                                       |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| | ``class`` *C* ``{`` ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `ConstMemberFunction <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$ConstMemberFunction.html>`__                                       |                                                                                                                                                                                                    |
+| | ``class`` *C* ``{`` ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `ConstMemberFunction <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$ConstMemberFunction.html>`__                           |                                                                                                                                                                                                    |
 | |  ``int`` *func* ``( float`` *param* ``) const;`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                                                                                                                                                       |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``class`` *C* ``{`` ... ``virtual int`` *func* ``(`` ... ``) {`` ... ``} };``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `VirtualFunction <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$VirtualFunction.html>`__                                               |                                                                                                                                                                                                    |
+| ``class`` *C* ``{`` ... ``virtual int`` *func* ``(`` ... ``) {`` ... ``} };``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `VirtualFunction <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$VirtualFunction.html>`__                                   |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``class`` *C* ``{`` ... *C* ``(`` ... ``) {`` ... ``}`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `Constructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$Constructor.html>`__                                                       |                                                                                                                                                                                                    |
+| ``class`` *C* ``{`` ... *C* ``(`` ... ``) {`` ... ``}`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `Constructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$Constructor.html>`__                                           |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``C::operator float () const;``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `ConversionOperator <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$ConversionOperator.html>`__                                         |                                                                                                                                                                                                    |
+| ``C::operator float () const;``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `ConversionOperator <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$ConversionOperator.html>`__                             |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``class`` *C* ``{`` ...  ``~`` *C* ``( void ) {`` ... ``}`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `Destructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$Destructor.html>`__                                                         |                                                                                                                                                                                                    |
+| ``class`` *C* ``{`` ...  ``~`` *C* ``( void ) {`` ... ``}`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `Destructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$Destructor.html>`__                                             |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| | ``class`` *C* ``{`` ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `ConversionConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$ConversionConstructor.html>`__                                   |                                                                                                                                                                                                    |
+| | ``class`` *C* ``{`` ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `ConversionConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$ConversionConstructor.html>`__                       |                                                                                                                                                                                                    |
 | |  *C* ``( const`` *D* ``&`` *d* ``) {`` ... ``}`` ... ``};``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                                                                                                                                                       |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| *C* ``&`` *C* ``:: operator= (const`` *C* ``& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `CopyAssignmentOperator <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$CopyAssignmentOperator.html>`__                                 |                                                                                                                                                                                                    |
+| *C* ``&`` *C* ``:: operator= (const`` *C* ``& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `CopyAssignmentOperator <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$CopyAssignmentOperator.html>`__                     |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| *C* ``&`` *C* ``:: operator= (`` *C* ``&& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `MoveAssignmentOperator <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$MoveAssignmentOperator.html>`__                                 |                                                                                                                                                                                                    |
+| *C* ``&`` *C* ``:: operator= (`` *C* ``&& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `MoveAssignmentOperator <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$MoveAssignmentOperator.html>`__                     |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| *C* ``::`` *C* ``(const`` *C* ``& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `CopyConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$CopyConstructor.html>`__                                               |                                                                                                                                                                                                    |
+| *C* ``::`` *C* ``(const`` *C* ``& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `CopyConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$CopyConstructor.html>`__                                   |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| *C* ``::`` *C* ``(`` *C* ``&& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `MoveConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$MoveConstructor.html>`__                                               |                                                                                                                                                                                                    |
+| *C* ``::`` *C* ``(`` *C* ``&& );``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `MoveConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$MoveConstructor.html>`__                                   |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| *C* ``::`` *C* ``(void);``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `NoArgConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Function.qll/type.Function$NoArgConstructor.html>`__                                             | Default constructor                                                                                                                                                                                |
+| *C* ``::`` *C* ``(void);``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `NoArgConstructor <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/MemberFunction.qll/type.MemberFunction$NoArgConstructor.html>`__                                 | Default constructor                                                                                                                                                                                |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | ``enum`` *en* ``{`` *val1* ``,`` *val2* ... ``}``                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `EnumConstant <https://help.semmle.com/qldoc/cpp/semmle/code/cpp/Enum.qll/type.Enum$EnumConstant.html>`__                                                             |                                                                                                                                                                                                    |
 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

docs/codeql/codeql-language-guides/codeql-for-cpp/conversions-and-classes-in-cpp.rst

@@ -1,3 +1,5 @@
+.. _conversions-and-classes-in-cpp:
+
 Conversions and classes in C and C++
 ====================================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/detecting-a-potential-buffer-overflow.rst

@@ -1,3 +1,5 @@
+.. _detecting-a-potential-buffer-overflow:
+
 Detecting a potential buffer overflow
 =====================================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/expressions-types-and-statements-in-cpp.rst

@@ -1,3 +1,5 @@
+.. _expressions-types-and-statements-in-cpp:
+
 Expressions, types, and statements in C and C++
 ===============================================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/functions-in-cpp.rst

@@ -1,3 +1,5 @@
+.. _functions-in-cpp:
+
 Functions in C and C++
 =======================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/hash-consing-and-value-numbering.rst

@@ -1,3 +1,5 @@
+.. _hash-consing-and-value-numbering:
+
 Hash consing and value numbering
 ================================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/index.rst

@@ -1,3 +1,5 @@
+.. _codeql-for-cpp:
+
 CodeQL for C and C++
 ====================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/refining-a-query-to-account-for-edge-cases.rst

@@ -1,3 +1,5 @@
+.. _refining-a-query-to-account-for-edge-cases:
+
 Refining a query to account for edge cases
 ==========================================
 
@@ -6,7 +8,7 @@ You can improve the results generated by a CodeQL query by adding conditions to
 Overview
 --------
 
-This topic describes how a C++ query was developed. The example introduces recursive predicates and demonstrates the typical workflow used to refine a query. For a full overview of the topics available for learning to write queries for C/C++ code, see ":doc:`CodeQL for C and C++ <codeql-for-cpp>`."
+This topic describes how a C++ query was developed. The example introduces recursive predicates and demonstrates the typical workflow used to refine a query. For a full overview of the topics available for learning to write queries for C/C++ code, see ":ref:`CodeQL for C and C++ <codeql-for-cpp>`."
 
 Finding every private field and checking for initialization
 -----------------------------------------------------------
@@ -102,7 +104,7 @@ You may also wish to consider methods called by constructors that assign to the
      int m_value;
    };
 
-This case can be excluded by creating a recursive predicate. The recursive predicate is given a function and a field, then checks whether the function assigns to the field. The predicate runs itself on all the functions called by the function that it has been given. By passing the constructor to this predicate, we can check for assignments of a field in all functions called by the constructor, and then do the same for all functions called by those functions all the way down the tree of function calls. For more information, see "`Recursion <https://help.semmle.com/QL/ql-language-reference/recursion.html>`__" in the QL language reference.
+This case can be excluded by creating a recursive predicate. The recursive predicate is given a function and a field, then checks whether the function assigns to the field. The predicate runs itself on all the functions called by the function that it has been given. By passing the constructor to this predicate, we can check for assignments of a field in all functions called by the constructor, and then do the same for all functions called by those functions all the way down the tree of function calls. For more information, see ":ref:`Recursion <recursion>`" in the QL language reference.
 
 .. code-block:: ql
 
@@ -126,7 +128,7 @@ This case can be excluded by creating a recursive predicate. The recursive predi
 Refinement 4—simplifying the query
 ----------------------------------
 
-Finally we can simplify the query by using the transitive closure operator. In this final version of the query, ``c.calls*(fun)`` resolves to the set of all functions that are ``c`` itself, are called by ``c``, are called by a function that is called by ``c``, and so on. This eliminates the need to make a new predicate all together. For more information, see "`Transitive closures <https://help.semmle.com/QL/ql-language-reference/recursion.html#transitive-closures>`__" in the QL language reference.
+Finally we can simplify the query by using the transitive closure operator. In this final version of the query, ``c.calls*(fun)`` resolves to the set of all functions that are ``c`` itself, are called by ``c``, are called by a function that is called by ``c``, and so on. This eliminates the need to make a new predicate all together. For more information, see ":ref:`Transitive closures <transitive-closures>`" in the QL language reference.
 
 .. code-block:: ql
 

docs/codeql/codeql-language-guides/codeql-for-cpp/using-range-analsis-in-cpp.rst

@@ -1,3 +1,5 @@
+.. _using-range-analsis-in-cpp:
+
 Using range analysis for C and C++
 ==================================
 

docs/codeql/codeql-language-guides/codeql-for-cpp/using-the-guards-library-in-cpp.rst

@@ -1,3 +1,5 @@
+.. _using-the-guards-library-in-cpp:
+
 Using the guards library in C and C++
 =====================================
 

docs/codeql/codeql-language-guides/codeql-for-csharp/analyzing-data-flow-in-csharp.rst

@@ -1,3 +1,5 @@
+.. _analyzing-data-flow-in-csharp:
+
 Analyzing data flow in C#
 =========================
 
@@ -8,7 +10,7 @@ About this article
 
 This article describes how data flow analysis is implemented in the CodeQL libraries for C# and includes examples to help you write your own data flow queries.
 The following sections describe how to use the libraries for local data flow, global data flow, and taint tracking.
-For a more general introduction to modeling data flow, see ":doc:`About data flow analysis <../about-data-flow-analysis>`."
+For a more general introduction to modeling data flow, see ":ref:`About data flow analysis <about-data-flow-analysis>`."
 
 Local data flow
 ---------------
@@ -553,7 +555,7 @@ This can be adapted from the ``SystemUriFlow`` class:
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/csharp-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-csharp/basic-query-for-csharp-code.rst

@@ -1,3 +1,5 @@
+.. _basic-query-for-csharp-code:
+
 Basic query for C# code
 =======================
 

docs/codeql/codeql-language-guides/codeql-for-csharp/codeql-library-for-csharp.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-csharp:
+
 CodeQL library for C#
 =====================
 
@@ -14,7 +16,7 @@ There is an extensive core library for analyzing CodeQL databases extracted from
 
 Since this is required for all C# queries, it's omitted from code snippets below.
 
-The core library contains all the program elements, including `files <#files>`__, `types <#types>`__, methods, `variables <#variables>`__, `statements <#statements>`__, and `expressions <#expressions>`__. This is sufficient for most queries, however additional libraries can be imported for bespoke functionality such as control flow and data flow. For information about these additional libraries, see ":doc:`CodeQL for C# <codeql-for-csharp>`." 
+The core library contains all the program elements, including `files <#files>`__, `types <#types>`__, methods, `variables <#variables>`__, `statements <#statements>`__, and `expressions <#expressions>`__. This is sufficient for most queries, however additional libraries can be imported for bespoke functionality such as control flow and data flow. For information about these additional libraries, see ":ref:`CodeQL for C# <codeql-for-csharp>`." 
 
 Class hierarchies
 ~~~~~~~~~~~~~~~~~

docs/codeql/codeql-language-guides/codeql-for-csharp/index.rst

@@ -1,3 +1,5 @@
+.. _codeql-for-csharp:
+
 CodeQL for C#
 =============
 

docs/codeql/codeql-language-guides/codeql-for-go/abstract-syntax-tree-classes-for-working-with-go-programs.rst

@@ -1,3 +1,5 @@
+.. _abstract-syntax-tree-classes-for-working-with-go-programs:
+
 Abstract syntax tree classes for working with Go programs
 =========================================================
 

docs/codeql/codeql-language-guides/codeql-for-go/basic-query-for-go-code.rst

@@ -1,3 +1,5 @@
+.. _basic-query-for-go-code:
+
 Basic query for Go code
 =======================
 
@@ -99,7 +101,7 @@ After the initial ``import`` statement, this simple query comprises three parts
 | ``where recv = m.getReceiver() and                            | Defines a condition on the variables.                                                                             | ``recv = m.getReceiver()`` states that ``recv`` must be the receiver variable of ``m``.                                              |
 | w.writesField(recv.getARead(), f, _) and                      |                                                                                                                   |                                                                                                                                      |
 | not recv.getType() instanceof PointerType``                   |                                                                                                                   | ``w.writesField(recv.getARead(), f, _)`` states that ``w`` must be a location in the code where field ``f`` of ``recv`` is modified. |
-|                                                               |                                                                                                                   | We use a `'don't-care' expression <https://help.semmle.com/QL/ql-language-reference/expressions.html#don-t-care-expressions>`__ ``_``|
+|                                                               |                                                                                                                   | We use a :ref:`'don't-care' expression <don-t-care-expressions>` ``_``                                                               |
 |                                                               |                                                                                                                   | for the value that is written to ``f``—the actual value doesn't matter in this query.                                                |
 |                                                               |                                                                                                                   |                                                                                                                                      |
 |                                                               |                                                                                                                   | ``not recv.getType() instanceof PointerType`` states that ``m`` is not a pointer method.                                             |

docs/codeql/codeql-language-guides/codeql-for-go/codeql-library-for-go.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-go:
+
 CodeQL library for Go
 =====================
 
@@ -502,7 +504,7 @@ taint, you can define a subclass of ``TaintTracking::Configuration``, which work
 data-flow configurations.
 
 A detailed exposition of global data flow and taint tracking is out of scope for this brief
-introduction. For a general overview of data flow and taint tracking, see "`About data flow analysis <https://help.semmle.com/QL/learn-ql/about-data-flow-analysis.html>`__."
+introduction. For a general overview of data flow and taint tracking, see ":ref:`About data flow analysis <about-data-flow-analysis>`."
 
 Advanced libraries
 ------------------

docs/codeql/codeql-language-guides/codeql-for-go/index.rst

@@ -1,3 +1,5 @@
+.. _codeql-for-go:
+
 CodeQL for Go
 =============
 

docs/codeql/codeql-language-guides/codeql-for-go/modeling-data-flow-in-go-libraries.rst

@@ -1,3 +1,5 @@
+.. _modeling-data-flow-in-go-libraries:
+
 Modeling data flow in Go libraries
 ==================================
 

docs/codeql/codeql-language-guides/codeql-for-java/abstract-syntax-tree-classes-for-working-with-java-programs.rst

@@ -1,3 +1,5 @@
+.. _abstract-syntax-tree-classes-for-working-with-java-programs:
+
 Abstract syntax tree classes for working with Java programs
 ===========================================================
 

docs/codeql/codeql-language-guides/codeql-for-java/analyzing-data-flow-in-java.rst

@@ -1,3 +1,5 @@
+.. _analyzing-data-flow-in-java:
+
 Analyzing data flow in Java
 ===========================
 
@@ -9,7 +11,7 @@ About this article
 This article describes how data flow analysis is implemented in the CodeQL libraries for Java and includes examples to help you write your own data flow queries.
 The following sections describe how to use the libraries for local data flow, global data flow, and taint tracking.
 
-For a more general introduction to modeling data flow, see ":doc:`About data flow analysis <../about-data-flow-analysis>`."
+For a more general introduction to modeling data flow, see ":ref:`About data flow analysis <about-data-flow-analysis>`."
 
 Local data flow
 ---------------
@@ -358,7 +360,7 @@ Exercise 4
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/java-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-java/annotations-in-java.rst

@@ -1,3 +1,5 @@
+.. _annotations-in-java:
+
 Annotations in Java
 ===================
 

docs/codeql/codeql-language-guides/codeql-for-java/basic-query-for-java-code.rst

@@ -1,3 +1,5 @@
+.. _basic-query-for-java-code:
+
 Basic query for Java code
 =========================
 

docs/codeql/codeql-language-guides/codeql-for-java/codeql-library-for-java.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-java:
+
 CodeQL library for Java
 =======================
 
@@ -196,7 +198,7 @@ The wildcards ``? extends Number`` and ``? super Float`` are represented by clas
 
 For dealing with generic methods, there are classes ``GenericMethod``, ``ParameterizedMethod`` and ``RawMethod``, which are entirely analogous to the like-named classes for representing generic types.
 
-For more information on working with types, see the :doc:`article on Java types <types-in-java>`.
+For more information on working with types, see the :doc:`Types in Java <types-in-java>`.
 
 Variables
 ~~~~~~~~~
@@ -210,7 +212,7 @@ Class ``Variable`` represents a variable `in the Java sense <https://docs.oracle
 Abstract syntax tree
 --------------------
 
-Classes in this category represent abstract syntax tree (AST) nodes, that is, statements (class ``Stmt``) and expressions (class ``Expr``). For a full list of expression and statement types available in the standard QL library, see ":doc:`Abstract syntax tree classes for working with Java programs <abstract-syntax-tree-classes-for-working-with-go-programs>`."
+Classes in this category represent abstract syntax tree (AST) nodes, that is, statements (class ``Stmt``) and expressions (class ``Expr``). For a full list of expression and statement types available in the standard QL library, see ":doc:`Abstract syntax tree classes for working with Java programs <abstract-syntax-tree-classes-for-working-with-java-programs>`."
 
 Both ``Expr`` and ``Stmt`` provide member predicates for exploring the abstract syntax tree of a program:
 

docs/codeql/codeql-language-guides/codeql-for-java/index.rst

@@ -1,3 +1,5 @@
+.. _codeql-for-java:
+
 CodeQL for Java
 ===============
 
@@ -15,7 +17,7 @@ Experiment and learn how to write effective and efficient queries for CodeQL dat
    annotations-in-java
    javadoc
    working-with-source-locations
-   abstract-syntax-tree-classes-for-working-with-go-programs
+   abstract-syntax-tree-classes-for-working-with-java-programs
 
 -  :doc:`Basic query for Java code <basic-query-for-java-code>`: Learn to write and run a simple CodeQL query using LGTM.
 
@@ -35,5 +37,5 @@ Experiment and learn how to write effective and efficient queries for CodeQL dat
 
 -  :doc:`Working with source locations <working-with-source-locations>`: You can use the location of entities within Java code to look for potential errors. Locations allow you to deduce the presence, or absence, of white space which, in some cases, may indicate a problem.
 
--  :doc:`Abstract syntax tree classes for working with Java programs <abstract-syntax-tree-classes-for-working-with-go-programs>`: CodeQL has a large selection of classes for representing the abstract syntax tree of Java programs.
+-  :doc:`Abstract syntax tree classes for working with Java programs <abstract-syntax-tree-classes-for-working-with-java-programs>`: CodeQL has a large selection of classes for representing the abstract syntax tree of Java programs.
 

docs/codeql/codeql-language-guides/codeql-for-java/javadoc.rst

@@ -1,3 +1,5 @@
+.. _javadoc:
+
 Javadoc
 =======
 

docs/codeql/codeql-language-guides/codeql-for-java/navigating-the-call-graph.rst

@@ -1,3 +1,5 @@
+.. _navigating-the-call-graph:
+
 Navigating the call graph
 =========================
 

docs/codeql/codeql-language-guides/codeql-for-java/overflow-prone-comparisons-in-java.rst

@@ -1,3 +1,5 @@
+.. _overflow-prone-comparisons-in-java:
+
 Overflow-prone comparisons in Java
 ==================================
 
@@ -26,7 +28,7 @@ If ``l`` is bigger than 2\ :sup:`31`\ - 1 (the largest positive value of type ``
 
    All primitive numeric types have a maximum value, beyond which they will wrap around to their lowest possible value (called an "overflow"). For ``int``, this maximum value is 2\ :sup:`31`\ - 1. Type ``long`` can accommodate larger values up to a maximum of 2\ :sup:`63`\ - 1. In this example, this means that ``l`` can take on a value that is higher than the maximum for type ``int``; ``i`` will never be able to reach this value, instead overflowing and returning to a low value.
 
-We're going to develop a query that finds code that looks like it might exhibit this kind of behavior. We'll be using several of the standard library classes for representing statements and functions. For a full list, see ":doc:`Abstract syntax tree classes for working with Java programs <abstract-syntax-tree-classes-for-working-with-go-programs>`."
+We're going to develop a query that finds code that looks like it might exhibit this kind of behavior. We'll be using several of the standard library classes for representing statements and functions. For a full list, see ":doc:`Abstract syntax tree classes for working with Java programs <abstract-syntax-tree-classes-for-working-with-java-programs>`."
 
 Initial query
 -------------
@@ -61,7 +63,7 @@ Notice that we use the predicate ``getType`` (available on all subclasses of ``E
 
 The class ``LoopStmt`` is a common superclass of all loops, including, in particular, ``for`` loops as in our example above. While different kinds of loops have different syntax, they all have a loop condition, which can be accessed through predicate ``getCondition``. We use the reflexive transitive closure operator ``*`` applied to the ``getAChildExpr`` predicate to express the requirement that ``expr`` should be nested inside the loop condition. In particular, it can be the loop condition itself.
 
-The final conjunct in the ``where`` clause takes advantage of the fact that `predicates <https://help.semmle.com/QL/ql-language-reference/predicates.html>`__ can return more than one value (they are really relations). In particular, ``getAnOperand`` may return *either* operand of ``expr``, so ``expr.getAnOperand().isCompileTimeConstant()`` holds if at least one of the operands is constant. Negating this condition means that the query will only find expressions where *neither* of the operands is constant.
+The final conjunct in the ``where`` clause takes advantage of the fact that :ref:`predicates <predicates>` can return more than one value (they are really relations). In particular, ``getAnOperand`` may return *either* operand of ``expr``, so ``expr.getAnOperand().isCompileTimeConstant()`` holds if at least one of the operands is constant. Negating this condition means that the query will only find expressions where *neither* of the operands is constant.
 
 Generalizing the query
 ----------------------

docs/codeql/codeql-language-guides/codeql-for-java/types-in-java.rst

@@ -1,5 +1,7 @@
-Java types
-==========
+.. _types-in-java:
+
+Types in Java
+=============
 
 You can use CodeQL to find out information about data types used in Java code. This allows you to write queries to identify specific type-related issues.
 

docs/codeql/codeql-language-guides/codeql-for-java/working-with-source-locations.rst

@@ -1,3 +1,5 @@
+.. _working-with-source-locations:
+
 Working with source locations
 =============================
 

docs/codeql/codeql-language-guides/codeql-for-javascript/abstract-syntax-tree-classes-for-working-with-javascript-and-typescript-programs.rst

@@ -1,3 +1,5 @@
+.. _abstract-syntax-tree-classes-for-working-with-javascript-and-typescript-programs:
+
 Abstract syntax tree classes for working with JavaScript and TypeScript programs
 ================================================================================
 

docs/codeql/codeql-language-guides/codeql-for-javascript/analyzing-data-flow-in-javascript.rst

@@ -1,3 +1,5 @@
+.. _analyzing-data-flow-in-javascript:
+
 Analyzing data flow in JavaScript and TypeScript
 ================================================
 
@@ -9,7 +11,7 @@ The various sections in this article describe how to utilize the libraries for l
 As our running example, we will develop a query that identifies command-line arguments that are passed as a file path to the standard Node.js ``readFile`` function.
 While this is not a problematic pattern as such, it is typical of the kind of reasoning that is frequently used in security queries.
 
-For a more general introduction to modeling data flow, see ":doc:`About data flow analysis <../about-data-flow-analysis>`."
+For a more general introduction to modeling data flow, see ":ref:`About data flow analysis <about-data-flow-analysis>`."
 
 Data flow nodes
 ---------------
@@ -554,7 +556,7 @@ Exercise 4
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/java-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-javascript/basic-query-for-javascript-code.rst

@@ -1,3 +1,5 @@
+.. _basic-query-for-javascript-code:
+
 Basic query for JavaScript code
 ===============================
 

docs/codeql/codeql-language-guides/codeql-for-javascript/codeql-library-for-javascript.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-javascript:
+
 CodeQL library for JavaScript
 =============================
 

docs/codeql/codeql-language-guides/codeql-for-javascript/codeql-library-for-typescript.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-typescript:
+
 CodeQL library for TypeScript
 =============================
 
@@ -177,7 +179,7 @@ Ambient nodes are mostly ignored by control flow and data flow analysis. The out
 Static type information
 -----------------------
 
-Static type information and global name binding is available for projects with "full" TypeScript extraction enabled. This option is enabled by default for projects on LGTM.com and when you create databases with the `CodeQL CLI <https://help.semmle.com/codeql/codeql-cli.html>`__.
+Static type information and global name binding is available for projects with "full" TypeScript extraction enabled. This option is enabled by default for projects on LGTM.com and when you create databases with the :ref:`CodeQL CLI <codeql-cli>`.
 
 .. pull-quote:: Note
 

docs/codeql/codeql-language-guides/codeql-for-javascript/data-flow-cheat-sheet-for-javascript.rst

@@ -1,3 +1,5 @@
+.. _data-flow-cheat-sheet-for-javascript:
+
 Data flow cheat sheet for JavaScript
 ====================================
 
@@ -34,7 +36,7 @@ This query reports flow paths which:
 - Step through variables, function calls, properties, strings, arrays, promises, exceptions, and steps added by `isAdditionalTaintStep <https://help.semmle.com/qldoc/javascript/semmle/javascript/dataflow/TaintTracking.qll/predicate.TaintTracking$TaintTracking$Configuration$isAdditionalTaintStep.2.html>`__.
 - End at a node matched by `isSink <https://help.semmle.com/qldoc/javascript/semmle/javascript/dataflow/Configuration.qll/predicate.Configuration$Configuration$isSink.1.html>`__.
 
-See also: "`Global data flow <https://help.semmle.com/QL/learn-ql/javascript/dataflow.html#global-data-flow>`__" and ":doc:`Creating path queries <../writing-queries/creating-path-queries>`."
+See also: "`Global data flow <analyzing-data-flow-in-javascript.html#global-data-flow>`__" and ":ref:`Creating path queries <creating-path-queries>`."
 
 DataFlow module
 ---------------
@@ -220,7 +222,7 @@ Troubleshooting
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/javascript-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-javascript/index.rst

@@ -1,3 +1,5 @@
+.. _codeql-for-javascript:
+
 CodeQL for JavaScript
 =====================
 

docs/codeql/codeql-language-guides/codeql-for-javascript/using-flow-labels-for-precise-data-flow-analysis.rst

@@ -1,3 +1,5 @@
+.. _using-flow-labels-for-precise-data-flow-analysis:
+
 Using flow labels for precise data flow analysis
 ================================================
 
@@ -267,7 +269,7 @@ sanitized value:
     }
   }
 
-Here is the final query, expressed as a :doc:`path query <../writing-queries/creating-path-queries>` so we can examine paths from sources to sinks
+Here is the final query, expressed as a :ref:`path query <creating-path-queries>` so we can examine paths from sources to sinks
 step by step in the UI:
 
 .. code-block:: ql
@@ -398,7 +400,7 @@ string may be an absolute path and whether it may contain ``..`` components.
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/javascript-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-javascript/using-type-tracking-for-api-modeling.rst

@@ -1,3 +1,5 @@
+.. _using-type-tracking-for-api-modeling:
+
 Using type tracking for API modeling
 ====================================
 
@@ -9,9 +11,9 @@ The type-tracking library makes it possible to track values through properties a
 usually to recognize method calls and properties accessed on a specific type of object.
 
 This is an advanced topic and is intended for readers already familiar with the
-`SourceNode <https://help.semmle.com/QL/learn-ql/javascript/dataflow.html#source-nodes>`__ class as well as
-`taint tracking <https://help.semmle.com/QL/learn-ql/javascript/dataflow.html#using-global-analyzing-data-flow-and-tracking-tainted-data-in-python>`__.
-For TypeScript analysis also consider reading about `static type information <https://help.semmle.com/QL/learn-ql/javascript/introduce-libraries-ts.html#static-type-information>`__ first.
+`SourceNode <analyzing-data-flow-in-javascript.html#source-nodes>`__ class as well as
+`taint tracking <analyzing-data-flow-in-javascript.html#using-global-analyzing-data-flow-and-tracking-tainted-data-in-python>`__.
+For TypeScript analysis also consider reading about `static type information <codeql-library-for-typescript.html.html#static-type-information>`__ first.
 
 
 The problem of recognizing method calls
@@ -456,7 +458,7 @@ Here's an example that the model from this tutorial won't find:
   let wrapper = wrapDB(firebase.database())
   wrapper.db.ref("forecast"); // <-- not found
 
-This is an example of where `data-flow configurations <https://help.semmle.com/QL/learn-ql/javascript/dataflow.html#global-data-flow>`__ are more powerful.
+This is an example of where `data-flow configurations <analyzing-data-flow-in-javascript.html#global-data-flow>`__ are more powerful.
 
 When to use type tracking
 -------------------------
@@ -489,15 +491,15 @@ Prefer type tracking when:
 
 Prefer data-flow configurations when:
 
-- Tracking user-controlled data -- use `taint tracking <https://help.semmle.com/QL/learn-ql/javascript/dataflow.html#using-global-analyzing-data-flow-and-tracking-tainted-data-in-python>`__.
+- Tracking user-controlled data -- use `taint tracking <analyzing-data-flow-in-javascript.html#using-global-analyzing-data-flow-and-tracking-tainted-data-in-python>`__.
 - Differentiating between different kinds of user-controlled data -- see ":doc:`Using flow labels for precise data flow analysis <using-flow-labels-for-precise-data-flow-analysis>`."
 - Tracking transformations of a value through generic utility functions.
 - Tracking values through string manipulation.
-- Generating a path from source to sink -- see ":doc:`Creating path queries <../writing-queries/creating-path-queries>`."
+- Generating a path from source to sink -- see ":ref:`Creating path queries <creating-path-queries>`."
 
 Lastly, depending on the code base being analyzed, some alternatives to consider are:
 
-- Using `static type information <https://help.semmle.com/QL/learn-ql/javascript/introduce-libraries-ts.html#static-type-information>`__,
+- Using `static type information <codeql-library-for-typescript.html.html#static-type-information>`__,
   if analyzing TypeScript code.
 
 - Relying on local data flow.

docs/codeql/codeql-language-guides/codeql-for-python/analyzing-control-flow-in-python.rst

@@ -1,3 +1,5 @@
+.. _analyzing-control-flow-in-python:
+
 Analyzing control flow in Python
 ================================
 

docs/codeql/codeql-language-guides/codeql-for-python/analyzing-data-flow-and-tracking-tainted-data-in-python.rst

@@ -1,3 +1,5 @@
+.. _analyzing-data-flow-and-tracking-tainted-data-in-python:
+
 Analyzing data flow and tracking tainted data in Python
 =======================================================
 
@@ -15,10 +17,10 @@ Taint tracking differs from basic data flow in that it considers non-value-prese
 For example, in the assignment ``dir = path + "/"``, if ``path`` is tainted then ``dir`` is also tainted,
 even though there is no data flow from ``path`` to ``path + "/"``.
 
-Separate CodeQL libraries have been written to handle 'normal' data flow and taint tracking in :doc:`C/C++ <../cpp/analyzing-data-flow-in-cpp>`, :doc:`C# <../csharp/analyzing-data-flow-in-csharp>`, :doc:`Java <../java/analyzing-data-flow-in-java>`, and :doc:`JavaScript <../javascript/analyzing-data-flow-in-javascript>`. You can access the appropriate classes and predicates that reason about these different modes of data flow by importing the appropriate library in your query.
+Separate CodeQL libraries have been written to handle 'normal' data flow and taint tracking in :doc:`C/C++ <../codeql-for-cpp/analyzing-data-flow-in-cpp>`, :doc:`C# <../codeql-for-csharp/analyzing-data-flow-in-csharp>`, :doc:`Java <../codeql-for-java/analyzing-data-flow-in-java>`, and :doc:`JavaScript <../codeql-for-javascript/analyzing-data-flow-in-javascript>`. You can access the appropriate classes and predicates that reason about these different modes of data flow by importing the appropriate library in your query.
 In Python analysis, we can use the same taint tracking library to model both 'normal' data flow and taint flow, but we are still able make the distinction between steps that preserve values and those that don't by defining additional data flow properties.
 
-For further information on data flow and taint tracking with CodeQL, see ":doc:`Introduction to data flow <../about-data-flow-analysis>`."
+For further information on data flow and taint tracking with CodeQL, see ":ref:`Introduction to data flow <about-data-flow-analysis>`."
 
 Fundamentals of taint tracking using data flow analysis
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -259,7 +261,7 @@ which defines the simplest possible taint kind class, ``HardcodedValue``, and cu
 Further reading
 ---------------
 
-- "`Exploring data flow with path queries <https://help.semmle.com/codeql/codeql-for-vscode/procedures/exploring-data-flow-with-path-queries.html>`__"
+- ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`"
 
 
 .. include:: ../../reusables/python-further-reading.rst

docs/codeql/codeql-language-guides/codeql-for-python/basic-query-for-python-code.rst

@@ -1,3 +1,5 @@
+.. _basic-query-for-python-code:
+
 Basic query for Python code
 ===========================
 

docs/codeql/codeql-language-guides/codeql-for-python/codeql-library-for-python.rst

@@ -1,3 +1,5 @@
+.. _codeql-library-for-python:
+
 CodeQL library for Python
 =========================
 

docs/codeql/codeql-language-guides/codeql-for-python/expressions-and-statements-in-python.rst

@@ -1,3 +1,5 @@
+.. _expressions-and-statements-in-python:
+
 Expressions and statements in Python
 ====================================
 

docs/codeql/codeql-language-guides/codeql-for-python/functions-in-python.rst

@@ -1,3 +1,5 @@
+.. _functions-in-python:
+
 Functions in Python
 ===================
 

docs/codeql/codeql-language-guides/codeql-for-python/index.rst

@@ -1,3 +1,5 @@
+.. _codeql-for-python:
+
 CodeQL for Python
 =================
 

docs/codeql/codeql-language-guides/codeql-for-python/pointer-analysis-and-type-inference-in-python.rst

@@ -1,3 +1,5 @@
+.. _pointer-analysis-and-type-inference-in-python:
+
 Pointer analysis and type inference in Python
 =============================================
 

docs/codeql/codeql-language-guides/index.rst

@@ -0,0 +1,14 @@
+.. _codeql-language-guides:
+
+CodeQL language guides
+======================
+
+.. toctree::
+
+   codeql-for-cpp/index
+   codeql-for-csharp/index
+   codeql-for-go/index
+   codeql-for-java/index
+   codeql-for-javascript/index
+   codeql-for-python/index
+   

docs/codeql/codeql-overview/about-codeql.rst

@@ -0,0 +1,125 @@
+.. _about-codeql:
+
+About CodeQL
+============
+
+CodeQL is the analysis engine used by developers to automate security checks, and by
+security researchers to perform variant analysis. 
+
+In CodeQL, code is treated like data. Security vulnerabilities, bugs, 
+and other errors are modeled as queries that can be executed against databases
+extracted from code. You can run the standard CodeQL queries, written by GitHub
+researchers and community contributors, or write your own to use in custom
+analyses. Queries that find potential bugs highlight the result directly in the
+source file.
+
+About variant analysis
+----------------------
+
+Variant analysis is the process of using a known security vulnerability as a
+seed to find similar problems in your code. It's a technique that security
+engineers use to identify potential vulnerabilities, and ensure these threats
+are properly fixed across multiple codebases.
+
+Querying code using CodeQL is the most efficient way to perform variant
+analysis. You can use the standard CodeQL queries to identify seed
+vulnerabilities, or find new vulnerabilities by writing your own custom CodeQL
+queries. Then, develop or iterate over the query to automatically find logical
+variants of the same bug that could be missed using traditional manual
+techniques.
+
+CodeQL analysis
+---------------
+
+CodeQL analysis consists of three steps:
+
+#. Preparing the code, by creating a CodeQL database
+#. Running CodeQL queries against the database
+#. Interpreting the query results
+
+Database creation
+~~~~~~~~~~~~~~~~~
+
+To create a database, CodeQL first extracts a single relational representation
+of each source file in the codebase. 
+
+For compiled languages, extraction works by monitoring the normal build process.
+Each time a compiler is invoked to process a source file, a copy of that file is
+made, and all relevant information about the source code is collected. This includes
+syntactic data about the abstract syntax tree and semantic data about name
+binding and type information.
+
+For interpreted languages, the extractor runs directly on the source code,
+resolving dependencies to give an accurate representation of the codebase. 
+
+There is one :ref:`extractor <extractor>` for each language supported by CodeQL
+to ensure that the extraction process is as accurate as possible. For
+multi-language codebases, databases are generated one language at a time.
+
+After extraction, all the data required for analysis (relational data, copied
+source files, and a language-specific :ref:`database schema
+<codeql-database-schema>`, which specifies the mutual relations in the data) is
+imported into a single directory, known as a :ref:`CodeQL database
+<codeql-database>`.
+
+Query execution
+~~~~~~~~~~~~~~~
+
+After you've created a CodeQL database, one or more queries are executed
+against it. CodeQL queries are written in a specially-designed object-oriented
+query language called QL. You can run the queries checked out from the CodeQL
+repo (or custom queries that you've written yourself) using the :ref:`CodeQL
+for VS Code extension <codeql-for-visual-studio-code>` or the :ref:`CodeQL CLI
+<codeql-cli>`. For more information about queries, see ":ref:`About CodeQL queries <about-codeql-queries>`." 
+
+.. _interpret-query-results:
+
+Query results
+~~~~~~~~~~~~~
+
+The final step converts results produced during query execution into a form that
+is more meaningful in the context of the source code. That is, the results are
+interpreted in a way that highlights the potential issue that the queries are
+designed to find.
+
+Queries contain metadata properties that indicate how the results should be
+interpreted. For instance, some queries display a simple message at a single
+location in the code. Others display a series of locations that represent steps
+along a data-flow or control-flow path, along with a message explaining the
+significance of the result. Queries that don't have metadata are not
+interpreted---their results are output as a table and not displayed in the source
+code.
+
+Following interpretation, results are output for code review and triaging. In
+CodeQL for Visual Studio Code, interpreted query results are automatically
+displayed in the source code. Results generated by the CodeQL CLI can be output
+into a number of different formats for use with different tools. 
+
+
+About CodeQL databases
+----------------------
+
+CodeQL databases contain queryable data extracted from a codebase, for a single
+language at a particular point in time. The database contains a full,
+hierarchical representation of the code, including a representation of the
+abstract syntax tree, the data flow graph, and the control flow graph. 
+
+Each language has its own unique database schema that defines the relations used
+to create a database. The schema provides an interface between the initial
+lexical analysis during the extraction process, and the actual complex analysis
+using CodeQL. The schema specifies, for instance, that there is a table for
+every language construct.
+
+For each language, the CodeQL libraries define classes to provide a layer of
+abstraction over the database tables. This provides an object-oriented view of
+the data which makes it easier to write queries. 
+
+For example, in a CodeQL database for a Java program, two key tables are:
+
+-  The ``expressions`` table containing a row for every single expression in the
+   source code that was analyzed during the build process. 
+-  The ``statements`` table containing a row for every single statement in the 
+   source code that was analyzed during the build process.
+
+The CodeQL library defines classes to provide a layer of abstraction over each
+of these tables (and the related auxiliary tables): ``Expr`` and ``Stmt``.

docs/codeql/codeql-overview/codeql-glossary.rst

@@ -0,0 +1,130 @@
+.. _codeql-glossary:
+
+CodeQL glossary
+===============
+
+An overview of the technical terms and concepts in coCodeQL. 
+
+.. _bqrs-file:
+
+``.bqrs`` file
+--------------
+
+A binary query result set (BQRS) file. BQRS is the binary representation of the raw
+result of a query, with the extension ``.bqrs``. A BQRS file can be
+interpreted into meaningful results and related to your source code. For
+example, alert query results are interpreted to display a string at the
+location in the source code where the alert occurs, as specified in the query.
+Similarly, path query results are interpreted as pairs of locations
+(sources and sinks) between which information can flow. These results can be
+exported as a variety of different formats, including SARIF.
+
+.. _codeql-database:
+
+CodeQL database
+---------------
+
+A database (or CodeQL database) is a directory containing: 
+
+- queryable data, extracted from the code.
+- a source reference, for displaying query results directly in the code.
+- query results. 
+- log files generated during database creation, query
+  execution, and other operations.
+
+.. _dil:
+
+DIL
+---
+
+DIL stands for Datalog Intermediary Language. It is an intermediate 
+representation between QL and relation algebra (RA) that is generated 
+during query compilation. DIL is useful for advanced users as an aid 
+for debugging query performance. 
+The DIL format may change without warning between CLI releases.
+
+When you specify the ``--dump-dil`` option for ``codeql query compile``, CodeQL 
+prints DIL to standard output for the queries it compiles. You can also 
+view results in DIL format when you run queries in VS Code. 
+For more information, see ":ref:`Analyzing your projects <viewing-query-results>`" in the CodeQL for VS Code help.
+
+.. _extractor:
+
+Extractor
+---------
+
+An extractor is a tool that produces the relational data and source
+reference for each input file, from which a CodeQL database can be built. 
+
+.. _codeql-database-schema:
+
+QL database schema
+------------------
+
+A QL database schema is a file describing the column types and
+extensional relations that make up a raw QL dataset. It is a text file
+with the ``.dbscheme`` extension.
+
+The extractor and core QL pack for a language each declare the database 
+schema that they use. This defines the database layout they create or
+expect. When you create a CodeQL database, the extractor copies
+its schema into the database. The CLI uses this to check whether the
+CodeQL database is compatible with a particular CodeQL library.
+If they aren't compatible you can use ``database upgrade`` to upgrade
+the schema for the CodeQL database.
+
+There is currently no public-facing specification for the syntax of schemas.
+
+.. _qlo:
+
+``.qlo`` files
+--------------
+
+``.qlo`` files are optionally generated during query compilation.
+If you specify the ``--dump-qlo`` option for ``codeql query compile``, 
+CodeQL writes ``.qlo`` files for the queries it compiles. They can be used
+as an aid for debugging and performance tuning for advanced users. 
+
+``.qlo`` is a binary format that represents a compiled 
+and optimized query in terms of relational algebra (RA) or the 
+intermediate :ref:`DIL <dil>` format. ``.qlo`` files can be expanded to 
+readable text using ``codeql query decompile``. 
+
+The exact details of the ``.qlo`` format may change without warning between CLI releases. 
+
+.. _sarif-file:
+
+SARIF file
+----------
+
+Static analysis results interchange format (SARIF) is an output format used for
+sharing static analysis results. For more information, see ":ref:`SARIF
+output <sarif-output>`." 
+
+.. _source-reference:
+
+Source reference
+----------------
+
+A source reference is a mechanism that allows the retrieval of the 
+contents of a source file, given an absolute filename at which that file 
+resided during extraction. Specific examples include:
+
+- A source archive directory, within which the requested absolute 
+  filename maps to a UTF8-encoded file.
+- A source archive, typically in ZIP format, which contains the UTF8-encoded 
+  content of all source files.
+- A source archive repository, typically in ``git`` format, typically bare, 
+  which contains the UTF8-encoded content of all source files.
+
+Source references are typically included in CodeQL databases.
+
+.. _trap-file:
+
+TRAP file
+---------
+
+A TRAP file is a UTF-8 encoded file generated by a CodeQL extractor 
+with the extension ``.trap``. To save space, they are usually archived. They
+contain the information that, when interpreted relative to a QL database
+schema, is used to create a QL dataset.

docs/codeql/codeql-overview/codeql-tools.rst

@@ -0,0 +1,24 @@
+.. _codeql-tools:
+
+CodeQL tools
+============
+
+GitHub provides the CodeQL command-line interface and CodeQL for Visual Studio
+Code for performing CodeQL analysis on open source codebases.
+
+CodeQL command-line interface
+-----------------------------
+
+The CodeQL command-line interface (CLI) is primarily used to create databases for 
+security research. You can also query CodeQL databases directly from the command line 
+or using the Visual Studio Code extension. For more information, see
+":ref:`CodeQL CLI <codeql-cli>`."
+
+
+CodeQL for Visual Studio Code
+-----------------------------
+
+You can analyze CodeQL databases in Visual Studio Code using the CodeQL
+extension, which provides an enhanced environment for writing and running custom
+queries and viewing the results. For more information, see ":ref:`CodeQL
+for Visual Studio Code <codeql-for-visual-studio-code>`."

docs/codeql/codeql-overview/index.rst

@@ -0,0 +1,15 @@
+.. _codeql-overview:
+
+CodeQL overview
+===============
+
+.. toctree:: 
+   :maxdepth: 1
+
+   about-codeql
+   supported-languages-and-frameworks
+   codeql-tools
+   codeql-glossary
+
+
+

docs/codeql/codeql-overview/supported-languages-and-frameworks.rst

@@ -0,0 +1,29 @@
+.. _supported-languages-and-frameworks:
+
+Supported languages and frameworks
+----------------------------------
+
+View the languages, libraries, and frameworks supported in the latest version of
+CodeQL.
+
+Languages and compilers
+#######################
+
+CodeQL supports the following languages and compilers.
+
+.. include:: ../support/reusables/versions-compilers.rst
+
+Frameworks and libraries
+########################
+
+The libraries and queries in the current version of CodeQL have been explicitly checked against the libraries and frameworks listed below.
+
+.. pull-quote::
+
+    Tip
+    
+    If you're interested in other libraries or frameworks, you can extend the analysis to cover them. 
+    For example, by extending the data flow libraries to include data sources
+    and sinks for additional libraries or frameworks.
+    
+.. include:: ../support/reusables/frameworks.rst

docs/codeql/conf.py

@@ -26,16 +26,25 @@
 # source_suffix = ['.rst', '.md']
 source_suffix = '.rst'
 
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'CodeQL documentation'
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+html_title = 'CodeQL'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'CodeQL'
+
 # If your documentation needs a minimal Sphinx version, state it here.
 #needs_sphinx = '1.0'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = [
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.mathjax',
-]
 
 # The encoding of source files.
 source_encoding = 'utf-8-sig'
@@ -44,10 +53,11 @@ source_encoding = 'utf-8-sig'
 pygments_style = 'sphinx'
 
 # Import the QL Lexer to use for syntax highlighting
+import os
 import sys
 
 def setup(sphinx):
-    sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), os.path.pardir, 'global-sphinx-files')))
+    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))	
     from qllexer import QLLexer
     sphinx.add_lexer("ql", QLLexer())
 
@@ -56,9 +66,9 @@ def setup(sphinx):
 # built documents.
 #
 # The short X.Y version.
-version = u'1.25'
+version = u'3.0'
 # The full version, including alpha/beta/rc tags.
-release = u'1.25'
+release = u'3.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -92,13 +102,12 @@ html_theme_options = {'font_size': '16px',
                       }
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['../global-sphinx-files/_templates']
+templates_path = ['_templates']
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['../global-sphinx-files/_static']
+html_static_path = ['_static']
 
-intersphinx_mapping = {}
-
-##############################################################################
+exclude_patterns = ['vale*', '_static', '_templates', 'reusables', 'images', 'support', 'ql-training', 'query-help','_build', '*.py*', 'README.rst']
+##############################################################################