mirror of
https://github.com/github/codeql.git
synced 2026-04-26 09:15:12 +02:00
Merge branch 'main' into promote-sqlalchemy
This commit is contained in:
Rasmus Wriedt Larsen
@@ -165,7 +165,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
<ul class="list-style-none d-flex text-gray">
|
||||
<li class="mr-3">© 2021 GitHub, Inc.</li>
|
||||
<li class="mr-3">©
|
||||
<script type="text/javascript">document.write(new Date().getFullYear());</script> GitHub, Inc.</li>
|
||||
<li class="mr-3"><a
|
||||
href="https://docs.github.com/github/site-policy/github-terms-of-service"
|
||||
class="link-gray">Terms </a></li>
|
||||
|
||||
@@ -81,7 +81,7 @@ The following properties are supported in ``qlpack.yml`` files.
|
||||
- All packs
|
||||
- A version number for this QL pack. This must be a valid semantic version that meets the `SemVer v2.0.0 specification <https://semver.org/spec/v2.0.0.html>`__.
|
||||
* - ``libraryPathDependencies``
|
||||
- ``codeql-javascript``
|
||||
- ``codeql/javascript-all``
|
||||
- Optional
|
||||
- The names of any QL packs that this QL pack depends on, as a sequence. This gives the pack access to any libraries, database schema, and query suites defined in the dependency.
|
||||
* - ``suites``
|
||||
@@ -104,7 +104,18 @@ The following properties are supported in ``qlpack.yml`` files.
|
||||
- ``.``
|
||||
- Packs with upgrades
|
||||
- The path to a directory within the pack that contains upgrade scripts, defined relative to the pack directory. The ``database upgrade`` action uses these scripts to update databases that were created by an older version of an extractor so they're compatible with the current extractor (see `Upgrade scripts for a language <#upgrade-scripts-for-a-language>`__ below.)
|
||||
* - ``dependencies``
|
||||
- .. code-block:: yaml
|
||||
|
||||
dependencies:
|
||||
codeql/cpp-all: ^0.0.2
|
||||
|
||||
- Packs that define CodeQL package dependencies on other packs
|
||||
- A map from pack references to the semantic version range that is compatible with this pack. Supported from CLI version v2.6.0 and onwards.
|
||||
* - ``defaultSuiteFile``
|
||||
- ``defaultSuiteFile: cpp-code-scanning.qls``
|
||||
- Packs that export a set of default queries to run
|
||||
- The path to a query suite file containing all of the queries that are run by default when this pack is passed to the ``codeql database analyze`` command. Supported from CLI version v2.6.0 and onwards.
|
||||
|
||||
.. _custom-ql-packs:
|
||||
|
||||
@@ -138,7 +149,7 @@ and libraries may contain:
|
||||
libraryPathDependencies: codeql/cpp-all
|
||||
suites: my-custom-suites
|
||||
|
||||
where ``codeql-cpp`` is the name of the QL pack for C/C++ analysis included in
|
||||
where ``codeql/cpp-all`` is the name of the QL pack for C/C++ analysis included in
|
||||
the CodeQL repository.
|
||||
|
||||
.. pull-quote::
|
||||
@@ -166,38 +177,61 @@ For more information about running tests, see ":doc:`Testing custom queries
|
||||
Examples of QL packs in the CodeQL repository
|
||||
---------------------------------------------
|
||||
|
||||
Each of the languages in the CodeQL repository has three main QL packs:
|
||||
Each of the languages in the CodeQL repository has four main QL packs:
|
||||
|
||||
- Core QL pack for the language, with the :ref:`database schema <codeql-database-schema>`
|
||||
used by the language, CodeQL libraries, and queries at ``ql/<language>/ql/src``
|
||||
- Tests for the core language libraries and queries pack at ``ql/<language>/ql/test``
|
||||
- Core library pack for the language, with the :ref:`database schema <codeql-database-schema>`
|
||||
used by the language, and CodeQL libraries, and queries at ``ql/<language>/ql/lib``
|
||||
- Core query pack for the language that includes the default queries for the language, along
|
||||
with their query suites at ``ql/<language>/ql/src``
|
||||
- Tests for the core language libraries and queries at ``ql/<language>/ql/test``
|
||||
- Upgrade scripts for the language at ``ql/<language>/upgrades``
|
||||
|
||||
Core QL pack
|
||||
~~~~~~~~~~~~
|
||||
Core library pack
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ``qlpack.yml`` file for a core QL pack uses the following properties:
|
||||
``name``, ``version``, ``dbscheme``, and ``suites``.
|
||||
The ``qlpack.yml`` file for a core library pack uses the following properties:
|
||||
``name``, ``version``, ``dbscheme``.
|
||||
The ``dbscheme`` property should only be defined in the core QL
|
||||
pack for a language.
|
||||
|
||||
For example, the ``qlpack.yml`` file for `C/C++ analysis
|
||||
<https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml>`__
|
||||
For example, the ``qlpack.yml`` file for `C/C++ analysis libraries
|
||||
<https://github.com/github/codeql/blob/main/cpp/ql/lib/qlpack.yml>`__
|
||||
contains:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
name: codeql-cpp
|
||||
name: codeql/cpp-all
|
||||
version: 0.0.0
|
||||
dbscheme: semmlecode.cpp.dbscheme
|
||||
dependencies:
|
||||
codeql/cpp-upgrades: "*"
|
||||
|
||||
Core query pack
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
The ``qlpack.yml`` file for a core query pack uses the following properties:
|
||||
``name``, ``version``, ``suites``, ``defaultSuiteFile``, ``dependencies`` .
|
||||
|
||||
For example, the ``qlpack.yml`` file for `C/C++ analysis queries
|
||||
<https://github.com/github/codeql/blob/main/cpp/ql/lib/qlpack.yml>`__
|
||||
contains:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
name: codeql/cpp-queries
|
||||
version: 0.0.0
|
||||
suites: codeql-suites
|
||||
defaultSuiteFile: codeql-suites/cpp-code-scanning.qls
|
||||
dependencies:
|
||||
codeql/cpp-all: "*"
|
||||
codeql/suite-helpers: "*"
|
||||
|
||||
Tests for the core QL pack
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ``qlpack.yml`` file for the tests for the core QL packs use the following
|
||||
properties: ``name``, ``version``, and ``libraryPathDependencies``.
|
||||
The ``libraryPathDependencies`` always specifies the core QL pack.
|
||||
properties: ``name``, ``version``, and ``dependencies``.
|
||||
The ``dependencies`` always specifies the core QL pack.
|
||||
|
||||
For example, the ``qlpack.yml`` file for `C/C++ analysis tests
|
||||
<https://github.com/github/codeql/blob/main/cpp/ql/test/qlpack.yml>`__
|
||||
@@ -205,15 +239,12 @@ contains:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
name: codeql-cpp-tests
|
||||
name: codeql/cpp-tests
|
||||
version: 0.0.0
|
||||
libraryPathDependencies: codeql/cpp-all
|
||||
dependencies:
|
||||
codeql/cpp-all: "*"
|
||||
codeql/cpp-queries: "*"
|
||||
|
||||
Notice that, unlike the example QL pack for custom tests, this file does not define
|
||||
an ``extractor`` or ``tests`` property. These properties have been added to
|
||||
the QL pack file since the release of CodeQL CLI 2.0.1.
|
||||
They haven't been added yet to ensure compatibility for LGTM Enterprise users.
|
||||
After the next release of LGTM Enterprise, these files can be updated.
|
||||
|
||||
.. _upgrade-ql-packs:
|
||||
|
||||
@@ -229,5 +260,5 @@ contains:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
name: codeql-cpp-upgrades
|
||||
name: codeql/cpp-upgrades
|
||||
upgrades: .
|
||||
|
||||
@@ -228,6 +228,123 @@ commands that you can specify for compiled languages.
|
||||
This command runs a custom script that contains all of the commands required
|
||||
to build the project.
|
||||
|
||||
Using indirect build tracing
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with ``codeql database trace-command``, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
|
||||
|
||||
To create a CodeQL database with indirect build tracing, run the following command from the checkout root of your project:
|
||||
|
||||
::
|
||||
|
||||
codeql database init ... --begin-tracing <database>
|
||||
|
||||
You must specify:
|
||||
|
||||
- ``<database>``: a path to the new database to be created. This directory will
|
||||
be created when you execute the command---you cannot specify an existing
|
||||
directory.
|
||||
- ``--begin-tracing``: creates scripts that can be used to set up an environment in which build commands will be traced.
|
||||
|
||||
You may specify other options for the ``codeql database init`` command as normal.
|
||||
|
||||
.. pull-quote:: Note
|
||||
|
||||
If the build runs on Windows, you must set either ``--trace-process-level <number>`` or ``--trace-process-name <parent process name>`` so that the option points to a parent CI process that will observe all build steps for the code being analyzed.
|
||||
|
||||
|
||||
The ``codeql database init`` command will output a message::
|
||||
|
||||
Created skeleton <database>. This in-progress database is ready to be populated by an extractor.
|
||||
In order to initialise tracing, some environment variables need to be set in the shell your build will run in.
|
||||
A number of scripts to do this have been created in <database>/temp/tracingEnvironment.
|
||||
Please run one of these scripts before invoking your build command.
|
||||
|
||||
Based on your operating system, we recommend you run: ...
|
||||
|
||||
The ``codeql database init`` command creates ``<database>/temp/tracingEnvironment`` with files that contain environment variables and values that will enable CodeQL to trace a sequence of build steps. These files are named ``start-tracing.{json,sh,bat,ps1}``. Use one of these files with your CI system's mechanism for setting environment variables for future steps. You can:
|
||||
|
||||
* Read the JSON file, process it, and print out environment variables in the format expected by your CI system. For example, Azure DevOps expects ``echo "##vso[task.setvariable variable=NAME]VALUE"``.
|
||||
* Or, if your CI system persists the environment, source the appropriate ``start-tracing`` script to set the CodeQL variables in the shell environment of the CI system.
|
||||
|
||||
Build your code; optionally, unset the environment variables using an ``end-tracing.{json,sh,bat,ps1}`` script from the directory where the ``start-tracing`` scripts are stored; and then run the command ``codeql database finalize <database>``.
|
||||
|
||||
Once you have created a CodeQL database using indirect build tracing, you can work with it like any other CodeQL database. For example, analyze the database, and upload the results to GitHub if you use code scanning.
|
||||
|
||||
Example of creating a CodeQL database using indirect build tracing
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following example shows how you could use indirect build tracing in an Azure DevOps pipeline to create a CodeQL database::
|
||||
|
||||
steps:
|
||||
# Download the CodeQL CLI and query packs...
|
||||
# Check out the repository ...
|
||||
|
||||
# Run any pre-build tasks, for example, restore NuGet dependencies...
|
||||
|
||||
# Initialize the CodeQL database.
|
||||
# In this example, the CodeQL CLI has been downloaded and placed on the PATH.
|
||||
- task: CmdLine@1
|
||||
displayName: Initialize CodeQL database
|
||||
inputs:
|
||||
# Assumes the source code is checked out to the current working directory.
|
||||
# Creates a database at `<current working directory>/db`.
|
||||
# Running on Windows, so specifies a trace process level.
|
||||
script: "codeql database init --language csharp --trace-process-name Agent.Worker.exe --source-root . --begin-tracing db"
|
||||
|
||||
# Read the generated environment variables and values,
|
||||
# and set them so they are available for subsequent commands
|
||||
# in the build pipeline. This is done in PowerShell in this example.
|
||||
- task: PowerShell@1
|
||||
displayName: Set CodeQL environment variables
|
||||
inputs:
|
||||
targetType: inline
|
||||
script: >
|
||||
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/start-tracing.json | ConvertFrom-Json
|
||||
$json.PSObject.Properties | ForEach-Object {
|
||||
$template = "##vso[task.setvariable variable="
|
||||
$template += $_.Name
|
||||
$template += "]"
|
||||
$template += $_.Value
|
||||
echo "$template"
|
||||
}
|
||||
|
||||
# Execute the pre-defined build step. Note the `msbuildArgs` variable.
|
||||
- task: VSBuild@1
|
||||
inputs:
|
||||
solution: '**/*.sln'
|
||||
# Disable MSBuild shared compilation for C# builds.
|
||||
msbuildArgs: /p:OutDir=$(Build.ArtifactStagingDirectory) /p:UseSharedCompilation=false
|
||||
platform: Any CPU
|
||||
configuration: Release
|
||||
# Execute a clean build, in order to remove any existing build artifacts prior to the build.
|
||||
clean: True
|
||||
displayName: Visual Studio Build
|
||||
|
||||
# Read and set the generated environment variables to end build tracing. This is done in PowerShell in this example.
|
||||
- task: PowerShell@1
|
||||
displayName: Clear CodeQL environment variables
|
||||
inputs:
|
||||
targetType: inline
|
||||
script: >
|
||||
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/end-tracing.json | ConvertFrom-Json
|
||||
$json.PSObject.Properties | ForEach-Object {
|
||||
$template = "##vso[task.setvariable variable="
|
||||
$template += $_.Name
|
||||
$template += "]"
|
||||
$template += $_.Value
|
||||
echo "$template"
|
||||
}
|
||||
|
||||
- task: CmdLine@2
|
||||
displayName: Finalize CodeQL database
|
||||
inputs:
|
||||
script: 'codeql database finalize db'
|
||||
|
||||
# Other tasks go here, for example:
|
||||
# `codeql database analyze`
|
||||
# then `codeql github upload-results` ...
|
||||
|
||||
Obtaining databases from LGTM.com
|
||||
---------------------------------
|
||||
|
||||
|
||||
@@ -4,8 +4,8 @@ Creating CodeQL query suites
|
||||
============================
|
||||
|
||||
CodeQL query suites provide a way of selecting queries, based on their
|
||||
filename, location on disk or in a QL pack, or metadata properties.
|
||||
Create query suites for the queries that you want to frequently use in
|
||||
filename, location on disk or in a QL pack, or metadata properties.
|
||||
Create query suites for the queries that you want to frequently use in
|
||||
your CodeQL analyses.
|
||||
|
||||
Query suites allow you to pass multiple queries to
|
||||
@@ -19,7 +19,7 @@ suite definition have been executed, the result is a set of selected queries.
|
||||
.. pull-quote:: Note
|
||||
|
||||
Any custom queries that you want to add to a query suite must be in a :doc:`QL
|
||||
pack <about-ql-packs>` and contain the correct query metadata.
|
||||
pack <about-ql-packs>` and contain the correct query metadata.
|
||||
For more information, see
|
||||
":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-the-codeql-cli>`."
|
||||
|
||||
@@ -50,10 +50,15 @@ queries using:
|
||||
- queries: <path-to-subdirectory>
|
||||
from: <ql-pack-name>
|
||||
|
||||
- A ``qlpack`` instruction---tells CodeQL to look for queries in a named QL pack::
|
||||
- A ``qlpack`` instruction---tells CodeQL to resolve queries in the default suite of the
|
||||
named QL pack::
|
||||
|
||||
- qlpack: <qlpack-name>
|
||||
|
||||
The default suite of a query pack includes a recommended set of queries
|
||||
inside of that query pack. Not all query packs have a default suite. If the given query pack does not
|
||||
define a default suite, the `qlpack` instruction will resolve to all of the queries within the pack.
|
||||
|
||||
.. pull-quote:: Note
|
||||
|
||||
When pathnames appear in query suite definitions, they must always
|
||||
@@ -73,7 +78,7 @@ Filtering the queries in a query suite
|
||||
After you have defined the initial set of queries to add to your suite by
|
||||
specifying ``query``, ``queries``, or ``qlpack`` instructions, you can add
|
||||
``include`` and ``exclude`` instructions. These instructions define selection
|
||||
criteria based on specific properties:
|
||||
criteria based on specific properties:
|
||||
|
||||
- When you execute an ``include`` instruction on a set of queries, any
|
||||
queries that match your conditions are retained in the selection, and queries
|
||||
@@ -99,12 +104,12 @@ For both instructions, the argument is a constraint block---that is, a YAML map
|
||||
representing the constraints. Each constraint is a map entry, where the key is
|
||||
typically a query metadata property. The value can be:
|
||||
|
||||
- A single string.
|
||||
- A single string.
|
||||
- A ``/``\ -enclosed `regular expression <https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html>`__.
|
||||
- A list containing strings, regular expressions, or both.
|
||||
|
||||
To match a constraint, a metadata value must match one of the strings or
|
||||
regular expressions. When there is more than one metadata key, each key must be matched.
|
||||
regular expressions. When there is more than one metadata key, each key must be matched.
|
||||
For more information about query metadata properties, see ":ref:`Metadata for CodeQL queries
|
||||
<metadata-for-codeql-queries>`."
|
||||
|
||||
@@ -112,7 +117,7 @@ In addition to metadata tags, the keys in the constraint block can also be:
|
||||
|
||||
- ``query filename``---matches on the last path component of the query file name.
|
||||
- ``query path``---matches on the path to the query file relative to its
|
||||
enclosing QL pack.
|
||||
enclosing QL pack.
|
||||
- ``tags contain``---one of the given match strings must match
|
||||
one of the space-separated components of the value of the ``@tags`` metadata property.
|
||||
- ``tags contain all``---each of the given match strings must match one of the
|
||||
@@ -121,6 +126,14 @@ In addition to metadata tags, the keys in the constraint block can also be:
|
||||
Examples
|
||||
~~~~~~~~
|
||||
|
||||
To define a suite that selects all queries in the default suite of the
|
||||
``codeql/cpp-queries`` QL pack, and then refines them to only include
|
||||
security queries, use::
|
||||
|
||||
- qlpack: codeql/cpp-queries
|
||||
- include:
|
||||
tags contain: security
|
||||
|
||||
To define a suite that selects all queries with ``@kind problem``
|
||||
and ``@precision high`` from the ``my-custom-queries`` directory, use::
|
||||
|
||||
@@ -136,34 +149,35 @@ recommendation``, use::
|
||||
- queries: my-custom-queries
|
||||
- include:
|
||||
kind: problem
|
||||
- exclude:
|
||||
- exclude:
|
||||
problem.severity: recommendation
|
||||
|
||||
To create a suite that selects all queries with ``@tag security`` and
|
||||
``@problem.severity high`` or ``very-high`` from the ``codeql-cpp`` QL pack,
|
||||
``@problem.severity high`` or ``very-high`` from the ``codeql/cpp-queries`` QL pack,
|
||||
use::
|
||||
|
||||
- qlpack: codeql-cpp
|
||||
- include:
|
||||
- queries: .
|
||||
from: codeql/cpp-queries
|
||||
- include:
|
||||
tags contain: security
|
||||
problem.severity:
|
||||
problem.severity:
|
||||
- high
|
||||
- very-high
|
||||
|
||||
Reusing existing query suite definitions
|
||||
-----------------------------------------
|
||||
|
||||
Existing query suite definitions can be reused by specifying:
|
||||
Existing query suite definitions can be reused by specifying:
|
||||
|
||||
- An ``import`` instruction---adds the queries selected by a
|
||||
previously defined ``.qls`` file to the current suite::
|
||||
|
||||
|
||||
- import: <path-to-query-suite>
|
||||
|
||||
The path to the imported suite must be relative to the QL pack containing the
|
||||
current suite definition. If the imported query suite is in a different QL
|
||||
pack you can use::
|
||||
|
||||
|
||||
- import: <path-to-query-suite>
|
||||
from: <ql-pack>
|
||||
|
||||
@@ -175,12 +189,12 @@ Existing query suite definitions can be reused by specifying:
|
||||
applied ``.qls`` file are executed as if they appear in place of ``apply``.
|
||||
Any ``include`` and ``exclude`` instructions from the applied suite also act on
|
||||
queries added by any earlier instructions::
|
||||
|
||||
|
||||
- apply: <path-to-query-suite>
|
||||
|
||||
The ``apply`` instruction can also be used to apply a set of reusable
|
||||
conditions, saved in a ``.yml`` file, to multiple query definitions. For more
|
||||
information, see the `example <#example>`__ below.
|
||||
information, see the `example <#example>`__ below.
|
||||
|
||||
- An ``eval`` instruction---performs the same function as an ``import``
|
||||
instruction, but takes a full suite definition as the argument, rather than the
|
||||
@@ -236,7 +250,7 @@ instruction::
|
||||
This value is displayed when you run `codeql resolve queries
|
||||
<../manual/resolve-queries>`__, if the suite is added to a "well-known"
|
||||
directory. For more information, see "`Specifying well-known query suites
|
||||
<#specifying-well-known-query-suites>`__."
|
||||
<#specifying-well-known-query-suites>`__."
|
||||
|
||||
Saving a query suite
|
||||
--------------------
|
||||
@@ -248,8 +262,8 @@ Specifying well-known query suites
|
||||
----------------------------------
|
||||
|
||||
You can use QL packs to declare directories that contain "well-known" query
|
||||
suites. You can use "well-known" query suites on the command line by referring
|
||||
to their file name,
|
||||
suites. You can use "well-known" query suites on the command line by referring
|
||||
to their file name,
|
||||
without providing their full path. This gives you a simple way of specifying a
|
||||
set of queries, without needing to search inside QL packs and distributions.
|
||||
To declare a directory that contains "well-known" query suites, add the directory
|
||||
@@ -263,7 +277,7 @@ You can specify query suites on the command line for any command that accepts
|
||||
``.qls`` files. For example, you can compile the queries selected by a suite
|
||||
definition using ``query compile``, or use the queries in an analysis using
|
||||
``database analyze``. For more information about analyzing CodeQL databases, see
|
||||
":doc:`Analyzing databases with the CodeQL CLI <analyzing-databases-with-the-codeql-cli>`."
|
||||
":doc:`Analyzing databases with the CodeQL CLI <analyzing-databases-with-the-codeql-cli>`."
|
||||
|
||||
Viewing the query suites used on LGTM.com
|
||||
-----------------------------------------
|
||||
|
||||
@@ -79,6 +79,8 @@ You can see all quick queries that you've run in the current session in the Quer
|
||||
|
||||
Once you're happy with your quick query, you should save it in a QL pack so you can access it later. For more information, see ":ref:`About QL packs <about-ql-packs>`."
|
||||
|
||||
.. _running-a-specific-part-of-a-query-or-library:
|
||||
|
||||
Running a specific part of a query or library
|
||||
----------------------------------------------
|
||||
|
||||
|
||||
@@ -349,11 +349,12 @@
|
||||
</li>
|
||||
</ul>
|
||||
<ul class="list-style-none d-flex text-gray">
|
||||
<li class="mr-3">© 2021 GitHub, Inc.</li>
|
||||
<li class="mr-3">©
|
||||
<script type="text/javascript">document.write(new Date().getFullYear());</script> GitHub, Inc.</li>
|
||||
<li class="mr-3"><a
|
||||
href="https://docs.github.com/en/free-pro-team@latest/github/site-policy/github-terms-of-service"
|
||||
href="https://docs.github.com/github/site-policy/github-terms-of-service"
|
||||
class="link-gray">Terms </a></li>
|
||||
<li><a href="https://docs.github.com/en/free-pro-team@latest/github/site-policy/github-privacy-statement"
|
||||
<li><a href="https://docs.github.com/github/site-policy/github-privacy-statement"
|
||||
class="link-gray">Privacy </a></li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
@@ -201,9 +201,10 @@ based on the `Unicode value <https://en.wikipedia.org/wiki/List_of_Unicode_chara
|
||||
of each character.
|
||||
|
||||
To specify a different order, follow ``<expression>`` with the keywords ``order by``, then
|
||||
the expression that specifies the order, and optionally the keyword ``asc`` or ``desc``
|
||||
one or more comma-separated expressions that specify the order, and optionally the keyword ``asc`` or ``desc`` after each expression
|
||||
(to determine whether to order the expression in ascending or descending order). If you don't
|
||||
specify an ordering, it defaults to ``asc``.
|
||||
specify an ordering, it defaults to ``asc``. For example, ``order by o.getName() asc, o.getSize() desc``
|
||||
might be used to order some object by name, breaking ties by descending size.
|
||||
|
||||
The following aggregates are available in QL:
|
||||
|
||||
@@ -229,11 +230,11 @@ The following aggregates are available in QL:
|
||||
In this case, ``<expression>`` must be of numeric type or of type ``string``.
|
||||
|
||||
For example, the following aggregation returns the name of the ``.js`` file (or files) with the
|
||||
largest number of lines:
|
||||
largest number of lines, using the number of lines of code to break ties:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
max(File f | f.getExtension() = "js" | f.getBaseName() order by f.getTotalNumberOfLines())
|
||||
max(File f | f.getExtension() = "js" | f.getBaseName() order by f.getTotalNumberOfLines(), f.getNumberOfLinesOfCode())
|
||||
|
||||
The following aggregation returns the minimum string ``s`` out of the three strings mentioned
|
||||
below, that is, the string that comes first in the lexicographic ordering of all the possible
|
||||
|
||||
@@ -78,7 +78,7 @@ To define a class, you write:
|
||||
#. The keyword ``class``.
|
||||
#. The name of the class. This is an `identifier <https://codeql.github.com/docs/ql-language-reference/ql-language-specification/#identifiers>`_
|
||||
starting with an uppercase letter.
|
||||
#. The types to extend.
|
||||
#. The supertypes that the class is derived from via `extends` and/or `instanceof`
|
||||
#. The :ref:`body of the class <class-bodies>`, enclosed in braces.
|
||||
|
||||
For example:
|
||||
@@ -106,12 +106,14 @@ This defines a class ``OneTwoThree``, which contains the values ``1``, ``2``, an
|
||||
.. index:: extends
|
||||
|
||||
``OneTwoThree`` extends ``int``, that is, it is a subtype of ``int``. A class in QL must always
|
||||
extend at least one existing type. Those types are called the **base types** of the class. The
|
||||
values of a class are contained within the intersection of the base types (that is, they are in
|
||||
the :ref:`class domain type <domain-types>`). A class inherits all member predicates from its
|
||||
base types.
|
||||
have at least one supertype. Supertypes that are referenced with the `extends` keyword are called
|
||||
the **base types** of the class. The values of a class are contained within the intersection of
|
||||
the supertypes (that is, they are in the :ref:`class domain type <domain-types>`).
|
||||
A class inherits all member predicates from its base types.
|
||||
|
||||
A class can extend multiple types. For more information, see ":ref:`multiple-inheritance`."
|
||||
Classes can also specialise other types without extending the class interface via `instanceof`,
|
||||
see ":ref:`instanceof-extensions`.".
|
||||
|
||||
To be valid, a class:
|
||||
- Must not extend itself.
|
||||
@@ -228,7 +230,7 @@ Concrete classes
|
||||
|
||||
The classes in the above examples are all **concrete** classes. They are defined by
|
||||
restricting the values in a larger type. The values in a concrete class are precisely those
|
||||
values in the intersection of the base types that also satisfy the
|
||||
values in the intersection of the supertypes that also satisfy the
|
||||
:ref:`characteristic predicate <characteristic-predicates>` of the class.
|
||||
|
||||
.. _abstract-classes:
|
||||
@@ -380,6 +382,59 @@ from ``OneTwoThree`` and ``int``.
|
||||
must :ref:`override <overriding-member-predicates>` those definitions to avoid ambiguity.
|
||||
:ref:`Super expressions <super>` are often useful in this situation.
|
||||
|
||||
.. _instanceof-extensions:
|
||||
|
||||
Non-extending subtypes
|
||||
======================
|
||||
|
||||
Besides extending base types, classes can also declare `instanceof` relationships with other types.
|
||||
Declaring a class as `instanceof Foo` is roughly equivalent to saying `this instanceof Foo` in the characteristic predicate.
|
||||
The main differences are that you can call methods on Bar via `super` and you can get better optimisation.
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
class Foo extends int {
|
||||
Foo() { this in [1 .. 10] }
|
||||
|
||||
string foo_method() { result = "foo" }
|
||||
}
|
||||
|
||||
class Bar instanceof Foo {
|
||||
string toString() { result = super.foo_method() }
|
||||
}
|
||||
|
||||
In this example, the characteristic predicate from `Foo` also applies to `Bar`.
|
||||
However, `foo_method` is not exposed in `Bar`, so the query `select any(Bar b).foo_method()`
|
||||
results in a compile time error. Note from the example that it is still possible to access
|
||||
methods from instanceof supertypes from within the specialising class with the `super` keyword.
|
||||
|
||||
Crucially, the instanceof **supertypes** are not **base types**.
|
||||
This means that these supertypes do not participate in overriding, and any fields of such
|
||||
supertypes are not part of the new class.
|
||||
This has implications on method resolution when complex class hierarchies are involved.
|
||||
The following example demonstrates this.
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
class Interface extends int {
|
||||
Interface() { this in [1 .. 10] }
|
||||
string foo() { result = "" }
|
||||
}
|
||||
|
||||
class Foo extends int {
|
||||
Foo() { this in [1 .. 5] }
|
||||
string foo() { result = "foo" }
|
||||
}
|
||||
|
||||
class Bar extends Interface instanceof Foo {
|
||||
override string foo() { result = "bar" }
|
||||
}
|
||||
|
||||
Here, the method `Bar::foo` does not override `Foo::foo`.
|
||||
Instead, it overrides only `Interface::foo`.
|
||||
This means that `select any(Foo f).foo()` yields only `foo`.
|
||||
Had `Bar` been defined as `extends Foo`, then `select any(Foo b)` would yield `bar`.
|
||||
|
||||
.. _character-types:
|
||||
.. _domain-types:
|
||||
|
||||
|
||||
@@ -22,14 +22,14 @@
|
||||
Eclipse compiler for Java (ECJ) [5]_",``.java``
|
||||
JavaScript,ECMAScript 2021 or lower,Not applicable,"``.js``, ``.jsx``, ``.mjs``, ``.es``, ``.es6``, ``.htm``, ``.html``, ``.xhm``, ``.xhtml``, ``.vue``, ``.json``, ``.yaml``, ``.yml``, ``.raml``, ``.xml`` [6]_"
|
||||
Python,"2.7, 3.5, 3.6, 3.7, 3.8, 3.9",Not applicable,``.py``
|
||||
TypeScript [7]_,"2.6-4.2",Standard TypeScript compiler,"``.ts``, ``.tsx``"
|
||||
TypeScript [7]_,"2.6-4.4",Standard TypeScript compiler,"``.ts``, ``.tsx``"
|
||||
|
||||
.. container:: footnote-group
|
||||
|
||||
.. [1] C++20 support is currently in beta. Supported for GCC on Linux only. Modules are *not* supported.
|
||||
.. [2] Support for the clang-cl compiler is preliminary.
|
||||
.. [3] Support for the Arm Compiler (armcc) is preliminary.
|
||||
.. [4] Builds that execute on Java 7 to 16 can be analyzed. The analysis understands Java 15 standard language features.
|
||||
.. [4] Builds that execute on Java 7 to 16 can be analyzed. The analysis understands Java 16 standard language features.
|
||||
.. [5] ECJ is supported when the build invokes it via the Maven Compiler plugin or the Takari Lifecycle plugin.
|
||||
.. [6] JSX and Flow code, YAML, JSON, HTML, and XML files may also be analyzed with JavaScript files.
|
||||
.. [7] TypeScript analysis is performed by running the JavaScript extractor with TypeScript enabled. This is the default for LGTM.
|
||||
|
||||
@@ -16,6 +16,7 @@ CodeQL queries are used in code scanning analyses to find problems in source cod
|
||||
about-data-flow-analysis
|
||||
creating-path-queries
|
||||
troubleshooting-query-performance
|
||||
debugging-data-flow-queries-using-partial-flow
|
||||
|
||||
- :doc:`About CodeQL queries <about-codeql-queries>`: CodeQL queries are used to analyze code for issues related to security, correctness, maintainability, and readability.
|
||||
- :doc:`Metadata for CodeQL queries <metadata-for-codeql-queries>`: Metadata tells users important information about CodeQL queries. You must include the correct query metadata in a query to be able to view query results in source code.
|
||||
@@ -25,3 +26,4 @@ CodeQL queries are used in code scanning analyses to find problems in source cod
|
||||
- :doc:`About data flow analysis <about-data-flow-analysis>`: Data flow analysis is used to compute the possible values that a variable can hold at various points in a program, determining how those values propagate through the program and where they are used.
|
||||
- :doc:`Creating path queries <creating-path-queries>`: You can create path queries to visualize the flow of information through a codebase.
|
||||
- :doc:`Troubleshooting query performance <troubleshooting-query-performance>`: Improve the performance of your CodeQL queries by following a few simple guidelines.
|
||||
- :doc:`Debugging data-flow queries using partial flow <debugging-data-flow-queries-using-partial-flow>`: If a data-flow query doesn't produce the results you expect to see, you can use partial flow to debug the problem..
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
.. _debugging-data-flow-queries-using-partial-flow:
|
||||
|
||||
Debugging data-flow queries using partial flow
|
||||
==============================================
|
||||
|
||||
If a data-flow query doesn't produce the results you expect to see, you can use partial flow to debug the problem.
|
||||
|
||||
In CodeQL, you can use :ref:`data flow analysis <about-data-flow-analysis>` to compute the possible values that a variable can hold at various points in a program.
|
||||
A typical data-flow query looks like this:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
::
|
||||
|
||||
class MyConfig extends TaintTracking::Configuration {
|
||||
MyConfig() { this = "MyConfig" }
|
||||
|
||||
override predicate isSource(DataFlow::Node node) { node instanceof MySource }
|
||||
|
||||
override predicate isSink(DataFlow::Node node) { node instanceof MySink }
|
||||
}
|
||||
|
||||
from MyConfig config, DataFlow::PathNode source, DataFlow::PathNode sink
|
||||
where config.hasFlowPath(source, sink)
|
||||
select sink.getNode(), source, sink, "Sink is reached from $@.", source.getNode(), "here"
|
||||
|
||||
The same query can be slightly simplified by rewriting it without :ref:`path explanations <creating-path-queries>`:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
from MyConfig config, DataFlow::Node source, DataFlow::Node sink
|
||||
where config.hasPath(source, sink)
|
||||
select sink, "Sink is reached from $@.", source.getNode(), "here"
|
||||
|
||||
If a data-flow query that you have written doesn't produce the results you expect it to, there may be a problem with your query.
|
||||
You can try to debug the potential problem by following the steps described below.
|
||||
|
||||
Checking sources and sinks
|
||||
--------------------------
|
||||
|
||||
Initially, you should make sure that the source and sink definitions contain what you expect. If either the source or sink is empty then there can never be any data flow. The easiest way to check this is using quick evaluation in CodeQL for VS Code. Select the text ``node instanceof MySource``, right-click, and choose "CodeQL: Quick Evaluation". This will evaluate the highlighted text, which in this case means the set of sources. For more information, see :ref:`Analyzing your projects <running-a-specific-part-of-a-query-or-library>` in the CodeQL for VS Code help.
|
||||
|
||||
If both source and sink definitions look good then we will need to look for missing flow steps.
|
||||
|
||||
``fieldFlowBranchLimit``
|
||||
------------------------
|
||||
|
||||
Data-flow configurations contain a parameter called ``fieldFlowBranchLimit``. If the value is set too high, you may experience performance degradation, but if it's too low you may miss results. When debugging data flow try setting ``fieldFlowBranchLimit`` to a high value and see whether your query generates more results. For example, try adding the following to your configuration:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
override int fieldFlowBranchLimit() { result = 5000 }
|
||||
|
||||
If there are still no results and performance is still useable, then it is best to leave this set to a high value while doing further debugging.
|
||||
|
||||
Partial flow
|
||||
------------
|
||||
|
||||
A naive next step could be to change the sink definition to ``any()``. This would mean that we would get a lot of flow to all the places that are reachable from the sources. While this approach may work in some cases, you might find that it produces so many results that it's very hard to explore the findings. It can can also dramatically affect query performance. More importantly, you might not even see all the partial flow paths. This is because the data-flow library tries very hard to prune impossible paths and, since field stores and reads must be evenly matched along a path, we will never see paths going through a store that fail to reach a corresponding read. This can make it hard to see where flow actually stops.
|
||||
|
||||
To avoid these problems, a data-flow ``Configuration`` comes with a mechanism for exploring partial flow that tries to deal with these caveats. This is the ``Configuration.hasPartialFlow`` predicate:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
/**
|
||||
* Holds if there is a partial data flow path from `source` to `node`. The
|
||||
* approximate distance between `node` and the closest source is `dist` and
|
||||
* is restricted to be less than or equal to `explorationLimit()`. This
|
||||
* predicate completely disregards sink definitions.
|
||||
*
|
||||
* This predicate is intended for dataflow exploration and debugging and may
|
||||
* perform poorly if the number of sources is too big and/or the exploration
|
||||
* limit is set too high without using barriers.
|
||||
*
|
||||
* This predicate is disabled (has no results) by default. Override
|
||||
* `explorationLimit()` with a suitable number to enable this predicate.
|
||||
*
|
||||
* To use this in a `path-problem` query, import the module `PartialPathGraph`.
|
||||
*/
|
||||
final predicate hasPartialFlow(PartialPathNode source, PartialPathNode node, int dist) {
|
||||
|
||||
As noted in the documentation for ``hasPartialFlow`` (for example, in the `CodeQL for Java documentation <https://codeql.github.com/codeql-standard-libraries/java/semmle/code/java/dataflow/internal/DataFlowImpl2.qll/predicate.DataFlowImpl2$Configuration$hasPartialFlow.3.html>__`) you must first enable this by adding an override of ``explorationLimit``. For example:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
override int explorationLimit() { result = 5 }
|
||||
|
||||
This defines the exploration radius within which ``hasPartialFlow`` returns results.
|
||||
|
||||
It is also useful to focus on a single source at a time as the starting point for the flow exploration. This is most easily done by adding a temporary restriction in the ``isSource`` predicate.
|
||||
|
||||
To do quick evaluations of partial flow it is often easiest to add a predicate to the query that is solely intended for quick evaluation (right-click the predicate name and choose "CodeQL: Quick Evaluation"). A good starting point is something like:
|
||||
|
||||
.. code-block:: ql
|
||||
|
||||
predicate adhocPartialFlow(Callable c, PartialPathNode n, Node src, int dist) {
|
||||
exists(MyConfig conf, PartialPathNode source |
|
||||
conf.hasPartialFlow(source, n, dist) and
|
||||
src = source.getNode() and
|
||||
c = n.getNode().getEnclosingCallable()
|
||||
)
|
||||
}
|
||||
|
||||
If you are focusing on a single source then the ``src`` column is superfluous. You may of course also add other columns of interest based on ``n``, but including the enclosing callable and the distance to the source at the very least is generally recommended, as they can be useful columns to sort on to better inspect the results.
|
||||
|
||||
|
||||
If you see a large number of partial flow results, you can focus them in a couple of ways:
|
||||
|
||||
- If flow travels a long distance following an expected path, that can result in a lot of uninteresting flow being included in the exploration radius. To reduce the amount of uninteresting flow, you can replace the source definition with a suitable ``node`` that appears along the path and restart the partial flow exploration from that point.
|
||||
- Creative use of barriers and sanitizers can be used to cut off flow paths that are uninteresting. This also reduces the number of partial flow results to explore while debugging.
|
||||
|
||||
Further reading
|
||||
----------------
|
||||
|
||||
- :ref:`About data flow analysis <about-data-flow-analysis>`
|
||||
- :ref:`Creating path queries <creating-path-queries>`
|
||||
Reference in New Issue
Block a user