Merge branch 'main' into rc/3.7

This commit is contained in:
Andrew Eisenberg
2022-09-20 08:33:58 -07:00
2309 changed files with 133758 additions and 43219 deletions

View File

@@ -1,6 +1,6 @@
.. _about-codeql-workspaces:
About CodeQL Workspaces
About CodeQL workspaces
=======================
.. include:: ../reusables/beta-note-package-management.rst
@@ -20,7 +20,7 @@ A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file
* The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace.
* The ``registries`` block contains a list of GHES URLs and package patterns that control which container registry is used for publishing CodeQL packs. For more information, see :ref:`Working with CodeQL packs on GitHub Enterprise Server <working-with-codeql-packs-on-ghes>`.
Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see" `@actions/glob <https://github.com/actions/toolkit/tree/main/packages/glob#patterns>`__ .
Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see "`@actions/glob <https://github.com/actions/toolkit/tree/main/packages/glob#patterns>`__ ."
For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all the CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory. The ``registries`` block specifies that ``codeql/*`` packs should be downloaded from https://ghcr.io/v2/, which is GitHub's default container registry. All other packs should be downloaded from and published to the regsitry at ``GHE_HOSTNAME``.
@@ -55,7 +55,7 @@ This is particularly useful in the following situations:
CodeQL workspaces and query resolution
--------------------------------------
All CodeQL packs in a workspace are available as source dependencies for each other when you run any CodeQL command that resolves queries or packs. For example, when you run ``codeql pack install`` in a pack directory in a workspace, any dependency that can be found in the workspace will be used instead of downloading that dependency to the package cache and adding it to the ``codeql-pack.lock.yml`` file. For more information, see ":ref:Adding and Installing Dependencies <adding-and-installing-dependencies>`."
All CodeQL packs in a workspace are available as source dependencies for each other when you run any CodeQL command that resolves queries or packs. For example, when you run ``codeql pack install`` in a pack directory in a workspace, any dependency that can be found in the workspace will be used instead of downloading that dependency to the package cache and adding it to the ``codeql-pack.lock.yml`` file. For more information, see ":ref:`Adding and Installing Dependencies <adding-and-installing-dependencies>`."
Similarly, when you publish a CodeQL query pack to the GitHub container registry using ``codeql pack publish`` the command will always use the dependencies from the workspace instead of using dependencies found in the local package cache.

View File

@@ -0,0 +1,12 @@
.. _about-ql-packs:
About QL packs
==============
This page has been moved to ":doc:`About CodeQL packs <about-codeql-packs>`."
.. toctree::
:hidden:
about-ql-packs

View File

@@ -11,11 +11,11 @@ or download them from GitHub.com.
CodeQL analysis relies on extracting relational data from your code, and
using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL
databases contain all of the important information about a codebase, which can
be analyzed by executing CodeQL queries against it. GitHub creates and
be analyzed by executing CodeQL queries against it. GitHub creates and
stores CodeQL databases for a large number of open-source projects. For more information,
see ":ref:`Downloading CodeQL databases from GitHub.com <downloading-codeql-databases-from-github-com>`."
You can also create CodeQL databases yourself using the CodeQL CLI.
You can also create CodeQL databases yourself using the CodeQL CLI.
Before you generate a CodeQL database, you need to:
- Install and set up the CodeQL CLI. For more information, see
@@ -214,10 +214,10 @@ commands that you can specify for compiled languages.
codeql database create cpp-database --language=cpp --command=make
- C# project built using ``dotnet build``::
- C# project built using ``dotnet build``:
It is a good idea to add `/t:rebuild` to ensure that all code will be built, or do a
prior `dotnet clean` (code that is not built will not be included in the CodeQL database):
It is a good idea to add `/t:rebuild` to ensure that all code will be built, or do a
prior `dotnet clean` (code that is not built will not be included in the CodeQL database)::
codeql database create csharp-database --language=csharp --command='dotnet build /t:rebuild'
@@ -391,9 +391,9 @@ Downloading databases from GitHub.com
-------------------------------------
.. include:: ../reusables/download-github-database.rst
Before running an analysis with the CodeQL CLI, you must unzip the databases.
Further reading
---------------

View File

@@ -13,9 +13,40 @@ CodeQL command-line interface
The CodeQL command-line interface (CLI) is primarily used to create databases for
security research. You can also query CodeQL databases directly from the command line
or using the Visual Studio Code extension. For more information, see
":ref:`CodeQL CLI <codeql-cli>`."
or using the Visual Studio Code extension.
The CodeQL CLI can be downloaded from `GitHub releases <https://github.com/github/codeql-cli-binaries/releases>`__.
For more information, see ":ref:`CodeQL CLI <codeql-cli>`" and the `CLI changelog <https://github.com/github/codeql-cli-binaries/blob/main/CHANGELOG.md>`__.
CodeQL packs
-----------------------------
The standard CodeQL query and library packs
(`source <https://github.com/github/codeql/tree/codeql-cli/latest>`__)
maintained by GitHub are:
- ``codeql/cpp-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/src>`__)
- ``codeql/cpp-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/lib>`__)
- ``codeql/csharp-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/src>`__)
- ``codeql/csharp-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/lib>`__)
- ``codeql/go-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/src>`__)
- ``codeql/go-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/lib>`__)
- ``codeql/java-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/src>`__)
- ``codeql/java-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/lib>`__)
- ``codeql/javascript-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/src>`__)
- ``codeql/javascript-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/lib>`__)
- ``codeql/python-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/src>`__)
- ``codeql/python-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/lib>`__)
- ``codeql/ruby-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/src>`__)
- ``codeql/ruby-all`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/lib/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/lib>`__)
For more information, see ":ref:`About CodeQL packs <about-codeql-packs>`."
CodeQL bundle
-----------------------------
The CodeQL bundle consists of the CodeQL CLI together with the standard CodeQL query and library packs
maintained by GitHub. The bundle can be downloaded from `GitHub releases <https://github.com/github/codeql-action/releases>`__.
Use this when running `code scanning with CodeQL <https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql>`__ on GitHub Actions or in another CI system.
CodeQL for Visual Studio Code
-----------------------------

View File

@@ -15,6 +15,8 @@ Learn all about QL, the powerful query language that underlies the code scanning
- :doc:`Modules <modules>`: Modules provide a way of organizing QL code by grouping together related types, predicates, and other modules.
- :doc:`Signatures <signatures>`: Signatures provide a typing mechanism to parameters of parameterized modules.
- :doc:`Aliases <aliases>`: An alias is an alternative name for an existing QL entity.
- :doc:`Variables <variables>`: Variables in QL are used in a similar way to variables in algebra or logic. They represent sets of values, and those values are usually restricted by a formula.
@@ -44,6 +46,7 @@ Learn all about QL, the powerful query language that underlies the code scanning
queries
types
modules
signatures
aliases
variables
expressions

View File

@@ -133,6 +133,92 @@ defined :ref:`above <library-modules>`:
This defines an explicit module named ``M``. The body of this module defines
the class ``OneTwo``.
.. _parameterized-modules:
Parameterized modules
=====================
Parameterized modules are QL's approach to generic programming.
Similar to explicit modules, parameterized modules are defined within other modules using the keywork ``module``.
In addition to the module name, parameterized modules declare one or more parameters between the name and the module body.
For example, consider the module ``M``, which takes two predicate parameters and defines a new predicate
that applies them one after the other:
.. code-block:: ql
module M<transformer/1 first, transformer/1 second> {
bindingset[x]
int applyBoth(int x) {
result = second(first(x))
}
}
Parameterized modules cannot be directly referenced.
Instead, you instantiate a parameterized module by passing arguments enclosed in angle brackets (``<`` and ``>``) to the module.
Instantiated parameterized modules can be used as a :ref:`module expression <name-resolution>`, identical to explicit module references.
For example, we can instantiate ``M`` with two identical arguments ``increment``, creating a module
containing a predicate that adds 2:
.. code-block:: ql
bindingset[result] bindingset[x]
int increment(int x) { result = x + 1 }
module IncrementTwice = M<increment/1, increment/1>;
select IncrementTwice::applyBoth(40) // 42
The parameters of a parameterized module are (meta-)typed with :ref:`signatures <signatures>`.
For example, in the previous two snippets, we relied on the predicate signature ``transformer``:
.. code-block:: ql
bindingset[x]
signature int transformer(int x);
The instantiation of parameterized modules is applicative.
That is, if you instantiate a parameterized module twice with identical arguments, the resulting object is the same.
This is particularly relevant for type definitions inside parameterized modules as :ref:`classes <classes>`
or via :ref:`newtype <algebraic-datatypes>`, because the duplication of such type definitions would result in
incompatible types.
The following example instantiates module ``M`` inside calls to predicate ``foo`` twice.
The first call is valid but the second call generates an error.
.. code-block:: ql
bindingset[this]
signature class TSig;
module M<TSig T> {
newtype A = B() or C()
}
string foo(M<int>::A a) { ... }
select foo(M<int>::B()), // valid: repeated identical instantiation of M does not duplicate A, B, C
foo(M<float>::B()) // ERROR: M<float>::B is not compatible with M<int>::A
Module parameters are dependently typed, meaning that signature expressions in parameter definitions can reference
preceding parameters.
For example, we can declare the signature for ``T2`` dependent on ``T1``, enforcing a subtyping relationship
between the two parameters:
.. code-block:: ql
signature class TSig;
module Extends<TSig T> { signature class Type extends T; }
module ParameterizedModule<TSig T1, Extends<T1>::Type T2> { ... }
Dependently typed parameters are particularly useful in combination with
:ref:`parameterized module signatures <parameterized-module-signatures>`.
.. _module-bodies:
Module bodies

View File

@@ -21,7 +21,7 @@ In summary, the kinds of expressions are:
- **Module expressions**
- These refer to modules.
- They can be simple :ref:`names <names>`, :ref:`qualified references <qualified-references>`
(in import statements), or :ref:`selections <selections>`.
(in import statements), :ref:`selections <selections>`, or :ref:`instantiations <parameterized-modules>`.
- **Type expressions**
- These refer to types.
- They can be simple :ref:`names <names>` or :ref:`selections <selections>`.

View File

@@ -57,44 +57,27 @@ construction of the library path.
First, determine the *query directory* of the ``.ql`` file being
compiled. Starting with the directory containing the ``.ql`` file, and
walking up the directory structure, each directory is checked for a
file called ``queries.xml`` or ``qlpack.yml``. The first directory
file called ``qlpack.yml`` or ``codeql-pack.yml``. The first directory
where such a file is found is the query directory. If there is no such
directory, the directory of the ``.ql`` file itself is the query
directory.
A ``queries.xml`` file that defines a query directory must always
contain a single top-level tag named
``queries``, which has a ``language`` attribute set to the identifier
of the active database schema (for example, ``<queries
language="java"/>``).
A ``qlpack.yml`` file defines a :ref:`CodeQL pack <about-codeql-packs>`.
The content of a ``qlpack.yml`` file is described in the CodeQL CLI documentation.
``codeql-pack.yml`` is an alias for ``qlpack.yml``.
If both a ``queries.xml`` and a ``qlpack.yml`` exist in the same
directory, the latter takes precedence (and the former is assumed to
exist for compatibility with older tooling).
The CodeQL CLI and newer tools based on it (such as,
The CodeQL CLI and tools based on it (such as,
GitHub code scanning and the CodeQL extension for Visual Studio Code)
construct a library path using CodeQL packs. For each CodeQL pack
added to the library path, the CodeQL packs named in its
``libraryPathDependencies`` will be subsequently added to the library
``dependencies`` will be subsequently added to the library
path, and the process continues until all packs have been
resolved. The actual library path consists of the root directories of
the selected CodeQL packs. This process depends on a mechanism for finding
CodeQL packs by pack name, as described in the :ref:`CodeQL CLI documentation <codeql-cli>`.
CodeQL packs by pack name and version, as described in the :ref:`CodeQL CLI documentation <codeql-cli>`.
When the query directory contains a ``queries.xml`` file but no
``qlpack.yml``, the CodeQL pack resolution behaves as if it defines a QL
pack with no name and a single library path dependency named
``legacy-libraries-LANGUAGE`` where ``LANGUAGE`` is taken from
``queries.xml``. The ``github/codeql`` repository provides packs with
names following this pattern, which themselves depend on the actual
CodeQL libraries for each language.
When the query directory contains neither a ``queries.xml`` nor
``qlpack.yml`` file, it is considered to be a CodeQL pack with no name and
When the query directory contains neither a ``qlpack.yml`` nor
``codeql-pack.yml`` file, it is considered to be a CodeQL pack with no name and
no library dependencies. This causes the library path to consist of
*only* the query directory itself. This is not generally useful,
but it suffices for running toy examples of QL code that don't
@@ -160,7 +143,7 @@ Module definitions
A QL module definition has the following syntax:
::
::
module ::= annotation* "module" modulename "{" moduleBody "}"
@@ -196,7 +179,7 @@ An import directive refers to a module identifier:
import ::= annotations "import" importModuleId ("as" modulename)?
qualId ::= simpleId | qualId "." simpleId
qualId ::= simpleId | qualId "." simpleId
importModuleId ::= qualId
| importModuleId "::" simpleId
@@ -268,7 +251,7 @@ With the exception of class domain types and character types (which cannot be re
type ::= (moduleId "::")? classname | dbasetype | "boolean" | "date" | "float" | "int" | "string"
moduleId ::= simpleId | moduleId "::" simpleId
moduleId ::= simpleId | moduleId "::" simpleId
A type reference is resolved to a type as follows:
@@ -598,7 +581,7 @@ An integer literal is a possibly negated sequence of decimal digits (``0`` throu
0
42
123
-2147483648
-2147483648
Float literals (float)
~~~~~~~~~~~~~~~~~~~~~~
@@ -760,7 +743,7 @@ it is parsed as part of the first declaration.
Inheriting QLDoc
~~~~~~~~~~~~~~~~
If no QLDoc is provided then it may be inherited.
If no QLDoc is provided then it may be inherited.
In the case of an alias then it may be inherited from the right-hand side of the alias.
@@ -818,7 +801,7 @@ The body of a predicate is of one of three forms:
::
optbody ::= ";"
| "{" formula "}"
| "{" formula "}"
| "=" literalId "(" (predicateRef "/" int ("," predicateRef "/" int)*)? ")" "(" (exprs)? ")"
In the first form, with just a semicolon, the predicate is said to not have a body. In the second form, the body of the predicate is the given formula (see "`Formulas <#formulas>`__"). In the third form, the body is a higher-order relation.
@@ -873,7 +856,7 @@ Each member of a class is either a *character*, a predicate, or a field:
::
member ::= character | predicate | field
character ::= qldoc? annotations classname "(" ")" "{" formula "}"
character ::= qldoc? annotations classname "(" ")" "{" formula "}"
field ::= qldoc? annotations var_decl ";"
Characters
@@ -924,7 +907,7 @@ A valid class may not inherit from two different classes that include a field wi
A valid field must override another field if it is annotated ``override``.
When field ``f`` overrides field ``g`` the type of ``f`` must be a subtype of the type of ``g``. ``f`` may not be a final field.
When field ``f`` overrides field ``g`` the type of ``f`` must be a subtype of the type of ``g``. ``f`` may not be a final field.
Select clauses
~~~~~~~~~~~~~~
@@ -993,8 +976,8 @@ There are several kinds of expressions:
| binop
| cast
| primary
primary ::= eparen
primary ::= eparen
| literal
| variable
| super_expr
@@ -1103,7 +1086,7 @@ A super expression has the following syntax:
::
super_expr ::= "super" | type "." "super"
super_expr ::= "super" | type "." "super"
For a super expression to be valid, the ``this`` keyword must have a type and value in the typing environment. The type of the expression is the same as the type of ``this`` in the typing environment.
@@ -1163,11 +1146,11 @@ A valid call with results *resolves* to a set of predicates. The ways a call can
- If the call has no receiver and the predicate name is a selection identifier, then the qualifier is resolved as a module (see "`Module resolution <#module-resolution>`__"). The identifier is then resolved in the exported predicate environment of the qualifier module.
- If the call has a super expression as the receiver, then it resolves to a member predicate in a class that the enclosing class inherits from:
- If the super expression is unqualified and there is a single class that the current class inherits from, then the super-class is that class.
- If the super expression is unqualified and there is a single class that the current class inherits from, then the super-class is that class.
- If the super expression is unqualified and there are multiple classes that the current class inherits from, then the super-class is the domain type.
- Otherwise, the super-class is the class named by the qualifier of the super expression.
- Otherwise, the super-class is the class named by the qualifier of the super expression.
The predicate is resolved by looking up its name and arity in the exported predicate environment of the super-class.
The predicate is resolved by looking up its name and arity in the exported predicate environment of the super-class.
- If the type of the receiver is the same as the enclosing class, the predicate is resolved by looking up its name and arity in the visible predicate environment of the class.
@@ -1226,15 +1209,15 @@ The rank expression must be present if the aggregate id is ``rank``; otherwise i
Apart from the presence or absence of the rank variable, all other reduced forms of an aggregation are equivalent to a full form using the following steps:
- If the formula is omitted, then it is taken to be ``any()``.
- If there are no aggregation expressions, then either:
- If there are no aggregation expressions, then either:
- The aggregation id is ``count`` or ``strictcount`` and the expression is taken to be ``1``.
- The aggregation id is ``count`` or ``strictcount`` and the expression is taken to be ``1``.
- There must be precisely one variable declaration, and the aggregation expression is taken to be a reference to that variable.
- If the aggregation id is ``concat`` or ``strictconcat`` and it has a single expression then the second expression is taken to be ``""``.
- If the ``monotonicAggregates`` language pragma is not enabled, or the original formula and variable declarations are both omitted, then the aggregate is transformed as follows:
- If the ``monotonicAggregates`` language pragma is not enabled, or the original formula and variable declarations are both omitted, then the aggregate is transformed as follows:
- For each aggregation expression ``expr_i``, a fresh variable ``v_i`` is declared with the same type as the expression in addition to the original variable declarations.
- For each aggregation expression ``expr_i``, a fresh variable ``v_i`` is declared with the same type as the expression in addition to the original variable declarations.
- The new range is the conjunction of the original range and a term ``v_i = expr_i`` for each aggregation expression ``expr_i``.
- Each original aggregation expression ``expr_i`` is replaced by a new aggregation expression ``v_i``.
@@ -1320,11 +1303,11 @@ Expression pragmas can be used to guide optimization.
The values of an expression pragma are the values of the contained expression.
The type `only_bind_out` hints that uses of the result of the expression pragma should not be used to guide the evaluation of the result of the contained expression.
When checking to see that all values are bound the compiler does not assume that if the result of the expression pragma is bound then the result of the contained
When checking to see that all values are bound the compiler does not assume that if the result of the expression pragma is bound then the result of the contained
expression is bound.
The type `only_bind_into` hints that uses of the contained expression should not be used to guide the evaluation of the result of the expression pragma.
When checking to see that all values are bound the compiler does not assume that if the result of the contained expression is bound then the result of the
When checking to see that all values are bound the compiler does not assume that if the result of the contained expression is bound then the result of the
expression pragma is bound.
Ranges
@@ -1498,7 +1481,7 @@ A comparison formula is two expressions separated by a comparison operator:
::
comparison ::= expr compop expr
compop ::= "=" | "!=" | "<" | ">" | "<=" | ">="
compop ::= "=" | "!=" | "<" | ">" | "<=" | ">="
A comparison formula matches if there is one value of the left expression that is in the given ordering with one of the values of the right expression. The ordering used is specified in "`Ordering <#ordering>`__." If one of the values is an integer and the other is a float value, then the integer is converted to a float value before the comparison.
@@ -1582,7 +1565,7 @@ Aliases define new names for existing QL entities.
alias ::= qldoc? annotations "predicate" literalId "=" predicateRef "/" int ";"
| qldoc? annotations "class" classname "=" type ";"
| qldoc? annotations "module" modulename "=" moduleId ";"
An alias introduces a binding from the new name to the entity referred to by the right-hand side in the current module's declared predicate, type, or module environment respectively.
@@ -1973,13 +1956,13 @@ Each layer of the stratification is *populated* in order. To populate a layer, e
- To populate a predicate that has a formula as a body, find each named tuple ``t`` that has the following properties:
- The tuple matches the body formula.
- The variables should be the predicate's arguments.
- The variables should be the predicate's arguments.
- If the predicate has a result, then the tuples should additionally have a value for ``result``.
- If the predicate is a member predicate or characteristic predicate of a class ``C`` then the tuples should additionally have a value for ``this`` and each visible field on the class.
- The values corresponding to the arguments should all be a member of the declared types of the arguments.
- The values corresponding to ``result`` should all be a member of the result type.
- The values corresponding to the fields should all be a member of the declared types of the fields.
- If the predicate is a member predicate of a class ``C`` and not a characteristic predicate, then the tuples should additionally extend some tuple in ``C.class``.
- If the predicate is a member predicate of a class ``C`` and not a characteristic predicate, then the tuples should additionally extend some tuple in ``C.class``.
- If the predicate is a characteristic predicate of a class ``C``, then there should be a tuple ``t'`` in ``C.extends`` such that for each visible field in ``C``, any field that is equal to or overrides a field in ``t'`` should have the same value in ``t``. ``this`` should also map to the same value in ``t`` and ``t'``.
For each such tuple remove any components that correspond to fields and add it to the predicate in the store.
@@ -1998,7 +1981,7 @@ Each layer of the stratification is *populated* in order. To populate a layer, e
- To populate the type ``C.C`` for a class ``C``, if ``C`` has a characteristic predicate, then add all tuples from that predicate to the store. Otherwise add all tuples ``t`` such that:
- The variables of ``t`` should be ``this`` and the visible fields of ``C``.
- The variables of ``t`` should be ``this`` and the visible fields of ``C``.
- The values corresponding to the fields should all be a member of the declared types of the fields.
- If the predicate is a characteristic predicate of a class ``C``, then there should be a tuple ``t'`` in ``C.extends`` such that for each visible field in ``C``, any field that is equal to or overrides a field in ``t'`` should have the same value in ``t``. ``this`` should also map to the same value in ``t`` and ``t'``.
@@ -2033,7 +2016,7 @@ The complete grammar for QL is as follows:
import ::= annotations "import" importModuleId ("as" modulename)?
qualId ::= simpleId | qualId "." simpleId
qualId ::= simpleId | qualId "." simpleId
importModuleId ::= qualId
| importModuleId "::" simpleId
@@ -2072,18 +2055,18 @@ The complete grammar for QL is as follows:
head ::= ("predicate" | type) predicateName "(" var_decls ")"
optbody ::= ";"
| "{" formula "}"
| "{" formula "}"
| "=" literalId "(" (predicateRef "/" int ("," predicateRef "/" int)*)? ")" "(" (exprs)? ")"
class ::= qldoc? annotations "class" classname "extends" type ("," type)* "{" member* "}"
member ::= character | predicate | field
character ::= qldoc? annotations classname "(" ")" "{" formula "}"
character ::= qldoc? annotations classname "(" ")" "{" formula "}"
field ::= qldoc? annotations var_decl ";"
moduleId ::= simpleId | moduleId "::" simpleId
moduleId ::= simpleId | moduleId "::" simpleId
type ::= (moduleId "::")? classname | dbasetype | "boolean" | "date" | "float" | "int" | "string"
@@ -2092,7 +2075,7 @@ The complete grammar for QL is as follows:
alias ::= qldoc? annotations "predicate" literalId "=" predicateRef "/" int ";"
| qldoc? annotations "class" classname "=" type ";"
| qldoc? annotations "module" modulename "=" moduleId ";"
var_decls ::= (var_decl ("," var_decl)*)?
var_decl ::= type lowerId
@@ -2108,7 +2091,7 @@ The complete grammar for QL is as follows:
| instanceof
| inrange
| call
fparen ::= "(" formula ")"
disjunction ::= formula "or" formula
@@ -2128,7 +2111,7 @@ The complete grammar for QL is as follows:
comparison ::= expr compop expr
compop ::= "=" | "!=" | "<" | ">" | "<=" | ">="
compop ::= "=" | "!=" | "<" | ">" | "<=" | ">="
instanceof ::= expr "instanceof" type
@@ -2146,7 +2129,7 @@ The complete grammar for QL is as follows:
| primary
primary ::= eparen
primary ::= eparen
| literal
| variable
| super_expr
@@ -2175,7 +2158,7 @@ The complete grammar for QL is as follows:
variable ::= varname | "this" | "result"
super_expr ::= "super" | type "." "super"
super_expr ::= "super" | type "." "super"
cast ::= "(" type ")" expr
@@ -2184,7 +2167,7 @@ The complete grammar for QL is as follows:
aggregation ::= aggid ("[" expr "]")? "(" var_decls ("|" (formula)? ("|" as_exprs ("order" "by" aggorderbys)?)?)? ")"
| aggid ("[" expr "]")? "(" as_exprs ("order" "by" aggorderbys)? ")"
| "unique" "(" var_decls "|" (formula)? ("|" as_exprs)? ")"
expression_pragma ::= "pragma" "[" expression_pragma_type "]" "(" expr ")"
expression_pragma_type ::= "only_bind_out" | "only_bind_into"
@@ -2199,9 +2182,9 @@ The complete grammar for QL is as follows:
callwithresults ::= predicateRef (closure)? "(" (exprs)? ")"
| primary "." predicateName (closure)? "(" (exprs)? ")"
range ::= "[" expr ".." expr "]"
setliteral ::= "[" expr ("," expr)* ","? "]"
simpleId ::= lowerId | upperId

View File

@@ -0,0 +1,120 @@
:tocdepth: 1
.. index:: signature
.. _signatures:
Signatures
##########
Parameterized modules use signatures as a type system for their parameters.
There are three categories of signatures: **predicate signatures**, **type signatures**, and **module signatures**.
Predicate signatures
====================
Predicate signatures declare module parameters that will be substituted with predicates when the module is instantiated.
The substitution of predicate signatures relies on structural typing. That is, predicates do not have to be explicitly
defined as implementing a predicate signature - they just have to match the return and argument types.
Predicate signatures are defined much like predicates themselves, but they do not have a body.
In detail, a predicate signature definition consists of:
#. The keyword ``signature``.
#. The keyword ``predicate`` (allows subsitution with a :ref:`predicate without result <predicates-without-result>`),
or the type of the result (allows subsitution with a :ref:`predicate with result <predicates-with-result>`).
#. The name of the predicate signature. This is an `identifier <https://codeql.github.com/docs/ql-language-reference/ql-language-specification/#identifiers>`_
starting with a lowercase letter.
#. The arguments to the predicate signature, if any, separated by commas.
For each argument, specify the argument type and an identifier for the argument variable.
#. A semicolon ``;``.
For example:
.. code-block:: ql
signature int operator(int lhs, int rhs);
Type signatures
===============
Type signatures declare module parameters that will be substituted with types when the module is instantiated.
Type signatures are used to specify supertypes and are the simplest category of signatures.
The substitution of type signatures relies on structural typing. That is, types do not have to be explicitly defined as
implementing a type signature - they just need to have the specified (transitive) supertypes.
In detail, a type signature definition consists of:
#. The keyword ``signature``.
#. The keyword ``class``.
#. The name of the type signature. This is an `identifier <https://codeql.github.com/docs/ql-language-reference/ql-language-specification/#identifiers>`_
starting with a uppercase letter.
#. Optionally, the keyword ``extends`` followed by a list of types, separated by commas.
#. A semicolon ``;``.
For example:
.. code-block:: ql
signature class ExtendsInt extends int;
Module signatures
=================
Module signatures declare module parameters that will be substituted with modules when the module is instantiated.
Module signatures specify a collection of types and predicates that a module needs to contain under given names and
matching given signatures.
Unlike type signatures and predicate signatures, the substitution of type signatures relies on nominal typing.
That is, the definition of a module must declare the module signatures it implements.
In detail, a type signature definition consists of:
#. The keyword ``signature``.
#. The keyword ``module``.
#. The name of the module signature. This is an `identifier <https://codeql.github.com/docs/ql-language-reference/ql-language-specification/#identifiers>`_
starting with a uppercase letter.
#. Optionally, a list of parameters for :ref:`parameterized module signatures <parameterized-module-signatures>`.
#. The module signature body, consisting of type signatures and predicate signatures enclosed in braces.
The ``signature`` keyword is omitted for these contained signatures.
For example:
.. code-block:: ql
signature module MSig {
class T;
predicate restriction(T t);
}
module Module implements MSig {
newtype T = A() or B();
predicate restriction(T t) { t = A() }
}
.. _parameterized-module-signatures:
Parameterized module signatures
-------------------------------
Module signatures can themselves be parameterized in exactly the same way as parameterized modules.
This is particularly useful in combination with the dependent typing of module parameters.
For example:
.. code-block:: ql
signature class NodeSig;
signature module EdgeSig<NodeSig Node> {
predicate apply(Node src, Node dst);
}
module Reachability<NodeSig Node, EdgeSig<Node> Edge> {
Node reachableFrom(Node src) {
Edge::apply+(src, result)
}
}

View File

@@ -7,8 +7,8 @@ For example, to check for CodeQL databases using the `GitHub CLI <https://cli.gi
This command returns information about any CodeQL databases that are available for a repository, including the language the database represents, and when the database was last updated. If no CodeQL databases are available, the response is empty.
When you have confirmed that a CodeQL database exists for the language you are interested in, you can download it using the following command::
When you have confirmed that a CodeQL database exists for the language you are interested in, you can download it using the following command::
gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/<language> -H 'Accept: application/zip' > path/to/local/database.zip
For more information, see the documentation for the `Get CodeQL database <https://docs.github.com/en/rest/reference/code-scanning#get-codeql-database>`__ endpoint in the GitHub REST API documentation.
For more information, see the documentation for the `Get CodeQL database <https://docs.github.com/en/rest/code-scanning#get-a-codeql-database-for-a-repository>`__ endpoint in the GitHub REST API documentation.

View File

@@ -23,7 +23,7 @@
JavaScript,ECMAScript 2022 or lower,Not applicable,"``.js``, ``.jsx``, ``.mjs``, ``.es``, ``.es6``, ``.htm``, ``.html``, ``.xhtm``, ``.xhtml``, ``.vue``, ``.hbs``, ``.ejs``, ``.njk``, ``.json``, ``.yaml``, ``.yml``, ``.raml``, ``.xml`` [6]_"
Python,"2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10",Not applicable,``.py``
Ruby [7]_,"up to 3.0.2",Not applicable,"``.rb``, ``.erb``, ``.gemspec``, ``Gemfile``"
TypeScript [8]_,"2.6-4.7",Standard TypeScript compiler,"``.ts``, ``.tsx``, ``.mts``, ``.cts``"
TypeScript [8]_,"2.6-4.8",Standard TypeScript compiler,"``.ts``, ``.tsx``, ``.mts``, ``.cts``"
.. container:: footnote-group

View File

@@ -31,13 +31,9 @@ Applying the `::Range` pattern yields the following:
* Extend this class to refine existing API models. If you want to model new APIs,
* extend `MySpecialExpr::Range` instead.
*/
class MySpecialExpr extends Expr {
MySpecialExpr::Range range;
MySpecialExpr() { this = range }
class MySpecialExpr extends Expr instanceof MySpecialExpr::Range {
/** <QLDoc...> */
int memberPredicate() { result = range.memberPredicate() }
int memberPredicate() { result = super.memberPredicate() }
}
/** Provides a class for modeling new <...> APIs. */
@@ -56,22 +52,18 @@ module MySpecialExpr {
```
Now, a concrete subclass can derive from `MySpecialExpr::Range` if it wants to extend the set of values in `MySpecialExpr`, and it will be required to implement the abstract `memberPredicate()`. Conversely, if it wants to refine `MySpecialExpr` and override `memberPredicate` for all extensions, it can do so by deriving from `MySpecialExpr` directly.
The key element of the pattern is to provide a field of type `MySpecialExpr::Range`, equating it to `this` in the characteristic predicate of `MySpecialExpr`. In member predicates, we can use either `this` or `range`, depending on which type has the API we need.
</details>
Note that in some libraries, the `range` field is in fact called `self`. While we do recommend using `range` for consistency, the name of the field does not matter (and using `range` avoids confusion in contexts like Python analysis that has strong usage of `self`).
### Rationale
Let's use an example from the Go libraries: https://github.com/github/codeql-go/blob/2ba9bbfd8ba1818b5ee9f6009c86a605189c9ef3/ql/src/semmle/go/Concepts.qll#L119-L157
Let's use an example from the Python libraries: https://github.com/github/codeql/blob/46751e515c40c6b4c9b61758cc840eec1894a624/python/ql/lib/semmle/python/Concepts.qll#L601-L683
`EscapeFunction`, as the name suggests, models various APIs that escape meta-characters. It has a member-predicate `kind()` that tells you what sort of escaping the modelled function does. For example, if the result of that predicate is `"js"`, then this means that the escaping function is meant to make things safe to embed inside JavaScript.
`EscapeFunction::Range` is subclassed to model various APIs, and `kind()` is implemented accordingly.
But we can also subclass `EscapeFunction` to, as in the above example, talk about all JS-escaping functions.
`Escaping`, as the name suggests, models various APIs that escape meta-characters. It has a member-predicate `getKind()` that tells you what sort of escaping the modeled function does. For example, if the result of that predicate is `"html"`, then this means that the escaping function is meant to make things safe to embed inside HTML.
`Escaping::Range` is subclassed to model various APIs, and `kind()` is implemented accordingly (this typically happens in library models).
But we can also subclass `Escaping`, as in the above example, where `HtmlEscaping` represents all HTML-escaping functions.
You can, of course, do the same without the `::Range` pattern, but it's a little cumbersome:
If you only had an `abstract class EscapeFunction { ... }`, then `JsEscapeFunction` would need to be implemented in a slightly tricky way to prevent it from extending `EscapeFunction` (instead of refining it). You would have to give it a charpred `this instanceof EscapeFunction`, which looks useless but isn't. And additionally, you'd have to provide trivial `none()` overrides of all the abstract predicates defined in `EscapeFunction`. This is all pretty awkward, and we can avoid it by distinguishing between `EscapeFunction` and `EscapeFunction::Range`.
If you only had an `abstract class Escaping { ... }`, then `HtmlEscaping` would need to be implemented in a slightly tricky way to prevent it from extending `Escaping` (instead of refining it). You would have to give it a charpred `this instanceof Escaping`, which looks useless but isn't. And additionally, you'd have to provide trivial `none()` overrides of all the abstract predicates defined in `Escaping`. This is all pretty awkward, and we can avoid it by distinguishing between `Escaping` and `Escaping::Range`.
## Importing all subclasses of a class

View File

@@ -179,7 +179,15 @@ The select clause of each alert query defines the alert message that is displaye
* The message should factually describe the problem that is being highlightedit should not contain recommendations about how to fix the problem or value judgements.
* Program element references should be in 'single quotes' to distinguish them from ordinary words. Quotes are not needed around substitutions (`$@`).
* Avoid constant alert message strings and include some context, if possible. For example, `The class 'Foo' is duplicated as 'Bar'.` is preferable to `This class is duplicated here.`
* If a reference to the current location can't be avoided use "this location" instead of "here". For example, `Bad thing at this location.` is preferable to `Bad thing here.`. This avoids the "click here" anti-pattern.
* For path queries, if possible, try to follow the template: `This path depends on a [user-provided value].`, or alternatively (if the first option doesn't work) `[User-provided value] flows to this location and is used in a path.`.
* Taint tracking queries generally have a sink that "depends on" the source, and dataflow queries generally have a source that "flows to" the sink.
### Links in alert messages
* Where you reference another program element, link to it if possible using a substitution (`$@`). Links should be used inline in the sentence, rather than as parenthesised lists or appositions.
* Avoid using link texts that don't describe what they link to. For example, rewrite `This sensitive data is written to a logfile unescaped [here]` to `This sensitive data is [written to a logfile unescaped]`.
* Make link text as concise and precise as possible. For example, avoid starting a link text with an indefinite article (a, an). `Path construction depends on a [user-provided value]` is preferable to `Path construction depends on [a user-provided value]`. (Where the square brackets indicate a link.) See [the W3C guide on link texts](https://www.w3.org/WAI/WCAG22/Understanding/link-purpose-in-context.html) for further information.
* When a message contains multiple links, construct a sentence that has the most variable link (that is, the link with most targets) last. For further information, see [Defining the results of a query](https://codeql.github.com/docs/writing-codeql-queries/defining-the-results-of-a-query/).
For examples of select clauses and alert messages, see the query source files at the following pages: