From 53d8bf27ff1d491b3c0ff6b66c886a5f94310ce5 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Wed, 24 Aug 2022 14:11:24 -0700 Subject: [PATCH 01/11] Add docs for codeql workspaces See https://github.com/github/codeql-core/issues/2687 --- docs/codeql/codeql-cli/about-codeql-packs.rst | 4 +- .../codeql-cli/about-codeql-workspaces.rst | 77 +++++++++++++++++++ .../codeql-cli/codeql-cli-reference.rst | 2 + ...creating-and-working-with-codeql-packs.rst | 2 + 4 files changed, 82 insertions(+), 3 deletions(-) create mode 100644 docs/codeql/codeql-cli/about-codeql-workspaces.rst diff --git a/docs/codeql/codeql-cli/about-codeql-packs.rst b/docs/codeql/codeql-cli/about-codeql-packs.rst index 3824d79e0fe..94db2a96018 100644 --- a/docs/codeql/codeql-cli/about-codeql-packs.rst +++ b/docs/codeql/codeql-cli/about-codeql-packs.rst @@ -199,10 +199,8 @@ The ``codeql-pack.lock.yml`` file will contain something like the following: my-user/transitive-dependency: version: 1.2.4 -.. - TODO: Add a link to the CodeQL CLI documentation for query resolution, specifically in regards to resolving from source -The ``codeql/cpp-all`` dependency is locked to version 0.1.4. The ``my-user/my-lib`` dependency is locked to version 0.2.4. The ``my-user/transitive-dependency``, which is a transitive dependency and is not specified in the ``qlpack.yml`` file, is locked to version 1.2.4. The ``other-dependency/from-source`` is absent from the lock file since it is resolved from source. This dependency must be available in the same CodeQL workspace as the pack. +The ``codeql/cpp-all`` dependency is locked to version 0.1.4. The ``my-user/my-lib`` dependency is locked to version 0.2.4. The ``my-user/transitive-dependency``, which is a transitive dependency and is not specified in the ``qlpack.yml`` file, is locked to version 1.2.4. The ``other-dependency/from-source`` is absent from the lock file since it is resolved from source. This dependency must be available in the same CodeQL workspace as the pack. For more information about CodeQL workspaces and resolving dependencies from source see ":doc:`About CodeQL Workspaces `." .. _custom-codeql-packs: diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst new file mode 100644 index 00000000000..261a770235b --- /dev/null +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -0,0 +1,77 @@ +.. _about-codeql-workspaces: + +About CodeQL Workspaces +======================= + +.. include:: ../reusables/beta-note-package-management.rst + +CodeQL workspaces are used to group multiple CodeQL packs together. CodeQL packs in the same workspace are automatically available as source dependencies for each other when running any CodeQL command that resolves queries. This makes it easier to develope and maintain multiple, related CodeQL packs. A typical use case for a CodeQL workspace is for developing one or more CodeQL library packs and one or more query packs that depends on it in them in the same location. + +The main benefit of a CodeQl workspace is that it is easier to develop and maintain multiple CodeQL packs. When a CodeQL workspace is used, all CodeQL packs in the workspace are available as source dependencies for each other when running any CodeQL command that resolves queries. This makes it easier to develope and maintain multiple, related CodeQL packs. + +In most cases, the CodeQL workspace and all CodeQL packs contained in it should be stored in the same git repository so the development environment is more easily sharable. + +The ``codeql-workspae.yml`` file +-------------------------------- + +A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. Each entry in the ``provide`` or ``ignore`` section must map to a path to a ``qlpack.yml`` file. All glob patterns are relative to the directory containing the workspace file. See `@actions/glob ` for a list of patterns accepted in this file. + +For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all CodeQl packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: + +.. code-block:: yaml + + provide: + - "*/codeql-packs/**/qlpack.yml" + ignore: + - "*/codeql-packs/**/experimental/**/qlpack.yml" + +To verify that you have the correct ``codeql-workspace.yml`` file, run ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. + +CodeQL workspaces and query resolution +-------------------------------------- + +All CodeQL packs in a workspace are available as source dependencies for each other when running any CodeQL command that resolves queries or packs. For example, when ``codeql pack install`` is run in a pack directory in a workspace, any dependency found in the workspace will not be downloaded to the package cache, nor will it be added to the resulting ``codeql-pack.lock.yml`` file. See `:ref:Adding and Installing Dependencies ` for more information. + +Similarly, publishing a CodeQL query pack to the GitHub container registry using ``codeql pack publish`` will always use dependencies found in the workspace instead of using dependencies found in the local package cache. + +This ensures that any local change to a query library in a dependency in the same workspace will be automatically reflected in the published query pack. + +Example +~~~~~~~ + +Consider the following ``codeql-workspace.yml`` file: + +.. code-block:: yaml + + provide: + - "**/qlpack.yml" + +And the following CodeQL library pack ``qlpack.yml`` file in the workspace: + +.. code-block:: yaml + + name: my-company/my-library + library: true + version: 1.0.0 + +And the following CodeQL query pack ``qlpack.yml`` file in the workspace: + +.. code-block:: yaml + + name: my-company/my-queries + version: 1.0.0 + dependencies: + my-company/my-library: "*" + codeql/cpp-all: ~0.2.0 + +Notice that ``"*"`` is specified as the version constraint for the library pack. Because the library pack is a source dependency, the version constraint is not needed since the library pack's content is always resolved from inside of the workspace. Any version constraint will be ignored in this case, but it is recommended to use ``"*"`` for source dependencies to avoid confusion. + +When ``codeql pack install`` is executed from the query pack directory, an appropriate version of ``codeql/cpp-all`` will be downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file will be created that contains the resolved version of ``codeql/cpp-all``, but no entry for ``my-company/my-library`` since it is resolved from source. The ``codeql-pack.lock.yml`` file will look something like this: + +.. code-block:: yaml + + dependencies: + codeql/cpp-all: + version: 0.2.2 + +When ``codeql pack publish`` is executed from the query pack directory, the ``codeql/cpp-all`` dependency from the package cache and the ``my-company/my-library`` from the workspace will be bundled with ``my-company/my-queries`` and published to the GitHub container registry. diff --git a/docs/codeql/codeql-cli/codeql-cli-reference.rst b/docs/codeql/codeql-cli/codeql-cli-reference.rst index fadf5ada79f..43ac1adbee6 100644 --- a/docs/codeql/codeql-cli/codeql-cli-reference.rst +++ b/docs/codeql/codeql-cli/codeql-cli-reference.rst @@ -14,6 +14,7 @@ Learn more about the files you can use when running CodeQL processes and the res sarif-output exit-codes extractor-options + about-codeql-workspaces - :doc:`About CodeQL packs `: CodeQL packs are created with the CodeQL CLI and are used to create, depend on, publish, and run CodeQL queries, libraries, and query suites. - :doc:`Query reference files `: A query reference file is text file that defines the location of one query to test. @@ -21,3 +22,4 @@ Learn more about the files you can use when running CodeQL processes and the res - :doc:`Exit codes `: The CodeQL CLI reports the status of each command it runs as an exit code. This exit code provides information for subsequent commands or for other tools that rely on the CodeQL CLI. - :doc:`Extractor options `: You can customize the behavior of extractors by setting options through the CodeQL CLI. +- :doc:`About CodeQL workspaces `: CodeQL workspaces are used to group multiple CodeQL packs together. diff --git a/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst b/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst index 3b3f1c7e2df..7d2001dded5 100644 --- a/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst +++ b/docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst @@ -45,6 +45,8 @@ If you already have a ``qlpack.yml`` file, you can edit it manually to convert i For more information about the properties, see ":ref:`About CodeQL packs `." +.. _adding-and-installing-dependencies: + Adding and installing dependencies to a CodeQL pack --------------------------------------------------- You can add dependencies on CodeQL packs using the command ``codeql pack add``. You must specify the scope, name, and (optionally) a compatible version range. From 68c2c16928fb8be60fe596f7e7267990ff06a2ad Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 26 Aug 2022 14:35:59 -0700 Subject: [PATCH 02/11] Address changes from PR --- docs/codeql/codeql-cli/about-codeql-workspaces.rst | 13 ++++++++++--- .../analyzing-databases-with-the-codeql-cli.rst | 2 +- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index 261a770235b..5bce962281d 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -5,9 +5,9 @@ About CodeQL Workspaces .. include:: ../reusables/beta-note-package-management.rst -CodeQL workspaces are used to group multiple CodeQL packs together. CodeQL packs in the same workspace are automatically available as source dependencies for each other when running any CodeQL command that resolves queries. This makes it easier to develope and maintain multiple, related CodeQL packs. A typical use case for a CodeQL workspace is for developing one or more CodeQL library packs and one or more query packs that depends on it in them in the same location. +CodeQL workspaces are used to group multiple CodeQL packs together. A typical use case for a CodeQL workspace is for developing a set of CodeQL library and query packs that are mutually dependent. For more information on CodeQL packs, see ":doc:`About CodeQL packs `." -The main benefit of a CodeQl workspace is that it is easier to develop and maintain multiple CodeQL packs. When a CodeQL workspace is used, all CodeQL packs in the workspace are available as source dependencies for each other when running any CodeQL command that resolves queries. This makes it easier to develope and maintain multiple, related CodeQL packs. +The main benefit of a CodeQl workspace is that it is easier to develop and maintain multiple CodeQL packs. When a CodeQL workspace is used, all CodeQL packs in the workspace are available as *source dependencies* for each other when running any CodeQL command that resolves queries. This makes it easier to develope and maintain multiple, related CodeQL packs. In most cases, the CodeQL workspace and all CodeQL packs contained in it should be stored in the same git repository so the development environment is more easily sharable. @@ -16,7 +16,7 @@ The ``codeql-workspae.yml`` file A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. Each entry in the ``provide`` or ``ignore`` section must map to a path to a ``qlpack.yml`` file. All glob patterns are relative to the directory containing the workspace file. See `@actions/glob ` for a list of patterns accepted in this file. -For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all CodeQl packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: +For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: .. code-block:: yaml @@ -27,6 +27,7 @@ For example, the following ``codeql-workspace.yml`` file defines a workspace tha To verify that you have the correct ``codeql-workspace.yml`` file, run ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. + CodeQL workspaces and query resolution -------------------------------------- @@ -36,6 +37,12 @@ Similarly, publishing a CodeQL query pack to the GitHub container registry using This ensures that any local change to a query library in a dependency in the same workspace will be automatically reflected in the published query pack. +.. pull-quote:: + + Note + + Source dependencies are CodeQL packs that are resolved from the filesystem. They might be in the same CodeQL workspace, or specified a path option in the ``--additional-packs`` argument. Source dependencies override any dependencies found in the local package cache and version constraints are ignored. This ensures that during local development version mismatches can be ignored. + Example ~~~~~~~ diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 7a47b98032d..57f223f1991 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -80,7 +80,7 @@ You can also specify: - .. include:: ../reusables/threads-query-execution.rst -. pull-quote:: +.. pull-quote:: Upgrading databases From d98d1b7455d7ddc0e005eecc5b560befc280417d Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Wed, 31 Aug 2022 10:07:28 -0700 Subject: [PATCH 03/11] Apply suggestions from code review Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com> --- docs/codeql/codeql-cli/about-codeql-packs.rst | 2 +- docs/codeql/codeql-cli/about-codeql-workspaces.rst | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-packs.rst b/docs/codeql/codeql-cli/about-codeql-packs.rst index 94db2a96018..9ce42c56cdc 100644 --- a/docs/codeql/codeql-cli/about-codeql-packs.rst +++ b/docs/codeql/codeql-cli/about-codeql-packs.rst @@ -200,7 +200,7 @@ The ``codeql-pack.lock.yml`` file will contain something like the following: version: 1.2.4 -The ``codeql/cpp-all`` dependency is locked to version 0.1.4. The ``my-user/my-lib`` dependency is locked to version 0.2.4. The ``my-user/transitive-dependency``, which is a transitive dependency and is not specified in the ``qlpack.yml`` file, is locked to version 1.2.4. The ``other-dependency/from-source`` is absent from the lock file since it is resolved from source. This dependency must be available in the same CodeQL workspace as the pack. For more information about CodeQL workspaces and resolving dependencies from source see ":doc:`About CodeQL Workspaces `." +The ``codeql/cpp-all`` dependency is locked to version 0.1.4. The ``my-user/my-lib`` dependency is locked to version 0.2.4. The ``my-user/transitive-dependency``, which is a transitive dependency and is not specified in the ``qlpack.yml`` file, is locked to version 1.2.4. The ``other-dependency/from-source`` is absent from the lock file since it is resolved from source. This dependency must be available in the same CodeQL workspace as the pack. For more information about CodeQL workspaces and resolving dependencies from source, see ":doc:`About CodeQL Workspaces `." .. _custom-codeql-packs: diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index 5bce962281d..260b3d7e88b 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -41,7 +41,7 @@ This ensures that any local change to a query library in a dependency in the sam Note - Source dependencies are CodeQL packs that are resolved from the filesystem. They might be in the same CodeQL workspace, or specified a path option in the ``--additional-packs`` argument. Source dependencies override any dependencies found in the local package cache and version constraints are ignored. This ensures that during local development version mismatches can be ignored. + Source dependencies are CodeQL packs that are resolved from the filesystem. They might be in the same CodeQL workspace, or specified as a path option in the ``--additional-packs`` argument. Source dependencies override any dependencies found in the local package cache and version constraints are ignored. This ensures that during local development version mismatches can be ignored. Example ~~~~~~~ @@ -71,9 +71,9 @@ And the following CodeQL query pack ``qlpack.yml`` file in the workspace: my-company/my-library: "*" codeql/cpp-all: ~0.2.0 -Notice that ``"*"`` is specified as the version constraint for the library pack. Because the library pack is a source dependency, the version constraint is not needed since the library pack's content is always resolved from inside of the workspace. Any version constraint will be ignored in this case, but it is recommended to use ``"*"`` for source dependencies to avoid confusion. +Notice that, for ``my-company/my-queries``, ``"*"`` is specified as the version constraint for the library pack in the ``dependencies`` block. The library pack is defined as a source dependency in ``codeql-workspace.yml``, so the version constraint is not needed since the library pack's content is always resolved from inside of the workspace. Any version constraint will be ignored in this case, but it is recommended to use ``"*"`` for source dependencies to avoid confusion. -When ``codeql pack install`` is executed from the query pack directory, an appropriate version of ``codeql/cpp-all`` will be downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file will be created that contains the resolved version of ``codeql/cpp-all``, but no entry for ``my-company/my-library`` since it is resolved from source. The ``codeql-pack.lock.yml`` file will look something like this: +When ``codeql pack install`` is executed from the query pack directory, an appropriate version of ``codeql/cpp-all`` will be downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file will be created that contains the resolved version of ``codeql/cpp-all``. The lock file won't contain an entry for ``my-company/my-library`` since it is resolved from source. The ``codeql-pack.lock.yml`` file will look something like this: .. code-block:: yaml From 6240b6e6997cb2ec85459b44ed61f66b41c6dad3 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Wed, 31 Aug 2022 11:05:26 -0700 Subject: [PATCH 04/11] Update docs/codeql/codeql-cli/about-codeql-workspaces.rst Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com> --- docs/codeql/codeql-cli/about-codeql-workspaces.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index 260b3d7e88b..0b0ca53ea17 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -14,7 +14,7 @@ In most cases, the CodeQL workspace and all CodeQL packs contained in it should The ``codeql-workspae.yml`` file -------------------------------- -A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. Each entry in the ``provide`` or ``ignore`` section must map to a path to a ``qlpack.yml`` file. All glob patterns are relative to the directory containing the workspace file. See `@actions/glob ` for a list of patterns accepted in this file. +A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. Each entry in the ``provide`` or ``ignore`` section must map to a path to a ``qlpack.yml`` file. All glob patterns are relative to the directory containing the workspace file. See `@actions/glob `__ for a list of patterns accepted in this file. For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: From 93ade495c2e9bb741b486e4bb1e40384b7be8be8 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 2 Sep 2022 13:19:45 -0700 Subject: [PATCH 05/11] Apply suggestions from code review Co-authored-by: Felicity Chapman --- .../codeql-cli/about-codeql-workspaces.rst | 29 +++++++++++-------- 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index 0b0ca53ea17..cf695a3b6fa 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -5,18 +5,23 @@ About CodeQL Workspaces .. include:: ../reusables/beta-note-package-management.rst -CodeQL workspaces are used to group multiple CodeQL packs together. A typical use case for a CodeQL workspace is for developing a set of CodeQL library and query packs that are mutually dependent. For more information on CodeQL packs, see ":doc:`About CodeQL packs `." +CodeQL workspaces are used to group multiple CodeQL packs together. A typical use case for a CodeQL workspace is to develop a set of CodeQL library and query packs that are mutually dependent. For more information on CodeQL packs, see ":doc:`About CodeQL packs `." -The main benefit of a CodeQl workspace is that it is easier to develop and maintain multiple CodeQL packs. When a CodeQL workspace is used, all CodeQL packs in the workspace are available as *source dependencies* for each other when running any CodeQL command that resolves queries. This makes it easier to develope and maintain multiple, related CodeQL packs. +The main benefit of a CodeQL workspace is that it makes it easier for you to develop and maintain multiple CodeQL packs. When you use a CodeQL workspace, all the CodeQL packs in the workspace are available as *source dependencies* for each other when you run a CodeQL command that resolves queries. This makes it easier to develop, maintain, and publish multiple, related CodeQL packs. -In most cases, the CodeQL workspace and all CodeQL packs contained in it should be stored in the same git repository so the development environment is more easily sharable. +In most cases, you should store the CodeQL workspace and the CodeQL packs contained in it in one git repository. This makes it easier to share your CodeQL development environment. -The ``codeql-workspae.yml`` file +The ``codeql-workspace.yml`` file -------------------------------- -A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. Each entry in the ``provide`` or ``ignore`` section must map to a path to a ``qlpack.yml`` file. All glob patterns are relative to the directory containing the workspace file. See `@actions/glob `__ for a list of patterns accepted in this file. +A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. -For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: +* The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. +* The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. + +Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see" `@actions/glob `__ . + +For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all the CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: .. code-block:: yaml @@ -31,11 +36,11 @@ To verify that you have the correct ``codeql-workspace.yml`` file, run ``codeql CodeQL workspaces and query resolution -------------------------------------- -All CodeQL packs in a workspace are available as source dependencies for each other when running any CodeQL command that resolves queries or packs. For example, when ``codeql pack install`` is run in a pack directory in a workspace, any dependency found in the workspace will not be downloaded to the package cache, nor will it be added to the resulting ``codeql-pack.lock.yml`` file. See `:ref:Adding and Installing Dependencies ` for more information. +All CodeQL packs in a workspace are available as source dependencies for each other when you run any CodeQL command that resolves queries or packs. For example, when you run ``codeql pack install`` in a pack directory in a workspace, any dependency that can be found in the workspace will be instead of downloading that dependency to the package cache and adding it to the ``codeql-pack.lock.yml`` file. For more information, see `:ref:Adding and Installing Dependencies `__. -Similarly, publishing a CodeQL query pack to the GitHub container registry using ``codeql pack publish`` will always use dependencies found in the workspace instead of using dependencies found in the local package cache. +Similarly, when you publish a CodeQL query pack to the GitHub container registry using ``codeql pack publish`` the command will always use the dependencies from the workspace instead of using dependencies found in the local package cache. -This ensures that any local change to a query library in a dependency in the same workspace will be automatically reflected in the published query pack. +This ensures that any local changes you make to a query library in a dependency are automatically reflected in any query packs you publish from that workspace. .. pull-quote:: @@ -71,9 +76,9 @@ And the following CodeQL query pack ``qlpack.yml`` file in the workspace: my-company/my-library: "*" codeql/cpp-all: ~0.2.0 -Notice that, for ``my-company/my-queries``, ``"*"`` is specified as the version constraint for the library pack in the ``dependencies`` block. The library pack is defined as a source dependency in ``codeql-workspace.yml``, so the version constraint is not needed since the library pack's content is always resolved from inside of the workspace. Any version constraint will be ignored in this case, but it is recommended to use ``"*"`` for source dependencies to avoid confusion. +Notice that the ``dependencies`` block for the CodeQL query pack,``my-company/my-queries``, specifies ``"*"`` as the version of the library pack. Since the library pack is already defined as a source dependency in ``codeql-workspace.yml``, the library pack's content is always resolved from inside the workspace. Any version constraint you define will be ignored in this case. We recommend that you use ``"*"`` for source dependencies to make it clear that the version is inherited from the workspace. -When ``codeql pack install`` is executed from the query pack directory, an appropriate version of ``codeql/cpp-all`` will be downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file will be created that contains the resolved version of ``codeql/cpp-all``. The lock file won't contain an entry for ``my-company/my-library`` since it is resolved from source. The ``codeql-pack.lock.yml`` file will look something like this: +When you execute ``codeql pack install`` from the query pack directory, an appropriate version of ``codeql/cpp-all`` is downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file is created that contains the resolved version of ``codeql/cpp-all``. The lock file won't contain an entry for ``my-company/my-library`` since it is resolved from source dependencies. The ``codeql-pack.lock.yml`` file will look something like this: .. code-block:: yaml @@ -81,4 +86,4 @@ When ``codeql pack install`` is executed from the query pack directory, an appro codeql/cpp-all: version: 0.2.2 -When ``codeql pack publish`` is executed from the query pack directory, the ``codeql/cpp-all`` dependency from the package cache and the ``my-company/my-library`` from the workspace will be bundled with ``my-company/my-queries`` and published to the GitHub container registry. +When you execute``codeql pack publish`` from the query pack directory, the ``codeql/cpp-all`` dependency from the package cache and the ``my-company/my-library`` from the workspace are bundled with ``my-company/my-queries`` and published to the GitHub container registry. From 94a6506cbb6899c709a47edca0a2a5109d20dba4 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 2 Sep 2022 14:05:31 -0700 Subject: [PATCH 06/11] Clarify section on source resolution --- .../codeql/codeql-cli/about-codeql-workspaces.rst | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index cf695a3b6fa..89ed2e2b9b6 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -14,10 +14,10 @@ In most cases, you should store the CodeQL workspace and the CodeQL packs contai The ``codeql-workspace.yml`` file -------------------------------- -A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. +A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. -* The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. -* The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. +* The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. +* The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see" `@actions/glob `__ . @@ -30,7 +30,7 @@ For example, the following ``codeql-workspace.yml`` file defines a workspace tha ignore: - "*/codeql-packs/**/experimental/**/qlpack.yml" -To verify that you have the correct ``codeql-workspace.yml`` file, run ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. +To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that you expect, run ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. CodeQL workspaces and query resolution @@ -46,7 +46,12 @@ This ensures that any local changes you make to a query library in a dependency Note - Source dependencies are CodeQL packs that are resolved from the filesystem. They might be in the same CodeQL workspace, or specified as a path option in the ``--additional-packs`` argument. Source dependencies override any dependencies found in the local package cache and version constraints are ignored. This ensures that during local development version mismatches can be ignored. + Source dependencies are CodeQL packs that are resolved from the local file system. These dependencies can be in the same CodeQL workspace, or specified as a path option using the ``--additional-packs`` argument. When you compile and run queries locally, source dependencies override any dependencies found in the local package cache as well as version constraints defined in the ``qlpack.yml``. + + This is particularly useful in the following situations: + + - One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to refernce that pack. + - You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it. Example ~~~~~~~ From 9a7d74f2ba471acc666fd0a8fc0bea73b08a1df3 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 2 Sep 2022 14:30:47 -0700 Subject: [PATCH 07/11] Reorder table of contents --- docs/codeql/codeql-cli/codeql-cli-reference.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/codeql/codeql-cli/codeql-cli-reference.rst b/docs/codeql/codeql-cli/codeql-cli-reference.rst index 43ac1adbee6..a1928e2439d 100644 --- a/docs/codeql/codeql-cli/codeql-cli-reference.rst +++ b/docs/codeql/codeql-cli/codeql-cli-reference.rst @@ -10,16 +10,16 @@ Learn more about the files you can use when running CodeQL processes and the res :hidden: about-codeql-packs + about-codeql-workspaces query-reference-files sarif-output exit-codes extractor-options - about-codeql-workspaces - :doc:`About CodeQL packs `: CodeQL packs are created with the CodeQL CLI and are used to create, depend on, publish, and run CodeQL queries, libraries, and query suites. +- :doc:`About CodeQL workspaces `: CodeQL workspaces are used to group multiple CodeQL packs together. - :doc:`Query reference files `: A query reference file is text file that defines the location of one query to test. - :doc:`SARIF output `: CodeQL supports SARIF as an output format for sharing static analysis results. - :doc:`Exit codes `: The CodeQL CLI reports the status of each command it runs as an exit code. This exit code provides information for subsequent commands or for other tools that rely on the CodeQL CLI. - :doc:`Extractor options `: You can customize the behavior of extractors by setting options through the CodeQL CLI. -- :doc:`About CodeQL workspaces `: CodeQL workspaces are used to group multiple CodeQL packs together. From 634e88322161ec078a96f49999117e57b0bb9e4e Mon Sep 17 00:00:00 2001 From: Felicity Chapman Date: Mon, 5 Sep 2022 09:14:33 +0100 Subject: [PATCH 08/11] Update docs/codeql/codeql-cli/about-codeql-workspaces.rst --- docs/codeql/codeql-cli/about-codeql-workspaces.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index 89ed2e2b9b6..b3b75063e2b 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -12,7 +12,7 @@ The main benefit of a CodeQL workspace is that it makes it easier for you to dev In most cases, you should store the CodeQL workspace and the CodeQL packs contained in it in one git repository. This makes it easier to share your CodeQL development environment. The ``codeql-workspace.yml`` file --------------------------------- +---------------------------------- A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file contains a ``provide`` block, and optionally an ``ignore`` block. From 2cacba5f66495df31fbba52c446bebc16de221cf Mon Sep 17 00:00:00 2001 From: Felicity Chapman Date: Mon, 5 Sep 2022 09:59:20 +0100 Subject: [PATCH 09/11] Apply suggestions to fix typos and formatting glitches --- docs/codeql/codeql-cli/about-codeql-workspaces.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index b3b75063e2b..af86f3b936e 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -19,7 +19,7 @@ A CodeQL workspace is defined by a ``codeql-workspace.yml`` yaml file. This file * The ``provide`` block contains a list of glob patterns that define the CodeQL packs that are available in the workspace. * The ``ignore`` block contains a list of glob patterns that define CodeQL packs that are not available in the workspace. -Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see" `@actions/glob `__ . +Each entry in the ``provide`` or ``ignore`` section must map to the location of a ``qlpack.yml`` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see "`@actions/glob `__ ." For example, the following ``codeql-workspace.yml`` file defines a workspace that contains all the CodeQL packs recursively found in the ``codeql-packs`` directory, except for the packs in the ``experimental`` directory: @@ -30,13 +30,13 @@ For example, the following ``codeql-workspace.yml`` file defines a workspace tha ignore: - "*/codeql-packs/**/experimental/**/qlpack.yml" -To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that you expect, run ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. +To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that you expect, run the ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. CodeQL workspaces and query resolution -------------------------------------- -All CodeQL packs in a workspace are available as source dependencies for each other when you run any CodeQL command that resolves queries or packs. For example, when you run ``codeql pack install`` in a pack directory in a workspace, any dependency that can be found in the workspace will be instead of downloading that dependency to the package cache and adding it to the ``codeql-pack.lock.yml`` file. For more information, see `:ref:Adding and Installing Dependencies `__. +All CodeQL packs in a workspace are available as source dependencies for each other when you run any CodeQL command that resolves queries or packs. For example, when you run ``codeql pack install`` in a pack directory in a workspace, any dependency that can be found in the workspace will be used instead of downloading that dependency to the package cache and adding it to the ``codeql-pack.lock.yml`` file. For more information, see ":ref:`Adding and Installing Dependencies `." Similarly, when you publish a CodeQL query pack to the GitHub container registry using ``codeql pack publish`` the command will always use the dependencies from the workspace instead of using dependencies found in the local package cache. @@ -50,7 +50,7 @@ This ensures that any local changes you make to a query library in a dependency This is particularly useful in the following situations: - - One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to refernce that pack. + - One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to reference that pack. - You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it. Example @@ -81,7 +81,7 @@ And the following CodeQL query pack ``qlpack.yml`` file in the workspace: my-company/my-library: "*" codeql/cpp-all: ~0.2.0 -Notice that the ``dependencies`` block for the CodeQL query pack,``my-company/my-queries``, specifies ``"*"`` as the version of the library pack. Since the library pack is already defined as a source dependency in ``codeql-workspace.yml``, the library pack's content is always resolved from inside the workspace. Any version constraint you define will be ignored in this case. We recommend that you use ``"*"`` for source dependencies to make it clear that the version is inherited from the workspace. +Notice that the ``dependencies`` block for the CodeQL query pack, ``my-company/my-queries``, specifies ``"*"`` as the version of the library pack. Since the library pack is already defined as a source dependency in ``codeql-workspace.yml``, the library pack's content is always resolved from inside the workspace. Any version constraint you define will be ignored in this case. We recommend that you use ``"*"`` for source dependencies to make it clear that the version is inherited from the workspace. When you execute ``codeql pack install`` from the query pack directory, an appropriate version of ``codeql/cpp-all`` is downloaded to the local package cache. Also, a ``codeql-pack.lock.yml`` file is created that contains the resolved version of ``codeql/cpp-all``. The lock file won't contain an entry for ``my-company/my-library`` since it is resolved from source dependencies. The ``codeql-pack.lock.yml`` file will look something like this: @@ -91,4 +91,4 @@ When you execute ``codeql pack install`` from the query pack directory, an appro codeql/cpp-all: version: 0.2.2 -When you execute``codeql pack publish`` from the query pack directory, the ``codeql/cpp-all`` dependency from the package cache and the ``my-company/my-library`` from the workspace are bundled with ``my-company/my-queries`` and published to the GitHub container registry. +When you execute ``codeql pack publish`` from the query pack directory, the ``codeql/cpp-all`` dependency from the package cache and the ``my-company/my-library`` from the workspace are bundled with ``my-company/my-queries`` and published to the GitHub container registry. From 137b068e4c85b3075fa72230f5ae64f8dbd5f91b Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 6 Sep 2022 11:46:47 -0700 Subject: [PATCH 10/11] Move Source Dependencies description to its own section --- .../codeql-cli/about-codeql-workspaces.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index af86f3b936e..600f97bf6e4 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -32,6 +32,17 @@ For example, the following ``codeql-workspace.yml`` file defines a workspace tha To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that you expect, run the ``codeql pack ls`` command in the same directory as your workspace. The result of the command is a list of all CodeQL packs in the workspace. +.. _source-dependencies: + +Source dependencies +------------------- + +Source dependencies are CodeQL packs that are resolved from the local file system outside of the CodeQL package cache. These dependencies can be in the same CodeQL workspace, or specified as a path option using the ``--additional-packs`` argument. When you compile and run queries locally, source dependencies override any dependencies found in the CodeQL package cache as well as version constraints defined in the ``qlpack.yml``. All references to CodeQL packs in the same workspace are resolved as source dependencies. + +This is particularly useful in the following situations: + +- One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to reference that pack. +- You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it. CodeQL workspaces and query resolution -------------------------------------- @@ -42,17 +53,6 @@ Similarly, when you publish a CodeQL query pack to the GitHub container registry This ensures that any local changes you make to a query library in a dependency are automatically reflected in any query packs you publish from that workspace. -.. pull-quote:: - - Note - - Source dependencies are CodeQL packs that are resolved from the local file system. These dependencies can be in the same CodeQL workspace, or specified as a path option using the ``--additional-packs`` argument. When you compile and run queries locally, source dependencies override any dependencies found in the local package cache as well as version constraints defined in the ``qlpack.yml``. - - This is particularly useful in the following situations: - - - One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to reference that pack. - - You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it. - Example ~~~~~~~ From 65c2477409c4bab69965c1300bdd492046d65a06 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 6 Sep 2022 11:53:17 -0700 Subject: [PATCH 11/11] Add reference to Source Dependencies section --- docs/codeql/codeql-cli/about-codeql-packs.rst | 2 +- docs/codeql/codeql-cli/about-codeql-workspaces.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/codeql/codeql-cli/about-codeql-packs.rst b/docs/codeql/codeql-cli/about-codeql-packs.rst index 0856404d03a..dca601c530a 100644 --- a/docs/codeql/codeql-cli/about-codeql-packs.rst +++ b/docs/codeql/codeql-cli/about-codeql-packs.rst @@ -312,7 +312,7 @@ core query pack: Some extra notes on the following properties: -- ``dependencies``: This query pack depends on ``codeql/cpp-all`` and ``codeql/suite-helpers``. Since these dependencies are resolved from source, it does not matter what version of the CodeQL pack they are compatible with. +- ``dependencies``: This query pack depends on ``codeql/cpp-all`` and ``codeql/suite-helpers``. Since these dependencies are resolved from source, it does not matter what version of the CodeQL pack they are compatible with. For more information about resolving dependencies from source, see ":ref:`Source Dependencies `." - ``suites``: Indicates the directory containing "well-known" query suites. - ``defaultSuiteFile``: The name of the default query suite file that is used when no query suite is specified. diff --git a/docs/codeql/codeql-cli/about-codeql-workspaces.rst b/docs/codeql/codeql-cli/about-codeql-workspaces.rst index 600f97bf6e4..3215c44042e 100644 --- a/docs/codeql/codeql-cli/about-codeql-workspaces.rst +++ b/docs/codeql/codeql-cli/about-codeql-workspaces.rst @@ -34,7 +34,7 @@ To verify that your ``codeql-workspace.yml`` file includes the CodeQL packs that .. _source-dependencies: -Source dependencies +Source Dependencies ------------------- Source dependencies are CodeQL packs that are resolved from the local file system outside of the CodeQL package cache. These dependencies can be in the same CodeQL workspace, or specified as a path option using the ``--additional-packs`` argument. When you compile and run queries locally, source dependencies override any dependencies found in the CodeQL package cache as well as version constraints defined in the ``qlpack.yml``. All references to CodeQL packs in the same workspace are resolved as source dependencies.