From 3bfa8681057459964464f59201a5bfc2acdfe2c2 Mon Sep 17 00:00:00 2001 From: james Date: Thu, 4 Nov 2021 09:53:32 +0000 Subject: [PATCH 1/7] add new option to database analyze tutorial --- .../codeql-cli/analyzing-databases-with-the-codeql-cli.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index effc34f3a1d..0ea124501e1 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -63,6 +63,11 @@ You can also specify: language to the GitHub code scanning API. For more information about this use case, see `Configuring CodeQL CLI in your CI system `__ in the GitHub documentation. +- ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written + in Markdown to SARIF (v2.1.0 or later) analysis results. For each query that runs, CodeQL command + replaces the ``.ql`` extension of the query file with ``.md``. If the .md file exists, + its contents are treated as Markdown-formatted help text for the query and included in the SARIF output. + - .. include:: ../reusables/threads-query-execution.rst From af0f32fdb69e254f9c07f72ddb167442b69d6196 Mon Sep 17 00:00:00 2001 From: james Date: Thu, 4 Nov 2021 12:36:27 +0000 Subject: [PATCH 2/7] further changes for query help in sarif --- ...nalyzing-databases-with-the-codeql-cli.rst | 20 ++++++++++++++++--- .../codeql-cli/testing-query-help-files.rst | 4 ++++ .../query-help-files.rst | 7 +++++++ 3 files changed, 28 insertions(+), 3 deletions(-) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 0ea124501e1..6ffe08708bd 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -64,9 +64,9 @@ You can also specify: see `Configuring CodeQL CLI in your CI system `__ in the GitHub documentation. - ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written - in Markdown to SARIF (v2.1.0 or later) analysis results. For each query that runs, CodeQL command - replaces the ``.ql`` extension of the query file with ``.md``. If the .md file exists, - its contents are treated as Markdown-formatted help text for the query and included in the SARIF output. + in Markdown to SARIF (v2.1.0 or later) analysis results. Query help stored in ``.qhelp`` files must be + converted to ``.md`` before running the analysis. For further information, + see ":doc:`Testing query help files `." - .. include:: ../reusables/threads-query-execution.rst @@ -206,6 +206,20 @@ A SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures that the results are formatted according to the most recent SARIF specification supported by CodeQL. +.. _including-query-help-for-custom-codeql-queries-in-sarif-files: + +Including query help for custom CodeQL queries in SARIF files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Code scanning does not process ``.qhelp`` files for custom CodeQL queries, so to show +query help for custom queries in the code scanning UI you must include markdown-rendered query help +in SARIF files generated during an analysis. + +For CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files +by providing the ``--sarif-add-query-help`` option when running +``codeql database analyze``. Query help written in ``.qhelp`` files cannot be include in SARIF files, +somust be converted to markdown before running the analysis. For more information, see +":doc:`Testing query help files `." Results ------- diff --git a/docs/codeql/codeql-cli/testing-query-help-files.rst b/docs/codeql/codeql-cli/testing-query-help-files.rst index ba5cf3901e7..0aafed4f8e7 100644 --- a/docs/codeql/codeql-cli/testing-query-help-files.rst +++ b/docs/codeql/codeql-cli/testing-query-help-files.rst @@ -15,6 +15,10 @@ in the CodeQL repository. The CodeQL CLI includes a command to test query help and render the content as markdown, so that you can easily preview the content in your IDE. Use the command to validate query help files before uploading them to the CodeQL repository or sharing them with other users. +For CodeQL CLI 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files +generated during CodeQL analyses so that the query help can be displayed in the code scanning UI. +For more information, see +":ref:`Analyzing databases with the CodeQL CLI `." Prerequisites ------------- diff --git a/docs/codeql/writing-codeql-queries/query-help-files.rst b/docs/codeql/writing-codeql-queries/query-help-files.rst index b19ff548a66..60c37fce84f 100644 --- a/docs/codeql/writing-codeql-queries/query-help-files.rst +++ b/docs/codeql/writing-codeql-queries/query-help-files.rst @@ -40,6 +40,13 @@ Query help files are written using a custom XML format, and stored in a file wit The header and single top-level ``qhelp`` element are both mandatory. The following sections explain additional elements that you may include in your query help files. +.. pull-quote:: + + Code scanning does not process ``.qhelp`` files for custom CodeQL queries, so to show + query help for custom queries in the code scanning UI you must convert the ``.qhelp`` files to markdown + and then include the markdown-rendered query help in SARIF files generated during an analysis. + For more information, see + ":ref:`Analyzing databases with the CodeQL CLI `." Section-level elements ====================== From 7236f3b4b6d5d4260ab0426d75d119f553f6bdbd Mon Sep 17 00:00:00 2001 From: james Date: Thu, 4 Nov 2021 12:41:30 +0000 Subject: [PATCH 3/7] improve description of new option --- .../analyzing-databases-with-the-codeql-cli.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 6ffe08708bd..bc59af7b2fb 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -64,9 +64,9 @@ You can also specify: see `Configuring CodeQL CLI in your CI system `__ in the GitHub documentation. - ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written - in Markdown to SARIF (v2.1.0 or later) analysis results. Query help stored in ``.qhelp`` files must be + in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be converted to ``.md`` before running the analysis. For further information, - see ":doc:`Testing query help files `." + see ":doc:`Including query help for custom CodeQL queries in SARIF files `." - .. include:: ../reusables/threads-query-execution.rst @@ -218,8 +218,8 @@ in SARIF files generated during an analysis. For CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files by providing the ``--sarif-add-query-help`` option when running ``codeql database analyze``. Query help written in ``.qhelp`` files cannot be include in SARIF files, -somust be converted to markdown before running the analysis. For more information, see -":doc:`Testing query help files `." +so must be converted to markdown before running the analysis. For more information, see +":ref:`Testing query help files `." Results ------- From 5ea93d64474e5acfc0f37e24bb40cc025602c823 Mon Sep 17 00:00:00 2001 From: james Date: Thu, 4 Nov 2021 14:54:30 +0000 Subject: [PATCH 4/7] further imrpovements --- ...nalyzing-databases-with-the-codeql-cli.rst | 22 ++++++++++++------- .../codeql-cli/testing-query-help-files.rst | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index bc59af7b2fb..0493d1e4985 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -66,7 +66,7 @@ You can also specify: - ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be converted to ``.md`` before running the analysis. For further information, - see ":doc:`Including query help for custom CodeQL queries in SARIF files `." + see ":ref:`Including query help for custom CodeQL queries in SARIF files `." - .. include:: ../reusables/threads-query-execution.rst @@ -211,15 +211,21 @@ supported by CodeQL. Including query help for custom CodeQL queries in SARIF files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Code scanning does not process ``.qhelp`` files for custom CodeQL queries, so to show -query help for custom queries in the code scanning UI you must include markdown-rendered query help -in SARIF files generated during an analysis. +If you use the CodeQL CLI to to run code scanning analyses on third party CI/CD systems, +you can include the query help for your custom queries in SARIF files generated during an analysis. +After uploading the SARIF file to GitHub, the query help is shown in the code scanning UI for any +alerts generated by the custom queries. -For CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files +From CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files by providing the ``--sarif-add-query-help`` option when running -``codeql database analyze``. Query help written in ``.qhelp`` files cannot be include in SARIF files, -so must be converted to markdown before running the analysis. For more information, see -":ref:`Testing query help files `." +``codeql database analyze``. + +You can write query help for custom queries directly in markdown file and save it alongside the +corresponding query. Alternatively, for consistency with the standard CodeQL queries, +you can write query help in the ``.qhelp`` format and convert it to markdown before +running the analysis. Query help written in ``.qhelp`` files can't be included in SARIF files, +and they can't be processed by code scanning. For more information, see +":doc:`Testing query help files `." Results ------- diff --git a/docs/codeql/codeql-cli/testing-query-help-files.rst b/docs/codeql/codeql-cli/testing-query-help-files.rst index 0aafed4f8e7..26cea264043 100644 --- a/docs/codeql/codeql-cli/testing-query-help-files.rst +++ b/docs/codeql/codeql-cli/testing-query-help-files.rst @@ -15,7 +15,7 @@ in the CodeQL repository. The CodeQL CLI includes a command to test query help and render the content as markdown, so that you can easily preview the content in your IDE. Use the command to validate query help files before uploading them to the CodeQL repository or sharing them with other users. -For CodeQL CLI 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files +From CodeQL CLI 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files generated during CodeQL analyses so that the query help can be displayed in the code scanning UI. For more information, see ":ref:`Analyzing databases with the CodeQL CLI `." From c94bfc306a044e627827ec6c18d9141c9c7a326e Mon Sep 17 00:00:00 2001 From: james Date: Mon, 8 Nov 2021 11:18:41 +0000 Subject: [PATCH 5/7] improve links --- .../analyzing-databases-with-the-codeql-cli.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 0493d1e4985..8b8b8dabf47 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -219,13 +219,16 @@ alerts generated by the custom queries. From CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files by providing the ``--sarif-add-query-help`` option when running ``codeql database analyze``. +For more information, see `Configuring CodeQL CLI in your CI system ` +in the GitHub documentation. -You can write query help for custom queries directly in markdown file and save it alongside the +You can write query help for custom queries directly in a markdown file and save it alongside the corresponding query. Alternatively, for consistency with the standard CodeQL queries, -you can write query help in the ``.qhelp`` format and convert it to markdown before -running the analysis. Query help written in ``.qhelp`` files can't be included in SARIF files, -and they can't be processed by code scanning. For more information, see -":doc:`Testing query help files `." +you can write query help in the ``.qhelp`` format. Query help written in ``.qhelp`` +files can't be included in SARIF files, and they can't be processed by code +scanning so must be converted to markdown before running +the analysis. For more information, see ":doc:`Query help files `" +and ":doc:`Testing query help files `." Results ------- From dfe77f844f68cacdb75db4249074a20fed120e6a Mon Sep 17 00:00:00 2001 From: james Date: Mon, 8 Nov 2021 11:59:53 +0000 Subject: [PATCH 6/7] fix errors in debugging-data-flow-queries-using-partial-flow.rst --- .../debugging-data-flow-queries-using-partial-flow.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/codeql/writing-codeql-queries/debugging-data-flow-queries-using-partial-flow.rst b/docs/codeql/writing-codeql-queries/debugging-data-flow-queries-using-partial-flow.rst index 05a12ad98ee..e3587550e7f 100644 --- a/docs/codeql/writing-codeql-queries/debugging-data-flow-queries-using-partial-flow.rst +++ b/docs/codeql/writing-codeql-queries/debugging-data-flow-queries-using-partial-flow.rst @@ -10,7 +10,6 @@ A typical data-flow query looks like this: .. code-block:: ql -:: class MyConfig extends TaintTracking::Configuration { MyConfig() { this = "MyConfig" } @@ -79,7 +78,8 @@ To avoid these problems, a data-flow ``Configuration`` comes with a mechanism fo */ final predicate hasPartialFlow(PartialPathNode source, PartialPathNode node, int dist) { -As noted in the documentation for ``hasPartialFlow`` (for example, in the `CodeQL for Java documentation __`) you must first enable this by adding an override of ``explorationLimit``. For example: +As noted in the documentation for ``hasPartialFlow`` (for example, in the +`CodeQL for Java documentation `__) you must first enable this by adding an override of ``explorationLimit``. For example: .. code-block:: ql From 96ff2f5125571ea28988c55b902499e178a91d87 Mon Sep 17 00:00:00 2001 From: james Date: Mon, 8 Nov 2021 12:06:20 +0000 Subject: [PATCH 7/7] use correct type of link --- .../codeql-cli/analyzing-databases-with-the-codeql-cli.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst index 8b8b8dabf47..c180d9b5a9d 100644 --- a/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst +++ b/docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst @@ -227,7 +227,7 @@ corresponding query. Alternatively, for consistency with the standard CodeQL que you can write query help in the ``.qhelp`` format. Query help written in ``.qhelp`` files can't be included in SARIF files, and they can't be processed by code scanning so must be converted to markdown before running -the analysis. For more information, see ":doc:`Query help files `" +the analysis. For more information, see ":ref:`Query help files `" and ":doc:`Testing query help files `." Results