From 970232623239301b10cf8596ce51b6dfd0abdea2 Mon Sep 17 00:00:00 2001 From: james Date: Mon, 23 Nov 2020 16:18:45 +0000 Subject: [PATCH] add leftover docs to support project --- docs/codeql/support/index.rst | 8 ++- docs/codeql/support/notes-ql-users.rst | 91 ++++++++++++++++++++++++++ docs/codeql/support/ql-training.rst | 63 ++++++++++++++++++ 3 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 docs/codeql/support/notes-ql-users.rst create mode 100644 docs/codeql/support/ql-training.rst diff --git a/docs/codeql/support/index.rst b/docs/codeql/support/index.rst index 6236b4c1543..3eac7708109 100644 --- a/docs/codeql/support/index.rst +++ b/docs/codeql/support/index.rst @@ -11,4 +11,10 @@ For details see: language-support.rst framework-support.rst -For details of the CodeQL libraries, see `CodeQL standard libraries `_. \ No newline at end of file +For details of the CodeQL libraries, see `CodeQL standard libraries `_. + +.. toctree:: + :hidden: + + notes-ql-users + ql-training \ No newline at end of file diff --git a/docs/codeql/support/notes-ql-users.rst b/docs/codeql/support/notes-ql-users.rst new file mode 100644 index 00000000000..7e5f032279e --- /dev/null +++ b/docs/codeql/support/notes-ql-users.rst @@ -0,0 +1,91 @@ +Notes for legacy QL CLI users +============================= + +If you've previously used the QL command-line tools (``odasa``), you'll notice a +few key differences when you use the new CodeQL products: + +* "QL snapshots" are now called `CodeQL databases `__. +* The process of creating a CodeQL database is much simpler and more streamlined. + There's no need to create ``projects`` or ``snapshots``---just check out the + code and build it using the CodeQL CLI ``codeql database create`` command. +* Queries are run against CodeQL databases using the CodeQL CLI ``codeql + database analyze`` command. + +For more information, see `Creating CodeQL databases +`__ and +`Analyzing databases with the CodeQL CLI `__. +For detailed guidance about equivalent commands, see `Overview of common commands +<#overview-of-common-commands>`__ below. + +.. _database-compatibiilty-notes: + +Database compatibility notes +---------------------------- + +A CodeQL database created by the CodeQL CLI serves the same purpose as a QL +snapshot created using ``odasa``. They both contain a code database to query and +usually a source reference for results display. However, they are not identical +formats and, if you use the legacy QL tools alongside the CodeQL tools, you need +to be aware of the following: + +* Existing QL snapshots, exported using the legacy CLI, can be used with the new + CodeQL tools. Unzip the snapshot and treat the directory as a database. If it + was built with an earlier version of the legacy CLI, you may need to upgrade + the database using ``codeql database upgrade``. For more information, see the + `database upgrade reference documentation + `__. + +* CodeQL databases are not directly compatible with CodeQL for Eclipse. + However, you can "bundle" a CodeQL database into the equivalent of a QL + exported snapshot by running:: + + codeql database bundle --include-uncompressed-source -o + + The resulting database can be imported into CodeQL for Eclipse. For more + information, see the `database bundle reference documentation `__. + +* .. include:: ../reusables/index-files-note.rst + +* CodeQL databases cannot be directly uploaded to an LGTM Enterprise instance. + For more information, see `Preparing CodeQL databases to upload to LGTM + `__ + in the LGTM admin help. + +Query suites +------------ + +CodeQL includes a new, more flexible, format for query suites. Legacy query +suite definitions are not compatible with the new CodeQL tools. For more +information about CodeQL query suites, see `Creating CodeQL query suites +`__. + +Overview of common commands +--------------------------- + +If you're switching from the legacy ODASA CLI to the new CodeQL CLI, +the table below shows which commands replace the most +common ODASA processes. + ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``odasa`` command | Corresponding ``codeql`` command | Notes | ++==========================================+===================================================================================================+=========================================================================================================================================================================================================================================================================================================================================================================================================================================================================+ +| ``bootstrap`` | n/a | CodeQL analysis does not use ``project`` files during database creation. For more information about creating databases, see `Creating CodeQL databases `__. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``addSnapshot``, ``addLatestSnapshot`` | n/a | To obtain the version of the code you want to analyze, just run your normal check-out commands. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``buildSnapshot`` | `database create `__ | When creating a CodeQL database, you specfiy build commands in the command line, rather than in a project file. For more information, see `Creating CodeQL databases `__. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``analyzeSnapshot`` | `database analyze `__ | For more information, see `Analyzing databases with the CodeQL CLI `__. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``archiveSnapshot`` | `database cleanup `__ | Use ``database cleanup`` to reduce the size of a CodeQL database by deleting temporary data. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``export`` | `database bundle `__ | You don't need to export databases before adding them to VS Code. However, you should "bundle" CodeQL databases before using them with LGTM Enterprise, CodeQL for Eclipse, or CodeQL for Visual Studio. For more information, see `Preparing CodeQL databases to upload to LGTM `__ in the LGTM admin help and the `Database compatibility notes <#database-compatibility-notes>`__. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``prepareQueries`` | `query compile `__ | Queries are compiled when you run ``database analyze`` and other query-running commands. You can speed up compilation by running ``query compile`` separately using more threads. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``qltest`` | `test run `__ | For more information about running regression tests, see `Testing custom queries `__. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``runQuery`` | `query run `__ | Use ``query run`` to quickly view results in your terminal. To generate interpreted results that can be viewed in source code, use ``database analyze``. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ``upgrade`` | `database upgrade `__ | For more information, see `Upgrading CodeQL databases `__. | ++------------------------------------------+---------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ \ No newline at end of file diff --git a/docs/codeql/support/ql-training.rst b/docs/codeql/support/ql-training.rst new file mode 100644 index 00000000000..bb4dc9a3f2e --- /dev/null +++ b/docs/codeql/support/ql-training.rst @@ -0,0 +1,63 @@ +CodeQL training and variant analysis examples +============================================= + +CodeQL and variant analysis +--------------------------- + +Variant analysis is the process of using a known vulnerability as a seed to find similar problems in your code. Security engineers typically perform variant analysis to identify possible vulnerabilities and to ensure that these threats are properly fixed across multiple code bases. + +CodeQL is the code analysis engine that underpins LGTM, the community driven security analysis platform. Together, CodeQL and LGTM provide continuous monitoring and scalable variant analysis for your projects, even if you don’t have your own team of dedicated security engineers. You can read more about using CodeQL and LGTM in variant analysis on the `Security Lab research page `__. + +CodeQL is easy to learn, and exploring code using CodeQL is the most efficient way to perform variant analysis. + +Learning CodeQL for variant analysis +------------------------------------ + +Start learning how to use CodeQL in variant analysis for a specific language by looking at the topics below. Each topic links to a short presentation on CodeQL, its libraries, or an example variant discovered using CodeQL. + +.. |arrow-l| unicode:: U+2190 + +.. |arrow-r| unicode:: U+2192 + +.. |info| unicode:: U+24D8 + +When you have selected a presentation, use |arrow-r| and |arrow-l| to navigate between slides. +Press **p** to view the additional notes on slides that have an information icon |info| in the top right corner, and press **f** to enter full-screen mode. + +The presentations contain a number of query examples. +We recommend that you download `CodeQL for Visual Studio Code `__ and add the example database for each presentation so that you can find the bugs mentioned in the slides. + + +.. pull-quote:: + + Information + + The presentations listed below are used in CodeQL and variant analysis training sessions run by GitHub engineers. + Therefore, be aware that the slides are designed to be presented by an instructor. + If you are using the slides without an instructor, please use the additional notes to help guide you through the examples. + +CodeQL and variant analysis for C/C++ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- `Introduction to variant analysis: CodeQL for C/C++ `__–an introduction to variant analysis and CodeQL for C/C++ programmers. +- `Example: Bad overflow guard `__–an example of iterative query development to find bad overflow guards in a C++ project. +- `Program representation: CodeQL for C/C++ `__–information on how CodeQL analysis represents C/C++ programs. +- `Introduction to local data flow `__–an introduction to analyzing local data flow in C/C++ using CodeQL, including an example demonstrating how to develop a query to find a real CVE. +- `Exercise: snprintf overflow `__–an example demonstrating how to develop a data flow query. +- `Introduction to global data flow `__–an introduction to analyzing global data flow in C/C++ using CodeQL. +- `Analyzing control flow: CodeQL for C/C++ `__–an introduction to analyzing control flow in C/C++ using CodeQL. + +CodeQL and variant analysis for Java +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- `Introduction to variant analysis: CodeQL for Java `__–an introduction to variant analysis and CodeQL for Java programmers. +- `Example: Query injection `__–an example of iterative query development to find unsanitized SPARQL injections in a Java project. +- `Program representation: CodeQL for Java `__–information on how CodeQL analysis represents Java programs. +- `Introduction to local data flow `__–an introduction to analyzing local data flow in Java using CodeQL, including an example demonstrating how to develop a query to find a real CVE. +- `Exercise: Apache Struts `__–an example demonstrating how to develop a data flow query. +- `Introduction to global data flow `__–an introduction to analyzing global data flow in Java using CodeQL. + +Further reading +~~~~~~~~~~~~~~~ + +- `GitHub Security Lab `__ \ No newline at end of file