From 8661de11f2ff9c5651899095b3465f11faaf5216 Mon Sep 17 00:00:00 2001 From: james Date: Tue, 5 Nov 2019 14:50:28 +0000 Subject: [PATCH 1/8] docs: update ql terminology --- docs/language/learn-ql/ql-training.rst | 52 +++++++++---------- .../ql-training/cpp/bad-overflow-guard.rst | 24 ++++----- .../ql-training/cpp/control-flow-cpp.rst | 4 +- .../ql-training/cpp/data-flow-cpp.rst | 4 +- .../ql-training/cpp/global-data-flow-cpp.rst | 4 +- .../language/ql-training/cpp/intro-ql-cpp.rst | 38 +++++++------- .../cpp/program-representation-cpp.rst | 10 ++-- docs/language/ql-training/cpp/snprintf.rst | 2 +- docs/language/ql-training/index.rst | 4 +- .../ql-training/java/apache-struts-java.rst | 2 +- .../java/global-data-flow-java.rst | 2 +- .../ql-training/java/intro-ql-java.rst | 36 ++++++------- .../java/program-representation-java.rst | 2 +- .../ql-training/java/query-injection-java.rst | 20 +++---- .../slide-snippets/abstract-syntax-tree.rst | 10 ++-- .../ql-training/slide-snippets/info.rst | 33 ------------ .../slide-snippets/intro-ql-general.rst | 16 +++--- 17 files changed, 115 insertions(+), 148 deletions(-) delete mode 100644 docs/language/ql-training/slide-snippets/info.rst diff --git a/docs/language/learn-ql/ql-training.rst b/docs/language/learn-ql/ql-training.rst index 66afa06d754..757de7daed6 100644 --- a/docs/language/learn-ql/ql-training.rst +++ b/docs/language/learn-ql/ql-training.rst @@ -1,19 +1,19 @@ -QL training and variant analysis examples -######################################### +CodeQL training and variant analysis examples +============================================= QL and variant analysis -======================= +----------------------- `Variant analysis `__ is the process of using a known vulnerability as a seed to find similar problems in your code. Security engineers typically perform variant analysis to identify possible vulnerabilities and to ensure that these threats are properly fixed across multiple code bases. -`QL `__ is Semmle's variant analysis engine, and it is also the technology that underpins LGTM, Semmle's community driven security analysis platform. Together, QL and LGTM provide continuous monitoring and scalable variant analysis for your projects, even if you don’t have your own team of dedicated security engineers. You can read more about using QL and LGTM in variant analysis in the `Semmle blog `__. +`CodeQL `__ is the code analysis engine that underpins LGTM, Semmle's community driven security analysis platform. Together, CodeQL and LGTM provide continuous monitoring and scalable variant analysis for your projects, even if you don’t have your own team of dedicated security engineers. You can read more about using CodeQL and LGTM in variant analysis in the `Semmle blog `__. -The QL language is easy to learn, and exploring code using QL is the most efficient way to perform variant analysis. +CodeQL is easy to learn, and exploring code using CodeQL is the most efficient way to perform variant analysis. Learning QL for variant analysis -================================ +-------------------------------- -Start learning how to use QL in variant analysis for a specific language by looking at the topics below. Each topic links to a short presentation on the QL language, QL libraries, or an example variant discovered using QL. +Start learning how to use CodeQL in variant analysis for a specific language by looking at the topics below. Each topic links to a short presentation on CodeQL, its libraries, or an example variant discovered using CodeQL. .. |arrow-l| unicode:: U+2190 @@ -24,7 +24,7 @@ Start learning how to use QL in variant analysis for a specific language by look When you have selected a presentation, use |arrow-r| and |arrow-l| to navigate between slides. Press **p** to view the additional notes on slides that have an information icon |info| in the top right corner, and press **f** to enter full-screen mode. -The presentations contain a number of QL query examples. +The presentations contain a number of query examples. We recommend that you download `QL for Eclipse `__ and import the example snapshot for each presentation so that you can find the bugs mentioned in the slides. @@ -32,35 +32,35 @@ We recommend that you download `QL for Eclipse `__–an introduction to variant analysis and QL for C/C++ programmers. +- `Introduction to variant analysis: CodeQL for C/C++ <../ql-training/cpp/intro-ql-cpp.html>`__–an introduction to variant analysis and CodeQL for C/C++ programmers. - `Example: Bad overflow guard <../ql-training/cpp/bad-overflow-guard.html>`__–an example of iterative query development to find bad overflow guards in a C++ project. -- `Program representation: QL for C/C++ <../ql-training/cpp/program-representation-cpp.html>`__–information on how QL analysis represents C/C++ programs. -- `Introduction to local data flow <../ql-training/cpp/data-flow-cpp.html>`__–an introduction to analyzing local data flow in C/C++ using QL, including an example demonstrating how to develop a query to find a real CVE. +- `Program representation: CodeQL for C/C++ <../ql-training/cpp/program-representation-cpp.html>`__–information on how CodeQL analysis represents C/C++ programs. +- `Introduction to local data flow <../ql-training/cpp/data-flow-cpp.html>`__–an introduction to analyzing local data flow in C/C++ using CodeQL, including an example demonstrating how to develop a query to find a real CVE. - `Exercise: snprintf overflow <../ql-training/cpp/snprintf.html>`__–an example demonstrating how to develop a data flow query. -- `Introduction to global data flow <../ql-training/cpp/global-data-flow-cpp.html>`__–an introduction to analyzing global data flow in C/C++ using QL. -- `Analyzing control flow: QL for C/C++ <../ql-training/cpp/control-flow-cpp.html>`__–an introduction to analyzing control flow in C/C++ using QL. +- `Introduction to global data flow <../ql-training/cpp/global-data-flow-cpp.html>`__–an introduction to analyzing global data flow in C/C++ using CodeQL. +- `Analyzing control flow: CodeQL for C/C++ <../ql-training/cpp/control-flow-cpp.html>`__–an introduction to analyzing control flow in C/C++ using QL. -QL and variant analysis for Java --------------------------------- +CodeQL and variant analysis for Java +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- `Introduction to variant analysis: QL for Java <../ql-training/java/intro-ql-java.html>`__–an introduction to variant analysis and QL for Java programmers. +- `Introduction to variant analysis: CodeQL for Java <../ql-training/java/intro-ql-java.html>`__–an introduction to variant analysis and CodeQL for Java programmers. - `Example: Query injection <../ql-training/java/query-injection-java.html>`__–an example of iterative query development to find unsanitized SPARQL injections in a Java project. -- `Program representation: QL for Java <../ql-training/java/program-representation-java.html>`__–information on how QL analysis represents Java programs. -- `Introduction to local data flow <../ql-training/java/data-flow-java.html>`__–an introduction to analyzing local data flow in Java using QL, including an example demonstrating how to develop a query to find a real CVE. +- `Program representation: CodeQL for Java <../ql-training/java/program-representation-java.html>`__–information on how CodeQL analysis represents Java programs. +- `Introduction to local data flow <../ql-training/java/data-flow-java.html>`__–an introduction to analyzing local data flow in Java using CodeQL, including an example demonstrating how to develop a query to find a real CVE. - `Exercise: Apache Struts <../ql-training/java/apache-struts-java.html>`__–an example demonstrating how to develop a data flow query. -- `Introduction to global data flow <../ql-training/java/global-data-flow-java.html>`__–an introduction to analyzing global data flow in Java using QL. +- `Introduction to global data flow <../ql-training/java/global-data-flow-java.html>`__–an introduction to analyzing global data flow in Java using CodeQL. More resources --------------- +~~~~~~~~~~~~~~ -- If you are completely new to QL, look at our introductory topics in :ref:`Getting started `. -- To find more detailed information about how to write QL queries for specific languages, visit the links in :ref:`Writing QL queries `. -- To read more about how QL queries have been used in Semmle's security research, and to read about new QL developments, visit the `Semmle blog `__. +- If you are completely new to CodeQL, look at our introductory topics in :doc:`Learning CodeQL `. +- To find more detailed information about how to write CodeQL queries for specific languages, visit the links in :ref:`Writing CodeQL queries `. +- To read more about how CodeQL queries have been used in Semmle's security research, and to read about new CodeQL developments, visit the `Semmle blog `__. - Find more examples of queries written by Semmle's own security researchers in the `Semmle Demos repository `__ on GitHub. diff --git a/docs/language/ql-training/cpp/bad-overflow-guard.rst b/docs/language/ql-training/cpp/bad-overflow-guard.rst index 12f35440906..5e9fbcfc700 100644 --- a/docs/language/ql-training/cpp/bad-overflow-guard.rst +++ b/docs/language/ql-training/cpp/bad-overflow-guard.rst @@ -2,7 +2,7 @@ Example: Bad overflow guard =========================== -QL for C/C++ +CodeQL for C/C++ .. container:: semmle-logo @@ -127,13 +127,13 @@ This happens even though the overflow check passed! .. rst-class:: background2 -Developing a QL query -===================== +Developing a CodeQL query +========================= Finding bad overflow guards -QL query: bad overflow guards -============================= +CodeQL query: bad overflow guards +================================== Let’s look for overflow guards of the form ``v + b < v``, using the classes ``AddExpr``, ``Variable`` and ``RelationalOperation`` from the ``cpp`` library. @@ -153,10 +153,10 @@ Let’s look for overflow guards of the form ``v + b < v``, using the classes - a ``RelationalOperation``: the overflow comparison check. - a ``Variable``: used as an argument to both the addition and comparison. - - The ``where`` part of the query ties these three QL variables together using `predicates `__ defined in the `standard QL for C/C++ library `__. + - The ``where`` part of the query ties these three variables together using `predicates `__ defined in the `standard CodeQL for C/C++ library `__. -QL query: bad overflow guards -============================= +CodeQL query: bad overflow guards +================================= We want to ensure the operands being added have size less than 4 bytes. @@ -180,8 +180,8 @@ We can get the size (in bytes) of a type using the ``getSize()`` method. - We therefore write a helper predicate for small expressions. - This predicate effectively represents the set of all expressions in the database where the size of the type of the expression is less than 4 bytes, that is, less than 32-bits. -QL query: bad overflow guards -============================= +CodeQL query: bad overflow guards +================================== We can ensure the operands being added have size less than 4 bytes, using our new predicate. @@ -216,8 +216,8 @@ Now our query becomes: - The “range” part, ``op = a.getAnOperand()``, restricts ``op`` to being one of the two operands to the addition. - The “condition” part, ``isSmall(op)``, says that the ``forall`` holds only if the condition (that the ``op`` is small) holds for everything in the range–that is, both the arguments to the addition. -QL query: bad overflow guards -============================= +CodeQL query: bad overflow guards +================================= Sometimes the result of the addition is cast to a small type of size less than 4 bytes, preventing automatic widening. We don’t want our query to flag these instances. diff --git a/docs/language/ql-training/cpp/control-flow-cpp.rst b/docs/language/ql-training/cpp/control-flow-cpp.rst index ea8e2c1b158..a8171a44b5d 100644 --- a/docs/language/ql-training/cpp/control-flow-cpp.rst +++ b/docs/language/ql-training/cpp/control-flow-cpp.rst @@ -2,7 +2,7 @@ Analyzing control flow ====================== -QL for C/C++ +CodeQL for C/C++ .. container:: semmle-logo @@ -226,7 +226,7 @@ A ``GuardCondition`` is a ``Boolean`` condition that controls one or more basic Further materials ================= -- QL for C/C++: https://help.semmle.com/QL/learn-ql/ql/cpp/ql-for-cpp.html +- CodeQL for C/C++: https://help.semmle.com/QL/learn-ql/ql/cpp/ql-for-cpp.html - API reference: https://help.semmle.com/qldoc/cpp .. rst-class:: end-slide diff --git a/docs/language/ql-training/cpp/data-flow-cpp.rst b/docs/language/ql-training/cpp/data-flow-cpp.rst index 2b020499eb0..5377d8d52d1 100644 --- a/docs/language/ql-training/cpp/data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/data-flow-cpp.rst @@ -86,9 +86,9 @@ Write a query that flags ``printf`` calls where the format argument is not a ``S .. note:: - This first query is about finding places where the format specifier is not a constant string. In QL for C/C++, constant strings are modeled as ``StringLiteral`` nodes, so we are looking for calls to format functions where the format specifier argument is not a string literal. + This first query is about finding places where the format specifier is not a constant string. In CodeQL for C/C++, constant strings are modeled as ``StringLiteral`` nodes, so we are looking for calls to format functions where the format specifier argument is not a string literal. - The `C/C++ standard libraries `__ include many different formatting functions that may be vulnerable to this particular attack–including ``printf``, ``snprintf``, and others. Furthermore, each of these different formatting functions may include the format string in a different position in the argument list. Instead of laboriously listing all these different variants, we can make use of the QL for C/C++ standard library class ``FormattingFunction``, which provides an interface that models common formatting functions in C/C++. + The `C/C++ standard libraries `__ include many different formatting functions that may be vulnerable to this particular attack–including ``printf``, ``snprintf``, and others. Furthermore, each of these different formatting functions may include the format string in a different position in the argument list. Instead of laboriously listing all these different variants, we can make use of the CodeQL for C/C++ standard library class ``FormattingFunction``, which provides an interface that models common formatting functions in C/C++. Meh... ====== diff --git a/docs/language/ql-training/cpp/global-data-flow-cpp.rst b/docs/language/ql-training/cpp/global-data-flow-cpp.rst index 6033581ffc3..bc048ac59f4 100644 --- a/docs/language/ql-training/cpp/global-data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/global-data-flow-cpp.rst @@ -2,7 +2,7 @@ Introduction to global data flow ================================ -QL for C/C++ +CodeQL for C/C++ .. container:: semmle-logo @@ -77,7 +77,7 @@ The library class ``SecurityOptions`` provides a (configurable) model of what co .. note:: - We first define what it means to be a *source* of tainted data for this particular problem. In this case, what we care about is whether the format string can be provided by an external user to our application or service. As there are many such ways external data could be introduced into the system, the standard QL libraries for C/C++ include an extensible API for modeling user input. In this case, we will simply use the predefined set of *user inputs*, which includes arguments provided to command line applications. + We first define what it means to be a *source* of tainted data for this particular problem. In this case, what we care about is whether the format string can be provided by an external user to our application or service. As there are many such ways external data could be introduced into the system, the standard CodeQL libraries for C/C++ include an extensible API for modeling user input. In this case, we will simply use the predefined set of *user inputs*, which includes arguments provided to command line applications. Defining sinks (exercise) diff --git a/docs/language/ql-training/cpp/intro-ql-cpp.rst b/docs/language/ql-training/cpp/intro-ql-cpp.rst index 82eb62a3ba8..dddc86ca77f 100644 --- a/docs/language/ql-training/cpp/intro-ql-cpp.rst +++ b/docs/language/ql-training/cpp/intro-ql-cpp.rst @@ -2,7 +2,7 @@ Introduction to variant analysis ================================ -QL for C/C++ +CodeQL for C/C++ .. container:: semmle-logo @@ -56,14 +56,14 @@ Oops .. note:: - Here’s a simple (artificial) bug, which we’ll develop a QL query to catch. + Here’s a simple (artificial) bug, which we’ll develop a query to catch. This function writes a value to a given location in an array, first trying to do a bounds check to validate that the location is within bounds. However, the return statement has been commented out, leaving a redundant if statement and no bounds checking. This case can act as our “patient zero” in the variant analysis game. -A simple QL query -================= +A simple CodeQL query +===================== .. literalinclude:: ../query-examples/cpp/empty-if-cpp.ql :language: ql @@ -72,9 +72,9 @@ A simple QL query We are going to write a simple query which finds “if statements” with empty “then” blocks, so we can highlight the results like those on the previous slide. The query can be run in the `query console on LGTM `__, or in your `IDE `__. - A `QL query `__ consists of a “select” clause that indicates what results should be returned. Typically it will also provide a “from” clause to declare some variables, and a “where” clause to state conditions over those variables. For more information on the structure of query files (including links to useful topics in the `QL language handbook `__), see `Introduction to query files `__. + A `query `__ consists of a “select” clause that indicates what results should be returned. Typically it will also provide a “from” clause to declare some variables, and a “where” clause to state conditions over those variables. For more information on the structure of query files (including links to useful topics in the `QL language handbook `__), see `Introduction to query files `__. - In our example here, the first line of the query imports the `C/C++ standard QL library `__, which defines concepts like ``IfStmt`` and ``Block``. + In our example here, the first line of the query imports the `CodeQL for C/C++ standard library `__, which defines concepts like ``IfStmt`` and ``Block``. The query proper starts by declaring two variables–ifStmt and block. These variables represent sets of values in the database, according to the type of each of the variables. For example, ifStmt has the type IfStmt, which means it represents the set of all if statements in the program. If we simply selected these two variables:: @@ -97,8 +97,8 @@ A simple QL query -Structure of a QL query -======================= +Structure of a query +==================== A **query file** has the extension ``.ql`` and contains a **query clause**, and optionally **predicates**, **classes**, and **modules**. @@ -110,14 +110,14 @@ Each query library also implicitly defines a module. .. note:: - QL queries are always contained in query files with the file extension ``.ql``. `Quick queries `__, run in `QL for Eclipse `__, are no exception: the quick query window maintains a temporary QL file in the background. + Queries are always contained in query files with the file extension ``.ql``. `Quick queries `__, run in `QL for Eclipse `__, are no exception: the quick query window maintains a temporary query file in the background. - Parts of queries can be lifted into `QL library files `__ with the extension ``qll``. Definitions within such libraries can be brought into scope using ``import`` statements, and similarly QLL files can import each other’s definitions using “import” statements. + Parts of queries can be lifted into `library files `__ with the extension ``qll``. Definitions within such libraries can be brought into scope using ``import`` statements, and similarly QLL files can import each other’s definitions using “import” statements. Logic can be encapsulated as user-defined `predicates `__ and `classes `__, and organized into `modules `__. Each QLL file implicitly defines a module, but QL and QLL files can also contain explicit module definitions, as we will see later. -Predicates in QL -================ +Predicates +========== A predicate allows you to pull out and name parts of a query. @@ -135,14 +135,14 @@ A predicate allows you to pull out and name parts of a query. .. note:: - A QL predicate takes zero or more parameters, and its body is a condition on those parameters. The predicate may (or may not) hold. Predicates may also be recursive, simply by referring to themselves (directly or indirectly). + A `predicate `__ takes zero or more parameters, and its body is a condition on those parameters. The predicate may (or may not) hold. Predicates may also be `recursive `__, simply by referring to themselves (directly or indirectly). You can imagine a predicate to be a self-contained from-where-select statement, that produces an intermediate relation, or table. In this case, the ``isEmpty`` predicate will be the set of all blocks which are empty. -Classes in QL -============= +Classes +======= -A QL class allows you to name a set of values and define (member) predicates on them. +A class allows you to name a set of values and define (member) predicates on them. A class has at least one supertype and optionally a **characteristic predicate**; it contains the values that belong to *all* supertypes *and* satisfy the characteristic predicate, if provided. @@ -162,8 +162,8 @@ Member predicates are inherited and can be overridden. In the example, declaring a variable “EmptyBlock e” will allow it to range over only those blocks that have zero statements. -Classes in QL continued -======================= +Classes continued +================= .. container:: column-left @@ -198,7 +198,7 @@ Iterative query refinement .. note:: - QL makes it very easy to experiment with analysis ideas. A common workflow is to start with a simple query (like our “redundant if-statement” example), examine a few results, refine the query based on any patterns that emerge and repeat. + CodeQL makes it very easy to experiment with analysis ideas. A common workflow is to start with a simple query (like our “redundant if-statement” example), examine a few results, refine the query based on any patterns that emerge and repeat. As an exercise, refine the redundant-if query based on the observation that if the if-statement has an “else” clause, then even if the body of the “then” clause is empty, it’s not actually redundant. diff --git a/docs/language/ql-training/cpp/program-representation-cpp.rst b/docs/language/ql-training/cpp/program-representation-cpp.rst index 1850e3e5671..b21677fccab 100644 --- a/docs/language/ql-training/cpp/program-representation-cpp.rst +++ b/docs/language/ql-training/cpp/program-representation-cpp.rst @@ -2,7 +2,7 @@ Program representation ====================== -QL for C/C++ +CodeQL for C/C++ .. container:: semmle-logo @@ -25,8 +25,8 @@ Agenda .. resume slides -AST QL classes -============== +AST classes +=========== Important AST classes include: @@ -66,9 +66,9 @@ Working with variables Working with functions ====================== -Functions are represented by the Function QL class. Each declaration or definition of a function is represented by a ``FunctionDeclarationEntry``. +Functions are represented by the Function class. Each declaration or definition of a function is represented by a ``FunctionDeclarationEntry``. -Calls to functions are modeled by QL class Call and its subclasses: +Calls to functions are modeled by class Call and its subclasses: - ``Call.getTarget()`` gets the declared target of the call; undefined for calls through function pointers - ``Function.getACallToThisFunction()`` gets a call to this function diff --git a/docs/language/ql-training/cpp/snprintf.rst b/docs/language/ql-training/cpp/snprintf.rst index 77e46933fcb..dafebc4a720 100644 --- a/docs/language/ql-training/cpp/snprintf.rst +++ b/docs/language/ql-training/cpp/snprintf.rst @@ -2,7 +2,7 @@ Exercise: ``snprintf`` overflow =============================== -QL for C/C++ +CodeQL for C/C++ .. container:: semmle-logo diff --git a/docs/language/ql-training/index.rst b/docs/language/ql-training/index.rst index 0cc4ca9b024..309fe92c052 100644 --- a/docs/language/ql-training/index.rst +++ b/docs/language/ql-training/index.rst @@ -1,5 +1,5 @@ -QL training and variant analysis examples -========================================= +CodeQL training and variant analysis examples +============================================= .. container:: semmle-logo diff --git a/docs/language/ql-training/java/apache-struts-java.rst b/docs/language/ql-training/java/apache-struts-java.rst index 7fcd2e003cd..5c27ec73979 100644 --- a/docs/language/ql-training/java/apache-struts-java.rst +++ b/docs/language/ql-training/java/apache-struts-java.rst @@ -67,7 +67,7 @@ RCE in Apache Struts Finding the RCE yourself ======================== -#. Create a QL class to find the interface ``org.apache.struts2.rest.handler.ContentTypeHandler`` +#. Create a class to find the interface ``org.apache.struts2.rest.handler.ContentTypeHandler`` **Hint**: Use predicate ``hasQualifiedName(...)`` diff --git a/docs/language/ql-training/java/global-data-flow-java.rst b/docs/language/ql-training/java/global-data-flow-java.rst index 665899f8459..f43c31e0f4f 100644 --- a/docs/language/ql-training/java/global-data-flow-java.rst +++ b/docs/language/ql-training/java/global-data-flow-java.rst @@ -2,7 +2,7 @@ Introduction to global data flow ================================ -QL for Java +CodeQL for Java .. container:: semmle-logo diff --git a/docs/language/ql-training/java/intro-ql-java.rst b/docs/language/ql-training/java/intro-ql-java.rst index 392c18309cb..f80f878492d 100644 --- a/docs/language/ql-training/java/intro-ql-java.rst +++ b/docs/language/ql-training/java/intro-ql-java.rst @@ -2,7 +2,7 @@ Introduction to variant analysis ================================ -QL for Java +CodeQL for Java .. container:: semmle-logo @@ -56,14 +56,14 @@ Oops .. note:: - Here’s a simple (artificial) bug, which we’ll develop a QL query to catch. + Here’s a simple (artificial) bug, which we’ll develop a query to catch. This function writes a value to a given location in an array, first trying to do a bounds check to validate that the location is within bounds. However, the return statement has been commented out, leaving a redundant if statement and no bounds checking. This case can act as our “patient zero” in the variant analysis game. -A simple QL query -================= +A simple CodeQL query +===================== .. literalinclude:: ../query-examples/java/empty-if-java.ql :language: ql @@ -72,9 +72,9 @@ A simple QL query We are going to write a simple query which finds “if statements” with empty “then” blocks, so we can highlight the results like those on the previous slide. The query can be run in the `query console on LGTM `__, or in your `IDE `__. - A `QL query `__ consists of a “select” clause that indicates what results should be returned. Typically it will also provide a “from” clause to declare some variables, and a “where” clause to state conditions over those variables. For more information on the structure of query files (including links to useful topics in the `QL language handbook `__), see `Introduction to query files `__. + A `query `__ consists of a “select” clause that indicates what results should be returned. Typically it will also provide a “from” clause to declare some variables, and a “where” clause to state conditions over those variables. For more information on the structure of query files (including links to useful topics in the `QL language handbook `__), see `Introduction to query files `__. - In our example here, the first line of the query imports the `Java standard QL library `__, which defines concepts like ``IfStmt`` and ``Block``. + In our example here, the first line of the query imports the `CodeQL for Java library `__, which defines concepts like ``IfStmt`` and ``Block``. The query proper starts by declaring two variables–ifStmt and block. These variables represent sets of values in the database, according to the type of each of the variables. For example, ``ifStmt`` has the type ``IfStmt``, which means it represents the set of all if statements in the program. If we simply selected these two variables:: @@ -96,8 +96,8 @@ A simple QL query Finally, we select a location, at which to report the problem, and a message, to explain what the problem is. -Structure of a QL query -======================= +Structure of a query +===================== A **query file** has the extension ``.ql`` and contains a **query clause**, and optionally **predicates**, **classes**, and **modules**. @@ -109,14 +109,14 @@ Each query library also implicitly defines a module. .. note:: - QL queries are always contained in query files with the file extension ``.ql``. `Quick queries `__, run in `QL for Eclipse `__, are no exception: the quick query window maintains a temporary QL file in the background. + Queries are always contained in query files with the file extension ``.ql``. `Quick queries `__, run in `QL for Eclipse `__, are no exception: the quick query window maintains a temporary QL file in the background. - Parts of queries can be lifted into `QL library files `__ with the extension ``.qll``. Definitions within such libraries can be brought into scope using “import” statements, and similarly QLL files can import each other’s definitions using “import” statements. + Parts of queries can be lifted into `library files `__ with the extension ``.qll``. Definitions within such libraries can be brought into scope using “import” statements, and similarly QLL files can import each other’s definitions using “import” statements. Logic can be encapsulated as user-defined `predicates `__ and `classes `__, and organized into `modules `__. Each QLL file implicitly defines a module, but QL and QLL files can also contain explicit module definitions, as we will see later. -Predicates in QL -================ +Predicates +========== A predicate allows you to pull out and name parts of a query. @@ -134,15 +134,15 @@ A predicate allows you to pull out and name parts of a query. .. note:: - A `QL predicate `__ takes zero or more parameters, and its body is a condition on those parameters. The predicate may (or may not) hold. Predicates may also be `recursive `__, simply by referring to themselves (directly or indirectly). + A `predicate `__ takes zero or more parameters, and its body is a condition on those parameters. The predicate may (or may not) hold. Predicates may also be `recursive `__, simply by referring to themselves (directly or indirectly). You can imagine a predicate to be a self-contained from-where-select statement, that produces an intermediate relation, or table. In this case, the ``isEmpty`` predicate will be the set of all blocks which are empty. -Classes in QL -============= +Classes +======= -A QL class allows you to name a set of values and define (member) predicates on them. +A class allows you to name a set of values and define (member) predicates on them. A class has at least one supertype and optionally a **characteristic predicate**; it contains the values that belong to *all* supertypes *and* satisfy the characteristic predicate, if provided. @@ -162,7 +162,7 @@ Member predicates are inherited and can be overridden. In the example, declaring a variable “EmptyBlock e” will allow it to range over only those blocks that have zero statements. -Classes in QL continued +Classes in continued ======================= .. container:: column-left @@ -197,7 +197,7 @@ Iterative query refinement .. note:: - QL makes it very easy to experiment with analysis ideas. A common workflow is to start with a simple query (like our “redundant if-statement” example), examine a few results, refine the query based on any patterns that emerge and repeat. + CodeQL makes it very easy to experiment with analysis ideas. A common workflow is to start with a simple query (like our “redundant if-statement” example), examine a few results, refine the query based on any patterns that emerge and repeat. As an exercise, refine the redundant-if query based on the observation that if the if-statement has an “else” clause, then even if the body of the “then” clause is empty, it’s not actually redundant. diff --git a/docs/language/ql-training/java/program-representation-java.rst b/docs/language/ql-training/java/program-representation-java.rst index d090c30aebe..ef078b53562 100644 --- a/docs/language/ql-training/java/program-representation-java.rst +++ b/docs/language/ql-training/java/program-representation-java.rst @@ -2,7 +2,7 @@ Program representation ====================== -QL for Java +CodeQL for Java .. container:: semmle-logo diff --git a/docs/language/ql-training/java/query-injection-java.rst b/docs/language/ql-training/java/query-injection-java.rst index 67f7fe21a76..b3ef7d7e182 100644 --- a/docs/language/ql-training/java/query-injection-java.rst +++ b/docs/language/ql-training/java/query-injection-java.rst @@ -2,7 +2,7 @@ Example: Query injection ======================== -QL for Java +CodeQL for Java .. container:: semmle-logo @@ -65,13 +65,13 @@ SPARQL injection .. rst-class:: background2 -Developing a QL query -====================== +Developing a query +=================== Finding a query concatenation -QL query: find SPARQL methods -============================= +CodeQL query: find SPARQL methods +================================= Let’s start by looking for calls to methods with names of the form ``sparql*Query``, using the classes ``Method`` and ``MethodAccess`` from the Java library. @@ -88,10 +88,10 @@ Let’s start by looking for calls to methods with names of the form ``sparql*Qu - a ``MethodAccess``: the call to a SPARQL query method - a ``Method``: the SPARQL query method. - - The ``where`` part of the query ties these three QL variables together using `predicates `__ defined in the `standard QL for Java library `__. + - The ``where`` part of the query ties these variables together using `predicates `__ defined in the `standard CodeQL for Java library `__. -QL query: find string concatenation -=================================== +CodeQL query: find string concatenation +======================================= - We now need to define what would make these API calls unsafe. - A simple heuristic would be to look for string concatenation used in the query argument. @@ -113,8 +113,8 @@ Looking at autocomplete suggestions, we see that we can get the type of an expre - We therefore write a helper predicate for finding string concatenation. - This predicate effectively represents the set of all ``add`` expressions in the database where the type of the expression is ``TypeString`` - that is, the addition produces a ``String`` value. -QL query: SPARQL injection -========================== +CodeQL query: SPARQL injection +============================== We can now combine our predicate with the existing query. Note that we do not need to specify that the argument of the method access is an ``AddExpr`` - this is implied by the ``isStringConcat`` requirement. diff --git a/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst b/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst index c640aa7a16a..6a26b5c74a4 100644 --- a/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst +++ b/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst @@ -35,9 +35,9 @@ The basic representation of an analyzed program is an *abstract syntax tree (AST .. note:: - When writing queries in QL it is important to have in mind the underlying representation of the program which is stored in the database. Typically queries make use of the “AST” representation of the program - a tree structure where program elements are nested within other program elements. + When writing queries it is important to have in mind the underlying representation of the program which is stored in the database. Typically queries make use of the “AST” representation of the program - a tree structure where program elements are nested within other program elements. - The following topics contain overviews of the important AST classes and QL libraries for C/C++, C#, and Java: + The following topics contain overviews of the important AST classes and CodeQL libraries for C/C++, C#, and Java: - `Introducing the C/C++ libraries `__ - `Introducing the C# libraries `__ @@ -47,19 +47,19 @@ The basic representation of an analyzed program is an *abstract syntax tree (AST Database representations of ASTs ================================ -AST nodes and other program elements are encoded in the database as *entity values*. Entities are implemented as integers, but in QL they are opaque - all one can do with them is to check their equality. +AST nodes and other program elements are encoded in the database as *entity values*. Entities are implemented as integers, but in QL they are opaque---all one can do with them is to check their equality. Each entity belongs to an entity type. Entity types have names starting with “@” and are defined in the database schema (not in QL). Properties of AST nodes and their relationships to each other are encoded by database relations, which are predicates defined in the database (not in QL). -Entity types are rarely used directly, the usual pattern is to define a QL class that extends the type and exposes properties of its entities through member predicates. +Entity types are rarely used directly, the usual pattern is to define a class that extends the type and exposes properties of its entities through member predicates. .. note:: ASTs are a typical example of the kind of data representation one finds in object-oriented programming, with data-carrying nodes that reference each other. At first glance, QL, which can only work with atomic values, does not seem to be well suited for working with this kind of data. However, ultimately all that we require of the nodes in an AST is that they have an identity. The relationships among nodes, usually implemented by reference-valued object fields in other languages, can just as well (and arguably more naturally) be represented as relations over nodes. Attaching data (such as strings or numbers) to nodes can also be represented with relations over nodes and primitive values. All we need is a way for relations to reference nodes. This is achieved in QL (as in other database languages) by means of *entity values* (or entities, for short), which are opaque atomic values, implemented as integers under the hood. - It is the job of the extractor to create entity values for all AST nodes and populate database relations that encode the relationship between AST nodes and any values associated with them. These relations are *extensional*, that is, explicitly stored in the database, unlike the relations described by QL predicates, which we also refer to as *intensional* relations. Entity values belong to *entity types*, whose name starts with “@” to set them apart from primitive types and classes. + It is the job of the extractor to create entity values for all AST nodes and populate database relations that encode the relationship between AST nodes and any values associated with them. These relations are *extensional*, that is, explicitly stored in the database, unlike the relations described by predicates, which we also refer to as *intensional* relations. Entity values belong to *entity types*, whose name starts with “@” to set them apart from primitive types and classes. The interface between entity types and extensional relations on the one hand and QL predicates and classes on the other hand is provided by the *database schema*, which defines the available entity types and the schema of each extensional relation, that is, how many columns the relation has, and which entity type or primitive type the values in each column come from. QL programs can refer to entity types and extensional relations just as they would refer to QL classes and predicates, with the restriction that entity types cannot be directly selected in a ``select`` clause, since they do not have a well-defined string representation. diff --git a/docs/language/ql-training/slide-snippets/info.rst b/docs/language/ql-training/slide-snippets/info.rst deleted file mode 100644 index 583a456207b..00000000000 --- a/docs/language/ql-training/slide-snippets/info.rst +++ /dev/null @@ -1,33 +0,0 @@ -Information -=========== - -To try the examples in this presentation we recommend you download `QL for Eclipse `__. - -**QL language resources** - -- If you are new to QL, try the QL language tutorials at `Learning CodeQL `__. -- To learn more about the main features of QL, try looking at the `QL language handbook `__. -- For further information about writing queries in QL, see `Writing QL queries `__. - -**QL queries** - -The QL queries included in the latest Semmle release are open source. View them in the `semmle/ql repository `__. - -**Extra information** - -.. |arrow-l| unicode:: U+2190 - -.. |arrow-r| unicode:: U+2192 - -- Press |arrow-l| and |arrow-r| to navigate between slides -- Pressing **p** toggles between the slide and any extra notes (where they're available) -- Pressing **f** toggles full screen viewing on/off - -.. note:: - - To run the queries featured in this training presentation, we recommend you download the free-to-use `QL for Eclipse plugin `__. - - This plugin allows you to locally access the latest features of QL, including the standard QL libraries and queries. It also provides standard IDE features such as syntax highlighting, jump-to-definition, and tab completion. - - When you have setup QL for Eclipse we recommend increasing the “Memory for running queries” from the default setting of 4096MB to 8192MB, to ensure that all the queries complete quickly. - \ No newline at end of file diff --git a/docs/language/ql-training/slide-snippets/intro-ql-general.rst b/docs/language/ql-training/slide-snippets/intro-ql-general.rst index f0c00c131fc..6fa68c7243e 100644 --- a/docs/language/ql-training/slide-snippets/intro-ql-general.rst +++ b/docs/language/ql-training/slide-snippets/intro-ql-general.rst @@ -67,9 +67,9 @@ Complete text of the analysis (nothing left out!): .. note:: - Once the mission critical bug was discovered on Curiosity, JPL contacted Semmle for help discovering whether variants of the problem might exist elsewhere in the Curiosity control software. In 20 minutes, research engineers from Semmle produced a QL query and shared it with the JPL team. It finds all functions that are passed an array as an argument whose size is smaller than expected. + Once the mission critical bug was discovered on Curiosity, JPL contacted Semmle for help discovering whether variants of the problem might exist elsewhere in the Curiosity control software. In 20 minutes, research engineers from Semmle produced a CodeQL query and shared it with the JPL team. It finds all functions that are passed an array as an argument whose size is smaller than expected. - (The goal here is not to fully understand the QL, but to illustrate the power of the language and its standard libraries.) + (The goal here is not to fully understand the query, but to illustrate the power of the language and its standard libraries.) Find all instances! @@ -105,16 +105,16 @@ Analysis overview Semmle’s analysis works by extracting a queryable database from your project. For compiled languages, Semmle’s tools observe an ordinary build of the source code. Each time a compiler is invoked to process a source file, a copy of that file is made, and all relevant information about the source code (syntactic data about the abstract syntax tree, semantic data like name binding and type information, data on the operation of the C preprocessor, etc.) is collected. For interpreted languages, the extractor gathers similar information by running directly on the source code. Multi-language code bases are analyzed one language at a time. - Once the extraction finishes, all this information is collected into a single `snapshot database `__, which is then ready to query, possibly on a different machine. A copy of the source files, made at the time the database was created, is also included in the snapshot so analysis results can be displayed at the correct location in the code. The database schema is (source) language specific. + Once the extraction finishes, all this information is collected into a single `CodeQL database `__, which is then ready to query, possibly on a different machine. A copy of the source files, made at the time the database was created, is also included in the snapshot so analysis results can be displayed at the correct location in the code. The database schema is (source) language specific. - Queries are written in `QL `__ and usually depend on one or more of the `standard QL libraries `__ (and of course you can write your own custom libraries). They are compiled into an efficiently executable format by the QL compiler and then run on a snapshot database by the QL evaluator, either on a remote worker machine or locally on a developer’s machine. + Queries are written in `QL `__ and usually depend on one or more of the `standard CodeQL libraries `__ (and of course you can write your own custom libraries). They are compiled into an efficiently executable format by the QL compiler and then run on a CodeQL database by the QL evaluator, either on a remote worker machine or locally on a developer’s machine. Query results can be interpreted and presented in a variety of ways, including displaying them in an `IDE plugin `__ such as QL for Eclipse, or in a web dashboard as on `LGTM `__. Introducing QL ============== -QL is the query language running all Semmle analysis. +QL is the query language running all CodeQL analysis. QL is: @@ -126,12 +126,12 @@ QL is: .. note:: - QL is the high-level, object-oriented logic language that underpins all of Semmle’s libraries and analyses. You can learn lots more about QL by visiting `Introduction to the QL language `__ and `About QL `__. + QL is the high-level, object-oriented logic language that underpins all CodeQL libraries and analyses. You can learn lots more about QL by visiting `Introduction to the QL language `__ and `About QL `__. The key features of QL are: - All common logic connectives are available, including quantifiers like ``exist``, which can also introduce new variables. - The language is declarative–the user focuses on stating what they would like to find, and leaves the details of how to evaluate the query to the engine. - - The object-oriented layer allows Semmle to distribute rich standard libraries for program analysis. These model the common AST node types, control flow and name lookup, and define further layers on top–for example control flow or data flow analysis. The `standard QL libraries and queries `__ ship as source and can be inspected by the user, and new abstractions are readily defined. + - The object-oriented layer allows Semmle to distribute rich standard libraries for program analysis. These model the common AST node types, control flow and name lookup, and define further layers on top–for example control flow or data flow analysis. The `standard CodeQL libraries and queries `__ ship as source and can be inspected by the user, and new abstractions are readily defined. - The database generated by Semmle’s tools is treated as read-only; queries cannot insert new data into it, though they can inspect its contents in various ways. - You can start writing QL and running QL queries on open source projects in the `query console `__ on LGTM.com. You can also download snapshots from LGTM.com to query locally, by `running queries in your IDE `__ equipped with a QL plugin or extension. + You can start writing running queries on open source projects in the `query console `__ on LGTM.com. You can also download snapshots from LGTM.com to query locally, by `running queries in your IDE `__. From 69d7baa9bcfc4ac4ce72459e92bc66436197859c Mon Sep 17 00:00:00 2001 From: james Date: Tue, 5 Nov 2019 15:35:18 +0000 Subject: [PATCH 2/8] docs: update snapshot terminology --- docs/language/ql-training/cpp/bad-overflow-guard.rst | 8 ++++---- docs/language/ql-training/cpp/control-flow-cpp.rst | 6 +++--- docs/language/ql-training/cpp/data-flow-cpp.rst | 6 +++--- docs/language/ql-training/cpp/global-data-flow-cpp.rst | 6 +++--- docs/language/ql-training/cpp/intro-ql-cpp.rst | 6 +++--- .../ql-training/cpp/program-representation-cpp.rst | 4 ++-- docs/language/ql-training/cpp/snprintf.rst | 6 +++--- docs/language/ql-training/java/apache-struts-java.rst | 6 +++--- docs/language/ql-training/java/data-flow-java.rst | 6 +++--- docs/language/ql-training/java/global-data-flow-java.rst | 6 +++--- docs/language/ql-training/java/intro-ql-java.rst | 6 +++--- docs/language/ql-training/java/query-injection-java.rst | 6 +++--- .../ql-training/slide-snippets/abstract-syntax-tree.rst | 2 +- .../{snapshot-note.rst => database-note.rst} | 2 +- .../ql-training/slide-snippets/global-data-flow.rst | 6 +++--- .../ql-training/slide-snippets/intro-ql-general.rst | 6 +++--- .../ql-training/slide-snippets/local-data-flow.rst | 4 ++-- 17 files changed, 46 insertions(+), 46 deletions(-) rename docs/language/ql-training/slide-snippets/{snapshot-note.rst => database-note.rst} (66%) diff --git a/docs/language/ql-training/cpp/bad-overflow-guard.rst b/docs/language/ql-training/cpp/bad-overflow-guard.rst index 5e9fbcfc700..6937e9a1a4f 100644 --- a/docs/language/ql-training/cpp/bad-overflow-guard.rst +++ b/docs/language/ql-training/cpp/bad-overflow-guard.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `ChakraCore snapshot `__ +- `ChakraCore database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides @@ -233,4 +233,4 @@ The final query .. literalinclude:: ../query-examples/cpp/bad-overflow-guard-3.ql :language: ql -This query finds a single result in our historic snapshot, which was `a genuine bug in ChakraCore `__. +This query finds a single result in our historic database, which was `a genuine bug in ChakraCore `__. diff --git a/docs/language/ql-training/cpp/control-flow-cpp.rst b/docs/language/ql-training/cpp/control-flow-cpp.rst index a8171a44b5d..4e6ebbf4292 100644 --- a/docs/language/ql-training/cpp/control-flow-cpp.rst +++ b/docs/language/ql-training/cpp/control-flow-cpp.rst @@ -18,7 +18,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `ChakraCore snapshot `__ +- `ChakraCore database `__ .. note:: @@ -26,9 +26,9 @@ For this example you should download: You can query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/cpp/data-flow-cpp.rst b/docs/language/ql-training/cpp/data-flow-cpp.rst index 5377d8d52d1..cb7ed7b5133 100644 --- a/docs/language/ql-training/cpp/data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/data-flow-cpp.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `dotnet/coreclr snapshot `__ +- `dotnet/coreclr database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/cpp/global-data-flow-cpp.rst b/docs/language/ql-training/cpp/global-data-flow-cpp.rst index bc048ac59f4..5a5e93aafda 100644 --- a/docs/language/ql-training/cpp/global-data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/global-data-flow-cpp.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `dotnet/coreclr snapshot `__ +- `dotnet/coreclr database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/cpp/intro-ql-cpp.rst b/docs/language/ql-training/cpp/intro-ql-cpp.rst index dddc86ca77f..e7d4b318b25 100644 --- a/docs/language/ql-training/cpp/intro-ql-cpp.rst +++ b/docs/language/ql-training/cpp/intro-ql-cpp.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `exiv2 snapshot `__ +- `exiv2 database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/cpp/program-representation-cpp.rst b/docs/language/ql-training/cpp/program-representation-cpp.rst index b21677fccab..077b36520f6 100644 --- a/docs/language/ql-training/cpp/program-representation-cpp.rst +++ b/docs/language/ql-training/cpp/program-representation-cpp.rst @@ -107,7 +107,7 @@ Working with macros #define square(x) x*x y = square(y0), z = square(z0) -is represented in the snapshot database as: +is represented in the CodeQL database database as: - A Macro entity representing the text of the *head* and *body* of the macro - Assignment nodes, representing the two assignments after preprocessing @@ -121,4 +121,4 @@ Useful predicates on ``Element``: ``isInMacroExpansion()``, ``isAffectedByMacro( .. note:: - The snapshot also contains information about macro definitions, which are represented by class ``Macro``. These macro definitions are related to the AST nodes resulting from their uses by the class ``MacroAccess``. \ No newline at end of file + The CodeQL database also contains information about macro definitions, which are represented by class ``Macro``. These macro definitions are related to the AST nodes resulting from their uses by the class ``MacroAccess``. \ No newline at end of file diff --git a/docs/language/ql-training/cpp/snprintf.rst b/docs/language/ql-training/cpp/snprintf.rst index dafebc4a720..26c54708c0a 100644 --- a/docs/language/ql-training/cpp/snprintf.rst +++ b/docs/language/ql-training/cpp/snprintf.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `rsyslog snapshot `__ +- `rsyslog database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/java/apache-struts-java.rst b/docs/language/ql-training/java/apache-struts-java.rst index 5c27ec73979..73d0122b7ea 100644 --- a/docs/language/ql-training/java/apache-struts-java.rst +++ b/docs/language/ql-training/java/apache-struts-java.rst @@ -20,7 +20,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `Apache Struts snapshot `__ +- `Apache Struts database `__ .. note:: @@ -28,9 +28,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/java/data-flow-java.rst b/docs/language/ql-training/java/data-flow-java.rst index be9ba98456e..0b9026aadfb 100644 --- a/docs/language/ql-training/java/data-flow-java.rst +++ b/docs/language/ql-training/java/data-flow-java.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `VIVO Vitro snapshot `__ +- `VIVO Vitro database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/java/global-data-flow-java.rst b/docs/language/ql-training/java/global-data-flow-java.rst index f43c31e0f4f..062ce30b04c 100644 --- a/docs/language/ql-training/java/global-data-flow-java.rst +++ b/docs/language/ql-training/java/global-data-flow-java.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `Apache Struts snapshot `__ +- `Apache Struts database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/java/intro-ql-java.rst b/docs/language/ql-training/java/intro-ql-java.rst index f80f878492d..393e35cf5a8 100644 --- a/docs/language/ql-training/java/intro-ql-java.rst +++ b/docs/language/ql-training/java/intro-ql-java.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `Apache Struts snapshot `__ +- `Apache Struts database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/java/query-injection-java.rst b/docs/language/ql-training/java/query-injection-java.rst index b3ef7d7e182..dc345d3b39c 100644 --- a/docs/language/ql-training/java/query-injection-java.rst +++ b/docs/language/ql-training/java/query-injection-java.rst @@ -16,7 +16,7 @@ Setup For this example you should download: - `QL for Eclipse `__ -- `VIVO Vitro snapshot `__ +- `VIVO Vitro database `__ .. note:: @@ -24,9 +24,9 @@ For this example you should download: You can also query the project in `the query console `__ on LGTM.com. - .. insert snapshot-note.rst to explain differences between snapshot available to download and the version available in the query console. + .. insert database-note.rst to explain differences between database available to download and the version available in the query console. - .. include:: ../slide-snippets/snapshot-note.rst + .. include:: ../slide-snippets/database-note.rst .. resume slides diff --git a/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst b/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst index 6a26b5c74a4..aaa2cd23d71 100644 --- a/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst +++ b/docs/language/ql-training/slide-snippets/abstract-syntax-tree.rst @@ -63,7 +63,7 @@ Entity types are rarely used directly, the usual pattern is to define a class th The interface between entity types and extensional relations on the one hand and QL predicates and classes on the other hand is provided by the *database schema*, which defines the available entity types and the schema of each extensional relation, that is, how many columns the relation has, and which entity type or primitive type the values in each column come from. QL programs can refer to entity types and extensional relations just as they would refer to QL classes and predicates, with the restriction that entity types cannot be directly selected in a ``select`` clause, since they do not have a well-defined string representation. - For example, the database schemas for C/++, C#, and Java snapshot databases are here: + For example, the database schemas for C/++, C#, and Java CodeQL databases are here: - https://github.com/Semmle/ql/blob/master/cpp/ql/src/semmlecode.cpp.dbscheme - https://github.com/Semmle/ql/blob/master/csharp/ql/src/semmlecode.csharp.dbscheme diff --git a/docs/language/ql-training/slide-snippets/snapshot-note.rst b/docs/language/ql-training/slide-snippets/database-note.rst similarity index 66% rename from docs/language/ql-training/slide-snippets/snapshot-note.rst rename to docs/language/ql-training/slide-snippets/database-note.rst index 4a32243211b..af6dce23728 100644 --- a/docs/language/ql-training/slide-snippets/snapshot-note.rst +++ b/docs/language/ql-training/slide-snippets/database-note.rst @@ -1 +1 @@ -Note that results generated in the query console are likely to differ to those generated in the QL plugin as LGTM.com analyzes the most recent revisions of each project that has been added–the snapshot available to download above is based on an historical version of the codebase. \ No newline at end of file +Note that results generated in the query console are likely to differ to those generated in the QL plugin as LGTM.com analyzes the most recent revisions of each project that has been added–the CodeQL database available to download above is based on an historical version of the codebase. \ No newline at end of file diff --git a/docs/language/ql-training/slide-snippets/global-data-flow.rst b/docs/language/ql-training/slide-snippets/global-data-flow.rst index f18dc22d178..6cb52af0f84 100644 --- a/docs/language/ql-training/slide-snippets/global-data-flow.rst +++ b/docs/language/ql-training/slide-snippets/global-data-flow.rst @@ -17,15 +17,15 @@ Global data flow and taint tracking - Recap: - - Local (“intra-procedural”) data flow models flow within one function; feasible to compute for all functions in a snapshot - - Global (“inter-procedural”) data flow models flow across function calls; not feasible to compute for all functions in a snapshot + - Local (“intra-procedural”) data flow models flow within one function; feasible to compute for all functions in a CodeQL database + - Global (“inter-procedural”) data flow models flow across function calls; not feasible to compute for all functions in a CodeQL database - For global data flow (and taint tracking), we must therefore provide restrictions to ensure the problem is tractable. - Typically, this involves specifying the *source* and *sink*. .. note:: - As we mentioned in the previous slide deck, while local data flow is feasible to compute for all functions in a snapshot, global data flow is not. This is because the number of paths becomes exponentially larger for global data flow. + As we mentioned in the previous slide deck, while local data flow is feasible to compute for all functions in a CodeQL database, global data flow is not. This is because the number of paths becomes exponentially larger for global data flow. The global data flow (and taint tracking) avoids this problem by requiring that the query author specifies which ``sources`` and ``sinks`` are applicable. This allows the implementation to compute paths between the restricted set of nodes, rather than the full graph. diff --git a/docs/language/ql-training/slide-snippets/intro-ql-general.rst b/docs/language/ql-training/slide-snippets/intro-ql-general.rst index 6fa68c7243e..d559c5bbeb7 100644 --- a/docs/language/ql-training/slide-snippets/intro-ql-general.rst +++ b/docs/language/ql-training/slide-snippets/intro-ql-general.rst @@ -105,7 +105,7 @@ Analysis overview Semmle’s analysis works by extracting a queryable database from your project. For compiled languages, Semmle’s tools observe an ordinary build of the source code. Each time a compiler is invoked to process a source file, a copy of that file is made, and all relevant information about the source code (syntactic data about the abstract syntax tree, semantic data like name binding and type information, data on the operation of the C preprocessor, etc.) is collected. For interpreted languages, the extractor gathers similar information by running directly on the source code. Multi-language code bases are analyzed one language at a time. - Once the extraction finishes, all this information is collected into a single `CodeQL database `__, which is then ready to query, possibly on a different machine. A copy of the source files, made at the time the database was created, is also included in the snapshot so analysis results can be displayed at the correct location in the code. The database schema is (source) language specific. + Once the extraction finishes, all this information is collected into a single `CodeQL database `__, which is then ready to query, possibly on a different machine. A copy of the source files, made at the time the database was created, is also included in the CodeQL database so analysis results can be displayed at the correct location in the code. The database schema is (source) language specific. Queries are written in `QL `__ and usually depend on one or more of the `standard CodeQL libraries `__ (and of course you can write your own custom libraries). They are compiled into an efficiently executable format by the QL compiler and then run on a CodeQL database by the QL evaluator, either on a remote worker machine or locally on a developer’s machine. @@ -121,7 +121,7 @@ QL is: - a **logic** language based on first-order logic - a **declarative** language without side effects - an **object-oriented** language -- a **query** language working on a read-only snapshot database +- a **query** language working on a read-only CodeQL database database - equipped with rich standard libraries **for program analysis** .. note:: @@ -134,4 +134,4 @@ QL is: - The object-oriented layer allows Semmle to distribute rich standard libraries for program analysis. These model the common AST node types, control flow and name lookup, and define further layers on top–for example control flow or data flow analysis. The `standard CodeQL libraries and queries `__ ship as source and can be inspected by the user, and new abstractions are readily defined. - The database generated by Semmle’s tools is treated as read-only; queries cannot insert new data into it, though they can inspect its contents in various ways. - You can start writing running queries on open source projects in the `query console `__ on LGTM.com. You can also download snapshots from LGTM.com to query locally, by `running queries in your IDE `__. + You can start writing running queries on open source projects in the `query console `__ on LGTM.com. You can also download CodeQL databases from LGTM.com to query locally, by `running queries in your IDE `__. diff --git a/docs/language/ql-training/slide-snippets/local-data-flow.rst b/docs/language/ql-training/slide-snippets/local-data-flow.rst index 8a0eeac6374..068dfc9170a 100644 --- a/docs/language/ql-training/slide-snippets/local-data-flow.rst +++ b/docs/language/ql-training/slide-snippets/local-data-flow.rst @@ -61,8 +61,8 @@ Data flow graphs Local vs global data flow ========================= -- Local (“intra-procedural”) data flow models flow within one function; feasible to compute for all functions in a snapshot -- Global (“inter-procedural”) data flow models flow across function calls; not feasible to compute for all functions in a snapshot +- Local (“intra-procedural”) data flow models flow within one function; feasible to compute for all functions in a CodeQL database +- Global (“inter-procedural”) data flow models flow across function calls; not feasible to compute for all functions in a CodeQL database - Different APIs, so discussed separately - This slide deck focuses on the former From 2d00ca5773b7773184b3beabbd3799876d3fc06b Mon Sep 17 00:00:00 2001 From: james Date: Fri, 4 Oct 2019 09:18:19 +0100 Subject: [PATCH 3/8] docs: semmle logo (cherry picked from commit 4a8e8fa0de4470aaeed94357f70b54b68c0b84a7) --- .../static/theme/css/default.css | 18 +-- .../_static-training/title-slide.svg | 149 +----------------- .../ql-training/cpp/bad-overflow-guard.rst | 4 - .../ql-training/cpp/control-flow-cpp.rst | 4 - .../ql-training/cpp/data-flow-cpp.rst | 4 - .../ql-training/cpp/global-data-flow-cpp.rst | 4 - .../language/ql-training/cpp/intro-ql-cpp.rst | 4 - .../cpp/program-representation-cpp.rst | 4 - docs/language/ql-training/cpp/snprintf.rst | 4 - docs/language/ql-training/index.rst | 4 - .../ql-training/java/apache-struts-java.rst | 4 - .../ql-training/java/data-flow-java.rst | 4 - .../java/global-data-flow-java.rst | 4 - .../ql-training/java/intro-ql-java.rst | 4 - .../java/program-representation-java.rst | 4 - .../ql-training/java/query-injection-java.rst | 4 - docs/language/ql-training/template.rst | 4 - 17 files changed, 4 insertions(+), 223 deletions(-) diff --git a/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css b/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css index 2c05ea5e93b..4f706120747 100644 --- a/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css +++ b/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css @@ -1300,13 +1300,13 @@ aside.gdbar img { .title-slide hgroup h1 { font-size: 2em; line-height: 1.4; - /*letter-spacing: -3px;*/ color: white; margin: auto; display: block; position: absolute; top: 0; bottom: 10%; + left: 1.25em; height: 0; } /* line 898, ../scss/default.scss */ @@ -1430,31 +1430,19 @@ hgroup .pre { color: #5c31ff; } -/* title slide (deck title, subtitle, semmle logo)*/ +/* title slide (deck title, subtitle)*/ .title-slide { background-image: url("../../title-slide.svg"); background-size: cover; } -.semmle-logo sup { - vertical-align: super; - font-size: 0.3em; - font-weight: 100; -} - -.title-slide .semmle-logo { - color: white; - font-size: 1.2em; - position: absolute; - top: 10%; -} - .title-slide p { color: white; font-size: 1em; position: absolute; bottom: 30%; + left: 2.6em; } .title-slide hgroup .pre { diff --git a/docs/language/ql-training/_static-training/title-slide.svg b/docs/language/ql-training/_static-training/title-slide.svg index 6f9a19f4a1b..13eb2d34fef 100644 --- a/docs/language/ql-training/_static-training/title-slide.svg +++ b/docs/language/ql-training/_static-training/title-slide.svg @@ -1,148 +1 @@ - - - - - - image/svg+xml - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + \ No newline at end of file diff --git a/docs/language/ql-training/cpp/bad-overflow-guard.rst b/docs/language/ql-training/cpp/bad-overflow-guard.rst index 6937e9a1a4f..dceeec3320f 100644 --- a/docs/language/ql-training/cpp/bad-overflow-guard.rst +++ b/docs/language/ql-training/cpp/bad-overflow-guard.rst @@ -4,10 +4,6 @@ Example: Bad overflow guard CodeQL for C/C++ -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/cpp/control-flow-cpp.rst b/docs/language/ql-training/cpp/control-flow-cpp.rst index 4e6ebbf4292..3fddc3f8072 100644 --- a/docs/language/ql-training/cpp/control-flow-cpp.rst +++ b/docs/language/ql-training/cpp/control-flow-cpp.rst @@ -4,10 +4,6 @@ Analyzing control flow CodeQL for C/C++ -.. container:: semmle-logo - - Semmle :sup:`TM` - .. Include information slides here .. rst-class:: setup diff --git a/docs/language/ql-training/cpp/data-flow-cpp.rst b/docs/language/ql-training/cpp/data-flow-cpp.rst index cb7ed7b5133..e78e8e86220 100644 --- a/docs/language/ql-training/cpp/data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/data-flow-cpp.rst @@ -4,10 +4,6 @@ Introduction to data flow Finding string formatting vulnerabilities in C/C++ -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/cpp/global-data-flow-cpp.rst b/docs/language/ql-training/cpp/global-data-flow-cpp.rst index 5a5e93aafda..aaa567e7a8e 100644 --- a/docs/language/ql-training/cpp/global-data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/global-data-flow-cpp.rst @@ -3,10 +3,6 @@ Introduction to global data flow ================================ CodeQL for C/C++ - -.. container:: semmle-logo - - Semmle :sup:`TM` .. rst-class:: setup diff --git a/docs/language/ql-training/cpp/intro-ql-cpp.rst b/docs/language/ql-training/cpp/intro-ql-cpp.rst index e7d4b318b25..cbe29e6c520 100644 --- a/docs/language/ql-training/cpp/intro-ql-cpp.rst +++ b/docs/language/ql-training/cpp/intro-ql-cpp.rst @@ -4,10 +4,6 @@ Introduction to variant analysis CodeQL for C/C++ -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/cpp/program-representation-cpp.rst b/docs/language/ql-training/cpp/program-representation-cpp.rst index 077b36520f6..3c062089428 100644 --- a/docs/language/ql-training/cpp/program-representation-cpp.rst +++ b/docs/language/ql-training/cpp/program-representation-cpp.rst @@ -4,10 +4,6 @@ Program representation CodeQL for C/C++ -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: agenda Agenda diff --git a/docs/language/ql-training/cpp/snprintf.rst b/docs/language/ql-training/cpp/snprintf.rst index 26c54708c0a..c15bf74d286 100644 --- a/docs/language/ql-training/cpp/snprintf.rst +++ b/docs/language/ql-training/cpp/snprintf.rst @@ -4,10 +4,6 @@ Exercise: ``snprintf`` overflow CodeQL for C/C++ -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/index.rst b/docs/language/ql-training/index.rst index 309fe92c052..021bb58d3c4 100644 --- a/docs/language/ql-training/index.rst +++ b/docs/language/ql-training/index.rst @@ -1,10 +1,6 @@ CodeQL training and variant analysis examples ============================================= -.. container:: semmle-logo - - Semmle :sup:`TM` - .. toctree:: :glob: :maxdepth: 1 diff --git a/docs/language/ql-training/java/apache-struts-java.rst b/docs/language/ql-training/java/apache-struts-java.rst index 73d0122b7ea..be071d41562 100644 --- a/docs/language/ql-training/java/apache-struts-java.rst +++ b/docs/language/ql-training/java/apache-struts-java.rst @@ -8,10 +8,6 @@ Exercise: Apache Struts CVE-2017-9805 -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/java/data-flow-java.rst b/docs/language/ql-training/java/data-flow-java.rst index 0b9026aadfb..93ccc87b7d1 100644 --- a/docs/language/ql-training/java/data-flow-java.rst +++ b/docs/language/ql-training/java/data-flow-java.rst @@ -2,10 +2,6 @@ Introduction to data flow ========================= -.. container:: semmle-logo - - Semmle :sup:`TM` - Finding SPARQL injection vulnerabilities in Java .. rst-class:: setup diff --git a/docs/language/ql-training/java/global-data-flow-java.rst b/docs/language/ql-training/java/global-data-flow-java.rst index 062ce30b04c..9edacde93c3 100644 --- a/docs/language/ql-training/java/global-data-flow-java.rst +++ b/docs/language/ql-training/java/global-data-flow-java.rst @@ -4,10 +4,6 @@ Introduction to global data flow CodeQL for Java -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/java/intro-ql-java.rst b/docs/language/ql-training/java/intro-ql-java.rst index 393e35cf5a8..f7afe713c4a 100644 --- a/docs/language/ql-training/java/intro-ql-java.rst +++ b/docs/language/ql-training/java/intro-ql-java.rst @@ -4,10 +4,6 @@ Introduction to variant analysis CodeQL for Java -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/java/program-representation-java.rst b/docs/language/ql-training/java/program-representation-java.rst index ef078b53562..ba9b1a9d80d 100644 --- a/docs/language/ql-training/java/program-representation-java.rst +++ b/docs/language/ql-training/java/program-representation-java.rst @@ -4,10 +4,6 @@ Program representation CodeQL for Java -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: agenda Agenda diff --git a/docs/language/ql-training/java/query-injection-java.rst b/docs/language/ql-training/java/query-injection-java.rst index dc345d3b39c..a5b46eabda5 100644 --- a/docs/language/ql-training/java/query-injection-java.rst +++ b/docs/language/ql-training/java/query-injection-java.rst @@ -4,10 +4,6 @@ Example: Query injection CodeQL for Java -.. container:: semmle-logo - - Semmle :sup:`TM` - .. rst-class:: setup Setup diff --git a/docs/language/ql-training/template.rst b/docs/language/ql-training/template.rst index 0cce4a11435..21c5abef144 100644 --- a/docs/language/ql-training/template.rst +++ b/docs/language/ql-training/template.rst @@ -27,10 +27,6 @@ Template slide deck Second subheading -.. container:: semmle-logo - - Semmle :sup:`TM` - .. Set up slide. Include link to QL4E snapshots required for examples .. rst-class:: setup From 0b6592f650421c5b5ff001674042577e038367bb Mon Sep 17 00:00:00 2001 From: james Date: Mon, 30 Sep 2019 11:51:17 +0100 Subject: [PATCH 4/8] docs: version number and small css tweaks (cherry picked from commit ff78feeeeaec99bdf779242c33e25f57c7bcab29) --- .../slides-semmle-2/static/theme/css/default.css | 6 ++++-- docs/language/ql-training/conf.py | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css b/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css index 4f706120747..3f8a89c6555 100644 --- a/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css +++ b/docs/language/ql-training/_static-training/slides-semmle-2/static/theme/css/default.css @@ -485,6 +485,7 @@ ul { margin-left: 2.2em; margin-bottom: 1em; position: relative; + width: 90%; } /* line 300, ../scss/default.scss */ ul li { @@ -1452,6 +1453,7 @@ hgroup .pre { .subheading { position: absolute; top: 62.5%; + left: 0; } .subheading p { @@ -1557,7 +1559,7 @@ p.first.admonition-title { text-align: left; font-size: 0.8em; width: 100%; - overflow: scroll; + overflow: auto; border: 1px solid black; } @@ -1596,7 +1598,7 @@ p.first.admonition-title { display: block; position: fixed; top: 0; - right: -1%; + right: 0; font-size: 1.2em; } diff --git a/docs/language/ql-training/conf.py b/docs/language/ql-training/conf.py index de712cd06a4..8f8a13569b5 100644 --- a/docs/language/ql-training/conf.py +++ b/docs/language/ql-training/conf.py @@ -86,9 +86,9 @@ htmlhelp_basename = 'QL training' # built documents. # # The short X.Y version. -version = u'1.21' +version = u'1.22' # The full version, including alpha/beta/rc tags. -release = u'1.21' +release = u'1.22' copyright = u'2019 Semmle Ltd' author = u'Semmle Ltd' From 488ce158887ed174a854730d81dd353c962a5c1d Mon Sep 17 00:00:00 2001 From: James Fletcher <42464962+jf205@users.noreply.github.com> Date: Tue, 5 Nov 2019 19:57:24 +0000 Subject: [PATCH 5/8] Apply suggestions from code review Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com> Co-Authored-By: Felicity Chapman --- docs/language/learn-ql/ql-training.rst | 12 ++++++------ docs/language/ql-training/cpp/data-flow-cpp.rst | 4 ++-- docs/language/ql-training/cpp/intro-ql-cpp.rst | 2 +- .../ql-training/cpp/program-representation-cpp.rst | 6 +++--- docs/language/ql-training/java/intro-ql-java.rst | 2 +- .../ql-training/slide-snippets/intro-ql-general.rst | 2 +- 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/language/learn-ql/ql-training.rst b/docs/language/learn-ql/ql-training.rst index 757de7daed6..4187ae621b5 100644 --- a/docs/language/learn-ql/ql-training.rst +++ b/docs/language/learn-ql/ql-training.rst @@ -1,8 +1,8 @@ CodeQL training and variant analysis examples ============================================= -QL and variant analysis ------------------------ +CodeQL and variant analysis +--------------------------- `Variant analysis `__ is the process of using a known vulnerability as a seed to find similar problems in your code. Security engineers typically perform variant analysis to identify possible vulnerabilities and to ensure that these threats are properly fixed across multiple code bases. @@ -10,8 +10,8 @@ QL and variant analysis CodeQL is easy to learn, and exploring code using CodeQL is the most efficient way to perform variant analysis. -Learning QL for variant analysis --------------------------------- +Learning CodeQL for variant analysis +------------------------------------ Start learning how to use CodeQL in variant analysis for a specific language by looking at the topics below. Each topic links to a short presentation on CodeQL, its libraries, or an example variant discovered using CodeQL. @@ -45,7 +45,7 @@ CodeQL and variant analysis for C/C++ - `Introduction to local data flow <../ql-training/cpp/data-flow-cpp.html>`__–an introduction to analyzing local data flow in C/C++ using CodeQL, including an example demonstrating how to develop a query to find a real CVE. - `Exercise: snprintf overflow <../ql-training/cpp/snprintf.html>`__–an example demonstrating how to develop a data flow query. - `Introduction to global data flow <../ql-training/cpp/global-data-flow-cpp.html>`__–an introduction to analyzing global data flow in C/C++ using CodeQL. -- `Analyzing control flow: CodeQL for C/C++ <../ql-training/cpp/control-flow-cpp.html>`__–an introduction to analyzing control flow in C/C++ using QL. +- `Analyzing control flow: CodeQL for C/C++ <../ql-training/cpp/control-flow-cpp.html>`__–an introduction to analyzing control flow in C/C++ using CodeQL. CodeQL and variant analysis for Java ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -61,6 +61,6 @@ More resources ~~~~~~~~~~~~~~ - If you are completely new to CodeQL, look at our introductory topics in :doc:`Learning CodeQL `. -- To find more detailed information about how to write CodeQL queries for specific languages, visit the links in :ref:`Writing CodeQL queries `. +- To find more detailed information about how to write queries for specific languages, visit the links in :ref:`Writing CodeQL queries `. - To read more about how CodeQL queries have been used in Semmle's security research, and to read about new CodeQL developments, visit the `Semmle blog `__. - Find more examples of queries written by Semmle's own security researchers in the `Semmle Demos repository `__ on GitHub. diff --git a/docs/language/ql-training/cpp/data-flow-cpp.rst b/docs/language/ql-training/cpp/data-flow-cpp.rst index e78e8e86220..32a3dfa233b 100644 --- a/docs/language/ql-training/cpp/data-flow-cpp.rst +++ b/docs/language/ql-training/cpp/data-flow-cpp.rst @@ -82,9 +82,9 @@ Write a query that flags ``printf`` calls where the format argument is not a ``S .. note:: - This first query is about finding places where the format specifier is not a constant string. In CodeQL for C/C++, constant strings are modeled as ``StringLiteral`` nodes, so we are looking for calls to format functions where the format specifier argument is not a string literal. + This first query is about finding places where the format specifier is not a constant string. In the CodeQL libraries for C/C++, constant strings are modeled as ``StringLiteral`` nodes, so we are looking for calls to format functions where the format specifier argument is not a string literal. - The `C/C++ standard libraries `__ include many different formatting functions that may be vulnerable to this particular attack–including ``printf``, ``snprintf``, and others. Furthermore, each of these different formatting functions may include the format string in a different position in the argument list. Instead of laboriously listing all these different variants, we can make use of the CodeQL for C/C++ standard library class ``FormattingFunction``, which provides an interface that models common formatting functions in C/C++. + The `C/C++ standard libraries `__ include many different formatting functions that may be vulnerable to this particular attack–including ``printf``, ``snprintf``, and others. Furthermore, each of these different formatting functions may include the format string in a different position in the argument list. Instead of laboriously listing all these different variants, we can make use of the standard CodeQL class ``FormattingFunction``, which provides an interface that models common formatting functions in C/C++. Meh... ====== diff --git a/docs/language/ql-training/cpp/intro-ql-cpp.rst b/docs/language/ql-training/cpp/intro-ql-cpp.rst index cbe29e6c520..9cfb1d3b442 100644 --- a/docs/language/ql-training/cpp/intro-ql-cpp.rst +++ b/docs/language/ql-training/cpp/intro-ql-cpp.rst @@ -108,7 +108,7 @@ Each query library also implicitly defines a module. Queries are always contained in query files with the file extension ``.ql``. `Quick queries `__, run in `QL for Eclipse `__, are no exception: the quick query window maintains a temporary query file in the background. - Parts of queries can be lifted into `library files `__ with the extension ``qll``. Definitions within such libraries can be brought into scope using ``import`` statements, and similarly QLL files can import each other’s definitions using “import” statements. + Parts of queries can be lifted into `library files `__ with the extension ``.qll``. Definitions within such libraries can be brought into scope using ``import`` statements, and similarly QLL files can import each other’s definitions using “import” statements. Logic can be encapsulated as user-defined `predicates `__ and `classes `__, and organized into `modules `__. Each QLL file implicitly defines a module, but QL and QLL files can also contain explicit module definitions, as we will see later. diff --git a/docs/language/ql-training/cpp/program-representation-cpp.rst b/docs/language/ql-training/cpp/program-representation-cpp.rst index 3c062089428..156cb0a4a68 100644 --- a/docs/language/ql-training/cpp/program-representation-cpp.rst +++ b/docs/language/ql-training/cpp/program-representation-cpp.rst @@ -64,7 +64,7 @@ Working with functions Functions are represented by the Function class. Each declaration or definition of a function is represented by a ``FunctionDeclarationEntry``. -Calls to functions are modeled by class Call and its subclasses: +Calls to functions are modeled by class ``Call`` and its subclasses: - ``Call.getTarget()`` gets the declared target of the call; undefined for calls through function pointers - ``Function.getACallToThisFunction()`` gets a call to this function @@ -103,7 +103,7 @@ Working with macros #define square(x) x*x y = square(y0), z = square(z0) -is represented in the CodeQL database database as: +is represented in the CodeQL database as: - A Macro entity representing the text of the *head* and *body* of the macro - Assignment nodes, representing the two assignments after preprocessing @@ -117,4 +117,4 @@ Useful predicates on ``Element``: ``isInMacroExpansion()``, ``isAffectedByMacro( .. note:: - The CodeQL database also contains information about macro definitions, which are represented by class ``Macro``. These macro definitions are related to the AST nodes resulting from their uses by the class ``MacroAccess``. \ No newline at end of file + The CodeQL database also contains information about macro definitions, which are represented by class ``Macro``. These macro definitions are related to the AST nodes resulting from their uses by the class ``MacroAccess``. diff --git a/docs/language/ql-training/java/intro-ql-java.rst b/docs/language/ql-training/java/intro-ql-java.rst index f7afe713c4a..46106204ed4 100644 --- a/docs/language/ql-training/java/intro-ql-java.rst +++ b/docs/language/ql-training/java/intro-ql-java.rst @@ -158,7 +158,7 @@ Member predicates are inherited and can be overridden. In the example, declaring a variable “EmptyBlock e” will allow it to range over only those blocks that have zero statements. -Classes in continued +Classes continued ======================= .. container:: column-left diff --git a/docs/language/ql-training/slide-snippets/intro-ql-general.rst b/docs/language/ql-training/slide-snippets/intro-ql-general.rst index d559c5bbeb7..ae0c0cdabb5 100644 --- a/docs/language/ql-training/slide-snippets/intro-ql-general.rst +++ b/docs/language/ql-training/slide-snippets/intro-ql-general.rst @@ -121,7 +121,7 @@ QL is: - a **logic** language based on first-order logic - a **declarative** language without side effects - an **object-oriented** language -- a **query** language working on a read-only CodeQL database database +- a **query** language working on a read-only CodeQL database - equipped with rich standard libraries **for program analysis** .. note:: From 8d02a740dda1b8ed961249b4e40d7e3dfd0c7409 Mon Sep 17 00:00:00 2001 From: james Date: Tue, 5 Nov 2019 20:03:47 +0000 Subject: [PATCH 6/8] docs: address remaining comments --- docs/language/ql-training/conf.py | 8 ++++---- .../ql-training/slide-snippets/local-data-flow.rst | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/language/ql-training/conf.py b/docs/language/ql-training/conf.py index 8f8a13569b5..117f515a54f 100644 --- a/docs/language/ql-training/conf.py +++ b/docs/language/ql-training/conf.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # -# QL training slides build configuration file +# CodeQL training slides build configuration file # # This file is execfile()d with the current directory set to its # containing dir. @@ -59,7 +59,7 @@ highlight_language = 'ql' master_doc = 'index' # General information about the project. -project = u'QL training and variant analysis examples' +project = u'CodeQL training and variant analysis examples' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, @@ -77,10 +77,10 @@ slide_theme_path = ["_static-training/"] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". -html_title = 'QL training and variant analysis examples' +html_title = 'CodeQL training and variant analysis examples' # Output file base name for HTML help builder. -htmlhelp_basename = 'QL training' +htmlhelp_basename = 'CodeQL training' # The Semmle version info for the current release you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. diff --git a/docs/language/ql-training/slide-snippets/local-data-flow.rst b/docs/language/ql-training/slide-snippets/local-data-flow.rst index 068dfc9170a..85b851a9ce0 100644 --- a/docs/language/ql-training/slide-snippets/local-data-flow.rst +++ b/docs/language/ql-training/slide-snippets/local-data-flow.rst @@ -70,7 +70,7 @@ Local vs global data flow For further information, see: - - `Introduction to data flow analysis in QL `__ + - `Introduction to data flow analysis in CodeQL `__ .. rst-class:: background2 From 24615f2bdbb73398e129cc28fa8a95326c5e53c8 Mon Sep 17 00:00:00 2001 From: james Date: Wed, 6 Nov 2019 12:37:20 +0000 Subject: [PATCH 7/8] docs: further review comments --- docs/language/ql-training/cpp/control-flow-cpp.rst | 2 +- docs/language/ql-training/cpp/intro-ql-cpp.rst | 12 ++++++------ .../ql-training/cpp/program-representation-cpp.rst | 10 +++++----- docs/language/ql-training/java/intro-ql-java.rst | 10 +++++----- .../ql-training/java/program-representation-java.rst | 12 ++++++------ .../ql-training/java/query-injection-java.rst | 6 +++--- .../ql-training/slide-snippets/local-data-flow.rst | 2 +- 7 files changed, 27 insertions(+), 27 deletions(-) diff --git a/docs/language/ql-training/cpp/control-flow-cpp.rst b/docs/language/ql-training/cpp/control-flow-cpp.rst index 3fddc3f8072..f72633b714f 100644 --- a/docs/language/ql-training/cpp/control-flow-cpp.rst +++ b/docs/language/ql-training/cpp/control-flow-cpp.rst @@ -89,7 +89,7 @@ Control flow graphs Modeling control flow ===================== -The control flow is modeled with a QL class, ``ControlFlowNode``. Examples of control flow nodes include statements and expressions. +The control flow is modeled with a CodeQL class, ``ControlFlowNode``. Examples of control flow nodes include statements and expressions. - ``ControlFlowNode`` provides API for traversing the control flow graph: diff --git a/docs/language/ql-training/cpp/intro-ql-cpp.rst b/docs/language/ql-training/cpp/intro-ql-cpp.rst index 9cfb1d3b442..4e2cd4ca0b4 100644 --- a/docs/language/ql-training/cpp/intro-ql-cpp.rst +++ b/docs/language/ql-training/cpp/intro-ql-cpp.rst @@ -70,7 +70,7 @@ A simple CodeQL query A `query `__ consists of a “select” clause that indicates what results should be returned. Typically it will also provide a “from” clause to declare some variables, and a “where” clause to state conditions over those variables. For more information on the structure of query files (including links to useful topics in the `QL language handbook `__), see `Introduction to query files `__. - In our example here, the first line of the query imports the `CodeQL for C/C++ standard library `__, which defines concepts like ``IfStmt`` and ``Block``. + In our example here, the first line of the query imports the `CodeQL library for C/C++ `__, which defines concepts like ``IfStmt`` and ``Block``. The query proper starts by declaring two variables–ifStmt and block. These variables represent sets of values in the database, according to the type of each of the variables. For example, ifStmt has the type IfStmt, which means it represents the set of all if statements in the program. If we simply selected these two variables:: @@ -135,10 +135,10 @@ A predicate allows you to pull out and name parts of a query. You can imagine a predicate to be a self-contained from-where-select statement, that produces an intermediate relation, or table. In this case, the ``isEmpty`` predicate will be the set of all blocks which are empty. -Classes -======= +Classes in QL +============= -A class allows you to name a set of values and define (member) predicates on them. +A QL class allows you to name a set of values and define (member) predicates on them. A class has at least one supertype and optionally a **characteristic predicate**; it contains the values that belong to *all* supertypes *and* satisfy the characteristic predicate, if provided. @@ -158,8 +158,8 @@ Member predicates are inherited and can be overridden. In the example, declaring a variable “EmptyBlock e” will allow it to range over only those blocks that have zero statements. -Classes continued -================= +Classes in QL continued +======================= .. container:: column-left diff --git a/docs/language/ql-training/cpp/program-representation-cpp.rst b/docs/language/ql-training/cpp/program-representation-cpp.rst index 156cb0a4a68..8f7a4d43de9 100644 --- a/docs/language/ql-training/cpp/program-representation-cpp.rst +++ b/docs/language/ql-training/cpp/program-representation-cpp.rst @@ -21,16 +21,16 @@ Agenda .. resume slides -AST classes -=========== +AST CodeQL classes +================== -Important AST classes include: +Important AST CodeQL classes include: - ``Expr``: expressions such as assignments, variable references, function calls, ... - ``Stmt``: statements such as conditionals, loops, try statements, ... - ``DeclarationEntry``: places where functions, variables or types are declared and/or defined -These three (and all other AST classes) are subclasses of ``Element``. +These three (and all other AST CodeQL classes) are subclasses of ``Element``. Symbol table ============ @@ -64,7 +64,7 @@ Working with functions Functions are represented by the Function class. Each declaration or definition of a function is represented by a ``FunctionDeclarationEntry``. -Calls to functions are modeled by class ``Call`` and its subclasses: +Calls to functions are modeled by CodeQL class ``Call`` and its subclasses: - ``Call.getTarget()`` gets the declared target of the call; undefined for calls through function pointers - ``Function.getACallToThisFunction()`` gets a call to this function diff --git a/docs/language/ql-training/java/intro-ql-java.rst b/docs/language/ql-training/java/intro-ql-java.rst index 46106204ed4..1666d9caca8 100644 --- a/docs/language/ql-training/java/intro-ql-java.rst +++ b/docs/language/ql-training/java/intro-ql-java.rst @@ -70,7 +70,7 @@ A simple CodeQL query A `query `__ consists of a “select” clause that indicates what results should be returned. Typically it will also provide a “from” clause to declare some variables, and a “where” clause to state conditions over those variables. For more information on the structure of query files (including links to useful topics in the `QL language handbook `__), see `Introduction to query files `__. - In our example here, the first line of the query imports the `CodeQL for Java library `__, which defines concepts like ``IfStmt`` and ``Block``. + In our example here, the first line of the query imports the `CodeQL library for Java `__, which defines concepts like ``IfStmt`` and ``Block``. The query proper starts by declaring two variables–ifStmt and block. These variables represent sets of values in the database, according to the type of each of the variables. For example, ``ifStmt`` has the type ``IfStmt``, which means it represents the set of all if statements in the program. If we simply selected these two variables:: @@ -135,10 +135,10 @@ A predicate allows you to pull out and name parts of a query. You can imagine a predicate to be a self-contained from-where-select statement, that produces an intermediate relation, or table. In this case, the ``isEmpty`` predicate will be the set of all blocks which are empty. -Classes -======= +Classes in QL +============= -A class allows you to name a set of values and define (member) predicates on them. +A QL class allows you to name a set of values and define (member) predicates on them. A class has at least one supertype and optionally a **characteristic predicate**; it contains the values that belong to *all* supertypes *and* satisfy the characteristic predicate, if provided. @@ -158,7 +158,7 @@ Member predicates are inherited and can be overridden. In the example, declaring a variable “EmptyBlock e” will allow it to range over only those blocks that have zero statements. -Classes continued +Classes in QL continued ======================= .. container:: column-left diff --git a/docs/language/ql-training/java/program-representation-java.rst b/docs/language/ql-training/java/program-representation-java.rst index ba9b1a9d80d..a66a81d6083 100644 --- a/docs/language/ql-training/java/program-representation-java.rst +++ b/docs/language/ql-training/java/program-representation-java.rst @@ -12,7 +12,7 @@ Agenda - Abstract syntax trees - Database representation - Program elements -- AST classes +- AST CodeQL classes .. insert abstract-syntax-tree.rst @@ -23,7 +23,7 @@ Agenda Program elements ================ -- The QL class ``Element`` represents program elements with a name. +- The CodeQL class ``Element`` represents program elements with a name. - This includes: packages (``Package``), compilation units (``CompilationUnit``), types (``Type``), methods (``Method``), constructors (``Constructor``), and variables (``Variable``). - It is often convenient to refer to an element that might either be a method or a constructor; the class ``Callable``, which is a common superclass of ``Method`` and ``Constructor``, can be used for this purpose. @@ -31,7 +31,7 @@ Program elements AST === -There are two primary AST classes, used within ``Callables``: +There are two primary AST CodeQL classes, used within ``Callables``: - ``Expr``: expressions such as assignments, variable references, function calls, ... - ``Stmt``: statements such as conditionals, loops, try statements, ... @@ -47,7 +47,7 @@ Types The database also includes information about the types used in a program: -- ``PrimitiveType`` represents a `primitive type `__, that is, one of ``boolean``, ``byte``, ``char``, ``double``, ``float``, ``int``, ``long``, ``short``. QL also classifies ``void`` and ```` (the type of the ``null`` literal) as primitive types. +- ``PrimitiveType`` represents a `primitive type `__, that is, one of ``boolean``, ``byte``, ``char``, ``double``, ``float``, ``int``, ``long``, ``short``. CodeQL also classifies ``void`` and ```` (the type of the ``null`` literal) as primitive types. - ``RefType`` represents a reference type; it has several subclasses: - ``Class`` represents a Java class. @@ -74,9 +74,9 @@ Working with variables Working with callables ====================== -Callables are represented by the ``Callable`` QL class. +Callables are represented by the ``Callable`` CodeQL class. -Calls to callables are modeled by the QL class ``Call`` and its subclasses: +Calls to callables are modeled by the CodeQL class ``Call`` and its subclasses: - ``Call.getCallee()`` gets the declared target of the call - ``Call.getAReference()`` gets a call to this function diff --git a/docs/language/ql-training/java/query-injection-java.rst b/docs/language/ql-training/java/query-injection-java.rst index a5b46eabda5..799c99e5209 100644 --- a/docs/language/ql-training/java/query-injection-java.rst +++ b/docs/language/ql-training/java/query-injection-java.rst @@ -77,14 +77,14 @@ Let’s start by looking for calls to methods with names of the form ``sparql*Qu .. note:: - - When performing `variant analysis `__, it is usually helpful to write a simple query that finds the simple syntactic pattern, before trying to go on to describe the cases where it goes wrong. - - In this case, we start by looking for all the method calls which appear to run, before trying to refine the query to find cases which are vulnerable to query injection. + - When performing `variant analysis `__, it is usually helpful to write a simple query that finds the simple syntactic pattern, before trying to go on to describe the cases where it goes wrong. + - In this case, we start by looking for all the method calls that appear to run, before trying to refine the query to find cases which are vulnerable to query injection. - The ``select`` clause defines what this query is looking for: - a ``MethodAccess``: the call to a SPARQL query method - a ``Method``: the SPARQL query method. - - The ``where`` part of the query ties these variables together using `predicates `__ defined in the `standard CodeQL for Java library `__. + - The ``where`` part of the query ties these variables together using `predicates `__ defined in the `standard CodeQL library for Java `__. CodeQL query: find string concatenation ======================================= diff --git a/docs/language/ql-training/slide-snippets/local-data-flow.rst b/docs/language/ql-training/slide-snippets/local-data-flow.rst index 85b851a9ce0..ed681b0398a 100644 --- a/docs/language/ql-training/slide-snippets/local-data-flow.rst +++ b/docs/language/ql-training/slide-snippets/local-data-flow.rst @@ -70,7 +70,7 @@ Local vs global data flow For further information, see: - - `Introduction to data flow analysis in CodeQL `__ + - `Introduction to data flow analysis with CodeQL `__ .. rst-class:: background2 From 2e7bd4db6f163b3d19eda68fab2b460dc8f43196 Mon Sep 17 00:00:00 2001 From: James Fletcher <42464962+jf205@users.noreply.github.com> Date: Wed, 6 Nov 2019 12:59:45 +0000 Subject: [PATCH 8/8] Update docs/language/ql-training/cpp/program-representation-cpp.rst Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com> --- docs/language/ql-training/cpp/program-representation-cpp.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/language/ql-training/cpp/program-representation-cpp.rst b/docs/language/ql-training/cpp/program-representation-cpp.rst index 8f7a4d43de9..7048251aa02 100644 --- a/docs/language/ql-training/cpp/program-representation-cpp.rst +++ b/docs/language/ql-training/cpp/program-representation-cpp.rst @@ -64,7 +64,7 @@ Working with functions Functions are represented by the Function class. Each declaration or definition of a function is represented by a ``FunctionDeclarationEntry``. -Calls to functions are modeled by CodeQL class ``Call`` and its subclasses: +Calls to functions are modeled by the CodeQL class ``Call`` and its subclasses: - ``Call.getTarget()`` gets the declared target of the call; undefined for calls through function pointers - ``Function.getACallToThisFunction()`` gets a call to this function