Revert "C#: Revert "Merge pull request #4653 from tamasvajk/feature/csharp9-relational-pattern""

Approved by erik-krogh, mchammer01

.github/codeql/codeql-config.yml

@@ -7,3 +7,5 @@ paths-ignore:
   - '/cpp/'
   - '/java/'
   - '/python/'
+  - '/javascript/ql/test'
+  - '/javascript/extractor/tests'

.vscode/extensions.json

@@ -3,8 +3,8 @@
     // Extension identifier format: ${publisher}.${name}. Example: vscode.csharp
     // List of extensions which should be recommended for users of this workspace.
     "recommendations": [
-        "github.vscode-codeql"
+        "GitHub.vscode-codeql"
     ],
     // List of extensions recommended by VS Code that should not be recommended for users of this workspace.
     "unwantedRecommendations": []
-}
+}

change-notes/1.25/analysis-java.md

@@ -4,20 +4,26 @@ The following changes in version 1.25 affect Java analysis in all applications.
 
 ## General improvements
 
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
+The Java autobuilder has been improved to detect more Gradle Java versions.
 
 ## Changes to existing queries
 
 | **Query**                    | **Expected impact**    | **Change**                        |
 |------------------------------|------------------------|-----------------------------------|
-
+| Hard-coded credential in API call (`java/hardcoded-credential-api-call`) | More results | The query now recognizes the `BasicAWSCredentials` class of the Amazon client SDK library with hardcoded access key/secret key. |
+| Deserialization of user-controlled data (`java/unsafe-deserialization`) | Fewer false positive results | The query no longer reports results using `org.apache.commons.io.serialization.ValidatingObjectInputStream`. |
+| Use of a broken or risky cryptographic algorithm (`java/weak-cryptographic-algorithm`) | More results | The query now recognizes the `MessageDigest.getInstance` method. |
+| Use of a potentially broken or risky cryptographic algorithm (`java/potentially-weak-cryptographic-algorithm`) | More results | The query now recognizes the `MessageDigest.getInstance` method. |
+| Reading from a world writable file (`java/world-writable-file-read`) | More results | The query now recognizes more JDK file operations. |
 
 ## Changes to libraries
 
+* The data-flow library has been improved with more taint flow modeling for the
+  Collections framework and other classes of the JDK. This affects all security
+  queries using data flow and can yield additional results.
+* The data-flow library has been improved with more taint flow modeling for the
+  Spring framework. This affects all security queries using data flow and can
+  yield additional results on project that rely on the Spring framework.
 * The data-flow library has been improved, which affects most security queries by potentially
   adding more results. Flow through methods now takes nested field reads/writes into account.
   For example, the library is able to track flow from `"taint"` to `sink()` via the method
@@ -39,3 +45,5 @@ The following changes in version 1.25 affect Java analysis in all applications.
     }
   }
   ```
+* The library has been extended with more support for Java 14 features
+  (`switch` expressions and pattern-matching for `instanceof`).

change-notes/1.25/analysis-python.md

@@ -1,22 +1,9 @@
 # Improvements to Python analysis
 
-The following changes in version 1.25 affect Python analysis in all applications.
-
-## General improvements
-
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
-
-## Changes to existing queries
-
-| **Query**                  | **Expected impact**    | **Change**                                                       |
-|----------------------------|------------------------|------------------------------------------------------------------|
-
-
-## Changes to libraries
-
 * Importing `semmle.python.web.HttpRequest` will no longer import `UntrustedStringKind` transitively. `UntrustedStringKind` is the most commonly used non-abstract subclass of `ExternalStringKind`. If not imported (by one mean or another), taint-tracking queries that concern `ExternalStringKind` will not produce any results. Please ensure such queries contain an explicit import (`import semmle.python.security.strings.Untrusted`).
+* Added model of taint sources for HTTP servers using `http.server`.
+* Added taint modeling of routed parameters in Flask.
+* Improved modeling of built-in methods on strings for taint tracking.
+* Improved classification of test files.
+* New class `BoundMethodValue` represents a bound method during runtime.
+* The query `py/command-line-injection` now recognizes command execution with the `fabric` and `invoke` Python libraries.

change-notes/1.26/analysis-cpp.md

@@ -23,7 +23,9 @@ The following changes in version 1.26 affect C/C++ analysis in all applications.
 * The QL class `Block`, denoting the `{ ... }` statement, is renamed to `BlockStmt`.
 * The models library now models many taint flows through `std::array`, `std::vector`, `std::deque`, `std::list` and `std::forward_list`.
 * The models library now models many more taint flows through `std::string`.
-* The models library now models some taint flows through `std::ostream`.
+* The models library now models many taint flows through `std::istream` and `std::ostream`.
 * The models library now models some taint flows through `std::shared_ptr`, `std::unique_ptr`, `std::make_shared` and `std::make_unique`.
+* The models library now models many taint flows through `std::pair`, `std::map`, `std::unordered_map`, `std::set` and `std::unordered_set`.
+* The models library now models `bcopy`.
 * The `SimpleRangeAnalysis` library now supports multiplications of the form
   `e1 * e2` and `x *= e2` when `e1` and `e2` are unsigned or constant.

change-notes/1.26/analysis-csharp.md

@@ -1,35 +0,0 @@
-# Improvements to C# analysis
-
-The following changes in version 1.26 affect C# analysis in all applications.
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
-
-## Changes to existing queries
-
-| **Query**                    | **Expected impact**    | **Change**                        |
-|------------------------------|------------------------|-----------------------------------|
-
-
-## Removal of old queries
-
-## Changes to code extraction
-
-* Partial method bodies are extracted. Previously, partial method bodies were skipped completely.
-* Inferring the lengths of implicitely sized arrays is fixed. Previously, multidimensional arrays were always extracted with the same length for
-each dimension. With the fix, the array sizes `2` and `1` are extracted for `new int[,]{{1},{2}}`. Previously `2` and `2` were extracted.
-* The extractor is now assembly-insensitive by default. This means that two entities with the same
-  fully-qualified name are now mapped to the same entity in the resulting database, regardless of
-  whether they belong to different assemblies. Assembly sensitivity can be reenabled by passing
-  `--assemblysensitivetrap` to the extractor.
-
-## Changes to libraries
-
-## Changes to autobuilder
-
-## Changes to tooling support
-
-* The Abstract Syntax Tree of C# files can be printed in Visual Studio Code.

change-notes/1.26/analysis-java.md

@@ -18,4 +18,3 @@ The following changes in version 1.26 affect Java analysis in all applications.
 
 ## Changes to libraries
 
-* The QL class `Block`, denoting the `{ ... }` statement, is renamed to `BlockStmt`.

change-notes/1.26/analysis-javascript.md

@@ -2,19 +2,49 @@
 
 ## General improvements
 
+* Angular-specific taint sources and sinks are now recognized by the security queries.
+
+* Support for React has improved, with better handling of react hooks, react-router path parameters, lazy-loaded components, and components transformed using `react-redux` and/or `styled-components`.
+
+* Dynamic imports are now analyzed more precisely.
+
 * Support for the following frameworks and libraries has been improved:
+  - [@angular/*](https://www.npmjs.com/package/@angular/core)
+  - [AWS Serverless](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html)
+  - [Alibaba Serverless](https://www.alibabacloud.com/help/doc-detail/156876.htm)
+  - [debounce](https://www.npmjs.com/package/debounce)
+  - [bluebird](https://www.npmjs.com/package/bluebird)
+  - [call-limit](https://www.npmjs.com/package/call-limit)
+  - [classnames](https://www.npmjs.com/package/classnames)
+  - [clsx](https://www.npmjs.com/package/clsx)
+  - [express](https://www.npmjs.com/package/express)
   - [fast-json-stable-stringify](https://www.npmjs.com/package/fast-json-stable-stringify)
   - [fast-safe-stringify](https://www.npmjs.com/package/fast-safe-stringify)
+  - [http](https://nodejs.org/api/http.html)
   - [javascript-stringify](https://www.npmjs.com/package/javascript-stringify)
   - [js-stringify](https://www.npmjs.com/package/js-stringify)
   - [json-stable-stringify](https://www.npmjs.com/package/json-stable-stringify)
   - [json-stringify-safe](https://www.npmjs.com/package/json-stringify-safe)
   - [json3](https://www.npmjs.com/package/json3)
+  - [jQuery throttle / debounce](https://github.com/cowboy/jquery-throttle-debounce)
+  - [lodash](https://www.npmjs.com/package/lodash)
+  - [lodash.debounce](https://www.npmjs.com/package/lodash.debounce)
+  - [lodash.throttle](https://www.npmjs.com/package/lodash.throttle)
+  - [needle](https://www.npmjs.com/package/needle)
   - [object-inspect](https://www.npmjs.com/package/object-inspect)
   - [pretty-format](https://www.npmjs.com/package/pretty-format)
+  - [react](https://www.npmjs.com/package/react)
+  - [react-router-dom](https://www.npmjs.com/package/react-router-dom)
+  - [react-redux](https://www.npmjs.com/package/react-redux)
+  - [redis](https://www.npmjs.com/package/redis)
+  - [redux](https://www.npmjs.com/package/redux)
   - [stringify-object](https://www.npmjs.com/package/stringify-object)
+  - [styled-components](https://www.npmjs.com/package/styled-components)
+  - [throttle-debounce](https://www.npmjs.com/package/throttle-debounce)
+  - [underscore](https://www.npmjs.com/package/underscore)
 
 * Analyzing files with the ".cjs" extension is now supported.
+* ES2021 features are now supported.
 
 ## New queries
 
@@ -32,7 +62,12 @@
 | Unused loop iteration variable (`js/unused-loop-variable`) | Fewer results | This query no longer flags variables in a destructuring array assignment that are not the last variable in the destructed array. |
 | Unsafe shell command constructed from library input (`js/shell-command-constructed-from-input`) | More results | This query now recognizes more commands where colon, dash, and underscore are used. |
 | Unsafe jQuery plugin (`js/unsafe-jquery-plugin`) | More results | This query now detects more unsafe uses of nested option properties. |
+| Client-side URL redirect (`js/client-side-unvalidated-url-redirection`) | More results | This query now recognizes some unsafe uses of `importScripts()` inside WebWorkers. |
+| Missing CSRF middleware (`js/missing-token-validation`) | More results | This query now recognizes writes to cookie and session variables as potentially vulnerable to CSRF attacks. |
+| Missing CSRF middleware (`js/missing-token-validation`) | Fewer results | This query now recognizes more ways of protecting against CSRF attacks. |
+| Client-side cross-site scripting (`js/xss`) | More results | This query now tracks data flow from `location.hash` more precisely. |
 
 
 ## Changes to libraries
 * The predicate `TypeAnnotation.hasQualifiedName` now works in more cases when the imported library was not present during extraction.
+* The class `DomBasedXss::Configuration` has been deprecated, as it has been split into `DomBasedXss::HtmlInjectionConfiguration` and `DomBasedXss::JQueryHtmlOrSelectorInjectionConfiguration`. Unless specifically working with jQuery sinks, subclasses should instead be based on `HtmlInjectionConfiguration`. To use both configurations in a query, see [Xss.ql](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/Xss.ql) for an example.

config/identical-files.json

@@ -19,15 +19,17 @@
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImpl3.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImpl4.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImpl5.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImpl.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImpl2.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl2.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl3.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImpl4.qll"
   ],
   "DataFlow Java/C++/C#/Python Common": [
     "java/ql/src/semmle/code/java/dataflow/internal/DataFlowImplCommon.qll",
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowImplCommon.qll",
     "cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowImplCommon.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImplCommon.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImplCommon.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll"
   ],
   "TaintTracking::Configuration Java/C++/C#/Python": [
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
@@ -41,14 +43,37 @@
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/tainttracking5/TaintTrackingImpl.qll",
     "java/ql/src/semmle/code/java/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
     "java/ql/src/semmle/code/java/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
-    "python/ql/src/experimental/dataflow/internal/tainttracking1/TaintTrackingImpl.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking1/TaintTrackingImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking2/TaintTrackingImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking3/TaintTrackingImpl.qll",
+    "python/ql/src/semmle/python/dataflow/new/internal/tainttracking4/TaintTrackingImpl.qll"
   ],
   "DataFlow Java/C++/C#/Python Consistency checks": [
     "java/ql/src/semmle/code/java/dataflow/internal/DataFlowImplConsistency.qll",
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowImplConsistency.qll",
     "cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowImplConsistency.qll",
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImplConsistency.qll",
-    "python/ql/src/experimental/dataflow/internal/DataFlowImplConsistency.qll"
+    "python/ql/src/semmle/python/dataflow/new/internal/DataFlowImplConsistency.qll"
+  ],
+  "SsaReadPosition Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll"
+  ],
+  "Sign Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/Sign.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/Sign.qll"
+  ],
+  "SignAnalysis Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll"
+  ],
+  "Bound Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/Bound.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/Bound.qll"
+  ],
+  "ModulusAnalysis Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/ModulusAnalysis.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/ModulusAnalysis.qll"
   ],
   "C++ SubBasicBlocks": [
     "cpp/ql/src/semmle/code/cpp/controlflow/SubBasicBlocks.qll",
@@ -87,7 +112,7 @@
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/Operand.qll",
     "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/Operand.qll",
     "csharp/ql/src/experimental/ir/implementation/raw/Operand.qll",
-    "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Operand.qll" 
+    "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Operand.qll"
   ],
   "IR IRType": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/IRType.qll",
@@ -109,11 +134,11 @@
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/OperandTag.qll",
     "csharp/ql/src/experimental/ir/implementation/internal/OperandTag.qll"
   ],
-  "IR TInstruction":[
+  "IR TInstruction": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/TInstruction.qll",
     "csharp/ql/src/experimental/ir/implementation/internal/TInstruction.qll"
   ],
-  "IR TIRVariable":[
+  "IR TIRVariable": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/TIRVariable.qll",
     "csharp/ql/src/experimental/ir/implementation/internal/TIRVariable.qll"
   ],
@@ -325,10 +350,22 @@
     "csharp/ql/src/experimental/ir/implementation/raw/gvn/internal/ValueNumberingImports.qll",
     "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll"
   ],
+  "C# ControlFlowReachability": [
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/ControlFlowReachability.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/ControlFlowReachability.qll"
+  ],
   "Inline Test Expectations": [
     "cpp/ql/test/TestUtilities/InlineExpectationsTest.qll",
     "python/ql/test/TestUtilities/InlineExpectationsTest.qll"
   ],
+  "C++ ExternalAPIs": [
+    "cpp/ql/src/Security/CWE/CWE-020/ExternalAPIs.qll",
+    "cpp/ql/src/Security/CWE/CWE-020/ir/ExternalAPIs.qll"
+  ],
+  "C++ SafeExternalAPIFunction": [
+    "cpp/ql/src/Security/CWE/CWE-020/SafeExternalAPIFunction.qll",
+    "cpp/ql/src/Security/CWE/CWE-020/ir/SafeExternalAPIFunction.qll"
+  ],
   "XML": [
     "cpp/ql/src/semmle/code/cpp/XML.qll",
     "csharp/ql/src/semmle/code/csharp/XML.qll",
@@ -380,5 +417,12 @@
     "java/ql/src/Metrics/Files/CommentedOutCodeReferences.qhelp",
     "javascript/ql/src/Comments/CommentedOutCodeReferences.qhelp",
     "python/ql/src/Lexical/CommentedOutCodeReferences.qhelp"
+  ],
+  "IDE Contextual Queries": [
+    "cpp/ql/src/IDEContextual.qll",
+    "csharp/ql/src/IDEContextual.qll",
+    "java/ql/src/IDEContextual.qll",
+    "javascript/ql/src/IDEContextual.qll",
+    "python/ql/src/analysis/IDEContextual.qll"
   ]
 }

cpp/autobuilder/Semmle.Autobuild.Cpp.Tests/Semmle.Autobuild.Cpp.Tests.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFramework>netcoreapp3.0</TargetFramework>
+    <TargetFramework>netcoreapp3.1</TargetFramework>
     <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
     <RuntimeIdentifiers>win-x64;linux-x64;osx-x64</RuntimeIdentifiers>
     <Nullable>enable</Nullable>

cpp/autobuilder/Semmle.Autobuild.Cpp/Semmle.Autobuild.Cpp.csproj

@@ -1,7 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>netcoreapp3.0</TargetFramework>
+    <TargetFramework>netcoreapp3.1</TargetFramework>
     <AssemblyName>Semmle.Autobuild.Cpp</AssemblyName>
     <RootNamespace>Semmle.Autobuild.Cpp</RootNamespace>
     <ApplicationIcon />

cpp/change-notes/2020-09-29-range-analysis-rollup.md

@@ -0,0 +1,14 @@
+lgtm,codescanning
+* The `SimpleRangeAnalysis` library has gained support for several language
+  constructs it did not support previously. These improvements primarily affect
+  the queries `cpp/constant-comparison`, `cpp/comparison-with-wider-type`, and
+  `cpp/integer-multiplication-cast-to-long`. The newly supported language
+  features are:
+    * Multiplication of unsigned numbers.
+    * Multiplication by a constant.
+    * Reference-typed function parameters.
+    * Comparing a variable not equal to an endpoint of its range, thus narrowing the range by one.
+    * Using `if (x)` or `if (!x)` or similar to test for equality to zero.
+* The `SimpleRangeAnalysis` library can now be extended with custom rules. See
+  examples in
+  `cpp/ql/src/experimental/semmle/code/cpp/rangeanalysis/extensions/`.

cpp/change-notes/2020-10-21-erroneous-types.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The `cpp/wrong-type-format-argument` and `cpp/non-portable-printf` queries have been hardened so that they do not produce nonsensical results on databases that contain errors (specifically the `ErroneousType`).

cpp/change-notes/2020-10-21-size-check-queries.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The 'Not enough memory allocated for pointer type' (cpp/allocation-too-small) and 'Not enough memory allocated for array of pointer type' (cpp/suspicious-allocation-size) queries have been improved. Previously some allocations would be reported by both queries, this no longer occurs. In addition more allocation functions are now understood by both queries.

cpp/change-notes/2020-11-02-unused-local-variable.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* Two issues causing the 'Unused local variable' query (`cpp/unused-local-variable`) to produce false positive results have been fixed.

cpp/change-notes/2020-11-05-formatting-function.md

@@ -0,0 +1,4 @@
+lgtm,codescanning
+* `FormattingFunction.getOutputParameterIndex` now has a parameter identifying whether the output at that index is a buffer or a stream.
+* `FormattingFunction` now has a predicate `isOutputGlobal` indicating when the output is to a global stream.
+* The `primitiveVariadicFormatter` and `variadicFormatter` predicates have more parameters exposing information about the function.

cpp/change-notes/2020-11-05-private-models.md

@@ -0,0 +1,3 @@
+lgtm,codescanning
+* Various classes in `semmle.code.cpp.models.implementations` have been made private. Users should not depend on library implementation details.
+* The `OperatorNewAllocationFunction`, `OperatorDeleteDeallocationFunction`, `Iterator` and `Snprintf` classes now have interfaces in `semmle.code.cpp.models.interfaces`.

cpp/change-notes/2020-11-12-unsafe-use-of-this.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* A new query (`cpp/unsafe-use-of-this`) has been added. The query finds pure virtual function calls whose qualifier is an object under construction.

cpp/change-notes/2020-11-27-downgrade-to-recommendation.md

@@ -0,0 +1,2 @@
+lgtm,codescanning
+* The queries `cpp/local-variable-hides-global-variable` and `cpp/missing-header-guard` now have severity `recommendation` instead of `warning`.

cpp/config/suites/cpp/correctness

@@ -9,6 +9,7 @@
 + semmlecode-cpp-queries/Likely Bugs/Conversion/CastArrayPointerArithmetic.ql: /Correctness/Dangerous Conversions
 + semmlecode-cpp-queries/Likely Bugs/Underspecified Functions/MistypedFunctionArguments.ql: /Correctness/Dangerous Conversions
 + semmlecode-cpp-queries/Security/CWE/CWE-253/HResultBooleanConversion.ql: /Correctness/Dangerous Conversions
++ semmlecode-cpp-queries/Likely Bugs/OO/UnsafeUseOfThis.ql: /Correctness/Dangerous Conversions
   # Consistent Use
 + semmlecode-cpp-queries/Critical/ReturnValueIgnored.ql: /Correctness/Consistent Use
 + semmlecode-cpp-queries/Likely Bugs/InconsistentCheckReturnNull.ql: /Correctness/Consistent Use

cpp/ql/src/Best Practices/Exceptions/LeakyCatch.qhelp

@@ -39,7 +39,7 @@ void good() {
 </example>
 <references>
 
-  <li>MSDN Library for MFC: <a href="http://msdn.microsoft.com/en-us/library/0e5twxsh(v=vs.110).aspx">Exceptions: Catching and Deleting Exceptions</a>.</li>
+  <li>MSDN Library for MFC: <a href="https://docs.microsoft.com/en-us/cpp/mfc/exceptions-catching-and-deleting-exceptions">Exceptions: Catching and Deleting Exceptions</a>.</li>
 
 
 </references>

cpp/ql/src/Best Practices/Hiding/LocalVariableHidesGlobalVariable.ql

@@ -2,7 +2,7 @@
  * @name Local variable hides global variable
  * @description A local variable or parameter that hides a global variable of the same name. This may be confusing. Consider renaming one of the variables.
  * @kind problem
- * @problem.severity warning
+ * @problem.severity recommendation
  * @precision very-high
  * @id cpp/local-variable-hides-global-variable
  * @tags maintainability

cpp/ql/src/Best Practices/Magic Constants/MagicConstants.qll

@@ -291,8 +291,7 @@ predicate arrayInitializerChild(AggregateLiteral parent, Expr e) {
 
 // i.e. not a constant folded expression
 predicate literallyLiteral(Literal lit) {
-  lit
-      .getValueText()
+  lit.getValueText()
       .regexpMatch(".*\".*|\\s*+[-+]?+\\s*+(0[xob][0-9a-fA-F]|[0-9])[0-9a-fA-F,._]*+([eE][-+]?+[0-9,._]*+)?+\\s*+[a-zA-Z]*+\\s*+")
 }
 

cpp/ql/src/Best Practices/Unused Entities/UnusedLocals.ql

@@ -57,5 +57,12 @@ where
   not declarationHasSideEffects(v) and
   not exists(AsmStmt s | f = s.getEnclosingFunction()) and
   not v.getAnAttribute().getName() = "unused" and
-  not any(ErrorExpr e).getEnclosingFunction() = f // unextracted expr likely used `v`
+  not any(ErrorExpr e).getEnclosingFunction() = f and // unextracted expr may use `v`
+  not exists(
+    Literal l // this case can be removed when the `myFunction2( [obj](){} );` test case doesn't depend on this exclusion
+  |
+    l.getEnclosingFunction() = f and
+    not exists(l.getValue())
+  ) and
+  not any(ConditionDeclExpr cde).getEnclosingFunction() = f // this case can be removed when the `if (a = b; a)` test case doesn't depend on this exclusion
 select v, "Variable " + v.getName() + " is not used"

cpp/ql/src/Best Practices/Unused Entities/UnusedStaticVariables.qhelp

@@ -27,7 +27,7 @@ then removing it will make code more readable. If the static variable is needed
   <a href="https://www.securecoding.cert.org/confluence/display/c/MSC12-C.+Detect+and+remove+code+that+has+no+effect+or+is+never+executed">Detect and remove code that has no effect</a>
 </li>
 <li>
-  <a href="https://www.securecoding.cert.org/confluence/display/cplusplus/DCL07-CPP.+Minimize+the+scope+of+variables+and+methods">Minimize the scope of variables and methods</a>
+  <a href="https://wiki.sei.cmu.edu/confluence/display/c/DCL19-C.+Minimize+the+scope+of+variables+and+functions">Minimize the scope of variables and functions</a>
 </li>
 
 

cpp/ql/src/Best Practices/UseOfGoto.qhelp

@@ -41,7 +41,7 @@ this rule.
   E. W. Dijkstra Archive: <a href="http://www.cs.utexas.edu/users/EWD/transcriptions/EWD02xx/EWD215.html">A Case against the GO TO Statement (EWD-215)</a>.
 </li>
 <li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-gb/library/b34dt9cd%28v=vs.80%29.aspx">The goto Statement</a>.
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/cpp/goto-statement-cpp">goto Statement (C++)</a>.
 </li>
 <li>
   Mats Henricson and Erik Nyquist, <i>Industrial Strength C++</i>, Rule 4.6. Prentice Hall PTR, 1997.

cpp/ql/src/Critical/MissingNullTest.qhelp

@@ -27,6 +27,6 @@ this cannot happen.
 </example>
 
 <references>
-<li>SEI CERT C Coding Standard: <a href="https://wiki.sei.cmu.edu/confluence/display/c/EXP34-C.+Do+not+dereference+null+pointerss">EXP34-C. Do not dereference null pointers</a>.</li>
+<li>SEI CERT C Coding Standard: <a href="https://wiki.sei.cmu.edu/confluence/display/c/EXP34-C.+Do+not+dereference+null+pointers">EXP34-C. Do not dereference null pointers</a>.</li>
 </references>
 </qhelp>

cpp/ql/src/Critical/NewDelete.qll

@@ -5,8 +5,6 @@
 import cpp
 import semmle.code.cpp.controlflow.SSA
 import semmle.code.cpp.dataflow.DataFlow
-import semmle.code.cpp.models.implementations.Allocation
-import semmle.code.cpp.models.implementations.Deallocation
 
 /**
  * Holds if `alloc` is a use of `malloc` or `new`.  `kind` is

cpp/ql/src/Critical/OverflowDestination.ql

@@ -23,10 +23,7 @@ import semmle.code.cpp.security.TaintTracking
  * ```
  */
 predicate sourceSized(FunctionCall fc, Expr src) {
-  exists(string name |
-    (name = "strncpy" or name = "strncat" or name = "memcpy" or name = "memmove") and
-    fc.getTarget().hasGlobalOrStdName(name)
-  ) and
+  fc.getTarget().hasGlobalOrStdName(["strncpy", "strncat", "memcpy", "memmove"]) and
   exists(Expr dest, Expr size, Variable v |
     fc.getArgument(0) = dest and
     fc.getArgument(1) = src and

cpp/ql/src/Critical/SizeCheck.ql

@@ -13,30 +13,9 @@
  */
 
 import cpp
+import semmle.code.cpp.models.Models
 
-class Allocation extends FunctionCall {
-  Allocation() {
-    exists(string name |
-      this.getTarget().hasGlobalOrStdName(name) and
-      (name = "malloc" or name = "calloc" or name = "realloc")
-    )
-  }
-
-  private string getName() { this.getTarget().hasGlobalOrStdName(result) }
-
-  int getSize() {
-    this.getName() = "malloc" and
-    this.getArgument(0).getValue().toInt() = result
-    or
-    this.getName() = "realloc" and
-    this.getArgument(1).getValue().toInt() = result
-    or
-    this.getName() = "calloc" and
-    result = this.getArgument(0).getValue().toInt() * this.getArgument(1).getValue().toInt()
-  }
-}
-
-predicate baseType(Allocation alloc, Type base) {
+predicate baseType(AllocationExpr alloc, Type base) {
   exists(PointerType pointer |
     pointer.getBaseType() = base and
     (
@@ -54,11 +33,12 @@ predicate decideOnSize(Type t, int size) {
   size = min(t.getSize())
 }
 
-from Allocation alloc, Type base, int basesize, int allocated
+from AllocationExpr alloc, Type base, int basesize, int allocated
 where
   baseType(alloc, base) and
-  allocated = alloc.getSize() and
+  allocated = alloc.getSizeBytes() and
   decideOnSize(base, basesize) and
+  alloc.(FunctionCall).getTarget() instanceof AllocationFunction and // exclude `new` and similar
   basesize > allocated
 select alloc,
   "Type '" + base.getName() + "' is " + basesize.toString() + " bytes, but only " +

cpp/ql/src/Critical/SizeCheck2.ql

@@ -13,30 +13,9 @@
  */
 
 import cpp
+import semmle.code.cpp.models.Models
 
-class Allocation extends FunctionCall {
-  Allocation() {
-    exists(string name |
-      this.getTarget().hasGlobalOrStdName(name) and
-      (name = "malloc" or name = "calloc" or name = "realloc")
-    )
-  }
-
-  private string getName() { this.getTarget().hasGlobalOrStdName(result) }
-
-  int getSize() {
-    this.getName() = "malloc" and
-    this.getArgument(0).getValue().toInt() = result
-    or
-    this.getName() = "realloc" and
-    this.getArgument(1).getValue().toInt() = result
-    or
-    this.getName() = "calloc" and
-    result = this.getArgument(0).getValue().toInt() * this.getArgument(1).getValue().toInt()
-  }
-}
-
-predicate baseType(Allocation alloc, Type base) {
+predicate baseType(AllocationExpr alloc, Type base) {
   exists(PointerType pointer |
     pointer.getBaseType() = base and
     (
@@ -49,16 +28,23 @@ predicate baseType(Allocation alloc, Type base) {
   )
 }
 
-from Allocation alloc, Type base, int basesize, int allocated
+predicate decideOnSize(Type t, int size) {
+  // If the codebase has more than one type with the same name, it can have more than one size.
+  size = min(t.getSize())
+}
+
+from AllocationExpr alloc, Type base, int basesize, int allocated
 where
   baseType(alloc, base) and
-  allocated = alloc.getSize() and
+  allocated = alloc.getSizeBytes() and
+  decideOnSize(base, basesize) and
+  alloc.(FunctionCall).getTarget() instanceof AllocationFunction and // exclude `new` and similar
   // If the codebase has more than one type with the same name, check if any matches
   not exists(int size | base.getSize() = size |
     size = 0 or
     (allocated / size) * size = allocated
   ) and
-  basesize = min(base.getSize())
+  not basesize > allocated // covered by SizeCheck.ql
 select alloc,
   "Allocated memory (" + allocated.toString() + " bytes) is not a multiple of the size of '" +
     base.getName() + "' (" + basesize.toString() + " bytes)."

cpp/ql/src/IDEContextual.qll

@@ -0,0 +1,22 @@
+/**
+ * Provides shared predicates related to contextual queries in the code viewer.
+ */
+
+import semmle.files.FileSystem
+
+/**
+ * Returns the `File` matching the given source file name as encoded by the VS
+ * Code extension.
+ */
+cached
+File getFileBySourceArchiveName(string name) {
+  // The name provided for a file in the source archive by the VS Code extension
+  // has some differences from the absolute path in the database:
+  // 1. colons are replaced by underscores
+  // 2. there's a leading slash, even for Windows paths: "C:/foo/bar" ->
+  //    "/C_/foo/bar"
+  // 3. double slashes in UNC prefixes are replaced with a single slash
+  // We can handle 2 and 3 together by unconditionally adding a leading slash
+  // before replacing double slashes.
+  name = ("/" + result.getAbsolutePath().replaceAll(":", "_")).replaceAll("//", "/")
+}

cpp/ql/src/JPL_C/LOC-2/Rule 11/SimpleControlFlowJmp.ql

@@ -13,14 +13,7 @@
 import cpp
 
 class ForbiddenFunction extends Function {
-  ForbiddenFunction() {
-    exists(string name | name = this.getName() |
-      name = "setjmp" or
-      name = "longjmp" or
-      name = "sigsetjmp" or
-      name = "siglongjmp"
-    )
-  }
+  ForbiddenFunction() { this.getName() = ["setjmp", "longjmp", "sigsetjmp", "siglongjmp"] }
 }
 
 from FunctionCall call

cpp/ql/src/Likely Bugs/Arithmetic/BadCheckOdd.qhelp

@@ -23,7 +23,7 @@ As a result, this check incorrectly considers all negative numbers as even.
 <references>
 
 <li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-us/library/ty2ax9z9%28v=vs.71%29.aspx">Multiplicative Operators: *, /, and %</a>.
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/cpp/multiplicative-operators-and-the-modulus-operator">Multiplicative Operators and the Modulus Operator</a>.
 </li>
 <li>
   Wikipedia: <a href="http://en.wikipedia.org/wiki/Modulo_operation#Common_pitfalls">Modulo Operation - Common pitfalls</a>.

cpp/ql/src/Likely Bugs/Arithmetic/BitwiseSignCheck.qhelp

@@ -24,7 +24,7 @@
   Code Project: <a href="http://www.codeproject.com/Articles/2247/An-introduction-to-bitwise-operators">An introduction to bitwise operators</a>
 </li>
 <li>
-  MSDN Library: <a href="https://msdn.microsoft.com/en-us/library/dxda59dh.aspx">Signed Bitwise Operations</a>
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/c-language/signed-bitwise-operations">Signed Bitwise Operations</a>
 </li>
 
 

cpp/ql/src/Likely Bugs/Arithmetic/ComparisonPrecedence.qhelp

@@ -21,7 +21,7 @@ It is best to fully parenthesize complex comparison expressions to explicitly de
 <references>
 
 <li>
-  <a href="http://msdn.microsoft.com/en-us/library/126fe14k%28v=VS.80%29.aspx">Operator Precedence and Associativity</a>
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/cpp/cpp-built-in-operators-precedence-and-associativity">C++ built-in operators, precedence, and associativity</a>
 </li>
 <li>
   <a href="http://www.cplusplus.com/doc/tutorial/operators/">Operators</a>

cpp/ql/src/Likely Bugs/Arithmetic/FloatComparison.qhelp

@@ -24,7 +24,7 @@ as rounding errors will be more prominent when using such values.
 
 <li>
   D. Goldberg, <em>What Every Computer Scientist Should Know About Floating-Point Arithmetic</em>,
-ACM Computing Surveys, Volume 23, Issue 1, March 1991 (<a href="http://docs.sun.com/source/806-3568/ncg_goldberg.html">available online</a>).
+ACM Computing Surveys, Volume 23, Issue 1, March 1991 (<a href="https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html">available online</a>).
 </li>
 
 

cpp/ql/src/Likely Bugs/Arithmetic/IntMultToLong.cpp

@@ -4,3 +4,5 @@ long j = i * i; //Wrong: due to overflow on the multiplication between ints,
 
 long k = (long) i * i; //Correct: the multiplication is done on longs instead of ints, 
                        //and will not overflow
+
+long l = static_cast<long>(i) * i; //Correct: modern C++

cpp/ql/src/Likely Bugs/Arithmetic/IntMultToLong.qhelp

@@ -23,7 +23,7 @@ the expression would produce a result that would be too large to fit in the smal
 <references>
 
 <li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-us/library/ty2ax9z9%28v=vs.71%29.aspx">Multiplicative Operators: *, /, and %</a>.
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/cpp/multiplicative-operators-and-the-modulus-operator">Multiplicative Operators and the Modulus Operator</a>.
 </li>
 <li>
   Cplusplus.com: <a href="http://www.cplusplus.com/articles/DE18T05o/">Integer overflow</a>.

cpp/ql/src/Likely Bugs/Conversion/LossyPointerCast.qhelp

@@ -23,7 +23,7 @@ the latter occupies eight bytes on a 64-bit machine.</p>
 <references>
 
 <li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-us/library/hh279667.aspx">Type Conversions and Type Safety (Modern C++)</a>.
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/cpp/type-conversions-and-type-safety-modern-cpp">Type Conversions and Type Safety</a>.
 </li>
 <li>
   Cplusplus.com: <a href="http://www.cplusplus.com/doc/tutorial/typecasting/">Type conversions</a>.

cpp/ql/src/Likely Bugs/Format/WrongTypeFormatArguments.qhelp

@@ -23,7 +23,7 @@ the function.
 <li>CERT C Coding
 Standard: <a href="https://www.securecoding.cert.org/confluence/display/c/FIO30-C.+Exclude+user+input+from+format+strings">FIO30-C. Exclude user input from format strings</a>.</li>
 <li>cplusplus.com: <a href="http://www.tutorialspoint.com/cplusplus/cpp_functions.htm">C++ Functions</a>.</li>
-<li>MSDN Alphabetical Function Reference: <a href="http://msdn.microsoft.com/en-us/library/wc7014hz%28VS.71%29.aspx">printf, wprintf</a>.</li>
+<li>CRT Alphabetical Function Reference: <a href="https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/printf-printf-l-wprintf-wprintf-l">printf, _printf_l, wprintf, _wprintf_l</a>.</li>
 
 
 

cpp/ql/src/Likely Bugs/Format/WrongTypeFormatArguments.ql

@@ -155,7 +155,8 @@ where
     not actual.getUnspecifiedType().(IntegralType).getSize() = sizeof_IntType()
   ) and
   not arg.isAffectedByMacro() and
-  not arg.isFromUninstantiatedTemplate(_)
+  not arg.isFromUninstantiatedTemplate(_) and
+  not actual.getUnspecifiedType() instanceof ErroneousType
 select arg,
   "This argument should be of type '" + expected.getName() + "' but is of type '" +
     actual.getUnspecifiedType().getName() + "'"

cpp/ql/src/Likely Bugs/Leap Year/Adding365DaysPerYear.qhelp

@@ -15,7 +15,7 @@ of days.  Alternatively, use an established library routine that already contain
 </recommendation>
 
 <references>
-  <li>U.S. Naval Observatory Website - <a href="https://aa.usno.navy.mil/faq/docs/calendars.php"> Introduction to Calendars</a></li>
+  <li>NASA / Goddard Space Flight Center - <a href="https://eclipse.gsfc.nasa.gov/SEhelp/calendars.html">Calendars</a></li>
   <li>Wikipedia - <a href="https://en.wikipedia.org/wiki/Leap_year_bug"> Leap year bug</a> </li>
   <li>Microsoft Azure blog - <a href="https://azure.microsoft.com/en-us/blog/is-your-code-ready-for-the-leap-year/"> Is your code ready for the leap year?</a> </li>
 </references>

cpp/ql/src/Likely Bugs/Leap Year/UncheckedLeapYearAfterYearModification.qhelp

@@ -22,7 +22,7 @@
 </example>
 
 <references>
-  <li>U.S. Naval Observatory Website - <a href="https://aa.usno.navy.mil/faq/docs/calendars.php"> Introduction to Calendars</a></li>
+  <li>NASA / Goddard Space Flight Center - <a href="https://eclipse.gsfc.nasa.gov/SEhelp/calendars.html">Calendars</a></li>
   <li>Wikipedia - <a href="https://en.wikipedia.org/wiki/Leap_year_bug"> Leap year bug</a> </li>
   <li>Microsoft Azure blog - <a href="https://azure.microsoft.com/en-us/blog/is-your-code-ready-for-the-leap-year/"> Is your code ready for the leap year?</a> </li>
 </references>

cpp/ql/src/Likely Bugs/Leap Year/UncheckedReturnValueForTimeFunctions.qhelp

@@ -34,7 +34,7 @@
 </example>
 
 <references>
-  <li>U.S. Naval Observatory Website - <a href="https://aa.usno.navy.mil/faq/docs/calendars.php"> Introduction to Calendars</a></li>
+  <li>NASA / Goddard Space Flight Center - <a href="https://eclipse.gsfc.nasa.gov/SEhelp/calendars.html">Calendars</a></li>
   <li>Wikipedia - <a href="https://en.wikipedia.org/wiki/Leap_year_bug"> Leap year bug</a> </li>
   <li>Microsoft Azure blog - <a href="https://azure.microsoft.com/en-us/blog/is-your-code-ready-for-the-leap-year/"> Is your code ready for the leap year?</a> </li>
 </references>

cpp/ql/src/Likely Bugs/Leap Year/UncheckedReturnValueForTimeFunctions.ql

@@ -40,9 +40,7 @@ class DateStructModifiedFieldAccess extends LeapYearFieldAccess {
  */
 class SafeTimeGatheringFunction extends Function {
   SafeTimeGatheringFunction() {
-    this.getQualifiedName() = "GetFileTime" or
-    this.getQualifiedName() = "GetSystemTime" or
-    this.getQualifiedName() = "NtQuerySystemTime"
+    this.getQualifiedName() = ["GetFileTime", "GetSystemTime", "NtQuerySystemTime"]
   }
 }
 
@@ -51,15 +49,13 @@ class SafeTimeGatheringFunction extends Function {
  */
 class TimeConversionFunction extends Function {
   TimeConversionFunction() {
-    this.getQualifiedName() = "FileTimeToSystemTime" or
-    this.getQualifiedName() = "SystemTimeToFileTime" or
-    this.getQualifiedName() = "SystemTimeToTzSpecificLocalTime" or
-    this.getQualifiedName() = "SystemTimeToTzSpecificLocalTimeEx" or
-    this.getQualifiedName() = "TzSpecificLocalTimeToSystemTime" or
-    this.getQualifiedName() = "TzSpecificLocalTimeToSystemTimeEx" or
-    this.getQualifiedName() = "RtlLocalTimeToSystemTime" or
-    this.getQualifiedName() = "RtlTimeToSecondsSince1970" or
-    this.getQualifiedName() = "_mkgmtime"
+    this.getQualifiedName() =
+      [
+        "FileTimeToSystemTime", "SystemTimeToFileTime", "SystemTimeToTzSpecificLocalTime",
+        "SystemTimeToTzSpecificLocalTimeEx", "TzSpecificLocalTimeToSystemTime",
+        "TzSpecificLocalTimeToSystemTimeEx", "RtlLocalTimeToSystemTime",
+        "RtlTimeToSecondsSince1970", "_mkgmtime"
+      ]
   }
 }
 

cpp/ql/src/Likely Bugs/Leap Year/UnsafeArrayForDaysOfYear.qhelp

@@ -23,7 +23,7 @@
 </example>
 
 <references>
-  <li>U.S. Naval Observatory Website - <a href="https://aa.usno.navy.mil/faq/docs/calendars.php"> Introduction to Calendars</a></li>
+  <li>NASA / Goddard Space Flight Center - <a href="https://eclipse.gsfc.nasa.gov/SEhelp/calendars.html">Calendars</a></li>
   <li>Wikipedia - <a href="https://en.wikipedia.org/wiki/Leap_year_bug"> Leap year bug</a> </li>
   <li>Microsoft Azure blog - <a href="https://azure.microsoft.com/en-us/blog/is-your-code-ready-for-the-leap-year/"> Is your code ready for the leap year?</a> </li>
 </references>

cpp/ql/src/Likely Bugs/Likely Typos/MissingEnumCaseInSwitch.qhelp

@@ -23,7 +23,7 @@ indication that there may be cases unhandled by the <code>switch</code> statemen
   Tutorialspoint - The C++ Programming Language: <a href="http://www.tutorialspoint.com/cplusplus/cpp_switch_statement.htm">C++ switch statement</a>
 </li>
 <li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-us/library/k0t5wee3%28v=VS.80%29.aspx">The switch Statement</a>
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/cpp/switch-statement-cpp">switch statement (C++)</a>
 </li>
 <li>
   M. Henricson and E. Nyquist, <i>Industrial Strength C++</i>, Chapter 4: Control Flow, Rec 4.5. Prentice Hall PTR, 1997 (<a href="http://mongers.org/industrial-c++/">available online</a>).

cpp/ql/src/Likely Bugs/Memory Management/Padding/NonPortablePrintf.ql

@@ -88,7 +88,8 @@ where
   not arg.isAffectedByMacro() and
   size32 = ilp32.paddedSize(actual) and
   size64 = lp64.paddedSize(actual) and
-  size64 != size32
+  size64 != size32 and
+  not actual instanceof ErroneousType
 select arg,
   "This argument should be of type '" + expected.getName() + "' but is of type '" + actual.getName()
     + "' (which changes size from " + size32 + " to " + size64 + " on 64-bit systems)."

cpp/ql/src/Likely Bugs/Memory Management/SuspiciousCallToMemset.qhelp

@@ -30,7 +30,7 @@ For an array, the size is the number of elements of the array multiplied by the
   Cplusplus.comn: <a href="http://www.cplusplus.com/reference/clibrary/cstring/memset/">memset</a>
 </li>
 <li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-us/library/aa246471%28v=VS.60%29.aspx">memset</a>, <a href="http://msdn.microsoft.com/en-us/library/4s7x1k91%28v=VS.71%29.aspx">sizeof Operator</a>
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/memset-wmemset">memset, wmemset</a>, <a href="https://docs.microsoft.com/en-us/cpp/cpp/sizeof-operator">sizeof Operator</a>
 </li>
 
 

cpp/ql/src/Likely Bugs/NestedLoopSameVar.qhelp

@@ -25,9 +25,6 @@ outer loop. </p>
 <li>
   Tutorialspoint - The C++ Programming Language: <a href="http://www.tutorialspoint.com/cplusplus/cpp_nested_loops.htm">C++ nested loops</a>
 </li>
-<li>
-  MSDN Library: <a href="http://msdn.microsoft.com/en-us/library/8y82wx12%28v=VS.80%29.aspx">Nested Control Structures</a>
-</li>
 
 
 

cpp/ql/src/Likely Bugs/OO/NonVirtualDestructor.qhelp

@@ -20,7 +20,7 @@ object instance).</p>
 
 </example>
 <references>
-<li>R. Chen, <a href="http://blogs.msdn.com/oldnewthing/archive/2004/05/07/127826.aspx">When should your destructor be virtual?</a>.</li>
+<li>R. Chen, <a href="https://devblogs.microsoft.com/oldnewthing/20040507-00/?p=39443">When should your destructor be virtual?</a>.</li>
 <li>S. Meyers. <em>Effective C++ 3d ed.</em> pp 40-44. Addison-Wesley Professional, 2005.</li>
 </references>
 </qhelp>

cpp/ql/src/Likely Bugs/OO/UnsafeUseOfThis.cpp

@@ -0,0 +1,20 @@
+class Base {
+private:
+    // pure virtual member function used for initialization of derived classes.
+    virtual void construct() = 0;
+public:
+    Base() {
+        // wrong: the virtual table of `Derived` has not been initialized yet. So this
+        // call will resolve to `Base::construct`, which cannot be called as it is a pure
+        // virtual function.
+        construct();
+    }
+};
+
+class Derived : public Base {
+    int field;
+
+    void construct() override {
+        field = 1;
+    }
+};

cpp/ql/src/Likely Bugs/OO/UnsafeUseOfThis.qhelp

@@ -0,0 +1,30 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+
+
+<overview>
+<p>This rule finds calls to pure virtual member functions in constructors and destructors. When executing the body of a constructor of class <code>T</code>, the virtual table of <code>T</code> refers to the virtual table of one of <code>T</code>'s base classes. This can produce unexpected behavior, including program abort that can lead to denial of service attacks. The same problem exists during destruction of an object.</p>
+
+</overview>
+<recommendation>
+<p>Do not rely on virtual dispatch in constructors and destructors. Instead, each class should be responsible for acquiring and releasing its resources. If a base class needs to refer to a derived class during initialization, use the Dynamic Binding During Initialization idiom.</p>
+
+</recommendation>
+<example><sample src="UnsafeUseOfThis.cpp" />
+
+
+
+</example>
+<references>
+
+<li>ISO C++ FAQ: <a href="https://isocpp.org/wiki/faq/strange-inheritance#calling-virtuals-from-ctors">When my base class's constructor calls a virtual function on its this object, why doesn't my derived class's override of that virtual function get invoked?</a>
+</li>
+<li>SEI CERT C++ Coding Standard <a href="https://wiki.sei.cmu.edu/confluence/display/cplusplus/OOP50-CPP.+Do+not+invoke+virtual+functions+from+constructors+or+destructors">OOP50-CPP. Do not invoke virtual functions from constructors or destructors</a>
+</li>
+<li>ISO C++ FAQ: <a href="https://isocpp.org/wiki/faq/strange-inheritance#calling-virtuals-from-ctor-idiom">Okay, but is there a way to simulate that behavior as if dynamic binding worked on the this object within my base class's constructor?</a>
+</li>
+
+
+</references></qhelp>

cpp/ql/src/Likely Bugs/OO/UnsafeUseOfThis.ql

@@ -0,0 +1,212 @@
+/**
+ * @name Unsafe use of this in constructor
+ * @description A call to a pure virtual function using a 'this'
+ *              pointer of an object that is under construction
+ *              may lead to undefined behavior.
+ * @kind path-problem
+ * @id cpp/unsafe-use-of-this
+ * @problem.severity error
+ * @precision very-high
+ * @tags correctness
+ *       language-features
+ *       security
+ */
+
+import cpp
+// We don't actually use the global value numbering library in this query, but without it we end up
+// recomputing the IR.
+private import semmle.code.cpp.valuenumbering.GlobalValueNumbering
+private import semmle.code.cpp.ir.IR
+
+bindingset[n, result]
+int unbind(int n) { result >= n and result <= n }
+
+/** Holds if `p` is the `n`'th parameter of the non-virtual function `f`. */
+predicate parameterOf(Parameter p, Function f, int n) {
+  not f.isVirtual() and f.getParameter(n) = p
+}
+
+/**
+ * Holds if `instr` is the `n`'th argument to a call to the non-virtual function `f`, and
+ * `init` is the corresponding initiazation instruction that receives the value of `instr` in `f`.
+ */
+predicate flowIntoParameter(
+  CallInstruction call, Instruction instr, Function f, int n, InitializeParameterInstruction init
+) {
+  not f.isVirtual() and
+  call.getPositionalArgument(n) = instr and
+  f = call.getStaticCallTarget() and
+  getEnclosingNonVirtualFunctionInitializeParameter(init, f) and
+  init.getParameter().getIndex() = unbind(n)
+}
+
+/**
+ * Holds if `instr` is an argument to a call to the function `f`, and `init` is the
+ * corresponding initialization instruction that receives the value of `instr` in `f`.
+ */
+pragma[noinline]
+predicate getPositionalArgumentInitParam(
+  CallInstruction call, Instruction instr, InitializeParameterInstruction init, Function f
+) {
+  exists(int n |
+    parameterOf(_, f, n) and
+    flowIntoParameter(call, instr, f, unbind(n), init)
+  )
+}
+
+/**
+ * Holds if `instr` is the qualifier to a call to the non-virtual function `f`, and
+ * `init` is the corresponding initiazation instruction that receives the value of
+ * `instr` in `f`.
+ */
+pragma[noinline]
+predicate getThisArgumentInitParam(
+  CallInstruction call, Instruction instr, InitializeParameterInstruction init, Function f
+) {
+  not f.isVirtual() and
+  call.getStaticCallTarget() = f and
+  getEnclosingNonVirtualFunctionInitializeParameter(init, f) and
+  call.getThisArgument() = instr and
+  init.getIRVariable() instanceof IRThisVariable
+}
+
+/** Holds if `instr` is a `this` pointer used by the call instruction `call`. */
+predicate isSink(Instruction instr, CallInstruction call) {
+  exists(PureVirtualFunction func |
+    call.getStaticCallTarget() = func and
+    call.getThisArgument() = instr and
+    // Weed out implicit calls to destructors of a base class
+    not func instanceof Destructor
+  )
+}
+
+/** Holds if `init` initializes the `this` pointer in class `c`. */
+predicate isSource(InitializeParameterInstruction init, string msg, Class c) {
+  (
+    exists(Constructor func |
+      not func instanceof CopyConstructor and
+      not func instanceof MoveConstructor and
+      func = init.getEnclosingFunction() and
+      msg = "construction"
+    )
+    or
+    init.getEnclosingFunction() instanceof Destructor and msg = "destruction"
+  ) and
+  init.getIRVariable() instanceof IRThisVariable and
+  init.getEnclosingFunction().getDeclaringType() = c
+}
+
+/**
+ * Holds if `instr` flows to a sink (which is a use of the value of `instr` as a `this` pointer).
+ */
+predicate flowsToSink(Instruction instr, Instruction sink) {
+  flowsFromSource(instr) and
+  (
+    isSink(instr, _) and instr = sink
+    or
+    exists(Instruction mid |
+      successor(instr, mid) and
+      flowsToSink(mid, sink)
+    )
+  )
+}
+
+/** Holds if `instr` flows from a source. */
+predicate flowsFromSource(Instruction instr) {
+  isSource(instr, _, _)
+  or
+  exists(Instruction mid |
+    successor(mid, instr) and
+    flowsFromSource(mid)
+  )
+}
+
+/** Holds if `f` is the enclosing non-virtual function of `init`. */
+predicate getEnclosingNonVirtualFunctionInitializeParameter(
+  InitializeParameterInstruction init, Function f
+) {
+  not f.isVirtual() and
+  init.getEnclosingFunction() = f
+}
+
+/** Holds if `f` is the enclosing non-virtual function of `init`. */
+predicate getEnclosingNonVirtualFunctionInitializeIndirection(
+  InitializeIndirectionInstruction init, Function f
+) {
+  not f.isVirtual() and
+  init.getEnclosingFunction() = f
+}
+
+/**
+ * Holds if `instr` is an argument (or argument indirection) to a call, and
+ * `succ` is the corresponding initialization instruction in the call target.
+ */
+predicate flowThroughCallable(Instruction instr, Instruction succ) {
+  // Flow from an argument to a parameter
+  exists(CallInstruction call, InitializeParameterInstruction init | init = succ |
+    getPositionalArgumentInitParam(call, instr, init, call.getStaticCallTarget())
+    or
+    getThisArgumentInitParam(call, instr, init, call.getStaticCallTarget())
+  )
+  or
+  // Flow from argument indirection to parameter indirection
+  exists(
+    CallInstruction call, ReadSideEffectInstruction read, InitializeIndirectionInstruction init
+  |
+    init = succ and
+    read.getPrimaryInstruction() = call and
+    getEnclosingNonVirtualFunctionInitializeIndirection(init, call.getStaticCallTarget())
+  |
+    exists(int n |
+      read.getSideEffectOperand().getAnyDef() = instr and
+      read.getIndex() = n and
+      init.getParameter().getIndex() = unbind(n)
+    )
+    or
+    call.getThisArgument() = instr and
+    init.getIRVariable() instanceof IRThisVariable
+  )
+}
+
+/** Holds if `instr` flows to `succ`. */
+predicate successor(Instruction instr, Instruction succ) {
+  succ.(CopyInstruction).getSourceValue() = instr or
+  succ.(CheckedConvertOrNullInstruction).getUnary() = instr or
+  succ.(ChiInstruction).getTotal() = instr or
+  succ.(ConvertInstruction).getUnary() = instr or
+  succ.(InheritanceConversionInstruction).getUnary() = instr or
+  flowThroughCallable(instr, succ)
+}
+
+/**
+ * Holds if:
+ * - `source` is an initialization of a `this` pointer of type `sourceClass`, and
+ * - `sink` is a use of the `this` pointer, and
+ * - `call` invokes a pure virtual function using `sink` as the `this` pointer, and
+ * - `msg` is a string describing whether `source` is from a constructor or destructor.
+ */
+predicate flows(
+  Instruction source, string msg, Class sourceClass, Instruction sink, CallInstruction call
+) {
+  isSource(source, msg, sourceClass) and
+  flowsToSink(source, sink) and
+  isSink(sink, call)
+}
+
+query predicate edges(Instruction a, Instruction b) { successor(a, b) and flowsToSink(b, _) }
+
+query predicate nodes(Instruction n, string key, string val) {
+  flowsToSink(n, _) and
+  key = "semmle.label" and
+  val = n.toString()
+}
+
+from Instruction source, Instruction sink, CallInstruction call, string msg, Class sourceClass
+where
+  flows(source, msg, sourceClass, sink, call) and
+  // Only raise an alert if there is no override of the pure virtual function in any base class.
+  not exists(Class c | c = sourceClass.getABaseClass*() |
+    c.getAMemberFunction().getAnOverriddenFunction() = call.getStaticCallTarget()
+  )
+select call.getUnconvertedResultExpression(), source, sink,
+  "Call to pure virtual function during " + msg

cpp/ql/src/Metrics/Dependencies/ExternalDependencies.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name External dependencies
  * @description Count the number of dependencies a C/C++ source file has on external libraries.
  * @kind treemap

cpp/ql/src/Metrics/Dependencies/ExternalDependenciesSourceLinks.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name External dependency source links
  * @kind source-link
  * @metricType externalDependency

cpp/ql/src/Metrics/Files/FLinesOfDuplicatedCode.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name Duplicated lines in files
  * @description The number of lines in a file, including code, comment
  *              and whitespace lines, which are duplicated in at least

cpp/ql/src/Metrics/Files/FMacroRatio.qhelp

@@ -27,7 +27,7 @@ and IDE support than macros.</p>
 <references>
 
 <li>
-  <a href="http://msdn.microsoft.com/en-us/library/503x3e3s%28v=vs.80%29.aspx">Macros</a>
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/preprocessor/macros-c-cpp">Macros (C/C++)</a>
 </li>
 <li>
   <a href="http://www.stroustrup.com/icsm-2012-demacro.pdf">Rejuvenating C++ Programs through

cpp/ql/src/Metrics/Files/FTimeInFrontend.qhelp

@@ -15,7 +15,7 @@
 <references>
 
 <li>
-  <a href="http://msdn.microsoft.com/en-us/library/36k2cdd4%28v=VS.80%29.aspx">The #include Directive</a>
+  MSDN Library: <a href="https://docs.microsoft.com/en-us/cpp/preprocessor/hash-include-directive-c-cpp">#include directive (C/C++)</a>
 </li>
 <li>
   <a href="http://gcc.gnu.org/onlinedocs/cpp/Include-Operation.html#Include-Operation">Include operation</a>

cpp/ql/src/Microsoft/SAL.qll

@@ -10,13 +10,8 @@ import cpp
  */
 class SALMacro extends Macro {
   SALMacro() {
-    exists(string filename | filename = this.getFile().getBaseName() |
-      filename = "sal.h" or
-      filename = "specstrings_strict.h" or
-      filename = "specstrings.h" or
-      filename = "w32p.h" or
-      filename = "minwindef.h"
-    ) and
+    this.getFile().getBaseName() =
+      ["sal.h", "specstrings_strict.h", "specstrings.h", "w32p.h", "minwindef.h"] and
     (
       // Dialect for Windows 8 and above
       this.getName().matches("\\_%\\_")
@@ -58,10 +53,7 @@ class SALAnnotation extends MacroInvocation {
  */
 class SALCheckReturn extends SALAnnotation {
   SALCheckReturn() {
-    exists(SALMacro m | m = this.getMacro() |
-      m.getName() = "_Check_return_" or
-      m.getName() = "_Must_inspect_result_"
-    )
+    this.getMacro().(SALMacro).getName() = ["_Check_return_", "_Must_inspect_result_"]
   }
 }
 

cpp/ql/src/Security/CWE/CWE-020/CountUntrustedDataToExternalAPI.qhelp

@@ -0,0 +1,48 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<overview>
+<p>Using unsanitized untrusted data in an external API can cause a variety of security issues. This query reports 
+all external APIs that are used with untrusted data, along with how frequently the API is used, and how many 
+unique sources of untrusted data flow to this API. This query is designed primarily to help identify which APIs 
+may be relevant for security analysis of this application.</p>
+
+<p>An external API is defined as a call to a function that is not defined in the source code, and is not
+modeled as a taint step in the default taint library. External APIs may be from the C++ standard library,
+third party dependencies or from internal dependencies. The query will report the function name, along with
+either <code>[param x]</code>, where <code>x</code> indicates the position of the parameter receiving the
+untrusted data or <code>[qualifier]</code> indicating the untrusted data is used as the qualifier to the
+function call.</p>
+
+</overview>
+<recommendation>
+
+<p>For each result:</p>
+
+<ul>
+  <li>If the result highlights a known sink, no action is required.</li>
+  <li>If the result highlights an unknown sink for a problem, then add modeling for the sink to the relevant query.</li>
+  <li>If the result represents a call to an external API which transfers taint, add the appropriate modeling, and 
+  re-run the query to determine what new results have appeared due to this additional modeling.</li>
+</ul>
+
+<p>Otherwise, the result is likely uninteresting. Custom versions of this query can extend the <code>SafeExternalAPIFunction</code> 
+class to exclude known safe external APIs from future analysis.</p>
+
+</recommendation>
+<example>
+
+<p>If the query were to return the API <code>fputs [param 1]</code> 
+then we should first consider whether this a security relevant sink. In this case, this is writing to a <code>FILE*</code>, so we should 
+consider whether this is an XSS sink. If it is, we should confirm that it is handled by the XSS query.</p>
+
+<p>If the query were to return the API <code>strcat [param 1]</code>, then this should be
+reviewed as a possible taint step, because tainted data would flow from the 1st argument to the 0th argument of the call.</p>
+
+<p>Note that both examples are correctly handled by the standard taint tracking library and XSS query.</p>
+</example>
+<references>
+
+</references>
+</qhelp>

cpp/ql/src/Security/CWE/CWE-020/CountUntrustedDataToExternalAPI.ql

@@ -0,0 +1,17 @@
+/**
+ * @name Frequency counts for external APIs that are used with untrusted data
+ * @description This reports the external APIs that are used with untrusted data, along with how
+ *              frequently the API is called, and how many unique sources of untrusted data flow
+ *              to it.
+ * @id cpp/count-untrusted-data-external-api
+ * @kind table
+ * @tags security external/cwe/cwe-20
+ */
+
+import cpp
+import ExternalAPIs
+
+from ExternalAPIUsedWithUntrustedData externalAPI
+select externalAPI, count(externalAPI.getUntrustedDataNode()) as numberOfUses,
+  externalAPI.getNumberOfUntrustedSources() as numberOfUntrustedSources order by
+    numberOfUntrustedSources desc

cpp/ql/src/Security/CWE/CWE-020/ExternalAPISinkExample.cpp

@@ -0,0 +1,13 @@
+#include <cstdio>
+
+void do_get(FILE* request, FILE* response) {
+  char page[1024];
+  fgets(page, 1024, request);
+
+  char buffer[1024];
+  strcat(buffer, "The page \"");
+  strcat(buffer, page);
+  strcat(buffer, "\" was not found.");
+
+  fputs(buffer, response);
+}

cpp/ql/src/Security/CWE/CWE-020/ExternalAPITaintStepExample.cpp

@@ -0,0 +1,13 @@
+#include <cstdio>
+
+void do_get(FILE* request, FILE* response) {
+  char user_id[1024];
+  fgets(user_id, 1024, request);
+
+  char buffer[1024];
+  strcat(buffer, "SELECT * FROM user WHERE user_id='");
+  strcat(buffer, user_id);
+  strcat(buffer, "'");
+
+  // ...
+}

cpp/ql/src/Security/CWE/CWE-020/ExternalAPIs.qll

@@ -0,0 +1,50 @@
+/**
+ * Definitions for reasoning about untrusted data used in APIs defined outside the
+ * database.
+ */
+
+private import cpp
+private import semmle.code.cpp.models.interfaces.DataFlow
+private import semmle.code.cpp.models.interfaces.Taint
+import ExternalAPIsSpecific
+
+/** A node representing untrusted data being passed to an external API. */
+class UntrustedExternalAPIDataNode extends ExternalAPIDataNode {
+  UntrustedExternalAPIDataNode() { any(UntrustedDataToExternalAPIConfig c).hasFlow(_, this) }
+
+  /** Gets a source of untrusted data which is passed to this external API data node. */
+  DataFlow::Node getAnUntrustedSource() {
+    any(UntrustedDataToExternalAPIConfig c).hasFlow(result, this)
+  }
+}
+
+private newtype TExternalAPI =
+  TExternalAPIParameter(Function f, int index) {
+    exists(UntrustedExternalAPIDataNode n |
+      f = n.getExternalFunction() and
+      index = n.getIndex()
+    )
+  }
+
+/** An external API which is used with untrusted data. */
+class ExternalAPIUsedWithUntrustedData extends TExternalAPI {
+  /** Gets a possibly untrusted use of this external API. */
+  UntrustedExternalAPIDataNode getUntrustedDataNode() {
+    this = TExternalAPIParameter(result.getExternalFunction(), result.getIndex())
+  }
+
+  /** Gets the number of untrusted sources used with this external API. */
+  int getNumberOfUntrustedSources() {
+    result = strictcount(getUntrustedDataNode().getAnUntrustedSource())
+  }
+
+  /** Gets a textual representation of this element. */
+  string toString() {
+    exists(Function f, int index, string indexString |
+      if index = -1 then indexString = "qualifier" else indexString = "param " + index
+    |
+      this = TExternalAPIParameter(f, index) and
+      result = f.toString() + " [" + indexString + "]"
+    )
+  }
+}

cpp/ql/src/Security/CWE/CWE-020/ExternalAPIsSpecific.qll

@@ -0,0 +1,56 @@
+/**
+ * Provides AST-specific definitions for use in the `ExternalAPI` library.
+ */
+
+import semmle.code.cpp.dataflow.TaintTracking
+import semmle.code.cpp.models.interfaces.FlowSource
+import semmle.code.cpp.models.interfaces.DataFlow
+import SafeExternalAPIFunction
+
+/** A node representing untrusted data being passed to an external API. */
+class ExternalAPIDataNode extends DataFlow::Node {
+  Call call;
+  int i;
+
+  ExternalAPIDataNode() {
+    // Argument to call to a function
+    (
+      this.asExpr() = call.getArgument(i)
+      or
+      i = -1 and this.asExpr() = call.getQualifier()
+    ) and
+    exists(Function f |
+      f = call.getTarget() and
+      // Defined outside the source archive
+      not f.hasDefinition() and
+      // Not already modeled as a dataflow or taint step
+      not f instanceof DataFlowFunction and
+      not f instanceof TaintFunction and
+      // Not a call to a known safe external API
+      not f instanceof SafeExternalAPIFunction
+    )
+  }
+
+  /** Gets the called API `Function`. */
+  Function getExternalFunction() { result = call.getTarget() }
+
+  /** Gets the index which is passed untrusted data (where -1 indicates the qualifier). */
+  int getIndex() { result = i }
+
+  /** Gets the description of the function being called. */
+  string getFunctionDescription() { result = getExternalFunction().toString() }
+}
+
+/** A configuration for tracking flow from `RemoteFlowSource`s to `ExternalAPIDataNode`s. */
+class UntrustedDataToExternalAPIConfig extends TaintTracking::Configuration {
+  UntrustedDataToExternalAPIConfig() { this = "UntrustedDataToExternalAPIConfig" }
+
+  override predicate isSource(DataFlow::Node source) {
+    exists(RemoteFlowFunction remoteFlow |
+      remoteFlow = source.asExpr().(Call).getTarget() and
+      remoteFlow.hasRemoteFlowSource(_, _)
+    )
+  }
+
+  override predicate isSink(DataFlow::Node sink) { sink instanceof ExternalAPIDataNode }
+}

cpp/ql/src/Security/CWE/CWE-020/IRCountUntrustedDataToExternalAPI.qhelp

@@ -0,0 +1,48 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<overview>
+<p>Using unsanitized untrusted data in an external API can cause a variety of security issues. This query reports 
+all external APIs that are used with untrusted data, along with how frequently the API is used, and how many 
+unique sources of untrusted data flow to this API. This query is designed primarily to help identify which APIs 
+may be relevant for security analysis of this application.</p>
+
+<p>An external API is defined as a call to a function that is not defined in the source code, and is not
+modeled as a taint step in the default taint library. External APIs may be from the C++ standard library,
+third party dependencies or from internal dependencies. The query will report the function name, along with
+either <code>[param x]</code>, where <code>x</code> indicates the position of the parameter receiving the
+untrusted data or <code>[qualifier]</code> indicating the untrusted data is used as the qualifier to the
+function call.</p>
+
+</overview>
+<recommendation>
+
+<p>For each result:</p>
+
+<ul>
+  <li>If the result highlights a known sink, no action is required.</li>
+  <li>If the result highlights an unknown sink for a problem, then add modeling for the sink to the relevant query.</li>
+  <li>If the result represents a call to an external API which transfers taint, add the appropriate modeling, and 
+  re-run the query to determine what new results have appeared due to this additional modeling.</li>
+</ul>
+
+<p>Otherwise, the result is likely uninteresting. Custom versions of this query can extend the <code>SafeExternalAPIFunction</code> 
+class to exclude known safe external APIs from future analysis.</p>
+
+</recommendation>
+<example>
+
+<p>If the query were to return the API <code>fputs [param 1]</code> 
+then we should first consider whether this a security relevant sink. In this case, this is writing to a <code>FILE*</code>, so we should 
+consider whether this is an XSS sink. If it is, we should confirm that it is handled by the XSS query.</p>
+
+<p>If the query were to return the API <code>strcat [param 1]</code>, then this should be
+reviewed as a possible taint step, because tainted data would flow from the 1st argument to the 0th argument of the call.</p>
+
+<p>Note that both examples are correctly handled by the standard taint tracking library and XSS query.</p>
+</example>
+<references>
+
+</references>
+</qhelp>

cpp/ql/src/Security/CWE/CWE-020/IRCountUntrustedDataToExternalAPI.ql

@@ -0,0 +1,17 @@
+/**
+ * @name Frequency counts for external APIs that are used with untrusted data
+ * @description This reports the external APIs that are used with untrusted data, along with how
+ *              frequently the API is called, and how many unique sources of untrusted data flow
+ *              to it.
+ * @id cpp/count-untrusted-data-external-api-ir
+ * @kind table
+ * @tags security external/cwe/cwe-20
+ */
+
+import cpp
+import ir.ExternalAPIs
+
+from ExternalAPIUsedWithUntrustedData externalAPI
+select externalAPI, count(externalAPI.getUntrustedDataNode()) as numberOfUses,
+  externalAPI.getNumberOfUntrustedSources() as numberOfUntrustedSources order by
+    numberOfUntrustedSources desc

cpp/ql/src/Security/CWE/CWE-020/IRUntrustedDataToExternalAPI.qhelp

@@ -0,0 +1,59 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<overview>
+<p>Using unsanitized untrusted data in an external API can cause a variety of security issues. This query reports 
+external APIs that use untrusted data. The results are not filtered, so you can audit all examples.
+The query provides data for security reviews of the application and you can also use it to identify external APIs
+that should be modeled as either taint steps, or sinks for specific problems.</p>
+
+<p>An external API is defined as a call to a function that is not defined in the source code, and is not modeled
+as a taint step in the default taint library. External APIs may be from the
+C++ standard library, third-party dependencies or from internal dependencies. The query reports uses of 
+untrusted data in either the qualifier or as one of the arguments of external APIs.</p>
+
+</overview>
+<recommendation>
+
+<p>For each result:</p>
+
+<ul>
+  <li>If the result highlights a known sink, confirm that the result is reported by the relevant query, or 
+  that the result is a false positive because this data is sanitized.</li>
+  <li>If the result highlights an unknown sink for a problem, then add modeling for the sink to the relevant query, 
+  and confirm that the result is either found, or is safe due to appropriate sanitization.</li>
+  <li>If the result represents a call to an external API that transfers taint, add the appropriate modeling, and 
+  re-run the query to determine what new results have appeared due to this additional modeling.</li>
+</ul>
+
+<p>Otherwise, the result is likely uninteresting. Custom versions of this query can extend the <code>SafeExternalAPIFunction</code> 
+class to exclude known safe external APIs from future analysis.</p>
+
+</recommendation>
+<example>
+
+<p>In this first example, input is read from <code>fgets</code> and then ultimately used in a call to the 
+<code>fputs</code> external API:</p>
+
+<sample src="ExternalAPISinkExample.cpp" />
+
+<p>This is an XSS sink. The XSS query should therefore be reviewed to confirm that this sink is appropriately modeled, 
+and if it is, to confirm that the query reports this particular result, or that the result is a false positive due to 
+some existing sanitization.</p>
+
+<p>In this second example, again a request parameter is read from <code>fgets</code>.</p>
+
+<sample src="ExternalAPITaintStepExample.cpp" />
+
+<p>If the query reported the call to <code>strcat</code> on line 9, this would suggest that this external API is 
+not currently modeled as a taint step in the taint tracking library. The next step would be to model this as a taint step, then
+re-run the query to determine what additional results might be found. In this example, it seems likely that <code>buffer</code>
+will be executed as an SQL query, potentially leading to an SQL injection vulnerability.</p>
+
+<p>Note that both examples are correctly handled by the standard taint tracking library and XSS query.</p>
+</example>
+<references>
+
+</references>
+</qhelp>

cpp/ql/src/Security/CWE/CWE-020/IRUntrustedDataToExternalAPI.ql

@@ -0,0 +1,21 @@
+/**
+ * @name Untrusted data passed to external API
+ * @description Data provided remotely is used in this external API without sanitization, which could be a security risk.
+ * @id cpp/untrusted-data-to-external-api-ir
+ * @kind path-problem
+ * @precision low
+ * @problem.severity error
+ * @tags security external/cwe/cwe-20
+ */
+
+import cpp
+import semmle.code.cpp.ir.dataflow.TaintTracking
+import ir.ExternalAPIs
+import semmle.code.cpp.security.FlowSources
+import DataFlow::PathGraph
+
+from UntrustedDataToExternalAPIConfig config, DataFlow::PathNode source, DataFlow::PathNode sink
+where config.hasFlowPath(source, sink)
+select sink, source, sink,
+  "Call to " + sink.getNode().(ExternalAPIDataNode).getExternalFunction().toString() +
+    " with untrusted data from $@.", source, source.getNode().(RemoteFlowSource).getSourceType()

cpp/ql/src/Security/CWE/CWE-020/SafeExternalAPIFunction.qll

@@ -0,0 +1,24 @@
+/**
+ * Provides a class for modeling external functions that are "safe" from a security perspective.
+ */
+
+private import cpp
+private import semmle.code.cpp.models.interfaces.SideEffect
+
+/**
+ * A `Function` that is considered a "safe" external API from a security perspective.
+ */
+abstract class SafeExternalAPIFunction extends Function { }
+
+/** The default set of "safe" external APIs. */
+private class DefaultSafeExternalAPIFunction extends SafeExternalAPIFunction {
+  DefaultSafeExternalAPIFunction() {
+    // If a function does not write to any of its arguments, we consider it safe to
+    // pass untrusted data to it. This means that string functions such as `strcmp`
+    // and `strlen`, as well as memory functions such as `memcmp`, are considered safe.
+    exists(SideEffectFunction model | model = this |
+      model.hasOnlySpecificWriteSideEffects() and
+      not model.hasSpecificWriteSideEffect(_, _, _)
+    )
+  }
+}

cpp/ql/src/Security/CWE/CWE-020/UntrustedDataToExternalAPI.qhelp

@@ -0,0 +1,59 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<overview>
+<p>Using unsanitized untrusted data in an external API can cause a variety of security issues. This query reports 
+external APIs that use untrusted data. The results are not filtered, so you can audit all examples.
+The query provides data for security reviews of the application and you can also use it to identify external APIs
+that should be modeled as either taint steps, or sinks for specific problems.</p>
+
+<p>An external API is defined as a call to a function that is not defined in the source code, and is not modeled
+as a taint step in the default taint library. External APIs may be from the
+C++ standard library, third-party dependencies or from internal dependencies. The query reports uses of 
+untrusted data in either the qualifier or as one of the arguments of external APIs.</p>
+
+</overview>
+<recommendation>
+
+<p>For each result:</p>
+
+<ul>
+  <li>If the result highlights a known sink, confirm that the result is reported by the relevant query, or 
+  that the result is a false positive because this data is sanitized.</li>
+  <li>If the result highlights an unknown sink for a problem, then add modeling for the sink to the relevant query, 
+  and confirm that the result is either found, or is safe due to appropriate sanitization.</li>
+  <li>If the result represents a call to an external API that transfers taint, add the appropriate modeling, and 
+  re-run the query to determine what new results have appeared due to this additional modeling.</li>
+</ul>
+
+<p>Otherwise, the result is likely uninteresting. Custom versions of this query can extend the <code>SafeExternalAPIFunction</code> 
+class to exclude known safe external APIs from future analysis.</p>
+
+</recommendation>
+<example>
+
+<p>In this first example, input is read from <code>fgets</code> and then ultimately used in a call to the 
+<code>fputs</code> external API:</p>
+
+<sample src="ExternalAPISinkExample.cpp" />
+
+<p>This is an XSS sink. The XSS query should therefore be reviewed to confirm that this sink is appropriately modeled, 
+and if it is, to confirm that the query reports this particular result, or that the result is a false positive due to 
+some existing sanitization.</p>
+
+<p>In this second example, again a request parameter is read from <code>fgets</code>.</p>
+
+<sample src="ExternalAPITaintStepExample.cpp" />
+
+<p>If the query reported the call to <code>strcat</code> on line 9, this would suggest that this external API is 
+not currently modeled as a taint step in the taint tracking library. The next step would be to model this as a taint step, then
+re-run the query to determine what additional results might be found. In this example, it seems likely that <code>buffer</code>
+will be executed as an SQL query, potentially leading to an SQL injection vulnerability.</p>
+
+<p>Note that both examples are correctly handled by the standard taint tracking library and XSS query.</p>
+</example>
+<references>
+
+</references>
+</qhelp>

cpp/ql/src/Security/CWE/CWE-020/UntrustedDataToExternalAPI.ql

@@ -0,0 +1,20 @@
+/**
+ * @name Untrusted data passed to external API
+ * @description Data provided remotely is used in this external API without sanitization, which could be a security risk.
+ * @id cpp/untrusted-data-to-external-api
+ * @kind path-problem
+ * @precision low
+ * @problem.severity error
+ * @tags security external/cwe/cwe-20
+ */
+
+import cpp
+import semmle.code.cpp.dataflow.TaintTracking
+import ExternalAPIs
+import DataFlow::PathGraph
+
+from UntrustedDataToExternalAPIConfig config, DataFlow::PathNode source, DataFlow::PathNode sink
+where config.hasFlowPath(source, sink)
+select sink, source, sink,
+  "Call to " + sink.getNode().(ExternalAPIDataNode).getExternalFunction().toString() +
+    " with untrusted data from $@.", source, source.toString()

cpp/ql/src/Security/CWE/CWE-020/ir/ExternalAPIs.qll

@@ -0,0 +1,50 @@
+/**
+ * Definitions for reasoning about untrusted data used in APIs defined outside the
+ * database.
+ */
+
+private import cpp
+private import semmle.code.cpp.models.interfaces.DataFlow
+private import semmle.code.cpp.models.interfaces.Taint
+import ExternalAPIsSpecific
+
+/** A node representing untrusted data being passed to an external API. */
+class UntrustedExternalAPIDataNode extends ExternalAPIDataNode {
+  UntrustedExternalAPIDataNode() { any(UntrustedDataToExternalAPIConfig c).hasFlow(_, this) }
+
+  /** Gets a source of untrusted data which is passed to this external API data node. */
+  DataFlow::Node getAnUntrustedSource() {
+    any(UntrustedDataToExternalAPIConfig c).hasFlow(result, this)
+  }
+}
+
+private newtype TExternalAPI =
+  TExternalAPIParameter(Function f, int index) {
+    exists(UntrustedExternalAPIDataNode n |
+      f = n.getExternalFunction() and
+      index = n.getIndex()
+    )
+  }
+
+/** An external API which is used with untrusted data. */
+class ExternalAPIUsedWithUntrustedData extends TExternalAPI {
+  /** Gets a possibly untrusted use of this external API. */
+  UntrustedExternalAPIDataNode getUntrustedDataNode() {
+    this = TExternalAPIParameter(result.getExternalFunction(), result.getIndex())
+  }
+
+  /** Gets the number of untrusted sources used with this external API. */
+  int getNumberOfUntrustedSources() {
+    result = strictcount(getUntrustedDataNode().getAnUntrustedSource())
+  }
+
+  /** Gets a textual representation of this element. */
+  string toString() {
+    exists(Function f, int index, string indexString |
+      if index = -1 then indexString = "qualifier" else indexString = "param " + index
+    |
+      this = TExternalAPIParameter(f, index) and
+      result = f.toString() + " [" + indexString + "]"
+    )
+  }
+}

cpp/ql/src/Security/CWE/CWE-020/ir/ExternalAPIsSpecific.qll

@@ -0,0 +1,51 @@
+/**
+ * Provides IR-specific definitions for use in the `ExternalAPI` library.
+ */
+
+import semmle.code.cpp.ir.dataflow.TaintTracking
+private import semmle.code.cpp.security.FlowSources
+private import semmle.code.cpp.models.interfaces.DataFlow
+import SafeExternalAPIFunction
+
+/** A node representing untrusted data being passed to an external API. */
+class ExternalAPIDataNode extends DataFlow::Node {
+  Call call;
+  int i;
+
+  ExternalAPIDataNode() {
+    // Argument to call to a function
+    (
+      this.asExpr() = call.getArgument(i)
+      or
+      i = -1 and this.asExpr() = call.getQualifier()
+    ) and
+    exists(Function f |
+      f = call.getTarget() and
+      // Defined outside the source archive
+      not f.hasDefinition() and
+      // Not already modeled as a dataflow or taint step
+      not f instanceof DataFlowFunction and
+      not f instanceof TaintFunction and
+      // Not a call to a known safe external API
+      not f instanceof SafeExternalAPIFunction
+    )
+  }
+
+  /** Gets the called API `Function`. */
+  Function getExternalFunction() { result = call.getTarget() }
+
+  /** Gets the index which is passed untrusted data (where -1 indicates the qualifier). */
+  int getIndex() { result = i }
+
+  /** Gets the description of the function being called. */
+  string getFunctionDescription() { result = getExternalFunction().toString() }
+}
+
+/** A configuration for tracking flow from `RemoteFlowSource`s to `ExternalAPIDataNode`s. */
+class UntrustedDataToExternalAPIConfig extends TaintTracking::Configuration {
+  UntrustedDataToExternalAPIConfig() { this = "UntrustedDataToExternalAPIConfigIR" }
+
+  override predicate isSource(DataFlow::Node source) { source instanceof RemoteFlowSource }
+
+  override predicate isSink(DataFlow::Node sink) { sink instanceof ExternalAPIDataNode }
+}

cpp/ql/src/Security/CWE/CWE-020/ir/SafeExternalAPIFunction.qll

@@ -0,0 +1,24 @@
+/**
+ * Provides a class for modeling external functions that are "safe" from a security perspective.
+ */
+
+private import cpp
+private import semmle.code.cpp.models.interfaces.SideEffect
+
+/**
+ * A `Function` that is considered a "safe" external API from a security perspective.
+ */
+abstract class SafeExternalAPIFunction extends Function { }
+
+/** The default set of "safe" external APIs. */
+private class DefaultSafeExternalAPIFunction extends SafeExternalAPIFunction {
+  DefaultSafeExternalAPIFunction() {
+    // If a function does not write to any of its arguments, we consider it safe to
+    // pass untrusted data to it. This means that string functions such as `strcmp`
+    // and `strlen`, as well as memory functions such as `memcmp`, are considered safe.
+    exists(SideEffectFunction model | model = this |
+      model.hasOnlySpecificWriteSideEffects() and
+      not model.hasSpecificWriteSideEffect(_, _, _)
+    )
+  }
+}

cpp/ql/src/Security/CWE/CWE-022/TaintedPath.qhelp

@@ -39,7 +39,7 @@ access all the system's passwords.</p>
 
 <li>
 OWASP:
-<a href="https://www.owasp.org/index.php/Path_traversal">Path Traversal</a>.
+<a href="https://owasp.org/www-community/attacks/Path_Traversal">Path Traversal</a>.
 </li>
 
 </references>

cpp/ql/src/Security/CWE/CWE-089/SqlTainted.qhelp

@@ -21,7 +21,7 @@ string before converting it to SQL.</p>
 </example>
 <references>
 
-<li>Microsoft Developer Network: <a href="http://msdn.microsoft.com/en-us/library/ms161953.aspx">SQL Injection</a>.</li>
+<li>MSDN Library: <a href="https://docs.microsoft.com/en-us/sql/relational-databases/security/sql-injection">SQL Injection</a>.</li>
 
 
 <!--  LocalWords:  SQL CWE

cpp/ql/src/Security/CWE/CWE-121/UnterminatedVarargsCall.ql

@@ -56,7 +56,7 @@ class VarargsFunction extends Function {
   }
 
   string normalTerminator(int cnt) {
-    (result = "0" or result = "-1") and
+    result = ["0", "-1"] and
     cnt = trailingArgValueCount(result) and
     2 * cnt > totalCount() and
     not exists(FunctionCall fc, int index |

cpp/ql/src/Security/CWE/CWE-457/InitializationFunctions.qll

@@ -189,8 +189,7 @@ class InitializationFunction extends Function {
       // Field wise assignment to the parameter
       any(Assignment e).getLValue() = getAFieldAccess(this.getParameter(i)) or
       i =
-        this
-            .(MemberFunction)
+        this.(MemberFunction)
             .getAnOverridingFunction+()
             .(InitializationFunction)
             .initializedParameter() or
@@ -475,12 +474,9 @@ class ConditionalInitializationCall extends FunctionCall {
       fa.getASuccessor+() = result
     ) and
     result =
-      this
-          .getArgument(getTarget(this)
-                .(ConditionalInitializationFunction)
-                .conditionallyInitializedParameter(_))
-          .(AddressOfExpr)
-          .getOperand()
+      this.getArgument(getTarget(this)
+            .(ConditionalInitializationFunction)
+            .conditionallyInitializedParameter(_)).(AddressOfExpr).getOperand()
   }
 
   Variable getStatusVariable() {

cpp/ql/src/Security/CWE/CWE-676/DangerousUseOfCin.ql

@@ -66,10 +66,7 @@ class IFStream extends Type {
  */
 class CinVariable extends NamespaceVariable {
   CinVariable() {
-    (
-      getName() = "cin" or
-      getName() = "wcin"
-    ) and
+    getName() = ["cin", "wcin"] and
     getNamespace().getName() = "std"
   }
 }

cpp/ql/src/Security/CWE/CWE-676/PotentiallyDangerousFunction.ql

@@ -14,12 +14,7 @@ import cpp
 
 predicate potentiallyDangerousFunction(Function f, string message) {
   exists(string name | f.hasGlobalName(name) |
-    (
-      name = "gmtime" or
-      name = "localtime" or
-      name = "ctime" or
-      name = "asctime"
-    ) and
+    name = ["gmtime", "localtime", "ctime", "asctime"] and
     message = "Call to " + name + " is potentially dangerous"
   )
 }

cpp/ql/src/Security/CWE/CWE-732/DoNotCreateWorldWritable.ql

@@ -19,12 +19,7 @@ predicate worldWritableCreation(FileCreationExpr fc, int mode) {
 }
 
 predicate setWorldWritable(FunctionCall fc, int mode) {
-  exists(string name | fc.getTarget().getName() = name |
-    name = "chmod" or
-    name = "fchmod" or
-    name = "_chmod" or
-    name = "_wchmod"
-  ) and
+  fc.getTarget().getName() = ["chmod", "fchmod", "_chmod", "_wchmod"] and
   mode = fc.getArgument(1).getValue().toInt() and
   sets(mode, s_iwoth())
 }

cpp/ql/src/Security/CWE/CWE-732/FilePermissions.qll

@@ -31,11 +31,7 @@ predicate sets(int mask, int fields) { mask.bitAnd(fields) != 0 }
  * one of the `umask` family of functions.
  */
 private int umask(FunctionCall fc) {
-  exists(string name | name = fc.getTarget().getName() |
-    name = "umask" or
-    name = "_umask" or
-    name = "_umask_s"
-  ) and
+  fc.getTarget().getName() = ["umask", "_umask", "_umask_s"] and
   result = fc.getArgument(0).getValue().toInt()
 }
 
@@ -89,11 +85,7 @@ abstract class FileCreationExpr extends FunctionCall {
 
 class OpenCreationExpr extends FileCreationExpr {
   OpenCreationExpr() {
-    exists(string name | name = this.getTarget().getName() |
-      name = "open" or
-      name = "_open" or
-      name = "_wopen"
-    ) and
+    this.getTarget().getName() = ["open", "_open", "_wopen"] and
     sets(this.getArgument(1).getValue().toInt(), o_creat())
   }
 
@@ -134,14 +126,9 @@ private int fopenMode() {
 
 class FopenCreationExpr extends FileCreationExpr {
   FopenCreationExpr() {
-    exists(string name | name = this.getTarget().getName() |
-      name = "fopen" or
-      name = "_wfopen" or
-      name = "fsopen" or
-      name = "_wfsopen"
-    ) and
+    this.getTarget().getName() = ["fopen", "_wfopen", "fsopen", "_wfsopen"] and
     exists(string mode |
-      (mode = "w" or mode = "a") and
+      mode = ["w", "a"] and
       this.getArgument(1).getValue().matches(mode + "%")
     )
   }

cpp/ql/src/codeql-suites/cpp-lgtm-full.qls

@@ -9,10 +9,7 @@
     tags contain:
       - ide-contextual-queries/local-definitions
       - ide-contextual-queries/local-references
-- query: Metrics/Dependencies/ExternalDependencies.ql
-- query: Metrics/Dependencies/ExternalDependenciesSourceLinks.ql
 - query: Metrics/Files/FLinesOfCode.ql
 - query: Metrics/Files/FLinesOfCommentedOutCode.ql
 - query: Metrics/Files/FLinesOfComments.ql
-- query: Metrics/Files/FLinesOfDuplicatedCode.ql
 - query: Metrics/Files/FNumberOfTests.ql

cpp/ql/src/definitions.ql

@@ -1,7 +1,7 @@
 /**
  * @name Jump-to-definition links
  * @description Generates use-definition pairs that provide the data
- *              for jump-to-definition in the code viewer.
+ *              for jump-to-definition in the code viewer of LGTM.
  * @kind definitions
  * @id cpp/jump-to-definition
  */
@@ -9,5 +9,10 @@
 import definitions
 
 from Top e, Top def, string kind
-where def = definitionOf(e, kind)
+where
+  def = definitionOf(e, kind) and
+  // We need to exclude definitions for elements inside template instantiations,
+  // as these often lead to multiple links to definitions from the same source location.
+  // LGTM does not support this bevaviour.
+  not e.isFromTemplateInstantiation(_)
 select e, def, kind

cpp/ql/src/definitions.qll

@@ -4,6 +4,7 @@
  */
 
 import cpp
+import IDEContextual
 
 /**
  * Any element that might be the source or target of a jump-to-definition
@@ -124,6 +125,7 @@ private predicate constructorCallTypeMention(ConstructorCall cc, TypeMention tm)
 
 /**
  * Gets an element, of kind `kind`, that element `e` uses, if any.
+ * Attention: This predicate yields multiple definitions for a single location.
  *
  * The `kind` is a string representing what kind of use it is:
  *  - `"M"` for function and method calls
@@ -196,15 +198,7 @@ Top definitionOf(Top e, string kind) {
     not e.(Element).isInMacroExpansion() and
     // exclude nested macro invocations, as they will overlap with
     // the top macro invocation.
-    not exists(e.(MacroAccess).getParentInvocation()) and
-    // exclude results from template instantiations, as:
-    // (1) these dependencies will often be caused by a choice of
-    // template parameter, which is non-local to this part of code; and
-    // (2) overlapping results pointing to different locations will
-    // be very common.
-    // It's possible we could allow a subset of these dependencies
-    // in future, if we're careful to ensure the above don't apply.
-    not e.isFromTemplateInstantiation(_)
+    not exists(e.(MacroAccess).getParentInvocation())
   ) and
   // Some entities have many locations. This can arise for an external
   // function that is frequently declared but not defined, or perhaps
@@ -214,11 +208,3 @@ Top definitionOf(Top e, string kind) {
   // later on.
   strictcount(result.getLocation()) < 10
 }
-
-/**
- * Returns an appropriately encoded version of a filename `name`
- * passed by the VS Code extension in order to coincide with the
- * output of `.getFile()` on locatable entities.
- */
-cached
-File getEncodedFile(string name) { result.getAbsolutePath().replaceAll(":", "_") = name }

cpp/ql/src/experimental/Security/CWE/CWE-191/UnsignedDifferenceExpressionComparedZero.c

@@ -0,0 +1,11 @@
+unsigned long sizeArray;
+
+// BAD: let's consider several values, taking ULONG_MAX =18446744073709551615
+// sizeArray = 60; (sizeArray - 10) = 50; true
+// sizeArray = 10; (sizeArray - 10) = 0; false
+// sizeArray = 1; (sizeArray - 10) = 18446744073709551607; true
+// sizeArray = 0; (sizeArray - 10) = 18446744073709551606; true
+if (sizeArray - 10 > 0)
+
+// GOOD: Prevent overflow by checking the input
+if (sizeArray > 10)

cpp/ql/src/experimental/Security/CWE/CWE-191/UnsignedDifferenceExpressionComparedZero.qhelp

@@ -0,0 +1,33 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<overview>
+<p>The code compares the unsigned difference with zero. 
+It is highly probable that the condition is wrong if the difference expression has the unsigned type.  
+The condition holds in all the cases when difference is not equal to zero. 
+It means that we may use condition not equal. But the programmer probably wanted to compare the difference of elements.</p>
+
+<p>False positives include code in which the first difference element is always greater than or equal to the second. 
+For comparison, ">" such conditions are equivalent to "! =", And are recommended for replacement. 
+For comparison "> =", the conditions are always true and are recommended to be excluded.</p>
+
+</overview>
+<recommendation>
+
+<p>Use a simple comparison of two elements, instead of comparing their difference to zero.</p>
+
+</recommendation>
+<example>
+<p>The following example demonstrates an erroneous and corrected use of comparison.</p>
+<sample src="UnsignedDifferenceExpressionComparedZero.c" />
+
+</example>
+<references>
+
+<li>CERT C Coding Standard:
+<a href="https://wiki.sei.cmu.edu/confluence/display/c/INT02-C.+Understand+integer+conversion+rules">INT02-C. Understand integer conversion rules</a>.
+</li>
+
+</references>
+</qhelp>

cpp/ql/src/experimental/Security/CWE/CWE-191/UnsignedDifferenceExpressionComparedZero.ql

@@ -0,0 +1,23 @@
+/**
+ * @name Unsigned difference expression compared to zero
+ * @description It is highly probable that the condition is wrong if the difference expression has the unsigned type.
+ *              The condition holds in all the cases when difference is not equal to zero. It means that we may use condition not equal.
+ *              But the programmer probably wanted to compare the difference of elements.
+ * @kind problem
+ * @id cpp/unsigned-difference-expression-compared-zero
+ * @problem.severity warning
+ * @precision medium
+ * @tags security
+ *       external/cwe/cwe-191
+ */
+
+import cpp
+import semmle.code.cpp.commons.Exclusions
+
+from RelationalOperation ro, SubExpr sub
+where
+  not isFromMacroDefinition(ro) and
+  ro.getLesserOperand().getValue().toInt() = 0 and
+  ro.getGreaterOperand() = sub and
+  sub.getFullyConverted().getUnspecifiedType().(IntegralType).isUnsigned()
+select ro, "Difference in condition is always greater than or equal to zero"

cpp/ql/src/experimental/semmle/code/cpp/models/interfaces/SimpleRangeAnalysisDefinition.qll

@@ -0,0 +1,65 @@
+/**
+ * EXPERIMENTAL: The API of this module may change without notice.
+ *
+ * Provides a class for modeling `RangeSsaDefinition`s with a restricted range.
+ */
+
+import cpp
+import semmle.code.cpp.rangeanalysis.SimpleRangeAnalysis
+
+/**
+ * EXPERIMENTAL: The API of this class may change without notice.
+ *
+ * An SSA definition for which a range can be deduced. As with
+ * `RangeSsaDefinition` and `SsaDefinition`, instances of this class
+ * correspond to points in the program where one or more variables are defined
+ * or have their value constrained in some way.
+ *
+ * Extend this class to add functionality to the range analysis library.
+ */
+abstract class SimpleRangeAnalysisDefinition extends RangeSsaDefinition {
+  /**
+   * Holds if this `SimpleRangeAnalysisDefinition` adds range information for
+   * `v`. Because a `SimpleRangeAnalysisDefinition` is just a point in the
+   * program, it's possible that more than one variable might be defined at
+   * this point. This predicate clarifies which variable(s) should get range
+   * information from `this`.
+   *
+   * This predicate **must be overridden** to hold for any `v` that can show
+   * up in the other members of `SimpleRangeAnalysisDefinition`. Conversely,
+   * the other members **must be accurate** for any `v` in this predicate.
+   */
+  abstract predicate hasRangeInformationFor(StackVariable v);
+
+  /**
+   * Holds if `(this, v)` depends on the range of the unconverted expression
+   * `e`. This information is used to inform the range analysis about cyclic
+   * dependencies. Without this information, range analysis might work for
+   * simple cases but will go into infinite loops on complex code.
+   *
+   * For example, when modelling the definition by reference in a call to an
+   * overloaded `operator=`, written as `v = e`, the definition of `(this, v)`
+   * depends on `e`.
+   */
+  abstract predicate dependsOnExpr(StackVariable v, Expr e);
+
+  /**
+   * Gets the lower bound of the variable `v` defined by this definition.
+   *
+   * Implementations of this predicate should use
+   * `getFullyConvertedLowerBounds` and `getFullyConvertedUpperBounds` for
+   * recursive calls to get the bounds of their dependencies.
+   */
+  abstract float getLowerBounds(StackVariable v);
+
+  /**
+   * Gets the upper bound of the variable `v` defined by this definition.
+   *
+   * Implementations of this predicate should use
+   * `getFullyConvertedLowerBounds` and `getFullyConvertedUpperBounds` for
+   * recursive calls to get the bounds of their dependencies.
+   */
+  abstract float getUpperBounds(StackVariable v);
+}
+
+import SimpleRangeAnalysisInternal

cpp/ql/src/experimental/semmle/code/cpp/rangeanalysis/ExtendedRangeAnalysis.qll

@@ -0,0 +1,5 @@
+import semmle.code.cpp.rangeanalysis.SimpleRangeAnalysis
+//
+// Import each extension we want to enable
+import extensions.SubtractSelf
+import extensions.ConstantBitwiseAndExprRange

cpp/ql/src/experimental/semmle/code/cpp/rangeanalysis/extensions/ConstantBitwiseAndExprRange.qll

@@ -0,0 +1,90 @@
+private import cpp
+private import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisExpr
+private import semmle.code.cpp.rangeanalysis.RangeAnalysisUtils
+
+/**
+ * Holds if `e` is a constant or if it is a variable with a constant value
+ */
+float evaluateConstantExpr(Expr e) {
+  result = e.getValue().toFloat()
+  or
+  exists(SsaDefinition defn, StackVariable sv |
+    defn.getAUse(sv) = e and
+    result = defn.getDefiningValue(sv).getValue().toFloat()
+  )
+}
+
+/**
+ * The current implementation for `BitwiseAndExpr` only handles cases where both operands are
+ * either unsigned or non-negative constants. This class not only covers these cases, but also
+ * adds support for `&` expressions between a signed integer with a non-negative range and a
+ * non-negative constant. It also adds support for `&=` for the same set of cases as `&`.
+ */
+private class ConstantBitwiseAndExprRange extends SimpleRangeAnalysisExpr {
+  ConstantBitwiseAndExprRange() {
+    exists(Expr l, Expr r |
+      l = this.(BitwiseAndExpr).getLeftOperand() and
+      r = this.(BitwiseAndExpr).getRightOperand()
+      or
+      l = this.(AssignAndExpr).getLValue() and
+      r = this.(AssignAndExpr).getRValue()
+    |
+      // No operands can be negative constants
+      not (evaluateConstantExpr(l) < 0 or evaluateConstantExpr(r) < 0) and
+      // At least one operand must be a non-negative constant
+      (evaluateConstantExpr(l) >= 0 or evaluateConstantExpr(r) >= 0)
+    )
+  }
+
+  Expr getLeftOperand() {
+    result = this.(BitwiseAndExpr).getLeftOperand() or
+    result = this.(AssignAndExpr).getLValue()
+  }
+
+  Expr getRightOperand() {
+    result = this.(BitwiseAndExpr).getRightOperand() or
+    result = this.(AssignAndExpr).getRValue()
+  }
+
+  override float getLowerBounds() {
+    // If an operand can have negative values, the lower bound is unconstrained.
+    // Otherwise, the lower bound is zero.
+    exists(float lLower, float rLower |
+      lLower = getFullyConvertedLowerBounds(getLeftOperand()) and
+      rLower = getFullyConvertedLowerBounds(getRightOperand()) and
+      (
+        (lLower < 0 or rLower < 0) and
+        result = exprMinVal(this)
+        or
+        // This technically results in two lowerBounds when an operand range is negative, but
+        // that's fine since `exprMinVal(x) <= 0`. We can't use an if statement here without
+        // non-monotonic recursion issues
+        result = 0
+      )
+    )
+  }
+
+  override float getUpperBounds() {
+    // If an operand can have negative values, the upper bound is unconstrained.
+    // Otherwise, the upper bound is the minimum of the upper bounds of the operands
+    exists(float lLower, float lUpper, float rLower, float rUpper |
+      lLower = getFullyConvertedLowerBounds(getLeftOperand()) and
+      lUpper = getFullyConvertedUpperBounds(getLeftOperand()) and
+      rLower = getFullyConvertedLowerBounds(getRightOperand()) and
+      rUpper = getFullyConvertedUpperBounds(getRightOperand()) and
+      (
+        (lLower < 0 or rLower < 0) and
+        result = exprMaxVal(this)
+        or
+        // This technically results in two upperBounds when an operand range is negative, but
+        // that's fine since `exprMaxVal(b) >= result`. We can't use an if statement here without
+        // non-monotonic recursion issues
+        result = rUpper.minimum(lUpper)
+      )
+    )
+  }
+
+  override predicate dependsOnChild(Expr child) {
+    child = getLeftOperand() or child = getRightOperand()
+  }
+}

cpp/ql/src/experimental/semmle/code/cpp/rangeanalysis/extensions/SubtractSelf.qll

@@ -0,0 +1,15 @@
+import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisExpr
+
+private class SelfSub extends SimpleRangeAnalysisExpr, SubExpr {
+  SelfSub() {
+    // Match `x - x` but not `myInt - (unsigned char)myInt`.
+    getLeftOperand().getExplicitlyConverted().(VariableAccess).getTarget() =
+      getRightOperand().getExplicitlyConverted().(VariableAccess).getTarget()
+  }
+
+  override float getLowerBounds() { result = 0 }
+
+  override float getUpperBounds() { result = 0 }
+
+  override predicate dependsOnChild(Expr child) { none() }
+}

cpp/ql/src/experimental/semmle/code/cpp/security/PrivateCleartextWrite.qll

@@ -7,7 +7,6 @@ import semmle.code.cpp.dataflow.TaintTracking
 import experimental.semmle.code.cpp.security.PrivateData
 import semmle.code.cpp.security.FileWrite
 import semmle.code.cpp.security.BufferWrite
-import semmle.code.cpp.dataflow.TaintTracking
 
 module PrivateCleartextWrite {
   /**

cpp/ql/src/external/DuplicateBlock.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name Duplicate code
  * @description This block of code is duplicated elsewhere. If possible, the shared code should be refactored so there is only one occurrence left. It may not always be possible to address these issues; other duplicate code checks (such as duplicate function, duplicate class) give subsets of the results with higher confidence.
  * @kind problem

cpp/ql/src/external/DuplicateFunction.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name Duplicate function
  * @description There is another identical implementation of this function. Extract the code to a common file or superclass or delegate to improve sharing.
  * @kind problem

cpp/ql/src/external/MostlyDuplicateClass.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name Mostly duplicate class
  * @description More than 80% of the methods in this class are duplicated in another class. Create a common supertype to improve code sharing.
  * @kind problem

cpp/ql/src/external/MostlyDuplicateFile.ql

@@ -1,4 +1,5 @@
 /**
+ * @deprecated
  * @name Mostly duplicate file
  * @description There is another file that shares a lot of the code with this file. Merge the two files to improve maintainability.
  * @kind problem