Merge branch 'main' into map

change-notes/1.25/analysis-java.md

@@ -4,20 +4,26 @@ The following changes in version 1.25 affect Java analysis in all applications.
 
 ## General improvements
 
-## New queries
+The Java autobuilder has been improved to detect more Gradle Java versions.
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
 
 ## Changes to existing queries
 
 | **Query**                    | **Expected impact**    | **Change**                        |
 |------------------------------|------------------------|-----------------------------------|
-
+| Hard-coded credential in API call (`java/hardcoded-credential-api-call`) | More results | The query now recognizes the `BasicAWSCredentials` class of the Amazon client SDK library with hardcoded access key/secret key. |
+| Deserialization of user-controlled data (`java/unsafe-deserialization`) | Fewer false positive results | The query no longer reports results using `org.apache.commons.io.serialization.ValidatingObjectInputStream`. |
+| Use of a broken or risky cryptographic algorithm (`java/weak-cryptographic-algorithm`) | More results | The query now recognizes the `MessageDigest.getInstance` method. |
+| Use of a potentially broken or risky cryptographic algorithm (`java/potentially-weak-cryptographic-algorithm`) | More results | The query now recognizes the `MessageDigest.getInstance` method. |
+| Reading from a world writable file (`java/world-writable-file-read`) | More results | The query now recognizes more JDK file operations. |
 
 ## Changes to libraries
 
+* The data-flow library has been improved with more taint flow modeling for the
+  Collections framework and other classes of the JDK. This affects all security
+  queries using data flow and can yield additional results.
+* The data-flow library has been improved with more taint flow modeling for the
+  Spring framework. This affects all security queries using data flow and can
+  yield additional results on project that rely on the Spring framework.
 * The data-flow library has been improved, which affects most security queries by potentially
   adding more results. Flow through methods now takes nested field reads/writes into account.
   For example, the library is able to track flow from `"taint"` to `sink()` via the method
@@ -39,3 +45,5 @@ The following changes in version 1.25 affect Java analysis in all applications.
     }
   }
   ```
+* The library has been extended with more support for Java 14 features
+  (`switch` expressions and pattern-matching for `instanceof`).

change-notes/1.25/analysis-python.md

@@ -1,22 +1,9 @@
 # Improvements to Python analysis
 
-The following changes in version 1.25 affect Python analysis in all applications.
-
-## General improvements
-
-
-## New queries
-
-| **Query**                   | **Tags**  | **Purpose**                                                        |
-|-----------------------------|-----------|--------------------------------------------------------------------|
-
-
-## Changes to existing queries
-
-| **Query**                  | **Expected impact**    | **Change**                                                       |
-|----------------------------|------------------------|------------------------------------------------------------------|
-
-
-## Changes to libraries
-
 * Importing `semmle.python.web.HttpRequest` will no longer import `UntrustedStringKind` transitively. `UntrustedStringKind` is the most commonly used non-abstract subclass of `ExternalStringKind`. If not imported (by one mean or another), taint-tracking queries that concern `ExternalStringKind` will not produce any results. Please ensure such queries contain an explicit import (`import semmle.python.security.strings.Untrusted`).
+* Added model of taint sources for HTTP servers using `http.server`.
+* Added taint modeling of routed parameters in Flask.
+* Improved modeling of built-in methods on strings for taint tracking.
+* Improved classification of test files.
+* New class `BoundMethodValue` represents a bound method during runtime.
+* The query `py/command-line-injection` now recognizes command execution with the `fabric` and `invoke` Python libraries.

change-notes/1.26/analysis-csharp.md

@@ -12,7 +12,7 @@ The following changes in version 1.26 affect C# analysis in all applications.
 
 | **Query**                    | **Expected impact**    | **Change**                        |
 |------------------------------|------------------------|-----------------------------------|
-
+| Weak encryption: Insufficient key size (`cs/insufficient-key-size`) | More results | The required key size has been increased from 1024 to 2048. |
 
 ## Removal of old queries
 

change-notes/1.26/analysis-javascript.md

@@ -14,12 +14,14 @@
   - [json-stringify-safe](https://www.npmjs.com/package/json-stringify-safe)
   - [json3](https://www.npmjs.com/package/json3)
   - [lodash](https://www.npmjs.com/package/lodash)
+  - [needle](https://www.npmjs.com/package/needle)
   - [object-inspect](https://www.npmjs.com/package/object-inspect)
   - [pretty-format](https://www.npmjs.com/package/pretty-format)
   - [stringify-object](https://www.npmjs.com/package/stringify-object)
   - [underscore](https://www.npmjs.com/package/underscore)
 
 * Analyzing files with the ".cjs" extension is now supported.
+* ES2021 features are now supported.
 
 ## New queries
 
@@ -38,6 +40,7 @@
 | Unsafe shell command constructed from library input (`js/shell-command-constructed-from-input`) | More results | This query now recognizes more commands where colon, dash, and underscore are used. |
 | Unsafe jQuery plugin (`js/unsafe-jquery-plugin`) | More results | This query now detects more unsafe uses of nested option properties. |
 | Client-side URL redirect (`js/client-side-unvalidated-url-redirection`) | More results | This query now recognizes some unsafe uses of `importScripts()` inside WebWorkers. |
+| Missing CSRF middleware (`js/missing-token-validation`) | More results | This query now recognizes writes to cookie and session variables as potentially vulnerable to CSRF attacks. |
 
 
 ## Changes to libraries

config/identical-files.json

@@ -50,6 +50,18 @@
     "csharp/ql/src/semmle/code/csharp/dataflow/internal/DataFlowImplConsistency.qll",
     "python/ql/src/experimental/dataflow/internal/DataFlowImplConsistency.qll"
   ],
+  "SsaReadPosition Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll"
+  ],
+  "Sign Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/Sign.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/Sign.qll"
+  ],
+  "SignAnalysis Java/C#": [
+    "java/ql/src/semmle/code/java/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll",
+    "csharp/ql/src/semmle/code/csharp/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll"
+  ],
   "C++ SubBasicBlocks": [
     "cpp/ql/src/semmle/code/cpp/controlflow/SubBasicBlocks.qll",
     "cpp/ql/src/semmle/code/cpp/dataflow/internal/SubBasicBlocks.qll"
@@ -87,7 +99,7 @@
     "cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/Operand.qll",
     "cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/Operand.qll",
     "csharp/ql/src/experimental/ir/implementation/raw/Operand.qll",
-    "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Operand.qll" 
+    "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Operand.qll"
   ],
   "IR IRType": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/IRType.qll",
@@ -109,11 +121,11 @@
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/OperandTag.qll",
     "csharp/ql/src/experimental/ir/implementation/internal/OperandTag.qll"
   ],
-  "IR TInstruction":[
+  "IR TInstruction": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/TInstruction.qll",
     "csharp/ql/src/experimental/ir/implementation/internal/TInstruction.qll"
   ],
-  "IR TIRVariable":[
+  "IR TIRVariable": [
     "cpp/ql/src/semmle/code/cpp/ir/implementation/internal/TIRVariable.qll",
     "csharp/ql/src/experimental/ir/implementation/internal/TIRVariable.qll"
   ],
@@ -381,4 +393,4 @@
     "javascript/ql/src/Comments/CommentedOutCodeReferences.qhelp",
     "python/ql/src/Lexical/CommentedOutCodeReferences.qhelp"
   ]
-}
+}

cpp/ql/src/Security/CWE/CWE-022/TaintedPath.qhelp

@@ -39,7 +39,7 @@ access all the system's passwords.</p>
 
 <li>
 OWASP:
-<a href="https://www.owasp.org/index.php/Path_traversal">Path Traversal</a>.
+<a href="https://owasp.org/www-community/attacks/Path_Traversal">Path Traversal</a>.
 </li>
 
 </references>

cpp/ql/src/experimental/semmle/code/cpp/models/interfaces/SimpleRangeAnalysisDefinition.qll

@@ -0,0 +1,65 @@
+/**
+ * EXPERIMENTAL: The API of this module may change without notice.
+ *
+ * Provides a class for modeling `RangeSsaDefinition`s with a restricted range.
+ */
+
+import cpp
+import semmle.code.cpp.rangeanalysis.SimpleRangeAnalysis
+
+/**
+ * EXPERIMENTAL: The API of this class may change without notice.
+ *
+ * An SSA definition for which a range can be deduced. As with
+ * `RangeSsaDefinition` and `SsaDefinition`, instances of this class
+ * correspond to points in the program where one or more variables are defined
+ * or have their value constrained in some way.
+ *
+ * Extend this class to add functionality to the range analysis library.
+ */
+abstract class SimpleRangeAnalysisDefinition extends RangeSsaDefinition {
+  /**
+   * Holds if this `SimpleRangeAnalysisDefinition` adds range information for
+   * `v`. Because a `SimpleRangeAnalysisDefinition` is just a point in the
+   * program, it's possible that more than one variable might be defined at
+   * this point. This predicate clarifies which variable(s) should get range
+   * information from `this`.
+   *
+   * This predicate **must be overridden** to hold for any `v` that can show
+   * up in the other members of `SimpleRangeAnalysisDefinition`. Conversely,
+   * the other members **must be accurate** for any `v` in this predicate.
+   */
+  abstract predicate hasRangeInformationFor(StackVariable v);
+
+  /**
+   * Holds if `(this, v)` depends on the range of the unconverted expression
+   * `e`. This information is used to inform the range analysis about cyclic
+   * dependencies. Without this information, range analysis might work for
+   * simple cases but will go into infinite loops on complex code.
+   *
+   * For example, when modelling the definition by reference in a call to an
+   * overloaded `operator=`, written as `v = e`, the definition of `(this, v)`
+   * depends on `e`.
+   */
+  abstract predicate dependsOnExpr(StackVariable v, Expr e);
+
+  /**
+   * Gets the lower bound of the variable `v` defined by this definition.
+   *
+   * Implementations of this predicate should use
+   * `getFullyConvertedLowerBounds` and `getFullyConvertedUpperBounds` for
+   * recursive calls to get the bounds of their dependencies.
+   */
+  abstract float getLowerBounds(StackVariable v);
+
+  /**
+   * Gets the upper bound of the variable `v` defined by this definition.
+   *
+   * Implementations of this predicate should use
+   * `getFullyConvertedLowerBounds` and `getFullyConvertedUpperBounds` for
+   * recursive calls to get the bounds of their dependencies.
+   */
+  abstract float getUpperBounds(StackVariable v);
+}
+
+import SimpleRangeAnalysisInternal

cpp/ql/src/experimental/semmle/code/cpp/rangeanalysis/ExtendedRangeAnalysis.qll

@@ -0,0 +1,4 @@
+import semmle.code.cpp.rangeanalysis.SimpleRangeAnalysis
+//
+// Import each extension we want to enable
+import extensions.SubtractSelf

cpp/ql/src/experimental/semmle/code/cpp/rangeanalysis/extensions/SubtractSelf.qll

@@ -0,0 +1,15 @@
+import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisExpr
+
+private class SelfSub extends SimpleRangeAnalysisExpr, SubExpr {
+  SelfSub() {
+    // Match `x - x` but not `myInt - (unsigned char)myInt`.
+    getLeftOperand().getExplicitlyConverted().(VariableAccess).getTarget() =
+      getRightOperand().getExplicitlyConverted().(VariableAccess).getTarget()
+  }
+
+  override float getLowerBounds() { result = 0 }
+
+  override float getUpperBounds() { result = 0 }
+
+  override predicate dependsOnChild(Expr child) { none() }
+}

cpp/ql/src/semmle/code/cpp/Variable.qll

@@ -144,6 +144,11 @@ class Variable extends Declaration, @variable {
    */
   predicate isConstexpr() { this.hasSpecifier("is_constexpr") }
 
+  /**
+   * Holds if this variable is declared `constinit`.
+   */
+  predicate isConstinit() { this.hasSpecifier("declared_constinit") }
+
   /**
    * Holds if this variable is `thread_local`.
    */

cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowPrivate.qll

@@ -1,6 +1,7 @@
 private import cpp
 private import DataFlowUtil
 private import DataFlowDispatch
+private import FlowVar
 
 /** Gets the instance argument of a non-static call. */
 private Node getInstanceArgument(Call call) {
@@ -106,7 +107,7 @@ private class ExprOutNode extends OutNode, ExprNode {
   override DataFlowCall getCall() { result = this.getExpr() }
 }
 
-private class RefOutNode extends OutNode, DefinitionByReferenceNode {
+private class RefOutNode extends OutNode, DefinitionByReferenceOrIteratorNode {
   /** Gets the underlying call. */
   override DataFlowCall getCall() { result = this.getArgument().getParent() }
 }
@@ -120,7 +121,7 @@ OutNode getAnOutNode(DataFlowCall call, ReturnKind kind) {
   kind = TNormalReturnKind()
   or
   exists(int i |
-    result.asDefiningArgument() = call.getArgument(i) and
+    result.(DefinitionByReferenceOrIteratorNode).getArgument() = call.getArgument(i) and
     kind = TRefReturnKind(i)
   )
 }

cpp/ql/src/semmle/code/cpp/dataflow/internal/DataFlowUtil.qll

@@ -183,28 +183,29 @@ class ImplicitParameterNode extends ParameterNode, TInstanceParameterNode {
 }
 
 /**
- * A node that represents the value of a variable after a function call that
+ * INTERNAL: do not use.
- * may have changed the variable because it's passed by reference.
  *
- * A typical example would be a call `f(&x)`. Firstly, there will be flow into
+ * A node that represents the value of a variable after a function call that
- * `x` from previous definitions of `x`. Secondly, there will be a
+ * may have changed the variable because it's passed by reference or because an
- * `DefinitionByReferenceNode` to represent the value of `x` after the call has
+ * iterator for it was passed by value or by reference.
- * returned. This node will have its `getArgument()` equal to `&x`.
  */
-class DefinitionByReferenceNode extends PartialDefinitionNode {
+class DefinitionByReferenceOrIteratorNode extends PartialDefinitionNode {
   Expr inner;
   Expr argument;
 
-  DefinitionByReferenceNode() {
+  DefinitionByReferenceOrIteratorNode() {
-    this.getPartialDefinition().(DefinitionByReference).definesExpressions(inner, argument)
+    this.getPartialDefinition().definesExpressions(inner, argument) and
+    (
+      this.getPartialDefinition() instanceof DefinitionByReference
+      or
+      this.getPartialDefinition() instanceof DefinitionByIterator
+    )
   }
 
   override Function getFunction() { result = inner.getEnclosingFunction() }
 
   override Type getType() { result = inner.getType() }
 
-  override string toString() { result = "ref arg " + argument.toString() }
-
   override Location getLocation() { result = argument.getLocation() }
 
   override ExprNode getPreUpdateNode() { result.getExpr() = argument }
@@ -221,6 +222,21 @@ class DefinitionByReferenceNode extends PartialDefinitionNode {
   }
 }
 
+/**
+ * A node that represents the value of a variable after a function call that
+ * may have changed the variable because it's passed by reference.
+ *
+ * A typical example would be a call `f(&x)`. Firstly, there will be flow into
+ * `x` from previous definitions of `x`. Secondly, there will be a
+ * `DefinitionByReferenceNode` to represent the value of `x` after the call has
+ * returned. This node will have its `getArgument()` equal to `&x`.
+ */
+class DefinitionByReferenceNode extends DefinitionByReferenceOrIteratorNode {
+  override VariablePartialDefinition pd;
+
+  override string toString() { result = "ref arg " + argument.toString() }
+}
+
 /**
  * The value of an uninitialized local variable, viewed as a node in a data
  * flow graph.
@@ -284,13 +300,11 @@ abstract class PostUpdateNode extends Node {
   override Location getLocation() { result = getPreUpdateNode().getLocation() }
 }
 
-private class PartialDefinitionNode extends PostUpdateNode, TPartialDefinitionNode {
+abstract private class PartialDefinitionNode extends PostUpdateNode, TPartialDefinitionNode {
   PartialDefinition pd;
 
   PartialDefinitionNode() { this = TPartialDefinitionNode(pd) }
 
-  override Node getPreUpdateNode() { pd.definesExpressions(_, result.asExpr()) }
-
   override Location getLocation() { result = pd.getActualLocation() }
 
   PartialDefinition getPartialDefinition() { result = pd }
@@ -298,6 +312,24 @@ private class PartialDefinitionNode extends PostUpdateNode, TPartialDefinitionNo
   override string toString() { result = getPreUpdateNode().toString() + " [post update]" }
 }
 
+private class VariablePartialDefinitionNode extends PartialDefinitionNode {
+  override VariablePartialDefinition pd;
+
+  override Node getPreUpdateNode() { pd.definesExpressions(_, result.asExpr()) }
+}
+
+/**
+ * INTERNAL: do not use.
+ *
+ * A synthetic data flow node used for flow into a collection when an iterator
+ * write occurs in a callee.
+ */
+class IteratorPartialDefinitionNode extends PartialDefinitionNode {
+  override IteratorPartialDefinition pd;
+
+  override Node getPreUpdateNode() { pd.definesExpressions(_, result.asExpr()) }
+}
+
 /**
  * A post-update node on the `e->f` in `f(&e->f)` (and other forms).
  */

cpp/ql/src/semmle/code/cpp/dataflow/internal/FlowVar.qll

@@ -6,6 +6,7 @@ import cpp
 private import semmle.code.cpp.controlflow.SSA
 private import semmle.code.cpp.dataflow.internal.SubBasicBlocks
 private import semmle.code.cpp.dataflow.internal.AddressFlow
+private import semmle.code.cpp.models.implementations.Iterator
 
 /**
  * A conceptual variable that is assigned only once, like an SSA variable. This
@@ -108,21 +109,12 @@ class FlowVar extends TFlowVar {
  * ```
  */
 private module PartialDefinitions {
-  class PartialDefinition extends Expr {
+  abstract class PartialDefinition extends Expr {
-    Expr innerDefinedExpr;
     ControlFlowNode node;
 
-    PartialDefinition() {
+    abstract deprecated predicate partiallyDefines(Variable v);
-      exists(Expr convertedInner |
-        valueToUpdate(convertedInner, this.getFullyConverted(), node) and
-        innerDefinedExpr = convertedInner.getUnconverted() and
-        not this instanceof Conversion
-      )
-    }
 
-    deprecated predicate partiallyDefines(Variable v) { innerDefinedExpr = v.getAnAccess() }
+    abstract deprecated predicate partiallyDefinesThis(ThisExpr e);
-
-    deprecated predicate partiallyDefinesThis(ThisExpr e) { innerDefinedExpr = e }
 
     /**
      * Gets the subBasicBlock where this `PartialDefinition` is defined.
@@ -133,11 +125,9 @@ private module PartialDefinitions {
      * Holds if this `PartialDefinition` defines variable `v` at control-flow
      * node `cfn`.
      */
+    // does this work with a dispred?
     pragma[noinline]
-    predicate partiallyDefinesVariableAt(Variable v, ControlFlowNode cfn) {
+    abstract predicate partiallyDefinesVariableAt(Variable v, ControlFlowNode cfn);
-      innerDefinedExpr = v.getAnAccess() and
-      cfn = node
-    }
 
     /**
      * Holds if this partial definition may modify `inner` (or what it points
@@ -147,10 +137,7 @@ private module PartialDefinitions {
      * - `inner` = `... .x`, `outer` = `&...`
      * - `inner` = `a`, `outer` = `*`
      */
-    predicate definesExpressions(Expr inner, Expr outer) {
+    abstract predicate definesExpressions(Expr inner, Expr outer);
-      inner = innerDefinedExpr and
-      outer = this
-    }
 
     /**
      * Gets the location of this element, adjusted to avoid unknown locations
@@ -166,10 +153,107 @@ private module PartialDefinitions {
     }
   }
 
+  class IteratorPartialDefinition extends PartialDefinition {
+    Variable collection;
+    Expr innerDefinedExpr;
+
+    IteratorPartialDefinition() {
+      exists(Expr convertedInner |
+        not this instanceof Conversion and
+        valueToUpdate(convertedInner, this.getFullyConverted(), node) and
+        innerDefinedExpr = convertedInner.getUnconverted() and
+        (
+          innerDefinedExpr.(Call).getQualifier() = getAnIteratorAccess(collection)
+          or
+          innerDefinedExpr.(Call).getQualifier() = collection.getAnAccess() and
+          collection instanceof IteratorParameter
+        ) and
+        innerDefinedExpr.(Call).getTarget() instanceof IteratorPointerDereferenceMemberOperator
+      )
+      or
+      // iterators passed by value without a copy constructor
+      exists(Call call |
+        call = node and
+        call.getAnArgument() = innerDefinedExpr and
+        innerDefinedExpr = this and
+        this = getAnIteratorAccess(collection) and
+        not call.getTarget() instanceof IteratorPointerDereferenceMemberOperator
+      )
+      or
+      // iterators passed by value with a copy constructor
+      exists(Call call, ConstructorCall copy |
+        copy.getTarget() instanceof CopyConstructor and
+        call = node and
+        call.getAnArgument() = copy and
+        copy.getArgument(0) = getAnIteratorAccess(collection) and
+        innerDefinedExpr = this and
+        this = copy and
+        not call.getTarget() instanceof IteratorPointerDereferenceMemberOperator
+      )
+    }
+
+    deprecated override predicate partiallyDefines(Variable v) { v = collection }
+
+    deprecated override predicate partiallyDefinesThis(ThisExpr e) { none() }
+
+    override predicate definesExpressions(Expr inner, Expr outer) {
+      inner = innerDefinedExpr and
+      outer = this
+    }
+
+    override predicate partiallyDefinesVariableAt(Variable v, ControlFlowNode cfn) {
+      v = collection and
+      cfn = node
+    }
+  }
+
+  class VariablePartialDefinition extends PartialDefinition {
+    Expr innerDefinedExpr;
+
+    VariablePartialDefinition() {
+      not this instanceof Conversion and
+      exists(Expr convertedInner |
+        valueToUpdate(convertedInner, this.getFullyConverted(), node) and
+        innerDefinedExpr = convertedInner.getUnconverted()
+      )
+    }
+
+    deprecated override predicate partiallyDefines(Variable v) {
+      innerDefinedExpr = v.getAnAccess()
+    }
+
+    deprecated override predicate partiallyDefinesThis(ThisExpr e) { innerDefinedExpr = e }
+
+    /**
+     * Holds if this partial definition may modify `inner` (or what it points
+     * to) through `outer`. These expressions will never be `Conversion`s.
+     *
+     * For example, in `f(& (*a).x)`, there are two results:
+     * - `inner` = `... .x`, `outer` = `&...`
+     * - `inner` = `a`, `outer` = `*`
+     */
+    override predicate definesExpressions(Expr inner, Expr outer) {
+      inner = innerDefinedExpr and
+      outer = this
+    }
+
+    override predicate partiallyDefinesVariableAt(Variable v, ControlFlowNode cfn) {
+      innerDefinedExpr = v.getAnAccess() and
+      cfn = node
+    }
+  }
+
+  /**
+   * A partial definition that's a definition via an output iterator.
+   */
+  class DefinitionByIterator extends IteratorPartialDefinition {
+    DefinitionByIterator() { exists(Call c | this = c.getAnArgument() or this = c.getQualifier()) }
+  }
+
   /**
    * A partial definition that's a definition by reference.
    */
-  class DefinitionByReference extends PartialDefinition {
+  class DefinitionByReference extends VariablePartialDefinition {
     DefinitionByReference() { exists(Call c | this = c.getAnArgument() or this = c.getQualifier()) }
   }
 }
@@ -211,7 +295,8 @@ module FlowVar_internal {
     // The SSA library has a theoretically accurate treatment of reference types,
     // treating them as immutable, but for data flow it gives better results in
     // practice to make the variable synonymous with its contents.
-    not v.getUnspecifiedType() instanceof ReferenceType
+    not v.getUnspecifiedType() instanceof ReferenceType and
+    not v instanceof IteratorParameter
   }
 
   /**
@@ -240,7 +325,7 @@ module FlowVar_internal {
       (
         initializer(v, sbb.getANode())
         or
-        assignmentLikeOperation(sbb, v, _, _)
+        assignmentLikeOperation(sbb, v, _)
         or
         exists(PartialDefinition p | p.partiallyDefinesVariableAt(v, sbb))
         or
@@ -359,7 +444,7 @@ module FlowVar_internal {
     }
 
     override predicate definedByExpr(Expr e, ControlFlowNode node) {
-      assignmentLikeOperation(node, v, _, e) and
+      assignmentLikeOperation(node, v, e) and
       node = sbb
       or
       // We pick the defining `ControlFlowNode` of an `Initializer` to be its
@@ -449,7 +534,7 @@ module FlowVar_internal {
     pragma[noinline]
     private Variable getAVariableAssignedInLoop() {
       exists(BasicBlock bbAssign |
-        assignmentLikeOperation(bbAssign.getANode(), result, _, _) and
+        assignmentLikeOperation(bbAssign.getANode(), result, _) and
         this.bbInLoop(bbAssign)
       )
     }
@@ -487,7 +572,7 @@ module FlowVar_internal {
 
   pragma[noinline]
   private predicate assignsToVar(BasicBlock bb, Variable v) {
-    assignmentLikeOperation(bb.getANode(), v, _, _) and
+    assignmentLikeOperation(bb.getANode(), v, _) and
     exists(AlwaysTrueUponEntryLoop loop | v = loop.getARelevantVariable())
   }
 
@@ -524,7 +609,7 @@ module FlowVar_internal {
       result = mid.getASuccessor() and
       variableLiveInSBB(result, v) and
       forall(AlwaysTrueUponEntryLoop loop | skipLoop(mid, result, v, loop) | loop.sbbInLoop(sbbDef)) and
-      not assignmentLikeOperation(result, v, _, _)
+      not assignmentLikeOperation(result, v, _)
     )
   }
 
@@ -560,13 +645,15 @@ module FlowVar_internal {
       refType = p.getUnderlyingType() and
       not refType.getBaseType().isConst()
     )
+    or
+    p instanceof IteratorParameter
   }
 
   /**
    * Holds if liveness of `v` should stop propagating backwards from `sbb`.
    */
   private predicate variableNotLiveBefore(SubBasicBlock sbb, Variable v) {
-    assignmentLikeOperation(sbb, v, _, _)
+    assignmentLikeOperation(sbb, v, _)
     or
     // Liveness of `v` is killed when going backwards from a block that declares it
     exists(DeclStmt ds | ds.getADeclaration().(LocalVariable) = v and sbb.contains(ds))
@@ -686,21 +773,17 @@ module FlowVar_internal {
    * `node instanceof Initializer` is covered by `initializer` instead of this
    * predicate.
    */
-  predicate assignmentLikeOperation(
+  predicate assignmentLikeOperation(ControlFlowNode node, Variable v, Expr assignedExpr) {
-    ControlFlowNode node, Variable v, VariableAccess va, Expr assignedExpr
-  ) {
     // Together, the two following cases cover `Assignment`
     node =
       any(AssignExpr ae |
-        va = ae.getLValue() and
+        v.getAnAccess() = ae.getLValue() and
-        v = va.getTarget() and
         assignedExpr = ae.getRValue()
       )
     or
     node =
       any(AssignOperation ao |
-        va = ao.getLValue() and
+        v.getAnAccess() = ao.getLValue() and
-        v = va.getTarget() and
         // Here and in the `PrefixCrementOperation` case, we say that the assigned
         // expression is the operation itself. For example, we say that `x += 1`
         // assigns `x += 1` to `x`. The justification is that after this operation,
@@ -712,12 +795,24 @@ module FlowVar_internal {
     // `PrefixCrementOperation` is itself a source
     node =
       any(CrementOperation op |
-        va = op.getOperand() and
+        v.getAnAccess() = op.getOperand() and
-        v = va.getTarget() and
         assignedExpr = op
       )
   }
 
+  Expr getAnIteratorAccess(Variable collection) {
+    exists(Call c, SsaDefinition def, Variable iterator |
+      c.getQualifier() = collection.getAnAccess() and
+      c.getTarget() instanceof BeginOrEndFunction and
+      def.getAnUltimateDefiningValue(iterator) = c and
+      result = def.getAUse(iterator)
+    )
+  }
+
+  class IteratorParameter extends Parameter {
+    IteratorParameter() { this.getUnspecifiedType() instanceof Iterator }
+  }
+
   /**
    * Holds if `v` is initialized to have value `assignedExpr`.
    */
@@ -749,7 +844,7 @@ module FlowVar_internal {
   class DataFlowSubBasicBlockCutNode extends SubBasicBlockCutNode {
     DataFlowSubBasicBlockCutNode() {
       exists(Variable v | not fullySupportedSsaVariable(v) |
-        assignmentLikeOperation(this, v, _, _)
+        assignmentLikeOperation(this, v, _)
         or
         exists(PartialDefinition p | p.partiallyDefinesVariableAt(v, this))
         // It is not necessary to cut the basic blocks at `Initializer` nodes

cpp/ql/src/semmle/code/cpp/dataflow/internal/TaintTrackingUtil.qll

@@ -10,6 +10,7 @@
 
 private import semmle.code.cpp.models.interfaces.DataFlow
 private import semmle.code.cpp.models.interfaces.Taint
+private import semmle.code.cpp.models.interfaces.Iterator
 
 private module DataFlow {
   import semmle.code.cpp.dataflow.internal.DataFlowUtil
@@ -255,4 +256,12 @@ private predicate exprToPartialDefinitionStep(Expr exprIn, Expr exprOut) {
       exprIn = call.getArgument(argInIndex)
     )
   )
+  or
+  exists(Assignment a |
+    iteratorDereference(exprOut) and
+    a.getLValue() = exprOut and
+    a.getRValue() = exprIn
+  )
 }
+
+private predicate iteratorDereference(Call c) { c.getTarget() instanceof IteratorReferenceFunction }

cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivate.qll

@@ -355,14 +355,16 @@ private class ArrayToPointerConvertInstruction extends ConvertInstruction {
   }
 }
 
-private Instruction skipOneCopyValueInstruction(Instruction instr) {
+private Instruction skipOneCopyValueInstructionRec(CopyValueInstruction copy) {
-  not instr instanceof CopyValueInstruction and result = instr
+  copy.getUnary() = result and not result instanceof CopyValueInstruction
   or
-  result = instr.(CopyValueInstruction).getUnary()
+  result = skipOneCopyValueInstructionRec(copy.getUnary())
 }
 
 private Instruction skipCopyValueInstructions(Instruction instr) {
-  result = skipOneCopyValueInstruction*(instr) and not result instanceof CopyValueInstruction
+  not result instanceof CopyValueInstruction and result = instr
+  or
+  result = skipOneCopyValueInstructionRec(instr)
 }
 
 private predicate arrayReadStep(Node node1, ArrayContent a, Node node2) {

cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/Operand.qll

@@ -79,7 +79,8 @@ private PhiOperandBase phiOperand(
 }
 
 /**
- * A source operand of an `Instruction`. The operand represents a value consumed by the instruction.
+ * An operand of an `Instruction`. The operand represents a use of the result of one instruction
+ * (the defining instruction) in another instruction (the use instruction)
  */
 class Operand extends TOperand {
   /** Gets a textual representation of this element. */

cpp/ql/src/semmle/code/cpp/ir/implementation/aliased_ssa/internal/AliasedSSA.qll

@@ -133,6 +133,12 @@ abstract class MemoryLocation extends TMemoryLocation {
   predicate isAlwaysAllocatedOnStack() { none() }
 }
 
+/**
+ * Represents a set of `MemoryLocation`s that cannot overlap with
+ * `MemoryLocation`s outside of the set. The `VirtualVariable` will be
+ * represented by a `MemoryLocation` that totally overlaps all other
+ * `MemoryLocations` in the set.
+ */
 abstract class VirtualVariable extends MemoryLocation { }
 
 abstract class AllocationMemoryLocation extends MemoryLocation {

cpp/ql/src/semmle/code/cpp/ir/implementation/raw/Operand.qll

@@ -79,7 +79,8 @@ private PhiOperandBase phiOperand(
 }
 
 /**
- * A source operand of an `Instruction`. The operand represents a value consumed by the instruction.
+ * An operand of an `Instruction`. The operand represents a use of the result of one instruction
+ * (the defining instruction) in another instruction (the use instruction)
  */
 class Operand extends TOperand {
   /** Gets a textual representation of this element. */

cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/Operand.qll

@@ -79,7 +79,8 @@ private PhiOperandBase phiOperand(
 }
 
 /**
- * A source operand of an `Instruction`. The operand represents a value consumed by the instruction.
+ * An operand of an `Instruction`. The operand represents a use of the result of one instruction
+ * (the defining instruction) in another instruction (the use instruction)
  */
 class Operand extends TOperand {
   /** Gets a textual representation of this element. */

cpp/ql/src/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll

@@ -59,6 +59,12 @@ class MemoryLocation extends TMemoryLocation {
   final string getUniqueId() { result = var.getUniqueId() }
 }
 
+/**
+ * Represents a set of `MemoryLocation`s that cannot overlap with
+ * `MemoryLocation`s outside of the set. The `VirtualVariable` will be
+ * represented by a `MemoryLocation` that totally overlaps all other
+ * `MemoryLocations` in the set.
+ */
 class VirtualVariable extends MemoryLocation { }
 
 /** A virtual variable that groups all escaped memory within a function. */

cpp/ql/src/semmle/code/cpp/ir/internal/Overlap.qll

@@ -3,18 +3,33 @@ private newtype TOverlap =
   TMustTotallyOverlap() or
   TMustExactlyOverlap()
 
+/**
+ * Represents a possible overlap between two memory ranges.
+ */
 abstract class Overlap extends TOverlap {
   abstract string toString();
 }
 
+/**
+ * Represents a partial overlap between two memory ranges, which may or may not
+ * actually occur in practice.
+ */
 class MayPartiallyOverlap extends Overlap, TMayPartiallyOverlap {
   final override string toString() { result = "MayPartiallyOverlap" }
 }
 
+/**
+ * Represents an overlap in which the first memory range is known to include all
+ * bits of the second memory range, but may be larger or have a different type.
+ */
 class MustTotallyOverlap extends Overlap, TMustTotallyOverlap {
   final override string toString() { result = "MustTotallyOverlap" }
 }
 
+/**
+ * Represents an overlap between two memory ranges that have the same extent and
+ * the same type.
+ */
 class MustExactlyOverlap extends Overlap, TMustExactlyOverlap {
   final override string toString() { result = "MustExactlyOverlap" }
 }

cpp/ql/src/semmle/code/cpp/models/implementations/Iterator.qll

@@ -8,6 +8,7 @@
 import cpp
 import semmle.code.cpp.models.interfaces.Taint
 import semmle.code.cpp.models.interfaces.DataFlow
+import semmle.code.cpp.models.interfaces.Iterator
 
 /**
  * An instantiation of the `std::iterator_traits` template.
@@ -80,7 +81,7 @@ private FunctionInput getIteratorArgumentInput(Operator op, int index) {
 /**
  * A non-member prefix `operator*` function for an iterator type.
  */
-class IteratorPointerDereferenceOperator extends Operator, TaintFunction {
+class IteratorPointerDereferenceOperator extends Operator, TaintFunction, IteratorReferenceFunction {
   FunctionInput iteratorInput;
 
   IteratorPointerDereferenceOperator() {
@@ -169,7 +170,8 @@ class IteratorAssignArithmeticOperator extends Operator, DataFlowFunction, Taint
 /**
  * A prefix `operator*` member function for an iterator type.
  */
-class IteratorPointerDereferenceMemberOperator extends MemberFunction, TaintFunction {
+class IteratorPointerDereferenceMemberOperator extends MemberFunction, TaintFunction,
+  IteratorReferenceFunction {
   IteratorPointerDereferenceMemberOperator() {
     this.hasName("operator*") and
     this.getDeclaringType() instanceof Iterator
@@ -260,7 +262,7 @@ class IteratorAssignArithmeticMemberOperator extends MemberFunction, DataFlowFun
 /**
  * An `operator[]` member function of an iterator class.
  */
-class IteratorArrayMemberOperator extends MemberFunction, TaintFunction {
+class IteratorArrayMemberOperator extends MemberFunction, TaintFunction, IteratorReferenceFunction {
   IteratorArrayMemberOperator() {
     this.hasName("operator[]") and
     this.getDeclaringType() instanceof Iterator
@@ -271,3 +273,19 @@ class IteratorArrayMemberOperator extends MemberFunction, TaintFunction {
     output.isReturnValue()
   }
 }
+
+/**
+ * A `begin` or `end` member function, or a related member function, that
+ * returns an iterator.
+ */
+class BeginOrEndFunction extends MemberFunction, TaintFunction {
+  BeginOrEndFunction() {
+    this.hasName(["begin", "cbegin", "rbegin", "crbegin", "end", "cend", "rend", "crend"]) and
+    this.getType().getUnspecifiedType() instanceof Iterator
+  }
+
+  override predicate hasTaintFlow(FunctionInput input, FunctionOutput output) {
+    input.isQualifierObject() and
+    output.isReturnValue()
+  }
+}

cpp/ql/src/semmle/code/cpp/models/implementations/StdString.qll

@@ -216,23 +216,6 @@ class StdStringAssign extends TaintFunction {
   }
 }
 
-/**
- * The standard functions `std::string.begin` and `std::string.end` and their
- * variants.
- */
-class StdStringBeginEnd extends TaintFunction {
-  StdStringBeginEnd() {
-    this
-        .hasQualifiedName("std", "basic_string",
-          ["begin", "cbegin", "rbegin", "crbegin", "end", "cend", "rend", "crend"])
-  }
-
-  override predicate hasTaintFlow(FunctionInput input, FunctionOutput output) {
-    input.isQualifierObject() and
-    output.isReturnValue()
-  }
-}
-
 /**
  * The standard function `std::string.copy`.
  */

cpp/ql/src/semmle/code/cpp/models/interfaces/Iterator.qll

@@ -0,0 +1,17 @@
+/**
+ * Provides an abstract class for accurate modeling of flow through output
+ * iterators.  To use this QL library, create a QL class extending
+ * `IteratorReferenceFunction` with a characteristic predicate that selects the
+ * function or set of functions you are modeling. Within that class, override
+ * the predicates provided by `AliasFunction` to match the flow within that
+ * function.
+ */
+
+import cpp
+import semmle.code.cpp.models.Models
+
+/**
+ * A function which takes an iterator argument and returns a reference that
+ * can be used to write to the iterator's underlying collection.
+ */
+abstract class IteratorReferenceFunction extends Function { }

cpp/ql/src/semmle/code/cpp/rangeanalysis/SimpleRangeAnalysis.qll

@@ -45,6 +45,7 @@
 import cpp
 private import RangeAnalysisUtils
 private import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisExpr
+private import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisDefinition
 import RangeSSA
 import SimpleRangeAnalysisCached
 private import NanAnalysis
@@ -335,6 +336,11 @@ private predicate defDependsOnDef(
   or
   // Phi nodes.
   phiDependsOnDef(def, v, srcDef, srcVar)
+  or
+  // Extensions
+  exists(Expr expr | def.(SimpleRangeAnalysisDefinition).dependsOnExpr(v, expr) |
+    exprDependsOnDef(expr, srcDef, srcVar)
+  )
 }
 
 /**
@@ -492,6 +498,9 @@ private predicate analyzableDef(RangeSsaDefinition def, StackVariable v) {
   v = def.getAVariable()
   or
   phiDependsOnDef(def, v, _, _)
+  or
+  // A modeled def for range analysis
+  def.(SimpleRangeAnalysisDefinition).hasRangeInformationFor(v)
 }
 
 /**
@@ -1215,6 +1224,9 @@ private float getDefLowerBoundsImpl(RangeSsaDefinition def, StackVariable v) {
   // Phi nodes.
   result = getPhiLowerBounds(v, def)
   or
+  // A modeled def for range analysis
+  result = def.(SimpleRangeAnalysisDefinition).getLowerBounds(v)
+  or
   // Unanalyzable definitions.
   unanalyzableDefBounds(def, v, result, _)
 }
@@ -1248,6 +1260,9 @@ private float getDefUpperBoundsImpl(RangeSsaDefinition def, StackVariable v) {
   // Phi nodes.
   result = getPhiUpperBounds(v, def)
   or
+  // A modeled def for range analysis
+  result = def.(SimpleRangeAnalysisDefinition).getUpperBounds(v)
+  or
   // Unanalyzable definitions.
   unanalyzableDefBounds(def, v, _, result)
 }

cpp/ql/test/experimental/library-tests/rangeanalysis/extended/extended.cpp

@@ -0,0 +1,9 @@
+
+
+void test_overridability_sub(int x) {
+  int zero = x - x;
+  zero; // 0
+
+  int nonzero = x - (unsigned char)x;
+  nonzero; // full range
+}

cpp/ql/test/experimental/library-tests/rangeanalysis/extended/extended.expected

@@ -0,0 +1,6 @@
+| extended.cpp:4:14:4:14 | x | -2.147483648E9 | 2.147483647E9 |
+| extended.cpp:4:18:4:18 | x | -2.147483648E9 | 2.147483647E9 |
+| extended.cpp:5:3:5:6 | zero | 0.0 | 0.0 |
+| extended.cpp:7:17:7:17 | x | -2.147483648E9 | 2.147483647E9 |
+| extended.cpp:7:36:7:36 | x | -2.147483648E9 | 2.147483647E9 |
+| extended.cpp:8:3:8:9 | nonzero | -2.147483648E9 | 2.147483647E9 |

cpp/ql/test/experimental/library-tests/rangeanalysis/extended/extended.ql

@@ -0,0 +1,7 @@
+import experimental.semmle.code.cpp.rangeanalysis.ExtendedRangeAnalysis
+
+from VariableAccess expr, float lower, float upper
+where
+  lower = lowerBound(expr) and
+  upper = upperBound(expr)
+select expr, lower, upper

cpp/ql/test/experimental/library-tests/rangeanalysis/extensibility/extensibility.c

@@ -9,6 +9,11 @@ int test_extensibility_add(int x) {
 }
 
 int test_overridability_sub(int x) {
-  int result = x - x; // Returns 0 due to custom modeling in QL
+  int result = x - (unsigned char)x; // Returns 0 due to custom modeling for this test being deliberately wrong
   return result; // 0
-}
+}
+
+void test_parameter_override(int magic_name_at_most_10, int magic_name_at_most_20) {
+  magic_name_at_most_10;
+  magic_name_at_most_20;
+}

cpp/ql/test/experimental/library-tests/rangeanalysis/extensibility/extensibility.expected

@@ -3,5 +3,7 @@
 | extensibility.c:6:38:6:38 | x | -10.0 | 10.0 |
 | extensibility.c:7:12:7:17 | result | 90.0 | 110.0 |
 | extensibility.c:12:16:12:16 | x | -2.147483648E9 | 2.147483647E9 |
-| extensibility.c:12:20:12:20 | x | -2.147483648E9 | 2.147483647E9 |
+| extensibility.c:12:35:12:35 | x | -2.147483648E9 | 2.147483647E9 |
 | extensibility.c:13:10:13:15 | result | 0.0 | 0.0 |
+| extensibility.c:17:3:17:23 | magic_name_at_most_10 | -2.147483648E9 | 10.0 |
+| extensibility.c:18:3:18:23 | magic_name_at_most_20 | -2.147483648E9 | 20.0 |

cpp/ql/test/experimental/library-tests/rangeanalysis/extensibility/extensibility.ql

@@ -1,5 +1,7 @@
 import semmle.code.cpp.rangeanalysis.SimpleRangeAnalysis
+import semmle.code.cpp.rangeanalysis.RangeAnalysisUtils
 import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisExpr
+import experimental.semmle.code.cpp.models.interfaces.SimpleRangeAnalysisDefinition
 
 class CustomAddFunctionCall extends SimpleRangeAnalysisExpr, FunctionCall {
   CustomAddFunctionCall() { this.getTarget().hasGlobalName("custom_add_function") }
@@ -37,6 +39,40 @@ class SelfSub extends SimpleRangeAnalysisExpr, SubExpr {
   override predicate dependsOnChild(Expr child) { child = this.getAnOperand() }
 }
 
+/**
+ * A definition for test purposes of a parameter `p` that starts with a
+ * special prefix. This class is written to exploit how QL behaves when class
+ * fields are not functionally determined by `this`. When multiple parameters
+ * of the same function have the special prefix, there is still only one
+ * instance of this class.
+ */
+class MagicParameterName extends SimpleRangeAnalysisDefinition {
+  Parameter p;
+  float value;
+
+  MagicParameterName() {
+    this.definedByParameter(p) and
+    value = p.getName().regexpCapture("magic_name_at_most_(\\d+)", 1).toFloat()
+  }
+
+  override predicate hasRangeInformationFor(StackVariable v) { v = p }
+
+  override predicate dependsOnExpr(StackVariable v, Expr e) {
+    // No dependencies. This sample class yields constant values.
+    none()
+  }
+
+  override float getLowerBounds(StackVariable var) {
+    var = p and
+    result = typeLowerBound(p.getUnspecifiedType())
+  }
+
+  override float getUpperBounds(StackVariable var) {
+    var = p and
+    result = value
+  }
+}
+
 from VariableAccess expr, float lower, float upper
 where
   lower = lowerBound(expr) and

cpp/ql/test/library-tests/dataflow/taint-tests/localTaint.expected

@@ -1884,23 +1884,29 @@
 | smart_pointer.cpp:65:28:65:46 | call to make_unique | smart_pointer.cpp:67:10:67:10 | p |  |
 | smart_pointer.cpp:65:48:65:53 | call to source | smart_pointer.cpp:65:28:65:46 | call to make_unique | TAINT |
 | smart_pointer.cpp:65:58:65:58 | 0 | smart_pointer.cpp:65:28:65:46 | call to make_unique | TAINT |
+| standalone_iterators.cpp:39:45:39:51 | source1 | standalone_iterators.cpp:39:45:39:51 | source1 |  |
 | standalone_iterators.cpp:39:45:39:51 | source1 | standalone_iterators.cpp:40:11:40:17 | source1 |  |
 | standalone_iterators.cpp:39:45:39:51 | source1 | standalone_iterators.cpp:41:12:41:18 | source1 |  |
 | standalone_iterators.cpp:39:45:39:51 | source1 | standalone_iterators.cpp:42:14:42:20 | source1 |  |
 | standalone_iterators.cpp:40:11:40:17 | source1 | standalone_iterators.cpp:40:10:40:10 | call to operator* | TAINT |
+| standalone_iterators.cpp:41:12:41:18 | ref arg source1 | standalone_iterators.cpp:39:45:39:51 | source1 |  |
 | standalone_iterators.cpp:41:12:41:18 | ref arg source1 | standalone_iterators.cpp:42:14:42:20 | source1 |  |
 | standalone_iterators.cpp:41:12:41:18 | source1 | standalone_iterators.cpp:41:19:41:19 | call to operator++ |  |
 | standalone_iterators.cpp:41:19:41:19 | call to operator++ | standalone_iterators.cpp:41:10:41:10 | call to operator* | TAINT |
 | standalone_iterators.cpp:42:12:42:12 | call to operator++ | standalone_iterators.cpp:42:10:42:10 | call to operator* | TAINT |
+| standalone_iterators.cpp:42:14:42:20 | ref arg source1 | standalone_iterators.cpp:39:45:39:51 | source1 |  |
 | standalone_iterators.cpp:42:14:42:20 | source1 | standalone_iterators.cpp:42:12:42:12 | call to operator++ |  |
+| standalone_iterators.cpp:45:39:45:45 | source1 | standalone_iterators.cpp:45:39:45:45 | source1 |  |
 | standalone_iterators.cpp:45:39:45:45 | source1 | standalone_iterators.cpp:46:11:46:17 | source1 |  |
 | standalone_iterators.cpp:45:39:45:45 | source1 | standalone_iterators.cpp:47:12:47:18 | source1 |  |
 | standalone_iterators.cpp:45:39:45:45 | source1 | standalone_iterators.cpp:48:14:48:20 | source1 |  |
 | standalone_iterators.cpp:46:11:46:17 | source1 | standalone_iterators.cpp:46:10:46:10 | call to operator* | TAINT |
+| standalone_iterators.cpp:47:12:47:18 | ref arg source1 | standalone_iterators.cpp:45:39:45:45 | source1 |  |
 | standalone_iterators.cpp:47:12:47:18 | ref arg source1 | standalone_iterators.cpp:48:14:48:20 | source1 |  |
 | standalone_iterators.cpp:47:12:47:18 | source1 | standalone_iterators.cpp:47:19:47:19 | call to operator++ |  |
 | standalone_iterators.cpp:47:19:47:19 | call to operator++ | standalone_iterators.cpp:47:10:47:10 | call to operator* | TAINT |
 | standalone_iterators.cpp:48:12:48:12 | call to operator++ | standalone_iterators.cpp:48:10:48:10 | call to operator* | TAINT |
+| standalone_iterators.cpp:48:14:48:20 | ref arg source1 | standalone_iterators.cpp:45:39:45:45 | source1 |  |
 | standalone_iterators.cpp:48:14:48:20 | source1 | standalone_iterators.cpp:48:12:48:12 | call to operator++ |  |
 | standalone_iterators.cpp:51:37:51:43 | source1 | standalone_iterators.cpp:52:11:52:17 | source1 |  |
 | standalone_iterators.cpp:51:37:51:43 | source1 | standalone_iterators.cpp:53:12:53:18 | source1 |  |
@@ -5309,11 +5315,14 @@
 | vector.cpp:255:3:255:4 | ref arg v6 | vector.cpp:261:8:261:9 | v6 |  |
 | vector.cpp:255:3:255:4 | ref arg v6 | vector.cpp:262:2:262:2 | v6 |  |
 | vector.cpp:255:13:255:14 | call to iterator | vector.cpp:255:3:255:4 | ref arg v6 | TAINT |
+| vector.cpp:255:13:255:14 | call to iterator [post update] | vector.cpp:277:1:277:1 | v3 |  |
 | vector.cpp:255:13:255:14 | i1 | vector.cpp:255:13:255:14 | call to iterator |  |
+| vector.cpp:255:13:255:14 | i1 [post update] | vector.cpp:277:1:277:1 | v3 |  |
 | vector.cpp:255:17:255:18 | call to iterator | vector.cpp:255:3:255:4 | ref arg v6 | TAINT |
 | vector.cpp:255:17:255:18 | i2 | vector.cpp:255:17:255:18 | call to iterator |  |
 | vector.cpp:257:8:257:9 | ref arg v4 | vector.cpp:262:2:262:2 | v4 |  |
 | vector.cpp:258:8:258:9 | ref arg v5 | vector.cpp:262:2:262:2 | v5 |  |
+| vector.cpp:259:8:259:9 | ref arg i1 | vector.cpp:277:1:277:1 | v3 |  |
 | vector.cpp:261:8:261:9 | ref arg v6 | vector.cpp:262:2:262:2 | v6 |  |
 | vector.cpp:265:22:265:23 | call to vector | vector.cpp:269:3:269:4 | v7 |  |
 | vector.cpp:265:22:265:23 | call to vector | vector.cpp:273:8:273:9 | v7 |  |
@@ -5516,3 +5525,265 @@
 | vector.cpp:324:7:324:8 | ref arg v2 | vector.cpp:327:1:327:1 | v2 |  |
 | vector.cpp:325:7:325:8 | ref arg v3 | vector.cpp:327:1:327:1 | v3 |  |
 | vector.cpp:326:7:326:8 | ref arg v4 | vector.cpp:327:1:327:1 | v4 |  |
+| vector.cpp:329:62:329:65 | iter | vector.cpp:329:62:329:65 | iter |  |
+| vector.cpp:329:62:329:65 | iter | vector.cpp:330:3:330:6 | iter |  |
+| vector.cpp:330:2:330:2 | call to operator* [post update] | vector.cpp:329:62:329:65 | iter |  |
+| vector.cpp:330:2:330:17 | ... = ... | vector.cpp:330:2:330:2 | call to operator* [post update] |  |
+| vector.cpp:330:3:330:6 | iter | vector.cpp:330:2:330:2 | call to operator* | TAINT |
+| vector.cpp:330:10:330:15 | call to source | vector.cpp:330:2:330:2 | call to operator* [post update] | TAINT |
+| vector.cpp:330:10:330:15 | call to source | vector.cpp:330:2:330:17 | ... = ... |  |
+| vector.cpp:333:64:333:67 | iter | vector.cpp:333:64:333:67 | iter |  |
+| vector.cpp:333:64:333:67 | iter | vector.cpp:334:3:334:6 | iter |  |
+| vector.cpp:333:74:333:74 | i | vector.cpp:334:10:334:10 | i |  |
+| vector.cpp:334:2:334:2 | call to operator* [post update] | vector.cpp:333:64:333:67 | iter |  |
+| vector.cpp:334:2:334:10 | ... = ... | vector.cpp:334:2:334:2 | call to operator* [post update] |  |
+| vector.cpp:334:3:334:6 | iter | vector.cpp:334:2:334:2 | call to operator* | TAINT |
+| vector.cpp:334:10:334:10 | i | vector.cpp:334:2:334:2 | call to operator* [post update] | TAINT |
+| vector.cpp:334:10:334:10 | i | vector.cpp:334:2:334:10 | ... = ... |  |
+| vector.cpp:337:38:337:38 | b | vector.cpp:372:5:372:5 | b |  |
+| vector.cpp:338:22:338:24 | call to vector | vector.cpp:340:34:340:35 | v1 |  |
+| vector.cpp:338:22:338:24 | call to vector | vector.cpp:342:7:342:8 | v1 |  |
+| vector.cpp:338:22:338:24 | call to vector | vector.cpp:401:1:401:1 | v1 |  |
+| vector.cpp:338:30:338:32 | call to vector | vector.cpp:344:38:344:39 | v2 |  |
+| vector.cpp:338:30:338:32 | call to vector | vector.cpp:344:56:344:57 | v2 |  |
+| vector.cpp:338:30:338:32 | call to vector | vector.cpp:347:7:347:8 | v2 |  |
+| vector.cpp:338:30:338:32 | call to vector | vector.cpp:401:1:401:1 | v2 |  |
+| vector.cpp:338:38:338:40 | call to vector | vector.cpp:349:15:349:16 | v3 |  |
+| vector.cpp:338:38:338:40 | call to vector | vector.cpp:352:7:352:8 | v3 |  |
+| vector.cpp:338:38:338:40 | call to vector | vector.cpp:401:1:401:1 | v3 |  |
+| vector.cpp:338:46:338:48 | call to vector | vector.cpp:354:38:354:39 | v4 |  |
+| vector.cpp:338:46:338:48 | call to vector | vector.cpp:354:56:354:57 | v4 |  |
+| vector.cpp:338:46:338:48 | call to vector | vector.cpp:357:7:357:8 | v4 |  |
+| vector.cpp:338:46:338:48 | call to vector | vector.cpp:401:1:401:1 | v4 |  |
+| vector.cpp:338:54:338:56 | call to vector | vector.cpp:359:34:359:35 | v5 |  |
+| vector.cpp:338:54:338:56 | call to vector | vector.cpp:361:7:361:8 | v5 |  |
+| vector.cpp:338:54:338:56 | call to vector | vector.cpp:363:7:363:8 | v5 |  |
+| vector.cpp:338:54:338:56 | call to vector | vector.cpp:401:1:401:1 | v5 |  |
+| vector.cpp:338:62:338:64 | call to vector | vector.cpp:365:34:365:35 | v6 |  |
+| vector.cpp:338:62:338:64 | call to vector | vector.cpp:367:7:367:8 | v6 |  |
+| vector.cpp:338:62:338:64 | call to vector | vector.cpp:368:2:368:3 | v6 |  |
+| vector.cpp:338:62:338:64 | call to vector | vector.cpp:369:7:369:8 | v6 |  |
+| vector.cpp:338:62:338:64 | call to vector | vector.cpp:401:1:401:1 | v6 |  |
+| vector.cpp:338:70:338:72 | call to vector | vector.cpp:371:34:371:35 | v7 |  |
+| vector.cpp:338:70:338:72 | call to vector | vector.cpp:374:8:374:9 | v7 |  |
+| vector.cpp:338:70:338:72 | call to vector | vector.cpp:377:8:377:9 | v7 |  |
+| vector.cpp:338:70:338:72 | call to vector | vector.cpp:379:7:379:8 | v7 |  |
+| vector.cpp:338:70:338:72 | call to vector | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:338:78:338:80 | call to vector | vector.cpp:381:34:381:35 | v8 |  |
+| vector.cpp:338:78:338:80 | call to vector | vector.cpp:383:7:383:8 | v8 |  |
+| vector.cpp:338:78:338:80 | call to vector | vector.cpp:385:7:385:8 | v8 |  |
+| vector.cpp:338:78:338:80 | call to vector | vector.cpp:401:1:401:1 | v8 |  |
+| vector.cpp:338:86:338:88 | call to vector | vector.cpp:387:34:387:35 | v9 |  |
+| vector.cpp:338:86:338:88 | call to vector | vector.cpp:392:7:392:8 | v9 |  |
+| vector.cpp:338:86:338:88 | call to vector | vector.cpp:401:1:401:1 | v9 |  |
+| vector.cpp:338:95:338:97 | call to vector | vector.cpp:394:35:394:37 | v10 |  |
+| vector.cpp:338:95:338:97 | call to vector | vector.cpp:396:7:396:9 | v10 |  |
+| vector.cpp:338:95:338:97 | call to vector | vector.cpp:401:1:401:1 | v10 |  |
+| vector.cpp:338:104:338:106 | call to vector | vector.cpp:398:35:398:37 | v11 |  |
+| vector.cpp:338:104:338:106 | call to vector | vector.cpp:400:7:400:9 | v11 |  |
+| vector.cpp:338:104:338:106 | call to vector | vector.cpp:401:1:401:1 | v11 |  |
+| vector.cpp:340:34:340:35 | ref arg v1 | vector.cpp:342:7:342:8 | v1 |  |
+| vector.cpp:340:34:340:35 | ref arg v1 | vector.cpp:401:1:401:1 | v1 |  |
+| vector.cpp:340:34:340:35 | v1 | vector.cpp:340:37:340:41 | call to begin | TAINT |
+| vector.cpp:340:37:340:41 | call to begin | vector.cpp:341:3:341:4 | i1 |  |
+| vector.cpp:341:2:341:2 | call to operator* [post update] | vector.cpp:342:7:342:8 | v1 |  |
+| vector.cpp:341:2:341:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v1 |  |
+| vector.cpp:341:2:341:15 | ... = ... | vector.cpp:341:2:341:2 | call to operator* [post update] |  |
+| vector.cpp:341:3:341:4 | i1 | vector.cpp:341:2:341:2 | call to operator* | TAINT |
+| vector.cpp:341:8:341:13 | call to source | vector.cpp:341:2:341:2 | call to operator* [post update] | TAINT |
+| vector.cpp:341:8:341:13 | call to source | vector.cpp:341:2:341:15 | ... = ... |  |
+| vector.cpp:342:7:342:8 | ref arg v1 | vector.cpp:401:1:401:1 | v1 |  |
+| vector.cpp:344:38:344:39 | ref arg v2 | vector.cpp:344:56:344:57 | v2 |  |
+| vector.cpp:344:38:344:39 | ref arg v2 | vector.cpp:347:7:347:8 | v2 |  |
+| vector.cpp:344:38:344:39 | ref arg v2 | vector.cpp:401:1:401:1 | v2 |  |
+| vector.cpp:344:38:344:39 | v2 | vector.cpp:344:41:344:45 | call to begin | TAINT |
+| vector.cpp:344:41:344:45 | call to begin | vector.cpp:344:50:344:51 | it |  |
+| vector.cpp:344:41:344:45 | call to begin | vector.cpp:344:68:344:69 | it |  |
+| vector.cpp:344:41:344:45 | call to begin | vector.cpp:345:4:345:5 | it |  |
+| vector.cpp:344:56:344:57 | ref arg v2 | vector.cpp:344:56:344:57 | v2 |  |
+| vector.cpp:344:56:344:57 | ref arg v2 | vector.cpp:347:7:347:8 | v2 |  |
+| vector.cpp:344:56:344:57 | ref arg v2 | vector.cpp:401:1:401:1 | v2 |  |
+| vector.cpp:344:56:344:57 | v2 | vector.cpp:344:59:344:61 | call to end | TAINT |
+| vector.cpp:344:68:344:69 | it | vector.cpp:344:66:344:66 | call to operator++ |  |
+| vector.cpp:344:68:344:69 | ref arg it | vector.cpp:344:50:344:51 | it |  |
+| vector.cpp:344:68:344:69 | ref arg it | vector.cpp:344:68:344:69 | it |  |
+| vector.cpp:344:68:344:69 | ref arg it | vector.cpp:345:4:345:5 | it |  |
+| vector.cpp:345:3:345:3 | call to operator* [post update] | vector.cpp:344:56:344:57 | v2 |  |
+| vector.cpp:345:3:345:3 | call to operator* [post update] | vector.cpp:347:7:347:8 | v2 |  |
+| vector.cpp:345:3:345:3 | call to operator* [post update] | vector.cpp:401:1:401:1 | v2 |  |
+| vector.cpp:345:3:345:16 | ... = ... | vector.cpp:345:3:345:3 | call to operator* [post update] |  |
+| vector.cpp:345:4:345:5 | it | vector.cpp:345:3:345:3 | call to operator* | TAINT |
+| vector.cpp:345:9:345:14 | call to source | vector.cpp:345:3:345:3 | call to operator* [post update] | TAINT |
+| vector.cpp:345:9:345:14 | call to source | vector.cpp:345:3:345:16 | ... = ... |  |
+| vector.cpp:347:7:347:8 | ref arg v2 | vector.cpp:401:1:401:1 | v2 |  |
+| vector.cpp:349:15:349:15 | (__begin) | vector.cpp:349:15:349:15 | call to operator* | TAINT |
+| vector.cpp:349:15:349:15 | (__begin) | vector.cpp:349:15:349:15 | call to operator++ |  |
+| vector.cpp:349:15:349:15 | (__end) | vector.cpp:349:15:349:15 | call to iterator |  |
+| vector.cpp:349:15:349:15 | (__range) | vector.cpp:349:15:349:15 | call to begin | TAINT |
+| vector.cpp:349:15:349:15 | (__range) | vector.cpp:349:15:349:15 | call to end | TAINT |
+| vector.cpp:349:15:349:15 | call to begin | vector.cpp:349:15:349:15 | (__begin) |  |
+| vector.cpp:349:15:349:15 | call to begin | vector.cpp:349:15:349:15 | (__begin) |  |
+| vector.cpp:349:15:349:15 | call to begin | vector.cpp:349:15:349:15 | (__begin) |  |
+| vector.cpp:349:15:349:15 | call to end | vector.cpp:349:15:349:15 | (__end) |  |
+| vector.cpp:349:15:349:15 | ref arg (__begin) | vector.cpp:349:15:349:15 | (__begin) |  |
+| vector.cpp:349:15:349:15 | ref arg (__begin) | vector.cpp:349:15:349:15 | (__begin) |  |
+| vector.cpp:349:15:349:15 | ref arg (__begin) | vector.cpp:349:15:349:15 | (__begin) |  |
+| vector.cpp:349:15:349:15 | ref arg (__range) | vector.cpp:349:15:349:15 | (__range) |  |
+| vector.cpp:349:15:349:16 | v3 | vector.cpp:349:15:349:15 | (__range) |  |
+| vector.cpp:349:15:349:16 | v3 | vector.cpp:349:15:349:15 | (__range) |  |
+| vector.cpp:349:15:349:16 | v3 | vector.cpp:349:15:349:15 | call to operator* | TAINT |
+| vector.cpp:350:3:350:14 | ... = ... | vector.cpp:350:3:350:3 | x [post update] |  |
+| vector.cpp:350:7:350:12 | call to source | vector.cpp:350:3:350:14 | ... = ... |  |
+| vector.cpp:352:7:352:8 | ref arg v3 | vector.cpp:401:1:401:1 | v3 |  |
+| vector.cpp:354:38:354:39 | ref arg v4 | vector.cpp:354:56:354:57 | v4 |  |
+| vector.cpp:354:38:354:39 | ref arg v4 | vector.cpp:357:7:357:8 | v4 |  |
+| vector.cpp:354:38:354:39 | ref arg v4 | vector.cpp:401:1:401:1 | v4 |  |
+| vector.cpp:354:38:354:39 | v4 | vector.cpp:354:41:354:45 | call to begin | TAINT |
+| vector.cpp:354:41:354:45 | call to begin | vector.cpp:354:50:354:51 | it |  |
+| vector.cpp:354:41:354:45 | call to begin | vector.cpp:354:68:354:69 | it |  |
+| vector.cpp:354:41:354:45 | call to begin | vector.cpp:355:32:355:33 | it |  |
+| vector.cpp:354:56:354:57 | ref arg v4 | vector.cpp:354:56:354:57 | v4 |  |
+| vector.cpp:354:56:354:57 | ref arg v4 | vector.cpp:357:7:357:8 | v4 |  |
+| vector.cpp:354:56:354:57 | ref arg v4 | vector.cpp:401:1:401:1 | v4 |  |
+| vector.cpp:354:56:354:57 | v4 | vector.cpp:354:59:354:61 | call to end | TAINT |
+| vector.cpp:354:68:354:69 | it | vector.cpp:354:66:354:66 | call to operator++ |  |
+| vector.cpp:354:68:354:69 | ref arg it | vector.cpp:354:50:354:51 | it |  |
+| vector.cpp:354:68:354:69 | ref arg it | vector.cpp:354:68:354:69 | it |  |
+| vector.cpp:354:68:354:69 | ref arg it | vector.cpp:355:32:355:33 | it |  |
+| vector.cpp:355:32:355:33 | call to iterator [post update] | vector.cpp:354:56:354:57 | v4 |  |
+| vector.cpp:355:32:355:33 | call to iterator [post update] | vector.cpp:357:7:357:8 | v4 |  |
+| vector.cpp:355:32:355:33 | call to iterator [post update] | vector.cpp:401:1:401:1 | v4 |  |
+| vector.cpp:355:32:355:33 | it | vector.cpp:355:32:355:33 | call to iterator |  |
+| vector.cpp:355:32:355:33 | it [post update] | vector.cpp:354:56:354:57 | v4 |  |
+| vector.cpp:355:32:355:33 | it [post update] | vector.cpp:357:7:357:8 | v4 |  |
+| vector.cpp:355:32:355:33 | it [post update] | vector.cpp:401:1:401:1 | v4 |  |
+| vector.cpp:357:7:357:8 | ref arg v4 | vector.cpp:401:1:401:1 | v4 |  |
+| vector.cpp:359:34:359:35 | ref arg v5 | vector.cpp:361:7:361:8 | v5 |  |
+| vector.cpp:359:34:359:35 | ref arg v5 | vector.cpp:363:7:363:8 | v5 |  |
+| vector.cpp:359:34:359:35 | ref arg v5 | vector.cpp:401:1:401:1 | v5 |  |
+| vector.cpp:359:34:359:35 | v5 | vector.cpp:359:37:359:41 | call to begin | TAINT |
+| vector.cpp:359:37:359:41 | call to begin | vector.cpp:360:3:360:4 | i5 |  |
+| vector.cpp:359:37:359:41 | call to begin | vector.cpp:362:3:362:4 | i5 |  |
+| vector.cpp:360:2:360:2 | call to operator* [post update] | vector.cpp:361:7:361:8 | v5 |  |
+| vector.cpp:360:2:360:2 | call to operator* [post update] | vector.cpp:363:7:363:8 | v5 |  |
+| vector.cpp:360:2:360:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v5 |  |
+| vector.cpp:360:2:360:15 | ... = ... | vector.cpp:360:2:360:2 | call to operator* [post update] |  |
+| vector.cpp:360:3:360:4 | i5 | vector.cpp:360:2:360:2 | call to operator* | TAINT |
+| vector.cpp:360:8:360:13 | call to source | vector.cpp:360:2:360:2 | call to operator* [post update] | TAINT |
+| vector.cpp:360:8:360:13 | call to source | vector.cpp:360:2:360:15 | ... = ... |  |
+| vector.cpp:361:7:361:8 | ref arg v5 | vector.cpp:363:7:363:8 | v5 |  |
+| vector.cpp:361:7:361:8 | ref arg v5 | vector.cpp:401:1:401:1 | v5 |  |
+| vector.cpp:362:2:362:2 | call to operator* [post update] | vector.cpp:363:7:363:8 | v5 |  |
+| vector.cpp:362:2:362:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v5 |  |
+| vector.cpp:362:2:362:8 | ... = ... | vector.cpp:362:2:362:2 | call to operator* [post update] |  |
+| vector.cpp:362:3:362:4 | i5 | vector.cpp:362:2:362:2 | call to operator* | TAINT |
+| vector.cpp:362:8:362:8 | 1 | vector.cpp:362:2:362:2 | call to operator* [post update] | TAINT |
+| vector.cpp:362:8:362:8 | 1 | vector.cpp:362:2:362:8 | ... = ... |  |
+| vector.cpp:363:7:363:8 | ref arg v5 | vector.cpp:401:1:401:1 | v5 |  |
+| vector.cpp:365:34:365:35 | ref arg v6 | vector.cpp:367:7:367:8 | v6 |  |
+| vector.cpp:365:34:365:35 | ref arg v6 | vector.cpp:368:2:368:3 | v6 |  |
+| vector.cpp:365:34:365:35 | ref arg v6 | vector.cpp:369:7:369:8 | v6 |  |
+| vector.cpp:365:34:365:35 | ref arg v6 | vector.cpp:401:1:401:1 | v6 |  |
+| vector.cpp:365:34:365:35 | v6 | vector.cpp:365:37:365:41 | call to begin | TAINT |
+| vector.cpp:365:37:365:41 | call to begin | vector.cpp:366:3:366:4 | i6 |  |
+| vector.cpp:366:2:366:2 | call to operator* [post update] | vector.cpp:367:7:367:8 | v6 |  |
+| vector.cpp:366:2:366:2 | call to operator* [post update] | vector.cpp:368:2:368:3 | v6 |  |
+| vector.cpp:366:2:366:2 | call to operator* [post update] | vector.cpp:369:7:369:8 | v6 |  |
+| vector.cpp:366:2:366:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v6 |  |
+| vector.cpp:366:2:366:15 | ... = ... | vector.cpp:366:2:366:2 | call to operator* [post update] |  |
+| vector.cpp:366:3:366:4 | i6 | vector.cpp:366:2:366:2 | call to operator* | TAINT |
+| vector.cpp:366:8:366:13 | call to source | vector.cpp:366:2:366:2 | call to operator* [post update] | TAINT |
+| vector.cpp:366:8:366:13 | call to source | vector.cpp:366:2:366:15 | ... = ... |  |
+| vector.cpp:367:7:367:8 | ref arg v6 | vector.cpp:368:2:368:3 | v6 |  |
+| vector.cpp:367:7:367:8 | ref arg v6 | vector.cpp:369:7:369:8 | v6 |  |
+| vector.cpp:367:7:367:8 | ref arg v6 | vector.cpp:401:1:401:1 | v6 |  |
+| vector.cpp:368:2:368:3 | ref arg v6 | vector.cpp:369:7:369:8 | v6 |  |
+| vector.cpp:368:2:368:3 | ref arg v6 | vector.cpp:401:1:401:1 | v6 |  |
+| vector.cpp:368:7:368:26 | call to vector | vector.cpp:368:2:368:3 | ref arg v6 | TAINT |
+| vector.cpp:368:7:368:26 | call to vector | vector.cpp:368:5:368:5 | call to operator= | TAINT |
+| vector.cpp:369:7:369:8 | ref arg v6 | vector.cpp:401:1:401:1 | v6 |  |
+| vector.cpp:371:34:371:35 | ref arg v7 | vector.cpp:374:8:374:9 | v7 |  |
+| vector.cpp:371:34:371:35 | ref arg v7 | vector.cpp:377:8:377:9 | v7 |  |
+| vector.cpp:371:34:371:35 | ref arg v7 | vector.cpp:379:7:379:8 | v7 |  |
+| vector.cpp:371:34:371:35 | ref arg v7 | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:371:34:371:35 | v7 | vector.cpp:371:37:371:41 | call to begin | TAINT |
+| vector.cpp:371:37:371:41 | call to begin | vector.cpp:373:4:373:5 | i7 |  |
+| vector.cpp:371:37:371:41 | call to begin | vector.cpp:376:4:376:5 | i7 |  |
+| vector.cpp:373:3:373:3 | call to operator* [post update] | vector.cpp:374:8:374:9 | v7 |  |
+| vector.cpp:373:3:373:3 | call to operator* [post update] | vector.cpp:379:7:379:8 | v7 |  |
+| vector.cpp:373:3:373:3 | call to operator* [post update] | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:373:3:373:16 | ... = ... | vector.cpp:373:3:373:3 | call to operator* [post update] |  |
+| vector.cpp:373:4:373:5 | i7 | vector.cpp:373:3:373:3 | call to operator* | TAINT |
+| vector.cpp:373:9:373:14 | call to source | vector.cpp:373:3:373:3 | call to operator* [post update] | TAINT |
+| vector.cpp:373:9:373:14 | call to source | vector.cpp:373:3:373:16 | ... = ... |  |
+| vector.cpp:374:8:374:9 | ref arg v7 | vector.cpp:379:7:379:8 | v7 |  |
+| vector.cpp:374:8:374:9 | ref arg v7 | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:376:3:376:3 | call to operator* [post update] | vector.cpp:377:8:377:9 | v7 |  |
+| vector.cpp:376:3:376:3 | call to operator* [post update] | vector.cpp:379:7:379:8 | v7 |  |
+| vector.cpp:376:3:376:3 | call to operator* [post update] | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:376:3:376:9 | ... = ... | vector.cpp:376:3:376:3 | call to operator* [post update] |  |
+| vector.cpp:376:4:376:5 | i7 | vector.cpp:376:3:376:3 | call to operator* | TAINT |
+| vector.cpp:376:9:376:9 | 1 | vector.cpp:376:3:376:3 | call to operator* [post update] | TAINT |
+| vector.cpp:376:9:376:9 | 1 | vector.cpp:376:3:376:9 | ... = ... |  |
+| vector.cpp:377:8:377:9 | ref arg v7 | vector.cpp:379:7:379:8 | v7 |  |
+| vector.cpp:377:8:377:9 | ref arg v7 | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:379:7:379:8 | ref arg v7 | vector.cpp:401:1:401:1 | v7 |  |
+| vector.cpp:381:34:381:35 | ref arg v8 | vector.cpp:383:7:383:8 | v8 |  |
+| vector.cpp:381:34:381:35 | ref arg v8 | vector.cpp:385:7:385:8 | v8 |  |
+| vector.cpp:381:34:381:35 | ref arg v8 | vector.cpp:401:1:401:1 | v8 |  |
+| vector.cpp:381:34:381:35 | v8 | vector.cpp:381:37:381:41 | call to begin | TAINT |
+| vector.cpp:381:37:381:41 | call to begin | vector.cpp:382:3:382:4 | i8 |  |
+| vector.cpp:381:37:381:41 | call to begin | vector.cpp:384:3:384:4 | i8 |  |
+| vector.cpp:382:2:382:2 | call to operator* [post update] | vector.cpp:383:7:383:8 | v8 |  |
+| vector.cpp:382:2:382:2 | call to operator* [post update] | vector.cpp:385:7:385:8 | v8 |  |
+| vector.cpp:382:2:382:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v8 |  |
+| vector.cpp:382:2:382:15 | ... = ... | vector.cpp:382:2:382:2 | call to operator* [post update] |  |
+| vector.cpp:382:3:382:4 | i8 | vector.cpp:382:2:382:2 | call to operator* | TAINT |
+| vector.cpp:382:8:382:13 | call to source | vector.cpp:382:2:382:2 | call to operator* [post update] | TAINT |
+| vector.cpp:382:8:382:13 | call to source | vector.cpp:382:2:382:15 | ... = ... |  |
+| vector.cpp:383:7:383:8 | ref arg v8 | vector.cpp:385:7:385:8 | v8 |  |
+| vector.cpp:383:7:383:8 | ref arg v8 | vector.cpp:401:1:401:1 | v8 |  |
+| vector.cpp:384:2:384:2 | call to operator* [post update] | vector.cpp:385:7:385:8 | v8 |  |
+| vector.cpp:384:2:384:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v8 |  |
+| vector.cpp:384:2:384:8 | ... = ... | vector.cpp:384:2:384:2 | call to operator* [post update] |  |
+| vector.cpp:384:3:384:4 | i8 | vector.cpp:384:2:384:2 | call to operator* | TAINT |
+| vector.cpp:384:8:384:8 | 1 | vector.cpp:384:2:384:2 | call to operator* [post update] | TAINT |
+| vector.cpp:384:8:384:8 | 1 | vector.cpp:384:2:384:8 | ... = ... |  |
+| vector.cpp:385:7:385:8 | ref arg v8 | vector.cpp:401:1:401:1 | v8 |  |
+| vector.cpp:387:34:387:35 | ref arg v9 | vector.cpp:392:7:392:8 | v9 |  |
+| vector.cpp:387:34:387:35 | ref arg v9 | vector.cpp:401:1:401:1 | v9 |  |
+| vector.cpp:387:34:387:35 | v9 | vector.cpp:387:37:387:41 | call to begin | TAINT |
+| vector.cpp:387:37:387:41 | call to begin | vector.cpp:389:3:389:4 | i9 |  |
+| vector.cpp:387:37:387:41 | call to begin | vector.cpp:390:31:390:32 | i9 |  |
+| vector.cpp:389:2:389:2 | call to operator* [post update] | vector.cpp:392:7:392:8 | v9 |  |
+| vector.cpp:389:2:389:2 | call to operator* [post update] | vector.cpp:401:1:401:1 | v9 |  |
+| vector.cpp:389:2:389:15 | ... = ... | vector.cpp:389:2:389:2 | call to operator* [post update] |  |
+| vector.cpp:389:3:389:4 | i9 | vector.cpp:389:2:389:2 | call to operator* | TAINT |
+| vector.cpp:389:8:389:13 | call to source | vector.cpp:389:2:389:2 | call to operator* [post update] | TAINT |
+| vector.cpp:389:8:389:13 | call to source | vector.cpp:389:2:389:15 | ... = ... |  |
+| vector.cpp:390:31:390:32 | call to iterator [post update] | vector.cpp:392:7:392:8 | v9 |  |
+| vector.cpp:390:31:390:32 | call to iterator [post update] | vector.cpp:401:1:401:1 | v9 |  |
+| vector.cpp:390:31:390:32 | i9 | vector.cpp:390:31:390:32 | call to iterator |  |
+| vector.cpp:390:31:390:32 | i9 [post update] | vector.cpp:392:7:392:8 | v9 |  |
+| vector.cpp:390:31:390:32 | i9 [post update] | vector.cpp:401:1:401:1 | v9 |  |
+| vector.cpp:392:7:392:8 | ref arg v9 | vector.cpp:401:1:401:1 | v9 |  |
+| vector.cpp:394:35:394:37 | ref arg v10 | vector.cpp:396:7:396:9 | v10 |  |
+| vector.cpp:394:35:394:37 | ref arg v10 | vector.cpp:401:1:401:1 | v10 |  |
+| vector.cpp:394:35:394:37 | v10 | vector.cpp:394:39:394:43 | call to begin | TAINT |
+| vector.cpp:394:39:394:43 | call to begin | vector.cpp:395:33:395:35 | i10 |  |
+| vector.cpp:395:33:395:35 | call to iterator [post update] | vector.cpp:396:7:396:9 | v10 |  |
+| vector.cpp:395:33:395:35 | call to iterator [post update] | vector.cpp:401:1:401:1 | v10 |  |
+| vector.cpp:395:33:395:35 | i10 | vector.cpp:395:33:395:35 | call to iterator |  |
+| vector.cpp:395:33:395:35 | i10 [post update] | vector.cpp:396:7:396:9 | v10 |  |
+| vector.cpp:395:33:395:35 | i10 [post update] | vector.cpp:401:1:401:1 | v10 |  |
+| vector.cpp:396:7:396:9 | ref arg v10 | vector.cpp:401:1:401:1 | v10 |  |
+| vector.cpp:398:35:398:37 | ref arg v11 | vector.cpp:400:7:400:9 | v11 |  |
+| vector.cpp:398:35:398:37 | ref arg v11 | vector.cpp:401:1:401:1 | v11 |  |
+| vector.cpp:398:35:398:37 | v11 | vector.cpp:398:39:398:43 | call to begin | TAINT |
+| vector.cpp:398:39:398:43 | call to begin | vector.cpp:399:33:399:35 | i11 |  |
+| vector.cpp:399:33:399:35 | call to iterator [post update] | vector.cpp:400:7:400:9 | v11 |  |
+| vector.cpp:399:33:399:35 | call to iterator [post update] | vector.cpp:401:1:401:1 | v11 |  |
+| vector.cpp:399:33:399:35 | i11 | vector.cpp:399:33:399:35 | call to iterator |  |
+| vector.cpp:399:33:399:35 | i11 [post update] | vector.cpp:400:7:400:9 | v11 |  |
+| vector.cpp:399:33:399:35 | i11 [post update] | vector.cpp:401:1:401:1 | v11 |  |
+| vector.cpp:400:7:400:9 | ref arg v11 | vector.cpp:401:1:401:1 | v11 |  |

cpp/ql/test/library-tests/dataflow/taint-tests/taint.expected

@@ -531,3 +531,17 @@
 | vector.cpp:312:7:312:7 | d | vector.cpp:303:14:303:19 | call to source |
 | vector.cpp:324:7:324:8 | v2 | vector.cpp:318:15:318:20 | call to source |
 | vector.cpp:326:7:326:8 | v4 | vector.cpp:318:15:318:20 | call to source |
+| vector.cpp:342:7:342:8 | v1 | vector.cpp:341:8:341:13 | call to source |
+| vector.cpp:347:7:347:8 | v2 | vector.cpp:345:9:345:14 | call to source |
+| vector.cpp:357:7:357:8 | v4 | vector.cpp:330:10:330:15 | call to source |
+| vector.cpp:361:7:361:8 | v5 | vector.cpp:360:8:360:13 | call to source |
+| vector.cpp:363:7:363:8 | v5 | vector.cpp:360:8:360:13 | call to source |
+| vector.cpp:367:7:367:8 | v6 | vector.cpp:366:8:366:13 | call to source |
+| vector.cpp:369:7:369:8 | v6 | vector.cpp:366:8:366:13 | call to source |
+| vector.cpp:374:8:374:9 | v7 | vector.cpp:373:9:373:14 | call to source |
+| vector.cpp:379:7:379:8 | v7 | vector.cpp:373:9:373:14 | call to source |
+| vector.cpp:383:7:383:8 | v8 | vector.cpp:382:8:382:13 | call to source |
+| vector.cpp:385:7:385:8 | v8 | vector.cpp:382:8:382:13 | call to source |
+| vector.cpp:392:7:392:8 | v9 | vector.cpp:330:10:330:15 | call to source |
+| vector.cpp:392:7:392:8 | v9 | vector.cpp:389:8:389:13 | call to source |
+| vector.cpp:400:7:400:9 | v11 | vector.cpp:399:38:399:43 | call to source |

cpp/ql/test/library-tests/dataflow/taint-tests/test_diff.expected

@@ -267,3 +267,17 @@
 | vector.cpp:292:7:292:18 | vector.cpp:289:17:289:30 | AST only |
 | vector.cpp:308:9:308:14 | vector.cpp:303:14:303:19 | AST only |
 | vector.cpp:311:9:311:14 | vector.cpp:303:14:303:19 | AST only |
+| vector.cpp:342:7:342:8 | vector.cpp:341:8:341:13 | AST only |
+| vector.cpp:347:7:347:8 | vector.cpp:345:9:345:14 | AST only |
+| vector.cpp:357:7:357:8 | vector.cpp:330:10:330:15 | AST only |
+| vector.cpp:361:7:361:8 | vector.cpp:360:8:360:13 | AST only |
+| vector.cpp:363:7:363:8 | vector.cpp:360:8:360:13 | AST only |
+| vector.cpp:367:7:367:8 | vector.cpp:366:8:366:13 | AST only |
+| vector.cpp:369:7:369:8 | vector.cpp:366:8:366:13 | AST only |
+| vector.cpp:374:8:374:9 | vector.cpp:373:9:373:14 | AST only |
+| vector.cpp:379:7:379:8 | vector.cpp:373:9:373:14 | AST only |
+| vector.cpp:383:7:383:8 | vector.cpp:382:8:382:13 | AST only |
+| vector.cpp:385:7:385:8 | vector.cpp:382:8:382:13 | AST only |
+| vector.cpp:392:7:392:8 | vector.cpp:330:10:330:15 | AST only |
+| vector.cpp:392:7:392:8 | vector.cpp:389:8:389:13 | AST only |
+| vector.cpp:400:7:400:9 | vector.cpp:399:38:399:43 | AST only |

cpp/ql/test/library-tests/dataflow/taint-tests/vector.cpp

@@ -21,7 +21,7 @@ void test_range_based_for_loop_vector(int source1) {
 	}
 
 	for(std::vector<int>::iterator it = v.begin(); it != v.end(); ++it) {
-		sink(*it); // tainted
+		sink(*it); // tainted [NOT DETECTED by IR]
 	}
 
 	for(int& x : v) {
@@ -325,3 +325,77 @@ void test_constructors_more() {
 	sink(v3);
 	sink(v4); // tainted
 }
+
+void taint_vector_output_iterator(std::vector<int>::iterator iter) {
+	*iter = source();
+}
+
+void vector_iterator_assign_wrapper(std::vector<int>::iterator iter, int i) {
+	*iter = i;
+}
+
+void test_vector_output_iterator(int b) {
+	std::vector<int> v1(10), v2(10), v3(10), v4(10), v5(10), v6(10), v7(10), v8(10), v9(10), v10(10), v11(10);
+
+	std::vector<int>::iterator i1 = v1.begin();
+	*i1 = source();
+	sink(v1); // tainted [NOT DETECTED by IR]
+
+	for(std::vector<int>::iterator it = v2.begin(); it != v2.end(); ++it) {
+		*it = source();
+	}
+	sink(v2); // tainted [NOT DETECTED by IR]
+
+	for(int& x : v3) {
+		x = source();
+	}
+	sink(v3); // tainted [NOT DETECTED]
+
+	for(std::vector<int>::iterator it = v4.begin(); it != v4.end(); ++it) {
+		taint_vector_output_iterator(it);
+	}
+	sink(v4); // tainted [NOT DETECTED by IR]
+	
+	std::vector<int>::iterator i5 = v5.begin();
+	*i5 = source();
+	sink(v5); // tainted [NOT DETECTED by IR]
+	*i5 = 1;
+	sink(v5); // tainted [NOT DETECTED by IR]
+
+	std::vector<int>::iterator i6 = v6.begin();
+	*i6 = source();
+	sink(v6); // tainted [NOT DETECTED by IR]
+	v6 = std::vector<int>(10);
+	sink(v6); // [FALSE POSITIVE in AST]
+
+	std::vector<int>::iterator i7 = v7.begin();
+	if(b) {
+		*i7 = source();
+		sink(v7); // tainted [NOT DETECTED by IR]
+	} else {
+		*i7 = 1;
+		sink(v7);
+	}
+	sink(v7); // tainted [NOT DETECTED by IR]
+
+	std::vector<int>::iterator i8 = v8.begin();
+	*i8 = source();
+	sink(v8); // tainted [NOT DETECTED by IR]
+	*i8 = 1;
+	sink(v8);
+
+	std::vector<int>::iterator i9 = v9.begin();
+
+	*i9 = source();
+	taint_vector_output_iterator(i9);
+
+	sink(v9);
+
+	std::vector<int>::iterator i10 = v10.begin();
+	vector_iterator_assign_wrapper(i10, 10);
+	sink(v10);
+
+	std::vector<int>::iterator i11 = v11.begin();
+	vector_iterator_assign_wrapper(i11, source());
+	sink(v11); // tainted [NOT DETECTED by IR]
+}

cpp/ql/test/library-tests/lambdas/captures/elements.expected

@@ -13,9 +13,9 @@
 | captures.cpp:3:5:3:5 | (constructor) |
 | captures.cpp:3:5:3:5 | (constructor) |
 | captures.cpp:3:5:3:5 | (constructor) |
-| captures.cpp:3:5:3:5 | declaration of (null) |
+| captures.cpp:3:5:3:5 | declaration of (constructor) |
-| captures.cpp:3:5:3:5 | declaration of (null) |
+| captures.cpp:3:5:3:5 | declaration of (constructor) |
-| captures.cpp:3:5:3:5 | definition of (null) |
+| captures.cpp:3:5:3:5 | definition of (constructor) |
 | captures.cpp:3:5:3:5 | definition of operator= |
 | captures.cpp:3:5:3:5 | operator= |
 | captures.cpp:3:5:5:5 | [...](...){...} |
@@ -50,9 +50,9 @@
 | captures.cpp:9:5:9:5 | (constructor) |
 | captures.cpp:9:5:9:5 | (constructor) |
 | captures.cpp:9:5:9:5 | (constructor) |
-| captures.cpp:9:5:9:5 | declaration of (null) |
+| captures.cpp:9:5:9:5 | declaration of (constructor) |
-| captures.cpp:9:5:9:5 | declaration of (null) |
+| captures.cpp:9:5:9:5 | declaration of (constructor) |
-| captures.cpp:9:5:9:5 | definition of (null) |
+| captures.cpp:9:5:9:5 | definition of (constructor) |
 | captures.cpp:9:5:9:5 | definition of operator= |
 | captures.cpp:9:5:9:5 | operator= |
 | captures.cpp:9:5:11:5 | [...](...){...} |
@@ -87,9 +87,9 @@
 | captures.cpp:15:5:15:5 | (constructor) |
 | captures.cpp:15:5:15:5 | (constructor) |
 | captures.cpp:15:5:15:5 | (constructor) |
-| captures.cpp:15:5:15:5 | declaration of (null) |
+| captures.cpp:15:5:15:5 | declaration of (constructor) |
-| captures.cpp:15:5:15:5 | declaration of (null) |
+| captures.cpp:15:5:15:5 | declaration of (constructor) |
-| captures.cpp:15:5:15:5 | definition of (null) |
+| captures.cpp:15:5:15:5 | definition of (constructor) |
 | captures.cpp:15:5:15:5 | definition of operator= |
 | captures.cpp:15:5:15:5 | operator= |
 | captures.cpp:15:5:17:5 | [...](...){...} |
@@ -129,9 +129,9 @@
 | captures.cpp:22:19:22:19 | Unknown literal |
 | captures.cpp:22:19:22:19 | constructor init of field x |
 | captures.cpp:22:19:22:19 | constructor init of field y |
-| captures.cpp:22:19:22:19 | declaration of (null) |
+| captures.cpp:22:19:22:19 | declaration of (constructor) |
-| captures.cpp:22:19:22:19 | definition of (null) |
+| captures.cpp:22:19:22:19 | definition of (constructor) |
-| captures.cpp:22:19:22:19 | definition of (null) |
+| captures.cpp:22:19:22:19 | definition of (constructor) |
 | captures.cpp:22:19:22:19 | definition of operator= |
 | captures.cpp:22:19:22:19 | operator= |
 | captures.cpp:22:19:22:19 | return ... |
@@ -187,9 +187,9 @@
 | end_pos.cpp:9:15:9:15 | (constructor) |
 | end_pos.cpp:9:15:9:15 | Unknown literal |
 | end_pos.cpp:9:15:9:15 | constructor init of field ii |
-| end_pos.cpp:9:15:9:15 | declaration of (null) |
+| end_pos.cpp:9:15:9:15 | declaration of (constructor) |
-| end_pos.cpp:9:15:9:15 | definition of (null) |
+| end_pos.cpp:9:15:9:15 | definition of (constructor) |
-| end_pos.cpp:9:15:9:15 | definition of (null) |
+| end_pos.cpp:9:15:9:15 | definition of (constructor) |
 | end_pos.cpp:9:15:9:15 | definition of operator= |
 | end_pos.cpp:9:15:9:15 | operator= |
 | end_pos.cpp:9:15:9:15 | return ... |

cpp/ql/test/library-tests/noexcept/copy_from_prototype/copy_from_prototype.expected

@@ -2,8 +2,8 @@
 | copy_from_prototype.cpp:3:7:3:7 | a | a<int>::a(const a<int> &) -> void | copy_from_prototype.cpp:3:7:3:7 | a<int> | <no expr> |
 | copy_from_prototype.cpp:3:7:3:7 | operator= | a<int>::operator=(a<int> &&) -> a<int> & | copy_from_prototype.cpp:3:7:3:7 | a<int> | <no expr> |
 | copy_from_prototype.cpp:3:7:3:7 | operator= | a<int>::operator=(const a<int> &) -> a<int> & | copy_from_prototype.cpp:3:7:3:7 | a<int> | <no expr> |
-| copy_from_prototype.cpp:4:26:4:26 | a | a<<unnamed>>::a<(unnamed)>() -> void | copy_from_prototype.cpp:3:7:3:7 | a<<unnamed>> | 123 |
+| copy_from_prototype.cpp:4:26:4:26 | a | a<<unnamed>>::a<(unnamed template parameter)>() -> void | copy_from_prototype.cpp:3:7:3:7 | a<<unnamed>> | 123 |
-| copy_from_prototype.cpp:4:26:4:26 | a | a<int>::a<(unnamed)>() -> void | copy_from_prototype.cpp:3:7:3:7 | a<int> | <no expr> |
+| copy_from_prototype.cpp:4:26:4:26 | a | a<int>::a<(unnamed template parameter)>() -> void | copy_from_prototype.cpp:3:7:3:7 | a<int> | <no expr> |
 | copy_from_prototype.cpp:7:7:7:7 | b | b::b() -> void | copy_from_prototype.cpp:7:7:7:7 | b | <no expr> |
 | copy_from_prototype.cpp:7:7:7:7 | b | b::b(b &&) -> void | copy_from_prototype.cpp:7:7:7:7 | b | <no expr> |
 | copy_from_prototype.cpp:7:7:7:7 | b | b::b(const b &) -> void | copy_from_prototype.cpp:7:7:7:7 | b | <no expr> |
@@ -13,8 +13,8 @@
 | copy_from_prototype.cpp:13:7:13:7 | c | c<int>::c(const c<int> &) -> void | copy_from_prototype.cpp:13:7:13:7 | c<int> | <no expr> |
 | copy_from_prototype.cpp:13:7:13:7 | operator= | c<int>::operator=(c<int> &&) -> c<int> & | copy_from_prototype.cpp:13:7:13:7 | c<int> | <no expr> |
 | copy_from_prototype.cpp:13:7:13:7 | operator= | c<int>::operator=(const c<int> &) -> c<int> & | copy_from_prototype.cpp:13:7:13:7 | c<int> | <no expr> |
-| copy_from_prototype.cpp:14:26:14:26 | c | c<T>::c<(unnamed)>() -> void | copy_from_prototype.cpp:13:7:13:7 | c<T> | X |
+| copy_from_prototype.cpp:14:26:14:26 | c | c<T>::c<(unnamed template parameter)>() -> void | copy_from_prototype.cpp:13:7:13:7 | c<T> | X |
-| copy_from_prototype.cpp:14:26:14:26 | c | c<int>::c<(unnamed)>() -> void | copy_from_prototype.cpp:13:7:13:7 | c<int> | <no expr> |
+| copy_from_prototype.cpp:14:26:14:26 | c | c<int>::c<(unnamed template parameter)>() -> void | copy_from_prototype.cpp:13:7:13:7 | c<int> | <no expr> |
 | copy_from_prototype.cpp:17:7:17:7 | d | d::d() -> void | copy_from_prototype.cpp:17:7:17:7 | d | <no expr> |
 | copy_from_prototype.cpp:17:7:17:7 | d | d::d(const d &) -> void | copy_from_prototype.cpp:17:7:17:7 | d | <no expr> |
 | copy_from_prototype.cpp:17:7:17:7 | d | d::d(d &&) -> void | copy_from_prototype.cpp:17:7:17:7 | d | <no expr> |
@@ -24,7 +24,7 @@
 | copy_from_prototype.cpp:22:8:22:8 | e | e<int>::e(e<int> &&) -> void | copy_from_prototype.cpp:22:8:22:8 | e<int> | <no expr> |
 | copy_from_prototype.cpp:22:8:22:8 | operator= | e<int>::operator=(const e<int> &) -> e<int> & | copy_from_prototype.cpp:22:8:22:8 | e<int> | <no expr> |
 | copy_from_prototype.cpp:22:8:22:8 | operator= | e<int>::operator=(e<int> &&) -> e<int> & | copy_from_prototype.cpp:22:8:22:8 | e<int> | <no expr> |
-| copy_from_prototype.cpp:23:26:23:26 | e | e<T>::e<(unnamed)>() -> void | copy_from_prototype.cpp:22:8:22:8 | e<T> | 456 |
+| copy_from_prototype.cpp:23:26:23:26 | e | e<T>::e<(unnamed template parameter)>() -> void | copy_from_prototype.cpp:22:8:22:8 | e<T> | 456 |
-| copy_from_prototype.cpp:26:35:26:43 | e | e<int>::e<(unnamed)>() -> void | copy_from_prototype.cpp:22:8:22:8 | e<int> | 456 |
+| copy_from_prototype.cpp:26:35:26:43 | e | e<int>::e<(unnamed template parameter)>() -> void | copy_from_prototype.cpp:22:8:22:8 | e<int> | 456 |
 | file://:0:0:0:0 | operator= | __va_list_tag::operator=(__va_list_tag &&) -> __va_list_tag & | file://:0:0:0:0 | __va_list_tag | <none> |
 | file://:0:0:0:0 | operator= | __va_list_tag::operator=(const __va_list_tag &) -> __va_list_tag & | file://:0:0:0:0 | __va_list_tag | <none> |

cpp/ql/test/library-tests/special_members/generated_copy/functions.expected

@@ -78,8 +78,6 @@
 | copy.cpp:111:9:111:9 | MoveAssign | deleted |  |
 | copy.cpp:111:9:111:9 | operator= | deleted |  |
 | copy.cpp:113:17:113:25 | operator= |  |  |
-| copy.cpp:120:9:120:9 | OnlyCtor |  |  |
-| copy.cpp:120:9:120:9 | OnlyCtor |  |  |
 | copy.cpp:120:9:120:9 | OnlyCtor | deleted |  |
 | copy.cpp:120:9:120:9 | operator= | deleted |  |
 | copy.cpp:126:11:126:19 | operator= |  |  |

cpp/ql/test/library-tests/syntax-zoo/dataflow-ir-consistency.expected

@@ -539,8 +539,6 @@ uniqueNodeLocation
 | file://:0:0:0:0 | p#0 | Node should have one location but has 0. |
 | file://:0:0:0:0 | p#0 | Node should have one location but has 0. |
 | file://:0:0:0:0 | p#0 | Node should have one location but has 0. |
-| file://:0:0:0:0 | p#0 | Node should have one location but has 0. |
-| file://:0:0:0:0 | p#0 | Node should have one location but has 0. |
 | file://:0:0:0:0 | p#1 | Node should have one location but has 0. |
 | file://:0:0:0:0 | p#1 | Node should have one location but has 0. |
 | file://:0:0:0:0 | p#1 | Node should have one location but has 0. |
@@ -1418,7 +1416,7 @@ uniqueNodeLocation
 | whilestmt.c:39:6:39:11 | ReturnVoid | Node should have one location but has 4. |
 | whilestmt.c:39:6:39:11 | SideEffect | Node should have one location but has 4. |
 missingLocation
-| Nodes without location: 36 |
+| Nodes without location: 34 |
 uniqueNodeToString
 | break_labels.c:2:11:2:11 | i | Node should have one toString but has 2. |
 | break_labels.c:2:11:2:11 | i | Node should have one toString but has 2. |

cpp/ql/test/library-tests/templates/CPP-202/template_args.expected

@@ -1,7 +1,7 @@
 | file://:0:0:0:0 | __va_list_tag | <none> |
 | test.cpp:3:8:3:9 | s1<<expression>> | {...} |
-| test.cpp:3:8:3:9 | s1<<unnamed>> | (null) |
+| test.cpp:3:8:3:9 | s1<<unnamed>> | (unnamed template parameter constant) |
 | test.cpp:5:8:5:9 | s2<T> | T |
 | test.cpp:5:8:5:9 | s2<T> | T |
-| test.cpp:7:8:7:9 | s3<T, <unnamed>> | (unnamed) |
+| test.cpp:7:8:7:9 | s3<T, <unnamed>> | (unnamed template parameter) |
 | test.cpp:7:8:7:9 | s3<T, <unnamed>> | T |

cpp/ql/test/library-tests/templates/decls/decls.expected

@@ -7,7 +7,7 @@
 | decls.cpp:4:30:4:34 | p#0 |
 | decls.cpp:4:30:4:34 | p#0 |
 | decls.cpp:6:17:6:17 | f |
-| decls.cpp:8:18:8:18 | (unnamed) |
+| decls.cpp:8:18:8:18 | (unnamed template parameter) |
 | decls.cpp:8:25:8:25 | g |
 | file://:0:0:0:0 | __va_list_tag |
 | file://:0:0:0:0 | auto |

cpp/ql/test/library-tests/unnamed/elements.expected

@@ -1,6 +1,6 @@
 | file://:0:0:0:0 |  | Other |
 | file://:0:0:0:0 | (global namespace) | Other |
-| file://:0:0:0:0 | <unnamed> | Other |
+| file://:0:0:0:0 | (unnamed global/namespace variable) | Other |
 | file://:0:0:0:0 | _Complex __float128 | Other |
 | file://:0:0:0:0 | _Complex double | Other |
 | file://:0:0:0:0 | _Complex float | Other |
@@ -111,8 +111,8 @@
 | test.c:0:0:0:0 | test.c | Other |
 | test.c:2:6:2:6 | a | Other |
 | test.c:2:6:2:6 | definition of a | Other |
-| test.c:2:10:2:18 | <unnamed> | Variable access |
+| test.c:2:10:2:18 | (unnamed global/namespace variable) | Variable access |
 | test.c:2:10:2:18 | array to pointer conversion | Other |
 | test.c:2:10:2:18 | initializer for a | Other |
-| test.c:2:17:2:18 | initializer for <unnamed> | Other |
+| test.c:2:17:2:18 | initializer for (unnamed global/namespace variable) | Other |
 | test.c:2:17:2:18 | {...} | Other |

csharp/autobuilder/Semmle.Autobuild.CSharp.Tests/BuildScripts.cs

@@ -363,7 +363,6 @@ namespace Semmle.Autobuild.CSharp.Tests
             string cwd = @"C:\Project")
         {
             string codeqlUpperLanguage = Language.CSharp.UpperCaseName;
-            Actions.GetEnvironmentVariable[$"CODEQL_AUTOBUILDER_{codeqlUpperLanguage}_NO_INDEXING"] = "false";
             Actions.GetEnvironmentVariable[$"CODEQL_EXTRACTOR_{codeqlUpperLanguage}_TRAP_DIR"] = "";
             Actions.GetEnvironmentVariable[$"CODEQL_EXTRACTOR_{codeqlUpperLanguage}_SOURCE_ARCHIVE_DIR"] = "";
             Actions.GetEnvironmentVariable[$"CODEQL_EXTRACTOR_{codeqlUpperLanguage}_ROOT"] = $@"C:\codeql\{codeqlUpperLanguage.ToLowerInvariant()}";
@@ -400,8 +399,6 @@ namespace Semmle.Autobuild.CSharp.Tests
             Actions.RunProcess[@"cmd.exe /C dotnet clean C:\Project\test.csproj"] = 0;
             Actions.RunProcess[@"cmd.exe /C dotnet restore C:\Project\test.csproj"] = 0;
             Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto dotnet build --no-incremental C:\Project\test.csproj"] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project\test.csproj"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
@@ -419,7 +416,7 @@ namespace Semmle.Autobuild.CSharp.Tests
             Actions.LoadXml[@"C:\Project\test.csproj"] = xml;
 
             var autobuilder = CreateAutoBuilder(true);
-            TestAutobuilderScript(autobuilder, 0, 6);
+            TestAutobuilderScript(autobuilder, 0, 4);
         }
 
         [Fact]
@@ -432,8 +429,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"dotnet clean C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"dotnet restore C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto dotnet build --no-incremental /p:UseSharedCompilation=false C:\Project/test.csproj"] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project/test.csproj"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
@@ -451,7 +446,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.LoadXml[@"C:\Project/test.csproj"] = xml;
 
             var autobuilder = CreateAutoBuilder(false);
-            TestAutobuilderScript(autobuilder, 0, 7);
+            TestAutobuilderScript(autobuilder, 0, 5);
         }
 
         [Fact]
@@ -522,8 +517,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
         public void TestLinuxBuildlessExtractionSuccess()
         {
             Actions.RunProcess[@"C:\codeql\csharp/tools/linux64/Semmle.Extraction.CSharp.Standalone --references:."] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR"] = "";
@@ -531,7 +524,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.EnumerateDirectories[@"C:\Project"] = "";
 
             var autobuilder = CreateAutoBuilder(false, buildless: "true");
-            TestAutobuilderScript(autobuilder, 0, 3);
+            TestAutobuilderScript(autobuilder, 0, 1);
         }
 
         [Fact]
@@ -552,8 +545,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
         public void TestLinuxBuildlessExtractionSolution()
         {
             Actions.RunProcess[@"C:\codeql\csharp/tools/linux64/Semmle.Extraction.CSharp.Standalone foo.sln --references:."] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR"] = "";
@@ -561,7 +552,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.EnumerateDirectories[@"C:\Project"] = "";
 
             var autobuilder = CreateAutoBuilder(false, buildless: "true", solution: "foo.sln");
-            TestAutobuilderScript(autobuilder, 0, 3);
+            TestAutobuilderScript(autobuilder, 0, 1);
         }
 
         void SkipVsWhere()
@@ -598,8 +589,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess["dotnet --list-runtimes"] = 1;
             Actions.RunProcessOut["dotnet --list-runtimes"] = "";
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto ""./build.sh --skip-tests"""] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR"] = "";
@@ -609,7 +598,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             SkipVsWhere();
 
             var autobuilder = CreateAutoBuilder(false, buildCommand: "./build.sh --skip-tests");
-            TestAutobuilderScript(autobuilder, 0, 4);
+            TestAutobuilderScript(autobuilder, 0, 2);
         }
 
         [Fact]
@@ -624,12 +613,10 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcessOut["dotnet --list-runtimes"] = "";
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto C:\Project/build/build.sh"] = 0;
             Actions.RunProcessWorkingDirectory[@"C:\odasa/tools/odasa index --auto C:\Project/build/build.sh"] = @"C:\Project/build";
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
 
             var autobuilder = CreateAutoBuilder(false);
-            TestAutobuilderScript(autobuilder, 0, 5);
+            TestAutobuilderScript(autobuilder, 0, 3);
         }
 
         [Fact]
@@ -679,12 +666,10 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR"] = "";
             Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto C:\Project\build.bat"] = 0;
             Actions.RunProcessWorkingDirectory[@"cmd.exe /C C:\odasa\tools\odasa index --auto C:\Project\build.bat"] = @"C:\Project";
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
 
             var autobuilder = CreateAutoBuilder(true);
-            TestAutobuilderScript(autobuilder, 0, 3);
+            TestAutobuilderScript(autobuilder, 0, 1);
         }
 
         [Fact]
@@ -729,8 +714,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test1.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
             Actions.RunProcess[@"cmd.exe /C C:\Project\.nuget\nuget.exe restore C:\Project\test2.sln"] = 0;
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test2.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
             Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = false;
@@ -752,7 +735,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             autobuilder.ProjectsOrSolutionsToBuild.Add(testSolution1);
             autobuilder.ProjectsOrSolutionsToBuild.Add(testSolution2);
 
-            TestAutobuilderScript(autobuilder, 0, 6);
+            TestAutobuilderScript(autobuilder, 0, 4);
         }
 
         [Fact]
@@ -762,8 +745,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test1.csproj /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
             Actions.RunProcess[@"cmd.exe /C nuget restore C:\Project\test2.csproj"] = 0;
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test2.csproj /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project\test1.csproj"] = true;
             Actions.FileExists[@"C:\Project\test2.csproj"] = true;
@@ -799,7 +780,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             var autobuilder = CreateAutoBuilder(true, msBuildArguments: "/P:Fu=Bar", msBuildTarget: "Windows", msBuildPlatform: "x86", msBuildConfiguration: "Debug",
                 vsToolsVersion: "12");
 
-            TestAutobuilderScript(autobuilder, 0, 6);
+            TestAutobuilderScript(autobuilder, 0, 4);
         }
 
         [Fact]
@@ -834,8 +815,6 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
         {
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test1.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test2.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
             Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = false;
@@ -855,15 +834,13 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             autobuilder.ProjectsOrSolutionsToBuild.Add(testSolution1);
             autobuilder.ProjectsOrSolutionsToBuild.Add(testSolution2);
 
-            TestAutobuilderScript(autobuilder, 0, 4);
+            TestAutobuilderScript(autobuilder, 0, 2);
         }
 
         [Fact]
         public void TestSkipNugetBuildless()
         {
             Actions.RunProcess[@"C:\codeql\csharp/tools/linux64/Semmle.Extraction.CSharp.Standalone foo.sln --references:. --skip-nuget"] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR"] = "";
@@ -871,7 +848,7 @@ Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.EnumerateDirectories[@"C:\Project"] = "";
 
             var autobuilder = CreateAutoBuilder(false, buildless: "true", solution: "foo.sln", nugetRestore: "false");
-            TestAutobuilderScript(autobuilder, 0, 3);
+            TestAutobuilderScript(autobuilder, 0, 1);
         }
 
 
@@ -885,8 +862,6 @@ Microsoft.NETCore.App 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"dotnet clean C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"dotnet restore C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto dotnet build --no-incremental /p:UseSharedCompilation=false --no-restore C:\Project/test.csproj"] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project/test.csproj"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
@@ -904,7 +879,7 @@ Microsoft.NETCore.App 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.LoadXml[@"C:\Project/test.csproj"] = xml;
 
             var autobuilder = CreateAutoBuilder(false, dotnetArguments: "--no-restore");  // nugetRestore=false does not work for now.
-            TestAutobuilderScript(autobuilder, 0, 7);
+            TestAutobuilderScript(autobuilder, 0, 5);
         }
 
         [Fact]
@@ -922,8 +897,6 @@ Microsoft.NETCore.App 3.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"C:\Project/.dotnet/dotnet clean C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"C:\Project/.dotnet/dotnet restore C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto C:\Project/.dotnet/dotnet build --no-incremental C:\Project/test.csproj"] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists["test.csproj"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
@@ -943,7 +916,7 @@ Microsoft.NETCore.App 3.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.DownloadFiles.Add(("https://dot.net/v1/dotnet-install.sh", "dotnet-install.sh"));
 
             var autobuilder = CreateAutoBuilder(false, dotnetVersion: "2.1.3");
-            TestAutobuilderScript(autobuilder, 0, 11);
+            TestAutobuilderScript(autobuilder, 0, 9);
         }
 
         [Fact]
@@ -964,8 +937,6 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"C:\Project/.dotnet/dotnet clean C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"C:\Project/.dotnet/dotnet restore C:\Project/test.csproj"] = 0;
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto C:\Project/.dotnet/dotnet build --no-incremental /p:UseSharedCompilation=false C:\Project/test.csproj"] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists["test.csproj"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
@@ -985,7 +956,7 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.DownloadFiles.Add(("https://dot.net/v1/dotnet-install.sh", "dotnet-install.sh"));
 
             var autobuilder = CreateAutoBuilder(false, dotnetVersion: "2.1.3");
-            TestAutobuilderScript(autobuilder, 0, 11);
+            TestAutobuilderScript(autobuilder, 0, 9);
         }
 
         [Fact]
@@ -999,8 +970,6 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"cmd.exe /C C:\Project\.dotnet\dotnet clean C:\Project\test.csproj"] = 0;
             Actions.RunProcess[@"cmd.exe /C C:\Project\.dotnet\dotnet restore C:\Project\test.csproj"] = 0;
             Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto C:\Project\.dotnet\dotnet build --no-incremental C:\Project\test.csproj"] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project\test.csproj"] = true;
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
@@ -1019,7 +988,7 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.LoadXml[@"C:\Project\test.csproj"] = xml;
 
             var autobuilder = CreateAutoBuilder(true, dotnetVersion: "2.1.3");
-            TestAutobuilderScript(autobuilder, 0, 9);
+            TestAutobuilderScript(autobuilder, 0, 7);
         }
 
         [Fact]
@@ -1028,8 +997,6 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"cmd.exe /C nuget restore C:\Project\dirs.proj"] = 1;
             Actions.RunProcess[@"cmd.exe /C C:\Project\.nuget\nuget.exe restore C:\Project\dirs.proj"] = 0;
             Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && set Platform=&& type NUL && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\dirs.proj /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\codeql\tools\java\bin\java -jar C:\codeql\csharp\tools\extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project\a\test.csproj"] = true;
             Actions.FileExists[@"C:\Project\dirs.proj"] = true;
@@ -1065,7 +1032,7 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
 
             var autobuilder = CreateAutoBuilder(true, msBuildArguments: "/P:Fu=Bar", msBuildTarget: "Windows", msBuildPlatform: "x86", msBuildConfiguration: "Debug",
                 vsToolsVersion: "12", allSolutions: "true");
-            TestAutobuilderScript(autobuilder, 0, 5);
+            TestAutobuilderScript(autobuilder, 0, 3);
         }
 
         [Fact]
@@ -1074,8 +1041,6 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.RunProcess[@"nuget restore C:\Project/dirs.proj"] = 1;
             Actions.RunProcess[@"mono C:\Project/.nuget/nuget.exe restore C:\Project/dirs.proj"] = 0;
             Actions.RunProcess[@"C:\odasa/tools/odasa index --auto msbuild C:\Project/dirs.proj /p:UseSharedCompilation=false /t:rebuild /p:MvcBuildViews=true"] = 0;
-            Actions.RunProcess[@"C:\codeql\tools\java/bin/java -jar C:\codeql\csharp/tools/extractor-asp.jar ."] = 0;
-            Actions.RunProcess[@"C:\odasa/tools/odasa index --xml --extensions config csproj props xml"] = 0;
             Actions.FileExists["csharp.log"] = true;
             Actions.FileExists[@"C:\Project/a/test.csproj"] = true;
             Actions.FileExists[@"C:\Project/dirs.proj"] = true;
@@ -1104,7 +1069,7 @@ Microsoft.NETCore.App 2.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.Ap
             Actions.LoadXml[@"C:\Project/dirs.proj"] = dirsproj;
 
             var autobuilder = CreateAutoBuilder(false);
-            TestAutobuilderScript(autobuilder, 0, 5);
+            TestAutobuilderScript(autobuilder, 0, 3);
         }
 
         [Fact]

csharp/autobuilder/Semmle.Autobuild.CSharp/AspBuildRule.cs

@@ -1,23 +0,0 @@
-using Semmle.Autobuild.Shared;
-
-namespace Semmle.Autobuild.CSharp
-{
-    /// <summary>
-    /// ASP extraction.
-    /// </summary>
-    class AspBuildRule : IBuildRule
-    {
-        public BuildScript Analyse(Autobuilder builder, bool auto)
-        {
-            var javaHome = builder.JavaHome;
-            var dist = builder.Distribution;
-
-            var command = new CommandBuilder(builder.Actions).
-                RunCommand(builder.Actions.PathCombine(javaHome, "bin", "java")).
-                Argument("-jar").
-                QuoteArgument(builder.Actions.PathCombine(dist, "tools", "extractor-asp.jar")).
-                Argument(".");
-            return command.Script;
-        }
-    }
-}

csharp/autobuilder/Semmle.Autobuild.CSharp/CSharpAutobuilder.cs

@@ -85,10 +85,7 @@ namespace Semmle.Autobuild.CSharp
                     break;
             }
 
-            return
+            return attempt;
-                attempt &
-                (() => new AspBuildRule().Analyse(this, false)) &
-                (() => new XmlBuildRule().Analyse(this, false));
         }
 
         /// <summary>

csharp/autobuilder/Semmle.Autobuild.CSharp/XmlBuildRule.cs

@@ -1,21 +0,0 @@
-using Semmle.Autobuild.Shared;
-
-namespace Semmle.Autobuild.CSharp
-{
-    /// <summary>
-    /// XML extraction.
-    /// </summary>
-    class XmlBuildRule : IBuildRule
-    {
-        public BuildScript Analyse(Autobuilder builder, bool auto)
-        {
-            if (!builder.Options.Indexing || builder.Odasa is null)
-                return BuildScript.Success;
-
-            var command = new CommandBuilder(builder.Actions).
-                RunCommand(builder.Odasa).
-                Argument("index --xml --extensions config csproj props xml");
-            return command.Script;
-        }
-    }
-}

csharp/autobuilder/Semmle.Autobuild.Shared/AutobuildOptions.cs

@@ -29,7 +29,6 @@ namespace Semmle.Autobuild.Shared
         public readonly bool NugetRestore;
 
         public readonly Language Language;
-        public readonly bool Indexing;
 
         /// <summary>
         /// Reads options from environment variables.
@@ -54,8 +53,6 @@ namespace Semmle.Autobuild.Shared
             NugetRestore = actions.GetEnvironmentVariable(prefix + "NUGET_RESTORE").AsBool("nuget_restore", true);
 
             Language = language;
-
-            Indexing = !actions.GetEnvironmentVariable($"CODEQL_AUTOBUILDER_{Language.UpperCaseName}_NO_INDEXING").AsBool("no_indexing", false);
         }
     }
 

csharp/autobuilder/Semmle.Autobuild.Shared/Autobuilder.cs

@@ -1,4 +1,4 @@
-using Semmle.Util.Logging;
+using Semmle.Util.Logging;
 using System;
 using System.Collections.Generic;
 using System.IO;
@@ -190,21 +190,10 @@ namespace Semmle.Autobuild.Shared
             });
 
             CodeQLExtractorLangRoot = Actions.GetEnvironmentVariable($"CODEQL_EXTRACTOR_{this.Options.Language.UpperCaseName}_ROOT");
-            SemmleDist = Actions.GetEnvironmentVariable("SEMMLE_DIST");
             SemmlePlatformTools = Actions.GetEnvironmentVariable("SEMMLE_PLATFORM_TOOLS");
 
             CodeQlPlatform = Actions.GetEnvironmentVariable("CODEQL_PLATFORM");
 
-            JavaHome =
-                Actions.GetEnvironmentVariable("CODEQL_JAVA_HOME") ??
-                Actions.GetEnvironmentVariable("SEMMLE_JAVA_HOME") ??
-                throw new InvalidEnvironmentException("The environment variable CODEQL_JAVA_HOME or SEMMLE_JAVA_HOME has not been set.");
-
-            Distribution =
-                CodeQLExtractorLangRoot ??
-                SemmleDist ??
-                throw new InvalidEnvironmentException($"The environment variable CODEQL_EXTRACTOR_{this.Options.Language.UpperCaseName}_ROOT or SEMMLE_DIST has not been set.");
-
             TrapDir =
                 Actions.GetEnvironmentVariable($"CODEQL_EXTRACTOR_{this.Options.Language.UpperCaseName}_TRAP_DIR") ??
                 Actions.GetEnvironmentVariable("TRAP_FOLDER") ??
@@ -275,15 +264,6 @@ namespace Semmle.Autobuild.Shared
         /// </summary>
         public string? CodeQLExtractorLangRoot { get; }
 
-        /// <summary>
-        /// Value of SEMMLE_DIST environment variable.
-        /// </summary>
-        private string? SemmleDist { get; }
-
-        public string Distribution { get; }
-
-        public string JavaHome { get; }
-
         /// <summary>
         /// Value of SEMMLE_PLATFORM_TOOLS environment variable.
         /// </summary>
@@ -298,13 +278,20 @@ namespace Semmle.Autobuild.Shared
         /// The absolute path of the odasa executable.
         /// null if we are running in CodeQL.
         /// </summary>
-        public string? Odasa => SemmleDist is null ? null : Actions.PathCombine(SemmleDist, "tools", "odasa");
+        public string? Odasa
+        {
+            get
+            {
+                var semmleDist = Actions.GetEnvironmentVariable("SEMMLE_DIST");
+                return semmleDist is null ? null : Actions.PathCombine(semmleDist, "tools", "odasa");
+            }
+        }
 
         /// <summary>
         /// Construct a command that executed the given <paramref name="cmd"/> wrapped in
         /// an <code>odasa --index</code>, unless indexing has been disabled, in which case
         /// <paramref name="cmd"/> is run directly.
         /// </summary>
-        public CommandBuilder MaybeIndex(CommandBuilder builder, string cmd) => Options.Indexing && !(Odasa is null) ? builder.IndexCommand(Odasa, cmd) : builder.RunCommand(cmd);
+        public CommandBuilder MaybeIndex(CommandBuilder builder, string cmd) => Odasa is null ? builder.RunCommand(cmd) : builder.IndexCommand(Odasa, cmd);
     }
 }

csharp/extractor/Semmle.Extraction.CIL/Context.cs

@@ -43,7 +43,7 @@ namespace Semmle.Extraction.CIL
             namespaceFactory = new CachedFunction<StringHandle, Entities.Namespace>(n => CreateNamespace(mdReader.GetString(n)));
             namespaceDefinitionFactory = new CachedFunction<NamespaceDefinitionHandle, Entities.Namespace>(CreateNamespace);
             sourceFiles = new CachedFunction<PDB.ISourceFile, Entities.PdbSourceFile>(path => new Entities.PdbSourceFile(this, path));
-            folders = new CachedFunction<string, Entities.Folder>(path => new Entities.Folder(this, path));
+            folders = new CachedFunction<PathTransformer.ITransformedPath, Entities.Folder>(path => new Entities.Folder(this, path));
             sourceLocations = new CachedFunction<PDB.Location, Entities.PdbSourceLocation>(location => new Entities.PdbSourceLocation(this, location));
 
             defaultGenericContext = new EmptyContext(this);

csharp/extractor/Semmle.Extraction.CIL/Entities/Assembly.cs

@@ -5,6 +5,7 @@ using Semmle.Util.Logging;
 using System;
 using Semmle.Extraction.Entities;
 using System.IO;
+using Semmle.Util;
 
 namespace Semmle.Extraction.CIL.Entities
 {
@@ -134,9 +135,12 @@ namespace Semmle.Extraction.CIL.Entities
             extracted = false;
             try
             {
-                var extractor = new Extractor(false, assemblyPath, logger);
+                var canonicalPathCache = CanonicalPathCache.Create(logger, 1000);
-                var project = layout.LookupProjectOrDefault(assemblyPath);
+                var pathTransformer = new PathTransformer(canonicalPathCache);
-                using (var trapWriter = project.CreateTrapWriter(logger, assemblyPath + ".cil", true, trapCompression))
+                var extractor = new Extractor(false, assemblyPath, logger, pathTransformer);
+                var transformedAssemblyPath = pathTransformer.Transform(assemblyPath);
+                var project = layout.LookupProjectOrDefault(transformedAssemblyPath);
+                using (var trapWriter = project.CreateTrapWriter(logger, transformedAssemblyPath.WithSuffix(".cil"), true, trapCompression))
                 {
                     trapFile = trapWriter.TrapFile;
                     if (nocache || !System.IO.File.Exists(trapFile))

csharp/extractor/Semmle.Extraction.CIL/Entities/File.cs

@@ -1,4 +1,4 @@
-using System.Collections.Generic;
+using System.Collections.Generic;
 using System.IO;
 
 namespace Semmle.Extraction.CIL.Entities
@@ -13,37 +13,38 @@ namespace Semmle.Extraction.CIL.Entities
 
     public class File : LabelledEntity, IFile
     {
-        protected readonly string path;
+        protected readonly string OriginalPath;
+        protected readonly PathTransformer.ITransformedPath TransformedPath;
 
         public File(Context cx, string path) : base(cx)
         {
-            this.path = Semmle.Extraction.Entities.File.PathAsDatabaseString(path);
+            this.OriginalPath = path;
+            TransformedPath = cx.cx.Extractor.PathTransformer.Transform(OriginalPath);
         }
 
         public override void WriteId(TextWriter trapFile)
         {
-            trapFile.Write(Semmle.Extraction.Entities.File.PathAsDatabaseId(path));
+            trapFile.Write(TransformedPath.DatabaseId);
         }
 
         public override bool Equals(object? obj)
         {
-            return GetType() == obj?.GetType() && path == ((File)obj).path;
+            return GetType() == obj?.GetType() && OriginalPath == ((File)obj).OriginalPath;
         }
 
-        public override int GetHashCode() => 11 * path.GetHashCode();
+        public override int GetHashCode() => 11 * OriginalPath.GetHashCode();
 
         public override IEnumerable<IExtractionProduct> Contents
         {
             get
             {
-                var directoryName = System.IO.Path.GetDirectoryName(path);
+                if (TransformedPath.ParentDirectory is PathTransformer.ITransformedPath dir)
-                if (directoryName is null)
+                {
-                    throw new InternalError($"Directory name for path '{path}' is null.");
+                    var parent = cx.CreateFolder(dir);
-
+                    yield return parent;
-                var parent = cx.CreateFolder(directoryName);
+                    yield return Tuples.containerparent(parent, this);
-                yield return parent;
+                }
-                yield return Tuples.containerparent(parent, this);
+                yield return Tuples.files(this, TransformedPath.Value, TransformedPath.NameWithoutExtension, TransformedPath.Extension);
-                yield return Tuples.files(this, path, System.IO.Path.GetFileNameWithoutExtension(path), System.IO.Path.GetExtension(path).Substring(1));
             }
         }
 
@@ -69,9 +70,9 @@ namespace Semmle.Extraction.CIL.Entities
                 var text = file.Contents;
 
                 if (text == null)
-                    cx.cx.Extractor.Logger.Log(Util.Logging.Severity.Warning, string.Format("PDB source file {0} could not be found", path));
+                    cx.cx.Extractor.Logger.Log(Util.Logging.Severity.Warning, string.Format("PDB source file {0} could not be found", OriginalPath));
                 else
-                    cx.cx.TrapWriter.Archive(path, text);
+                    cx.cx.TrapWriter.Archive(TransformedPath, text);
 
                 yield return Tuples.file_extraction_mode(this, 2);
             }

csharp/extractor/Semmle.Extraction.CIL/Entities/Folder.cs

@@ -9,16 +9,16 @@ namespace Semmle.Extraction.CIL.Entities
 
     public sealed class Folder : LabelledEntity, IFolder
     {
-        readonly string path;
+        readonly PathTransformer.ITransformedPath TransformedPath;
 
-        public Folder(Context cx, string path) : base(cx)
+        public Folder(Context cx, PathTransformer.ITransformedPath path) : base(cx)
         {
-            this.path = path;
+            this.TransformedPath = path;
         }
 
         public override void WriteId(TextWriter trapFile)
         {
-            trapFile.Write(Semmle.Extraction.Entities.File.PathAsDatabaseId(path));
+            trapFile.Write(TransformedPath.DatabaseId);
         }
 
         public override string IdSuffix => ";folder";
@@ -27,25 +27,21 @@ namespace Semmle.Extraction.CIL.Entities
         {
             get
             {
-                // On Posix, we could get a Windows directory of the form "C:"
+                if (TransformedPath.ParentDirectory is PathTransformer.ITransformedPath parent)
-                bool windowsDriveLetter = path.Length == 2 && char.IsLetter(path[0]) && path[1] == ':';
-
-                var parent = Path.GetDirectoryName(path);
-                if (parent != null && !windowsDriveLetter)
                 {
                     var parentFolder = cx.CreateFolder(parent);
                     yield return parentFolder;
                     yield return Tuples.containerparent(parentFolder, this);
                 }
-                yield return Tuples.folders(this, Semmle.Extraction.Entities.File.PathAsDatabaseString(path), Path.GetFileName(path));
+                yield return Tuples.folders(this, TransformedPath.Value, TransformedPath.NameWithoutExtension);
             }
         }
 
         public override bool Equals(object? obj)
         {
-            return obj is Folder folder && path == folder.path;
+            return obj is Folder folder && TransformedPath == folder.TransformedPath;
         }
 
-        public override int GetHashCode() => path.GetHashCode();
+        public override int GetHashCode() => TransformedPath.GetHashCode();
     }
 }

csharp/extractor/Semmle.Extraction.CIL/Factories.cs

@@ -201,7 +201,7 @@ namespace Semmle.Extraction.CIL
 
         #region Locations
         readonly CachedFunction<PDB.ISourceFile, PdbSourceFile> sourceFiles;
-        readonly CachedFunction<string, Folder> folders;
+        readonly CachedFunction<PathTransformer.ITransformedPath, Folder> folders;
         readonly CachedFunction<PDB.Location, PdbSourceLocation> sourceLocations;
 
         /// <summary>
@@ -216,7 +216,7 @@ namespace Semmle.Extraction.CIL
         /// </summary>
         /// <param name="path">The path of the folder.</param>
         /// <returns>A folder entity.</returns>
-        public Folder CreateFolder(string path) => folders[path];
+        public Folder CreateFolder(PathTransformer.ITransformedPath path) => folders[path];
 
         /// <summary>
         /// Creates a source location.

csharp/extractor/Semmle.Extraction.CSharp/Analyser.cs

@@ -27,13 +27,16 @@ namespace Semmle.Extraction.CSharp
 
         public readonly bool AddAssemblyTrapPrefix;
 
-        public Analyser(IProgressMonitor pm, ILogger logger, bool addAssemblyTrapPrefix)
+        public readonly PathTransformer PathTransformer;
+
+        public Analyser(IProgressMonitor pm, ILogger logger, bool addAssemblyTrapPrefix, PathTransformer pathTransformer)
         {
             Logger = logger;
             AddAssemblyTrapPrefix = addAssemblyTrapPrefix;
             Logger.Log(Severity.Info, "EXTRACTION STARTING at {0}", DateTime.Now);
             stopWatch.Start();
             progressMonitor = pm;
+            PathTransformer = pathTransformer;
         }
 
         CSharpCompilation compilation;
@@ -67,7 +70,7 @@ namespace Semmle.Extraction.CSharp
             layout = new Layout();
             this.options = options;
             this.compilation = compilation;
-            extractor = new Extraction.Extractor(false, GetOutputName(compilation, commandLineArguments), Logger);
+            extractor = new Extraction.Extractor(false, GetOutputName(compilation, commandLineArguments), Logger, PathTransformer);
             LogDiagnostics();
 
             SetReferencePaths();
@@ -117,7 +120,7 @@ namespace Semmle.Extraction.CSharp
         {
             compilation = compilationIn;
             layout = new Layout();
-            extractor = new Extraction.Extractor(true, null, Logger);
+            extractor = new Extraction.Extractor(true, null, Logger, PathTransformer);
             this.options = options;
             LogExtractorInfo(Extraction.Extractor.Version);
             SetReferencePaths();
@@ -230,9 +233,10 @@ namespace Semmle.Extraction.CSharp
             try
             {
                 var assemblyPath = extractor.OutputPath;
+                var transformedAssemblyPath = PathTransformer.Transform(assemblyPath);
                 var assembly = compilation.Assembly;
-                var projectLayout = layout.LookupProjectOrDefault(assemblyPath);
+                var projectLayout = layout.LookupProjectOrDefault(transformedAssemblyPath);
-                var trapWriter = projectLayout.CreateTrapWriter(Logger, assemblyPath, true, options.TrapCompression);
+                var trapWriter = projectLayout.CreateTrapWriter(Logger, transformedAssemblyPath, true, options.TrapCompression);
                 compilationTrapFile = trapWriter;  // Dispose later
                 var cx = extractor.CreateContext(compilation.Clone(), trapWriter, new AssemblyScope(assembly, assemblyPath, true), AddAssemblyTrapPrefix);
 
@@ -260,8 +264,9 @@ namespace Semmle.Extraction.CSharp
                 stopwatch.Start();
 
                 var assemblyPath = r.FilePath;
-                var projectLayout = layout.LookupProjectOrDefault(assemblyPath);
+                var transformedAssemblyPath = PathTransformer.Transform(assemblyPath);
-                using (var trapWriter = projectLayout.CreateTrapWriter(Logger, assemblyPath, true, options.TrapCompression))
+                var projectLayout = layout.LookupProjectOrDefault(transformedAssemblyPath);
+                using (var trapWriter = projectLayout.CreateTrapWriter(Logger, transformedAssemblyPath, true, options.TrapCompression))
                 {
                     var skipExtraction = options.Cache && File.Exists(trapWriter.TrapFile);
 
@@ -360,16 +365,17 @@ namespace Semmle.Extraction.CSharp
                 var stopwatch = new Stopwatch();
                 stopwatch.Start();
                 var sourcePath = tree.FilePath;
+                var transformedSourcePath = PathTransformer.Transform(sourcePath);
 
-                var projectLayout = layout.LookupProjectOrNull(sourcePath);
+                var projectLayout = layout.LookupProjectOrNull(transformedSourcePath);
                 bool excluded = projectLayout == null;
-                string trapPath = excluded ? "" : projectLayout.GetTrapPath(Logger, sourcePath, options.TrapCompression);
+                string trapPath = excluded ? "" : projectLayout.GetTrapPath(Logger, transformedSourcePath, options.TrapCompression);
                 bool upToDate = false;
 
                 if (!excluded)
                 {
                     // compilation.Clone() is used to allow symbols to be garbage collected.
-                    using (var trapWriter = projectLayout.CreateTrapWriter(Logger, sourcePath, false, options.TrapCompression))
+                    using (var trapWriter = projectLayout.CreateTrapWriter(Logger, transformedSourcePath, false, options.TrapCompression))
                     {
                         upToDate = options.Fast && FileIsUpToDate(sourcePath, trapWriter.TrapFile);
 

csharp/extractor/Semmle.Extraction.CSharp/Entities/Compilation.cs

@@ -3,6 +3,7 @@ using System;
 using System.Collections.Generic;
 using System.IO;
 using System.Linq;
+using Semmle.Util;
 
 namespace Semmle.Extraction.CSharp.Entities
 {
@@ -22,7 +23,7 @@ namespace Semmle.Extraction.CSharp.Entities
         {
             Extraction.Entities.Assembly.CreateOutputAssembly(cx);
 
-            trapFile.compilations(this, Extraction.Entities.File.PathAsDatabaseString(cwd));
+            trapFile.compilations(this, FileUtils.ConvertToUnix(cwd));
 
             // Arguments
             int index = 0;

csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/ImplicitCast.cs

@@ -86,6 +86,15 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
                 return new ImplicitCast(info);
             }
 
+            if (conversion.IsIdentity && conversion.IsImplicit &&
+                convertedType.Symbol is IPointerTypeSymbol &&
+                !(resolvedType.Symbol is IPointerTypeSymbol))
+            {
+                // int[] -> int*
+                // string -> char*
+                return new ImplicitCast(info);
+            }
+
             // Default: Just create the expression without a conversion.
             return Factory.Create(info);
         }

csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Switch.cs

@@ -18,10 +18,9 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
         protected override void PopulateExpression(TextWriter trapFile)
         {
             SwitchedExpr = Expression.Create(cx, Syntax.GoverningExpression, this, -1);
-            int child = 0;
+            for (var i = 0; i < Syntax.Arms.Count; i++)
-            foreach (var arm in Syntax.Arms)
             {
-                new SwitchCase(cx, arm, this, child++);
+                new SwitchCase(cx, Syntax.Arms[i], this, i);
             }
         }
     }
@@ -29,7 +28,9 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
     class SwitchCase : Expression
     {
         internal SwitchCase(Context cx, SwitchExpressionArmSyntax arm, Switch parent, int child) :
-            base(new ExpressionInfo(cx, parent.SwitchedExpr.Type, cx.Create(arm.GetLocation()), ExprKind.SWITCH_CASE, parent, child, false, null))
+            base(new ExpressionInfo(
+                cx, Entities.Type.Create(cx, cx.GetType(arm.Expression)), cx.Create(arm.GetLocation()),
+                ExprKind.SWITCH_CASE, parent, child, false, null))
         {
             cx.CreatePattern(arm.Pattern, this, 0);
             if (arm.WhenClause is WhenClauseSyntax when)

csharp/extractor/Semmle.Extraction.CSharp/Entities/Field.cs

@@ -68,13 +68,12 @@ namespace Semmle.Extraction.CSharp.Entities
                 Context.PopulateLater(() =>
                 {
                     var loc = Context.Create(initializer.GetLocation());
-                    var simpleAssignExpr = new Expression(new ExpressionInfo(Context, Type, loc, ExprKind.SIMPLE_ASSIGN, this, child++, false, null));
+
-                    Expression.CreateFromNode(new ExpressionNodeInfo(Context, initializer.Initializer.Value, simpleAssignExpr, 0));
+                    var fieldAccess = AddInitializerAssignment(trapFile, initializer.Initializer.Value, loc, null, ref child);
-                    var access = new Expression(new ExpressionInfo(Context, Type, Location, ExprKind.FIELD_ACCESS, simpleAssignExpr, 1, false, null));
+
-                    trapFile.expr_access(access, this);
                     if (!symbol.IsStatic)
                     {
-                        This.CreateImplicit(Context, Entities.Type.Create(Context, symbol.ContainingType), Location, access, -1);
+                        This.CreateImplicit(Context, Entities.Type.Create(Context, symbol.ContainingType), Location, fieldAccess, -1);
                     }
                 });
             }
@@ -85,8 +84,13 @@ namespace Semmle.Extraction.CSharp.Entities
                 Where(n => n.EqualsValue != null))
             {
                 // Mark fields that have explicit initializers.
-                var expr = new Expression(new ExpressionInfo(Context, Type, Context.Create(initializer.EqualsValue.Value.FixedLocation()), Kinds.ExprKind.FIELD_ACCESS, this, child++, false, null));
+                var constValue = symbol.HasConstantValue
-                trapFile.expr_access(expr, this);
+                    ? Expression.ValueAsString(symbol.ConstantValue)
+                    : null;
+
+                var loc = Context.Create(initializer.GetLocation());
+
+                AddInitializerAssignment(trapFile, initializer.EqualsValue.Value, loc, constValue, ref child);
             }
 
             if (IsSourceDeclaration)
@@ -96,6 +100,16 @@ namespace Semmle.Extraction.CSharp.Entities
                     TypeMention.Create(Context, syntax.Type, this, Type);
         }
 
+        private Expression AddInitializerAssignment(TextWriter trapFile, ExpressionSyntax initializer, Extraction.Entities.Location loc,
+            string constValue, ref int child)
+        {
+            var simpleAssignExpr = new Expression(new ExpressionInfo(Context, Type, loc, ExprKind.SIMPLE_ASSIGN, this, child++, false, constValue));
+            Expression.CreateFromNode(new ExpressionNodeInfo(Context, initializer, simpleAssignExpr, 0));
+            var access = new Expression(new ExpressionInfo(Context, Type, Location, ExprKind.FIELD_ACCESS, simpleAssignExpr, 1, false, constValue));
+            trapFile.expr_access(access, this);
+            return access;
+        }
+
         readonly Lazy<AnnotatedType> type;
         public AnnotatedType Type => type.Value;
 

csharp/extractor/Semmle.Extraction.CSharp/Entities/LocalFunction.cs

@@ -51,5 +51,7 @@ namespace Semmle.Extraction.CSharp.Entities
             trapFile.local_functions(this, symbol.Name, returnType, originalDefinition);
             ExtractRefReturn(trapFile);
         }
+
+        public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.NeedsLabel;
     }
 }

csharp/extractor/Semmle.Extraction.CSharp/Extractor.cs

@@ -76,16 +76,16 @@ namespace Semmle.Extraction.CSharp
                 return ExitCode.Ok;
             }
 
-            using (var analyser = new Analyser(new LogProgressMonitor(logger), logger, commandLineArguments.AssemblySensitiveTrap))
+            var canonicalPathCache = CanonicalPathCache.Create(logger, 1000);
+            var pathTransformer = new PathTransformer(canonicalPathCache);
+
+            using (var analyser = new Analyser(new LogProgressMonitor(logger), logger, commandLineArguments.AssemblySensitiveTrap, pathTransformer))
             using (var references = new BlockingCollection<MetadataReference>())
             {
                 try
                 {
                     var compilerVersion = new CompilerVersion(commandLineArguments);
 
-                    bool preserveSymlinks = Environment.GetEnvironmentVariable("SEMMLE_PRESERVE_SYMLINKS") == "true";
-                    var canonicalPathCache = CanonicalPathCache.Create(logger, 1000, preserveSymlinks ? CanonicalPathCache.Symlinks.Preserve : CanonicalPathCache.Symlinks.Follow);
-
                     if (compilerVersion.SkipExtraction)
                     {
                         logger.Log(Severity.Warning, "  Unrecognized compiler '{0}' because {1}", compilerVersion.SpecifiedCompiler, compilerVersion.SkipReason);
@@ -318,7 +318,10 @@ namespace Semmle.Extraction.CSharp
             ILogger logger,
             CommonOptions options)
         {
-            using (var analyser = new Analyser(pm, logger, false))
+            var canonicalPathCache = CanonicalPathCache.Create(logger, 1000);
+            var pathTransformer = new PathTransformer(canonicalPathCache);
+
+            using (var analyser = new Analyser(pm, logger, false, pathTransformer))
             using (var references = new BlockingCollection<MetadataReference>())
             {
                 try

csharp/extractor/Semmle.Extraction.Tests/FilePattern.cs

@@ -0,0 +1,48 @@
+using Xunit;
+
+namespace Semmle.Extraction.Tests
+{
+    public class FilePatternTests
+    {
+        [Fact]
+        public void TestRegexCompilation()
+        {
+            var fp = new FilePattern("/hadoop*");
+            Assert.Equal("^hadoop[^/]*.*", fp.RegexPattern);
+            fp = new FilePattern("**/org/apache/hadoop");
+            Assert.Equal("^.*/org/apache/hadoop.*", fp.RegexPattern);
+            fp = new FilePattern("hadoop-common/**/test//    ");
+            Assert.Equal("^hadoop-common/.*/test(?<doubleslash>/).*", fp.RegexPattern);
+            fp = new FilePattern(@"-C:\agent\root\asdf//");
+            Assert.Equal("^C:/agent/root/asdf(?<doubleslash>/).*", fp.RegexPattern);
+            fp = new FilePattern(@"-C:\agent+\[root]\asdf//");
+            Assert.Equal(@"^C:/agent\+/\[root]/asdf(?<doubleslash>/).*", fp.RegexPattern);
+        }
+
+        [Fact]
+        public void TestMatching()
+        {
+            var fp1 = new FilePattern(@"C:\agent\root\abc//");
+            var fp2 = new FilePattern(@"C:\agent\root\def//ghi");
+            var patterns = new[] { fp1, fp2 };
+
+            var success = FilePattern.Matches(patterns, @"C:\agent\root\abc\file.cs", out var s);
+            Assert.True(success);
+            Assert.Equal("/file.cs", s);
+
+            success = FilePattern.Matches(patterns, @"C:\agent\root\def\ghi\file.cs", out s);
+            Assert.True(success);
+            Assert.Equal("/ghi/file.cs", s);
+
+            success = FilePattern.Matches(patterns, @"C:\agent\root\def\file.cs", out s);
+            Assert.False(success);
+        }
+
+        [Fact]
+        public void TestInvalidPatterns()
+        {
+            Assert.Throws<InvalidFilePatternException>(() => new FilePattern("/abc//def//ghi"));
+            Assert.Throws<InvalidFilePatternException>(() => new FilePattern("/abc**def"));
+        }
+    }
+}

csharp/extractor/Semmle.Extraction.Tests/Layout.cs

@@ -1,10 +1,30 @@
-using System.IO;
+using System.IO;
 using Xunit;
 using Semmle.Util.Logging;
 using System.Runtime.InteropServices;
 
 namespace Semmle.Extraction.Tests
 {
+    struct TransformedPathStub : PathTransformer.ITransformedPath
+    {
+        readonly string value;
+        public TransformedPathStub(string value) => this.value = value;
+        public string Value => value;
+
+        public string Extension => throw new System.NotImplementedException();
+
+        public string NameWithoutExtension => throw new System.NotImplementedException();
+
+        public PathTransformer.ITransformedPath ParentDirectory => throw new System.NotImplementedException();
+
+        public string DatabaseId => throw new System.NotImplementedException();
+
+        public PathTransformer.ITransformedPath WithSuffix(string suffix)
+        {
+            throw new System.NotImplementedException();
+        }
+    }
+
     public class Layout
     {
         readonly ILogger Logger = new LoggerMock();
@@ -13,12 +33,12 @@ namespace Semmle.Extraction.Tests
         public void TestDefaultLayout()
         {
             var layout = new Semmle.Extraction.Layout(null, null, null);
-            var project = layout.LookupProjectOrNull("foo.cs");
+            var project = layout.LookupProjectOrNull(new TransformedPathStub("foo.cs"));
 
             Assert.NotNull(project);
 
             // All files are mapped when there's no layout file.
-            Assert.True(layout.FileInLayout("foo.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("foo.cs")));
 
             // Test trap filename
             var tmpDir = Path.GetTempPath();
@@ -30,13 +50,13 @@ namespace Semmle.Extraction.Tests
                 Assert.NotEqual(Directory.GetCurrentDirectory(), tmpDir);
                 return;
             }
-            var f1 = project!.GetTrapPath(Logger, "foo.cs", TrapWriter.CompressionMode.Gzip);
+            var f1 = project!.GetTrapPath(Logger, new TransformedPathStub("foo.cs"), TrapWriter.CompressionMode.Gzip);
-            var g1 = TrapWriter.NestPaths(Logger, tmpDir, "foo.cs.trap.gz", TrapWriter.InnerPathComputation.ABSOLUTE);
+            var g1 = TrapWriter.NestPaths(Logger, tmpDir, "foo.cs.trap.gz");
             Assert.Equal(f1, g1);
 
             // Test trap file generation
-            var trapwriterFilename = project.GetTrapPath(Logger, "foo.cs", TrapWriter.CompressionMode.Gzip);
+            var trapwriterFilename = project.GetTrapPath(Logger, new TransformedPathStub("foo.cs"), TrapWriter.CompressionMode.Gzip);
-            using (var trapwriter = project.CreateTrapWriter(Logger, "foo.cs", false, TrapWriter.CompressionMode.Gzip))
+            using (var trapwriter = project.CreateTrapWriter(Logger, new TransformedPathStub("foo.cs"), false, TrapWriter.CompressionMode.Gzip))
             {
                 trapwriter.Emit("1=*");
                 Assert.False(File.Exists(trapwriterFilename));
@@ -65,25 +85,24 @@ namespace Semmle.Extraction.Tests
             var layout = new Semmle.Extraction.Layout(null, null, "layout.txt");
 
             // Test general pattern matching
-            Assert.True(layout.FileInLayout("bar.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("bar.cs")));
-            Assert.False(layout.FileInLayout("foo.cs"));
+            Assert.False(layout.FileInLayout(new TransformedPathStub("foo.cs")));
-            Assert.False(layout.FileInLayout("goo.cs"));
+            Assert.False(layout.FileInLayout(new TransformedPathStub("goo.cs")));
-            Assert.False(layout.FileInLayout("excluded/bar.cs"));
+            Assert.False(layout.FileInLayout(new TransformedPathStub("excluded/bar.cs")));
-            Assert.True(layout.FileInLayout("excluded/foo.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("excluded/foo.cs")));
-            Assert.True(layout.FileInLayout("included/foo.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("included/foo.cs")));
 
             // Test the trap file
-            var project = layout.LookupProjectOrNull("bar.cs");
+            var project = layout.LookupProjectOrNull(new TransformedPathStub("bar.cs"));
             Assert.NotNull(project);
-
+            var trapwriterFilename = project!.GetTrapPath(Logger, new TransformedPathStub("bar.cs"), TrapWriter.CompressionMode.Gzip);
-            var trapwriterFilename = project!.GetTrapPath(Logger, "bar.cs", TrapWriter.CompressionMode.Gzip);
+            Assert.Equal(TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap"), "bar.cs.trap.gz"),
-            Assert.Equal(TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap"), "bar.cs.trap.gz", TrapWriter.InnerPathComputation.ABSOLUTE),
                 trapwriterFilename);
 
             // Test the source archive
-            var trapWriter = project.CreateTrapWriter(Logger, "bar.cs", false, TrapWriter.CompressionMode.Gzip);
+            var trapWriter = project.CreateTrapWriter(Logger, new TransformedPathStub("bar.cs"), false, TrapWriter.CompressionMode.Gzip);
-            trapWriter.Archive("layout.txt", System.Text.Encoding.ASCII);
+            trapWriter.Archive("layout.txt", new TransformedPathStub("layout.txt"), System.Text.Encoding.ASCII);
-            var writtenFile = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\archive"), "layout.txt", TrapWriter.InnerPathComputation.ABSOLUTE);
+            var writtenFile = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\archive"), "layout.txt");
             Assert.True(File.Exists(writtenFile));
             File.Delete("layout.txt");
         }
@@ -93,11 +112,11 @@ namespace Semmle.Extraction.Tests
         {
             // When you specify both a trap file and a layout, use the trap file.
             var layout = new Semmle.Extraction.Layout(Path.GetFullPath("snapshot\\trap"), null, "something.txt");
-            Assert.True(layout.FileInLayout("bar.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("bar.cs")));
-            var subProject = layout.LookupProjectOrNull("foo.cs");
+            var subProject = layout.LookupProjectOrNull(new TransformedPathStub("foo.cs"));
             Assert.NotNull(subProject);
-            var f1 = subProject!.GetTrapPath(Logger, "foo.cs", TrapWriter.CompressionMode.Gzip);
+            var f1 = subProject!.GetTrapPath(Logger, new TransformedPathStub("foo.cs"), TrapWriter.CompressionMode.Gzip);
-            var g1 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap"), "foo.cs.trap.gz", TrapWriter.InnerPathComputation.ABSOLUTE);
+            var g1 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap"), "foo.cs.trap.gz");
             Assert.Equal(f1, g1);
         }
 
@@ -123,30 +142,30 @@ namespace Semmle.Extraction.Tests
             var layout = new Semmle.Extraction.Layout(null, null, "layout.txt");
 
             // Use Section 2
-            Assert.True(layout.FileInLayout("bar.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("bar.cs")));
-            var subProject = layout.LookupProjectOrNull("bar.cs");
+            var subProject = layout.LookupProjectOrNull(new TransformedPathStub("bar.cs"));
             Assert.NotNull(subProject);
-            var f1 = subProject!.GetTrapPath(Logger, "bar.cs", TrapWriter.CompressionMode.Gzip);
+            var f1 = subProject!.GetTrapPath(Logger, new TransformedPathStub("bar.cs"), TrapWriter.CompressionMode.Gzip);
-            var g1 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap2"), "bar.cs.trap.gz", TrapWriter.InnerPathComputation.ABSOLUTE);
+            var g1 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap2"), "bar.cs.trap.gz");
             Assert.Equal(f1, g1);
 
             // Use Section 1
-            Assert.True(layout.FileInLayout("foo.cs"));
+            Assert.True(layout.FileInLayout(new TransformedPathStub("foo.cs")));
-            subProject = layout.LookupProjectOrNull("foo.cs");
+            subProject = layout.LookupProjectOrNull(new TransformedPathStub("foo.cs"));
             Assert.NotNull(subProject);
-            var f2 = subProject!.GetTrapPath(Logger, "foo.cs", TrapWriter.CompressionMode.Gzip);
+            var f2 = subProject!.GetTrapPath(Logger, new TransformedPathStub("foo.cs"), TrapWriter.CompressionMode.Gzip);
-            var g2 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap1"), "foo.cs.trap.gz", TrapWriter.InnerPathComputation.ABSOLUTE);
+            var g2 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap1"), "foo.cs.trap.gz");
             Assert.Equal(f2, g2);
 
             // boo.dll is not in the layout, so use layout from first section.
-            Assert.False(layout.FileInLayout("boo.dll"));
+            Assert.False(layout.FileInLayout(new TransformedPathStub("boo.dll")));
-            var f3 = layout.LookupProjectOrDefault("boo.dll").GetTrapPath(Logger, "boo.dll", TrapWriter.CompressionMode.Gzip);
+            var f3 = layout.LookupProjectOrDefault(new TransformedPathStub("boo.dll")).GetTrapPath(Logger, new TransformedPathStub("boo.dll"), TrapWriter.CompressionMode.Gzip);
-            var g3 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap1"), "boo.dll.trap.gz", TrapWriter.InnerPathComputation.ABSOLUTE);
+            var g3 = TrapWriter.NestPaths(Logger, Path.GetFullPath("snapshot\\trap1"), "boo.dll.trap.gz");
             Assert.Equal(f3, g3);
 
             // boo.cs is not in the layout, so return null
-            Assert.False(layout.FileInLayout("boo.cs"));
+            Assert.False(layout.FileInLayout(new TransformedPathStub("boo.cs")));
-            Assert.Null(layout.LookupProjectOrNull("boo.cs"));
+            Assert.Null(layout.LookupProjectOrNull(new TransformedPathStub("boo.cs")));
         }
 
         [Fact]

csharp/extractor/Semmle.Extraction.Tests/PathTransformer.cs

@@ -0,0 +1,45 @@
+using Semmle.Util;
+using Xunit;
+
+namespace Semmle.Extraction.Tests
+{
+    class PathCacheStub : IPathCache
+    {
+        public string GetCanonicalPath(string path) => path;
+    }
+
+    public class PathTransformerTests
+    {
+        [Fact]
+        public void TestTransformerFile()
+        {
+            var spec = new string[]
+            {
+                @"#D:\src",
+                @"C:\agent*\src//",
+                @"-C:\agent*\src\external",
+                @"",
+                @"#empty",
+                @"",
+                @"#src2",
+                @"/agent*//src",
+                @"",
+                @"#optsrc",
+                @"opt/src//"
+            };
+
+            var pathTransformer = new PathTransformer(new PathCacheStub(), spec);
+
+            // Windows-style matching
+            Assert.Equal(@"C:/bar.cs", pathTransformer.Transform(@"C:\bar.cs").Value);
+            Assert.Equal("D:/src/file.cs", pathTransformer.Transform(@"C:\agent42\src\file.cs").Value);
+            Assert.Equal("D:/src/file.cs", pathTransformer.Transform(@"C:\agent43\src\file.cs").Value);
+            Assert.Equal(@"C:/agent43/src/external/file.cs", pathTransformer.Transform(@"C:\agent43\src\external\file.cs").Value);
+
+            // Linux-style matching
+            Assert.Equal(@"src2/src/file.cs", pathTransformer.Transform(@"/agent/src/file.cs").Value);
+            Assert.Equal(@"src2/src/file.cs", pathTransformer.Transform(@"/agent42/src/file.cs").Value);
+            Assert.Equal(@"optsrc/file.cs", pathTransformer.Transform(@"/opt/src/file.cs").Value);
+        }
+    }
+}

csharp/extractor/Semmle.Extraction.Tests/TrapWriter.cs

@@ -14,7 +14,7 @@ namespace Semmle.Extraction.Tests
             string tempDir = System.IO.Path.GetTempPath();
             string root1, root2, root3;
 
-            if(Win32.IsWindows())
+            if (Win32.IsWindows())
             {
                 root1 = "E:";
                 root2 = "e:";
@@ -27,32 +27,21 @@ namespace Semmle.Extraction.Tests
                 root3 = "/";
             }
 
-            string formattedTempDir = tempDir.Replace('/', '\\').Replace(':', '_').Trim('\\');
-
             var logger = new LoggerMock();
-            System.IO.Directory.SetCurrentDirectory(tempDir);
 
-            if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
+            Assert.Equal($@"C:\Temp\source_archive\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", "def.cs").Replace('/', '\\'));
-            {
-                // `Directory.SetCurrentDirectory()` doesn't seem to work on macOS,
-                // so disable this test on macOS, for now
-                Assert.NotEqual(Directory.GetCurrentDirectory(), tempDir);
-                return;
-            }
 
-            Assert.Equal($@"C:\Temp\source_archive\{formattedTempDir}\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", "def.cs", TrapWriter.InnerPathComputation.ABSOLUTE).Replace('/','\\'));
+            Assert.Equal(@"C:\Temp\source_archive\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", "def.cs").Replace('/', '\\'));
 
-            Assert.Equal(@"C:\Temp\source_archive\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", "def.cs", TrapWriter.InnerPathComputation.RELATIVE).Replace('/', '\\'));
+            Assert.Equal(@"C:\Temp\source_archive\E_\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root1}\source\def.cs").Replace('/', '\\'));
 
-            Assert.Equal(@"C:\Temp\source_archive\E_\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root1}\source\def.cs", TrapWriter.InnerPathComputation.ABSOLUTE).Replace('/', '\\'));
+            Assert.Equal(@"C:\Temp\source_archive\e_\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root2}\source\def.cs").Replace('/', '\\'));
 
-            Assert.Equal(@"C:\Temp\source_archive\e_\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root2}\source\def.cs", TrapWriter.InnerPathComputation.RELATIVE).Replace('/', '\\'));
+            Assert.Equal(@"C:\Temp\source_archive\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root3}source\def.cs").Replace('/', '\\'));
 
-            Assert.Equal(@"C:\Temp\source_archive\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root3}source\def.cs", TrapWriter.InnerPathComputation.ABSOLUTE).Replace('/', '\\'));
+            Assert.Equal(@"C:\Temp\source_archive\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root3}source\def.cs").Replace('/', '\\'));
 
-            Assert.Equal(@"C:\Temp\source_archive\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root3}source\def.cs", TrapWriter.InnerPathComputation.RELATIVE).Replace('/', '\\'));
+            Assert.Equal(@"C:\Temp\source_archive\diskstation\share\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root3}{root3}diskstation\share\source\def.cs").Replace('/', '\\'));
-
-            Assert.Equal(@"C:\Temp\source_archive\diskstation\share\source\def.cs", TrapWriter.NestPaths(logger, @"C:\Temp\source_archive", $@"{root3}{root3}diskstation\share\source\def.cs", TrapWriter.InnerPathComputation.ABSOLUTE).Replace('/', '\\'));
         }
 
         class LoggerMock : ILogger

csharp/extractor/Semmle.Extraction/Entities/File.cs

@@ -10,93 +10,55 @@ namespace Semmle.Extraction.Entities
         File(Context cx, string path)
             : base(cx, path)
         {
-            Path = path;
+            OriginalPath = path;
+            TransformedPathLazy = new Lazy<PathTransformer.ITransformedPath>(() => Context.Extractor.PathTransformer.Transform(OriginalPath));
         }
 
-        public string Path
+        readonly string OriginalPath;
-        {
+        readonly Lazy<PathTransformer.ITransformedPath> TransformedPathLazy;
-            get;
+        PathTransformer.ITransformedPath TransformedPath => TransformedPathLazy.Value;
-            private set;
-        }
 
-        public string DatabasePath => PathAsDatabaseId(Path);
+        public override bool NeedsPopulation => Context.DefinesFile(OriginalPath) || OriginalPath == Context.Extractor.OutputPath;
-
-        public override bool NeedsPopulation => Context.DefinesFile(Path) || Path == Context.Extractor.OutputPath;
 
         public override void Populate(TextWriter trapFile)
         {
-            if (Path == null)
+            trapFile.files(this, TransformedPath.Value, TransformedPath.NameWithoutExtension, TransformedPath.Extension);
+
+            if (TransformedPath.ParentDirectory is PathTransformer.ITransformedPath dir)
+                trapFile.containerparent(Folder.Create(Context, dir), this);
+
+            var fromSource = TransformedPath.Extension.ToLowerInvariant().Equals("cs");
+            if (fromSource)
             {
-                trapFile.files(this, "", "", "");
+                foreach (var text in Context.Compilation.SyntaxTrees.
-            }
+                    Where(t => t.FilePath == OriginalPath).
-            else
+                    Select(tree => tree.GetText()))
-            {
-                var fi = new FileInfo(Path);
-
-                string extension = fi.Extension ?? "";
-                string name = fi.Name;
-                name = name.Substring(0, name.Length - extension.Length);
-                int fromSource = extension.ToLowerInvariant().Equals(".cs") ? 1 : 2;
-
-                // remove the dot from the extension
-                if (extension.Length > 0)
-                    extension = extension.Substring(1);
-                trapFile.files(this, PathAsDatabaseString(Path), name, extension);
-
-                trapFile.containerparent(Folder.Create(Context, fi.Directory), this);
-                if (fromSource == 1)
                 {
-                    foreach (var text in Context.Compilation.SyntaxTrees.
+                    var rawText = text.ToString() ?? "";
-                        Where(t => t.FilePath == Path).
+                    var lineCounts = LineCounter.ComputeLineCounts(rawText);
-                        Select(tree => tree.GetText()))
+                    if (rawText.Length > 0 && rawText[rawText.Length - 1] != '\n') lineCounts.Total++;
-                    {
-                        var rawText = text.ToString() ?? "";
-                        var lineCounts = LineCounter.ComputeLineCounts(rawText);
-                        if (rawText.Length > 0 && rawText[rawText.Length - 1] != '\n') lineCounts.Total++;
 
-                        trapFile.numlines(this, lineCounts);
+                    trapFile.numlines(this, lineCounts);
-                        Context.TrapWriter.Archive(fi.FullName, text.Encoding ?? System.Text.Encoding.Default);
+                    Context.TrapWriter.Archive(OriginalPath, TransformedPath, text.Encoding ?? System.Text.Encoding.Default);
-                    }
                 }
-
-                trapFile.file_extraction_mode(this, Context.Extractor.Standalone ? 1 : 0);
             }
+
+            trapFile.file_extraction_mode(this, Context.Extractor.Standalone ? 1 : 0);
         }
 
         public override void WriteId(System.IO.TextWriter trapFile)
         {
-            if (Path is null)
+            trapFile.Write(TransformedPath.DatabaseId);
-                trapFile.Write("GENERATED;sourcefile");
+            trapFile.Write(";sourcefile");
-            else
-            {
-                trapFile.Write(DatabasePath);
-                trapFile.Write(";sourcefile");
-            }
         }
 
-        /// <summary>
-        /// Converts a path string into a string to use as an ID
-        /// in the QL database.
-        /// </summary>
-        /// <param name="path">An absolute path.</param>
-        /// <returns>The database ID.</returns>
-        public static string PathAsDatabaseId(string path)
-        {
-            if (path.Length >= 2 && path[1] == ':' && Char.IsLower(path[0]))
-                path = Char.ToUpper(path[0]) + "_" + path.Substring(2);
-            return path.Replace('\\', '/').Replace(":", "_");
-        }
-
-        public static string PathAsDatabaseString(string path) => path.Replace('\\', '/');
-
         public static File Create(Context cx, string path) => FileFactory.Instance.CreateEntity(cx, (typeof(File), path), path);
 
         public static File CreateGenerated(Context cx) => GeneratedFile.Create(cx);
 
         class GeneratedFile : File
         {
-            GeneratedFile(Context cx)
+            GeneratedFile(Context cx) : base(cx, "") { }
-                : base(cx, "") { }
 
             public override bool NeedsPopulation => true;
 

csharp/extractor/Semmle.Extraction/Entities/Folder.cs

@@ -2,65 +2,44 @@ using System.IO;
 
 namespace Semmle.Extraction.Entities
 {
-    sealed class Folder : CachedEntity<DirectoryInfo>
+    sealed class Folder : CachedEntity<PathTransformer.ITransformedPath>
     {
-        Folder(Context cx, DirectoryInfo init)
+        Folder(Context cx, PathTransformer.ITransformedPath init) : base(cx, init) { }
-            : base(cx, init)
-        {
-            Path = init.FullName;
-        }
-
-        public string Path
-        {
-            get;
-            private set;
-        }
-
-        public string DatabasePath => File.PathAsDatabaseId(Path);
 
         public override void Populate(TextWriter trapFile)
         {
-            // Ensure that the name of the root directory is consistent
+            trapFile.folders(this, symbol.Value, symbol.NameWithoutExtension);
-            // with the XmlTrapWriter.
+            if (symbol.ParentDirectory is PathTransformer.ITransformedPath parent)
-            // Linux/Windows: java.io.File.getName() returns ""
+                trapFile.containerparent(Create(Context, parent), this);
-            // On Linux: System.IO.DirectoryInfo.Name returns "/"
-            // On Windows: System.IO.DirectoryInfo.Name returns "L:\"
-            string shortName = symbol.Parent == null ? "" : symbol.Name;
-
-            trapFile.folders(this, File.PathAsDatabaseString(Path), shortName);
-            if (symbol.Parent != null)
-            {
-                trapFile.containerparent(Create(Context, symbol.Parent), this);
-            }
         }
 
         public override bool NeedsPopulation => true;
 
         public override void WriteId(System.IO.TextWriter trapFile)
         {
-            trapFile.Write(DatabasePath);
+            trapFile.Write(symbol.DatabaseId);
             trapFile.Write(";folder");
         }
 
-        public static Folder Create(Context cx, DirectoryInfo folder) =>
+        public static Folder Create(Context cx, PathTransformer.ITransformedPath folder) =>
             FolderFactory.Instance.CreateEntity(cx, folder, folder);
 
         public override Microsoft.CodeAnalysis.Location? ReportingLocation => null;
 
-        class FolderFactory : ICachedEntityFactory<DirectoryInfo, Folder>
+        class FolderFactory : ICachedEntityFactory<PathTransformer.ITransformedPath, Folder>
         {
             public static readonly FolderFactory Instance = new FolderFactory();
 
-            public Folder Create(Context cx, DirectoryInfo init) => new Folder(cx, init);
+            public Folder Create(Context cx, PathTransformer.ITransformedPath init) => new Folder(cx, init);
         }
 
         public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.NoLabel;
 
-        public override int GetHashCode() => Path.GetHashCode();
+        public override int GetHashCode() => symbol.GetHashCode();
 
         public override bool Equals(object? obj)
         {
-            return obj is Folder folder && folder.Path == Path;
+            return obj is Folder folder && Equals(folder.symbol, symbol);
         }
     }
 }

csharp/extractor/Semmle.Extraction/Extractor.cs

@@ -81,6 +81,11 @@ namespace Semmle.Extraction
         /// </summary>
         ILogger Logger { get; }
 
+        /// <summary>
+        /// The path transformer to apply.
+        /// </summary>
+        PathTransformer PathTransformer { get; }
+
         /// <summary>
         /// Creates a new context.
         /// </summary>
@@ -112,11 +117,14 @@ namespace Semmle.Extraction
         /// </summary>
         /// <param name="standalone">If the extraction is standalone.</param>
         /// <param name="outputPath">The name of the output DLL/EXE, or null if not specified (standalone extraction).</param>
-        public Extractor(bool standalone, string outputPath, ILogger logger)
+        /// <param name="logger">The object used for logging.</param>
+        /// <param name="pathTransformer">The object used for path transformations.</param>
+        public Extractor(bool standalone, string outputPath, ILogger logger, PathTransformer pathTransformer)
         {
             Standalone = standalone;
             OutputPath = outputPath;
             Logger = logger;
+            PathTransformer = pathTransformer;
         }
 
         // Limit the number of error messages in the log file
@@ -206,5 +214,7 @@ namespace Semmle.Extraction
         public ILogger Logger { get; private set; }
 
         public static string Version => $"{ThisAssembly.Git.BaseTag} ({ThisAssembly.Git.Sha})";
+
+        public PathTransformer PathTransformer { get; }
     }
 }

csharp/extractor/Semmle.Extraction/FilePattern.cs

@@ -0,0 +1,131 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Text.RegularExpressions;
+using System.Diagnostics.CodeAnalysis;
+using Semmle.Util;
+
+namespace Semmle.Extraction
+{
+    public sealed class InvalidFilePatternException : Exception
+    {
+        public InvalidFilePatternException(string pattern, string message) :
+            base($"Invalid file pattern '{pattern}': {message}")
+        { }
+    }
+
+    /// <summary>
+    /// A file pattern, as used in either an extractor layout file or
+    /// a path transformer file.
+    /// </summary>
+    public sealed class FilePattern
+    {
+        /// <summary>
+        /// Whether this is an inclusion pattern.
+        /// </summary>
+        public bool Include { get; }
+
+        public FilePattern(string pattern)
+        {
+            Include = true;
+            if (pattern.StartsWith("-"))
+            {
+                pattern = pattern.Substring(1);
+                Include = false;
+            }
+            pattern = FileUtils.ConvertToUnix(pattern.Trim()).TrimStart('/');
+            RegexPattern = BuildRegex(pattern).ToString();
+        }
+
+        /// <summary>
+        /// Constructs a regex string from a file pattern. Throws
+        /// `InvalidFilePatternException` for invalid patterns.
+        /// </summary>
+        static StringBuilder BuildRegex(string pattern)
+        {
+            bool HasCharAt(int i, Predicate<char> p) =>
+                i >= 0 && i < pattern.Length && p(pattern[i]);
+            var sb = new StringBuilder();
+            var i = 0;
+            var seenDoubleSlash = false;
+            sb.Append('^');
+            while (i < pattern.Length)
+            {
+                if (pattern[i] == '/')
+                {
+                    if (HasCharAt(i + 1, c => c == '/'))
+                    {
+                        if (seenDoubleSlash)
+                            throw new InvalidFilePatternException(pattern, "'//' is allowed at most once.");
+                        sb.Append("(?<doubleslash>/)");
+                        i += 2;
+                        seenDoubleSlash = true;
+                    }
+                    else
+                    {
+                        sb.Append('/');
+                        i++;
+                    }
+                }
+                else if (pattern[i] == '*')
+                {
+                    if (HasCharAt(i + 1, c => c == '*'))
+                    {
+                        if (HasCharAt(i - 1, c => c != '/'))
+                            throw new InvalidFilePatternException(pattern, "'**' preceeded by non-`/` character.");
+                        if (HasCharAt(i + 2, c => c != '/'))
+                            throw new InvalidFilePatternException(pattern, "'**' succeeded by non-`/` character");
+                        sb.Append(".*");
+                        i += 2;
+                    }
+                    else
+                    {
+                        sb.Append("[^/]*");
+                        i++;
+                    }
+                }
+                else
+                    sb.Append(Regex.Escape(pattern[i++].ToString()));
+            }
+            return sb.Append(".*");
+        }
+
+
+        /// <summary>
+        /// The regex pattern compiled from this file pattern.
+        /// </summary>
+        public string RegexPattern { get; }
+
+        /// <summary>
+        /// Returns `true` if the set of file patterns `patterns` match the path `path`.
+        /// If so, `transformerSuffix` will contain the part of `path` that needs to be
+        /// suffixed when using path transformers.
+        /// </summary>
+        public static bool Matches(IEnumerable<FilePattern> patterns, string path, [NotNullWhen(true)] out string? transformerSuffix)
+        {
+            path = FileUtils.ConvertToUnix(path).TrimStart('/');
+
+            foreach (var pattern in patterns.Reverse())
+            {
+                var m = new Regex(pattern.RegexPattern).Match(path);
+                if (m.Success)
+                {
+                    if (pattern.Include)
+                    {
+                        transformerSuffix = m.Groups.TryGetValue("doubleslash", out var group)
+                            ? path.Substring(group.Index)
+                            : path;
+                        return true;
+                    }
+
+                    transformerSuffix = null;
+                    return false;
+                }
+            }
+
+            transformerSuffix = null;
+            return false;
+        }
+    }
+}

csharp/extractor/Semmle.Extraction/Layout.cs

@@ -54,14 +54,15 @@ namespace Semmle.Extraction
             /// </summary>
             /// <param name="srcFile">The source file.</param>
             /// <returns>The full filepath of the trap file.</returns>
-            public string GetTrapPath(ILogger logger, string srcFile, TrapWriter.CompressionMode trapCompression) => TrapWriter.TrapPath(logger, TRAP_FOLDER, srcFile, trapCompression);
+            public string GetTrapPath(ILogger logger, PathTransformer.ITransformedPath srcFile, TrapWriter.CompressionMode trapCompression) =>
+                TrapWriter.TrapPath(logger, TRAP_FOLDER, srcFile, trapCompression);
 
             /// <summary>
             /// Creates a trap writer for a given source/assembly file.
             /// </summary>
             /// <param name="srcFile">The source file.</param>
             /// <returns>A newly created TrapWriter.</returns>
-            public TrapWriter CreateTrapWriter(ILogger logger, string srcFile, bool discardDuplicates, TrapWriter.CompressionMode trapCompression) =>
+            public TrapWriter CreateTrapWriter(ILogger logger, PathTransformer.ITransformedPath srcFile, bool discardDuplicates, TrapWriter.CompressionMode trapCompression) =>
                 new TrapWriter(logger, srcFile, TRAP_FOLDER, SOURCE_ARCHIVE, discardDuplicates, trapCompression);
         }
 
@@ -73,7 +74,7 @@ namespace Semmle.Extraction
         /// </summary>
         /// <param name="sourceFile">The file to look up.</param>
         /// <returns>The relevant subproject, or null if not found.</returns>
-        public SubProject? LookupProjectOrNull(string sourceFile)
+        public SubProject? LookupProjectOrNull(PathTransformer.ITransformedPath sourceFile)
         {
             if (!useLayoutFile) return DefaultProject;
 
@@ -89,7 +90,7 @@ namespace Semmle.Extraction
         /// </summary>
         /// <param name="sourceFile">The file to look up.</param>
         /// <returns>The relevant subproject, or DefaultProject if not found.</returns>
-        public SubProject LookupProjectOrDefault(string sourceFile)
+        public SubProject LookupProjectOrDefault(PathTransformer.ITransformedPath sourceFile)
         {
             return LookupProjectOrNull(sourceFile) ?? DefaultProject;
         }
@@ -134,7 +135,7 @@ namespace Semmle.Extraction
         /// </summary>
         /// <param name="path">The absolute path of the file to query.</param>
         /// <returns>True iff there is no layout file or the layout file specifies the file.</returns>
-        public bool FileInLayout(string path) => LookupProjectOrNull(path) != null;
+        public bool FileInLayout(PathTransformer.ITransformedPath path) => LookupProjectOrNull(path) != null;
 
         void ReadLayoutFile(string layout)
         {
@@ -167,33 +168,7 @@ namespace Semmle.Extraction
 
     sealed class LayoutBlock
     {
-        struct Condition
+        private readonly List<FilePattern> filePatterns = new List<FilePattern>();
-        {
-            private readonly bool include;
-            private readonly string prefix;
-
-            public bool Include => include;
-
-            public string Prefix => prefix;
-
-            public Condition(string line)
-            {
-                include = false;
-                if (line.StartsWith("-"))
-                    line = line.Substring(1);
-                else
-                    include = true;
-                prefix = Normalise(line.Trim());
-            }
-
-            static public string Normalise(string path)
-            {
-                path = Path.GetFullPath(path);
-                return path.Replace('\\', '/');
-            }
-        }
-
-        private readonly List<Condition> conditions = new List<Condition>();
 
         public readonly Layout.SubProject Directories;
 
@@ -219,22 +194,10 @@ namespace Semmle.Extraction
             ReadVariable("ODASA_BUILD_ERROR_DIR", lines[i++]);
             while (i < lines.Length && !lines[i].StartsWith("#"))
             {
-                conditions.Add(new Condition(lines[i++]));
+                filePatterns.Add(new FilePattern(lines[i++]));
             }
         }
 
-        public bool Matches(string path)
+        public bool Matches(PathTransformer.ITransformedPath path) => FilePattern.Matches(filePatterns, path.Value, out var _);
-        {
-            bool matches = false;
-            path = Condition.Normalise(path);
-            foreach (Condition condition in conditions)
-            {
-                if (condition.Include)
-                    matches |= path.StartsWith(condition.Prefix);
-                else
-                    matches &= !path.StartsWith(condition.Prefix);
-            }
-            return matches;
-        }
     }
 }

csharp/extractor/Semmle.Extraction/PathTransformer.cs

@@ -0,0 +1,177 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Diagnostics.CodeAnalysis;
+using Semmle.Util;
+
+namespace Semmle.Extraction
+{
+    /// <summary>
+    /// A class for interpreting path transformers specified using the environment
+    /// variable `CODEQL_PATH_TRANSFORMER`.
+    /// </summary>
+    public sealed class PathTransformer
+    {
+        public class InvalidPathTransformerException : Exception
+        {
+            public InvalidPathTransformerException(string message) :
+                base($"Invalid path transformer specification: {message}")
+            { }
+        }
+
+        /// <summary>
+        /// A transformed path.
+        /// </summary>
+        public interface ITransformedPath
+        {
+            string Value { get; }
+
+            string Extension { get; }
+
+            string NameWithoutExtension { get; }
+
+            ITransformedPath? ParentDirectory { get; }
+
+            ITransformedPath WithSuffix(string suffix);
+
+            string DatabaseId { get; }
+        }
+
+        struct TransformedPath : ITransformedPath
+        {
+            public TransformedPath(string value) { this.value = value; }
+            readonly string value;
+
+            public string Value => value;
+
+            public string Extension => Path.GetExtension(value)?.Substring(1) ?? "";
+
+            public string NameWithoutExtension => Path.GetFileNameWithoutExtension(value);
+
+            public ITransformedPath? ParentDirectory
+            {
+                get
+                {
+                    var dir = Path.GetDirectoryName(value);
+                    if (dir is null)
+                        return null;
+                    var isWindowsDriveLetter = dir.Length == 2 && char.IsLetter(dir[0]) && dir[1] == ':';
+                    if (isWindowsDriveLetter)
+                        return null;
+                    return new TransformedPath(FileUtils.ConvertToUnix(dir));
+                }
+            }
+
+            public ITransformedPath WithSuffix(string suffix) => new TransformedPath(value + suffix);
+
+            public string DatabaseId
+            {
+                get
+                {
+                    var ret = value;
+                    if (ret.Length >= 2 && ret[1] == ':' && Char.IsLower(ret[0]))
+                        ret = Char.ToUpper(ret[0]) + "_" + ret.Substring(2);
+                    return ret.Replace('\\', '/').Replace(":", "_");
+                }
+            }
+
+            public override int GetHashCode() => 11 * value.GetHashCode();
+
+            public override bool Equals(object? obj) => obj is TransformedPath tp && tp.value == value;
+
+            public override string ToString() => value;
+        }
+
+        readonly Func<string, string> transform;
+
+        /// <summary>
+        /// Returns the path obtained by transforming `path`.
+        /// </summary>
+        public ITransformedPath Transform(string path) => new TransformedPath(transform(path));
+
+        /// <summary>
+        /// Default constructor reads parameters from the environment.
+        /// </summary>
+        public PathTransformer(IPathCache pathCache) :
+            this(pathCache, Environment.GetEnvironmentVariable("CODEQL_PATH_TRANSFORMER") is string file ? File.ReadAllLines(file) : null)
+        {
+        }
+
+        /// <summary>
+        /// Creates a path transformer based on the specification in `lines`.
+        /// Throws `InvalidPathTransformerException` for invalid specifications.
+        /// </summary>
+        public PathTransformer(IPathCache pathCache, string[]? lines)
+        {
+            if (lines is null)
+            {
+                transform = path => FileUtils.ConvertToUnix(pathCache.GetCanonicalPath(path));
+                return;
+            }
+
+            var sections = ParsePathTransformerSpec(lines);
+            transform = path =>
+            {
+                path = FileUtils.ConvertToUnix(pathCache.GetCanonicalPath(path));
+                foreach (var section in sections)
+                {
+                    if (section.Matches(path, out var transformed))
+                        return transformed;
+                }
+                return path;
+            };
+        }
+
+        static IEnumerable<TransformerSection> ParsePathTransformerSpec(string[] lines)
+        {
+            var sections = new List<TransformerSection>();
+            try
+            {
+                int i = 0;
+                while (i < lines.Length && !lines[i].StartsWith("#"))
+                    i++;
+                while (i < lines.Length)
+                {
+                    var section = new TransformerSection(lines, ref i);
+                    sections.Add(section);
+                }
+
+                if (sections.Count == 0)
+                    throw new InvalidPathTransformerException("contains no sections.");
+            }
+            catch (InvalidFilePatternException ex)
+            {
+                throw new InvalidPathTransformerException(ex.Message);
+            }
+            return sections;
+        }
+    }
+
+    sealed class TransformerSection
+    {
+        readonly string name;
+        readonly List<FilePattern> filePatterns = new List<FilePattern>();
+
+        public TransformerSection(string[] lines, ref int i)
+        {
+            name = lines[i++].Substring(1); // skip the '#'
+            for (; i < lines.Length && !lines[i].StartsWith("#"); i++)
+            {
+                var line = lines[i];
+                if (!string.IsNullOrWhiteSpace(line))
+                    filePatterns.Add(new FilePattern(line));
+            }
+        }
+
+        public bool Matches(string path, [NotNullWhen(true)] out string? transformed)
+        {
+            if (FilePattern.Matches(filePatterns, path, out var suffix))
+            {
+                transformed = FileUtils.ConvertToUnix(name) + suffix;
+                return true;
+            }
+            transformed = null;
+            return false;
+        }
+    }
+}

csharp/extractor/Semmle.Extraction/TrapWriter.cs

@@ -14,12 +14,6 @@ namespace Semmle.Extraction
 
     public sealed class TrapWriter : IDisposable
     {
-        public enum InnerPathComputation
-        {
-            ABSOLUTE,
-            RELATIVE
-        }
-
         public enum CompressionMode
         {
             None,
@@ -45,7 +39,7 @@ namespace Semmle.Extraction
 
         readonly CompressionMode TrapCompression;
 
-        public TrapWriter(ILogger logger, string outputfile, string? trap, string? archive, bool discardDuplicates, CompressionMode trapCompression)
+        public TrapWriter(ILogger logger, PathTransformer.ITransformedPath outputfile, string? trap, string? archive, bool discardDuplicates, CompressionMode trapCompression)
         {
             Logger = logger;
             TrapCompression = trapCompression;
@@ -107,16 +101,17 @@ namespace Semmle.Extraction
         /// Adds the specified input file to the source archive. It may end up in either the normal or long path area
         /// of the source archive, depending on the length of its full path.
         /// </summary>
-        /// <param name="inputPath">The path to the input file.</param>
+        /// <param name="originalPath">The path to the input file.</param>
+        /// <param name="transformedPath">The transformed path to the input file.</param>
         /// <param name="inputEncoding">The encoding used by the input file.</param>
-        public void Archive(string inputPath, Encoding inputEncoding)
+        public void Archive(string originalPath, PathTransformer.ITransformedPath transformedPath, Encoding inputEncoding)
         {
             if (string.IsNullOrEmpty(archive)) return;
 
             // Calling GetFullPath makes this use the canonical capitalisation, if the file exists.
-            string fullInputPath = Path.GetFullPath(inputPath);
+            string fullInputPath = Path.GetFullPath(originalPath);
 
-            ArchivePath(fullInputPath, inputEncoding);
+            ArchivePath(fullInputPath, transformedPath, inputEncoding);
         }
 
         /// <summary>
@@ -124,14 +119,11 @@ namespace Semmle.Extraction
         /// </summary>
         /// <param name="inputPath">The path of the file.</param>
         /// <param name="contents">The contents of the file.</param>
-        public void Archive(string inputPath, string contents)
+        public void Archive(PathTransformer.ITransformedPath inputPath, string contents)
         {
             if (string.IsNullOrEmpty(archive)) return;
 
-            // Calling GetFullPath makes this use the canonical capitalisation, if the file exists.
+            ArchiveContents(inputPath, contents);
-            string fullInputPath = Path.GetFullPath(inputPath);
-
-            ArchiveContents(fullInputPath, contents);
         }
 
         /// <summary>
@@ -210,18 +202,19 @@ namespace Semmle.Extraction
         /// source archive less than the system path limit of 260 characters.
         /// </summary>
         /// <param name="fullInputPath">The full path to the input file.</param>
+        /// <param name="transformedPath">The transformed path to the input file.</param>
         /// <param name="inputEncoding">The encoding used by the input file.</param>
         /// <exception cref="PathTooLongException">If the output path in the source archive would
         /// exceed the system path limit of 260 characters.</exception>
-        private void ArchivePath(string fullInputPath, Encoding inputEncoding)
+        private void ArchivePath(string fullInputPath, PathTransformer.ITransformedPath transformedPath, Encoding inputEncoding)
         {
             string contents = File.ReadAllText(fullInputPath, inputEncoding);
-            ArchiveContents(fullInputPath, contents);
+            ArchiveContents(transformedPath, contents);
         }
 
-        private void ArchiveContents(string fullInputPath, string contents)
+        private void ArchiveContents(PathTransformer.ITransformedPath transformedPath, string contents)
         {
-            string dest = NestPaths(Logger, archive, fullInputPath, InnerPathComputation.ABSOLUTE);
+            string dest = NestPaths(Logger, archive, transformedPath.Value);
             string tmpSrcFile = Path.GetTempFileName();
             File.WriteAllText(tmpSrcFile, contents, UTF8);
             try
@@ -236,14 +229,11 @@ namespace Semmle.Extraction
             }
         }
 
-        public static string NestPaths(ILogger logger, string? outerpath, string innerpath, InnerPathComputation innerPathComputation)
+        public static string NestPaths(ILogger logger, string? outerpath, string innerpath)
         {
             string nested = innerpath;
             if (!string.IsNullOrEmpty(outerpath))
             {
-                if (!Path.IsPathRooted(innerpath) && innerPathComputation == InnerPathComputation.ABSOLUTE)
-                    innerpath = Path.GetFullPath(innerpath);
-
                 // Remove all leading path separators / or \
                 // For example, UNC paths have two leading \\
                 innerpath = innerpath.TrimStart(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
@@ -276,13 +266,13 @@ namespace Semmle.Extraction
             }
         }
 
-        public static string TrapPath(ILogger logger, string? folder, string filename, TrapWriter.CompressionMode trapCompression)
+        public static string TrapPath(ILogger logger, string? folder, PathTransformer.ITransformedPath path, TrapWriter.CompressionMode trapCompression)
         {
-            filename = $"{Path.GetFullPath(filename)}.trap{TrapExtension(trapCompression)}";
+            var filename = $"{path.Value}.trap{TrapExtension(trapCompression)}";
             if (string.IsNullOrEmpty(folder))
                 folder = Directory.GetCurrentDirectory();
 
-            return NestPaths(logger, folder, filename, InnerPathComputation.ABSOLUTE); ;
+            return NestPaths(logger, folder, filename);
         }
     }
 }

csharp/extractor/Semmle.Util/CanonicalPathCache.cs

@@ -222,6 +222,29 @@ namespace Semmle.Util
             this.pathStrategy = pathStrategy;
         }
 
+
+        /// <summary>
+        /// Create a CanonicalPathCache.
+        /// </summary>
+        ///
+        /// <remarks>
+        /// Creates the appropriate PathStrategy object which encapsulates
+        /// the correct algorithm. Falls back to different implementations
+        /// depending on platform.
+        /// </remarks>
+        ///
+        /// <param name="maxCapacity">Size of the cache.</param>
+        /// <param name="symlinks">Policy for following symlinks.</param>
+        /// <returns>A new CanonicalPathCache.</returns>
+        public static CanonicalPathCache Create(ILogger logger, int maxCapacity)
+        {
+            var preserveSymlinks =
+                Environment.GetEnvironmentVariable("CODEQL_PRESERVE_SYMLINKS") == "true" ||
+                Environment.GetEnvironmentVariable("SEMMLE_PRESERVE_SYMLINKS") == "true";
+            return Create(logger, maxCapacity, preserveSymlinks ? CanonicalPathCache.Symlinks.Preserve : CanonicalPathCache.Symlinks.Follow);
+
+        }
+
         /// <summary>
         /// Create a CanonicalPathCache.
         /// </summary>

csharp/ql/src/Likely Bugs/SelfAssignment.ql

@@ -20,7 +20,9 @@ class StructuralComparisonConfig extends StructuralComparisonConfiguration {
     exists(AssignExpr ae |
       // Member initializers are never self-assignments, in particular
       // not initializers such as `new C { F = F };`
-      not ae instanceof MemberInitializer
+      not ae instanceof MemberInitializer and
+      // Enum field initializers are never self assignments. `enum E { A = 42 }`
+      not ae.getParent().(Field).getDeclaringType() instanceof Enum
     |
       ae.getLValue() = x and
       ae.getRValue() = y

csharp/ql/src/Linq/Helpers.qll

@@ -132,6 +132,17 @@ class AnyCall extends MethodCall {
   }
 }
 
+/** A LINQ Count(...) call. */
+class CountCall extends MethodCall {
+  CountCall() {
+    exists(Method m |
+      m = getTarget() and
+      isEnumerableType(m.getDeclaringType()) and
+      m.hasName("Count")
+    )
+  }
+}
+
 /** A variable of type IEnumerable<T>, for some T. */
 class IEnumerableSequence extends Variable {
   IEnumerableSequence() { isIEnumerableType(getType()) }

csharp/ql/src/Security Features/CWE-022/TaintedPath.qhelp

@@ -41,7 +41,7 @@ sent back to the user, giving them access to all the system's passwords.</p>
 
 <li>
 OWASP:
-<a href="https://www.owasp.org/index.php/Path_traversal">Path Traversal</a>.
+<a href="https://owasp.org/www-community/attacks/Path_Traversal">Path Traversal</a>.
 </li>
 
 </references>

csharp/ql/src/Security Features/CWE-022/ZipSlip.qhelp

@@ -71,7 +71,7 @@ Snyk:
 </li>
 <li>
 OWASP:
-<a href="https://www.owasp.org/index.php/Path_traversal">Path Traversal</a>.
+<a href="https://owasp.org/www-community/attacks/Path_Traversal">Path Traversal</a>.
 </li>
 
 </references>

csharp/ql/src/Security Features/CWE-643/XPathInjection.qhelp

@@ -42,7 +42,7 @@ variables in an <code>XsltArgumentList</code>.
 </example>
 
 <references>
-<li>OWASP: <a href="https://www.owasp.org/index.php?title=Testing_for_XPath_Injection_(OTG-INPVAL-010)">Testing for XPath Injection</a>.</li>
+<li>OWASP: <a href="https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/07-Input_Validation_Testing/09-Testing_for_XPath_Injection">Testing for XPath Injection</a>.</li>
 <li>OWASP: <a href="https://www.owasp.org/index.php/XPATH_Injection">XPath Injection</a>.</li>
 <li>MSDN: <a href="https://msdn.microsoft.com/en-us/library/dd567715.aspx">User Defined Functions and Variables</a>.</li>
 </references>

csharp/ql/src/Security Features/InsufficientKeySize.cs

@@ -11,7 +11,7 @@ namespace InsufficientKeySize
         {
             try
             {
-                RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(512); // BAD
+                RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(1024); // BAD
                 rsa.ImportParameters(key);
                 return rsa.Encrypt(plaintext, true);
             }
@@ -27,7 +27,7 @@ namespace InsufficientKeySize
             try
             {
                 RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(); // BAD
-                rsa = new RSACryptoServiceProvider(1024); // GOOD
+                rsa = new RSACryptoServiceProvider(2048); // GOOD
                 rsa.ImportParameters(key);
                 return rsa.Encrypt(plaintext, true);
             }
@@ -58,7 +58,7 @@ namespace InsufficientKeySize
             try
             {
                 DSACryptoServiceProvider dsa = new DSACryptoServiceProvider(); // BAD
-                dsa = new DSACryptoServiceProvider(1024); // GOOD
+                dsa = new DSACryptoServiceProvider(2048); // GOOD
                 dsa.ImportParameters(key);
                 return dsa.SignData(plaintext);
             }
@@ -121,7 +121,7 @@ namespace InsufficientKeySize
             try
             {
                 // Create a new instance of DSACryptoServiceProvider.
-                using (DSACryptoServiceProvider DSA = new DSACryptoServiceProvider(1024)) // GOOD
+                using (DSACryptoServiceProvider DSA = new DSACryptoServiceProvider(2048)) // GOOD
                 {
                     // Import the key information.
                     DSA.ImportParameters(DSAKeyInfo);

csharp/ql/src/Security Features/InsufficientKeySize.qhelp

@@ -8,7 +8,7 @@ are vulnerable to brute force attack when too small a key size is used.</p>
 
 </overview>
 <recommendation>
-<p>The key should be at least 1024-bit long when using RSA encryption, and 128-bit long when using 
+<p>The key should be at least 2048-bit long when using RSA encryption, and 128-bit long when using 
 symmetric encryption.</p>
 
 </recommendation>

csharp/ql/src/Security Features/InsufficientKeySize.ql

@@ -29,8 +29,8 @@ predicate incorrectUseOfDSA(ObjectCreation e, string msg) {
       .getTarget()
       .getDeclaringType()
       .hasQualifiedName("System.Security.Cryptography", "DSACryptoServiceProvider") and
-  exists(Expr i | e.getArgument(0) = i and i.getValue().toInt() < 1024) and
+  exists(Expr i | e.getArgument(0) = i and i.getValue().toInt() < 2048) and
-  msg = "Key size should be at least 1024 bits for DSA encryption."
+  msg = "Key size should be at least 2048 bits for DSA encryption."
 }
 
 predicate incorrectUseOfRSA(ObjectCreation e, string msg) {
@@ -38,8 +38,8 @@ predicate incorrectUseOfRSA(ObjectCreation e, string msg) {
       .getTarget()
       .getDeclaringType()
       .hasQualifiedName("System.Security.Cryptography", "RSACryptoServiceProvider") and
-  exists(Expr i | e.getArgument(0) = i and i.getValue().toInt() < 1024) and
+  exists(Expr i | e.getArgument(0) = i and i.getValue().toInt() < 2048) and
-  msg = "Key size should be at least 1024 bits for RSA encryption."
+  msg = "Key size should be at least 2048 bits for RSA encryption."
 }
 
 from Expr e, string msg

csharp/ql/src/experimental/CWE-099/TaintedWebClient.qhelp

@@ -51,7 +51,7 @@ system's passwords.</p>
 
 <li>
 OWASP:
-<a href="https://www.owasp.org/index.php/Path_traversal">Path Traversal</a>.
+<a href="https://owasp.org/www-community/attacks/Path_Traversal">Path Traversal</a>.
 </li>
 
 </references>

csharp/ql/src/experimental/ir/implementation/raw/Operand.qll

@@ -79,7 +79,8 @@ private PhiOperandBase phiOperand(
 }
 
 /**
- * A source operand of an `Instruction`. The operand represents a value consumed by the instruction.
+ * An operand of an `Instruction`. The operand represents a use of the result of one instruction
+ * (the defining instruction) in another instruction (the use instruction)
  */
 class Operand extends TOperand {
   /** Gets a textual representation of this element. */

csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Operand.qll

@@ -79,7 +79,8 @@ private PhiOperandBase phiOperand(
 }
 
 /**
- * A source operand of an `Instruction`. The operand represents a value consumed by the instruction.
+ * An operand of an `Instruction`. The operand represents a use of the result of one instruction
+ * (the defining instruction) in another instruction (the use instruction)
  */
 class Operand extends TOperand {
   /** Gets a textual representation of this element. */

csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll

@@ -59,6 +59,12 @@ class MemoryLocation extends TMemoryLocation {
   final string getUniqueId() { result = var.getUniqueId() }
 }
 
+/**
+ * Represents a set of `MemoryLocation`s that cannot overlap with
+ * `MemoryLocation`s outside of the set. The `VirtualVariable` will be
+ * represented by a `MemoryLocation` that totally overlaps all other
+ * `MemoryLocations` in the set.
+ */
 class VirtualVariable extends MemoryLocation { }
 
 /** A virtual variable that groups all escaped memory within a function. */

csharp/ql/src/experimental/ir/internal/Overlap.qll

@@ -3,18 +3,33 @@ private newtype TOverlap =
   TMustTotallyOverlap() or
   TMustExactlyOverlap()
 
+/**
+ * Represents a possible overlap between two memory ranges.
+ */
 abstract class Overlap extends TOverlap {
   abstract string toString();
 }
 
+/**
+ * Represents a partial overlap between two memory ranges, which may or may not
+ * actually occur in practice.
+ */
 class MayPartiallyOverlap extends Overlap, TMayPartiallyOverlap {
   final override string toString() { result = "MayPartiallyOverlap" }
 }
 
+/**
+ * Represents an overlap in which the first memory range is known to include all
+ * bits of the second memory range, but may be larger or have a different type.
+ */
 class MustTotallyOverlap extends Overlap, TMustTotallyOverlap {
   final override string toString() { result = "MustTotallyOverlap" }
 }
 
+/**
+ * Represents an overlap between two memory ranges that have the same extent and
+ * the same type.
+ */
 class MustExactlyOverlap extends Overlap, TMustExactlyOverlap {
   final override string toString() { result = "MustExactlyOverlap" }
 }

csharp/ql/src/external/examples/filters/BumpMetricBy10.ql

@@ -1,11 +0,0 @@
-/**
- * @name Edit the value of a metric
- * @description Add 10 to a metric's value
- * @deprecated
- */
-
-import csharp
-import external.MetricFilter
-
-from MetricResult res
-select res, res.getValue() + 10

csharp/ql/src/external/examples/filters/EditDefectMessage.ql

@@ -1,11 +0,0 @@
-/**
- * @name Edit the message of a query
- * @description Change the string in the select to edit the message
- * @deprecated
- */
-
-import csharp
-import external.DefectFilter
-
-from DefectResult res
-select res, "Filtered query result: " + res.getMessage()

csharp/ql/src/external/examples/filters/ExcludeGeneratedCode.ql

@@ -1,14 +0,0 @@
-/**
- * @name Filter: removed results from generated code
- * @description Shows how to exclude certain files or folders from results.
- * @deprecated
- */
-
-import csharp
-import external.DefectFilter
-
-predicate generatedFile(File f) { f.getAbsolutePath().matches("%generated%") }
-
-from DefectResult res
-where not generatedFile(res.getFile())
-select res, res.getMessage()

csharp/ql/src/external/examples/filters/FromSource.ql

@@ -1,12 +0,0 @@
-/**
- * @name Filter: only keep results from source
- * @description Shows how to filter for only certain files
- * @deprecated
- */
-
-import csharp
-import external.DefectFilter
-
-from DefectResult res
-where res.getFile().fromSource()
-select res, res.getMessage()

csharp/ql/src/external/tests/DefectFromExternalData.ql

@@ -1,19 +0,0 @@
-/**
- * @name Defect from external data
- * @description Insert description here...
- * @kind problem
- * @problem.severity warning
- * @deprecated
- */
-
-import csharp
-import external.ExternalArtifact
-
-// custom://[FileUtil][2011-01-02][false][1.1][6][Message 2]
-from ExternalData d, File u
-where
-  d.getQueryPath() = "external-data.ql" and
-  u.getStem() = d.getField(0)
-select u,
-  d.getField(5) + ", " + d.getFieldAsDate(1) + ", " + d.getField(2) + ", " + d.getFieldAsFloat(3) +
-    ", " + d.getFieldAsInt(4) + ": " + d.getNumFields()

csharp/ql/src/external/tests/DefectFromExternalDefect.ql

@@ -1,17 +0,0 @@
-/**
- * @name Defect from external defect
- * @description Create a defect from external data
- * @kind problem
- * @problem.severity warning
- * @deprecated
- */
-
-import csharp
-import external.ExternalArtifact
-
-class DuplicateCode extends ExternalDefect {
-  DuplicateCode() { getQueryPath() = "duplicate-code/duplicateCode.ql" }
-}
-
-from DuplicateCode d
-select d, "External Defect " + d.getMessage()

csharp/ql/src/external/tests/DefectFromExternalMetric.ql

@@ -1,17 +0,0 @@
-/**
- * @name Defect from external metric
- * @description Create a defect from external data
- * @kind problem
- * @problem.severity warning
- * @deprecated
- */
-
-import csharp
-import external.ExternalArtifact
-
-from ExternalMetric m, File f
-where
-  m.getQueryPath() = "filesBuilt.ql" and
-  m.getValue() = 1.0 and
-  m.getFile() = f
-select f, "File is built"

csharp/ql/src/external/tests/MetricFilter.ql

@@ -1,13 +0,0 @@
-/**
- * @name Metric filter
- * @description Only include results in large files (200) lines of code.
- * @kind treemap
- * @deprecated
- */
-
-import csharp
-import external.MetricFilter
-
-from MetricResult res
-where res.getFile().getNumberOfLinesOfCode() > 200
-select res, res.getValue()

csharp/ql/src/external/tests/MetricFromExternalDefect.ql

@@ -1,21 +0,0 @@
-/**
- * @name Metric from external defect
- * @description Find number of duplicate code entries in a file
- * @treemap.warnOn lowValues
- * @metricType file
- * @kind treemap
- * @deprecated
- */
-
-import csharp
-import external.ExternalArtifact
-
-class DuplicateCode extends ExternalDefect {
-  DuplicateCode() { getQueryPath() = "duplicate-code/duplicateCode.ql" }
-}
-
-predicate numDuplicateEntries(File f, int i) { i = count(DuplicateCode d | d.getFile() = f) }
-
-from File f, int i
-where numDuplicateEntries(f, i)
-select f, i

csharp/ql/src/external/tests/MetricFromExternalMetric.ql

@@ -1,27 +0,0 @@
-/**
- * @name Metric from external metric
- * @description Each file in a folder gets as metric value the number of files built in that folder
- * @treemap.warnOn lowValues
- * @metricType file
- * @kind treemap
- * @deprecated
- */
-
-import csharp
-import external.ExternalArtifact
-
-predicate numBuiltFiles(Folder fold, int i) {
-  i =
-    count(File f |
-      exists(ExternalMetric m |
-        m.getQueryPath() = "filesBuilt.ql" and
-        m.getValue() = 1.0 and
-        m.getFile() = f
-      ) and
-      f.getParentContainer() = fold
-    )
-}
-
-from File f, int i
-where numBuiltFiles(f.getParentContainer(), i)
-select f, i

csharp/ql/src/semmle/code/csharp/commons/ComparisonTest.qll

@@ -2,7 +2,7 @@
  * Provides classes for capturing various ways of performing comparison tests.
  */
 
-import csharp
+private import csharp
 private import semmle.code.csharp.frameworks.System
 private import semmle.code.csharp.frameworks.system.Collections
 private import semmle.code.csharp.frameworks.system.collections.Generic

csharp/ql/src/semmle/code/csharp/controlflow/Guards.qll

@@ -31,6 +31,25 @@ class Guard extends Expr {
   predicate controlsNode(ControlFlow::Nodes::ElementNode cfn, AccessOrCallExpr sub, AbstractValue v) {
     isGuardedByNode(cfn, this, sub, v)
   }
+
+  /**
+   * Holds if basic block `bb` is guarded by this expression having value `v`.
+   */
+  predicate controlsBasicBlock(BasicBlock bb, AbstractValue v) {
+    Internal::guardControls(this, bb, v)
+  }
+
+  /**
+   * Holds if this guard is an equality test between `e1` and `e2`. If the test is
+   * negated, that is `!=`, then `polarity` is false, otherwise `polarity` is
+   * true.
+   */
+  predicate isEquality(Expr e1, Expr e2, boolean polarity) {
+    exists(BooleanValue v |
+      this = Internal::getAnEqualityCheck(e1, v, e2) and
+      polarity = v.getValue()
+    )
+  }
 }
 
 /** An abstract value. */
@@ -943,13 +962,14 @@ module Internal {
     e = any(BinaryArithmeticOperation bao | result = bao.getAnOperand())
   }
 
-  /** Holds if basic block `bb` only is reached when guard `g` has abstract value `v`. */
+  pragma[noinline]
-  private predicate guardControls(Guard g, BasicBlock bb, AbstractValue v) {
+  private predicate assertionControlsNodeInSameBasicBlock0(
-    exists(ControlFlowElement cfe, ConditionalSuccessor s, AbstractValue v0, Guard g0 |
+    Guard g, AbstractValue v, BasicBlock bb, int i
-      cfe.controlsBlock(bb, s)
+  ) {
-    |
+    exists(Assertion a, Guard g0, AbstractValue v0 |
-      v0.branch(cfe, s, g0) and
+      asserts(a, g0, v0) and
-      impliesSteps(g0, v0, g, v)
+      impliesSteps(g0, v0, g, v) and
+      bb.getNode(i) = a.getAControlFlowNode()
     )
   }
 
@@ -957,17 +977,13 @@ module Internal {
    * Holds if control flow node `cfn` only is reached when guard `g` evaluates to `v`,
    * because of an assertion.
    */
-  private predicate guardAssertionControlsNode(Guard g, ControlFlow::Node cfn, AbstractValue v) {
+  private predicate assertionControlsNodeInSameBasicBlock(
-    exists(Assertion a, Guard g0, AbstractValue v0 |
+    Guard g, ControlFlow::Node cfn, AbstractValue v
-      asserts(a, g0, v0) and
+  ) {
-      impliesSteps(g0, v0, g, v)
+    exists(BasicBlock bb, int i, int j |
-    |
+      assertionControlsNodeInSameBasicBlock0(g, v, bb, i) and
-      a.strictlyDominates(cfn.getBasicBlock())
+      bb.getNode(j) = cfn and
-      or
+      j > i
-      exists(BasicBlock bb, int i, int j | bb.getNode(i) = a.getAControlFlowNode() |
-        bb.getNode(j) = cfn and
-        j > i
-      )
     )
   }
 
@@ -977,7 +993,7 @@ module Internal {
    */
   private predicate guardAssertionControlsElement(Guard g, ControlFlowElement cfe, AbstractValue v) {
     forex(ControlFlow::Node cfn | cfn = cfe.getAControlFlowNode() |
-      guardAssertionControlsNode(g, cfn, v)
+      assertionControlsNodeInSameBasicBlock(g, cfn, v)
     )
   }
 
@@ -1291,24 +1307,6 @@ module Internal {
       )
     }
 
-    /**
-     * Gets an expression that tests whether expression `e1` is equal to
-     * expression `e2`.
-     *
-     * If the returned expression has abstract value `v`, then expression `e1` is
-     * guaranteed to be equal to `e2`, and if the returned expression has abstract
-     * value `v.getDualValue()`, then this expression is guaranteed to be
-     * non-equal to `e`.
-     *
-     * For example, if the expression `x != ""` evaluates to `false` then the
-     * expression `x` is guaranteed to be equal to `""`.
-     */
-    Expr getAnEqualityCheck(Expr e1, AbstractValue v, Expr e2) {
-      result = getABooleanEqualityCheck(e1, v, e2)
-      or
-      result = getAMatchingEqualityCheck(e1, v, e2)
-    }
-
     private Expr getAnEqualityCheckVal(Expr e, AbstractValue v, AbstractValue vExpr) {
       result = getAnEqualityCheck(e, v, vExpr.getAnExpr())
     }
@@ -1464,6 +1462,29 @@ module Internal {
         not e = any(LocalVariableDeclStmt s).getAVariableDeclExpr()
       }
 
+      /**
+       * Gets an expression that tests whether expression `e1` is equal to
+       * expression `e2`.
+       *
+       * If the returned expression has abstract value `v`, then expression `e1` is
+       * guaranteed to be equal to `e2`, and if the returned expression has abstract
+       * value `v.getDualValue()`, then this expression is guaranteed to be
+       * non-equal to `e`.
+       *
+       * For example, if the expression `x != ""` evaluates to `false` then the
+       * expression `x` is guaranteed to be equal to `""`.
+       */
+      cached
+      Expr getAnEqualityCheck(Expr e1, AbstractValue v, Expr e2) {
+        result = getABooleanEqualityCheck(e1, v, e2)
+        or
+        result = getABooleanEqualityCheck(e2, v, e1)
+        or
+        result = getAMatchingEqualityCheck(e1, v, e2)
+        or
+        result = getAMatchingEqualityCheck(e2, v, e1)
+      }
+
       cached
       predicate isCustomNullCheck(Call call, Expr arg, BooleanValue v, boolean isNull) {
         exists(Callable callable, Parameter p |
@@ -1749,7 +1770,7 @@ module Internal {
       exists(Guard g | e = getAChildExprStar(g) |
         guardControls(g, bb, _)
         or
-        guardAssertionControlsNode(g, bb.getANode(), _)
+        assertionControlsNodeInSameBasicBlock(g, bb.getANode(), _)
       )
     }
   }
@@ -1758,6 +1779,21 @@ module Internal {
   private module Cached {
     private import semmle.code.csharp.Caching
 
+    /** Holds if basic block `bb` only is reached when guard `g` has abstract value `v`. */
+    cached
+    predicate guardControls(Guard g, BasicBlock bb, AbstractValue v) {
+      exists(AbstractValue v0, Guard g0 | impliesSteps(g0, v0, g, v) |
+        exists(ControlFlowElement cfe, ConditionalSuccessor s |
+          v0.branch(cfe, s, g0) and cfe.controlsBlock(bb, s)
+        )
+        or
+        exists(Assertion a |
+          asserts(a, g0, v0) and
+          a.strictlyDominates(bb)
+        )
+      )
+    }
+
     pragma[noinline]
     private predicate isGuardedByNode0(
       ControlFlow::Node cfn, AccessOrCallExpr guarded, Guard g, AccessOrCallExpr sub,
@@ -1813,7 +1849,7 @@ module Internal {
     ) {
       isGuardedByNode0(guarded, _, g, sub, v)
       or
-      guardAssertionControlsNode(g, guarded, v) and
+      assertionControlsNodeInSameBasicBlock(g, guarded, v) and
       exists(ConditionOnExprComparisonConfig c | c.same(sub, guarded.getElement()))
     }
 

csharp/ql/src/semmle/code/csharp/dataflow/Nullness.qll

@@ -21,10 +21,8 @@ import csharp
 private import ControlFlow
 private import internal.CallableReturns
 private import semmle.code.csharp.commons.Assertions
-private import semmle.code.csharp.commons.ComparisonTest
 private import semmle.code.csharp.controlflow.Guards as G
 private import semmle.code.csharp.controlflow.Guards::AbstractValues
-private import semmle.code.csharp.dataflow.SSA
 private import semmle.code.csharp.frameworks.System
 private import semmle.code.csharp.frameworks.Test
 

csharp/ql/src/semmle/code/csharp/dataflow/SSA.qll

@@ -706,9 +706,9 @@ module Ssa {
 
     /**
      * Holds if `def` is accessed in basic block `bb1` (either a read or a write),
-     * `bb2` is a transitive successor of `bb1`, and `def` is *maybe* read in `bb2`
+     * `bb2` is a transitive successor of `bb1`, `def` is live at the end of `bb1`,
-     * or one of its transitive successors, but not in any block on the path between
+     * and the underlying variable for `def` is neither read nor written in any block
-     * `bb1` and `bb2`.
+     * on the path between `bb1` and `bb2`.
      */
     private predicate varBlockReaches(TrackedDefinition def, BasicBlock bb1, BasicBlock bb2) {
       varOccursInBlock(def, bb1, _) and
@@ -2537,6 +2537,13 @@ module Ssa {
       )
     }
 
+    /** Holds if `inp` is an input to the phi node along the edge originating in `bb`. */
+    predicate hasInputFromBlock(Definition inp, BasicBlock bb) {
+      this.getAnInput() = inp and
+      this.getBasicBlock().getAPredecessor() = bb and
+      inp.isLiveAtEndOfBlock(bb)
+    }
+
     override string toString() {
       result = getToStringPrefix(this) + "SSA phi(" + getSourceVariable() + ")"
     }

csharp/ql/src/semmle/code/csharp/dataflow/SignAnalysis.qll

@@ -0,0 +1,9 @@
+/**
+ * Provides sign analysis to determine whether expression are always positive
+ * or negative.
+ *
+ * The analysis is implemented as an abstract interpretation over the
+ * three-valued domain `{negative, zero, positive}`.
+ */
+
+import semmle.code.csharp.dataflow.internal.rangeanalysis.SignAnalysisCommon