Merge remote-tracking branch 'upstream/master' into DefaultTaintTracking-Configuration

.gitignore

@@ -19,5 +19,6 @@
 
 # It's useful (though not required) to be able to unpack codeql in the ql checkout itself
 /codeql/
-.vscode/settings.json
+
 csharp/extractor/Semmle.Extraction.CSharp.Driver/Properties/launchSettings.json
+.vscode

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing to CodeQL
 
-We welcome contributions to our CodeQL libraries and queries. Got an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request!
+We welcome contributions to our CodeQL libraries and queries. Got an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request! Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE).
 
 There is lots of useful documentation to help you write queries, ranging from information about query file structure to tutorials for specific target languages. For more information on the documentation available, see [Writing CodeQL queries](https://help.semmle.com/QL/learn-ql/writing-queries/writing-queries.html) on [help.semmle.com](https://help.semmle.com).
 
@@ -47,10 +47,6 @@ If you have an idea for a query that you would like to share with other CodeQL u
 
     - The query must have at least one true positive result on some revision of a real project.
 
-6. **Contributor License Agreement**
-
-    - The contributor can satisfy the [CLA](#contributor-license-agreement).
-
 Experimental queries and libraries may not be actively maintained as the [supported](docs/supported-queries.md) libraries evolve. They may also be changed in backwards-incompatible ways or may be removed entirely in the future without deprecation warnings.
 
 After the experimental query is merged, we welcome pull requests to improve it. Before a query can be moved out of the `experimental` subdirectory, it must satisfy [the requirements for being a supported query](docs/supported-queries.md).
@@ -65,33 +61,6 @@ normal course of software development. We also store records of your
 CLA agreements. Under GDPR legislation, we do this
 on the basis of our legitimate interest in creating the CodeQL product.
 
-Please do get in touch (privacy@semmle.com) if you have any questions about
+Please do get in touch (privacy@github.com) if you have any questions about
 this or our data protection policies.
 
-## Contributor License Agreement
-
-This Contributor License Agreement (“Agreement”) is entered into between Semmle Limited (“Semmle,” “we” or “us” etc.), and You (as defined and further identified below).
-
-Accordingly, You hereby agree to the following terms for Your present and future Contributions submitted to Semmle:
-
-1. **Definitions**.
-
-   * "You" (or "Your") shall mean the Contribution copyright owner (whether an individual or organization) or legal entity authorized by the copyright owner that is making this Agreement with Semmle. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
-
-   * "Contribution(s)" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, submitted by You to Semmle for inclusion in, or documentation of, any of the products or projects owned or managed by Semmle (the "Work(s)").  For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to Semmle or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Semmle for the purpose of discussing and/or improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
-
-2. **Grant of Copyright License**. You hereby grant to Semmle and to recipients of software distributed by Semmle a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.
-
-3. **Grant of Patent License**. You hereby grant to Semmle and to recipients of software distributed by Semmle a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that Your Contribution, or the Work to which You have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.  
-
-4. **Ownership**.  Except as set out above, You keep all right, title, and interest in Your Contribution. The rights that You grant to us under this Agreement are effective on the date You first submitted a Contribution to us, even if Your submission took place before the date You entered this Agreement.
-
-5. **Representations**.  You represent and warrant that: (i) the Contributions are an original work and that You can legally grant the rights set out in this Agreement; (ii) the Contributions and Semmle’s exercise of any license rights granted hereunder, does not and will not, infringe the rights of any third party; (iii) You are not aware of any pending or threatened claims, suits, actions, or charges pertaining to the Contributions, including without limitation any claims or allegations that any or all of the Contributions infringes, violates, or misappropriate the intellectual property rights of any third party (You further agree that You will notify Semmle immediately if You become aware of any such actual or potential claims, suits, actions, allegations or charges).
-
-6. **Employer**.  If Your employer(s) has rights to intellectual property that You create that includes Your Contributions, You represent and warrant that Your employer has waived such rights for Your Contributions to Semmle, or that You have received permission to make Contributions on behalf of that employer and that You are authorized to execute this Agreement on behalf of Your employer.
-
-7. **Inclusion of Code**. We determine the code that is in our Works. You understand that the decision to include the Contribution in any project or source repository is entirely that of Semmle, and this agreement does not guarantee that the Contributions will be included in any product.
-
-8. **Disclaimer**.  You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Except as set forth herein, and unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND.
-
-9. **General**.  The failure of either party to enforce its rights under this Agreement for any period shall not be construed as a waiver of such rights.  No changes or modifications or waivers to this Agreement will be effective unless in writing and signed by both parties.  In the event that any provision of this Agreement shall be determined to be illegal or unenforceable, that provision will be limited or eliminated to the minimum extent necessary so that this Agreement shall otherwise remain in full force and effect and enforceable.  This Agreement shall be governed by and construed in accordance with the laws of the State of California in the United States without regard to the conflicts of laws provisions thereof.  In any action or proceeding to enforce rights under this Agreement, the prevailing party will be entitled to recover costs and attorneys’ fees.  

COPYRIGHT

@@ -1,13 +0,0 @@
-Copyright (c) Semmle Inc and other contributors. All rights reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use
-this file except in compliance with the License. You may obtain a copy of the
-License at http://www.apache.org/licenses/LICENSE-2.0
-
-THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
-WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
-MERCHANTABLITY OR NON-INFRINGEMENT.
-
-See the Apache Version 2.0 License for specific language governing permissions
-and limitations under the License.

LICENSE

@@ -1,176 +1,21 @@
-                                 Apache License
+MIT License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
 
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+Copyright (c) 2006-2020 GitHub, Inc.
 
-   1. Definitions.
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-      "License" shall mean the terms and conditions for use, reproduction,
+The above copyright notice and this permission notice shall be included in all
-      and distribution as defined by Sections 1 through 9 of this document.
+copies or substantial portions of the Software.
 
-      "Licensor" shall mean the copyright owner or entity authorized by
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-      the copyright owner that is granting the License.
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-      "Legal Entity" shall mean the union of the acting entity and all
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-      other entities that control, are controlled by, or are under common
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-      control with that entity. For the purposes of this definition,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-      "control" means (i) the power, direct or indirect, to cause the
+SOFTWARE.
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS

change-notes/1.24/analysis-csharp.md

@@ -22,6 +22,8 @@ The following changes in version 1.24 affect C# analysis in all applications.
 | Dereferenced variable may be null (`cs/dereferenced-value-may-be-null`) | More results | Results are reported from parameters with a default value of `null`. |
 | Useless assignment to local variable (`cs/useless-assignment-to-local`) | Fewer false positive results | Results have been removed when the value assigned is an (implicitly or explicitly) cast default-like value. For example, `var s = (string)null` and `string s = default`. |
 | XPath injection (`cs/xml/xpath-injection`) | More results | The query now recognizes calls to methods on `System.Xml.XPath.XPathNavigator` objects. |
+| Information exposure through transmitted data (`cs/sensitive-data-transmission`) | More results | The query now recognizes writes to cookies and writes to ASP.NET (`Inner`)`Text` properties as additional sinks. |
+| Information exposure through an exception (`cs/information-exposure-through-exception`) | More results | The query now recognizes writes to cookies, writes to ASP.NET (`Inner`)`Text` properties, and email contents as additional sinks. |
 
 ## Removal of old queries
 
@@ -42,5 +44,6 @@ The following changes in version 1.24 affect C# analysis in all applications.
 * [Code contracts](https://docs.microsoft.com/en-us/dotnet/framework/debug-trace-profile/code-contracts) are now recognized, and are treated like any other assertion methods.
 * Expression nullability flow state is given by the predicates `Expr.hasNotNullFlowState()` and `Expr.hasMaybeNullFlowState()`.
 * `stackalloc` array creations are now represented by the QL class `Stackalloc`. Previously they were represented by the class `ArrayCreation`.
+* A new class `RemoteFlowSink` has been added to model sinks where data might be exposed to external users. Examples include web page output, e-mails, and cookies.
 
 ## Changes to autobuilder

change-notes/1.24/analysis-javascript.md

@@ -86,6 +86,7 @@
 | Useless regular-expression character escape (`js/useless-regexp-character-escape`) | Fewer false positive results | This query now distinguishes escapes in strings and regular expression literals. |
 | Identical operands (`js/redundant-operation`) | Fewer results | This query now recognizes cases where the operands change a value using ++/-- expressions. |
 | Superfluous trailing arguments (`js/superfluous-trailing-arguments`) | Fewer results | This query now recognizes cases where a function uses the `Function.arguments` value to process a variable number of parameters. |
+| Incomplete URL scheme check (`js/incomplete-url-scheme-check`) | More results | This query now recognizes more variations of URL scheme checks. |
 
 ## Changes to libraries
 

cpp/ql/src/codeql-suites/cpp-code-scanning.qls

@@ -0,0 +1,4 @@
+- description: Standard Code Scanning queries for C and C++
+- qlpack: codeql-cpp
+- apply: code-scanning-selectors.yml
+  from: codeql-suite-helpers

cpp/ql/test/library-tests/dataflow/taint-tests/localTaint.expected

@@ -152,6 +152,16 @@
 | stl.cpp:105:2:105:4 | ss1 [post update] | stl.cpp:110:7:110:9 | ss1 |  |
 | stl.cpp:106:2:106:4 | ss2 [post update] | stl.cpp:109:7:109:9 | ss2 |  |
 | stl.cpp:106:2:106:4 | ss2 [post update] | stl.cpp:111:7:111:9 | ss2 |  |
+| stl.cpp:124:16:124:28 | call to basic_string | stl.cpp:125:7:125:11 | path1 |  |
+| stl.cpp:124:17:124:26 | call to user_input | stl.cpp:124:16:124:28 | call to basic_string | TAINT |
+| stl.cpp:125:7:125:11 | path1 | stl.cpp:125:13:125:17 | call to c_str | TAINT |
+| stl.cpp:128:10:128:19 | call to user_input | stl.cpp:128:10:128:21 | call to basic_string | TAINT |
+| stl.cpp:128:10:128:21 | call to basic_string | stl.cpp:128:2:128:21 | ... = ... |  |
+| stl.cpp:128:10:128:21 | call to basic_string | stl.cpp:129:7:129:11 | path2 |  |
+| stl.cpp:129:7:129:11 | path2 | stl.cpp:129:13:129:17 | call to c_str | TAINT |
+| stl.cpp:131:15:131:24 | call to user_input | stl.cpp:131:15:131:27 | call to basic_string | TAINT |
+| stl.cpp:131:15:131:27 | call to basic_string | stl.cpp:132:7:132:11 | path3 |  |
+| stl.cpp:132:7:132:11 | path3 | stl.cpp:132:13:132:17 | call to c_str | TAINT |
 | taint.cpp:4:27:4:33 | source1 | taint.cpp:6:13:6:19 | source1 |  |
 | taint.cpp:4:40:4:45 | clean1 | taint.cpp:5:8:5:13 | clean1 |  |
 | taint.cpp:4:40:4:45 | clean1 | taint.cpp:6:3:6:8 | clean1 |  |

cpp/ql/test/library-tests/dataflow/taint-tests/stl.cpp

@@ -110,3 +110,24 @@ void test_stringstream_int(int source)
 	sink(ss1.str());
 	sink(ss2.str()); // tainted [NOT DETECTED]
 }
+
+using namespace std;
+
+char *user_input() {
+  return source();
+}
+
+void sink(const char *filename, const char *mode);
+
+void test_strings2()
+{
+	string path1 = user_input();
+	sink(path1.c_str(), "r"); // tainted
+
+	string path2;
+	path2 = user_input();
+	sink(path2.c_str(), "r"); // tainted
+
+	string path3(user_input());
+	sink(path3.c_str(), "r"); // tainted
+}

cpp/ql/test/library-tests/dataflow/taint-tests/taint.expected

@@ -11,6 +11,9 @@
 | stl.cpp:71:7:71:7 | a | stl.cpp:67:12:67:17 | call to source |
 | stl.cpp:73:7:73:7 | c | stl.cpp:69:16:69:21 | call to source |
 | stl.cpp:75:9:75:13 | call to c_str | stl.cpp:69:16:69:21 | call to source |
+| stl.cpp:125:13:125:17 | call to c_str | stl.cpp:117:10:117:15 | call to source |
+| stl.cpp:129:13:129:17 | call to c_str | stl.cpp:117:10:117:15 | call to source |
+| stl.cpp:132:13:132:17 | call to c_str | stl.cpp:117:10:117:15 | call to source |
 | taint.cpp:8:8:8:13 | clean1 | taint.cpp:4:27:4:33 | source1 |
 | taint.cpp:16:8:16:14 | source1 | taint.cpp:12:22:12:27 | call to source |
 | taint.cpp:17:8:17:16 | ++ ... | taint.cpp:12:22:12:27 | call to source |

cpp/ql/test/library-tests/dataflow/taint-tests/test_diff.expected

@@ -10,6 +10,9 @@
 | format.cpp:106:8:106:14 | format.cpp:105:38:105:52 | AST only |
 | stl.cpp:73:7:73:7 | stl.cpp:69:16:69:21 | AST only |
 | stl.cpp:75:9:75:13 | stl.cpp:69:16:69:21 | AST only |
+| stl.cpp:125:13:125:17 | stl.cpp:117:10:117:15 | AST only |
+| stl.cpp:129:13:129:17 | stl.cpp:117:10:117:15 | AST only |
+| stl.cpp:132:13:132:17 | stl.cpp:117:10:117:15 | AST only |
 | taint.cpp:41:7:41:13 | taint.cpp:35:12:35:17 | AST only |
 | taint.cpp:42:7:42:13 | taint.cpp:35:12:35:17 | AST only |
 | taint.cpp:43:7:43:13 | taint.cpp:37:22:37:27 | AST only |

csharp/autobuilder/Semmle.Autobuild.Tests/BuildScripts.cs

@@ -341,6 +341,8 @@ namespace Semmle.Extraction.Tests
             string cwd = @"C:\Project")
         {
             Actions.GetEnvironmentVariable["CODEQL_AUTOBUILDER_CSHARP_NO_INDEXING"] = "false";
+            Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_TRAP_DIR"] = "";
+            Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR"] = "";
             Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CSHARP_ROOT"] = @"C:\codeql\csharp";
             Actions.GetEnvironmentVariable["CODEQL_JAVA_HOME"] = @"C:\codeql\tools\java";
             Actions.GetEnvironmentVariable["SEMMLE_DIST"] = @"C:\odasa";
@@ -364,8 +366,7 @@ namespace Semmle.Extraction.Tests
             Actions.GetCurrentDirectory = cwd;
             Actions.IsWindows = isWindows;
 
-            var options = new AutobuildOptions();
+            var options = new AutobuildOptions(Actions);
-            options.ReadEnvironment(Actions);
             return new Autobuilder(Actions, options);
         }
 

csharp/autobuilder/Semmle.Autobuild/AspBuildRule.cs

@@ -7,10 +7,9 @@
     {
         public BuildScript Analyse(Autobuilder builder, bool auto)
         {
-            (var javaHome, var dist) =
+            var javaHome = builder.JavaHome;
-                builder.CodeQLJavaHome != null ?
+            var dist = builder.Distribution;
-                (builder.CodeQLJavaHome, builder.CodeQLExtractorCSharpRoot) :
+
-                (builder.SemmleJavaHome, builder.SemmleDist);
             var command = new CommandBuilder(builder.Actions).
                 RunCommand(builder.Actions.PathCombine(javaHome, "bin", "java")).
                 Argument("-jar").

csharp/autobuilder/Semmle.Autobuild/AutobuildOptions.cs

@@ -10,40 +10,40 @@ namespace Semmle.Autobuild
     public class AutobuildOptions
     {
         public readonly int SearchDepth = 3;
-        public string RootDirectory = null;
+        public readonly string RootDirectory;
-        static readonly string prefix = "LGTM_INDEX_";
+        private const string prefix = "LGTM_INDEX_";
 
-        public string VsToolsVersion;
+        public readonly string? VsToolsVersion;
-        public string MsBuildArguments;
+        public readonly string? MsBuildArguments;
-        public string MsBuildPlatform;
+        public readonly string? MsBuildPlatform;
-        public string MsBuildConfiguration;
+        public readonly string? MsBuildConfiguration;
-        public string MsBuildTarget;
+        public readonly string? MsBuildTarget;
-        public string DotNetArguments;
+        public readonly string? DotNetArguments;
-        public string DotNetVersion;
+        public readonly string? DotNetVersion;
-        public string BuildCommand;
+        public readonly string? BuildCommand;
-        public string[] Solution;
+        public readonly string[] Solution;
 
-        public bool IgnoreErrors;
+        public readonly bool IgnoreErrors;
-        public bool Buildless;
+        public readonly bool Buildless;
-        public bool AllSolutions;
+        public readonly bool AllSolutions;
-        public bool NugetRestore;
+        public readonly bool NugetRestore;
 
-        public Language Language;
+        public readonly Language Language;
-        public bool Indexing;
+        public readonly bool Indexing;
 
         /// <summary>
         /// Reads options from environment variables.
         /// Throws ArgumentOutOfRangeException for invalid arguments.
         /// </summary>
-        public void ReadEnvironment(IBuildActions actions)
+        public AutobuildOptions(IBuildActions actions)
         {
             RootDirectory = actions.GetCurrentDirectory();
             VsToolsVersion = actions.GetEnvironmentVariable(prefix + "VSTOOLS_VERSION");
-            MsBuildArguments = actions.GetEnvironmentVariable(prefix + "MSBUILD_ARGUMENTS").AsStringWithExpandedEnvVars(actions);
+            MsBuildArguments = actions.GetEnvironmentVariable(prefix + "MSBUILD_ARGUMENTS")?.AsStringWithExpandedEnvVars(actions);
             MsBuildPlatform = actions.GetEnvironmentVariable(prefix + "MSBUILD_PLATFORM");
             MsBuildConfiguration = actions.GetEnvironmentVariable(prefix + "MSBUILD_CONFIGURATION");
             MsBuildTarget = actions.GetEnvironmentVariable(prefix + "MSBUILD_TARGET");
-            DotNetArguments = actions.GetEnvironmentVariable(prefix + "DOTNET_ARGUMENTS").AsStringWithExpandedEnvVars(actions);
+            DotNetArguments = actions.GetEnvironmentVariable(prefix + "DOTNET_ARGUMENTS")?.AsStringWithExpandedEnvVars(actions);
             DotNetVersion = actions.GetEnvironmentVariable(prefix + "DOTNET_VERSION");
             BuildCommand = actions.GetEnvironmentVariable(prefix + "BUILD_COMMAND");
             Solution = actions.GetEnvironmentVariable(prefix + "SOLUTION").AsListWithExpandedEnvVars(actions, new string[0]);
@@ -60,7 +60,7 @@ namespace Semmle.Autobuild
 
     public static class OptionsExtensions
     {
-        public static bool AsBool(this string value, string param, bool defaultValue)
+        public static bool AsBool(this string? value, string param, bool defaultValue)
         {
             if (value == null) return defaultValue;
             switch (value.ToLower())
@@ -80,7 +80,7 @@ namespace Semmle.Autobuild
             }
         }
 
-        public static Language AsLanguage(this string key)
+        public static Language AsLanguage(this string? key)
         {
             switch (key)
             {
@@ -95,7 +95,7 @@ namespace Semmle.Autobuild
             }
         }
 
-        public static string[] AsListWithExpandedEnvVars(this string value, IBuildActions actions, string[] defaultValue)
+        public static string[] AsListWithExpandedEnvVars(this string? value, IBuildActions actions, string[] defaultValue)
         {
             if (value == null)
                 return defaultValue;

csharp/autobuilder/Semmle.Autobuild/Autobuilder.cs

@@ -20,6 +20,14 @@ namespace Semmle.Autobuild
         BuildScript Analyse(Autobuilder builder, bool auto);
     }
 
+    /// <summary>
+    /// Exception indicating that environment variables are missing or invalid.
+    /// </summary>
+    class InvalidEnvironmentException : Exception
+    {
+        public InvalidEnvironmentException(string m) : base(m) { }
+    }
+
     /// <summary>
     /// Main application logic, containing all data
     /// gathered from the project and filesystem.
@@ -69,7 +77,7 @@ namespace Semmle.Autobuild
         /// List of project/solution files to build.
         /// </summary>
         public IList<IProjectOrSolution> ProjectsOrSolutionsToBuild => projectsOrSolutionsToBuildLazy.Value;
-        readonly Lazy<IList<IProjectOrSolution>> projectsOrSolutionsToBuildLazy;
+        private readonly Lazy<IList<IProjectOrSolution>> projectsOrSolutionsToBuildLazy;
 
         /// <summary>
         /// Holds if a given path was found.
@@ -129,7 +137,7 @@ namespace Semmle.Autobuild
 
             projectsOrSolutionsToBuildLazy = new Lazy<IList<IProjectOrSolution>>(() =>
             {
-                List<IProjectOrSolution> ret;
+                List<IProjectOrSolution>? ret;
                 if (options.Solution.Any())
                 {
                     ret = new List<IProjectOrSolution>();
@@ -143,7 +151,7 @@ namespace Semmle.Autobuild
                     return ret;
                 }
 
-                IEnumerable<IProjectOrSolution> FindFiles(string extension, Func<string, ProjectOrSolution> create)
+                IEnumerable<IProjectOrSolution>? FindFiles(string extension, Func<string, ProjectOrSolution> create)
                 {
                     var matchingFiles = GetExtensions(extension).
                         Select(p => (ProjectOrSolution: create(p.Item1), DistanceFromRoot: p.Item2)).
@@ -177,19 +185,34 @@ namespace Semmle.Autobuild
             });
 
             CodeQLExtractorCSharpRoot = Actions.GetEnvironmentVariable("CODEQL_EXTRACTOR_CSHARP_ROOT");
-
-            CodeQLJavaHome = Actions.GetEnvironmentVariable("CODEQL_JAVA_HOME");
-
             SemmleDist = Actions.GetEnvironmentVariable("SEMMLE_DIST");
-
-            SemmleJavaHome = Actions.GetEnvironmentVariable("SEMMLE_JAVA_HOME");
-
             SemmlePlatformTools = Actions.GetEnvironmentVariable("SEMMLE_PLATFORM_TOOLS");
 
-            if (CodeQLExtractorCSharpRoot == null && SemmleDist == null)
+            JavaHome = 
-                Log(Severity.Error, "The environment variables CODEQL_EXTRACTOR_CSHARP_ROOT and SEMMLE_DIST have not been set.");
+                Actions.GetEnvironmentVariable("CODEQL_JAVA_HOME") ??
+                Actions.GetEnvironmentVariable("SEMMLE_JAVA_HOME") ?? 
+                throw new InvalidEnvironmentException("The environment variable CODEQL_JAVA_HOME or SEMMLE_JAVA_HOME has not been set.");
+
+            Distribution =
+                CodeQLExtractorCSharpRoot ??
+                SemmleDist ??
+                throw new InvalidEnvironmentException("The environment variable CODEQL_EXTRACTOR_CSHARP_ROOT or SEMMLE_DIST has not been set.");
+
+            TrapDir =
+                Actions.GetEnvironmentVariable("CODEQL_EXTRACTOR_CSHARP_TRAP_DIR") ??
+                Actions.GetEnvironmentVariable("TRAP_FOLDER") ??
+                throw new InvalidEnvironmentException("The environment variable CODEQL_EXTRACTOR_CSHARP_TRAP_DIR or TRAP_FOLDER has not been set.");
+
+            SourceArchiveDir =
+                Actions.GetEnvironmentVariable("CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR") ??
+                Actions.GetEnvironmentVariable("SOURCE_ARCHIVE") ??
+                throw new InvalidEnvironmentException("The environment variable CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR or SOURCE_ARCHIVE has not been set.");
         }
 
+        private string TrapDir { get; }
+
+        private string SourceArchiveDir { get; }
+
         readonly ILogger logger = new ConsoleLogger(Verbosity.Info);
 
         /// <summary>
@@ -271,9 +294,9 @@ namespace Semmle.Autobuild
                     break;
                 case CSharpBuildStrategy.Auto:
                     var cleanTrapFolder =
-                        BuildScript.DeleteDirectory(Actions.GetEnvironmentVariable("CODEQL_EXTRACTOR_CSHARP_TRAP_DIR") ?? Actions.GetEnvironmentVariable("TRAP_FOLDER"));
+                        BuildScript.DeleteDirectory(TrapDir);
                     var cleanSourceArchive =
-                        BuildScript.DeleteDirectory(Actions.GetEnvironmentVariable("CODEQL_EXTRACTOR_CSHARP_SOURCE_ARCHIVE_DIR") ?? Actions.GetEnvironmentVariable("SOURCE_ARCHIVE"));
+                        BuildScript.DeleteDirectory(SourceArchiveDir);
                     var tryCleanExtractorArgsLogs =
                         BuildScript.Create(actions =>
                         {
@@ -376,38 +399,33 @@ namespace Semmle.Autobuild
         /// <summary>
         /// Value of CODEQL_EXTRACTOR_CSHARP_ROOT environment variable.
         /// </summary>
-        public string CodeQLExtractorCSharpRoot { get; private set; }
+        private string? CodeQLExtractorCSharpRoot { get; }
-
-        /// <summary>
-        /// Value of CODEQL_JAVA_HOME environment variable.
-        /// </summary>
-        public string CodeQLJavaHome { get; private set; }
 
         /// <summary>
         /// Value of SEMMLE_DIST environment variable.
         /// </summary>
-        public string SemmleDist { get; private set; }
+        private string? SemmleDist { get; }
 
-        /// <summary>
+        public string Distribution { get; }
-        /// Value of SEMMLE_JAVA_HOME environment variable.
+
-        /// </summary>
+        public string JavaHome { get; }
-        public string SemmleJavaHome { get; private set; }
 
         /// <summary>
         /// Value of SEMMLE_PLATFORM_TOOLS environment variable.
         /// </summary>
-        public string SemmlePlatformTools { get; private set; }
+        public string? SemmlePlatformTools { get; }
 
         /// <summary>
         /// The absolute path of the odasa executable.
+        /// null if we are running in CodeQL.
         /// </summary>
-        public string Odasa => SemmleDist == null ? null : Actions.PathCombine(SemmleDist, "tools", "odasa");
+        public string? Odasa => SemmleDist is null ? null : Actions.PathCombine(SemmleDist, "tools", "odasa");
 
         /// <summary>
         /// Construct a command that executed the given <paramref name="cmd"/> wrapped in
         /// an <code>odasa --index</code>, unless indexing has been disabled, in which case
         /// <paramref name="cmd"/> is run directly.
         /// </summary>
-        internal CommandBuilder MaybeIndex(CommandBuilder builder, string cmd) => Options.Indexing ? builder.IndexCommand(Odasa, cmd) : builder.RunCommand(cmd);
+        internal CommandBuilder MaybeIndex(CommandBuilder builder, string cmd) => Options.Indexing && !(Odasa is null) ? builder.IndexCommand(Odasa, cmd) : builder.RunCommand(cmd);
     }
 }

csharp/autobuilder/Semmle.Autobuild/BuildActions.cs

@@ -21,7 +21,7 @@ namespace Semmle.Autobuild
         /// <param name="env">Additional environment variables.</param>
         /// <param name="stdOut">The lines of stdout.</param>
         /// <returns>The process exit code.</returns>
-        int RunProcess(string exe, string args, string workingDirectory, IDictionary<string, string> env, out IList<string> stdOut);
+        int RunProcess(string exe, string args, string? workingDirectory, IDictionary<string, string>? env, out IList<string> stdOut);
 
         /// <summary>
         /// Runs a process but does not capture its output.
@@ -31,7 +31,7 @@ namespace Semmle.Autobuild
         /// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
         /// <param name="env">Additional environment variables.</param>
         /// <returns>The process exit code.</returns>
-        int RunProcess(string exe, string args, string workingDirectory, IDictionary<string, string> env);
+        int RunProcess(string exe, string args, string? workingDirectory, IDictionary<string, string>? env);
 
         /// <summary>
         /// Tests whether a file exists, File.Exists().
@@ -63,7 +63,7 @@ namespace Semmle.Autobuild
         /// </summary>
         /// <param name="name">The name of the variable.</param>
         /// <returns>The string value, or null if the variable is not defined.</returns>
-        string GetEnvironmentVariable(string name);
+        string? GetEnvironmentVariable(string name);
 
         /// <summary>
         /// Gets the current directory, Directory.GetCurrentDirectory().
@@ -130,7 +130,7 @@ namespace Semmle.Autobuild
 
         bool IBuildActions.FileExists(string file) => File.Exists(file);
 
-        ProcessStartInfo GetProcessStartInfo(string exe, string arguments, string workingDirectory, IDictionary<string, string> environment, bool redirectStandardOutput)
+        ProcessStartInfo GetProcessStartInfo(string exe, string arguments, string? workingDirectory, IDictionary<string, string>? environment, bool redirectStandardOutput)
         {
             var pi = new ProcessStartInfo(exe, arguments)
             {
@@ -146,7 +146,7 @@ namespace Semmle.Autobuild
             return pi;
         }
 
-        int IBuildActions.RunProcess(string cmd, string args, string workingDirectory, IDictionary<string, string> environment)
+        int IBuildActions.RunProcess(string cmd, string args, string? workingDirectory, IDictionary<string, string>? environment)
         {
             var pi = GetProcessStartInfo(cmd, args, workingDirectory, environment, false);
             using (var p = Process.Start(pi))
@@ -156,7 +156,7 @@ namespace Semmle.Autobuild
             }
         }
 
-        int IBuildActions.RunProcess(string cmd, string args, string workingDirectory, IDictionary<string, string> environment, out IList<string> stdOut)
+        int IBuildActions.RunProcess(string cmd, string args, string? workingDirectory, IDictionary<string, string>? environment, out IList<string> stdOut)
         {
             var pi = GetProcessStartInfo(cmd, args, workingDirectory, environment, true);
             return pi.ReadOutput(out stdOut);
@@ -166,7 +166,7 @@ namespace Semmle.Autobuild
 
         bool IBuildActions.DirectoryExists(string dir) => Directory.Exists(dir);
 
-        string IBuildActions.GetEnvironmentVariable(string name) => Environment.GetEnvironmentVariable(name);
+        string? IBuildActions.GetEnvironmentVariable(string name) => Environment.GetEnvironmentVariable(name);
 
         string IBuildActions.GetCurrentDirectory() => Directory.GetCurrentDirectory();
 

csharp/autobuilder/Semmle.Autobuild/BuildCommandAutoRule.cs

@@ -40,7 +40,7 @@ namespace Semmle.Autobuild
             chmod.RunCommand("/bin/chmod", $"u+x {scriptPath}");
             var chmodScript = builder.Actions.IsWindows() ? BuildScript.Success : BuildScript.Try(chmod.Script);
 
-            var dir = Path.GetDirectoryName(scriptPath);
+            string? dir = Path.GetDirectoryName(scriptPath);
 
             // A specific .NET Core version may be required
             return chmodScript & DotNetRule.WithDotNet(builder, environment =>

csharp/autobuilder/Semmle.Autobuild/BuildScript.cs

@@ -48,8 +48,9 @@ namespace Semmle.Autobuild
 
         class BuildCommand : BuildScript
         {
-            readonly string exe, arguments, workingDirectory;
+            readonly string exe, arguments;
-            readonly IDictionary<string, string> environment;
+            readonly string? workingDirectory;
+            readonly IDictionary<string, string>? environment;
             readonly bool silent;
 
             /// <summary>
@@ -60,7 +61,7 @@ namespace Semmle.Autobuild
             /// <param name="silent">Whether this command should run silently.</param>
             /// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
             /// <param name="environment">Additional environment variables.</param>
-            public BuildCommand(string exe, string argumentsOpt, bool silent, string workingDirectory = null, IDictionary<string, string> environment = null)
+            public BuildCommand(string exe, string argumentsOpt, bool silent, string? workingDirectory = null, IDictionary<string, string>? environment = null)
             {
                 this.exe = exe;
                 this.arguments = argumentsOpt ?? "";
@@ -131,8 +132,8 @@ namespace Semmle.Autobuild
         class BindBuildScript : BuildScript
         {
             readonly BuildScript s1;
-            readonly Func<IList<string>, int, BuildScript> s2a;
+            readonly Func<IList<string>, int, BuildScript>? s2a;
-            readonly Func<int, BuildScript> s2b;
+            readonly Func<int, BuildScript>? s2b;
             public BindBuildScript(BuildScript s1, Func<IList<string>, int, BuildScript> s2)
             {
                 this.s1 = s1;
@@ -154,14 +155,19 @@ namespace Semmle.Autobuild
                     return s2a(stdout1, ret1).Run(actions, startCallback, exitCallBack);
                 }
 
-                ret1 = s1.Run(actions, startCallback, exitCallBack);
+                if (s2b != null)
-                return s2b(ret1).Run(actions, startCallback, exitCallBack);
+                {
+                    ret1 = s1.Run(actions, startCallback, exitCallBack);
+                    return s2b(ret1).Run(actions, startCallback, exitCallBack);
+                }
+
+                throw new InvalidOperationException("Unexpected error");
             }
 
             public override int Run(IBuildActions actions, Action<string, bool> startCallback, Action<int, string, bool> exitCallBack, out IList<string> stdout)
             {
                 var ret1 = s1.Run(actions, startCallback, exitCallBack, out var stdout1);
-                var ret2 = (s2a != null ? s2a(stdout1, ret1) : s2b(ret1)).Run(actions, startCallback, exitCallBack, out var stdout2);
+                var ret2 = (s2a != null ? s2a(stdout1, ret1) : s2b!(ret1)).Run(actions, startCallback, exitCallBack, out var stdout2);
                 var @out = new List<string>();
                 @out.AddRange(stdout1);
                 @out.AddRange(stdout2);
@@ -177,7 +183,7 @@ namespace Semmle.Autobuild
         /// <param name="silent">Whether the executable should run silently.</param>
         /// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
         /// <param name="environment">Additional environment variables.</param>
-        public static BuildScript Create(string exe, string argumentsOpt, bool silent, string workingDirectory, IDictionary<string, string> environment) =>
+        public static BuildScript Create(string exe, string argumentsOpt, bool silent, string? workingDirectory, IDictionary<string, string>? environment) =>
             new BuildCommand(exe, argumentsOpt, silent, workingDirectory, environment);
 
         /// <summary>

csharp/autobuilder/Semmle.Autobuild/CommandBuilder.cs

@@ -13,10 +13,10 @@ namespace Semmle.Autobuild
 
         readonly StringBuilder arguments;
         bool firstCommand;
-        string executable;
+        string? executable;
         readonly EscapeMode escapingMode;
-        readonly string workingDirectory;
+        readonly string? workingDirectory;
-        readonly IDictionary<string, string> environment;
+        readonly IDictionary<string, string>? environment;
         readonly bool silent;
 
         /// <summary>
@@ -25,7 +25,7 @@ namespace Semmle.Autobuild
         /// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
         /// <param name="environment">Additional environment variables.</param>
         /// <param name="silent">Whether this command should be run silently.</param>
-        public CommandBuilder(IBuildActions actions, string workingDirectory = null, IDictionary<string, string> environment = null, bool silent = false)
+        public CommandBuilder(IBuildActions actions, string? workingDirectory = null, IDictionary<string, string>? environment = null, bool silent = false)
         {
             arguments = new StringBuilder();
             if (actions.IsWindows())
@@ -50,7 +50,7 @@ namespace Semmle.Autobuild
             RunCommand(odasa, "index --auto");
         }
 
-        public CommandBuilder CallBatFile(string batFile, string argumentsOpt = null)
+        public CommandBuilder CallBatFile(string batFile, string? argumentsOpt = null)
         {
             NextCommand();
             arguments.Append(" CALL");
@@ -66,7 +66,7 @@ namespace Semmle.Autobuild
         /// <param name="command">The command to run.</param>
         /// <param name="argumentsOpt">Additional arguments.</param>
         /// <returns>this for chaining calls.</returns>
-        public CommandBuilder IndexCommand(string odasa, string command, string argumentsOpt = null)
+        public CommandBuilder IndexCommand(string odasa, string command, string? argumentsOpt = null)
         {
             OdasaIndex(odasa);
             QuoteArgument(command);
@@ -151,7 +151,7 @@ namespace Semmle.Autobuild
                 arguments.Append(' ');
         }
 
-        public CommandBuilder Argument(string argumentsOpt)
+        public CommandBuilder Argument(string? argumentsOpt)
         {
             if (argumentsOpt != null)
             {
@@ -169,7 +169,7 @@ namespace Semmle.Autobuild
                 arguments.Append(" &&");
         }
 
-        public CommandBuilder RunCommand(string exe, string argumentsOpt = null)
+        public CommandBuilder RunCommand(string exe, string? argumentsOpt = null)
         {
             var (exe0, arg0) =
                 escapingMode == EscapeMode.Process && exe.EndsWith(".exe", System.StringComparison.Ordinal)
@@ -193,6 +193,14 @@ namespace Semmle.Autobuild
         /// <summary>
         /// Returns a build script that contains just this command.
         /// </summary>
-        public BuildScript Script => BuildScript.Create(executable, arguments.ToString(), silent, workingDirectory, environment);
+        public BuildScript Script
+        {
+            get
+            {
+                if (executable is null)
+                    throw new System.InvalidOperationException("executable is null");
+                return BuildScript.Create(executable, arguments.ToString(), silent, workingDirectory, environment);
+            }
+        }
     }
 }

csharp/autobuilder/Semmle.Autobuild/DotNetRule.cs

@@ -56,13 +56,13 @@ namespace Semmle.Autobuild
                 });
         }
 
-        static BuildScript WithDotNet(Autobuilder builder, Func<string, IDictionary<string, string>, bool, BuildScript> f)
+        static BuildScript WithDotNet(Autobuilder builder, Func<string?, IDictionary<string, string>?, bool, BuildScript> f)
         {
-            var installDir = builder.Actions.PathCombine(builder.Options.RootDirectory, ".dotnet");
+            string? installDir = builder.Actions.PathCombine(builder.Options.RootDirectory, ".dotnet");
             var installScript = DownloadDotNet(builder, installDir);
             return BuildScript.Bind(installScript, installed =>
             {
-                Dictionary<string, string> env;
+                Dictionary<string, string>? env;
                 if (installed == 0)
                 {
                     // The installation succeeded, so use the newly installed .NET Core
@@ -120,7 +120,7 @@ namespace Semmle.Autobuild
         /// variables needed by the installed .NET Core (<code>null</code> when no variables
         /// are needed).
         /// </summary>
-        public static BuildScript WithDotNet(Autobuilder builder, Func<IDictionary<string, string>, BuildScript> f)
+        public static BuildScript WithDotNet(Autobuilder builder, Func<IDictionary<string, string>?, BuildScript> f)
             => WithDotNet(builder, (_1, env, _2) => f(env));
 
         /// <summary>
@@ -265,10 +265,10 @@ Invoke-Command -ScriptBlock $ScriptBlock";
             return listSdks.Script;
         }
 
-        static string DotNetCommand(IBuildActions actions, string dotNetPath) =>
+        static string DotNetCommand(IBuildActions actions, string? dotNetPath) =>
             dotNetPath != null ? actions.PathCombine(dotNetPath, "dotnet") : "dotnet";
 
-        BuildScript GetInfoCommand(IBuildActions actions, string dotNetPath, IDictionary<string, string> environment)
+        BuildScript GetInfoCommand(IBuildActions actions, string? dotNetPath, IDictionary<string, string>? environment)
         {
             var info = new CommandBuilder(actions, null, environment).
                 RunCommand(DotNetCommand(actions, dotNetPath)).
@@ -276,7 +276,7 @@ Invoke-Command -ScriptBlock $ScriptBlock";
             return info.Script;
         }
 
-        CommandBuilder GetCleanCommand(IBuildActions actions, string dotNetPath, IDictionary<string, string> environment)
+        CommandBuilder GetCleanCommand(IBuildActions actions, string? dotNetPath, IDictionary<string, string>? environment)
         {
             var clean = new CommandBuilder(actions, null, environment).
                 RunCommand(DotNetCommand(actions, dotNetPath)).
@@ -284,7 +284,7 @@ Invoke-Command -ScriptBlock $ScriptBlock";
             return clean;
         }
 
-        CommandBuilder GetRestoreCommand(IBuildActions actions, string dotNetPath, IDictionary<string, string> environment)
+        CommandBuilder GetRestoreCommand(IBuildActions actions, string? dotNetPath, IDictionary<string, string>? environment)
         {
             var restore = new CommandBuilder(actions, null, environment).
                 RunCommand(DotNetCommand(actions, dotNetPath)).
@@ -292,7 +292,7 @@ Invoke-Command -ScriptBlock $ScriptBlock";
             return restore;
         }
 
-        static BuildScript GetInstalledRuntimesScript(IBuildActions actions, string dotNetPath, IDictionary<string, string> environment)
+        static BuildScript GetInstalledRuntimesScript(IBuildActions actions, string? dotNetPath, IDictionary<string, string>? environment)
         {
             var listSdks = new CommandBuilder(actions, environment: environment, silent: true).
                 RunCommand(DotNetCommand(actions, dotNetPath)).
@@ -309,7 +309,7 @@ Invoke-Command -ScriptBlock $ScriptBlock";
         /// hence the need for CLR tracing), by adding a
         /// `/p:UseSharedCompilation=false` argument.
         /// </summary>
-        BuildScript GetBuildScript(Autobuilder builder, string dotNetPath, IDictionary<string, string> environment, bool compatibleClr, string projOrSln)
+        BuildScript GetBuildScript(Autobuilder builder, string? dotNetPath, IDictionary<string, string>? environment, bool compatibleClr, string projOrSln)
         {
             var build = new CommandBuilder(builder.Actions, null, environment);
             var script = builder.MaybeIndex(build, DotNetCommand(builder.Actions, dotNetPath)).

csharp/autobuilder/Semmle.Autobuild/MsBuildRule.cs

@@ -67,10 +67,10 @@ namespace Semmle.Autobuild
                 string target = builder.Options.MsBuildTarget != null
                                        ? builder.Options.MsBuildTarget
                                        : "rebuild";
-                string platform = builder.Options.MsBuildPlatform != null
+                string? platform = builder.Options.MsBuildPlatform != null
                                          ? builder.Options.MsBuildPlatform
                                          : projectOrSolution is ISolution s1 ? s1.DefaultPlatformName : null;
-                string configuration = builder.Options.MsBuildConfiguration != null
+                string? configuration = builder.Options.MsBuildConfiguration != null
                                               ? builder.Options.MsBuildConfiguration
                                               : projectOrSolution is ISolution s2 ? s2.DefaultConfigurationName : null;
 
@@ -96,9 +96,9 @@ namespace Semmle.Autobuild
         ///
         /// Returns <code>null</code> when no version is specified.
         /// </summary>
-        public static VcVarsBatFile GetVcVarsBatFile(Autobuilder builder)
+        public static VcVarsBatFile? GetVcVarsBatFile(Autobuilder builder)
         {
-            VcVarsBatFile vsTools = null;
+            VcVarsBatFile? vsTools = null;
 
             if (builder.Options.VsToolsVersion != null)
             {

csharp/autobuilder/Semmle.Autobuild/Program.cs

@@ -6,23 +6,27 @@ namespace Semmle.Autobuild
     {
         static int Main()
         {
-            var options = new AutobuildOptions();
-            var actions = SystemBuildActions.Instance;
 
             try
             {
-                options.ReadEnvironment(actions);
+                var actions = SystemBuildActions.Instance;
+                var options = new AutobuildOptions(actions);
+                try
+                {
+                    Console.WriteLine($"Semmle autobuilder for {options.Language}");
+                    var builder = new Autobuilder(actions, options);
+                    return builder.AttemptBuild();
+                }
+                catch(InvalidEnvironmentException ex)
+                {
+                    Console.WriteLine("The environment is invalid: {0}", ex.Message);
+                }
             }
             catch (ArgumentOutOfRangeException ex)
             {
                 Console.WriteLine("The value \"{0}\" for parameter \"{1}\" is invalid", ex.ActualValue, ex.ParamName);
             }
-
+            return 1;
-            var builder = new Autobuilder(actions, options);
-
-            Console.WriteLine($"Semmle autobuilder for {options.Language}");
-
-            return builder.AttemptBuild();
         }
     }
 }

csharp/autobuilder/Semmle.Autobuild/Project.cs

@@ -82,7 +82,7 @@ namespace Semmle.Autobuild
                     foreach (var include in projectFileIncludes.Concat(projectFilesIncludes))
                     {
                         var includePath = builder.Actions.PathCombine(include.Value.Split('\\', StringSplitOptions.RemoveEmptyEntries));
-                        ret.Add(new Project(builder, builder.Actions.PathCombine(Path.GetDirectoryName(this.FullPath), includePath)));
+                        ret.Add(new Project(builder, builder.Actions.PathCombine(DirectoryName, includePath)));
                     }
                     return ret;
                 });

csharp/autobuilder/Semmle.Autobuild/ProjectOrSolution.cs

@@ -1,4 +1,5 @@
 using System.Collections.Generic;
+using System.IO;
 using System.Linq;
 
 namespace Semmle.Autobuild
@@ -24,6 +25,8 @@ namespace Semmle.Autobuild
     {
         public string FullPath { get; private set; }
 
+        public string DirectoryName => Path.GetDirectoryName(FullPath) ?? "";
+
         protected ProjectOrSolution(Autobuilder builder, string path)
         {
             FullPath = builder.Actions.GetFullPath(path);

csharp/autobuilder/Semmle.Autobuild/Semmle.Autobuild.csproj

@@ -9,6 +9,7 @@
     <StartupObject />
     <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
     <RuntimeIdentifiers>win-x64;linux-x64;osx-x64</RuntimeIdentifiers>
+    <Nullable>enable</Nullable>
   </PropertyGroup>
 
   <ItemGroup>

csharp/autobuilder/Semmle.Autobuild/Solution.cs

@@ -43,7 +43,7 @@ namespace Semmle.Autobuild
     /// </summary>
     class Solution : ProjectOrSolution, ISolution
     {
-        readonly SolutionFile solution;
+        readonly SolutionFile? solution;
 
         readonly IEnumerable<Project> includedProjects;
         public override IEnumerable<IProjectOrSolution> IncludedProjects => includedProjects;
@@ -81,7 +81,7 @@ namespace Semmle.Autobuild
             includedProjects =
                 solution.ProjectsInOrder.
                 Where(p => p.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat).
-                Select(p => builder.Actions.PathCombine(Path.GetDirectoryName(path), builder.Actions.PathCombine(p.RelativePath.Split('\\', StringSplitOptions.RemoveEmptyEntries)))).
+                Select(p => builder.Actions.PathCombine(DirectoryName, builder.Actions.PathCombine(p.RelativePath.Split('\\', StringSplitOptions.RemoveEmptyEntries)))).
                 Select(p => new Project(builder, p)).
                 ToArray();
         }

csharp/autobuilder/Semmle.Autobuild/StandaloneBuildRule.cs

@@ -1,6 +1,4 @@
-using System.IO;
+namespace Semmle.Autobuild
-
-namespace Semmle.Autobuild
 {
     /// <summary>
     /// Build using standalone extraction.
@@ -9,8 +7,11 @@ namespace Semmle.Autobuild
     {
         public BuildScript Analyse(Autobuilder builder, bool auto)
         {
-            BuildScript GetCommand(string solution)
+            BuildScript GetCommand(string? solution)
             {
+                if (builder.SemmlePlatformTools is null)
+                    return BuildScript.Failure;
+
                 var standalone = builder.Actions.PathCombine(builder.SemmlePlatformTools, "csharp", "Semmle.Extraction.CSharp.Standalone");
                 var cmd = new CommandBuilder(builder.Actions);
                 cmd.RunCommand(standalone);

csharp/autobuilder/Semmle.Autobuild/XmlBuildRule.cs

@@ -7,7 +7,7 @@
     {
         public BuildScript Analyse(Autobuilder builder, bool auto)
         {
-            if (!builder.Options.Indexing)
+            if (!builder.Options.Indexing || builder.Odasa is null)
                 return BuildScript.Success;
 
             var command = new CommandBuilder(builder.Actions).

csharp/extractor/Semmle.Extraction.CSharp.Standalone/AssemblyCache.cs

@@ -163,7 +163,19 @@ namespace Semmle.BuildAnalyser
         /// </summary>
         /// <param name="filepath">The filename to query.</param>
         /// <returns>The assembly info.</returns>
-        public AssemblyInfo GetAssemblyInfo(string filepath) => assemblyInfo[filepath];
+        public AssemblyInfo GetAssemblyInfo(string filepath)
+        {
+            if(assemblyInfo.TryGetValue(filepath, out var info))
+            {
+                return info;
+            }
+            else
+            {
+                info = AssemblyInfo.ReadFromFile(filepath);
+                assemblyInfo.Add(filepath, info);
+                return info;
+            }
+        }
 
         // List of pending DLLs to index.
         readonly List<string> dlls = new List<string>();

csharp/extractor/Semmle.Extraction.CSharp.Standalone/BuildAnalysis.cs

@@ -1,10 +1,13 @@
-using System;
+using Semmle.Util;
+using System;
 using System.Collections.Generic;
 using System.IO;
 using System.Linq;
-using System.Runtime.InteropServices;
-using Semmle.Util;
 using Semmle.Extraction.CSharp.Standalone;
+using System.Threading.Tasks;
+using System.Collections.Concurrent;
+using System.Text;
+using System.Security.Cryptography;
 
 namespace Semmle.BuildAnalyser
 {
@@ -43,19 +46,18 @@ namespace Semmle.BuildAnalyser
     /// <summary>
     /// Main implementation of the build analysis.
     /// </summary>
-    class BuildAnalysis : IBuildAnalysis
+    class BuildAnalysis : IBuildAnalysis, IDisposable
     {
-        readonly AssemblyCache assemblyCache;
+        private readonly AssemblyCache assemblyCache;
-        readonly NugetPackages nuget;
+        private readonly NugetPackages nuget;
-        readonly IProgressMonitor progressMonitor;
+        private readonly IProgressMonitor progressMonitor;
-        HashSet<string> usedReferences = new HashSet<string>();
+        private readonly IDictionary<string, bool> usedReferences = new ConcurrentDictionary<string, bool>();
-        readonly HashSet<string> usedSources = new HashSet<string>();
+        private readonly IDictionary<string, bool> sources = new ConcurrentDictionary<string, bool>();
-        readonly HashSet<string> missingSources = new HashSet<string>();
+        private readonly IDictionary<string, string> unresolvedReferences = new ConcurrentDictionary<string, string>();
-        readonly Dictionary<string, string> unresolvedReferences = new Dictionary<string, string>();
+        private readonly DirectoryInfo sourceDir;
-        readonly DirectoryInfo sourceDir;
+        private int failedProjects, succeededProjects;
-        int failedProjects, succeededProjects;
+        private readonly string[] allSources;
-        readonly string[] allSources;
+        private int conflictedReferences = 0;
-        int conflictedReferences = 0;
 
         /// <summary>
         /// Performs a C# build analysis.
@@ -64,6 +66,8 @@ namespace Semmle.BuildAnalyser
         /// <param name="progress">Display of analysis progress.</param>
         public BuildAnalysis(Options options, IProgressMonitor progress)
         {
+            var startTime = DateTime.Now;
+
             progressMonitor = progress;
             sourceDir = new DirectoryInfo(options.SrcDir);
 
@@ -74,36 +78,43 @@ namespace Semmle.BuildAnalyser
                 Where(d => !options.ExcludesFile(d)).
                 ToArray();
 
-            var dllDirNames = options.DllDirs.Select(Path.GetFullPath);
+            var dllDirNames = options.DllDirs.Select(Path.GetFullPath).ToList();
+            PackageDirectory = new TemporaryDirectory(ComputeTempDirectory(sourceDir.FullName));
 
             if (options.UseNuGet)
             {
-                nuget = new NugetPackages(sourceDir.FullName);
+                try
-                ReadNugetFiles();
+                {
-                dllDirNames = dllDirNames.Concat(Enumerators.Singleton(nuget.PackageDirectory));
+                    nuget = new NugetPackages(sourceDir.FullName, PackageDirectory);
+                    ReadNugetFiles();
+                }
+                catch(FileNotFoundException)
+                {
+                    progressMonitor.MissingNuGet();
+                }
             }
 
             // Find DLLs in the .Net Framework
             if (options.ScanNetFrameworkDlls)
             {
-                dllDirNames = dllDirNames.Concat(Runtime.Runtimes.Take(1));
+                dllDirNames.Add(Runtime.Runtimes.First());
             }
 
-            assemblyCache = new BuildAnalyser.AssemblyCache(dllDirNames, progress);
+            // These files can sometimes prevent `dotnet restore` from working correctly.
+            using (new FileRenamer(sourceDir.GetFiles("global.json", SearchOption.AllDirectories)))
+            using (new FileRenamer(sourceDir.GetFiles("Directory.Build.props", SearchOption.AllDirectories)))
+            {
+                var solutions = options.SolutionFile != null ?
+                        new[] { options.SolutionFile } :
+                        sourceDir.GetFiles("*.sln", SearchOption.AllDirectories).Select(d => d.FullName);
 
-            // Analyse all .csproj files in the source tree.
+                RestoreSolutions(solutions);
-            if (options.SolutionFile != null)
+                dllDirNames.Add(PackageDirectory.DirInfo.FullName);
-            {
+                assemblyCache = new BuildAnalyser.AssemblyCache(dllDirNames, progress);
-                AnalyseSolution(options.SolutionFile);
+                AnalyseSolutions(solutions);
-            }
-            else if (options.AnalyseCsProjFiles)
-            {
-                AnalyseProjectFiles();
-            }
 
-            if (!options.AnalyseCsProjFiles)
+                foreach (var filename in assemblyCache.AllAssemblies.Select(a => a.Filename))
-            {
+                    UseReference(filename);
-                usedReferences = new HashSet<string>(assemblyCache.AllAssemblies.Select(a => a.Filename));
             }
 
             ResolveConflicts();
@@ -114,7 +125,7 @@ namespace Semmle.BuildAnalyser
             }
 
             // Output the findings
-            foreach (var r in usedReferences)
+            foreach (var r in usedReferences.Keys)
             {
                 progressMonitor.ResolvedReference(r);
             }
@@ -132,7 +143,27 @@ namespace Semmle.BuildAnalyser
                 UnresolvedReferences.Count(),
                 conflictedReferences,
                 succeededProjects + failedProjects,
-                failedProjects);
+                failedProjects,
+                DateTime.Now - startTime);
+        }
+
+        /// <summary>
+        /// Computes a unique temp directory for the packages associated
+        /// with this source tree. Use a SHA1 of the directory name.
+        /// </summary>
+        /// <param name="srcDir"></param>
+        /// <returns>The full path of the temp directory.</returns>
+        private static string ComputeTempDirectory(string srcDir)
+        {
+            var bytes = Encoding.Unicode.GetBytes(srcDir);
+
+            using var sha1 = new SHA1CryptoServiceProvider();
+            var sha = sha1.ComputeHash(bytes);
+            var sb = new StringBuilder();
+            foreach (var b in sha.Take(8))
+                sb.AppendFormat("{0:x2}", b);
+
+            return Path.Combine(Path.GetTempPath(), "GitHub", "packages", sb.ToString());
         }
 
         /// <summary>
@@ -143,7 +174,7 @@ namespace Semmle.BuildAnalyser
         void ResolveConflicts()
         {
             var sortedReferences = usedReferences.
-                Select(r => assemblyCache.GetAssemblyInfo(r)).
+                Select(r => assemblyCache.GetAssemblyInfo(r.Key)).
                 OrderBy(r => r.Version).
                 ToArray();
 
@@ -154,7 +185,9 @@ namespace Semmle.BuildAnalyser
                 finalAssemblyList[r.Name] = r;
 
             // Update the used references list
-            usedReferences = new HashSet<string>(finalAssemblyList.Select(r => r.Value.Filename));
+            usedReferences.Clear();
+            foreach (var r in finalAssemblyList.Select(r => r.Value.Filename))
+                UseReference(r);
 
             // Report the results
             foreach (var r in sortedReferences)
@@ -183,7 +216,7 @@ namespace Semmle.BuildAnalyser
         /// <param name="reference">The filename of the reference.</param>
         void UseReference(string reference)
         {
-            usedReferences.Add(reference);
+            usedReferences[reference] = true;
         }
 
         /// <summary>
@@ -192,25 +225,18 @@ namespace Semmle.BuildAnalyser
         /// <param name="sourceFile">The source file.</param>
         void UseSource(FileInfo sourceFile)
         {
-            if (sourceFile.Exists)
+            sources[sourceFile.FullName] = sourceFile.Exists;
-            {
-                usedSources.Add(sourceFile.FullName);
-            }
-            else
-            {
-                missingSources.Add(sourceFile.FullName);
-            }
         }
 
         /// <summary>
         /// The list of resolved reference files.
         /// </summary>
-        public IEnumerable<string> ReferenceFiles => this.usedReferences;
+        public IEnumerable<string> ReferenceFiles => this.usedReferences.Keys;
 
         /// <summary>
         /// The list of source files used in projects.
         /// </summary>
-        public IEnumerable<string> ProjectSourceFiles => usedSources;
+        public IEnumerable<string> ProjectSourceFiles => sources.Where(s => s.Value).Select(s => s.Key);
 
         /// <summary>
         /// All of the source files in the source directory.
@@ -226,7 +252,7 @@ namespace Semmle.BuildAnalyser
         /// List of source files which were mentioned in project files but
         /// do not exist on the file system.
         /// </summary>
-        public IEnumerable<string> MissingSourceFiles => missingSources;
+        public IEnumerable<string> MissingSourceFiles => sources.Where(s => !s.Value).Select(s => s.Key);
 
         /// <summary>
         /// Record that a particular reference couldn't be resolved.
@@ -239,74 +265,101 @@ namespace Semmle.BuildAnalyser
             unresolvedReferences[id] = projectFile;
         }
 
-        /// <summary>
+        readonly TemporaryDirectory PackageDirectory;
-        /// Performs an analysis of all .csproj files.
-        /// </summary>
-        void AnalyseProjectFiles()
-        {
-            AnalyseProjectFiles(sourceDir.GetFiles("*.csproj", SearchOption.AllDirectories));
-        }
 
         /// <summary>
         /// Reads all the source files and references from the given list of projects.
         /// </summary>
         /// <param name="projectFiles">The list of projects to analyse.</param>
-        void AnalyseProjectFiles(FileInfo[] projectFiles)
+        void AnalyseProjectFiles(IEnumerable<FileInfo> projectFiles)
         {
-            progressMonitor.AnalysingProjectFiles(projectFiles.Count());
-
             foreach (var proj in projectFiles)
+                AnalyseProject(proj);
+        }
+
+        void AnalyseProject(FileInfo project)
+        {
+            if(!project.Exists)
             {
-                try
+                progressMonitor.MissingProject(project.FullName);
-                {
+                return;
-                    var csProj = new CsProjFile(proj);
+            }
 
-                    foreach (var @ref in csProj.References)
+            try
-                    {
+            {
-                        AssemblyInfo resolved = assemblyCache.ResolveReference(@ref);
+                var csProj = new CsProjFile(project);
-                        if (!resolved.Valid)
-                        {
-                            UnresolvedReference(@ref, proj.FullName);
-                        }
-                        else
-                        {
-                            UseReference(resolved.Filename);
-                        }
-                    }
 
-                    foreach (var src in csProj.Sources)
+                foreach (var @ref in csProj.References)
-                    {
-                        // Make a note of which source files the projects use.
-                        // This information doesn't affect the build but is dumped
-                        // as diagnostic output.
-                        UseSource(new FileInfo(src));
-                    }
-                    ++succeededProjects;
-                }
-                catch (Exception ex)  // lgtm[cs/catch-of-all-exceptions]
                 {
-                    ++failedProjects;
+                    AssemblyInfo resolved = assemblyCache.ResolveReference(@ref);
-                    progressMonitor.FailedProjectFile(proj.FullName, ex.Message);
+                    if (!resolved.Valid)
+                    {
+                        UnresolvedReference(@ref, project.FullName);
+                    }
+                    else
+                    {
+                        UseReference(resolved.Filename);
+                    }
                 }
+
+                foreach (var src in csProj.Sources)
+                {
+                    // Make a note of which source files the projects use.
+                    // This information doesn't affect the build but is dumped
+                    // as diagnostic output.
+                    UseSource(new FileInfo(src));
+                }
+
+                ++succeededProjects;
+            }
+            catch (Exception ex)  // lgtm[cs/catch-of-all-exceptions]
+            {
+                ++failedProjects;
+                progressMonitor.FailedProjectFile(project.FullName, ex.Message);
+            }
+
+        }
+
+        void Restore(string projectOrSolution)
+        {
+            int exit = DotNet.RestoreToDirectory(projectOrSolution, PackageDirectory.DirInfo.FullName);
+            switch(exit)
+            {
+                case 0:
+                case 1:
+                    // No errors
+                    break;
+                default:
+                    progressMonitor.CommandFailed("dotnet", $"restore \"{projectOrSolution}\"", exit);
+                    break;
             }
         }
 
-        /// <summary>
+        public void RestoreSolutions(IEnumerable<string> solutions)
-        /// Delete packages directory.
-        /// </summary>
-        public void Cleanup()
         {
-            if (nuget != null) nuget.Cleanup(progressMonitor);
+            Parallel.ForEach(solutions, new ParallelOptions { MaxDegreeOfParallelism = 4 }, Restore);
         }
 
-        /// <summary>
+        public void AnalyseSolutions(IEnumerable<string> solutions)
-        /// Analyse all project files in a given solution only.
-        /// </summary>
-        /// <param name="solutionFile">The filename of the solution.</param>
-        public void AnalyseSolution(string solutionFile)
         {
-            var sln = new SolutionFile(solutionFile);
+            Parallel.ForEach(solutions, new ParallelOptions { MaxDegreeOfParallelism = 4 } , solutionFile =>
-            AnalyseProjectFiles(sln.Projects.Select(p => new FileInfo(p)).ToArray());
+            {
+                try
+                {
+                    var sln = new SolutionFile(solutionFile);
+                    progressMonitor.AnalysingSolution(solutionFile);
+                    AnalyseProjectFiles(sln.Projects.Select(p => new FileInfo(p)).Where(p => p.Exists));
+                }
+                catch (Microsoft.Build.Exceptions.InvalidProjectFileException ex)
+                {
+                    progressMonitor.FailedProjectFile(solutionFile, ex.BaseMessage);
+                }
+            });
+        }
+
+        public void Dispose()
+        {
+            PackageDirectory?.Dispose();
         }
     }
 }

csharp/extractor/Semmle.Extraction.CSharp.Standalone/CsProjFile.cs

@@ -10,12 +10,18 @@ namespace Semmle.BuildAnalyser
     /// </summary>
     class CsProjFile
     {
+        private string Filename { get; }
+
+        private string Directory => Path.GetDirectoryName(Filename);
+
         /// <summary>
         /// Reads the .csproj file.
         /// </summary>
         /// <param name="filename">The .csproj file.</param>
         public CsProjFile(FileInfo filename)
         {
+            Filename = filename.FullName;
+
             try
             {
                 // This can fail if the .csproj is invalid or has
@@ -39,7 +45,7 @@ namespace Semmle.BuildAnalyser
         /// and there seems to be no way to make it succeed. Fails on Linux.
         /// </summary>
         /// <param name="filename">The file to read.</param>
-        public void ReadMsBuildProject(FileInfo filename)
+        private void ReadMsBuildProject(FileInfo filename)
         {
             var msbuildProject = new Microsoft.Build.Execution.ProjectInstance(filename.FullName);
 
@@ -62,7 +68,7 @@ namespace Semmle.BuildAnalyser
         /// fallback if ReadMsBuildProject() fails.
         /// </summary>
         /// <param name="filename">The .csproj file.</param>
-        public void ReadProjectFileAsXml(FileInfo filename)
+        private void ReadProjectFileAsXml(FileInfo filename)
         {
             var projFile = new XmlDocument();
             var mgr = new XmlNamespaceManager(projFile.NameTable);
@@ -71,22 +77,48 @@ namespace Semmle.BuildAnalyser
             var projDir = filename.Directory;
             var root = projFile.DocumentElement;
 
-            references =
+            // Figure out if it's dotnet core
-                root.SelectNodes("/msbuild:Project/msbuild:ItemGroup/msbuild:Reference/@Include", mgr).
-                NodeList().
-                Select(node => node.Value).
-                ToArray();
 
-            var relativeCsIncludes =
+            bool netCoreProjectFile = root.GetAttribute("Sdk") == "Microsoft.NET.Sdk";
-                root.SelectNodes("/msbuild:Project/msbuild:ItemGroup/msbuild:Compile/@Include", mgr).
-                NodeList().
-                Select(node => node.Value).
-                ToArray();
 
-            csFiles = relativeCsIncludes.
+            if (netCoreProjectFile)
-                Select(cs => Path.DirectorySeparatorChar == '/' ? cs.Replace("\\", "/") : cs).
+            {
-                Select(f => Path.GetFullPath(Path.Combine(projDir.FullName, f))).
+                var relativeCsIncludes =
-                ToArray();
+                    root.SelectNodes("/Project/ItemGroup/Compile/@Include", mgr).
+                    NodeList().
+                    Select(node => node.Value).
+                    ToArray();
+
+                var explicitCsFiles = relativeCsIncludes.
+                    Select(cs => Path.DirectorySeparatorChar == '/' ? cs.Replace("\\", "/") : cs).
+                    Select(f => Path.GetFullPath(Path.Combine(projDir.FullName, f)));
+
+                var additionalCsFiles = System.IO.Directory.GetFiles(Directory, "*.cs", SearchOption.AllDirectories);
+
+                csFiles = explicitCsFiles.Concat(additionalCsFiles).ToArray();
+
+                references = new string[0];
+            }
+            else
+            {
+
+                references =
+                    root.SelectNodes("/msbuild:Project/msbuild:ItemGroup/msbuild:Reference/@Include", mgr).
+                    NodeList().
+                    Select(node => node.Value).
+                    ToArray();
+
+                var relativeCsIncludes =
+                    root.SelectNodes("/msbuild:Project/msbuild:ItemGroup/msbuild:Compile/@Include", mgr).
+                    NodeList().
+                    Select(node => node.Value).
+                    ToArray();
+
+                csFiles = relativeCsIncludes.
+                    Select(cs => Path.DirectorySeparatorChar == '/' ? cs.Replace("\\", "/") : cs).
+                    Select(f => Path.GetFullPath(Path.Combine(projDir.FullName, f))).
+                    ToArray();
+            }
         }
 
         string[] references;

csharp/extractor/Semmle.Extraction.CSharp.Standalone/DotNet.cs

@@ -0,0 +1,21 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.IO;
+using System.Linq;
+
+namespace Semmle.BuildAnalyser
+{
+    /// <summary>
+    /// Utilities to run the "dotnet" command.
+    /// </summary>
+    static class DotNet
+    {
+        public static int RestoreToDirectory(string projectOrSolutionFile, string packageDirectory)
+        {
+            using var proc = Process.Start("dotnet", $"restore --no-dependencies \"{projectOrSolutionFile}\" --packages \"{packageDirectory}\" /p:DisableImplicitNuGetFallbackFolder=true");
+            proc.WaitForExit();
+            return proc.ExitCode;
+        }
+    }
+}

csharp/extractor/Semmle.Extraction.CSharp.Standalone/NugetPackages.cs

@@ -1,10 +1,9 @@
-using System;
+using Semmle.Util;
+using System;
 using System.Collections.Generic;
 using System.Diagnostics;
 using System.IO;
 using System.Linq;
-using System.Security.Cryptography;
-using System.Text;
 
 namespace Semmle.BuildAnalyser
 {
@@ -19,10 +18,10 @@ namespace Semmle.BuildAnalyser
         /// Create the package manager for a specified source tree.
         /// </summary>
         /// <param name="sourceDir">The source directory.</param>
-        public NugetPackages(string sourceDir)
+        public NugetPackages(string sourceDir, TemporaryDirectory packageDirectory)
         {
             SourceDirectory = sourceDir;
-            PackageDirectory = computeTempDirectory(sourceDir);
+            PackageDirectory = packageDirectory;
 
             // Expect nuget.exe to be in a `nuget` directory under the directory containing this exe.
             var currentAssembly = System.Reflection.Assembly.GetExecutingAssembly().Location;
@@ -50,45 +49,12 @@ namespace Semmle.BuildAnalyser
         /// </summary>
         public IEnumerable<FileInfo> PackageFiles => packages;
 
-        // Whether to delete the packages directory prior to each run.
-        // Makes each build more reproducible.
-        const bool cleanupPackages = true;
-
-        public void Cleanup(IProgressMonitor pm)
-        {
-            var packagesDirectory = new DirectoryInfo(PackageDirectory);
-
-            if (packagesDirectory.Exists)
-            {
-                try
-                {
-                    packagesDirectory.Delete(true);
-                }
-                catch (System.IO.IOException ex)
-                {
-                    pm.Warning(string.Format("Couldn't delete package directory - it's probably held open by something else: {0}", ex.Message));
-                }
-            }
-        }
-
         /// <summary>
         /// Download the packages to the temp folder.
         /// </summary>
         /// <param name="pm">The progress monitor used for reporting errors etc.</param>
         public void InstallPackages(IProgressMonitor pm)
         {
-            if (cleanupPackages)
-            {
-                Cleanup(pm);
-            }
-
-            var packagesDirectory = new DirectoryInfo(PackageDirectory);
-
-            if (!Directory.Exists(PackageDirectory))
-            {
-                packagesDirectory.Create();
-            }
-
             foreach (var package in packages)
             {
                 RestoreNugetPackage(package.FullName, pm);
@@ -109,31 +75,7 @@ namespace Semmle.BuildAnalyser
         /// This will be in the Temp location
         /// so as to not trample the source tree.
         /// </summary>
-        public string PackageDirectory
+        public TemporaryDirectory PackageDirectory { get; }
-        {
-            get;
-            private set;
-        }
-
-        readonly SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider();
-
-        /// <summary>
-        /// Computes a unique temp directory for the packages associated
-        /// with this source tree. Use a SHA1 of the directory name.
-        /// </summary>
-        /// <param name="srcDir"></param>
-        /// <returns>The full path of the temp directory.</returns>
-        string computeTempDirectory(string srcDir)
-        {
-            var bytes = Encoding.Unicode.GetBytes(srcDir);
-
-            var sha = sha1.ComputeHash(bytes);
-            var sb = new StringBuilder();
-            foreach (var b in sha.Take(8))
-                sb.AppendFormat("{0:x2}", b);
-
-            return Path.Combine(Path.GetTempPath(), "Semmle", "packages", sb.ToString());
-        }
 
         /// <summary>
         /// Restore all files in a specified package.
@@ -171,16 +113,15 @@ namespace Semmle.BuildAnalyser
 
             try
             {
-                using (var p = Process.Start(pi))
+                using var p = Process.Start(pi);
-                {
-                    string output = p.StandardOutput.ReadToEnd();
-                    string error = p.StandardError.ReadToEnd();
 
-                    p.WaitForExit();
+                string output = p.StandardOutput.ReadToEnd();
-                    if (p.ExitCode != 0)
+                string error = p.StandardError.ReadToEnd();
-                    {
+
-                        pm.FailedNugetCommand(pi.FileName, pi.Arguments, output + error);
+                p.WaitForExit();
-                    }
+                if (p.ExitCode != 0)
+                {
+                    pm.FailedNugetCommand(pi.FileName, pi.Arguments, output + error);
                 }
             }
             catch (Exception ex)

csharp/extractor/Semmle.Extraction.CSharp.Standalone/Program.cs

@@ -23,7 +23,7 @@ namespace Semmle.Extraction.CSharp.Standalone
     /// <summary>
     ///     Searches for source/references and creates separate extractions.
     /// </summary>
-    class Analysis
+    class Analysis : IDisposable
     {
         readonly ILogger logger;
 
@@ -71,12 +71,9 @@ namespace Semmle.Extraction.CSharp.Standalone
             projectExtraction.Sources.AddRange(options.SolutionFile == null ? buildAnalysis.AllSourceFiles : buildAnalysis.ProjectSourceFiles);
         }
 
-        /// <summary>
+        public void Dispose()
-        /// Delete any Nuget assemblies.
-        /// </summary>
-        public void Cleanup()
         {
-            buildAnalysis.Cleanup();
+            buildAnalysis.Dispose();
         }
     };
 
@@ -85,8 +82,9 @@ namespace Semmle.Extraction.CSharp.Standalone
         static int Main(string[] args)
         {
             var options = Options.Create(args);
+            // options.CIL = true;  // To do: Enable this
             var output = new ConsoleLogger(options.Verbosity);
-            var a = new Analysis(output);
+            using var a = new Analysis(output);
 
             if (options.Help)
             {
@@ -97,6 +95,8 @@ namespace Semmle.Extraction.CSharp.Standalone
             if (options.Errors)
                 return 1;
 
+            var start = DateTime.Now;
+
             output.Log(Severity.Info, "Running C# standalone extractor");
             a.AnalyseProjects(options);
             int sourceFiles = a.Extraction.Sources.Count();
@@ -117,10 +117,9 @@ namespace Semmle.Extraction.CSharp.Standalone
                     new ExtractionProgress(output),
                     new FileLogger(options.Verbosity, Extractor.GetCSharpLogPath()),
                     options);
-                output.Log(Severity.Info, "Extraction complete");
+                output.Log(Severity.Info, $"Extraction completed in {DateTime.Now-start}");
             }
 
-            a.Cleanup();
             return 0;
         }
 
@@ -151,7 +150,7 @@ namespace Semmle.Extraction.CSharp.Standalone
 
             public void MissingSummary(int missingTypes, int missingNamespaces)
             {
-                logger.Log(Severity.Info, "Failed to resolve {0} types and {1} namespaces", missingTypes, missingNamespaces);
+                logger.Log(Severity.Info, "Failed to resolve {0} types in {1} namespaces", missingTypes, missingNamespaces);
             }
         }
     }

csharp/extractor/Semmle.Extraction.CSharp.Standalone/ProgressMonitor.cs

@@ -1,4 +1,5 @@
 using Semmle.Util.Logging;
+using System;
 
 namespace Semmle.BuildAnalyser
 {
@@ -9,15 +10,17 @@ namespace Semmle.BuildAnalyser
     {
         void FindingFiles(string dir);
         void UnresolvedReference(string id, string project);
-        void AnalysingProjectFiles(int count);
+        void AnalysingSolution(string filename);
         void FailedProjectFile(string filename, string reason);
         void FailedNugetCommand(string exe, string args, string message);
         void NugetInstall(string package);
         void ResolvedReference(string filename);
-        void Summary(int existingSources, int usedSources, int missingSources, int references, int unresolvedReferences, int resolvedConflicts, int totalProjects, int failedProjects);
+        void Summary(int existingSources, int usedSources, int missingSources, int references, int unresolvedReferences, int resolvedConflicts, int totalProjects, int failedProjects, TimeSpan analysisTime);
         void Warning(string message);
         void ResolvedConflict(string asm1, string asm2);
         void MissingProject(string projectFile);
+        void CommandFailed(string exe, string arguments, int exitCode);
+        void MissingNuGet();
     }
 
     class ProgressMonitor : IProgressMonitor
@@ -46,9 +49,9 @@ namespace Semmle.BuildAnalyser
             logger.Log(Severity.Debug, "Unresolved {0} referenced by {1}", id, project);
         }
 
-        public void AnalysingProjectFiles(int count)
+        public void AnalysingSolution(string filename)
         {
-            logger.Log(Severity.Info, "Analyzing project files...");
+            logger.Log(Severity.Info, $"Analyzing {filename}...");
         }
 
         public void FailedProjectFile(string filename, string reason)
@@ -73,7 +76,9 @@ namespace Semmle.BuildAnalyser
         }
 
         public void Summary(int existingSources, int usedSources, int missingSources,
-            int references, int unresolvedReferences, int resolvedConflicts, int totalProjects, int failedProjects)
+            int references, int unresolvedReferences,
+            int resolvedConflicts, int totalProjects, int failedProjects,
+            TimeSpan analysisTime)
         {
             logger.Log(Severity.Info, "");
             logger.Log(Severity.Info, "Build analysis summary:");
@@ -85,6 +90,7 @@ namespace Semmle.BuildAnalyser
             logger.Log(Severity.Info, "{0, 6} resolved assembly conflicts", resolvedConflicts);
             logger.Log(Severity.Info, "{0, 6} projects", totalProjects);
             logger.Log(Severity.Info, "{0, 6} missing/failed projects", failedProjects);
+            logger.Log(Severity.Info, "Build analysis completed in {0}", analysisTime);
         }
 
         public void Warning(string message)
@@ -94,12 +100,22 @@ namespace Semmle.BuildAnalyser
 
         public void ResolvedConflict(string asm1, string asm2)
         {
-            logger.Log(Severity.Info, "Resolved {0} as {1}", asm1, asm2);
+            logger.Log(Severity.Debug, "Resolved {0} as {1}", asm1, asm2);
         }
 
         public void MissingProject(string projectFile)
         {
             logger.Log(Severity.Info, "Solution is missing {0}", projectFile);
         }
+
+        public void CommandFailed(string exe, string arguments, int exitCode)
+        {
+            logger.Log(Severity.Error, $"Command {exe} {arguments} failed with exit code {exitCode}");
+        }
+
+        public void MissingNuGet()
+        {
+            logger.Log(Severity.Error, "Missing nuget.exe");
+        }
     }
 }

csharp/extractor/Semmle.Extraction.CSharp.Standalone/Semmle.Extraction.CSharp.Standalone.csproj

@@ -1,4 +1,4 @@
-<Project Sdk="Microsoft.NET.Sdk">
+<Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
@@ -14,6 +14,7 @@
 
   <ItemGroup>
     <ProjectReference Include="..\Semmle.Extraction.CSharp\Semmle.Extraction.CSharp.csproj" />
+    <ProjectReference Include="..\Semmle.Util\Semmle.Util.csproj" />
   </ItemGroup>
 
   <ItemGroup>

csharp/extractor/Semmle.Extraction.CSharp.Standalone/SolutionFile.cs

@@ -12,6 +12,8 @@ namespace Semmle.BuildAnalyser
     {
         readonly Microsoft.Build.Construction.SolutionFile solutionFile;
 
+        private string FullPath { get; }
+
         /// <summary>
         /// Read the file.
         /// </summary>
@@ -19,8 +21,8 @@ namespace Semmle.BuildAnalyser
         public SolutionFile(string filename)
         {
             // SolutionFile.Parse() expects a rooted path.
-            var fullPath = Path.GetFullPath(filename);
+            FullPath = Path.GetFullPath(filename);
-            solutionFile = Microsoft.Build.Construction.SolutionFile.Parse(fullPath);
+            solutionFile = Microsoft.Build.Construction.SolutionFile.Parse(FullPath);
         }
 
         /// <summary>

csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Access.cs

@@ -45,7 +45,10 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
         Access(ExpressionNodeInfo info, ISymbol symbol, bool implicitThis, IEntity target)
             : base(info.SetKind(AccessKind(info.Context, symbol)))
         {
-            cx.TrapWriter.Writer.expr_access(this, target);
+            if (!(target is null))
+            {
+                cx.TrapWriter.Writer.expr_access(this, target);
+            }
 
             if (implicitThis && !symbol.IsStatic)
             {

csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/MemberAccess.cs

@@ -71,7 +71,9 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
             if (symbol == null)
             {
                 info.Context.ModelError(info.Node, "Failed to determine symbol for member access");
-                return new MemberAccess(info.SetKind(ExprKind.UNKNOWN), expression, symbol);
+                // Default to property access - this can still give useful results but
+                // the target of the expression should be checked in QL.
+                return new MemberAccess(info.SetKind(ExprKind.PROPERTY_ACCESS), expression, symbol);
             }
 
             ExprKind kind;

csharp/extractor/Semmle.Extraction.CSharp/Entities/NamespaceDeclaration.cs

@@ -23,7 +23,8 @@ namespace Semmle.Extraction.CSharp.Entities
 
         protected override void Populate(TextWriter trapFile)
         {
-            var ns = Namespace.Create(cx, (INamespaceSymbol)cx.GetModel(Node).GetSymbolInfo(Node.Name).Symbol);
+            var @namespace = (INamespaceSymbol) cx.GetModel(Node).GetSymbolInfo(Node.Name).Symbol;
+            var ns = Namespace.Create(cx, @namespace);
             trapFile.namespace_declarations(this, ns);
             trapFile.namespace_declaration_location(this, cx.Create(Node.Name.GetLocation()));
 

csharp/extractor/Semmle.Extraction.CSharp/Entities/Types/NamedType.cs

@@ -25,7 +25,7 @@ namespace Semmle.Extraction.CSharp.Entities
         {
             if (symbol.TypeKind == TypeKind.Error)
             {
-                Context.Extractor.MissingType(symbol.ToString());
+                Context.Extractor.MissingType(symbol.ToString(), Context.FromSource);
                 return;
             }
 

csharp/extractor/Semmle.Extraction.CSharp/Entities/UsingDirective.cs

@@ -32,7 +32,7 @@ namespace Semmle.Extraction.CSharp.Entities
 
                 if (namespaceSymbol == null)
                 {
-                    cx.Extractor.MissingNamespace(Node.Name.ToFullString());
+                    cx.Extractor.MissingNamespace(Node.Name.ToFullString(), cx.FromSource);
                     cx.ModelError(Node, "Namespace not found");
                     return;
                 }

csharp/extractor/Semmle.Extraction.CSharp/SymbolExtensions.cs

@@ -214,7 +214,6 @@ namespace Semmle.Extraction.CSharp
         static void BuildNamedTypeId(this INamedTypeSymbol named, Context cx, TextWriter trapFile, Action<Context, TextWriter, ITypeSymbol> subTermAction)
         {
             bool prefixAssembly = true;
-            if (cx.Extractor.Standalone) prefixAssembly = false;
             if (named.ContainingAssembly is null) prefixAssembly = false;
 
             if (named.IsTupleType)

csharp/extractor/Semmle.Extraction/Context.cs

@@ -155,7 +155,7 @@ namespace Semmle.Extraction
 #if DEBUG_LABELS
                 using (var id = new StringWriter())
                 {
-                    entity.WriteId(id);
+                    entity.WriteQuotedId(id);
                     CheckEntityHasUniqueLabel(id.ToString(), entity);
                 }
 #endif
@@ -270,6 +270,8 @@ namespace Semmle.Extraction
             TrapWriter = trapWriter;
         }
 
+        public bool FromSource => Scope.FromSource;
+
         public bool IsGlobalContext => Scope.IsGlobalScope;
 
         public readonly ICommentGenerator CommentGenerator = new CommentProcessor();

csharp/extractor/Semmle.Extraction/ExtractionScope.cs

@@ -25,6 +25,8 @@ namespace Semmle.Extraction
         bool InFileScope(string path);
 
         bool IsGlobalScope { get; }
+
+        bool FromSource { get; }
     }
 
     /// <summary>
@@ -49,6 +51,8 @@ namespace Semmle.Extraction
         public bool InScope(ISymbol symbol) =>
             SymbolEqualityComparer.Default.Equals(symbol.ContainingAssembly, assembly) ||
             SymbolEqualityComparer.Default.Equals(symbol, assembly);
+
+        public bool FromSource => false;
     }
 
     /// <summary>
@@ -68,5 +72,7 @@ namespace Semmle.Extraction
         public bool InFileScope(string path) => path == sourceTree.FilePath;
 
         public bool InScope(ISymbol symbol) => symbol.Locations.Any(loc => loc.SourceTree == sourceTree);
+
+        public bool FromSource => true;
     }
 }

csharp/extractor/Semmle.Extraction/Extractor.cs

@@ -50,13 +50,15 @@ namespace Semmle.Extraction
         /// Record a new error type.
         /// </summary>
         /// <param name="fqn">The display name of the type, qualified where possible.</param>
-        void MissingType(string fqn);
+        /// <param name="fromSource">If the missing type was referenced from a source file.</param>
+        void MissingType(string fqn, bool fromSource);
 
         /// <summary>
         /// Record an unresolved `using namespace` directive.
         /// </summary>
         /// <param name="fqn">The full name of the namespace.</param>
-        void MissingNamespace(string fqn);
+        /// <param name="fromSource">If the missing namespace was referenced from a source file.</param>
+        void MissingNamespace(string fqn, bool fromSource);
 
         /// <summary>
         /// The list of missing types.
@@ -167,16 +169,22 @@ namespace Semmle.Extraction
         readonly ISet<string> missingTypes = new SortedSet<string>();
         readonly ISet<string> missingNamespaces = new SortedSet<string>();
 
-        public void MissingType(string fqn)
+        public void MissingType(string fqn, bool fromSource)
         {
-            lock (mutex)
+            if (fromSource)
-                missingTypes.Add(fqn);
+            {
+                lock (mutex)
+                    missingTypes.Add(fqn);
+            }
         }
 
-        public void MissingNamespace(string fqdn)
+        public void MissingNamespace(string fqdn, bool fromSource)
         {
-            lock (mutex)
+            if (fromSource)
-                missingNamespaces.Add(fqdn);
+            {
+                lock (mutex)
+                    missingNamespaces.Add(fqdn);
+            }
         }
 
         public Context CreateContext(Compilation c, TrapWriter trapWriter, IExtractionScope scope)

csharp/extractor/Semmle.Util.Tests/Semmle.Util.Tests.csproj

@@ -5,6 +5,7 @@
     <TargetFramework>netcoreapp3.0</TargetFramework>
     <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
     <RuntimeIdentifiers>win-x64;linux-x64;osx-x64</RuntimeIdentifiers>
+    <Nullable>enable</Nullable>
   </PropertyGroup>
 
   <ItemGroup>

csharp/extractor/Semmle.Util/ActionMap.cs

@@ -9,20 +9,19 @@ namespace Semmle.Util
     /// </summary>
     /// <typeparam name="Key"></typeparam>
     /// <typeparam name="Value"></typeparam>
-    public class ActionMap<Key, Value>
+    public class ActionMap<Key, Value> where Key : notnull
     {
         public void Add(Key key, Value value)
         {
-            Action<Value> a;
+
-            if (actions.TryGetValue(key, out a))
+            if (actions.TryGetValue(key, out var a))
                 a(value);
             values[key] = value;
         }
 
         public void OnAdd(Key key, Action<Value> action)
         {
-            Action<Value> a;
+            if (actions.TryGetValue(key, out var a))
-            if (actions.TryGetValue(key, out a))
             {
                 actions[key] = a + action;
             }

csharp/extractor/Semmle.Util/CanonicalPathCache.cs

@@ -127,8 +127,8 @@ namespace Semmle.Util
 
             if (parent != null)
             {
-                string name = Path.GetFileName(path);
+                var name = Path.GetFileName(path);
-                string parentPath = cache.GetCanonicalPath(parent.FullName);
+                var parentPath = cache.GetCanonicalPath(parent.FullName);
                 try
                 {
                     string[] entries = Directory.GetFileSystemEntries(parentPath, name);
@@ -313,14 +313,15 @@ namespace Semmle.Util
         /// <returns>The canonical path.</returns>
         public string GetCanonicalPath(string path)
         {
-            string canonicalPath;
             lock (cache)
-                if (!cache.TryGetValue(path, out canonicalPath))
+            {
+                if (!cache.TryGetValue(path, out var canonicalPath))
                 {
                     canonicalPath = pathStrategy.GetCanonicalPath(path, this);
                     AddToCache(path, canonicalPath);
                 }
-            return canonicalPath;
+                return canonicalPath;
+            }
         }
     }
 }

csharp/extractor/Semmle.Util/CommandLineExtensions.cs

@@ -18,7 +18,7 @@ namespace Semmle.Util
             var found = false;
             foreach (var arg in commandLineArguments.Where(arg => arg.StartsWith('@')).Select(arg => arg.Substring(1)))
             {
-                string line;
+                string? line;
                 using (StreamReader file = new StreamReader(arg))
                     while ((line = file.ReadLine()) != null)
                         textWriter.WriteLine(line);

csharp/extractor/Semmle.Util/DictionaryExtensions.cs

@@ -9,10 +9,9 @@ namespace Semmle.Util
         /// dictionary. If a list does not already exist, a new list is
         /// created.
         /// </summary>
-        public static void AddAnother<T1, T2>(this Dictionary<T1, List<T2>> dict, T1 key, T2 element)
+        public static void AddAnother<T1, T2>(this Dictionary<T1, List<T2>> dict, T1 key, T2 element) where T1:notnull
         {
-            List<T2> list;
+            if (!dict.TryGetValue(key, out var list))
-            if (!dict.TryGetValue(key, out list))
             {
                 list = new List<T2>();
                 dict[key] = list;

csharp/extractor/Semmle.Util/FileRenamer.cs

@@ -0,0 +1,34 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+
+namespace Semmle.Util
+{
+    /// <summary>
+    /// Utility to temporarily rename a set of files.
+    /// </summary>
+    public sealed class FileRenamer : IDisposable
+    {
+        readonly string[] files;
+        const string suffix = ".codeqlhidden";
+
+        public FileRenamer(IEnumerable<FileInfo> oldFiles)
+        {
+            files = oldFiles.Select(f => f.FullName).ToArray();
+
+            foreach (var file in files)
+            {
+                File.Move(file, file + suffix);
+            }
+        }
+
+        public void Dispose()
+        {
+            foreach (var file in files)
+            {
+                File.Move(file + suffix, file);
+            }
+        }
+    }
+}

csharp/extractor/Semmle.Util/FileUtils.cs

@@ -62,7 +62,7 @@ namespace Semmle.Util
         ///
         /// Returns <code>null</code> of no path can be found.
         /// </summary>
-        public static string FindProgramOnPath(string prog)
+        public static string? FindProgramOnPath(string prog)
         {
             var paths = Environment.GetEnvironmentVariable("PATH")?.Split(Path.PathSeparator);
             string[] exes;

csharp/extractor/Semmle.Util/FuzzyDictionary.cs

@@ -37,7 +37,7 @@ namespace Semmle.Util
     /// </remarks>
     ///
     /// <typeparam name="T">The value type.</typeparam>
-    public class FuzzyDictionary<T>
+    public class FuzzyDictionary<T> where T:class
     {
         // All data items indexed by the "base string" (stripped of numbers)
         readonly Dictionary<string, List<KeyValuePair<string, T>>> index = new Dictionary<string, List<KeyValuePair<string, T>>>();
@@ -61,7 +61,7 @@ namespace Semmle.Util
         /// <param name="v1">Vector 1</param>
         /// <param name="v2">Vector 2</param>
         /// <returns>The Hamming Distance.</returns>
-        static int HammingDistance<U>(IEnumerable<U> v1, IEnumerable<U> v2)
+        static int HammingDistance<U>(IEnumerable<U> v1, IEnumerable<U> v2) where U: notnull
         {
             return v1.Zip(v2, (x, y) => x.Equals(y) ? 0 : 1).Sum();
         }
@@ -72,11 +72,10 @@ namespace Semmle.Util
         /// <param name="query">The query string.</param>
         /// <param name="distance">The distance between the query string and the stored string.</param>
         /// <returns>The best match, or null (default).</returns>
-        public T FindMatch(string query, out int distance)
+        public T? FindMatch(string query, out int distance)
         {
             string root = StripDigits(query);
-            List<KeyValuePair<string, T>> list;
+            if (!index.TryGetValue(root, out var list))
-            if (!index.TryGetValue(root, out list))
             {
                 distance = 0;
                 return default(T);
@@ -93,9 +92,9 @@ namespace Semmle.Util
         /// <param name="distance">The distance function.</param>
         /// <param name="bestDistance">The distance between the query and the stored string.</param>
         /// <returns>The stored value.</returns>
-        static T BestMatch(string query, IEnumerable<KeyValuePair<string, T>> candidates, Func<string, string, int> distance, out int bestDistance)
+        static T? BestMatch(string query, IEnumerable<KeyValuePair<string, T>> candidates, Func<string, string, int> distance, out int bestDistance)
         {
-            T bestMatch = default(T);
+            T? bestMatch = default(T);
             bestDistance = 0;
             bool first = true;
 

csharp/extractor/Semmle.Util/IEnumerableExtensions.cs

@@ -93,7 +93,7 @@ namespace Semmle.Util
         /// <typeparam name="T">The type of the item.</typeparam>
         /// <param name="items">The list of items to hash.</param>
         /// <returns>The hash code.</returns>
-        public static int SequenceHash<T>(this IEnumerable<T> items)
+        public static int SequenceHash<T>(this IEnumerable<T> items) where T: notnull
         {
             int h = 0;
             foreach (var i in items)

csharp/extractor/Semmle.Util/LineCounter.cs

@@ -31,9 +31,9 @@ namespace Semmle.Util
         //#################### PUBLIC METHODS ####################
         #region
 
-        public override bool Equals(Object other)
+        public override bool Equals(object? other)
         {
-            LineCounts rhs = other as LineCounts;
+            var rhs = other as LineCounts;
             return rhs != null && Total == rhs.Total && Code == rhs.Code && Comment == rhs.Comment;
         }
 

csharp/extractor/Semmle.Util/Logger.cs

@@ -67,8 +67,8 @@ namespace Semmle.Util.Logging
 
             try
             {
-                var dir = Path.GetDirectoryName(outputFile);
+                string? dir = Path.GetDirectoryName(outputFile);
-                if (dir.Length > 0 && !System.IO.Directory.Exists(dir))
+                if (!string.IsNullOrEmpty(dir) && !System.IO.Directory.Exists(dir))
                     Directory.CreateDirectory(dir);
                 writer = new PidStreamWriter(new FileStream(outputFile, FileMode.Append, FileAccess.Write,
                     FileShare.ReadWrite, 8192));

csharp/extractor/Semmle.Util/LoggerUtils.cs

@@ -21,7 +21,7 @@ namespace Semmle.Util
 
         private readonly string prefix = "[" + Process.GetCurrentProcess().Id + "] ";
 
-        public override void WriteLine(string value)
+        public override void WriteLine(string? value)
         {
             lock (mutex)
             {
@@ -29,9 +29,9 @@ namespace Semmle.Util
             }
         }
 
-        public override void WriteLine(string value, object[] args)
+        public override void WriteLine(string? value, object?[] args)
         {
-            WriteLine(String.Format(value, args));
+            WriteLine(value is null ? value : String.Format(value, args));
         }
 
         readonly object mutex = new object();

csharp/extractor/Semmle.Util/ProcessStartInfoExtensions.cs

@@ -14,7 +14,7 @@ namespace Semmle.Util
             stdout = new List<string>();
             using (var process = Process.Start(pi))
             {
-                string s;
+                string? s;
                 do
                 {
                     s = process.StandardOutput.ReadLine();

csharp/extractor/Semmle.Util/Semmle.Util.csproj

@@ -6,6 +6,7 @@
     <RootNamespace>Semmle.Util</RootNamespace>
     <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
     <RuntimeIdentifiers>win-x64;linux-x64;osx-x64</RuntimeIdentifiers>
+    <Nullable>enable</Nullable>
   </PropertyGroup>
 
   <ItemGroup>

csharp/extractor/Semmle.Util/SharedReference.cs

@@ -13,6 +13,6 @@ namespace Semmle.Util
         /// <summary>
         /// The shared object to which different parts of the code want to refer.
         /// </summary>
-        public T Obj { get; set; }
+        public T? Obj { get; set; }
     }
 }

csharp/extractor/Semmle.Util/TemporaryDirectory.cs

@@ -0,0 +1,30 @@
+using System;
+using System.IO;
+using System.Linq;
+using System.Security.Cryptography;
+using System.Text;
+
+namespace Semmle.Util
+{
+    /// <summary>
+    /// A temporary directory that is created within the system temp directory.
+    /// When this object is disposed, the directory is deleted.
+    /// </summary>
+    public sealed class TemporaryDirectory : IDisposable
+    {
+        public DirectoryInfo DirInfo { get; }
+
+        public TemporaryDirectory(string name)
+        {
+            DirInfo = new DirectoryInfo(name);
+            DirInfo.Create();
+        }
+
+        public void Dispose()
+        {
+            DirInfo.Delete(true);
+        }
+
+        public override string ToString() => DirInfo.FullName.ToString();
+    }
+}

csharp/ql/src/Security Features/CWE-091/XMLInjection.ql

@@ -11,7 +11,7 @@
  */
 
 import csharp
-import semmle.code.csharp.dataflow.flowsources.Remote
+import semmle.code.csharp.security.dataflow.flowsources.Remote
 import semmle.code.csharp.frameworks.system.Xml
 
 /**

csharp/ql/src/Security Features/CWE-114/AssemblyPathInjection.ql

@@ -12,7 +12,7 @@
  */
 
 import csharp
-import semmle.code.csharp.dataflow.flowsources.Remote
+import semmle.code.csharp.security.dataflow.flowsources.Remote
 import semmle.code.csharp.commons.Util
 
 /**

csharp/ql/src/Security Features/CWE-134/UncontrolledFormatString.ql

@@ -11,8 +11,8 @@
  */
 
 import csharp
-import semmle.code.csharp.dataflow.flowsources.Remote
+import semmle.code.csharp.security.dataflow.flowsources.Remote
-import semmle.code.csharp.dataflow.flowsources.Local
+import semmle.code.csharp.security.dataflow.flowsources.Local
 import semmle.code.csharp.dataflow.TaintTracking
 import semmle.code.csharp.frameworks.Format
 import DataFlow::PathGraph

csharp/ql/src/Security Features/CWE-201/ExposureInTransmittedData.ql

@@ -11,8 +11,7 @@
 
 import csharp
 import semmle.code.csharp.security.SensitiveActions
-import semmle.code.csharp.security.dataflow.XSS
+import semmle.code.csharp.security.dataflow.flowsinks.Remote
-import semmle.code.csharp.security.dataflow.Email
 import semmle.code.csharp.frameworks.system.data.Common
 import semmle.code.csharp.frameworks.System
 import semmle.code.csharp.dataflow.DataFlow::DataFlow::PathGraph
@@ -42,11 +41,7 @@ class TaintTrackingConfiguration extends TaintTracking::Configuration {
     )
   }
 
-  override predicate isSink(DataFlow::Node sink) {
+  override predicate isSink(DataFlow::Node sink) { sink instanceof RemoteFlowSink }
-    sink instanceof XSS::Sink
-    or
-    sink instanceof Email::Sink
-  }
 }
 
 from TaintTrackingConfiguration configuration, DataFlow::PathNode source, DataFlow::PathNode sink

csharp/ql/src/Security Features/CWE-209/ExceptionInformationExposure.ql

@@ -14,7 +14,7 @@
 
 import csharp
 import semmle.code.csharp.frameworks.System
-import semmle.code.csharp.security.dataflow.XSS
+import semmle.code.csharp.security.dataflow.flowsinks.Remote
 import semmle.code.csharp.dataflow.DataFlow::DataFlow::PathGraph
 
 /**
@@ -46,7 +46,7 @@ class TaintTrackingConfiguration extends TaintTracking::Configuration {
     )
   }
 
-  override predicate isSink(DataFlow::Node sink) { sink instanceof XSS::Sink }
+  override predicate isSink(DataFlow::Node sink) { sink instanceof RemoteFlowSink }
 
   override predicate isSanitizer(DataFlow::Node sanitizer) {
     // Do not flow through Message

csharp/ql/src/Security Features/CWE-838/InappropriateEncoding.ql

@@ -16,7 +16,7 @@ import semmle.code.csharp.frameworks.system.Net
 import semmle.code.csharp.frameworks.system.Web
 import semmle.code.csharp.frameworks.system.web.UI
 import semmle.code.csharp.security.dataflow.SqlInjection
-import semmle.code.csharp.security.dataflow.XSS
+import semmle.code.csharp.security.dataflow.flowsinks.Html
 import semmle.code.csharp.security.dataflow.UrlRedirect
 import semmle.code.csharp.security.Sanitizers
 import semmle.code.csharp.dataflow.DataFlow2::DataFlow2
@@ -114,7 +114,7 @@ module EncodingConfigurations {
 
     override string getKind() { result = "HTML expression" }
 
-    override predicate requiresEncoding(Node n) { n instanceof XSS::HtmlSink }
+    override predicate requiresEncoding(Node n) { n instanceof HtmlSink }
 
     override predicate isPossibleEncodedValue(Expr e) { e instanceof HtmlSanitizedExpr }
   }

csharp/ql/src/Security Features/InsecureRandomness.ql

@@ -16,7 +16,7 @@ import semmle.code.csharp.frameworks.Test
 import semmle.code.csharp.dataflow.DataFlow::DataFlow::PathGraph
 
 module Random {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.security.SensitiveActions
 
   /**

csharp/ql/src/codeql-suites/csharp-code-scanning.qls

@@ -0,0 +1,4 @@
+- description: Standard Code Scanning queries for C#
+- qlpack: codeql-csharp
+- apply: code-scanning-selectors.yml
+  from: codeql-suite-helpers

csharp/ql/src/semmle/code/csharp/dataflow/flowsources/PublicCallableParameter.qll

@@ -1,9 +1,10 @@
 /**
+ * DEPRECATED.
+ *
  * Provides classes representing data flow sources for parameters of public callables.
  */
 
 import csharp
-private import semmle.code.csharp.frameworks.WCF
 
 /**
  * A parameter of a public callable, for example `p` in

csharp/ql/src/semmle/code/csharp/dataflow/flowsources/Remote.qll

@@ -1,218 +1,7 @@
 /**
- * Provides classes representing data flow sources for remote user input.
+ * DEPRECATED.
+ *
+ * Use `semmle.code.csharp.security.dataflow.flowsources.Remote` instead.
  */
 
-import csharp
+import semmle.code.csharp.security.dataflow.flowsources.Remote
-private import semmle.code.csharp.frameworks.system.Net
-private import semmle.code.csharp.frameworks.system.Web
-private import semmle.code.csharp.frameworks.system.web.Http
-private import semmle.code.csharp.frameworks.system.web.Mvc
-private import semmle.code.csharp.frameworks.system.web.Services
-private import semmle.code.csharp.frameworks.system.web.ui.WebControls
-private import semmle.code.csharp.frameworks.WCF
-private import semmle.code.csharp.frameworks.microsoft.Owin
-private import semmle.code.csharp.frameworks.microsoft.AspNetCore
-
-/** A data flow source of remote user input. */
-abstract class RemoteFlowSource extends DataFlow::Node {
-  /** Gets a string that describes the type of this remote flow source. */
-  abstract string getSourceType();
-}
-
-/** A data flow source of remote user input (ASP.NET). */
-abstract class AspNetRemoteFlowSource extends RemoteFlowSource { }
-
-/** A member containing an ASP.NET query string. */
-class AspNetQueryStringMember extends Member {
-  AspNetQueryStringMember() {
-    exists(RefType t |
-      t instanceof SystemWebHttpRequestClass or
-      t instanceof SystemNetHttpListenerRequestClass or
-      t instanceof SystemWebHttpRequestBaseClass
-    |
-      this = t.getProperty(getHttpRequestFlowPropertyNames()) or
-      this.(Field).getType() = t or
-      this.(Property).getType() = t or
-      this.(Callable).getReturnType() = t
-    )
-  }
-}
-
-/**
- * Gets the names of the properties in `HttpRequest` classes that should propagate taint out of the
- * request.
- */
-private string getHttpRequestFlowPropertyNames() {
-  result = "QueryString" or
-  result = "Headers" or
-  result = "RawUrl" or
-  result = "Url" or
-  result = "Cookies" or
-  result = "Form" or
-  result = "Params" or
-  result = "Path" or
-  result = "PathInfo"
-}
-
-/** A data flow source of remote user input (ASP.NET query string). */
-class AspNetQueryStringRemoteFlowSource extends AspNetRemoteFlowSource, DataFlow::ExprNode {
-  AspNetQueryStringRemoteFlowSource() {
-    exists(RefType t |
-      t instanceof SystemWebHttpRequestClass or
-      t instanceof SystemNetHttpListenerRequestClass or
-      t instanceof SystemWebHttpRequestBaseClass
-    |
-      // A request object can be indexed, so taint the object as well
-      this.getExpr().getType() = t
-    )
-    or
-    this.getExpr() = any(AspNetQueryStringMember m).getAnAccess()
-  }
-
-  override string getSourceType() { result = "ASP.NET query string" }
-}
-
-/** A data flow source of remote user input (ASP.NET unvalidated request data). */
-class AspNetUnvalidatedQueryStringRemoteFlowSource extends AspNetRemoteFlowSource,
-  DataFlow::ExprNode {
-  AspNetUnvalidatedQueryStringRemoteFlowSource() {
-    this.getExpr() = any(SystemWebUnvalidatedRequestValues c).getAProperty().getGetter().getACall() or
-    this.getExpr() =
-      any(SystemWebUnvalidatedRequestValuesBase c).getAProperty().getGetter().getACall()
-  }
-
-  override string getSourceType() { result = "ASP.NET unvalidated request data" }
-}
-
-/** A data flow source of remote user input (ASP.NET user input). */
-class AspNetUserInputRemoteFlowSource extends AspNetRemoteFlowSource, DataFlow::ExprNode {
-  AspNetUserInputRemoteFlowSource() { getType() instanceof SystemWebUIWebControlsTextBoxClass }
-
-  override string getSourceType() { result = "ASP.NET user input" }
-}
-
-/** A data flow source of remote user input (WCF based web service). */
-class WcfRemoteFlowSource extends RemoteFlowSource, DataFlow::ParameterNode {
-  WcfRemoteFlowSource() { exists(OperationMethod om | om.getAParameter() = this.getParameter()) }
-
-  override string getSourceType() { result = "web service input" }
-}
-
-/** A data flow source of remote user input (ASP.NET web service). */
-class AspNetServiceRemoteFlowSource extends RemoteFlowSource, DataFlow::ParameterNode {
-  AspNetServiceRemoteFlowSource() {
-    exists(Method m |
-      m.getAParameter() = this.getParameter() and
-      m.getAnAttribute().getType() instanceof SystemWebServicesWebMethodAttributeClass
-    )
-  }
-
-  override string getSourceType() { result = "ASP.NET web service input" }
-}
-
-/** A data flow source of remote user input (ASP.NET request message). */
-class SystemNetHttpRequestMessageRemoteFlowSource extends RemoteFlowSource, DataFlow::ExprNode {
-  SystemNetHttpRequestMessageRemoteFlowSource() {
-    getType() instanceof SystemWebHttpRequestMessageClass
-  }
-
-  override string getSourceType() { result = "ASP.NET request message" }
-}
-
-/**
- * A data flow source of remote user input (Microsoft Owin, a query, request,
- * or path string).
- */
-class MicrosoftOwinStringFlowSource extends RemoteFlowSource, DataFlow::ExprNode {
-  MicrosoftOwinStringFlowSource() {
-    this.getExpr() = any(MicrosoftOwinString owinString).getValueProperty().getGetter().getACall()
-  }
-
-  override string getSourceType() { result = "Microsoft Owin request or query string" }
-}
-
-/** A data flow source of remote user input (`Microsoft Owin IOwinRequest`). */
-class MicrosoftOwinRequestRemoteFlowSource extends RemoteFlowSource, DataFlow::ExprNode {
-  MicrosoftOwinRequestRemoteFlowSource() {
-    exists(Property p, MicrosoftOwinIOwinRequestClass owinRequest |
-      this.getExpr() = p.getGetter().getACall()
-    |
-      p = owinRequest.getAcceptProperty() or
-      p = owinRequest.getBodyProperty() or
-      p = owinRequest.getCacheControlProperty() or
-      p = owinRequest.getContentTypeProperty() or
-      p = owinRequest.getContextProperty() or
-      p = owinRequest.getCookiesProperty() or
-      p = owinRequest.getHeadersProperty() or
-      p = owinRequest.getHostProperty() or
-      p = owinRequest.getMediaTypeProperty() or
-      p = owinRequest.getMethodProperty() or
-      p = owinRequest.getPathProperty() or
-      p = owinRequest.getPathBaseProperty() or
-      p = owinRequest.getQueryProperty() or
-      p = owinRequest.getQueryStringProperty() or
-      p = owinRequest.getRemoteIpAddressProperty() or
-      p = owinRequest.getSchemeProperty() or
-      p = owinRequest.getURIProperty()
-    )
-  }
-
-  override string getSourceType() { result = "Microsoft Owin request" }
-}
-
-/** A parameter to an Mvc controller action method, viewed as a source of remote user input. */
-class ActionMethodParameter extends RemoteFlowSource, DataFlow::ParameterNode {
-  ActionMethodParameter() {
-    exists(Parameter p |
-      p = this.getParameter() and
-      p.fromSource()
-    |
-      p = any(Controller c).getAnActionMethod().getAParameter() or
-      p = any(ApiController c).getAnActionMethod().getAParameter()
-    )
-  }
-
-  override string getSourceType() { result = "ASP.NET MVC action method parameter" }
-}
-
-/** A data flow source of remote user input (ASP.NET Core). */
-abstract class AspNetCoreRemoteFlowSource extends RemoteFlowSource { }
-
-/** A data flow source of remote user input (ASP.NET query collection). */
-class AspNetCoreQueryRemoteFlowSource extends AspNetCoreRemoteFlowSource, DataFlow::ExprNode {
-  AspNetCoreQueryRemoteFlowSource() {
-    exists(ValueOrRefType t |
-      t instanceof MicrosoftAspNetCoreHttpHttpRequest or
-      t instanceof MicrosoftAspNetCoreHttpQueryCollection or
-      t instanceof MicrosoftAspNetCoreHttpQueryString
-    |
-      this.getExpr().(Call).getTarget().getDeclaringType() = t or
-      this.asExpr().(Access).getTarget().getDeclaringType() = t
-    )
-    or
-    exists(Call c |
-      c
-          .getTarget()
-          .getDeclaringType()
-          .hasQualifiedName("Microsoft.AspNetCore.Http", "IQueryCollection") and
-      c.getTarget().getName() = "TryGetValue" and
-      this.asExpr() = c.getArgumentForName("value")
-    )
-  }
-
-  override string getSourceType() { result = "ASP.NET Core query string" }
-}
-
-/** A parameter to a `Mvc` controller action method, viewed as a source of remote user input. */
-class AspNetCoreActionMethodParameter extends RemoteFlowSource, DataFlow::ParameterNode {
-  AspNetCoreActionMethodParameter() {
-    exists(Parameter p |
-      p = this.getParameter() and
-      p.fromSource()
-    |
-      p = any(MicrosoftAspNetCoreMvcController c).getAnActionMethod().getAParameter()
-    )
-  }
-
-  override string getSourceType() { result = "ASP.NET Core MVC action method parameter" }
-}

csharp/ql/src/semmle/code/csharp/exprs/Access.qll

@@ -388,7 +388,12 @@ library class PropertyAccessExpr extends Expr, @property_access_expr {
   /** Gets the target of this property access. */
   Property getProperty() { expr_access(this, result) }
 
-  override string toString() { result = "access to property " + this.getProperty().getName() }
+  override string toString() {
+    result = "access to property " + this.getProperty().getName()
+    or
+    not exists(this.getProperty()) and
+    result = "access to property (unknown)"
+  }
 }
 
 /**

csharp/ql/src/semmle/code/csharp/security/dataflow/CleartextStorage.qll

@@ -5,10 +5,10 @@
 import csharp
 
 module CleartextStorage {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.Web
   import semmle.code.csharp.security.SensitiveActions
-  import semmle.code.csharp.security.sinks.ExternalLocationSink
+  import semmle.code.csharp.security.dataflow.flowsinks.ExternalLocationSink
 
   /**
    * A data flow source for cleartext storage of sensitive information.

csharp/ql/src/semmle/code/csharp/security/dataflow/CodeInjection.qll

@@ -5,8 +5,8 @@
 import csharp
 
 module CodeInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
-  import semmle.code.csharp.dataflow.flowsources.Local
+  import semmle.code.csharp.security.dataflow.flowsources.Local
   import semmle.code.csharp.frameworks.system.codedom.Compiler
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/CommandInjection.qll

@@ -5,7 +5,7 @@
 import csharp
 
 module CommandInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.Diagnostics
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/ConditionalBypass.qll

@@ -8,7 +8,7 @@ import csharp
 module UserControlledBypassOfSensitiveMethod {
   import semmle.code.csharp.controlflow.Guards
   import semmle.code.csharp.controlflow.BasicBlocks
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.System
   import semmle.code.csharp.frameworks.system.Net
   import semmle.code.csharp.security.SensitiveActions

csharp/ql/src/semmle/code/csharp/security/dataflow/ExposureOfPrivateInformation.qll

@@ -5,8 +5,8 @@
 import csharp
 
 module ExposureOfPrivateInformation {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
-  import semmle.code.csharp.security.sinks.ExternalLocationSink
+  import semmle.code.csharp.security.dataflow.flowsinks.ExternalLocationSink
   import semmle.code.csharp.security.PrivateData
 
   /**

csharp/ql/src/semmle/code/csharp/security/dataflow/LDAPInjection.qll

@@ -6,7 +6,7 @@
 import csharp
 
 module LDAPInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.DirectoryServices
   import semmle.code.csharp.frameworks.system.directoryservices.Protocols
   import semmle.code.csharp.security.Sanitizers

csharp/ql/src/semmle/code/csharp/security/dataflow/LogForging.qll

@@ -5,11 +5,11 @@
 import csharp
 
 module LogForging {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.System
   import semmle.code.csharp.frameworks.system.text.RegularExpressions
   import semmle.code.csharp.security.Sanitizers
-  import semmle.code.csharp.security.sinks.ExternalLocationSink
+  import semmle.code.csharp.security.dataflow.flowsinks.ExternalLocationSink
 
   /**
    * A data flow source for untrusted user input used in log entries.

csharp/ql/src/semmle/code/csharp/security/dataflow/MissingXMLValidation.qll

@@ -6,7 +6,7 @@
 import csharp
 
 module MissingXMLValidation {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.Xml
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/ReDoS.qll

@@ -7,7 +7,7 @@ import csharp
 
 module ReDoS {
   private import semmle.code.csharp.dataflow.DataFlow2
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.text.RegularExpressions
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/RegexInjection.qll

@@ -6,7 +6,7 @@
 import csharp
 
 module RegexInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.text.RegularExpressions
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/ResourceInjection.qll

@@ -5,8 +5,8 @@
 import csharp
 
 module ResourceInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
-  import semmle.code.csharp.dataflow.flowsources.Local
+  import semmle.code.csharp.security.dataflow.flowsources.Local
   import semmle.code.csharp.frameworks.system.Data
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/SqlInjection.qll

@@ -5,8 +5,8 @@
 import csharp
 
 module SqlInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
-  import semmle.code.csharp.dataflow.flowsources.Local
+  import semmle.code.csharp.security.dataflow.flowsources.Local
   import semmle.code.csharp.frameworks.Sql
   import semmle.code.csharp.security.Sanitizers
 

csharp/ql/src/semmle/code/csharp/security/dataflow/TaintedPath.qll

@@ -7,7 +7,7 @@ import csharp
 
 module TaintedPath {
   import semmle.code.csharp.controlflow.Guards
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.IO
   import semmle.code.csharp.frameworks.system.Web
   import semmle.code.csharp.security.Sanitizers

csharp/ql/src/semmle/code/csharp/security/dataflow/UnsafeDeserialization.qll

@@ -6,7 +6,7 @@
 import csharp
 
 module UnsafeDeserialization {
-  private import semmle.code.csharp.dataflow.flowsources.Remote
+  private import semmle.code.csharp.security.dataflow.flowsources.Remote
   private import semmle.code.csharp.serialization.Deserializers
 
   /**

csharp/ql/src/semmle/code/csharp/security/dataflow/UrlRedirect.qll

@@ -5,7 +5,7 @@
 import csharp
 
 module UrlRedirect {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.controlflow.Guards
   import semmle.code.csharp.frameworks.system.Web
   import semmle.code.csharp.frameworks.system.web.Mvc

csharp/ql/src/semmle/code/csharp/security/dataflow/XMLEntityInjection.qll

@@ -5,7 +5,7 @@
 import csharp
 
 module XMLEntityInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.System
   import semmle.code.csharp.frameworks.system.text.RegularExpressions
   import semmle.code.csharp.security.xml.InsecureXML

csharp/ql/src/semmle/code/csharp/security/dataflow/XPathInjection.qll

@@ -5,7 +5,7 @@
 import csharp
 
 module XPathInjection {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
   import semmle.code.csharp.frameworks.system.xml.XPath
   import semmle.code.csharp.frameworks.system.Xml
   import semmle.code.csharp.security.Sanitizers

csharp/ql/src/semmle/code/csharp/security/dataflow/XSS.qll

@@ -6,17 +6,14 @@
 import csharp
 
 module XSS {
-  import semmle.code.csharp.dataflow.flowsources.Remote
+  import semmle.code.asp.AspNet
-  import semmle.code.csharp.frameworks.microsoft.AspNetCore
   import semmle.code.csharp.frameworks.system.Net
   import semmle.code.csharp.frameworks.system.Web
-  import semmle.code.csharp.frameworks.system.web.Mvc
-  import semmle.code.csharp.frameworks.system.web.WebPages
   import semmle.code.csharp.frameworks.system.web.UI
-  import semmle.code.csharp.frameworks.system.web.ui.WebControls
-  import semmle.code.csharp.frameworks.system.windows.Forms
   import semmle.code.csharp.security.Sanitizers
-  import semmle.code.asp.AspNet
+  import semmle.code.csharp.security.dataflow.flowsinks.Html
+  import semmle.code.csharp.security.dataflow.flowsinks.Remote
+  import semmle.code.csharp.security.dataflow.flowsources.Remote
 
   /**
    * Holds if there is tainted flow from `source` to `sink` that may lead to a
@@ -112,8 +109,11 @@ module XSS {
 
   /**
    * A data flow sink for cross-site scripting (XSS) vulnerabilities.
+   *
+   * Any XSS sink is also a remote flow sink, so this class contributes
+   * to the abstract class `RemoteFlowSink`.
    */
-  abstract class Sink extends DataFlow::ExprNode {
+  abstract class Sink extends DataFlow::ExprNode, RemoteFlowSink {
     string explanation() { none() }
   }
 
@@ -166,78 +166,21 @@ module XSS {
     UrlEncodeSanitizer() { this.getExpr() instanceof UrlSanitizedExpr }
   }
 
-  /** A sink where the value of the expression may be rendered as HTML. */
+  private class HtmlSinkSink extends Sink {
-  abstract class HtmlSink extends DataFlow::Node { }
+    HtmlSinkSink() { this instanceof HtmlSink }
 
-  /**
+    override string explanation() {
-   * An expression that is used as an argument to an XSS sink method on
+      this instanceof WebPageWriteLiteralSink and
-   * `HttpResponse`.
+      result = "System.Web.WebPages.WebPage.WriteLiteral() method"
-   */
+      or
-  private class HttpResponseSink extends Sink, HtmlSink {
+      this instanceof WebPageWriteLiteralToSink and
-    HttpResponseSink() {
+      result = "System.Web.WebPages.WebPage.WriteLiteralTo() method"
-      exists(Method m, SystemWebHttpResponseClass responseClass |
+      or
-        m = responseClass.getAWriteMethod() or
+      this instanceof MicrosoftAspNetCoreMvcHtmlHelperRawSink and
-        m = responseClass.getAWriteFileMethod() or
+      result = "Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.Raw() method"
-        m = responseClass.getATransmitFileMethod() or
+      or
-        m = responseClass.getABinaryWriteMethod()
+      this instanceof MicrosoftAspNetRazorPageWriteLiteralSink and
-      |
+      result = "Microsoft.AspNetCore.Mvc.Razor.RazorPageBase.WriteLiteral() method"
-        // Calls to these methods, or overrides of them
-        this.getExpr() = m.getAnOverrider*().getParameter(0).getAnAssignedArgument()
-      )
-    }
-  }
-
-  /**
-   * An expression that is used as an argument to an XSS sink method on
-   * `HtmlTextWriter`.
-   */
-  private class HtmlTextWriterSink extends Sink, HtmlSink {
-    HtmlTextWriterSink() {
-      exists(SystemWebUIHtmlTextWriterClass writeClass, Method m, Call c, int paramPos |
-        paramPos = 0 and
-        (
-          m = writeClass.getAWriteMethod() or
-          m = writeClass.getAWriteLineMethod() or
-          m = writeClass.getAWriteLineNoTabsMethod() or
-          m = writeClass.getAWriteBeginTagMethod() or
-          m = writeClass.getAWriteAttributeMethod()
-        )
-        or
-        // The second parameter to the `WriteAttribute` method is the attribute value, which we
-        // should only consider as tainted if the call does not ask for the attribute value to be
-        // encoded using the final parameter.
-        m = writeClass.getAWriteAttributeMethod() and
-        paramPos = 1 and
-        not c.getArgumentForParameter(m.getParameter(2)).(BoolLiteral).getBoolValue() = true
-      |
-        c = m.getACall() and
-        this.getExpr() = c.getArgumentForParameter(m.getParameter(paramPos))
-      )
-    }
-  }
-
-  /**
-   * An expression that is used as an argument to an XSS sink method on
-   * `AttributeCollection`.
-   */
-  private class AttributeCollectionSink extends Sink, HtmlSink {
-    AttributeCollectionSink() {
-      exists(SystemWebUIAttributeCollectionClass ac, Parameter p |
-        p = ac.getAddMethod().getParameter(1) or
-        p = ac.getItemProperty().getSetter().getParameter(0)
-      |
-        this.getExpr() = p.getAnAssignedArgument()
-      )
-    }
-  }
-
-  /**
-   * An expression that is used as the second argument `HtmlElement.SetAttribute`.
-   */
-  private class SetAttributeSink extends Sink, HtmlSink {
-    SetAttributeSink() {
-      this.getExpr() =
-        any(SystemWindowsFormsHtmlElement c).getSetAttributeMethod().getACall().getArgument(1)
     }
   }
 
@@ -285,31 +228,6 @@ module XSS {
     }
   }
 
-  /**
-   * An expression that is used as an argument to an XSS sink setter, on
-   * a class within the `System.Web.UI` namespace.
-   */
-  private class SystemWebSetterHtmlSink extends Sink, HtmlSink {
-    SystemWebSetterHtmlSink() {
-      exists(Property p, string name, ValueOrRefType declaringType |
-        declaringType = p.getDeclaringType() and
-        any(SystemWebUINamespace n).getAChildNamespace*() = declaringType.getNamespace() and
-        this.getExpr() = p.getSetter().getParameter(0).getAnAssignedArgument() and
-        p.hasName(name)
-      |
-        name = "Caption" and
-        (declaringType.hasName("Calendar") or declaringType.hasName("Table"))
-        or
-        name = "InnerHtml"
-      )
-      or
-      exists(SystemWebUIWebControlsLabelClass c |
-        // Unlike `Text` properties of other web controls, `Label.Text` is not automatically HTML encoded
-        this.getExpr() = c.getTextProperty().getSetter().getParameter(0).getAnAssignedArgument()
-      )
-    }
-  }
-
   /**
    * An expression that is used as an argument to an XSS sink setter, on
    * a class within the `System.Web.UI` namespace.
@@ -345,16 +263,6 @@ module XSS {
     }
   }
 
-  /**
-   * An expression that is used as an argument to `HtmlHelper.Raw`, typically in
-   * a `.cshtml` file.
-   */
-  private class SystemWebMvcHtmlHelperRawSink extends Sink, HtmlSink {
-    SystemWebMvcHtmlHelperRawSink() {
-      this.getExpr() = any(SystemWebMvcHtmlHelperClass h).getRawMethod().getACall().getAnArgument()
-    }
-  }
-
   /**
    * Gets a member which is accessed by the given `AspInlineCode`.
    * The code body must consist only of an access to the member, possibly with qualified
@@ -493,31 +401,6 @@ module XSS {
     }
   }
 
-  /** An expression that is returned from a `ToHtmlString` method. */
-  private class ToHtmlString extends Sink, HtmlSink {
-    ToHtmlString() {
-      exists(Method toHtmlString |
-        toHtmlString =
-          any(SystemWebIHtmlString i).getToHtmlStringMethod().getAnUltimateImplementor() and
-        toHtmlString.canReturn(this.getExpr())
-      )
-    }
-  }
-
-  /**
-   * An expression passed to the constructor of an `HtmlString` or a `MvcHtmlString`.
-   */
-  private class HtmlString extends Sink, HtmlSink {
-    HtmlString() {
-      exists(Class c |
-        c = any(SystemWebMvcMvcHtmlString m) or
-        c = any(SystemWebHtmlString m)
-      |
-        this.getExpr() = c.getAConstructor().getACall().getAnArgument()
-      )
-    }
-  }
-
   /**
    * An expression passed as the `content` argument to the constructor of `StringContent`.
    */
@@ -529,75 +412,6 @@ module XSS {
         ).getArgumentForName("content")
     }
   }
-
-  /**
-   * An expression that is used as an argument to `Page.WriteLiteral`, typically in
-   * a `.cshtml` file.
-   */
-  class WebPageWriteLiteralSink extends Sink, HtmlSink {
-    WebPageWriteLiteralSink() {
-      this.getExpr() = any(WebPageClass h).getWriteLiteralMethod().getACall().getAnArgument()
-    }
-
-    override string explanation() { result = "System.Web.WebPages.WebPage.WriteLiteral() method" }
-  }
-
-  /**
-   * An expression that is used as an argument to `Page.WriteLiteralTo`, typically in
-   * a `.cshtml` file.
-   */
-  class WebPageWriteLiteralToSink extends Sink, HtmlSink {
-    WebPageWriteLiteralToSink() {
-      this.getExpr() = any(WebPageClass h).getWriteLiteralToMethod().getACall().getAnArgument()
-    }
-
-    override string explanation() { result = "System.Web.WebPages.WebPage.WriteLiteralTo() method" }
-  }
-
-  abstract class AspNetCoreSink extends Sink, HtmlSink { }
-
-  /**
-   * An expression that is used as an argument to `HtmlHelper.Raw`, typically in
-   * a `.cshtml` file.
-   */
-  class MicrosoftAspNetCoreMvcHtmlHelperRawSink extends AspNetCoreSink {
-    MicrosoftAspNetCoreMvcHtmlHelperRawSink() {
-      this.getExpr() =
-        any(MicrosoftAspNetCoreMvcHtmlHelperClass h).getRawMethod().getACall().getAnArgument()
-    }
-
-    override string explanation() {
-      result = "Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.Raw() method"
-    }
-  }
-
-  /**
-   * An expression that is used as an argument to `Page.WriteLiteral` in ASP.NET 6.0 razor page, typically in
-   * a `.cshtml` file.
-   */
-  class MicrosoftAspNetRazorPageWriteLiteralSink extends AspNetCoreSink {
-    MicrosoftAspNetRazorPageWriteLiteralSink() {
-      this.getExpr() =
-        any(MicrosoftAspNetCoreMvcRazorPageBase h)
-            .getWriteLiteralMethod()
-            .getACall()
-            .getAnArgument()
-    }
-
-    override string explanation() {
-      result = "Microsoft.AspNetCore.Mvc.Razor.RazorPageBase.WriteLiteral() method"
-    }
-  }
-
-  /** `HtmlString` that may be rendered as is need to have sanitized value. */
-  class MicrosoftAspNetHtmlStringSink extends AspNetCoreSink {
-    MicrosoftAspNetHtmlStringSink() {
-      exists(ObjectCreation c, MicrosoftAspNetCoreHttpHtmlString s |
-        c.getTarget() = s.getAConstructor() and
-        this.asExpr() = c.getAnArgument()
-      )
-    }
-  }
 }
 
 private Type getMemberType(Member m) {

csharp/ql/src/semmle/code/csharp/security/dataflow/flowsinks/Email.qll

@@ -1,11 +1,13 @@
 /** Provides data flow sinks for sending email. */
 
 import csharp
+private import Remote
 private import semmle.code.csharp.frameworks.system.net.Mail
 
+/** Provides sinks for emails. */
 module Email {
   /** A data flow sink for sending email. */
-  abstract class Sink extends DataFlow::ExprNode { }
+  abstract class Sink extends DataFlow::ExprNode, RemoteFlowSink { }
 
   /** A data flow sink for sending email via `System.Net.Mail.MailMessage`. */
   class MailMessageSink extends Sink {

csharp/ql/src/semmle/code/csharp/security/dataflow/flowsinks/ExternalLocationSink.qll

@@ -3,6 +3,7 @@
  */
 
 import csharp
+private import Remote
 private import semmle.code.csharp.commons.Loggers
 private import semmle.code.csharp.frameworks.system.Web
 
@@ -45,7 +46,7 @@ class TraceMessageSink extends ExternalLocationSink {
 /**
  * An expression set as a value on a cookie instance.
  */
-class CookieStorageSink extends ExternalLocationSink {
+class CookieStorageSink extends ExternalLocationSink, RemoteFlowSink {
   CookieStorageSink() {
     exists(Expr e | e = this.getExpr() |
       e = any(SystemWebHttpCookie cookie).getAConstructor().getACall().getArgumentForName("value")

csharp/ql/src/semmle/code/csharp/security/dataflow/flowsinks/Html.qll

@@ -0,0 +1,208 @@
+/**
+ * Provides classes representing HTML data flow sinks.
+ */
+
+import csharp
+private import Remote
+private import semmle.code.csharp.frameworks.microsoft.AspNetCore
+private import semmle.code.csharp.frameworks.system.Net
+private import semmle.code.csharp.frameworks.system.Web
+private import semmle.code.csharp.frameworks.system.web.Mvc
+private import semmle.code.csharp.frameworks.system.web.WebPages
+private import semmle.code.csharp.frameworks.system.web.UI
+private import semmle.code.csharp.frameworks.system.web.ui.WebControls
+private import semmle.code.csharp.frameworks.system.windows.Forms
+private import semmle.code.csharp.security.dataflow.flowsources.Remote
+private import semmle.code.asp.AspNet
+
+/**
+ * A sink where the value of the expression may be rendered as HTML,
+ * without implicit HTML encoding.
+ */
+abstract class HtmlSink extends DataFlow::ExprNode, RemoteFlowSink { }
+
+/**
+ * An expression that is used as an argument to an HTML sink method on
+ * `HttpResponse`.
+ */
+class HttpResponseSink extends HtmlSink {
+  HttpResponseSink() {
+    exists(Method m, SystemWebHttpResponseClass responseClass |
+      m = responseClass.getAWriteMethod() or
+      m = responseClass.getAWriteFileMethod() or
+      m = responseClass.getATransmitFileMethod() or
+      m = responseClass.getABinaryWriteMethod()
+    |
+      // Calls to these methods, or overrides of them
+      this.getExpr() = m.getAnOverrider*().getParameter(0).getAnAssignedArgument()
+    )
+  }
+}
+
+/**
+ * An expression that is used as an argument to an HTML sink method on
+ * `HtmlTextWriter`.
+ */
+class HtmlTextWriterSink extends HtmlSink {
+  HtmlTextWriterSink() {
+    exists(SystemWebUIHtmlTextWriterClass writeClass, Method m, Call c, int paramPos |
+      paramPos = 0 and
+      (
+        m = writeClass.getAWriteMethod() or
+        m = writeClass.getAWriteLineMethod() or
+        m = writeClass.getAWriteLineNoTabsMethod() or
+        m = writeClass.getAWriteBeginTagMethod() or
+        m = writeClass.getAWriteAttributeMethod()
+      )
+      or
+      // The second parameter to the `WriteAttribute` method is the attribute value, which we
+      // should only consider as tainted if the call does not ask for the attribute value to be
+      // encoded using the final parameter.
+      m = writeClass.getAWriteAttributeMethod() and
+      paramPos = 1 and
+      not c.getArgumentForParameter(m.getParameter(2)).(BoolLiteral).getBoolValue() = true
+    |
+      c = m.getACall() and
+      this.getExpr() = c.getArgumentForParameter(m.getParameter(paramPos))
+    )
+  }
+}
+
+/**
+ * An expression that is used as an argument to an HTML sink method on
+ * `AttributeCollection`.
+ */
+class AttributeCollectionSink extends HtmlSink {
+  AttributeCollectionSink() {
+    exists(SystemWebUIAttributeCollectionClass ac, Parameter p |
+      p = ac.getAddMethod().getParameter(1) or
+      p = ac.getItemProperty().getSetter().getParameter(0)
+    |
+      this.getExpr() = p.getAnAssignedArgument()
+    )
+  }
+}
+
+/**
+ * An expression that is used as the second argument `HtmlElement.SetAttribute`.
+ */
+class SetAttributeSink extends HtmlSink {
+  SetAttributeSink() {
+    this.getExpr() =
+      any(SystemWindowsFormsHtmlElement c).getSetAttributeMethod().getACall().getArgument(1)
+  }
+}
+
+/**
+ * An expression that is used as an argument to an HTML sink setter, on
+ * a class within the `System.Web.UI` namespace.
+ */
+class SystemWebSetterHtmlSink extends HtmlSink {
+  SystemWebSetterHtmlSink() {
+    exists(Property p, string name, ValueOrRefType declaringType |
+      declaringType = p.getDeclaringType() and
+      any(SystemWebUINamespace n).getAChildNamespace*() = declaringType.getNamespace() and
+      this.getExpr() = p.getAnAssignedValue() and
+      p.hasName(name)
+    |
+      name = "Caption" and
+      (declaringType.hasName("Calendar") or declaringType.hasName("Table"))
+      or
+      name = "InnerHtml"
+    )
+    or
+    exists(SystemWebUIWebControlsLabelClass c |
+      // Unlike `Text` properties of other web controls, `Label.Text` is not automatically HTML encoded
+      this.getExpr() = c.getTextProperty().getSetter().getParameter(0).getAnAssignedArgument()
+    )
+  }
+}
+
+/**
+ * An expression that is used as an argument to `HtmlHelper.Raw`, typically in
+ * a `.cshtml` file.
+ */
+class SystemWebMvcHtmlHelperRawSink extends HtmlSink {
+  SystemWebMvcHtmlHelperRawSink() {
+    this.getExpr() = any(SystemWebMvcHtmlHelperClass h).getRawMethod().getACall().getAnArgument()
+  }
+}
+
+/** An expression that is returned from a `ToHtmlString` method. */
+class ToHtmlString extends HtmlSink {
+  ToHtmlString() {
+    exists(Method toHtmlString |
+      toHtmlString = any(SystemWebIHtmlString i).getToHtmlStringMethod().getAnUltimateImplementor() and
+      toHtmlString.canReturn(this.getExpr())
+    )
+  }
+}
+
+/**
+ * An expression passed to the constructor of an `HtmlString` or a `MvcHtmlString`.
+ */
+class HtmlString extends HtmlSink {
+  HtmlString() {
+    exists(Class c |
+      c = any(SystemWebMvcMvcHtmlString m) or
+      c = any(SystemWebHtmlString m)
+    |
+      this.getExpr() = c.getAConstructor().getACall().getAnArgument()
+    )
+  }
+}
+
+/**
+ * An expression that is used as an argument to `Page.WriteLiteral`, typically in
+ * a `.cshtml` file.
+ */
+class WebPageWriteLiteralSink extends HtmlSink {
+  WebPageWriteLiteralSink() {
+    this.getExpr() = any(WebPageClass h).getWriteLiteralMethod().getACall().getAnArgument()
+  }
+}
+
+/**
+ * An expression that is used as an argument to `Page.WriteLiteralTo`, typically in
+ * a `.cshtml` file.
+ */
+class WebPageWriteLiteralToSink extends HtmlSink {
+  WebPageWriteLiteralToSink() {
+    this.getExpr() = any(WebPageClass h).getWriteLiteralToMethod().getACall().getAnArgument()
+  }
+}
+
+/** An ASP.NET Core HTML sink. */
+abstract class AspNetCoreHtmlSink extends HtmlSink { }
+
+/**
+ * An expression that is used as an argument to `HtmlHelper.Raw`, typically in
+ * a `.cshtml` file.
+ */
+class MicrosoftAspNetCoreMvcHtmlHelperRawSink extends AspNetCoreHtmlSink {
+  MicrosoftAspNetCoreMvcHtmlHelperRawSink() {
+    this.getExpr() =
+      any(MicrosoftAspNetCoreMvcHtmlHelperClass h).getRawMethod().getACall().getAnArgument()
+  }
+}
+
+/**
+ * An expression that is used as an argument to `Page.WriteLiteral` in ASP.NET 6.0 razor page, typically in
+ * a `.cshtml` file.
+ */
+class MicrosoftAspNetRazorPageWriteLiteralSink extends AspNetCoreHtmlSink {
+  MicrosoftAspNetRazorPageWriteLiteralSink() {
+    this.getExpr() =
+      any(MicrosoftAspNetCoreMvcRazorPageBase h).getWriteLiteralMethod().getACall().getAnArgument()
+  }
+}
+
+/** `HtmlString` that may be rendered as is need to have sanitized value. */
+class MicrosoftAspNetHtmlStringSink extends AspNetCoreHtmlSink {
+  MicrosoftAspNetHtmlStringSink() {
+    exists(ObjectCreation c, MicrosoftAspNetCoreHttpHtmlString s |
+      c.getTarget() = s.getAConstructor() and
+      this.asExpr() = c.getAnArgument()
+    )
+  }
+}

csharp/ql/src/semmle/code/csharp/security/dataflow/flowsinks/Remote.qll

@@ -0,0 +1,31 @@
+/**
+ * Provides classes representing data flow sinks for remote user output.
+ */
+
+import csharp
+private import Email::Email
+private import ExternalLocationSink
+private import Html
+private import semmle.code.csharp.security.dataflow.XSS
+private import semmle.code.csharp.frameworks.system.web.UI
+
+/** A data flow sink of remote user output. */
+abstract class RemoteFlowSink extends DataFlow::Node { }
+
+/**
+ * A value written to the `[Inner]Text` property of an object defined in the
+ * `System.Web.UI` namespace.
+ */
+class SystemWebUIText extends RemoteFlowSink {
+  SystemWebUIText() {
+    exists(Property p, string name |
+      p.getDeclaringType().getNamespace().getParentNamespace*() instanceof SystemWebUINamespace and
+      this.asExpr() = p.getAnAssignedValue() and
+      p.hasName(name)
+    |
+      name = "Text"
+      or
+      name = "InnerText"
+    )
+  }
+}

csharp/ql/src/semmle/code/csharp/security/dataflow/flowsources/Remote.qll

@@ -0,0 +1,218 @@
+/**
+ * Provides classes representing data flow sources for remote user input.
+ */
+
+import csharp
+private import semmle.code.csharp.frameworks.system.Net
+private import semmle.code.csharp.frameworks.system.Web
+private import semmle.code.csharp.frameworks.system.web.Http
+private import semmle.code.csharp.frameworks.system.web.Mvc
+private import semmle.code.csharp.frameworks.system.web.Services
+private import semmle.code.csharp.frameworks.system.web.ui.WebControls
+private import semmle.code.csharp.frameworks.WCF
+private import semmle.code.csharp.frameworks.microsoft.Owin
+private import semmle.code.csharp.frameworks.microsoft.AspNetCore
+
+/** A data flow source of remote user input. */
+abstract class RemoteFlowSource extends DataFlow::Node {
+  /** Gets a string that describes the type of this remote flow source. */
+  abstract string getSourceType();
+}
+
+/** A data flow source of remote user input (ASP.NET). */
+abstract class AspNetRemoteFlowSource extends RemoteFlowSource { }
+
+/** A member containing an ASP.NET query string. */
+class AspNetQueryStringMember extends Member {
+  AspNetQueryStringMember() {
+    exists(RefType t |
+      t instanceof SystemWebHttpRequestClass or
+      t instanceof SystemNetHttpListenerRequestClass or
+      t instanceof SystemWebHttpRequestBaseClass
+    |
+      this = t.getProperty(getHttpRequestFlowPropertyNames()) or
+      this.(Field).getType() = t or
+      this.(Property).getType() = t or
+      this.(Callable).getReturnType() = t
+    )
+  }
+}
+
+/**
+ * Gets the names of the properties in `HttpRequest` classes that should propagate taint out of the
+ * request.
+ */
+private string getHttpRequestFlowPropertyNames() {
+  result = "QueryString" or
+  result = "Headers" or
+  result = "RawUrl" or
+  result = "Url" or
+  result = "Cookies" or
+  result = "Form" or
+  result = "Params" or
+  result = "Path" or
+  result = "PathInfo"
+}
+
+/** A data flow source of remote user input (ASP.NET query string). */
+class AspNetQueryStringRemoteFlowSource extends AspNetRemoteFlowSource, DataFlow::ExprNode {
+  AspNetQueryStringRemoteFlowSource() {
+    exists(RefType t |
+      t instanceof SystemWebHttpRequestClass or
+      t instanceof SystemNetHttpListenerRequestClass or
+      t instanceof SystemWebHttpRequestBaseClass
+    |
+      // A request object can be indexed, so taint the object as well
+      this.getExpr().getType() = t
+    )
+    or
+    this.getExpr() = any(AspNetQueryStringMember m).getAnAccess()
+  }
+
+  override string getSourceType() { result = "ASP.NET query string" }
+}
+
+/** A data flow source of remote user input (ASP.NET unvalidated request data). */
+class AspNetUnvalidatedQueryStringRemoteFlowSource extends AspNetRemoteFlowSource,
+  DataFlow::ExprNode {
+  AspNetUnvalidatedQueryStringRemoteFlowSource() {
+    this.getExpr() = any(SystemWebUnvalidatedRequestValues c).getAProperty().getGetter().getACall() or
+    this.getExpr() =
+      any(SystemWebUnvalidatedRequestValuesBase c).getAProperty().getGetter().getACall()
+  }
+
+  override string getSourceType() { result = "ASP.NET unvalidated request data" }
+}
+
+/** A data flow source of remote user input (ASP.NET user input). */
+class AspNetUserInputRemoteFlowSource extends AspNetRemoteFlowSource, DataFlow::ExprNode {
+  AspNetUserInputRemoteFlowSource() { getType() instanceof SystemWebUIWebControlsTextBoxClass }
+
+  override string getSourceType() { result = "ASP.NET user input" }
+}
+
+/** A data flow source of remote user input (WCF based web service). */
+class WcfRemoteFlowSource extends RemoteFlowSource, DataFlow::ParameterNode {
+  WcfRemoteFlowSource() { exists(OperationMethod om | om.getAParameter() = this.getParameter()) }
+
+  override string getSourceType() { result = "web service input" }
+}
+
+/** A data flow source of remote user input (ASP.NET web service). */
+class AspNetServiceRemoteFlowSource extends RemoteFlowSource, DataFlow::ParameterNode {
+  AspNetServiceRemoteFlowSource() {
+    exists(Method m |
+      m.getAParameter() = this.getParameter() and
+      m.getAnAttribute().getType() instanceof SystemWebServicesWebMethodAttributeClass
+    )
+  }
+
+  override string getSourceType() { result = "ASP.NET web service input" }
+}
+
+/** A data flow source of remote user input (ASP.NET request message). */
+class SystemNetHttpRequestMessageRemoteFlowSource extends RemoteFlowSource, DataFlow::ExprNode {
+  SystemNetHttpRequestMessageRemoteFlowSource() {
+    getType() instanceof SystemWebHttpRequestMessageClass
+  }
+
+  override string getSourceType() { result = "ASP.NET request message" }
+}
+
+/**
+ * A data flow source of remote user input (Microsoft Owin, a query, request,
+ * or path string).
+ */
+class MicrosoftOwinStringFlowSource extends RemoteFlowSource, DataFlow::ExprNode {
+  MicrosoftOwinStringFlowSource() {
+    this.getExpr() = any(MicrosoftOwinString owinString).getValueProperty().getGetter().getACall()
+  }
+
+  override string getSourceType() { result = "Microsoft Owin request or query string" }
+}
+
+/** A data flow source of remote user input (`Microsoft Owin IOwinRequest`). */
+class MicrosoftOwinRequestRemoteFlowSource extends RemoteFlowSource, DataFlow::ExprNode {
+  MicrosoftOwinRequestRemoteFlowSource() {
+    exists(Property p, MicrosoftOwinIOwinRequestClass owinRequest |
+      this.getExpr() = p.getGetter().getACall()
+    |
+      p = owinRequest.getAcceptProperty() or
+      p = owinRequest.getBodyProperty() or
+      p = owinRequest.getCacheControlProperty() or
+      p = owinRequest.getContentTypeProperty() or
+      p = owinRequest.getContextProperty() or
+      p = owinRequest.getCookiesProperty() or
+      p = owinRequest.getHeadersProperty() or
+      p = owinRequest.getHostProperty() or
+      p = owinRequest.getMediaTypeProperty() or
+      p = owinRequest.getMethodProperty() or
+      p = owinRequest.getPathProperty() or
+      p = owinRequest.getPathBaseProperty() or
+      p = owinRequest.getQueryProperty() or
+      p = owinRequest.getQueryStringProperty() or
+      p = owinRequest.getRemoteIpAddressProperty() or
+      p = owinRequest.getSchemeProperty() or
+      p = owinRequest.getURIProperty()
+    )
+  }
+
+  override string getSourceType() { result = "Microsoft Owin request" }
+}
+
+/** A parameter to an Mvc controller action method, viewed as a source of remote user input. */
+class ActionMethodParameter extends RemoteFlowSource, DataFlow::ParameterNode {
+  ActionMethodParameter() {
+    exists(Parameter p |
+      p = this.getParameter() and
+      p.fromSource()
+    |
+      p = any(Controller c).getAnActionMethod().getAParameter() or
+      p = any(ApiController c).getAnActionMethod().getAParameter()
+    )
+  }
+
+  override string getSourceType() { result = "ASP.NET MVC action method parameter" }
+}
+
+/** A data flow source of remote user input (ASP.NET Core). */
+abstract class AspNetCoreRemoteFlowSource extends RemoteFlowSource { }
+
+/** A data flow source of remote user input (ASP.NET query collection). */
+class AspNetCoreQueryRemoteFlowSource extends AspNetCoreRemoteFlowSource, DataFlow::ExprNode {
+  AspNetCoreQueryRemoteFlowSource() {
+    exists(ValueOrRefType t |
+      t instanceof MicrosoftAspNetCoreHttpHttpRequest or
+      t instanceof MicrosoftAspNetCoreHttpQueryCollection or
+      t instanceof MicrosoftAspNetCoreHttpQueryString
+    |
+      this.getExpr().(Call).getTarget().getDeclaringType() = t or
+      this.asExpr().(Access).getTarget().getDeclaringType() = t
+    )
+    or
+    exists(Call c |
+      c
+          .getTarget()
+          .getDeclaringType()
+          .hasQualifiedName("Microsoft.AspNetCore.Http", "IQueryCollection") and
+      c.getTarget().getName() = "TryGetValue" and
+      this.asExpr() = c.getArgumentForName("value")
+    )
+  }
+
+  override string getSourceType() { result = "ASP.NET Core query string" }
+}
+
+/** A parameter to a `Mvc` controller action method, viewed as a source of remote user input. */
+class AspNetCoreActionMethodParameter extends RemoteFlowSource, DataFlow::ParameterNode {
+  AspNetCoreActionMethodParameter() {
+    exists(Parameter p |
+      p = this.getParameter() and
+      p.fromSource()
+    |
+      p = any(MicrosoftAspNetCoreMvcController c).getAnActionMethod().getAParameter()
+    )
+  }
+
+  override string getSourceType() { result = "ASP.NET Core MVC action method parameter" }
+}

csharp/ql/test/library-tests/dataflow/flowsources/remote/remoteFlowSource.ql

@@ -1,4 +1,4 @@
-import semmle.code.csharp.dataflow.flowsources.Remote
+import semmle.code.csharp.security.dataflow.flowsources.Remote
 
 from RemoteFlowSource source
 select source, source.getSourceType()

csharp/ql/test/library-tests/standalone/controlflow/cfg.expected

@@ -7,9 +7,14 @@
 | ControlFlow.cs:10:9:10:43 | Call (unknown target) | ControlFlow.cs:12:9:12:87 | ...; |
 | ControlFlow.cs:10:9:10:43 | call to method  | ControlFlow.cs:12:9:12:87 | ...; |
 | ControlFlow.cs:10:9:10:44 | ...; | ControlFlow.cs:10:9:10:13 | Expression |
-| ControlFlow.cs:10:22:10:22 | access to local variable v | ControlFlow.cs:10:22:10:24 | Expression |
+| ControlFlow.cs:10:22:10:22 | access to local variable v | ControlFlow.cs:10:22:10:24 | Call (unknown target) |
-| ControlFlow.cs:10:22:10:24 | Expression | ControlFlow.cs:10:22:10:26 | Expression |
+| ControlFlow.cs:10:22:10:22 | access to local variable v | ControlFlow.cs:10:22:10:24 | access to property (unknown) |
-| ControlFlow.cs:10:22:10:26 | Expression | ControlFlow.cs:10:29:10:42 | "This is true" |
+| ControlFlow.cs:10:22:10:24 | Call (unknown target) | ControlFlow.cs:10:22:10:26 | Call (unknown target) |
+| ControlFlow.cs:10:22:10:24 | Call (unknown target) | ControlFlow.cs:10:22:10:26 | access to property (unknown) |
+| ControlFlow.cs:10:22:10:24 | access to property (unknown) | ControlFlow.cs:10:22:10:26 | Call (unknown target) |
+| ControlFlow.cs:10:22:10:24 | access to property (unknown) | ControlFlow.cs:10:22:10:26 | access to property (unknown) |
+| ControlFlow.cs:10:22:10:26 | Call (unknown target) | ControlFlow.cs:10:29:10:42 | "This is true" |
+| ControlFlow.cs:10:22:10:26 | access to property (unknown) | ControlFlow.cs:10:29:10:42 | "This is true" |
 | ControlFlow.cs:10:29:10:42 | "This is true" | ControlFlow.cs:10:9:10:43 | Call (unknown target) |
 | ControlFlow.cs:10:29:10:42 | "This is true" | ControlFlow.cs:10:9:10:43 | call to method  |
 | ControlFlow.cs:12:9:12:86 | Call (unknown target) | ControlFlow.cs:12:37:12:47 | Expression |
@@ -20,5 +25,7 @@
 | ControlFlow.cs:12:51:12:62 | access to field Empty | ControlFlow.cs:12:37:12:62 | ... = ... |
 | ControlFlow.cs:12:65:12:75 | Expression | ControlFlow.cs:12:79:12:79 | access to local variable v |
 | ControlFlow.cs:12:65:12:84 | ... = ... | ControlFlow.cs:12:35:12:86 | { ..., ... } |
-| ControlFlow.cs:12:79:12:79 | access to local variable v | ControlFlow.cs:12:79:12:84 | Expression |
+| ControlFlow.cs:12:79:12:79 | access to local variable v | ControlFlow.cs:12:79:12:84 | Call (unknown target) |
-| ControlFlow.cs:12:79:12:84 | Expression | ControlFlow.cs:12:65:12:84 | ... = ... |
+| ControlFlow.cs:12:79:12:79 | access to local variable v | ControlFlow.cs:12:79:12:84 | access to property (unknown) |
+| ControlFlow.cs:12:79:12:84 | Call (unknown target) | ControlFlow.cs:12:65:12:84 | ... = ... |
+| ControlFlow.cs:12:79:12:84 | access to property (unknown) | ControlFlow.cs:12:65:12:84 | ... = ... |

csharp/ql/test/library-tests/standalone/errorrecovery/ErrorCalls.expected

@@ -2,5 +2,5 @@
 | errors.cs:43:21:43:28 | errors.cs:43:21:43:28 | object creation of type C1 | C1 |
 | errors.cs:44:13:44:19 | errors.cs:44:13:44:19 | call to method m1 | m1 |
 | errors.cs:45:13:45:19 | errors.cs:45:13:45:19 | call to method m2 | m2 |
-| errors.cs:46:13:46:38 | errors.cs:46:13:46:38 | call to method  | none |
+| errors.cs:46:13:46:38 | errors.cs:46:13:46:38 | call to method WriteLine | WriteLine |
 | errors.cs:53:17:53:25 | errors.cs:53:17:53:25 | object creation of type C2 | none |