hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2026-03-06 15:49:08 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into mergeback-2018-10-11

Browse Source
This commit is contained in:
Tom Hvitved
2018-10-11 14:36:44 +02:00
parent 95abf557ac 68dae60927
commit b29b314f4e
321 changed files with 27787 additions and 3927 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
change-notes/1.19/analysis-cpp.md
View File

@@ -6,7 +6,9 @@
| **Query** | **Tags** | **Purpose** |
|-----------------------------|-----------|--------------------------------------------------------------------|
| *@name of query (Query ID)* | *Tags* |*Aim of the new query and whether it is enabled by default or not* |
| Cast between HRESULT and a Boolean type (`cpp/hresult-boolean-conversion`) | external/cwe/cwe-253 | Finds logic errors caused by mistakenly treating the Windows `HRESULT` type as a Boolean instead of testing it with the appropriate macros. Enabled by default. |
| Setting a DACL to `NULL` in a `SECURITY_DESCRIPTOR` (`cpp/unsafe-dacl-security-descriptor`) | external/cwe/cwe-732 | This query finds code that creates world-writable objects on Windows by setting their DACL to `NULL`. Enabled by default. |
| Cast from char* to wchar_t* | security, external/cwe/cwe-704 | Detects potentially dangerous casts from char* to wchar_t*. Enabled by default on LGTM. |
## Changes to existing queries
@@ -14,6 +16,8 @@
|----------------------------|------------------------|------------------------------------------------------------------|
| Resource not released in destructor | Fewer false positive results | Placement new is now excluded from the query. |
| Missing return statement (`cpp/missing-return`) | Visible by default | The precision of this query has been increased from 'medium' to 'high', which makes it visible by default in LGTM. It was 'medium' in release 1.17 and 1.18 because it had false positives due to an extractor bug that was fixed in 1.18. |
| Call to memory access function may overflow buffer | More correct results | Array indexing with a negative index is now detected by this query. |
| Suspicious add with sizeof | Fewer false positive results | Arithmetic with void pointers (where allowed) is now excluded from this query. |
| Wrong type of arguments to formatting function | Fewer false positive results | False positive results involving typedefs have been removed. Expected argument types are determined more accurately, especially for wide string and pointer types. Custom (non-standard) formatting functions are also identified more accurately. |
## Changes to QL libraries

16
change-notes/1.19/analysis-java.md Normal file
View File

@@ -0,0 +1,16 @@
# Improvements to Java analysis
## General improvements
## New queries
| **Query** | **Tags** | **Purpose** |
|-----------------------------------------------|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
## Changes to existing queries
| **Query** | **Expected impact** | **Change** |
| Unreachable catch clause (`java/unreachable-catch-clause`) | Fewer false-positive results | This rule now accounts for calls to generic methods that throw generic exceptions. |
## Changes to QL libraries

8
change-notes/1.19/analysis-javascript.md
View File

@@ -16,10 +16,12 @@
| **Query** | **Tags** | **Purpose** |
|-----------------------------------------------|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Enabling Node.js integration for Electron web content renderers (`js/enabling-electron-renderer-node-integration`) | security, frameworks/electron, external/cwe/cwe-094 | Highlights Electron web content renderer preferences with Node.js integration enabled, indicating a violation of [CWE-94](https://cwe.mitre.org/data/definitions/94.html). Results are not shown on LGTM by default. |
| File data in outbound network request | security, external/cwe/cwe-200 | Highlights locations where file data is sent in a network request. Results are not shown on LGTM by default. |
| Host header poisoning in email generation | security, external/cwe/cwe-640 | Highlights code that generates emails with links that can be hijacked by HTTP host header poisoning, indicating a violation of [CWE-640](https://cwe.mitre.org/data/definitions/640.html). Results shown on LGTM by default. |
| Replacement of a substring with itself (`js/identity-replacement`) | correctness, security, external/cwe/cwe-116 | Highlights string replacements that replace a string with itself, which usually indicates a mistake. Results shown on LGTM by default. |
| Stored cross-site scripting (`js/stored-xss`) | security, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights uncontrolled stored values flowing into HTML content, indicating a violation of [CWE-079](https://cwe.mitre.org/data/definitions/79.html). Results shown on LGTM by default. |
| Unclear precedence of nested operators (`js/unclear-operator-precedence`) | maintainability, correctness, external/cwe/cwe-783 | Highlights nested binary operators whose relative precedence is easy to misunderstand. Results shown on LGTM by default. |
| User-controlled data in file | security, external/cwe/cwe-912 | Highlights locations where user-controlled data is written to a file. Results are not shown on LGTM by default. |
## Changes to existing queries
@@ -35,4 +37,8 @@
## Changes to QL libraries
* The flow configuration framework now supports distinguishing and tracking different kinds of taint, specified by an extensible class `FlowLabel` (which can also be referred to by its alias `TaintKind`).
* The flow configuration framework now supports distinguishing and tracking different kinds of taint, specified by an extensible class `FlowLabel` (which can also be referred to by its alias `TaintKind`).
* The `DataFlow::ThisNode` class now corresponds to the implicit receiver parameter of a function, as opposed to an indivdual `this` expression. This means `getALocalSource` now maps all `this` expressions within a given function to the same source. The data-flow node associated with a `ThisExpr` can no longer be cast to `DataFlow::SourceNode` or `DataFlow::ThisNode` - it is recomended to use `getALocalSource` before casting or instead of casting.
* `ReactComponent::getAThisAccess` has been renamed to `getAThisNode`. The old name is still usable but is deprecated. It no longer gets individual `this` expressions, but the `ThisNode` mentioned above.

1
cpp/config/suites/c/correctness
View File

@@ -6,6 +6,7 @@
+ semmlecode-cpp-queries/Likely Bugs/Arithmetic/IntMultToLong.ql: /Correctness/Dangerous Conversions
+ semmlecode-cpp-queries/Likely Bugs/Conversion/NonzeroValueCastToPointer.ql: /Correctness/Dangerous Conversions
+ semmlecode-cpp-queries/Likely Bugs/Conversion/ImplicitDowncastFromBitfield.ql: /Correctness/Dangerous Conversions
+ semmlecode-cpp-queries/Security/CWE/CWE-253/HResultBooleanConversion.ql: /Correctness/Dangerous Conversions
# Consistent Use
+ semmlecode-cpp-queries/Critical/ReturnValueIgnored.ql: /Correctness/Consistent Use
+ semmlecode-cpp-queries/Likely Bugs/InconsistentCheckReturnNull.ql: /Correctness/Consistent Use

1
cpp/config/suites/cpp/correctness
View File

@@ -7,6 +7,7 @@
+ semmlecode-cpp-queries/Likely Bugs/Conversion/NonzeroValueCastToPointer.ql: /Correctness/Dangerous Conversions
+ semmlecode-cpp-queries/Likely Bugs/Conversion/ImplicitDowncastFromBitfield.ql: /Correctness/Dangerous Conversions
+ semmlecode-cpp-queries/Likely Bugs/Conversion/CastArrayPointerArithmetic.ql: /Correctness/Dangerous Conversions
+ semmlecode-cpp-queries/Security/CWE/CWE-253/HResultBooleanConversion.ql: /Correctness/Dangerous Conversions
# Consistent Use
+ semmlecode-cpp-queries/Critical/ReturnValueIgnored.ql: /Correctness/Consistent Use
+ semmlecode-cpp-queries/Likely Bugs/InconsistentCheckReturnNull.ql: /Correctness/Consistent Use

3
cpp/config/suites/security/cwe-253 Normal file
View File

@@ -0,0 +1,3 @@
# CWE-253: Incorrect Check of Function Return Value
+ semmlecode-cpp-queries/Security/CWE/CWE-253/HResultBooleanConversion.ql: /CWE/CWE-253
@name Cast between HRESULT and a Boolean type (CWE-253)

2
cpp/config/suites/security/cwe-732
View File

@@ -1,3 +1,5 @@
# CWE-732: Incorrect Permission Assignment for Critical Resource
+ semmlecode-cpp-queries/Security/CWE/CWE-732/DoNotCreateWorldWritable.ql: /CWE/CWE-732
@name File created without restricting permissions (CWE-732)
+ semmlecode-cpp-queries/Security/CWE/CWE-732/UnsafeDaclSecurityDescriptor.ql: /CWE/CWE-732
@name Setting a DACL to NULL in a SECURITY_DESCRIPTOR (CWE-732)

1
cpp/config/suites/security/default
View File

@@ -13,6 +13,7 @@
@import "cwe-170"
@import "cwe-190"
@import "cwe-242"
@import "cwe-253"
@import "cwe-290"
@import "cwe-311"
@import "cwe-327"

8
cpp/ql/src/Security/CWE/CWE-253/HResultBooleanConversion.qhelp
View File

@@ -4,13 +4,13 @@
<qhelp>
<overview>
<p>This query indicates that an <code>HRESULT</code> is being cast to a boolean type or vice versa.</p>
<p>The typical success value (<code>S_OK</code>) of an <code>HRESULT</code> equals 0. However, 0 indicates failure for a boolean type.</p>
<p>Casting an <code>HRESULT</code> to a boolean type and then using it in a test expression will yield an incorrect result.</p>
<p>This query indicates that an <code>HRESULT</code> is being cast to a Boolean type or vice versa.</p>
<p>The typical success value (<code>S_OK</code>) of an <code>HRESULT</code> equals 0. However, 0 indicates failure for a Boolean type.</p>
<p>Casting an <code>HRESULT</code> to a Boolean type and then using it in a test expression will yield an incorrect result.</p>
</overview>
<recommendation>
<p>To check if a call that returns an HRESULT succeeded use the <code>FAILED</code> macro.</p>
<p>To check if a call that returns an <code>HRESULT</code> succeeded use the <code>FAILED</code> macro.</p>
</recommendation>
<example>

8
cpp/ql/src/Security/CWE/CWE-253/HResultBooleanConversion.ql
View File

@@ -1,8 +1,6 @@
/**
* @name Cast between semantically different integer types: HRESULT to/from a Boolean type
* @description Cast between semantically different integer types: HRESULT to/from a Boolean type.
* Boolean types indicate success by a non-zero value, whereas success (S_OK) in HRESULT is indicated by a value of 0.
* Casting an HRESULT to/from a Boolean type and then using it in a test expression will yield an incorrect result.
* @name Cast between HRESULT and a Boolean type
* @description Casting an HRESULT to/from a Boolean type and then using it in a test expression will yield an incorrect result because success (S_OK) in HRESULT is indicated by a value of 0.
* @kind problem
* @id cpp/hresult-boolean-conversion
* @problem.severity error
@@ -68,4 +66,4 @@ where exists
)
and not isHresultBooleanConverted(e1)
)
select e1, msg
select e1, msg

36
cpp/ql/src/semmle/code/cpp/commons/Buffer.qll
View File

@@ -1,4 +1,5 @@
import cpp
import semmle.code.cpp.dataflow.DataFlow
/**
* Holds if `sizeof(s)` occurs as part of the parameter of a dynamic
@@ -47,6 +48,7 @@ predicate memberMayBeVarSize(Class c, MemberVariable v) {
/**
* Get the size in bytes of the buffer pointed to by an expression (if this can be determined).
*/
language[monotonicAggregates]
int getBufferSize(Expr bufferExpr, Element why) {
exists(Variable bufferVar | bufferVar = bufferExpr.(VariableAccess).getTarget() |
(
@@ -58,6 +60,10 @@ int getBufferSize(Expr bufferExpr, Element why) {
// buffer is an initialized array
// e.g. int buffer[] = {1, 2, 3};
why = bufferVar.getInitializer().getExpr() and
(
why instanceof AggregateLiteral or
why instanceof StringLiteral
) and
result = why.(Expr).getType().(ArrayType).getSize() and
not exists(bufferVar.getType().getUnspecifiedType().(ArrayType).getSize())
) or exists(Class parentClass, VariableAccess parentPtr |
@@ -71,19 +77,25 @@ int getBufferSize(Expr bufferExpr, Element why) {
bufferVar.getType().getSize() -
parentClass.getSize()
)
) or exists(Expr def |
// buffer is assigned with an allocation
definitionUsePair(_, def, bufferExpr) and
exprDefinition(_, def, why) and
isFixedSizeAllocationExpr(why, result)
) or exists(Expr def, Expr e, Element why2 |
// buffer is assigned with another buffer
definitionUsePair(_, def, bufferExpr) and
exprDefinition(_, def, e) and
result = getBufferSize(e, why2) and
(
) or (
// buffer is a fixed size dynamic allocation
isFixedSizeAllocationExpr(bufferExpr, result) and
why = bufferExpr
) or (
// dataflow (all sources must be the same size)
result = min(Expr def |
DataFlow::localFlowStep(DataFlow::exprNode(def), DataFlow::exprNode(bufferExpr)) |
getBufferSize(def, _)
) and result = max(Expr def |
DataFlow::localFlowStep(DataFlow::exprNode(def), DataFlow::exprNode(bufferExpr)) |
getBufferSize(def, _)
) and
// find reason
exists(Expr def |
DataFlow::localFlowStep(DataFlow::exprNode(def), DataFlow::exprNode(bufferExpr)) |
why = def or
why = why2
exists(getBufferSize(def, why))
)
) or exists(Type bufferType |
// buffer is the address of a variable

13
csharp/extractor/.gitignore vendored Normal file
View File

@@ -0,0 +1,13 @@
obj/
TestResults/
*.manifest
*.pdb
*.suo
*.mdb
*.vsmdi
csharp.log
**/bin/Debug
**/bin/Release
*.tlog
.vs
*.user

83
csharp/extractor/CSharpExtractor.sln Normal file
View File

@@ -0,0 +1,83 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 15
VisualStudioVersion = 15.0.27130.2036
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Util", "Semmle.Util\Semmle.Util.csproj", "{CDD7AD69-0FD8-40F0-A9DA-F1077A2A85D6}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction", "Semmle.Extraction\Semmle.Extraction.csproj", "{81EAAD75-4BE1-44E4-91DF-20778216DB64}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction.CSharp", "Semmle.Extraction.CSharp\Semmle.Extraction.CSharp.csproj", "{C4D62DA0-B64B-440B-86DC-AB52318CB8BF}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction.CIL", "Semmle.Extraction.CIL\Semmle.Extraction.CIL.csproj", "{399A1579-68F0-40F4-9A23-F241BA697F9C}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Autobuild", "Semmle.Autobuild\Semmle.Autobuild.csproj", "{5131EF00-0BA9-4436-A3B0-C5CDAB4B194C}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction.CSharp.Standalone", "Semmle.Extraction.CSharp.Standalone\Semmle.Extraction.CSharp.Standalone.csproj", "{D00E7D25-0FA0-48EC-B048-CD60CE1B30D8}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction.CIL.Driver", "Semmle.Extraction.CIL.Driver\Semmle.Extraction.CIL.Driver.csproj", "{EFA400B3-C1CE-446F-A4E2-8B44E61EF47C}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction.CSharp.Driver", "Semmle.Extraction.CSharp.Driver\Semmle.Extraction.CSharp.Driver.csproj", "{C36453BF-0C82-448A-B15D-26947503A2D3}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Extraction.Tests", "Semmle.Extraction.Tests\Semmle.Extraction.Tests.csproj", "{CD8D3F90-AD2E-4BB5-8E82-B94AA293864A}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Util.Tests", "Semmle.Util.Tests\Semmle.Util.Tests.csproj", "{55A620F0-23F6-440D-A5BA-0567613B3C0F}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Semmle.Autobuild.Tests", "Semmle.Autobuild.Tests\Semmle.Autobuild.Tests.csproj", "{CE267461-D762-4F53-A275-685A0A4EC48D}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{CDD7AD69-0FD8-40F0-A9DA-F1077A2A85D6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CDD7AD69-0FD8-40F0-A9DA-F1077A2A85D6}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CDD7AD69-0FD8-40F0-A9DA-F1077A2A85D6}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CDD7AD69-0FD8-40F0-A9DA-F1077A2A85D6}.Release|Any CPU.Build.0 = Release|Any CPU
{81EAAD75-4BE1-44E4-91DF-20778216DB64}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{81EAAD75-4BE1-44E4-91DF-20778216DB64}.Debug|Any CPU.Build.0 = Debug|Any CPU
{81EAAD75-4BE1-44E4-91DF-20778216DB64}.Release|Any CPU.ActiveCfg = Release|Any CPU
{81EAAD75-4BE1-44E4-91DF-20778216DB64}.Release|Any CPU.Build.0 = Release|Any CPU
{C4D62DA0-B64B-440B-86DC-AB52318CB8BF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C4D62DA0-B64B-440B-86DC-AB52318CB8BF}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C4D62DA0-B64B-440B-86DC-AB52318CB8BF}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C4D62DA0-B64B-440B-86DC-AB52318CB8BF}.Release|Any CPU.Build.0 = Release|Any CPU
{399A1579-68F0-40F4-9A23-F241BA697F9C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{399A1579-68F0-40F4-9A23-F241BA697F9C}.Debug|Any CPU.Build.0 = Debug|Any CPU
{399A1579-68F0-40F4-9A23-F241BA697F9C}.Release|Any CPU.ActiveCfg = Release|Any CPU
{399A1579-68F0-40F4-9A23-F241BA697F9C}.Release|Any CPU.Build.0 = Release|Any CPU
{5131EF00-0BA9-4436-A3B0-C5CDAB4B194C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5131EF00-0BA9-4436-A3B0-C5CDAB4B194C}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5131EF00-0BA9-4436-A3B0-C5CDAB4B194C}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5131EF00-0BA9-4436-A3B0-C5CDAB4B194C}.Release|Any CPU.Build.0 = Release|Any CPU
{D00E7D25-0FA0-48EC-B048-CD60CE1B30D8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{D00E7D25-0FA0-48EC-B048-CD60CE1B30D8}.Debug|Any CPU.Build.0 = Debug|Any CPU
{D00E7D25-0FA0-48EC-B048-CD60CE1B30D8}.Release|Any CPU.ActiveCfg = Release|Any CPU
{D00E7D25-0FA0-48EC-B048-CD60CE1B30D8}.Release|Any CPU.Build.0 = Release|Any CPU
{EFA400B3-C1CE-446F-A4E2-8B44E61EF47C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{EFA400B3-C1CE-446F-A4E2-8B44E61EF47C}.Debug|Any CPU.Build.0 = Debug|Any CPU
{EFA400B3-C1CE-446F-A4E2-8B44E61EF47C}.Release|Any CPU.ActiveCfg = Release|Any CPU
{EFA400B3-C1CE-446F-A4E2-8B44E61EF47C}.Release|Any CPU.Build.0 = Release|Any CPU
{C36453BF-0C82-448A-B15D-26947503A2D3}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C36453BF-0C82-448A-B15D-26947503A2D3}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C36453BF-0C82-448A-B15D-26947503A2D3}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C36453BF-0C82-448A-B15D-26947503A2D3}.Release|Any CPU.Build.0 = Release|Any CPU
{CD8D3F90-AD2E-4BB5-8E82-B94AA293864A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CD8D3F90-AD2E-4BB5-8E82-B94AA293864A}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CD8D3F90-AD2E-4BB5-8E82-B94AA293864A}.Release|Any CPU.ActiveCfg = Release|Any CPU
{55A620F0-23F6-440D-A5BA-0567613B3C0F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{55A620F0-23F6-440D-A5BA-0567613B3C0F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{55A620F0-23F6-440D-A5BA-0567613B3C0F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CE267461-D762-4F53-A275-685A0A4EC48D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CE267461-D762-4F53-A275-685A0A4EC48D}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CE267461-D762-4F53-A275-685A0A4EC48D}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CE267461-D762-4F53-A275-685A0A4EC48D}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {E2B2BAC0-D55C-45DB-8CB3-8CEBA86FB547}
EndGlobalSection
EndGlobal

883
csharp/extractor/Semmle.Autobuild.Tests/BuildScripts.cs Normal file
View File

@@ -0,0 +1,883 @@
using Xunit;
using Semmle.Autobuild;
using System.Collections.Generic;
using System;
using System.Linq;
using Microsoft.Build.Construction;
namespace Semmle.Extraction.Tests
{
/// <summary>
/// Test class to script Autobuilder scenarios.
/// For most methods, it uses two fields:
/// - an IList to capture the the arguments passed to it
/// - an IDictionary of possible return values.
/// </summary>
class TestActions : IBuildActions
{
/// <summary>
/// List of strings passed to FileDelete.
/// </summary>
public IList<string> FileDeleteIn = new List<string>();
void IBuildActions.FileDelete(string file)
{
FileDeleteIn.Add(file);
}
public IList<string> FileExistsIn = new List<string>();
public IDictionary<string, bool> FileExists = new Dictionary<string, bool>();
bool IBuildActions.FileExists(string file)
{
FileExistsIn.Add(file);
if (FileExists.TryGetValue(file, out var ret))
return ret;
throw new ArgumentException("Missing FileExists " + file);
}
public IList<string> RunProcessIn = new List<string>();
public IDictionary<string, int> RunProcess = new Dictionary<string, int>();
public IDictionary<string, string> RunProcessOut = new Dictionary<string, string>();
public IDictionary<string, string> RunProcessWorkingDirectory = new Dictionary<string, string>();
int IBuildActions.RunProcess(string cmd, string args, string workingDirectory, IDictionary<string, string> env, out IList<string> stdOut)
{
var pattern = cmd + " " + args;
RunProcessIn.Add(pattern);
if (RunProcessOut.TryGetValue(pattern, out var str))
stdOut = str.Split("\n");
else
throw new ArgumentException("Missing RunProcessOut " + pattern);
RunProcessWorkingDirectory.TryGetValue(pattern, out var wd);
if (wd != workingDirectory)
throw new ArgumentException("Missing RunProcessWorkingDirectory " + pattern);
if (RunProcess.TryGetValue(pattern, out var ret))
return ret;
throw new ArgumentException("Missing RunProcess " + pattern);
}
int IBuildActions.RunProcess(string cmd, string args, string workingDirectory, IDictionary<string, string> env)
{
var pattern = cmd + " " + args;
RunProcessIn.Add(pattern);
RunProcessWorkingDirectory.TryGetValue(pattern, out var wd);
if (wd != workingDirectory)
throw new ArgumentException("Missing RunProcessWorkingDirectory " + pattern);
if (RunProcess.TryGetValue(pattern, out var ret))
return ret;
throw new ArgumentException("Missing RunProcess " + pattern);
}
public IList<string> DirectoryDeleteIn = new List<string>();
void IBuildActions.DirectoryDelete(string dir, bool recursive)
{
DirectoryDeleteIn.Add(dir);
}
public IDictionary<string, bool> DirectoryExists = new Dictionary<string, bool>();
public IList<string> DirectoryExistsIn = new List<string>();
bool IBuildActions.DirectoryExists(string dir)
{
DirectoryExistsIn.Add(dir);
if (DirectoryExists.TryGetValue(dir, out var ret))
return ret;
throw new ArgumentException("Missing DirectoryExists " + dir);
}
public IDictionary<string, string> GetEnvironmentVariable = new Dictionary<string, string>();
string IBuildActions.GetEnvironmentVariable(string name)
{
if (GetEnvironmentVariable.TryGetValue(name, out var ret))
return ret;
throw new ArgumentException("Missing GetEnvironmentVariable " + name);
}
public string GetCurrentDirectory;
string IBuildActions.GetCurrentDirectory()
{
return GetCurrentDirectory;
}
public IDictionary<string, string> EnumerateFiles = new Dictionary<string, string>();
IEnumerable<string> IBuildActions.EnumerateFiles(string dir)
{
if (EnumerateFiles.TryGetValue(dir, out var str))
return str.Split("\n");
throw new ArgumentException("Missing EnumerateFiles " + dir);
}
public IDictionary<string, string> EnumerateDirectories = new Dictionary<string, string>();
IEnumerable<string> IBuildActions.EnumerateDirectories(string dir)
{
if (EnumerateDirectories.TryGetValue(dir, out var str))
return string.IsNullOrEmpty(str) ? Enumerable.Empty<string>() : str.Split("\n");
throw new ArgumentException("Missing EnumerateDirectories " + dir);
}
public bool IsWindows;
bool IBuildActions.IsWindows() => IsWindows;
string IBuildActions.PathCombine(params string[] parts)
{
return string.Join('\\', parts);
}
void IBuildActions.WriteAllText(string filename, string contents)
{
}
}
/// <summary>
/// A fake solution to build.
/// </summary>
class TestSolution : ISolution
{
public IEnumerable<Project> Projects => throw new NotImplementedException();
public IEnumerable<SolutionConfigurationInSolution> Configurations => throw new NotImplementedException();
public string DefaultConfigurationName => "Release";
public string DefaultPlatformName => "x86";
public string Path { get; set; }
public int ProjectCount => throw new NotImplementedException();
public Version ToolsVersion => new Version("14.0");
public TestSolution(string path)
{
Path = path;
}
}
public class BuildScriptTests
{
TestActions Actions = new TestActions();
// Records the arguments passed to StartCallback.
IList<string> StartCallbackIn = new List<string>();
void StartCallback(string s)
{
StartCallbackIn.Add(s);
}
// Records the arguments passed to EndCallback
IList<string> EndCallbackIn = new List<string>();
IList<int> EndCallbackReturn = new List<int>();
void EndCallback(int ret, string s)
{
EndCallbackReturn.Add(ret);
EndCallbackIn.Add(s);
}
[Fact]
public void TestBuildCommand()
{
var cmd = BuildScript.Create("abc", "def ghi", null, null);
Actions.RunProcess["abc def ghi"] = 1;
cmd.Run(Actions, StartCallback, EndCallback);
Assert.Equal("abc def ghi", Actions.RunProcessIn[0]);
Assert.Equal("abc def ghi", StartCallbackIn[0]);
Assert.Equal("", EndCallbackIn[0]);
Assert.Equal(1, EndCallbackReturn[0]);
}
[Fact]
public void TestAnd1()
{
var cmd = BuildScript.Create("abc", "def ghi", null, null) & BuildScript.Create("odasa", null, null, null);
Actions.RunProcess["abc def ghi"] = 1;
cmd.Run(Actions, StartCallback, EndCallback);
Assert.Equal("abc def ghi", Actions.RunProcessIn[0]);
Assert.Equal("abc def ghi", StartCallbackIn[0]);
Assert.Equal("", EndCallbackIn[0]);
Assert.Equal(1, EndCallbackReturn[0]);
}
[Fact]
public void TestAnd2()
{
var cmd = BuildScript.Create("odasa", null, null, null) & BuildScript.Create("abc", "def ghi", null, null);
Actions.RunProcess["abc def ghi"] = 1;
Actions.RunProcess["odasa "] = 0;
cmd.Run(Actions, StartCallback, EndCallback);
Assert.Equal("odasa ", Actions.RunProcessIn[0]);
Assert.Equal("odasa ", StartCallbackIn[0]);
Assert.Equal("", EndCallbackIn[0]);
Assert.Equal(0, EndCallbackReturn[0]);
Assert.Equal("abc def ghi", Actions.RunProcessIn[1]);
Assert.Equal("abc def ghi", StartCallbackIn[1]);
Assert.Equal("", EndCallbackIn[1]);
Assert.Equal(1, EndCallbackReturn[1]);
}
[Fact]
public void TestOr1()
{
var cmd = BuildScript.Create("odasa", null, null, null) | BuildScript.Create("abc", "def ghi", null, null);
Actions.RunProcess["abc def ghi"] = 1;
Actions.RunProcess["odasa "] = 0;
cmd.Run(Actions, StartCallback, EndCallback);
Assert.Equal("odasa ", Actions.RunProcessIn[0]);
Assert.Equal("odasa ", StartCallbackIn[0]);
Assert.Equal("", EndCallbackIn[0]);
Assert.Equal(0, EndCallbackReturn[0]);
Assert.Equal(1, EndCallbackReturn.Count);
}
[Fact]
public void TestOr2()
{
var cmd = BuildScript.Create("abc", "def ghi", null, null) | BuildScript.Create("odasa", null, null, null);
Actions.RunProcess["abc def ghi"] = 1;
Actions.RunProcess["odasa "] = 0;
cmd.Run(Actions, StartCallback, EndCallback);
Assert.Equal("abc def ghi", Actions.RunProcessIn[0]);
Assert.Equal("abc def ghi", StartCallbackIn[0]);
Assert.Equal("", EndCallbackIn[0]);
Assert.Equal(1, EndCallbackReturn[0]);
Assert.Equal("odasa ", Actions.RunProcessIn[1]);
Assert.Equal("odasa ", StartCallbackIn[1]);
Assert.Equal("", EndCallbackIn[1]);
Assert.Equal(0, EndCallbackReturn[1]);
}
[Fact]
public void TestSuccess()
{
Assert.Equal(0, BuildScript.Success.Run(Actions, StartCallback, EndCallback));
}
[Fact]
public void TestFailure()
{
Assert.NotEqual(0, BuildScript.Failure.Run(Actions, StartCallback, EndCallback));
}
[Fact]
public void TestDeleteDirectorySuccess()
{
Actions.DirectoryExists["trap"] = true;
Assert.Equal(0, BuildScript.DeleteDirectory("trap").Run(Actions, StartCallback, EndCallback));
Assert.Equal("trap", Actions.DirectoryDeleteIn[0]);
}
[Fact]
public void TestDeleteDirectoryFailure()
{
Actions.DirectoryExists["trap"] = false;
Assert.NotEqual(0, BuildScript.DeleteDirectory("trap").Run(Actions, StartCallback, EndCallback));
}
[Fact]
public void TestDeleteFileSuccess()
{
Actions.FileExists["csharp.log"] = true;
Assert.Equal(0, BuildScript.DeleteFile("csharp.log").Run(Actions, StartCallback, EndCallback));
Assert.Equal("csharp.log", Actions.FileExistsIn[0]);
Assert.Equal("csharp.log", Actions.FileDeleteIn[0]);
}
[Fact]
public void TestDeleteFileFailure()
{
Actions.FileExists["csharp.log"] = false;
Assert.NotEqual(0, BuildScript.DeleteFile("csharp.log").Run(Actions, StartCallback, EndCallback));
Assert.Equal("csharp.log", Actions.FileExistsIn[0]);
}
[Fact]
public void TestTry()
{
Assert.Equal(0, BuildScript.Try(BuildScript.Failure).Run(Actions, StartCallback, EndCallback));
}
Autobuilder CreateAutoBuilder(string lgtmLanguage, bool isWindows,
string buildless=null, string solution=null, string buildCommand=null, string ignoreErrors=null,
string msBuildArguments=null, string msBuildPlatform=null, string msBuildConfiguration=null, string msBuildTarget=null,
string dotnetArguments=null, string dotnetVersion=null, string vsToolsVersion=null,
string nugetRestore=null, string allSolutions=null,
string cwd=@"C:\Project")
{
Actions.GetEnvironmentVariable["SEMMLE_DIST"] = @"C:\odasa";
Actions.GetEnvironmentVariable["SEMMLE_JAVA_HOME"] = @"C:\odasa\tools\java";
Actions.GetEnvironmentVariable["LGTM_PROJECT_LANGUAGE"] = lgtmLanguage;
Actions.GetEnvironmentVariable["SEMMLE_PLATFORM_TOOLS"] = @"C:\odasa\tools";
Actions.GetEnvironmentVariable["LGTM_INDEX_VSTOOLS_VERSION"] = vsToolsVersion;
Actions.GetEnvironmentVariable["LGTM_INDEX_MSBUILD_ARGUMENTS"] = msBuildArguments;
Actions.GetEnvironmentVariable["LGTM_INDEX_MSBUILD_PLATFORM"] = msBuildPlatform;
Actions.GetEnvironmentVariable["LGTM_INDEX_MSBUILD_CONFIGURATION"] = msBuildConfiguration;
Actions.GetEnvironmentVariable["LGTM_INDEX_MSBUILD_TARGET"] = msBuildTarget;
Actions.GetEnvironmentVariable["LGTM_INDEX_DOTNET_ARGUMENTS"] = dotnetArguments;
Actions.GetEnvironmentVariable["LGTM_INDEX_DOTNET_VERSION"] = dotnetVersion;
Actions.GetEnvironmentVariable["LGTM_INDEX_BUILD_COMMAND"] = buildCommand;
Actions.GetEnvironmentVariable["LGTM_INDEX_SOLUTION"] = solution;
Actions.GetEnvironmentVariable["LGTM_INDEX_IGNORE_ERRORS"] = ignoreErrors;
Actions.GetEnvironmentVariable["LGTM_INDEX_BUILDLESS"] = buildless;
Actions.GetEnvironmentVariable["LGTM_INDEX_ALL_SOLUTIONS"] = allSolutions;
Actions.GetEnvironmentVariable["LGTM_INDEX_NUGET_RESTORE"] = nugetRestore;
Actions.GetEnvironmentVariable["ProgramFiles(x86)"] = isWindows ? @"C:\Program Files (x86)" : null;
Actions.GetCurrentDirectory = cwd;
Actions.IsWindows = isWindows;
var options = new AutobuildOptions();
options.ReadEnvironment(Actions);
return new Autobuilder(Actions, options);
}
[Fact]
public void TestDefaultCSharpAutoBuilder()
{
Actions.RunProcess["cmd.exe /C dotnet --info"] = 0;
Actions.RunProcess["cmd.exe /C dotnet clean"] = 0;
Actions.RunProcess["cmd.exe /C dotnet restore"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto dotnet build --no-incremental /p:UseSharedCompilation=false"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbar.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", true);
TestAutobuilderScript(autobuilder, 0, 6);
}
[Fact]
public void TestLinuxCSharpAutoBuilder()
{
Actions.RunProcess["dotnet --info"] = 0;
Actions.RunProcess["dotnet clean"] = 0;
Actions.RunProcess["dotnet restore"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto dotnet build --no-incremental /p:UseSharedCompilation=false"] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false);
TestAutobuilderScript(autobuilder, 0, 6);
}
[Fact]
public void TestLinuxCSharpAutoBuilderExtractorFailed()
{
Actions.RunProcess["dotnet --info"] = 0;
Actions.RunProcess["dotnet clean"] = 0;
Actions.RunProcess["dotnet restore"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto dotnet build --no-incremental /p:UseSharedCompilation=false"] = 0;
Actions.FileExists["csharp.log"] = false;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false);
TestAutobuilderScript(autobuilder, 1, 4);
}
[Fact]
public void TestDefaultCppAutobuilder()
{
Actions.EnumerateFiles[@"C:\Project"] = "";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("cpp", true);
var script = autobuilder.GetBuildScript();
// Fails due to no solutions present.
Assert.NotEqual(0, script.Run(Actions, StartCallback, EndCallback));
}
[Fact]
public void TestCppAutobuilderSuccess()
{
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\csharp\nuget\nuget.exe restore C:\Project\test.sln"] = 1;
Actions.RunProcess[@"cmd.exe /C CALL ^""C:\Program Files ^(x86^)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat^"" && C:\odasa\tools\odasa index --auto msbuild C:\Project\test.sln /p:UseSharedCompilation=false /t:rebuild /p:Platform=""x86"" /p:Configuration=""Release"" /p:MvcBuildViews=true"] = 0;
Actions.RunProcessOut[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationPath"] = "";
Actions.RunProcess[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationPath"] = 1;
Actions.RunProcess[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationVersion"] = 0;
Actions.RunProcessOut[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationVersion"] = "";
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = true;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.slx";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("cpp", true);
var solution = new TestSolution(@"C:\Project\test.sln");
autobuilder.SolutionsToBuild.Add(solution);
TestAutobuilderScript(autobuilder, 0, 2);
}
[Fact]
public void TestVsWhereSucceeded()
{
Actions.GetEnvironmentVariable["ProgramFiles(x86)"] = @"C:\Program Files (x86)";
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = true;
Actions.RunProcess[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationPath"] = 0;
Actions.RunProcessOut[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationPath"] = "C:\\VS1\nC:\\VS2";
Actions.RunProcessOut[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationVersion"] = "10.0\n11.0";
Actions.RunProcess[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationVersion"] = 0;
var candidates = BuildTools.GetCandidateVcVarsFiles(Actions).ToArray();
Assert.Equal("C:\\VS1\\VC\\vcvarsall.bat", candidates[0].Path);
Assert.Equal(10, candidates[0].ToolsVersion);
Assert.Equal("C:\\VS2\\VC\\vcvarsall.bat", candidates[1].Path);
Assert.Equal(11, candidates[1].ToolsVersion);
}
[Fact]
public void TestVsWhereNotExist()
{
Actions.GetEnvironmentVariable["ProgramFiles(x86)"] = @"C:\Program Files (x86)";
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
var candidates = BuildTools.GetCandidateVcVarsFiles(Actions).ToArray();
Assert.Equal(4, candidates.Length);
}
[Fact]
public void TestVcVarsAllBatFiles()
{
Actions.GetEnvironmentVariable["ProgramFiles(x86)"] = @"C:\Program Files (x86)";
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = false;
var vcvarsfiles = BuildTools.VcVarsAllBatFiles(Actions).ToArray();
Assert.Equal(2, vcvarsfiles.Length);
}
[Fact]
public void TestLinuxBuildlessExtractionSuccess()
{
Actions.RunProcess[@"C:\odasa\tools\csharp\Semmle.Extraction.CSharp.Standalone --references:."] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.sln";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, buildless:"true");
TestAutobuilderScript(autobuilder, 0, 3);
}
[Fact]
public void TestLinuxBuildlessExtractionFailed()
{
Actions.RunProcess[@"C:\odasa\tools\csharp\Semmle.Extraction.CSharp.Standalone --references:."] = 10;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.sln";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, buildless: "true");
TestAutobuilderScript(autobuilder, 10, 1);
}
[Fact]
public void TestLinuxBuildlessExtractionSolution()
{
Actions.RunProcess[@"C:\odasa\tools\csharp\Semmle.Extraction.CSharp.Standalone foo.sln --references:."] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.sln";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, buildless: "true", solution: "foo.sln");
TestAutobuilderScript(autobuilder, 0, 3);
}
void SkipVsWhere()
{
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = false;
}
void TestAutobuilderScript(Autobuilder autobuilder, int expectedOutput, int commandsRun)
{
Assert.Equal(expectedOutput, autobuilder.GetBuildScript().Run(Actions, StartCallback, EndCallback));
// Check expected commands actually ran
Assert.Equal(commandsRun, StartCallbackIn.Count);
Assert.Equal(commandsRun, EndCallbackIn.Count);
Assert.Equal(commandsRun, EndCallbackReturn.Count);
var action = Actions.RunProcess.GetEnumerator();
for(int cmd=0; cmd<commandsRun; ++cmd)
{
Assert.True(action.MoveNext());
Assert.Equal(action.Current.Key, StartCallbackIn[cmd]);
Assert.Equal(action.Current.Value, EndCallbackReturn[cmd]);
}
}
[Fact]
public void TestLinuxBuildCommand()
{
Actions.RunProcess["C:\\odasa\\tools\\odasa index --auto \"./build.sh --skip-tests\""] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.sln";
Actions.EnumerateDirectories[@"C:\Project"] = "";
SkipVsWhere();
var autobuilder = CreateAutoBuilder("csharp", false, buildCommand:"./build.sh --skip-tests");
TestAutobuilderScript(autobuilder, 0, 3);
}
[Fact]
public void TestLinuxBuildSh()
{
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbuild/build.sh";
Actions.EnumerateDirectories[@"C:\Project"] = "";
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.RunProcess["dotnet --info"] = 1;
Actions.RunProcess["/bin/chmod u+x build/build.sh"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto build/build.sh"] = 0;
Actions.RunProcessWorkingDirectory[@"C:\odasa\tools\odasa index --auto build/build.sh"] = "build";
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
var autobuilder = CreateAutoBuilder("csharp", false);
TestAutobuilderScript(autobuilder, 0, 5);
}
[Fact]
public void TestLinuxBuildShCSharpLogMissing()
{
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbuild.sh";
Actions.EnumerateDirectories[@"C:\Project"] = "";
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.RunProcess["dotnet --info"] = 1;
Actions.RunProcess["/bin/chmod u+x build.sh"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto build.sh"] = 0;
Actions.RunProcessWorkingDirectory[@"C:\odasa\tools\odasa index --auto build.sh"] = "";
Actions.FileExists["csharp.log"] = false;
var autobuilder = CreateAutoBuilder("csharp", false);
TestAutobuilderScript(autobuilder, 1, 3);
}
[Fact]
public void TestLinuxBuildShFailed()
{
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbuild.sh";
Actions.EnumerateDirectories[@"C:\Project"] = "";
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.RunProcess["dotnet --info"] = 1;
Actions.RunProcess["/bin/chmod u+x build.sh"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto build.sh"] = 5;
Actions.RunProcessWorkingDirectory[@"C:\odasa\tools\odasa index --auto build.sh"] = "";
Actions.FileExists["csharp.log"] = true;
var autobuilder = CreateAutoBuilder("csharp", false);
TestAutobuilderScript(autobuilder, 1, 3);
}
[Fact]
public void TestWindowsBuildBat()
{
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbuild.bat";
Actions.EnumerateDirectories[@"C:\Project"] = "";
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.RunProcess["cmd.exe /C dotnet --info"] = 1;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto build.bat"] = 0;
Actions.RunProcessWorkingDirectory[@"cmd.exe /C C:\odasa\tools\odasa index --auto build.bat"] = "";
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
var autobuilder = CreateAutoBuilder("csharp", true);
TestAutobuilderScript(autobuilder, 0, 4);
}
[Fact]
public void TestWindowsBuildBatIgnoreErrors()
{
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbuild.bat";
Actions.EnumerateDirectories[@"C:\Project"] = "";
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.RunProcess["cmd.exe /C dotnet --info"] = 1;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto build.bat"] = 1;
Actions.RunProcessWorkingDirectory[@"cmd.exe /C C:\odasa\tools\odasa index --auto build.bat"] = "";
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
var autobuilder = CreateAutoBuilder("csharp", true, ignoreErrors:"true");
TestAutobuilderScript(autobuilder, 1, 2);
}
[Fact]
public void TestWindowsCmdIgnoreErrors()
{
Actions.RunProcess["cmd.exe /C C:\\odasa\\tools\\odasa index --auto ^\"build.cmd --skip-tests^\""] = 3;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
SkipVsWhere();
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.sln";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", true, buildCommand: "build.cmd --skip-tests", ignoreErrors: "true");
TestAutobuilderScript(autobuilder, 3, 1);
}
[Fact]
public void TestWindowCSharpMsBuild()
{
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\csharp\nuget\nuget.exe restore C:\Project\test1.sln"] = 0;
Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test1.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\csharp\nuget\nuget.exe restore C:\Project\test2.sln"] = 0;
Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test2.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest1.cs\ntest2.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", true, msBuildArguments:"/P:Fu=Bar", msBuildTarget:"Windows", msBuildPlatform:"x86", msBuildConfiguration:"Debug",
vsToolsVersion:"12", allSolutions:"true");
var testSolution1 = new TestSolution(@"C:\Project\test1.sln");
var testSolution2 = new TestSolution(@"C:\Project\test2.sln");
autobuilder.SolutionsToBuild.Add(testSolution1);
autobuilder.SolutionsToBuild.Add(testSolution2);
TestAutobuilderScript(autobuilder, 0, 6);
}
[Fact]
public void TestWindowCSharpMsBuildFailed()
{
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\csharp\nuget\nuget.exe restore C:\Project\test1.sln"] = 0;
Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test1.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 1;
Actions.FileExists["csharp.log"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest1.cs\ntest2.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", true, msBuildArguments: "/P:Fu=Bar", msBuildTarget: "Windows", msBuildPlatform: "x86", msBuildConfiguration: "Debug",
vsToolsVersion: "12", allSolutions: "true");
var testSolution1 = new TestSolution(@"C:\Project\test1.sln");
var testSolution2 = new TestSolution(@"C:\Project\test2.sln");
autobuilder.SolutionsToBuild.Add(testSolution1);
autobuilder.SolutionsToBuild.Add(testSolution2);
TestAutobuilderScript(autobuilder, 1, 2);
}
[Fact]
public void TestSkipNugetMsBuild()
{
Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test1.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
Actions.RunProcess["cmd.exe /C CALL ^\"C:\\Program Files ^(x86^)\\Microsoft Visual Studio 12.0\\VC\\vcvarsall.bat^\" && C:\\odasa\\tools\\odasa index --auto msbuild C:\\Project\\test2.sln /p:UseSharedCompilation=false /t:Windows /p:Platform=\"x86\" /p:Configuration=\"Debug\" /p:MvcBuildViews=true /P:Fu=Bar"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = false;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest1.cs\ntest2.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", true, msBuildArguments: "/P:Fu=Bar", msBuildTarget: "Windows",
msBuildPlatform: "x86", msBuildConfiguration: "Debug", vsToolsVersion: "12",
allSolutions: "true", nugetRestore:"false");
var testSolution1 = new TestSolution(@"C:\Project\test1.sln");
var testSolution2 = new TestSolution(@"C:\Project\test2.sln");
autobuilder.SolutionsToBuild.Add(testSolution1);
autobuilder.SolutionsToBuild.Add(testSolution2);
TestAutobuilderScript(autobuilder, 0, 4);
}
[Fact]
public void TestSkipNugetBuildless()
{
Actions.RunProcess[@"C:\odasa\tools\csharp\Semmle.Extraction.CSharp.Standalone foo.sln --references:. --skip-nuget"] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.sln";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, buildless: "true", solution: "foo.sln", nugetRestore:"false");
TestAutobuilderScript(autobuilder, 0, 3);
}
[Fact]
public void TestSkipNugetDotnet()
{
Actions.RunProcess["dotnet --info"] = 0;
Actions.RunProcess["dotnet clean"] = 0;
Actions.RunProcess["dotnet restore"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto dotnet build --no-incremental /p:UseSharedCompilation=false --no-restore"] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, dotnetArguments:"--no-restore"); // nugetRestore=false does not work for now.
TestAutobuilderScript(autobuilder, 0, 6);
}
[Fact]
public void TestDotnetVersionNotInstalled()
{
Actions.RunProcess["dotnet --list-sdks"] = 0;
Actions.RunProcessOut["dotnet --list-sdks"] = "2.1.2 [C:\\Program Files\\dotnet\\sdks]\n2.1.4 [C:\\Program Files\\dotnet\\sdks]";
Actions.RunProcess[@"curl -sO https://dot.net/v1/dotnet-install.sh"] = 0;
Actions.RunProcess[@"chmod u+x dotnet-install.sh"] = 0;
Actions.RunProcess[@"./dotnet-install.sh --channel release --version 2.1.3 --install-dir C:\Project\.dotnet"] = 0;
Actions.RunProcess[@"C:\Project\.dotnet\dotnet --info"] = 0;
Actions.RunProcess[@"C:\Project\.dotnet\dotnet clean"] = 0;
Actions.RunProcess[@"C:\Project\.dotnet\dotnet restore"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto C:\Project\.dotnet\dotnet build --no-incremental /p:UseSharedCompilation=false"] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.GetEnvironmentVariable["PATH"] = "/bin:/usr/bin";
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, dotnetVersion:"2.1.3");
TestAutobuilderScript(autobuilder, 0, 10);
}
[Fact]
public void TestDotnetVersionAlreadyInstalled()
{
Actions.RunProcess["dotnet --list-sdks"] = 0;
Actions.RunProcessOut["dotnet --list-sdks"] = "2.1.3 [C:\\Program Files\\dotnet\\sdks]\n2.1.4 [C:\\Program Files\\dotnet\\sdks]";
Actions.RunProcess[@"curl -sO https://dot.net/v1/dotnet-install.sh"] = 0;
Actions.RunProcess[@"chmod u+x dotnet-install.sh"] = 0;
Actions.RunProcess[@"./dotnet-install.sh --channel release --version 2.1.3 --install-dir C:\Project\.dotnet"] = 0;
Actions.RunProcess[@"C:\Project\.dotnet\dotnet --info"] = 0;
Actions.RunProcess[@"C:\Project\.dotnet\dotnet clean"] = 0;
Actions.RunProcess[@"C:\Project\.dotnet\dotnet restore"] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --auto C:\Project\.dotnet\dotnet build --no-incremental /p:UseSharedCompilation=false"] = 0;
Actions.RunProcess[@"C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.GetEnvironmentVariable["PATH"] = "/bin:/usr/bin";
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\nbar.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", false, dotnetVersion: "2.1.3");
TestAutobuilderScript(autobuilder, 0, 10);
}
[Fact]
public void TestDotnetVersionWindows()
{
Actions.RunProcess["cmd.exe /C dotnet --list-sdks"] = 0;
Actions.RunProcessOut["cmd.exe /C dotnet --list-sdks"] = "2.1.3 [C:\\Program Files\\dotnet\\sdks]\n2.1.4 [C:\\Program Files\\dotnet\\sdks]";
Actions.RunProcess[@"cmd.exe /C powershell -NoProfile -ExecutionPolicy unrestricted -file C:\Project\install-dotnet.ps1 -Version 2.1.3 -InstallDir C:\Project\.dotnet"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\Project\.dotnet\dotnet --info"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\Project\.dotnet\dotnet clean"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\Project\.dotnet\dotnet restore"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --auto C:\Project\.dotnet\dotnet build --no-incremental /p:UseSharedCompilation=false"] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\java\bin\java -jar C:\odasa\tools\extractor-asp.jar ."] = 0;
Actions.RunProcess[@"cmd.exe /C C:\odasa\tools\odasa index --xml --extensions config"] = 0;
Actions.FileExists["csharp.log"] = true;
Actions.GetEnvironmentVariable["TRAP_FOLDER"] = null;
Actions.GetEnvironmentVariable["SOURCE_ARCHIVE"] = null;
Actions.GetEnvironmentVariable["PATH"] = "/bin:/usr/bin";
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.cs";
Actions.EnumerateDirectories[@"C:\Project"] = "";
var autobuilder = CreateAutoBuilder("csharp", true, dotnetVersion: "2.1.3");
TestAutobuilderScript(autobuilder, 0, 8);
}
}
}

32
csharp/extractor/Semmle.Autobuild.Tests/Properties/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,32 @@
using System.Reflection;
using System.Runtime.InteropServices;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Semmle.Autobuild.Tests")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("Semmle.Extraction.Tests")]
[assembly: AssemblyCopyright("Copyright © 2018")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

18
csharp/extractor/Semmle.Autobuild.Tests/Semmle.Autobuild.Tests.csproj Normal file
View File

@@ -0,0 +1,18 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.0</TargetFramework>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="xunit" Version="2.4.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Semmle.Autobuild\Semmle.Autobuild.csproj" />
</ItemGroup>
</Project>

20
csharp/extractor/Semmle.Autobuild/AspBuildRule.cs Normal file
View File

@@ -0,0 +1,20 @@
using System.IO;
namespace Semmle.Autobuild
{
/// <summary>
/// ASP extraction.
/// </summary>
class AspBuildRule : IBuildRule
{
public BuildScript Analyse(Autobuilder builder)
{
var command = new CommandBuilder(builder.Actions).
RunCommand(builder.Actions.PathCombine(builder.SemmleJavaHome, "bin", "java")).
Argument("-jar").
QuoteArgument(builder.Actions.PathCombine(builder.SemmleDist, "tools", "extractor-asp.jar")).
Argument(".");
return command.Script;
}
}
}

103
csharp/extractor/Semmle.Autobuild/AutobuildOptions.cs Normal file
View File

@@ -0,0 +1,103 @@
using System;
using System.Linq;
namespace Semmle.Autobuild
{
/// <summary>
/// Encapsulates build options.
/// </summary>
public class AutobuildOptions
{
public readonly int SearchDepth = 3;
public string RootDirectory = null;
static readonly string prefix = "LGTM_INDEX_";
public string VsToolsVersion;
public string MsBuildArguments;
public string MsBuildPlatform;
public string MsBuildConfiguration;
public string MsBuildTarget;
public string DotNetArguments;
public string DotNetVersion;
public string BuildCommand;
public string[] Solution;
public bool IgnoreErrors;
public bool Buildless;
public bool AllSolutions;
public bool NugetRestore;
public Language Language;
/// <summary>
/// Reads options from environment variables.
/// Throws ArgumentOutOfRangeException for invalid arguments.
/// </summary>
public void ReadEnvironment(IBuildActions actions)
{
RootDirectory = actions.GetCurrentDirectory();
VsToolsVersion = actions.GetEnvironmentVariable(prefix + "VSTOOLS_VERSION");
MsBuildArguments = actions.GetEnvironmentVariable(prefix + "MSBUILD_ARGUMENTS");
MsBuildPlatform = actions.GetEnvironmentVariable(prefix + "MSBUILD_PLATFORM");
MsBuildConfiguration = actions.GetEnvironmentVariable(prefix + "MSBUILD_CONFIGURATION");
MsBuildTarget = actions.GetEnvironmentVariable(prefix + "MSBUILD_TARGET");
DotNetArguments = actions.GetEnvironmentVariable(prefix + "DOTNET_ARGUMENTS");
DotNetVersion = actions.GetEnvironmentVariable(prefix + "DOTNET_VERSION");
BuildCommand = actions.GetEnvironmentVariable(prefix + "BUILD_COMMAND");
Solution = actions.GetEnvironmentVariable(prefix + "SOLUTION").AsList(new string[0]);
IgnoreErrors = actions.GetEnvironmentVariable(prefix + "IGNORE_ERRORS").AsBool("ignore_errors", false);
Buildless = actions.GetEnvironmentVariable(prefix + "BUILDLESS").AsBool("buildless", false);
AllSolutions = actions.GetEnvironmentVariable(prefix + "ALL_SOLUTIONS").AsBool("all_solutions", false);
NugetRestore = actions.GetEnvironmentVariable(prefix + "NUGET_RESTORE").AsBool("nuget_restore", true);
Language = actions.GetEnvironmentVariable("LGTM_PROJECT_LANGUAGE").AsLanguage();
}
}
public static class OptionsExtensions
{
public static bool AsBool(this string value, string param, bool defaultValue)
{
if (value == null) return defaultValue;
switch (value.ToLower())
{
case "on":
case "yes":
case "true":
case "enabled":
return true;
case "off":
case "no":
case "false":
case "disabled":
return false;
default:
throw new ArgumentOutOfRangeException(param, value, "The Boolean value is invalid.");
}
}
public static Language AsLanguage(this string key)
{
switch (key)
{
case null:
throw new ArgumentException("Environment variable required: LGTM_PROJECT_LANGUAGE");
case "csharp":
return Language.CSharp;
case "cpp":
return Language.Cpp;
default:
throw new ArgumentException("Language key not understood: '" + key + "'");
}
}
public static string[] AsList(this string value, string[] defaultValue)
{
if (value == null)
return defaultValue;
return value.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries).ToArray();
}
}
}

352
csharp/extractor/Semmle.Autobuild/Autobuilder.cs Normal file
View File

@@ -0,0 +1,352 @@
using Semmle.Extraction.CSharp;
using Semmle.Util.Logging;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace Semmle.Autobuild
{
/// <summary>
/// A build rule analyses the files in "builder" and outputs a build script.
/// </summary>
interface IBuildRule
{
/// <summary>
/// Analyse the files and produce a build script.
/// </summary>
/// <param name="builder">The files and options relating to the build.</param>
BuildScript Analyse(Autobuilder builder);
}
/// <summary>
/// Main application logic, containing all data
/// gathered from the project and filesystem.
///
/// The overall design is intended to be extensible so that in theory,
/// it should be possible to add new build rules without touching this code.
/// </summary>
public class Autobuilder
{
/// <summary>
/// Full file paths of files found in the project directory.
/// </summary>
public IEnumerable<string> Paths => pathsLazy.Value;
readonly Lazy<IEnumerable<string>> pathsLazy;
/// <summary>
/// Gets a list of paths matching a set of extensions
/// (including the ".").
/// </summary>
/// <param name="extensions">The extensions to find.</param>
/// <returns>The files matching the extension.</returns>
public IEnumerable<string> GetExtensions(params string[] extensions) =>
Paths.Where(p => extensions.Contains(Path.GetExtension(p)));
/// <summary>
/// Gets all paths matching a particular filename.
/// </summary>
/// <param name="name">The filename to find.</param>
/// <returns>Possibly empty sequence of paths with the given filename.</returns>
public IEnumerable<string> GetFilename(string name) => Paths.Where(p => Path.GetFileName(p) == name);
/// <summary>
/// Holds if a given path, relative to the root of the source directory
/// was found.
/// </summary>
/// <param name="path">The relative path.</param>
/// <returns>True iff the path was found.</returns>
public bool HasRelativePath(string path) => HasPath(Actions.PathCombine(RootDirectory, path));
/// <summary>
/// List of solution files to build.
/// </summary>
public IList<ISolution> SolutionsToBuild => solutionsToBuildLazy.Value;
readonly Lazy<IList<ISolution>> solutionsToBuildLazy;
/// <summary>
/// Holds if a given path was found.
/// </summary>
/// <param name="path">The path of the file.</param>
/// <returns>True iff the path was found.</returns>
public bool HasPath(string path) => Paths.Any(p => path == p);
void FindFiles(string dir, int depth, IList<string> results)
{
foreach (var f in Actions.EnumerateFiles(dir))
{
results.Add(f);
}
if (depth > 1)
{
foreach (var d in Actions.EnumerateDirectories(dir))
{
FindFiles(d, depth - 1, results);
}
}
}
/// <summary>
/// The root of the source directory.
/// </summary>
string RootDirectory => Options.RootDirectory;
/// <summary>
/// Gets the supplied build configuration.
/// </summary>
public AutobuildOptions Options { get; }
/// <summary>
/// The set of build actions used during the autobuilder.
/// Could be real system operations, or a stub for testing.
/// </summary>
public IBuildActions Actions { get; }
/// <summary>
/// Find all the relevant files and picks the best
/// solution file and tools.
/// </summary>
/// <param name="options">The command line options.</param>
public Autobuilder(IBuildActions actions, AutobuildOptions options)
{
Actions = actions;
Options = options;
pathsLazy = new Lazy<IEnumerable<string>>(() =>
{
var files = new List<string>();
FindFiles(options.RootDirectory, options.SearchDepth, files);
return files.
OrderBy(s => s.Count(c => c == Path.DirectorySeparatorChar)).
ThenBy(s => Path.GetFileName(s).Length).
ToArray();
});
solutionsToBuildLazy = new Lazy<IList<ISolution>>(() =>
{
if (options.Solution.Any())
{
var ret = new List<ISolution>();
foreach (var solution in options.Solution)
{
if (actions.FileExists(solution))
ret.Add(new Solution(this, solution));
else
Log(Severity.Error, "The specified solution file {0} was not found", solution);
}
return ret;
}
var solutions = GetExtensions(".sln").
Select(s => new Solution(this, s)).
Where(s => s.ProjectCount > 0).
OrderByDescending(s => s.ProjectCount).
ThenBy(s => s.Path.Length).
ToArray();
foreach (var sln in solutions)
{
Log(Severity.Info, $"Found {sln.Path} with {sln.ProjectCount} {this.Options.Language} projects, version {sln.ToolsVersion}, config {string.Join(" ", sln.Configurations.Select(c => c.FullName))}");
}
return new List<ISolution>(options.AllSolutions ?
solutions :
solutions.Take(1));
});
SemmleDist = Actions.GetEnvironmentVariable("SEMMLE_DIST");
SemmleJavaHome = Actions.GetEnvironmentVariable("SEMMLE_JAVA_HOME");
SemmlePlatformTools = Actions.GetEnvironmentVariable("SEMMLE_PLATFORM_TOOLS");
if (SemmleDist == null)
Log(Severity.Error, "The environment variable SEMMLE_DIST has not been set.");
}
readonly ILogger logger = new ConsoleLogger(Verbosity.Info);
/// <summary>
/// Log a given build event to the console.
/// </summary>
/// <param name="format">The format string.</param>
/// <param name="args">Inserts to the format string.</param>
public void Log(Severity severity, string format, params object[] args)
{
logger.Log(severity, format, args);
}
/// <summary>
/// Attempt to build this project.
/// </summary>
/// <returns>The exit code, 0 for success and non-zero for failures.</returns>
public int AttemptBuild()
{
Log(Severity.Info, $"Working directory: {Options.RootDirectory}");
var script = GetBuildScript();
if (Options.IgnoreErrors)
script |= BuildScript.Success;
void startCallback(string s) => Log(Severity.Info, $"\nRunning {s}");
void exitCallback(int ret, string msg) => Log(Severity.Info, $"Exit code {ret}{(string.IsNullOrEmpty(msg) ? "" : $": {msg}")}");
return script.Run(Actions, startCallback, exitCallback);
}
/// <summary>
/// Returns the build script to use for this project.
/// </summary>
public BuildScript GetBuildScript()
{
var isCSharp = Options.Language == Language.CSharp;
return isCSharp ? GetCSharpBuildScript() : GetCppBuildScript();
}
BuildScript GetCSharpBuildScript()
{
/// <summary>
/// A script that checks that the C# extractor has been executed.
/// </summary>
BuildScript CheckExtractorRun(bool warnOnFailure) =>
BuildScript.Create(actions =>
{
if (actions.FileExists(Extractor.GetCSharpLogPath()))
return 0;
if (warnOnFailure)
Log(Severity.Error, "No C# code detected during build.");
return 1;
});
var attempt = BuildScript.Failure;
switch (GetCSharpBuildStrategy())
{
case CSharpBuildStrategy.CustomBuildCommand:
attempt = new BuildCommandRule().Analyse(this) & CheckExtractorRun(true);
break;
case CSharpBuildStrategy.Buildless:
// No need to check that the extractor has been executed in buildless mode
attempt = new StandaloneBuildRule().Analyse(this);
break;
case CSharpBuildStrategy.MSBuild:
attempt = new MsBuildRule().Analyse(this) & CheckExtractorRun(true);
break;
case CSharpBuildStrategy.DotNet:
attempt = new DotNetRule().Analyse(this) & CheckExtractorRun(true);
break;
case CSharpBuildStrategy.Auto:
var cleanTrapFolder =
BuildScript.DeleteDirectory(Actions.GetEnvironmentVariable("TRAP_FOLDER"));
var cleanSourceArchive =
BuildScript.DeleteDirectory(Actions.GetEnvironmentVariable("SOURCE_ARCHIVE"));
var cleanExtractorLog =
BuildScript.DeleteFile(Extractor.GetCSharpLogPath());
var attemptExtractorCleanup =
BuildScript.Try(cleanTrapFolder) &
BuildScript.Try(cleanSourceArchive) &
BuildScript.Try(cleanExtractorLog);
/// <summary>
/// Execute script `s` and check that the C# extractor has been executed.
/// If either fails, attempt to cleanup any artifacts produced by the extractor,
/// and exit with code 1, in order to proceed to the next attempt.
/// </summary>
BuildScript IntermediateAttempt(BuildScript s) =>
(s & CheckExtractorRun(false)) |
(attemptExtractorCleanup & BuildScript.Failure);
attempt =
// First try .NET Core
IntermediateAttempt(new DotNetRule().Analyse(this)) |
// Then MSBuild
(() => IntermediateAttempt(new MsBuildRule().Analyse(this))) |
// And finally look for a script that might be a build script
(() => new BuildCommandAutoRule().Analyse(this) & CheckExtractorRun(true)) |
// All attempts failed: print message
AutobuildFailure();
break;
}
return
attempt &
(() => new AspBuildRule().Analyse(this)) &
(() => new XmlBuildRule().Analyse(this));
}
/// <summary>
/// Gets the build strategy that the autobuilder should apply, based on the
/// options in the `lgtm.yml` file.
/// </summary>
CSharpBuildStrategy GetCSharpBuildStrategy()
{
if (Options.BuildCommand != null)
return CSharpBuildStrategy.CustomBuildCommand;
if (Options.Buildless)
return CSharpBuildStrategy.Buildless;
if (Options.MsBuildArguments != null
|| Options.MsBuildConfiguration != null
|| Options.MsBuildPlatform != null
|| Options.MsBuildTarget != null)
return CSharpBuildStrategy.MSBuild;
if (Options.DotNetArguments != null || Options.DotNetVersion != null)
return CSharpBuildStrategy.DotNet;
return CSharpBuildStrategy.Auto;
}
enum CSharpBuildStrategy
{
CustomBuildCommand,
Buildless,
MSBuild,
DotNet,
Auto
}
BuildScript GetCppBuildScript()
{
if (Options.BuildCommand != null)
return new BuildCommandRule().Analyse(this);
return
// First try MSBuild
new MsBuildRule().Analyse(this) |
// Then look for a script that might be a build script
(() => new BuildCommandAutoRule().Analyse(this)) |
// All attempts failed: print message
AutobuildFailure();
}
BuildScript AutobuildFailure() =>
BuildScript.Create(actions =>
{
Log(Severity.Error, "Could not auto-detect a suitable build method");
return 1;
});
/// <summary>
/// Value of SEMMLE_DIST environment variable.
/// </summary>
public string SemmleDist { get; private set; }
/// <summary>
/// Value of SEMMLE_JAVA_HOME environment variable.
/// </summary>
public string SemmleJavaHome { get; private set; }
/// <summary>
/// Value of SEMMLE_PLATFORM_TOOLS environment variable.
/// </summary>
public string SemmlePlatformTools { get; private set; }
/// <summary>
/// The absolute path of the odasa executable.
/// </summary>
public string Odasa => SemmleDist == null ? null : Actions.PathCombine(SemmleDist, "tools", "odasa");
}
}

176
csharp/extractor/Semmle.Autobuild/BuildActions.cs Normal file
View File

@@ -0,0 +1,176 @@
using Semmle.Util;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
namespace Semmle.Autobuild
{
/// <summary>
/// Wrapper around system calls so that the build scripts can be unit-tested.
/// </summary>
public interface IBuildActions
{
/// <summary>
/// Runs a process and captures its output.
/// </summary>
/// <param name="exe">The exe to run.</param>
/// <param name="args">The other command line arguments.</param>
/// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
/// <param name="env">Additional environment variables.</param>
/// <param name="stdOut">The lines of stdout.</param>
/// <returns>The process exit code.</returns>
int RunProcess(string exe, string args, string workingDirectory, IDictionary<string, string> env, out IList<string> stdOut);
/// <summary>
/// Runs a process but does not capture its output.
/// </summary>
/// <param name="exe">The exe to run.</param>
/// <param name="args">The other command line arguments.</param>
/// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
/// <param name="env">Additional environment variables.</param>
/// <returns>The process exit code.</returns>
int RunProcess(string exe, string args, string workingDirectory, IDictionary<string, string> env);
/// <summary>
/// Tests whether a file exists, File.Exists().
/// </summary>
/// <param name="file">The filename.</param>
/// <returns>True iff the file exists.</returns>
bool FileExists(string file);
/// <summary>
/// Tests whether a directory exists, Directory.Exists().
/// </summary>
/// <param name="dir">The directory name.</param>
/// <returns>True iff the directory exists.</returns>
bool DirectoryExists(string dir);
/// <summary>
/// Deletes a file, File.Delete().
/// </summary>
/// <param name="file">The filename.</param>
void FileDelete(string file);
/// <summary>
/// Deletes a directory, Directory.Delete().
/// </summary>
void DirectoryDelete(string dir, bool recursive);
/// <summary>
/// Gets an environment variable, Environment.GetEnvironmentVariable().
/// </summary>
/// <param name="name">The name of the variable.</param>
/// <returns>The string value, or null if the variable is not defined.</returns>
string GetEnvironmentVariable(string name);
/// <summary>
/// Gets the current directory, Directory.GetCurrentDirectory().
/// </summary>
/// <returns>The current directory.</returns>
string GetCurrentDirectory();
/// <summary>
/// Enumerates files in a directory, Directory.EnumerateFiles().
/// </summary>
/// <param name="dir">The directory to enumerate.</param>
/// <returns>A list of filenames, or an empty list.</returns>
IEnumerable<string> EnumerateFiles(string dir);
/// <summary>
/// Enumerates the directories in a directory, Directory.EnumerateDirectories().
/// </summary>
/// <param name="dir">The directory to enumerate.</param>
/// <returns>List of subdirectories, or empty list.</returns>
IEnumerable<string> EnumerateDirectories(string dir);
/// <summary>
/// True if we are running on Windows.
/// </summary>
bool IsWindows();
/// <summary>
/// Combine path segments, Path.Combine().
/// </summary>
/// <param name="parts">The parts of the path.</param>
/// <returns>The combined path.</returns>
string PathCombine(params string[] parts);
/// <summary>
/// Writes contents to file, File.WriteAllText().
/// </summary>
/// <param name="filename">The filename.</param>
/// <param name="contents">The text.</param>
void WriteAllText(string filename, string contents);
}
/// <summary>
/// An implementation of IBuildActions that actually performs the requested operations.
/// </summary>
class SystemBuildActions : IBuildActions
{
void IBuildActions.FileDelete(string file) => File.Delete(file);
bool IBuildActions.FileExists(string file) => File.Exists(file);
ProcessStartInfo GetProcessStartInfo(string exe, string arguments, string workingDirectory, IDictionary<string, string> environment, bool redirectStandardOutput)
{
var pi = new ProcessStartInfo(exe, arguments)
{
UseShellExecute = false,
RedirectStandardOutput = redirectStandardOutput
};
if (workingDirectory != null)
pi.WorkingDirectory = workingDirectory;
// Environment variables can only be used when not redirecting stdout
if (!redirectStandardOutput)
{
pi.Environment["UseSharedCompilation"] = "false";
if (environment != null)
environment.ForEach(kvp => pi.Environment[kvp.Key] = kvp.Value);
}
return pi;
}
int IBuildActions.RunProcess(string cmd, string args, string workingDirectory, IDictionary<string, string> environment)
{
var pi = GetProcessStartInfo(cmd, args, workingDirectory, environment, false);
using (var p = Process.Start(pi))
{
p.WaitForExit();
return p.ExitCode;
}
}
int IBuildActions.RunProcess(string cmd, string args, string workingDirectory, IDictionary<string, string> environment, out IList<string> stdOut)
{
var pi = GetProcessStartInfo(cmd, args, workingDirectory, environment, true);
return pi.ReadOutput(out stdOut);
}
void IBuildActions.DirectoryDelete(string dir, bool recursive) => Directory.Delete(dir, recursive);
bool IBuildActions.DirectoryExists(string dir) => Directory.Exists(dir);
string IBuildActions.GetEnvironmentVariable(string name) => Environment.GetEnvironmentVariable(name);
string IBuildActions.GetCurrentDirectory() => Directory.GetCurrentDirectory();
IEnumerable<string> IBuildActions.EnumerateFiles(string dir) => Directory.EnumerateFiles(dir);
IEnumerable<string> IBuildActions.EnumerateDirectories(string dir) => Directory.EnumerateDirectories(dir);
bool IBuildActions.IsWindows() => Win32.IsWindows();
string IBuildActions.PathCombine(params string[] parts) => Path.Combine(parts);
void IBuildActions.WriteAllText(string filename, string contents) => File.WriteAllText(filename, contents);
private SystemBuildActions()
{
}
public static readonly IBuildActions Instance = new SystemBuildActions();
}
}

61
csharp/extractor/Semmle.Autobuild/BuildCommandAutoRule.cs Normal file
View File

@@ -0,0 +1,61 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Semmle.Util;
using Semmle.Util.Logging;
namespace Semmle.Autobuild
{
/// <summary>
/// Auto-detection of build scripts.
/// </summary>
class BuildCommandAutoRule : IBuildRule
{
readonly IEnumerable<string> winExtensions = new List<string> {
".bat",
".cmd",
".exe"
};
readonly IEnumerable<string> linuxExtensions = new List<string> {
"",
".sh"
};
readonly IEnumerable<string> buildScripts = new List<string> {
"build"
};
public BuildScript Analyse(Autobuilder builder)
{
builder.Log(Severity.Info, "Attempting to locate build script");
var extensions = builder.Actions.IsWindows() ? winExtensions : linuxExtensions;
var scripts = buildScripts.SelectMany(s => extensions.Select(e => s + e));
var scriptPath = builder.Paths.Where(p => scripts.Any(p.ToLower().EndsWith)).OrderBy(p => p.Length).FirstOrDefault();
if (scriptPath == null)
return BuildScript.Failure;
var chmod = new CommandBuilder(builder.Actions);
chmod.RunCommand("/bin/chmod", $"u+x {scriptPath}");
var chmodScript = builder.Actions.IsWindows() ? BuildScript.Success : BuildScript.Try(chmod.Script);
var path = Path.GetDirectoryName(scriptPath);
// A specific .NET Core version may be required
return chmodScript & DotNetRule.WithDotNet(builder, dotNet =>
{
var command = new CommandBuilder(builder.Actions, path, dotNet?.Environment);
// A specific Visual Studio version may be required
var vsTools = MsBuildRule.GetVcVarsBatFile(builder);
if (vsTools != null)
command.CallBatFile(vsTools.Path);
command.IndexCommand(builder.Odasa, scriptPath);
return command.Script;
});
}
}
}

28
csharp/extractor/Semmle.Autobuild/BuildCommandRule.cs Normal file
View File

@@ -0,0 +1,28 @@
namespace Semmle.Autobuild
{
/// <summary>
/// Execute the build_command rule.
/// </summary>
class BuildCommandRule : IBuildRule
{
public BuildScript Analyse(Autobuilder builder)
{
if (builder.Options.BuildCommand == null)
return BuildScript.Failure;
// Custom build commands may require a specific .NET Core version
return DotNetRule.WithDotNet(builder, dotNet =>
{
var command = new CommandBuilder(builder.Actions, null, dotNet?.Environment);
// Custom build commands may require a specific Visual Studio version
var vsTools = MsBuildRule.GetVcVarsBatFile(builder);
if (vsTools != null)
command.CallBatFile(vsTools.Path);
command.IndexCommand(builder.Odasa, builder.Options.BuildCommand);
return command.Script;
});
}
}
}

274
csharp/extractor/Semmle.Autobuild/BuildScript.cs Normal file
View File

@@ -0,0 +1,274 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using Semmle.Util;
namespace Semmle.Autobuild
{
/// <summary>
/// A build script.
/// </summary>
public abstract class BuildScript
{
/// <summary>
/// Run this build script.
/// </summary>
/// <param name="actions">
/// The interface used to implement the build actions.
/// </param>
/// <param name="startCallback">
/// A call back that is called every time a new process is started. The
/// argument to the call back is a textual representation of the process.
/// </param>
/// <param name="exitCallBack">
/// A call back that is called every time a new process exits. The first
/// argument to the call back is the exit code, and the second argument is
/// an exit message.
/// </param>
/// <returns>The exit code from this build script.</returns>
public abstract int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack);
/// <summary>
/// Run this build command.
/// </summary>
/// <param name="actions">
/// The interface used to implement the build actions.
/// </param>
/// <param name="startCallback">
/// A call back that is called every time a new process is started. The
/// argument to the call back is a textual representation of the process.
/// </param>
/// <param name="exitCallBack">
/// A call back that is called every time a new process exits. The first
/// argument to the call back is the exit code, and the second argument is
/// an exit message.
/// </param>
/// <param name="stdout">Contents of standard out.</param>
/// <returns>The exit code from this build script.</returns>
public abstract int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack, out IList<string> stdout);
class BuildCommand : BuildScript
{
readonly string exe, arguments, workingDirectory;
readonly IDictionary<string, string> environment;
/// <summary>
/// Create a simple build command.
/// </summary>
/// <param name="exe">The executable to run.</param>
/// <param name="argumentsOpt">The arguments to the executable, or null.</param>
/// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
/// <param name="environment">Additional environment variables.</param>
public BuildCommand(string exe, string argumentsOpt, string workingDirectory = null, IDictionary<string, string> environment = null)
{
this.exe = exe;
this.arguments = argumentsOpt ?? "";
this.workingDirectory = workingDirectory;
this.environment = environment;
}
public override string ToString() => exe + " " + arguments;
public override int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack)
{
startCallback(this.ToString());
var ret = 1;
var retMessage = "";
try
{
ret = actions.RunProcess(exe, arguments, workingDirectory, environment);
}
catch (Exception ex)
when (ex is System.ComponentModel.Win32Exception || ex is FileNotFoundException)
{
retMessage = ex.Message;
}
exitCallBack(ret, retMessage);
return ret;
}
public override int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack, out IList<string> stdout)
{
startCallback(this.ToString());
var ret = 1;
var retMessage = "";
try
{
ret = actions.RunProcess(exe, arguments, workingDirectory, environment, out stdout);
}
catch (Exception ex)
when (ex is System.ComponentModel.Win32Exception || ex is FileNotFoundException)
{
retMessage = ex.Message;
stdout = new string[0];
}
exitCallBack(ret, retMessage);
return ret;
}
}
class ReturnBuildCommand : BuildScript
{
readonly Func<IBuildActions, int> func;
public ReturnBuildCommand(Func<IBuildActions, int> func)
{
this.func = func;
}
public override int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack) => func(actions);
public override int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack, out IList<string> stdout)
{
stdout = new string[0];
return func(actions);
}
}
class BindBuildScript : BuildScript
{
readonly BuildScript s1;
readonly Func<IList<string>, int, BuildScript> s2a;
readonly Func<int, BuildScript> s2b;
public BindBuildScript(BuildScript s1, Func<IList<string>, int, BuildScript> s2)
{
this.s1 = s1;
this.s2a = s2;
}
public BindBuildScript(BuildScript s1, Func<int, BuildScript> s2)
{
this.s1 = s1;
this.s2b = s2;
}
public override int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack)
{
int ret1;
if (s2a != null)
{
ret1 = s1.Run(actions, startCallback, exitCallBack, out var stdout1);
return s2a(stdout1, ret1).Run(actions, startCallback, exitCallBack);
}
ret1 = s1.Run(actions, startCallback, exitCallBack);
return s2b(ret1).Run(actions, startCallback, exitCallBack);
}
public override int Run(IBuildActions actions, Action<string> startCallback, Action<int, string> exitCallBack, out IList<string> stdout)
{
var ret1 = s1.Run(actions, startCallback, exitCallBack, out var stdout1);
var ret2 = (s2a != null ? s2a(stdout1, ret1) : s2b(ret1)).Run(actions, startCallback, exitCallBack, out var stdout2);
var @out = new List<string>();
@out.AddRange(stdout1);
@out.AddRange(stdout2);
stdout = @out;
return ret2;
}
}
/// <summary>
/// Creates a simple build script that runs the specified exe.
/// </summary>
/// <param name="argumentsOpt">The arguments to the executable, or null.</param>
/// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
/// <param name="environment">Additional environment variables.</param>
public static BuildScript Create(string exe, string argumentsOpt, string workingDirectory, IDictionary<string, string> environment) =>
new BuildCommand(exe, argumentsOpt, workingDirectory, environment);
/// <summary>
/// Creates a simple build script that runs the specified function.
/// </summary>
public static BuildScript Create(Func<IBuildActions, int> func) =>
new ReturnBuildCommand(func);
/// <summary>
/// Creates a build script that runs <paramref name="s1"/>, followed by running the script
/// produced by <paramref name="s2"/> on the exit code from <paramref name="s1"/>.
/// </summary>
public static BuildScript Bind(BuildScript s1, Func<int, BuildScript> s2) =>
new BindBuildScript(s1, s2);
/// <summary>
/// Creates a build script that runs <paramref name="s1"/>, followed by running the script
/// produced by <paramref name="s2"/> on the exit code and standard output from
/// <paramref name="s1"/>.
/// </summary>
public static BuildScript Bind(BuildScript s1, Func<IList<string>, int, BuildScript> s2) =>
new BindBuildScript(s1, s2);
const int SuccessCode = 0;
/// <summary>
/// The empty build script that always returns exit code 0.
/// </summary>
public static readonly BuildScript Success = Create(actions => SuccessCode);
const int FailureCode = 1;
/// <summary>
/// The empty build script that always returns exit code 1.
/// </summary>
public static readonly BuildScript Failure = Create(actions => FailureCode);
static bool Succeeded(int i) => i == SuccessCode;
public static BuildScript operator &(BuildScript s1, BuildScript s2) =>
new BindBuildScript(s1, ret1 => Succeeded(ret1) ? s2 : Create(actions => ret1));
public static BuildScript operator &(BuildScript s1, Func<BuildScript> s2) =>
new BindBuildScript(s1, ret1 => Succeeded(ret1) ? s2() : Create(actions => ret1));
public static BuildScript operator |(BuildScript s1, BuildScript s2) =>
new BindBuildScript(s1, ret1 => Succeeded(ret1) ? Success : s2);
public static BuildScript operator |(BuildScript s1, Func<BuildScript> s2) =>
new BindBuildScript(s1, ret1 => Succeeded(ret1) ? Success : s2());
/// <summary>
/// Creates a build script that attempts to run the build script <paramref name="s"/>,
/// always returning a successful exit code.
/// </summary>
public static BuildScript Try(BuildScript s) => s | Success;
/// <summary>
/// Creates a build script that deletes the given directory.
/// </summary>
public static BuildScript DeleteDirectory(string dir) =>
Create(actions =>
{
if (string.IsNullOrEmpty(dir) || !actions.DirectoryExists(dir))
return FailureCode;
try
{
actions.DirectoryDelete(dir, true);
}
catch
{
return FailureCode;
}
return SuccessCode;
});
/// <summary>
/// Creates a build script that deletes the given file.
/// </summary>
public static BuildScript DeleteFile(string file) =>
Create(actions =>
{
if (string.IsNullOrEmpty(file) || !actions.FileExists(file))
return FailureCode;
try
{
actions.FileDelete(file);
}
catch
{
return FailureCode;
}
return SuccessCode;
});
}
}

99
csharp/extractor/Semmle.Autobuild/BuildTools.cs Normal file
View File

@@ -0,0 +1,99 @@
using System.Collections.Generic;
using System.Linq;
namespace Semmle.Autobuild
{
/// <summary>
/// A BAT file used to initialise the appropriate
/// Visual Studio version/platform.
/// </summary>
public class VcVarsBatFile
{
public readonly int ToolsVersion;
public readonly string Path;
public readonly string[] Platform;
public VcVarsBatFile(string path, int version, params string[] platform)
{
Path = path;
ToolsVersion = version;
Platform = platform;
}
};
/// <summary>
/// Collection of available Visual Studio build tools.
/// </summary>
public static class BuildTools
{
public static IEnumerable<VcVarsBatFile> GetCandidateVcVarsFiles(IBuildActions actions)
{
var programFilesx86 = actions.GetEnvironmentVariable("ProgramFiles(x86)");
if (programFilesx86 == null)
yield break;
// Attempt to use vswhere to find installations of Visual Studio
string vswhere = actions.PathCombine(programFilesx86, "Microsoft Visual Studio", "Installer", "vswhere.exe");
if (actions.FileExists(vswhere))
{
int exitCode1 = actions.RunProcess(vswhere, "-prerelease -legacy -property installationPath", null, null, out var installationList);
int exitCode2 = actions.RunProcess(vswhere, "-prerelease -legacy -property installationVersion", null, null, out var versionList);
if (exitCode1 == 0 && exitCode2 == 0 && versionList.Count == installationList.Count)
{
// vswhere ran successfully and produced the expected output
foreach (var vsInstallation in versionList.Zip(installationList, (v, i) => (Version: v, InstallationPath: i)))
{
var dot = vsInstallation.Version.IndexOf('.');
var majorVersionString = dot == -1 ? vsInstallation.Version : vsInstallation.Version.Substring(0, dot);
if (int.TryParse(majorVersionString, out int majorVersion))
{
if (majorVersion < 15)
{
yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"VC\vcvarsall.bat"), majorVersion, "x86");
}
else
{
yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"VC\Auxiliary\Build\vcvars32.bat"), majorVersion, "x86");
yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"VC\Auxiliary\Build\vcvars64.bat"), majorVersion, "x64");
}
}
// else: Skip installation without a version
}
yield break;
}
}
// vswhere not installed or didn't run correctly - return legacy Visual Studio versions
yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 14.0\VC\vcvarsall.bat"), 14, "x86");
yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 12.0\VC\vcvarsall.bat"), 12, "x86");
yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 11.0\VC\vcvarsall.bat"), 11, "x86");
yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 10.0\VC\vcvarsall.bat"), 10, "x86");
}
/// <summary>
/// Enumerates all available tools.
/// </summary>
public static IEnumerable<VcVarsBatFile> VcVarsAllBatFiles(IBuildActions actions) =>
GetCandidateVcVarsFiles(actions).Where(v => actions.FileExists(v.Path));
/// <summary>
/// Finds a VcVars file that provides a compatible environment for the given solution.
/// </summary>
/// <param name="sln">The solution file.</param>
/// <returns>A compatible file, or throws an exception.</returns>
public static VcVarsBatFile FindCompatibleVcVars(IBuildActions actions, ISolution sln) =>
FindCompatibleVcVars(actions, sln.ToolsVersion.Major);
/// <summary>
/// Finds a VcVars that provides a compatible environment for the given tools version.
/// </summary>
/// <param name="targetVersion">The tools version.</param>
/// <returns>A compatible file, or null.</returns>
public static VcVarsBatFile FindCompatibleVcVars(IBuildActions actions, int targetVersion) =>
targetVersion < 10 ?
VcVarsAllBatFiles(actions).OrderByDescending(b => b.ToolsVersion).FirstOrDefault() :
VcVarsAllBatFiles(actions).Where(b => b.ToolsVersion >= targetVersion).OrderBy(b => b.ToolsVersion).FirstOrDefault();
}
}

195
csharp/extractor/Semmle.Autobuild/CommandBuilder.cs Normal file
View File

@@ -0,0 +1,195 @@
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace Semmle.Autobuild
{
/// <summary>
/// Utility to construct a build command.
/// </summary>
class CommandBuilder
{
enum EscapeMode { Process, Cmd };
readonly StringBuilder arguments;
bool firstCommand;
string executable;
readonly EscapeMode escapingMode;
readonly string workingDirectory;
readonly IDictionary<string, string> environment;
/// <summary>
/// Initializes a new instance of the <see cref="T:Semmle.Autobuild.CommandBuilder"/> class.
/// </summary>
/// <param name="workingDirectory">The working directory (<code>null</code> for current directory).</param>
/// <param name="environment">Additional environment variables.</param>
public CommandBuilder(IBuildActions actions, string workingDirectory = null, IDictionary<string, string> environment = null)
{
arguments = new StringBuilder();
if (actions.IsWindows())
{
executable = "cmd.exe";
arguments.Append("/C");
escapingMode = EscapeMode.Cmd;
}
else
{
escapingMode = EscapeMode.Process;
}
firstCommand = true;
this.workingDirectory = workingDirectory;
this.environment = environment;
}
void OdasaIndex(string odasa)
{
RunCommand(odasa, "index --auto");
}
public CommandBuilder CallBatFile(string batFile, string argumentsOpt = null)
{
NextCommand();
arguments.Append(" CALL");
QuoteArgument(batFile);
Argument(argumentsOpt);
return this;
}
/// <summary>
/// Perform odasa index on a given command or BAT file.
/// </summary>
/// <param name="odasa">The odasa executable.</param>
/// <param name="command">The command to run.</param>
/// <param name="argumentsOpt">Additional arguments.</param>
/// <returns>this for chaining calls.</returns>
public CommandBuilder IndexCommand(string odasa, string command, string argumentsOpt = null)
{
OdasaIndex(odasa);
QuoteArgument(command);
Argument(argumentsOpt);
return this;
}
static readonly char[] specialChars = { ' ', '\t', '\n', '\v', '\"' };
static readonly char[] cmdMetacharacter = { '(', ')', '%', '!', '^', '\"', '<', '>', '&', '|' };
/// <summary>
/// Appends the given argument to the command line.
/// </summary>
/// <param name="argument">The argument to append.</param>
/// <param name="force">Whether to always quote the argument.</param>
/// <param name="cmd">Whether to escape for cmd.exe</param>
///
/// <remarks>
/// This implementation is copied from
/// https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/
/// </remarks>
void ArgvQuote(string argument, bool force)
{
bool cmd = escapingMode == EscapeMode.Cmd;
if (!force &&
!string.IsNullOrEmpty(argument) &&
argument.IndexOfAny(specialChars) == -1)
{
arguments.Append(argument);
}
else
{
if (cmd) arguments.Append('^');
arguments.Append('\"');
for (int it = 0; ; ++it)
{
var numBackslashes = 0;
while (it != argument.Length && argument[it] == '\\')
{
++it;
++numBackslashes;
}
if (it == argument.Length)
{
arguments.Append('\\', numBackslashes * 2);
break;
}
else if (argument[it] == '\"')
{
arguments.Append('\\', numBackslashes * 2 + 1);
if (cmd) arguments.Append('^');
arguments.Append(arguments[it]);
}
else
{
arguments.Append('\\', numBackslashes);
if (cmd && cmdMetacharacter.Any(c => c == argument[it]))
arguments.Append('^');
arguments.Append(argument[it]);
}
}
if (cmd) arguments.Append('^');
arguments.Append('\"');
}
}
public CommandBuilder QuoteArgument(string argumentsOpt)
{
if (argumentsOpt != null)
{
NextArgument();
ArgvQuote(argumentsOpt, false);
}
return this;
}
void NextArgument()
{
if (arguments.Length > 0)
arguments.Append(' ');
}
public CommandBuilder Argument(string argumentsOpt)
{
if (argumentsOpt != null)
{
NextArgument();
arguments.Append(argumentsOpt);
}
return this;
}
void NextCommand()
{
if (firstCommand)
firstCommand = false;
else
arguments.Append(" &&");
}
public CommandBuilder RunCommand(string exe, string argumentsOpt = null)
{
var (exe0, arg0) =
escapingMode == EscapeMode.Process && exe.EndsWith(".exe", System.StringComparison.Ordinal)
? ("mono", exe) // Linux
: (exe, null);
NextCommand();
if (executable == null)
{
executable = exe0;
}
else
{
QuoteArgument(exe0);
}
Argument(arg0);
Argument(argumentsOpt);
return this;
}
/// <summary>
/// Returns a build script that contains just this command.
/// </summary>
public BuildScript Script => BuildScript.Create(executable, arguments.ToString(), workingDirectory, environment);
}
}

268
csharp/extractor/Semmle.Autobuild/DotNetRule.cs Normal file
View File

@@ -0,0 +1,268 @@
using System;
using Semmle.Util.Logging;
using System.Linq;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.IO;
namespace Semmle.Autobuild
{
/// <summary>
/// A build rule where the build command is of the form "dotnet build".
/// Currently unused because the tracer does not work with dotnet.
/// </summary>
class DotNetRule : IBuildRule
{
public BuildScript Analyse(Autobuilder builder)
{
builder.Log(Severity.Info, "Attempting to build using .NET Core");
var projects = builder.SolutionsToBuild.Any()
? builder.SolutionsToBuild.SelectMany(s => s.Projects).ToArray()
: builder.GetExtensions(Language.CSharp.ProjectExtension).Select(p => new Project(builder, p)).ToArray();
var notDotNetProject = projects.FirstOrDefault(p => !p.DotNetProject);
if (notDotNetProject != null)
{
builder.Log(Severity.Info, "Not using .NET Core because of incompatible project {0}", notDotNetProject);
return BuildScript.Failure;
}
if (!builder.SolutionsToBuild.Any())
// Attempt dotnet build in root folder
return WithDotNet(builder, dotNet =>
{
var info = GetInfoCommand(builder.Actions, dotNet);
var clean = GetCleanCommand(builder.Actions, dotNet).Script;
var restore = GetRestoreCommand(builder.Actions, dotNet).Script;
var build = GetBuildCommand(builder, dotNet).Script;
return info & clean & BuildScript.Try(restore) & build;
});
// Attempt dotnet build on each solution
return WithDotNet(builder, dotNet =>
{
var ret = GetInfoCommand(builder.Actions, dotNet);
foreach (var solution in builder.SolutionsToBuild)
{
var cleanCommand = GetCleanCommand(builder.Actions, dotNet);
cleanCommand.QuoteArgument(solution.Path);
var clean = cleanCommand.Script;
var restoreCommand = GetRestoreCommand(builder.Actions, dotNet);
restoreCommand.QuoteArgument(solution.Path);
var restore = restoreCommand.Script;
var buildCommand = GetBuildCommand(builder, dotNet);
buildCommand.QuoteArgument(solution.Path);
var build = buildCommand.Script;
ret &= clean & BuildScript.Try(restore) & build;
}
return ret;
});
}
/// <summary>
/// Returns a script that attempts to download relevant version(s) of the
/// .NET Core SDK, followed by running the script generated by <paramref name="f"/>.
///
/// The first element <code>DotNetPath</code> of the argument to <paramref name="f"/>
/// is the path where .NET Core was installed, and the second element <code>Environment</code>
/// is any additional required environment variables. The tuple argument is <code>null</code>
/// when the installation failed.
/// </summary>
public static BuildScript WithDotNet(Autobuilder builder, Func<(string DotNetPath, IDictionary<string, string> Environment)?, BuildScript> f)
{
var installDir = builder.Actions.PathCombine(builder.Options.RootDirectory, ".dotnet");
var installScript = DownloadDotNet(builder, installDir);
return BuildScript.Bind(installScript, installed =>
{
if (installed == 0)
{
// The installation succeeded, so use the newly installed .NET Core
var path = builder.Actions.GetEnvironmentVariable("PATH");
var delim = builder.Actions.IsWindows() ? ";" : ":";
var env = new Dictionary<string, string>{
{ "DOTNET_MULTILEVEL_LOOKUP", "false" }, // prevent look up of other .NET Core SDKs
{ "DOTNET_SKIP_FIRST_TIME_EXPERIENCE", "true" },
{ "PATH", installDir + delim + path }
};
return f((installDir, env));
}
return f(null);
});
}
/// <summary>
/// Returns a script for downloading relevant versions of the
/// .NET Core SDK. The SDK(s) will be installed at <code>installDir</code>
/// (provided that the script succeeds).
/// </summary>
static BuildScript DownloadDotNet(Autobuilder builder, string installDir)
{
if (!string.IsNullOrEmpty(builder.Options.DotNetVersion))
// Specific version supplied in configuration: always use that
return DownloadDotNetVersion(builder, installDir, builder.Options.DotNetVersion);
// Download versions mentioned in `global.json` files
// See https://docs.microsoft.com/en-us/dotnet/core/tools/global-json
var installScript = BuildScript.Success;
var validGlobalJson = false;
foreach (var path in builder.Paths.Where(p => p.EndsWith("global.json", StringComparison.Ordinal)))
{
string version;
try
{
var o = JObject.Parse(File.ReadAllText(path));
version = (string)o["sdk"]["version"];
}
catch
{
// not a valid global.json file
continue;
}
installScript &= DownloadDotNetVersion(builder, installDir, version);
validGlobalJson = true;
}
return validGlobalJson ? installScript : BuildScript.Failure;
}
/// <summary>
/// Returns a script for downloading a specific .NET Core SDK version, if the
/// version is not already installed.
///
/// See https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-install-script.
/// </summary>
static BuildScript DownloadDotNetVersion(Autobuilder builder, string path, string version)
{
return BuildScript.Bind(GetInstalledSdksScript(builder.Actions), (sdks, sdksRet) =>
{
if (sdksRet == 0 && sdks.Count() == 1 && sdks[0].StartsWith(version + " ", StringComparison.Ordinal))
// The requested SDK is already installed (and no other SDKs are installed), so
// no need to reinstall
return BuildScript.Failure;
builder.Log(Severity.Info, "Attempting to download .NET Core {0}", version);
if (builder.Actions.IsWindows())
{
var psScript = @"param([string]$Version, [string]$InstallDir)
add-type @""
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy
{
public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem)
{
return true;
}
}
""@
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$Script = Invoke-WebRequest -useb 'https://dot.net/v1/dotnet-install.ps1'
$arguments = @{
Channel = 'release'
Version = $Version
InstallDir = $InstallDir
}
$ScriptBlock = [scriptblock]::create("".{$($Script)} $(&{$args} @arguments)"")
Invoke-Command -ScriptBlock $ScriptBlock";
var psScriptFile = builder.Actions.PathCombine(builder.Options.RootDirectory, "install-dotnet.ps1");
builder.Actions.WriteAllText(psScriptFile, psScript);
var install = new CommandBuilder(builder.Actions).
RunCommand("powershell").
Argument("-NoProfile").
Argument("-ExecutionPolicy").
Argument("unrestricted").
Argument("-file").
Argument(psScriptFile).
Argument("-Version").
Argument(version).
Argument("-InstallDir").
Argument(path);
return install.Script;
}
else
{
var curl = new CommandBuilder(builder.Actions).
RunCommand("curl").
Argument("-sO").
Argument("https://dot.net/v1/dotnet-install.sh");
var chmod = new CommandBuilder(builder.Actions).
RunCommand("chmod").
Argument("u+x").
Argument("dotnet-install.sh");
var install = new CommandBuilder(builder.Actions).
RunCommand("./dotnet-install.sh").
Argument("--channel").
Argument("release").
Argument("--version").
Argument(version).
Argument("--install-dir").
Argument(path);
return curl.Script & chmod.Script & install.Script;
}
});
}
static BuildScript GetInstalledSdksScript(IBuildActions actions)
{
var listSdks = new CommandBuilder(actions).
RunCommand("dotnet").
Argument("--list-sdks");
return listSdks.Script;
}
static string DotNetCommand(IBuildActions actions, string dotNetPath) =>
dotNetPath != null ? actions.PathCombine(dotNetPath, "dotnet") : "dotnet";
BuildScript GetInfoCommand(IBuildActions actions, (string DotNetPath, IDictionary<string, string> Environment)? arg)
{
var info = new CommandBuilder(actions, null, arg?.Environment).
RunCommand(DotNetCommand(actions, arg?.DotNetPath)).
Argument("--info");
return info.Script;
}
CommandBuilder GetCleanCommand(IBuildActions actions, (string DotNetPath, IDictionary<string, string> Environment)? arg)
{
var clean = new CommandBuilder(actions, null, arg?.Environment).
RunCommand(DotNetCommand(actions, arg?.DotNetPath)).
Argument("clean");
return clean;
}
CommandBuilder GetRestoreCommand(IBuildActions actions, (string DotNetPath, IDictionary<string, string> Environment)? arg)
{
var restore = new CommandBuilder(actions, null, arg?.Environment).
RunCommand(DotNetCommand(actions, arg?.DotNetPath)).
Argument("restore");
return restore;
}
CommandBuilder GetBuildCommand(Autobuilder builder, (string DotNetPath, IDictionary<string, string> Environment)? arg)
{
var build = new CommandBuilder(builder.Actions, null, arg?.Environment).
IndexCommand(builder.Odasa, DotNetCommand(builder.Actions, arg?.DotNetPath)).
Argument("build").
Argument("--no-incremental").
Argument("/p:UseSharedCompilation=false").
Argument(builder.Options.DotNetArguments);
return build;
}
}
}

24
csharp/extractor/Semmle.Autobuild/Language.cs Normal file
View File

@@ -0,0 +1,24 @@
namespace Semmle.Autobuild
{
public sealed class Language
{
public static readonly Language Cpp = new Language(".vcxproj");
public static readonly Language CSharp = new Language(".csproj");
public bool ProjectFileHasThisLanguage(string path) =>
System.IO.Path.GetExtension(path) == ProjectExtension;
public static bool IsProjectFileForAnySupportedLanguage(string path) =>
Cpp.ProjectFileHasThisLanguage(path) || CSharp.ProjectFileHasThisLanguage(path);
public readonly string ProjectExtension;
private Language(string extension)
{
ProjectExtension = extension;
}
public override string ToString() =>
ProjectExtension == Cpp.ProjectExtension ? "C/C++" : "C#";
}
}

118
csharp/extractor/Semmle.Autobuild/MsBuildRule.cs Normal file
View File

@@ -0,0 +1,118 @@
using Semmle.Util.Logging;
using System.IO;
using System.Linq;
namespace Semmle.Autobuild
{
/// <summary>
/// A build rule using msbuild.
/// </summary>
class MsBuildRule : IBuildRule
{
/// <summary>
/// The name of the msbuild command.
/// </summary>
const string MsBuild = "msbuild";
public BuildScript Analyse(Autobuilder builder)
{
builder.Log(Severity.Info, "Attempting to build using MSBuild");
if (!builder.SolutionsToBuild.Any())
{
builder.Log(Severity.Info, "Could not find a suitable solution file to build");
return BuildScript.Failure;
}
var vsTools = GetVcVarsBatFile(builder);
if (vsTools == null && builder.SolutionsToBuild.Any())
{
vsTools = BuildTools.FindCompatibleVcVars(builder.Actions, builder.SolutionsToBuild.First());
}
if (vsTools == null && builder.Actions.IsWindows())
{
builder.Log(Severity.Warning, "Could not find a suitable version of vcvarsall.bat");
}
var nuget = builder.Actions.PathCombine(builder.SemmlePlatformTools, "csharp", "nuget", "nuget.exe");
var ret = BuildScript.Success;
foreach (var solution in builder.SolutionsToBuild)
{
if (builder.Options.NugetRestore)
{
var nugetCommand = new CommandBuilder(builder.Actions).
RunCommand(nuget).
Argument("restore").
QuoteArgument(solution.Path);
ret &= BuildScript.Try(nugetCommand.Script);
}
var command = new CommandBuilder(builder.Actions);
if (vsTools != null)
{
command.CallBatFile(vsTools.Path);
}
command.IndexCommand(builder.Odasa, MsBuild);
command.QuoteArgument(solution.Path);
command.Argument("/p:UseSharedCompilation=false");
string target = builder.Options.MsBuildTarget != null ? builder.Options.MsBuildTarget : "rebuild";
string platform = builder.Options.MsBuildPlatform != null ? builder.Options.MsBuildPlatform : solution.DefaultPlatformName;
string configuration = builder.Options.MsBuildConfiguration != null ? builder.Options.MsBuildConfiguration : solution.DefaultConfigurationName;
command.Argument("/t:" + target);
command.Argument(string.Format("/p:Platform=\"{0}\"", platform));
command.Argument(string.Format("/p:Configuration=\"{0}\"", configuration));
command.Argument("/p:MvcBuildViews=true");
command.Argument(builder.Options.MsBuildArguments);
ret &= command.Script;
}
return ret;
}
/// <summary>
/// Gets the BAT file used to initialize the appropriate Visual Studio
/// version/platform, as specified by the `vstools_version` property in
/// lgtm.yml.
///
/// Returns <code>null</code> when no version is specified.
/// </summary>
public static VcVarsBatFile GetVcVarsBatFile(Autobuilder builder)
{
VcVarsBatFile vsTools = null;
if (builder.Options.VsToolsVersion != null)
{
if (int.TryParse(builder.Options.VsToolsVersion, out var msToolsVersion))
{
foreach (var b in BuildTools.VcVarsAllBatFiles(builder.Actions))
{
builder.Log(Severity.Info, "Found {0} version {1}", b.Path, b.ToolsVersion);
}
vsTools = BuildTools.FindCompatibleVcVars(builder.Actions, msToolsVersion);
if (vsTools == null)
builder.Log(Severity.Warning, "Could not find build tools matching version {0}", msToolsVersion);
else
builder.Log(Severity.Info, "Setting Visual Studio tools to {0}", vsTools.Path);
}
else
{
builder.Log(Severity.Error, "The format of vstools_version is incorrect. Please specify an integer.");
}
}
return vsTools;
}
}
}

28
csharp/extractor/Semmle.Autobuild/Program.cs Normal file
View File

@@ -0,0 +1,28 @@
using System;
namespace Semmle.Autobuild
{
class Program
{
static int Main()
{
var options = new AutobuildOptions();
var actions = SystemBuildActions.Instance;
try
{
options.ReadEnvironment(actions);
}
catch (ArgumentOutOfRangeException ex)
{
Console.WriteLine("The value \"{0}\" for parameter \"{1}\" is invalid", ex.ActualValue, ex.ParamName);
}
var builder = new Autobuilder(actions, options);
Console.WriteLine($"Semmle autobuilder for {options.Language}");
return builder.AttemptBuild();
}
}
}

69
csharp/extractor/Semmle.Autobuild/Project.cs Normal file
View File

@@ -0,0 +1,69 @@
using System;
using System.IO;
using System.Xml;
using Semmle.Util.Logging;
namespace Semmle.Autobuild
{
/// <summary>
/// Representation of a .csproj (C#) or .vcxproj (C++) file.
/// C# project files come in 2 flavours, .Net core and msbuild, but they
/// have the same file extension.
/// </summary>
public class Project
{
/// <summary>
/// Holds if this project is for .Net core.
/// </summary>
public bool DotNetProject { get; private set; }
public bool ValidToolsVersion { get; private set; }
public Version ToolsVersion { get; private set; }
readonly string filename;
public Project(Autobuilder builder, string filename)
{
this.filename = filename;
ToolsVersion = new Version();
if (!File.Exists(filename))
return;
var projFile = new XmlDocument();
projFile.Load(filename);
var root = projFile.DocumentElement;
if (root.Name == "Project")
{
if (root.HasAttribute("Sdk"))
{
DotNetProject = true;
}
else
{
var toolsVersion = root.GetAttribute("ToolsVersion");
if (string.IsNullOrEmpty(toolsVersion))
{
builder.Log(Severity.Warning, "Project {0} is missing a tools version", filename);
}
else
{
try
{
ToolsVersion = new Version(toolsVersion);
ValidToolsVersion = true;
}
catch // Generic catch clause - Version constructor throws about 5 different exceptions.
{
builder.Log(Severity.Warning, "Project {0} has invalid tools version {1}", filename, toolsVersion);
}
}
}
}
}
public override string ToString() => filename;
}
}

35
csharp/extractor/Semmle.Autobuild/Properties/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,35 @@
using System.Reflection;
using System.Runtime.InteropServices;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Semmle.Autobuild")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("Semmle")]
[assembly: AssemblyProduct("Semmle Visual Studio Autobuild")]
[assembly: AssemblyCopyright("Copyright © Semmle 2017")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("1d9920ad-7b00-4df1-8b01-9ff5b687828e")]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

27
csharp/extractor/Semmle.Autobuild/Semmle.Autobuild.csproj Normal file
View File

@@ -0,0 +1,27 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp2.0</TargetFramework>
<AssemblyName>Semmle.Autobuild</AssemblyName>
<RootNamespace>Semmle.Autobuild</RootNamespace>
<ApplicationIcon />
<OutputType>Exe</OutputType>
<StartupObject />
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
</PropertyGroup>
<ItemGroup>
<Folder Include="Properties\" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Build" Version="15.8.166" />
<PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Semmle.Util\Semmle.Util.csproj" />
<ProjectReference Include="..\Semmle.Extraction.CSharp\Semmle.Extraction.CSharp.csproj" />
</ItemGroup>
</Project>

107
csharp/extractor/Semmle.Autobuild/Solution.cs Normal file
View File

@@ -0,0 +1,107 @@
using Microsoft.Build.Construction;
using Microsoft.Build.Exceptions;
using System;
using System.Collections.Generic;
using System.Linq;
using Semmle.Util;
namespace Semmle.Autobuild
{
/// <summary>
/// A solution file, extension .sln.
/// </summary>
public interface ISolution
{
/// <summary>
/// List of C# or C++ projects contained in the solution.
/// (There could be other project types as well - these are ignored.)
/// </summary>
IEnumerable<Project> Projects { get; }
/// <summary>
/// Solution configurations.
/// </summary>
IEnumerable<SolutionConfigurationInSolution> Configurations { get; }
/// <summary>
/// The default configuration name, e.g. "Release"
/// </summary>
string DefaultConfigurationName { get; }
/// <summary>
/// The default platform name, e.g. "x86"
/// </summary>
string DefaultPlatformName { get; }
/// <summary>
/// The path of the solution file.
/// </summary>
string Path { get; }
/// <summary>
/// The number of C# or C++ projects.
/// </summary>
int ProjectCount { get; }
/// <summary>
/// Gets the "best" tools version for this solution.
/// If there are several versions, because the project files
/// are inconsistent, then pick the highest/latest version.
/// If no tools versions are present, return 0.0.0.0.
/// </summary>
Version ToolsVersion { get; }
}
/// <summary>
/// A solution file on the filesystem, read using Microsoft.Build.
/// </summary>
class Solution : ISolution
{
readonly SolutionFile solution;
public IEnumerable<Project> Projects { get; private set; }
public IEnumerable<SolutionConfigurationInSolution> Configurations =>
solution == null ? Enumerable.Empty<SolutionConfigurationInSolution>() : solution.SolutionConfigurations;
public string DefaultConfigurationName =>
solution == null ? "" : solution.GetDefaultConfigurationName();
public string DefaultPlatformName =>
solution == null ? "" : solution.GetDefaultPlatformName();
public Solution(Autobuilder builder, string path)
{
Path = System.IO.Path.GetFullPath(path);
try
{
solution = SolutionFile.Parse(Path);
Projects =
solution.ProjectsInOrder.
Where(p => p.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat).
Select(p => System.IO.Path.GetFullPath(FileUtils.ConvertToNative(p.AbsolutePath))).
Where(p => builder.Options.Language.ProjectFileHasThisLanguage(p)).
Select(p => new Project(builder, p)).
ToArray();
}
catch (InvalidProjectFileException)
{
// We allow specifying projects as solutions in lgtm.yml, so model
// that scenario as a solution with just that one project
Projects = Language.IsProjectFileForAnySupportedLanguage(Path)
? new[] { new Project(builder, Path) }
: new Project[0];
}
}
public string Path { get; private set; }
public int ProjectCount => Projects.Count();
IEnumerable<Version> ToolsVersions => Projects.Where(p => p.ValidToolsVersion).Select(p => p.ToolsVersion);
public Version ToolsVersion => ToolsVersions.Any() ? ToolsVersions.Max() : new Version();
}
}

46
csharp/extractor/Semmle.Autobuild/StandaloneBuildRule.cs Normal file
View File

@@ -0,0 +1,46 @@
using System.IO;
namespace Semmle.Autobuild
{
/// <summary>
/// Build using standalone extraction.
/// </summary>
class StandaloneBuildRule : IBuildRule
{
public BuildScript Analyse(Autobuilder builder)
{
BuildScript GetCommand(string solution)
{
var standalone = builder.Actions.PathCombine(builder.SemmlePlatformTools, "csharp", "Semmle.Extraction.CSharp.Standalone");
var cmd = new CommandBuilder(builder.Actions);
cmd.RunCommand(standalone);
if (solution != null)
cmd.QuoteArgument(solution);
cmd.Argument("--references:.");
if (!builder.Options.NugetRestore)
{
cmd.Argument("--skip-nuget");
}
return cmd.Script;
}
if (!builder.Options.Buildless)
return BuildScript.Failure;
var solutions = builder.Options.Solution.Length;
if (solutions == 0)
return GetCommand(null);
var script = BuildScript.Success;
foreach (var solution in builder.Options.Solution)
script &= GetCommand(solution);
return script;
}
}
}

18
csharp/extractor/Semmle.Autobuild/XmlBuildRule.cs Normal file
View File

@@ -0,0 +1,18 @@
using System.Collections.Generic;
namespace Semmle.Autobuild
{
/// <summary>
/// XML extraction.
/// </summary>
class XmlBuildRule : IBuildRule
{
public BuildScript Analyse(Autobuilder builder)
{
var command = new CommandBuilder(builder.Actions).
RunCommand(builder.Odasa).
Argument("index --xml --extensions config");
return command.Script;
}
}
}

267
csharp/extractor/Semmle.Extraction.CIL.Driver/ExtractorOptions.cs Normal file
View File

@@ -0,0 +1,267 @@
using System.Collections.Generic;
using System.Linq;
using System.IO;
using System.Reflection.PortableExecutable;
using System.Reflection.Metadata;
using System.Reflection;
using System.Runtime.InteropServices;
using System.Globalization;
using Semmle.Util.Logging;
namespace Semmle.Extraction.CIL.Driver
{
/// <summary>
/// Information about a single assembly.
/// In particular, provides references between assemblies.
/// </summary>
class AssemblyInfo
{
public override string ToString() => filename;
static AssemblyName CreateAssemblyName(MetadataReader mdReader, StringHandle name, System.Version version, StringHandle culture)
{
var cultureString = mdReader.GetString(culture);
var assemblyName = new AssemblyName()
{
Name = mdReader.GetString(name),
Version = version
};
if (cultureString != "neutral")
assemblyName.CultureInfo = CultureInfo.GetCultureInfo(cultureString);
return assemblyName;
}
static AssemblyName CreateAssemblyName(MetadataReader mdReader, AssemblyReference ar)
{
var an = CreateAssemblyName(mdReader, ar.Name, ar.Version, ar.Culture);
if (!ar.PublicKeyOrToken.IsNil)
an.SetPublicKeyToken(mdReader.GetBlobBytes(ar.PublicKeyOrToken));
return an;
}
static AssemblyName CreateAssemblyName(MetadataReader mdReader, AssemblyDefinition ad)
{
var an = CreateAssemblyName(mdReader, ad.Name, ad.Version, ad.Culture);
if (!ad.PublicKey.IsNil)
an.SetPublicKey(mdReader.GetBlobBytes(ad.PublicKey));
return an;
}
public AssemblyInfo(string path)
{
filename = path;
// Attempt to open the file and see if it's a valid assembly.
using (var stream = File.OpenRead(path))
using (var peReader = new PEReader(stream))
{
try
{
isAssembly = peReader.HasMetadata;
if (!isAssembly) return;
var mdReader = peReader.GetMetadataReader();
isAssembly = mdReader.IsAssembly;
if (!mdReader.IsAssembly) return;
// Get our own assembly name
name = CreateAssemblyName(mdReader, mdReader.GetAssemblyDefinition());
references = mdReader.AssemblyReferences.
Select(r => mdReader.GetAssemblyReference(r)).
Select(ar => CreateAssemblyName(mdReader, ar)).
ToArray();
}
catch (System.BadImageFormatException)
{
// This failed on one of the Roslyn tests that includes
// a deliberately malformed assembly.
// In this case, we just skip the extraction of this assembly.
isAssembly = false;
}
}
}
public readonly AssemblyName name;
public readonly string filename;
public bool extract;
public readonly bool isAssembly;
public readonly AssemblyName[] references;
}
/// <summary>
/// Helper to manage a collection of assemblies.
/// Resolves references between assemblies and determines which
/// additional assemblies need to be extracted.
/// </summary>
class AssemblyList
{
class AssemblyNameComparer : IEqualityComparer<AssemblyName>
{
bool IEqualityComparer<AssemblyName>.Equals(AssemblyName x, AssemblyName y) =>
x.Name == y.Name && x.Version == y.Version;
int IEqualityComparer<AssemblyName>.GetHashCode(AssemblyName obj) =>
obj.Name.GetHashCode() + 7 * obj.Version.GetHashCode();
}
readonly Dictionary<AssemblyName, AssemblyInfo> assembliesRead = new Dictionary<AssemblyName, AssemblyInfo>(new AssemblyNameComparer());
public void AddFile(string assemblyPath, bool extractAll)
{
if (!filesAnalyzed.Contains(assemblyPath))
{
filesAnalyzed.Add(assemblyPath);
var info = new AssemblyInfo(assemblyPath);
if (info.isAssembly)
{
info.extract = extractAll;
if (!assembliesRead.ContainsKey(info.name))
assembliesRead.Add(info.name, info);
}
}
}
public IEnumerable<AssemblyInfo> AssembliesToExtract => assembliesRead.Values.Where(info => info.extract);
IEnumerable<AssemblyName> AssembliesToReference => AssembliesToExtract.SelectMany(info => info.references);
public void ResolveReferences()
{
var assembliesToReference = new Stack<AssemblyName>(AssembliesToReference);
while (assembliesToReference.Any())
{
var item = assembliesToReference.Pop();
AssemblyInfo info;
if (assembliesRead.TryGetValue(item, out info))
{
if (!info.extract)
{
info.extract = true;
foreach (var reference in info.references)
assembliesToReference.Push(reference);
}
}
else
{
missingReferences.Add(item);
}
}
}
readonly HashSet<string> filesAnalyzed = new HashSet<string>();
public readonly HashSet<AssemblyName> missingReferences = new HashSet<AssemblyName>();
}
/// <summary>
/// Parses the command line and collates a list of DLLs/EXEs to extract.
/// </summary>
class ExtractorOptions
{
readonly AssemblyList assemblyList = new AssemblyList();
public void AddDirectory(string directory, bool extractAll)
{
foreach (var file in
Directory.EnumerateFiles(directory, "*.dll", SearchOption.AllDirectories).
Concat(Directory.EnumerateFiles(directory, "*.exe", SearchOption.AllDirectories)))
{
assemblyList.AddFile(file, extractAll);
}
}
void AddFrameworkDirectories(bool extractAll)
{
AddDirectory(RuntimeEnvironment.GetRuntimeDirectory(), extractAll);
}
public Verbosity Verbosity { get; private set; }
public bool NoCache { get; private set; }
public int Threads { get; private set; }
public bool PDB { get; private set; }
void AddFileOrDirectory(string path)
{
path = Path.GetFullPath(path);
if (File.Exists(path))
{
assemblyList.AddFile(path, true);
AddDirectory(Path.GetDirectoryName(path), false);
}
else if (Directory.Exists(path))
{
AddDirectory(path, true);
}
}
void ResolveReferences()
{
assemblyList.ResolveReferences();
AssembliesToExtract = assemblyList.AssembliesToExtract.ToArray();
}
public IEnumerable<AssemblyInfo> AssembliesToExtract { get; private set; }
/// <summary>
/// Gets the assemblies that were referenced but were not available to be
/// extracted. This is not an error, it just means that the database is not
/// as complete as it could be.
/// </summary>
public IEnumerable<AssemblyName> MissingReferences => assemblyList.missingReferences;
public static ExtractorOptions ParseCommandLine(string[] args)
{
var options = new ExtractorOptions();
options.Verbosity = Verbosity.Info;
options.Threads = System.Environment.ProcessorCount;
options.PDB = true;
foreach (var arg in args)
{
if (arg == "--verbose")
{
options.Verbosity = Verbosity.All;
}
else if (arg == "--silent")
{
options.Verbosity = Verbosity.Off;
}
else if (arg.StartsWith("--verbosity:"))
{
options.Verbosity = (Verbosity)int.Parse(arg.Substring(12));
}
else if (arg == "--dotnet")
{
options.AddFrameworkDirectories(true);
}
else if (arg == "--nocache")
{
options.NoCache = true;
}
else if (arg.StartsWith("--threads:"))
{
options.Threads = int.Parse(arg.Substring(10));
}
else if (arg == "--no-pdb")
{
options.PDB = false;
}
else
{
options.AddFileOrDirectory(arg);
}
}
options.AddFrameworkDirectories(false);
options.ResolveReferences();
return options;
}
}
}

68
csharp/extractor/Semmle.Extraction.CIL.Driver/Program.cs Normal file
View File

@@ -0,0 +1,68 @@
using System;
using System.Linq;
using System.Threading.Tasks;
using System.IO;
using Semmle.Util.Logging;
using System.Diagnostics;
namespace Semmle.Extraction.CIL.Driver
{
class Program
{
static void DisplayHelp()
{
Console.WriteLine("CIL command line extractor");
Console.WriteLine();
Console.WriteLine("Usage: Semmle.Extraction.CIL.Driver.exe [options] path ...");
Console.WriteLine(" --verbose Turn on verbose output");
Console.WriteLine(" --dotnet Extract the .Net Framework");
Console.WriteLine(" --nocache Overwrite existing trap files");
Console.WriteLine(" --no-pdb Do not extract PDB files");
Console.WriteLine(" path A directory/dll/exe to analyze");
}
static void ExtractAssembly(Layout layout, string assemblyPath, ILogger logger, bool nocache, bool extractPdbs)
{
string trapFile;
bool extracted;
var sw = new Stopwatch();
sw.Start();
Entities.Assembly.ExtractCIL(layout, assemblyPath, logger, nocache, extractPdbs, out trapFile, out extracted);
sw.Stop();
logger.Log(Severity.Info, " {0} ({1})", assemblyPath, sw.Elapsed);
}
static void Main(string[] args)
{
if (args.Length == 0)
{
DisplayHelp();
return;
}
var options = ExtractorOptions.ParseCommandLine(args);
var layout = new Layout();
var logger = new ConsoleLogger(options.Verbosity);
var actions = options.
AssembliesToExtract.Select(asm => asm.filename).
Select<string, Action>(filename => () => ExtractAssembly(layout, filename, logger, options.NoCache, options.PDB)).
ToArray();
foreach (var missingRef in options.MissingReferences)
logger.Log(Severity.Info, " Missing assembly " + missingRef);
var sw = new Stopwatch();
sw.Start();
var piOptions = new ParallelOptions
{
MaxDegreeOfParallelism = options.Threads
};
Parallel.Invoke(piOptions, actions);
sw.Stop();
logger.Log(Severity.Info, "Extraction completed in {0}", sw.Elapsed);
}
}
}

35
csharp/extractor/Semmle.Extraction.CIL.Driver/Properties/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,35 @@
using System.Reflection;
using System.Runtime.InteropServices;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Semmle.Extraction.CIL.Driver")]
[assembly: AssemblyDescription("Semmle CIL extractor")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("Semmle Ltd")]
[assembly: AssemblyProduct("Semmle.Extraction.CIL.Driver")]
[assembly: AssemblyCopyright("Copyright © Semmle 2017")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("5642ae68-9c26-43c9-bd3c-49923dddf02d")]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

21
csharp/extractor/Semmle.Extraction.CIL.Driver/Semmle.Extraction.CIL.Driver.csproj Normal file
View File

@@ -0,0 +1,21 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.0</TargetFramework>
<AssemblyName>Semmle.Extraction.CIL.Driver</AssemblyName>
<RootNamespace>Semmle.Extraction.CIL.Driver</RootNamespace>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
</PropertyGroup>
<ItemGroup>
<Folder Include="Properties\" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Semmle.Extraction.CIL\Semmle.Extraction.CIL.csproj" />
<ProjectReference Include="..\Semmle.Extraction\Semmle.Extraction.csproj" />
<ProjectReference Include="..\Semmle.Util\Semmle.Util.csproj" />
</ItemGroup>
</Project>

69
csharp/extractor/Semmle.Extraction.CIL/CachedFunction.cs Normal file
View File

@@ -0,0 +1,69 @@
using System;
using System.Collections.Generic;
namespace Semmle.Extraction.CIL
{
/// <summary>
/// A factory and a cache for mapping source entities to target entities.
/// Could be considered as a memoizer.
/// </summary>
/// <typeparam name="SrcType">The type of the source.</typeparam>
/// <typeparam name="TargetType">The type of the generated object.</typeparam>
public class CachedFunction<SrcType, TargetType>
{
readonly Func<SrcType, TargetType> generator;
readonly Dictionary<SrcType, TargetType> cache;
/// <summary>
/// Initializes the factory with a given mapping.
/// </summary>
/// <param name="g">The mapping.</param>
public CachedFunction(Func<SrcType, TargetType> g)
{
generator = g;
cache = new Dictionary<SrcType, TargetType>();
}
/// <summary>
/// Gets the target for a given source.
/// Create it if it does not exist.
/// </summary>
/// <param name="src">The source object.</param>
/// <returns>The created object.</returns>
public TargetType this[SrcType src]
{
get
{
TargetType result;
if (!cache.TryGetValue(src, out result))
{
result = generator(src);
cache[src] = result;
}
return result;
}
}
}
/// <summary>
/// A factory for mapping a pair of source entities to a target entity.
/// </summary>
/// <typeparam name="Src1">Source entity type 1.</typeparam>
/// <typeparam name="Src2">Source entity type 2.</typeparam>
/// <typeparam name="Target">The target type.</typeparam>
public class CachedFunction<Src1, Src2, Target>
{
readonly CachedFunction<(Src1, Src2), Target> factory;
/// <summary>
/// Initializes the factory with a given mapping.
/// </summary>
/// <param name="g">The mapping.</param>
public CachedFunction(Func<Src1, Src2, Target> g)
{
factory = new CachedFunction<(Src1, Src2), Target>(p => g(p.Item1, p.Item2));
}
public Target this[Src1 s1, Src2 s2] => factory[(s1, s2)];
}
}

173
csharp/extractor/Semmle.Extraction.CIL/Context.cs Normal file
View File

@@ -0,0 +1,173 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
namespace Semmle.Extraction.CIL
{
/// <summary>
/// Extraction context for CIL extraction.
/// Adds additional context that is specific for CIL extraction.
/// One context = one DLL/EXE.
/// </summary>
partial class Context : IDisposable
{
public Extraction.Context cx;
readonly FileStream stream;
public readonly MetadataReader mdReader;
public readonly PEReader peReader;
public readonly string assemblyPath;
public Entities.Assembly assembly;
public PDB.IPdb pdb;
public Context(Extraction.Context cx, string assemblyPath, bool extractPdbs)
{
this.cx = cx;
this.assemblyPath = assemblyPath;
stream = File.OpenRead(assemblyPath);
peReader = new PEReader(stream, PEStreamOptions.PrefetchEntireImage);
mdReader = peReader.GetMetadataReader();
TypeSignatureDecoder = new Entities.TypeSignatureDecoder(this);
globalNamespace = new Lazy<Entities.Namespace>(() => Populate(new Entities.Namespace(this, GetId(""), null)));
systemNamespace = new Lazy<Entities.Namespace>(() => Populate(new Entities.Namespace(this, "System")));
genericHandleFactory = new CachedFunction<GenericContext, Handle, ILabelledEntity>(CreateGenericHandle);
namespaceFactory = new CachedFunction<StringHandle, Entities.Namespace>(n => CreateNamespace(mdReader.GetString(n)));
namespaceDefinitionFactory = new CachedFunction<NamespaceDefinitionHandle, Entities.Namespace>(CreateNamespace);
sourceFiles = new CachedFunction<PDB.ISourceFile, Entities.PdbSourceFile>(path => new Entities.PdbSourceFile(this, path));
folders = new CachedFunction<string, Entities.Folder>(path => new Entities.Folder(this, path));
sourceLocations = new CachedFunction<PDB.Location, Entities.PdbSourceLocation>(location => new Entities.PdbSourceLocation(this, location));
defaultGenericContext = new EmptyContext(this);
var def = mdReader.GetAssemblyDefinition();
AssemblyPrefix = GetId(def.Name) + "_" + def.Version.ToString() + "::";
if (extractPdbs)
{
pdb = PDB.PdbReader.Create(assemblyPath, peReader);
if (pdb != null)
{
cx.Extractor.Logger.Log(Util.Logging.Severity.Info, string.Format("Found PDB information for {0}", assemblyPath));
}
}
}
void IDisposable.Dispose()
{
if (pdb != null)
pdb.Dispose();
peReader.Dispose();
stream.Dispose();
}
/// <summary>
/// Extract the contents of a given entity.
/// </summary>
/// <param name="entity">The entity to extract.</param>
public void Extract(IExtractedEntity entity)
{
foreach (var content in entity.Contents)
{
content.Extract(this);
}
}
public readonly Id AssemblyPrefix;
public readonly Entities.TypeSignatureDecoder TypeSignatureDecoder;
/// <summary>
/// A type used to signify something we can't handle yet.
/// Specifically, function pointers (used in C++).
/// </summary>
public Entities.Type ErrorType
{
get
{
var errorType = new Entities.ErrorType(this);
Populate(errorType);
return errorType;
}
}
/// <summary>
/// Attempt to locate debugging information for a particular method.
///
/// Returns null on failure, for example if there was no PDB information found for the
/// DLL, or if the particular method is compiler generated or doesn't come from source code.
/// </summary>
/// <param name="handle">The handle of the method.</param>
/// <returns>The debugging information, or null if the information could not be located.</returns>
public PDB.IMethod GetMethodDebugInformation(MethodDefinitionHandle handle)
{
return pdb == null ? null : pdb.GetMethod(handle.ToDebugInformationHandle());
}
}
/// <summary>
/// When we decode a type/method signature, we need access to
/// generic parameters.
/// </summary>
public abstract class GenericContext
{
public Context cx;
public GenericContext(Context cx)
{
this.cx = cx;
}
/// <summary>
/// The list of generic type parameters.
/// </summary>
public abstract IEnumerable<Entities.Type> TypeParameters { get; }
/// <summary>
/// The list of generic method parameters.
/// </summary>
public abstract IEnumerable<Entities.Type> MethodParameters { get; }
/// <summary>
/// Gets the `p`th type parameter.
/// </summary>
/// <param name="p">The index of the parameter.</param>
/// <returns>
/// For constructed types, the supplied type.
/// For unbound types, the type parameter.
/// </returns>
public Entities.Type GetGenericTypeParameter(int p)
{
return TypeParameters.ElementAt(p);
}
/// <summary>
/// Gets the `p`th method type parameter.
/// </summary>
/// <param name="p">The index of the parameter.</param>
/// <returns>
/// For constructed types, the supplied type.
/// For unbound types, the type parameter.
/// </returns>
public Entities.Type GetGenericMethodParameter(int p)
{
return MethodParameters.ElementAt(p);
}
}
/// <summary>
/// A generic context which does not contain any type parameters.
/// </summary>
public class EmptyContext : GenericContext
{
public EmptyContext(Context cx) : base(cx)
{
}
public override IEnumerable<Entities.Type> TypeParameters { get { yield break; } }
public override IEnumerable<Entities.Type> MethodParameters { get { yield break; } }
}
}

152
csharp/extractor/Semmle.Extraction.CIL/Entities/Assembly.cs Normal file
View File

@@ -0,0 +1,152 @@
using System.Reflection;
using System.Globalization;
using System.Collections.Generic;
using Semmle.Util.Logging;
using System;
namespace Semmle.Extraction.CIL.Entities
{
public interface ILocation : IEntity
{
}
interface IAssembly : ILocation
{
}
/// <summary>
/// An assembly to extract.
/// </summary>
public class Assembly : LabelledEntity, IAssembly
{
public override Id IdSuffix => suffix;
readonly File file;
readonly AssemblyName assemblyName;
public Assembly(Context cx) : base(cx)
{
cx.assembly = this;
var def = cx.mdReader.GetAssemblyDefinition();
assemblyName = new AssemblyName();
assemblyName.Name = cx.mdReader.GetString(def.Name);
assemblyName.Version = def.Version;
assemblyName.CultureInfo = new CultureInfo(cx.mdReader.GetString(def.Culture));
if (!def.PublicKey.IsNil)
assemblyName.SetPublicKey(cx.mdReader.GetBlobBytes(def.PublicKey));
ShortId = cx.GetId(assemblyName.FullName) + "#file:///" + cx.assemblyPath.Replace("\\", "/");
file = new File(cx, cx.assemblyPath);
}
static readonly Id suffix = new StringId(";assembly");
public override IEnumerable<IExtractionProduct> Contents
{
get
{
yield return file;
yield return Tuples.assemblies(this, file, assemblyName.FullName, assemblyName.Name, assemblyName.Version.ToString());
if (cx.pdb != null)
{
foreach (var f in cx.pdb.SourceFiles)
{
yield return cx.CreateSourceFile(f);
}
}
foreach (var handle in cx.mdReader.TypeDefinitions)
{
IExtractionProduct product = null;
try
{
product = cx.Create(handle);
}
catch (InternalError e)
{
cx.cx.Extractor.Message(new Message
{
exception = e,
message = "Error processing type definition",
severity = Semmle.Util.Logging.Severity.Error
});
}
// Limitation of C#: Cannot yield return inside a try-catch.
if (product != null)
yield return product;
}
foreach (var handle in cx.mdReader.MethodDefinitions)
{
IExtractionProduct product = null;
try
{
product = cx.Create(handle);
}
catch (InternalError e)
{
cx.cx.Extractor.Message(new Message
{
exception = e,
message = "Error processing bytecode",
severity = Semmle.Util.Logging.Severity.Error
});
}
if (product != null)
yield return product;
}
}
}
static void ExtractCIL(Extraction.Context cx, string assemblyPath, bool extractPdbs)
{
using (var cilContext = new Context(cx, assemblyPath, extractPdbs))
{
cilContext.Populate(new Assembly(cilContext));
cilContext.cx.PopulateAll();
}
}
/// <summary>
/// Main entry point to the CIL extractor.
/// Call this to extract a given assembly.
/// </summary>
/// <param name="layout">The trap layout.</param>
/// <param name="assemblyPath">The full path of the assembly to extract.</param>
/// <param name="logger">The logger.</param>
/// <param name="nocache">True to overwrite existing trap file.</param>
/// <param name="extractPdbs">Whether to extract PDBs.</param>
/// <param name="trapFile">The path of the trap file.</param>
/// <param name="extracted">Whether the file was extracted (false=cached).</param>
public static void ExtractCIL(Layout layout, string assemblyPath, ILogger logger, bool nocache, bool extractPdbs, out string trapFile, out bool extracted)
{
trapFile = "";
extracted = false;
try
{
var extractor = new Extractor(false, assemblyPath, logger);
var project = layout.LookupProjectOrDefault(assemblyPath);
using (var trapWriter = project.CreateTrapWriter(logger, assemblyPath + ".cil", true))
{
trapFile = trapWriter.TrapFile;
if (nocache || !System.IO.File.Exists(trapFile))
{
var cx = new Extraction.Context(extractor, null, trapWriter, null);
ExtractCIL(cx, assemblyPath, extractPdbs);
extracted = true;
}
}
}
catch (Exception ex)
{
logger.Log(Severity.Error, string.Format("Exception extracting {0}: {1}", assemblyPath, ex));
}
}
}
}

100
csharp/extractor/Semmle.Extraction.CIL/Entities/Attribute.cs Normal file
View File

@@ -0,0 +1,100 @@
using System;
using System.Collections.Generic;
using System.Reflection.Metadata;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// A CIL attribute.
/// </summary>
interface IAttribute : IExtractedEntity
{
}
/// <summary>
/// Entity representing a CIL attribute.
/// </summary>
class Attribute : UnlabelledEntity, IAttribute
{
readonly CustomAttribute attrib;
readonly IEntity @object;
public Attribute(Context cx, IEntity @object, CustomAttributeHandle handle) : base(cx)
{
attrib = cx.mdReader.GetCustomAttribute(handle);
this.@object = @object;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
var constructor = (Method)cx.Create(attrib.Constructor);
yield return constructor;
yield return Tuples.cil_attribute(this, @object, constructor);
CustomAttributeValue<Type> decoded;
try
{
decoded = attrib.DecodeValue(new CustomAttributeDecoder(cx));
}
catch (NotImplementedException)
{
// Attribute decoding is only partial at this stage.
yield break;
}
for (int index = 0; index < decoded.FixedArguments.Length; ++index)
{
object value = decoded.FixedArguments[index].Value;
yield return Tuples.cil_attribute_positional_argument(this, index, value == null ? "null" : value.ToString());
}
foreach (var p in decoded.NamedArguments)
{
object value = p.Value;
yield return Tuples.cil_attribute_named_argument(this, p.Name, value == null ? "null" : value.ToString());
}
}
}
public static IEnumerable<IExtractionProduct> Populate(Context cx, IEntity @object, CustomAttributeHandleCollection attributes)
{
foreach (var attrib in attributes)
{
yield return new Attribute(cx, @object, attrib);
}
}
}
/// <summary>
/// Helper class to decode the attribute structure.
/// Note that there are some unhandled cases that should be fixed in due course.
/// </summary>
class CustomAttributeDecoder : ICustomAttributeTypeProvider<Type>
{
readonly Context cx;
public CustomAttributeDecoder(Context cx) { this.cx = cx; }
public Type GetPrimitiveType(PrimitiveTypeCode typeCode) => cx.Populate(new PrimitiveType(cx, typeCode));
public Type GetSystemType() => throw new NotImplementedException();
public Type GetSZArrayType(Type elementType) =>
cx.Populate(new ArrayType(cx, elementType));
public Type GetTypeFromDefinition(MetadataReader reader, TypeDefinitionHandle handle, byte rawTypeKind) =>
(Type)cx.Create(handle);
public Type GetTypeFromReference(MetadataReader reader, TypeReferenceHandle handle, byte rawTypeKind) =>
(Type)cx.Create(handle);
public Type GetTypeFromSerializedName(string name) => throw new NotImplementedException();
public PrimitiveTypeCode GetUnderlyingEnumType(Type type) => throw new NotImplementedException();
public bool IsSystemType(Type type) => type is PrimitiveType; // ??
}
}

67
csharp/extractor/Semmle.Extraction.CIL/Entities/Event.cs Normal file
View File

@@ -0,0 +1,67 @@
using System.Collections.Generic;
using System.Reflection.Metadata;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// An event.
/// </summary>
interface IEvent : ILabelledEntity
{
}
/// <summary>
/// An event entity.
/// </summary>
class Event : LabelledEntity, IEvent
{
readonly Type parent;
readonly EventDefinition ed;
static readonly Id suffix = CIL.Id.Create(";cil-event");
public Event(Context cx, Type parent, EventDefinitionHandle handle) : base(cx)
{
this.parent = parent;
ed = cx.mdReader.GetEventDefinition(handle);
ShortId = parent.ShortId + cx.Dot + cx.ShortName(ed.Name) + suffix;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
var signature = (Type)cx.CreateGeneric(parent, ed.Type);
yield return signature;
yield return Tuples.cil_event(this, parent, cx.ShortName(ed.Name), signature);
var accessors = ed.GetAccessors();
if (!accessors.Adder.IsNil)
{
var adder = (Method)cx.CreateGeneric(parent, accessors.Adder);
yield return adder;
yield return Tuples.cil_adder(this, adder);
}
if (!accessors.Remover.IsNil)
{
var remover = (Method)cx.CreateGeneric(parent, accessors.Remover);
yield return remover;
yield return Tuples.cil_remover(this, remover);
}
if (!accessors.Raiser.IsNil)
{
var raiser = (Method)cx.CreateGeneric(parent, accessors.Raiser);
yield return raiser;
yield return Tuples.cil_raiser(this, raiser);
}
foreach (var c in Attribute.Populate(cx, this, ed.GetCustomAttributes()))
yield return c;
}
}
public override Id IdSuffix => suffix;
}
}

63
csharp/extractor/Semmle.Extraction.CIL/Entities/ExceptionRegion.cs Normal file
View File

@@ -0,0 +1,63 @@
using System.Collections.Generic;
namespace Semmle.Extraction.CIL.Entities
{
interface IExceptionRegion : IExtractedEntity
{
}
/// <summary>
/// An exception region entity.
/// </summary>
class ExceptionRegion : UnlabelledEntity, IExceptionRegion
{
readonly GenericContext gc;
readonly MethodImplementation method;
readonly int index;
readonly System.Reflection.Metadata.ExceptionRegion r;
readonly Dictionary<int, IInstruction> jump_table;
public ExceptionRegion(GenericContext gc, MethodImplementation method, int index, System.Reflection.Metadata.ExceptionRegion r, Dictionary<int, IInstruction> jump_table) : base(gc.cx)
{
this.gc = gc;
this.method = method;
this.index = index;
this.r = r;
this.jump_table = jump_table;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
IInstruction try_start, try_end, handler_start;
if (!jump_table.TryGetValue(r.TryOffset, out try_start))
throw new InternalError("Failed to retrieve handler");
if (!jump_table.TryGetValue(r.TryOffset + r.TryLength, out try_end))
throw new InternalError("Failed to retrieve handler");
if (!jump_table.TryGetValue(r.HandlerOffset, out handler_start))
throw new InternalError("Failed to retrieve handler");
yield return Tuples.cil_handler(this, method, index, (int)r.Kind, try_start, try_end, handler_start);
if (r.FilterOffset != -1)
{
IInstruction filter_start;
if (!jump_table.TryGetValue(r.FilterOffset, out filter_start))
throw new InternalError("ExceptionRegion filter clause");
yield return Tuples.cil_handler_filter(this, filter_start);
}
if (!r.CatchType.IsNil)
{
var catchType = (Type)cx.CreateGeneric(gc, r.CatchType);
yield return catchType;
yield return Tuples.cil_handler_type(this, catchType);
}
}
}
}
}

147
csharp/extractor/Semmle.Extraction.CIL/Entities/Field.cs Normal file
View File

@@ -0,0 +1,147 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using System.Reflection.Metadata;
using System.Reflection;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// An entity represting a member.
/// Used to type tuples correctly.
/// </summary>
interface IMember : ILabelledEntity
{
}
/// <summary>
/// An entity representing a field.
/// </summary>
interface IField : IMember
{
}
/// <summary>
/// An entity representing a field.
/// </summary>
abstract class Field : GenericContext, IField
{
protected Field(Context cx) : base(cx)
{
}
public bool NeedsPopulation { get { return true; } }
public Label Label { get; set; }
public IId Id => ShortId + IdSuffix;
public Id IdSuffix => fieldSuffix;
static readonly StringId fieldSuffix = new StringId(";cil-field");
public Id ShortId
{
get; set;
}
public abstract Id Name { get; }
public abstract Type DeclaringType { get; }
public Location ReportingLocation => throw new NotImplementedException();
abstract public Type Type { get; }
public virtual IEnumerable<IExtractionProduct> Contents
{
get
{
yield return Tuples.cil_field(this, DeclaringType, Name.Value, Type);
}
}
public void Extract(Context cx)
{
cx.Populate(this);
}
TrapStackBehaviour IEntity.TrapStackBehaviour => TrapStackBehaviour.NoLabel;
}
sealed class DefinitionField : Field
{
readonly FieldDefinition fd;
readonly GenericContext gc;
public DefinitionField(GenericContext gc, FieldDefinitionHandle handle) : base(gc.cx)
{
this.gc = gc;
fd = cx.mdReader.GetFieldDefinition(handle);
ShortId = DeclaringType.ShortId + cx.Dot + Name;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
foreach (var c in base.Contents)
yield return c;
if (fd.Attributes.HasFlag(FieldAttributes.Private))
yield return Tuples.cil_private(this);
if (fd.Attributes.HasFlag(FieldAttributes.Public))
yield return Tuples.cil_public(this);
if (fd.Attributes.HasFlag(FieldAttributes.Family))
yield return Tuples.cil_protected(this);
if (fd.Attributes.HasFlag(FieldAttributes.Static))
yield return Tuples.cil_static(this);
if (fd.Attributes.HasFlag(FieldAttributes.Assembly))
yield return Tuples.cil_internal(this);
foreach (var c in Attribute.Populate(cx, this, fd.GetCustomAttributes()))
yield return c;
}
}
public override Id Name => cx.GetId(fd.Name);
public override Type DeclaringType => (Type)cx.Create(fd.GetDeclaringType());
public override Type Type => fd.DecodeSignature(cx.TypeSignatureDecoder, DeclaringType);
public override IEnumerable<Type> TypeParameters => throw new NotImplementedException();
public override IEnumerable<Type> MethodParameters => throw new NotImplementedException();
}
sealed class MemberReferenceField : Field
{
readonly MemberReference mr;
readonly GenericContext gc;
readonly Type declType;
public MemberReferenceField(GenericContext gc, MemberReferenceHandle handle) : base(gc.cx)
{
this.gc = gc;
mr = cx.mdReader.GetMemberReference(handle);
declType = (Type)cx.CreateGeneric(gc, mr.Parent);
ShortId = declType.ShortId + cx.Dot + Name;
}
public override Id Name => cx.GetId(mr.Name);
public override Type DeclaringType => declType;
public override Type Type => mr.DecodeFieldSignature(cx.TypeSignatureDecoder, this);
public override IEnumerable<Type> TypeParameters => gc.TypeParameters.Concat(declType.TypeParameters);
public override IEnumerable<Type> MethodParameters => gc.MethodParameters.Concat(declType.MethodParameters);
}
}

66
csharp/extractor/Semmle.Extraction.CIL/Entities/File.cs Normal file
View File

@@ -0,0 +1,66 @@
using System.Collections.Generic;
namespace Semmle.Extraction.CIL.Entities
{
interface IFileOrFolder : IEntity
{
}
interface IFile : IFileOrFolder
{
}
public class File : LabelledEntity, IFile
{
protected readonly string path;
public File(Context cx, string path) : base(cx)
{
this.path = path.Replace("\\", "/");
ShortId = new StringId(path.Replace(":", "_"));
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
var parent = cx.CreateFolder(System.IO.Path.GetDirectoryName(path));
yield return parent;
yield return Tuples.containerparent(parent, this);
yield return Tuples.files(this, path, System.IO.Path.GetFileNameWithoutExtension(path), System.IO.Path.GetExtension(path).Substring(1));
}
}
public override Id IdSuffix => suffix;
static readonly Id suffix = new StringId(";sourcefile");
}
public class PdbSourceFile : File
{
readonly PDB.ISourceFile file;
public PdbSourceFile(Context cx, PDB.ISourceFile file) : base(cx, file.Path)
{
this.file = file;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
foreach (var c in base.Contents)
yield return c;
var text = file.Contents;
if (text == null)
cx.cx.Extractor.Logger.Log(Util.Logging.Severity.Warning, string.Format("PDB source file {0} could not be found", path));
else
cx.cx.TrapWriter.Archive(path, text);
yield return Tuples.file_extraction_mode(this, 2);
}
}
}
}

42
csharp/extractor/Semmle.Extraction.CIL/Entities/Folder.cs Normal file
View File

@@ -0,0 +1,42 @@
using System.Collections.Generic;
using System.IO;
namespace Semmle.Extraction.CIL.Entities
{
interface IFolder : IFileOrFolder
{
}
public class Folder : LabelledEntity, IFolder
{
readonly string path;
public Folder(Context cx, string path) : base(cx)
{
this.path = path;
ShortId = new StringId(path.Replace("\\", "/").Replace(":", "_"));
}
static readonly Id suffix = new StringId(";folder");
public override IEnumerable<IExtractionProduct> Contents
{
get
{
// On Posix, we could get a Windows directory of the form "C:"
bool windowsDriveLetter = path.Length == 2 && char.IsLetter(path[0]) && path[1] == ':';
var parent = Path.GetDirectoryName(path);
if (parent != null && !windowsDriveLetter)
{
var parentFolder = cx.CreateFolder(parent);
yield return parentFolder;
yield return Tuples.containerparent(parentFolder, this);
}
yield return Tuples.folders(this, path, Path.GetFileName(path));
}
}
public override Id IdSuffix => suffix;
}
}

486
csharp/extractor/Semmle.Extraction.CIL/Entities/Instruction.cs Normal file
View File

@@ -0,0 +1,486 @@
using System;
using System.Collections.Generic;
using System.Reflection.Metadata;
using System.Reflection.Metadata.Ecma335;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// A CIL instruction.
/// </summary>
interface IInstruction : IExtractedEntity
{
/// <summary>
/// Gets the extraction products for branches.
/// </summary>
/// <param name="jump_table">The map from offset to instruction.</param>
/// <returns>The extraction products.</returns>
IEnumerable<IExtractionProduct> JumpContents(Dictionary<int, IInstruction> jump_table);
}
/// <summary>
/// A CIL instruction.
/// </summary>
class Instruction : UnlabelledEntity, IInstruction
{
/// <summary>
/// The additional data following the opcode, if any.
/// </summary>
public enum Payload
{
None, TypeTok, Field, Target8, Class,
Method, Arg8, Local8, Target32, Int8,
Int16, Int32, Int64, Float32, Float64,
CallSiteDesc, Switch, String, Constructor, ValueType,
Type, Arg16, Ignore8, Token, Local16, MethodRef
}
/// <summary>
/// For each Payload, how many additional bytes in the bytestream need to be read.
/// </summary>
internal static readonly int[] payloadSizes = {
0, 4, 4, 1, 4,
4, 1, 1, 4, 1,
2, 4, 8, 4, 8,
4, -1, 4, 4, 4,
4, 2, 1, 4, 2, 4 };
// Maps opcodes to payloads for each instruction.
public static readonly Dictionary<ILOpCode, Payload> opPayload = new Dictionary<ILOpCode, Payload>()
{
{ ILOpCode.Nop, Payload.None },
{ ILOpCode.Break, Payload.None },
{ ILOpCode.Ldarg_0, Payload.None },
{ ILOpCode.Ldarg_1, Payload.None },
{ ILOpCode.Ldarg_2, Payload.None },
{ ILOpCode.Ldarg_3, Payload.None },
{ ILOpCode.Ldloc_0, Payload.None },
{ ILOpCode.Ldloc_1, Payload.None },
{ ILOpCode.Ldloc_2, Payload.None },
{ ILOpCode.Ldloc_3, Payload.None },
{ ILOpCode.Stloc_0, Payload.None },
{ ILOpCode.Stloc_1, Payload.None },
{ ILOpCode.Stloc_2, Payload.None },
{ ILOpCode.Stloc_3, Payload.None },
{ ILOpCode.Ldarg_s, Payload.Arg8 },
{ ILOpCode.Ldarga_s, Payload.Arg8 },
{ ILOpCode.Starg_s, Payload.Arg8 },
{ ILOpCode.Ldloc_s, Payload.Local8 },
{ ILOpCode.Ldloca_s, Payload.Local8 },
{ ILOpCode.Stloc_s, Payload.Local8 },
{ ILOpCode.Ldnull, Payload.None },
{ ILOpCode.Ldc_i4_m1, Payload.None },
{ ILOpCode.Ldc_i4_0, Payload.None },
{ ILOpCode.Ldc_i4_1, Payload.None },
{ ILOpCode.Ldc_i4_2, Payload.None },
{ ILOpCode.Ldc_i4_3, Payload.None },
{ ILOpCode.Ldc_i4_4, Payload.None },
{ ILOpCode.Ldc_i4_5, Payload.None },
{ ILOpCode.Ldc_i4_6, Payload.None },
{ ILOpCode.Ldc_i4_7, Payload.None },
{ ILOpCode.Ldc_i4_8, Payload.None },
{ ILOpCode.Ldc_i4_s, Payload.Int8 },
{ ILOpCode.Ldc_i4, Payload.Int32 },
{ ILOpCode.Ldc_i8, Payload.Int64 },
{ ILOpCode.Ldc_r4, Payload.Float32 },
{ ILOpCode.Ldc_r8, Payload.Float64 },
{ ILOpCode.Dup, Payload.None },
{ ILOpCode.Pop, Payload.None },
{ ILOpCode.Jmp, Payload.Method },
{ ILOpCode.Call, Payload.Method },
{ ILOpCode.Calli, Payload.CallSiteDesc },
{ ILOpCode.Ret, Payload.None },
{ ILOpCode.Br_s, Payload.Target8 },
{ ILOpCode.Brfalse_s, Payload.Target8 },
{ ILOpCode.Brtrue_s, Payload.Target8 },
{ ILOpCode.Beq_s, Payload.Target8 },
{ ILOpCode.Bge_s, Payload.Target8 },
{ ILOpCode.Bgt_s, Payload.Target8 },
{ ILOpCode.Ble_s, Payload.Target8 },
{ ILOpCode.Blt_s, Payload.Target8 },
{ ILOpCode.Bne_un_s, Payload.Target8 },
{ ILOpCode.Bge_un_s, Payload.Target8 },
{ ILOpCode.Bgt_un_s, Payload.Target8 },
{ ILOpCode.Ble_un_s, Payload.Target8 },
{ ILOpCode.Blt_un_s, Payload.Target8 },
{ ILOpCode.Br, Payload.Target32 },
{ ILOpCode.Brfalse, Payload.Target32 },
{ ILOpCode.Brtrue, Payload.Target32 },
{ ILOpCode.Beq, Payload.Target32 },
{ ILOpCode.Bge, Payload.Target32 },
{ ILOpCode.Bgt, Payload.Target32 },
{ ILOpCode.Ble, Payload.Target32 },
{ ILOpCode.Blt, Payload.Target32 },
{ ILOpCode.Bne_un, Payload.Target32 },
{ ILOpCode.Bge_un, Payload.Target32 },
{ ILOpCode.Bgt_un, Payload.Target32 },
{ ILOpCode.Ble_un, Payload.Target32 },
{ ILOpCode.Blt_un, Payload.Target32 },
{ ILOpCode.Switch, Payload.Switch },
{ ILOpCode.Ldind_i1, Payload.None },
{ ILOpCode.Ldind_u1, Payload.None },
{ ILOpCode.Ldind_i2, Payload.None },
{ ILOpCode.Ldind_u2, Payload.None },
{ ILOpCode.Ldind_i4, Payload.None },
{ ILOpCode.Ldind_u4, Payload.None },
{ ILOpCode.Ldind_i8, Payload.None },
{ ILOpCode.Ldind_i, Payload.None },
{ ILOpCode.Ldind_r4, Payload.None },
{ ILOpCode.Ldind_r8, Payload.None },
{ ILOpCode.Ldind_ref, Payload.None },
{ ILOpCode.Stind_ref, Payload.None },
{ ILOpCode.Stind_i1, Payload.None },
{ ILOpCode.Stind_i2, Payload.None },
{ ILOpCode.Stind_i4, Payload.None },
{ ILOpCode.Stind_i8, Payload.None },
{ ILOpCode.Stind_r4, Payload.None },
{ ILOpCode.Stind_r8, Payload.None },
{ ILOpCode.Add, Payload.None },
{ ILOpCode.Sub, Payload.None },
{ ILOpCode.Mul, Payload.None },
{ ILOpCode.Div, Payload.None },
{ ILOpCode.Div_un, Payload.None },
{ ILOpCode.Rem, Payload.None },
{ ILOpCode.Rem_un, Payload.None },
{ ILOpCode.And, Payload.None },
{ ILOpCode.Or, Payload.None },
{ ILOpCode.Xor, Payload.None },
{ ILOpCode.Shl, Payload.None },
{ ILOpCode.Shr, Payload.None },
{ ILOpCode.Shr_un, Payload.None },
{ ILOpCode.Neg, Payload.None },
{ ILOpCode.Not, Payload.None },
{ ILOpCode.Conv_i1, Payload.None },
{ ILOpCode.Conv_i2, Payload.None },
{ ILOpCode.Conv_i4, Payload.None },
{ ILOpCode.Conv_i8, Payload.None },
{ ILOpCode.Conv_r4, Payload.None },
{ ILOpCode.Conv_r8, Payload.None },
{ ILOpCode.Conv_u4, Payload.None },
{ ILOpCode.Conv_u8, Payload.None },
{ ILOpCode.Callvirt, Payload.MethodRef },
{ ILOpCode.Cpobj, Payload.TypeTok },
{ ILOpCode.Ldobj, Payload.TypeTok },
{ ILOpCode.Ldstr, Payload.String },
{ ILOpCode.Newobj, Payload.Constructor },
{ ILOpCode.Castclass, Payload.Class },
{ ILOpCode.Isinst, Payload.Class },
{ ILOpCode.Conv_r_un, Payload.None },
{ ILOpCode.Unbox, Payload.ValueType },
{ ILOpCode.Throw, Payload.None },
{ ILOpCode.Ldfld, Payload.Field },
{ ILOpCode.Ldflda, Payload.Field },
{ ILOpCode.Stfld, Payload.Field },
{ ILOpCode.Ldsfld, Payload.Field },
{ ILOpCode.Ldsflda, Payload.Field },
{ ILOpCode.Stsfld, Payload.Field },
{ ILOpCode.Stobj, Payload.Field },
{ ILOpCode.Conv_ovf_i1_un, Payload.None },
{ ILOpCode.Conv_ovf_i2_un, Payload.None },
{ ILOpCode.Conv_ovf_i4_un, Payload.None },
{ ILOpCode.Conv_ovf_i8_un, Payload.None },
{ ILOpCode.Conv_ovf_u1_un, Payload.None },
{ ILOpCode.Conv_ovf_u2_un, Payload.None },
{ ILOpCode.Conv_ovf_u4_un, Payload.None },
{ ILOpCode.Conv_ovf_u8_un, Payload.None },
{ ILOpCode.Conv_ovf_i_un, Payload.None },
{ ILOpCode.Conv_ovf_u_un, Payload.None },
{ ILOpCode.Box, Payload.TypeTok },
{ ILOpCode.Newarr, Payload.TypeTok },
{ ILOpCode.Ldlen, Payload.None },
{ ILOpCode.Ldelema, Payload.Class },
{ ILOpCode.Ldelem_i1, Payload.None },
{ ILOpCode.Ldelem_u1, Payload.None },
{ ILOpCode.Ldelem_i2, Payload.None },
{ ILOpCode.Ldelem_u2, Payload.None },
{ ILOpCode.Ldelem_i4, Payload.None },
{ ILOpCode.Ldelem_u4, Payload.None },
{ ILOpCode.Ldelem_i8, Payload.None },
{ ILOpCode.Ldelem_i, Payload.None },
{ ILOpCode.Ldelem_r4, Payload.None },
{ ILOpCode.Ldelem_r8, Payload.None },
{ ILOpCode.Ldelem_ref, Payload.None },
{ ILOpCode.Stelem_i, Payload.None },
{ ILOpCode.Stelem_i1, Payload.None },
{ ILOpCode.Stelem_i2, Payload.None },
{ ILOpCode.Stelem_i4, Payload.None },
{ ILOpCode.Stelem_i8, Payload.None },
{ ILOpCode.Stelem_r4, Payload.None },
{ ILOpCode.Stelem_r8, Payload.None },
{ ILOpCode.Stelem_ref, Payload.None },
{ ILOpCode.Ldelem, Payload.TypeTok },
{ ILOpCode.Stelem, Payload.TypeTok },
{ ILOpCode.Unbox_any, Payload.TypeTok },
{ ILOpCode.Conv_ovf_i1, Payload.None },
{ ILOpCode.Conv_ovf_u1, Payload.None },
{ ILOpCode.Conv_ovf_i2, Payload.None },
{ ILOpCode.Conv_ovf_u2, Payload.None },
{ ILOpCode.Conv_ovf_i4, Payload.None },
{ ILOpCode.Conv_ovf_u4, Payload.None },
{ ILOpCode.Conv_ovf_i8, Payload.None },
{ ILOpCode.Conv_ovf_u8, Payload.None },
{ ILOpCode.Refanyval, Payload.Type },
{ ILOpCode.Ckfinite, Payload.None },
{ ILOpCode.Mkrefany, Payload.Class },
{ ILOpCode.Ldtoken, Payload.Token },
{ ILOpCode.Conv_u2, Payload.None },
{ ILOpCode.Conv_u1, Payload.None },
{ ILOpCode.Conv_i, Payload.None },
{ ILOpCode.Conv_ovf_i, Payload.None },
{ ILOpCode.Conv_ovf_u, Payload.None },
{ ILOpCode.Add_ovf, Payload.None },
{ ILOpCode.Add_ovf_un, Payload.None },
{ ILOpCode.Mul_ovf, Payload.None },
{ ILOpCode.Mul_ovf_un, Payload.None },
{ ILOpCode.Sub_ovf, Payload.None },
{ ILOpCode.Sub_ovf_un, Payload.None },
{ ILOpCode.Endfinally, Payload.None },
{ ILOpCode.Leave, Payload.Target32 },
{ ILOpCode.Leave_s, Payload.Target8 },
{ ILOpCode.Stind_i, Payload.None },
{ ILOpCode.Conv_u, Payload.None },
{ ILOpCode.Arglist, Payload.None },
{ ILOpCode.Ceq, Payload.None },
{ ILOpCode.Cgt, Payload.None },
{ ILOpCode.Cgt_un, Payload.None },
{ ILOpCode.Clt, Payload.None },
{ ILOpCode.Clt_un, Payload.None },
{ ILOpCode.Ldftn, Payload.Method },
{ ILOpCode.Ldvirtftn, Payload.Method },
{ ILOpCode.Ldarg, Payload.Arg16 },
{ ILOpCode.Ldarga, Payload.Arg16 },
{ ILOpCode.Starg, Payload.Arg16 },
{ ILOpCode.Ldloc, Payload.Local16 },
{ ILOpCode.Ldloca, Payload.Local16 },
{ ILOpCode.Stloc, Payload.Local16 },
{ ILOpCode.Localloc, Payload.None },
{ ILOpCode.Endfilter, Payload.None },
{ ILOpCode.Unaligned, Payload.Ignore8 },
{ ILOpCode.Volatile, Payload.None },
{ ILOpCode.Tail, Payload.None },
{ ILOpCode.Initobj, Payload.TypeTok },
{ ILOpCode.Constrained, Payload.Type },
{ ILOpCode.Cpblk, Payload.None },
{ ILOpCode.Initblk, Payload.None },
{ ILOpCode.Rethrow, Payload.None },
{ ILOpCode.Sizeof, Payload.TypeTok },
{ ILOpCode.Refanytype, Payload.None },
{ ILOpCode.Readonly, Payload.None }
};
public readonly DefinitionMethod Method;
public readonly ILOpCode OpCode;
public readonly int Offset;
public readonly int Index;
readonly int PayloadValue;
readonly uint UnsignedPayloadValue;
public Payload PayloadType
{
get
{
Payload result;
if (!opPayload.TryGetValue(OpCode, out result))
throw new InternalError("Unknown op code " + OpCode);
return result;
}
}
public override string ToString() => Index + ": " + OpCode;
/// <summary>
/// The number of bytes of this instruction,
/// including the payload (if any).
/// </summary>
public int Width
{
get
{
if (OpCode == ILOpCode.Switch) return 5 + 4 * PayloadValue;
return ((int)OpCode > 255 ? 2 : 1) + PayloadSize;
}
}
Label IEntity.Label
{
get; set;
}
readonly byte[] data;
int PayloadSize => payloadSizes[(int)PayloadType];
/// <summary>
/// Reads the instruction from a byte stream.
/// </summary>
/// <param name="data">The byte stream.</param>
/// <param name="offset">The offset of the instruction.</param>
/// <param name="index">The index of this instruction in the callable.</param>
public Instruction(Context cx, DefinitionMethod method, byte[] data, int offset, int index) : base(cx)
{
Method = method;
Offset = offset;
Index = index;
this.data = data;
int opcode = data[offset];
++offset;
/*
* An opcode is either 1 or 2 bytes, followed by an optional payload depending on the instruction.
* Instructions where the first byte is 0xfe are 2-byte instructions.
*/
if (opcode == 0xfe)
opcode = opcode << 8 | data[offset++];
OpCode = (ILOpCode)opcode;
switch (PayloadSize)
{
case 0:
PayloadValue = 0;
break;
case 1:
PayloadValue = (sbyte)data[offset];
UnsignedPayloadValue = data[offset];
break;
case 2:
PayloadValue = BitConverter.ToInt16(data, offset);
UnsignedPayloadValue = BitConverter.ToUInt16(data, offset);
break;
case -1: // Switch
case 4:
PayloadValue = BitConverter.ToInt32(data, offset);
break;
case 8: // Not handled here.
break;
default:
throw new InternalError("Unhandled CIL instruction Payload");
}
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
int offset = Offset;
yield return Tuples.cil_instruction(this, (int)OpCode, Index, Method.Implementation);
switch (PayloadType)
{
case Payload.String:
yield return Tuples.cil_value(this, cx.mdReader.GetUserString(MetadataTokens.UserStringHandle(PayloadValue)));
break;
case Payload.Float32:
yield return Tuples.cil_value(this, BitConverter.ToSingle(data, offset).ToString());
break;
case Payload.Float64:
yield return Tuples.cil_value(this, BitConverter.ToDouble(data, offset).ToString());
break;
case Payload.Int8:
yield return Tuples.cil_value(this, data[offset].ToString());
break;
case Payload.Int16:
yield return Tuples.cil_value(this, BitConverter.ToInt16(data, offset).ToString());
break;
case Payload.Int32:
yield return Tuples.cil_value(this, BitConverter.ToInt32(data, offset).ToString());
break;
case Payload.Int64:
yield return Tuples.cil_value(this, BitConverter.ToInt64(data, offset).ToString());
break;
case Payload.Constructor:
case Payload.Method:
case Payload.MethodRef:
case Payload.Class:
case Payload.TypeTok:
case Payload.Token:
case Payload.Type:
case Payload.Field:
case Payload.ValueType:
// A generic EntityHandle.
var handle = MetadataTokens.EntityHandle(PayloadValue);
var target = cx.CreateGeneric(Method, handle);
yield return target;
if (target != null)
{
yield return Tuples.cil_access(this, target);
}
else
{
throw new InternalError("Unable to create payload type {0} for opcode {1}", PayloadType, OpCode);
}
break;
case Payload.Arg8:
case Payload.Arg16:
yield return Tuples.cil_access(this, Method.Parameters[(int)UnsignedPayloadValue]);
break;
case Payload.Local8:
case Payload.Local16:
yield return Tuples.cil_access(this, Method.LocalVariables[(int)UnsignedPayloadValue]);
break;
case Payload.None:
case Payload.Target8:
case Payload.Target32:
case Payload.Switch:
case Payload.Ignore8:
case Payload.CallSiteDesc:
// These are not handled here.
// Some of these are handled by JumpContents().
break;
default:
throw new InternalError("Unhandled payload type {0}", PayloadType);
}
}
}
// Called to populate the jumps in each instruction.
public IEnumerable<IExtractionProduct> JumpContents(Dictionary<int, IInstruction> jump_table)
{
int target;
IInstruction inst;
switch (PayloadType)
{
case Payload.Target8:
target = Offset + PayloadValue + 2;
break;
case Payload.Target32:
target = Offset + PayloadValue + 5;
break;
case Payload.Switch:
int end = Offset + Width;
int offset = Offset + 5;
for (int b = 0; b < PayloadValue; ++b, offset += 4)
{
target = BitConverter.ToInt32(data, offset) + end;
if (!jump_table.TryGetValue(target, out inst))
throw new InternalError("Invalid jump target");
yield return Tuples.cil_switch(this, b, inst);
}
yield break;
default:
// Not a jump
yield break;
}
if (jump_table.TryGetValue(target, out inst))
{
yield return Tuples.cil_jump(this, inst);
}
else
{
// Sometimes instructions can jump outside the current method.
// TODO: Find a solution to this.
// For now, just log the error
cx.cx.Extractor.Message(new Message { message = "A CIL instruction jumps outside the current method", severity = Util.Logging.Severity.Warning });
}
}
}
}

36
csharp/extractor/Semmle.Extraction.CIL/Entities/LocalVariable.cs Normal file
View File

@@ -0,0 +1,36 @@
using System.Collections.Generic;
namespace Semmle.Extraction.CIL.Entities
{
interface ILocal : ILabelledEntity
{
}
class LocalVariable : LabelledEntity, ILocal
{
readonly MethodImplementation method;
readonly int index;
readonly Type type;
public LocalVariable(Context cx, MethodImplementation m, int i, Type t) : base(cx)
{
method = m;
index = i;
type = t;
ShortId = CIL.Id.Create(method.Label) + underscore + index;
}
static readonly Id underscore = CIL.Id.Create("_");
static readonly Id suffix = CIL.Id.Create(";cil-local");
public override Id IdSuffix => suffix;
public override IEnumerable<IExtractionProduct> Contents
{
get
{
yield return type;
yield return Tuples.cil_local_variable(this, method, index, type);
}
}
}
}

515
csharp/extractor/Semmle.Extraction.CIL/Entities/Method.cs Normal file
View File

@@ -0,0 +1,515 @@
using System;
using System.Collections.Immutable;
using System.Reflection.Metadata;
using Microsoft.CodeAnalysis;
using System.Collections.Generic;
using System.Reflection;
using System.Linq;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// A method entity.
/// </summary>
interface IMethod : IMember
{
}
/// <summary>
/// A method entity.
/// </summary>
abstract class Method : TypeContainer, IMethod
{
protected Method(GenericContext gc) : base(gc.cx)
{
this.gc = gc;
}
public override IEnumerable<Type> TypeParameters => gc.TypeParameters.Concat(declaringType.TypeParameters);
public override IEnumerable<Type> MethodParameters => genericParams == null ? Enumerable.Empty<Type>() : genericParams;
public int GenericParameterCount => signature.GenericParameterCount;
public virtual Method SourceDeclaration => this;
public abstract Type DeclaringType { get; }
public abstract string Name { get; }
public virtual IList<LocalVariable> LocalVariables => throw new NotImplementedException();
public IList<Parameter> Parameters { get; private set; }
static readonly Id tick = CIL.Id.Create("`");
static readonly Id space = CIL.Id.Create(" ");
static readonly Id dot = CIL.Id.Create(".");
static readonly Id open = CIL.Id.Create("(");
static readonly Id close = CIL.Id.Create(")");
internal protected Id MakeMethodId(Type parent, Id methodName)
{
var id = signature.ReturnType.MakeId(gc) + space + parent.ShortId + dot + methodName;
if (signature.GenericParameterCount > 0)
{
id += tick + signature.GenericParameterCount;
}
id += open + CIL.Id.CommaSeparatedList(signature.ParameterTypes.Select(p => p.MakeId(gc))) + close;
return id;
}
protected MethodTypeParameter[] genericParams;
protected Type declaringType;
protected GenericContext gc;
protected MethodSignature<ITypeSignature> signature;
protected Id name;
static readonly StringId methodSuffix = new StringId(";cil-method");
public override Id IdSuffix => methodSuffix;
protected void PopulateParameters(IEnumerable<Type> parameterTypes)
{
Parameters = MakeParameters(parameterTypes).ToArray();
}
protected IEnumerable<IExtractionProduct> PopulateFlags
{
get
{
if (IsStatic)
yield return Tuples.cil_static(this);
}
}
public abstract bool IsStatic { get; }
private IEnumerable<Parameter> MakeParameters(IEnumerable<Type> parameterTypes)
{
int i = 0;
if (!IsStatic)
{
yield return cx.Populate(new Parameter(cx, this, i++, DeclaringType));
}
foreach (var p in parameterTypes)
yield return cx.Populate(new Parameter(cx, this, i++, p));
}
}
/// <summary>
/// A method implementation entity.
/// </summary>
interface IMethodImplementation : IExtractedEntity
{
}
/// <summary>
/// A method implementation entity.
/// In the database, the same method could in principle have multiple implementations.
/// </summary>
class MethodImplementation : UnlabelledEntity, IMethodImplementation
{
readonly Method m;
public MethodImplementation(Method m) : base(m.cx)
{
this.m = m;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
yield return Tuples.cil_method_implementation(this, m, cx.assembly);
}
}
}
/// <summary>
/// A definition method - a method defined in the current assembly.
/// </summary>
class DefinitionMethod : Method, IMember
{
readonly MethodDefinition md;
readonly PDB.IMethod methodDebugInformation;
LocalVariable[] locals;
public MethodImplementation Implementation { get; private set; }
public override IList<LocalVariable> LocalVariables => locals;
public DefinitionMethod(GenericContext gc, MethodDefinitionHandle handle) : base(gc)
{
md = cx.mdReader.GetMethodDefinition(handle);
this.gc = gc;
name = cx.GetId(md.Name);
declaringType = (Type)cx.CreateGeneric(this, md.GetDeclaringType());
signature = md.DecodeSignature(new SignatureDecoder(), this);
ShortId = MakeMethodId(declaringType, name);
methodDebugInformation = cx.GetMethodDebugInformation(handle);
}
public override bool IsStatic => !signature.Header.IsInstance;
public override Type DeclaringType => declaringType;
public override string Name => cx.ShortName(md.Name);
/// <summary>
/// Holds if this method has bytecode.
/// </summary>
public bool HasBytecode => md.ImplAttributes == MethodImplAttributes.IL && md.RelativeVirtualAddress != 0;
public override IEnumerable<IExtractionProduct> Contents
{
get
{
if (md.GetGenericParameters().Any())
{
// We need to perform a 2-phase population because some type parameters can
// depend on other type parameters (as a constraint).
genericParams = new MethodTypeParameter[md.GetGenericParameters().Count];
for (int i = 0; i < genericParams.Length; ++i)
genericParams[i] = cx.Populate(new MethodTypeParameter(this, this, i));
for (int i = 0; i < genericParams.Length; ++i)
genericParams[i].PopulateHandle(this, md.GetGenericParameters()[i]);
foreach (var p in genericParams)
yield return p;
}
var typeSignature = md.DecodeSignature(cx.TypeSignatureDecoder, this);
PopulateParameters(typeSignature.ParameterTypes);
foreach (var c in Parameters)
yield return c;
foreach (var c in PopulateFlags)
yield return c;
foreach (var p in md.GetParameters().Select(h => cx.mdReader.GetParameter(h)).Where(p => p.SequenceNumber > 0))
{
var pe = Parameters[IsStatic ? p.SequenceNumber - 1 : p.SequenceNumber];
if (p.Attributes.HasFlag(ParameterAttributes.Out))
yield return Tuples.cil_parameter_out(pe);
if (p.Attributes.HasFlag(ParameterAttributes.In))
yield return Tuples.cil_parameter_in(pe);
Attribute.Populate(cx, pe, p.GetCustomAttributes());
}
yield return Tuples.cil_method(this, Name, declaringType, typeSignature.ReturnType);
yield return Tuples.cil_method_source_declaration(this, this);
yield return Tuples.cil_method_location(this, cx.assembly);
if (HasBytecode)
{
Implementation = new MethodImplementation(this);
yield return Implementation;
var body = cx.peReader.GetMethodBody(md.RelativeVirtualAddress);
if (!body.LocalSignature.IsNil)
{
var locals = cx.mdReader.GetStandaloneSignature(body.LocalSignature);
var localVariableTypes = locals.DecodeLocalSignature(cx.TypeSignatureDecoder, this);
this.locals = new LocalVariable[localVariableTypes.Length];
for (int l = 0; l < this.locals.Length; ++l)
{
this.locals[l] = cx.Populate(new LocalVariable(cx, Implementation, l, localVariableTypes[l]));
yield return this.locals[l];
}
}
var jump_table = new Dictionary<int, IInstruction>();
foreach (var c in Decode(body.GetILBytes(), jump_table))
yield return c;
int filter_index = 0;
foreach (var region in body.ExceptionRegions)
{
yield return new ExceptionRegion(this, Implementation, filter_index++, region, jump_table);
}
yield return Tuples.cil_method_stack_size(Implementation, body.MaxStack);
if (methodDebugInformation != null)
{
var sourceLocation = cx.CreateSourceLocation(methodDebugInformation.Location);
yield return sourceLocation;
yield return Tuples.cil_method_location(this, sourceLocation);
}
}
// Flags
if (md.Attributes.HasFlag(MethodAttributes.Private))
yield return Tuples.cil_private(this);
if (md.Attributes.HasFlag(MethodAttributes.Public))
yield return Tuples.cil_public(this);
if (md.Attributes.HasFlag(MethodAttributes.Family))
yield return Tuples.cil_protected(this);
if (md.Attributes.HasFlag(MethodAttributes.Final))
yield return Tuples.cil_sealed(this);
if (md.Attributes.HasFlag(MethodAttributes.Virtual))
yield return Tuples.cil_virtual(this);
if (md.Attributes.HasFlag(MethodAttributes.Abstract))
yield return Tuples.cil_abstract(this);
if (md.Attributes.HasFlag(MethodAttributes.HasSecurity))
yield return Tuples.cil_security(this);
if (md.Attributes.HasFlag(MethodAttributes.RequireSecObject))
yield return Tuples.cil_requiresecobject(this);
if (md.Attributes.HasFlag(MethodAttributes.SpecialName))
yield return Tuples.cil_specialname(this);
if (md.Attributes.HasFlag(MethodAttributes.NewSlot))
yield return Tuples.cil_newslot(this);
// Populate attributes
Attribute.Populate(cx, this, md.GetCustomAttributes());
}
}
IEnumerable<IExtractionProduct> Decode(byte[] ilbytes, Dictionary<int, IInstruction> jump_table)
{
// Sequence points are stored in order of offset.
// We use an enumerator to locate the correct sequence point for each instruction.
// The sequence point gives the location of each instruction.
// The location of an instruction is given by the sequence point *after* the
// instruction.
IEnumerator<PDB.SequencePoint> nextSequencePoint = null;
PdbSourceLocation instructionLocation = null;
if (methodDebugInformation != null)
{
nextSequencePoint = methodDebugInformation.SequencePoints.GetEnumerator();
if (nextSequencePoint.MoveNext())
{
instructionLocation = cx.CreateSourceLocation(nextSequencePoint.Current.Location);
yield return instructionLocation;
}
else
{
nextSequencePoint = null;
}
}
int child = 0;
for (int offset = 0; offset < ilbytes.Length;)
{
var instruction = new Instruction(cx, this, ilbytes, offset, child++);
yield return instruction;
if (nextSequencePoint != null && offset >= nextSequencePoint.Current.Offset)
{
instructionLocation = cx.CreateSourceLocation(nextSequencePoint.Current.Location);
yield return instructionLocation;
if (!nextSequencePoint.MoveNext())
nextSequencePoint = null;
}
if (instructionLocation != null)
yield return Tuples.cil_instruction_location(instruction, instructionLocation);
jump_table.Add(instruction.Offset, instruction);
offset += instruction.Width;
}
foreach (var i in jump_table)
{
foreach (var t in i.Value.JumpContents(jump_table))
yield return t;
}
}
/// <summary>
/// Display the instructions in the method in the debugger.
/// This is only used for debugging, not in the code itself.
/// </summary>
public IEnumerable<Instruction> DebugInstructions
{
get
{
if (md.ImplAttributes == MethodImplAttributes.IL && md.RelativeVirtualAddress != 0)
{
var body = cx.peReader.GetMethodBody(md.RelativeVirtualAddress);
var ilbytes = body.GetILBytes();
int child = 0;
for (int offset = 0; offset < ilbytes.Length;)
{
Instruction decoded;
try
{
decoded = new Instruction(cx, this, ilbytes, offset, child++);
offset += decoded.Width;
}
catch
{
yield break;
}
yield return decoded;
}
}
}
}
}
/// <summary>
/// This is a late-bound reference to a method.
/// </summary>
class MemberReferenceMethod : Method
{
readonly MemberReference mr;
readonly Type declType;
readonly GenericContext parent;
readonly Method sourceDeclaration;
public MemberReferenceMethod(GenericContext gc, MemberReferenceHandle handle) : base(gc)
{
this.gc = gc;
mr = cx.mdReader.GetMemberReference(handle);
signature = mr.DecodeMethodSignature(new SignatureDecoder(), gc);
parent = (GenericContext)cx.CreateGeneric(gc, mr.Parent);
var parentMethod = parent as Method;
var nameLabel = cx.GetId(mr.Name);
declType = parentMethod == null ? parent as Type : parentMethod.DeclaringType;
ShortId = MakeMethodId(declType, nameLabel);
var typeSourceDeclaration = declType.SourceDeclaration;
sourceDeclaration = typeSourceDeclaration == declType ? (Method)this : typeSourceDeclaration.LookupMethod(mr.Name, mr.Signature);
}
public override Method SourceDeclaration => sourceDeclaration;
public override bool IsStatic => !signature.Header.IsInstance;
public override Type DeclaringType => declType;
public override string Name => cx.ShortName(mr.Name);
public override IEnumerable<Type> TypeParameters => parent.TypeParameters.Concat(gc.TypeParameters);
public override IEnumerable<IExtractionProduct> Contents
{
get
{
genericParams = new MethodTypeParameter[signature.GenericParameterCount];
for (int p = 0; p < genericParams.Length; ++p)
genericParams[p] = cx.Populate(new MethodTypeParameter(this, this, p));
foreach (var p in genericParams)
yield return p;
var typeSignature = mr.DecodeMethodSignature(cx.TypeSignatureDecoder, this);
PopulateParameters(typeSignature.ParameterTypes);
foreach (var p in Parameters) yield return p;
foreach (var f in PopulateFlags) yield return f;
yield return Tuples.cil_method(this, Name, DeclaringType, typeSignature.ReturnType);
if (SourceDeclaration != null)
yield return Tuples.cil_method_source_declaration(this, SourceDeclaration);
}
}
}
/// <summary>
/// A constructed method.
/// </summary>
class MethodSpecificationMethod : Method
{
readonly MethodSpecification ms;
readonly Method unboundMethod;
readonly ImmutableArray<Type> typeParams;
public MethodSpecificationMethod(GenericContext gc, MethodSpecificationHandle handle) : base(gc)
{
ms = cx.mdReader.GetMethodSpecification(handle);
typeParams = ms.DecodeSignature(cx.TypeSignatureDecoder, gc);
unboundMethod = (Method)cx.CreateGeneric(gc, ms.Method);
declaringType = unboundMethod.DeclaringType;
ShortId = unboundMethod.ShortId + openAngle + CIL.Id.CommaSeparatedList(typeParams.Select(p => p.ShortId)) + closeAngle;
}
static readonly Id openAngle = CIL.Id.Create("<");
static readonly Id closeAngle = CIL.Id.Create(">");
public override Method SourceDeclaration => unboundMethod;
public override Type DeclaringType => unboundMethod.DeclaringType;
public override string Name => unboundMethod.Name;
public override bool IsStatic => unboundMethod.IsStatic;
public override IEnumerable<Type> MethodParameters => typeParams;
public override IEnumerable<IExtractionProduct> Contents
{
get
{
MethodSignature<Type> constructedTypeSignature;
switch (ms.Method.Kind)
{
case HandleKind.MemberReference:
var mr = cx.mdReader.GetMemberReference((MemberReferenceHandle)ms.Method);
constructedTypeSignature = mr.DecodeMethodSignature(cx.TypeSignatureDecoder, this);
break;
case HandleKind.MethodDefinition:
var md = cx.mdReader.GetMethodDefinition((MethodDefinitionHandle)ms.Method);
constructedTypeSignature = md.DecodeSignature(cx.TypeSignatureDecoder, this);
break;
default:
throw new InternalError("Unexpected constructed method handle kind {0}", ms.Method.Kind);
}
PopulateParameters(constructedTypeSignature.ParameterTypes);
foreach (var p in Parameters)
yield return p;
foreach (var f in PopulateFlags)
yield return f;
yield return Tuples.cil_method(this, Name, DeclaringType, constructedTypeSignature.ReturnType);
yield return Tuples.cil_method_source_declaration(this, SourceDeclaration);
if (typeParams.Count() != unboundMethod.GenericParameterCount)
throw new InternalError("Method type parameter mismatch");
for (int p = 0; p < typeParams.Length; ++p)
{
yield return Tuples.cil_type_argument(this, p, typeParams[p]);
}
}
}
}
}

78
csharp/extractor/Semmle.Extraction.CIL/Entities/Namespace.cs Normal file
View File

@@ -0,0 +1,78 @@
using System;
using System.Collections.Generic;
using Semmle.Extraction.Entities;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// A namespace.
/// </summary>
interface INamespace : ITypeContainer
{
}
/// <summary>
/// A namespace.
/// </summary>
public class Namespace : TypeContainer, INamespace
{
public Namespace ParentNamespace;
public readonly StringId Name;
public bool IsGlobalNamespace => ParentNamespace == null;
static readonly Id suffix = CIL.Id.Create(";namespace");
public Id CreateId
{
get
{
if (ParentNamespace != null && !ParentNamespace.IsGlobalNamespace)
{
return ParentNamespace.ShortId + cx.Dot + Name;
}
return Name;
}
}
public override Id IdSuffix => suffix;
public override IEnumerable<Type> TypeParameters => throw new NotImplementedException();
public override IEnumerable<Type> MethodParameters => throw new NotImplementedException();
static string parseNamespaceName(string fqn)
{
var i = fqn.LastIndexOf('.');
return i == -1 ? fqn : fqn.Substring(i + 1);
}
static Namespace createParentNamespace(Context cx, string fqn)
{
if (fqn == "") return null;
var i = fqn.LastIndexOf('.');
return i == -1 ? cx.GlobalNamespace : cx.Populate(new Namespace(cx, fqn.Substring(0, i)));
}
public Namespace(Context cx, string fqn) : this(cx, cx.GetId(parseNamespaceName(fqn)), createParentNamespace(cx, fqn))
{
}
public Namespace(Context cx, StringId name, Namespace parent) : base(cx)
{
Name = name;
ParentNamespace = parent;
ShortId = CreateId;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
yield return Tuples.namespaces(this, Name.Value);
if (!IsGlobalNamespace)
yield return Tuples.parent_namespace(this, ParentNamespace);
}
}
}
}

43
csharp/extractor/Semmle.Extraction.CIL/Entities/Parameter.cs Normal file
View File

@@ -0,0 +1,43 @@
using System.Collections.Generic;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// A parameter entity.
/// </summary>
interface IParameter : ILabelledEntity
{
}
/// <summary>
/// A parameter entity.
/// </summary>
class Parameter : LabelledEntity, IParameter
{
readonly Method method;
readonly int index;
readonly Type type;
public Parameter(Context cx, Method m, int i, Type t) : base(cx)
{
method = m;
index = i;
type = t;
ShortId = openCurly + method.Label.Value + closeCurly + index;
}
static readonly Id parameterSuffix = CIL.Id.Create(";cil-parameter");
static readonly Id openCurly = CIL.Id.Create("{#");
static readonly Id closeCurly = CIL.Id.Create("}_");
public override Id IdSuffix => parameterSuffix;
public override IEnumerable<IExtractionProduct> Contents
{
get
{
yield return Tuples.cil_parameter(this, method, index, type);
}
}
}
}

64
csharp/extractor/Semmle.Extraction.CIL/Entities/Property.cs Normal file
View File

@@ -0,0 +1,64 @@
using System.Collections.Generic;
using System.Reflection.Metadata;
using System.Linq;
namespace Semmle.Extraction.CIL.Entities
{
/// <summary>
/// A property.
/// </summary>
interface IProperty : ILabelledEntity
{
}
/// <summary>
/// A property.
/// </summary>
class Property : LabelledEntity, IProperty
{
readonly Type type;
readonly PropertyDefinition pd;
static readonly Id suffix = CIL.Id.Create(";cil-property");
public Property(GenericContext gc, Type type, PropertyDefinitionHandle handle) : base(gc.cx)
{
pd = cx.mdReader.GetPropertyDefinition(handle);
this.type = type;
var id = type.ShortId + gc.cx.Dot + cx.ShortName(pd.Name);
var signature = pd.DecodeSignature(new SignatureDecoder(), gc);
id += "(" + CIL.Id.CommaSeparatedList(signature.ParameterTypes.Select(p => p.MakeId(gc))) + ")";
ShortId = id;
}
public override IEnumerable<IExtractionProduct> Contents
{
get
{
var sig = pd.DecodeSignature(cx.TypeSignatureDecoder, type);
yield return Tuples.cil_property(this, type, cx.ShortName(pd.Name), sig.ReturnType);
var accessors = pd.GetAccessors();
if (!accessors.Getter.IsNil)
{
var getter = (Method)cx.CreateGeneric(type, accessors.Getter);
yield return getter;
yield return Tuples.cil_getter(this, getter);
}
if (!accessors.Setter.IsNil)
{
var setter = (Method)cx.CreateGeneric(type, accessors.Setter);
yield return setter;
yield return Tuples.cil_setter(this, setter);
}
foreach (var c in Attribute.Populate(cx, this, pd.GetCustomAttributes()))
yield return c;
}
}
public override Id IdSuffix => suffix;
}
}

37
csharp/extractor/Semmle.Extraction.CIL/Entities/SourceLocation.cs Normal file
View File

@@ -0,0 +1,37 @@
using System.Collections.Generic;
using Semmle.Extraction.PDB;
namespace Semmle.Extraction.CIL.Entities
{
public interface ISourceLocation : ILocation
{
}
public sealed class PdbSourceLocation : LabelledEntity, ISourceLocation
{
readonly Location location;
readonly PdbSourceFile file;
public PdbSourceLocation(Context cx, PDB.Location location) : base(cx)
{
this.location = location;
file = cx.CreateSourceFile(location.File);
ShortId = file.ShortId + separator + new IntId(location.StartLine) + separator + new IntId(location.StartColumn) + separator + new IntId(location.EndLine) + separator + new IntId(location.EndColumn);
}
static readonly Id suffix = new StringId(";sourcelocation");
static readonly Id separator = new StringId(",");
public override IEnumerable<IExtractionProduct> Contents
{
get
{
yield return file;
yield return Tuples.locations_default(this, file, location.StartLine, location.StartColumn, location.EndLine, location.EndColumn);
}
}
public override Id IdSuffix => suffix;
}
}

1285
csharp/extractor/Semmle.Extraction.CIL/Entities/Type.cs Normal file
View File

File diff suppressed because it is too large Load Diff

127
csharp/extractor/Semmle.Extraction.CIL/ExtractionProduct.cs Normal file
View File

@@ -0,0 +1,127 @@
using System;
using System.Collections.Generic;
namespace Semmle.Extraction.CIL
{
/// <summary>
/// Something that is extracted from an entity.
/// </summary>
///
/// <remarks>
/// The extraction algorithm proceeds as follows:
/// - Construct entity
/// - Call Extract()
/// - ILabelledEntity check if already extracted
/// - Enumerate Contents to produce more extraction products
/// - Extract these until there is nothing left to extract
/// </remarks>
public interface IExtractionProduct
{
/// <summary>
/// Perform further extraction/population of this item as necessary.
/// </summary>
///
/// <param name="cx">The extraction context.</param>
void Extract(Context cx);
}
/// <summary>
/// An entity which has been extracted.
/// </summary>
public interface IExtractedEntity : IEntity, IExtractionProduct
{
/// <summary>
/// The contents of the entity.
/// </summary>
IEnumerable<IExtractionProduct> Contents { get; }
}
/// <summary>
/// An entity that has contents to extract. There is no need to populate
/// a key as it's done in the contructor.
/// </summary>
public abstract class UnlabelledEntity : IExtractedEntity
{
public abstract IEnumerable<IExtractionProduct> Contents { get; }
public Label Label { get; set; }
public Microsoft.CodeAnalysis.Location ReportingLocation => throw new NotImplementedException();
public virtual IId Id => FreshId.Instance;
public virtual void Extract(Context cx)
{
cx.Extract(this);
}
public readonly Context cx;
protected UnlabelledEntity(Context cx)
{
this.cx = cx;
cx.cx.AddFreshLabel(this);
}
TrapStackBehaviour IEntity.TrapStackBehaviour => TrapStackBehaviour.NoLabel;
}
/// <summary>
/// An entity that needs to be populated during extraction.
/// This assigns a key and optionally extracts its contents.
/// </summary>
public abstract class LabelledEntity : ILabelledEntity
{
public abstract IEnumerable<IExtractionProduct> Contents { get; }
public Label Label { get; set; }
public Microsoft.CodeAnalysis.Location ReportingLocation => throw new NotImplementedException();
public Id ShortId { get; set; }
public abstract Id IdSuffix { get; }
public IId Id => ShortId + IdSuffix;
public void Extract(Context cx)
{
cx.Populate(this);
}
public readonly Context cx;
protected LabelledEntity(Context cx)
{
this.cx = cx;
}
public override string ToString() => Id.ToString();
TrapStackBehaviour IEntity.TrapStackBehaviour => TrapStackBehaviour.NoLabel;
}
/// <summary>
/// An entity with a defined ID.
/// </summary>
public interface ILabelledEntity : IExtractedEntity
{
Id ShortId { get; set; }
Id IdSuffix { get; }
}
/// <summary>
/// A tuple that is an extraction product.
/// </summary>
class Tuple : IExtractionProduct
{
readonly Extraction.Tuple tuple;
public Tuple(string name, params object[] args)
{
tuple = new Extraction.Tuple(name, args);
}
public void Extract(Context cx)
{
cx.cx.Emit(tuple);
}
public override string ToString() => tuple.ToString();
}
}

233
csharp/extractor/Semmle.Extraction.CIL/Factories.cs Normal file
View File

@@ -0,0 +1,233 @@
using System;
using System.Collections.Generic;
using System.Reflection.Metadata;
namespace Semmle.Extraction.CIL
{
/// <summary>
/// Provides methods for creating and caching various entities.
/// </summary>
public partial class Context
{
readonly Dictionary<Id, (Label, Id)> ids = new Dictionary<Id, (Label, Id)>();
public T Populate<T>(T e) where T : ILabelledEntity
{
Id id = e.ShortId;
if (ids.TryGetValue(id, out var existing))
{
// It exists already
e.Label = existing.Item1;
e.ShortId = existing.Item2; // Reuse ID for efficiency
}
else
{
cx.DefineLabel(e);
ids.Add(id, (e.Label, id));
cx.PopulateLater(() =>
{
foreach (var c in e.Contents)
c.Extract(this);
});
}
return e;
}
public IExtractedEntity Create(Handle h)
{
var entity = CreateGeneric(defaultGenericContext, h);
return entity;
}
/// <summary>
/// Creates an entity from a Handle in a GenericContext.
/// The type of the returned entity depends on the type of the handle.
/// The GenericContext is needed because some handles are generics which
/// need to be expanded in terms of the current instantiation. If this sounds
/// complex, you are right.
///
/// The pair (h,genericContext) is cached in case it is needed again.
/// </summary>
/// <param name="h">The handle of the entity.</param>
/// <param name="genericContext">The generic context.</param>
/// <returns></returns>
public ILabelledEntity CreateGeneric(GenericContext genericContext, Handle h) => genericHandleFactory[genericContext, h];
readonly GenericContext defaultGenericContext;
ILabelledEntity CreateGenericHandle(GenericContext gc, Handle handle)
{
ILabelledEntity entity;
switch (handle.Kind)
{
case HandleKind.MethodDefinition:
entity = new Entities.DefinitionMethod(gc, (MethodDefinitionHandle)handle);
break;
case HandleKind.MemberReference:
entity = Create(gc, (MemberReferenceHandle)handle);
break;
case HandleKind.MethodSpecification:
entity = new Entities.MethodSpecificationMethod(gc, (MethodSpecificationHandle)handle);
break;
case HandleKind.FieldDefinition:
entity = new Entities.DefinitionField(gc, (FieldDefinitionHandle)handle);
break;
case HandleKind.TypeReference:
entity = new Entities.TypeReferenceType(this, (TypeReferenceHandle)handle);
break;
case HandleKind.TypeSpecification:
entity = new Entities.TypeSpecificationType(gc, (TypeSpecificationHandle)handle);
break;
case HandleKind.TypeDefinition:
entity = new Entities.TypeDefinitionType(this, (TypeDefinitionHandle)handle);
break;
default:
throw new InternalError("Unhandled handle kind " + handle.Kind);
}
Populate(entity);
return entity;
}
ILabelledEntity Create(GenericContext gc, MemberReferenceHandle handle)
{
var mr = mdReader.GetMemberReference(handle);
switch (mr.GetKind())
{
case MemberReferenceKind.Method:
return new Entities.MemberReferenceMethod(gc, handle);
case MemberReferenceKind.Field:
return new Entities.MemberReferenceField(gc, handle);
default:
throw new InternalError("Unhandled member reference handle");
}
}
#region Strings
readonly Dictionary<StringHandle, StringId> stringHandleIds = new Dictionary<StringHandle, StringId>();
readonly Dictionary<string, StringId> stringIds = new Dictionary<string, StringId>();
/// <summary>
/// Return an ID containing the given string.
/// </summary>
/// <param name="h">The string handle.</param>
/// <returns>An ID.</returns>
public StringId GetId(StringHandle h)
{
StringId result;
if (!stringHandleIds.TryGetValue(h, out result))
{
result = new StringId(mdReader.GetString(h));
stringHandleIds.Add(h, result);
}
return result;
}
public readonly StringId Dot = new StringId(".");
/// <summary>
/// Gets an ID containing the given string.
/// Caches existing IDs for more compact storage.
/// </summary>
/// <param name="str">The string.</param>
/// <returns>An ID containing the string.</returns>
public StringId GetId(string str)
{
StringId result;
if (!stringIds.TryGetValue(str, out result))
{
result = new StringId(str);
stringIds.Add(str, result);
}
return result;
}
#endregion
#region Namespaces
readonly CachedFunction<StringHandle, Entities.Namespace> namespaceFactory;
public Entities.Namespace CreateNamespace(StringHandle fqn) => namespaceFactory[fqn];
readonly Lazy<Entities.Namespace> globalNamespace, systemNamespace;
/// <summary>
/// The entity representing the global namespace.
/// </summary>
public Entities.Namespace GlobalNamespace => globalNamespace.Value;
/// <summary>
/// The entity representing the System namespace.
/// </summary>
public Entities.Namespace SystemNamespace => systemNamespace.Value;
/// <summary>
/// Creates a namespace from a fully-qualified name.
/// </summary>
/// <param name="fqn">The fully-qualified namespace name.</param>
/// <returns>The namespace entity.</returns>
Entities.Namespace CreateNamespace(string fqn) => Populate(new Entities.Namespace(this, fqn));
readonly CachedFunction<NamespaceDefinitionHandle, Entities.Namespace> namespaceDefinitionFactory;
/// <summary>
/// Creates a namespace from a namespace handle.
/// </summary>
/// <param name="handle">The handle of the namespace.</param>
/// <returns>The namespace entity.</returns>
public Entities.Namespace Create(NamespaceDefinitionHandle handle) => namespaceDefinitionFactory[handle];
Entities.Namespace CreateNamespace(NamespaceDefinitionHandle handle)
{
if (handle.IsNil) return GlobalNamespace;
NamespaceDefinition nd = mdReader.GetNamespaceDefinition(handle);
return Populate(new Entities.Namespace(this, GetId(nd.Name), Create(nd.Parent)));
}
#endregion
#region Locations
readonly CachedFunction<PDB.ISourceFile, Entities.PdbSourceFile> sourceFiles;
readonly CachedFunction<string, Entities.Folder> folders;
readonly CachedFunction<PDB.Location, Entities.PdbSourceLocation> sourceLocations;
/// <summary>
/// Creates a source file entity from a PDB source file.
/// </summary>
/// <param name="file">The PDB source file.</param>
/// <returns>A source file entity.</returns>
public Entities.PdbSourceFile CreateSourceFile(PDB.ISourceFile file) => sourceFiles[file];
/// <summary>
/// Creates a folder entitiy with the given path.
/// </summary>
/// <param name="path">The path of the folder.</param>
/// <returns>A folder entity.</returns>
public Entities.Folder CreateFolder(string path) => folders[path];
/// <summary>
/// Creates a source location.
/// </summary>
/// <param name="loc">The source location from PDB.</param>
/// <returns>A source location entity.</returns>
public Entities.PdbSourceLocation CreateSourceLocation(PDB.Location loc) => sourceLocations[loc];
#endregion
readonly CachedFunction<GenericContext, Handle, ILabelledEntity> genericHandleFactory;
/// <summary>
/// Gets the short name of a member, without the preceding interface qualifier.
/// </summary>
/// <param name="handle">The handle of the name.</param>
/// <returns>The short name.</returns>
public string ShortName(StringHandle handle)
{
string str = mdReader.GetString(handle);
if (str.EndsWith(".ctor")) return ".ctor";
if (str.EndsWith(".cctor")) return ".cctor";
var dot = str.LastIndexOf('.');
return dot == -1 ? str : str.Substring(dot + 1);
}
}
}

204
csharp/extractor/Semmle.Extraction.CIL/Id.cs Normal file
View File

@@ -0,0 +1,204 @@
using System.Collections.Generic;
using System.Reflection.Metadata;
namespace Semmle.Extraction.CIL
{
/// <summary>
/// An ID fragment which is designed to be shared, reused
/// and composed using the + operator.
/// </summary>
public abstract class Id : IId
{
public void AppendTo(ITrapBuilder tb)
{
tb.Append("@\"");
BuildParts(tb);
tb.Append("\"");
}
public abstract void BuildParts(ITrapBuilder tb);
public static Id operator +(Id l1, Id l2) => Create(l1, l2);
public static Id operator +(Id l1, string l2) => Create(l1, Create(l2));
public static Id operator +(Id l1, int l2) => Create(l1, Create(l2));
public static Id operator +(string l1, Id l2) => Create(Create(l1), l2);
public static Id operator +(int l1, Id l2) => Create(Create(l1), l2);
public static Id Create(string s) => s == null ? null : new StringId(s);
public static Id Create(int i) => new IntId(i);
static readonly Id openCurly = Create("{#");
static readonly Id closeCurly = Create("}");
public static Id Create(Label l) => openCurly + l.Value + closeCurly;
static readonly Id comma = Id.Create(",");
public static Id CommaSeparatedList(IEnumerable<Id> items)
{
Id result = null;
bool first = true;
foreach (var i in items)
{
if (first) first = false; else result += comma;
result += i;
}
return result;
}
public static Id Create(Id l1, Id l2)
{
return l1 == null ? l2 : l2 == null ? l1 : new ConsId(l1, l2);
}
public abstract string Value { get; }
public override string ToString() => Value;
}
/// <summary>
/// An ID concatenating two other IDs.
/// </summary>
public sealed class ConsId : Id
{
readonly Id left, right;
readonly int hash;
public ConsId(Id l1, Id l2)
{
left = l1;
right = l2;
hash = unchecked(12 + 3 * (left.GetHashCode() + 51 * right.GetHashCode()));
}
public override void BuildParts(ITrapBuilder tb)
{
left.BuildParts(tb);
right.BuildParts(tb);
}
public override bool Equals(object other)
{
return other is ConsId && Equals((ConsId)other);
}
public bool Equals(ConsId other)
{
return this == other ||
(hash == other.hash && left.Equals(other.left) && right.Equals(other.right));
}
public override int GetHashCode() => hash;
public override string Value => left.Value + right.Value;
}
/// <summary>
/// A leaf ID storing a string.
/// </summary>
public sealed class StringId : Id
{
readonly string value;
public override string Value => value;
public StringId(string s)
{
value = s;
}
public override void BuildParts(ITrapBuilder tb)
{
tb.Append(value);
}
public override bool Equals(object obj)
{
return obj is StringId && ((StringId)obj).value == value;
}
public override int GetHashCode() => Value.GetHashCode() * 31 + 9;
}
/// <summary>
/// A leaf ID storing an integer.
/// </summary>
public sealed class IntId : Id
{
readonly int value;
public override string Value => value.ToString();
public IntId(int i)
{
value = i;
}
public override void BuildParts(ITrapBuilder tb)
{
tb.Append(value);
}
public override bool Equals(object obj)
{
return obj is IntId && ((IntId)obj).value == value;
}
public override int GetHashCode() => unchecked(12 + value * 17);
}
/// <summary>
/// Some predefined IDs.
/// </summary>
public static class IdUtils
{
public static StringId boolId = new StringId("Boolean");
public static StringId byteId = new StringId("Byte");
public static StringId charId = new StringId("Char");
public static StringId doubleId = new StringId("Double");
public static StringId shortId = new StringId("Int16");
public static StringId intId = new StringId("Int32");
public static StringId longId = new StringId("Int64");
public static StringId intptrId = new StringId("IntPtr");
public static StringId objectId = new StringId("Object");
public static StringId sbyteId = new StringId("SByte");
public static StringId floatId = new StringId("Single");
public static StringId stringId = new StringId("String");
public static StringId ushortId = new StringId("UInt16");
public static StringId uintId = new StringId("UInt32");
public static StringId ulongId = new StringId("UInt64");
public static StringId uintptrId = new StringId("UIntPtr");
public static StringId voidId = new StringId("Void");
public static StringId typedReferenceId = new StringId("TypedReference");
public static StringId Id(this PrimitiveTypeCode typeCode)
{
switch (typeCode)
{
case PrimitiveTypeCode.Boolean: return boolId;
case PrimitiveTypeCode.Byte: return byteId;
case PrimitiveTypeCode.Char: return charId;
case PrimitiveTypeCode.Double: return doubleId;
case PrimitiveTypeCode.Int16: return shortId;
case PrimitiveTypeCode.Int32: return intId;
case PrimitiveTypeCode.Int64: return longId;
case PrimitiveTypeCode.IntPtr: return intptrId;
case PrimitiveTypeCode.Object: return objectId;
case PrimitiveTypeCode.SByte: return sbyteId;
case PrimitiveTypeCode.Single: return floatId;
case PrimitiveTypeCode.String: return stringId;
case PrimitiveTypeCode.UInt16: return ushortId;
case PrimitiveTypeCode.UInt32: return uintId;
case PrimitiveTypeCode.UInt64: return ulongId;
case PrimitiveTypeCode.UIntPtr: return uintptrId;
case PrimitiveTypeCode.Void: return voidId;
case PrimitiveTypeCode.TypedReference: return typedReferenceId;
default: throw new InternalError("Unhandled type code {0}", typeCode);
}
}
}
}

93
csharp/extractor/Semmle.Extraction.CIL/PDB/MetadataPdbReader.cs Normal file
View File

@@ -0,0 +1,93 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
namespace Semmle.Extraction.PDB
{
/// <summary>
/// A reader of PDB information using System.Reflection.Metadata.
/// This is cross platform, and the future of PDB.
///
/// PDB information can be in a separate PDB file, or embedded in the DLL.
/// </summary>
class MetadataPdbReader : IPdb
{
class SourceFile : ISourceFile
{
public SourceFile(MetadataReader reader, DocumentHandle handle)
{
var doc = reader.GetDocument(handle);
Path = reader.GetString(doc.Name);
}
public string Path { get; private set; }
public string Contents => File.Exists(Path) ? File.ReadAllText(Path, System.Text.Encoding.Default) : null;
}
// Turns out to be very important to keep the MetadataReaderProvider live
// or the reader will crash.
readonly MetadataReaderProvider provider;
readonly MetadataReader reader;
public MetadataPdbReader(MetadataReaderProvider provider)
{
this.provider = provider;
reader = provider.GetMetadataReader();
}
public IEnumerable<ISourceFile> SourceFiles => reader.Documents.Select(handle => new SourceFile(reader, handle));
public IMethod GetMethod(MethodDebugInformationHandle handle)
{
var debugInfo = reader.GetMethodDebugInformation(handle);
var sequencePoints = debugInfo.GetSequencePoints().
Where(p => !p.Document.IsNil && !p.IsHidden).
Select(p => new SequencePoint(p.Offset, new Location(new SourceFile(reader, p.Document), p.StartLine, p.StartColumn, p.EndLine, p.EndColumn))).
Where(p => p.Location.File.Path != null).
ToArray();
return sequencePoints.Any() ? new Method() { SequencePoints = sequencePoints } : null;
}
public static MetadataPdbReader CreateFromAssembly(string assemblyPath, PEReader peReader)
{
foreach (var provider in peReader.
ReadDebugDirectory().
Where(d => d.Type == DebugDirectoryEntryType.EmbeddedPortablePdb).
Select(dirEntry => peReader.ReadEmbeddedPortablePdbDebugDirectoryData(dirEntry)))
{
return new MetadataPdbReader(provider);
}
try
{
MetadataReaderProvider provider;
string pdbPath;
if (peReader.TryOpenAssociatedPortablePdb(assemblyPath, s => new FileStream(s, FileMode.Open, FileAccess.Read, FileShare.Read), out provider, out pdbPath))
{
return new MetadataPdbReader(provider);
}
}
catch (BadImageFormatException)
{
// Something is wrong with the file.
}
catch (FileNotFoundException)
{
// The PDB file was not found.
}
return null;
}
public void Dispose()
{
provider.Dispose();
}
}
}

143
csharp/extractor/Semmle.Extraction.CIL/PDB/NativePdbReader.cs Normal file
View File

@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection.PortableExecutable;
using Microsoft.DiaSymReader;
using System.Reflection.Metadata.Ecma335;
using System.Reflection.Metadata;
using System.IO;
using System.Reflection;
namespace Semmle.Extraction.PDB
{
/// <summary>
/// A PDB reader using Microsoft.DiaSymReader.Native.
/// This is an unmanaged Windows DLL, which therefore only works on Windows.
/// </summary>
class NativePdbReader : IPdb
{
sealed class Document : ISourceFile
{
readonly ISymUnmanagedDocument document;
public Document(ISymUnmanagedDocument doc)
{
document = doc;
contents = new Lazy<string>(() =>
{
bool isEmbedded;
if (document.HasEmbeddedSource(out isEmbedded) == 0 && isEmbedded)
{
var rawContents = document.GetEmbeddedSource().ToArray();
return System.Text.Encoding.Default.GetString(rawContents);
}
else
{
return File.Exists(Path) ? File.ReadAllText(Path) : null;
}
});
}
public override bool Equals(object obj)
{
var otherDoc = obj as Document;
return otherDoc != null && Path.Equals(otherDoc.Path);
}
public override int GetHashCode() => Path.GetHashCode();
public string Path => document.GetName();
public override string ToString() => Path;
readonly Lazy<string> contents;
public string Contents => contents.Value;
}
public IEnumerable<ISourceFile> SourceFiles => reader.GetDocuments().Select(d => new Document(d));
public IMethod GetMethod(MethodDebugInformationHandle h)
{
int methodToken = MetadataTokens.GetToken(h.ToDefinitionHandle());
var method = reader.GetMethod(methodToken);
if (method != null)
{
int count;
if (method.GetSequencePointCount(out count) != 0 || count == 0)
return null;
var s = method.GetSequencePoints().
Where(sp => !sp.IsHidden).
Select(sp => new SequencePoint(sp.Offset, new Location(new Document(sp.Document), sp.StartLine, sp.StartColumn, sp.EndLine, sp.EndColumn))).
ToArray();
return s.Any() ? new Method { SequencePoints = s } : null;
}
return null;
}
NativePdbReader(string path)
{
pdbStream = new FileStream(path, FileMode.Open);
var metadataProvider = new MdProvider();
reader = SymUnmanagedReaderFactory.CreateReader<ISymUnmanagedReader5>(pdbStream, metadataProvider);
}
readonly ISymUnmanagedReader5 reader;
readonly FileStream pdbStream;
public static NativePdbReader CreateFromAssembly(string assemblyPath, PEReader peReader)
{
// The Native PDB reader uses an unmanaged Windows DLL
// so only works on Windows.
if (!Semmle.Util.Win32.IsWindows())
return null;
var debugDirectory = peReader.ReadDebugDirectory();
foreach (var path in debugDirectory.
Where(d => d.Type == DebugDirectoryEntryType.CodeView).
Select(peReader.ReadCodeViewDebugDirectoryData).
Select(cv => cv.Path).
Where(path => File.Exists(path)))
{
return new NativePdbReader(path);
}
return null;
}
public void Dispose()
{
pdbStream.Dispose();
}
}
/// <summary>
/// This is not used but is seemingly needed in order to use DiaSymReader.
/// </summary>
class MdProvider : ISymReaderMetadataProvider
{
public MdProvider()
{
}
public object GetMetadataImport() => null;
public unsafe bool TryGetStandaloneSignature(int standaloneSignatureToken, out byte* signature, out int length) =>
throw new NotImplementedException();
public bool TryGetTypeDefinitionInfo(int typeDefinitionToken, out string namespaceName, out string typeName, out TypeAttributes attributes, out int baseTypeToken) =>
throw new NotImplementedException();
public bool TryGetTypeDefinitionInfo(int typeDefinitionToken, out string namespaceName, out string typeName, out TypeAttributes attributes) =>
throw new NotImplementedException();
public bool TryGetTypeReferenceInfo(int typeReferenceToken, out string namespaceName, out string typeName, out int resolutionScopeToken) =>
throw new NotImplementedException();
public bool TryGetTypeReferenceInfo(int typeReferenceToken, out string namespaceName, out string typeName) =>
throw new NotImplementedException();
}
}

151
csharp/extractor/Semmle.Extraction.CIL/PDB/PdbReader.cs Normal file
View File

@@ -0,0 +1,151 @@
using Microsoft.DiaSymReader;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
namespace Semmle.Extraction.PDB
{
/// <summary>
/// A sequencepoint is a marker in the source code where you can put a breakpoint, and
/// maps instructions to source code.
/// </summary>
public struct SequencePoint
{
/// <summary>
/// The byte-offset of the instruction.
/// </summary>
public readonly int Offset;
/// <summary>
/// The source location of the instruction.
/// </summary>
public readonly Location Location;
public override string ToString()
{
return string.Format("{0} = {1}", Offset, Location);
}
public SequencePoint(int offset, Location location)
{
Offset = offset;
Location = location;
}
}
/// <summary>
/// A location in source code.
/// </summary>
public sealed class Location
{
/// <summary>
/// The file containing the code.
/// </summary>
public readonly ISourceFile File;
/// <summary>
/// The span of text within the text file.
/// </summary>
public readonly int StartLine, StartColumn, EndLine, EndColumn;
public override string ToString()
{
return string.Format("({0},{1})-({2},{3})", StartLine, StartColumn, EndLine, EndColumn);
}
public override bool Equals(object obj)
{
var otherLocation = obj as Location;
return otherLocation != null &&
File.Equals(otherLocation.File) &&
StartLine == otherLocation.StartLine &&
StartColumn == otherLocation.StartColumn &&
EndLine == otherLocation.EndLine &&
EndColumn == otherLocation.EndColumn;
}
public override int GetHashCode()
{
var h1 = StartLine + 37 * (StartColumn + 51 * (EndLine + 97 * EndColumn));
return File.GetHashCode() + 17 * h1;
}
public Location(ISourceFile file, int startLine, int startCol, int endLine, int endCol)
{
File = file;
StartLine = startLine;
StartColumn = startCol;
EndLine = endLine;
EndColumn = endCol;
}
}
public interface IMethod
{
IEnumerable<SequencePoint> SequencePoints { get; }
Location Location { get; }
}
class Method : IMethod
{
public IEnumerable<SequencePoint> SequencePoints { get; set; }
public Location Location => SequencePoints.First().Location;
}
/// <summary>
/// A source file reference in a PDB file.
/// </summary>
public interface ISourceFile
{
string Path { get; }
/// <summary>
/// The contents of the file.
/// This property is needed in case the contents
/// of the file are embedded in the PDB instead of being on the filesystem.
///
/// null if the contents are unavailable.
/// E.g. if the PDB file exists but the corresponding source files are missing.
/// </summary>
string Contents { get; }
}
/// <summary>
/// Wrapper for reading PDB files.
/// This is needed because there are different libraries for dealing with
/// different types of PDB file, even though they share the same file extension.
/// </summary>
public interface IPdb : IDisposable
{
/// <summary>
/// Gets all source files in this PDB.
/// </summary>
IEnumerable<ISourceFile> SourceFiles { get; }
/// <summary>
/// Look up a method from a given handle.
/// </summary>
/// <param name="methodHandle">The handle to query.</param>
/// <returns>The method information, or null if the method does not have debug information.</returns>
IMethod GetMethod(MethodDebugInformationHandle methodHandle);
}
class PdbReader
{
/// <summary>
/// Returns the PDB information associated with an assembly.
/// </summary>
/// <param name="assemblyPath">The path to the assembly.</param>
/// <param name="peReader">The PE reader for the assembky.</param>
/// <returns>A PdbReader, or null if no PDB information is available.</returns>
public static IPdb Create(string assemblyPath, PEReader peReader)
{
return (IPdb)MetadataPdbReader.CreateFromAssembly(assemblyPath, peReader) ??
NativePdbReader.CreateFromAssembly(assemblyPath, peReader);
}
}
}

35
csharp/extractor/Semmle.Extraction.CIL/Properties/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,35 @@
using System.Reflection;
using System.Runtime.InteropServices;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Semmle.Extraction.CIL")]
[assembly: AssemblyDescription("Semme CIL extractor.")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("Semmle.Extraction.CIL")]
[assembly: AssemblyCopyright("Copyright © Semmle 2017")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("a23d9ec2-8aae-43da-97cb-579f640b89cd")]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

26
csharp/extractor/Semmle.Extraction.CIL/Semmle.Extraction.CIL.csproj Normal file
View File

@@ -0,0 +1,26 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp2.0</TargetFramework>
<AssemblyName>Semmle.Extraction.CIL</AssemblyName>
<RootNamespace>Semmle.Extraction.CIL</RootNamespace>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\Semmle.Extraction\Semmle.Extraction.csproj" />
<ProjectReference Include="..\Semmle.Util\Semmle.Util.csproj" />
</ItemGroup>
<ItemGroup>
<Folder Include="Properties\" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.DiaSymReader" Version="1.3.0" />
<PackageReference Include="Microsoft.DiaSymReader.Native" Version="1.7.0" />
<PackageReference Include="Microsoft.DiaSymReader.PortablePdb" Version="1.5.0" />
</ItemGroup>
</Project>

209
csharp/extractor/Semmle.Extraction.CIL/Tuples.cs Normal file
View File

@@ -0,0 +1,209 @@
using Semmle.Extraction.CIL.Entities;
namespace Semmle.Extraction.CIL
{
internal static class Tuples
{
internal static Tuple assemblies(Assembly assembly, File file, string identifier, string name, string version) =>
new Tuple("assemblies", assembly, file, identifier, name, version);
internal static Tuple cil_abstract(IMember method) =>
new Tuple("cil_abstract", method);
internal static Tuple cil_adder(IEvent member, IMethod method) =>
new Tuple("cil_adder", member, method);
internal static Tuple cil_access(IInstruction i, IEntity m) =>
new Tuple("cil_access", i, m);
internal static Tuple cil_attribute(IAttribute attribute, IEntity @object, IMethod constructor) =>
new Tuple("cil_attribute", attribute, @object, constructor);
internal static Tuple cil_attribute_named_argument(IAttribute attribute, string name, string value) =>
new Tuple("cil_attribute_named_argument", attribute, name, value);
internal static Tuple cil_attribute_positional_argument(IAttribute attribute, int index, string value) =>
new Tuple("cil_attribute_positional_argument", attribute, index, value);
internal static Tuple cil_array_type(IArrayType array, IType element, int rank) =>
new Tuple("cil_array_type", array, element, rank);
internal static Tuple cil_base_class(IType t, IType @base) =>
new Tuple("cil_base_class", t, @base);
internal static Tuple cil_base_interface(IType t, IType @base) =>
new Tuple("cil_base_interface", t, @base);
internal static Tuple cil_class(IMember method) =>
new Tuple("cil_class", method);
internal static Tuple cil_event(IEvent e, IType parent, string name, IType type) =>
new Tuple("cil_event", e, parent, name, type);
internal static Tuple cil_field(IField field, IType parent, string name, IType fieldType) =>
new Tuple("cil_field", field, parent, name, fieldType);
internal static Tuple cil_getter(IProperty member, IMethod method) =>
new Tuple("cil_getter", member, method);
internal static Tuple cil_handler(IExceptionRegion region, IMethodImplementation method, int index, int kind,
IInstruction region_start,
IInstruction region_end,
IInstruction handler_start) =>
new Tuple("cil_handler", region, method, index, kind, region_start, region_end, handler_start);
internal static Tuple cil_handler_filter(IExceptionRegion region, IInstruction filter_start) =>
new Tuple("cil_handler_filter", region, filter_start);
internal static Tuple cil_handler_type(IExceptionRegion region, Type t) =>
new Tuple("cil_handler_type", region, t);
internal static Tuple cil_implements(IMethod derived, IMethod declaration) =>
new Tuple("cil_implements", derived, declaration);
internal static Tuple cil_instruction(IInstruction instruction, int opcode, int index, IMethodImplementation parent) =>
new Tuple("cil_instruction", instruction, opcode, index, parent);
internal static Tuple cil_instruction_location(Instruction i, ILocation loc) =>
new Tuple("cil_instruction_location", i, loc);
internal static Tuple cil_interface(IMember method) =>
new Tuple("cil_interface", method);
internal static Tuple cil_internal(IMember modifiable) =>
new Tuple("cil_internal", modifiable);
internal static Tuple cil_jump(IInstruction from, IInstruction to) =>
new Tuple("cil_jump", from, to);
internal static Tuple cil_local_variable(ILocal l, IMethodImplementation m, int i, Type t) =>
new Tuple("cil_local_variable", l, m, i, t);
internal static Tuple cil_method(IMethod method, string name, IType declType, IType returnType) =>
new Tuple("cil_method", method, name, declType, returnType);
internal static Tuple cil_method_implementation(IMethodImplementation impl, IMethod method, IAssembly assembly) =>
new Tuple("cil_method_implementation", impl, method, assembly);
internal static Tuple cil_method_location(IMethod m, ILocation a) =>
new Tuple("cil_method_location", m, a);
internal static Tuple cil_method_source_declaration(IMethod method, IMethod sourceDecl) =>
new Tuple("cil_method_source_declaration", method, sourceDecl);
internal static Tuple cil_method_stack_size(IMethodImplementation method, int stackSize) =>
new Tuple("cil_method_stack_size", method, stackSize);
internal static Tuple cil_newslot(IMethod method) =>
new Tuple("cil_newslot", method);
internal static Tuple cil_parameter(IParameter p, IMethod m, int i, IType t) =>
new Tuple("cil_parameter", p, m, i, t);
internal static Tuple cil_parameter_in(IParameter p) =>
new Tuple("cil_parameter_in", p);
internal static Tuple cil_parameter_out(IParameter p) =>
new Tuple("cil_parameter_out", p);
internal static Tuple cil_pointer_type(IPointerType t, IType pointee) =>
new Tuple("cil_pointer_type", t, pointee);
internal static Tuple cil_private(IMember modifiable) =>
new Tuple("cil_private", modifiable);
internal static Tuple cil_protected(IMember modifiable) =>
new Tuple("cil_protected", modifiable);
internal static Tuple cil_property(IProperty p, IType parent, string name, IType propType) =>
new Tuple("cil_property", p, parent, name, propType);
internal static Tuple cil_public(IMember modifiable) =>
new Tuple("cil_public", modifiable);
internal static Tuple cil_raiser(IEvent member, IMethod method) =>
new Tuple("cil_raiser", member, method);
internal static Tuple cil_requiresecobject(IMethod method) =>
new Tuple("cil_requiresecobject", method);
internal static Tuple cil_remover(IEvent member, IMethod method) =>
new Tuple("cil_remover", member, method);
internal static Tuple cil_sealed(IMember modifiable) =>
new Tuple("cil_sealed", modifiable);
internal static Tuple cil_security(IMember method) =>
new Tuple("cil_security", method);
internal static Tuple cil_setter(IProperty member, IMethod method) =>
new Tuple("cil_setter", member, method);
internal static Tuple cil_specialname(IMethod method) =>
new Tuple("cil_specialname", method);
internal static Tuple cil_static(IMember modifiable) =>
new Tuple("cil_static", modifiable);
internal static Tuple cil_switch(IInstruction from, int index, IInstruction to) =>
new Tuple("cil_switch", from, index, to);
internal static Tuple cil_type(IType t, string name, CilTypeKind kind, ITypeContainer parent, IType sourceDecl) =>
new Tuple("cil_type", t, name, (int)kind, parent, sourceDecl);
internal static Tuple cil_type_argument(ITypeContainer constructedTypeOrMethod, int index, IType argument) =>
new Tuple("cil_type_argument", constructedTypeOrMethod, index, argument);
internal static Tuple cil_type_location(IType t, IAssembly a) =>
new Tuple("cil_type_location", t, a);
internal static Tuple cil_type_parameter(ITypeContainer unboundTypeOrMethod, int index, ITypeParameter parameter) =>
new Tuple("cil_type_parameter", unboundTypeOrMethod, index, parameter);
internal static Tuple cil_typeparam_covariant(ITypeParameter p) =>
new Tuple("cil_typeparam_covariant", p);
internal static Tuple cil_typeparam_contravariant(ITypeParameter p) =>
new Tuple("cil_typeparam_contravariant", p);
internal static Tuple cil_typeparam_class(ITypeParameter p) =>
new Tuple("cil_typeparam_class", p);
internal static Tuple cil_typeparam_constraint(ITypeParameter p, IType constraint) =>
new Tuple("cil_typeparam_constraint", p, constraint);
internal static Tuple cil_typeparam_new(ITypeParameter p) =>
new Tuple("cil_typeparam_new", p);
internal static Tuple cil_typeparam_struct(ITypeParameter p) =>
new Tuple("cil_typeparam_struct", p);
internal static Tuple cil_value(IInstruction i, string value) =>
new Tuple("cil_value", i, value);
internal static Tuple cil_virtual(IMethod method) =>
new Tuple("cil_virtual", method);
internal static Tuple containerparent(IFolder parent, IFileOrFolder child) =>
new Tuple("containerparent", parent, child);
internal static Tuple files(IFile file, string fullName, string name, string extension) =>
new Tuple("files", file, fullName, name, extension, 0);
internal static Tuple file_extraction_mode(IFile file, int mode) =>
new Tuple("file_extraction_mode", file, mode);
internal static Tuple folders(IFolder folder, string path, string name) =>
new Tuple("folders", folder, path, name);
internal static Tuple locations_default(ISourceLocation label, IFile file, int startLine, int startCol, int endLine, int endCol) =>
new Tuple("locations_default", label, file, startLine, startCol, endLine, endCol);
internal static Tuple namespaces(INamespace ns, string name) =>
new Tuple("namespaces", ns, name);
internal static Tuple parent_namespace(ITypeContainer child, INamespace parent) =>
new Tuple("parent_namespace", child, parent);
}
}

13
csharp/extractor/Semmle.Extraction.CSharp.Driver/Driver.cs Normal file
View File

@@ -0,0 +1,13 @@
namespace Semmle.Extraction.CSharp
{
/// <summary>
/// A command-line driver for the extractor.
/// </summary>
public class Driver
{
static int Main(string[] args)
{
return (int)Extractor.Run(args);
}
}
}

35
csharp/extractor/Semmle.Extraction.CSharp.Driver/Properties/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,35 @@
using System.Reflection;
using System.Runtime.InteropServices;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Semmle.Extraction.CSharp.Driver")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("Semmle.Extraction.CSharp.Driver")]
[assembly: AssemblyCopyright("Copyright © 2015")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("c02a2b0e-8884-4b82-8275-ea282403a775")]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

15
csharp/extractor/Semmle.Extraction.CSharp.Driver/Semmle.Extraction.CSharp.Driver.csproj Normal file
View File

@@ -0,0 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.0</TargetFramework>
<AssemblyName>Semmle.Extraction.CSharp.Driver</AssemblyName>
<RootNamespace>Semmle.Extraction.CSharp.Driver</RootNamespace>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\Semmle.Extraction.CSharp\Semmle.Extraction.CSharp.csproj" />
</ItemGroup>
</Project>

184
csharp/extractor/Semmle.Extraction.CSharp.Standalone/AssemblyCache.cs Normal file
View File

@@ -0,0 +1,184 @@
using System.Collections.Generic;
using System.Linq;
using System.IO;
using System;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// Manages the set of assemblies.
/// Searches for assembly DLLs, indexes them and provides
/// a lookup facility from assembly ID to filename.
/// </summary>
class AssemblyCache
{
/// <summary>
/// Locate all reference files and index them.
/// </summary>
/// <param name="dirs">Directories to search.</param>
/// <param name="progress">Callback for progress.</param>
public AssemblyCache(IEnumerable<string> dirs, IProgressMonitor progress)
{
foreach (var dir in dirs)
{
progress.FindingFiles(dir);
AddReferenceDirectory(dir);
}
IndexReferences();
}
/// <summary>
/// Finds all assemblies nested within a directory
/// and adds them to its index.
/// (Indexing is performed at a later stage by IndexReferences()).
/// </summary>
/// <param name="dir">The directory to index.</param>
/// <returns>The number of DLLs within this directory.</returns>
int AddReferenceDirectory(string dir)
{
int count = 0;
foreach (var dll in new DirectoryInfo(dir).EnumerateFiles("*.dll", SearchOption.AllDirectories))
{
dlls.Add(dll.FullName);
++count;
}
return count;
}
/// <summary>
/// Indexes all DLLs we have located.
/// Because this is a potentially time-consuming operation, it is put into a separate stage.
/// </summary>
void IndexReferences()
{
// Read all of the files
foreach (var filename in dlls)
{
var info = AssemblyInfo.ReadFromFile(filename);
if (info.Valid)
{
assemblyInfo[filename] = info;
}
else
{
failedDlls.Add(filename);
}
}
// Index "assemblyInfo" by version string
// The OrderBy is used to ensure that we by default select the highest version number.
foreach (var info in assemblyInfo.Values.OrderBy(info => info.Id))
{
foreach (var index in info.IndexStrings)
references[index] = info;
}
}
/// <summary>
/// The number of DLLs which are assemblies.
/// </summary>
public int AssemblyCount => assemblyInfo.Count;
/// <summary>
/// The number of DLLs which weren't assemblies. (E.g. C++).
/// </summary>
public int NonAssemblyCount => failedDlls.Count;
/// <summary>
/// Given an assembly id, determine its full info.
/// </summary>
/// <param name="id">The given assembly id.</param>
/// <returns>The information about the assembly.</returns>
public AssemblyInfo ResolveReference(string id)
{
// Fast path if we've already seen this before.
if (failedReferences.Contains(id))
return AssemblyInfo.Invalid;
var query = AssemblyInfo.MakeFromId(id);
id = query.Id; // Sanitise the id.
// Look up the id in our references map.
AssemblyInfo result;
if (references.TryGetValue(id, out result))
{
// The string is in the references map.
return result;
}
else
{
// Attempt to load the reference from the GAC.
try
{
var loadedAssembly = System.Reflection.Assembly.ReflectionOnlyLoad(id);
if (loadedAssembly != null)
{
// The assembly was somewhere we haven't indexed before.
// Add this assembly to our index so that subsequent lookups are faster.
result = AssemblyInfo.MakeFromAssembly(loadedAssembly);
references[id] = result;
assemblyInfo[loadedAssembly.Location] = result;
return result;
}
}
catch (FileNotFoundException)
{
// A suitable assembly could not be found
}
catch (FileLoadException)
{
// The assembly cannot be loaded for some reason
// e.g. The name is malformed.
}
catch (PlatformNotSupportedException)
{
// .NET Core does not have a GAC.
}
// Fallback position - locate the assembly by its lower-case name only.
var asmName = query.Name.ToLowerInvariant();
if (references.TryGetValue(asmName, out result))
{
references[asmName] = result; // Speed up the next time the same string is resolved
return result;
}
failedReferences.Add(id); // Fail early next time
return AssemblyInfo.Invalid;
}
}
/// <summary>
/// All the assemblies we have indexed.
/// </summary>
public IEnumerable<AssemblyInfo> AllAssemblies => assemblyInfo.Select(a => a.Value);
/// <summary>
/// Retrieve the assembly info of a pre-cached assembly.
/// </summary>
/// <param name="filepath">The filename to query.</param>
/// <returns>The assembly info.</returns>
public AssemblyInfo GetAssemblyInfo(string filepath) => assemblyInfo[filepath];
// List of pending DLLs to index.
readonly List<string> dlls = new List<string>();
// Map from filename to assembly info.
readonly Dictionary<string, AssemblyInfo> assemblyInfo = new Dictionary<string, AssemblyInfo>();
// List of DLLs which are not assemblies.
// We probably don't need to keep this
readonly List<string> failedDlls = new List<string>();
// Map from assembly id (in various formats) to the full info.
readonly Dictionary<string, AssemblyInfo> references = new Dictionary<string, AssemblyInfo>();
// Set of failed assembly ids.
readonly HashSet<string> failedReferences = new HashSet<string>();
}
}

179
csharp/extractor/Semmle.Extraction.CSharp.Standalone/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,179 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Security.Cryptography;
using System.Text;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// Stores information about an assembly file (DLL).
/// </summary>
sealed class AssemblyInfo
{
/// <summary>
/// The file containing the assembly.
/// </summary>
public string Filename { get; private set; }
/// <summary>
/// Was the information correctly determined?
/// </summary>
public bool Valid { get; private set; }
/// <summary>
/// The short name of this assembly.
/// </summary>
public string Name { get; private set; }
/// <summary>
/// The version number of this assembly.
/// </summary>
public System.Version Version { get; private set; }
/// <summary>
/// The public key token of the assembly.
/// </summary>
public string PublicKeyToken { get; private set; }
/// <summary>
/// The culture.
/// </summary>
public string Culture { get; private set; }
/// <summary>
/// Get/parse a canonical ID of this assembly.
/// </summary>
public string Id
{
get
{
var result = Name;
if (Version != null)
result = string.Format("{0}, Version={1}", result, Version);
if (Culture != null)
result = string.Format("{0}, Culture={1}", result, Culture);
if (PublicKeyToken != null)
result = string.Format("{0}, PublicKeyToken={1}", result, PublicKeyToken);
return result;
}
private set
{
var sections = value.Split(new string[] { ", " }, StringSplitOptions.None);
Name = sections.First();
foreach (var section in sections.Skip(1))
{
if (section.StartsWith("Version="))
Version = new Version(section.Substring(8));
else if (section.StartsWith("Culture="))
Culture = section.Substring(8);
else if (section.StartsWith("PublicKeyToken="))
PublicKeyToken = section.Substring(15);
// else: Some other field like processorArchitecture - ignore.
}
}
}
public override string ToString() => Id;
/// <summary>
/// Gets a list of canonical search strings for this assembly.
/// </summary>
public IEnumerable<string> IndexStrings
{
get
{
yield return Id;
if (Version != null)
{
if (Culture != null) yield return string.Format("{0}, Version={1}, Culture={2}", Name, Version, Culture);
yield return string.Format("{0}, Version={1}", Name, Version);
}
yield return Name;
yield return Name.ToLowerInvariant();
}
}
/// <summary>
/// Get an invalid assembly info (Valid==false).
/// </summary>
public static AssemblyInfo Invalid { get; } = new AssemblyInfo();
private AssemblyInfo() { }
/// <summary>
/// Get AssemblyInfo from a loaded Assembly.
/// </summary>
/// <param name="assembly">The assembly.</param>
/// <returns>Info about the assembly.</returns>
public static AssemblyInfo MakeFromAssembly(Assembly assembly) => new AssemblyInfo() { Valid = true, Filename = assembly.Location, Id = assembly.FullName };
/// <summary>
/// Parse an assembly name/Id into an AssemblyInfo,
/// populating the available fields and leaving the others null.
/// </summary>
/// <param name="id">The assembly name/Id.</param>
/// <returns>The deconstructed assembly info.</returns>
public static AssemblyInfo MakeFromId(string id) => new AssemblyInfo() { Valid = true, Id = id };
/// <summary>
/// Reads the assembly info from a file.
/// This uses System.Reflection.Metadata, which is a very performant and low-level
/// library. This is very convenient when scanning hundreds of DLLs at a time.
/// </summary>
/// <param name="filename">The full filename of the assembly.</param>
/// <returns>The information about the assembly.</returns>
public static AssemblyInfo ReadFromFile(string filename)
{
var result = new AssemblyInfo() { Filename = filename };
try
{
/* This method is significantly faster and more lightweight than using
* System.Reflection.Assembly.ReflectionOnlyLoadFrom. It also allows
* loading the same assembly from different locations.
*/
using (var pereader = new System.Reflection.PortableExecutable.PEReader(new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Read)))
{
var metadata = pereader.GetMetadata();
unsafe
{
var reader = new System.Reflection.Metadata.MetadataReader(metadata.Pointer, metadata.Length);
var def = reader.GetAssemblyDefinition();
// This is how you compute the public key token from the full public key.
// The last 8 bytes of the SHA1 of the public key.
var publicKey = reader.GetBlobBytes(def.PublicKey);
var publicKeyToken = sha1.ComputeHash(publicKey);
var publicKeyString = new StringBuilder();
foreach (var b in publicKeyToken.Skip(12).Reverse())
publicKeyString.AppendFormat("{0:x2}", b);
result.Name = reader.GetString(def.Name);
result.Version = def.Version;
result.Culture = def.Culture.IsNil ? "neutral" : reader.GetString(def.Culture);
result.PublicKeyToken = publicKeyString.ToString();
result.Valid = true;
}
}
}
catch (BadImageFormatException)
{
// The DLL wasn't an assembly -> result.Valid = false.
}
catch (InvalidOperationException)
{
// Some other failure -> result.Valid = false.
}
return result;
}
static readonly SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider();
}
}

312
csharp/extractor/Semmle.Extraction.CSharp.Standalone/BuildAnalysis.cs Normal file
View File

@@ -0,0 +1,312 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using Semmle.Util;
using Semmle.Extraction.CSharp.Standalone;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// The output of a build analysis.
/// </summary>
interface IBuildAnalysis
{
/// <summary>
/// Full filepaths of external references.
/// </summary>
IEnumerable<string> ReferenceFiles { get; }
/// <summary>
/// Full filepaths of C# source files from project files.
/// </summary>
IEnumerable<string> ProjectSourceFiles { get; }
/// <summary>
/// Full filepaths of C# source files in the filesystem.
/// </summary>
IEnumerable<string> AllSourceFiles { get; }
/// <summary>
/// The assembly IDs which could not be resolved.
/// </summary>
IEnumerable<string> UnresolvedReferences { get; }
/// <summary>
/// List of source files referenced by projects but
/// which were not found in the filesystem.
/// </summary>
IEnumerable<string> MissingSourceFiles { get; }
}
/// <summary>
/// Main implementation of the build analysis.
/// </summary>
class BuildAnalysis : IBuildAnalysis
{
readonly AssemblyCache assemblyCache;
readonly NugetPackages nuget;
readonly IProgressMonitor progressMonitor;
HashSet<string> usedReferences = new HashSet<string>();
readonly HashSet<string> usedSources = new HashSet<string>();
readonly HashSet<string> missingSources = new HashSet<string>();
readonly Dictionary<string, string> unresolvedReferences = new Dictionary<string, string>();
readonly DirectoryInfo sourceDir;
int failedProjects, succeededProjects;
readonly string[] allSources;
int conflictedReferences = 0;
/// <summary>
/// Performs a C# build analysis.
/// </summary>
/// <param name="options">Analysis options from the command line.</param>
/// <param name="progress">Display of analysis progress.</param>
public BuildAnalysis(Options options, IProgressMonitor progress)
{
progressMonitor = progress;
sourceDir = new DirectoryInfo(options.SrcDir);
progressMonitor.FindingFiles(options.SrcDir);
allSources = sourceDir.GetFiles("*.cs", SearchOption.AllDirectories).
Select(d => d.FullName).
Where(d => !options.ExcludesFile(d)).
ToArray();
var dllDirNames = options.DllDirs.Select(Path.GetFullPath);
if (options.UseNuGet)
{
nuget = new NugetPackages(sourceDir.FullName);
ReadNugetFiles();
dllDirNames = dllDirNames.Concat(Enumerators.Singleton(nuget.PackageDirectory));
}
// Find DLLs in the .Net Framework
if (options.ScanNetFrameworkDlls)
{
dllDirNames = dllDirNames.Concat(Runtime.Runtimes.Take(1));
}
assemblyCache = new BuildAnalyser.AssemblyCache(dllDirNames, progress);
// Analyse all .csproj files in the source tree.
if (options.SolutionFile != null)
{
AnalyseSolution(options.SolutionFile);
}
else if (options.AnalyseCsProjFiles)
{
AnalyseProjectFiles();
}
if (!options.AnalyseCsProjFiles)
{
usedReferences = new HashSet<string>(assemblyCache.AllAssemblies.Select(a => a.Filename));
}
ResolveConflicts();
if (options.UseMscorlib)
{
UseReference(typeof(object).Assembly.Location);
}
// Output the findings
foreach (var r in usedReferences)
{
progressMonitor.ResolvedReference(r);
}
foreach (var r in unresolvedReferences)
{
progressMonitor.UnresolvedReference(r.Key, r.Value);
}
progressMonitor.Summary(
AllSourceFiles.Count(),
ProjectSourceFiles.Count(),
MissingSourceFiles.Count(),
ReferenceFiles.Count(),
UnresolvedReferences.Count(),
conflictedReferences,
succeededProjects + failedProjects,
failedProjects);
}
/// <summary>
/// Resolves conflicts between all of the resolved references.
/// If the same assembly name is duplicated with different versions,
/// resolve to the higher version number.
/// </summary>
void ResolveConflicts()
{
var sortedReferences = usedReferences.
Select(r => assemblyCache.GetAssemblyInfo(r)).
OrderBy(r => r.Version).
ToArray();
Dictionary<string, AssemblyInfo> finalAssemblyList = new Dictionary<string, AssemblyInfo>();
// Pick the highest version for each assembly name
foreach (var r in sortedReferences)
finalAssemblyList[r.Name] = r;
// Update the used references list
usedReferences = new HashSet<string>(finalAssemblyList.Select(r => r.Value.Filename));
// Report the results
foreach (var r in sortedReferences)
{
var resolvedInfo = finalAssemblyList[r.Name];
if (resolvedInfo.Version != r.Version)
{
progressMonitor.ResolvedConflict(r.Id, resolvedInfo.Id);
++conflictedReferences;
}
}
}
/// <summary>
/// Find and restore NuGet packages.
/// </summary>
void ReadNugetFiles()
{
nuget.FindPackages();
nuget.InstallPackages(progressMonitor);
}
/// <summary>
/// Store that a particular reference file is used.
/// </summary>
/// <param name="reference">The filename of the reference.</param>
void UseReference(string reference)
{
usedReferences.Add(reference);
}
/// <summary>
/// Store that a particular source file is used (by a project file).
/// </summary>
/// <param name="sourceFile">The source file.</param>
void UseSource(FileInfo sourceFile)
{
if (sourceFile.Exists)
{
usedSources.Add(sourceFile.FullName);
}
else
{
missingSources.Add(sourceFile.FullName);
}
}
/// <summary>
/// The list of resolved reference files.
/// </summary>
public IEnumerable<string> ReferenceFiles => this.usedReferences;
/// <summary>
/// The list of source files used in projects.
/// </summary>
public IEnumerable<string> ProjectSourceFiles => usedSources;
/// <summary>
/// All of the source files in the source directory.
/// </summary>
public IEnumerable<string> AllSourceFiles => allSources;
/// <summary>
/// List of assembly IDs which couldn't be resolved.
/// </summary>
public IEnumerable<string> UnresolvedReferences => this.unresolvedReferences.Select(r => r.Key);
/// <summary>
/// List of source files which were mentioned in project files but
/// do not exist on the file system.
/// </summary>
public IEnumerable<string> MissingSourceFiles => missingSources;
/// <summary>
/// Record that a particular reference couldn't be resolved.
/// Note that this records at most one project file per missing reference.
/// </summary>
/// <param name="id">The assembly ID.</param>
/// <param name="projectFile">The project file making the reference.</param>
void UnresolvedReference(string id, string projectFile)
{
unresolvedReferences[id] = projectFile;
}
/// <summary>
/// Performs an analysis of all .csproj files.
/// </summary>
void AnalyseProjectFiles()
{
AnalyseProjectFiles(sourceDir.GetFiles("*.csproj", SearchOption.AllDirectories));
}
/// <summary>
/// Reads all the source files and references from the given list of projects.
/// </summary>
/// <param name="projectFiles">The list of projects to analyse.</param>
void AnalyseProjectFiles(FileInfo[] projectFiles)
{
progressMonitor.AnalysingProjectFiles(projectFiles.Count());
foreach (var proj in projectFiles)
{
try
{
var csProj = new CsProjFile(proj);
foreach (var @ref in csProj.References)
{
AssemblyInfo resolved = assemblyCache.ResolveReference(@ref);
if (!resolved.Valid)
{
UnresolvedReference(@ref, proj.FullName);
}
else
{
UseReference(resolved.Filename);
}
}
foreach (var src in csProj.Sources)
{
// Make a note of which source files the projects use.
// This information doesn't affect the build but is dumped
// as diagnostic output.
UseSource(new FileInfo(src));
}
++succeededProjects;
}
catch (Exception ex)
{
++failedProjects;
progressMonitor.FailedProjectFile(proj.FullName, ex.Message);
}
}
}
/// <summary>
/// Delete packages directory.
/// </summary>
public void Cleanup()
{
if (nuget != null) nuget.Cleanup(progressMonitor);
}
/// <summary>
/// Analyse all project files in a given solution only.
/// </summary>
/// <param name="solutionFile">The filename of the solution.</param>
public void AnalyseSolution(string solutionFile)
{
var sln = new SolutionFile(solutionFile);
AnalyseProjectFiles(sln.Projects.Select(p => new FileInfo(p)).ToArray());
}
}
}

120
csharp/extractor/Semmle.Extraction.CSharp.Standalone/CsProjFile.cs Normal file
View File

@@ -0,0 +1,120 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Xml;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// Represents a .csproj file and reads information from it.
/// </summary>
class CsProjFile
{
/// <summary>
/// Reads the .csproj file.
/// </summary>
/// <param name="filename">The .csproj file.</param>
public CsProjFile(FileInfo filename)
{
try
{
// This can fail if the .csproj is invalid or has
// unrecognised content or is the wrong version.
// This currently always fails on Linux because
// Microsoft.Build is not cross platform.
ReadMsBuildProject(filename);
}
catch
{
// There was some reason why the project couldn't be loaded.
// Fall back to reading the Xml document directly.
// This method however doesn't handle variable expansion.
ReadProjectFileAsXml(filename);
}
}
/// <summary>
/// Read the .csproj file using Microsoft Build.
/// This occasionally fails if the project file is incompatible for some reason,
/// and there seems to be no way to make it succeed. Fails on Linux.
/// </summary>
/// <param name="filename">The file to read.</param>
public void ReadMsBuildProject(FileInfo filename)
{
var msbuildProject = new Microsoft.Build.Execution.ProjectInstance(filename.FullName);
references = msbuildProject.
Items.
Where(item => item.ItemType == "Reference").
Select(item => item.EvaluatedInclude).
ToArray();
csFiles = msbuildProject.Items
.Where(item => item.ItemType == "Compile")
.Select(item => item.GetMetadataValue("FullPath"))
.Where(fn => fn.EndsWith(".cs"))
.ToArray();
}
/// <summary>
/// Reads the .csproj file directly as XML.
/// This doesn't handle variables etc, and should only used as a
/// fallback if ReadMsBuildProject() fails.
/// </summary>
/// <param name="filename">The .csproj file.</param>
public void ReadProjectFileAsXml(FileInfo filename)
{
var projFile = new XmlDocument();
var mgr = new XmlNamespaceManager(projFile.NameTable);
mgr.AddNamespace("msbuild", "http://schemas.microsoft.com/developer/msbuild/2003");
projFile.Load(filename.FullName);
var projDir = filename.Directory;
var root = projFile.DocumentElement;
references =
root.SelectNodes("/msbuild:Project/msbuild:ItemGroup/msbuild:Reference/@Include", mgr).
NodeList().
Select(node => node.Value).
ToArray();
var relativeCsIncludes =
root.SelectNodes("/msbuild:Project/msbuild:ItemGroup/msbuild:Compile/@Include", mgr).
NodeList().
Select(node => node.Value).
ToArray();
csFiles = relativeCsIncludes.
Select(cs => Path.DirectorySeparatorChar == '/' ? cs.Replace("\\", "/") : cs).
Select(f => Path.GetFullPath(Path.Combine(projDir.FullName, f))).
ToArray();
}
string[] references;
string[] csFiles;
/// <summary>
/// The list of references as a list of assembly IDs.
/// </summary>
public IEnumerable<string> References => references;
/// <summary>
/// The list of C# source files in full path format.
/// </summary>
public IEnumerable<string> Sources => csFiles;
}
static class XmlNodeHelper
{
/// <summary>
/// Helper to convert an XmlNodeList into an IEnumerable.
/// This allows it to be used with Linq.
/// </summary>
/// <param name="list">The list to convert.</param>
/// <returns>A more useful data type.</returns>
public static IEnumerable<XmlNode> NodeList(this XmlNodeList list)
{
foreach (var i in list)
yield return i as XmlNode;
}
}
}

195
csharp/extractor/Semmle.Extraction.CSharp.Standalone/NugetPackages.cs Normal file
View File

@@ -0,0 +1,195 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// Manage the downloading of NuGet packages.
/// Locates packages in a source tree and downloads all of the
/// referenced assemblies to a temp folder.
/// </summary>
class NugetPackages
{
/// <summary>
/// Create the package manager for a specified source tree.
/// </summary>
/// <param name="sourceDir">The source directory.</param>
public NugetPackages(string sourceDir)
{
SourceDirectory = sourceDir;
PackageDirectory = computeTempDirectory(sourceDir);
// Expect nuget.exe to be in a `nuget` directory under the directory containing this exe.
var currentAssembly = System.Reflection.Assembly.GetExecutingAssembly().Location;
nugetExe = Path.Combine(Path.GetDirectoryName(currentAssembly), "nuget", "nuget.exe");
if (!File.Exists(nugetExe))
throw new FileNotFoundException(string.Format("NuGet could not be found at {0}", nugetExe));
}
/// <summary>
/// Locate all NuGet packages but don't download them yet.
/// </summary>
public void FindPackages()
{
packages = new DirectoryInfo(SourceDirectory).
EnumerateFiles("packages.config", SearchOption.AllDirectories).
ToArray();
}
// List of package files to download.
FileInfo[] packages;
/// <summary>
/// The list of package files.
/// </summary>
public IEnumerable<FileInfo> PackageFiles => packages;
// Whether to delete the packages directory prior to each run.
// Makes each build more reproducible.
const bool cleanupPackages = true;
public void Cleanup(IProgressMonitor pm)
{
var packagesDirectory = new DirectoryInfo(PackageDirectory);
if (packagesDirectory.Exists)
{
try
{
packagesDirectory.Delete(true);
}
catch (System.IO.IOException ex)
{
pm.Warning(string.Format("Couldn't delete package directory - it's probably held open by something else: {0}", ex.Message));
}
}
}
/// <summary>
/// Download the packages to the temp folder.
/// </summary>
/// <param name="pm">The progress monitor used for reporting errors etc.</param>
public void InstallPackages(IProgressMonitor pm)
{
if (cleanupPackages)
{
Cleanup(pm);
}
var packagesDirectory = new DirectoryInfo(PackageDirectory);
if (!Directory.Exists(PackageDirectory))
{
packagesDirectory.Create();
}
foreach (var package in packages)
{
RestoreNugetPackage(package.FullName, pm);
}
}
/// <summary>
/// The source directory used.
/// </summary>
public string SourceDirectory
{
get;
private set;
}
/// <summary>
/// The computed packages directory.
/// This will be in the Temp location
/// so as to not trample the source tree.
/// </summary>
public string PackageDirectory
{
get;
private set;
}
readonly SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider();
/// <summary>
/// Computes a unique temp directory for the packages associated
/// with this source tree. Use a SHA1 of the directory name.
/// </summary>
/// <param name="srcDir"></param>
/// <returns>The full path of the temp directory.</returns>
string computeTempDirectory(string srcDir)
{
var bytes = Encoding.Unicode.GetBytes(srcDir);
var sha = sha1.ComputeHash(bytes);
var sb = new StringBuilder();
foreach (var b in sha.Take(8))
sb.AppendFormat("{0:x2}", b);
return Path.Combine(Path.GetTempPath(), "Semmle", "packages", sb.ToString());
}
/// <summary>
/// Restore all files in a specified package.
/// </summary>
/// <param name="package">The package file.</param>
/// <param name="pm">Where to log progress/errors.</param>
void RestoreNugetPackage(string package, IProgressMonitor pm)
{
pm.NugetInstall(package);
/* Use nuget.exe to install a package.
* Note that there is a clutch of NuGet assemblies which could be used to
* invoke this directly, which would arguably be nicer. However they are
* really unwieldy and this solution works for now.
*/
string exe, args;
if (Util.Win32.IsWindows())
{
exe = nugetExe;
args = string.Format("install -OutputDirectory {0} {1}", PackageDirectory, package);
}
else
{
exe = "mono";
args = string.Format("{0} install -OutputDirectory {1} {2}", nugetExe, PackageDirectory, package);
}
var pi = new ProcessStartInfo(exe, args)
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
try
{
using (var p = Process.Start(pi))
{
string output = p.StandardOutput.ReadToEnd();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();
if (p.ExitCode != 0)
{
pm.FailedNugetCommand(pi.FileName, pi.Arguments, output + error);
}
}
}
catch (Exception e)
when (e is System.ComponentModel.Win32Exception || e is FileNotFoundException)
{
pm.FailedNugetCommand(pi.FileName, pi.Arguments, e.Message);
}
}
readonly string nugetExe;
}
}

178
csharp/extractor/Semmle.Extraction.CSharp.Standalone/Options.cs Normal file
View File

@@ -0,0 +1,178 @@
using Semmle.Util.Logging;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Semmle.Util;
namespace Semmle.Extraction.CSharp.Standalone
{
/// <summary>
/// The options controlling standalone extraction.
/// </summary>
public sealed class Options : CommonOptions
{
public override bool handleFlag(string key, bool value)
{
switch(key)
{
case "silent":
Verbosity = value ? Verbosity.Off : Verbosity.Info;
return true;
case "help":
Help = true;
return true;
case "dry-run":
SkipExtraction = value;
return true;
case "skip-nuget":
UseNuGet = !value;
return true;
case "all-references":
AnalyseCsProjFiles = !value;
return true;
case "stdlib":
UseMscorlib = value;
return true;
case "skip-dotnet":
ScanNetFrameworkDlls = !value;
return true;
default:
return base.handleFlag(key, value);
}
}
public override bool handleOption(string key, string value)
{
switch(key)
{
case "exclude":
Excludes.Add(value);
return true;
case "references":
DllDirs.Add(value);
return true;
default:
return base.handleOption(key, value);
}
}
public override bool handleArgument(string arg)
{
SolutionFile = arg;
var fi = new FileInfo(SolutionFile);
if (!fi.Exists)
{
System.Console.WriteLine("Error: The solution {0} does not exist", fi.FullName);
Errors = true;
}
return true;
}
public override void invalidArgument(string argument)
{
System.Console.WriteLine($"Error: Invalid argument {argument}");
Errors = true;
}
/// <summary>
/// Files/patterns to exclude.
/// </summary>
public IList<string> Excludes = new List<string>();
/// <summary>
/// The number of concurrent threads to use.
/// </summary>
public int NumberOfThreads = Semmle.Extraction.Extractor.DefaultNumberOfThreads;
/// <summary>
/// The directory containing the source code;
/// </summary>
public readonly string SrcDir = System.IO.Directory.GetCurrentDirectory();
/// <summary>
/// Whether to analyse NuGet packages.
/// </summary>
public bool UseNuGet = true;
/// <summary>
/// Directories to search DLLs in.
/// </summary>
public IList<string> DllDirs = new List<string>();
/// <summary>
/// Whether to search the .Net framework directory.
/// </summary>
public bool ScanNetFrameworkDlls = true;
/// <summary>
/// Whether to use mscorlib as a reference.
/// </summary>
public bool UseMscorlib = true;
/// <summary>
/// Whether to search .csproj files.
/// </summary>
public bool AnalyseCsProjFiles = true;
/// <summary>
/// The solution file to analyse, or null if not specified.
/// </summary>
public string SolutionFile;
/// <summary>
/// Whether the extraction phase should be skipped (dry-run).
/// </summary>
public bool SkipExtraction = false;
/// <summary>
/// Whether errors were encountered parsing the arguments.
/// </summary>
public bool Errors = false;
/// <summary>
/// Whether to show help.
/// </summary>
public bool Help = false;
/// <summary>
/// Determine whether the given path should be excluded.
/// </summary>
/// <param name="path">The path to query.</param>
/// <returns>True iff the path matches an exclusion.</returns>
public bool ExcludesFile(string path)
{
return Excludes.Any(ex => path.Contains(ex));
}
/// <summary>
/// Outputs the command line options to the console.
/// </summary>
public void ShowHelp(System.IO.TextWriter output)
{
output.WriteLine("C# standalone extractor\n\nExtracts a C# project in the current directory without performing a build.\n");
output.WriteLine("Additional options:\n");
output.WriteLine(" xxx.sln Restrict sources to given solution");
output.WriteLine(" --exclude:xxx Exclude a file or directory (can be specified multiple times)");
output.WriteLine(" --references:xxx Scan additional files or directories for assemblies (can be specified multiple times)");
output.WriteLine(" --skip-dotnet Do not reference the .Net Framework");
output.WriteLine(" --dry-run Stop before extraction");
output.WriteLine(" --skip-nuget Do not download nuget packages");
output.WriteLine(" --all-references Use all references (default is to only use references in .csproj files)");
output.WriteLine(" --nostdlib Do not link mscorlib.dll (use only for extracting mscorlib itself)");
output.WriteLine(" --threads:nnn Specify number of threads (default=CPU cores)");
output.WriteLine(" --verbose Produce more output");
output.WriteLine(" --pdb Cross-reference information from PDBs where available");
}
private Options()
{
}
public static Options Create(string[] args)
{
var options = new Options();
options.ParseArguments(args);
return options;
}
}
}

158
csharp/extractor/Semmle.Extraction.CSharp.Standalone/Program.cs Normal file
View File

@@ -0,0 +1,158 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Semmle.BuildAnalyser;
using Semmle.Util.Logging;
namespace Semmle.Extraction.CSharp.Standalone
{
/// <summary>
/// One independent run of the extractor.
/// </summary>
class Extraction
{
public Extraction(string directory)
{
this.directory = directory;
}
public readonly string directory;
public readonly List<string> Sources = new List<string>();
};
/// <summary>
/// Searches for source/references and creates separate extractions.
/// </summary>
class Analysis
{
readonly ILogger logger;
public Analysis(ILogger logger)
{
this.logger = logger;
}
// The extraction configuration for the entire project.
Extraction projectExtraction;
public IEnumerable<string> References
{
get; private set;
}
/// <summary>
/// The extraction configuration.
/// </summary>
public Extraction Extraction => projectExtraction;
/// <summary>
/// Creates an extraction for the current directory
/// and adds it to the list of all extractions.
/// </summary>
/// <param name="dir">The directory of the extraction.</param>
/// <returns>The extraction.</returns>
void CreateExtraction(string dir)
{
projectExtraction = new Extraction(dir);
}
BuildAnalysis buildAnalysis;
/// <summary>
/// Analyse projects/solution and resolves references.
/// </summary>
/// <param name="options">The build analysis options.</param>
public void AnalyseProjects(Options options)
{
CreateExtraction(options.SrcDir);
var progressMonitor = new ProgressMonitor(logger);
buildAnalysis = new BuildAnalysis(options, progressMonitor);
References = buildAnalysis.ReferenceFiles;
projectExtraction.Sources.AddRange(options.SolutionFile == null ? buildAnalysis.AllSourceFiles : buildAnalysis.ProjectSourceFiles);
}
/// <summary>
/// Delete any Nuget assemblies.
/// </summary>
public void Cleanup()
{
buildAnalysis.Cleanup();
}
};
public class Program
{
static int Main(string[] args)
{
var options = Options.Create(args);
var output = new ConsoleLogger(options.Verbosity);
var a = new Analysis(output);
if (options.Help)
{
options.ShowHelp(System.Console.Out);
return 0;
}
if (options.Errors)
return 1;
output.Log(Severity.Info, "Running C# standalone extractor");
a.AnalyseProjects(options);
int sourceFiles = a.Extraction.Sources.Count();
if (sourceFiles == 0)
{
output.Log(Severity.Error, "No source files found");
return 1;
}
if (!options.SkipExtraction)
{
output.Log(Severity.Info, "");
output.Log(Severity.Info, "Extracting...");
Extractor.ExtractStandalone(
a.Extraction.Sources,
a.References,
new ExtractionProgress(output),
new FileLogger(options.Verbosity, Extractor.GetCSharpLogPath()),
options);
output.Log(Severity.Info, "Extraction complete");
}
a.Cleanup();
return 0;
}
class ExtractionProgress : IProgressMonitor
{
public ExtractionProgress(ILogger output)
{
logger = output;
}
readonly ILogger logger;
public void Analysed(int item, int total, string source, string output, TimeSpan time, AnalysisAction action)
{
logger.Log(Severity.Info, "[{0}/{1}] {2} ({3})", item, total, source,
action == AnalysisAction.Extracted ? time.ToString() : action == AnalysisAction.Excluded ? "excluded" : "up to date");
}
public void MissingType(string type)
{
logger.Log(Severity.Debug, "Missing type {0}", type);
}
public void MissingNamespace(string @namespace)
{
logger.Log(Severity.Info, "Missing namespace {0}", @namespace);
}
public void MissingSummary(int missingTypes, int missingNamespaces)
{
logger.Log(Severity.Info, "Failed to resolve {0} types and {1} namespaces", missingTypes, missingNamespaces);
}
}
}
}

105
csharp/extractor/Semmle.Extraction.CSharp.Standalone/ProgressMonitor.cs Normal file
View File

@@ -0,0 +1,105 @@
using Semmle.Util.Logging;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// Callback for various events that may happen during the build analysis.
/// </summary>
interface IProgressMonitor
{
void FindingFiles(string dir);
void UnresolvedReference(string id, string project);
void AnalysingProjectFiles(int count);
void FailedProjectFile(string filename, string reason);
void FailedNugetCommand(string exe, string args, string message);
void NugetInstall(string package);
void ResolvedReference(string filename);
void Summary(int existingSources, int usedSources, int missingSources, int references, int unresolvedReferences, int resolvedConflicts, int totalProjects, int failedProjects);
void Warning(string message);
void ResolvedConflict(string asm1, string asm2);
void MissingProject(string projectFile);
}
class ProgressMonitor : IProgressMonitor
{
readonly ILogger logger;
public ProgressMonitor(ILogger logger)
{
this.logger = logger;
}
public void FindingFiles(string dir)
{
logger.Log(Severity.Info, "Finding files in {0}...", dir);
}
public void IndexingReferences(int count)
{
logger.Log(Severity.Info, "Indexing...");
logger.Log(Severity.Debug, "Indexing {0} DLLs...", count);
}
public void UnresolvedReference(string id, string project)
{
logger.Log(Severity.Info, "Unresolved reference {0}", id);
logger.Log(Severity.Debug, "Unresolved {0} referenced by {1}", id, project);
}
public void AnalysingProjectFiles(int count)
{
logger.Log(Severity.Info, "Analyzing project files...");
}
public void FailedProjectFile(string filename, string reason)
{
logger.Log(Severity.Info, "Couldn't read project file {0}: {1}", filename, reason);
}
public void FailedNugetCommand(string exe, string args, string message)
{
logger.Log(Severity.Info, "Command failed: {0} {1}", exe, args);
logger.Log(Severity.Info, " {0}", message);
}
public void NugetInstall(string package)
{
logger.Log(Severity.Info, "Restoring {0}...", package);
}
public void ResolvedReference(string filename)
{
logger.Log(Severity.Info, "Resolved {0}", filename);
}
public void Summary(int existingSources, int usedSources, int missingSources,
int references, int unresolvedReferences, int resolvedConflicts, int totalProjects, int failedProjects)
{
logger.Log(Severity.Info, "");
logger.Log(Severity.Info, "Build analysis summary:");
logger.Log(Severity.Info, "{0, 6} source files in the filesystem", existingSources);
logger.Log(Severity.Info, "{0, 6} source files in project files", usedSources);
logger.Log(Severity.Info, "{0, 6} sources missing from project files", missingSources);
logger.Log(Severity.Info, "{0, 6} resolved references", references);
logger.Log(Severity.Info, "{0, 6} unresolved references", unresolvedReferences);
logger.Log(Severity.Info, "{0, 6} resolved assembly conflicts", resolvedConflicts);
logger.Log(Severity.Info, "{0, 6} projects", totalProjects);
logger.Log(Severity.Info, "{0, 6} missing/failed projects", failedProjects);
}
public void Warning(string message)
{
logger.Log(Severity.Warning, message);
}
public void ResolvedConflict(string asm1, string asm2)
{
logger.Log(Severity.Info, "Resolved {0} as {1}", asm1, asm2);
}
public void MissingProject(string projectFile)
{
logger.Log(Severity.Info, "Solution is missing {0}", projectFile);
}
}
}

35
csharp/extractor/Semmle.Extraction.CSharp.Standalone/Properties/AssemblyInfo.cs Normal file
View File

@@ -0,0 +1,35 @@
using System.Reflection;
using System.Runtime.InteropServices;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Semmle.Extraction.CSharp.Standalone")]
[assembly: AssemblyDescription("Standalone extractor for C#")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("Semmle Ltd.")]
[assembly: AssemblyProduct("Semmle.Extraction.CSharp.Standalone")]
[assembly: AssemblyCopyright("Copyright © Semmle 2016")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("bb71e9da-7e0a-43e8-989c-c8e87c828e7c")]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

74
csharp/extractor/Semmle.Extraction.CSharp.Standalone/Runtime.cs Normal file
View File

@@ -0,0 +1,74 @@
using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using System.IO;
using System.Linq;
namespace Semmle.Extraction.CSharp.Standalone
{
/// <summary>
/// Locates .NET Runtimes.
/// </summary>
static class Runtime
{
static string ExecutingRuntime => RuntimeEnvironment.GetRuntimeDirectory();
/// <summary>
/// Locates .NET Core Runtimes.
/// </summary>
public static IEnumerable<string> CoreRuntimes
{
get
{
string[] dotnetDirs = { "/usr/share/dotnet", @"C:\Program Files\dotnet" };
foreach (var dir in dotnetDirs.Where(Directory.Exists))
return Directory.EnumerateDirectories(Path.Combine(dir, "shared", "Microsoft.NETCore.App")).
OrderByDescending(d => Path.GetFileName(d));
return Enumerable.Empty<string>();
}
}
/// <summary>
/// Locates .NET Desktop Runtimes.
/// This includes Mono and Microsoft.NET.
/// </summary>
public static IEnumerable<string> DesktopRuntimes
{
get
{
string[] monoDirs = { "/usr/lib/mono", @"C:\Program Files\Mono\lib\mono" };
if (Directory.Exists(@"C:\Windows\Microsoft.NET\Framework64"))
{
return System.IO.Directory.EnumerateDirectories(@"C:\Windows\Microsoft.NET\Framework64", "v*").
OrderByDescending(d => Path.GetFileName(d));
}
foreach (var dir in monoDirs.Where(Directory.Exists))
{
return System.IO.Directory.EnumerateDirectories(dir).
Where(d => Char.IsDigit(Path.GetFileName(d)[0])).
OrderByDescending(d => Path.GetFileName(d));
}
return Enumerable.Empty<string>();
}
}
public static IEnumerable<string> Runtimes
{
get
{
foreach (var r in CoreRuntimes)
yield return r;
foreach (var r in DesktopRuntimes)
yield return r;
// A bad choice if it's the self-contained runtime distributed in odasa dist.
yield return ExecutingRuntime;
}
}
}
}

30
csharp/extractor/Semmle.Extraction.CSharp.Standalone/Semmle.Extraction.CSharp.Standalone.csproj Normal file
View File

@@ -0,0 +1,30 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.0</TargetFramework>
<AssemblyName>Semmle.Extraction.CSharp.Standalone</AssemblyName>
<RootNamespace>Semmle.Extraction.CSharp.Standalone</RootNamespace>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<TreatWarningsAsErrors>false</TreatWarningsAsErrors>
<WarningsAsErrors />
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\Semmle.Extraction.CSharp\Semmle.Extraction.CSharp.csproj" />
</ItemGroup>
<ItemGroup>
<Folder Include="Properties\" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Build" Version="15.8.166" />
<PackageReference Include="Microsoft.Win32.Primitives" Version="4.3.0" />
<PackageReference Include="System.Net.Primitives" Version="4.3.0" />
<PackageReference Include="System.Security.Principal" Version="4.3.0" />
<PackageReference Include="System.Threading.ThreadPool" Version="4.3.0" />
</ItemGroup>
</Project>

68
csharp/extractor/Semmle.Extraction.CSharp.Standalone/SolutionFile.cs Normal file
View File

@@ -0,0 +1,68 @@
using Microsoft.Build.Construction;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace Semmle.BuildAnalyser
{
/// <summary>
/// Access data in a .sln file.
/// </summary>
class SolutionFile
{
readonly Microsoft.Build.Construction.SolutionFile solutionFile;
/// <summary>
/// Read the file.
/// </summary>
/// <param name="filename">The filename of the .sln.</param>
public SolutionFile(string filename)
{
// SolutionFile.Parse() expects a rooted path.
var fullPath = Path.GetFullPath(filename);
solutionFile = Microsoft.Build.Construction.SolutionFile.Parse(fullPath);
}
/// <summary>
/// Projects directly included in the .sln file.
/// </summary>
public IEnumerable<string> MsBuildProjects
{
get
{
return solutionFile.ProjectsInOrder.
Where(p => p.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat).
Select(p => p.AbsolutePath).
Select(p => Path.DirectorySeparatorChar == '/' ? p.Replace("\\", "/") : p);
}
}
/// <summary>
/// Projects included transitively via a subdirectory.
/// </summary>
public IEnumerable<string> NestedProjects
{
get
{
return solutionFile.ProjectsInOrder.
Where(p => p.ProjectType == SolutionProjectType.SolutionFolder).
Where(p => Directory.Exists(p.AbsolutePath)).
SelectMany(p => new DirectoryInfo(p.AbsolutePath).EnumerateFiles("*.csproj", SearchOption.AllDirectories)).
Select(f => f.FullName);
}
}
/// <summary>
/// List of projects which were mentioned but don't exist on disk.
/// </summary>
public IEnumerable<string> MissingProjects =>
// Only projects in the solution file can be missing.
// (NestedProjects are located on disk so always exist.)
MsBuildProjects.Where(p => !File.Exists(p));
/// <summary>
/// The list of project files.
/// </summary>
public IEnumerable<string> Projects => MsBuildProjects.Concat(NestedProjects);
}
}

505
csharp/extractor/Semmle.Extraction.CSharp/Analyser.cs Normal file
View File

@@ -0,0 +1,505 @@
using System;
using System.Collections.Immutable;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Diagnostics;
using System.IO;
using System.Linq;
using Semmle.Extraction.CSharp.Populators;
using System.Runtime.InteropServices;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;
using Semmle.Util.Logging;
using Semmle.Util;
namespace Semmle.Extraction.CSharp
{
/// <summary>
/// Encapsulates a C# analysis task.
/// </summary>
public class Analyser : IDisposable
{
IExtractor extractor;
readonly Stopwatch stopWatch = new Stopwatch();
readonly IProgressMonitor progressMonitor;
public readonly ILogger Logger;
public Analyser(IProgressMonitor pm, ILogger logger)
{
Logger = logger;
Logger.Log(Severity.Info, "EXTRACTION STARTING at {0}", DateTime.Now);
stopWatch.Start();
progressMonitor = pm;
}
CSharpCompilation compilation;
Layout layout;
/// <summary>
/// Initialize the analyser.
/// </summary>
/// <param name="commandLineArguments">Arguments passed to csc.</param>
/// <param name="compilationIn">The Roslyn compilation.</param>
/// <param name="options">Extractor options.</param>
public void Initialize(
CSharpCommandLineArguments commandLineArguments,
CSharpCompilation compilationIn,
Options options)
{
compilation = compilationIn;
layout = new Layout();
this.options = options;
extractor = new Extraction.Extractor(false, GetOutputName(compilation, commandLineArguments), Logger);
LogDiagnostics();
SetReferencePaths();
CompilationErrors += FilteredDiagnostics.Count();
}
/// <summary>
/// Constructs the map from assembly string to its filename.
///
/// Roslyn doesn't record the relationship between a filename and its assembly
/// information, so we need to retrieve this information manually.
/// </summary>
void SetReferencePaths()
{
foreach (var reference in compilation.References.OfType<PortableExecutableReference>())
{
try
{
var refPath = reference.FilePath;
/* This method is significantly faster and more lightweight than using
* System.Reflection.Assembly.ReflectionOnlyLoadFrom. It is also allows
* loading the same assembly from different locations.
*/
using (var pereader = new System.Reflection.PortableExecutable.PEReader(new FileStream(refPath, FileMode.Open, FileAccess.Read, FileShare.Read)))
{
var metadata = pereader.GetMetadata();
string assemblyIdentity;
unsafe
{
var reader = new System.Reflection.Metadata.MetadataReader(metadata.Pointer, metadata.Length);
var def = reader.GetAssemblyDefinition();
assemblyIdentity = reader.GetString(def.Name) + " " + def.Version;
}
extractor.SetAssemblyFile(assemblyIdentity, refPath);
}
}
catch (Exception ex)
{
extractor.Message(new Message
{
exception = ex,
message = string.Format("Exception reading reference file {0}: {1}",
reference.FilePath, ex)
});
}
}
}
public void InitializeStandalone(CSharpCompilation compilationIn, CommonOptions options)
{
compilation = compilationIn;
layout = new Layout();
extractor = new Extraction.Extractor(true, null, Logger);
this.options = options;
LogDiagnostics();
SetReferencePaths();
}
readonly HashSet<string> errorsToIgnore = new HashSet<string>
{
"CS7027", // Code signing failure
"CS1589", // XML referencing not supported
"CS1569" // Error writing XML documentation
};
IEnumerable<Diagnostic> FilteredDiagnostics
{
get
{
return extractor == null || extractor.Standalone || compilation == null ? Enumerable.Empty<Diagnostic>() :
compilation.
GetDiagnostics().
Where(e => e.Severity >= DiagnosticSeverity.Error && !errorsToIgnore.Contains(e.Id));
}
}
public IEnumerable<string> MissingTypes => extractor.MissingTypes;
public IEnumerable<string> MissingNamespaces => extractor.MissingNamespaces;
/// <summary>
/// Determine the path of the output dll/exe.
/// </summary>
/// <param name="compilation">Information about the compilation.</param>
/// <param name="cancel">Cancellation token required.</param>
/// <returns>The filename.</returns>
static string GetOutputName(CSharpCompilation compilation,
CSharpCommandLineArguments commandLineArguments)
{
// There's no apparent way to access the output filename from the compilation,
// so we need to re-parse the command line arguments.
if (commandLineArguments.OutputFileName == null)
{
// No output specified: Use name based on first filename
var entry = compilation.GetEntryPoint(System.Threading.CancellationToken.None);
if (entry == null)
{
if (compilation.SyntaxTrees.Length == 0)
throw new ArgumentNullException("No source files seen");
// Probably invalid, but have a go anyway.
var entryPointFile = compilation.SyntaxTrees.First().FilePath;
return Path.ChangeExtension(entryPointFile, ".exe");
}
else
{
var entryPointFilename = entry.Locations.First().SourceTree.FilePath;
return Path.ChangeExtension(entryPointFilename, ".exe");
}
}
else
{
return Path.Combine(commandLineArguments.OutputDirectory, commandLineArguments.OutputFileName);
}
}
/// <summary>
/// Perform an analysis on a source file/syntax tree.
/// </summary>
/// <param name="tree">Syntax tree to analyse.</param>
public void AnalyseTree(SyntaxTree tree)
{
extractionTasks.Add(() => DoExtractTree(tree));
}
/// <summary>
/// Perform an analysis on an assembly.
/// </summary>
/// <param name="assembly">Assembly to analyse.</param>
void AnalyseAssembly(PortableExecutableReference assembly)
{
// CIL first - it takes longer.
if (options.CIL)
extractionTasks.Add(() => DoExtractCIL(assembly));
extractionTasks.Add(() => DoAnalyseAssembly(assembly));
}
readonly object progressMutex = new object();
int taskCount = 0;
CommonOptions options;
static bool FileIsUpToDate(string src, string dest)
{
return File.Exists(dest) &&
File.GetLastWriteTime(dest) >= File.GetLastWriteTime(src);
}
bool FileIsCached(string src, string dest)
{
return options.Cache && FileIsUpToDate(src, dest);
}
/// <summary>
/// Extract an assembly to a new trap file.
/// If the trap file exists, skip extraction to avoid duplicating
/// extraction within the snapshot.
/// </summary>
/// <param name="r">The assembly to extract.</param>
void DoAnalyseAssembly(PortableExecutableReference r)
{
try
{
var stopwatch = new Stopwatch();
stopwatch.Start();
var assemblyPath = r.FilePath;
var projectLayout = layout.LookupProjectOrDefault(assemblyPath);
using (var trapWriter = projectLayout.CreateTrapWriter(Logger, assemblyPath, true))
{
var skipExtraction = FileIsCached(assemblyPath, trapWriter.TrapFile);
if (!skipExtraction)
{
/* Note on parallel builds:
*
* The trap writer and source archiver both perform atomic moves
* of the file to the final destination.
*
* If the same source file or trap file are generated concurrently
* (by different parallel invocations of the extractor), then
* last one wins.
*
* Specifically, if two assemblies are analysed concurrently in a build,
* then there is a small amount of duplicated work but the output should
* still be correct.
*/
// compilation.Clone() reduces memory footprint by allowing the symbols
// in c to be garbage collected.
Compilation c = compilation.Clone();
var assembly = c.GetAssemblyOrModuleSymbol(r) as IAssemblySymbol;
if (assembly != null)
{
var cx = new Context(extractor, c, trapWriter, new AssemblyScope(assembly, assemblyPath));
foreach (var module in assembly.Modules)
{
AnalyseNamespace(cx, module.GlobalNamespace);
}
cx.PopulateAll();
}
}
ReportProgress(assemblyPath, trapWriter.TrapFile, stopwatch.Elapsed, skipExtraction ? AnalysisAction.UpToDate : AnalysisAction.Extracted);
}
}
catch (Exception ex)
{
Logger.Log(Severity.Error, " Unhandled exception analyzing {0}: {1}", r.FilePath, ex);
}
}
void DoExtractCIL(PortableExecutableReference r)
{
var stopwatch = new Stopwatch();
stopwatch.Start();
string trapFile;
bool extracted;
CIL.Entities.Assembly.ExtractCIL(layout, r.FilePath, Logger, !options.Cache, options.PDB, out trapFile, out extracted);
stopwatch.Stop();
ReportProgress(r.FilePath, trapFile, stopwatch.Elapsed, extracted ? AnalysisAction.Extracted : AnalysisAction.UpToDate);
}
void AnalyseNamespace(Context cx, INamespaceSymbol ns)
{
foreach (var memberNamespace in ns.GetNamespaceMembers())
{
AnalyseNamespace(cx, memberNamespace);
}
foreach (var memberType in ns.GetTypeMembers())
{
Entities.Type.Create(cx, memberType).ExtractRecursive();
}
}
/// <summary>
/// Enqueue all reference analysis tasks.
/// </summary>
public void AnalyseReferences()
{
foreach (var r in compilation.References.OfType<PortableExecutableReference>())
{
AnalyseAssembly(r);
}
}
// The bulk of the extraction work, potentially executed in parallel.
readonly List<Action> extractionTasks = new List<Action>();
void ReportProgress(string src, string output, TimeSpan time, AnalysisAction action)
{
lock (progressMutex)
progressMonitor.Analysed(++taskCount, extractionTasks.Count, src, output, time, action);
}
void DoExtractTree(SyntaxTree tree)
{
try
{
var stopwatch = new Stopwatch();
stopwatch.Start();
var sourcePath = tree.FilePath;
var projectLayout = layout.LookupProjectOrNull(sourcePath);
bool excluded = projectLayout == null;
string trapPath = excluded ? "" : projectLayout.GetTrapPath(Logger, sourcePath);
bool upToDate = false;
if (!excluded)
{
// compilation.Clone() is used to allow symbols to be garbage collected.
using (var trapWriter = projectLayout.CreateTrapWriter(Logger, sourcePath, false))
{
upToDate = options.Fast && FileIsUpToDate(sourcePath, trapWriter.TrapFile);
if (!upToDate)
{
Context cx = new Context(extractor, compilation.Clone(), trapWriter, new SourceScope(tree));
Populators.CompilationUnit.Extract(cx, tree.GetRoot());
cx.PopulateAll();
cx.ExtractComments(cx.CommentGenerator);
}
}
}
ReportProgress(sourcePath, trapPath, stopwatch.Elapsed, excluded ? AnalysisAction.Excluded : upToDate ? AnalysisAction.UpToDate : AnalysisAction.Extracted);
}
catch (Exception ex)
{
extractor.Message(new Message { exception = ex, message = string.Format("Unhandled exception processing {0}: {1}", tree.FilePath, ex), severity = Severity.Error });
}
}
/// <summary>
/// Run all extraction tasks.
/// </summary>
/// <param name="numberOfThreads">The number of threads to use.</param>
public void PerformExtraction(int numberOfThreads)
{
Parallel.Invoke(
new ParallelOptions { MaxDegreeOfParallelism = numberOfThreads },
extractionTasks.ToArray());
}
public void Dispose()
{
stopWatch.Stop();
Logger.Log(Severity.Info, " Peak working set = {0} MB", Process.GetCurrentProcess().PeakWorkingSet64 / (1024 * 1024));
if (TotalErrors > 0)
Logger.Log(Severity.Info, "EXTRACTION FAILED with {0} error{1} in {2}", TotalErrors, TotalErrors == 1 ? "" : "s", stopWatch.Elapsed);
else
Logger.Log(Severity.Info, "EXTRACTION SUCCEEDED in {0}", stopWatch.Elapsed);
Logger.Dispose();
}
/// <summary>
/// Number of errors encountered during extraction.
/// </summary>
public int ExtractorErrors => extractor == null ? 0 : extractor.Errors;
/// <summary>
/// Number of errors encountered by the compiler.
/// </summary>
public int CompilationErrors { get; set; }
/// <summary>
/// Total number of errors reported.
/// </summary>
public int TotalErrors => CompilationErrors + ExtractorErrors;
void AppendQuoted(StringBuilder sb, string s)
{
if (s.IndexOf(' ') != -1)
sb.Append('\"').Append(s).Append('\"');
else
sb.Append(s);
}
/// <summary>
/// Logs detailed information about this invocation,
/// in the event that errors were detected.
/// </summary>
public void LogDiagnostics()
{
Logger.Log(Severity.Info, " Current working directory: {0}", Directory.GetCurrentDirectory());
Logger.Log(Severity.Info, " Extractor: {0}", Environment.GetCommandLineArgs().First());
if (extractor != null)
Logger.Log(Severity.Info, " Extractor version: {0}", extractor.Version);
var sb = new StringBuilder();
sb.Append(" Expanded command line: ");
bool first = true;
foreach (var arg in Environment.GetCommandLineArgs().Skip(1))
{
if (arg[0] == '@')
{
foreach (var line in File.ReadAllLines(arg.Substring(1)))
{
if (first) first = false;
else sb.Append(" ");
sb.Append(line);
}
}
else
{
if (first) first = false;
else sb.Append(" ");
AppendQuoted(sb, arg);
}
}
Logger.Log(Severity.Info, sb.ToString());
foreach (var error in FilteredDiagnostics)
{
Logger.Log(Severity.Error, " Compilation error: {0}", error);
}
if (FilteredDiagnostics.Any())
{
foreach (var reference in compilation.References)
{
Logger.Log(Severity.Info, " Resolved reference {0}", reference.Display);
}
}
}
}
/// <summary>
/// What action was performed when extracting a file.
/// </summary>
public enum AnalysisAction
{
Extracted,
UpToDate,
Excluded
}
/// <summary>
/// Callback for various extraction events.
/// (Used for display of progress).
/// </summary>
public interface IProgressMonitor
{
/// <summary>
/// Callback that a particular item has been analysed.
/// </summary>
/// <param name="item">The item number being processed.</param>
/// <param name="total">The total number of items to process.</param>
/// <param name="source">The name of the item, e.g. a source file.</param>
/// <param name="output">The name of the item being output, e.g. a trap file.</param>
/// <param name="time">The time to extract the item.</param>
/// <param name="action">What action was taken for the file.</param>
void Analysed(int item, int total, string source, string output, TimeSpan time, AnalysisAction action);
/// <summary>
/// A "using namespace" directive was seen but the given
/// namespace could not be found.
/// Only called once for each @namespace.
/// </summary>
/// <param name="namespace"></param>
void MissingNamespace(string @namespace);
/// <summary>
/// An ErrorType was found.
/// Called once for each type name.
/// </summary>
/// <param name="type">The full/partial name of the type.</param>
void MissingType(string type);
/// <summary>
/// Report a summary of missing entities.
/// </summary>
/// <param name="types">The number of missing types.</param>
/// <param name="namespaces">The number of missing using namespace declarations.</param>
void MissingSummary(int types, int namespaces);
}
}

107
csharp/extractor/Semmle.Extraction.CSharp/CompilerVersion.cs Normal file
View File

@@ -0,0 +1,107 @@
using System.Diagnostics;
using System.IO;
using System.Runtime.InteropServices;
namespace Semmle.Extraction.CSharp
{
/// <summary>
/// Identifies the compiler and framework from the command line arguments.
/// --compiler specifies the compiler
/// --framework specifies the .net framework
/// </summary>
public class CompilerVersion
{
const string csc_rsp = "csc.rsp";
readonly string specifiedFramework = null;
/// <summary>
/// The value specified by --compiler, or null.
/// </summary>
public string SpecifiedCompiler
{
get;
private set;
}
/// <summary>
/// Why was the candidate exe rejected as a compiler?
/// </summary>
public string SkipReason
{
get;
private set;
}
/// <summary>
/// Probes the compiler (if specified).
/// </summary>
/// <param name="options">The command line arguments.</param>
public CompilerVersion(Options options)
{
SpecifiedCompiler = options.CompilerName;
specifiedFramework = options.Framework;
if (SpecifiedCompiler != null)
{
if (!File.Exists(SpecifiedCompiler))
{
SkipExtractionBecause("the specified file does not exist");
return;
}
// Reads the file details from the .exe
var versionInfo = FileVersionInfo.GetVersionInfo(SpecifiedCompiler);
var compilerDir = Path.GetDirectoryName(SpecifiedCompiler);
bool known_compiler_name = versionInfo.OriginalFilename == "csc.exe" || versionInfo.OriginalFilename == "csc2.exe";
bool copyright_microsoft = versionInfo.LegalCopyright != null && versionInfo.LegalCopyright.Contains("Microsoft");
bool mscorlib_exists = File.Exists(Path.Combine(compilerDir, "mscorlib.dll"));
if (specifiedFramework == null && mscorlib_exists)
{
specifiedFramework = compilerDir;
}
if (!known_compiler_name)
{
SkipExtractionBecause("the exe name is not recognised");
}
else if (!copyright_microsoft)
{
SkipExtractionBecause("the exe isn't copyright Microsoft");
}
}
}
void SkipExtractionBecause(string reason)
{
SkipExtraction = true;
SkipReason = reason;
}
/// <summary>
/// The directory containing the .Net Framework.
/// </summary>
public string FrameworkPath => specifiedFramework ?? RuntimeEnvironment.GetRuntimeDirectory();
/// <summary>
/// The file csc.rsp.
/// </summary>
public string CscRsp => Path.Combine(FrameworkPath, csc_rsp);
/// <summary>
/// Should we skip extraction?
/// Only if csc.exe was specified but it wasn't a compiler.
/// </summary>
public bool SkipExtraction
{
get;
private set;
}
/// <summary>
/// Gets additional reference directories - the compiler directory.
/// </summary>
public string AdditionalReferenceDirectories => SpecifiedCompiler != null ? Path.GetDirectoryName(SpecifiedCompiler) : null;
}
}

88
csharp/extractor/Semmle.Extraction.CSharp/Entities/Accessor.cs Normal file
View File

@@ -0,0 +1,88 @@
using Microsoft.CodeAnalysis;
using System.Linq;
namespace Semmle.Extraction.CSharp.Entities
{
class Accessor : Method
{
protected Accessor(Context cx, IMethodSymbol init)
: base(cx, init) { }
/// <summary>
/// Gets the property symbol associated with this accessor.
/// </summary>
IPropertySymbol PropertySymbol
{
get
{
// Usually, the property/indexer can be fetched from the associated symbol
var prop = symbol.AssociatedSymbol as IPropertySymbol;
if (prop != null)
return prop;
// But for properties/indexers that implement explicit interfaces, Roslyn
// does not properly populate `AssociatedSymbol`
var props = symbol.ContainingType.GetMembers().OfType<IPropertySymbol>();
props = props.Where(p => symbol.Equals(p.GetMethod) || symbol.Equals(p.SetMethod));
return props.SingleOrDefault();
}
}
public new Accessor OriginalDefinition => Create(Context, symbol.OriginalDefinition);
public override void Populate()
{
PopulateMethod();
ExtractModifiers();
ContainingType.ExtractGenerics();
var prop = PropertySymbol;
if (prop == null)
{
Context.ModelError(symbol, "Unhandled accessor associated symbol");
return;
}
var parent = Property.Create(Context, prop);
int kind;
Accessor unboundAccessor;
if (symbol.Equals(prop.GetMethod))
{
kind = 1;
unboundAccessor = Create(Context, prop.OriginalDefinition.GetMethod);
}
else if (symbol.Equals(prop.SetMethod))
{
kind = 2;
unboundAccessor = Create(Context, prop.OriginalDefinition.SetMethod);
}
else
{
Context.ModelError(symbol, "Undhandled accessor kind");
return;
}
Context.Emit(Tuples.accessors(this, kind, symbol.Name, parent, unboundAccessor));
foreach (var l in Locations)
Context.Emit(Tuples.accessor_location(this, l));
Overrides();
if (symbol.FromSource() && Block == null)
{
Context.Emit(Tuples.compiler_generated(this));
}
}
public new static Accessor Create(Context cx, IMethodSymbol symbol) =>
AccessorFactory.Instance.CreateEntity(cx, symbol);
class AccessorFactory : ICachedEntityFactory<IMethodSymbol, Accessor>
{
public static readonly AccessorFactory Instance = new AccessorFactory();
public Accessor Create(Context cx, IMethodSymbol init) => new Accessor(cx, init);
}
}
}

79
csharp/extractor/Semmle.Extraction.CSharp/Entities/Attribute.cs Normal file
View File

@@ -0,0 +1,79 @@
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.CSharp.Populators;
using Semmle.Extraction.Entities;
using System.Collections.Generic;
using System.Linq;
namespace Semmle.Extraction.CSharp.Entities
{
class Attribute : FreshEntity, IExpressionParentEntity
{
bool IExpressionParentEntity.IsTopLevelParent => true;
public Attribute(Context cx, AttributeData attribute, IEntity entity)
: base(cx)
{
if (attribute.ApplicationSyntaxReference != null)
{
// !! Extract attributes from assemblies.
// This is harder because the "expression" entities presume the
// existence of a syntax tree. This is not the case for compiled
// attributes.
var syntax = attribute.ApplicationSyntaxReference.GetSyntax() as AttributeSyntax;
ExtractAttribute(cx, syntax, attribute.AttributeClass, entity);
}
}
public Attribute(Context cx, AttributeSyntax attribute, IEntity entity)
: base(cx)
{
var info = cx.GetSymbolInfo(attribute);
ExtractAttribute(cx, attribute, info.Symbol.ContainingType, entity);
}
void ExtractAttribute(Context cx, AttributeSyntax syntax, ITypeSymbol attributeClass, IEntity entity)
{
var type = Type.Create(cx, attributeClass);
cx.Emit(Tuples.attributes(this, type.TypeRef, entity));
cx.Emit(Tuples.attribute_location(this, cx.Create(syntax.Name.GetLocation())));
if (cx.Extractor.OutputPath != null)
cx.Emit(Tuples.attribute_location(this, Assembly.CreateOutputAssembly(cx)));
TypeMention.Create(cx, syntax.Name, this, type);
if (syntax.ArgumentList != null)
{
cx.PopulateLater(() =>
{
int child = 0;
foreach (var arg in syntax.ArgumentList.Arguments)
{
Expression.Create(cx, arg.Expression, this, child++);
}
// !! Handle named arguments
});
}
}
public static void ExtractAttributes(Context cx, ISymbol symbol, IEntity entity)
{
foreach (var attribute in symbol.GetAttributes())
{
new Attribute(cx, attribute, entity);
}
}
public static void ExtractAttributes(Context cx, IEnumerable<AttributeListSyntax> attributes, IEntity entity)
{
foreach (var attributeSyntax in attributes.SelectMany(l => l.Attributes))
{
new Attribute(cx, attributeSyntax, entity);
}
}
public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.OptionalLabel;
}
}

51
csharp/extractor/Semmle.Extraction.CSharp/Entities/CommentBlock.cs Normal file
View File

@@ -0,0 +1,51 @@
using Semmle.Extraction.CommentProcessing;
using Semmle.Extraction.Entities;
namespace Semmle.Extraction.CSharp.Entities
{
class CommentBlock : CachedEntity<ICommentBlock>
{
CommentBlock(Context cx, ICommentBlock init)
: base(cx, init) { }
public override void Populate()
{
Context.Emit(Tuples.commentblock(this));
int child = 0;
Context.Emit(Tuples.commentblock_location(this, Context.Create(symbol.Location)));
foreach (var l in symbol.CommentLines)
{
Context.Emit(Tuples.commentblock_child(this, (CommentLine)l, child++));
}
}
public override bool NeedsPopulation => true;
public override IId Id
{
get
{
var loc = Context.Create(symbol.Location);
return new Key(loc, ";commentblock");
}
}
public override Microsoft.CodeAnalysis.Location ReportingLocation => symbol.Location;
public void BindTo(Label entity, Binding binding)
{
Context.Emit(Tuples.commentblock_binding(this, entity, binding));
}
public static CommentBlock Create(Context cx, ICommentBlock block) => CommentBlockFactory.Instance.CreateEntity(cx, block);
class CommentBlockFactory : ICachedEntityFactory<ICommentBlock, CommentBlock>
{
public static readonly CommentBlockFactory Instance = new CommentBlockFactory();
public CommentBlock Create(Context cx, ICommentBlock init) => new CommentBlock(cx, init);
}
public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.NoLabel;
}
}

147
csharp/extractor/Semmle.Extraction.CSharp/Entities/CommentLine.cs Normal file
View File

@@ -0,0 +1,147 @@
using Semmle.Extraction.CommentProcessing;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Semmle.Extraction.Entities;
using System;
namespace Semmle.Extraction.CSharp.Entities
{
class CommentLine : CachedEntity<(Microsoft.CodeAnalysis.Location, string)>, ICommentLine
{
CommentLine(Context cx, Microsoft.CodeAnalysis.Location loc, CommentType type, string text, string raw)
: base(cx, (loc, text))
{
Type = type;
RawText = raw;
}
public Microsoft.CodeAnalysis.Location Location => symbol.Item1;
public CommentType Type { get; private set; }
public string Text { get { return symbol.Item2; } }
public string RawText { get; private set; }
public static void Extract(Context cx, SyntaxTrivia trivia)
{
switch (trivia.Kind())
{
case SyntaxKind.SingleLineDocumentationCommentTrivia:
/*
This is actually a multi-line comment consisting of /// lines.
So split it up.
*/
var text = trivia.ToFullString();
var split = text.Split('\n');
var currentLocation = trivia.GetLocation().SourceSpan.Start - 3;
for (int line = 0; line < split.Length - 1; ++line)
{
string fullLine = split[line];
var nextLineLocation = currentLocation + fullLine.Length + 1;
fullLine = fullLine.TrimEnd('\r');
string trimmedLine = fullLine;
int leadingSpaces = trimmedLine.IndexOf('/');
if (leadingSpaces != -1)
{
fullLine = fullLine.Substring(leadingSpaces);
currentLocation += leadingSpaces;
trimmedLine = trimmedLine.Substring(leadingSpaces + 3); // Remove leading spaces and the "///"
trimmedLine = trimmedLine.Trim();
var span = Microsoft.CodeAnalysis.Text.TextSpan.FromBounds(currentLocation, currentLocation + fullLine.Length);
var location = Microsoft.CodeAnalysis.Location.Create(trivia.SyntaxTree, span);
var commentType = CommentType.XmlDoc;
cx.CommentGenerator.AddComment(Create(cx, location, commentType, trimmedLine, fullLine));
}
else
{
cx.ModelError("Unexpected comment format");
}
currentLocation = nextLineLocation;
}
break;
case SyntaxKind.SingleLineCommentTrivia:
{
string contents = trivia.ToString().Substring(2);
var commentType = CommentType.Singleline;
if (contents.Length > 0 && contents[0] == '/')
{
commentType = CommentType.XmlDoc;
contents = contents.Substring(1); // An XML comment.
}
cx.CommentGenerator.AddComment(Create(cx, trivia.GetLocation(), commentType, contents.Trim(), trivia.ToFullString()));
}
break;
case SyntaxKind.MultiLineDocumentationCommentTrivia:
case SyntaxKind.MultiLineCommentTrivia:
/* We receive a single SyntaxTrivia for a multiline block spanning several lines.
So we split it into separate lines
*/
text = trivia.ToFullString();
split = text.Split('\n');
currentLocation = trivia.GetLocation().SourceSpan.Start;
for (int line = 0; line < split.Length; ++line)
{
string fullLine = split[line];
var nextLineLocation = currentLocation + fullLine.Length + 1;
fullLine = fullLine.TrimEnd('\r');
string trimmedLine = fullLine;
if (line == 0) trimmedLine = trimmedLine.Substring(2);
if (line == split.Length - 1) trimmedLine = trimmedLine.Substring(0, trimmedLine.Length - 2);
trimmedLine = trimmedLine.Trim();
var span = Microsoft.CodeAnalysis.Text.TextSpan.FromBounds(currentLocation, currentLocation + fullLine.Length);
var location = Microsoft.CodeAnalysis.Location.Create(trivia.SyntaxTree, span);
var commentType = line == 0 ? CommentType.Multiline : CommentType.MultilineContinuation;
cx.CommentGenerator.AddComment(Create(cx, location, commentType, trimmedLine, fullLine));
currentLocation = nextLineLocation;
}
break;
// Strangely, these are reported as SingleLineCommentTrivia.
case SyntaxKind.DocumentationCommentExteriorTrivia:
cx.ModelError("Unhandled comment type {0} for {1}", trivia.Kind(), trivia);
break;
}
}
Extraction.Entities.Location location;
public override void Populate()
{
location = Context.Create(Location);
Context.Emit(Tuples.commentline(this, Type == CommentType.MultilineContinuation ? CommentType.Multiline : Type, Text, RawText));
Context.Emit(Tuples.commentline_location(this, location));
}
public override Microsoft.CodeAnalysis.Location ReportingLocation => location.symbol;
public override bool NeedsPopulation => true;
public override IId Id
{
get
{
var loc = Context.Create(Location);
return new Key(loc, ";commentline");
}
}
static CommentLine Create(Context cx, Microsoft.CodeAnalysis.Location loc, CommentType type, string text, string raw) => CommentLineFactory.Instance.CreateEntity(cx, loc, type, text, raw);
class CommentLineFactory : ICachedEntityFactory<(Microsoft.CodeAnalysis.Location, CommentType, string, string), CommentLine>
{
public static readonly CommentLineFactory Instance = new CommentLineFactory();
public CommentLine Create(Context cx, (Microsoft.CodeAnalysis.Location, CommentType, string, string) init) =>
new CommentLine(cx, init.Item1, init.Item2, init.Item3, init.Item4);
}
public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.OptionalLabel;
}
}

158
csharp/extractor/Semmle.Extraction.CSharp/Entities/Constructor.cs Normal file
View File

@@ -0,0 +1,158 @@
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.CSharp.Populators;
using Semmle.Util;
using System.Linq;
using Microsoft.CodeAnalysis.CSharp;
using Semmle.Extraction.Entities;
namespace Semmle.Extraction.CSharp.Entities
{
public class Constructor : Method
{
Constructor(Context cx, IMethodSymbol init)
: base(cx, init) { }
public override void Populate()
{
PopulateMethod();
ExtractModifiers();
ContainingType.ExtractGenerics();
Context.Emit(Tuples.constructors(this, symbol.ContainingType.Name, ContainingType, (Constructor)OriginalDefinition));
Context.Emit(Tuples.constructor_location(this, Location));
if (symbol.IsImplicitlyDeclared)
{
var lineCounts = new LineCounts() { Total = 2, Code = 1, Comment = 0 };
Context.Emit(Tuples.numlines(this, lineCounts));
}
ExtractCompilerGenerated();
}
protected override void ExtractInitializers()
{
// Do not extract initializers for constructed types.
if (!IsSourceDeclaration) return;
var syntax = Syntax;
var initializer = syntax == null ? null : syntax.Initializer;
if (initializer == null) return;
Type initializerType;
var symbolInfo = Context.GetSymbolInfo(initializer);
switch (initializer.Kind())
{
case SyntaxKind.BaseConstructorInitializer:
initializerType = Type.Create(Context, symbol.ContainingType.BaseType);
break;
case SyntaxKind.ThisConstructorInitializer:
initializerType = ContainingType;
break;
default:
Context.ModelError(initializer, "Unknown initializer");
return;
}
var initInfo = new ExpressionInfo(Context,
initializerType,
Context.Create(initializer.ThisOrBaseKeyword.GetLocation()),
Kinds.ExprKind.CONSTRUCTOR_INIT,
this,
-1,
false,
null);
var init = new Expression(initInfo);
var target = Constructor.Create(Context, (IMethodSymbol)symbolInfo.Symbol);
if (target == null)
{
Context.ModelError(symbol, "Unable to resolve call");
return;
}
Context.Emit(Tuples.expr_call(init, target));
int child = 0;
foreach (var arg in initializer.ArgumentList.Arguments)
{
Expression.Create(Context, arg.Expression, init, child++);
}
}
ConstructorDeclarationSyntax Syntax
{
get
{
return symbol.DeclaringSyntaxReferences.
Select(r => r.GetSyntax()).
OfType<ConstructorDeclarationSyntax>().
FirstOrDefault();
}
}
public new static Constructor Create(Context cx, IMethodSymbol constructor)
{
if (constructor == null) return null;
switch (constructor.MethodKind)
{
case MethodKind.StaticConstructor:
case MethodKind.Constructor:
return ConstructorFactory.Instance.CreateEntity(cx, constructor);
default:
throw new InternalError(constructor, "Attempt to create a Constructor from a symbol that isn't a constructor");
}
}
public override IId Id
{
get
{
return new Key(tb =>
{
if (symbol.IsStatic) tb.Append("static");
tb.Append(ContainingType);
AddParametersToId(Context, tb, symbol);
tb.Append("; constructor");
});
}
}
ConstructorDeclarationSyntax GetSyntax() =>
symbol.DeclaringSyntaxReferences.Select(r => r.GetSyntax()).OfType<ConstructorDeclarationSyntax>().FirstOrDefault();
public override Microsoft.CodeAnalysis.Location FullLocation => ReportingLocation;
public override Microsoft.CodeAnalysis.Location ReportingLocation
{
get
{
var syn = GetSyntax();
if (syn != null)
{
return syn.Identifier.GetLocation();
}
else if (symbol.IsImplicitlyDeclared)
{
return ContainingType.ReportingLocation;
}
else
{
return symbol.ContainingType.Locations.FirstOrDefault();
}
}
}
class ConstructorFactory : ICachedEntityFactory<IMethodSymbol, Constructor>
{
public static readonly ConstructorFactory Instance = new ConstructorFactory();
public Constructor Create(Context cx, IMethodSymbol init) => new Constructor(cx, init);
}
}
}

37
csharp/extractor/Semmle.Extraction.CSharp/Entities/Conversion.cs Normal file
View File

@@ -0,0 +1,37 @@
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.CSharp.Populators;
using System.Linq;
namespace Semmle.Extraction.CSharp.Entities
{
class Conversion : UserOperator
{
Conversion(Context cx, IMethodSymbol init)
: base(cx, init) { }
public new static Conversion Create(Context cx, IMethodSymbol symbol) =>
ConversionFactory.Instance.CreateEntity(cx, symbol);
public override Microsoft.CodeAnalysis.Location ReportingLocation
{
get
{
return symbol.
DeclaringSyntaxReferences.
Select(r => r.GetSyntax()).
OfType<ConversionOperatorDeclarationSyntax>().
Select(s => s.FixedLocation()).
Concat(symbol.Locations).
FirstOrDefault();
}
}
class ConversionFactory : ICachedEntityFactory<IMethodSymbol, Conversion>
{
public static readonly ConversionFactory Instance = new ConversionFactory();
public Conversion Create(Context cx, IMethodSymbol init) => new Conversion(cx, init);
}
}
}

35
csharp/extractor/Semmle.Extraction.CSharp/Entities/Destructor.cs Normal file
View File

@@ -0,0 +1,35 @@
using Microsoft.CodeAnalysis;
namespace Semmle.Extraction.CSharp.Entities
{
class Destructor : Method
{
Destructor(Context cx, IMethodSymbol init)
: base(cx, init) { }
public override void Populate()
{
PopulateMethod();
ExtractModifiers();
ContainingType.ExtractGenerics();
Context.Emit(Tuples.destructors(this, string.Format("~{0}", symbol.ContainingType.Name), ContainingType, OriginalDefinition(Context, this, symbol)));
Context.Emit(Tuples.destructor_location(this, Location));
}
static new Destructor OriginalDefinition(Context cx, Destructor original, IMethodSymbol symbol)
{
return symbol.OriginalDefinition == null || Equals(symbol.OriginalDefinition, symbol) ? original : Create(cx, symbol.OriginalDefinition);
}
public new static Destructor Create(Context cx, IMethodSymbol symbol) =>
DestructorFactory.Instance.CreateEntity(cx, symbol);
class DestructorFactory : ICachedEntityFactory<IMethodSymbol, Destructor>
{
public static readonly DestructorFactory Instance = new DestructorFactory();
public Destructor Create(Context cx, IMethodSymbol init) => new Destructor(cx, init);
}
}
}

76
csharp/extractor/Semmle.Extraction.CSharp/Entities/Event.cs Normal file
View File

@@ -0,0 +1,76 @@
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using System.Linq;
namespace Semmle.Extraction.CSharp.Entities
{
class Event : CachedSymbol<IEventSymbol>
{
Event(Context cx, IEventSymbol init)
: base(cx, init) { }
public override IId Id
{
get
{
return new Key(tb =>
{
tb.Append(ContainingType);
tb.Append(".");
Method.AddExplicitInterfaceQualifierToId(Context, tb, symbol.ExplicitInterfaceImplementations);
tb.Append(symbol.Name);
tb.Append(";event");
});
}
}
public override void Populate()
{
var type = Type.Create(Context, symbol.Type);
Context.Emit(Tuples.events(this, symbol.GetName(), ContainingType, type.TypeRef, Create(Context, symbol.OriginalDefinition)));
var adder = symbol.AddMethod;
if (adder != null)
EventAccessor.Create(Context, adder);
var remover = symbol.RemoveMethod;
if (remover != null)
EventAccessor.Create(Context, remover);
ExtractModifiers();
BindComments();
var declSyntaxReferences = IsSourceDeclaration
? symbol.DeclaringSyntaxReferences.Select(d => d.GetSyntax()).ToArray()
: Enumerable.Empty<SyntaxNode>();
foreach (var explicitInterface in symbol.ExplicitInterfaceImplementations.Select(impl => Type.Create(Context, impl.ContainingType)))
{
Context.Emit(Tuples.explicitly_implements(this, explicitInterface.TypeRef));
foreach (var syntax in declSyntaxReferences.OfType<EventDeclarationSyntax>())
TypeMention.Create(Context, syntax.ExplicitInterfaceSpecifier.Name, this, explicitInterface);
}
foreach (var l in Locations)
Context.Emit(Tuples.event_location(this, l));
foreach (var syntaxType in declSyntaxReferences.OfType<VariableDeclaratorSyntax>().
Select(d => d.Parent).
OfType<VariableDeclarationSyntax>().
Select(syntax => syntax.Type))
TypeMention.Create(Context, syntaxType, this, type);
}
public static Event Create(Context cx, IEventSymbol symbol) => EventFactory.Instance.CreateEntity(cx, symbol);
class EventFactory : ICachedEntityFactory<IEventSymbol, Event>
{
public static readonly EventFactory Instance = new EventFactory();
public Event Create(Context cx, IEventSymbol init) => new Event(cx, init);
}
public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.NoLabel;
}
}

64
csharp/extractor/Semmle.Extraction.CSharp/Entities/EventAccessor.cs Normal file
View File

@@ -0,0 +1,64 @@
using Microsoft.CodeAnalysis;
namespace Semmle.Extraction.CSharp.Entities
{
class EventAccessor : Accessor
{
EventAccessor(Context cx, IMethodSymbol init)
: base(cx, init) { }
/// <summary>
/// Gets the event symbol associated with this accessor.
/// </summary>
IEventSymbol EventSymbol => symbol.AssociatedSymbol as IEventSymbol;
public override void Populate()
{
PopulateMethod();
ContainingType.ExtractGenerics();
var @event = EventSymbol;
if (@event == null)
{
Context.ModelError(symbol, "Unhandled event accessor associated symbol");
return;
}
var parent = Event.Create(Context, @event);
int kind;
EventAccessor unboundAccessor;
if (symbol.Equals(@event.AddMethod))
{
kind = 1;
unboundAccessor = Create(Context, @event.OriginalDefinition.AddMethod);
}
else if (symbol.Equals(@event.RemoveMethod))
{
kind = 2;
unboundAccessor = Create(Context, @event.OriginalDefinition.RemoveMethod);
}
else
{
Context.ModelError(symbol, "Undhandled event accessor kind");
return;
}
Context.Emit(Tuples.event_accessors(this, kind, symbol.Name, parent, unboundAccessor));
foreach (var l in Locations)
Context.Emit(Tuples.event_accessor_location(this, l));
Overrides();
}
public new static EventAccessor Create(Context cx, IMethodSymbol symbol) =>
EventAccessorFactory.Instance.CreateEntity(cx, symbol);
class EventAccessorFactory : ICachedEntityFactory<IMethodSymbol, EventAccessor>
{
public static readonly EventAccessorFactory Instance = new EventAccessorFactory();
public EventAccessor Create(Context cx, IMethodSymbol init) => new EventAccessor(cx, init);
}
}
}

527
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expression.cs Normal file
View File

@@ -0,0 +1,527 @@
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.CSharp.Populators;
using Semmle.Extraction.Entities;
using Semmle.Extraction.Kinds;
using System.Linq;
namespace Semmle.Extraction.CSharp.Entities
{
public interface IExpressionParentEntity : IEntity
{
/// <summary>
/// Whether this entity is the parent of a top-level expression.
/// </summary>
bool IsTopLevelParent { get; }
}
class Expression : FreshEntity, IExpressionParentEntity
{
public readonly Type Type;
public readonly Extraction.Entities.Location Location;
public readonly ExprKind Kind;
internal Expression(IExpressionInfo info)
: base(info.Context)
{
Location = info.Location;
Kind = info.Kind;
Type = info.Type ?? NullType.Create(cx);
cx.Emit(Tuples.expressions(this, Kind, Type.TypeRef));
if (info.Parent.IsTopLevelParent)
cx.Emit(Tuples.expr_parent_top_level(this, info.Child, info.Parent));
else
cx.Emit(Tuples.expr_parent(this, info.Child, info.Parent));
cx.Emit(Tuples.expr_location(this, Location));
if (info.IsCompilerGenerated)
cx.Emit(Tuples.expr_compiler_generated(this));
if (info.ExprValue is string value)
cx.Emit(Tuples.expr_value(this, value));
Type.ExtractGenerics();
}
public override Microsoft.CodeAnalysis.Location ReportingLocation => Location.symbol;
bool IExpressionParentEntity.IsTopLevelParent => false;
/// <summary>
/// Gets a string represention of a constant value.
/// </summary>
/// <param name="obj">The value.</param>
/// <returns>The string representation.</returns>
public static string ValueAsString(object value)
{
return value == null ? "null" : value is bool ? ((bool)value ? "true" : "false") : value.ToString();
}
/// <summary>
/// Creates an expression from a syntax node.
/// Inserts type conversion as required.
/// </summary>
/// <param name="cx">The extraction context.</param>
/// <param name="node">The node to extract.</param>
/// <param name="parent">The parent entity.</param>
/// <param name="child">The child index.</param>
/// <param name="type">A type hint.</param>
/// <returns>The new expression.</returns>
public static Expression Create(Context cx, ExpressionSyntax node, IExpressionParentEntity parent, int child) =>
CreateFromNode(new ExpressionNodeInfo(cx, node, parent, child));
public static Expression CreateFromNode(ExpressionNodeInfo info) => Expressions.ImplicitCast.Create(info);
/// <summary>
/// Creates an expression from a syntax node.
/// Inserts type conversion as required.
/// Population is deferred to avoid overflowing the stack.
/// </summary>
/// <param name="cx">The extraction context.</param>
/// <param name="node">The node to extract.</param>
/// <param name="parent">The parent entity.</param>
/// <param name="child">The child index.</param>
/// <param name="type">A type hint.</param>
public static void CreateDeferred(Context cx, ExpressionSyntax node, IExpressionParentEntity parent, int child)
{
if (ContainsPattern(node))
// Expressions with patterns should be created right away, as they may introduce
// local variables referenced in `LocalVariable::GetAlreadyCreated()`
Create(cx, node, parent, child);
else
cx.PopulateLater(() => Create(cx, node, parent, child));
}
static bool ContainsPattern(SyntaxNode node) =>
node is PatternSyntax || node is VariableDesignationSyntax || node.ChildNodes().Any(ContainsPattern);
/// <summary>
/// Adapt the operator kind depending on whether it's a dynamic call or a user-operator call.
/// </summary>
/// <param name="cx"></param>
/// <param name="node"></param>
/// <param name="originalKind"></param>
/// <returns></returns>
public static ExprKind UnaryOperatorKind(Context cx, ExprKind originalKind, ExpressionSyntax node) =>
GetCallType(cx, node).AdjustKind(originalKind);
/// <summary>
/// If the expression calls an operator, add an expr_call()
/// to show the target of the call. Also note the dynamic method
/// name if available.
/// </summary>
/// <param name="cx">Context</param>
/// <param name="node">The expression.</param>
public void OperatorCall(ExpressionSyntax node)
{
var @operator = cx.GetSymbolInfo(node);
var method = @operator.Symbol as IMethodSymbol;
if (GetCallType(cx, node) == CallType.Dynamic)
{
UserOperator.OperatorSymbol(method.Name, out string operatorName);
cx.Emit(Tuples.dynamic_member_name(this, operatorName));
return;
}
if (method != null)
cx.Emit(Tuples.expr_call(this, Method.Create(cx, method)));
}
public enum CallType
{
None,
BuiltInOperator,
Dynamic,
UserOperator
}
/// <summary>
/// Determine what type of method was called for this expression.
/// </summary>
/// <param name="cx">The context.</param>
/// <param name="node">The expression</param>
/// <returns>The call type.</returns>
public static CallType GetCallType(Context cx, ExpressionSyntax node)
{
var @operator = cx.GetSymbolInfo(node);
if (@operator.Symbol != null)
{
var method = @operator.Symbol as IMethodSymbol;
var containingSymbol = method.ContainingSymbol as ITypeSymbol;
if (containingSymbol != null && containingSymbol.TypeKind == Microsoft.CodeAnalysis.TypeKind.Dynamic)
{
return CallType.Dynamic;
}
switch (method.MethodKind)
{
case MethodKind.BuiltinOperator:
if (method.ContainingType != null && method.ContainingType.TypeKind == Microsoft.CodeAnalysis.TypeKind.Delegate)
return CallType.UserOperator;
return CallType.BuiltInOperator;
default:
return CallType.UserOperator;
}
}
return CallType.None;
}
public static bool IsDynamic(Context cx, ExpressionSyntax node)
{
var ti = cx.GetTypeInfo(node).ConvertedType;
return ti != null && ti.TypeKind == Microsoft.CodeAnalysis.TypeKind.Dynamic;
}
/// <summary>
/// Given b in a?.b.c, return a.
/// </summary>
/// <param name="node">A MemberBindingExpression.</param>
/// <returns>The qualifier of the conditional access.</returns>
protected static ExpressionSyntax FindConditionalQualifier(ExpressionSyntax node)
{
for (SyntaxNode n = node; n != null; n = n.Parent)
{
var conditionalAccess = n.Parent as ConditionalAccessExpressionSyntax;
if (conditionalAccess != null && conditionalAccess.WhenNotNull == n)
return conditionalAccess.Expression;
}
throw new InternalError(node, "Unable to locate a ConditionalAccessExpression");
}
public void MakeConditional()
{
cx.Emit(Tuples.conditional_access(this));
}
public void PopulateArguments(Context cx, BaseArgumentListSyntax args, int child)
{
foreach (var arg in args.Arguments)
PopulateArgument(cx, arg, child++);
}
private void PopulateArgument(Context cx, ArgumentSyntax arg, int child)
{
var expr = Create(cx, arg.Expression, this, child);
int mode;
switch (arg.RefOrOutKeyword.Kind())
{
case SyntaxKind.RefKeyword:
mode = 1;
break;
case SyntaxKind.OutKeyword:
mode = 2;
break;
case SyntaxKind.None:
mode = 0;
break;
default:
throw new InternalError(arg, "Unknown argument type");
}
cx.Emit(Tuples.expr_argument(expr, mode));
if (arg.NameColon != null)
{
cx.Emit(Tuples.expr_argument_name(expr, arg.NameColon.Name.Identifier.Text));
}
}
public override string ToString() => Label.ToString();
public override TrapStackBehaviour TrapStackBehaviour => TrapStackBehaviour.OptionalLabel;
}
static class CallTypeExtensions
{
/// <summary>
/// Adjust the expression kind <paramref name="k"/> to match this call type.
/// </summary>
public static ExprKind AdjustKind(this Expression.CallType ct, ExprKind k)
{
switch (ct)
{
case Expression.CallType.Dynamic:
case Expression.CallType.UserOperator:
return ExprKind.OPERATOR_INVOCATION;
default:
return k;
}
}
}
abstract class Expression<SyntaxNode> : Expression
where SyntaxNode : ExpressionSyntax
{
public readonly SyntaxNode Syntax;
protected Expression(ExpressionNodeInfo info)
: base(info)
{
Syntax = (SyntaxNode)info.Node;
}
/// <summary>
/// Populates expression-type specific relations in the trap file. The general relations
/// <code>expressions</code> and <code>expr_location</code> are populated by the constructor
/// (should not fail), so even if expression-type specific population fails (e.g., in
/// standalone extraction), the expression created via
/// <see cref="Expression.Create(Context, ExpressionSyntax, IEntity, int, ITypeSymbol)"/> will
/// still be valid.
/// </summary>
protected abstract void Populate();
protected Expression TryPopulate()
{
cx.Try(Syntax, null, Populate);
return this;
}
}
/// <summary>
/// Holds all information required to create an Expression entity.
/// </summary>
interface IExpressionInfo
{
Context Context { get; }
/// <summary>
/// The type of the expression.
/// </summary>
Type Type { get; }
/// <summary>
/// The location of the expression.
/// </summary>
Extraction.Entities.Location Location { get; }
/// <summary>
/// The kind of the expression.
/// </summary>
ExprKind Kind { get; }
/// <summary>
/// The parent of the expression.
/// </summary>
IExpressionParentEntity Parent { get; }
/// <summary>
/// The child index of the expression.
/// </summary>
int Child { get; }
/// <summary>
/// Holds if this is an implicit expression.
/// </summary>
bool IsCompilerGenerated { get; }
/// <summary>
/// Gets a string representation of the value.
/// null is encoded as the string "null".
/// If the expression does not have a value, then this
/// is null.
/// </summary>
string ExprValue { get; }
}
/// <summary>
/// Explicitly constructed expression information.
/// </summary>
class ExpressionInfo : IExpressionInfo
{
public Context Context { get; }
public Type Type { get; }
public Extraction.Entities.Location Location { get; }
public ExprKind Kind { get; }
public IExpressionParentEntity Parent { get; }
public int Child { get; }
public bool IsCompilerGenerated { get; }
public string ExprValue { get; }
public ExpressionInfo(Context cx, Type type, Extraction.Entities.Location location, ExprKind kind, IExpressionParentEntity parent, int child, bool isCompilerGenerated, string value)
{
Context = cx;
Type = type;
Location = location;
Kind = kind;
Parent = parent;
Child = child;
ExprValue = value;
IsCompilerGenerated = isCompilerGenerated;
}
}
/// <summary>
/// Expression information constructed from a syntax node.
/// </summary>
class ExpressionNodeInfo : IExpressionInfo
{
public ExpressionNodeInfo(Context cx, ExpressionSyntax node, IExpressionParentEntity parent, int child) :
this(cx, node, parent, child, cx.GetTypeInfo(node))
{
}
public ExpressionNodeInfo(Context cx, ExpressionSyntax node, IExpressionParentEntity parent, int child, TypeInfo typeInfo)
{
Context = cx;
Node = node;
Parent = parent;
Child = child;
TypeInfo = typeInfo;
Conversion = cx.Model(node).GetConversion(node);
}
public ExpressionNodeInfo(Context cx, ExpressionSyntax node, IExpressionParentEntity parent, int child, ITypeSymbol type) :
this(cx, node, parent, child)
{
Type = Type.Create(cx, type);
}
public Context Context { get; }
public ExpressionSyntax Node { get; private set; }
public IExpressionParentEntity Parent { get; set; }
public int Child { get; set; }
public TypeInfo TypeInfo { get; }
public Microsoft.CodeAnalysis.CSharp.Conversion Conversion { get; }
public ITypeSymbol ResolvedType => Context.DisambiguateType(TypeInfo.Type);
public ITypeSymbol ConvertedType => Context.DisambiguateType(TypeInfo.ConvertedType);
public ITypeSymbol ExpressionType
{
get
{
var type = ResolvedType;
if (type == null)
type = Context.DisambiguateType(TypeInfo.Type ?? TypeInfo.ConvertedType);
// Roslyn workaround: It can't work out the type of "new object[0]"
// Clearly a bug.
if (type != null && type.TypeKind == Microsoft.CodeAnalysis.TypeKind.Error)
{
var arrayCreation = Node as ArrayCreationExpressionSyntax;
if (arrayCreation != null)
{
var elementType = Context.GetType(arrayCreation.Type.ElementType);
if (elementType != null)
return Context.Compilation.CreateArrayTypeSymbol(elementType, arrayCreation.Type.RankSpecifiers.Count);
}
Context.ModelError(Node, "Failed to determine type");
}
return type;
}
}
Microsoft.CodeAnalysis.Location location;
public Microsoft.CodeAnalysis.Location CodeAnalysisLocation
{
get
{
if (location == null)
location = Node.FixedLocation();
return location;
}
set
{
location = value;
}
}
public SemanticModel Model => Context.Model(Node);
public string ExprValue
{
get
{
var c = Model.GetConstantValue(Node);
return c.HasValue ? Expression.ValueAsString(c.Value) : null;
}
}
Type cachedType;
public Type Type
{
get
{
if (cachedType == null)
cachedType = Type.Create(Context, ExpressionType);
return cachedType;
}
set
{
cachedType = value;
}
}
Extraction.Entities.Location cachedLocation;
public Extraction.Entities.Location Location
{
get
{
if (cachedLocation == null)
cachedLocation = Context.Create(CodeAnalysisLocation);
return cachedLocation;
}
set
{
cachedLocation = value;
}
}
public ExprKind Kind { get; set; } = ExprKind.UNKNOWN;
public bool IsCompilerGenerated { get; set; }
public ExpressionNodeInfo SetParent(IExpressionParentEntity parent, int child)
{
Parent = parent;
Child = child;
return this;
}
public ExpressionNodeInfo SetKind(ExprKind kind)
{
Kind = kind;
return this;
}
public ExpressionNodeInfo SetType(Type type)
{
Type = type;
return this;
}
public ExpressionNodeInfo SetNode(ExpressionSyntax node)
{
Node = node;
return this;
}
SymbolInfo cachedSymbolInfo;
public SymbolInfo SymbolInfo
{
get
{
if (cachedSymbolInfo.Symbol == null && cachedSymbolInfo.CandidateReason == CandidateReason.None)
cachedSymbolInfo = Model.GetSymbolInfo(Node);
return cachedSymbolInfo;
}
}
}
}

55
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Access.cs Normal file
View File

@@ -0,0 +1,55 @@
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class Access : Expression
{
static ExprKind AccessKind(Context cx, ISymbol symbol)
{
switch (symbol.Kind)
{
case SymbolKind.TypeParameter:
case SymbolKind.NamedType:
return ExprKind.TYPE_ACCESS;
case SymbolKind.Field:
return ExprKind.FIELD_ACCESS;
case SymbolKind.Property:
return ExprKind.PROPERTY_ACCESS;
case SymbolKind.Event:
return ExprKind.EVENT_ACCESS;
case SymbolKind.Method:
return ExprKind.METHOD_ACCESS;
case SymbolKind.Local:
case SymbolKind.RangeVariable:
return ExprKind.LOCAL_VARIABLE_ACCESS;
case SymbolKind.Parameter:
return ExprKind.PARAMETER_ACCESS;
default:
cx.ModelError(symbol, "Unhandled access kind '{0}'", symbol.Kind);
return ExprKind.UNKNOWN;
}
}
Access(ExpressionNodeInfo info, ISymbol symbol, bool implicitThis, IEntity target)
: base(info.SetKind(AccessKind(info.Context, symbol)))
{
cx.Emit(Tuples.expr_access(this, target));
if (implicitThis && !symbol.IsStatic)
{
This.CreateImplicit(cx, Type.Create(cx, symbol.ContainingType), Location, this, -1);
}
}
public static Expression Create(ExpressionNodeInfo info, ISymbol symbol, bool implicitThis, IEntity target) => new Access(info, symbol, implicitThis, target);
}
}

18
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/ArgList.cs Normal file
View File

@@ -0,0 +1,18 @@
using Semmle.Extraction.Kinds;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using System;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class ArgList : Expression<ExpressionSyntax>
{
ArgList(ExpressionNodeInfo info) : base(info.SetKind(ExprKind.UNKNOWN)) { }
protected override void Populate()
{
throw new NotImplementedException();
}
public static ArgList Create(ExpressionNodeInfo info) => new ArgList(info);
}
}

108
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/ArrayCreation.cs Normal file
View File

@@ -0,0 +1,108 @@
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds;
using System.Linq;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
abstract class ArrayCreation<SyntaxNode> : Expression<SyntaxNode> where SyntaxNode : ExpressionSyntax
{
protected ArrayCreation(ExpressionNodeInfo info) : base(info) { }
}
class StackAllocArrayCreation : ArrayCreation<StackAllocArrayCreationExpressionSyntax>
{
StackAllocArrayCreation(ExpressionNodeInfo info) : base(info.SetKind(ExprKind.ARRAY_CREATION)) { }
public static Expression Create(ExpressionNodeInfo info) => new StackAllocArrayCreation(info).TryPopulate();
protected override void Populate()
{
var arrayType = Syntax.Type as ArrayTypeSyntax;
if (arrayType == null)
{
cx.ModelError(Syntax, "Unexpected array type");
return;
}
var child = 0;
foreach (var rank in arrayType.RankSpecifiers.SelectMany(rs => rs.Sizes))
{
Create(cx, rank, this, child++);
}
cx.Emit(Tuples.explicitly_sized_array_creation(this));
}
}
class ExplicitArrayCreation : ArrayCreation<ArrayCreationExpressionSyntax>
{
ExplicitArrayCreation(ExpressionNodeInfo info) : base(info.SetKind(ExprKind.ARRAY_CREATION)) { }
public static Expression Create(ExpressionNodeInfo info) => new ExplicitArrayCreation(info).TryPopulate();
protected override void Populate()
{
var child = 0;
bool explicitlySized = false;
foreach (var rank in Syntax.Type.RankSpecifiers.SelectMany(rs => rs.Sizes))
{
if (rank is OmittedArraySizeExpressionSyntax)
{
// Create an expression which simulates the explicit size of the array
if (Syntax.Initializer != null)
{
// An implicitly-sized array must have an initializer.
// Guard it just in case.
var size = Syntax.Initializer.Expressions.Count;
var info = new ExpressionInfo(
cx,
Type.Create(cx, cx.Compilation.GetSpecialType(Microsoft.CodeAnalysis.SpecialType.System_Int32)),
Location,
ExprKind.INT_LITERAL,
this,
child,
false,
size.ToString());
new Expression(info);
}
}
else
{
Create(cx, rank, this, child);
explicitlySized = true;
}
child++;
}
if (Syntax.Initializer != null)
{
ArrayInitializer.Create(new ExpressionNodeInfo(cx, Syntax.Initializer, this, -1));
}
if (explicitlySized)
cx.Emit(Tuples.explicitly_sized_array_creation(this));
}
}
class ImplicitArrayCreation : ArrayCreation<ImplicitArrayCreationExpressionSyntax>
{
ImplicitArrayCreation(ExpressionNodeInfo info) : base(info.SetKind(ExprKind.ARRAY_CREATION)) { }
public static Expression Create(ExpressionNodeInfo info) => new ImplicitArrayCreation(info).TryPopulate();
protected override void Populate()
{
if (Syntax.Initializer != null)
{
ArrayInitializer.Create(new ExpressionNodeInfo(cx, Syntax.Initializer, this, -1));
}
cx.Emit(Tuples.implicitly_typed_array_creation(this));
}
}
}

154
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Assignment.cs Normal file
View File

@@ -0,0 +1,154 @@
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.CodeAnalysis.CSharp;
using Semmle.Extraction.CSharp.Populators;
using Semmle.Extraction.Kinds;
using Microsoft.CodeAnalysis;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class Assignment : Expression<AssignmentExpressionSyntax>
{
Assignment(ExpressionNodeInfo info)
: base(info.SetKind(GetKind(info.Context, (AssignmentExpressionSyntax)info.Node)))
{
}
public static Assignment Create(ExpressionNodeInfo info)
{
var ret = new Assignment(info);
ret.TryPopulate();
return ret;
}
protected override void Populate()
{
var operatorKind = OperatorKind;
if (operatorKind.HasValue)
{
// Convert assignment such as `a += b` into `a = a + b`.
var simpleAssignExpr = new Expression(new ExpressionInfo(cx, Type, Location, ExprKind.SIMPLE_ASSIGN, this, 2, false, null));
Create(cx, Syntax.Left, simpleAssignExpr, 1);
var opexpr = new Expression(new ExpressionInfo(cx, Type, Location, operatorKind.Value, simpleAssignExpr, 0, false, null));
Create(cx, Syntax.Left, opexpr, 0);
Create(cx, Syntax.Right, opexpr, 1);
opexpr.OperatorCall(Syntax);
}
else
{
Create(cx, Syntax.Left, this, 1);
Create(cx, Syntax.Right, this, 0);
if (Kind == ExprKind.ADD_EVENT || Kind == ExprKind.REMOVE_EVENT)
{
OperatorCall(Syntax);
}
}
}
static ExprKind GetAssignmentOperation(Context cx, AssignmentExpressionSyntax syntax)
{
switch (syntax.OperatorToken.Kind())
{
case SyntaxKind.PlusEqualsToken:
return ExprKind.ASSIGN_ADD;
case SyntaxKind.MinusEqualsToken:
return ExprKind.ASSIGN_SUB;
case SyntaxKind.EqualsToken:
return ExprKind.SIMPLE_ASSIGN;
case SyntaxKind.BarEqualsToken:
return ExprKind.ASSIGN_OR;
case SyntaxKind.AmpersandEqualsToken:
return ExprKind.ASSIGN_AND;
case SyntaxKind.CaretEqualsToken:
return ExprKind.ASSIGN_XOR;
case SyntaxKind.AsteriskEqualsToken:
return ExprKind.ASSIGN_MUL;
case SyntaxKind.PercentEqualsToken:
return ExprKind.ASSIGN_REM;
case SyntaxKind.SlashEqualsToken:
return ExprKind.ASSIGN_DIV;
case SyntaxKind.LessThanLessThanEqualsToken:
return ExprKind.ASSIGN_LSHIFT;
case SyntaxKind.GreaterThanGreaterThanEqualsToken:
return ExprKind.ASSIGN_RSHIFT;
default:
cx.ModelError(syntax, "Unrecognised assignment type " + GetKind(cx, syntax));
return ExprKind.UNKNOWN;
}
}
static ExprKind GetKind(Context cx, AssignmentExpressionSyntax syntax)
{
var leftSymbol = cx.GetSymbolInfo(syntax.Left);
bool assignEvent = leftSymbol.Symbol != null && leftSymbol.Symbol is IEventSymbol;
var kind = GetAssignmentOperation(cx, syntax);
var leftType = cx.GetType(syntax.Left);
if (leftType != null && leftType.SpecialType != SpecialType.None)
{
// In Mono, the builtin types did not specify their operator invocation
// even though EVERY operator has an invocation in C#. (This is a flaw in the dbscheme and should be fixed).
return kind;
}
if (kind == ExprKind.ASSIGN_ADD && assignEvent)
{
return ExprKind.ADD_EVENT;
}
if (kind == ExprKind.ASSIGN_SUB && assignEvent)
{
return ExprKind.REMOVE_EVENT;
}
return kind;
}
/// <summary>
/// Gets the kind of this assignment operator (<code>null</code> if the
/// assignment is not an assignment operator). For example, the operator
/// kind of `*=` is `*`.
/// </summary>
ExprKind? OperatorKind
{
get
{
var kind = Kind;
if (kind == ExprKind.REMOVE_EVENT || kind == ExprKind.ADD_EVENT || kind == ExprKind.SIMPLE_ASSIGN)
return null;
if (CallType.AdjustKind(kind) == ExprKind.OPERATOR_INVOCATION)
return ExprKind.OPERATOR_INVOCATION;
switch (kind)
{
case ExprKind.ASSIGN_ADD:
return ExprKind.ADD;
case ExprKind.ASSIGN_AND:
return ExprKind.BIT_AND;
case ExprKind.ASSIGN_DIV:
return ExprKind.DIV;
case ExprKind.ASSIGN_LSHIFT:
return ExprKind.LSHIFT;
case ExprKind.ASSIGN_MUL:
return ExprKind.MUL;
case ExprKind.ASSIGN_OR:
return ExprKind.BIT_OR;
case ExprKind.ASSIGN_REM:
return ExprKind.REM;
case ExprKind.ASSIGN_RSHIFT:
return ExprKind.RSHIFT;
case ExprKind.ASSIGN_SUB:
return ExprKind.SUB;
case ExprKind.ASSIGN_XOR:
return ExprKind.BIT_XOR;
default:
cx.ModelError(Syntax, "Couldn't unfold assignment of type " + kind);
return ExprKind.UNKNOWN;
}
}
}
public new CallType CallType => GetCallType(cx, Syntax);
}
}

17
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Await.cs Normal file
View File

@@ -0,0 +1,17 @@
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class Await : Expression<AwaitExpressionSyntax>
{
Await(ExpressionNodeInfo info) : base(info.SetKind(ExprKind.AWAIT)) { }
public static Expression Create(ExpressionNodeInfo info) => new Await(info).TryPopulate();
protected override void Populate()
{
Create(cx, Syntax.Expression, this, 0);
}
}
}

12
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Base.cs Normal file
View File

@@ -0,0 +1,12 @@
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class Base : Expression
{
Base(ExpressionNodeInfo info) : base(info.SetKind(ExprKind.BASE_ACCESS)) { }
public static Base Create(ExpressionNodeInfo info) => new Base(info);
}
}

61
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Binary.cs Normal file
View File

@@ -0,0 +1,61 @@
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class Binary : Expression<BinaryExpressionSyntax>
{
Binary(ExpressionNodeInfo info)
: base(info.SetKind(GetKind(info.Context, (BinaryExpressionSyntax)info.Node)))
{
}
public static Expression Create(ExpressionNodeInfo info) => new Binary(info).TryPopulate();
protected override void Populate()
{
OperatorCall(Syntax);
CreateDeferred(cx, Syntax.Left, this, 0);
CreateDeferred(cx, Syntax.Right, this, 1);
}
static ExprKind GetKind(Context cx, BinaryExpressionSyntax node)
{
var k = GetBinaryTokenKind(cx, node.OperatorToken.Kind());
return GetCallType(cx, node).AdjustKind(k);
}
static ExprKind GetBinaryTokenKind(Context cx, SyntaxKind kind)
{
switch (kind)
{
case SyntaxKind.LessThanToken: return ExprKind.LT;
case SyntaxKind.PlusToken: return ExprKind.ADD;
case SyntaxKind.LessThanEqualsToken: return ExprKind.LE;
case SyntaxKind.GreaterThanToken: return ExprKind.GT;
case SyntaxKind.AsteriskToken: return ExprKind.MUL;
case SyntaxKind.AmpersandAmpersandToken: return ExprKind.LOG_AND;
case SyntaxKind.EqualsEqualsToken: return ExprKind.EQ;
case SyntaxKind.PercentToken: return ExprKind.REM;
case SyntaxKind.MinusToken: return ExprKind.SUB;
case SyntaxKind.AmpersandToken: return ExprKind.BIT_AND;
case SyntaxKind.BarToken: return ExprKind.BIT_OR;
case SyntaxKind.SlashToken: return ExprKind.DIV;
case SyntaxKind.ExclamationEqualsToken: return ExprKind.NE;
case SyntaxKind.AsKeyword: return ExprKind.AS;
case SyntaxKind.IsKeyword: return ExprKind.IS;
case SyntaxKind.BarBarToken: return ExprKind.LOG_OR;
case SyntaxKind.GreaterThanEqualsToken: return ExprKind.GE;
case SyntaxKind.GreaterThanGreaterThanToken: return ExprKind.RSHIFT;
case SyntaxKind.LessThanLessThanToken: return ExprKind.LSHIFT;
case SyntaxKind.CaretToken: return ExprKind.BIT_XOR;
case SyntaxKind.QuestionQuestionToken: return ExprKind.NULL_COALESCING;
// !! And the rest
default:
cx.ModelError("Unhandled operator type {0}", kind);
return ExprKind.UNKNOWN;
}
}
}
}

29
csharp/extractor/Semmle.Extraction.CSharp/Entities/Expressions/Cast.cs Normal file
View File

@@ -0,0 +1,29 @@
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds;
namespace Semmle.Extraction.CSharp.Entities.Expressions
{
class Cast : Expression<CastExpressionSyntax>
{
Cast(ExpressionNodeInfo info) : base(info.SetKind(UnaryOperatorKind(info.Context, ExprKind.CAST, info.Node))) { }
public static Expression Create(ExpressionNodeInfo info) => new Cast(info).TryPopulate();
protected override void Populate()
{
Create(cx, Syntax.Expression, this, 0);
if (Kind == ExprKind.CAST)
// Type cast
TypeAccess.Create(new ExpressionNodeInfo(cx, Syntax.Type, this, 1));
else
{
// Type conversion
OperatorCall(Syntax);
TypeMention.Create(cx, Syntax.Type, this, Type);
}
}
public override Microsoft.CodeAnalysis.Location ReportingLocation => Syntax.GetLocation();
}
}

Some files were not shown because too many files have changed in this diff Show More