Compare commits

..

105 Commits

Author SHA1 Message Date
tiferet
07b0a742cf Update TaintedPathConfig after rebase on main 2023-03-15 16:42:58 -07:00
tiferet
68133cb8ca Delete unneeded imports 2023-03-15 16:25:50 -07:00
tiferet
a6841875fc Extract a higher percent of negative examples because some of the kinds are relatively rare. 2023-03-14 12:49:32 -07:00
tiferet
9071f54e4e Add a new sink type: command-injection 2023-03-14 12:49:32 -07:00
tiferet
fad491d1f6 Add a filter that excludes an endpoint if it is unexploitable because it's a call to exists or notExists. 2023-03-14 12:49:32 -07:00
tiferet
cea30cc463 Remove last vestiges of flow from source from the query that extracts sink candidates. 2023-03-14 12:49:32 -07:00
tiferet
103e778a3a The string of each NotASinkCharacteristic should be a clear explanation of why such endpoints are not interesting, and this should be the message exported to the sarif. 2023-03-14 12:49:32 -07:00
tiferet
a3cbf9fb46 Add a subclass of EndpointCharacteristics that surfaces sinks rather than negative examples. Make all the existing ones inherit from this one. 2023-03-14 12:49:32 -07:00
tiferet
7342ac3db2 Add a negative characteristic that filters out calls to undocumented methods in undocumented classes inside the current DB. 2023-03-14 12:49:31 -07:00
tiferet
1e4dd7a9c9 Break up hasMetadata into one predicate that extracts information and another that concatenates it into a string of the needed format, so that we can reuse the information-extraction predicate for other purposes. 2023-03-14 12:49:31 -07:00
tiferet
06b218f0a7 Another small fix to the syntax of the extracted MaD declarations. 2023-03-14 12:49:31 -07:00
tiferet
6e9eeb8eb3 Add a negative characteristic that filters out calls for which the package the call originates from and the package where the callee is defined are the same up to at least three levels, and the callee package is at least four levels deep. 2023-03-14 12:49:31 -07:00
tiferet
3c847f7ce4 Add a negative characteristic that filters out non-public methods. 2023-03-14 12:49:31 -07:00
tiferet
b400b583d6 Add a class of filters indicating not necessarily that an endpoint is not a sink, but rather that it’s not a sink that’s interesting to model in the standard Java libraries. 2023-03-14 12:49:31 -07:00
tiferet
9244912880 Remove IsConstantExpressionCharacteristic, because the value passed into the argument has nothing to do with whether or not that argument is a sink. 2023-03-14 12:49:31 -07:00
tiferet
9307bf9dcd Add a filter that excludes an endpoint if it is unexploitable. 2023-03-14 12:49:31 -07:00
tiferet
06b546f015 Add a filter that excludes an endpoint if it's a non-sink argument to a method whose sinks have already been modeled. 2023-03-14 12:49:31 -07:00
tiferet
f1efc62e16 Promote NeutralModelCharacteristic from LikelyNotASinkCharacteristic to NotASinkCharacteristic. 2023-03-14 12:49:31 -07:00
tiferet
b3e05e9fb5 Fixes to the syntax of the extracted MaD declarations. 2023-03-14 12:49:31 -07:00
tiferet
5b7147a56f Downgrade CreatePathSinkCharacteristic to medium confidence. 2023-03-14 12:49:31 -07:00
tiferet
ced7a33419 Add a negative characteristic that indicates that an endpoint was manually modeled as a neutral model. 2023-03-14 12:49:31 -07:00
tiferet
084f0ee57a Add an endpoint filter that indicates that an endpoint is not a to node for any known taint step. Such a node cannot be tainted, because taint can't flow into it. 2023-03-14 12:49:31 -07:00
tiferet
b48c6badba Add an endpoint filter to filter out MaD-modeled taint steps.
This filter currently has some overlap with `CreatePathSinkCharacteristic`. We add a flag to `erroneousEndpoints` such that these known modeling errors can optionally be ignored.

We turn the flag off when extracting prompt examples, to ensure the prompt contains only examples we're highly certain about.

If there are errors even with this flag turned on, we return an error message in the query that extracts positive examples, to prevent us from accidentally running it when there's a codex-generated data extension file in `java/ql/lib/ext`.
2023-03-14 12:49:31 -07:00
tiferet
25f103a010 Cleanup of EndpointCharacteristics, to get rid of historical naming such as "endpoint filters" and of classes that are used nowhere. 2023-03-14 12:49:31 -07:00
tiferet
d47007a930 Rename AtmConfig to AtmConfigs and fix some imports. 2023-03-14 12:49:31 -07:00
tiferet
e9da1f3751 Rename isEffectiveSink to isSinkCandidate 2023-03-14 12:49:30 -07:00
tiferet
dbb4fa0b1c Replace EndpointType with either SinkType or SourceType wherever possible. 2023-03-14 12:49:30 -07:00
tiferet
f5833ffc3d Simplify AtmConfig:
- We no longer create new configs for each query we want to boost with ATM.
- Instead the `AtmConfig` module imports the configs for the Java queries it can and copies the configs for the ones that are defined in a ql file.
- The predicates that used to be defined in the `AtmConfig` class are now defined either in candidate extraction query or(in the case of `isKnownSink` which is used in more than one file) in `EndpointCharacteristic.qll`.
- Delete all the derived classes of AtmConfig.
- Surface all candidates that pass the endpoint filters, regardless of flow from a source.
2023-03-14 12:49:30 -07:00
tiferet
efb6522656 EndpointType.getKind is final and just returns this. The name of the endpoint type is its MaD kind. Human-readable descriptions of these kinds are encoded only in Python, not in CodeQL. 2023-03-14 12:49:30 -07:00
tiferet
1d5afaec0e Get rid of EndpointType.getDescription 2023-03-14 12:49:30 -07:00
tiferet
43db83a28f Delete some commented out code that was copied directly from JS 2023-03-14 12:49:30 -07:00
tiferet
2e4cc7efd0 Delete EndpointType.getEncoding, which is not needed anywhere.
If we need this down the line for model training, we can add it back in then.
2023-03-14 12:49:30 -07:00
tiferet
bcd1ac1bb0 Delete EndpointType.getEncoding, which is not needed anywhere.
If we need this down the line for model training, we can add it back in then.
2023-03-14 12:49:30 -07:00
tiferet
10b81eebb7 Improve EndpointTypes:
- Create two derived classes for EndpointType: SinkType and SourceType.
- EndpointTypes don't use a `newtype`, but rather extend string, with their characteristic predicate replacing the current getDescription predicate.
2023-03-14 12:49:30 -07:00
tiferet
91109c826d List the MaD provenance as "ai-generated" rather than "manual"
See https://github.com/github/codeql/pull/12228
2023-03-14 12:49:30 -07:00
tiferet
abe3a2dae1 Improve positive prompt examples:
Include only sinks that are arguments to an external API call, because these are the sinks we are most interested in.
2023-03-14 12:49:30 -07:00
tiferet
4db03cf4ae Remove IsMaDTaintStepCharacteristic for now because it's catching all our known sinks as well as taint steps 2023-03-14 12:49:30 -07:00
tiferet
f73b3e0d97 Add endpoint filters:
- Filter out MaD taint steps
2023-03-14 12:49:30 -07:00
tiferet
3b508f7879 Remove redundancy from ExceptionCharacteristic 2023-03-14 12:49:30 -07:00
tiferet
9b028476b8 Add endpoint filters:
- Filter out exceptions
- Filter out endpoints in test files
2023-03-14 12:49:30 -07:00
tiferet
24e01104a2 As part of the metadata extraction predicate, surface whether or not the argument is being passed to an external API 2023-03-14 12:49:29 -07:00
tiferet
8f6db6b244 Switch back to one sink type per supported query, rather than existing MaD kinds. 2023-03-14 12:49:29 -07:00
tiferet
d6c897c9fd Small bug fix for handling queries with multiple sink types:
`getAReasonSinkExcluded` excludes endpoints that have a characteristic that implies they're not sinks for this particular sink type _for every sink type relevant to this query_.
2023-03-14 12:49:29 -07:00
tiferet
8d8a21b100 Fix a bug that allowed some known sinks to end up as sink candidates for codex 2023-03-14 12:49:29 -07:00
tiferet
a27ae27101 In the MaD data, set the subtypes field to false for final classes / methods. 2023-03-14 12:49:29 -07:00
tiferet
4b6d1f7b78 Create a new class other sink:
See https://github.com/github/atm-codex/pull/3

- Add a sink type `OtherMaDSinkType`, and corresponding characteristic `OtherMaDSinkCharacteristic`, for other sinks modeled by a MaD `kind` but not belonging to any of the existing sink types.
- Extract positive prompt examples for the new sink type, together with the corresponding MaD `kind`.
2023-03-14 12:49:29 -07:00
tiferet
66c77e890c Bug fix 2023-03-14 12:49:29 -07:00
tiferet
be9c6500b8 In the MaD data, extract the argument index as an int rather than a string wrapped up in "Argument[]" 2023-03-14 12:49:29 -07:00
tiferet
831830831c Fix the MaD signature to the correct format 2023-03-14 12:49:29 -07:00
tiferet
ae69a2bcd9 Separate out the sink types to align with the MaD kinds that currently exist, adding a sink type for all sinks of a given query that are not currently mapped in the MaD kinds. 2023-03-14 12:49:29 -07:00
tiferet
65923ed2c1 Add support for multiple sink types per query 2023-03-14 12:49:29 -07:00
tiferet
a7269075e2 As part of the metadata extraction predicate, surface whether or not the callee is a public method 2023-03-14 12:49:29 -07:00
tiferet
d3a5ee53c6 Refactor the CodeQL code that extracts metadata for methods presented to Codex, to make it easy to add another field 2023-03-14 12:49:29 -07:00
tiferet
f32bb65c54 Refactor the CodeQL code that extracts metadata for methods presented to Codex, to make it easy to add another field 2023-03-14 12:49:29 -07:00
tiferet
633bfdba28 Make the endpoint filter to filter out flow steps in Java a bit broader, and document it 2023-03-14 12:49:28 -07:00
tiferet
db9cec6ea6 Add an endpoint filter to filter out flow steps 2023-03-14 12:49:28 -07:00
tiferet
ec5425d952 When extracting positive and negative examples for the Java prompt, extract the data used in the MaD extensible predicate.
This will enable the codex prompt to optionally use this data in additional columns.
2023-03-14 12:49:28 -07:00
tiferet
7666843316 Resolve two TODO items 2023-03-14 12:49:28 -07:00
tiferet
e06bcc3112 Exclude negative examples that are type access nodes.
These will never be on a flow path so they're not useful negative examples.
2023-03-14 12:49:28 -07:00
tiferet
3229b37436 Increase diversity of negative prompt examples by creating finer sub-types 2023-03-14 12:49:28 -07:00
tiferet
559570419d If a node satisfies the logic for both isSink and isSanitizer, don't include it as a positive or negative example in the prompt, because it's too ambiguous and will confuse the model. 2023-03-14 12:49:28 -07:00
tiferet
844171a28e Simplify the definition of ExtractPositiveExamples.ql 2023-03-14 12:49:28 -07:00
tiferet
ecf4d4dc02 Avoid accidentally extracting positive prompt examples when there is a codex-generated data extension file in java/ql/lib/ext 2023-03-14 12:49:28 -07:00
tiferet
0d4e85ff93 Add a predicate that finds endpoints with logically-inconsistent characteristics, and exclude such endpoints from both positive and negative examples extracted for the codex prompt. 2023-03-14 12:49:28 -07:00
tiferet
1211197914 Fix codeql-pack.lock.yml so it's not looking for an ML model 2023-03-14 12:49:28 -07:00
tiferet
41df8df182 Typo fix 2023-03-14 12:49:28 -07:00
tiferet
125245aa62 Delete TODO items that are done 2023-03-14 12:49:28 -07:00
tiferet
8bb2b2eaea Have each EndpointType keep track of the sink/source kind for this endpoint type as used in Models as Data 2023-03-14 12:49:28 -07:00
tiferet
27efe524da Fix the extraction of data for the data extension YML file. 2023-03-14 12:49:28 -07:00
tiferet
ae4668c488 Add data needed for the data extension YML file to ExtractSinkCandidatesWithFlow.ql: first pass. 2023-03-14 12:49:28 -07:00
tiferet
3987d8d374 Small update to SafeExternalApiMethodCharacteristic 2023-03-14 12:49:28 -07:00
tiferet
fd75952c1e Improvements to ExtractSinkCandidatesWithFlow.ql 2023-03-14 12:49:28 -07:00
tiferet
4db0dec82e Minor improvement 2023-03-14 12:49:28 -07:00
tiferet
a73b52adef Improvements to ExtractSinkCandidatesWithFlow.ql 2023-03-14 12:49:28 -07:00
tiferet
39a4513fcc Delete the queries the Java team isn't currently interested in boosting 2023-03-14 12:49:28 -07:00
tiferet
3c44332f17 Move isFlowLikelyInBaseQuery to the ATMConfig and delete AdaptiveThreatModeling.qll 2023-03-14 12:49:27 -07:00
tiferet
06c7f1012c Rename request forgery sink to server-side request forgery sink 2023-03-14 12:49:27 -07:00
tiferet
9421ba5303 Add and implementation of request forgery sinks and corresponding positive EndpointCharacteristic in Java 2023-03-14 12:49:27 -07:00
tiferet
f5109be2ac Bug fixes 2023-03-14 12:49:27 -07:00
tiferet
c14a4c4d93 Add an implementation of TaintedPathATM.qll and corresponding positive EndpointCharacteristic in Java 2023-03-14 12:49:27 -07:00
tiferet
4546dbe51b Subsample negative examples to 1% to prevent huge numbers. 2023-03-14 12:49:26 -07:00
tiferet
5d62dc3d2e Add a Java NotASinkCharacteristic safe external API method 2023-03-14 12:49:26 -07:00
tiferet
0acd06a6d3 Add queries to surface high-confidence Java sinks and non-sinks to use as examples in the codex prompt. 2023-03-14 12:49:26 -07:00
tiferet
04abb87fef Rewrite ExtractSinkCandidatesWithFlow.ql as a problem query so we can run it with codeql database analyze to output SARIF results. 2023-03-14 12:49:26 -07:00
tiferet
5dc5c3fb3f Add a couple of endpoint filters for Java 2023-03-14 12:49:26 -07:00
tiferet
653b0128f5 Try implementing SqlInjectionATM.qll in Java 2023-03-14 12:49:26 -07:00
tiferet
c0f58371b4 Start making the additions needed to surface candidate Java sinks for codex classification outside the evaluator. 2023-03-14 12:49:26 -07:00
tiferet
cf289d57e9 Go back to the prompt of https://github.com/github/codeql-dca-main/issues/9475 2023-03-14 12:49:26 -07:00
tiferet
459050151a Give more explicit instructions in the codex prompt, but don't solicit rare sink types. 2023-03-14 12:49:26 -07:00
tiferet
01979aeb62 Give more explicit instructions in the codex prompt. 2023-03-14 12:49:26 -07:00
tiferet
ef95f4c419 Minor prompt improvements:
- Tell codex explicitly that this is JavaScript code
- Replace "Dataflow node" with "Code snippet"
2023-03-14 12:49:26 -07:00
tiferet
ac5434b3f3 Minor prompt improvements:
Remove spaces that break the code syntax or make for strange code styling.
2023-03-14 12:49:26 -07:00
tiferet
ce17d94f80 In-line predicates that are costing a lot of compute time 2023-03-14 12:49:26 -07:00
tiferet
bcc4cdd376 Add a test that can be used to determine the alerts codex will surface for each query. 2023-03-14 12:49:25 -07:00
tiferet
9aba7a0bca Bug fixes for things that interfere with using the codex model 2023-03-14 12:49:25 -07:00
tiferet
9a21539fca Add a test that can be used to determine how well codex reproduces the manual modeling for each sink type. 2023-03-14 12:49:25 -07:00
tiferet
d76d11bd27 Fix endpointScores 2023-03-14 12:49:25 -07:00
tiferet
4603a66411 Bug fix in selecting a node's location:
Locations only exist where there are locatable structures in the DB. Thus, select the largest location that contains the node and at most `neighborhoodSize` lines before and after the node.
2023-03-14 12:49:25 -07:00
tiferet
b130b2e82f Give endpoint types more intuitive names and then use those names directly in composing the codex prompt. 2023-03-14 12:49:25 -07:00
tiferet
94676ed713 Further improve the structure of endpoint scoring 2023-03-14 12:49:25 -07:00
tiferet
4ed57e71db Remove tokens from the prompt that the Java side can't handle 2023-03-14 12:49:25 -07:00
tiferet
12def779e6 Change the prompt to use sink names defined in EndpointType 2023-03-14 12:49:25 -07:00
tiferet
a6c01042eb Improve the structure of endpoint scoring 2023-03-14 12:49:25 -07:00
tiferet
fa36fc838b Pull in the prompt work from branch tiferet/codex-prompt 2023-03-14 12:49:25 -07:00
tiferet
09bf2218d4 Merge in aeisenberg/atm-codex 2023-03-14 12:49:24 -07:00
16849 changed files with 655492 additions and 1674233 deletions

View File

@@ -1,17 +1,3 @@
common --enable_platform_specific_config
common --enable_bzlmod
# because we use --override_module with `%workspace%`, the lock file is not stable
common --lockfile_mode=off
# when building from this repository in isolation, the internal repository will not be found at ..
# where `MODULE.bazel` looks for it. The following will get us past the module loading phase, so
# that we can build things that do not rely on that
common --override_module=semmle_code=%workspace%/misc/bazel/semmle_code_stub
build --repo_env=CC=clang --repo_env=CXX=clang++
build:linux --cxxopt=-std=c++20
build:macos --cxxopt=-std=c++20 --cpu=darwin_x86_64
build:windows --cxxopt=/std:c++20 --cxxopt=/Zc:preprocessor
build --repo_env=CC=clang --repo_env=CXX=clang++ --cxxopt="-std=c++17"
try-import %workspace%/local.bazelrc

View File

@@ -1 +1 @@
7.1.0
5.0.0

View File

@@ -1 +0,0 @@
DisableFormat: true

View File

@@ -1,6 +1,6 @@
{
"extensions": [
"rust-lang.rust-analyzer",
"rust-lang.rust",
"bungcip.better-toml",
"github.vscode-codeql",
"hbenl.vscode-test-explorer",

7
.gitattributes vendored
View File

@@ -71,10 +71,3 @@ go/extractor/opencsv/CSVReader.java -text
# `javascript/ql/experimental/adaptivethreatmodeling/test/update_endpoint_test_files.py`.
javascript/ql/experimental/adaptivethreatmodeling/test/endpoint_large_scale/autogenerated/**/*.js linguist-generated=true -merge
javascript/ql/experimental/adaptivethreatmodeling/test/endpoint_large_scale/autogenerated/**/*.ts linguist-generated=true -merge
# Auto-generated modeling for Python
python/ql/lib/semmle/python/frameworks/data/internal/subclass-capture/*.yml linguist-generated=true
# auto-generated bazel lock file
ruby/extractor/cargo-bazel-lock.json linguist-generated=true
ruby/extractor/cargo-bazel-lock.json -merge

View File

@@ -17,26 +17,3 @@ updates:
ignore:
- dependency-name: '*'
update-types: ['version-update:semver-patch', 'version-update:semver-minor']
- package-ecosystem: "gomod"
directory: "go/extractor"
schedule:
interval: "daily"
allow:
- dependency-name: "golang.org/x/mod"
- dependency-name: "golang.org/x/tools"
groups:
extractor-dependencies:
patterns:
- "golang.org/x/*"
reviewers:
- "github/codeql-go"
- package-ecosystem: "gomod"
directory: "go/ql/test"
schedule:
interval: "monthly"
ignore:
- dependency-name: "*"
reviewers:
- "github/codeql-go"

11
.github/labeler.yml vendored
View File

@@ -11,7 +11,7 @@ Go:
- change-notes/**/*go.*
Java:
- any: [ 'java/**/*', '!java/kotlin-extractor/**/*', '!java/ql/test/kotlin/**/*' ]
- any: [ 'java/**/*', '!java/kotlin-extractor/**/*', '!java/kotlin-explorer/**/*', '!java/ql/test/kotlin/**/*' ]
- change-notes/**/*java.*
JS:
@@ -20,7 +20,8 @@ JS:
Kotlin:
- java/kotlin-extractor/**/*
- java/ql/test-kotlin*/**/*
- java/kotlin-explorer/**/*
- java/ql/test/kotlin/**/*
Python:
- python/**/*
@@ -45,7 +46,11 @@ documentation:
# Since these are all shared files that need to be synced, just pick _one_ copy of each.
"DataFlow Library":
- "shared/dataflow/**/*"
- "java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl.qll"
- "java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplCommon.qll"
- "java/ql/lib/semmle/code/java/dataflow/internal/tainttracking1/TaintTrackingImpl.qll"
- "java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplConsistency.qll"
- "java/ql/lib/semmle/code/java/dataflow/internal/FlowSummaryImpl.qll"
"ATM":
- javascript/ql/experimental/adaptivethreatmodeling/**/*

View File

@@ -0,0 +1,102 @@
name: "ATM - Check query suite"
env:
QUERY_PACK: javascript/ql/experimental/adaptivethreatmodeling/src
QUERY_SUITE: codeql-suites/javascript-atm-code-scanning.qls
on:
pull_request:
paths:
- ".github/workflows/atm-check-query-suite.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/**"
workflow_dispatch:
jobs:
atm-check-query-suite:
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v3
- name: Setup CodeQL
uses: ./.github/actions/fetch-codeql
with:
channel: release
- name: Cache compilation cache
id: query-cache
uses: ./.github/actions/cache-query-compilation
with:
key: atm-suite
- name: Install ATM model
run: |
set -exu
# Install dependencies of ATM query pack, i.e. the ATM model
codeql pack install "${QUERY_PACK}"
# Retrieve model checksum
model_checksum=$(codeql resolve extensions "${QUERY_PACK}/${QUERY_SUITE}" | jq -r '.models[0].checksum')
# Trust the model so that we can use it in the ATM boosted queries
mkdir -p "$HOME/.config/codeql"
echo "--insecurely-execute-ml-model-checksums ${model_checksum}" >> "$HOME/.config/codeql/config"
- name: Create test DB
run: |
DB_PATH="${RUNNER_TEMP}/db"
echo "DB_PATH=${DB_PATH}" >> "${GITHUB_ENV}"
codeql database create "${DB_PATH}" --source-root config/atm --language javascript
- name: Run ATM query suite
run: |
SARIF_PATH="${RUNNER_TEMP}/sarif.json"
echo "SARIF_PATH=${SARIF_PATH}" >> "${GITHUB_ENV}"
codeql database analyze \
--threads=0 \
--ram 50000 \
--format sarif-latest \
--output "${SARIF_PATH}" \
--sarif-group-rules-by-pack \
-vv \
--compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" \
-- \
"${DB_PATH}" \
"${QUERY_PACK}/${QUERY_SUITE}"
- name: Upload SARIF
uses: actions/upload-artifact@v3
with:
name: javascript-ml-powered-queries.sarif
path: "${{ env.SARIF_PATH }}"
retention-days: 5
- name: Check results
run: |
# We should run at least the ML-powered queries in `expected_rules`.
expected_rules="js/ml-powered/nosql-injection js/ml-powered/path-injection js/ml-powered/sql-injection js/ml-powered/xss"
for rule in ${expected_rules}; do
found_rule=$(jq --arg rule "${rule}" '[.runs[0].tool.extensions[].rules | select(. != null) |
flatten | .[].id] | any(. == $rule)' "${SARIF_PATH}")
if [[ "${found_rule}" != "true" ]]; then
echo "Expected SARIF output to contain rule '${rule}', but found no such rule."
exit 1
else
echo "Found rule '${rule}'."
fi
done
# We should have at least one alert from an ML-powered query.
num_alerts=$(jq '[.runs[0].results[] |
select(.properties.score != null and (.rule.id | startswith("js/ml-powered/")))] | length' \
"${SARIF_PATH}")
if [[ "${num_alerts}" -eq 0 ]]; then
echo "Expected to find at least one alert from an ML-powered query but found ${num_alerts}."
exit 1
else
echo "Found ${num_alerts} alerts from ML-powered queries.";
fi

View File

@@ -0,0 +1,12 @@
name: ATM Model Integration Tests
on:
workflow_dispatch:
jobs:
hello-world:
runs-on: ubuntu-latest
steps:
- name: foo
run: echo "Hello world"

View File

@@ -1,8 +1,5 @@
name: Check change note
permissions:
pull-requests: read
on:
pull_request_target:
types: [labeled, unlabeled, opened, synchronize, reopened, ready_for_review]
@@ -11,43 +8,27 @@ on:
- "*/ql/src/**/*.qll"
- "*/ql/lib/**/*.ql"
- "*/ql/lib/**/*.qll"
- "*/ql/lib/**/*.yml"
- "shared/**/*.ql"
- "shared/**/*.qll"
- "!**/experimental/**"
- "!ql/**"
- "!swift/**"
- ".github/workflows/check-change-note.yml"
jobs:
check-change-note:
env:
REPO: ${{ github.repository }}
PULL_REQUEST_NUMBER: ${{ github.event.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
runs-on: ubuntu-latest
steps:
- name: Fail if no change note found. To fix, either add one, or add the `no-change-note-required` label.
if: |
github.event.pull_request.draft == false &&
!contains(github.event.pull_request.labels.*.name, 'no-change-note-required')
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
change_note_files=$(gh api "repos/$REPO/pulls/$PULL_REQUEST_NUMBER/files" --paginate --jq '.[].filename | select(test("/change-notes/.*[.]md$"))')
if [ -z "$change_note_files" ]; then
echo "No change note found. Either add one, or add the 'no-change-note-required' label."
exit 1
fi
echo "Change notes found:"
echo "$change_note_files"
- name: Fail if the change note filename doesn't match the expected format. The file name must be of the form 'YYYY-MM-DD.md', 'YYYY-MM-DD-{title}.md', where '{title}' is arbitrary text, or released/x.y.z.md for released change-notes
gh api 'repos/${{github.repository}}/pulls/${{github.event.number}}/files' --paginate --jq 'any(.[].filename ; test("/change-notes/.*[.]md$"))' |
grep true -c
- name: Fail if the change note filename doesn't match the expected format. The file name must be of the form 'YYYY-MM-DD.md' or 'YYYY-MM-DD-{title}.md', where '{title}' is arbitrary text.
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
bad_change_note_file_names=$(gh api "repos/$REPO/pulls/$PULL_REQUEST_NUMBER/files" --paginate --jq '[.[].filename | select(test("/change-notes/.*[.]md$"))][] | select((test("/change-notes/[0-9]{4}-[0-9]{2}-[0-9]{2}.*[.]md$") or test("/change-notes/released/[0-9]*[.][0-9]*[.][0-9]*[.]md$")) | not)')
if [ -n "$bad_change_note_file_names" ]; then
echo "The following change note file names are invalid:"
echo "$bad_change_note_file_names"
exit 1
fi
gh api 'repos/${{github.repository}}/pulls/${{github.event.number}}/files' --paginate --jq '[.[].filename | select(test("/change-notes/.*[.]md$"))] | all(test("/change-notes/[0-9]{4}-[0-9]{2}-[0-9]{2}.*[.]md$"))' |
grep true -c

View File

@@ -1,32 +0,0 @@
name: "Check implicit this warnings"
on:
workflow_dispatch:
pull_request:
paths:
- "**qlpack.yml"
branches:
- main
- "rc/*"
permissions:
contents: read
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check that implicit this warnings is enabled for all packs
shell: bash
run: |
EXIT_CODE=0
packs="$(find . -iname 'qlpack.yml')"
for pack_file in ${packs}; do
option="$(yq '.warnOnImplicitThis' ${pack_file})"
if [ "${option}" != "true" ]; then
echo "::error file=${pack_file}::warnOnImplicitThis property must be set to 'true' for pack ${pack_file}"
EXIT_CODE=1
fi
done
exit "${EXIT_CODE}"

View File

@@ -10,15 +10,12 @@ on:
- main
- "rc/*"
permissions:
contents: read
jobs:
qldoc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
with:
fetch-depth: 2

View File

@@ -11,14 +11,11 @@ on:
- "rc/*"
workflow_dispatch:
permissions:
contents: read
jobs:
check:
name: Check query IDs
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Check for duplicate query IDs
run: python3 misc/scripts/check-query-ids.py

View File

@@ -5,9 +5,6 @@ on:
schedule:
- cron: "30 1 * * *"
permissions:
issues: write
jobs:
stale:
if: github.repository == 'github/codeql'
@@ -15,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
- uses: actions/stale@v7
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue is stale because it has been open 14 days with no activity. Comment or remove the `Stale` label in order to avoid having this issue closed in 7 days.'

View File

@@ -28,12 +28,12 @@ jobs:
steps:
- name: Setup dotnet
uses: actions/setup-dotnet@v4
uses: actions/setup-dotnet@v3
with:
dotnet-version: 8.0.101
dotnet-version: 7.0.102
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v3
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL

View File

@@ -8,16 +8,12 @@ on:
- "codeql-cli-*"
pull_request:
permissions:
contents: read
jobs:
compile-queries:
if: github.repository_owner == 'github'
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Setup CodeQL
uses: ./.github/actions/fetch-codeql
with:
@@ -28,14 +24,14 @@ jobs:
with:
key: all-queries
- name: check formatting
run: find shared */ql -type f \( -name "*.qll" -o -name "*.ql" \) -print0 | xargs -0 -n 3000 -P 10 codeql query format -q --check-only
run: find */ql -type f \( -name "*.qll" -o -name "*.ql" \) -print0 | xargs -0 -n 3000 -P 10 codeql query format -q --check-only
- name: compile queries - check-only
# run with --check-only if running in a PR (github.sha != main)
if : ${{ github.event_name == 'pull_request' }}
shell: bash
run: codeql query compile -q -j0 */ql/{src,examples} --keep-going --warnings=error --check-only --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" --compilation-cache-size=500
run: codeql query compile -q -j0 */ql/{src,examples} --keep-going --warnings=error --check-only --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"
- name: compile queries - full
# do full compile if running on main - this populates the cache
if : ${{ github.event_name != 'pull_request' }}
shell: bash
run: codeql query compile -q -j0 */ql/{src,examples} --keep-going --warnings=error --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" --compilation-cache-size=500
run: codeql query compile -q -j0 */ql/{src,examples} --keep-going --warnings=error --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"

View File

@@ -1,55 +0,0 @@
name: "Code scanning - C++"
on:
push:
branches:
- main
- 'rc/*'
pull_request:
branches:
- main
- 'rc/*'
paths:
- 'swift/**'
- '.github/codeql/**'
- '.github/workflows/cpp-swift-analysis.yml'
schedule:
- cron: '0 9 * * 1'
jobs:
CodeQL-Build:
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write
pull-requests: read
steps:
- name: Checkout repository
uses: actions/checkout@v4
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@main
# Override language selection by uncommenting this and choosing your languages
with:
languages: cpp
config-file: ./.github/codeql/codeql-config.yml
- name: "[Ubuntu] Remove GCC 13 from runner image"
shell: bash
run: |
sudo rm -f /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-jammy.list
sudo apt-get update
sudo apt-get install -y --allow-downgrades libc6=2.35-* libc6-dev=2.35-* libstdc++6=12.3.0-* libgcc-s1=12.3.0-*
- name: "Build Swift extractor using Bazel"
run: |
bazel clean --expunge
bazel run //swift:create-extractor-pack --nouse_action_cache --noremote_accept_cached --noremote_upload_local_results --spawn_strategy=local --features=-layering_check
bazel shutdown
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@main

View File

@@ -25,14 +25,11 @@ defaults:
run:
working-directory: csharp
permissions:
contents: read
jobs:
qlupgrade:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- name: Check DB upgrade scripts
run: |
@@ -49,14 +46,14 @@ jobs:
xargs codeql execute upgrades testdb
diff -q testdb/semmlecode.csharp.dbscheme downgrades/initial/semmlecode.csharp.dbscheme
qltest:
if: github.repository_owner == 'github'
runs-on: ubuntu-latest-xl
strategy:
fail-fast: false
matrix:
slice: ["1/2", "2/2"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: ./csharp/actions/create-extractor-pack
- name: Cache compilation cache
id: query-cache
@@ -65,41 +62,25 @@ jobs:
key: csharp-qltest-${{ matrix.slice }}
- name: Run QL tests
run: |
codeql test run --threads=0 --ram 50000 --slice ${{ matrix.slice }} --search-path extractor-pack --check-databases --check-undefined-labels --check-repeated-labels --check-redefined-labels --consistency-queries ql/consistency-queries ql/test --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"
CODEQL_PATH=$(gh codeql version --format=json | jq -r .unpackedLocation)
# The legacy ASP extractor is not in this repo, so take the one from the nightly build
mv "$CODEQL_PATH/csharp/tools/extractor-asp.jar" "${{ github.workspace }}/csharp/extractor-pack/tools"
# Safe guard against using the bundled extractor
rm -rf "$CODEQL_PATH/csharp"
codeql test run --threads=0 --ram 50000 --slice ${{ matrix.slice }} --search-path "${{ github.workspace }}/csharp/extractor-pack" --check-databases --check-undefined-labels --check-repeated-labels --check-redefined-labels --consistency-queries ql/consistency-queries ql/test --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"
env:
GITHUB_TOKEN: ${{ github.token }}
unit-tests:
strategy:
matrix:
os: [ubuntu-latest, windows-2019]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Setup dotnet
uses: actions/setup-dotnet@v4
with:
dotnet-version: 8.0.101
- name: Extractor unit tests
run: |
dotnet test -p:RuntimeFrameworkVersion=8.0.1 extractor/Semmle.Util.Tests
dotnet test -p:RuntimeFrameworkVersion=8.0.1 extractor/Semmle.Extraction.Tests
dotnet test -p:RuntimeFrameworkVersion=8.0.1 autobuilder/Semmle.Autobuild.CSharp.Tests
dotnet test -p:RuntimeFrameworkVersion=8.0.1 "${{ github.workspace }}/cpp/autobuilder/Semmle.Autobuild.Cpp.Tests"
shell: bash
stubgentest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./csharp/actions/create-extractor-pack
- name: Run stub generator tests
- uses: actions/checkout@v3
- name: Setup dotnet
uses: actions/setup-dotnet@v3
with:
dotnet-version: 7.0.102
- name: Extractor unit tests
run: |
# Generate (Asp)NetCore stubs
STUBS_PATH=stubs_output
python3 scripts/stubs/make_stubs_nuget.py webapp Swashbuckle.AspNetCore.Swagger 6.5.0 "$STUBS_PATH"
rm -rf ql/test/resources/stubs/_frameworks
# Update existing stubs in the repo with the freshly generated ones
mv "$STUBS_PATH/output/stubs/_frameworks" ql/test/resources/stubs/
git status
codeql test run --threads=0 --search-path extractor-pack --check-databases --check-undefined-labels --check-repeated-labels --check-redefined-labels --consistency-queries ql/consistency-queries -- ql/test/library-tests/dataflow/flowsources/aspremote
env:
GITHUB_TOKEN: ${{ github.token }}
dotnet test -p:RuntimeFrameworkVersion=7.0.2 "${{ github.workspace }}/csharp/extractor/Semmle.Util.Tests"
dotnet test -p:RuntimeFrameworkVersion=7.0.2 "${{ github.workspace }}/csharp/extractor/Semmle.Extraction.Tests"
dotnet test -p:RuntimeFrameworkVersion=7.0.2 "${{ github.workspace }}/csharp/autobuilder/Semmle.Autobuild.CSharp.Tests"
dotnet test -p:RuntimeFrameworkVersion=7.0.2 "${{ github.workspace }}/cpp/autobuilder/Semmle.Autobuild.Cpp.Tests"

View File

@@ -14,16 +14,12 @@ on:
- ".github/workflows/csv-coverage-metrics.yml"
- ".github/actions/fetch-codeql/action.yml"
permissions:
contents: read
security-events: write
jobs:
publish-java:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Setup CodeQL
uses: ./.github/actions/fetch-codeql
- name: Create empty database
@@ -51,7 +47,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Setup CodeQL
uses: ./.github/actions/fetch-codeql
- name: Create empty database

View File

@@ -10,7 +10,6 @@ on:
- "*/ql/src/**/*.qll"
- "*/ql/lib/**/*.ql"
- "*/ql/lib/**/*.qll"
- "*/ql/lib/ext/**/*.yml"
- "misc/scripts/library-coverage/*.py"
# input data files
- "*/documentation/library-coverage/cwe-sink.csv"
@@ -19,10 +18,6 @@ on:
- main
- "rc/*"
permissions:
contents: read
pull-requests: read
jobs:
generate:
name: Generate framework coverage artifacts
@@ -35,11 +30,11 @@ jobs:
GITHUB_CONTEXT: ${{ toJSON(github.event) }}
run: echo "$GITHUB_CONTEXT"
- name: Clone self (github/codeql) - MERGE
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: merge
- name: Clone self (github/codeql) - BASE
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
fetch-depth: 2
path: base
@@ -93,32 +88,9 @@ jobs:
- name: Save PR number
run: |
mkdir -p pr
echo ${PR_NUMBER} > pr/NR
env:
PR_NUMBER: ${{ github.event.pull_request.number }}
echo ${{ github.event.pull_request.number }} > pr/NR
- name: Upload PR number
uses: actions/upload-artifact@v3
with:
name: pr
path: pr/
- name: Save comment ID (if it exists)
run: |
# Find the latest comment starting with COMMENT_PREFIX
COMMENT_PREFIX=":warning: The head of this PR and the base branch were compared for differences in the framework coverage reports."
COMMENT_ID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" --paginate | jq --arg prefix "${COMMENT_PREFIX}" 'map(select(.body|startswith($prefix)) | .id) | max // empty')
if [[ -z ${COMMENT_ID} ]]
then
echo "Comment not found. Not uploading 'comment/ID' artifact."
else
mkdir -p comment
echo ${COMMENT_ID} > comment/ID
fi
env:
GITHUB_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
- name: Upload comment ID (if it exists)
uses: actions/upload-artifact@v3
with:
name: comment
path: comment/
if-no-files-found: ignore

View File

@@ -6,10 +6,6 @@ on:
types:
- completed
permissions:
contents: read
pull-requests: write
jobs:
check:
name: Check framework coverage differences and comment
@@ -24,7 +20,7 @@ jobs:
GITHUB_CONTEXT: ${{ toJSON(github.event) }}
run: echo "$GITHUB_CONTEXT"
- name: Clone self (github/codeql)
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Set up Python 3.8
uses: actions/setup-python@v4
with:

View File

@@ -3,20 +3,17 @@ name: Build framework coverage timeseries reports
on:
workflow_dispatch:
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Clone self (github/codeql)
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: script
- name: Clone self (github/codeql) for analysis
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: codeqlModels
fetch-depth: 0

View File

@@ -5,10 +5,6 @@ on:
schedule:
- cron: "0 0 * * *"
permissions:
contents: write
pull-requests: write
jobs:
update:
name: Update framework coverage report
@@ -21,7 +17,7 @@ jobs:
GITHUB_CONTEXT: ${{ toJSON(github.event) }}
run: echo "$GITHUB_CONTEXT"
- name: Clone self (github/codeql)
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: ql
fetch-depth: 0

View File

@@ -7,20 +7,17 @@ on:
description: "github/codeql repo SHA used for looking up the CSV models"
required: false
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Clone self (github/codeql)
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: script
- name: Clone self (github/codeql) for analysis
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: codeqlModels
ref: ${{ github.event.inputs.qlModelShaOverride || github.ref }}

View File

@@ -1,51 +0,0 @@
# Fast-forwards the branch specified in BRANCH_NAME
# to the github.ref/sha that this workflow is run on.
# Used as part of the release process, to ensure
# external query writers can always access a branch of github/codeql
# that is compatible with the latest stable release.
name: Fast-forward tracking branch for selected CodeQL version
on:
workflow_dispatch:
permissions:
contents: write
jobs:
fast-forward:
name: Fast-forward tracking branch for selected CodeQL version
runs-on: ubuntu-latest
if: github.repository == 'github/codeql'
env:
BRANCH_NAME: 'lgtm.com'
steps:
- name: Validate chosen branch
if: ${{ !startsWith(github.ref_name, 'codeql-cli-') }}
shell: bash
run: |
echo "::error ::The $BRANCH_NAME tracking branch should only be fast-forwarded to the tip of a codeql-cli-* branch, got $GITHUB_REF_NAME instead."
exit 1
- name: Checkout
uses: actions/checkout@v4
- name: Git config
shell: bash
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
- name: Fetch
shell: bash
run: |
set -x
echo "Fetching $BRANCH_NAME"
# Explicitly unshallow and fetch to ensure the remote ref is available.
git fetch --unshallow origin "$BRANCH_NAME"
git checkout -b "$BRANCH_NAME" "origin/$BRANCH_NAME"
- name: Fast-forward
shell: bash
run: |
echo "Fast-forwarding $BRANCH_NAME to ${GITHUB_REF}@${GITHUB_SHA}"
git merge --ff-only "$GITHUB_SHA"
git push origin "$BRANCH_NAME"

View File

@@ -7,26 +7,19 @@ on:
- .github/workflows/go-tests-other-os.yml
- .github/actions/**
- codeql-workspace.yml
env:
GO_VERSION: '~1.22.0'
permissions:
contents: read
jobs:
test-mac:
name: Test MacOS
runs-on: macos-latest
steps:
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v5
- name: Set up Go 1.20
uses: actions/setup-go@v3
with:
go-version: ${{ env.GO_VERSION }}
cache: false
go-version: 1.20.0
id: go
- name: Check out code
uses: actions/checkout@v4
uses: actions/checkout@v2
- name: Set up CodeQL CLI
uses: ./.github/actions/fetch-codeql
@@ -51,19 +44,17 @@ jobs:
make test cache="${{ steps.query-cache.outputs.cache-dir }}"
test-win:
if: github.repository_owner == 'github'
name: Test Windows
runs-on: windows-latest-xl
steps:
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v5
- name: Set up Go 1.20
uses: actions/setup-go@v3
with:
go-version: ${{ env.GO_VERSION }}
cache: false
go-version: 1.20.0
id: go
- name: Check out code
uses: actions/checkout@v4
uses: actions/checkout@v2
- name: Set up CodeQL CLI
uses: ./.github/actions/fetch-codeql

View File

@@ -15,28 +15,19 @@ on:
- .github/workflows/go-tests.yml
- .github/actions/**
- codeql-workspace.yml
env:
GO_VERSION: '~1.22.0'
permissions:
contents: read
jobs:
test-linux:
if: github.repository_owner == 'github'
name: Test Linux (Ubuntu)
runs-on: ubuntu-latest-xl
steps:
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v5
- name: Set up Go 1.20
uses: actions/setup-go@v3
with:
go-version: ${{ env.GO_VERSION }}
cache: false
go-version: 1.20.0
id: go
- name: Check out code
uses: actions/checkout@v4
uses: actions/checkout@v2
- name: Set up CodeQL CLI
uses: ./.github/actions/fetch-codeql

65
.github/workflows/js-ml-tests.yml vendored Normal file
View File

@@ -0,0 +1,65 @@
name: JS ML-powered queries tests
on:
push:
paths:
- "javascript/ql/experimental/adaptivethreatmodeling/**"
- .github/workflows/js-ml-tests.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches:
- main
- "rc/*"
pull_request:
paths:
- "javascript/ql/experimental/adaptivethreatmodeling/**"
- .github/workflows/js-ml-tests.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
workflow_dispatch:
defaults:
run:
working-directory: javascript/ql/experimental/adaptivethreatmodeling
jobs:
qltest:
name: Test QL
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- name: Install pack dependencies
run: |
for pack in modelbuilding src test; do
codeql pack install --mode verify -- "${pack}"
done
- name: Cache compilation cache
id: query-cache
uses: ./.github/actions/cache-query-compilation
with:
key: js-ml-test
- name: Check QL compilation
run: |
codeql query compile \
--check-only \
--ram 50000 \
--additional-packs "${{ github.workspace }}" \
--threads=0 \
--compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" \
-- \
lib modelbuilding src
- name: Run QL tests
run: |
codeql test run \
--threads=0 \
--ram 50000 \
--additional-packs "${{ github.workspace }}" \
--compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" \
-- \
test

View File

@@ -2,12 +2,11 @@ name: "Pull Request Labeler"
on:
- pull_request_target
permissions:
contents: read
pull-requests: write
jobs:
triage:
permissions:
contents: read
pull-requests: write
runs-on: ubuntu-latest
steps:
- uses: actions/labeler@v4

View File

@@ -12,7 +12,6 @@ on:
- main
paths:
- "java/ql/src/utils/modelgenerator/**/*.*"
- "misc/scripts/models-as-data/*.*"
- ".github/workflows/mad_modelDiff.yml"
permissions:
@@ -28,12 +27,12 @@ jobs:
slug: ${{fromJson(github.event.inputs.projects || '["apache/commons-codec", "apache/commons-io", "apache/commons-beanutils", "apache/commons-logging", "apache/commons-fileupload", "apache/commons-lang", "apache/commons-validator", "apache/commons-csv", "apache/dubbo"]' )}}
steps:
- name: Clone github/codeql from PR
uses: actions/checkout@v4
uses: actions/checkout@v3
if: github.event.pull_request
with:
path: codeql-pr
- name: Clone github/codeql from main
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: codeql-main
ref: main
@@ -62,9 +61,8 @@ jobs:
DATABASE=$2
cd codeql-$QL_VARIANT
SHORTNAME=`basename $DATABASE`
python java/ql/src/utils/modelgenerator/GenerateFlowModel.py --with-summaries --with-sinks $DATABASE $SHORTNAME/$QL_VARIANT
mkdir -p $MODELS/$SHORTNAME
mv java/ql/lib/ext/generated/$SHORTNAME/$QL_VARIANT $MODELS/$SHORTNAME
python java/ql/src/utils/modelgenerator/GenerateFlowModel.py --with-summaries --with-sinks $DATABASE ${SHORTNAME}.temp.model.yml
mv java/ql/lib/ext/generated/${SHORTNAME}.temp.model.yml $MODELS/${SHORTNAME}Generated_${QL_VARIANT}.model.yml
cd ..
}
@@ -87,16 +85,16 @@ jobs:
set -x
MODELS=`pwd`/tmp-models
ls -1 tmp-models/
for m in $MODELS/*/main/*.model.yml ; do
for m in $MODELS/*_main.model.yml ; do
t="${m/main/"pr"}"
basename=`basename $m`
name="diff_${basename/.model.yml/""}"
name="diff_${basename/_main.model.yml/""}"
(diff -w -u $m $t | diff2html -i stdin -F $MODELS/$name.html) || true
done
- uses: actions/upload-artifact@v3
with:
name: models
path: tmp-models/**/**/*.model.yml
path: tmp-models/*.model.yml
retention-days: 20
- uses: actions/upload-artifact@v3
with:

View File

@@ -11,9 +11,6 @@ on:
- ".github/workflows/mad_regenerate-models.yml"
- ".github/actions/fetch-codeql/action.yml"
permissions:
contents: read
jobs:
regenerate-models:
runs-on: ubuntu-latest
@@ -30,11 +27,11 @@ jobs:
ref: "placeholder"
steps:
- name: Clone self (github/codeql)
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Setup CodeQL binaries
uses: ./.github/actions/fetch-codeql
- name: Clone repositories
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: repos/${{ matrix.ref }}
ref: ${{ matrix.ref }}

View File

@@ -43,7 +43,7 @@ jobs:
if-no-files-found: error
retention-days: 1
- uses: actions/checkout@v4
- uses: actions/checkout@v3
with:
fetch-depth: 2
persist-credentials: false
@@ -77,7 +77,7 @@ jobs:
done < "${RUNNER_TEMP}/paths.txt" >> comment_body.txt
exit "${EXIT_CODE}"
- if: ${{ !cancelled() }}
- if: always()
uses: actions/upload-artifact@v3
with:
name: comment

View File

@@ -9,22 +9,17 @@ on:
env:
CARGO_TERM_COLOR: always
permissions:
contents: read
security-events: write
jobs:
analyze:
if: github.repository_owner == 'github'
runs-on: ubuntu-latest-xl
steps:
### Build the queries ###
- uses: actions/checkout@v4
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Find codeql
id: find-codeql
uses: github/codeql-action/init@main
uses: github/codeql-action/init@v2
with:
languages: javascript # does not matter
- uses: ./.github/actions/os-version
@@ -37,7 +32,7 @@ jobs:
path: |
ql/extractor-pack/
ql/target/release/buramu
key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-extractor-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('shared/tree-sitter-extractor') }}-${{ hashFiles('ql/**/*.rs') }}
key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-extractor-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('ql/**/*.rs') }}
- name: Cache cargo
if: steps.cache-extractor.outputs.cache-hit != 'true'
uses: actions/cache@v3
@@ -70,7 +65,7 @@ jobs:
exclude:*/ql/lib/upgrades/
exclude:java/ql/integration-tests
- name: Upload sarif to code-scanning
uses: github/codeql-action/upload-sarif@main
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: ql-for-ql.sarif
category: ql-for-ql

View File

@@ -11,10 +11,6 @@ on:
- ql/ql/src/ql.dbscheme
workflow_dispatch:
permissions:
contents: read
security-events: read
jobs:
measure:
env:
@@ -25,11 +21,11 @@ jobs:
- github/codeql
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Find codeql
id: find-codeql
uses: github/codeql-action/init@main
uses: github/codeql-action/init@v2
with:
languages: javascript # does not matter
- uses: ./.github/actions/os-version
@@ -46,7 +42,7 @@ jobs:
env:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- name: Checkout ${{ matrix.repo }}
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
repository: ${{ matrix.repo }}
path: ${{ github.workspace }}/repo
@@ -75,7 +71,7 @@ jobs:
runs-on: ubuntu-latest
needs: measure
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: measurements

View File

@@ -17,17 +17,14 @@ on:
env:
CARGO_TERM_COLOR: always
permissions:
contents: read
jobs:
qltest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Find codeql
id: find-codeql
uses: github/codeql-action/init@main
uses: github/codeql-action/init@v2
with:
languages: javascript # does not matter
- uses: ./.github/actions/os-version
@@ -64,7 +61,7 @@ jobs:
needs: [qltest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Install GNU tar
if: runner.os == 'macOS'
run: |
@@ -72,7 +69,7 @@ jobs:
echo "/usr/local/opt/gnu-tar/libexec/gnubin" >> $GITHUB_PATH
- name: Find codeql
id: find-codeql
uses: github/codeql-action/init@main
uses: github/codeql-action/init@v2
with:
languages: javascript # does not matter
- uses: ./.github/actions/os-version

View File

@@ -13,9 +13,6 @@ on:
- '.github/actions/fetch-codeql/action.yml'
- 'misc/scripts/generate-code-scanning-query-list.py'
permissions:
contents: read
jobs:
build:
@@ -23,7 +20,7 @@ jobs:
steps:
- name: Clone self (github/codeql)
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
path: codeql
- name: Set up Python 3.8

View File

@@ -32,9 +32,6 @@ defaults:
run:
working-directory: ruby
permissions:
contents: read
jobs:
build:
strategy:
@@ -45,17 +42,12 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Install GNU tar
if: runner.os == 'macOS'
run: |
brew install gnu-tar
echo "/usr/local/opt/gnu-tar/libexec/gnubin" >> $GITHUB_PATH
- name: Prepare Windows
if: runner.os == 'Windows'
shell: powershell
run: |
git config --global core.longpaths true
- uses: ./.github/actions/os-version
id: os_version
- name: Cache entire extractor
@@ -63,10 +55,12 @@ jobs:
id: cache-extractor
with:
path: |
ruby/extractor/target/release/codeql-extractor-ruby
ruby/extractor/target/release/codeql-extractor-ruby.exe
ruby/extractor/ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-ruby-extractor-${{ hashFiles('ruby/extractor/rust-toolchain.toml', 'ruby/extractor/Cargo.lock') }}-${{ hashFiles('shared/tree-sitter-extractor') }}-${{ hashFiles('ruby/extractor/**/*.rs') }}
ruby/target/release/ruby-autobuilder
ruby/target/release/ruby-autobuilder.exe
ruby/target/release/ruby-extractor
ruby/target/release/ruby-extractor.exe
ruby/ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-ruby-extractor-${{ hashFiles('ruby/rust-toolchain.toml', 'ruby/**/Cargo.lock') }}--${{ hashFiles('ruby/**/*.rs') }}
- uses: actions/cache@v3
if: steps.cache-extractor.outputs.cache-hit != 'true'
with:
@@ -74,22 +68,22 @@ jobs:
~/.cargo/registry
~/.cargo/git
ruby/target
key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-ruby-rust-cargo-${{ hashFiles('ruby/extractor/rust-toolchain.toml', 'ruby/extractor/**/Cargo.lock') }}
key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-ruby-rust-cargo-${{ hashFiles('ruby/rust-toolchain.toml', 'ruby/**/Cargo.lock') }}
- name: Check formatting
if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd extractor && cargo fmt --all -- --check
run: cargo fmt --all -- --check
- name: Build
if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd extractor && cargo build --verbose
run: cargo build --verbose
- name: Run tests
if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd extractor && cargo test --verbose
run: cargo test --verbose
- name: Release build
if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd extractor && cargo build --release
run: cargo build --release
- name: Generate dbscheme
if: ${{ matrix.os == 'ubuntu-latest' && steps.cache-extractor.outputs.cache-hit != 'true'}}
run: extractor/target/release/codeql-extractor-ruby generate --dbscheme ql/lib/ruby.dbscheme --library ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
run: target/release/ruby-generator --dbscheme ql/lib/ruby.dbscheme --library ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
- uses: actions/upload-artifact@v3
if: ${{ matrix.os == 'ubuntu-latest' }}
with:
@@ -104,46 +98,46 @@ jobs:
with:
name: extractor-${{ matrix.os }}
path: |
ruby/extractor/target/release/codeql-extractor-ruby
ruby/extractor/target/release/codeql-extractor-ruby.exe
ruby/target/release/ruby-autobuilder
ruby/target/release/ruby-autobuilder.exe
ruby/target/release/ruby-extractor
ruby/target/release/ruby-extractor.exe
retention-days: 1
compile-queries:
if: github.repository_owner == 'github'
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Fetch CodeQL
uses: ./.github/actions/fetch-codeql
- name: Cache compilation cache
id: query-cache
uses: ./.github/actions/cache-query-compilation
with:
with:
key: ruby-build
- name: Build Query Pack
run: |
PACKS=${{ runner.temp }}/query-packs
rm -rf $PACKS
codeql pack create ../misc/suite-helpers --output "$PACKS"
codeql pack create ../shared/regex --output "$PACKS"
codeql pack create ../shared/ssa --output "$PACKS"
codeql pack create ../shared/tutorial --output "$PACKS"
codeql pack create ql/lib --output "$PACKS"
codeql pack create -j0 ql/src --output "$PACKS" --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"
PACK_FOLDER=$(readlink -f "$PACKS"/codeql/ruby-queries/*)
rm -rf target/packs
codeql pack create ../misc/suite-helpers --output target/packs
codeql pack create ../shared/regex --output target/packs
codeql pack create ../shared/ssa --output target/packs
codeql pack create ../shared/tutorial --output target/packs
codeql pack create ql/lib --output target/packs
codeql pack create -j0 ql/src --output target/packs --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"
PACK_FOLDER=$(readlink -f target/packs/codeql/ruby-queries/*)
codeql generate query-help --format=sarifv2.1.0 --output="${PACK_FOLDER}/rules.sarif" ql/src
(cd ql/src; find queries \( -name '*.qhelp' -o -name '*.rb' -o -name '*.erb' \) -exec bash -c 'mkdir -p "'"${PACK_FOLDER}"'/$(dirname "{}")"' \; -exec cp "{}" "${PACK_FOLDER}/{}" \;)
- uses: actions/upload-artifact@v3
with:
name: codeql-ruby-queries
path: |
${{ runner.temp }}/query-packs/*
ruby/target/packs/*
retention-days: 1
package:
runs-on: ubuntu-latest
needs: [build, compile-queries]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: ruby.dbscheme
@@ -164,10 +158,13 @@ jobs:
mkdir -p ruby
cp -r codeql-extractor.yml tools ql/lib/ruby.dbscheme.stats ruby/
mkdir -p ruby/tools/{linux64,osx64,win64}
cp linux64/codeql-extractor-ruby ruby/tools/linux64/extractor
cp osx64/codeql-extractor-ruby ruby/tools/osx64/extractor
cp win64/codeql-extractor-ruby.exe ruby/tools/win64/extractor.exe
chmod +x ruby/tools/{linux64,osx64}/extractor
cp linux64/ruby-autobuilder ruby/tools/linux64/autobuilder
cp osx64/ruby-autobuilder ruby/tools/osx64/autobuilder
cp win64/ruby-autobuilder.exe ruby/tools/win64/autobuilder.exe
cp linux64/ruby-extractor ruby/tools/linux64/extractor
cp osx64/ruby-extractor ruby/tools/osx64/extractor
cp win64/ruby-extractor.exe ruby/tools/win64/extractor.exe
chmod +x ruby/tools/{linux64,osx64}/{autobuilder,extractor}
zip -rq codeql-ruby.zip ruby
- uses: actions/upload-artifact@v3
with:
@@ -204,7 +201,7 @@ jobs:
runs-on: ${{ matrix.os }}
needs: [package]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Fetch CodeQL
uses: ./.github/actions/fetch-codeql

View File

@@ -17,9 +17,6 @@ on:
- .github/workflows/ruby-dataset-measure.yml
workflow_dispatch:
permissions:
contents: read
jobs:
measure:
env:
@@ -30,14 +27,14 @@ jobs:
repo: [rails/rails, discourse/discourse, spree/spree, ruby/ruby]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: ./ruby/actions/create-extractor-pack
- name: Checkout ${{ matrix.repo }}
uses: actions/checkout@v4
uses: actions/checkout@v3
with:
repository: ${{ matrix.repo }}
path: ${{ github.workspace }}/repo
@@ -62,7 +59,7 @@ jobs:
runs-on: ubuntu-latest
needs: measure
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: measurements

View File

@@ -4,7 +4,6 @@ on:
push:
paths:
- "ruby/**"
- "shared/**"
- .github/workflows/ruby-build.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
@@ -14,7 +13,6 @@ on:
pull_request:
paths:
- "ruby/**"
- "shared/**"
- .github/workflows/ruby-qltest.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
@@ -29,14 +27,11 @@ defaults:
run:
working-directory: ruby
permissions:
contents: read
jobs:
qlupgrade:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- name: Check DB upgrade scripts
run: |
@@ -53,12 +48,11 @@ jobs:
xargs codeql execute upgrades testdb
diff -q testdb/ruby.dbscheme downgrades/initial/ruby.dbscheme
qltest:
if: github.repository_owner == 'github'
runs-on: ubuntu-latest-xl
strategy:
fail-fast: false
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: ./ruby/actions/create-extractor-pack
- name: Cache compilation cache

View File

@@ -16,7 +16,6 @@ on:
branches:
- main
- rc/*
- codeql-cli-*
push:
paths:
- "swift/**"
@@ -31,79 +30,62 @@ on:
branches:
- main
- rc/*
- codeql-cli-*
permissions:
contents: read
jobs:
# not using a matrix as you cannot depend on a specific job in a matrix, and we want to start linux checks
# without waiting for the macOS build
build-and-test-macos:
if: github.repository_owner == 'github'
runs-on: macos-12-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./swift/actions/build-and-test
build-and-test-linux:
if: github.repository_owner == 'github'
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./swift/actions/build-and-test
qltests-linux:
if: github.repository_owner == 'github'
needs: build-and-test-linux
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./swift/actions/run-ql-tests
qltests-macos:
if: ${{ github.repository_owner == 'github' && github.event_name == 'pull_request' }}
if : ${{ github.event_name == 'pull_request' }}
needs: build-and-test-macos
runs-on: macos-12-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./swift/actions/run-ql-tests
integration-tests-linux:
if: github.repository_owner == 'github'
needs: build-and-test-linux
runs-on: ubuntu-latest-xl
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./swift/actions/run-integration-tests
integration-tests-macos:
if: ${{ github.repository_owner == 'github' && github.event_name == 'pull_request' }}
if : ${{ github.event_name == 'pull_request' }}
needs: build-and-test-macos
runs-on: macos-12-xl
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./swift/actions/run-integration-tests
clang-format:
if : ${{ github.event_name == 'pull_request' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: pre-commit/action@646c83fcd040023954eafda54b4db0192ce70507
name: Check that python code is properly formatted
with:
extra_args: clang-format --all-files
codegen:
if : ${{ github.event_name == 'pull_request' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: bazelbuild/setup-bazelisk@v2
- uses: actions/setup-python@v4
with:
python-version-file: 'swift/.python-version'
- uses: pre-commit/action@646c83fcd040023954eafda54b4db0192ce70507
- uses: pre-commit/action@v3.0.0
name: Check that python code is properly formatted
with:
extra_args: autopep8 --all-files
- uses: ./.github/actions/fetch-codeql
- uses: pre-commit/action@646c83fcd040023954eafda54b4db0192ce70507
- uses: pre-commit/action@v3.0.0
name: Check that QL generated code was checked in
with:
extra_args: swift-codegen --all-files
@@ -118,6 +100,6 @@ jobs:
if : ${{ github.event_name == 'pull_request' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: ./swift/actions/database-upgrade-scripts

View File

@@ -10,16 +10,11 @@ on:
- main
- 'rc/*'
permissions:
contents: read
jobs:
sync:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- name: Check synchronized files
run: python config/sync-files.py
- name: Check dbscheme fragments
run: python config/sync-dbscheme-fragments.py

View File

@@ -1,49 +0,0 @@
name: Test tree-sitter-extractor
on:
push:
paths:
- "shared/tree-sitter-extractor/**"
- .github/workflows/tree-sitter-extractor-test.yml
branches:
- main
- "rc/*"
pull_request:
paths:
- "shared/tree-sitter-extractor/**"
- .github/workflows/tree-sitter-extractor-test.yml
branches:
- main
- "rc/*"
env:
CARGO_TERM_COLOR: always
defaults:
run:
working-directory: shared/tree-sitter-extractor
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check formatting
run: cargo fmt --all -- --check
- name: Run tests
run: cargo test --verbose
fmt:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check formatting
run: cargo fmt --check
clippy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run clippy
run: cargo clippy -- --no-deps -D warnings -A clippy::new_without_default -A clippy::too_many_arguments

View File

@@ -15,15 +15,12 @@ on:
- ".github/workflows/validate-change-notes.yml"
- ".github/actions/fetch-codeql/action.yml"
permissions:
contents: read
jobs:
check-change-note:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v3
- name: Setup CodeQL
uses: ./.github/actions/fetch-codeql

3
.gitignore vendored
View File

@@ -39,9 +39,6 @@
# local bazel options
/local.bazelrc
# generated cmake directory
/.bazel-cmake
# CLion project files
/.clwb

View File

@@ -5,25 +5,21 @@ repos:
rev: v3.2.0
hooks:
- id: trailing-whitespace
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)|.*\.patch
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)
- id: end-of-file-fixer
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)|.*\.patch
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v17.0.6
rev: v13.0.1
hooks:
- id: clang-format
files: ^swift/.*\.(h|c|cpp)$
- repo: https://github.com/pre-commit/mirrors-autopep8
rev: v1.6.0
hooks:
- id: autopep8
files: ^misc/codegen/.*\.py
- repo: https://github.com/warchant/pre-commit-buildifier
rev: 0.0.2
hooks:
- id: buildifier
files: ^swift/.*\.py
- repo: local
hooks:

18
.vscode/tasks.json vendored
View File

@@ -22,22 +22,6 @@
"command": "${config:python.pythonPath}",
},
"problemMatcher": []
},
{
"label": "Accept .expected changes from CI",
"type": "process",
// Non-Windows OS will usually have Python 3 already installed at /usr/bin/python3.
"command": "python3",
"args": [
"misc/scripts/accept-expected-changes-from-ci.py"
],
"group": "build",
"windows": {
// On Windows, use whatever Python interpreter is configured for this workspace. The default is
// just `python`, so if Python is already on the path, this will find it.
"command": "${config:python.pythonPath}",
},
"problemMatcher": []
}
]
}
}

View File

@@ -1,5 +1,4 @@
/cpp/ @github/codeql-c-analysis
/cpp/autobuilder/ @github/codeql-c-extractor
/csharp/ @github/codeql-csharp
/go/ @github/codeql-go
/java/ @github/codeql-java
@@ -9,8 +8,7 @@
/swift/ @github/codeql-swift
/misc/codegen/ @github/codeql-swift
/java/kotlin-extractor/ @github/codeql-kotlin
/java/ql/test-kotlin1/ @github/codeql-kotlin
/java/ql/test-kotlin2/ @github/codeql-kotlin
/java/kotlin-explorer/ @github/codeql-kotlin
# ML-powered queries
/javascript/ql/experimental/adaptivethreatmodeling/ @github/codeql-ml-powered-queries-reviewers
@@ -26,7 +24,6 @@
# Bazel (excluding BUILD.bazel files)
WORKSPACE.bazel @github/codeql-ci-reviewers
MODULE.bazel @github/codeql-ci-reviewers
.bazelversion @github/codeql-ci-reviewers
.bazelrc @github/codeql-ci-reviewers
**/*.bzl @github/codeql-ci-reviewers
@@ -43,7 +40,3 @@ MODULE.bazel @github/codeql-ci-reviewers
/.github/workflows/ql-for-ql-* @github/codeql-ql-for-ql-reviewers
/.github/workflows/ruby-* @github/codeql-ruby
/.github/workflows/swift.yml @github/codeql-swift
# Misc
/misc/scripts/accept-expected-changes-from-ci.py @RasmusWL
/misc/scripts/generate-code-scanning-query-list.py @RasmusWL

View File

@@ -14,16 +14,14 @@ If you have an idea for a query that you would like to share with other CodeQL u
1. **Directory structure**
There are eight language-specific query directories in this repository:
There are six language-specific query directories in this repository:
* C/C++: `cpp/ql/src`
* C#: `csharp/ql/src`
* Go: `go/ql/src`
* Java/Kotlin: `java/ql/src`
* Java: `java/ql/src`
* JavaScript: `javascript/ql/src`
* Python: `python/ql/src`
* Ruby: `ruby/ql/src`
* Swift: `swift/ql/src`
Each language-specific directory contains further subdirectories that group queries based on their `@tags` or purpose.
- Experimental queries and libraries are stored in the `experimental` subdirectory within each language-specific directory in the [CodeQL repository](https://github.com/github/codeql). For example, experimental Java queries and libraries are stored in `java/ql/src/experimental` and any corresponding tests in `java/ql/test/experimental`.

View File

@@ -1,53 +0,0 @@
module(
name = "codeql",
version = "0.0",
)
# this points to our internal repository when `codeql` is checked out as a submodule thereof
# when building things from `codeql` independently this is stubbed out in `.bazelrc`
bazel_dep(name = "semmle_code", version = "0.0")
local_path_override(
module_name = "semmle_code",
path = "..",
)
# see https://registry.bazel.build/ for a list of available packages
bazel_dep(name = "platforms", version = "0.0.8")
bazel_dep(name = "rules_pkg", version = "0.9.1")
bazel_dep(name = "rules_nodejs", version = "6.0.3")
bazel_dep(name = "rules_python", version = "0.31.0")
bazel_dep(name = "bazel_skylib", version = "1.5.0")
bazel_dep(name = "abseil-cpp", version = "20240116.0", repo_name = "absl")
bazel_dep(name = "nlohmann_json", version = "3.11.3", repo_name = "json")
bazel_dep(name = "fmt", version = "10.0.0")
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
hub_name = "codegen_deps",
python_version = "3.11",
requirements_lock = "//misc/codegen:requirements_lock.txt",
)
use_repo(pip, "codegen_deps")
swift_deps = use_extension("//swift/third_party:load.bzl", "swift_deps")
use_repo(
swift_deps,
"binlog",
"picosha2",
"swift_prebuilt_darwin_x86_64",
"swift_prebuilt_linux",
"swift_toolchain_linux",
"swift_toolchain_macos",
)
node = use_extension("@rules_nodejs//nodejs:extensions.bzl", "node")
node.toolchain(
name = "nodejs",
node_version = "18.15.0",
)
use_repo(node, "nodejs", "nodejs_toolchains")
register_toolchains(
"@nodejs_toolchains//:all",
)

View File

@@ -1,2 +1,12 @@
# please use MODULE.bazel to add dependencies
# this empty file is required by internal repositories, don't remove it
# Please notice that any bazel targets and definitions in this repository are currently experimental
# and for internal use only.
workspace(name = "codeql")
load("//misc/bazel:workspace.bzl", "codeql_workspace")
codeql_workspace()
load("//misc/bazel:workspace_deps.bzl", "codeql_workspace_deps")
codeql_workspace_deps()

View File

@@ -1,26 +1,23 @@
provide:
- "*/ql/src/qlpack.yml"
- "*/ql/lib/qlpack.yml"
- "*/ql/test*/qlpack.yml"
- "*/ql/test/qlpack.yml"
- "*/ql/examples/qlpack.yml"
- "*/ql/consistency-queries/qlpack.yml"
- "*/ql/automodel/src/qlpack.yml"
- "*/ql/automodel/test/qlpack.yml"
- "python/extractor/qlpack.yml"
- "shared/**/qlpack.yml"
- "shared/*/qlpack.yml"
- "cpp/ql/test/query-tests/Security/CWE/CWE-190/semmle/tainted/qlpack.yml"
- "go/ql/config/legacy-support/qlpack.yml"
- "go/build/codeql-extractor-go/codeql-extractor.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/lib/qlpack.yml"
- "*/ql/experimental/adaptivethreatmodeling/lib/qlpack.yml"
# This pack is explicitly excluded from the workspace since most users
# will want to use a version of this pack from the package cache. Internal
# users can uncomment the following line and place a custom ML model
# in the corresponding pack to test a custom ML model within their local
# checkout.
# - "javascript/ql/experimental/adaptivethreatmodeling/model/qlpack.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/modelbuilding/qlpack.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/src/qlpack.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/test/qlpack.yml"
# - "*/ql/experimental/adaptivethreatmodeling/model/qlpack.yml"
- "*/ql/experimental/adaptivethreatmodeling/modelbuilding/qlpack.yml"
- "*/ql/experimental/adaptivethreatmodeling/src/qlpack.yml"
- "*/ql/experimental/adaptivethreatmodeling/test/qlpack.yml"
- "csharp/ql/campaigns/Solorigate/lib/qlpack.yml"
- "csharp/ql/campaigns/Solorigate/src/qlpack.yml"
- "csharp/ql/campaigns/Solorigate/test/qlpack.yml"
@@ -28,8 +25,8 @@ provide:
- "misc/suite-helpers/qlpack.yml"
- "ruby/extractor-pack/codeql-extractor.yml"
- "swift/extractor-pack/codeql-extractor.yml"
- "swift/integration-tests/qlpack.yml"
- "ql/extractor-pack/codeql-extractor.yml"
- ".github/codeql/extensions/**/codeql-pack.yml"
versionPolicies:
default:

View File

@@ -1,33 +0,0 @@
{
"files": [
"javascript/ql/lib/semmlecode.javascript.dbscheme",
"python/ql/lib/semmlecode.python.dbscheme",
"ruby/ql/lib/ruby.dbscheme",
"ql/ql/src/ql.dbscheme"
],
"fragments": [
"/*- External data -*/",
"/*- Files and folders -*/",
"/*- Diagnostic messages -*/",
"/*- Diagnostic messages: severity -*/",
"/*- Source location prefix -*/",
"/*- Lines of code -*/",
"/*- Configuration files with key value pairs -*/",
"/*- YAML -*/",
"/*- XML Files -*/",
"/*- XML: sourceline -*/",
"/*- DEPRECATED: External defects and metrics -*/",
"/*- DEPRECATED: Snapshot date -*/",
"/*- DEPRECATED: Duplicate code -*/",
"/*- DEPRECATED: Version control data -*/",
"/*- JavaScript-specific part -*/",
"/*- Ruby dbscheme -*/",
"/*- Erb dbscheme -*/",
"/*- QL dbscheme -*/",
"/*- Dbscheme dbscheme -*/",
"/*- Yaml dbscheme -*/",
"/*- Blame dbscheme -*/",
"/*- JSON dbscheme -*/",
"/*- Python dbscheme -*/"
]
}

View File

@@ -1,4 +1,26 @@
{
"DataFlow Java/C++/C#/Go/Python/Ruby/Swift": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlow.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlow.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlow.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlow.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlow.qll",
"go/ql/lib/semmle/go/dataflow/internal/DataFlow.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlow.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlow.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlow.qll"
],
"DataFlowImpl Java/C++/C#/Go/Python/Ruby/Swift": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlowImpl.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl.qll",
"go/ql/lib/semmle/go/dataflow/internal/DataFlowImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImpl.qll"
],
"DataFlow Java/C++/C#/Go/Python/Ruby/Swift Legacy Configuration": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl1.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl2.qll",
@@ -6,6 +28,8 @@
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl4.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl5.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImpl6.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplForSerializability.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplForOnActivityResult.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlowImpl1.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlowImpl2.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlowImpl3.qll",
@@ -15,27 +39,61 @@
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl2.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl3.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl4.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl1.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl2.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl3.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImpl4.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl1.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl2.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl3.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl4.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl5.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplForContentDataFlow.qll",
"go/ql/lib/semmle/go/dataflow/internal/DataFlowImpl1.qll",
"go/ql/lib/semmle/go/dataflow/internal/DataFlowImpl2.qll",
"go/ql/lib/semmle/go/dataflow/internal/DataFlowImplForStringsNewReplacer.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl1.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl2.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl3.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl4.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplForRegExp.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl1.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl2.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplForHttpClientLibraries.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplForPathname.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImpl1.qll"
],
"DataFlow Java/C++/C#/Go/Python/Ruby/Swift Common": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplCommon.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlowImplCommon.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImplCommon.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImplCommon.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplCommon.qll",
"go/ql/lib/semmle/go/dataflow/internal/DataFlowImplCommon.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplCommon.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImplCommon.qll"
],
"TaintTracking Java/C++/C#/Go/Python/Ruby/Swift": [
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/tainttracking1/TaintTracking.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/tainttracking1/TaintTracking.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/tainttracking1/TaintTracking.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/tainttracking1/TaintTracking.qll",
"go/ql/lib/semmle/go/dataflow/internal/tainttracking1/TaintTracking.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/tainttracking1/TaintTracking.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/tainttracking1/TaintTracking.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/tainttracking1/TaintTracking.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/tainttracking1/TaintTracking.qll"
],
"TaintTracking Legacy Configuration Java/C++/C#/Go/Python/Ruby/Swift": [
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/tainttracking3/TaintTrackingImpl.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/tainttracking3/TaintTrackingImpl.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/tainttracking3/TaintTrackingImpl.qll",
@@ -53,6 +111,24 @@
"ruby/ql/lib/codeql/ruby/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/tainttracking1/TaintTrackingImpl.qll"
],
"DataFlow Java/C++/C#/Python/Ruby/Swift Consistency checks": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplConsistency.qll",
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/DataFlowImplConsistency.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImplConsistency.qll",
"cpp/ql/lib/experimental/semmle/code/cpp/ir/dataflow/internal/DataFlowImplConsistency.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplConsistency.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplConsistency.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplConsistency.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImplConsistency.qll"
],
"DataFlow Java/C#/Go/Ruby/Python/Swift Flow Summaries": [
"java/ql/lib/semmle/code/java/dataflow/internal/FlowSummaryImpl.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/FlowSummaryImpl.qll",
"go/ql/lib/semmle/go/dataflow/internal/FlowSummaryImpl.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/FlowSummaryImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/FlowSummaryImpl.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/FlowSummaryImpl.qll"
],
"SsaReadPosition Java/C#": [
"java/ql/lib/semmle/code/java/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll"
@@ -61,10 +137,6 @@
"java/ql/src/utils/modelgenerator/internal/CaptureModels.qll",
"csharp/ql/src/utils/modelgenerator/internal/CaptureModels.qll"
],
"Model as Data Generation Java/C# - CaptureModelsPrinting": [
"java/ql/src/utils/modelgenerator/internal/CaptureModelsPrinting.qll",
"csharp/ql/src/utils/modelgenerator/internal/CaptureModelsPrinting.qll"
],
"Sign Java/C#": [
"java/ql/lib/semmle/code/java/dataflow/internal/rangeanalysis/Sign.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/rangeanalysis/Sign.qll"
@@ -88,46 +160,123 @@
"IR Instruction": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/Instruction.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/Instruction.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/Instruction.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/Instruction.qll",
"csharp/ql/src/experimental/ir/implementation/raw/Instruction.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Instruction.qll"
],
"IR IRBlock": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/IRBlock.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/IRBlock.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRBlock.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRBlock.qll",
"csharp/ql/src/experimental/ir/implementation/raw/IRBlock.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/IRBlock.qll"
],
"IR IRVariable": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/IRVariable.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/IRVariable.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRVariable.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRVariable.qll",
"csharp/ql/src/experimental/ir/implementation/raw/IRVariable.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/IRVariable.qll"
],
"IR IRFunction": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/IRFunction.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/IRFunction.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRFunction.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRFunction.qll",
"csharp/ql/src/experimental/ir/implementation/raw/IRFunction.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/IRFunction.qll"
],
"IR Operand": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/Operand.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/Operand.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/Operand.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/Operand.qll",
"csharp/ql/src/experimental/ir/implementation/raw/Operand.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/Operand.qll"
],
"IR IRType": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/IRType.qll",
"csharp/ql/src/experimental/ir/implementation/IRType.qll"
],
"IR IRConfiguration": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/IRConfiguration.qll",
"csharp/ql/src/experimental/ir/implementation/IRConfiguration.qll"
],
"IR UseSoundEscapeAnalysis": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/UseSoundEscapeAnalysis.qll",
"csharp/ql/src/experimental/ir/implementation/UseSoundEscapeAnalysis.qll"
],
"IR IRFunctionBase": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/internal/IRFunctionBase.qll",
"csharp/ql/src/experimental/ir/implementation/internal/IRFunctionBase.qll"
],
"IR Operand Tag": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/internal/OperandTag.qll",
"csharp/ql/src/experimental/ir/implementation/internal/OperandTag.qll"
],
"IR TInstruction": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/internal/TInstruction.qll",
"csharp/ql/src/experimental/ir/implementation/internal/TInstruction.qll"
],
"IR TIRVariable": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/internal/TIRVariable.qll",
"csharp/ql/src/experimental/ir/implementation/internal/TIRVariable.qll"
],
"IR IR": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/IR.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/IR.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IR.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IR.qll",
"csharp/ql/src/experimental/ir/implementation/raw/IR.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/IR.qll"
],
"IR IRConsistency": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/IRConsistency.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/IRConsistency.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRConsistency.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/IRConsistency.qll",
"csharp/ql/src/experimental/ir/implementation/raw/IRConsistency.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/IRConsistency.qll"
],
"IR PrintIR": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/PrintIR.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/PrintIR.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/PrintIR.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/PrintIR.qll",
"csharp/ql/src/experimental/ir/implementation/raw/PrintIR.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/PrintIR.qll"
],
"IR IntegerConstant": [
"cpp/ql/lib/semmle/code/cpp/ir/internal/IntegerConstant.qll",
"csharp/ql/src/experimental/ir/internal/IntegerConstant.qll"
],
"IR IntegerInteval": [
"cpp/ql/lib/semmle/code/cpp/ir/internal/IntegerInterval.qll",
"csharp/ql/src/experimental/ir/internal/IntegerInterval.qll"
],
"IR IntegerPartial": [
"cpp/ql/lib/semmle/code/cpp/ir/internal/IntegerPartial.qll",
"csharp/ql/src/experimental/ir/internal/IntegerPartial.qll"
],
"IR Overlap": [
"cpp/ql/lib/semmle/code/cpp/ir/internal/Overlap.qll",
"csharp/ql/src/experimental/ir/internal/Overlap.qll"
],
"IR EdgeKind": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/EdgeKind.qll",
"csharp/ql/src/experimental/ir/implementation/EdgeKind.qll"
],
"IR MemoryAccessKind": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/MemoryAccessKind.qll",
"csharp/ql/src/experimental/ir/implementation/MemoryAccessKind.qll"
],
"IR TempVariableTag": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/TempVariableTag.qll",
"csharp/ql/src/experimental/ir/implementation/TempVariableTag.qll"
],
"IR Opcode": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/Opcode.qll",
"csharp/ql/src/experimental/ir/implementation/Opcode.qll"
],
"IR SSAConsistency": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SSAConsistency.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConsistency.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConsistency.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/SSAConsistency.qll"
],
"C++ IR InstructionImports": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/InstructionImports.qll",
@@ -144,11 +293,6 @@
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/IRBlockImports.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/IRBlockImports.qll"
],
"C++ IR IRConsistencyImports": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/IRConsistencyImports.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/IRConsistencyImports.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/IRConsistencyImports.qll"
],
"C++ IR IRFunctionImports": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/IRFunctionImports.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/IRFunctionImports.qll",
@@ -175,7 +319,8 @@
],
"SSA AliasAnalysis": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/AliasAnalysis.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/AliasAnalysis.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/AliasAnalysis.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/AliasAnalysis.qll"
],
"SSA PrintAliasAnalysis": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/PrintAliasAnalysis.qll",
@@ -190,28 +335,44 @@
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/internal/ValueNumberingImports.qll"
],
"IR SSA SimpleSSA": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/SimpleSSA.qll"
],
"IR AliasConfiguration (unaliased_ssa)": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/AliasConfiguration.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/AliasConfiguration.qll"
],
"IR SSA SSAConstruction": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/SSAConstruction.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConstruction.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/SSAConstruction.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/SSAConstruction.qll"
],
"IR SSA PrintSSA": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/PrintSSA.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/PrintSSA.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/internal/PrintSSA.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/PrintSSA.qll"
],
"IR ValueNumberInternal": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/gvn/internal/ValueNumberingInternal.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingInternal.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/internal/ValueNumberingInternal.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/internal/ValueNumberingInternal.qll",
"csharp/ql/src/experimental/ir/implementation/raw/gvn/internal/ValueNumberingInternal.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingInternal.qll"
],
"C++ IR ValueNumber": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/gvn/ValueNumbering.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/gvn/ValueNumbering.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/ValueNumbering.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/ValueNumbering.qll",
"csharp/ql/src/experimental/ir/implementation/raw/gvn/ValueNumbering.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/ValueNumbering.qll"
],
"C++ IR PrintValueNumbering": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/gvn/PrintValueNumbering.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/gvn/PrintValueNumbering.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/PrintValueNumbering.qll"
"cpp/ql/lib/semmle/code/cpp/ir/implementation/aliased_ssa/gvn/PrintValueNumbering.qll",
"csharp/ql/src/experimental/ir/implementation/raw/gvn/PrintValueNumbering.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/PrintValueNumbering.qll"
],
"C++ IR ConstantAnalysis": [
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/constant/ConstantAnalysis.qll",
@@ -239,6 +400,38 @@
"cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/reachability/PrintDominance.qll",
"cpp/ql/lib/semmle/code/cpp/ir/implementation/unaliased_ssa/internal/reachability/PrintDominance.qll"
],
"C# IR InstructionImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/InstructionImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/InstructionImports.qll"
],
"C# IR IRImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/IRImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/IRImports.qll"
],
"C# IR IRBlockImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/IRBlockImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/IRBlockImports.qll"
],
"C# IR IRFunctionImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/IRFunctionImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/IRFunctionImports.qll"
],
"C# IR IRVariableImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/IRVariableImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/IRVariableImports.qll"
],
"C# IR OperandImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/OperandImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/OperandImports.qll"
],
"C# IR PrintIRImports": [
"csharp/ql/src/experimental/ir/implementation/raw/internal/PrintIRImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/internal/PrintIRImports.qll"
],
"C# IR ValueNumberingImports": [
"csharp/ql/src/experimental/ir/implementation/raw/gvn/internal/ValueNumberingImports.qll",
"csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll"
],
"C# ControlFlowReachability": [
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/ControlFlowReachability.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/rangeanalysis/ControlFlowReachability.qll"
@@ -255,6 +448,7 @@
"cpp/ql/lib/semmle/code/cpp/XML.qll",
"csharp/ql/lib/semmle/code/csharp/XML.qll",
"java/ql/lib/semmle/code/xml/XML.qll",
"javascript/ql/lib/semmle/javascript/XML.qll",
"python/ql/lib/semmle/python/xml/XML.qll"
],
"DuplicationProblems.inc.qhelp": [
@@ -304,6 +498,13 @@
"java/ql/src/experimental/Security/CWE/CWE-400/LocalThreadResourceAbuse.qhelp",
"java/ql/src/experimental/Security/CWE/CWE-400/ThreadResourceAbuse.qhelp"
],
"IDE Contextual Queries": [
"cpp/ql/lib/IDEContextual.qll",
"csharp/ql/lib/IDEContextual.qll",
"java/ql/lib/IDEContextual.qll",
"javascript/ql/lib/IDEContextual.qll",
"python/ql/lib/analysis/IDEContextual.qll"
],
"CryptoAlgorithms Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/CryptoAlgorithms.qll",
"python/ql/lib/semmle/python/concepts/CryptoAlgorithms.qll",
@@ -317,8 +518,25 @@
"SensitiveDataHeuristics Python/JS": [
"javascript/ql/lib/semmle/javascript/security/internal/SensitiveDataHeuristics.qll",
"python/ql/lib/semmle/python/security/internal/SensitiveDataHeuristics.qll",
"ruby/ql/lib/codeql/ruby/security/internal/SensitiveDataHeuristics.qll",
"swift/ql/lib/codeql/swift/security/internal/SensitiveDataHeuristics.qll"
"ruby/ql/lib/codeql/ruby/security/internal/SensitiveDataHeuristics.qll"
],
"CFG": [
"csharp/ql/lib/semmle/code/csharp/controlflow/internal/ControlFlowGraphImplShared.qll",
"ruby/ql/lib/codeql/ruby/controlflow/internal/ControlFlowGraphImplShared.qll",
"swift/ql/lib/codeql/swift/controlflow/internal/ControlFlowGraphImplShared.qll"
],
"TypeTracker": [
"python/ql/lib/semmle/python/dataflow/new/internal/TypeTracker.qll",
"ruby/ql/lib/codeql/ruby/typetracking/TypeTracker.qll"
],
"AccessPathSyntax": [
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/AccessPathSyntax.qll",
"go/ql/lib/semmle/go/dataflow/internal/AccessPathSyntax.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/AccessPathSyntax.qll",
"javascript/ql/lib/semmle/javascript/frameworks/data/internal/AccessPathSyntax.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/AccessPathSyntax.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/AccessPathSyntax.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/AccessPathSyntax.qll"
],
"IncompleteUrlSubstringSanitization": [
"javascript/ql/src/Security/CWE-020/IncompleteUrlSubstringSanitization.qll",
@@ -339,6 +557,26 @@
"ruby/ql/lib/codeql/ruby/frameworks/data/internal/ApiGraphModelsExtensions.qll",
"python/ql/lib/semmle/python/frameworks/data/internal/ApiGraphModelsExtensions.qll"
],
"TaintedFormatStringQuery Ruby/JS": [
"javascript/ql/lib/semmle/javascript/security/dataflow/TaintedFormatStringQuery.qll",
"ruby/ql/lib/codeql/ruby/security/TaintedFormatStringQuery.qll"
],
"TaintedFormatStringCustomizations Ruby/JS": [
"javascript/ql/lib/semmle/javascript/security/dataflow/TaintedFormatStringCustomizations.qll",
"ruby/ql/lib/codeql/ruby/security/TaintedFormatStringCustomizations.qll"
],
"HttpToFileAccessQuery JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/dataflow/HttpToFileAccessQuery.qll",
"ruby/ql/lib/codeql/ruby/security/HttpToFileAccessQuery.qll"
],
"HttpToFileAccessCustomizations JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/dataflow/HttpToFileAccessCustomizations.qll",
"ruby/ql/lib/codeql/ruby/security/HttpToFileAccessCustomizations.qll"
],
"Typo database": [
"javascript/ql/src/Expressions/TypoDatabase.qll",
"ql/ql/src/codeql_ql/style/TypoDatabase.qll"
],
"Swift declarations test file": [
"swift/ql/test/extractor-tests/declarations/declarations.swift",
"swift/ql/test/library-tests/ast/declarations.swift"
@@ -366,9 +604,5 @@
"EncryptionKeySizes Python/Java": [
"python/ql/lib/semmle/python/security/internal/EncryptionKeySizes.qll",
"java/ql/lib/semmle/code/java/security/internal/EncryptionKeySizes.qll"
],
"Python model summaries test extension": [
"python/ql/test/experimental/dataflow/model-summaries/InlineTaintTest.ext.yml",
"python/ql/test/experimental/dataflow/model-summaries/NormalDataflowTest.ext.yml"
]
}
}

View File

@@ -1,86 +0,0 @@
#!/usr/bin/env python3
import argparse
import json
import os
import pathlib
import re
def make_groups(blocks):
groups = {}
for block in blocks:
groups.setdefault("".join(block["lines"]), []).append(block)
return list(groups.values())
def validate_fragments(fragments):
ok = True
for header, blocks in fragments.items():
groups = make_groups(blocks)
if len(groups) > 1:
ok = False
print("Warning: dbscheme fragments with header '{}' are different for {}".format(header, ["{}:{}:{}".format(
group[0]["file"], group[0]["start"], group[0]["end"]) for group in groups]))
return ok
def main():
script_path = os.path.realpath(__file__)
script_dir = os.path.dirname(script_path)
parser = argparse.ArgumentParser(
prog=os.path.basename(script_path),
description='Sync dbscheme fragments across files.'
)
parser.add_argument('files', metavar='dbscheme_file', type=pathlib.Path, nargs='*', default=[],
help='dbscheme files to check')
args = parser.parse_args()
with open(os.path.join(script_dir, "dbscheme-fragments.json"), "r") as f:
config = json.load(f)
fragment_headers = set(config["fragments"])
fragments = {}
ok = True
for file in args.files + config["files"]:
with open(os.path.join(os.path.dirname(script_dir), file), "r") as dbscheme:
header = None
line_number = 1
block = {"file": file, "start": line_number,
"end": None, "lines": []}
def end_block():
block["end"] = line_number - 1
if len(block["lines"]) > 0:
if header is None:
if re.match(r'(?m)\A(\s|//.*$|/\*(\**[^\*])*\*+/)*\Z', "".join(block["lines"])):
# Ignore comments at the beginning of the file
pass
else:
ok = False
print("Warning: dbscheme fragment without header: {}:{}:{}".format(
block["file"], block["start"], block["end"]))
else:
fragments.setdefault(header, []).append(block)
for line in dbscheme:
m = re.match(r"^\/\*-.*-\*\/$", line)
if m:
end_block()
header = line.strip()
if header not in fragment_headers:
ok = False
print("Warning: unknown header for dbscheme fragment: '{}': {}:{}".format(
header, file, line_number))
block = {"file": file, "start": line_number,
"end": None, "lines": []}
block["lines"].append(line)
line_number += 1
block["lines"].append('\n')
line_number += 1
end_block()
if not ok or not validate_fragments(fragments):
exit(1)
if __name__ == "__main__":
main()

View File

@@ -1,17 +1,12 @@
load("@rules_pkg//:mappings.bzl", "pkg_filegroup")
package(default_visibility = ["//visibility:public"])
load("@rules_pkg//:mappings.bzl", "pkg_filegroup")
alias(
name = "dbscheme",
actual = "//cpp/ql/lib:dbscheme",
)
alias(
name = "dbscheme-stats",
actual = "//cpp/ql/lib:dbscheme-stats",
)
pkg_filegroup(
name = "db-files",
srcs = [

View File

@@ -145,9 +145,9 @@ namespace Semmle.Autobuild.Cpp.Tests
bool IBuildActions.IsMacOs() => IsMacOs;
public bool IsRunningOnAppleSilicon { get; set; }
public bool IsArm { get; set; }
bool IBuildActions.IsRunningOnAppleSilicon() => IsRunningOnAppleSilicon;
bool IBuildActions.IsArm() => IsArm;
string IBuildActions.PathCombine(params string[] parts)
{
@@ -326,8 +326,8 @@ namespace Semmle.Autobuild.Cpp.Tests
public void TestCppAutobuilderSuccess()
{
Actions.RunProcess[@"cmd.exe /C nuget restore C:\Project\test.sln -DisableParallelProcessing"] = 1;
Actions.RunProcess[@"cmd.exe /C scratch\.nuget\nuget.exe restore C:\Project\test.sln -DisableParallelProcessing"] = 0;
Actions.RunProcess[@"cmd.exe /C CALL ^""C:\Program^ Files^ ^(x86^)\Microsoft^ Visual^ Studio^ 14.0\VC\vcvarsall.bat^"" && set Platform=&& type NUL && msbuild C:\Project\test.sln /t:rebuild /p:Platform=""x86"" /p:Configuration=""Release"""] = 0;
Actions.RunProcess[@"cmd.exe /C C:\Project\.nuget\nuget.exe restore C:\Project\test.sln -DisableParallelProcessing"] = 0;
Actions.RunProcess[@"cmd.exe /C CALL ^""C:\Program Files ^(x86^)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat^"" && set Platform=&& type NUL && msbuild C:\Project\test.sln /t:rebuild /p:Platform=""x86"" /p:Configuration=""Release"""] = 0;
Actions.RunProcessOut[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationPath"] = "";
Actions.RunProcess[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationPath"] = 1;
Actions.RunProcess[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -prerelease -legacy -property installationVersion"] = 0;
@@ -337,11 +337,10 @@ namespace Semmle.Autobuild.Cpp.Tests
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"] = true;
Actions.FileExists[@"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"] = true;
Actions.GetEnvironmentVariable["CODEQL_EXTRACTOR_CPP_SCRATCH_DIR"] = "scratch";
Actions.EnumerateFiles[@"C:\Project"] = "foo.cs\ntest.slx";
Actions.EnumerateDirectories[@"C:\Project"] = "";
Actions.CreateDirectories.Add(@"scratch\.nuget");
Actions.DownloadFiles.Add(("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", @"scratch\.nuget\nuget.exe"));
Actions.CreateDirectories.Add(@"C:\Project\.nuget");
Actions.DownloadFiles.Add(("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", @"C:\Project\.nuget\nuget.exe"));
var autobuilder = CreateAutoBuilder(true);
var solution = new TestSolution(@"C:\Project\test.sln");

View File

@@ -2,7 +2,7 @@
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<TargetFramework>net7.0</TargetFramework>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
<RuntimeIdentifiers>win-x64;linux-x64;osx-x64</RuntimeIdentifiers>
<Nullable>enable</Nullable>
@@ -11,12 +11,12 @@
<ItemGroup>
<PackageReference Include="System.IO.FileSystem" Version="4.3.0" />
<PackageReference Include="System.IO.FileSystem.Primitives" Version="4.3.0" />
<PackageReference Include="xunit" Version="2.6.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.4">
<PackageReference Include="xunit" Version="2.4.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
</PackageReference>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.4.0" />
</ItemGroup>
<ItemGroup>

View File

@@ -1,7 +1,5 @@
using System;
using Semmle.Autobuild.Shared;
using Semmle.Util;
namespace Semmle.Autobuild.Cpp
{

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<TargetFramework>net7.0</TargetFramework>
<AssemblyName>Semmle.Autobuild.Cpp</AssemblyName>
<RootNamespace>Semmle.Autobuild.Cpp</RootNamespace>
<ApplicationIcon />
@@ -17,7 +17,7 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Build" Version="17.8.3" />
<PackageReference Include="Microsoft.Build" Version="17.3.2" />
</ItemGroup>
<ItemGroup>

View File

@@ -1,3 +0,0 @@
description: Expose whether a function was prototyped or not
compatibility: backwards
function_prototyped.rel: delete

View File

@@ -1,4 +0,0 @@
description: Revert support for repeated initializers, which are allowed in C with designated initializers.
compatibility: full
aggregate_field_init.rel: reorder aggregate_field_init.rel (int aggregate, int initializer, int field, int position) aggregate initializer field
aggregate_array_init.rel: reorder aggregate_array_init.rel (int aggregate, int initializer, int element_index, int position) aggregate initializer element_index

View File

@@ -1,19 +0,0 @@
class Element extends @element {
string toString() { none() }
}
class Expr extends @expr {
string toString() { none() }
}
class Stmt extends @stmt {
string toString() { none() }
}
predicate isStmtWithInitializer(Stmt stmt) { exists(int kind | stmts(stmt, kind, _) | kind = 29) }
from Expr child, int index, int index_new, Element parent
where
exprparents(child, index, parent) and
if isStmtWithInitializer(parent) then index_new = index - 1 else index_new = index
select child, index_new, parent

View File

@@ -1,9 +0,0 @@
class Stmt extends @stmt {
string toString() { none() }
}
from Stmt f, Stmt i
where
for_initialization(f, i) and
f instanceof @stmt_for
select f, i

View File

@@ -1,20 +0,0 @@
class Element extends @element {
string toString() { none() }
}
class Stmt extends @stmt {
string toString() { none() }
}
predicate isStmtWithInitializer(Stmt stmt) { exists(int kind | stmts(stmt, kind, _) | kind = 29) }
from Stmt child, int index, int index_new, Element parent
where
stmtparents(child, index, parent) and
(
not isStmtWithInitializer(parent)
or
index > 0
) and
if isStmtWithInitializer(parent) then index_new = index - 1 else index_new = index
select child, index_new, parent

View File

@@ -1,5 +0,0 @@
description: Support C++20 range-based for initializers
compatibility: partial
exprparents.rel: run exprparents.qlo
stmtparents.rel: run stmtparents.qlo
for_initialization.rel: run for_initialization.qlo

View File

@@ -1,11 +0,0 @@
class Declaration extends @declaration {
string toString() { none() }
}
class MangledName extends @mangledname {
string toString() { none() }
}
from Declaration d, MangledName n
where mangled_name(d, n, _)
select d, n

View File

@@ -1,3 +0,0 @@
description: Add completness information to mangled name table
compatibility: full
mangled_name.rel: run mangled_name.qlo

View File

@@ -1,19 +0,0 @@
class BuiltinType extends @builtintype {
string toString() { none() }
}
from BuiltinType type, string name, int kind, int kind_new, int size, int sign, int alignment
where
builtintypes(type, name, kind, size, sign, alignment) and
if
type instanceof @fp16 or
type instanceof @std_bfloat16 or
type instanceof @std_float16 or
type instanceof @complex_std_float32 or
type instanceof @complex_float32x or
type instanceof @complex_std_float64 or
type instanceof @complex_float64x or
type instanceof @complex_std_float128
then kind_new = 2
else kind_new = kind
select type, name, kind_new, size, sign, alignment

View File

@@ -1,3 +0,0 @@
description: Introduce new floating-point types from C23 and C++23
compatibility: backwards
builtintypes.rel: run builtintypes.qlo

View File

@@ -1,9 +0,0 @@
class Function extends @function {
string toString() { none() }
}
from Function fun, string name, int kind, int kind_new
where
functions(fun, name, kind) and
if kind = 7 or kind = 8 then kind_new = 0 else kind_new = kind
select fun, name, kind_new

View File

@@ -1,3 +0,0 @@
description: Support more function types
compatibility: full
functions.rel: run functions.qlo

View File

@@ -1,2 +0,0 @@
description: Removed @assignpaddexpr and @assignpsubexpr from @assign_bitwise_expr
compatibility: full

View File

@@ -1,13 +0,0 @@
class Expr extends @expr {
string toString() { none() }
}
class Location extends @location_expr {
string toString() { none() }
}
from Expr expr, int kind, int kind_new, Location loc
where
exprs(expr, kind, loc) and
if kind = 363 then kind_new = 1 else kind_new = kind
select expr, kind_new, loc

View File

@@ -1,4 +0,0 @@
description: Introduce re-use expressions
compatibility: partial
expr_reuse.rel: delete
exprs.rel: run exprs.qlo

View File

@@ -1,7 +0,0 @@
class Expr extends @expr {
string toString() { none() }
}
from Expr reuse, Expr original
where expr_reuse(reuse, original, _)
select reuse, original

View File

@@ -1,22 +0,0 @@
class Expr extends @expr {
string toString() { none() }
}
class Type extends @type {
string toString() { none() }
}
predicate existingType(Expr expr, Type type, int value_category) {
expr_types(expr, type, value_category)
}
predicate reuseType(Expr reuse, Type type, int value_category) {
exists(Expr original |
expr_reuse(reuse, original, value_category) and
expr_types(original, type, _)
)
}
from Expr expr, Type type, int value_category
where existingType(expr, type, value_category) or reuseType(expr, type, value_category)
select expr, type, value_category

View File

@@ -1,4 +0,0 @@
description: Add value category to expr_reuse table
compatibility: full
expr_reuse.rel: run expr_reuse.qlo
expr_types.rel: run expr_types.qlo

View File

@@ -1,6 +1,5 @@
description: Support C++17 if and switch initializers
compatibility: partial
constexpr_if_initialization.rel: delete
if_initialization.rel: delete
switch_initialization.rel: delete
exprparents.rel: run exprparents.qlo

View File

@@ -1,2 +0,0 @@
description: Remove _Float128 type
compatibility: full

View File

@@ -1,17 +0,0 @@
class AttributeArg extends @attribute_arg {
string toString() { none() }
}
class Attribute extends @attribute {
string toString() { none() }
}
class Location extends @location_default {
string toString() { none() }
}
from AttributeArg arg, int kind, int kind_new, Attribute attr, int index, Location location
where
attribute_args(arg, kind, attr, index, location) and
if arg instanceof @attribute_arg_expr then kind_new = 0 else kind_new = kind
select arg, kind_new, attr, index, location

Some files were not shown because too many files have changed in this diff Show More