Compare commits

..

25 Commits

Author SHA1 Message Date
Stephan Brandauer
9228ff5473 documentation for calleeImports ATM feature 2022-04-22 15:57:28 +02:00
Stephan Brandauer
090ae20909 ATM: new feature to list all imports that are used in a callee 2022-04-22 15:55:58 +02:00
Stephan Brandauer
6299753a2a documentation for new feature 2022-04-22 15:50:02 +02:00
Stephan Brandauer
5ff3d7287b ATM: new feature to list all imports in an endpoint's file 2022-04-22 15:48:40 +02:00
Stephan Brandauer
965551e65d add functionInterfacesInFile and surroundingFunctionParameters features 2022-04-22 13:55:07 +02:00
Esben Sparre Andreasen
ea13a999cc address review comments 2022-04-07 15:01:45 +02:00
Esben Sparre Andreasen
ea88253501 Apply suggestions from code review
Co-authored-by: Henry Mercer <henrymercer@github.com>
2022-04-07 15:01:45 +02:00
Esben Sparre Andreasen
e2ad791983 fix semantic merge conflict 2022-04-07 15:01:45 +02:00
Esben Sparre Andreasen
a4a95f0cda rename new features 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
e6e06b9530 add more features 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
5ca8509759 improve feature documentation 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
a201b77b11 improve feature tests with more cases 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
afd10e3949 improve access path strings 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
510a394307 support import in getSimpleAccessPath 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
e0ea4c4ccb support await in getSimpleAccessPath 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
d4f3f6516c avoid using new feautes by default 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
b84e5af050 add CompareFeatures.ql 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
57812c6934 add generic tests for features 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
22bbe9cc5d Document EndpointFeatures.qll 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
62667f431c add ParameterAccessPathSimpleFromArgumentTraversal 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
8de583b51b improve getSimpleAccessPath 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
a1a93ec9ae refactor calleeAccessPath feature to class 2022-04-07 15:01:44 +02:00
Stephan Brandauer
ccf76c9567 refactor getACallBasedTokenFeature to class-use 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
31c5c97b7d Add CalleeAccessPathSimpleFromArgumentTraversal 2022-04-07 15:01:44 +02:00
Esben Sparre Andreasen
c6fb05636e refactor EndpointFeatures.ql to use classes 2022-04-07 15:01:44 +02:00
8686 changed files with 162939 additions and 723565 deletions

View File

@@ -1,3 +0,0 @@
build --repo_env=CC=clang --repo_env=CXX=clang++ --copt="-std=c++17"
try-import %workspace%/local.bazelrc

View File

@@ -1 +0,0 @@
5.0.0

27
.codeqlmanifest.json Normal file
View File

@@ -0,0 +1,27 @@
{
"provide": [
"*/ql/src/qlpack.yml",
"*/ql/lib/qlpack.yml",
"*/ql/test/qlpack.yml",
"*/ql/examples/qlpack.yml",
"*/ql/consistency-queries/qlpack.yml",
"cpp/ql/test/query-tests/Security/CWE/CWE-190/semmle/tainted/qlpack.yml",
"javascript/ql/experimental/adaptivethreatmodeling/lib/qlpack.yml",
"javascript/ql/experimental/adaptivethreatmodeling/modelbuilding/qlpack.yml",
"javascript/ql/experimental/adaptivethreatmodeling/src/qlpack.yml",
"csharp/ql/campaigns/Solorigate/lib/qlpack.yml",
"csharp/ql/campaigns/Solorigate/src/qlpack.yml",
"csharp/ql/campaigns/Solorigate/test/qlpack.yml",
"misc/legacy-support/*/qlpack.yml",
"misc/suite-helpers/qlpack.yml",
"ruby/extractor-pack/codeql-extractor.yml",
"ql/extractor-pack/codeql-extractor.yml"
],
"versionPolicies": {
"default": {
"requireChangeNotes": true,
"committedPrereleaseSuffix": "dev",
"committedVersion": "nextPatchRelease"
}
}
}

View File

@@ -3,8 +3,6 @@
"rust-lang.rust", "rust-lang.rust",
"bungcip.better-toml", "bungcip.better-toml",
"github.vscode-codeql", "github.vscode-codeql",
"hbenl.vscode-test-explorer",
"ms-vscode.test-adapter-converter",
"slevesque.vscode-zipexplorer" "slevesque.vscode-zipexplorer"
], ],
"settings": { "settings": {

View File

@@ -1,9 +0,0 @@
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.236.0/containers/cpp/.devcontainer/base.Dockerfile
# [Choice] Debian / Ubuntu version (use Debian 11, Ubuntu 18.04/22.04 on local arm64/Apple Silicon): debian-11, debian-10, ubuntu-22.04, ubuntu-20.04, ubuntu-18.04
FROM mcr.microsoft.com/vscode/devcontainers/cpp:0-ubuntu-22.04
USER root
ADD root.sh /tmp/root.sh
ADD update-codeql.sh /usr/local/bin/update-codeql
RUN bash /tmp/root.sh && rm /tmp/root.sh

View File

@@ -1,25 +0,0 @@
{
"extensions": [
"github.vscode-codeql",
"hbenl.vscode-test-explorer",
"ms-vscode.test-adapter-converter",
"slevesque.vscode-zipexplorer",
"ms-vscode.cpptools"
],
"settings": {
"files.watcherExclude": {
"**/target/**": true
},
"codeQL.runningQueries.memory": 2048
},
"build": {
"dockerfile": "Dockerfile",
},
"runArgs": [
"--cap-add=SYS_PTRACE",
"--security-opt",
"seccomp=unconfined"
],
"remoteUser": "vscode",
"onCreateCommand": ".devcontainer/swift/user.sh"
}

View File

@@ -1,22 +0,0 @@
set -xe
BAZELISK_VERSION=v1.12.0
BAZELISK_DOWNLOAD_SHA=6b0bcb2ea15bca16fffabe6fda75803440375354c085480fe361d2cbf32501db
apt-get update
export DEBIAN_FRONTEND=noninteractive
apt-get -y install --no-install-recommends \
zlib1g-dev \
uuid-dev \
python3-distutils \
python3-pip \
bash-completion
# Install Bazel
curl -fSsL -o /usr/local/bin/bazelisk https://github.com/bazelbuild/bazelisk/releases/download/${BAZELISK_VERSION}/bazelisk-linux-amd64
echo "${BAZELISK_DOWNLOAD_SHA} */usr/local/bin/bazelisk" | sha256sum --check -
chmod 0755 /usr/local/bin/bazelisk
ln -s bazelisk /usr/local/bin/bazel
# install latest codeql
update-codeql

View File

@@ -1,20 +0,0 @@
#!/bin/bash -e
URL=https://github.com/github/codeql-cli-binaries/releases
LATEST_VERSION=$(curl -L -s -H 'Accept: application/json' $URL/latest | sed -e 's/.*"tag_name":"\([^"]*\)".*/\1/')
CURRENT_VERSION=v$(codeql version 2>/dev/null | sed -ne 's/.*release \([0-9.]*\)\./\1/p')
if [[ $CURRENT_VERSION != $LATEST_VERSION ]]; then
if [[ $UID != 0 ]]; then
echo "update required, please run this script with sudo:"
echo " sudo $0"
exit 1
fi
ZIP=$(mktemp codeql.XXXX.zip)
curl -fSqL -o $ZIP $URL/download/$LATEST_VERSION/codeql-linux64.zip
unzip -q $ZIP -d /opt
rm $ZIP
ln -sf /opt/codeql/codeql /usr/local/bin/codeql
echo installed version $LATEST_VERSION
else
echo current version $CURRENT_VERSION is up-to-date
fi

View File

@@ -1,13 +0,0 @@
set -xe
# add the workspace to the codeql search path
mkdir -p /home/vscode/.config/codeql
echo "--search-path /workspaces/codeql" > /home/vscode/.config/codeql/config
# create a swift extractor pack with the current state
cd /workspaces/codeql
bazel run swift/create-extractor-pack
#install and set up pre-commit
python3 -m pip install pre-commit --no-warn-script-location
$HOME/.local/bin/pre-commit install

9
.gitattributes vendored
View File

@@ -39,7 +39,6 @@
*.py text *.py text
*.lua text *.lua text
*.expected text *.expected text
*.go text
# Explicitly set a bunch of known extensions to binary, because Git < 2.10 will treat # Explicitly set a bunch of known extensions to binary, because Git < 2.10 will treat
# `* text=auto eol=lf` as `* text eol=lf` # `* text=auto eol=lf` as `* text eol=lf`
@@ -53,14 +52,6 @@
java/ql/test/stubs/**/*.java linguist-generated=true java/ql/test/stubs/**/*.java linguist-generated=true
java/ql/test/experimental/stubs/**/*.java linguist-generated=true java/ql/test/experimental/stubs/**/*.java linguist-generated=true
# Force git not to modify line endings for go or html files under the go/ql directory
go/ql/**/*.go -text
go/ql/**/*.html -text
# Force git not to modify line endings for go dbschemes
go/*.dbscheme -text
# Preserve unusual line ending from codeql-go merge
go/extractor/opencsv/CSVReader.java -text
# For some languages, upgrade script testing references really old dbscheme # For some languages, upgrade script testing references really old dbscheme
# files from legacy upgrades that have CRLF line endings. Since upgrade # files from legacy upgrades that have CRLF line endings. Since upgrade
# resolution relies on object hashes, we must suppress line ending conversion # resolution relies on object hashes, we must suppress line ending conversion

View File

@@ -6,9 +6,9 @@ runs:
- name: Fetch CodeQL - name: Fetch CodeQL
shell: bash shell: bash
run: | run: |
gh extension install github/gh-codeql LATEST=$(gh release list --repo https://github.com/github/codeql-cli-binaries | cut -f 1 | grep -v beta | sort --version-sort | tail -1)
gh codeql set-channel nightly gh release download --repo https://github.com/github/codeql-cli-binaries --pattern codeql-linux64.zip "$LATEST"
gh codeql version unzip -q -d "${RUNNER_TEMP}" codeql-linux64.zip
gh codeql version --format=json | jq -r .unpackedLocation >> "${GITHUB_PATH}" echo "${RUNNER_TEMP}/codeql" >> "${GITHUB_PATH}"
env: env:
GITHUB_TOKEN: ${{ github.token }} GITHUB_TOKEN: ${{ github.token }}

View File

@@ -16,11 +16,3 @@ updates:
directory: "ruby/autobuilder" directory: "ruby/autobuilder"
schedule: schedule:
interval: "daily" interval: "daily"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
ignore:
- dependency-name: '*'
update-types: ['version-update:semver-patch', 'version-update:semver-minor']

18
.github/labeler.yml vendored
View File

@@ -6,23 +6,14 @@
- csharp/**/* - csharp/**/*
- change-notes/**/*csharp* - change-notes/**/*csharp*
Go:
- go/**/*
- change-notes/**/*go.*
Java: Java:
- any: [ 'java/**/*', '!java/kotlin-extractor/**/*', '!java/kotlin-explorer/**/*', '!java/ql/test/kotlin/**/*' ] - java/**/*
- change-notes/**/*java.* - change-notes/**/*java.*
JS: JS:
- any: [ 'javascript/**/*', '!javascript/ql/experimental/adaptivethreatmodeling/**/*' ] - javascript/**/*
- change-notes/**/*javascript* - change-notes/**/*javascript*
Kotlin:
- java/kotlin-extractor/**/*
- java/kotlin-explorer/**/*
- java/ql/test/kotlin/**/*
Python: Python:
- python/**/* - python/**/*
- change-notes/**/*python* - change-notes/**/*python*
@@ -31,10 +22,6 @@ Ruby:
- ruby/**/* - ruby/**/*
- change-notes/**/*ruby* - change-notes/**/*ruby*
Swift:
- swift/**/*
- change-notes/**/*swift*
documentation: documentation:
- "**/*.qhelp" - "**/*.qhelp"
- "**/*.md" - "**/*.md"
@@ -42,4 +29,3 @@ documentation:
"QL-for-QL": "QL-for-QL":
- ql/**/* - ql/**/*
- .github/workflows/ql-for-ql*

View File

@@ -1,14 +0,0 @@
{
"problemMatcher": [
{
"owner": "codeql-query-format",
"pattern": [
{
"regexp": "^((.*) would change by autoformatting\\.)$",
"file": 2,
"message": 1
}
]
}
]
}

View File

@@ -1,17 +0,0 @@
{
"problemMatcher": [
{
"owner": "codeql-syntax-check",
"pattern": [
{
"regexp": "^((ERROR|WARNING): .* \\((.*):(\\d+),(\\d+)-\\d+\\))$",
"message": 1,
"file": 3,
"line": 4,
"col": 5,
"severity": 2
}
]
}
]
}

View File

@@ -1,14 +0,0 @@
{
"problemMatcher": [
{
"owner": "codeql-test-run",
"pattern": [
{
"regexp": "(\\[.*\\] FAILED\\((RESULT|COMPILATION)\\) (.*))$",
"file": 3,
"message": 1
}
]
}
]
}

View File

@@ -1,13 +0,0 @@
{
"problemMatcher": [
{
"owner": "make",
"pattern": [
{
"regexp": "^(make: \\*\\*\\* .*)$",
"message": 1
}
]
}
]
}

View File

@@ -10,7 +10,6 @@ on:
- "*/ql/lib/**/*.qll" - "*/ql/lib/**/*.qll"
- "!**/experimental/**" - "!**/experimental/**"
- "!ql/**" - "!ql/**"
- "!swift/**"
- ".github/workflows/check-change-note.yml" - ".github/workflows/check-change-note.yml"
jobs: jobs:

View File

@@ -5,7 +5,6 @@ on:
paths: paths:
- "*/ql/lib/**" - "*/ql/lib/**"
- .github/workflows/check-qldoc.yml - .github/workflows/check-qldoc.yml
- .github/actions/fetch-codeql/action.yml
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -15,22 +14,26 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - name: Install CodeQL
run: |
gh extension install github/gh-codeql
gh codeql set-channel nightly
gh codeql version
env:
GITHUB_TOKEN: ${{ github.token }}
- uses: actions/checkout@v2
with: with:
fetch-depth: 2 fetch-depth: 2
- name: Install CodeQL
uses: ./.github/actions/fetch-codeql
- name: Check QLdoc coverage - name: Check QLdoc coverage
shell: bash shell: bash
run: | run: |
EXIT_CODE=0 EXIT_CODE=0
# TODO: remove the swift exception from the regex when we fix generated QLdoc changed_lib_packs="$(git diff --name-only --diff-filter=ACMRT HEAD^ HEAD | { grep -o '^[a-z]*/ql/lib' || true; } | sort -u)"
changed_lib_packs="$(git diff --name-only --diff-filter=ACMRT HEAD^ HEAD | { grep -Po '^(?!swift)[a-z]*/ql/lib' || true; } | sort -u)"
for pack_dir in ${changed_lib_packs}; do for pack_dir in ${changed_lib_packs}; do
lang="${pack_dir%/ql/lib}" lang="${pack_dir%/ql/lib}"
codeql generate library-doc-coverage --output="${RUNNER_TEMP}/${lang}-current.txt" --dir="${pack_dir}" gh codeql generate library-doc-coverage --output="${RUNNER_TEMP}/${lang}-current.txt" --dir="${pack_dir}"
done done
git checkout HEAD^ git checkout HEAD^
for pack_dir in ${changed_lib_packs}; do for pack_dir in ${changed_lib_packs}; do
@@ -38,7 +41,7 @@ jobs:
# In this case the right thing to do is to skip the check. # In this case the right thing to do is to skip the check.
[[ ! -d "${pack_dir}" ]] && continue [[ ! -d "${pack_dir}" ]] && continue
lang="${pack_dir%/ql/lib}" lang="${pack_dir%/ql/lib}"
codeql generate library-doc-coverage --output="${RUNNER_TEMP}/${lang}-baseline.txt" --dir="${pack_dir}" gh codeql generate library-doc-coverage --output="${RUNNER_TEMP}/${lang}-baseline.txt" --dir="${pack_dir}"
awk -F, '{gsub(/"/,""); if ($4==0 && $6=="public") print "\""$3"\"" }' "${RUNNER_TEMP}/${lang}-current.txt" | sort -u > "${RUNNER_TEMP}/current-undocumented.txt" awk -F, '{gsub(/"/,""); if ($4==0 && $6=="public") print "\""$3"\"" }' "${RUNNER_TEMP}/${lang}-current.txt" | sort -u > "${RUNNER_TEMP}/current-undocumented.txt"
awk -F, '{gsub(/"/,""); if ($4==0 && $6=="public") print "\""$3"\"" }' "${RUNNER_TEMP}/${lang}-baseline.txt" | sort -u > "${RUNNER_TEMP}/baseline-undocumented.txt" awk -F, '{gsub(/"/,""); if ($4==0 && $6=="public") print "\""$3"\"" }' "${RUNNER_TEMP}/${lang}-baseline.txt" | sort -u > "${RUNNER_TEMP}/baseline-undocumented.txt"
UNDOCUMENTED="$(grep -f <(comm -13 "${RUNNER_TEMP}/baseline-undocumented.txt" "${RUNNER_TEMP}/current-undocumented.txt") "${RUNNER_TEMP}/${lang}-current.txt" || true)" UNDOCUMENTED="$(grep -f <(comm -13 "${RUNNER_TEMP}/baseline-undocumented.txt" "${RUNNER_TEMP}/current-undocumented.txt") "${RUNNER_TEMP}/${lang}-current.txt" || true)"

View File

@@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/stale@v5 - uses: actions/stale@v3
with: with:
repo-token: ${{ secrets.GITHUB_TOKEN }} repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue is stale because it has been open 14 days with no activity. Comment or remove the `Stale` label in order to avoid having this issue closed in 7 days.' stale-issue-message: 'This issue is stale because it has been open 14 days with no activity. Comment or remove the `Stale` label in order to avoid having this issue closed in 7 days.'

View File

@@ -28,12 +28,12 @@ jobs:
steps: steps:
- name: Setup dotnet - name: Setup dotnet
uses: actions/setup-dotnet@v2 uses: actions/setup-dotnet@v1
with: with:
dotnet-version: 6.0.202 dotnet-version: 6.0.101
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v3 uses: actions/checkout@v2
# Initializes the CodeQL tools for scanning. # Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL - name: Initialize CodeQL
@@ -49,7 +49,7 @@ jobs:
# uses: github/codeql-action/autobuild@main # uses: github/codeql-action/autobuild@main
# Command-line programs to run using the OS shell. # Command-line programs to run using the OS shell.
# 📚 https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun # 📚 https://git.io/JvXDl
# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
# and modify them (or add more) to build your code if your project # and modify them (or add more) to build your code if your project

View File

@@ -12,14 +12,13 @@ on:
- main - main
paths: paths:
- ".github/workflows/csv-coverage-metrics.yml" - ".github/workflows/csv-coverage-metrics.yml"
- ".github/actions/fetch-codeql/action.yml"
jobs: jobs:
publish-java: publish:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v3 uses: actions/checkout@v2
- name: Setup CodeQL - name: Setup CodeQL
uses: ./.github/actions/fetch-codeql uses: ./.github/actions/fetch-codeql
- name: Create empty database - name: Create empty database
@@ -32,40 +31,13 @@ jobs:
- name: Capture coverage information - name: Capture coverage information
run: | run: |
DATABASE="${{ runner.temp }}/java-database" DATABASE="${{ runner.temp }}/java-database"
codeql database analyze --format=sarif-latest --output=metrics-java.sarif -- "$DATABASE" ./java/ql/src/Metrics/Summaries/FrameworkCoverage.ql codeql database analyze --format=sarif-latest --output=metrics.sarif -- "$DATABASE" ./java/ql/src/Metrics/Summaries/FrameworkCoverage.ql
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: metrics-java.sarif name: metrics.sarif
path: metrics-java.sarif path: metrics.sarif
retention-days: 20 retention-days: 20
- name: Upload SARIF file - name: Upload SARIF file
uses: github/codeql-action/upload-sarif@main uses: github/codeql-action/upload-sarif@v1
with: with:
sarif_file: metrics-java.sarif sarif_file: metrics.sarif
publish-csharp:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Setup CodeQL
uses: ./.github/actions/fetch-codeql
- name: Create empty database
run: |
DATABASE="${{ runner.temp }}/csharp-database"
PROJECT="${{ runner.temp }}/csharp-project"
dotnet new classlib --language=C# --output="$PROJECT"
codeql database create "$DATABASE" --language=csharp --source-root="$PROJECT" --command 'dotnet build /t:rebuild csharp-project.csproj /p:UseSharedCompilation=false'
- name: Capture coverage information
run: |
DATABASE="${{ runner.temp }}/csharp-database"
codeql database analyze --format=sarif-latest --output=metrics-csharp.sarif -- "$DATABASE" ./csharp/ql/src/Metrics/Summaries/FrameworkCoverage.ql
- uses: actions/upload-artifact@v3
with:
name: metrics-csharp.sarif
path: metrics-csharp.sarif
retention-days: 20
- name: Upload SARIF file
uses: github/codeql-action/upload-sarif@main
with:
sarif_file: metrics-csharp.sarif

View File

@@ -3,20 +3,18 @@ name: Check framework coverage changes
on: on:
pull_request: pull_request:
paths: paths:
- ".github/workflows/csv-coverage-pr-comment.yml" - '.github/workflows/csv-coverage-pr-comment.yml'
- ".github/workflows/csv-coverage-pr-artifacts.yml" - '*/ql/src/**/*.ql'
- ".github/actions/fetch-codeql/action.yml" - '*/ql/src/**/*.qll'
- "*/ql/src/**/*.ql" - '*/ql/lib/**/*.ql'
- "*/ql/src/**/*.qll" - '*/ql/lib/**/*.qll'
- "*/ql/lib/**/*.ql" - 'misc/scripts/library-coverage/*.py'
- "*/ql/lib/**/*.qll"
- "misc/scripts/library-coverage/*.py"
# input data files # input data files
- "*/documentation/library-coverage/cwe-sink.csv" - '*/documentation/library-coverage/cwe-sink.csv'
- "*/documentation/library-coverage/frameworks.csv" - '*/documentation/library-coverage/frameworks.csv'
branches: branches:
- main - main
- "rc/*" - 'rc/*'
jobs: jobs:
generate: generate:
@@ -25,72 +23,77 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Dump GitHub context - name: Dump GitHub context
env: env:
GITHUB_CONTEXT: ${{ toJSON(github.event) }} GITHUB_CONTEXT: ${{ toJSON(github.event) }}
run: echo "$GITHUB_CONTEXT" run: echo "$GITHUB_CONTEXT"
- name: Clone self (github/codeql) - MERGE - name: Clone self (github/codeql) - MERGE
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: merge path: merge
- name: Clone self (github/codeql) - BASE - name: Clone self (github/codeql) - BASE
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
fetch-depth: 2 fetch-depth: 2
path: base path: base
- run: | - run: |
git checkout HEAD^1 git checkout HEAD^1
git log -1 --format='%H' git log -1 --format='%H'
working-directory: base working-directory: base
- name: Set up Python 3.8 - name: Set up Python 3.8
uses: actions/setup-python@v4 uses: actions/setup-python@v2
with: with:
python-version: 3.8 python-version: 3.8
- name: Download CodeQL CLI - name: Download CodeQL CLI
uses: ./merge/.github/actions/fetch-codeql env:
- name: Generate CSV files on merge commit of the PR GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: | run: |
echo "Running generator on merge" gh release download --repo "github/codeql-cli-binaries" --pattern "codeql-linux64.zip"
python merge/misc/scripts/library-coverage/generate-report.py ci merge merge - name: Unzip CodeQL CLI
mkdir out_merge run: unzip -d codeql-cli codeql-linux64.zip
cp framework-coverage-*.csv out_merge/ - name: Generate CSV files on merge commit of the PR
cp framework-coverage-*.rst out_merge/ run: |
- name: Generate CSV files on base commit of the PR echo "Running generator on merge"
run: | PATH="$PATH:codeql-cli/codeql" python merge/misc/scripts/library-coverage/generate-report.py ci merge merge
echo "Running generator on base" mkdir out_merge
python base/misc/scripts/library-coverage/generate-report.py ci base base cp framework-coverage-*.csv out_merge/
mkdir out_base cp framework-coverage-*.rst out_merge/
cp framework-coverage-*.csv out_base/ - name: Generate CSV files on base commit of the PR
cp framework-coverage-*.rst out_base/ run: |
- name: Generate diff of coverage reports echo "Running generator on base"
run: | PATH="$PATH:codeql-cli/codeql" python base/misc/scripts/library-coverage/generate-report.py ci base base
python base/misc/scripts/library-coverage/compare-folders.py out_base out_merge comparison.md mkdir out_base
- name: Upload CSV package list cp framework-coverage-*.csv out_base/
uses: actions/upload-artifact@v3 cp framework-coverage-*.rst out_base/
with: - name: Generate diff of coverage reports
name: csv-framework-coverage-merge run: |
path: | python base/misc/scripts/library-coverage/compare-folders.py out_base out_merge comparison.md
out_merge/framework-coverage-*.csv - name: Upload CSV package list
out_merge/framework-coverage-*.rst uses: actions/upload-artifact@v2
- name: Upload CSV package list with:
uses: actions/upload-artifact@v3 name: csv-framework-coverage-merge
with: path: |
name: csv-framework-coverage-base out_merge/framework-coverage-*.csv
path: | out_merge/framework-coverage-*.rst
out_base/framework-coverage-*.csv - name: Upload CSV package list
out_base/framework-coverage-*.rst uses: actions/upload-artifact@v2
- name: Upload comparison results with:
uses: actions/upload-artifact@v3 name: csv-framework-coverage-base
with: path: |
name: comparison out_base/framework-coverage-*.csv
path: | out_base/framework-coverage-*.rst
comparison.md - name: Upload comparison results
- name: Save PR number uses: actions/upload-artifact@v2
run: | with:
mkdir -p pr name: comparison
echo ${{ github.event.pull_request.number }} > pr/NR path: |
- name: Upload PR number comparison.md
uses: actions/upload-artifact@v3 - name: Save PR number
with: run: |
name: pr mkdir -p pr
path: pr/ echo ${{ github.event.pull_request.number }} > pr/NR
- name: Upload PR number
uses: actions/upload-artifact@v2
with:
name: pr
path: pr/

View File

@@ -20,9 +20,9 @@ jobs:
GITHUB_CONTEXT: ${{ toJSON(github.event) }} GITHUB_CONTEXT: ${{ toJSON(github.event) }}
run: echo "$GITHUB_CONTEXT" run: echo "$GITHUB_CONTEXT"
- name: Clone self (github/codeql) - name: Clone self (github/codeql)
uses: actions/checkout@v3 uses: actions/checkout@v2
- name: Set up Python 3.8 - name: Set up Python 3.8
uses: actions/setup-python@v4 uses: actions/setup-python@v2
with: with:
python-version: 3.8 python-version: 3.8

View File

@@ -5,29 +5,38 @@ on:
jobs: jobs:
build: build:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Clone self (github/codeql) - name: Clone self (github/codeql)
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: script path: script
- name: Clone self (github/codeql) for analysis - name: Clone self (github/codeql) for analysis
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: codeqlModels path: codeqlModels
fetch-depth: 0 fetch-depth: 0
- name: Set up Python 3.8 - name: Set up Python 3.8
uses: actions/setup-python@v4 uses: actions/setup-python@v2
with: with:
python-version: 3.8 python-version: 3.8
- name: Download CodeQL CLI - name: Download CodeQL CLI
uses: ./script/.github/actions/fetch-codeql env:
- name: Build modeled package list GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: | run: |
python script/misc/scripts/library-coverage/generate-timeseries.py codeqlModels gh release download --repo "github/codeql-cli-binaries" --pattern "codeql-linux64.zip"
- name: Upload timeseries CSV - name: Unzip CodeQL CLI
uses: actions/upload-artifact@v3 run: unzip -d codeql-cli codeql-linux64.zip
with: - name: Build modeled package list
name: framework-coverage-timeseries run: |
path: framework-coverage-timeseries-*.csv CLI=$(realpath "codeql-cli/codeql")
echo $CLI
PATH="$PATH:$CLI" python script/misc/scripts/library-coverage/generate-timeseries.py codeqlModels
- name: Upload timeseries CSV
uses: actions/upload-artifact@v2
with:
name: framework-coverage-timeseries
path: framework-coverage-timeseries-*.csv

View File

@@ -12,27 +12,33 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Dump GitHub context - name: Dump GitHub context
env: env:
GITHUB_CONTEXT: ${{ toJSON(github.event) }} GITHUB_CONTEXT: ${{ toJSON(github.event) }}
run: echo "$GITHUB_CONTEXT" run: echo "$GITHUB_CONTEXT"
- name: Clone self (github/codeql) - name: Clone self (github/codeql)
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: ql path: ql
fetch-depth: 0 fetch-depth: 0
- name: Set up Python 3.8 - name: Set up Python 3.8
uses: actions/setup-python@v4 uses: actions/setup-python@v2
with: with:
python-version: 3.8 python-version: 3.8
- name: Download CodeQL CLI - name: Download CodeQL CLI
uses: ./ql/.github/actions/fetch-codeql env:
- name: Generate coverage files GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: | run: |
python ql/misc/scripts/library-coverage/generate-report.py ci ql ql gh release download --repo "github/codeql-cli-binaries" --pattern "codeql-linux64.zip"
- name: Unzip CodeQL CLI
run: unzip -d codeql-cli codeql-linux64.zip
- name: Create pull request with changes - name: Generate coverage files
env: run: |
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} PATH="$PATH:codeql-cli/codeql" python ql/misc/scripts/library-coverage/generate-report.py ci ql ql
run: |
python ql/misc/scripts/library-coverage/create-pr.py ql "$GITHUB_REPOSITORY" - name: Create pull request with changes
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
python ql/misc/scripts/library-coverage/create-pr.py ql "$GITHUB_REPOSITORY"

View File

@@ -4,39 +4,46 @@ on:
workflow_dispatch: workflow_dispatch:
inputs: inputs:
qlModelShaOverride: qlModelShaOverride:
description: "github/codeql repo SHA used for looking up the CSV models" description: 'github/codeql repo SHA used for looking up the CSV models'
required: false required: false
jobs: jobs:
build: build:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Clone self (github/codeql) - name: Clone self (github/codeql)
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: script path: script
- name: Clone self (github/codeql) for analysis - name: Clone self (github/codeql) for analysis
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: codeqlModels path: codeqlModels
ref: ${{ github.event.inputs.qlModelShaOverride || github.ref }} ref: ${{ github.event.inputs.qlModelShaOverride || github.ref }}
- name: Set up Python 3.8 - name: Set up Python 3.8
uses: actions/setup-python@v4 uses: actions/setup-python@v2
with: with:
python-version: 3.8 python-version: 3.8
- name: Download CodeQL CLI - name: Download CodeQL CLI
uses: ./script/.github/actions/fetch-codeql env:
- name: Build modeled package list GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: | run: |
python script/misc/scripts/library-coverage/generate-report.py ci codeqlModels script gh release download --repo "github/codeql-cli-binaries" --pattern "codeql-linux64.zip"
- name: Upload CSV package list - name: Unzip CodeQL CLI
uses: actions/upload-artifact@v3 run: unzip -d codeql-cli codeql-linux64.zip
with: - name: Build modeled package list
name: framework-coverage-csv run: |
path: framework-coverage-*.csv PATH="$PATH:codeql-cli/codeql" python script/misc/scripts/library-coverage/generate-report.py ci codeqlModels script
- name: Upload RST package list - name: Upload CSV package list
uses: actions/upload-artifact@v3 uses: actions/upload-artifact@v2
with: with:
name: framework-coverage-rst name: framework-coverage-csv
path: framework-coverage-*.rst path: framework-coverage-*.csv
- name: Upload RST package list
uses: actions/upload-artifact@v2
with:
name: framework-coverage-rst
path: framework-coverage-*.rst

View File

@@ -1,114 +0,0 @@
name: "Go: Run Tests"
on:
pull_request:
paths:
- "go/**"
- .github/workflows/go-tests.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
jobs:
test-linux:
name: Test Linux (Ubuntu)
runs-on: ubuntu-latest
steps:
- name: Set up Go 1.19
uses: actions/setup-go@v3
with:
go-version: 1.19
id: go
- name: Check out code
uses: actions/checkout@v2
- name: Set up CodeQL CLI
uses: ./.github/actions/fetch-codeql
- name: Enable problem matchers in repository
shell: bash
run: 'find .github/problem-matchers -name \*.json -exec echo "::add-matcher::{}" \;'
- name: Build
run: |
cd go
make
- name: Check that all QL and Go code is autoformatted
run: |
cd go
make check-formatting
- name: Compile qhelp files to markdown
run: |
cd go
env QHELP_OUT_DIR=qhelp-out make qhelp-to-markdown
- name: Upload qhelp markdown
uses: actions/upload-artifact@v2
with:
name: qhelp-markdown
path: go/qhelp-out/**/*.md
- name: Test
run: |
cd go
make test
test-mac:
name: Test MacOS
runs-on: macos-latest
steps:
- name: Set up Go 1.19
uses: actions/setup-go@v3
with:
go-version: 1.19
id: go
- name: Check out code
uses: actions/checkout@v2
- name: Set up CodeQL CLI
uses: ./.github/actions/fetch-codeql
- name: Enable problem matchers in repository
shell: bash
run: 'find .github/problem-matchers -name \*.json -exec echo "::add-matcher::{}" \;'
- name: Build
run: |
cd go
make
- name: Test
run: |
cd go
make test
test-win:
name: Test Windows
runs-on: windows-2019
steps:
- name: Set up Go 1.19
uses: actions/setup-go@v3
with:
go-version: 1.19
id: go
- name: Check out code
uses: actions/checkout@v2
- name: Set up CodeQL CLI
uses: ./.github/actions/fetch-codeql
- name: Enable problem matchers in repository
shell: bash
run: 'find .github/problem-matchers -name \*.json -exec echo "::add-matcher::{}" \;'
- name: Build
run: |
cd go
make
- name: Test
run: |
cd go
make test

View File

@@ -5,8 +5,6 @@ on:
paths: paths:
- "javascript/ql/experimental/adaptivethreatmodeling/**" - "javascript/ql/experimental/adaptivethreatmodeling/**"
- .github/workflows/js-ml-tests.yml - .github/workflows/js-ml-tests.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -14,9 +12,6 @@ on:
paths: paths:
- "javascript/ql/experimental/adaptivethreatmodeling/**" - "javascript/ql/experimental/adaptivethreatmodeling/**"
- .github/workflows/js-ml-tests.yml - .github/workflows/js-ml-tests.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
workflow_dispatch:
defaults: defaults:
run: run:
@@ -27,7 +22,7 @@ jobs:
name: Check QL formatting name: Check QL formatting
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
@@ -40,7 +35,7 @@ jobs:
name: Check QL compilation name: Check QL compilation
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
@@ -64,7 +59,7 @@ jobs:
name: Run QL tests name: Run QL tests
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql

View File

@@ -4,11 +4,8 @@ on:
jobs: jobs:
triage: triage:
permissions:
contents: read
pull-requests: write
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/labeler@v4 - uses: actions/labeler@v2
with: with:
repo-token: "${{ secrets.GITHUB_TOKEN }}" repo-token: "${{ secrets.GITHUB_TOKEN }}"

View File

@@ -27,12 +27,12 @@ jobs:
slug: ${{fromJson(github.event.inputs.projects || '["apache/commons-codec", "apache/commons-io", "apache/commons-beanutils", "apache/commons-logging", "apache/commons-fileupload", "apache/commons-lang", "apache/commons-validator", "apache/commons-csv", "apache/dubbo"]' )}} slug: ${{fromJson(github.event.inputs.projects || '["apache/commons-codec", "apache/commons-io", "apache/commons-beanutils", "apache/commons-logging", "apache/commons-fileupload", "apache/commons-lang", "apache/commons-validator", "apache/commons-csv", "apache/dubbo"]' )}}
steps: steps:
- name: Clone github/codeql from PR - name: Clone github/codeql from PR
uses: actions/checkout@v3 uses: actions/checkout@v2
if: github.event.pull_request if: github.event.pull_request
with: with:
path: codeql-pr path: codeql-pr
- name: Clone github/codeql from main - name: Clone github/codeql from main
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: codeql-main path: codeql-main
ref: main ref: main
@@ -61,7 +61,7 @@ jobs:
DATABASE=$2 DATABASE=$2
cd codeql-$QL_VARIANT cd codeql-$QL_VARIANT
SHORTNAME=`basename $DATABASE` SHORTNAME=`basename $DATABASE`
python java/ql/src/utils/model-generator/GenerateFlowModel.py --with-summaries --with-sinks $DATABASE $MODELS/${SHORTNAME}.qll python java/ql/src/utils/model-generator/GenerateFlowModel.py $DATABASE $MODELS/${SHORTNAME}.qll
mv $MODELS/${SHORTNAME}.qll $MODELS/${SHORTNAME}Generated_${QL_VARIANT}.qll mv $MODELS/${SHORTNAME}.qll $MODELS/${SHORTNAME}Generated_${QL_VARIANT}.qll
cd .. cd ..
} }
@@ -91,12 +91,12 @@ jobs:
name="diff_${basename/_main.qll/""}" name="diff_${basename/_main.qll/""}"
(diff -w -u $m $t | diff2html -i stdin -F $MODELS/$name.html) || true (diff -w -u $m $t | diff2html -i stdin -F $MODELS/$name.html) || true
done done
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: models name: models
path: tmp-models/*.qll path: tmp-models/*.qll
retention-days: 20 retention-days: 20
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: diffs name: diffs
path: tmp-models/*.html path: tmp-models/*.html

View File

@@ -9,7 +9,6 @@ on:
- main - main
paths: paths:
- ".github/workflows/mad_regenerate-models.yml" - ".github/workflows/mad_regenerate-models.yml"
- ".github/actions/fetch-codeql/action.yml"
jobs: jobs:
regenerate-models: regenerate-models:
@@ -21,17 +20,17 @@ jobs:
ref: ["placeholder"] ref: ["placeholder"]
include: include:
- slug: "apache/commons-io" - slug: "apache/commons-io"
ref: "13258ce2d07aa0e764bbaa8020af4dcd3a02a620" ref: "8985de8fe74f6622a419b37a6eed0dbc484dc128"
exclude: exclude:
- slug: "placeholder" - slug: "placeholder"
ref: "placeholder" ref: "placeholder"
steps: steps:
- name: Clone self (github/codeql) - name: Clone self (github/codeql)
uses: actions/checkout@v3 uses: actions/checkout@v2
- name: Setup CodeQL binaries - name: Setup CodeQL binaries
uses: ./.github/actions/fetch-codeql uses: ./.github/actions/fetch-codeql
- name: Clone repositories - name: Clone repositories
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: repos/${{ matrix.ref }} path: repos/${{ matrix.ref }}
ref: ${{ matrix.ref }} ref: ${{ matrix.ref }}
@@ -56,7 +55,7 @@ jobs:
find java -name "*.qll" -print0 | xargs -0 git add find java -name "*.qll" -print0 | xargs -0 git add
git status git status
git diff --cached > models.patch git diff --cached > models.patch
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: patch name: patch
path: models.patch path: models.patch

View File

@@ -1,17 +1,12 @@
# This workflow is the second part of the process described in name: Post pull-request comment
# .github/workflows/qhelp-pr-preview.yml
# See that file for more info.
name: Post PR comment
on: on:
workflow_run: workflow_run:
workflows: [Render QHelp changes] workflows: ["Query help preview"]
types: types:
- completed - completed
permissions: permissions:
pull-requests: write pull-requests: write
actions: read
jobs: jobs:
post_comment: post_comment:
@@ -22,53 +17,15 @@ jobs:
env: env:
GITHUB_TOKEN: ${{ github.token }} GITHUB_TOKEN: ${{ github.token }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }} WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
- run: |
- name: Check that PR SHA matches workflow SHA PR="$(grep -o '^[0-9]\+$' pr.txt)"
run: |
PR="$(grep -o '^[0-9]\+$' pr_number.txt)"
PR_HEAD_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/pulls/${PR}" --jq .head.sha)" PR_HEAD_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/pulls/${PR}" --jq .head.sha)"
# Check that the pull-request head SHA matches the head SHA of the workflow run # Check that the pull-request head SHA matches the head SHA of the workflow run
if [ "${WORKFLOW_RUN_HEAD_SHA}" != "${PR_HEAD_SHA}" ]; then if [ "${WORKFLOW_RUN_HEAD_SHA}" != "${PR_HEAD_SHA}" ]; then
echo "PR head SHA ${PR_HEAD_SHA} does not match workflow_run event SHA ${WORKFLOW_RUN_HEAD_SHA}. Stopping." 1>&2 echo "PR head SHA ${PR_HEAD_SHA} does not match workflow_run event SHA ${WORKFLOW_RUN_HEAD_SHA}. Stopping." 1>&2
exit 1 exit 1
fi fi
gh pr comment "${PR}" --repo "${GITHUB_REPOSITORY}" -F comment.txt
env: env:
GITHUB_TOKEN: ${{ github.token }} GITHUB_TOKEN: ${{ github.token }}
WORKFLOW_RUN_HEAD_SHA: ${{ github.event.workflow_run.head_commit.id }} WORKFLOW_RUN_HEAD_SHA: ${{ github.event.workflow_run.head_commit.id }}
- name: Create or update comment
run: |
COMMENT_PREFIX="QHelp previews"
COMMENT_AUTHOR="github-actions[bot]"
PR_NUMBER="$(grep -o '^[0-9]\+$' pr_number.txt)"
# If there is no existing comment, comment_id.txt will contain just a
# newline (due to jq & gh behaviour). This will cause grep to fail, so
# we catch that.
RAW_COMMENT_ID=$(grep -o '^[0-9]\+$' comment_id.txt || true)
if [ $RAW_COMMENT_ID ]
then
# Fetch existing comment, and validate:
# - comment belongs to the PR with number $PR_NUMBER
# - comment starts with the expected prefix ("QHelp previews")
# - comment author is github-actions[bot]
FILTER='select(.issue_url | endswith($repo+"/issues/"+$pr))
| select(.body | startswith($prefix))
| select(.user.login == $author)
| .id'
COMMENT_ID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/comments/${RAW_COMMENT_ID}" | jq --arg repo "${GITHUB_REPOSITORY}" --arg pr "${PR_NUMBER}" --arg prefix "${COMMENT_PREFIX}" --arg author "${COMMENT_AUTHOR}" "${FILTER}")
if [ $COMMENT_ID ]
then
# Update existing comment
jq --rawfile body comment_body.txt '{"body":$body}' -n | gh api "repos/${GITHUB_REPOSITORY}/issues/comments/${COMMENT_ID}" -X PATCH --input -
else
echo "Comment ${RAW_COMMENT_ID} did not pass validations: not editing." >&2
exit 1
fi
else
# Create new comment
jq --rawfile body comment_body.txt '{"body":$body}' -n | gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" -X POST --input -
fi
env:
GITHUB_TOKEN: ${{ github.token }}

View File

@@ -1,25 +1,7 @@
# This workflow checks for any changes in .qhelp files in pull requests. name: Query help preview
# For any changed files, it renders them to markdown in a file called `comment_body.txt`.
# It then checks if there's an existing comment on the pull request generated by
# this workflow, and writes the comment ID to `comment_id.txt`.
# It also writes the PR number to `pr_number.txt`.
# These three files are uploaded as an artifact.
# When this workflow completes, the workflow "Post PR comment" runs.
# It downloads the artifact and adds a comment to the PR with the rendered
# QHelp.
# The task is split like this because creating PR comments requires extra
# permissions that we don't want to expose to PRs from external forks.
# For more info see:
# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_run
# https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token
name: Render QHelp changes
permissions: permissions:
contents: read contents: read
pull-requests: read
on: on:
pull_request: pull_request:
@@ -33,17 +15,13 @@ jobs:
qhelp: qhelp:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- run: echo "${PR_NUMBER}" > pr_number.txt - run: echo "${{ github.event.number }}" > pr.txt
env: - uses: actions/upload-artifact@v2
PR_NUMBER: ${{ github.event.number }}
- uses: actions/upload-artifact@v3
with: with:
name: comment name: comment
path: pr_number.txt path: pr.txt
if-no-files-found: error
retention-days: 1 retention-days: 1
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with: with:
fetch-depth: 2 fetch-depth: 2
persist-credentials: false persist-credentials: false
@@ -58,7 +36,7 @@ jobs:
- name: QHelp preview - name: QHelp preview
run: | run: |
EXIT_CODE=0 EXIT_CODE=0
echo "QHelp previews:" > comment_body.txt echo "QHelp previews:" > comment.txt
while read -r -d $'\0' path; do while read -r -d $'\0' path; do
if [ ! -f "${path}" ]; then if [ ! -f "${path}" ]; then
exit 1 exit 1
@@ -74,29 +52,12 @@ jobs:
echo '```' echo '```'
fi fi
echo "</details>" echo "</details>"
done < "${RUNNER_TEMP}/paths.txt" >> comment_body.txt done < "${RUNNER_TEMP}/paths.txt" >> comment.txt
exit "${EXIT_CODE}" exit "${EXIT_CODE}"
- if: always() - if: always()
uses: actions/upload-artifact@v3 uses: actions/upload-artifact@v2
with: with:
name: comment name: comment
path: comment_body.txt path: comment.txt
if-no-files-found: error
retention-days: 1
- name: Save ID of existing QHelp comment (if it exists)
run: |
# Find the latest comment starting with "QHelp previews"
COMMENT_PREFIX="QHelp previews"
gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" --paginate | jq --arg prefix "${COMMENT_PREFIX}" '[.[] | select(.body|startswith($prefix)) | .id] | max' > comment_id.txt
env:
GITHUB_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.number }}
- uses: actions/upload-artifact@v3
with:
name: comment
path: comment_id.txt
if-no-files-found: error
retention-days: 1 retention-days: 1

View File

@@ -10,16 +10,16 @@ env:
CARGO_TERM_COLOR: always CARGO_TERM_COLOR: always
jobs: jobs:
analyze: queries:
runs-on: ubuntu-latest-xl runs-on: ubuntu-latest
steps: steps:
### Build the queries ### - uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Find codeql - name: Find codeql
id: find-codeql id: find-codeql
uses: github/codeql-action/init@71a8b35ff4c80fcfcd05bc1cd932fe3c08f943ca uses: github/codeql-action/init@aa93aea877e5fb8841bcb1193f672abf6e9f2980
with: with:
languages: javascript # does not matter languages: javascript # does not matter
tools: latest
- name: Get CodeQL version - name: Get CodeQL version
id: get-codeql-version id: get-codeql-version
run: | run: |
@@ -27,39 +27,39 @@ jobs:
shell: bash shell: bash
env: env:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }} CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- name: Cache entire pack
id: cache-pack
uses: actions/cache@v3
with:
path: ${{ runner.temp }}/pack
key: ${{ runner.os }}-pack-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('ql/**/*.rs') }}-${{ hashFiles('ql/**/*.ql*') }}-${{ hashFiles('ql/**/qlpack.yml') }}-${{ hashFiles('ql/ql/src/ql.dbscheme*') }}-${{ steps.get-codeql-version.outputs.version }}--${{ hashFiles('.github/workflows/ql-for-ql-build.yml') }}
- name: Cache queries - name: Cache queries
if: steps.cache-pack.outputs.cache-hit != 'true'
id: cache-queries id: cache-queries
uses: actions/cache@v3 uses: actions/cache@v2
with: with:
path: ${{ runner.temp }}/queries path: ${{ runner.temp }}/query-pack.zip
key: queries-${{ hashFiles('ql/**/*.ql*') }}-${{ hashFiles('ql/**/qlpack.yml') }}-${{ hashFiles('ql/ql/src/ql.dbscheme*') }}-${{ steps.get-codeql-version.outputs.version }}--${{ hashFiles('.github/workflows/ql-for-ql-build.yml') }} key: queries-${{ hashFiles('ql/**/*.ql*') }}-${{ hashFiles('ql/**/qlpack.yml') }}-${{ hashFiles('ql/ql/src/ql.dbscheme*') }}-${{ steps.get-codeql-version.outputs.version }}
- name: Build query pack - name: Build query pack
if: steps.cache-queries.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-queries.outputs.cache-hit != 'true'
run: | run: |
cd ql/ql/src cd ql/ql/src
"${CODEQL}" pack create -j 16 "${CODEQL}" pack create
mv .codeql/pack/codeql/ql/0.0.0 ${{ runner.temp }}/queries cd .codeql/pack/codeql/ql/0.0.0
env: zip "${PACKZIP}" -r .
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- name: Move cache queries to pack
if: steps.cache-pack.outputs.cache-hit != 'true'
run: |
cp -r ${{ runner.temp }}/queries ${{ runner.temp }}/pack
env: env:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }} CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
PACKZIP: ${{ runner.temp }}/query-pack.zip
- name: Upload query pack
uses: actions/upload-artifact@v2
with:
name: query-pack-zip
path: ${{ runner.temp }}/query-pack.zip
### Build the extractor ### extractors:
strategy:
fail-fast: false
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Cache entire extractor - name: Cache entire extractor
if: steps.cache-pack.outputs.cache-hit != 'true'
id: cache-extractor id: cache-extractor
uses: actions/cache@v3 uses: actions/cache@v2
with: with:
path: | path: |
ql/target/release/ql-autobuilder ql/target/release/ql-autobuilder
@@ -68,8 +68,8 @@ jobs:
ql/target/release/ql-extractor.exe ql/target/release/ql-extractor.exe
key: ${{ runner.os }}-extractor-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('ql/**/*.rs') }} key: ${{ runner.os }}-extractor-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('ql/**/*.rs') }}
- name: Cache cargo - name: Cache cargo
if: steps.cache-extractor.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-extractor.outputs.cache-hit != 'true'
uses: actions/cache@v3 uses: actions/cache@v2
with: with:
path: | path: |
~/.cargo/registry ~/.cargo/registry
@@ -77,35 +77,87 @@ jobs:
ql/target ql/target
key: ${{ runner.os }}-rust-cargo-${{ hashFiles('ql/**/Cargo.lock') }} key: ${{ runner.os }}-rust-cargo-${{ hashFiles('ql/**/Cargo.lock') }}
- name: Check formatting - name: Check formatting
if: steps.cache-extractor.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd ql; cargo fmt --all -- --check run: cd ql; cargo fmt --all -- --check
- name: Build - name: Build
if: steps.cache-extractor.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd ql; cargo build --verbose run: cd ql; cargo build --verbose
- name: Run tests - name: Run tests
if: steps.cache-extractor.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd ql; cargo test --verbose run: cd ql; cargo test --verbose
- name: Release build - name: Release build
if: steps.cache-extractor.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-extractor.outputs.cache-hit != 'true'
run: cd ql; cargo build --release run: cd ql; cargo build --release
- name: Generate dbscheme - name: Generate dbscheme
if: steps.cache-extractor.outputs.cache-hit != 'true' && steps.cache-pack.outputs.cache-hit != 'true' if: steps.cache-extractor.outputs.cache-hit != 'true'
run: ql/target/release/ql-generator --dbscheme ql/ql/src/ql.dbscheme --library ql/ql/src/codeql_ql/ast/internal/TreeSitter.qll run: ql/target/release/ql-generator --dbscheme ql/ql/src/ql.dbscheme --library ql/ql/src/codeql_ql/ast/internal/TreeSitter.qll
- uses: actions/upload-artifact@v2
with:
name: extractor-ubuntu-latest
path: |
ql/target/release/ql-autobuilder
ql/target/release/ql-autobuilder.exe
ql/target/release/ql-extractor
ql/target/release/ql-extractor.exe
retention-days: 1
package:
runs-on: ubuntu-latest
### Package the queries and extractor ### needs:
- name: Package pack - extractors
if: steps.cache-pack.outputs.cache-hit != 'true' - queries
steps:
- uses: actions/checkout@v2
- uses: actions/download-artifact@v2
with:
name: query-pack-zip
path: query-pack-zip
- uses: actions/download-artifact@v2
with:
name: extractor-ubuntu-latest
path: linux64
- run: |
unzip query-pack-zip/*.zip -d pack
cp -r ql/codeql-extractor.yml ql/tools ql/ql/src/ql.dbscheme.stats pack/
mkdir -p pack/tools/linux64
if [[ -f linux64/ql-autobuilder ]]; then
cp linux64/ql-autobuilder pack/tools/linux64/autobuilder
chmod +x pack/tools/linux64/autobuilder
fi
if [[ -f linux64/ql-extractor ]]; then
cp linux64/ql-extractor pack/tools/linux64/extractor
chmod +x pack/tools/linux64/extractor
fi
cd pack
zip -rq ../codeql-ql.zip .
- uses: actions/upload-artifact@v2
with:
name: codeql-ql-pack
path: codeql-ql.zip
retention-days: 1
analyze:
runs-on: ubuntu-latest
strategy:
matrix:
folder: [cpp, csharp, java, javascript, python, ql, ruby]
needs:
- package
steps:
- name: Download pack
uses: actions/download-artifact@v2
with:
name: codeql-ql-pack
path: ${{ runner.temp }}/codeql-ql-pack-artifact
- name: Prepare pack
run: | run: |
cp -r ql/codeql-extractor.yml ql/tools ql/ql/src/ql.dbscheme.stats ${PACK}/ unzip "${PACK_ARTIFACT}/*.zip" -d "${PACK}"
mkdir -p ${PACK}/tools/linux64
cp ql/target/release/ql-autobuilder ${PACK}/tools/linux64/autobuilder
cp ql/target/release/ql-extractor ${PACK}/tools/linux64/extractor
chmod +x ${PACK}/tools/linux64/autobuilder
chmod +x ${PACK}/tools/linux64/extractor
env: env:
PACK_ARTIFACT: ${{ runner.temp }}/codeql-ql-pack-artifact
PACK: ${{ runner.temp }}/pack PACK: ${{ runner.temp }}/pack
### Run the analysis ###
- name: Hack codeql-action options - name: Hack codeql-action options
run: | run: |
JSON=$(jq -nc --arg pack "${PACK}" '.database."run-queries"=["--search-path", $pack] | .resolve.queries=["--search-path", $pack] | .resolve.extractor=["--search-path", $pack] | .database.init=["--search-path", $pack]') JSON=$(jq -nc --arg pack "${PACK}" '.database."run-queries"=["--search-path", $pack] | .resolve.queries=["--search-path", $pack] | .resolve.extractor=["--search-path", $pack] | .database.init=["--search-path", $pack]')
@@ -113,51 +165,39 @@ jobs:
env: env:
PACK: ${{ runner.temp }}/pack PACK: ${{ runner.temp }}/pack
- name: Checkout repository
uses: actions/checkout@v2
- name: Create CodeQL config file - name: Create CodeQL config file
run: | run: |
echo "paths:" > ${CONF}
echo " - ${FOLDER}" >> ${CONF}
echo "paths-ignore:" >> ${CONF} echo "paths-ignore:" >> ${CONF}
echo " - ql/ql/test" >> ${CONF} echo " - ql/ql/test" >> ${CONF}
echo " - \"*/ql/lib/upgrades/\"" >> ${CONF}
echo "disable-default-queries: true" >> ${CONF} echo "disable-default-queries: true" >> ${CONF}
echo "queries:" >> ${CONF} echo "packs:" >> ${CONF}
echo " - uses: ./ql/ql/src/codeql-suites/ql-code-scanning.qls" >> ${CONF} echo " - codeql/ql" >> ${CONF}
echo "Config file: " echo "Config file: "
cat ${CONF} cat ${CONF}
env: env:
CONF: ./ql-for-ql-config.yml CONF: ./ql-for-ql-config.yml
FOLDER: ${{ matrix.folder }}
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@71a8b35ff4c80fcfcd05bc1cd932fe3c08f943ca uses: github/codeql-action/init@aa93aea877e5fb8841bcb1193f672abf6e9f2980
with: with:
languages: ql languages: ql
db-location: ${{ runner.temp }}/db db-location: ${{ runner.temp }}/db
config-file: ./ql-for-ql-config.yml config-file: ./ql-for-ql-config.yml
- name: Move pack cache tools: latest
run: |
cp -r ${PACK}/.cache ql/ql/src/.cache
env:
PACK: ${{ runner.temp }}/pack
- name: Perform CodeQL Analysis - name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@71a8b35ff4c80fcfcd05bc1cd932fe3c08f943ca uses: github/codeql-action/analyze@aa93aea877e5fb8841bcb1193f672abf6e9f2980
with: with:
category: "ql-for-ql" category: "ql-for-ql-${{ matrix.folder }}"
- name: Copy sarif file to CWD - name: Copy sarif file to CWD
run: cp ../results/ql.sarif ./ql-for-ql.sarif run: cp ../results/ql.sarif ./${{ matrix.folder }}.sarif
- name: Fixup the $scema in sarif # Until https://github.com/microsoft/sarif-vscode-extension/pull/436/ is part in a stable release
run: |
sed -i 's/\$schema.*/\$schema": "https:\/\/raw.githubusercontent.com\/oasis-tcs\/sarif-spec\/master\/Schemata\/sarif-schema-2.1.0",/' ql-for-ql.sarif
- name: Sarif as artifact - name: Sarif as artifact
uses: actions/upload-artifact@v3 uses: actions/upload-artifact@v2
with: with:
name: ql-for-ql.sarif name: ${{ matrix.folder }}.sarif
path: ql-for-ql.sarif path: ${{ matrix.folder }}.sarif
- name: Split out the sarif file into langs
run: |
mkdir split-sarif
node ./ql/scripts/split-sarif.js ql-for-ql.sarif split-sarif
- name: Upload langs as artifacts
uses: actions/upload-artifact@v3
with:
name: ql-for-ql-langs
path: split-sarif
retention-days: 1

View File

@@ -19,16 +19,17 @@ jobs:
matrix: matrix:
repo: repo:
- github/codeql - github/codeql
- github/codeql-go
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- name: Find codeql - name: Find codeql
id: find-codeql id: find-codeql
uses: github/codeql-action/init@71a8b35ff4c80fcfcd05bc1cd932fe3c08f943ca uses: github/codeql-action/init@aa93aea877e5fb8841bcb1193f672abf6e9f2980
with: with:
languages: javascript # does not matter languages: javascript # does not matter
- uses: actions/cache@v3 - uses: actions/cache@v2
with: with:
path: | path: |
~/.cargo/registry ~/.cargo/registry
@@ -36,11 +37,11 @@ jobs:
ql/target ql/target
key: ${{ runner.os }}-qltest-cargo-${{ hashFiles('ql/**/Cargo.lock') }} key: ${{ runner.os }}-qltest-cargo-${{ hashFiles('ql/**/Cargo.lock') }}
- name: Build Extractor - name: Build Extractor
run: cd ql; env "PATH=$PATH:`dirname ${CODEQL}`" ./scripts/create-extractor-pack.sh run: cd ql; env "PATH=$PATH:`dirname ${CODEQL}`" ./create-extractor-pack.sh
env: env:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }} CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- name: Checkout ${{ matrix.repo }} - name: Checkout ${{ matrix.repo }}
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
repository: ${{ matrix.repo }} repository: ${{ matrix.repo }}
path: ${{ github.workspace }}/repo path: ${{ github.workspace }}/repo
@@ -59,7 +60,7 @@ jobs:
"${CODEQL}" dataset measure --threads 4 --output "stats/${{ matrix.repo }}/stats.xml" "${{ runner.temp }}/database/db-ql" "${CODEQL}" dataset measure --threads 4 --output "stats/${{ matrix.repo }}/stats.xml" "${{ runner.temp }}/database/db-ql"
env: env:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }} CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: measurements name: measurements
path: stats path: stats
@@ -69,15 +70,15 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
needs: measure needs: measure
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: measurements name: measurements
path: stats path: stats
- run: | - run: |
python -m pip install --user lxml python -m pip install --user lxml
find stats -name 'stats.xml' -print0 | sort -z | xargs -0 python ruby/scripts/merge_stats.py --output ql/ql/src/ql.dbscheme.stats --normalise ql_tokeninfo find stats -name 'stats.xml' -print0 | sort -z | xargs -0 python ql/scripts/merge_stats.py --output ql/ql/src/ql.dbscheme.stats --normalise ql_tokeninfo
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: ql.dbscheme.stats name: ql.dbscheme.stats
path: ql/ql/src/ql.dbscheme.stats path: ql/ql/src/ql.dbscheme.stats

View File

@@ -5,12 +5,10 @@ on:
branches: [main] branches: [main]
paths: paths:
- "ql/**" - "ql/**"
- codeql-workspace.yml
pull_request: pull_request:
branches: [main] branches: [main]
paths: paths:
- "ql/**" - "ql/**"
- codeql-workspace.yml
env: env:
CARGO_TERM_COLOR: always CARGO_TERM_COLOR: always
@@ -19,13 +17,13 @@ jobs:
qltest: qltest:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- name: Find codeql - name: Find codeql
id: find-codeql id: find-codeql
uses: github/codeql-action/init@71a8b35ff4c80fcfcd05bc1cd932fe3c08f943ca uses: github/codeql-action/init@aa93aea877e5fb8841bcb1193f672abf6e9f2980
with: with:
languages: javascript # does not matter languages: javascript # does not matter
- uses: actions/cache@v3 - uses: actions/cache@v2
with: with:
path: | path: |
~/.cargo/registry ~/.cargo/registry
@@ -36,7 +34,7 @@ jobs:
run: | run: |
cd ql; cd ql;
codeqlpath=$(dirname ${{ steps.find-codeql.outputs.codeql-path }}); codeqlpath=$(dirname ${{ steps.find-codeql.outputs.codeql-path }});
env "PATH=$PATH:$codeqlpath" ./scripts/create-extractor-pack.sh env "PATH=$PATH:$codeqlpath" ./create-extractor-pack.sh
- name: Run QL tests - name: Run QL tests
run: | run: |
"${CODEQL}" test run --check-databases --check-unused-labels --check-repeated-labels --check-redefined-labels --check-use-before-definition --search-path "${{ github.workspace }}/ql/extractor-pack" --consistency-queries ql/ql/consistency-queries ql/ql/test "${CODEQL}" test run --check-databases --check-unused-labels --check-repeated-labels --check-redefined-labels --check-use-before-definition --search-path "${{ github.workspace }}/ql/extractor-pack" --consistency-queries ql/ql/consistency-queries ql/ql/test
@@ -44,7 +42,7 @@ jobs:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }} CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- name: Check QL formatting - name: Check QL formatting
run: | run: |
find ql/ql/src "(" -name "*.ql" -or -name "*.qll" ")" -print0 | xargs -0 "${CODEQL}" query format --check-only find ql/ql "(" -name "*.ql" -or -name "*.qll" ")" -print0 | xargs -0 "${CODEQL}" query format --check-only
env: env:
CODEQL: ${{ steps.find-codeql.outputs.codeql-path }} CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
- name: Check QL compilation - name: Check QL compilation

View File

@@ -5,12 +5,9 @@ on:
branches: branches:
- main - main
- 'rc/**' - 'rc/**'
tags:
- 'codeql-cli/*'
pull_request: pull_request:
paths: paths:
- '.github/workflows/query-list.yml' - '.github/workflows/query-list.yml'
- '.github/actions/fetch-codeql/action.yml'
- 'misc/scripts/generate-code-scanning-query-list.py' - 'misc/scripts/generate-code-scanning-query-list.py'
jobs: jobs:
@@ -20,21 +17,33 @@ jobs:
steps: steps:
- name: Clone self (github/codeql) - name: Clone self (github/codeql)
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
path: codeql path: codeql
- name: Clone github/codeql-go
uses: actions/checkout@v2
with:
repository: 'github/codeql-go'
path: codeql-go
- name: Set up Python 3.8 - name: Set up Python 3.8
uses: actions/setup-python@v4 uses: actions/setup-python@v2
with: with:
python-version: 3.8 python-version: 3.8
- name: Download CodeQL CLI - name: Download CodeQL CLI
# Look under the `codeql` directory, as this is where we checked out the `github/codeql` repo uses: dsaltares/fetch-gh-release-asset@aa37ae5c44d3c9820bc12fe675e8670ecd93bd1c
uses: ./codeql/.github/actions/fetch-codeql with:
repo: "github/codeql-cli-binaries"
version: "latest"
file: "codeql-linux64.zip"
token: ${{ secrets.GITHUB_TOKEN }}
- name: Unzip CodeQL CLI
run: unzip -d codeql-cli codeql-linux64.zip
- name: Build code scanning query list - name: Build code scanning query list
run: | run: |
python codeql/misc/scripts/generate-code-scanning-query-list.py > code-scanning-query-list.csv PATH="$PATH:codeql-cli/codeql" python codeql/misc/scripts/generate-code-scanning-query-list.py > code-scanning-query-list.csv
- name: Upload code scanning query list - name: Upload code scanning query list
uses: actions/upload-artifact@v3 uses: actions/upload-artifact@v2
with: with:
name: code-scanning-query-list name: code-scanning-query-list
path: code-scanning-query-list.csv path: code-scanning-query-list.csv

View File

@@ -5,8 +5,6 @@ on:
paths: paths:
- "ruby/**" - "ruby/**"
- .github/workflows/ruby-build.yml - .github/workflows/ruby-build.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -14,8 +12,6 @@ on:
paths: paths:
- "ruby/**" - "ruby/**"
- .github/workflows/ruby-build.yml - .github/workflows/ruby-build.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -42,13 +38,13 @@ jobs:
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- name: Install GNU tar - name: Install GNU tar
if: runner.os == 'macOS' if: runner.os == 'macOS'
run: | run: |
brew install gnu-tar brew install gnu-tar
echo "/usr/local/opt/gnu-tar/libexec/gnubin" >> $GITHUB_PATH echo "/usr/local/opt/gnu-tar/libexec/gnubin" >> $GITHUB_PATH
- uses: actions/cache@v3 - uses: actions/cache@v2
with: with:
path: | path: |
~/.cargo/registry ~/.cargo/registry
@@ -66,17 +62,17 @@ jobs:
- name: Generate dbscheme - name: Generate dbscheme
if: ${{ matrix.os == 'ubuntu-latest' }} if: ${{ matrix.os == 'ubuntu-latest' }}
run: target/release/ruby-generator --dbscheme ql/lib/ruby.dbscheme --library ql/lib/codeql/ruby/ast/internal/TreeSitter.qll run: target/release/ruby-generator --dbscheme ql/lib/ruby.dbscheme --library ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
if: ${{ matrix.os == 'ubuntu-latest' }} if: ${{ matrix.os == 'ubuntu-latest' }}
with: with:
name: ruby.dbscheme name: ruby.dbscheme
path: ruby/ql/lib/ruby.dbscheme path: ruby/ql/lib/ruby.dbscheme
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
if: ${{ matrix.os == 'ubuntu-latest' }} if: ${{ matrix.os == 'ubuntu-latest' }}
with: with:
name: TreeSitter.qll name: TreeSitter.qll
path: ruby/ql/lib/codeql/ruby/ast/internal/TreeSitter.qll path: ruby/ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: extractor-${{ matrix.os }} name: extractor-${{ matrix.os }}
path: | path: |
@@ -90,18 +86,23 @@ jobs:
env: env:
CODEQL_THREADS: 4 # TODO: remove this once it's set by the CLI CODEQL_THREADS: 4 # TODO: remove this once it's set by the CLI
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- name: Fetch CodeQL - name: Fetch CodeQL
uses: ./.github/actions/fetch-codeql run: |
LATEST=$(gh release list --repo https://github.com/github/codeql-cli-binaries | cut -f 1 | grep -v beta | sort --version-sort | tail -1)
gh release download --repo https://github.com/github/codeql-cli-binaries --pattern codeql-linux64.zip "$LATEST"
unzip -q codeql-linux64.zip
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Build Query Pack - name: Build Query Pack
run: | run: |
codeql pack create ql/lib --output target/packs codeql/codeql pack create ql/lib --output target/packs
codeql pack install ql/src codeql/codeql pack install ql/src
codeql pack create ql/src --output target/packs codeql/codeql pack create ql/src --output target/packs
PACK_FOLDER=$(readlink -f target/packs/codeql/ruby-queries/*) PACK_FOLDER=$(readlink -f target/packs/codeql/ruby-queries/*)
codeql generate query-help --format=sarifv2.1.0 --output="${PACK_FOLDER}/rules.sarif" ql/src codeql/codeql generate query-help --format=sarifv2.1.0 --output="${PACK_FOLDER}/rules.sarif" ql/src
(cd ql/src; find queries \( -name '*.qhelp' -o -name '*.rb' -o -name '*.erb' \) -exec bash -c 'mkdir -p "'"${PACK_FOLDER}"'/$(dirname "{}")"' \; -exec cp "{}" "${PACK_FOLDER}/{}" \;) (cd ql/src; find queries \( -name '*.qhelp' -o -name '*.rb' -o -name '*.erb' \) -exec bash -c 'mkdir -p "'"${PACK_FOLDER}"'/$(dirname "{}")"' \; -exec cp "{}" "${PACK_FOLDER}/{}" \;)
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: codeql-ruby-queries name: codeql-ruby-queries
path: | path: |
@@ -112,20 +113,20 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
needs: [build, compile-queries] needs: [build, compile-queries]
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: ruby.dbscheme name: ruby.dbscheme
path: ruby/ruby path: ruby/ruby
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: extractor-ubuntu-latest name: extractor-ubuntu-latest
path: ruby/linux64 path: ruby/linux64
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: extractor-windows-latest name: extractor-windows-latest
path: ruby/win64 path: ruby/win64
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: extractor-macos-latest name: extractor-macos-latest
path: ruby/osx64 path: ruby/osx64
@@ -141,12 +142,12 @@ jobs:
cp win64/ruby-extractor.exe ruby/tools/win64/extractor.exe cp win64/ruby-extractor.exe ruby/tools/win64/extractor.exe
chmod +x ruby/tools/{linux64,osx64}/{autobuilder,extractor} chmod +x ruby/tools/{linux64,osx64}/{autobuilder,extractor}
zip -rq codeql-ruby.zip ruby zip -rq codeql-ruby.zip ruby
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: codeql-ruby-pack name: codeql-ruby-pack
path: ruby/codeql-ruby.zip path: ruby/codeql-ruby.zip
retention-days: 1 retention-days: 1
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: codeql-ruby-queries name: codeql-ruby-queries
path: ruby/qlpacks path: ruby/qlpacks
@@ -158,7 +159,7 @@ jobs:
] ]
}' > .codeqlmanifest.json }' > .codeqlmanifest.json
zip -rq codeql-ruby-bundle.zip .codeqlmanifest.json ruby qlpacks zip -rq codeql-ruby-bundle.zip .codeqlmanifest.json ruby qlpacks
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: codeql-ruby-bundle name: codeql-ruby-bundle
path: ruby/codeql-ruby-bundle.zip path: ruby/codeql-ruby-bundle.zip
@@ -176,17 +177,21 @@ jobs:
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
needs: [package] needs: [package]
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- name: Fetch CodeQL
uses: ./.github/actions/fetch-codeql
- uses: actions/checkout@v3
with: with:
repository: Shopify/example-ruby-app repository: Shopify/example-ruby-app
ref: 67a0decc5eb550f3a9228eda53925c3afd40dfe9 ref: 67a0decc5eb550f3a9228eda53925c3afd40dfe9
- name: Fetch CodeQL
shell: bash
run: |
LATEST=$(gh release list --repo https://github.com/github/codeql-cli-binaries | cut -f 1 | grep -v beta | sort --version-sort | tail -1)
gh release download --repo https://github.com/github/codeql-cli-binaries --pattern codeql.zip "$LATEST"
unzip -q codeql.zip
env:
GITHUB_TOKEN: ${{ github.token }}
working-directory: ${{ runner.temp }}
- name: Download Ruby bundle - name: Download Ruby bundle
uses: actions/download-artifact@v3 uses: actions/download-artifact@v2
with: with:
name: codeql-ruby-bundle name: codeql-ruby-bundle
path: ${{ runner.temp }} path: ${{ runner.temp }}
@@ -208,12 +213,12 @@ jobs:
- name: Run QL test - name: Run QL test
shell: bash shell: bash
run: | run: |
codeql test run --search-path "${{ runner.temp }}/ruby-bundle" --additional-packs "${{ runner.temp }}/ruby-bundle" . "${{ runner.temp }}/codeql/codeql" test run --search-path "${{ runner.temp }}/ruby-bundle" --additional-packs "${{ runner.temp }}/ruby-bundle" .
- name: Create database - name: Create database
shell: bash shell: bash
run: | run: |
codeql database create --search-path "${{ runner.temp }}/ruby-bundle" --language ruby --source-root . ../database "${{ runner.temp }}/codeql/codeql" database create --search-path "${{ runner.temp }}/ruby-bundle" --language ruby --source-root . ../database
- name: Analyze database - name: Analyze database
shell: bash shell: bash
run: | run: |
codeql database analyze --search-path "${{ runner.temp }}/ruby-bundle" --format=sarifv2.1.0 --output=out.sarif ../database ruby-code-scanning.qls "${{ runner.temp }}/codeql/codeql" database analyze --search-path "${{ runner.temp }}/ruby-bundle" --format=sarifv2.1.0 --output=out.sarif ../database ruby-code-scanning.qls

View File

@@ -27,14 +27,14 @@ jobs:
repo: [rails/rails, discourse/discourse, spree/spree, ruby/ruby] repo: [rails/rails, discourse/discourse, spree/spree, ruby/ruby]
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
- uses: ./ruby/actions/create-extractor-pack - uses: ./ruby/actions/create-extractor-pack
- name: Checkout ${{ matrix.repo }} - name: Checkout ${{ matrix.repo }}
uses: actions/checkout@v3 uses: actions/checkout@v2
with: with:
repository: ${{ matrix.repo }} repository: ${{ matrix.repo }}
path: ${{ github.workspace }}/repo path: ${{ github.workspace }}/repo
@@ -49,7 +49,7 @@ jobs:
run: | run: |
mkdir -p "stats/${{ matrix.repo }}" mkdir -p "stats/${{ matrix.repo }}"
codeql dataset measure --threads 4 --output "stats/${{ matrix.repo }}/stats.xml" "${{ runner.temp }}/database/db-ruby" codeql dataset measure --threads 4 --output "stats/${{ matrix.repo }}/stats.xml" "${{ runner.temp }}/database/db-ruby"
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: measurements name: measurements
path: stats path: stats
@@ -59,15 +59,15 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
needs: measure needs: measure
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: actions/download-artifact@v3 - uses: actions/download-artifact@v2
with: with:
name: measurements name: measurements
path: stats path: stats
- run: | - run: |
python -m pip install --user lxml python -m pip install --user lxml
find stats -name 'stats.xml' | sort | xargs python ruby/scripts/merge_stats.py --output ruby/ql/lib/ruby.dbscheme.stats --normalise ruby_tokeninfo find stats -name 'stats.xml' | sort | xargs python ruby/scripts/merge_stats.py --output ruby/ql/lib/ruby.dbscheme.stats --normalise ruby_tokeninfo
- uses: actions/upload-artifact@v3 - uses: actions/upload-artifact@v2
with: with:
name: ruby.dbscheme.stats name: ruby.dbscheme.stats
path: ruby/ql/lib/ruby.dbscheme.stats path: ruby/ql/lib/ruby.dbscheme.stats

View File

@@ -5,8 +5,6 @@ on:
paths: paths:
- "ruby/**" - "ruby/**"
- .github/workflows/ruby-qltest.yml - .github/workflows/ruby-qltest.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -14,8 +12,6 @@ on:
paths: paths:
- "ruby/**" - "ruby/**"
- .github/workflows/ruby-qltest.yml - .github/workflows/ruby-qltest.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -31,14 +27,14 @@ jobs:
qlformat: qlformat:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
- name: Check QL formatting - name: Check QL formatting
run: find ql "(" -name "*.ql" -or -name "*.qll" ")" -print0 | xargs -0 codeql query format --check-only run: find ql "(" -name "*.ql" -or -name "*.qll" ")" -print0 | xargs -0 codeql query format --check-only
qlcompile: qlcompile:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
- name: Check QL compilation - name: Check QL compilation
run: | run: |
@@ -48,7 +44,7 @@ jobs:
qlupgrade: qlupgrade:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
- name: Check DB upgrade scripts - name: Check DB upgrade scripts
run: | run: |
@@ -71,7 +67,7 @@ jobs:
matrix: matrix:
slice: ["1/2", "2/2"] slice: ["1/2", "2/2"]
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- uses: ./.github/actions/fetch-codeql - uses: ./.github/actions/fetch-codeql
- uses: ./ruby/actions/create-extractor-pack - uses: ./ruby/actions/create-extractor-pack
- name: Run QL tests - name: Run QL tests

View File

@@ -1,37 +0,0 @@
name: "Swift: Check code generation"
on:
pull_request:
paths:
- "swift/**"
- .github/workflows/swift-codegen.yml
- .github/actions/fetch-codeql/action.yml
branches:
- main
jobs:
codegen:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: bazelbuild/setup-bazelisk@v2
- uses: actions/setup-python@v3
- uses: pre-commit/action@v3.0.0
name: Check that python code is properly formatted
with:
extra_args: autopep8 --all-files
- name: Run unit tests
run: |
bazel test //swift/codegen/test --test_output=errors
- uses: pre-commit/action@v3.0.0
name: Check that QL generated code was checked in
with:
extra_args: swift-codegen --all-files
- name: Generate C++ files
run: |
bazel run //swift/codegen:codegen -- --generate=trap,cpp --cpp-output=$PWD/swift-generated-cpp-files
- uses: actions/upload-artifact@v3
with:
name: swift-generated-cpp-files
path: swift-generated-cpp-files/**

View File

@@ -1,35 +0,0 @@
name: "Swift: Run Integration Tests"
on:
pull_request:
paths:
- "swift/**"
- .github/workflows/swift-integration-tests.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches:
- main
defaults:
run:
working-directory: swift
jobs:
integration-tests:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os:
- ubuntu-20.04
# - macos-latest TODO
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: bazelbuild/setup-bazelisk@v2
- uses: actions/setup-python@v3
- name: Build Swift extractor
run: |
bazel run //swift:create-extractor-pack
- name: Run integration tests
run: |
python integration-tests/runner.py

View File

@@ -1,41 +0,0 @@
name: "Swift: Run QL Tests"
on:
pull_request:
paths:
- "swift/**"
- .github/workflows/swift-qltest.yml
- .github/actions/fetch-codeql/action.yml
- codeql-workspace.yml
branches:
- main
defaults:
run:
working-directory: swift
jobs:
qlformat:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- name: Check QL formatting
run: find ql "(" -name "*.ql" -or -name "*.qll" ")" -print0 | xargs -0 codeql query format --check-only
qltest:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os : [ubuntu-20.04, macos-latest]
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/fetch-codeql
- uses: bazelbuild/setup-bazelisk@v2
- name: Build Swift extractor
run: |
bazel run //swift:create-extractor-pack
- name: Run QL tests
run: |
codeql test run --threads=0 --ram 5000 --search-path "${{ github.workspace }}/swift/extractor-pack" --check-databases --check-unused-labels --check-repeated-labels --check-redefined-labels --check-use-before-definition ql/test
env:
GITHUB_TOKEN: ${{ github.token }}

View File

@@ -14,7 +14,7 @@ jobs:
sync: sync:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v2
- name: Check synchronized files - name: Check synchronized files
run: python config/sync-files.py run: python config/sync-files.py

View File

@@ -5,7 +5,6 @@ on:
paths: paths:
- "*/ql/*/change-notes/**/*" - "*/ql/*/change-notes/**/*"
- ".github/workflows/validate-change-notes.yml" - ".github/workflows/validate-change-notes.yml"
- ".github/actions/fetch-codeql/action.yml"
branches: branches:
- main - main
- "rc/*" - "rc/*"
@@ -13,14 +12,13 @@ on:
paths: paths:
- "*/ql/*/change-notes/**/*" - "*/ql/*/change-notes/**/*"
- ".github/workflows/validate-change-notes.yml" - ".github/workflows/validate-change-notes.yml"
- ".github/actions/fetch-codeql/action.yml"
jobs: jobs:
check-change-note: check-change-note:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v3 uses: actions/checkout@v2
- name: Setup CodeQL - name: Setup CodeQL
uses: ./.github/actions/fetch-codeql uses: ./.github/actions/fetch-codeql

30
.gitignore vendored
View File

@@ -9,7 +9,6 @@
# qltest projects and artifacts # qltest projects and artifacts
*/ql/test/**/*.testproj */ql/test/**/*.testproj
*/ql/test/**/*.actual */ql/test/**/*.actual
*/ql/test/**/go.sum
# Visual studio temporaries, except a file used by QL4VS # Visual studio temporaries, except a file used by QL4VS
.vs/* .vs/*
@@ -21,9 +20,6 @@
# python virtual environment folder # python virtual environment folder
.venv/ .venv/
# binary files created by pytest-cov
.coverage
# It's useful (though not required) to be able to unpack codeql in the ql checkout itself # It's useful (though not required) to be able to unpack codeql in the ql checkout itself
/codeql/ /codeql/
@@ -35,29 +31,5 @@ csharp/extractor/Semmle.Extraction.CSharp.Driver/Properties/launchSettings.json
# Compiled class file # Compiled class file
*.class *.class
# links created by bazel # links create by bazel
/bazel-* /bazel-*
# local bazel options
/local.bazelrc
# CLion project files
/.clwb
# Go build artifacts
go/build/*
# Go binaries
go/tools/bin
go/tools/linux64
go/tools/osx64
go/tools/win64
go/tools/tokenizer.jar
go/main
# node_modules folders except in the JS test suite
node_modules/
!/javascript/ql/test/**/node_modules/
# Temporary folders for working with generated models
.model-temp

View File

@@ -6,7 +6,6 @@ path_classifiers:
test: test:
- csharp/ql/src - csharp/ql/src
- csharp/ql/test - csharp/ql/test
- go/ql/test
- javascript/extractor/parser-tests - javascript/extractor/parser-tests
- javascript/extractor/tests - javascript/extractor/tests
- javascript/ql/src - javascript/ql/src
@@ -14,9 +13,6 @@ path_classifiers:
- python/ql/src - python/ql/src
- python/ql/test - python/ql/test
example:
- go/ql/src
queries: queries:
- include: "*" - include: "*"

View File

@@ -1,57 +1,29 @@
# See https://pre-commit.com for more information # See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks # See https://pre-commit.com/hooks.html for more hooks
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)
repos: repos:
- repo: https://github.com/pre-commit/pre-commit-hooks - repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0 rev: v3.2.0
hooks: hooks:
- id: trailing-whitespace - id: trailing-whitespace
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref) - id: end-of-file-fixer
- id: end-of-file-fixer
exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)
- repo: https://github.com/pre-commit/mirrors-clang-format - repo: local
rev: v13.0.1
hooks: hooks:
- id: clang-format - id: codeql-format
files: ^swift/.*\.(h|c|cpp)$
- repo: https://github.com/pre-commit/mirrors-autopep8
rev: v1.6.0
hooks:
- id: autopep8
files: ^swift/codegen/.*\.py
- repo: local
hooks:
- id: codeql-format
name: Fix QL file formatting name: Fix QL file formatting
files: \.qll?$ files: \.qll?$
language: system language: system
entry: codeql query format --in-place entry: codeql query format --in-place
- id: sync-files - id: sync-files
name: Fix files required to be identical name: Fix files required to be identical
files: \.(qll?|qhelp|swift)$
language: system language: system
entry: python3 config/sync-files.py --latest entry: python3 config/sync-files.py --latest
pass_filenames: false pass_filenames: false
- id: qhelp - id: qhelp
name: Check query help generation name: Check query help generation
files: \.qhelp$ files: \.qhelp$
language: system language: system
entry: python3 misc/scripts/check-qhelp.py entry: python3 misc/scripts/check-qhelp.py
- id: swift-codegen
name: Run Swift checked in code generation
files: ^swift/(codegen/|.*/generated/|ql/lib/(swift\.dbscheme$|codeql/swift/elements))
language: system
entry: bazel run //swift/codegen -- --quiet
pass_filenames: false
- id: swift-codegen-unit-tests
name: Run Swift code generation unit tests
files: ^swift/codegen/.*\.py$
language: system
entry: bazel test //swift/codegen/test
pass_filenames: false

View File

@@ -1,13 +1,9 @@
/cpp/ @github/codeql-c-analysis /cpp/ @github/codeql-c-analysis
/csharp/ @github/codeql-csharp /csharp/ @github/codeql-csharp
/go/ @github/codeql-go
/java/ @github/codeql-java /java/ @github/codeql-java
/javascript/ @github/codeql-javascript /javascript/ @github/codeql-javascript
/python/ @github/codeql-python /python/ @github/codeql-python
/ruby/ @github/codeql-ruby /ruby/ @github/codeql-ruby
/swift/ @github/codeql-c
/java/kotlin-extractor/ @github/codeql-kotlin
/java/kotlin-explorer/ @github/codeql-kotlin
# ML-powered queries # ML-powered queries
/javascript/ql/experimental/adaptivethreatmodeling/ @github/codeql-ml-powered-queries-reviewers /javascript/ql/experimental/adaptivethreatmodeling/ @github/codeql-ml-powered-queries-reviewers
@@ -28,18 +24,6 @@
# QL for QL reviewers # QL for QL reviewers
/ql/ @github/codeql-ql-for-ql-reviewers /ql/ @github/codeql-ql-for-ql-reviewers
# Bazel (excluding BUILD.bazel files) # Bazel
WORKSPACE.bazel @github/codeql-ci-reviewers **/*.bazel @github/codeql-ci-reviewers
**/*.bzl @github/codeql-ci-reviewers **/*.bzl @github/codeql-ci-reviewers
# Documentation etc
/*.md @github/code-scanning-product
/LICENSE @github/code-scanning-product
# Workflows
/.github/workflows/ @github/codeql-ci-reviewers
/.github/workflows/go-* @github/codeql-go
/.github/workflows/js-ml-tests.yml @github/codeql-ml-powered-queries-reviewers
/.github/workflows/ql-for-ql-* @github/codeql-ql-for-ql-reviewers
/.github/workflows/ruby-* @github/codeql-ruby
/.github/workflows/swift-* @github/codeql-c

View File

@@ -2,7 +2,7 @@
We welcome contributions to our CodeQL libraries and queries. Got an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request! Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE). We welcome contributions to our CodeQL libraries and queries. Got an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request! Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE).
There is lots of useful documentation to help you write queries, ranging from information about query file structure to tutorials for specific target languages. For more information on the documentation available, see [CodeQL queries](https://codeql.github.com/docs/writing-codeql-queries/codeql-queries) on [codeql.github.com](https://codeql.github.com). There is lots of useful documentation to help you write queries, ranging from information about query file structure to tutorials for specific target languages. For more information on the documentation available, see [CodeQL queries](https://help.semmle.com/QL/learn-ql/writing-queries/writing-queries.html) on [help.semmle.com](https://help.semmle.com).
## Change notes ## Change notes
@@ -40,7 +40,7 @@ If you have an idea for a query that you would like to share with other CodeQL u
3. **Formatting** 3. **Formatting**
- The queries and libraries must be autoformatted, for example using the "Format Document" command in [CodeQL for Visual Studio Code](https://codeql.github.com/docs/codeql-for-visual-studio-code/about-codeql-for-visual-studio-code). - The queries and libraries must be autoformatted, for example using the "Format Document" command in [CodeQL for Visual Studio Code](https://help.semmle.com/codeql/codeql-for-vscode/procedures/about-codeql-for-vscode.html).
If you prefer, you can either: If you prefer, you can either:
1. install the [pre-commit framework](https://pre-commit.com/) and install the configured hooks on this repo via `pre-commit install`, or 1. install the [pre-commit framework](https://pre-commit.com/) and install the configured hooks on this repo via `pre-commit install`, or

View File

@@ -1,6 +1,6 @@
# CodeQL # CodeQL
This open source repository contains the standard CodeQL libraries and queries that power [GitHub Advanced Security](https://github.com/features/security/code) and the other application security products that [GitHub](https://github.com/features/security/) makes available to its customers worldwide. This open source repository contains the standard CodeQL libraries and queries that power [GitHub Advanced Security](https://github.com/features/security/code) and the other application security products that [GitHub](https://github.com/features/security/) makes available to its customers worldwide. For the queries, libraries, and extractor that power Go analysis, visit the [CodeQL for Go repository](https://github.com/github/codeql-go).
## How do I learn CodeQL and run queries? ## How do I learn CodeQL and run queries?
@@ -13,9 +13,7 @@ We welcome contributions to our standard library and standard checks. Do you hav
## License ## License
The code in this repository is licensed under the [MIT License](LICENSE) by [GitHub](https://github.com). The code in this repository is licensed under the [MIT License](LICENSE) by [GitHub](https://github.com). The use of CodeQL on open source code is licensed under specific [Terms & Conditions](https://securitylab.github.com/tools/codeql/license/) UNLESS you have a commercial license in place. If you'd like to use CodeQL with a commercial codebase, please [contact us](https://github.com/enterprise/contact) for further help.
The CodeQL CLI (including the CodeQL engine) is hosted in a [different repository](https://github.com/github/codeql-cli-binaries) and is [licensed separately](https://github.com/github/codeql-cli-binaries/blob/main/LICENSE.md). If you'd like to use the CodeQL CLI to analyze closed-source code, you will need a separate commercial license; please [contact us](https://github.com/enterprise/contact) for further help.
## Visual Studio Code integration ## Visual Studio Code integration

View File

@@ -1,12 +1,2 @@
# Please notice that any bazel targets and definitions in this repository are currently experimental # Please notice that any bazel targets and definitions in this repository are currently experimental
# and for internal use only. # and for internal use only.
workspace(name = "codeql")
load("//misc/bazel:workspace.bzl", "codeql_workspace")
codeql_workspace()
load("//misc/bazel:workspace_deps.bzl", "codeql_workspace_deps")
codeql_workspace_deps()

View File

@@ -1,32 +0,0 @@
provide:
- "*/ql/src/qlpack.yml"
- "*/ql/lib/qlpack.yml"
- "*/ql/test/qlpack.yml"
- "*/ql/examples/qlpack.yml"
- "*/ql/consistency-queries/qlpack.yml"
- "cpp/ql/test/query-tests/Security/CWE/CWE-190/semmle/tainted/qlpack.yml"
- "go/ql/config/legacy-support/qlpack.yml"
- "go/build/codeql-extractor-go/codeql-extractor.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/lib/qlpack.yml"
# This pack is explicitly excluded from the workspace since most users
# will want to use a version of this pack from the package cache. Internal
# users can uncomment the following line and place a custom ML model
# in the corresponding pack to test a custom ML model within their local
# checkout.
# - "javascript/ql/experimental/adaptivethreatmodeling/model/qlpack.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/modelbuilding/qlpack.yml"
- "javascript/ql/experimental/adaptivethreatmodeling/src/qlpack.yml"
- "csharp/ql/campaigns/Solorigate/lib/qlpack.yml"
- "csharp/ql/campaigns/Solorigate/src/qlpack.yml"
- "csharp/ql/campaigns/Solorigate/test/qlpack.yml"
- "misc/legacy-support/*/qlpack.yml"
- "misc/suite-helpers/qlpack.yml"
- "ruby/extractor-pack/codeql-extractor.yml"
- "swift/extractor-pack/codeql-extractor.yml"
- "ql/extractor-pack/codeql-extractor.ym"
versionPolicies:
default:
requireChangeNotes: true
committedPrereleaseSuffix: dev
committedVersion: nextPatchRelease

View File

@@ -22,15 +22,13 @@
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl3.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl3.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl4.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl4.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl5.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImpl5.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplForContentDataFlow.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl.qll", "python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl2.qll", "python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl2.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl3.qll", "python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl3.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl4.qll", "python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImpl4.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl2.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImpl2.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplForLibraries.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplForLibraries.qll"
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImpl.qll"
], ],
"DataFlow Java/C++/C#/Python Common": [ "DataFlow Java/C++/C#/Python Common": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplCommon.qll", "java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplCommon.qll",
@@ -38,8 +36,7 @@
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImplCommon.qll", "cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImplCommon.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplCommon.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplCommon.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll", "python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplCommon.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplCommon.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplCommon.qll"
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImplCommon.qll"
], ],
"TaintTracking::Configuration Java/C++/C#/Python": [ "TaintTracking::Configuration Java/C++/C#/Python": [
"cpp/ql/lib/semmle/code/cpp/dataflow/internal/tainttracking1/TaintTrackingImpl.qll", "cpp/ql/lib/semmle/code/cpp/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
@@ -54,14 +51,12 @@
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/tainttracking5/TaintTrackingImpl.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/tainttracking5/TaintTrackingImpl.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/tainttracking1/TaintTrackingImpl.qll", "java/ql/lib/semmle/code/java/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/tainttracking2/TaintTrackingImpl.qll", "java/ql/lib/semmle/code/java/dataflow/internal/tainttracking2/TaintTrackingImpl.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/tainttracking3/TaintTrackingImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/tainttracking1/TaintTrackingImpl.qll", "python/ql/lib/semmle/python/dataflow/new/internal/tainttracking1/TaintTrackingImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/tainttracking2/TaintTrackingImpl.qll", "python/ql/lib/semmle/python/dataflow/new/internal/tainttracking2/TaintTrackingImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/tainttracking3/TaintTrackingImpl.qll", "python/ql/lib/semmle/python/dataflow/new/internal/tainttracking3/TaintTrackingImpl.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/tainttracking4/TaintTrackingImpl.qll", "python/ql/lib/semmle/python/dataflow/new/internal/tainttracking4/TaintTrackingImpl.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/tainttracking1/TaintTrackingImpl.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/tainttracking1/TaintTrackingImpl.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/tainttrackingforlibraries/TaintTrackingImpl.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/tainttrackingforlibraries/TaintTrackingImpl.qll"
"swift/ql/lib/codeql/swift/dataflow/internal/tainttracking1/TaintTrackingImpl.qll"
], ],
"DataFlow Java/C++/C#/Python Consistency checks": [ "DataFlow Java/C++/C#/Python Consistency checks": [
"java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplConsistency.qll", "java/ql/lib/semmle/code/java/dataflow/internal/DataFlowImplConsistency.qll",
@@ -69,14 +64,12 @@
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImplConsistency.qll", "cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowImplConsistency.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplConsistency.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/DataFlowImplConsistency.qll",
"python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplConsistency.qll", "python/ql/lib/semmle/python/dataflow/new/internal/DataFlowImplConsistency.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplConsistency.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/DataFlowImplConsistency.qll"
"swift/ql/lib/codeql/swift/dataflow/internal/DataFlowImplConsistency.qll"
], ],
"DataFlow Java/C# Flow Summaries": [ "DataFlow Java/C# Flow Summaries": [
"java/ql/lib/semmle/code/java/dataflow/internal/FlowSummaryImpl.qll", "java/ql/lib/semmle/code/java/dataflow/internal/FlowSummaryImpl.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/FlowSummaryImpl.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/FlowSummaryImpl.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/FlowSummaryImpl.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/FlowSummaryImpl.qll"
"swift/ql/lib/codeql/swift/dataflow/internal/FlowSummaryImpl.qll"
], ],
"SsaReadPosition Java/C#": [ "SsaReadPosition Java/C#": [
"java/ql/lib/semmle/code/java/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll", "java/ql/lib/semmle/code/java/dataflow/internal/rangeanalysis/SsaReadPositionCommon.qll",
@@ -390,10 +383,7 @@
"csharp/ql/test/TestUtilities/InlineExpectationsTest.qll", "csharp/ql/test/TestUtilities/InlineExpectationsTest.qll",
"java/ql/test/TestUtilities/InlineExpectationsTest.qll", "java/ql/test/TestUtilities/InlineExpectationsTest.qll",
"python/ql/test/TestUtilities/InlineExpectationsTest.qll", "python/ql/test/TestUtilities/InlineExpectationsTest.qll",
"ruby/ql/test/TestUtilities/InlineExpectationsTest.qll", "ruby/ql/test/TestUtilities/InlineExpectationsTest.qll"
"ql/ql/test/TestUtilities/InlineExpectationsTest.qll",
"go/ql/test/TestUtilities/InlineExpectationsTest.qll",
"swift/ql/test/TestUtilities/InlineExpectationsTest.qll"
], ],
"C++ ExternalAPIs": [ "C++ ExternalAPIs": [
"cpp/ql/src/Security/CWE/CWE-020/ExternalAPIs.qll", "cpp/ql/src/Security/CWE/CWE-020/ExternalAPIs.qll",
@@ -454,11 +444,11 @@
"python/ql/src/Lexical/CommentedOutCodeReferences.inc.qhelp" "python/ql/src/Lexical/CommentedOutCodeReferences.inc.qhelp"
], ],
"IDE Contextual Queries": [ "IDE Contextual Queries": [
"cpp/ql/lib/IDEContextual.qll", "cpp/ql/src/IDEContextual.qll",
"csharp/ql/lib/IDEContextual.qll", "csharp/ql/src/IDEContextual.qll",
"java/ql/lib/IDEContextual.qll", "java/ql/src/IDEContextual.qll",
"javascript/ql/lib/IDEContextual.qll", "javascript/ql/src/IDEContextual.qll",
"python/ql/lib/analysis/IDEContextual.qll" "python/ql/src/analysis/IDEContextual.qll"
], ],
"SSA C#": [ "SSA C#": [
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/SsaImplCommon.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/SsaImplCommon.qll",
@@ -466,8 +456,7 @@
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/basessa/SsaImplCommon.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/basessa/SsaImplCommon.qll",
"csharp/ql/lib/semmle/code/cil/internal/SsaImplCommon.qll", "csharp/ql/lib/semmle/code/cil/internal/SsaImplCommon.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/SsaImplCommon.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/SsaImplCommon.qll",
"cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/SsaImplCommon.qll", "cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/SsaImplCommon.qll"
"swift/ql/lib/codeql/swift/dataflow/internal/SsaImplCommon.qll"
], ],
"CryptoAlgorithms Python/JS/Ruby": [ "CryptoAlgorithms Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/CryptoAlgorithms.qll", "javascript/ql/lib/semmle/javascript/security/CryptoAlgorithms.qll",
@@ -484,44 +473,29 @@
"python/ql/lib/semmle/python/security/internal/SensitiveDataHeuristics.qll", "python/ql/lib/semmle/python/security/internal/SensitiveDataHeuristics.qll",
"ruby/ql/lib/codeql/ruby/security/internal/SensitiveDataHeuristics.qll" "ruby/ql/lib/codeql/ruby/security/internal/SensitiveDataHeuristics.qll"
], ],
"ReDoS Util Python/JS/Ruby/Java": [ "ReDoS Util Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/regexp/NfaUtils.qll", "javascript/ql/lib/semmle/javascript/security/performance/ReDoSUtil.qll",
"python/ql/lib/semmle/python/security/regexp/NfaUtils.qll", "python/ql/lib/semmle/python/security/performance/ReDoSUtil.qll",
"ruby/ql/lib/codeql/ruby/security/regexp/NfaUtils.qll", "ruby/ql/lib/codeql/ruby/security/performance/ReDoSUtil.qll"
"java/ql/lib/semmle/code/java/security/regexp/NfaUtils.qll"
], ],
"ReDoS Exponential Python/JS/Ruby/Java": [ "ReDoS Exponential Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/regexp/ExponentialBackTracking.qll", "javascript/ql/lib/semmle/javascript/security/performance/ExponentialBackTracking.qll",
"python/ql/lib/semmle/python/security/regexp/ExponentialBackTracking.qll", "python/ql/lib/semmle/python/security/performance/ExponentialBackTracking.qll",
"ruby/ql/lib/codeql/ruby/security/regexp/ExponentialBackTracking.qll", "ruby/ql/lib/codeql/ruby/security/performance/ExponentialBackTracking.qll"
"java/ql/lib/semmle/code/java/security/regexp/ExponentialBackTracking.qll"
], ],
"ReDoS Polynomial Python/JS/Ruby/Java": [ "ReDoS Polynomial Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/regexp/SuperlinearBackTracking.qll", "javascript/ql/lib/semmle/javascript/security/performance/SuperlinearBackTracking.qll",
"python/ql/lib/semmle/python/security/regexp/SuperlinearBackTracking.qll", "python/ql/lib/semmle/python/security/performance/SuperlinearBackTracking.qll",
"ruby/ql/lib/codeql/ruby/security/regexp/SuperlinearBackTracking.qll", "ruby/ql/lib/codeql/ruby/security/performance/SuperlinearBackTracking.qll"
"java/ql/lib/semmle/code/java/security/regexp/SuperlinearBackTracking.qll"
],
"RegexpMatching Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/regexp/RegexpMatching.qll",
"python/ql/lib/semmle/python/security/regexp/RegexpMatching.qll",
"ruby/ql/lib/codeql/ruby/security/regexp/RegexpMatching.qll"
], ],
"BadTagFilterQuery Python/JS/Ruby": [ "BadTagFilterQuery Python/JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/BadTagFilterQuery.qll", "javascript/ql/lib/semmle/javascript/security/BadTagFilterQuery.qll",
"python/ql/lib/semmle/python/security/BadTagFilterQuery.qll", "python/ql/lib/semmle/python/security/BadTagFilterQuery.qll",
"ruby/ql/lib/codeql/ruby/security/BadTagFilterQuery.qll" "ruby/ql/lib/codeql/ruby/security/BadTagFilterQuery.qll"
], ],
"OverlyLargeRange Python/JS/Ruby/Java": [
"javascript/ql/lib/semmle/javascript/security/OverlyLargeRangeQuery.qll",
"python/ql/lib/semmle/python/security/OverlyLargeRangeQuery.qll",
"ruby/ql/lib/codeql/ruby/security/OverlyLargeRangeQuery.qll",
"java/ql/lib/semmle/code/java/security/OverlyLargeRangeQuery.qll"
],
"CFG": [ "CFG": [
"csharp/ql/lib/semmle/code/csharp/controlflow/internal/ControlFlowGraphImplShared.qll", "csharp/ql/lib/semmle/code/csharp/controlflow/internal/ControlFlowGraphImplShared.qll",
"ruby/ql/lib/codeql/ruby/controlflow/internal/ControlFlowGraphImplShared.qll", "ruby/ql/lib/codeql/ruby/controlflow/internal/ControlFlowGraphImplShared.qll"
"swift/ql/lib/codeql/swift/controlflow/internal/ControlFlowGraphImplShared.qll"
], ],
"TypeTracker": [ "TypeTracker": [
"python/ql/lib/semmle/python/dataflow/new/internal/TypeTracker.qll", "python/ql/lib/semmle/python/dataflow/new/internal/TypeTracker.qll",
@@ -539,9 +513,7 @@
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/AccessPathSyntax.qll", "csharp/ql/lib/semmle/code/csharp/dataflow/internal/AccessPathSyntax.qll",
"java/ql/lib/semmle/code/java/dataflow/internal/AccessPathSyntax.qll", "java/ql/lib/semmle/code/java/dataflow/internal/AccessPathSyntax.qll",
"javascript/ql/lib/semmle/javascript/frameworks/data/internal/AccessPathSyntax.qll", "javascript/ql/lib/semmle/javascript/frameworks/data/internal/AccessPathSyntax.qll",
"ruby/ql/lib/codeql/ruby/dataflow/internal/AccessPathSyntax.qll", "ruby/ql/lib/codeql/ruby/dataflow/internal/AccessPathSyntax.qll"
"python/ql/lib/semmle/python/frameworks/data/internal/AccessPathSyntax.qll",
"swift/ql/lib/codeql/swift/dataflow/internal/AccessPathSyntax.qll"
], ],
"IncompleteUrlSubstringSanitization": [ "IncompleteUrlSubstringSanitization": [
"javascript/ql/src/Security/CWE-020/IncompleteUrlSubstringSanitization.qll", "javascript/ql/src/Security/CWE-020/IncompleteUrlSubstringSanitization.qll",
@@ -559,8 +531,7 @@
], ],
"ApiGraphModels": [ "ApiGraphModels": [
"javascript/ql/lib/semmle/javascript/frameworks/data/internal/ApiGraphModels.qll", "javascript/ql/lib/semmle/javascript/frameworks/data/internal/ApiGraphModels.qll",
"ruby/ql/lib/codeql/ruby/frameworks/data/internal/ApiGraphModels.qll", "ruby/ql/lib/codeql/ruby/frameworks/data/internal/ApiGraphModels.qll"
"python/ql/lib/semmle/python/frameworks/data/internal/ApiGraphModels.qll"
], ],
"TaintedFormatStringQuery Ruby/JS": [ "TaintedFormatStringQuery Ruby/JS": [
"javascript/ql/lib/semmle/javascript/security/dataflow/TaintedFormatStringQuery.qll", "javascript/ql/lib/semmle/javascript/security/dataflow/TaintedFormatStringQuery.qll",
@@ -577,29 +548,5 @@
"HttpToFileAccessCustomizations JS/Ruby": [ "HttpToFileAccessCustomizations JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/dataflow/HttpToFileAccessCustomizations.qll", "javascript/ql/lib/semmle/javascript/security/dataflow/HttpToFileAccessCustomizations.qll",
"ruby/ql/lib/codeql/ruby/security/HttpToFileAccessCustomizations.qll" "ruby/ql/lib/codeql/ruby/security/HttpToFileAccessCustomizations.qll"
],
"Typo database": [
"javascript/ql/src/Expressions/TypoDatabase.qll",
"ql/ql/src/codeql_ql/style/TypoDatabase.qll"
],
"Swift declarations test file": [
"swift/ql/test/extractor-tests/declarations/declarations.swift",
"swift/ql/test/library-tests/parent/declarations.swift"
],
"Swift statements test file": [
"swift/ql/test/extractor-tests/statements/statements.swift",
"swift/ql/test/library-tests/parent/statements.swift"
],
"Swift expressions test file": [
"swift/ql/test/extractor-tests/expressions/expressions.swift",
"swift/ql/test/library-tests/parent/expressions.swift"
],
"Swift patterns test file": [
"swift/ql/test/extractor-tests/patterns/patterns.swift",
"swift/ql/test/library-tests/parent/patterns.swift"
],
"IncompleteMultiCharacterSanitization JS/Ruby": [
"javascript/ql/lib/semmle/javascript/security/IncompleteMultiCharacterSanitizationQuery.qll",
"ruby/ql/lib/codeql/ruby/security/IncompleteMultiCharacterSanitizationQuery.qll"
] ]
} }

View File

@@ -1 +0,0 @@
# this empty file adds the repo root to PYTHON_PATH when running pytest

View File

@@ -1,3 +0,0 @@
description: Add relation for tracking C++ braced initializers
compatibility: full
braced_initialisers.rel: delete

View File

@@ -1,17 +0,0 @@
class Expr extends @expr {
string toString() { none() }
}
class Location extends @location_expr {
string toString() { none() }
}
predicate isExprWithNewBuiltin(Expr expr) {
exists(int kind | exprs(expr, kind, _) | 330 <= kind and kind <= 334)
}
from Expr expr, int kind, int kind_new, Location location
where
exprs(expr, kind, location) and
if isExprWithNewBuiltin(expr) then kind_new = 0 else kind_new = kind
select expr, kind_new, location

View File

@@ -1,3 +0,0 @@
description: Add new builtin operations
compatibility: partial
exprs.rel: run exprs.qlo

View File

@@ -1,17 +0,0 @@
class AttributeArgument extends @attribute_arg {
string toString() { none() }
}
class Attribute extends @attribute {
string toString() { none() }
}
class LocationDefault extends @location_default {
string toString() { none() }
}
from AttributeArgument arg, int kind, Attribute attr, int index, LocationDefault location
where
attribute_args(arg, kind, attr, index, location) and
not arg instanceof @attribute_arg_constant_expr
select arg, kind, attr, index, location

View File

@@ -1,4 +0,0 @@
description: Support all constant attribute arguments
compatibility: backwards
attribute_arg_constant.rel: delete
attribute_args.rel: run attribute_args.qlo

View File

@@ -1,13 +0,0 @@
class Expr extends @expr {
string toString() { none() }
}
class Location extends @location_expr {
string toString() { none() }
}
from Expr expr, int kind, int kind_new, Location location
where
exprs(expr, kind, location) and
if expr instanceof @blockassignexpr then kind_new = 0 else kind_new = kind
select expr, kind_new, location

View File

@@ -1,3 +0,0 @@
description: Support block assignment
compatibility: partial
exprs.rel: run exprs.qlo

View File

@@ -1,21 +0,0 @@
class Element extends @element {
string toString() { none() }
}
class Expr extends @expr {
string toString() { none() }
}
class Stmt extends @stmt {
string toString() { none() }
}
predicate isStmtWithInitializer(Stmt stmt) {
exists(int kind | stmts(stmt, kind, _) | kind = 2 or kind = 11 or kind = 35)
}
from Expr child, int index, int index_new, Element parent
where
exprparents(child, index, parent) and
if isStmtWithInitializer(parent) then index_new = index - 1 else index_new = index
select child, index_new, parent

View File

@@ -1,22 +0,0 @@
class Element extends @element {
string toString() { none() }
}
class Stmt extends @stmt {
string toString() { none() }
}
predicate isStmtWithInitializer(Stmt stmt) {
exists(int kind | stmts(stmt, kind, _) | kind = 2 or kind = 11 or kind = 35)
}
from Stmt child, int index, int index_new, Element parent
where
stmtparents(child, index, parent) and
(
not isStmtWithInitializer(parent)
or
index > 0
) and
if isStmtWithInitializer(parent) then index_new = index - 1 else index_new = index
select child, index_new, parent

View File

@@ -1,6 +0,0 @@
description: Support C++17 if and switch initializers
compatibility: partial
if_initialization.rel: delete
switch_initialization.rel: delete
exprparents.rel: run exprparents.qlo
stmtparents.rel: run stmtparents.qlo

View File

@@ -1,3 +0,0 @@
description: Add relation for orphaned local variables
compatibility: full
orphaned_variables.rel: delete

View File

@@ -1,121 +1,3 @@
## 0.3.5
## 0.3.4
### Deprecated APIs
* Many classes/predicates/modules with upper-case acronyms in their name have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
### New Features
* Added support for getting the link targets of global and namespace variables.
* Added a `BlockAssignExpr` class, which models a `memcpy`-like operation used in compiler generated copy/move constructors and assignment operations.
### Minor Analysis Improvements
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.
## 0.3.3
### New Features
* Added a predicate `getValueConstant` to `AttributeArgument` that yields the argument value as an `Expr` when the value is a constant expression.
* A new class predicate `MustFlowConfiguration::allowInterproceduralFlow` has been added to the `semmle.code.cpp.ir.dataflow.MustFlow` library. The new predicate can be overridden to disable interprocedural flow.
* Added subclasses of `BuiltInOperations` for `__builtin_bit_cast`, `__builtin_shuffle`, `__has_unique_object_representations`, `__is_aggregate`, and `__is_assignable`.
### Major Analysis Improvements
* The IR dataflow library now includes flow through global variables. This enables new findings in many scenarios.
## 0.3.2
### Bug Fixes
* Under certain circumstances a variable declaration that is not also a definition could be associated with a `Variable` that did not have the definition as a `VariableDeclarationEntry`. This is now fixed, and a unique `Variable` will exist that has both the declaration and the definition as a `VariableDeclarationEntry`.
## 0.3.1
### Minor Analysis Improvements
* `AnalysedExpr::isNullCheck` and `AnalysedExpr::isValidCheck` have been updated to handle variable accesses on the left-hand side of the C++ logical "and", and variable declarations in conditions.
## 0.3.0
### Deprecated APIs
* The `BarrierGuard` class has been deprecated. Such barriers and sanitizers can now instead be created using the new `BarrierGuard` parameterized module.
### Bug Fixes
* `UserType.getADeclarationEntry()` now yields all forward declarations when the user type is a `class`, `struct`, or `union`.
## 0.2.3
### New Features
* An `isBraced` predicate was added to the `Initializer` class which holds when a C++ braced initializer was used in the initialization.
## 0.2.2
### Deprecated APIs
* The `AnalysedString` class in the `StringAnalysis` module has been replaced with `AnalyzedString`, to follow our style guide. The old name still exists as a deprecated alias.
### New Features
* A `getInitialization` predicate was added to the `ConstexprIfStmt`, `IfStmt`, and `SwitchStmt` classes that yields the C++17-style initializer of the `if` or `switch` statement when it exists.
## 0.2.1
## 0.2.0
### Breaking Changes
* The signature of `allowImplicitRead` on `DataFlow::Configuration` and `TaintTracking::Configuration` has changed from `allowImplicitRead(DataFlow::Node node, DataFlow::Content c)` to `allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c)`.
### Minor Analysis Improvements
* More Windows pool allocation functions are now detected as `AllocationFunction`s.
* The `semmle.code.cpp.commons.Buffer` library has been enhanced to handle array members of classes that do not specify a size.
## 0.1.0
### Breaking Changes
* The recently added flow-state versions of `isBarrierIn`, `isBarrierOut`, `isSanitizerIn`, and `isSanitizerOut` in the data flow and taint tracking libraries have been removed.
### New Features
* A new library `semmle.code.cpp.security.PrivateData` has been added. The new library heuristically detects variables and functions dealing with sensitive private data, such as e-mail addresses and credit card numbers.
### Minor Analysis Improvements
* The `semmle.code.cpp.security.SensitiveExprs` library has been enhanced with some additional rules for detecting credentials.
## 0.0.13
## 0.0.12
### Breaking Changes
* The flow state variants of `isBarrier` and `isAdditionalFlowStep` are no longer exposed in the taint tracking library. The `isSanitizer` and `isAdditionalTaintStep` predicates should be used instead.
### Deprecated APIs
* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
### New Features
* The data flow and taint tracking libraries have been extended with versions of `isBarrierIn`, `isBarrierOut`, and `isBarrierGuard`, respectively `isSanitizerIn`, `isSanitizerOut`, and `isSanitizerGuard`, that support flow states.
### Minor Analysis Improvements
* `DefaultOptions::exits` now holds for C11 functions with the `_Noreturn` or `noreturn` specifier.
* `hasImplicitCopyConstructor` and `hasImplicitCopyAssignmentOperator` now correctly handle implicitly-deleted operators in templates.
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.
## 0.0.11 ## 0.0.11
### Minor Analysis Improvements ### Minor Analysis Improvements

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.

View File

@@ -0,0 +1,5 @@
---
category: deprecated
---
* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* `hasImplicitCopyConstructor` and `hasImplicitCopyAssignmentOperator` now correctly handle implicitly-deleted operators in templates.

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* `DefaultOptions::exits` now holds for C11 functions with the `_Noreturn` or `noreturn` specifier.

View File

@@ -0,0 +1,4 @@
---
category: feature
---
* The data flow and taint tracking libraries have been extended with versions of `isBarrierIn`, `isBarrierOut`, and `isBarrierGuard`, respectively `isSanitizerIn`, `isSanitizerOut`, and `isSanitizerGuard`, that support flow states.

View File

@@ -0,0 +1,4 @@
---
category: breaking
---
* The flow state variants of `isBarrier` and `isAdditionalFlowStep` are no longer exposed in the taint tracking library. The `isSanitizer` and `isAdditionalTaintStep` predicates should be used instead.

View File

@@ -0,0 +1,4 @@
---
category: feature
---
* A new library `semmle.code.cpp.security.PrivateData` has been added. The new library heuristically detects variables and functions dealing with sensitive private data, such as e-mail addresses and credit card numbers.

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* The `semmle.code.cpp.security.SensitiveExprs` library has been enhanced with some additional rules for detecting credentials.

View File

@@ -1,20 +0,0 @@
## 0.0.12
### Breaking Changes
* The flow state variants of `isBarrier` and `isAdditionalFlowStep` are no longer exposed in the taint tracking library. The `isSanitizer` and `isAdditionalTaintStep` predicates should be used instead.
### Deprecated APIs
* Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
### New Features
* The data flow and taint tracking libraries have been extended with versions of `isBarrierIn`, `isBarrierOut`, and `isBarrierGuard`, respectively `isSanitizerIn`, `isSanitizerOut`, and `isSanitizerGuard`, that support flow states.
### Minor Analysis Improvements
* `DefaultOptions::exits` now holds for C11 functions with the `_Noreturn` or `noreturn` specifier.
* `hasImplicitCopyConstructor` and `hasImplicitCopyAssignmentOperator` now correctly handle implicitly-deleted operators in templates.
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.

View File

@@ -1 +0,0 @@
## 0.0.13

View File

@@ -1,13 +0,0 @@
## 0.1.0
### Breaking Changes
* The recently added flow-state versions of `isBarrierIn`, `isBarrierOut`, `isSanitizerIn`, and `isSanitizerOut` in the data flow and taint tracking libraries have been removed.
### New Features
* A new library `semmle.code.cpp.security.PrivateData` has been added. The new library heuristically detects variables and functions dealing with sensitive private data, such as e-mail addresses and credit card numbers.
### Minor Analysis Improvements
* The `semmle.code.cpp.security.SensitiveExprs` library has been enhanced with some additional rules for detecting credentials.

View File

@@ -1,10 +0,0 @@
## 0.2.0
### Breaking Changes
* The signature of `allowImplicitRead` on `DataFlow::Configuration` and `TaintTracking::Configuration` has changed from `allowImplicitRead(DataFlow::Node node, DataFlow::Content c)` to `allowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c)`.
### Minor Analysis Improvements
* More Windows pool allocation functions are now detected as `AllocationFunction`s.
* The `semmle.code.cpp.commons.Buffer` library has been enhanced to handle array members of classes that do not specify a size.

View File

@@ -1 +0,0 @@
## 0.2.1

View File

@@ -1,9 +0,0 @@
## 0.2.2
### Deprecated APIs
* The `AnalysedString` class in the `StringAnalysis` module has been replaced with `AnalyzedString`, to follow our style guide. The old name still exists as a deprecated alias.
### New Features
* A `getInitialization` predicate was added to the `ConstexprIfStmt`, `IfStmt`, and `SwitchStmt` classes that yields the C++17-style initializer of the `if` or `switch` statement when it exists.

View File

@@ -1,5 +0,0 @@
## 0.2.3
### New Features
* An `isBraced` predicate was added to the `Initializer` class which holds when a C++ braced initializer was used in the initialization.

View File

@@ -1,9 +0,0 @@
## 0.3.0
### Deprecated APIs
* The `BarrierGuard` class has been deprecated. Such barriers and sanitizers can now instead be created using the new `BarrierGuard` parameterized module.
### Bug Fixes
* `UserType.getADeclarationEntry()` now yields all forward declarations when the user type is a `class`, `struct`, or `union`.

View File

@@ -1,5 +0,0 @@
## 0.3.1
### Minor Analysis Improvements
* `AnalysedExpr::isNullCheck` and `AnalysedExpr::isValidCheck` have been updated to handle variable accesses on the left-hand side of the C++ logical "and", and variable declarations in conditions.

View File

@@ -1,5 +0,0 @@
## 0.3.2
### Bug Fixes
* Under certain circumstances a variable declaration that is not also a definition could be associated with a `Variable` that did not have the definition as a `VariableDeclarationEntry`. This is now fixed, and a unique `Variable` will exist that has both the declaration and the definition as a `VariableDeclarationEntry`.

View File

@@ -1,11 +0,0 @@
## 0.3.3
### New Features
* Added a predicate `getValueConstant` to `AttributeArgument` that yields the argument value as an `Expr` when the value is a constant expression.
* A new class predicate `MustFlowConfiguration::allowInterproceduralFlow` has been added to the `semmle.code.cpp.ir.dataflow.MustFlow` library. The new predicate can be overridden to disable interprocedural flow.
* Added subclasses of `BuiltInOperations` for `__builtin_bit_cast`, `__builtin_shuffle`, `__has_unique_object_representations`, `__is_aggregate`, and `__is_assignable`.
### Major Analysis Improvements
* The IR dataflow library now includes flow through global variables. This enables new findings in many scenarios.

View File

@@ -1,15 +0,0 @@
## 0.3.4
### Deprecated APIs
* Many classes/predicates/modules with upper-case acronyms in their name have been renamed to follow our style-guide.
The old name still exists as a deprecated alias.
### New Features
* Added support for getting the link targets of global and namespace variables.
* Added a `BlockAssignExpr` class, which models a `memcpy`-like operation used in compiler generated copy/move constructors and assignment operations.
### Minor Analysis Improvements
* All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.

Some files were not shown because too many files have changed in this diff Show More