End-to-end demo of CodeQL command line usage
-
Want to run analyses (command line use - github)
-
Get collection of databases (already handy)
-
Get https://github.com/rvermeulen/codeql-workshop-vulnerable-linux-driver
cd ~/local git clone git@github.com:rvermeulen/codeql-workshop-vulnerable-linux-driver.git cd codeql-workshop-vulnerable-linux-driver/ unzip vulnerable-linux-driver.zip tree -L 2 vulnerable-linux-driver-db/ vulnerable-linux-driver-db/ ├── codeql-database.yml ├── db-cpp │ ├── default │ ├── semmlecode.cpp.dbscheme │ └── semmlecode.cpp.dbscheme.stats └── src.zip 3 directories, 4 files -
Quick check using VS Code. Same steps will repeat:
- select DB
- select query
- run query
- view results
-
Install codeql
-
Full docs: https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli#getting-started-with-the-codeql-cli https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system#setting-up-the-codeql-cli-in-your-ci-system In short:
cd ~/local/codeql-cli-end-to-endw # Decide on version / os via browser, then: wget https://github.com/github/codeql-action/releases/download/codeql-bundle-v2.13.4/codeql-bundle-osx64.tar.gz # Fix attributes on mac if [ `uname` = Darwin ] ; then xattr -c *.tar.gz fi # Extract tar zxf ./codeql-bundle-osx64.tar.gz # Check binary pwd # /Users/hohn/local/codeql-cli-end-to-end ./codeql/codeql --version # CodeQL command-line toolchain release 2.13.4. # Copyright (C) 2019-2023 GitHub, Inc. # Unpacked in: /Users/hohn/local/codeql-cli-end-to-end/codeql # Analysis results depend critically on separately distributed query and # extractor modules. To list modules that are visible to the toolchain, # use 'codeql resolve qlpacks' and 'codeql resolve languages'. # Check packs 0:$ ./codeql/codeql resolve qlpacks |head -5 # codeql/cpp-all (/Users/hohn/local/codeql-cli-end-to-end/codeql/qlpacks/codeql/cpp-all/0.7.3) # codeql/cpp-examples (/Users/hohn/local/codeql-cli-end-to-end/codeql/qlpacks/codeql/cpp-examples/0.0.0) # codeql/cpp-queries (/Users/hohn/local/codeql-cli-end-to-end/codeql/qlpacks/codeql/cpp-queries/0.6.3) # codeql/csharp-all (/Users/hohn/local/codeql-cli-end-to-end/codeql/qlpacks/codeql/csharp-all/0.6.3) # codeql/csharp-examples (/Users/hohn/local/codeql-cli-end-to-end/codeql/qlpacks/codeql/csharp-examples/0.0.0) # Fix the path export PATH=$(pwd -P)/codeql:"$PATH" # Check languages codeql resolve languages | head -5 # go (/Users/hohn/local/codeql-cli-end-to-end/codeql/go) # python (/Users/hohn/local/codeql-cli-end-to-end/codeql/python) # java (/Users/hohn/local/codeql-cli-end-to-end/codeql/java) # html (/Users/hohn/local/codeql-cli-end-to-end/codeql/html) # xml (/Users/hohn/local/codeql-cli-end-to-end/codeql/xml)A more fancy version:
# Reference urls: # https://github.com/github/codeql-cli-binaries/releases/download/v2.8.0/codeql-linux64.zip # https://github.com/github/codeql/archive/refs/tags/codeql-cli/v2.8.0.zip # # grab -- retrieve and extract codeql cli and library # Usage: grab version url prefix grab() { version=$1; shift platform=$1; shift prefix=$1; shift mkdir -p $prefix/codeql-$version && cd $prefix/codeql-$version || return # Get cli wget "https://github.com/github/codeql-cli-binaries/releases/download/$version/codeql-$platform.zip" # Get lib wget "https://github.com/github/codeql/archive/refs/tags/codeql-cli/$version.zip" # Fix attributes if [ `uname` = Darwin ] ; then xattr -c *.zip fi # Extract unzip -q codeql-$platform.zip unzip -q $version.zip # Rename library directory for VS Code mv codeql-codeql-cli-$version/ ql # remove archives? # rm codeql-$platform.zip # rm $version.zip } grab v2.7.6 osx64 $HOME/local grab v2.8.3 osx64 $HOME/local grab v2.8.4 osx64 $HOME/local grab v2.6.3 linux64 /opt grab v2.6.3 osx64 $HOME/local grab v2.4.6 osx64 $HOME/local -
Most flexible in use, but more initial setup: gh, the GitHub command-line tool from https://github.com/cli/cli
gh api repos/{owner}/{repo}/releases https://cli.github.com/manual/gh_api
gh extension create https://cli.github.com/manual/gh_extension
gh codeql extension https://github.com/github/gh-codeql install codeql cli and library?
gh gist list https://cli.github.com/manual/gh_gist_list
0:$ gh codeql GitHub command-line wrapper for the CodeQL CLI.
-
-
Install pack dependencies
-
-
-
Run queries
- Individual: 1 database -> N sarif files
- Use directory of queries: 1 database -> 1 sarif file (least effort)
- Use suite: 1 database -> 1 sarif file (more flexible, more effort)
-
Include versioning:
- codeql cli
- query set version
Checks:
-
Will include e.g.,
codeql database analyze --format=sarif-latest --rerun \ --output $QUERY_RES_SARIF \ --search-path $QLGIT \ -j6 \ --ram=24000 \ -- \ $DB \ $QLQUERY - Will include recommendations, e.g., 32 G ram, 4-6 cores.
- For building DBs: Common case: 15 minutes for || cpp compilation, can be 2 h with codeql.
-
Want to review results
- sarif viewer plugin
- raw sarif with
jq -
sarif-cli
- dump
- sql conversion
-
Running sequence
- Smallest query suite (security suite).
-
Check results.
- Lots of result (> 5000) -> cli review via compiler-style dump.
- Medium result sets (~ 2000) (sarif review plugin, can only load 5000 results)
- Few results (sarif review plugin, can only load 5000 results)
- Expand query
-
Compare results.
- sarif-cli using compiler-style dump.
Short end-to-end illustration
- Overall procedure
-
Command-line use
- For 3.2 also using sarif-cli
- sarif viewer plugin https://marketplace.visualstudio.com/items?itemName=MS-SarifVSCode.sarif-viewer Sarif Viewer v3.3.7 Microsoft DevLabs microsoft.com 53,335 (1)
- Details on query suite use (3. Use suite: 1 database -> 1 sarif file (more flexible, more effort))