Rust on CodeQL
Development
Dependencies
If you don't have the semmle-code repo you may need to install Bazel manually, e.g. from https://github.com/bazelbuild/bazelisk.
Building the Rust Extractor
This approach uses a released codeql version and is simpler to use for QL development. From anywhere under your semmle-code or codeql directory you can run:
bazel run @codeql//rust:install
You can use shorter versions of the above command:
bazel run //rust:install # if under the `codeql` checkout
bazel run rust:install # if at the root of the `codeql` checkout
bazel run :install # if at the `rust` directory of the `codeql` checkout
You now need to create a per-user CodeQL configuration file and specify the option:
--search-path PATH/TO/semmle-code/ql
(wherever the codeql checkout is on your system)
You can now use the Rust extractor e.g. to run Rust tests from the command line or in VSCode.
Building the Rust Extractor (as a sembuild target)
This approach allows you to build a Rust extractor with a CLI built from source. From your semmle-code directory run:
./build target/intree/codeql-rust
You can now invoke it directly, for example to run some tests:
./target/intree/codeql-rust/codeql test run ql/rust/ql/test/PATH/TO/TEST/
Building a Database
TODO
Code Generation
If you make changes to either
ast-generator/, orschema/*.py
you'll need to regenerate code. You can do so running
bazel run @codeql//rust/codegen
Sometimes, especially if resolving conflicts on generated files, you might need to run
bazel run @codeql//rust/codegen -- --force
for code generation to succeed.
Updating rust-analyzer
Here's a rundown of the typical actions to perform to do a rust-analyzer (and other dependencies) update. A one-time setup consists in
installing cargo-edit with cargo install cargo-edit. On Ubuntu that also requires
sudo apt install libssl-dev pkg-config.
-
From the root of the
codeqlrepo checkout, run a Cargo upgrade:cargo upgrade --incompatible --pinned -
Look at a diff of the
Cargo.tomlfiles: if allra_ap_prefixed dependencies have been updated to the same number, go on to the next step. Otherwise, it means the latestrust-analyzerupdate has not been fully rolled out to all its crates incrates.io. Allra_ap_versions must agree! Downgrade by hand to the minimum one you see, and run acargo updateafter that to fix theCargo.lockfile. -
Commit the changes, skipping
pre-commithooks if you have them enabled:git commit -am 'Cargo: upgrade dependencies' --no-verify -
Regenerate vendored bazel files (these allow faster builds, particularly on CI where it has to start from scratch each time), commit the changes:
misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh git add . git commit -am 'Bazel: regenerate vendored cargo dependencies' --no-verify -
Run codegen
bazel run //rust/codegenTake note whether
rust/schema/ast.pywas changed. That might need tweaks, new tests and/or downgrade/upgrade scripts down the line. -
Try compiling
bazel run //rust:install- if it succeeds: good! You can move on to the next step.
- if it fails while compiling rust-analyzer dependencies, you need to update the rust toolchain. Sometimes the error will tell you
so explicitly, but it may happen that the error is more obscure. To update the rust toolchain:
- you will need to open a PR on the internal repo updating
RUST_VERSIONinMODULE.bazel. In general you can have this merged independently of the changes incodeql. - in
codeql, update bothRUST_VERSIONinMODULE.bazelandrust-toolchain.tomlfiles. You may want to also update the nightly toolchain inrust/extractor/src/nightly-toolchain/rust-toolchain.tomlto a more recent date while you're at it.
- you will need to open a PR on the internal repo updating
- if it fails while compiling rust extractor code, you will need to adapt it to the new library version.
- for example updating annotations in
annotations.py, adding / removing generated tests.
- for example updating annotations in
If you had to do any changes, commit them. If you updated the rust toolchain, running
rust/lint.pymight reformat or apply new lints to the code. -
Check with CI if everything is in order.
-
Run DCA with database caching disabled. Iterate on the code if needed.
-
If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note. This is best done last to reduce the chance of merge conflicts (none of the other testing depends on having upgrade and downgrade scripts in place). See Upgrading a language database schema.