codeql/rust

Rust on CodeQL

Development

Dependencies

If you don't have the semmle-code repo you may need to install Bazel manually, e.g. from https://github.com/bazelbuild/bazelisk.

Building the Rust Extractor

This approach uses a released codeql version and is simpler to use for QL development. From anywhere under your semmle-code or codeql directory you can run:

bazel run @codeql//rust:install

You can use shorter versions of the above command:

bazel run //rust:install  # if under the `codeql` checkout
bazel run rust:install  # if at the root of the `codeql` checkout
bazel run :install  # if at the `rust` directory of the `codeql` checkout

You now need to create a per-user CodeQL configuration file and specify the option:

--search-path PATH/TO/semmle-code/ql

(wherever the codeql checkout is on your system)

You can now use the Rust extractor e.g. to run Rust tests from the command line or in VSCode.

Building the Rust Extractor (as a sembuild target)

This approach allows you to build a Rust extractor with a CLI built from source. From your semmle-code directory run:

./build target/intree/codeql-rust

You can now invoke it directly, for example to run some tests:

./target/intree/codeql-rust/codeql test run ql/rust/ql/test/PATH/TO/TEST/

Building a Database

TODO

Code Generation

If you make changes to either

• ast-generator/, or
• schema/*.py

you'll need to regenerate code. You can do so running

bazel run @codeql//rust/codegen

Sometimes, especially if resolving conflicts on generated files, you might need to run

bazel run @codeql//rust/codegen -- --force

for code generation to succeed.

Updating rust-analyzer

Here's a rundown of the typical actions to perform to do a rust-analyzer (and other dependencies) update. A one-time setup consists in installing cargo-edit with cargo install cargo-edit. On Ubuntu that also requires sudo apt install libssl-dev pkg-config.

1. From the root of the codeql repo checkout, run a Cargo upgrade:

   cargo upgrade --incompatible --pinned
   

2. Look at a diff of the Cargo.toml files: if all ra_ap_ prefixed dependencies have been updated to the same number, go on to the next step. Otherwise, it means the latest rust-analyzer update has not been fully rolled out to all its crates in crates.io. All ra_ap_ versions must agree! Downgrade by hand to the minimum one you see, and run a cargo update after that to fix the Cargo.lock file.

3. Commit the changes, skipping pre-commit hooks if you have them enabled:

   git commit -am 'Cargo: upgrade dependencies' --no-verify
   

4. Regenerate vendored bazel files (these allow faster builds, particularly on CI where it has to start from scratch each time), commit the changes:

   misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh
   git add .
   git commit -am 'Bazel: regenerate vendored cargo dependencies' --no-verify
   

5. Run codegen

   bazel run //rust/codegen
   

   Take note whether rust/schema/ast.py was changed. That might need tweaks, new tests and/or downgrade/upgrade scripts down the line.

6. Try compiling

   bazel run //rust:install
   

   • if it succeeds: good! You can move on to the next step.
   • if it fails while compiling rust-analyzer dependencies, you need to update the rust toolchain. Sometimes the error will tell you so explicitly, but it may happen that the error is more obscure. To update the rust toolchain:
     • you will need to open a PR on the internal repo updating RUST_VERSION in MODULE.bazel. In general you can have this merged independently of the changes in codeql.
     • in codeql, update both RUST_VERSION in MODULE.bazel and rust-toolchain.toml files. You may want to also update the nightly toolchain in rust/extractor/src/nightly-toolchain/rust-toolchain.toml to a more recent date while you're at it.
   • if it fails while compiling rust extractor code, you will need to adapt it to the new library version.
     • for example updating annotations in annotations.py, adding / removing generated tests.

   If you had to do any changes, commit them. If you updated the rust toolchain, running rust/lint.py might reformat or apply new lints to the code.

7. Check with CI if everything is in order.

8. Run DCA with database caching disabled. Iterate on the code if needed.

9. If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note. This is best done last to reduce the chance of merge conflicts (none of the other testing depends on having upgrade and downgrade scripts in place). See Upgrading a language database schema.