The rust-analyzer update will need more work as it seems to break rust
analysis on windows.
This was carried out using `cargo upgrade` from `cargo-edit`:
* getting exclusions options for rust-analyzer with
```bash
cargo upgrade -i --dry-run | grep -o 'ra_ap_\S\+' | sort -u | sed 's/^/--exclude=/' > /tmp/exclude
```
* running
```bash
cargo upgrade -i $(cat /tmp/exclude)
misc/bazel/3rdparty/update_cargo_deps.sh
```
* The ungram file is now taken from the rust-analyzer dependencies
pulled in by bazel
* the grammar parsing code is not published, so it must be taken
directly from rust-analyzer code. That part should be less prone to be
updated than the ungram file, so it does not necessarily need to be
in sync with the rust-analyzer version is used elsewhere.
* both need some patches. The former is patched during build, the latter
during loading in `MODULE.bazel`.
This hack is meant to be an optimization when using install for tests,
where the install step is skipped if nothing changed. If the
installation directory is somehow messed up, `bazel run` can be used to
force install.
This is added as a `<name>-installer-as-test` target, which we can now
use in our internal pytest integration to skip the installation step if
nothing changed on the CLI + language packs side.
Previously, we were using 8.0.0rc1.
In particular, this upgrade means we need to explicitly
import more rules, as they've been moved out of the core bazel repo.
This outputs some duration counts for various parts of the extraction
process in the database in the form of telemetry diagnostics.
The diagnostics format was preferred to putting things in the relational
database as that will scale better to code scanning and is more flexible
as for the data we can put into it without passing through the dbscheme.
Also, although it's not the case yet, it will be possible to output
diagnostics even if creation of the database fails.
This makes the first `codeql_pack` in a package add an `installer` target
aliasing the `<name>-installer` one. This makes it so that one can for
example do `bazel run //rust:installer` instead of the stuttering
`bazel run //rust:rust-installer`. If a bazel package defines multiple
`codeql_pack` targets, the first one only will get the `installer` alias.
We've been observing some performance issues using crate_universe on CI.
Therefore, we're moving to vendor the auto-generated BUILD files
in our repository. This should provide a nice speed boost, while
getting rid of the complexity of the "rust cache" job we've been using
when we had a lot of git dependencies.
This PR includes a vendor script, and I'll put up a CI job internally
that runs that vendor script on Cargo.toml and Cargo.lock changes, to check
that the vendored files are in sync.
We've hardcoded the tilde in a lot of places :(
This improves performance on Windows and gets us ready for Bazel 8.
We need an upgrade of rules_rust for this to work.
This is another shot at https://github.com/github/codeql/pull/17382,
using a different and more lightweight approach.
This allows building the ruby and python (and in the future also rust)
packs from within the codeql repository. This will:
* skip defining the glibc symbols checking, which only makes sense when
building the release from the internal repository
* stub out our `universal_binary` rule, which we only need when building
the release.
The case of an `HTTPError` was printed to stdout (and therefore globbed
by bazel).
While I'm at it, I also introduced a timeout to `urlopen` and improved
the `no endpoints found` error message.
This should allow us to build our python and ruby
code independently - in particular, we can now do shallow
checkouts of one without the other.
Previously, the modext introduced cross-dependency.
This also reduces the amount of work we do in the
crate universe processing for the other language, even
though it's unused.
This does need renaming the module, as otherwise
the generated paths from rules_rust get too long
for Windows :(
Ripunzip is great, but occasionally bugs out due to a parallelism issue.
As we don't want this to fail CI, retry the entire ripunzip/installation step
up to 3 times.
We need to clean up the working directory as ripunzip doesn't support overwriting
files. I've not been able to test this with the original issue (it doesn't seem
to reproduce locally for me), but I injected another error and got 3 retries
of installation.
