This outputs some duration counts for various parts of the extraction
process in the database in the form of telemetry diagnostics.
The diagnostics format was preferred to putting things in the relational
database as that will scale better to code scanning and is more flexible
as for the data we can put into it without passing through the dbscheme.
Also, although it's not the case yet, it will be possible to output
diagnostics even if creation of the database fails.
This makes the first `codeql_pack` in a package add an `installer` target
aliasing the `<name>-installer` one. This makes it so that one can for
example do `bazel run //rust:installer` instead of the stuttering
`bazel run //rust:rust-installer`. If a bazel package defines multiple
`codeql_pack` targets, the first one only will get the `installer` alias.
We've been observing some performance issues using crate_universe on CI.
Therefore, we're moving to vendor the auto-generated BUILD files
in our repository. This should provide a nice speed boost, while
getting rid of the complexity of the "rust cache" job we've been using
when we had a lot of git dependencies.
This PR includes a vendor script, and I'll put up a CI job internally
that runs that vendor script on Cargo.toml and Cargo.lock changes, to check
that the vendored files are in sync.
We've hardcoded the tilde in a lot of places :(
This improves performance on Windows and gets us ready for Bazel 8.
We need an upgrade of rules_rust for this to work.
This is another shot at https://github.com/github/codeql/pull/17382,
using a different and more lightweight approach.
This allows building the ruby and python (and in the future also rust)
packs from within the codeql repository. This will:
* skip defining the glibc symbols checking, which only makes sense when
building the release from the internal repository
* stub out our `universal_binary` rule, which we only need when building
the release.
The case of an `HTTPError` was printed to stdout (and therefore globbed
by bazel).
While I'm at it, I also introduced a timeout to `urlopen` and improved
the `no endpoints found` error message.
This should allow us to build our python and ruby
code independently - in particular, we can now do shallow
checkouts of one without the other.
Previously, the modext introduced cross-dependency.
This also reduces the amount of work we do in the
crate universe processing for the other language, even
though it's unused.
This does need renaming the module, as otherwise
the generated paths from rules_rust get too long
for Windows :(
Ripunzip is great, but occasionally bugs out due to a parallelism issue.
As we don't want this to fail CI, retry the entire ripunzip/installation step
up to 3 times.
We need to clean up the working directory as ripunzip doesn't support overwriting
files. I've not been able to test this with the original issue (it doesn't seem
to reproduce locally for me), but I injected another error and got 3 retries
of installation.
This PR introduces a `codeql_pack_rule` that does the heavy lifting
of extracting arch- and common zip files for production dist building.
It also factors out the installer targets for individual packs,
as well as pack groups.
This changes the contract between the internal build system and the pack
definition significantly, which is why an accompanying internal PR is required.
No backwards compatibility layer is provided, as the PR as complex enough as-is.
The individual `codeql_pack` rules are now much simpler,
as they mostly stuff their inputs into a new `_CodeQLPackInfo` provider,
and let the installer and `codeql_pack_group` rules do the heavy lifting.
For working in the external repo with self-contained packs,
the per-pack installer targets are still available.
Internally, we'll only use the new `codeql_pack_group` targets
going forward, both for defining intree-dists and for building
the production zip files.
This gets rid of our last workspace dependency.
In particular, this change also gets rid of the checked-in extra
lock files that took forever to generate.
