We've been observing some performance issues using crate_universe on CI.
Therefore, we're moving to vendor the auto-generated BUILD files
in our repository. This should provide a nice speed boost, while
getting rid of the complexity of the "rust cache" job we've been using
when we had a lot of git dependencies.
This PR includes a vendor script, and I'll put up a CI job internally
that runs that vendor script on Cargo.toml and Cargo.lock changes, to check
that the vendored files are in sync.
We've hardcoded the tilde in a lot of places :(
This improves performance on Windows and gets us ready for Bazel 8.
We need an upgrade of rules_rust for this to work.
This is another shot at https://github.com/github/codeql/pull/17382,
using a different and more lightweight approach.
This allows building the ruby and python (and in the future also rust)
packs from within the codeql repository. This will:
* skip defining the glibc symbols checking, which only makes sense when
building the release from the internal repository
* stub out our `universal_binary` rule, which we only need when building
the release.
The case of an `HTTPError` was printed to stdout (and therefore globbed
by bazel).
While I'm at it, I also introduced a timeout to `urlopen` and improved
the `no endpoints found` error message.
This should allow us to build our python and ruby
code independently - in particular, we can now do shallow
checkouts of one without the other.
Previously, the modext introduced cross-dependency.
This also reduces the amount of work we do in the
crate universe processing for the other language, even
though it's unused.
This does need renaming the module, as otherwise
the generated paths from rules_rust get too long
for Windows :(
Ripunzip is great, but occasionally bugs out due to a parallelism issue.
As we don't want this to fail CI, retry the entire ripunzip/installation step
up to 3 times.
We need to clean up the working directory as ripunzip doesn't support overwriting
files. I've not been able to test this with the original issue (it doesn't seem
to reproduce locally for me), but I injected another error and got 3 retries
of installation.
This PR introduces a `codeql_pack_rule` that does the heavy lifting
of extracting arch- and common zip files for production dist building.
It also factors out the installer targets for individual packs,
as well as pack groups.
This changes the contract between the internal build system and the pack
definition significantly, which is why an accompanying internal PR is required.
No backwards compatibility layer is provided, as the PR as complex enough as-is.
The individual `codeql_pack` rules are now much simpler,
as they mostly stuff their inputs into a new `_CodeQLPackInfo` provider,
and let the installer and `codeql_pack_group` rules do the heavy lifting.
For working in the external repo with self-contained packs,
the per-pack installer targets are still available.
Internally, we'll only use the new `codeql_pack_group` targets
going forward, both for defining intree-dists and for building
the production zip files.
This gets rid of our last workspace dependency.
In particular, this change also gets rid of the checked-in extra
lock files that took forever to generate.
This space breaks when upgrading bazelisk on Windows.
Somehow, a space inside the argument messes up some argument-parsin
somewhere. Porbably due to Windows argument parsing
being not well-defined in how it splits a string into argv.
Removing the space works around the problem.
If the space is included, cmake suddenly can't find the bazelisk
binary anymore, despite happily executing `info output_base` earlier.
First, the labels used in this file need to be wrapped in `Label`,
so that we can use the installer target from the internal repo.
Second, when unpacking zips, the path passed to `ripunzip` included a trailing
`\n`, which caused `ripunzip` to fail.
In order to build the language-independent parts of our dist with `pkg.bzl`,
we need two override features:
* A way to set the prefix in the zip files to the empty string, so that our top-level files
stay top-level.
* A way to put `codeql.exe` into the arch-specific zip, despite it not being under `CODEQL_PLATFORM`.
This PR implements both.
