We've hardcoded the tilde in a lot of places :(
This improves performance on Windows and gets us ready for Bazel 8.
We need an upgrade of rules_rust for this to work.
The case of an `HTTPError` was printed to stdout (and therefore globbed
by bazel).
While I'm at it, I also introduced a timeout to `urlopen` and improved
the `no endpoints found` error message.
This should allow us to build our python and ruby
code independently - in particular, we can now do shallow
checkouts of one without the other.
Previously, the modext introduced cross-dependency.
This also reduces the amount of work we do in the
crate universe processing for the other language, even
though it's unused.
This does need renaming the module, as otherwise
the generated paths from rules_rust get too long
for Windows :(
Ripunzip is great, but occasionally bugs out due to a parallelism issue.
As we don't want this to fail CI, retry the entire ripunzip/installation step
up to 3 times.
We need to clean up the working directory as ripunzip doesn't support overwriting
files. I've not been able to test this with the original issue (it doesn't seem
to reproduce locally for me), but I injected another error and got 3 retries
of installation.
This PR introduces a `codeql_pack_rule` that does the heavy lifting
of extracting arch- and common zip files for production dist building.
It also factors out the installer targets for individual packs,
as well as pack groups.
This changes the contract between the internal build system and the pack
definition significantly, which is why an accompanying internal PR is required.
No backwards compatibility layer is provided, as the PR as complex enough as-is.
The individual `codeql_pack` rules are now much simpler,
as they mostly stuff their inputs into a new `_CodeQLPackInfo` provider,
and let the installer and `codeql_pack_group` rules do the heavy lifting.
For working in the external repo with self-contained packs,
the per-pack installer targets are still available.
Internally, we'll only use the new `codeql_pack_group` targets
going forward, both for defining intree-dists and for building
the production zip files.
First, the labels used in this file need to be wrapped in `Label`,
so that we can use the installer target from the internal repo.
Second, when unpacking zips, the path passed to `ripunzip` included a trailing
`\n`, which caused `ripunzip` to fail.
This encapsulate arch specific logic, local installation and separation
of zip files into generic and arch-specific parts as required by the
internal build.
If the cache is prefilled, LFS rules were still trying to query LFS
urls.
Now the strategy is to first try to fetch the files from the repository
cache (which is possible by providing an empty url list and `allow_fail`
to `repository_ctx.download`), and only run the LFS protocol if that
fails. Technically this is possible by enhancing `git_lfs_probe.py` with
a `--hash-only` flag.
This is also an optimization where no uneeded access is done (including
the slightly slow SSH call) if the repository cache is warm.
