We've been observing some performance issues using crate_universe on CI.
Therefore, we're moving to vendor the auto-generated BUILD files
in our repository. This should provide a nice speed boost, while
getting rid of the complexity of the "rust cache" job we've been using
when we had a lot of git dependencies.
This PR includes a vendor script, and I'll put up a CI job internally
that runs that vendor script on Cargo.toml and Cargo.lock changes, to check
that the vendored files are in sync.
The `.cargo/config.toml` override based workaround wasn't really
working, as while `cargo build|check` was reading that, `cargo metadata`
wasn't, ending up in a completely broken IDE experience.
For the moment, we just use a unified workspace `Cargo.toml` for all
extractors using the shared tree-sitter code, which has the downside of
making bazel pull in dependencies for all of them, and not being able to
do sparse checkouts for them. We should investigate and rivist this in
the future.
This gets rid of our last workspace dependency.
In particular, this change also gets rid of the checked-in extra
lock files that took forever to generate.
Previously, we pulled in the shared tree-sitter extractor via a `git`
dependency in `Cargo.toml` to address a `rules_rust` limitation (no `path`
dependencies outside of the cargo workspace)). This was a problem,
as that means we're cloning `github/codeql` _again_ for the build, which is
quite slow.
I found another way that is faster, and still produces correct builds
for both `cargo`` and `rules_rust`:
* Cargo depends on a fake crate that has the same dependencies as the real crate (thanks to `sync-files.py`). Therefore, cargo pulls in the right dependencies into the lockfile, which bazel targets
* For local builds, we override the path to that dependency in a cargo config, so we're pulling in the correct code
* rules_rust only uses `path` dependencies for collecting transitive dependencies, it never pulls in the code from there. So far that, we manually provide a `BUILD.bazel` file for the shared extractor, and depend on that.
This PR introduces a bazel and `rules_rust`-based build system
for the ruby extractor and language pack.
This replacese the existing, `cargo` and `cross`-based build system.
For local development, nothing changes, and the existing `cargo`-based
build still keeps working as-is.
We no longer need to use `cross` to compile our Linux binaries,
as we now can link against our hermetic C++ toolchain, which ships
with an old enough glibc, so that we don't run into symbol version issues
when deploying the binaries to older systems.
Besides the one change in dependency (explained in detail in `Cargo.toml`
and in https://github.com/github/codeql/pull/15595), nothing ought to
change in how we build the extractor.
