If the cache is prefilled, LFS rules were still trying to query LFS
urls.
Now the strategy is to first try to fetch the files from the repository
cache (which is possible by providing an empty url list and `allow_fail`
to `repository_ctx.download`), and only run the LFS protocol if that
fails. Technically this is possible by enhancing `git_lfs_probe.py` with
a `--hash-only` flag.
This is also an optimization where no uneeded access is done (including
the slightly slow SSH call) if the repository cache is warm.
This reintroduces lazy lfs file rules that were removed in
https://github.com/github/codeql/pull/16117, now improved.
The new rules will make the actual file download go through bazel's
download manager, which includes:
* caching into the repository cache
* sane limiting of concurrent downloads
* retries
The bulk of the work is done by `git_lfs_probe.py`, which will use the
LFS protocol (with authentication via SSH) to output short lived
download URLs that can be consumed by `repository_ctx.download`.
The bazel -> cmake generator is currently not capable of handling
separate included generated cmake files making use of common C/C++
dependencies.
To work around this limitation, a single generated cmake is now in
place. Long-term, we should either:
* make the cmake generator handle common dependencies gracefully, or
* make the cmake generation aspect travel up `pkg_` rules `srcs`
attributes
so to avoid having to list the targets to be generated in the top-level
`BUILD` file.
Other things fixed:
* removed some warning spam about redefined `BAZEL_CURRENT_REPOSITORY`
* fixed the final link step, that was failing because `libswiftCore.so`
was not being linked.
An interesting byproduct was finding a problematic `assert` in the
Swift headers. An incomplete `FallthroughStmt` was asserting on having
a destination. I did not find any other sensible way of getting rid of
the crash when running in debug mode than to patch the header.
This replaces usages of `llvm::fs` and string manipulation with
`std::filesystem`, also replacing `std::string` with
`std::filesystem::path` where it made sense.
Moreover MD5 hashing used in macOS file remapping was replaced by
SHA256 hashing using a small header-only SHA256 C++ library with an
MIT license, https://github.com/okdshin/PicoSHA2.
File contents hashing was relocated to the newly created `file` library
for later planned reuse.
A cmake generator in bazel is introduced allowing to import the Swift
extractor as a CMake project while keeping Bazel files as the source of
truth for the build.
Using the CMake project:
* requires bazel and clang to be installed and available on the command
line
* does not require a previous bazel build, however
* will require a CMake reconfiguration for changes to generated code
(like changes to the schema)
