This reintroduces lazy lfs file rules that were removed in
https://github.com/github/codeql/pull/16117, now improved.
The new rules will make the actual file download go through bazel's
download manager, which includes:
* caching into the repository cache
* sane limiting of concurrent downloads
* retries
The bulk of the work is done by `git_lfs_probe.py`, which will use the
LFS protocol (with authentication via SSH) to output short lived
download URLs that can be consumed by `repository_ctx.download`.
The bazel -> cmake generator is currently not capable of handling
separate included generated cmake files making use of common C/C++
dependencies.
To work around this limitation, a single generated cmake is now in
place. Long-term, we should either:
* make the cmake generator handle common dependencies gracefully, or
* make the cmake generation aspect travel up `pkg_` rules `srcs`
attributes
so to avoid having to list the targets to be generated in the top-level
`BUILD` file.
Other things fixed:
* removed some warning spam about redefined `BAZEL_CURRENT_REPOSITORY`
* fixed the final link step, that was failing because `libswiftCore.so`
was not being linked.
An interesting byproduct was finding a problematic `assert` in the
Swift headers. An incomplete `FallthroughStmt` was asserting on having
a destination. I did not find any other sensible way of getting rid of
the crash when running in debug mode than to patch the header.
This replaces usages of `llvm::fs` and string manipulation with
`std::filesystem`, also replacing `std::string` with
`std::filesystem::path` where it made sense.
Moreover MD5 hashing used in macOS file remapping was replaced by
SHA256 hashing using a small header-only SHA256 C++ library with an
MIT license, https://github.com/okdshin/PicoSHA2.
File contents hashing was relocated to the newly created `file` library
for later planned reuse.
A cmake generator in bazel is introduced allowing to import the Swift
extractor as a CMake project while keeping Bazel files as the source of
truth for the build.
Using the CMake project:
* requires bazel and clang to be installed and available on the command
line
* does not require a previous bazel build, however
* will require a CMake reconfiguration for changes to generated code
(like changes to the schema)
