This adds the possibility to add a special `proc_macro.rs` source file
to QL tests, which will be generated into a `proc_macro` crate the
usual `lib` crate depends on.
This allow to define procedural macros in QL tests, and is here used to
move the `macro-expansion` integration test to be a language test
instead.
As the generated manifests involved were starting to get a bit complex,
they are now generated from a `mustache` template.
Turns out this is important for the black formatting tool to work
correctly. The formatting won't generally change between python versions
(it only depends on `black`'s version), but the formatted code needs to
be parseable by the system python version. One script uses
```python
def foo[T](x: T) -> T:
```
syntax, which is only supported in Python 3.12 and later.
Running `pre-commit` will now require a python 3.12 installation (which
is already what we mandate for internal developer environment setup).
The error in case of absence of such a version is pretty clear though.
The standalone MaD generator now uses `0` for threads and throttles the
RAM to use 2GB per thread by default.
Also, replaced the hand-written argument parsing with `argparse`.
* fix a bug where the order of model generation was determined by the
order in the `download.json` file of the experiment rather than the
order in the config file
* allow configuring `--ram` and `--threads` in the MaD generator scripts
* use no `--ram` and `--threads=0` by default in the bulk generator
(single generator defaults are left unchanged)
* allow to pass `--dca` multiple times, taking DBs from experiments
listed last. This allows to run a subset of the sources in a "fixup"
experiment and use it to "patch" a previous run without rerunning
everything.
* fix a bug where the order of model generation was determined by the
order in the `download.json` file of the experiment rather than the
order in the config file
* allow configuring `--ram` and `--threads` in the MaD generator scripts
* use no `--ram` and `--threads=0` by default in the bulk generator
(single generator defaults are left unchanged)
* allow to pass `--dca` multiple times, taking DBs from experiments
listed last. This allows to run a subset of the sources in a "fixup"
experiment and use it to "patch" a previous run without rerunning
everything.
This switched `codegen` from the `autopep8` formatting to the `black`
one, and applies it to `bulk_mad_generator.py` as well. We can enroll
more python scripts to it in the future.
