Turns out this is important for the black formatting tool to work
correctly. The formatting won't generally change between python versions
(it only depends on `black`'s version), but the formatted code needs to
be parseable by the system python version. One script uses
```python
def foo[T](x: T) -> T:
```
syntax, which is only supported in Python 3.12 and later.
Running `pre-commit` will now require a python 3.12 installation (which
is already what we mandate for internal developer environment setup).
The error in case of absence of such a version is pretty clear though.
This switched `codegen` from the `autopep8` formatting to the `black`
one, and applies it to `bulk_mad_generator.py` as well. We can enroll
more python scripts to it in the future.
This introduces tooling and enforcement for formatting bazel files.
The tooling is provided as a bazel run target from
[keith/buildifier-prebuilt](https://github.com/keith/buildifier-prebuilt).
This is used in a [`pre-commit`](https://pre-commit.com/) hook for those
having that installed. In turn this is used in a CI check. Relying on a
`pre-commit` action gives us easy checking that buildifying did not
change anything in the files and printing the diff, without having to
hand-roll the check ourselves.
This enforcement will make usage of gazelle easier, as gazelle itself
might reformat files, even outside of `go`. Having them properly
formatted will allow gazelle to leave them unchanged, without needing
to configure awkward exclude directives.
This is a developer QoL improvement, where running codegen will skip
writing (and especially formatting) any files that were not changed.
**Why?** While code generation in itself was pretty much instant, QL
formatting of generated code was starting to take a long time. This made
unconditionally running codegen quite annoying, for example before each
test run as part of an IDE workflow or as part of the pre-commit hook.
**How?** This was not completely straightforward as we could not work
with the contents of the file prior to code generation as that was
already post-processed by the QL formatting, so we had no chance of
comparing the output of template rendering with that. We therefore store
the hashes of the files _prior_ to QL formatting in a checked-in file
(`swift/ql/.generated.list`). We can therefore load those hashes at
the beginning of code generation, use them to compare the template
rendering output and update them in this special registry file.
**What else?** We also extend this mechanism to detect accidental
modification of generated files in a more robust way. Before this patch,
we were doing it with a rough regexp based heuristic. Now, we just store
the hashes of the files _after_ QL formatting in the same checked file,
so we can check that and stop generation if a generated file was
modified, or a stub was modified without removing the `// generated`
header.
While PEP8 mandates those, they look bad in the schema file.
`autopep8` already ignores those, and they were single newlines at some
point until an overeager IDE has "fixed" them at some point without me
realizing.
Also, the pre-commit configuration was updated to take `schema.py` into
account.
By explicitly marking children in the `schema.yml` file, an internal
`getAChild` predicate is implemented, that is in turn used in `AstNode`
to implement `getParent`.
This is yet to be used in the control flow library to replace the
hand-rolled implementation.
A further, more complex step is to use the same information to fully
generate the core implementation of `PrintAst` (including the
accessor string). This will be done later.
The `parent` tests use the same swift code as the extractor tests, and
this is currently enforced by `sync-files.py`. Notice that `qltest.sh`
had to be modified to deal with multiple files, which was not working
yet.
This checks in the trapgen script generating trap entries in C++.
The codegen suite has been slightly reorganized, moving the templates
directory up one level and chopping everything into smaller bazel
packages. Running tests is now done via
```
bazel run //swift/codegen/test
```
With respect to the PoC, the nested `codeql::trap` namespace has been
dropped in favour of a `Trap` prefix (or suffix in case of entries)
within the `codeql` namespace. Also, generated C++ code is not checked
in in git any more, and generated during build. Finally, labels get
printed in hex in the trap file.
`TrapLabel` is for the moment only default-constructible, so only one
single label is possible. `TrapArena`, that is responsible for creating
disjoint labels will come in a later commit.
Tests can be run with
```
bazel test //swift/codegen:tests
```
Coverage can be checked installing `pytest-cov` and running
```
pytest --cov=swift/codegen swift/codegen/test
```
This enables use of the `pre-commit` framework to run quick pre-commit
checks. In particular this allows to automatically fix:
* trailing white spaces
* absence or multiple newlines at the end of files
* QL code formatting
* file sync
More could be added in the future: anything that can be checked fast
can be added in the configuration (for example well-formedness of
`qldoc` files).
This is a purely opt-in feature. Instructions for enabling it and
possibly configuring its behaviour are in `pre-commit-hook-setup.md`.
