This is a developer QoL improvement, where running codegen will skip
writing (and especially formatting) any files that were not changed.
**Why?** While code generation in itself was pretty much instant, QL
formatting of generated code was starting to take a long time. This made
unconditionally running codegen quite annoying, for example before each
test run as part of an IDE workflow or as part of the pre-commit hook.
**How?** This was not completely straightforward as we could not work
with the contents of the file prior to code generation as that was
already post-processed by the QL formatting, so we had no chance of
comparing the output of template rendering with that. We therefore store
the hashes of the files _prior_ to QL formatting in a checked-in file
(`swift/ql/.generated.list`). We can therefore load those hashes at
the beginning of code generation, use them to compare the template
rendering output and update them in this special registry file.
**What else?** We also extend this mechanism to detect accidental
modification of generated files in a more robust way. Before this patch,
we were doing it with a rough regexp based heuristic. Now, we just store
the hashes of the files _after_ QL formatting in the same checked file,
so we can check that and stop generation if a generated file was
modified, or a stub was modified without removing the `// generated`
header.
In those kinds of tests the results may have different final classes
that are not necessarily visible (or tested) solely through the string
representation. For better testing and reading of expected results,
`getQlPrimaryClasses` is added in these cases.
The information that was contained in `schema.yml` is now in
`swift/schema.py`, which allows a more integrated IDE experience
for writing and navigating it.
Another minor change is that `schema.Class` now has a `str` `group`
field instead of a `pathlib.Path` `dir` one.
This makes the result of code generation independent of the order
in which classes are defined in the schema, and makes additional
topological sorting not required.
Being independent from schema order will be important for reviewing the
move to a pure python schema, as generated code will be left untouched.
There was an issue in case multiple inheritance from classes with
children was involved, where indexes would overlap.
The generated code structure has been reshuffled a bit, with
`Impl::getImmediateChildOf<Class>` predicates giving 0-based children
for a given class, including those coming from bases, and the final
`Impl::getImmediateChild` disjuncting the above on final classes only.
This removes the need of `getMaximumChildrenIndex<Class>`, and also
removes the code scanning alerts.
Also, comments were fixed addressing the review.
This introduces a `MethodRefExpr` node synthesized out of
`DotSyntaxCallExpr` under the `LookupExpr` hierarchy. This means that
much like
```free_function(1, 2)```
is a `CallExpr` with `getFunction` giving a `DeclRefExpr`,
```foo.method(1, 2)```
is now a `CallExpr` with `getFunction` giving a `MethodRefExpr`.
`ApplyExpr::getStaticTarget` has been made work with it (as well as
`ConstructorRefCallExpr` which for the moment has been left where it
is), a new `MethodApplyExpr` has been introduced deriving from it,
and control and data flow libraries have adapted.
A small but was fixed in `qlgen` where the default constructor for DB
types was not correctly subtracting derived IPA types depending on the
order of definitions in `schema.yml`.
There are still some occurrences of `DotSyntaxCallExpr`, and as already
mentioned the other `SelfApply` class (`ConstructorRefCallExpr`) was
left alone. Their treatment is left for a future PR.
It turns out the threshold of 5 lines for stub modification detection
was too strict: in case of a long class name the QL formatter will put
the closing brace of the empty class definition on a new line, leading
to codegen fail with an error thinking the stub was modified.
On the other side of things, also adding a base to a stub class was not
being detected as a modification.
Now the modification test is slightly smarter. If the stub still marked
as generated and
* has more than 6 lines, or
* the contents does not match a regexp aproximation of a plain stub
then codegen will abort. The test will still avoid reading the whole
contents of all the stubs.
If one modifies a QL stub but forgets to remove the `// generated`
header comment, codegen will now abort with an error rather than
silently reverting the change.
This is based on the rough heuristic of just counting the lines. If any
change is done to the stub class, the number of lines is bound to be
5 or more.
Python code was simplified, and now a `--generate` option can be used
to drive what can be generated.
The extractor pack creation now will use an internally generated
dbscheme. This should be the same as the checked in one, but doing so
allows `bazel run create-extractor-pack` and `bazel run codegen` to be
run independently from one another, while previously the former had to
follow the latter in case of a schema change. This is the change that
triggered the above simplification, as in order for the two dbscheme
files to be identical, the first `// generated` line had to state the
same generator script.
