Previously we were not extracting any `swiftmodule` file that was not
a system or a built-in one. This was done to avoid re-extracting
`swiftmodule` files that were built previously in the same build, but it
turned out to be too eager, as there are legitimate cases where a
non-system, non-built-in precompiled swift module can be used. An
example of that is the `PackageDescription` module used in Swift
Package Manager manifest files (`Package.swift`).
We now relax the test and trigger module extraction on all loaded
modules that do not have source files (we trigger source file extraction
for those). The catch, is that we also create empty trap files for
current output `swiftmodule` files (including possible alias locations
set up by XCode).
This means that if a following extractor run loads a previously built
`swiftmodule` file, although it will trigger module extraction, this
will however be skipped as it will find its target file already present
(this is done via the `TargetFile` semantics).
Firstly, this change reworks how inter-process races are resolved.
Moreover some responsability reorganization has led to merging
`TrapArena` and `TrapOutput` again into a `TrapDomain` class.
A `TargetFile` class is introduced, that is successfully created
only for the first process that starts processing a given trap output
file. From then on `TargetFile` simply wraps around `<<` stream
operations, dumping them to a temporary file. When `TargetFile::commit`
is called, the temporary file is moved on to the actual target trap
file.
Processes that lose the race can now just ignore the unneeded
extraction and go on, while previously all processes would carry out
all extractions overwriting each other at the end.
Some of the file system logic contained in `SwiftExtractor.cpp` has been
moved to this class, and two TODOs are solved:
* introducing a better inter process file collision avoidance strategy
* better error handling for trap output operations: if unable to write
to the trap file (or carry out other basic file operations), we just
abort.
The changes to `ExprVisitor` and `StmtVisitor` are due to wanting to
hide the raw `TrapDomain::createLabel` from them, and bring more
funcionality under the generic caching/dispatching mechanism.
As `ASTMangler` crashes when called on `ModuleDecl`, we simply use
its name.
This might probably not work reliably in a scenario where multiple
modules are compiled with the same name (like `main`), but this is left
for future work. At the moment this cannot create DB inconsistencies.
Currently, we have a number of assertions in the codebase and certain
assumptions about the AST. These don't always hold, sometimes leading to
a crash in the extractor.
The crashes leave incomplete TRAP files that cannot be imported into the
database.
With this change, we still get those incomplete TRAP files, but we also
get a database in the end (even thoough it is also incomplete as we
cannot import everything).
Visitor code has been split between header and sources to speed up
incremental build. Moreover the code was reorganized using a new `infra`
bazel package (and `visitors` got promoted to a bazel package as well).
This was temporarily broken as we were skipping full emission of all
entities without any valid location.
We now rely on `decl->getDeclContext()->getParentSourceFile()` which is
more robust.
`getLocation()` will now exists for all entities. When there is no
valid location, the location will still not be emitted in the DB, but
on the QL side we will then assign a special `UnknownLocation` with
empty filename and 0 for line/column start/end.
This unknown location is currently emitted (with a unique `@` key) at
the start of every extraction, but we can move it elsewhere (and
possibly in a unique global trap file) at a later stage, possibly after
or when we rework the trap file strategy.
This should solve flakiness that was observed on the control flow tests,
which is probably caused by the `nodes` predicate in the `TestOutput`
class in `ControlFlowGraphImplShared.qll` not able to assign a proper
rank when the node does not have a location.
While the (removed) comment is correct and the frontend can be called in
different modes, both `swift build` and `xcodebuild` always use
`-primary-files` when compiling Swift projects.
The other mode was present only within our test runner (`qltest.sh`), so
removing it and doing what the official build systems do simplifies our
code base.
Additionally, file archival is now a separate function/operation.
This checks in the trapgen script generating trap entries in C++.
The codegen suite has been slightly reorganized, moving the templates
directory up one level and chopping everything into smaller bazel
packages. Running tests is now done via
```
bazel run //swift/codegen/test
```
With respect to the PoC, the nested `codeql::trap` namespace has been
dropped in favour of a `Trap` prefix (or suffix in case of entries)
within the `codeql` namespace. Also, generated C++ code is not checked
in in git any more, and generated during build. Finally, labels get
printed in hex in the trap file.
`TrapLabel` is for the moment only default-constructible, so only one
single label is possible. `TrapArena`, that is responsible for creating
disjoint labels will come in a later commit.
