Adds a few tests that validate that higher-order functions are parsed
correctly into the commonAST representation.
Also removes a redundant assignment to `ctx.in_function_type` that
happend after all translations had taken place (and so nothing would
actually read this field).
The extractor shells out to a separate `swift-syntax-parse` binary, but
nothing placed it in the extractor pack, so a shipped Swift extraction
failed at the first spawn. Package it next to the extractor, the same
way `//swift/extractor` ships its Swift-linked binary: a small wrapper
points the dynamic loader at its own directory and execs the real
binary, whose Swift runtime libraries travel alongside it.
- `swift-syntax-parse.sh`: wrapper that sets `LD_LIBRARY_PATH` /
`DYLD_LIBRARY_PATH` to its directory and execs
`swift-syntax-parse.real`
(mirrors `swift/extractor/extractor.sh`).
- `runtime.bzl`: a `swift_runtime_libs` rule that selects just the Linux
Swift runtime shared objects (`usr/lib/swift/linux/*.so`) out of the
full toolchain, so only they — not the whole toolchain — travel with
the binary.
- `swift-syntax-rs/BUILD.bazel`: the `rust_binary` becomes
`swift-syntax-parse.real`
and carries the runtime libraries as runfiles on Linux; a `sh_binary`
(`swift-syntax-parse`) is the wrapper; `codeql_pkg_runfiles` flattens
the three (wrapper, real binary, runtime) into one directory.
- `BUILD.bazel`: ship that group under `tools/{CODEQL_PLATFORM}` next to
the extractor, on the platforms where swift-syntax builds
(Linux/macOS).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`languages::swift::adapter` embeds the swift-syntax node-types schema
with `include_str!("../../../swift_node_types.yml")`, but the file was
never listed in the extractor's Bazel `compile_data`. The `cargo` build
finds it on disk, so this went unnoticed, but the sandboxed Bazel build
cannot see it and fails to compile the extractor. List it alongside
`ast_types.yml`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two fixes prompted by review of the swift-syntax switch-over.
Parser resolution (`parse.rs`): `parse_bin` now resolves the
`swift-syntax-parse` executable in priority order — the
`CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` override, then a copy next
to the extractor executable (as a shipped extractor pack lays it out:
`tools/<platform>/{extractor,swift-syntax-parse}`), then a
bare `PATH` lookup. This lets a packaged extractor find its parser with
no environment setup. (Bundling the binary into the pack, together with
its Swift runtime, is a separate follow-up.)
Corpus test guard (`corpus_tests.rs`): `parser_available` previously
treated *any* parser error as "unavailable" and skipped the entire
corpus suite, so a parser that was present but crashed or emitted
invalid JSON would silently skip the exact regressions the suite exists
to catch. It now uses the new `binary_available`, which reports whether
the *executable* can be launched (false only when it cannot be found,
e.g. no Swift toolchain); a launchable-but-failing parser makes the
suite run and fail.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`swift-syntax-rs` is a workspace member, so its build script runs on a
plain `cargo check`/`fmt`/`clippy` at the repo root. Previously it
panicked when `swift build` could not be run, breaking those Swift-free
workflows for anyone without a Swift toolchain.
Instead, when `swift build` cannot be spawned, emit a `cargo:warning`
and skip the link directives rather than panicking. `cargo
check`/`fmt`/`clippy` don't link, so they keep working; only `cargo
build`/`cargo test` then fail, at link time — which is fair, since those
genuinely need Swift (and CI builds go through Bazel). A Swift toolchain
that is present but whose build fails is still surfaced as a hard error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerate `types/property-with-getter-and-setter`, whose mapped AST now
differs from the tree-sitter output: the backing `private var _v`
retains its `private` modifier (`modifier "var"` + `modifier
"private"`), whereas the tree-sitter path dropped it (its
`visibility_modifier` node carried no text). The swift-syntax
front-end preserves the modifier, so the mapped AST is strictly richer
here.
The raw (second) section is regenerated to the swift-syntax AST like the
other cases; the mapped (third) section gains the retained `private`
modifier.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that the Swift front-end is swift-syntax, regenerate the second (raw
parse tree) section of the corpus `.output` files to hold the
swift-syntax AST the adapter builds, instead of the old tree-sitter
parse tree.
These 98 cases map to a byte-for-byte identical mapped AST (the third
section), so only their raw section changes — the mapping rules were
ported to produce the same output. The one case whose mapped AST differs
(`types/property-with-getter-and-setter`) is handled separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip the runtime Swift front-end from tree-sitter to swift-syntax. The
mapping rules were already ported, so the mapped AST is unchanged; this
makes the switch live.
- `language_spec` now builds a language-free desugarer
(`ConcreteDesugarer::without_language`) and wires the swift-syntax
parser (`swift_parse::parse`) as the front-end, dropping the
tree-sitter language and node types. The desugarer supplies the output
schema, so `node_types` is left empty.
- The `swift_adapter`/`swift_parse` modules are no longer
`allow(dead_code)`: they are now reached from the live extraction
path.
- `corpus_tests` skips (rather than fails) when the external
`swift-syntax-parse` binary is unavailable, since it cannot run
without the Swift-backed parser.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the remaining member declarations to the swift-syntax AST. An
`initializerDecl` becomes a `constructor_declaration` (its body
optional, so a bodyless protocol requirement still maps);
`deinitializerDecl`, `typeAliasDecl`, and `associatedTypeDecl` map to
`destructor_declaration`, `type_alias_declaration`, and
`associated_type_declaration` respectively.
The tree-sitter subscript and preprocessor-diagnostic rules are dropped:
their swift-syntax counterparts (`subscriptDecl`, `ifConfigDecl`) fall
through to the `unsupported_node` fallback, producing the same output.
This also removes the now-unused `type` unwrap rule (swift-syntax has no
such wrapper node) and retires the last `ctx.literal` helper use in
favour of a `tree!` leaf.
PARITY(tree-sitter): the initializer's parameters are still not emitted,
because the tree-sitter path dropped them too. Emitting them is a future
improvement.
This completes the rule migration: every declaration now maps through
the swift-syntax front-end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget enum cases to the swift-syntax AST. An `enumCaseDecl` flattens
its comma-separated `enumCaseElement`s (non-first tagged
`chained_declaration`) and publishes any case modifiers (e.g.
`indirect`) into `ctx`; an element with a payload becomes a nested
`class_like_declaration` + constructor, an element with a raw value
(`case a = 1`) or a plain element a `variable_declaration`;
`enumCaseParameter` becomes a `parameter`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget property accessors to the swift-syntax AST, output unchanged.
An accessor-bearing `variableDecl` publishes the property name/type into
`ctx`; a computed property (`var v: T { get set }`) emits accessors
carrying the type, while a stored property with observers (`var x = e {
didSet {…} }`) emits the backing `variable_declaration` first.
swift-syntax models get/set/willSet/didSet uniformly as `accessorDecl`,
so a single `accessorDecl` rule — with an optional body distinguishing a
computed accessor from a bodyless protocol requirement — replaces the
tree-sitter grammar's separate computed-getter/setter/modify,
willset/didset, and getter-/setter-specifier rules.
The tree-sitter protocol property and function requirement rules
(`protocol_property_declaration`, `protocol_function_declaration`) are
dropped: swift-syntax models those requirements as ordinary
`variableDecl`s (with bodyless accessors) and `functionDecl`s, already
handled by the general rules.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the nominal type declarations to the swift-syntax AST, output
unchanged.
`classDecl`/`structDecl`/`enumDecl`/`protocolDecl`/`extensionDecl`
each become a `class_like_declaration` tagged with a modifier naming the
declaration keyword, with members drawn from the `memberBlock`; each
`memberBlockItem` unwraps to its contained declaration. `superExpr` maps
to `super_expr`; `self` needs no rule (swift-syntax models it as an
ordinary `declReferenceExpr`, already a `name_expr`).
Following the tree-sitter path (PARITY), the inheritance clause is not
emitted as a `base_type` (the tree-sitter rule captured it positionally,
but the grammar nests it under a field, so it never actually matched);
swift-syntax exposes it cleanly, so that is a correctness improvement to
make once tree-sitter is retired. The tree-sitter grammar's standalone
`self`, dead modifier (`visibility_modifier`, etc.), key-path, and
inheritance-specifier rules are dropped; `#selector`/`#keyPath` (now a
`macroExpansionExpr`) stays an `unsupported_node`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget imports to the swift-syntax AST, output unchanged. swift-syntax
represents the dotted path as a list of `importPathComponent`s, folded
into a `name_expr`/`member_access_expr` chain via `member_chain`. A
single rule handles both forms via an optional `importKindSpecifier`: a
scoped import (`import struct Foo.Bar`) has one and binds the last path
component as a `name_pattern`; a plain import (`import Foundation`) has
none and uses a `bulk_importing_pattern`. Leading attributes
(`@_exported`) and access modifiers (`public`) become `modifier`s.
The tree-sitter multi-part `identifier` rule is dropped: swift-syntax
qualified names are already `memberAccessExpr` chains, and import paths
are `importPathComponent`s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget optional chaining, `try`, `do`/`catch`, casts, type tests,
`await`, and force-unwrap to the swift-syntax AST, output unchanged:
- `optionalChainingExpr` (`x?`) unwraps transparently (the enclosing
member access / call carries the semantics).
- `tryExpr` -> prefix `unary_expr`; swift-syntax splits the operator
into a `try` keyword and an optional `?`/`!` mark, recombined into one
`prefix_operator` spelling.
- `doStmt` -> `try_expr` with `catchClause` -> `catch_clause`; a `catch`
binds the first `catchItem`'s pattern and optional `where` guard.
- `asExpr` (`x as`/`as?`/`as!` `T`) -> `type_cast_expr`, `isExpr`
(`x is T`) -> `type_test_expr`, and `awaitExpr` -> prefix
`unary_expr`.
- `forceUnwrapExpr` (`x!`) -> postfix `unary_expr` (swift-syntax has a
dedicated node; the tree-sitter path used the generic postfix
operator).
A couple of rewritten rules keep their opening inlined so this commit's
diff stays readable; a later commit restores the canonical formatting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget collection literals and subscripts to the swift-syntax AST,
output unchanged: `arrayExpr` -> `array_literal` (each `arrayElement`
unwraps to its expression); `dictionaryExpr` -> an opaque `map_literal`
leaf (its source span, matching the tree-sitter path); and
`subscriptCallExpr` (`xs[0]`) -> `call_expr`, mirroring the tree-sitter
grammar's treatment of a subscript as a call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget loops to the swift-syntax AST, output unchanged: `forStmt` ->
`for_each_stmt` (the optional `where` clause becomes the `guard`),
`whileStmt` -> `while_stmt`, `repeatStmt` -> `do_while_stmt`, and
`labeledStmt` -> `labeled_stmt`. Unlike the tree-sitter grammar,
swift-syntax stores a labeled statement's label and colon as separate
tokens, so the label token is already the bare name (no trailing `:` to
strip). A `repeat`-`while` loop has a single condition in swift-syntax
(not a condition list), so it needs no `and_chain`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `if`/`guard`/`switch`/ternary and the case/binding patterns to
the swift-syntax AST, output unchanged. swift-syntax distinguishes a
binding pattern (`valueBindingPattern`, `let x`) from a match pattern
(`expressionPattern`, `someConstant`) by node kind, so the tree-sitter
path's context-based `in_binding_pattern` disambiguation is removed:
- `ifExpr`/`guardStmt`/`ternaryExpr` ->
`if_expr`/`guard_if_stmt`/`if_expr`;
`switchExpr` + `switchCase` -> `switch_expr` + `switch_case` (comma
cases become an `or_pattern`); `conditionElement`/`switchCaseItem`
unwrap; a statement-position `if`/`switch`/`do` is unwrapped from its
`expressionStmt`.
- `optionalBindingCondition` (`if let`) and `matchingPatternCondition`
(`if case`) -> `pattern_guard_expr`.
- An `expressionPattern` wrapping a leading-dot or qualified call ->
`constructor_pattern` (setting `ctx.in_pattern`);
`valueBindingPattern` unwraps; a bare `expressionPattern` ->
`expr_equality_pattern`; a wildcard
(`discardAssignmentExpr`) -> `ignore_pattern`; a `tupleExpr` match
pattern -> `tuple_pattern`; `isTypePattern` (`case is T`) ->
`unsupported_node`.
- The `labeledExpr` argument rules gain `in_pattern`-aware pattern
variants so an enum-case pattern's arguments (`case .foo(let x, _)`)
become `pattern_element`s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget closures to the swift-syntax AST, output unchanged.
swift-syntax nests the whole closure header under an optional
`closureSignature`, so a single `closureExpr` rule (with optional
attributes, capture list, parameter clause, and return clause) replaces
the tree-sitter `lambda_literal` rule.
The parameter clause is a union of the parenthesised form
(`closureParameterClause`, unwrapped to its `closureParameter` children)
and the shorthand form (`closureShorthandParameter`, a bare name); one
rule each replaces the four `lambda_parameter` variants.
`closureCapture` -> `variable_declaration` (an optional ownership
specifier becomes a modifier; an explicit capture initializer becomes
the bound value).
The trailing-closure call form is already handled by the
`functionCallExpr`
`trailingClosure` variant, so the tree-sitter trailing-closure call rule
is
dropped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget function declarations, calls, member access, and control
transfer to the swift-syntax AST, output unchanged:
- `functionDecl` -> `function_declaration` (parameters and return type
nest under `signature`; the body is a `codeBlock`). A bodyless
function (a protocol requirement) still emits an empty `block`.
- `functionParameter` -> `parameter`: two names give the external label
and internal name, one name just the internal name; the default value
is handled inline, so the `ctx.default_value` threading (and its
`SwiftContext` field) is removed. The declared type is dropped: in the
tree-sitter path the untyped-parameter rule was ordered first and
shadowed the typed one, so the baseline emits no parameter type.
- A function reference spelled with argument labels (`f(x:y:z:)`) is a
`declReferenceExpr` with `argumentNames`; it is mapped to
`unsupported_node` (matched before the bare-name rule) so downstream
QL isn't handed a malformed reference, as in the tree-sitter path.
- `functionCallExpr` -> `call_expr` (a trailing closure becomes a final
unlabelled argument); `labeledExpr` -> `argument`;
`memberAccessExpr` -> `member_access_expr` (base-ful matched before
leading-dot).
- `returnStmt`/`breakStmt`/`continueStmt`/`throwStmt` ->
`return_expr`/`break_expr`/`continue_expr`/`throw_expr`, collapsing
the labelled/unlabelled variants via optional captures.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget type expressions to the swift-syntax AST, output unchanged:
- `user_type` -> `identifierType` (its `name` is the type-name token).
- The sugared types keep desugaring to `generic_type_expr`:
`optionalType` -> Optional<T>, `arrayType` -> Array<T>,
`dictionaryType` -> Dictionary<K, V>.
- A generic type with explicit arguments (`Set<Int>`) stays opaque (its
whole source text as the name), matched before the plain
`identifierType` rule.
This matches the tree-sitter `user_type` rule, which was also opaque.
- Tuple types (`(Int, String)`) -> `tuple_type_expr` and function types
(`(Int) -> Bool`) -> `function_type_expr`. swift-syntax holds both as
`tupleTypeElement`s but a tuple element maps to `tuple_type_element`
while a function parameter maps to `parameter`; the containers set
`SwiftContext::in_function_type` for their direct children so the
shared `tupleTypeElement` rule emits the right kind (and nested types
stay correct).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `let`/`var` bindings to the swift-syntax AST, preserving the
output:
- A `variableDecl` publishes its `bindingSpecifier` (`let`/`var`) as the
binding modifier, followed by its attributes and modifiers (`@objc`,
`public`, `static`, …), and flattens each `patternBinding` into its
own `variable_declaration`, tagging non-first ones
`chained_declaration`. - One `patternBinding` rule with optional
`typeAnnotation`/`initializer` covers `let x`, `let x = e`,
`let x: T`, and `let x: T = e`.
- `identifierPattern` -> `name_pattern`; `tuplePattern` /
`tuplePatternElement` -> `tuple_pattern` / `pattern_element` (tuple
destructuring), carrying an optional element label through as the
`pattern_element` key.
- `codeBlockItem` now captures `_*` / annotates `stmt*` so a
multi-binding declaration splices as several statements.
- Add a `declModifier` -> `modifier` rule (swift-syntax unifies the
visibility/function/member/mutation/ownership modifiers into one
kind).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget operator handling to the swift-syntax AST. Because Swift's
grammar has no operator precedence, the parser front-end folds operator
chains into nested `infixOperatorExpr`s (see swift-syntax-rs), so a
single rule replaces the tree-sitter grammar's per-precedence binary
rules (additive, multiplicative, comparison, equality, conjunction,
disjunction, bitwise, range, nil-coalescing). The output is unchanged;
only the input matching differs:
- `binaryOperatorExpr` unwraps to the `infix_operator` leaf.
- A `binaryOperator`-based `infixOperatorExpr` becomes `binary_expr`, or
`compound_assign_expr` when the operator's spelling is a compound
assignment — merging the tree-sitter grammar's separate binary and
compound-assignment rules (the operator kinds are structurally
identical, distinguishable only by spelling).
- An `assignmentExpr`-based `infixOperatorExpr` becomes `assign_expr`.
- An unresolved chain stays a flat `sequenceExpr` ->
`unresolved_operator_sequence`.
- `prefixOperatorExpr` -> prefix `unary_expr`; `tupleExpr` -> opaque
`tuple_expr`; `codeBlock` -> `block`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the top-level, literal, and name mapping rules from the
tree-sitter grammar to the swift-syntax AST. The output each rule builds
is unchanged; only the input pattern differs, reflecting the different
AST shape:
- `source_file` -> `sourceFile` (statements live in an elided
`statements` collection of `codeBlockItem` wrappers); the tree-sitter
`global_declaration` / `local_declaration` wrappers have no
swift-syntax counterpart.
- The lexical integer/string variants (`hex_literal`, `oct_literal`,
`multi_line_string_literal`, ...) collapse into single
`integerLiteralExpr` / `stringLiteralExpr` kinds.
- `simple_identifier` and `referenceable_operator` both become
`declReferenceExpr` (its `baseName` is the referenced name or
operator).
This is the first step of an in-place, rule-by-rule migration;
intermediate commits do not pass the corpus test (the runtime front-end
is still tree-sitter) — the corpus is regenerated once at the end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the plumbing for the swift-syntax front-end, without yet switching
the runtime over to it (the Swift front-end still parses with
tree-sitter):
- `languages/swift/parse.rs` shells out to the separate
`swift-syntax-parse` binary and adapts its JSON into a `yeast::Ast`
(plus side-channel `extra` tokens) via `swift_adapter`. Running the
parser out-of-process keeps the Swift toolchain out of the extractor's
own build. It is wired in as a module but left `allow(dead_code)`
until the runtime uses it.
- `swift_node_types.yml` is the authoritative swift-syntax input schema
(generated from swift-syntax by a one-off tool). The adapter seeds
every parse with it, pre-registering every input kind and field so
that rule matching never references a name absent from a given file's
tree. The adapter now emits `ExtraToken`s directly during its single
traversal, so `parse.rs` hands the parsed tree straight through with
no second pass.
- `ast_types.yml` gains an `unresolved_operator_sequence` type (with an
`expr_or_operator` union) for flat operator chains the parser can't
resolve — e.g. a chain using an operator imported from another module,
whose precedence is unknown. Nothing produces it yet; the mapping
rules that do are added when the rules are ported. The dbscheme and QL
library are regenerated to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In our swift-syntax wrapper, we now attempt to fold all operator
sequences (i.e. `sequenceExpr` nodes) into appropriate
`infixOperatorExpr` nodes, assuming the requisite operator definitions
are present.
Currently, we only consider operators that are defined in the standard
library, and operators that are defined in the current file, leaving
operators defined in separate modules as future work.
The folding is done maximally -- if an argument of an unknown operator
can be folded in isolation, then this is done. Each top-level sequence
is folded independently, so a single unknown operator leaves only its
own sequence flat rather than aborting folding elsewhere.
The AST dump previously emitted named fields in field-id order, which
made it dependant on registration order and so it could differ between
front-ends. We now emit them in the order declared in the node-types
YAML instead, so that the order is kept stable.
It seems that using swift_version_file has some issues when `codeql` is
consumed as a dependency module. I'm hoping hardcoding the version
instead will fix this, but ideally we should find a more robust
solution.
- BUILD.bazel: require x86_64 on the Linux branch of the compatibility
select. The only registered standalone Linux Swift toolchain is
x86_64-only, so without the CPU constraint Linux/aarch64 targets
stayed
"compatible" and then failed toolchain resolution instead of being
skipped cleanly.
- build.rs: emitting `rerun-if-changed` disables Cargo's default
whole-package scan, so list every input to `swift build` — the package
manifests, `Package.resolved`, `.swift-version`, and the whole Sources
tree — so stale shim/toolchain output can't linger. (`.build/` is left
unwatched, as it is this build's own output.)
- src/lib.rs: a null result from the shim is not a parse failure —
SwiftParser recovers from invalid syntax and always yields a tree — so
it
means the shim failed to serialize/allocate the JSON. Fix the error
message and the variant doc accordingly.
- README.md: `.swift-version` is not honored by the macOS Bazel build
(which uses the host Xcode toolchain); document that it pins the Linux
and local builds only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Also gathers these into a separate side-channel during the initial AST
construction. That way, we don't encounter these as weird extra nodes
while running yeast.
Introduces new yeast types for `Point`s and `Range`s (corresponding to
the tree-sitter equivalents), since it seemed silly to have
`swift-syntax-rs` pull in `tree-sitter` just to have those types
available.
This _does_ mean there is a slight overhead in the shared extractor when
converting between these types, but I think this is negligible.
Adds a preliminary mapping that decodes the JSON into the yeast AST. The
JSON AST itself is still very verbose, but it would be useful to see if
it actually contains all of the things we want it to contain.
Adds an initial prototype of an interface from Rust to Swift, which
enables us to use the `swift-syntax` package for parsing.
At present, the parsed AST is passed between Swift and Rust as a JSON
string.
Turns out, `(_)` would match both named and unnamed nodes, as we never
checked the value of the `match_unnamed` field. This is the real reason
why the final catch-all rule we removed in the last commit was
superfluous -- unnamed nodes were being caught by the penultimate rule
instead (and mapped to `unsupported_node`).
Having fixed the bug, we now (correctly) get errors due to unmatched
unnamed nodes in the input. To fix this, we change the catch-all rule to
match unnamed nodes as well. This restores the previous behaviour
exactly.
At some point, we should find a better way to handle unnamed nodes, as
it seems wasteful to map these to `unsupported_node` (since we in
practice only use them for their string content). Perhaps we should not
attempt to translate unnamed nodes at all?
In order to facilitate static type checking of rules (and to make it
easier for human readers as well), rust blocks at the root level (i.e.
rules of the form `... => { ... }`) must now have a type annotation in
front.
All other forms are unaffected: if the right hand side of a rule is a
tree, we can read the type of the root node directly. For interpolations
that happen inside of such a tree, we can recover the type by looking at
what field we're interpolating into, and consulting the output schema.
All existing uses have been updated to have the appropriate type
annotations, though these are of course not checked yet (and so could be
wrong).
Finally, this commit also removes the final catch-all rule `_ @node =>
{node}`. Because of the preceding rule that matches `(_) @node`, this
rule would only ever match unnamed nodes, and I think in practice it did
not match at all (at least not in our current set of tests).
To give it a proper type we would have to add some notion of an "any"
type, which I would like to avoid. If it _does_ turn out to be needed,
we can easily add it back (ideally with a test-case that shows why it's
still needed).
This macro allows the easy addition of multiple rules at the same time.
In addition, it also accepts an input and output schema, which
eventually will be used to check the validity of the rewrite rules.
In order to avoid having context changes bubble up through the tree (or
from one sibling to another) the current framework makes a copy of the
context before calling `translate` recursively, and then restores it
afterwards.
However, this is a bit silly -- after we're done with all of the
translations, there's really no need to restore the context (as it
doesn't get accessed again).
So instead we change it from "save and then restore" to "clone and then
drop". Each rule invocation gets its own copy of the context, and simply
drops it when it's done.
A previous commit added a translate_reset method on BuildCtx, which had
the effect of performing a translation in a completely empty context.
One issue with this is that this is an all-or-nothing proposition -- If
you want to preserve _some_ parts of the context, you have to do
something more complicated. Moreover, if you introduce a contextual
value that _should_ be preserved, all of the existing uses of
translate_reset now silently do the wrong thing.
There are two patterns that we want to address. The first one is "modify
the context in some way, then do a translation". If the translation is
the last step of a Rust block, then we don't actually need
translate_reset -- we could just reset the context and then call
`translate`. The fact that the outer context is restored afterwards
means it's okay to make destructive changes to `ctx.user_ctx` -- none of
these changes will persist.
The second pattern is the same, but where we want to do more
translations using the original context, after having performed a
translation with a modified context. In this case, we cannot just
overwrite the context, since that would invalidate the subsequent
translations.
Instead, we introduce a new method `ctx.scoped` which takes a closure as
an argument. With this we can now write
```
ctx.scoped(|ctx| ctx.reset(); ctx.translate(...));
```
and the closure is run with a copy of `ctx` that has a clone of
`user_ctx` on the inside, so no changes will persist.
(You may wonder: why not just clone `ctx` and use the clone? The answer
is that `ctx` owns mutable pointers to the AST etc., and this makes it
awkward to just "clone" it. The closure circumvents this issue nicely,
since it can borrow these pointers internally.)
For now, this rewrite has the same behaviour as the version that used
translate_reset -- we clear the entire `user_ctx`. However, we could
imagine being more fine-grained in this approach, by implementing, say,
SwiftContext::reset_modifiers (which would only affect what modifiers
are currently in the context, leaving everything else as-is).
