Now that the Swift front-end is swift-syntax, regenerate the second (raw
parse tree) section of the corpus `.output` files to hold the
swift-syntax AST the adapter builds, instead of the old tree-sitter
parse tree.
These 98 cases map to a byte-for-byte identical mapped AST (the third
section), so only their raw section changes — the mapping rules were
ported to produce the same output. The one case whose mapped AST differs
(`types/property-with-getter-and-setter`) is handled separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip the runtime Swift front-end from tree-sitter to swift-syntax. The
mapping rules were already ported, so the mapped AST is unchanged; this
makes the switch live.
- `language_spec` now builds a language-free desugarer
(`ConcreteDesugarer::without_language`) and wires the swift-syntax
parser (`swift_parse::parse`) as the front-end, dropping the
tree-sitter language and node types. The desugarer supplies the output
schema, so `node_types` is left empty.
- The `swift_adapter`/`swift_parse` modules are no longer
`allow(dead_code)`: they are now reached from the live extraction
path.
- `corpus_tests` skips (rather than fails) when the external
`swift-syntax-parse` binary is unavailable, since it cannot run
without the Swift-backed parser.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the remaining member declarations to the swift-syntax AST. An
`initializerDecl` becomes a `constructor_declaration` (its body
optional, so a bodyless protocol requirement still maps);
`deinitializerDecl`, `typeAliasDecl`, and `associatedTypeDecl` map to
`destructor_declaration`, `type_alias_declaration`, and
`associated_type_declaration` respectively.
The tree-sitter subscript and preprocessor-diagnostic rules are dropped:
their swift-syntax counterparts (`subscriptDecl`, `ifConfigDecl`) fall
through to the `unsupported_node` fallback, producing the same output.
This also removes the now-unused `type` unwrap rule (swift-syntax has no
such wrapper node) and retires the last `ctx.literal` helper use in
favour of a `tree!` leaf.
PARITY(tree-sitter): the initializer's parameters are still not emitted,
because the tree-sitter path dropped them too. Emitting them is a future
improvement.
This completes the rule migration: every declaration now maps through
the swift-syntax front-end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget enum cases to the swift-syntax AST. An `enumCaseDecl` flattens
its comma-separated `enumCaseElement`s (non-first tagged
`chained_declaration`) and publishes any case modifiers (e.g.
`indirect`) into `ctx`; an element with a payload becomes a nested
`class_like_declaration` + constructor, an element with a raw value
(`case a = 1`) or a plain element a `variable_declaration`;
`enumCaseParameter` becomes a `parameter`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget property accessors to the swift-syntax AST, output unchanged.
An accessor-bearing `variableDecl` publishes the property name/type into
`ctx`; a computed property (`var v: T { get set }`) emits accessors
carrying the type, while a stored property with observers (`var x = e {
didSet {…} }`) emits the backing `variable_declaration` first.
swift-syntax models get/set/willSet/didSet uniformly as `accessorDecl`,
so a single `accessorDecl` rule — with an optional body distinguishing a
computed accessor from a bodyless protocol requirement — replaces the
tree-sitter grammar's separate computed-getter/setter/modify,
willset/didset, and getter-/setter-specifier rules.
The tree-sitter protocol property and function requirement rules
(`protocol_property_declaration`, `protocol_function_declaration`) are
dropped: swift-syntax models those requirements as ordinary
`variableDecl`s (with bodyless accessors) and `functionDecl`s, already
handled by the general rules.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the nominal type declarations to the swift-syntax AST, output
unchanged.
`classDecl`/`structDecl`/`enumDecl`/`protocolDecl`/`extensionDecl`
each become a `class_like_declaration` tagged with a modifier naming the
declaration keyword, with members drawn from the `memberBlock`; each
`memberBlockItem` unwraps to its contained declaration. `superExpr` maps
to `super_expr`; `self` needs no rule (swift-syntax models it as an
ordinary `declReferenceExpr`, already a `name_expr`).
Following the tree-sitter path (PARITY), the inheritance clause is not
emitted as a `base_type` (the tree-sitter rule captured it positionally,
but the grammar nests it under a field, so it never actually matched);
swift-syntax exposes it cleanly, so that is a correctness improvement to
make once tree-sitter is retired. The tree-sitter grammar's standalone
`self`, dead modifier (`visibility_modifier`, etc.), key-path, and
inheritance-specifier rules are dropped; `#selector`/`#keyPath` (now a
`macroExpansionExpr`) stays an `unsupported_node`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget imports to the swift-syntax AST, output unchanged. swift-syntax
represents the dotted path as a list of `importPathComponent`s, folded
into a `name_expr`/`member_access_expr` chain via `member_chain`. A
single rule handles both forms via an optional `importKindSpecifier`: a
scoped import (`import struct Foo.Bar`) has one and binds the last path
component as a `name_pattern`; a plain import (`import Foundation`) has
none and uses a `bulk_importing_pattern`. Leading attributes
(`@_exported`) and access modifiers (`public`) become `modifier`s.
The tree-sitter multi-part `identifier` rule is dropped: swift-syntax
qualified names are already `memberAccessExpr` chains, and import paths
are `importPathComponent`s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget optional chaining, `try`, `do`/`catch`, casts, type tests,
`await`, and force-unwrap to the swift-syntax AST, output unchanged:
- `optionalChainingExpr` (`x?`) unwraps transparently (the enclosing
member access / call carries the semantics).
- `tryExpr` -> prefix `unary_expr`; swift-syntax splits the operator
into a `try` keyword and an optional `?`/`!` mark, recombined into one
`prefix_operator` spelling.
- `doStmt` -> `try_expr` with `catchClause` -> `catch_clause`; a `catch`
binds the first `catchItem`'s pattern and optional `where` guard.
- `asExpr` (`x as`/`as?`/`as!` `T`) -> `type_cast_expr`, `isExpr`
(`x is T`) -> `type_test_expr`, and `awaitExpr` -> prefix
`unary_expr`.
- `forceUnwrapExpr` (`x!`) -> postfix `unary_expr` (swift-syntax has a
dedicated node; the tree-sitter path used the generic postfix
operator).
A couple of rewritten rules keep their opening inlined so this commit's
diff stays readable; a later commit restores the canonical formatting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget collection literals and subscripts to the swift-syntax AST,
output unchanged: `arrayExpr` -> `array_literal` (each `arrayElement`
unwraps to its expression); `dictionaryExpr` -> an opaque `map_literal`
leaf (its source span, matching the tree-sitter path); and
`subscriptCallExpr` (`xs[0]`) -> `call_expr`, mirroring the tree-sitter
grammar's treatment of a subscript as a call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget loops to the swift-syntax AST, output unchanged: `forStmt` ->
`for_each_stmt` (the optional `where` clause becomes the `guard`),
`whileStmt` -> `while_stmt`, `repeatStmt` -> `do_while_stmt`, and
`labeledStmt` -> `labeled_stmt`. Unlike the tree-sitter grammar,
swift-syntax stores a labeled statement's label and colon as separate
tokens, so the label token is already the bare name (no trailing `:` to
strip). A `repeat`-`while` loop has a single condition in swift-syntax
(not a condition list), so it needs no `and_chain`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `if`/`guard`/`switch`/ternary and the case/binding patterns to
the swift-syntax AST, output unchanged. swift-syntax distinguishes a
binding pattern (`valueBindingPattern`, `let x`) from a match pattern
(`expressionPattern`, `someConstant`) by node kind, so the tree-sitter
path's context-based `in_binding_pattern` disambiguation is removed:
- `ifExpr`/`guardStmt`/`ternaryExpr` ->
`if_expr`/`guard_if_stmt`/`if_expr`;
`switchExpr` + `switchCase` -> `switch_expr` + `switch_case` (comma
cases become an `or_pattern`); `conditionElement`/`switchCaseItem`
unwrap; a statement-position `if`/`switch`/`do` is unwrapped from its
`expressionStmt`.
- `optionalBindingCondition` (`if let`) and `matchingPatternCondition`
(`if case`) -> `pattern_guard_expr`.
- An `expressionPattern` wrapping a leading-dot or qualified call ->
`constructor_pattern` (setting `ctx.in_pattern`);
`valueBindingPattern` unwraps; a bare `expressionPattern` ->
`expr_equality_pattern`; a wildcard
(`discardAssignmentExpr`) -> `ignore_pattern`; a `tupleExpr` match
pattern -> `tuple_pattern`; `isTypePattern` (`case is T`) ->
`unsupported_node`.
- The `labeledExpr` argument rules gain `in_pattern`-aware pattern
variants so an enum-case pattern's arguments (`case .foo(let x, _)`)
become `pattern_element`s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget closures to the swift-syntax AST, output unchanged.
swift-syntax nests the whole closure header under an optional
`closureSignature`, so a single `closureExpr` rule (with optional
attributes, capture list, parameter clause, and return clause) replaces
the tree-sitter `lambda_literal` rule.
The parameter clause is a union of the parenthesised form
(`closureParameterClause`, unwrapped to its `closureParameter` children)
and the shorthand form (`closureShorthandParameter`, a bare name); one
rule each replaces the four `lambda_parameter` variants.
`closureCapture` -> `variable_declaration` (an optional ownership
specifier becomes a modifier; an explicit capture initializer becomes
the bound value).
The trailing-closure call form is already handled by the
`functionCallExpr`
`trailingClosure` variant, so the tree-sitter trailing-closure call rule
is
dropped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget function declarations, calls, member access, and control
transfer to the swift-syntax AST, output unchanged:
- `functionDecl` -> `function_declaration` (parameters and return type
nest under `signature`; the body is a `codeBlock`). A bodyless
function (a protocol requirement) still emits an empty `block`.
- `functionParameter` -> `parameter`: two names give the external label
and internal name, one name just the internal name; the default value
is handled inline, so the `ctx.default_value` threading (and its
`SwiftContext` field) is removed. The declared type is dropped: in the
tree-sitter path the untyped-parameter rule was ordered first and
shadowed the typed one, so the baseline emits no parameter type.
- A function reference spelled with argument labels (`f(x:y:z:)`) is a
`declReferenceExpr` with `argumentNames`; it is mapped to
`unsupported_node` (matched before the bare-name rule) so downstream
QL isn't handed a malformed reference, as in the tree-sitter path.
- `functionCallExpr` -> `call_expr` (a trailing closure becomes a final
unlabelled argument); `labeledExpr` -> `argument`;
`memberAccessExpr` -> `member_access_expr` (base-ful matched before
leading-dot).
- `returnStmt`/`breakStmt`/`continueStmt`/`throwStmt` ->
`return_expr`/`break_expr`/`continue_expr`/`throw_expr`, collapsing
the labelled/unlabelled variants via optional captures.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget type expressions to the swift-syntax AST, output unchanged:
- `user_type` -> `identifierType` (its `name` is the type-name token).
- The sugared types keep desugaring to `generic_type_expr`:
`optionalType` -> Optional<T>, `arrayType` -> Array<T>,
`dictionaryType` -> Dictionary<K, V>.
- A generic type with explicit arguments (`Set<Int>`) stays opaque (its
whole source text as the name), matched before the plain
`identifierType` rule.
This matches the tree-sitter `user_type` rule, which was also opaque.
- Tuple types (`(Int, String)`) -> `tuple_type_expr` and function types
(`(Int) -> Bool`) -> `function_type_expr`. swift-syntax holds both as
`tupleTypeElement`s but a tuple element maps to `tuple_type_element`
while a function parameter maps to `parameter`; the containers set
`SwiftContext::in_function_type` for their direct children so the
shared `tupleTypeElement` rule emits the right kind (and nested types
stay correct).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `let`/`var` bindings to the swift-syntax AST, preserving the
output:
- A `variableDecl` publishes its `bindingSpecifier` (`let`/`var`) as the
binding modifier, followed by its attributes and modifiers (`@objc`,
`public`, `static`, …), and flattens each `patternBinding` into its
own `variable_declaration`, tagging non-first ones
`chained_declaration`. - One `patternBinding` rule with optional
`typeAnnotation`/`initializer` covers `let x`, `let x = e`,
`let x: T`, and `let x: T = e`.
- `identifierPattern` -> `name_pattern`; `tuplePattern` /
`tuplePatternElement` -> `tuple_pattern` / `pattern_element` (tuple
destructuring), carrying an optional element label through as the
`pattern_element` key.
- `codeBlockItem` now captures `_*` / annotates `stmt*` so a
multi-binding declaration splices as several statements.
- Add a `declModifier` -> `modifier` rule (swift-syntax unifies the
visibility/function/member/mutation/ownership modifiers into one
kind).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget operator handling to the swift-syntax AST. Because Swift's
grammar has no operator precedence, the parser front-end folds operator
chains into nested `infixOperatorExpr`s (see swift-syntax-rs), so a
single rule replaces the tree-sitter grammar's per-precedence binary
rules (additive, multiplicative, comparison, equality, conjunction,
disjunction, bitwise, range, nil-coalescing). The output is unchanged;
only the input matching differs:
- `binaryOperatorExpr` unwraps to the `infix_operator` leaf.
- A `binaryOperator`-based `infixOperatorExpr` becomes `binary_expr`, or
`compound_assign_expr` when the operator's spelling is a compound
assignment — merging the tree-sitter grammar's separate binary and
compound-assignment rules (the operator kinds are structurally
identical, distinguishable only by spelling).
- An `assignmentExpr`-based `infixOperatorExpr` becomes `assign_expr`.
- An unresolved chain stays a flat `sequenceExpr` ->
`unresolved_operator_sequence`.
- `prefixOperatorExpr` -> prefix `unary_expr`; `tupleExpr` -> opaque
`tuple_expr`; `codeBlock` -> `block`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the top-level, literal, and name mapping rules from the
tree-sitter grammar to the swift-syntax AST. The output each rule builds
is unchanged; only the input pattern differs, reflecting the different
AST shape:
- `source_file` -> `sourceFile` (statements live in an elided
`statements` collection of `codeBlockItem` wrappers); the tree-sitter
`global_declaration` / `local_declaration` wrappers have no
swift-syntax counterpart.
- The lexical integer/string variants (`hex_literal`, `oct_literal`,
`multi_line_string_literal`, ...) collapse into single
`integerLiteralExpr` / `stringLiteralExpr` kinds.
- `simple_identifier` and `referenceable_operator` both become
`declReferenceExpr` (its `baseName` is the referenced name or
operator).
This is the first step of an in-place, rule-by-rule migration;
intermediate commits do not pass the corpus test (the runtime front-end
is still tree-sitter) — the corpus is regenerated once at the end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the plumbing for the swift-syntax front-end, without yet switching
the runtime over to it (the Swift front-end still parses with
tree-sitter):
- `languages/swift/parse.rs` shells out to the separate
`swift-syntax-parse` binary and adapts its JSON into a `yeast::Ast`
(plus side-channel `extra` tokens) via `swift_adapter`. Running the
parser out-of-process keeps the Swift toolchain out of the extractor's
own build. It is wired in as a module but left `allow(dead_code)`
until the runtime uses it.
- `swift_node_types.yml` is the authoritative swift-syntax input schema
(generated from swift-syntax by a one-off tool). The adapter seeds
every parse with it, pre-registering every input kind and field so
that rule matching never references a name absent from a given file's
tree. The adapter now emits `ExtraToken`s directly during its single
traversal, so `parse.rs` hands the parsed tree straight through with
no second pass.
- `ast_types.yml` gains an `unresolved_operator_sequence` type (with an
`expr_or_operator` union) for flat operator chains the parser can't
resolve — e.g. a chain using an operator imported from another module,
whose precedence is unknown. Nothing produces it yet; the mapping
rules that do are added when the rules are ported. The dbscheme and QL
library are regenerated to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prepare yeast for front-ends that do not parse with tree-sitter (e.g.
the swift-syntax front-end, whose parser hands us a ready-built
`yeast::Ast`):
- `Runner`/`ConcreteDesugarer` now hold `Option<tree_sitter::Language>`.
New constructors `Runner::with_schema_no_language`,
`ConcreteDesugarer::without_language`, and
`DesugaringConfig::build_schema_no_language` build the schema from the
output node-types YAML alone. The parsing entry points
(`run`/`run_from_tree`) error when no language is present;
`run_from_ast` needs none.
- `BuildCtx::source_text` is a small convenience for Rust-block rules
that read a captured token's source text.
- AST-dump type validation now resolves field constraints and required
fields by field NAME rather than by field id. A field id is local to
the schema that assigned it, so an AST built by one schema (e.g. an
external parser's adapter) could not be validated against another (the
output node-types schema) without re-keying. Looking up by name keeps
the two schemas full independent: they share field names, not ids.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In our swift-syntax wrapper, we now attempt to fold all operator
sequences (i.e. `sequenceExpr` nodes) into appropriate
`infixOperatorExpr` nodes, assuming the requisite operator definitions
are present.
Currently, we only consider operators that are defined in the standard
library, and operators that are defined in the current file, leaving
operators defined in separate modules as future work.
The folding is done maximally -- if an argument of an unknown operator
can be folded in isolation, then this is done. Each top-level sequence
is folded independently, so a single unknown operator leaves only its
own sequence flat rather than aborting folding elsewhere.
The AST dump previously emitted named fields in field-id order, which
made it dependant on registration order and so it could differ between
front-ends. We now emit them in the order declared in the node-types
YAML instead, so that the order is kept stable.
Registers the `unified/swift-syntax-rs` workspace member (added in
"unified:
Add swift-syntax-rs") in the tree-sitter extractors crate universe. It
has no
external Rust dependencies, so its dependency entries are empty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
It seems that using swift_version_file has some issues when `codeql` is
consumed as a dependency module. I'm hoping hardcoding the version
instead will fix this, but ideally we should find a more robust
solution.
The doc claimed the schema passed to `Ast::with_schema` must already
have
every node kind and field name registered, contradicting the
registration
methods just below and the swift-syntax adapter, which starts from
`Schema::new()` and registers names on demand during construction.
Document
that up-front registration is optional.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- BUILD.bazel: require x86_64 on the Linux branch of the compatibility
select. The only registered standalone Linux Swift toolchain is
x86_64-only, so without the CPU constraint Linux/aarch64 targets
stayed
"compatible" and then failed toolchain resolution instead of being
skipped cleanly.
- build.rs: emitting `rerun-if-changed` disables Cargo's default
whole-package scan, so list every input to `swift build` — the package
manifests, `Package.resolved`, `.swift-version`, and the whole Sources
tree — so stale shim/toolchain output can't linger. (`.build/` is left
unwatched, as it is this build's own output.)
- src/lib.rs: a null result from the shim is not a parse failure —
SwiftParser recovers from invalid syntax and always yields a tree — so
it
means the shim failed to serialize/allocate the JSON. Fix the error
message and the variant doc accordingly.
- README.md: `.swift-version` is not honored by the macOS Bazel build
(which uses the host Xcode toolchain); document that it pins the Linux
and local builds only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
