Retarget `if`/`guard`/`switch`/ternary and the case/binding patterns to
the swift-syntax AST, output unchanged. swift-syntax distinguishes a
binding pattern (`valueBindingPattern`, `let x`) from a match pattern
(`expressionPattern`, `someConstant`) by node kind, so the tree-sitter
path's context-based `in_binding_pattern` disambiguation is removed:
- `ifExpr`/`guardStmt`/`ternaryExpr` ->
`if_expr`/`guard_if_stmt`/`if_expr`;
`switchExpr` + `switchCase` -> `switch_expr` + `switch_case` (comma
cases become an `or_pattern`); `conditionElement`/`switchCaseItem`
unwrap; a statement-position `if`/`switch`/`do` is unwrapped from its
`expressionStmt`.
- `optionalBindingCondition` (`if let`) and `matchingPatternCondition`
(`if case`) -> `pattern_guard_expr`.
- An `expressionPattern` wrapping a leading-dot or qualified call ->
`constructor_pattern` (setting `ctx.in_pattern`);
`valueBindingPattern` unwraps; a bare `expressionPattern` ->
`expr_equality_pattern`; a wildcard
(`discardAssignmentExpr`) -> `ignore_pattern`; a `tupleExpr` match
pattern -> `tuple_pattern`; `isTypePattern` (`case is T`) ->
`unsupported_node`.
- The `labeledExpr` argument rules gain `in_pattern`-aware pattern
variants so an enum-case pattern's arguments (`case .foo(let x, _)`)
become `pattern_element`s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget closures to the swift-syntax AST, output unchanged.
swift-syntax nests the whole closure header under an optional
`closureSignature`, so a single `closureExpr` rule (with optional
attributes, capture list, parameter clause, and return clause) replaces
the tree-sitter `lambda_literal` rule.
The parameter clause is a union of the parenthesised form
(`closureParameterClause`, unwrapped to its `closureParameter` children)
and the shorthand form (`closureShorthandParameter`, a bare name); one
rule each replaces the four `lambda_parameter` variants.
`closureCapture` -> `variable_declaration` (an optional ownership
specifier becomes a modifier; an explicit capture initializer becomes
the bound value).
The trailing-closure call form is already handled by the
`functionCallExpr`
`trailingClosure` variant, so the tree-sitter trailing-closure call rule
is
dropped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget function declarations, calls, member access, and control
transfer to the swift-syntax AST, output unchanged:
- `functionDecl` -> `function_declaration` (parameters and return type
nest under `signature`; the body is a `codeBlock`). A bodyless
function (a protocol requirement) still emits an empty `block`.
- `functionParameter` -> `parameter`: two names give the external label
and internal name, one name just the internal name; the default value
is handled inline, so the `ctx.default_value` threading (and its
`SwiftContext` field) is removed. The declared type is dropped: in the
tree-sitter path the untyped-parameter rule was ordered first and
shadowed the typed one, so the baseline emits no parameter type.
- A function reference spelled with argument labels (`f(x:y:z:)`) is a
`declReferenceExpr` with `argumentNames`; it is mapped to
`unsupported_node` (matched before the bare-name rule) so downstream
QL isn't handed a malformed reference, as in the tree-sitter path.
- `functionCallExpr` -> `call_expr` (a trailing closure becomes a final
unlabelled argument); `labeledExpr` -> `argument`;
`memberAccessExpr` -> `member_access_expr` (base-ful matched before
leading-dot).
- `returnStmt`/`breakStmt`/`continueStmt`/`throwStmt` ->
`return_expr`/`break_expr`/`continue_expr`/`throw_expr`, collapsing
the labelled/unlabelled variants via optional captures.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget type expressions to the swift-syntax AST, output unchanged:
- `user_type` -> `identifierType` (its `name` is the type-name token).
- The sugared types keep desugaring to `generic_type_expr`:
`optionalType` -> Optional<T>, `arrayType` -> Array<T>,
`dictionaryType` -> Dictionary<K, V>.
- A generic type with explicit arguments (`Set<Int>`) stays opaque (its
whole source text as the name), matched before the plain
`identifierType` rule.
This matches the tree-sitter `user_type` rule, which was also opaque.
- Tuple types (`(Int, String)`) -> `tuple_type_expr` and function types
(`(Int) -> Bool`) -> `function_type_expr`. swift-syntax holds both as
`tupleTypeElement`s but a tuple element maps to `tuple_type_element`
while a function parameter maps to `parameter`; the containers set
`SwiftContext::in_function_type` for their direct children so the
shared `tupleTypeElement` rule emits the right kind (and nested types
stay correct).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `let`/`var` bindings to the swift-syntax AST, preserving the
output:
- A `variableDecl` publishes its `bindingSpecifier` (`let`/`var`) as the
binding modifier, followed by its attributes and modifiers (`@objc`,
`public`, `static`, …), and flattens each `patternBinding` into its
own `variable_declaration`, tagging non-first ones
`chained_declaration`. - One `patternBinding` rule with optional
`typeAnnotation`/`initializer` covers `let x`, `let x = e`,
`let x: T`, and `let x: T = e`.
- `identifierPattern` -> `name_pattern`; `tuplePattern` /
`tuplePatternElement` -> `tuple_pattern` / `pattern_element` (tuple
destructuring), carrying an optional element label through as the
`pattern_element` key.
- `codeBlockItem` now captures `_*` / annotates `stmt*` so a
multi-binding declaration splices as several statements.
- Add a `declModifier` -> `modifier` rule (swift-syntax unifies the
visibility/function/member/mutation/ownership modifiers into one
kind).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget operator handling to the swift-syntax AST. Because Swift's
grammar has no operator precedence, the parser front-end folds operator
chains into nested `infixOperatorExpr`s (see swift-syntax-rs), so a
single rule replaces the tree-sitter grammar's per-precedence binary
rules (additive, multiplicative, comparison, equality, conjunction,
disjunction, bitwise, range, nil-coalescing). The output is unchanged;
only the input matching differs:
- `binaryOperatorExpr` unwraps to the `infix_operator` leaf.
- A `binaryOperator`-based `infixOperatorExpr` becomes `binary_expr`, or
`compound_assign_expr` when the operator's spelling is a compound
assignment — merging the tree-sitter grammar's separate binary and
compound-assignment rules (the operator kinds are structurally
identical, distinguishable only by spelling).
- An `assignmentExpr`-based `infixOperatorExpr` becomes `assign_expr`.
- An unresolved chain stays a flat `sequenceExpr` ->
`unresolved_operator_sequence`.
- `prefixOperatorExpr` -> prefix `unary_expr`; `tupleExpr` -> opaque
`tuple_expr`; `codeBlock` -> `block`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the top-level, literal, and name mapping rules from the
tree-sitter grammar to the swift-syntax AST. The output each rule builds
is unchanged; only the input pattern differs, reflecting the different
AST shape:
- `source_file` -> `sourceFile` (statements live in an elided
`statements` collection of `codeBlockItem` wrappers); the tree-sitter
`global_declaration` / `local_declaration` wrappers have no
swift-syntax counterpart.
- The lexical integer/string variants (`hex_literal`, `oct_literal`,
`multi_line_string_literal`, ...) collapse into single
`integerLiteralExpr` / `stringLiteralExpr` kinds.
- `simple_identifier` and `referenceable_operator` both become
`declReferenceExpr` (its `baseName` is the referenced name or
operator).
This is the first step of an in-place, rule-by-rule migration;
intermediate commits do not pass the corpus test (the runtime front-end
is still tree-sitter) — the corpus is regenerated once at the end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the plumbing for the swift-syntax front-end, without yet switching
the runtime over to it (the Swift front-end still parses with
tree-sitter):
- `languages/swift/parse.rs` shells out to the separate
`swift-syntax-parse` binary and adapts its JSON into a `yeast::Ast`
(plus side-channel `extra` tokens) via `swift_adapter`. Running the
parser out-of-process keeps the Swift toolchain out of the extractor's
own build. It is wired in as a module but left `allow(dead_code)`
until the runtime uses it.
- `swift_node_types.yml` is the authoritative swift-syntax input schema
(generated from swift-syntax by a one-off tool). The adapter seeds
every parse with it, pre-registering every input kind and field so
that rule matching never references a name absent from a given file's
tree. The adapter now emits `ExtraToken`s directly during its single
traversal, so `parse.rs` hands the parsed tree straight through with
no second pass.
- `ast_types.yml` gains an `unresolved_operator_sequence` type (with an
`expr_or_operator` union) for flat operator chains the parser can't
resolve — e.g. a chain using an operator imported from another module,
whose precedence is unknown. Nothing produces it yet; the mapping
rules that do are added when the rules are ported. The dbscheme and QL
library are regenerated to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prepare yeast for front-ends that do not parse with tree-sitter (e.g.
the swift-syntax front-end, whose parser hands us a ready-built
`yeast::Ast`):
- `Runner`/`ConcreteDesugarer` now hold `Option<tree_sitter::Language>`.
New constructors `Runner::with_schema_no_language`,
`ConcreteDesugarer::without_language`, and
`DesugaringConfig::build_schema_no_language` build the schema from the
output node-types YAML alone. The parsing entry points
(`run`/`run_from_tree`) error when no language is present;
`run_from_ast` needs none.
- `BuildCtx::source_text` is a small convenience for Rust-block rules
that read a captured token's source text.
- AST-dump type validation now resolves field constraints and required
fields by field NAME rather than by field id. A field id is local to
the schema that assigned it, so an AST built by one schema (e.g. an
external parser's adapter) could not be validated against another (the
output node-types schema) without re-keying. Looking up by name keeps
the two schemas full independent: they share field names, not ids.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In our swift-syntax wrapper, we now attempt to fold all operator
sequences (i.e. `sequenceExpr` nodes) into appropriate
`infixOperatorExpr` nodes, assuming the requisite operator definitions
are present.
Currently, we only consider operators that are defined in the standard
library, and operators that are defined in the current file, leaving
operators defined in separate modules as future work.
The folding is done maximally -- if an argument of an unknown operator
can be folded in isolation, then this is done. Each top-level sequence
is folded independently, so a single unknown operator leaves only its
own sequence flat rather than aborting folding elsewhere.
The AST dump previously emitted named fields in field-id order, which
made it dependant on registration order and so it could differ between
front-ends. We now emit them in the order declared in the node-types
YAML instead, so that the order is kept stable.
Registers the `unified/swift-syntax-rs` workspace member (added in
"unified:
Add swift-syntax-rs") in the tree-sitter extractors crate universe. It
has no
external Rust dependencies, so its dependency entries are empty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
It seems that using swift_version_file has some issues when `codeql` is
consumed as a dependency module. I'm hoping hardcoding the version
instead will fix this, but ideally we should find a more robust
solution.
The doc claimed the schema passed to `Ast::with_schema` must already
have
every node kind and field name registered, contradicting the
registration
methods just below and the swift-syntax adapter, which starts from
`Schema::new()` and registers names on demand during construction.
Document
that up-front registration is optional.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- BUILD.bazel: require x86_64 on the Linux branch of the compatibility
select. The only registered standalone Linux Swift toolchain is
x86_64-only, so without the CPU constraint Linux/aarch64 targets
stayed
"compatible" and then failed toolchain resolution instead of being
skipped cleanly.
- build.rs: emitting `rerun-if-changed` disables Cargo's default
whole-package scan, so list every input to `swift build` — the package
manifests, `Package.resolved`, `.swift-version`, and the whole Sources
tree — so stale shim/toolchain output can't linger. (`.build/` is left
unwatched, as it is this build's own output.)
- src/lib.rs: a null result from the shim is not a parse failure —
SwiftParser recovers from invalid syntax and always yields a tree — so
it
means the shim failed to serialize/allocate the JSON. Fix the error
message and the variant doc accordingly.
- README.md: `.swift-version` is not honored by the macOS Bazel build
(which uses the host Xcode toolchain); document that it pins the Linux
and local builds only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
