Adds a few tests that validate that higher-order functions are parsed
correctly into the commonAST representation.
Also removes a redundant assignment to `ctx.in_function_type` that
happend after all translations had taken place (and so nothing would
actually read this field).
Two fixes prompted by review of the swift-syntax switch-over.
Parser resolution (`parse.rs`): `parse_bin` now resolves the
`swift-syntax-parse` executable in priority order — the
`CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` override, then a copy next
to the extractor executable (as a shipped extractor pack lays it out:
`tools/<platform>/{extractor,swift-syntax-parse}`), then a
bare `PATH` lookup. This lets a packaged extractor find its parser with
no environment setup. (Bundling the binary into the pack, together with
its Swift runtime, is a separate follow-up.)
Corpus test guard (`corpus_tests.rs`): `parser_available` previously
treated *any* parser error as "unavailable" and skipped the entire
corpus suite, so a parser that was present but crashed or emitted
invalid JSON would silently skip the exact regressions the suite exists
to catch. It now uses the new `binary_available`, which reports whether
the *executable* can be launched (false only when it cannot be found,
e.g. no Swift toolchain); a launchable-but-failing parser makes the
suite run and fail.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerate `types/property-with-getter-and-setter`, whose mapped AST now
differs from the tree-sitter output: the backing `private var _v`
retains its `private` modifier (`modifier "var"` + `modifier
"private"`), whereas the tree-sitter path dropped it (its
`visibility_modifier` node carried no text). The swift-syntax
front-end preserves the modifier, so the mapped AST is strictly richer
here.
The raw (second) section is regenerated to the swift-syntax AST like the
other cases; the mapped (third) section gains the retained `private`
modifier.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that the Swift front-end is swift-syntax, regenerate the second (raw
parse tree) section of the corpus `.output` files to hold the
swift-syntax AST the adapter builds, instead of the old tree-sitter
parse tree.
These 98 cases map to a byte-for-byte identical mapped AST (the third
section), so only their raw section changes — the mapping rules were
ported to produce the same output. The one case whose mapped AST differs
(`types/property-with-getter-and-setter`) is handled separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip the runtime Swift front-end from tree-sitter to swift-syntax. The
mapping rules were already ported, so the mapped AST is unchanged; this
makes the switch live.
- `language_spec` now builds a language-free desugarer
(`ConcreteDesugarer::without_language`) and wires the swift-syntax
parser (`swift_parse::parse`) as the front-end, dropping the
tree-sitter language and node types. The desugarer supplies the output
schema, so `node_types` is left empty.
- The `swift_adapter`/`swift_parse` modules are no longer
`allow(dead_code)`: they are now reached from the live extraction
path.
- `corpus_tests` skips (rather than fails) when the external
`swift-syntax-parse` binary is unavailable, since it cannot run
without the Swift-backed parser.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The AST dump previously emitted named fields in field-id order, which
made it dependant on registration order and so it could differ between
front-ends. We now emit them in the order declared in the node-types
YAML instead, so that the order is kept stable.
This macro allows the easy addition of multiple rules at the same time.
In addition, it also accepts an input and output schema, which
eventually will be used to check the validity of the rewrite rules.
The corpus tests interleaved hand-written content (test cases) with
generated content (printed ASTs).
This made merge conflicts hard to resolve because you can't just
regnerate the printed ASTs without potentially throwing away new test
cases that came from either branch (or depending on whether the merge
conflict markers appeared, the corpus test could be ruined completely).
The old design did have one nice advantage: Reviewers could see the
printed ASTs alongside the source code from which it was generated.
To preserve this feature, the source code for the test case is itself
included in the generated output file.
Format the touched Rust crates (shared/tree-sitter-extractor,
shared/yeast, shared/yeast-macros, unified/extractor) so the
tree-sitter-extractor CI fmt check passes. No functional changes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoids more "mutation after creation" via prepend_field.
Also adds a test to the corpus for exercising this syntax. Although it's
not evident, the test output was unchanged by this refactoring.
This was necessary since otherwise the generic type of the
user-specified context (which should only be a concern for yeast) starts
to bleed out into the shared extractor. Instead, we type-erase it by
putting it inside the aforementioned trait.
Renames what was previously called `__yeast_ctx` into just `ctx`, and
adds a new field `user_ctx` to this context. Said field can contain a
struct of any user type (necessitating making various parts of the
implementation generic in said type).
Through some Deref magic, field accesses are delegated to the inner
struct (assuming they are not already defined on `ctx`), which should
hopefully make the interface a bit more ergonomic.
Patterns have an unusual parse tree, but now the matching should
at least be a bit easier to follow.
The TODO regarding not being able to pass down context to handle
var/let is still relevant, and can't be solved in the mapping alone.
Adds a test case 'Switch with labeled case pattern arguments' covering:
- case .implicit(isAcknowledged: false) — labeled bool literal
- case .thread(threadRowId: _, let rowId) — labeled wildcard + binding
The current output contains type errors: pattern_element::key is being
produced as name_expr instead of identifier. These will be fixed in the
following commit.
The switch_entry rule was capturing switch_pattern wrapper nodes instead of
drilling into them to extract the actual pattern nodes. This caused patterns
from switch cases to be lost during desugaring.
Changed the pattern match from:
(switch_entry pattern: (switch_pattern)* @pats ...)
to:
(switch_entry pattern: (switch_pattern pattern: @pats)* ...)
This now correctly extracts the pattern field from each switch_pattern node,
ensuring that patterns from cases like 'case 1:' and 'case .circle(let r):'
are preserved in the switch_case AST nodes.
Updated control-flow.txt corpus outputs to reflect the new behavior.
Introduce NodeRef as a typed wrapper around node arena IDs. Captures in
desugaring rules are now bound as NodeRef instead of raw usize, which
prevents accidental misuse and enables source-text-aware rendering.
Add the YeastDisplay trait as an alternative to Display: its
yeast_to_string method receives the Ast, allowing NodeRef to resolve to
the captured node's source text instead of printing a numeric ID.
Store the original source bytes in the Ast so that NodeContent::Range
values (from synthesized literal nodes) can be resolved back to text.
Update yeast-macros to emit NodeRef-typed capture bindings and use
Into::<usize>::into where raw IDs are needed. The #{expr} template
syntax now uses YeastDisplay instead of Display.
The effect is visible in the corpus tests: operator nodes now correctly
render as e.g. operator "+" instead of operator "3".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add corpus test cases for Swift covering closures, collections, control
flow, functions, literals, loops, operators, optionals/errors, types,
and variables. Update existing desugar.txt with raw parse sections.
Note: operator nodes currently render their node ID instead of the actual
operator text (e.g. operator "3" instead of operator "+"). This will be
fixed in the next commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add ast_types.yml defining the unified output AST schema with supertypes
(expr, stmt, condition, pattern) and named nodes (top_level, binary_expr,
name_expr, etc.).
Rewrite swift translation rules to map from tree-sitter Swift grammar to
the unified AST, using one-shot phase rules.
Update the generator to use the output AST schema for dbscheme/QL
generation, and normalize the extraction table prefix to 'unified'.
Improve the corpus test framework to include raw tree-sitter parse output,
type-error checking against the output schema, and better failure
reporting.
Regenerate Ast.qll, unified.dbscheme, and update BasicTest accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
