In order to avoid having context changes bubble up through the tree (or
from one sibling to another) the current framework makes a copy of the
context before calling `translate` recursively, and then restores it
afterwards.
However, this is a bit silly -- after we're done with all of the
translations, there's really no need to restore the context (as it
doesn't get accessed again).
So instead we change it from "save and then restore" to "clone and then
drop". Each rule invocation gets its own copy of the context, and simply
drops it when it's done.
A previous commit added a translate_reset method on BuildCtx, which had
the effect of performing a translation in a completely empty context.
One issue with this is that this is an all-or-nothing proposition -- If
you want to preserve _some_ parts of the context, you have to do
something more complicated. Moreover, if you introduce a contextual
value that _should_ be preserved, all of the existing uses of
translate_reset now silently do the wrong thing.
There are two patterns that we want to address. The first one is "modify
the context in some way, then do a translation". If the translation is
the last step of a Rust block, then we don't actually need
translate_reset -- we could just reset the context and then call
`translate`. The fact that the outer context is restored afterwards
means it's okay to make destructive changes to `ctx.user_ctx` -- none of
these changes will persist.
The second pattern is the same, but where we want to do more
translations using the original context, after having performed a
translation with a modified context. In this case, we cannot just
overwrite the context, since that would invalidate the subsequent
translations.
Instead, we introduce a new method `ctx.scoped` which takes a closure as
an argument. With this we can now write
```
ctx.scoped(|ctx| ctx.reset(); ctx.translate(...));
```
and the closure is run with a copy of `ctx` that has a clone of
`user_ctx` on the inside, so no changes will persist.
(You may wonder: why not just clone `ctx` and use the clone? The answer
is that `ctx` owns mutable pointers to the AST etc., and this makes it
awkward to just "clone" it. The closure circumvents this issue nicely,
since it can borrow these pointers internally.)
For now, this rewrite has the same behaviour as the version that used
translate_reset -- we clear the entire `user_ctx`. However, we could
imagine being more fine-grained in this approach, by implementing, say,
SwiftContext::reset_modifiers (which would only affect what modifiers
are currently in the context, leaving everything else as-is).
There's an awkward divide in yeast between returning a list of nodes or
optional node (both of which are iterable), and returning a single node
(which is not). In practice, we would like all of these cases to be
handled transparently: if a single node is returned, it behaves as if it
were a singleton list containing that node. This gives us a uniform
interface during translation -- no matter what is returned, it will be
an iterable of nodes.
To facilitate this, we make the slightly unorthodox choice of
implementing IntoIterator for Id, with the behaviour detailed above.
The trait had a single implementor (`AstCursor`), three type parameters
of which one (`T`) was never used in any method signature, and one
external consumer that needed `use yeast::Cursor;` in scope just to
call methods on the cursor. The abstraction was overhead without a
second implementor to justify it.
Move the six trait methods to an inherent `impl AstCursor` block;
delete `shared/yeast/src/cursor.rs`, the `pub mod cursor;` and
`pub use cursor::Cursor;` lines in `lib.rs`, and the `use yeast::Cursor;`
in `tree-sitter-extractor`'s `traverse_yeast`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `{expr}.map(p -> tpl)` and `{expr}.reduce_left(first -> init, acc,
elem -> fold)` post-fix chains on `{expr}` placeholders had no
remaining users in the codebase: `.map` was never used, and the
4 `.reduce_left` sites in `swift.rs` were rewritten to plain
`Iterator::reduce` via an `and_chain` helper in an earlier commit.
Removes the entire `parse_chain_suffix` function (~90 lines) and the
`has_chain` detection / dispatch branches at the two call sites
(field-position in `parse_direct_node_inner` and body-position in
`parse_direct_list`). The remaining `{expr}` path is the
trait-dispatched one introduced by the splice-syntax cleanup, which
handles single ids and iterables uniformly via `IntoFieldIds`.
Also strips the chain syntax from the `tree!` macro doc comment.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The empty error string passed to `expect_ident` was dead code (the
preceding lookahead has already confirmed the token is an ident),
but it would have been a confusing message if it ever fired. Replace
with an explicit "unreachable" string that makes the intent
clearer to readers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both accessors returned the same private `kind_name: &'static str`
field; `kind_name()` is widely used (mainly by dump.rs and schema
diagnostics) and `kind()` had only 2 internal callers in lib.rs and
a handful in tests. Pick the more descriptive name and update the
callers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`BuildCtx::prepend_field` and the underlying `Ast::prepend_field_child`
existed to support the create-then-mutate pattern in swift.rs (build
an output node, then prepend modifiers to its `modifier:` field). The
SwiftContext-based refactor on the previous branches eliminated all
such call sites: every emitted declaration now carries its modifiers
from birth, so the in-place prepend operation has no users.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`translate_opt` was a convenience for the manual_rule! body code,
collapsing `Option<I>` to `Option<Id>` via `translate`. Since the
`@@` raw-capture migration replaced manual_rule! with rule!, no
callers remain — the auto-translate prefix handles `Option<Id>`
captures directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`Captures::map_captures`, `Captures::map_captures_to`, and
`Captures::try_map_all_captures` had no callers. The last one was
subsumed by `try_map_captures_except` (which takes a skip list and
degenerates to the old behaviour when the list is empty).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In the initial implementation of yeast, the splice syntax was needed do
distinguish between splicing multiple nodes or just a single node.
However, this was always an ugly "wart" in the syntax, since the user
shouldn't have to worry about these things.
To fix this, we add an `IntoFieldIds` trait that dispatches on the
value's type: `Id` pushes a single id, and a blanket impl for
`IntoIterator<Item: Into<Id>>` handles `Vec<Id>`, `Option<Id>`, and
arbitrary iterator chains.
With this, we no longer need to use the special splice syntax, and hence
we can get rid of it.
Previously, the `Id` type was a bare usize alias. The `NodeRef` newtype
existed solely to carry the AST-aware `YeastDisplay` /
`YeastSourceRange` impls (so that `#{captured_node}` rendered source
text rather than the numeric id) without colliding with the impls for
raw integer types.
This commit promotes `Id` itself to a (transparent) newtype struct and
moves the AST-aware trait impls directly onto it. With `Id` and `usize`
now being different types, the integer-display impl (for `usize`) and
the source-text impl (for `Id`) coexist without conflict, and `NodeRef`
becomes redundant (and so we remove it).
- unified/swift: Mark `binding_kind` as a raw `@@` capture in the
property_declaration rule. It is only used to read its source text
(`ctx.ast.source_text`), never as a translated node. With `@` the
auto-translate prefix would route the unnamed `let`/`var` token
through the catch-all `_ @node => {node}` fallback for a no-op
roundtrip; `@@` makes the intent explicit and removes that reliance.
- shared/yeast/tests: Reword a stale comment in test_raw_capture_marker.
The text claimed a "second assertion" exists in this test, but the
explicit-translation check actually lives in the companion
test_raw_capture_marker_explicit_translate.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With `@@name` available, there's no longer a need to use `manual_rule!`.
Every place where it is used, we can instead just mark the relevant raw
captures as such. This results in quite a lot of cleanup! (Also, to me
at least, it makes these rules a lot easier to reason about.)
A first iteration of this approach resulted in a lot of
`.map(Into::into)` being needed, because `SwiftContext` stores `Id`s,
but captures produce `NodeRef`s. To avoid this, I swapped it around so
that the context stores `NodeRef`s. This does require adding `.into()`
in a few places, but it makes the rest of the code a lot more ergonomic.
The `@@name` capture marker in `rule!` queries skips the
auto-translate prefix for that specific capture, letting the body see
the original capture (and thus delay its translation using
`ctx.translate` until it becomes convenient).
Regular `@name` captures continue to be auto-translated as before.
Specifically these are translated _eagerly_, before the main body of the
rewrite rule is run.
I settled on `@@` as the syntax because it did not add new symbols that
the user has to keep track of (it's still a kind of capture), but it's
still visually distinct enough that the user should be able to tell that
there's something special going on. In principle one could accidentally
write one form of capture where the other was intended, but in practice
this would result in code that did not compile (because the types would
not match).
Format the touched Rust crates (shared/tree-sitter-extractor,
shared/yeast, shared/yeast-macros, unified/extractor) so the
tree-sitter-extractor CI fmt check passes. No functional changes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cleans up a few places where we were constructing trees piece by piece
rather than using the `tree!` macro.
In the process, Copilot noticed an issue that should probably be
addressed: the labeled_statement rule can never fire, since there are no
such nodes in the input. This is possibly a simple as making
_labeled_statement (which _does_ exist) named, but I haven't attempted
this.
Finally, a small change to yeast makes it so that the contents of a {}
interpolation can be a Rust block (previously it could only be a single
expression). This avoids the need to double-wrap instances where you
want to interpolate a single node produced as the final value of some
block.
This was necessary since otherwise the generic type of the
user-specified context (which should only be a concern for yeast) starts
to bleed out into the shared extractor. Instead, we type-erase it by
putting it inside the aforementioned trait.
Adds `manual_rule!` which provides a more low-level interface for
defining rewrites. (I'm not entirely sold on the name, so any
suggestions would be welcome.)
Notably, the captures bound in the body of such rules have _not_ been
translated yet -- they still come from the _input_ tree. It is the
user's duty to call ctx.translate on these (which has the effect of
recursively invoking the translation) before substituting them into the
output.
For _truly_ low-level access, the user can still construct a Rule
directly, but this is now somewhat cumbersome as the closure contained
therein takes quite a few parameters. Still, the possibility remains.
This enables users to specify how and when these captures get
translated. In conjunction with the context mechanism, this can be used
to e.g. translate some piece of information (e.g. the type of
something), record it in the context, and then recursively translate
some other capture that relies on this information. This allows
information to be cleanly passed into descendants (which can be written
using context accesses in the `rule!` macro form).
As a consequence of this change, we now need to pass around a
TranslatorHandle to perform the manual translation. For Repeating rules,
it doesn't really make sense to translate things, so in this case we
simply signal an error.
Also, the implementation of the `rule!` macro changes slightly (without
changing semantics): it now essentially delegates to `Rule::new`,
receiving raw captures, but then immediately applies the translation to
those captures (which, for the majority of cases, is likely the desired
behaviour).
Renames what was previously called `__yeast_ctx` into just `ctx`, and
adds a new field `user_ctx` to this context. Said field can contain a
struct of any user type (necessitating making various parts of the
implementation generic in said type).
Through some Deref magic, field accesses are delegated to the inner
struct (assuming they are not already defined on `ctx`), which should
hopefully make the interface a bit more ergonomic.
