mirror of
https://github.com/github/codeql.git
synced 2026-07-29 14:56:51 +02:00
bbf52ce7a7
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>