Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Canonicalize `current_dir()` to match canonicalized file paths (avoids
`\\?\` prefix mismatch on Windows), and normalize backslashes to
forward slashes in relative diagnostic paths.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Diagnostic `location.file` entries were using absolute paths (e.g.
`/home/runner/work/...`), causing broken links in the GitHub UI.
Now relativize against CWD (the source root during extraction), falling
back to a properly percent-encoded `file:` URI for paths outside it.
Fixes https://github.com/github/codeql/issues/21802
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Language and LanguageSpec gain optional output_node_types field.
When set, the generator produces dbscheme/QL from the output types
and the extractor validates TRAP against them.
All existing extractors pass None (no behavior change).
Ruby extract() calls gain vec![] for the new rules parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
extract() gains a rules parameter. When empty, uses tree-sitter native
traversal (no behavior change). When non-empty, runs yeast desugaring
and extracts via traverse_yeast.
Adds AstNode trait abstracting over tree_sitter::Node and yeast::Node,
with minimal changes to existing Visitor methods (Node -> &N in 6
signatures).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous implementation returned None if any of the paths in the
changes JSON couldn't be canonicalized. This could happen for files that
were deleted in the diff. Now, it just ignores paths for which
canonicalize() fails.
Supports only a minimal subset of the project layout specification;
enough to work with the transformers produced by the CLI when building
an overlay database.
