bbf52ce7a7
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
QL for QL
QL for QL is a CodeQL analysis designed to find common bug patterns in QL code.
This analysis is mostly used as a PR check in github/codeql.
QL for QL is experimental technology and not a supported product.
This directory contains the extractor, CodeQL libraries, and queries that power QL for QL.
Some setup is required to use QL for QL (see the below sections).
Building the tools from source
Install Rust (if using VSCode, you may also want the rust-analyzer extension), then run:
cargo build --release
Generating the database schema and QL library
The generated ql/src/ql.dbscheme and ql/src/codeql_ql/ast/internal/TreeSitter.qll files are included in the repository, but they can be re-generated as follows:
./scripts/create-extractor-pack.sh
Building a CodeQL database for a QL program
First, get an extractor pack:
Run ./scripts/create-extractor-pack.sh (Linux/Mac) or .\scripts\create-extractor-pack.ps1 (Windows PowerShell) and the pack will be created in the extractor-pack directory.
Then run
codeql database create <database-path> -l ql -s <project-source-path> --search-path <extractor-pack-path>
CodeQL can be configured to remember the extractor by setting the config file ~/.config/codeql/config to:
--search-path /full/path/to/extractor-pack
Running qltests
Run
codeql test run <test-path> --search-path <repository-root-path>