mirror of
https://github.com/github/codeql.git
synced 2025-12-16 16:53:25 +01:00
95 lines
5.4 KiB
Markdown
95 lines
5.4 KiB
Markdown
# How to update the Python parser
|
|
|
|
## Step 1: Add an extractor test
|
|
|
|
Extractor parser tests live in the `tests/parser` directory. There are two different kinds of tests
|
|
|
|
- Tests that compare the behavior of the old (Python-based) and new (`tree-sitter`-based) parsers, and verify that they yield the same output on the given source files, and
|
|
- Tests that compare the output of the parser (old or new) against a fixed `.expected` file.
|
|
|
|
What kind of test is run is determined based on the file name.
|
|
If it ends in either `_new.py` or `_old.py`, then the test is run against an `.expected` file. If not, it is used to compare old against new.
|
|
|
|
In most cases when adding new features, you'll only be interested in modifying the new parser (the old one is mostly there for legacy reasons).
|
|
Thus, you will almost certainly want to create a test that ends in `_new.py`.
|
|
|
|
It's a good habit to start by adding the parser test, as this makes it more easy to test when various bits of the parser have been added/modified successfully.
|
|
|
|
The rest of this document will only concern itself with the process of extending the _new_ parser.
|
|
|
|
To actually _run_ the tests, the easiest way is to use `pytest`.
|
|
In the main `extractor` directory (i.e. where this file is located) run
|
|
|
|
```sh
|
|
pytest tests/test_parser.py
|
|
```
|
|
|
|
and wait for the tests to complete. It is normal and expected that the test seemingly freezes on the first run.
|
|
This is simply because the `tsg-python` Rust binary is being built in the background.
|
|
|
|
Once you have added a new test (or modified an old one) and start making modifications to the parser itself, it quickly becomes tedious to run _all_ the parser tests.
|
|
To run just a single test using `pytest`, use the `::` syntax to specify specific tests.
|
|
For instance, if you want to just run the tests associated with the file `types_new.py`, you would write
|
|
|
|
```sh
|
|
pytest tests/test_parser.py::ParserTest::test_types_new
|
|
```
|
|
|
|
## Step 2: Extend the `tree-sitter-python` grammar
|
|
|
|
The new parser is based on `tree-sitter`, so the first task is to extend the existing `tree-sitter-python` grammar.
|
|
This grammar can be found in the `grammar.js` file in the `tsg-python/tsp` subdirectory of the extractor directory.
|
|
|
|
Note that whenever changes are made to `grammar.js`, you must regenerate the parser files by running
|
|
|
|
```sh
|
|
tree-sitter generate
|
|
```
|
|
|
|
inside the `tsp` directory.
|
|
You'll need to install the `tree-sitter` CLI in order to run this command.
|
|
One way to install it is to use `cargo`:
|
|
|
|
```sh
|
|
cargo install tree-sitter-cli
|
|
```
|
|
|
|
(This presupposes you have `cargo` available, but you'll need this anyway when compiling `tsg-python`.)
|
|
|
|
Once the parser files have been regenerated, they'll get picked up automatically when `tsg-python` is rebuilt.
|
|
|
|
> Pro-tip: When you're done with your parser changes, and go to commit these to a branch, put the autogenerated files in their own commit.
|
|
> This makes it easier to review the changes, and if you need to go back and regenerate the files again, it's easy to modify just that commit.
|
|
|
|
Once you have extended `grammar.js` and regenerated the parser files, you should be able to check that the grammar changes are sufficient by rerunning the parser test using `pytest`. If it fails while producing an AST that doesn't make sense, then you're probably on the right track. If it fails _without_ producing an AST, then something went wrong with the actual `tree-sitter` parse. To check if this is the case, you can run
|
|
|
|
```sh
|
|
tree-sitter parse path/to/test.py
|
|
```
|
|
|
|
and see what kind of errors are emitted (possibly as `ERROR` or `MISSING` nodes in the AST that is output).
|
|
|
|
## Step 3: Extend `python.tsg`
|
|
|
|
Once the grammar has been extended, we need to also tell `tsg-python` how to turn the `tree-sitter-python` AST into something that better matches the AST structure that we use in the Python extractor.
|
|
|
|
For an introduction to the language of `tree-sitter-graph` (and in particular how we use it in the Python extractor), see the `README.md` file in the `tsg-python` directory.
|
|
|
|
## Step 4: Extend the set of known AST nodes
|
|
|
|
If you added new node types, or added fields to known node types, then you'll need to update a few files in the Python extractor before it is able to reconstruct the output from `tsg-python`.
|
|
|
|
New AST nodes should be added in two places: `master.py` and `semmle/python/ast.py`. The former of these is used to automatically generate the Python dbscheme and `AstGenerated.qll`. The latter is what the parser actually uses as its internal representation of the AST.
|
|
|
|
## Step 5: Rebuild the autogenerated AST and database scheme
|
|
|
|
If you made changes to `master.py`, you'll need to regenerate a couple of files. This can be done from within the `extractor` directory using the `make dbscheme` and `make ast` commands. Note that for the latter, you need a copy of the CodeQL CLI present, as it is used to autoformat the `AstGenerated.qll` file.
|
|
|
|
## Step 6: Add dbscheme upgrade and downgrade scripts
|
|
|
|
If you ended up making changes to the database scheme inw step 5, then you'll need to add an appropriate pair of up- and downgrade scripts to handle any changes between the different versions of the dbscheme.
|
|
|
|
This can be a bit fiddly, but luckily there are [tools](https://github.com/github/codeql/blob/main/misc/scripts/prepare-db-upgrade.sh) that can help set up some of the necessary files for you.
|
|
|
|
See also the [guide](https://github.com/github/codeql/blob/main/docs/prepare-db-upgrade.md) for preparing database upgrades.
|