From a9817a028127c649a88ad3b30a1b76e77edf8117 Mon Sep 17 00:00:00 2001 From: Taus Date: Thu, 28 Nov 2024 12:32:00 +0000 Subject: [PATCH] Python: Add guide describing how to extend the parser --- python/extractor/extending-the-parser.md | 94 ++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 python/extractor/extending-the-parser.md diff --git a/python/extractor/extending-the-parser.md b/python/extractor/extending-the-parser.md new file mode 100644 index 00000000000..55bdd6c2a0d --- /dev/null +++ b/python/extractor/extending-the-parser.md @@ -0,0 +1,94 @@ +# How to update the Python parser + +## Step 1: Add an extractor test + +Extractor parser tests live in the `tests/parser` directory. There are two different kinds of tests + +- Tests that compare the behavior of the old (Python-based) and new (`tree-sitter`-based) parsers, and verify that they yield the same output on the given source files, and +- Tests that compare the output of the parser (old or new) against a fixed `.expected` file. + +What kind of test is run is determined based on the file name. +If it ends in either `_new.py` or `_old.py`, then the test is run against an `.expected` file. If not, it is used to compare old against new. + +In most cases when adding new features, you'll only be interested in modifying the new parser (the old one is mostly there for legacy reasons). +Thus, you will almost certainly want to create a test that ends in `_new.py`. + +It's a good habit to start by adding the parser test, as this makes it more easy to test when various bits of the parser have been added/modified successfully. + +The rest of this document will only concern itself with the process of extending the _new_ parser. + +To actually _run_ the tests, the easiest way is to use `pytest`. +In the main `extractor` directory (i.e. where this file is located) run + +```sh +pytest tests/test_parser.py +``` + +and wait for the tests to complete. It is normal and expected that the test seemingly freezes on the first run. +This is simply because the `tsg-python` Rust binary is being built in the background. + +Once you have added a new test (or modified an old one) and start making modifications to the parser itself, it quickly becomes tedious to run _all_ the parser tests. +To run just a single test using `pytest`, use the `::` syntax to specify specific tests. +For instance, if you want to just run the tests associated with the file `types_new.py`, you would write + +```sh +pytest tests/test_parser.py::ParserTest::test_types_new +``` + +## Step 2: Extend the `tree-sitter-python` grammar + +The new parser is based on `tree-sitter`, so the first task is to extend the existing `tree-sitter-python` grammar. +This grammar can be found in the `grammar.js` file in the `tsg-python/tsp` subdirectory of the extractor directory. + +Note that whenever changes are made to `grammar.js`, you must regenerate the parser files by running + +```sh +tree-sitter generate +``` + +inside the `tsp` directory. +You'll need to install the `tree-sitter` CLI in order to run this command. +One way to install it is to use `cargo`: + +```sh +cargo install tree-sitter-cli +``` + +(This presupposes you have `cargo` available, but you'll need this anyway when compiling `tsg-python`.) + +Once the parser files have been regenerated, they'll get picked up automatically when `tsg-python` is rebuilt. + +> Pro-tip: When you're done with your parser changes, and go to commit these to a branch, put the autogenerated files in their own commit. +> This makes it easier to review the changes, and if you need to go back and regenerate the files again, it's easy to modify just that commit. + +Once you have extended `grammar.js` and regenerated the parser files, you should be able to check that the grammar changes are sufficient by rerunning the parser test using `pytest`. If it fails while producing an AST that doesn't make sense, then you're probably on the right track. If it fails _without_ producing an AST, then something went wrong with the actual `tree-sitter` parse. To check if this is the case, you can run + +```sh +tree-sitter parse path/to/test.py +``` + +and see what kind of errors are emitted (possibly as `ERROR` or `MISSING` nodes in the AST that is output). + +## Step 3: Extend `python.tsg` + +Once the grammar has been extended, we need to also tell `tsg-python` how to turn the `tree-sitter-python` AST into something that better matches the AST structure that we use in the Python extractor. + +For an introduction to the language of `tree-sitter-graph` (and in particular how we use it in the Python extractor), see the `README.md` file in the `tsg-python` directory. + +## Step 4: Extend the set of known AST nodes + +If you added new node types, or added fields to known node types, then you'll need to update a few files in the Python extractor before it is able to reconstruct the output from `tsg-python`. + +New AST nodes should be added in two places: `master.py` and `semmle/python/ast.py`. The former of these is used to automatically generate the Python dbscheme and `AstGenerated.qll`. The latter is what the parser actually uses as its internal representation of the AST. + +## Step 5: Rebuild the autogenerated AST and database scheme + +If you made changes to `master.py`, you'll need to regenerate a couple of files. This can be done from within the `extractor` directory using the `make dbscheme` and `make ast` commands. Note that for the latter, you need a copy of the CodeQL CLI present, as it is used to autoformat the `AstGenerated.qll` file. + +## Step 6: Add dbscheme upgrade and downgrade scripts + +If you ended up making changes to the database scheme inw step 5, then you'll need to add an appropriate pair of up- and downgrade scripts to handle any changes between the different versions of the dbscheme. + +This can be a bit fiddly, but luckily there are [tools](https://github.com/github/codeql/blob/main/misc/scripts/prepare-db-upgrade.sh) that can help set up some of the necessary files for you. + +See also the [guide](https://github.com/github/codeql/blob/main/docs/prepare-db-upgrade.md) for preparing database upgrades.