Commit Graph

13 Commits

Author SHA1 Message Date
Taus
f75615b913 Merge pull request #17822 from github/tausbn/python-more-parser-fixes
Python: A few more parser fixes
2024-10-30 13:47:10 +01:00
Taus
5db601af3c Python: Allow comments in comprehensions
A somewhat complicated solution that necessitated adding a new custom
function to `tsg-python`. See the comments in `python.tsg` for why this
was necessary.
2024-10-23 14:24:47 +00:00
Taus
4f60494019 Python: Support assignments of the form [x,y,z] = w
Surprisingly, the new parser did not support these constructs (and the
relevant test was missing this case), so on files that required the new
parser we were unable to parse this construct.

To fix it, we add `list_pattern` (not to be confused with
`pattern_list`) as a `tree-sitter-python` node that results in a `List`
node in the AST.
2024-10-22 16:06:35 +00:00
Taus
9c913902c5 Python: Allow except* to be written as except *
Turns out, `except*` is actually not a token on its own according to the
Python grammar. This means it's legal to write `except *foo: ...`, which
we previously would consider a syntax error.

To fix it, we simply break up the `except*` into two separate tokens.
2024-10-22 15:39:29 +00:00
Taus
8053e0ed44 Python: Allow list_splats as type annotations
That is, the `*T` in `def foo(*args : *T): ...`.

This is apparently a piece of syntax we did not support correctly until
now.

In terms of the grammar, we simply add `list_splat` as a possible
alternative for `type` (which could previously only be an `expression`).
We also update `python.tsg` to not specify `expression` those places (as
the relevant stanzas will then not work for `list_splat`s).

This syntax is not supported by the old parser, hence we only add a new
parser test for it.
2024-10-22 15:17:12 +00:00
Taus
fcec8e0256 Python: Fail tests when errors/warnings are logged
This is primarily useful for ensuring that errors where a node does not
have an appropriate context set in `python.tsg` actually have an effect
on the pass/fail status of the parser tests. Previously, these would
just be logged to stdout, but test could still succeed when there were
errors present.

Also fixes one of the logging lines in `tsg_parser.py` to be more
consistent with the others.
2024-10-22 15:11:51 +00:00
Taus
9803bbdc4b Python: Update class parser test 2024-10-21 15:35:48 +00:00
Taus
819b3d77ab Python: Update test expectations
Note that this still includes the somewhat puzzling parsing of
`Spam[**P2]` as an exponentiation with an empty left hand side. When we
fix that bug, we should also update this test to contain actually valid
syntax.
2024-10-15 11:22:33 +00:00
Taus
1ced5b44d7 Python: Add test for type parameter defaults 2024-10-15 11:22:30 +00:00
yoff
0b0e8a4bf5 Update python/extractor/tests/parser/.gitignore
As suggested by @tausbn
2024-10-09 12:22:17 +02:00
Rasmus Lerchedahl Petersen
ad630bc6ff Python: ignore some extractor test output
If you test the extractor locally, you want to ignore these files.
2024-10-09 11:34:58 +02:00
Taus
04c9ed37a7 Python: Fix reference in unit test
The referenced file lives in the internal repo, so this is perhaps a bit
of a hack, but I think it should be fine in the short run.
2024-03-19 17:11:40 +00:00
Taus
6dec323cfc Python: Copy Python extractor to codeql repo 2024-03-07 13:59:16 +00:00