Adds CFG coverage for the binding 'Name's introduced by PEP 695
type-parameter syntax on functions, classes, and 'type' aliases:
def func[T](...): ...
class Box[T]: ...
def multi[T: int, *Ts, **P](...): ...
type Alias[T] = ...
For each parametrised AST node, the type-parameter names (and, for
'type' aliases, the alias name itself) are added as children of the
enclosing CFG node so that 'Name.defines(v)' has a corresponding
position. Bounds and defaults are intentionally not wired (they have
no SSA-relevant semantics for our purposes).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds `ImportStmt` and `ImportStarStmt` wrappers in `AstNodeImpl.qll`.
For each `Alias` in an import statement, both the value (module/member
expression) and the bound `asname` Name become children of the CFG node
for the import statement, in evaluation order.
Without this, every `Name` introduced by `import` / `from .. import ..`
lacked a CFG node, even though `Name.defines(v)` returns true for it on
the AST side. This was the highest-volume gap: 20,332 missing import
aliases across CPython.
Removes the corresponding MISSING: annotations from imports.py.
Verified: all 24 ControlFlow/evaluation-order tests still pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements `AstSig::Parameter` and `callableGetParameter(c, i)` in
`AstNodeImpl.qll`, following the C# template
(`csharp/.../ControlFlowGraph.qll:147-156`) rather than Java's
`Parameter() { none() }`.
Each Python parameter (positional, *args, keyword-only, **kwargs) now
becomes a CFG node at a stable position in the enclosing callable's
entry sequence. Defaults still evaluate at function-definition time
via `FunctionDefExpr.getDefault` / `LambdaExpr.getDefault`, so
`Parameter::getDefaultValue()` returns `none()` (the shared CFG
library calls this to model the missing-argument fallback, which
Python does not surface at the CFG level).
The bindings test now exercises parameters (the `py_expr_contexts(_, 4, ...)`
exclusion has been removed). A new `parameters.py` test case covers
positional, defaulted, vararg, kwarg, keyword-only, kitchen-sink,
method (self/cls), lambda, and PEP 570 positional-only parameters.
Several other test files were updated to annotate parameters that the
test had previously hidden (synthetic `.0` comprehension parameter,
method `self`, decorator `f`, etc.).
Verified:
- All 24 ControlFlow/evaluation-order tests still pass.
- CFG consistency query (`python/ql/consistency-queries/CfgConsistency.ql`)
shows zero violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds an `AnnAssignStmt` wrapper in `AstNodeImpl.qll` so that PEP 526
annotated assignments (`x: int = 1`, `x: int`) participate in the
control flow graph. Evaluation order follows CPython: annotation,
optional value, target binding.
Without this, `x: int = 1` had no CFG node for `x` even though
`Name.defines(v)` returns true for it on the AST side. SSA built on
the new CFG would therefore miss every annotated-assignment write.
Removes the corresponding MISSING: annotations from the CFG-binding
gap test:
- annassign.py — all four cases now green.
- match_pattern.py — class-body annotated fields (`x: int`, `y: int`).
- type_params.py — `item: T` inside class.
Verified: all 24 ControlFlow/evaluation-order tests still pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds inline-expectation tests for the new shared CFG implementation in
python/ql/lib/semmle/python/controlflow/internal/AstNodeImpl.qll,
covering every Python binding construct that introduces a variable.
The test files use MISSING: annotations to record bindings whose
defining Name AST node is *not* currently reachable from the new CFG.
These are the 'red' half of red-green commit pairs: subsequent commits
will extend AstNodeImpl to cover each construct and remove the
corresponding MISSING: marker.
Confirmed-broken categories:
- Import aliases (from x import a)
- Annotated assignment (x: int = 1)
- Exception handler (except E as e)
- Match patterns (case x, case [a,b], case ... as v)
- PEP 695 type params (def f[T], class C[T])
Confirmed-working (no MISSING:):
- Compound targets, with-as, comprehensions, decorated def/class,
walrus, starred.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Style cleanup: avoid naming newtype branch constructors (TPyStmt,
TPyExpr, TBlockStmt, TPattern, TBoolExprPair, TScope) outside the
char-preds that classify their wrappers. Method bodies and helper
predicates now use the as* projections instead:
// Before: result = TBlockStmt(ifStmt.getBody())
// After: result.asStmtList() = ifStmt.getBody()
// Before: result = TPyStmt(matchStmt.getCase(index))
// After: result.asStmt() = matchStmt.getCase(index)
Adds:
- AstNode.asStmtList() - the inverse of TBlockStmt(_).
- BinaryExpr.getIndex() - exposes the synthetic-pair index, used
internally by getRightOperand to find the next pair without
naming TBoolExprPair.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Style cleanup: when a class's characteristic predicate binds via a
'cast' helper like
IfStmt() { ifStmt = this.asStmt() }
prefer naming the newtype branch directly:
IfStmt() { this = TPyStmt(ifStmt) }
This makes the wrapped representation explicit. Apply throughout:
~30 charpreds (every Stmt/Expr leaf wrapper, plus LoopStmt, BreakStmt,
ContinueStmt, BooleanLiteral, UnaryExpr, ArithUnaryExpr, Comprehension).
Method bodies that use asStmt/asExpr to project an underlying
Python AST node (Stmt.toString, BlockStmt.getEnclosingCallable,
UnaryExpr.getOperand, etc.) keep that form - they're projections,
not classifications.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mirror the TStmt refactor for the Expr hierarchy: rename the TExpr
newtype branch to TPyExpr and add
private class TExpr = TPyExpr or TBoolExprPair;
This lets the public Expr class use TExpr directly:
class Expr extends AstNodeImpl, TExpr { ... }
instead of
class Expr extends AstNodeImpl {
Expr() { this instanceof TExpr or this instanceof TBoolExprPair }
...
}
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the 14-disjunct allow-list with a 2-conjunct exclusion list.
Of the 17 Py::StmtList getters in AstGenerated.qll, only Try.getHandlers()
and MatchStmt.getCases() should not be wrapped as BlockStmts (they are
iterated individually by the shared library's Try/Switch logic via
getCatch(int) and getCase(int)). All other StmtLists are imperative
block bodies.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename the TStmt newtype branch to TPyStmt, and add a private union
type alias
private class TStmt = TPyStmt or TBlockStmt;
This lets the public Stmt class use TStmt directly in its extends
clause:
class Stmt extends AstNodeImpl, TStmt { ... }
instead of the previous
class Stmt extends AstNodeImpl {
Stmt() { this instanceof TStmt or this instanceof TBlockStmt }
...
}
The same pattern is used in cpp/.../TInstruction.qll and
rust/.../Synth.qll.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert AstNode from a concrete class with empty default predicates into
a private abstract class plus a final alias, matching the pattern used
in cpp/.../EdgeKind.qll and cpp/.../IRVariable.qll:
abstract private class AstNodeImpl extends TAstNode {
abstract string toString();
abstract Py::Location getLocation();
abstract Callable getEnclosingCallable();
...
}
final class AstNode = AstNodeImpl;
This makes the compiler enforce that every concrete subclass implements
toString/getLocation/getEnclosingCallable, replacing the brittle
'empty default + per-branch override' arrangement. Sister classes
inside the module now extend AstNodeImpl instead of AstNode (which is
final and cannot be extended).
The empty Parameter stub gains explicit none() overrides for the
three abstract members, since QL requires them statically even when
the class has no instances.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explain that the shared library's Assignment / CompoundAssignment
hierarchy extends BinaryExpr, so it cannot host Python's statement-
level assignment forms (Assign, AugAssign), and that Python has no
short-circuiting compound operators (&&=, ||=, ??=) so all
subclasses remain empty.
No behaviour change; doc comments only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the two-key TBlockStmt(Py::AstNode parent, string slot) newtype
branch with the simpler TBlockStmt(Py::StmtList sl). Each Py::StmtList
that represents an imperative block (function/class/module body, if/
while/for branch, try/except/finally body, case body, except/except*
body) becomes one BlockStmt directly. The slot string disappears;
toString just defers to Py::StmtList.toString() ('StmtList').
The newtype branch keeps an explicit characteristic predicate listing
the slots that count as block bodies. This excludes Try.getHandlers(),
which is a Py::StmtList of ExceptStmt items already iterated by the
shared library's Try logic via getCatch(int) - including it would
produce parallel CFG edges (verified: a permissive
TBlockStmt(Py::StmtList sl) version regressed CPython to 1720
multipleSuccessors and 584 deadEnds before this restriction).
Drops the getBodyStmtList helper. Caller sites now use the StmtList
accessor directly: TBlockStmt(ifStmt.getBody()),
TBlockStmt(tryStmt.getFinalbody()), etc.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously a Py::BoolExpr appeared in two newtype branches: as TExpr(be)
(the outermost pair) and TBoolExprPair(be, i) for inner pairs of 3+
operand expressions. This forced BinaryExpr/LogicalAndExpr/LogicalOrExpr
to disjoin two cases, and the synthetic-pair handling spanned multiple
layers.
Restrict TExpr to non-BoolExpr Py::Expr, and extend TBoolExprPair to
cover every operand pair (index 0..n-2). Now every Py::BoolExpr is
represented uniformly as TBoolExprPair(_, 0) for the whole expression
and TBoolExprPair(_, i) for inner pairs.
Extend AstNode.asExpr() to also recover the underlying Py::BoolExpr
from TBoolExprPair(_, 0). This makes asExpr() the inverse of
construction: every 'result = TExpr(e)' turns into 'result.asExpr() = e',
which works uniformly for BoolExprs and non-BoolExprs alike.
Consequences:
- BinaryExpr now extends TBoolExprPair directly with a single uniform
rule for left/right operands.
- LogicalAndExpr/LogicalOrExpr are one-line char preds via
getBoolExpr().
- The private BoolExprPair wrapper class folds into BinaryExpr.
- 60+ leaf wrappers now read 'result.asExpr() = py_expr' instead of
'result = TExpr(py_expr)'.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Five of the six per-newtype-branch wrapper classes had a natural
public class corresponding to that branch:
TStmtAstNode -> Stmt (TStmt subset; BlockStmt overrides for TBlockStmt)
TExprAstNode -> Expr (TExpr subset; BoolExprPair overrides for TBoolExprPair)
TScopeAstNode -> Callable (= TScope exactly)
TPatternAstNode -> Pattern (= TPattern exactly)
TBlockStmtAstNode -> BlockStmt (= TBlockStmt exactly)
Move toString/getLocation/getEnclosingCallable onto these classes and
delete the wrappers.
The sixth wrapper (TBoolExprPair) has no exact public counterpart -
BinaryExpr is broader, including TExpr-branch BoolExprs - so it
remains as a small private class, renamed BoolExprPair.
No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the three big disjunctive predicates on AstNode with empty
defaults plus per-newtype-branch override classes:
AstNode.toString() { none() }
AstNode.getLocation() { none() }
AstNode.getEnclosingCallable() { none() }
Six private subclasses (one per newtype branch — TStmt, TExpr,
TScope, TPattern, TBoolExprPair, TBlockStmt) override these with
the branch-specific implementation. This mirrors the per-class
dispatch already used for getChild.
No behaviour change: all 24 NewCfg evaluation-order tests pass and
all 11 shared-CFG consistency queries still report 0 violations on
CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Main added two new requirements to AstSig:
- A 'Parameter' class with a 'getDefaultValue()' method, plus a
'callableGetParameter(Callable, int)' predicate.
- A 'CallableContext' class in InputSig1, replacing the previous
'CallableBodyPartContext'.
Add stub implementations: 'Parameter' is empty (none()) and
'callableGetParameter' returns nothing, mirroring Java's TODO. Rename
'CallableBodyPartContext = Void' to 'CallableContext = Void' in the
Python Input module.
NewCfg evaluation-order tests still pass at the 22/24 baseline; all
11 shared-CFG consistency queries still report 0 violations on
CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the single ~240-line top-level getChild predicate with one
override per AST class. AstNode declares a default
AstNode getChild(int index) { none() }
and each subclass with children overrides it (41 classes total).
The top-level predicate becomes a one-line dispatch:
AstNode getChild(AstNode n, int index) { result = n.getChild(index) }
No behavioral change: NewCfg evaluation-order tests still pass at the
same 22/24 baseline, and all 11 shared-CFG consistency queries still
report 0 violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The shared CFG library propagates abrupt completions from child to
parent via getChild(parent, _) = child. Python's try.getElse() was
wired into normal step rules but not listed in getChild(TryStmt, ...),
so return/break/continue/raise statements occurring inside a try-else
block had no parent path and ended up as dead-end CFG nodes.
Add the else block at index -2 (alongside finally at -1). This affects
only completion propagation; the normal-flow CFG is unchanged because
TryStmt has explicit step rules.
Verified on a CPython database: all 11 shared-CFG consistency queries
now pass with 0 violations (deadEnd: 244 -> 0).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two default predicates to AstSig:
default AstNode getWhileElse(WhileStmt loop) { none() }
default AstNode getForeachElse(ForeachStmt loop) { none() }
When defined, the explicit-step rules for While/Do and Foreach
route the loop's normal-completion exits through the else block
before reaching the after-loop node:
- WhileStmt: after-false condition -> before-else -> after-while
(instead of directly after-while).
- ForeachStmt: after-collection [empty] and the LoopHeader exit
are both routed through before-else -> after-foreach.
Python's Ast module overrides the predicates to return the
synthetic BlockStmt for the orelse slot, replacing the previous
customisations in Input::step. This eliminates parallel direct
successors emitted by the previous Python-side step additions
(verified: multipleSuccessors on a CPython database goes from
1340 to 0).
Java and C# CFG tests are unaffected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`Args.getDefault(int)` and `Args.getKwDefault(int)` are indexed by
argument position (with gaps for args without defaults), not by
default position. The CFG `getChild` predicate for FunctionDefExpr
and LambdaExpr therefore had gaps at low indices and collisions
where defaults and kwdefaults overlapped, producing parallel
edges before the FunctionExpr.
Use `rank` to compact-renumber `getDefault(n)` and `getKwDefault(n)`
in source order. Verified on a CPython database: removes ~536
`multipleSuccessors` consistency results (1340 -> 804); the rest are
`for/else` and `while/else`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merge the previous `Ast` and `AstSigImpl` modules into a single
`module Ast implements AstSig<Py::Location>`. Classes now use the
signature names (IfStmt, WhileStmt, ForeachStmt, etc.) and signature
predicates (getCondition, getThen, getElse, etc.) directly, with no
intermediate renaming layer.
Drop the TStmtListNode newtype branch entirely. Replace it with a
synthetic TBlockStmt(parent, slot) keyed by a parent AST node and a
slot label string ('body', 'orelse', 'finally'). Py::StmtList no
longer appears in the newtype; the BlockStmt class provides indexed
access to the underlying body items via getStmt(n).
All 22 of 24 evaluation-order tests still pass; the same 2
comprehension-related failures predate this refactor.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Currently we only instantiate them with the old CFG library, but in the
future we'll want to do this with the new library as well.
Co-authored-by: yoff <yoff@github.com>
This one is potentially a bit iffy -- it checks for a very powerful
propetry (that implies many of the other queries), but as the test
results show, it can produce false positives when there is in fact no
problem. We may want to get rid of it entirely, if it becomes too noisy.
This looks for nodes annotated with `t.never` in the test that are
reachable in the CFG. This should not happen (it messes with various
queries, e.g. the "mixed returns" query), but the test shows that in a
few particular cases (involving the `match` statement where all cases
contain `return`s), we _do_ have reachable nodes that shouldn't be.
