The new SSA's implicit entry-def predicate previously placed entries in
the variable's defining scope. For closure variables that's the outer
function, so inner functions had no entry def for the captured
variable — reads in the inner scope failed to resolve to any
definition.
Mirrors legacy ESSA's 'NonLocalVariable.getScopeEntryDefinition()':
place an implicit entry def at every reading scope's entry block,
independently of where the variable is *defined*. A closure variable
accessed in two nested functions and the outer one gets three entry
defs (one per reading scope).
Also makes 'ScopeEntryDefinition' extend 'EssaNodeDefinition' (matching
legacy ESSA), with 'getDefiningNode()' returning the scope's entry CFG
node. This requires extending the private 'writeDefNode' helper to
project i=-1 entries to bb.getNode(0).
Updates the new-vs-legacy comparison snapshot: closure-variable reads
('x:32:5'), nested global reads ('GLOBAL:52:1') now resolve. New
'def-only-new' entries appear for unbound names ('sum', 'open',
'compute') — the new SSA uniformly creates scope-entry defs for all
non-local reads, including those that legacy ESSA classifies as
builtin and excludes. This is a more uniform semantic and arguably
cleaner.
Updates the SsaTest 'some_undefined' annotation: previously documented
as a known limitation, now correctly resolves to a scope-entry def.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Phase 0.5 - Adapter API on top of the shared SSA:
Adds the legacy-ESSA-shaped class hierarchy that the dataflow library
consumes, layered on the shared 'Ssa::Make' instantiation:
* EssaDefinition / EssaNodeDefinition: the latter exposes
'getDefiningNode()' (the CFG node at the def's index in its BB)
and 'getVariable()' / 'getScope()'.
* AssignmentDefinition: matches Assign, AnnAssign with value,
AssignExpr and AugAssign target Names. Exposes 'getValue()'
pointing at the RHS' CFG node.
* ParameterDefinition: matches when the defining Name is in
parameter context.
* WithDefinition: matches 'with ... as x:' bindings.
* ScopeEntryDefinition: implicit entry defs at synthetic position
'-1' of the scope's entry basic block (non-local / global /
builtin / captured reads).
* PhiFunction (alias for PhiNode).
* EssaVariable adapter wrapping a 'Ssa::Definition' with 'getAUse()',
'getDefinition()', 'getAnUltimateDefinition()', and 'getName()'.
* AdjacentUses module with 'firstUse' and 'adjacentUseUse' predicates
bridging to 'Ssa::firstUse' / 'Ssa::adjacentUseUse'.
This is the minimum API the new dataflow's internals call into. The
richer legacy ESSA (refinement nodes, attribute refinements, edge
refinements) stays in 'semmle.python.essa.Essa' for legacy code.
Phase 0.6 - Comparison test:
Adds 'dataflow-new-ssa-vs-legacy/CmpTest.ql' that snapshots the
difference between definitions produced by new SSA vs legacy ESSA on
the same Python source. Baseline output records the current
'def-only-old' mismatches, grouped by category:
* function/class/global definitions with no in-scope read (intentional;
SSA is liveness-pruned)
* captured / closure variables (real gap in new SSA - no
closure-capture handling yet)
* module variables __name__ / __package__ / $ (legacy ESSA implicit
bindings)
* exception 'as' bindings (depend on raise modelling)
Zero 'def-only-new' mismatches: the new SSA never produces a spurious
definition compared to legacy ESSA on this corpus.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In the legacy CFG the same Python 'Name' that is the target of an
augmented assignment has two distinct CFG nodes — a load node (context
3) earlier in the basic block and a store node (context 5) later.
'augstore(load, store)' relates the pair via dominance.
The new (shared) CFG canonicalises each AST expression to a single
CFG node, so 'load' and 'store' collapse to one. The dominance-based
'augstore' from the legacy implementation no longer holds (it would
require 'load.strictlyDominates(load)'), so 'isAugLoad' / 'isAugStore'
never fired and 'isStore' missed the AugAssign target entirely.
Redefines 'augstore' as reflexive on the AugAssign target's canonical
CFG node. With this change:
* isAugLoad / isAugStore both fire on the single canonical node.
* isStore fires (via 'or augstore(_, this)') — matching the legacy
classification that an augmented-assignment target is a store.
* isLoad does not fire (excluded by 'not augstore(_, this)').
Adds 'python/ql/test/library-tests/ControlFlow/store-load/' covering
plain load/store/delete, parameters, augmented assignment, tuple
unpacking, attribute and subscript stores. The test asserts the
classification directly on the new-CFG facade.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds 'python/ql/lib/semmle/python/dataflow/new/internal/SsaImpl.qll', a
minimal Python SSA implementation built on the shared SSA library
('codeql.ssa.Ssa::Make<Location, Cfg, Input>'). The structure mirrors
Java's adapter at 'java/ql/lib/semmle/code/java/dataflow/internal/SsaImpl.qll'.
Key design choices:
* 'SourceVariable' wraps 'Py::Variable'. Only variables that are read
or deleted somewhere are tracked - write-only variables don't
benefit from SSA construction.
* Variable references are positional ('BasicBlock', 'int') pairs
looked up via 'Cfg::NameNode.defines'/'.uses'/'.deletes' (which
themselves are one-line bridges to AST-level 'Name.defines' etc.).
* Parameter writes are not synthesised: parameter Name nodes are
already wired into the CFG (per the earlier C#-style parameter
extension in 'AstNodeImpl.qll'), so the regular 'variableWrite'
path handles them at their natural CFG index.
* Non-local / captured / global / builtin variables read in a scope
but not written in it receive a synthetic entry definition at
index '-1' of the scope's entry basic block. This matches Java's
'hasEntryDef'.
* 'del x' is modelled as a certain write at the deletion site.
Includes an inline-expectations test under
'python/ql/test/library-tests/dataflow-new-ssa/' covering:
plain parameter pass-through, simple assignment + read, reassignment
with dead-write pruning, if/else with phi insertion at the join, and
an undefined-name read (currently a known limitation - no SSA flow
without an enclosing definition).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds 'dead_under_no_raise.py' to the bindings test suite, capturing the
three CPython patterns where bindings legitimately have no CFG node
because the surrounding code is unreachable under the 'no expressions
raise' abstraction:
1. Statements after a 'try: return X; except: pass' block.
2. The 'else:' clause of a try whose body always raises.
3. Cache-lookup pattern 'try: return cache[k]; except: pass' followed
by computation and store.
These bindings intentionally carry no 'cfgdefines=' annotations. If
raise modelling is later added to the CFG, the BindingsTest will surface
the new CFG nodes as unexpected results and this file will need to be
revisited.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds CFG coverage for the binding 'Name's introduced by PEP 695
type-parameter syntax on functions, classes, and 'type' aliases:
def func[T](...): ...
class Box[T]: ...
def multi[T: int, *Ts, **P](...): ...
type Alias[T] = ...
For each parametrised AST node, the type-parameter names (and, for
'type' aliases, the alias name itself) are added as children of the
enclosing CFG node so that 'Name.defines(v)' has a corresponding
position. Bounds and defaults are intentionally not wired (they have
no SSA-relevant semantics for our purposes).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds `ImportStmt` and `ImportStarStmt` wrappers in `AstNodeImpl.qll`.
For each `Alias` in an import statement, both the value (module/member
expression) and the bound `asname` Name become children of the CFG node
for the import statement, in evaluation order.
Without this, every `Name` introduced by `import` / `from .. import ..`
lacked a CFG node, even though `Name.defines(v)` returns true for it on
the AST side. This was the highest-volume gap: 20,332 missing import
aliases across CPython.
Removes the corresponding MISSING: annotations from imports.py.
Verified: all 24 ControlFlow/evaluation-order tests still pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements `AstSig::Parameter` and `callableGetParameter(c, i)` in
`AstNodeImpl.qll`, following the C# template
(`csharp/.../ControlFlowGraph.qll:147-156`) rather than Java's
`Parameter() { none() }`.
Each Python parameter (positional, *args, keyword-only, **kwargs) now
becomes a CFG node at a stable position in the enclosing callable's
entry sequence. Defaults still evaluate at function-definition time
via `FunctionDefExpr.getDefault` / `LambdaExpr.getDefault`, so
`Parameter::getDefaultValue()` returns `none()` (the shared CFG
library calls this to model the missing-argument fallback, which
Python does not surface at the CFG level).
The bindings test now exercises parameters (the `py_expr_contexts(_, 4, ...)`
exclusion has been removed). A new `parameters.py` test case covers
positional, defaulted, vararg, kwarg, keyword-only, kitchen-sink,
method (self/cls), lambda, and PEP 570 positional-only parameters.
Several other test files were updated to annotate parameters that the
test had previously hidden (synthetic `.0` comprehension parameter,
method `self`, decorator `f`, etc.).
Verified:
- All 24 ControlFlow/evaluation-order tests still pass.
- CFG consistency query (`python/ql/consistency-queries/CfgConsistency.ql`)
shows zero violations on CPython.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds an `AnnAssignStmt` wrapper in `AstNodeImpl.qll` so that PEP 526
annotated assignments (`x: int = 1`, `x: int`) participate in the
control flow graph. Evaluation order follows CPython: annotation,
optional value, target binding.
Without this, `x: int = 1` had no CFG node for `x` even though
`Name.defines(v)` returns true for it on the AST side. SSA built on
the new CFG would therefore miss every annotated-assignment write.
Removes the corresponding MISSING: annotations from the CFG-binding
gap test:
- annassign.py — all four cases now green.
- match_pattern.py — class-body annotated fields (`x: int`, `y: int`).
- type_params.py — `item: T` inside class.
Verified: all 24 ControlFlow/evaluation-order tests still pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds inline-expectation tests for the new shared CFG implementation in
python/ql/lib/semmle/python/controlflow/internal/AstNodeImpl.qll,
covering every Python binding construct that introduces a variable.
The test files use MISSING: annotations to record bindings whose
defining Name AST node is *not* currently reachable from the new CFG.
These are the 'red' half of red-green commit pairs: subsequent commits
will extend AstNodeImpl to cover each construct and remove the
corresponding MISSING: marker.
Confirmed-broken categories:
- Import aliases (from x import a)
- Annotated assignment (x: int = 1)
- Exception handler (except E as e)
- Match patterns (case x, case [a,b], case ... as v)
- PEP 695 type params (def f[T], class C[T])
Confirmed-working (no MISSING:):
- Compound targets, with-as, comprehensions, decorated def/class,
walrus, starred.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two default predicates to AstSig:
default AstNode getWhileElse(WhileStmt loop) { none() }
default AstNode getForeachElse(ForeachStmt loop) { none() }
When defined, the explicit-step rules for While/Do and Foreach
route the loop's normal-completion exits through the else block
before reaching the after-loop node:
- WhileStmt: after-false condition -> before-else -> after-while
(instead of directly after-while).
- ForeachStmt: after-collection [empty] and the LoopHeader exit
are both routed through before-else -> after-foreach.
Python's Ast module overrides the predicates to return the
synthetic BlockStmt for the orelse slot, replacing the previous
customisations in Input::step. This eliminates parallel direct
successors emitted by the previous Python-side step additions
(verified: multipleSuccessors on a CPython database goes from
1340 to 0).
Java and C# CFG tests are unaffected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Currently we only instantiate them with the old CFG library, but in the
future we'll want to do this with the new library as well.
Co-authored-by: yoff <yoff@github.com>
This one is potentially a bit iffy -- it checks for a very powerful
propetry (that implies many of the other queries), but as the test
results show, it can produce false positives when there is in fact no
problem. We may want to get rid of it entirely, if it becomes too noisy.
This looks for nodes annotated with `t.never` in the test that are
reachable in the CFG. This should not happen (it messes with various
queries, e.g. the "mixed returns" query), but the test shows that in a
few particular cases (involving the `match` statement where all cases
contain `return`s), we _do_ have reachable nodes that shouldn't be.
This one demonstrates a bug in the current CFG. In a dictionary
comprehension `{k: v for k, v in d.items()}`, we evaluate the value
before the key, which is incorrect. (A fix for this bug has been
implemented in a separate PR.)
These use the annotated, self-verifying test files to check various
consistency requirements.
Some of these may be expressing the same thing in different ways, but
it's fairly cheap to keep them around, so I have not attempted to
produce a minimal set of queries for this.
These tests consist of various Python constructions (hopefully a
somewhat comprehensive set) with specific timestamp annotations
scattered throughout. When the tests are run using the Python 3
interpreter, these annotations are checked and compared to the "current
timestamp" to see that they are in agreement. This is what makes the
tests "self-validating".
There are a few different kinds of annotations: the basic `t[4]` style
(meaning this is executed at timestamp 4), the `t.dead[4]` variant
(meaning this _would_ happen at timestamp 4, but it is in a dead
branch), and `t.never` (meaning this is never executed at all).
In addition to this, there is a query, MissingAnnotations, which checks
whether we have applied these annotations maximally. Many expression
nodes are not actually annotatable, so there is a sizeable list of
excluded nodes for that query.
We won't be able to run these tests until Python 3.15 is actually out
(and our CI is using it), so it seemed easiest to just put them in their
own test directory.
Adds `hasOverloadDecorator` as a predicate on functions. It looks for
decorators called `overload` or `something.overload` (usually
`typing.overload` or `t.overload`). These are then filtered out in the
predicates that (approximate) resolving methods according to the MRO.
As the test introduced in the previous commit shows, this removes the
spurious resolutions we had before.
The ones that no longer require points-to no longer import
`LegacyPointsTo`. The ones that do use the specific
`...MetricsWithPointsTo` classes that are applicable.
With `ModuleVariableNode`s now appearing for _all_ global variables (not
just the ones that actually seem to be used), some of the tests changed
a bit. Mostly this was in the form of new flow (because of new nodes
that popped into existence). For some inline expectation tests, I opted
to instead exclude these results, as there was no suitable location to
annotate. For the normal tests, I just accepted the output (after having
vetted it carefully, of course).
