The canonicalization work on the CFG facade PR removed the thin
`Cfg::CfgForBb` wrapper module ("we have CfgImpl::Cfg already"), so the
variable-capture Flow instantiation must reference the native
`CfgImpl::Cfg` (= BB::Make<Location, BbInput>, which implements
BB::CfgSig) directly. This is the only mechanical change needed when the
dataflow flip is rebased onto the canonicalized CFG facade.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e1faf86b-2dba-40bd-9419-f10184c53d72
The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's
`dominates`/`strictlyDominates` predicates, which are declared
`bindingset[this, that]` + `pragma[inline_late]` and are meant to be used
as bound-pair membership checks. The facade wrappers dropped these
annotations (using plain `pragma[inline]`), so even though the only
callers — the `with` / `async with` taint steps in DataFlowPrivate.qll
and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was
free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full
O(nodes^2) relation over the (larger) shared-CFG node set.
On some projects this dominated analysis time entirely (DCA showed e.g.
ICTU/quality-time and biosimulations regressing ~75-160x). Restoring
`bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns
the predicate back into a bound-pair check and is result-preserving (only
binding annotations change, the predicate body is unchanged).
Reproduced on ICTU/quality-time: full python-security-extended suite went
from stalling >20min on `strictlyDominates` to completing in ~6min; all
ControlFlow and dataflow/coverage library tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll)
and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade
(semmle.python.controlflow.internal.Cfg) and the new SSA adapter
(semmle.python.dataflow.new.internal.SsaImpl), both introduced
additively in the preceding PRs in this stack.
This is the trunk-flip equivalent of the original draft PR #21894 (kept
around as documentation), rebased on top of the four preparatory PRs:
P1: Remove AstNode.getAFlowNode() and rewrite callers (#21919).
P2: Qualify Flow.qll's AST references with Py:: prefix (#21920).
P3: Add new shared-CFG-backed control flow graph (#21921).
P4: Add new shared-SSA-backed SSA adapter (#21923).
The Python dataflow library (semmle/python/dataflow/new/) now imports
the new CFG facade and SSA adapter. All CFG-typed predicates
(ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are
qualified with the Cfg:: prefix; SSA references switch from
EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable.
GuardNode is redesigned to use the new CFG's outcome-node model
(isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock +
flipped indirection. Only BarrierGuard<...> is preserved as public
API.
Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib,
...) are updated to take CFG nodes from the new facade.
A handful of dataflow consistency tweaks for the new CFG:
- Augmented-assignment targets are treated as both load and store.
- 'from X import *' produces uncertain SSA writes for unknown names.
- CFG nodes are canonicalised so dataflow does not see equivalent
pre/post-order pairs as distinct nodes.
Two AST tweaks for the new CFG:
- AstNodeImpl: omit PEP 695 type-parameter names from
FunctionDefExpr / ClassDefExpr children.
- ImportResolution: drop the legacy essa import.
Test churn (~175 files): reblessed library- and query-test .expected
files reflect slightly different CFG granularity, different toString
output, and a handful of true alert deltas in security queries.
Verification: all 367 lib + src + consistency-queries compile clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preparatory refactor for the shared-CFG dataflow migration. Adds the
new Python SSA adapter additively, without changing any production
behaviour.
Library additions:
- semmle.python.dataflow.new.internal.SsaImpl — Python SSA
implementation built on the new (shared) CFG. Mirrors the Java SSA
adapter (java/ql/lib/semmle/code/java/dataflow/internal/SsaImpl.qll):
an InputSig is defined in terms of positional (BasicBlock, int)
variable references, and the shared
codeql.ssa.Ssa::Make<Location, Cfg, Input> module is then
instantiated.
SourceVariable is the AST-level Py::Variable. Variable references
are looked up via the new CFG facade's NameNode.defines/uses/deletes
predicates (added in the preceding PR), which themselves are
one-line bridges to AST-level Name.defines/uses/deletes.
Implicit-entry definitions are inserted for non-local/global/builtin
reads, captured variables, and (when needed) parameters.
Test additions:
- library-tests/dataflow-new-ssa/ — exercises the new SSA over a
representative test corpus and checks expected def/use chains.
- library-tests/dataflow-new-ssa-vs-legacy/ — runs both new SSA and
legacy ESSA over the same corpus and diffs the results, so any
semantic divergence shows up as a test failure.
Production impact:
None. The new SSA adapter has zero callers in lib/ and src/ — the
legacy ESSA SSA (semmle/python/essa/*) remains the default. The
dataflow library is not migrated yet; that lands in a follow-up PR.
Verified by:
- All 367 lib + src + consistency-queries compile clean.
- All 641 ControlFlow + PointsTo + dataflow + essa + consistency
library-tests pass.
- Both new dataflow-new-ssa[/vs-legacy] test packs pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- AST navigation such as `getObject` and `getChild` now only moves through nodes identified via `injects`
- identification such as `isSubscript` only holds for injected nodes
- subclasses such as `AttrNode` are injected nodes (no, say after-nodes)
The new (shared-CFG-based) Python control flow graph in
`semmle.python.controlflow.internal.Cfg` previously did not emit CFG
nodes for parameter type annotations (`def f(x: T): ...`) or for the
return type annotation (`-> T`). The legacy CFG emitted both, and a
small number of framework models rely on this: `LocalSources.qll`'s
`annotatedInstance` walks the parameter annotation expression by way
of its CFG node to track that a parameter receives an instance of the
annotated class.
After the dataflow flip to the new CFG/SSA this regression manifested
as lost flows in any test exercising annotation-based parameter
tracking: FastAPI `Depends()` receivers, Pydantic request bodies,
Starlette `WebSocket`, the call-graph type-annotation test, and so on.
Extend `FunctionDefExpr` to visit each annotation as a child of the
function-def expression, in CPython evaluation order: positional
parameter annotations, `*args` annotation, keyword-only parameter
annotations, `**kwargs` annotation, then the return annotation. (Lambda
expressions have no annotations in Python syntax, so `LambdaExpr` is
unchanged.) PEP 695 type parameters remain out of scope; they belong
to the inner annotation scope, not the enclosing CFG.
Restored test results across `framework/aiohttp`, `framework/fastapi`,
`framework/lxml`, the `CallGraph-type-annotations` test, and
`CWE-022-PathInjection`. Two FastAPI list-comprehension MISSING markers
become positive (`taint_test.py:41,55`). CPython CFG consistency
remains clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The new CFG previously only emitted exception edges for explicit `raise`
and `assert` statements. As a result, code that became reachable only
via the exception path of an arbitrary expression (e.g., the body of an
`except` handler following a try-body whose `call()` could raise) was
classified as dead, breaking analyses like StackTraceExposure,
FileNotAlwaysClosed, ExceptionInfo, UseOfExit, and CatchingBaseException.
This commit adds a `mayThrow` predicate over expressions that are known
sources of implicit exceptions in Python (calls, attribute access,
subscripts, arithmetic/comparison operators, imports, await/yield/yield
from) plus `from m import *` at the statement level, and routes them
through the shared CFG's `beginAbruptCompletion(_, _, ExceptionSuccessor,
always=false)` hook.
The set of exception sources is restricted to nodes that are
syntactically inside a `try`/`with` statement in the same scope.
This mirrors Java's `ControlFlowGraph::mayThrow`, which only emits
exception edges where local handling can observe them — outside such
contexts, the edges add CFG complexity (weakening BarrierGuard
precision and breaking SSA continuity around augmented assignments and
subscript stores) without analysis benefit, since exceptions just
propagate to the function exit anyway.
Net effect on the test suite: ~100 alerts restored across the exception-
related query tests (StackTraceExposure +29, ExceptionInfo +17,
FileNotAlwaysClosed +52, UseOfExit +1, CatchingBaseException restored)
with no precision regressions. Affected `.expected` files and the
regression-guard `dead_under_no_raise.py` are updated accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the public facade on top of the AstNodeImpl adapter from the
previous commit. Re-exposes the same API surface as
semmle/python/Flow.qll (ControlFlowNode, CallNode, BasicBlock,
NameNode, DefinitionNode, CompareNode, ...), backed by the shared
codeql.controlflow.ControlFlowGraph library.
- semmle.python.controlflow.internal.Cfg — public facade.
- ControlFlow/store-load/* — basic store/load coverage via the facade.
The new CFG library is added additively: it has zero callers in lib/
and src/, and the legacy CFG in semmle/python/Flow.qll remains the
default. Dataflow, SSA, and production query migration land in
follow-up PRs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preparatory refactor for the shared-CFG dataflow migration. Adds the
adapter that mediates between the Python AST and the shared
codeql.controlflow.ControlFlowGraph signature, plus the test suites
that validate the new CFG directly against this adapter. The public
facade is added in the following commit.
Library additions:
- semmle.python.controlflow.internal.AstNodeImpl — wraps Python's
Stmt/Expr/Scope/Pattern and adds two synthetic kinds of node
(BlockStmt for body slots, intermediate nodes for multi-operand
boolean expressions) to satisfy the shared CFG signature.
- lib/ide-contextual-queries/printCfg.ql — the IDE "Print CFG" query, retargeted to the new CFG.
- consistency-queries/CfgConsistency.ql — consistency query running
the shared CFG's standard checks against Python.
Test additions (all driven directly off AstNodeImpl):
- ControlFlow/bindings/* — annotation-driven SSA-binding tests
(annassign, compound, comprehension, decorated, except_handler,
imports, match_pattern, parameters, simple, type_params,
walrus_starred, with_stmt, dead_under_no_raise).
- ControlFlow/evaluation-order/NewCfg*.ql — mirrors of the existing
OldCfg evaluation-order self-validation suite, run against the
new CFG via NewCfgImpl.qll.
- Minor extensions to existing test_if.py / test_boolean.py +
cosmetic .expected churn on a handful of OldCfg tests.
No dataflow, SSA, or production query is migrated yet.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- unified/swift: Mark `binding_kind` as a raw `@@` capture in the
property_declaration rule. It is only used to read its source text
(`ctx.ast.source_text`), never as a translated node. With `@` the
auto-translate prefix would route the unnamed `let`/`var` token
through the catch-all `_ @node => {node}` fallback for a no-op
roundtrip; `@@` makes the intent explicit and removes that reliance.
- shared/yeast/tests: Reword a stale comment in test_raw_capture_marker.
The text claimed a "second assertion" exists in this test, but the
explicit-translation check actually lives in the companion
test_raw_capture_marker_explicit_translate.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With `@@name` available, there's no longer a need to use `manual_rule!`.
Every place where it is used, we can instead just mark the relevant raw
captures as such. This results in quite a lot of cleanup! (Also, to me
at least, it makes these rules a lot easier to reason about.)
A first iteration of this approach resulted in a lot of
`.map(Into::into)` being needed, because `SwiftContext` stores `Id`s,
but captures produce `NodeRef`s. To avoid this, I swapped it around so
that the context stores `NodeRef`s. This does require adding `.into()`
in a few places, but it makes the rest of the code a lot more ergonomic.
The `@@name` capture marker in `rule!` queries skips the
auto-translate prefix for that specific capture, letting the body see
the original capture (and thus delay its translation using
`ctx.translate` until it becomes convenient).
Regular `@name` captures continue to be auto-translated as before.
Specifically these are translated _eagerly_, before the main body of the
rewrite rule is run.
I settled on `@@` as the syntax because it did not add new symbols that
the user has to keep track of (it's still a kind of capture), but it's
still visually distinct enough that the user should be able to tell that
there's something special going on. In principle one could accidentally
write one form of capture where the other was intended, but in practice
this would result in code that did not compile (because the types would
not match).
