Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll)
and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade
(semmle.python.controlflow.internal.Cfg) and the new SSA adapter
(semmle.python.dataflow.new.internal.SsaImpl), both introduced
additively in the preceding PRs in this stack.
This is the trunk-flip equivalent of the original draft PR #21894 (kept
around as documentation), rebased on top of the four preparatory PRs:
P1: Remove AstNode.getAFlowNode() and rewrite callers (#21919).
P2: Qualify Flow.qll's AST references with Py:: prefix (#21920).
P3: Add new shared-CFG-backed control flow graph (#21921).
P4: Add new shared-SSA-backed SSA adapter (#21923).
The Python dataflow library (semmle/python/dataflow/new/) now imports
the new CFG facade and SSA adapter. All CFG-typed predicates
(ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are
qualified with the Cfg:: prefix; SSA references switch from
EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable.
GuardNode is redesigned to use the new CFG's outcome-node model
(isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock +
flipped indirection. Only BarrierGuard<...> is preserved as public
API.
Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib,
...) are updated to take CFG nodes from the new facade.
A handful of dataflow consistency tweaks for the new CFG:
- Augmented-assignment targets are treated as both load and store.
- 'from X import *' produces uncertain SSA writes for unknown names.
- CFG nodes are canonicalised so dataflow does not see equivalent
pre/post-order pairs as distinct nodes.
Two AST tweaks for the new CFG:
- AstNodeImpl: omit PEP 695 type-parameter names from
FunctionDefExpr / ClassDefExpr children.
- ImportResolution: drop the legacy essa import.
Test churn (~175 files): reblessed library- and query-test .expected
files reflect slightly different CFG granularity, different toString
output, and a handful of true alert deltas in security queries.
Verification: all 367 lib + src + consistency-queries compile clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preparatory refactor for the shared-CFG dataflow migration. Adds the
new Python CFG library additively, without changing any production
behaviour.
Library additions:
- semmle.python.controlflow.internal.AstNodeImpl — mediates between
the Python AST and the shared codeql.controlflow.ControlFlowGraph
signature. Wraps Python's Stmt/Expr/Scope/Pattern and adds two
synthetic kinds of node (BlockStmt for body slots, intermediate
nodes for multi-operand boolean expressions).
- semmle.python.controlflow.internal.Cfg — public facade
re-exposing the same API surface as semmle/python/Flow.qll
(ControlFlowNode, CallNode, BasicBlock, NameNode, DefinitionNode,
CompareNode, ...), backed by the shared CFG.
- lib/printCfgNew.ql — debug/visualisation query for the new CFG.
- consistency-queries/CfgConsistency.ql — consistency query running
the shared CFG's standard checks against Python.
Shared library:
- shared.controlflow.ControlFlowGraph — adds two defaulted
getWhileElse / getForeachElse predicates to AstSig so Python can
model while-else / for-else (no behavioural change for other
languages).
Test additions:
- ControlFlow/bindings/* — annotation-driven SSA-binding tests for
the new CFG (annassign, compound, comprehension, decorated,
except_handler, imports, match_pattern, parameters, simple,
type_params, walrus_starred, with_stmt, dead_under_no_raise).
- ControlFlow/store-load/* — basic store/load coverage.
- ControlFlow/evaluation-order/NewCfg*.ql — mirrors of the existing
OldCfg evaluation-order self-validation suite, run against the
new CFG via NewCfgImpl.qll.
- Minor extensions to existing test_if.py / test_boolean.py +
cosmetic .expected churn on a handful of OldCfg tests.
No dataflow, SSA, or production query is migrated yet — that lands in
follow-up PRs. The new CFG library has zero callers in lib/ and src/.
Verified by:
- All lib + src + consistency-queries compile clean (367 queries).
- All 56 ControlFlow library-tests pass.
- All 474 dataflow + PointsTo library-tests + consistency tests pass.
- syntax_error/CONSISTENCY/CfgConsistency passes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes the test failures that arose from making `ExtractedArgumentNode`
local.
For the consistency checks, we now explicitly exclude the
`ExtractedArgumentNode`s (now much more plentiful due to the
overapproximation) that don't have a corresponding `getCallArg` tuple.
For various queries/tests using `instanceof ArgumentNode`, we instead us
`isArgumentNode`, which explicitly filters out the ones for which
`isArgumentOf` doesn't hold (which, again, is the case for most of the
nodes in the overapproximation).
When looking things over a bit more, we could actually exclude the steps
that would never be used instead. A much more involved solution, but
more performance oriented and clear in terms of what is supported (at
least until we start supporting type-tracking with more than depth 1
access-path, if that ever happens)
This would have found the problem in
https://github.com/github/codeql/pull/15755.
As highlighted in the comment in the code, it's not a perfect solution
since we don't have an automatic way to ensure we don't introduce a new
PhaseDependentFlow use with a new step relation and forget to add it to
this consistency check... but I think this consistency check still adds
value!
These are the labmda self references. This is similar to
how `BlockParameterArgumentNode` is excluded for Ruby.
It is important that we restrict `call` in this logic.
Otherwise, we get a cartesian product and the consistency
check runs for a very long time...
