The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's
`dominates`/`strictlyDominates` predicates, which are declared
`bindingset[this, that]` + `pragma[inline_late]` and are meant to be used
as bound-pair membership checks. The facade wrappers dropped these
annotations (using plain `pragma[inline]`), so even though the only
callers — the `with` / `async with` taint steps in DataFlowPrivate.qll
and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was
free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full
O(nodes^2) relation over the (larger) shared-CFG node set.
On some projects this dominated analysis time entirely (DCA showed e.g.
ICTU/quality-time and biosimulations regressing ~75-160x). Restoring
`bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns
the predicate back into a bound-pair check and is result-preserving (only
binding annotations change, the predicate body is unchanged).
Reproduced on ICTU/quality-time: full python-security-extended suite went
from stalling >20min on `strictlyDominates` to completing in ~6min; all
ControlFlow and dataflow/coverage library tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll)
and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade
(semmle.python.controlflow.internal.Cfg) and the new SSA adapter
(semmle.python.dataflow.new.internal.SsaImpl), both introduced
additively in the preceding PRs in this stack.
This is the trunk-flip equivalent of the original draft PR #21894 (kept
around as documentation), rebased on top of the four preparatory PRs:
P1: Remove AstNode.getAFlowNode() and rewrite callers (#21919).
P2: Qualify Flow.qll's AST references with Py:: prefix (#21920).
P3: Add new shared-CFG-backed control flow graph (#21921).
P4: Add new shared-SSA-backed SSA adapter (#21923).
The Python dataflow library (semmle/python/dataflow/new/) now imports
the new CFG facade and SSA adapter. All CFG-typed predicates
(ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are
qualified with the Cfg:: prefix; SSA references switch from
EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable.
GuardNode is redesigned to use the new CFG's outcome-node model
(isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock +
flipped indirection. Only BarrierGuard<...> is preserved as public
API.
Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib,
...) are updated to take CFG nodes from the new facade.
A handful of dataflow consistency tweaks for the new CFG:
- Augmented-assignment targets are treated as both load and store.
- 'from X import *' produces uncertain SSA writes for unknown names.
- CFG nodes are canonicalised so dataflow does not see equivalent
pre/post-order pairs as distinct nodes.
Two AST tweaks for the new CFG:
- AstNodeImpl: omit PEP 695 type-parameter names from
FunctionDefExpr / ClassDefExpr children.
- ImportResolution: drop the legacy essa import.
Test churn (~175 files): reblessed library- and query-test .expected
files reflect slightly different CFG granularity, different toString
output, and a handful of true alert deltas in security queries.
Verification: all 367 lib + src + consistency-queries compile clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preparatory refactor for the shared-CFG dataflow migration. Adds the
new Python SSA adapter additively, without changing any production
behaviour.
Library additions:
- semmle.python.dataflow.new.internal.SsaImpl — Python SSA
implementation built on the new (shared) CFG. Mirrors the Java SSA
adapter (java/ql/lib/semmle/code/java/dataflow/internal/SsaImpl.qll):
an InputSig is defined in terms of positional (BasicBlock, int)
variable references, and the shared
codeql.ssa.Ssa::Make<Location, Cfg, Input> module is then
instantiated.
SourceVariable is the AST-level Py::Variable. Variable references
are looked up via the new CFG facade's NameNode.defines/uses/deletes
predicates (added in the preceding PR), which themselves are
one-line bridges to AST-level Name.defines/uses/deletes.
Implicit-entry definitions are inserted for non-local/global/builtin
reads, captured variables, and (when needed) parameters.
Test additions:
- library-tests/dataflow-new-ssa/ — exercises the new SSA over a
representative test corpus and checks expected def/use chains.
- library-tests/dataflow-new-ssa-vs-legacy/ — runs both new SSA and
legacy ESSA over the same corpus and diffs the results, so any
semantic divergence shows up as a test failure.
Production impact:
None. The new SSA adapter has zero callers in lib/ and src/ — the
legacy ESSA SSA (semmle/python/essa/*) remains the default. The
dataflow library is not migrated yet; that lands in a follow-up PR.
Verified by:
- All 367 lib + src + consistency-queries compile clean.
- All 641 ControlFlow + PointsTo + dataflow + essa + consistency
library-tests pass.
- Both new dataflow-new-ssa[/vs-legacy] test packs pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a few tests that validate that higher-order functions are parsed
correctly into the commonAST representation.
Also removes a redundant assignment to `ctx.in_function_type` that
happend after all translations had taken place (and so nothing would
actually read this field).
The extractor shells out to a separate `swift-syntax-parse` binary, but
nothing placed it in the extractor pack, so a shipped Swift extraction
failed at the first spawn. Package it next to the extractor, the same
way `//swift/extractor` ships its Swift-linked binary: a small wrapper
points the dynamic loader at its own directory and execs the real
binary, whose Swift runtime libraries travel alongside it.
- `swift-syntax-parse.sh`: wrapper that sets `LD_LIBRARY_PATH` /
`DYLD_LIBRARY_PATH` to its directory and execs
`swift-syntax-parse.real`
(mirrors `swift/extractor/extractor.sh`).
- `runtime.bzl`: a `swift_runtime_libs` rule that selects just the Linux
Swift runtime shared objects (`usr/lib/swift/linux/*.so`) out of the
full toolchain, so only they — not the whole toolchain — travel with
the binary.
- `swift-syntax-rs/BUILD.bazel`: the `rust_binary` becomes
`swift-syntax-parse.real`
and carries the runtime libraries as runfiles on Linux; a `sh_binary`
(`swift-syntax-parse`) is the wrapper; `codeql_pkg_runfiles` flattens
the three (wrapper, real binary, runtime) into one directory.
- `BUILD.bazel`: ship that group under `tools/{CODEQL_PLATFORM}` next to
the extractor, on the platforms where swift-syntax builds
(Linux/macOS).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`languages::swift::adapter` embeds the swift-syntax node-types schema
with `include_str!("../../../swift_node_types.yml")`, but the file was
never listed in the extractor's Bazel `compile_data`. The `cargo` build
finds it on disk, so this went unnoticed, but the sandboxed Bazel build
cannot see it and fails to compile the extractor. List it alongside
`ast_types.yml`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two fixes prompted by review of the swift-syntax switch-over.
Parser resolution (`parse.rs`): `parse_bin` now resolves the
`swift-syntax-parse` executable in priority order — the
`CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` override, then a copy next
to the extractor executable (as a shipped extractor pack lays it out:
`tools/<platform>/{extractor,swift-syntax-parse}`), then a
bare `PATH` lookup. This lets a packaged extractor find its parser with
no environment setup. (Bundling the binary into the pack, together with
its Swift runtime, is a separate follow-up.)
Corpus test guard (`corpus_tests.rs`): `parser_available` previously
treated *any* parser error as "unavailable" and skipped the entire
corpus suite, so a parser that was present but crashed or emitted
invalid JSON would silently skip the exact regressions the suite exists
to catch. It now uses the new `binary_available`, which reports whether
the *executable* can be launched (false only when it cannot be found,
e.g. no Swift toolchain); a launchable-but-failing parser makes the
suite run and fail.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`swift-syntax-rs` is a workspace member, so its build script runs on a
plain `cargo check`/`fmt`/`clippy` at the repo root. Previously it
panicked when `swift build` could not be run, breaking those Swift-free
workflows for anyone without a Swift toolchain.
Instead, when `swift build` cannot be spawned, emit a `cargo:warning`
and skip the link directives rather than panicking. `cargo
check`/`fmt`/`clippy` don't link, so they keep working; only `cargo
build`/`cargo test` then fail, at link time — which is fair, since those
genuinely need Swift (and CI builds go through Bazel). A Swift toolchain
that is present but whose build fails is still surfaced as a hard error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerate `types/property-with-getter-and-setter`, whose mapped AST now
differs from the tree-sitter output: the backing `private var _v`
retains its `private` modifier (`modifier "var"` + `modifier
"private"`), whereas the tree-sitter path dropped it (its
`visibility_modifier` node carried no text). The swift-syntax
front-end preserves the modifier, so the mapped AST is strictly richer
here.
The raw (second) section is regenerated to the swift-syntax AST like the
other cases; the mapped (third) section gains the retained `private`
modifier.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that the Swift front-end is swift-syntax, regenerate the second (raw
parse tree) section of the corpus `.output` files to hold the
swift-syntax AST the adapter builds, instead of the old tree-sitter
parse tree.
These 98 cases map to a byte-for-byte identical mapped AST (the third
section), so only their raw section changes — the mapping rules were
ported to produce the same output. The one case whose mapped AST differs
(`types/property-with-getter-and-setter`) is handled separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip the runtime Swift front-end from tree-sitter to swift-syntax. The
mapping rules were already ported, so the mapped AST is unchanged; this
makes the switch live.
- `language_spec` now builds a language-free desugarer
(`ConcreteDesugarer::without_language`) and wires the swift-syntax
parser (`swift_parse::parse`) as the front-end, dropping the
tree-sitter language and node types. The desugarer supplies the output
schema, so `node_types` is left empty.
- The `swift_adapter`/`swift_parse` modules are no longer
`allow(dead_code)`: they are now reached from the live extraction
path.
- `corpus_tests` skips (rather than fails) when the external
`swift-syntax-parse` binary is unavailable, since it cannot run
without the Swift-backed parser.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the remaining member declarations to the swift-syntax AST. An
`initializerDecl` becomes a `constructor_declaration` (its body
optional, so a bodyless protocol requirement still maps);
`deinitializerDecl`, `typeAliasDecl`, and `associatedTypeDecl` map to
`destructor_declaration`, `type_alias_declaration`, and
`associated_type_declaration` respectively.
The tree-sitter subscript and preprocessor-diagnostic rules are dropped:
their swift-syntax counterparts (`subscriptDecl`, `ifConfigDecl`) fall
through to the `unsupported_node` fallback, producing the same output.
This also removes the now-unused `type` unwrap rule (swift-syntax has no
such wrapper node) and retires the last `ctx.literal` helper use in
favour of a `tree!` leaf.
PARITY(tree-sitter): the initializer's parameters are still not emitted,
because the tree-sitter path dropped them too. Emitting them is a future
improvement.
This completes the rule migration: every declaration now maps through
the swift-syntax front-end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget enum cases to the swift-syntax AST. An `enumCaseDecl` flattens
its comma-separated `enumCaseElement`s (non-first tagged
`chained_declaration`) and publishes any case modifiers (e.g.
`indirect`) into `ctx`; an element with a payload becomes a nested
`class_like_declaration` + constructor, an element with a raw value
(`case a = 1`) or a plain element a `variable_declaration`;
`enumCaseParameter` becomes a `parameter`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget property accessors to the swift-syntax AST, output unchanged.
An accessor-bearing `variableDecl` publishes the property name/type into
`ctx`; a computed property (`var v: T { get set }`) emits accessors
carrying the type, while a stored property with observers (`var x = e {
didSet {…} }`) emits the backing `variable_declaration` first.
swift-syntax models get/set/willSet/didSet uniformly as `accessorDecl`,
so a single `accessorDecl` rule — with an optional body distinguishing a
computed accessor from a bodyless protocol requirement — replaces the
tree-sitter grammar's separate computed-getter/setter/modify,
willset/didset, and getter-/setter-specifier rules.
The tree-sitter protocol property and function requirement rules
(`protocol_property_declaration`, `protocol_function_declaration`) are
dropped: swift-syntax models those requirements as ordinary
`variableDecl`s (with bodyless accessors) and `functionDecl`s, already
handled by the general rules.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the nominal type declarations to the swift-syntax AST, output
unchanged.
`classDecl`/`structDecl`/`enumDecl`/`protocolDecl`/`extensionDecl`
each become a `class_like_declaration` tagged with a modifier naming the
declaration keyword, with members drawn from the `memberBlock`; each
`memberBlockItem` unwraps to its contained declaration. `superExpr` maps
to `super_expr`; `self` needs no rule (swift-syntax models it as an
ordinary `declReferenceExpr`, already a `name_expr`).
Following the tree-sitter path (PARITY), the inheritance clause is not
emitted as a `base_type` (the tree-sitter rule captured it positionally,
but the grammar nests it under a field, so it never actually matched);
swift-syntax exposes it cleanly, so that is a correctness improvement to
make once tree-sitter is retired. The tree-sitter grammar's standalone
`self`, dead modifier (`visibility_modifier`, etc.), key-path, and
inheritance-specifier rules are dropped; `#selector`/`#keyPath` (now a
`macroExpansionExpr`) stays an `unsupported_node`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget imports to the swift-syntax AST, output unchanged. swift-syntax
represents the dotted path as a list of `importPathComponent`s, folded
into a `name_expr`/`member_access_expr` chain via `member_chain`. A
single rule handles both forms via an optional `importKindSpecifier`: a
scoped import (`import struct Foo.Bar`) has one and binds the last path
component as a `name_pattern`; a plain import (`import Foundation`) has
none and uses a `bulk_importing_pattern`. Leading attributes
(`@_exported`) and access modifiers (`public`) become `modifier`s.
The tree-sitter multi-part `identifier` rule is dropped: swift-syntax
qualified names are already `memberAccessExpr` chains, and import paths
are `importPathComponent`s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
