Compare commits

..

81 Commits

Author SHA1 Message Date
yoff
03c1f772f8 Python: revert spurious Py2 hidden-test rebless
The earlier rebless added '.hidden/inner' and 'folder' to the expected
output, but those are namespace-package folders without __init__.py, so
under Py2 mode (respect_init=True) they are correctly *not* extracted
as packages. The previous local rebless picked up Py3 behaviour because
python/ql/test/2/extractor-tests/options doesn't propagate --lang=2
(only library-tests and query-tests do). CI runs it under Py2 and
produces the smaller output.

This brings the expected file back in line with main and CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-29 05:38:44 +00:00
yoff
ef74ec1ff1 Python: fold in evaluation-order review-comment fixes from main
After rebasing onto main, apply the substance of upstream review-comment
commits (1ef557c972, 35faec3db1):

- timer.py: stricter validation (raise TypeError for unknown subscript
  elements), bypass atexit via os._exit on failure.
- test_basic.py: simpler test cases per review (drop unnecessary parens,
  use call form in test_callable_syntax), updated docstring.
- TimerUtils.qll: docstring update reflecting the t[dead(n)] / t[never]
  forms.

The 'dead(2)' annotation in test_boolean.py:27 is kept because our
NewCfgBranchTimestamps check (added on this branch) requires it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:12:48 +00:00
yoff
2e82990dfd Python: rebless CONSISTENCY queries + revert LongPath
- LongPath.expected: revert wrong rebless from 69c27c53c1b. CI generates
  the long-path file during build, so the long-path entry is correct.

- 4 framework/query DataFlowConsistency.expected: pure toString polish
  (ControlFlowNode for X -> X / After X).

- essa/ssa-compute/CONSISTENCY/TypeTrackingConsistency.expected: deleted.
  The 6 prior 'unreachable node in step of kind ...' violations are gone
  under the new SSA; per CI auto-rebless convention the empty file is
  removed.

- extractor-tests/syntax_error/CONSISTENCY/CfgConsistency.expected: new.
  Documents one expected deadEnd on `break` outside any loop in the
  syntax-error test corpus.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
1bcaa56b10 Python: rebless second round after shared-CFG dataflow migration
Second batch of test reblessings, capturing changes in result content
(not just toString labels):

- Framework taint/concept tests (fastapi, sqlalchemy, aiohttp, lxml,
  stdlib, django-orm): mostly gained MISSING-tainted annotations where
  the new dataflow no longer reaches sinks. Some are real taint
  regressions; left as documented failures for follow-up.

- Exception-handler tests (CWE-209-StackTraceExposure, EmptyExcept,
  CatchingBaseException, IncorrectExceptOrder, FileNotAlwaysClosed,
  FindSubclass/Find, Statements/exit/UseOfExit): the no-raise shared CFG
  abstraction does not emit ExceptionSuccessor abrupt-completion edges
  from arbitrary expressions, so except-handler bodies (and their
  exception target Names) are statically dead. Tracked separately under
  cfg-modelling-exceptions.

- Dataflow-path / control-flow node toString polish across the security
  query suite (PathInjection, CodeInjection, UnsafeUnpacking,
  UnsafeUsageOfClientSideEncryptionVersion, RequestWithoutValidation,
  ReflectedXss, CallGraph): simple-leaf nodes now stringify as their
  AST text instead of 'After X'.

- SSA / call-graph improvements (CmpTest, CallGraph/InlineCallGraphTest):
  fewer SSA mismatches between new and old; two previously-MISSING tt=
  annotations resolved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
cf28c32fd8 Python: rebless toString churn from shared-CFG migration
The shared CFG library overrides ControlFlowNode.toString() as 'final'
(shared/controlflow/codeql/controlflow/Cfg.qll:1217), so the legacy
'ControlFlowNode for X' prefix is gone — the new toString returns just
'X' for normal nodes and 'After X' for after-nodes. This produces a
large cosmetic diff in test expected files with no semantic change.

Mass-rebless 78 .expected files whose actual output differs from the
checked-in expected only by this rename. Each file was verified to be
identical after normalising 'ControlFlowNode for ' and 'After ' away
from both sides.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
9696ee922e Python: migrate two more test queries off legacy CFG types
After the shared-CFG migration, DataFlow::Node.asCfgNode() returns
Cfg::ControlFlowNode rather than the legacy Flow::ControlFlowNode, and
funcValue.getACall() / dfCall.getNode() now return different CFG types
(legacy vs new). Update the two remaining test queries that still cast
to legacy NameNode/CallNode types to bridge through Cfg:: types or AST.

* experimental/import-resolution-namespace-relative/test.ql: cast to
  Cfg::NameNode instead of legacy NameNode.

* experimental/library-tests/CallGraph/InlineCallGraphTest.ql: change
  predicate signatures from CallNode to AST Call, and bridge to
  legacy CallNode (points-to) and Cfg::CallNode (type-tracking)
  via getNode() on each side.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
de7744950f Python: migrate remaining tests off getAFlowNode() and fix star-import SSA step
Sweep the last few uses of legacy AstNode.getAFlowNode() in tests over to
explicit ControlFlowNode joins after the shared-CFG migration. importflow.ql
needs the new Cfg::ControlFlowNode/CompareNode types because DataFlow::Node.
asCfgNode() now returns the shared-CFG node.

Also extend ImportResolution::allowedEssaImportStep to walk back through
uncertain-write SSA inputs, so that a later 'from X import *' does not hide
the preceding explicit (re)assignment from module-export resolution. Without
this, a reassigned name that survives a wildcard import was no longer
recognised as the module export. Rebless ModuleExport.expected to drop the
legacy 'ControlFlowNode for' toString prefix and pick up the two correct rows
exposed by the fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
04b130f57f Python: fix library-test compile errors and rebless after CFG migration
Library-test compile fixes after the shared-CFG migration:
- PointsTo/global, PointsTo/local: use `f.getNode() = s.getValue()`
  instead of `s.getValue().getAFlowNode() = f` (the new CFG does not
  surface getAFlowNode on AST nodes).
- PointsTo/new/ImpliesDataflow: bridge new Cfg::ControlFlowNode to the
  legacy ControlFlowNodeWithPointsTo via AST identity.
- frameworks/aiohttp + frameworks/modeling-example: qualify CallNode /
  NameNode / AttrNode casts with Cfg:: now that those names live in
  the new CFG facade.

Rebless 4 expected files for toString-only differences (renamed CFG
positions like 'CFG node for foo' vs 'foo' — no semantic change):
ImpliesDataflow, EnclosingCallable, NaiveModel, ProperModel.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
ba0f24f9f4 Python: canonicalize CFG nodes for dataflow
The shared CFG creates multiple ControlFlowNodes per AST node in
conditional contexts (e.g. afterTrue/afterFalse for boolean conditions,
empty/non-empty for for-loops, matched/unmatched for match cases).
These splits matter for control-flow analysis, but for dataflow — where
we ask 'what is the value of this expression?' — we need exactly one
representative per AST or we double-count calls, arguments, and store
steps.

This adds Cfg::isCanonicalAstNodeRepresentative as a purely structural
pick: for split ASTs it selects the 'positive' outcome variant; for
non-split ASTs it selects the unique variant. The picker is implemented
via genuine-outcome helpers that work around the shared CFG's
cross-kind isAfterValue fallback (ControlFlowGraph.qll:870-892), see
the doc on isGenuineAfterTrue for details.

The TCfgNode-family newtypes in DataFlowPublic, TNormalCall and
TPotentialLibraryCall in DataFlowDispatch, and the SSA-projected
use-use/def-use steps in DataFlowPrivate are all routed through the
canonical filter. DataFlowConsistency and the test UnresolvedCalls
helper qualify their CallNode casts with Cfg:: to keep working.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
b8c5e25c9d Python: migrate src queries to new shared CFG types + reformat
Migrate 27 queries under python/ql/src/ from legacy CFG types
(CallNode/AttrNode/NameNode/etc.) to the shared-CFG-based 'Cfg::'
namespace, matching the dataflow API surface introduced earlier on
this branch. ModificationOfParameterWithDefaultCustomizations.qll
is rewritten on top of BarrierGuard, removing the last legacy ESSA
dependency in that file. UnguardedNextInGenerator.ql still uses
ESSA and bridges to the new CFG via Cfg::CallNode.getNode().

Also reformat 14 library and query files that had drifted from
the formatter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:49 +00:00
yoff
2d997aaaf4 Python: omit PEP 695 type-param names from FunctionDefExpr/ClassDefExpr children
PEP 695 type-param names (e.g. `T` in `def func[T]:` or `class Box[T]:`)
bind in an annotation scope that nests the function/class body, so
their AST scope is the inner function/class — not the enclosing scope
where the FunctionDefExpr/ClassDefExpr CFG node lives. Visiting them
as children created scope-crossing CFG edges (nonLocalStep violations:
96 across CPython).

Drop them from the children list; the legacy CFG omitted them too.
TypeAliasStmt is unaffected (its type-params share scope with the
alias's enclosing scope).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:48 +00:00
yoff
d5f1e092dd Python: migrate remaining query-side files to new Cfg::
Four library/query files still referenced the legacy Flow.qll `ControlFlowNode`
and friends, which no longer match the dataflow library's `Cfg::ControlFlowNode`:

- SubclassFinder.qll: type `value` as `Cfg::ControlFlowNode`.
- ExceptionInfo.qll: replace `EssaNodeDefinition.getDefiningNode()` filter
  with `Cfg::NameNode.defines(_)` (the legacy ESSA class isn't reachable
  through the new dataflow API at the query-pack layer).
- ServerSideRequestForgeryCustomizations.qll: qualify `BinaryExprNode` with
  `Cfg::` and update `stringRestriction` to take `Cfg::ControlFlowNode`.
- TarSlipCustomizations.qll: qualify `CallNode`/`AttrNode`/`NameNode` and
  the `tarFileInfoSanitizer` parameter with `Cfg::`.

The three reblessed `.expected` files are purely cosmetic toString churn
("ControlFlowNode for X" -> "X", "After X"); verified set-equal after
normalising the toString prefixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:48 +00:00
yoff
81b2f340a4 Python: adapt AstNodeImpl to upstream shared-CFG signature changes
- ForStmt.getInit(int)/getUpdate(int) now return AstNode (was Expr)
- Case.getAPattern() renamed to getPattern(int index)

Both are stubs in Python (no C-style for, single match pattern).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:48 +00:00
yoff
6365b72160 Python: drop legacy essa import from ImportResolution
`ImportResolution.qll` was the last new-dataflow file with a direct
`import semmle.python.essa.SsaDefinitions`, used only for the
`SsaSource::init_module_submodule_defn` helper. Inline the 5-line body
as a local private predicate. No functional change — the inlined
predicate is clause-for-clause equivalent (the `f = init.getEntryNode()`
join only constrained `package = init`, since `Scope.getEntryNode()` is
unique per scope; we now express that constraint directly).

All 70 dataflow + ApiGraphs library-tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:48 +00:00
yoff
0a4ddf8685 Python: treat augmented-assignment targets as both load and store
The legacy CFG emitted two ControlFlowNodes for `x[i] += 42` (one load,
one store, with `load.strictlyDominates(store)`). The new CFG collapses
them to a single canonical node, mirroring Java's single-`VarAccess`
model where `isVarRead`/`isVarWrite` are non-disjoint on the same
expression. Reconcile two legacy two-node behaviours with the merged
single-node world:

1. `Cfg::ControlFlowNode.isLoad()` no longer excludes augmented
   targets — both `isLoad` and `isStore` hold on the merged canonical
   node, matching Java. `NameNode.defines` drops the now-redundant
   `not isLoad` guard; `Py::Name.defines` already filters by
   `isDefinition` (Store/Param/AugAssign-target ctx).

2. `LocalFlow::definitionFlowStep` is restricted to NameNode targets,
   matching legacy ESSA's `assignment_definition` which required
   `defn.(NameNode).defines(v)`. Subscript and attribute writes
   (`x[i] = 42`, `obj.attr = 42`) no longer emit a local-flow step
   *into* the LHS expression — that flow is handled by the AttrWrite
   and content-flow machinery. This is essential for keeping augmented
   Subscript/Attribute targets classifiable as `LocalSourceNode` on
   the read side, which the API graph requires for emitting Use edges.

`StoreLoadTest.ql` is updated to filter `isAugLoad` out of the regular
`load` tag, mirroring the pre-existing `not isAugStore` filter on the
`store` tag so augmented-assignment expectations remain
`augload=n augstore=n` (not also `load=n store=n`).

Closes the three remaining ApiGraphs library-test failures
(`getSubscript.ql` semantically, plus cosmetic toString updates in
`ModuleImportWithDots.ql` and `test_crosstalk.ql`).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:48 +00:00
yoff
438092fb47 Python: model from X import * as uncertain SSA writes
Add a 4th disjunct to `SsaImplInput::variableWrite` in the shared-SSA
adapter that mirrors legacy ESSA's `ImportStarRefinement`: every
variable whose scope is the import-star's scope, OR which is used in
the import-star's scope, gets an uncertain write at the `import *`
position.

Uncertain writes do not kill prior definitions; shared SSA's
`SsaUncertainWrite` joins the new value with the immediately-preceding
definition via `uncertainWriteDefinitionInput`. This is the equivalent
of legacy ESSA's two-input refinement.

Cannot depend on `ImportStar` / `ImportResolution` (those modules
import `SsaImpl`), so the predicate uses the structural heuristic on
`Cfg::ImportStarNode` directly.

This closes the two remaining failing dataflow library-tests:

- `import-star/global` — `module_export` chains via `from X import *`
  re-exports now resolve: the importing module has an SSA def of every
  re-exported name, so `lastUseVar` finds the read at the use site.
- `typetracking_imports/highlight_problem` — a direct `from .foo import
  foo` immediately followed by `from .other import *` is now correctly
  marked as dead at the direct import.

Two scope-entry-def noise rows in `highlight_problem.expected` are also
dropped — legacy ESSA needed them as refinement inputs, but shared SSA
handles uncertain writes without an explicit prior def. They were
always tagged `no use to normal exit` (dead).

Dataflow library-tests: 62/64 → 64/64 passing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:48 +00:00
yoff
8f6c24678f Python: update dataflow tests for new CFG + shared SSA
Test-side changes accompanying the dataflow migration:

  * Test queries (.ql) and shared test harness (TestSummaries,
    TestTaintLib) qualify CFG / SSA types with Cfg:: / SsaImpl::,
    bridge via AST (Name, Call, ...) instead of legacy NameNode /
    CallNode, and switch GlobalSsaVariable / EssaVariable usages
    to the new adapter API.

  * .expected files updated for legitimate precision and toString
    changes:
      - phi-node def-use edges newly exposed in def_use_counts.
      - scope-exit synthetic use surfaces one extra implicit use
        in use-use-counts.
      - For [empty]/[non-empty] outcome rows added in
        EnclosingCallable.
      - SsaSourceVariable / Global Variable label cosmetics
        normalised throughout.

  * Inline annotations:
      - typetracking/test.py: removed MISSING:tracked on lines
        93/95 (now found), added SPURIOUS:tracked on line 108
        (decorator over-reach).
      - global-flow/test.py: added SPURIOUS writes=g_mod on line
        20 (correctly reports immediately-overwritten write).
      - tainttracking/customSanitizer/test.py: marked
        try/except: ensure_tainted(s) cases as MISSING: tainted
        (no-raise CFG abstraction does not connect try body to
        except body).
      - coverage/test.py: marked
        SINK(return_from_inner_scope([])) as
        MISSING: flow=... pending closer investigation.

  * regression/{dataflow,custom_dataflow}.expected: accept two
    if/else cond-correlation over-reaches (documented limitation;
    same imprecision applies under legacy semantics by design).

After this change the dataflow library-tests stand at 62 of 64
passing; the two remaining failures are tracked under the
ImportStarRefinement workstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:47 +00:00
yoff
1037f5242e Python: migrate dataflow library to new CFG + shared SSA
Switches the trunk dataflow library and all in-tree consumers
(frameworks, ApiGraphs, Concepts, regexp, security customisations,
test harness) from the legacy Flow.qll/ESSA stack to the new
shared-CFG facade (Cfg.qll) and the ESSA-shaped adapter on the
shared-SSA library (SsaImpl.qll).

Highlights:

  * DataFlowPublic/Private/Dispatch, Attributes, VariableCapture,
    IterableUnpacking, ImportResolution, ImportStar, LocalSources,
    TaintTrackingPrivate, MatchUnpacking, TypeTrackingImpl,
    SsaImpl, Builtins all now qualify CFG/SSA references with
    Cfg:: / SsaImpl:: and stop pulling in semmle.python.essa.*.

  * AstNodeImpl.qll/Cfg.qll: ImportMember exposes its inner
    ImportExpr, DefinitionNode.getValue covers Alias / AnnAssign /
    AugAssign / AssignExpr / For-target / Parameter-default,
    ForNode is treated as an expression node, AnnotatedExitNode is
    canonical, and BoolExprNode.getAnOperand drops the dominance
    constraint that did not hold for short-circuit BBs.

  * SsaImpl.qll: parameters always get a ParameterDefinition (so
    unused parameters still have SSA defs), scope-entry defs for
    module globals require an actual store somewhere, scope-exit
    has a synthetic use so reaching-defs survives to module
    boundary, and the legacy SsaSourceVariable / EssaVariable
    surface (getName, getScope, getAUse, getASourceUse,
    getAnImplicitUse) is reinstated for downstream queries.

  * DataFlowPublic.qll: GuardNode redesigned around the new
    structural outcome nodes (isAfterTrue / isAfterFalse).  The
    legacy ConditionBlock + flipped indirection is gone;
    controlsBlock walks UP through 'not' / '==True' / 'is False'
    etc. via outcomeOfGuard, accumulating polarity cleanly.  Only
    BarrierGuard<...> is preserved as public API.

  * ModuleVariableNode.getAWrite and LocalFlow::definitionFlowStep
    bypass SSA and consult Cfg::NameNode.defines /
    Cfg::DefinitionNode.getValue directly, so that write defs
    pruned by shared SSA (because the variable has no in-scope
    read) still produce dataflow steps.

  * Frameworks + downstream consumers: replace
    EssaVariable.hasDefiningNode, getAReturnValueFlowNode,
    Parameter.getDefault, Scope.getEntryNode / getANormalExit etc.
    with CFG-side bridges through Cfg::ControlFlowNode.

The legacy Flow.qll / Essa.qll stack is untouched and remains
available for queries that import it directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:47 +00:00
yoff
f9398546a4 Python: remove getAFlowNode() — bridge AST→CFG only via CFG-side getNode()
Option 2: eliminates the AST→CFG bridge from the AST layer. Previously
'AstNode.getAFlowNode()' returned a 'ControlFlowNode' from the legacy
'Flow.qll' CFG via 'py_flow_bb_node' — this hardcoded the AST to know
about the legacy CFG, preventing files from cleanly switching to the
new shared CFG.

Removes:
  * 'AstNode.getAFlowNode()' from 'AstExtended.qll'
  * Type-narrowing overrides on 'Attribute' / 'Subscript' / 'Call' /
    'IfExp' / 'Name' / 'NameConstant' / 'ImportMember' (in Exprs.qll
    and Import.qll)

Rewrites ~130 call sites across 'python/ql/lib/' and 'python/ql/src/'
to bridge from the CFG side instead:

  Before:  node = expr.getAFlowNode()
  After:   node.getNode() = expr

  Before:  expr.getAFlowNode().(DefinitionNode).getValue()
  After:   exists(DefinitionNode d | d.getNode() = expr | d.getValue())

  Before:  cn.operands(const.getAFlowNode(), op, x)
  After:   exists(ControlFlowNode c | c.getNode() = const | cn.operands(c, op, x))

This is semantically a no-op — both forms are duals of the same predicate.
Verified by passing all library tests:
  * 64 dataflow tests
  * 28 ControlFlow + dataflow-new-ssa tests
  * 1 essa SSA-compute test
  * 93 tests total in the focused suite

Once committed, files that want to switch from the legacy 'Flow' CFG
to the new 'Cfg' facade only need to change their imports — the
bridge sites are CFG-side and respect whichever ControlFlowNode is in
scope.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:46 +00:00
yoff
d308c88de4 Python: SSA adapter: add MultiAssignmentDefinition, definedBy, useOfDef
Extends the ESSA-shaped adapter on top of the new shared SSA with the
remaining APIs consumed by the dataflow library:

  * MultiAssignmentDefinition: matches the AST pattern 'a, b = ...' where
    the LHS is a Tuple/List and the Name being defined is a sub-element.
    Used by IterableUnpacking.qll to recognise unpacking assignments.

  * EssaNodeDefinition.definedBy(var, defNode): a flatter equivalent of
    'getSourceVariable() = var and getDefiningNode() = defNode', matching
    legacy ESSA's signature. Used by DataFlowPublic.qll's
    ModuleVariableNode to enumerate writes of a global.

  * AdjacentUses::useOfDef(def, use): all reachable uses of a definition
    (firstUse plus transitive use-use adjacency). Used by guards in
    DataFlowPublic.qll.

These complete the API surface enumerated by grep across the dataflow
library. The remaining items (EssaNodeRefinement, EssaImportStep) are
ImportResolution-specific and will need separate treatment, possibly via
a different abstraction since the SSA library does not model heap-state
refinements like 'foo.bar = X'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:46 +00:00
yoff
ccd60a51c3 Python: SSA: handle closure variables via per-scope entry defs
The new SSA's implicit entry-def predicate previously placed entries in
the variable's defining scope. For closure variables that's the outer
function, so inner functions had no entry def for the captured
variable — reads in the inner scope failed to resolve to any
definition.

Mirrors legacy ESSA's 'NonLocalVariable.getScopeEntryDefinition()':
place an implicit entry def at every reading scope's entry block,
independently of where the variable is *defined*. A closure variable
accessed in two nested functions and the outer one gets three entry
defs (one per reading scope).

Also makes 'ScopeEntryDefinition' extend 'EssaNodeDefinition' (matching
legacy ESSA), with 'getDefiningNode()' returning the scope's entry CFG
node. This requires extending the private 'writeDefNode' helper to
project i=-1 entries to bb.getNode(0).

Updates the new-vs-legacy comparison snapshot: closure-variable reads
('x:32:5'), nested global reads ('GLOBAL:52:1') now resolve. New
'def-only-new' entries appear for unbound names ('sum', 'open',
'compute') — the new SSA uniformly creates scope-entry defs for all
non-local reads, including those that legacy ESSA classifies as
builtin and excludes. This is a more uniform semantic and arguably
cleaner.

Updates the SsaTest 'some_undefined' annotation: previously documented
as a known limitation, now correctly resolves to a scope-entry def.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:46 +00:00
yoff
2741492e5d Python: extend new SSA with ESSA-shaped adapter + baseline comparison test
Phase 0.5 - Adapter API on top of the shared SSA:

Adds the legacy-ESSA-shaped class hierarchy that the dataflow library
consumes, layered on the shared 'Ssa::Make' instantiation:

  * EssaDefinition / EssaNodeDefinition: the latter exposes
    'getDefiningNode()' (the CFG node at the def's index in its BB)
    and 'getVariable()' / 'getScope()'.
  * AssignmentDefinition: matches Assign, AnnAssign with value,
    AssignExpr and AugAssign target Names. Exposes 'getValue()'
    pointing at the RHS' CFG node.
  * ParameterDefinition: matches when the defining Name is in
    parameter context.
  * WithDefinition: matches 'with ... as x:' bindings.
  * ScopeEntryDefinition: implicit entry defs at synthetic position
    '-1' of the scope's entry basic block (non-local / global /
    builtin / captured reads).
  * PhiFunction (alias for PhiNode).
  * EssaVariable adapter wrapping a 'Ssa::Definition' with 'getAUse()',
    'getDefinition()', 'getAnUltimateDefinition()', and 'getName()'.
  * AdjacentUses module with 'firstUse' and 'adjacentUseUse' predicates
    bridging to 'Ssa::firstUse' / 'Ssa::adjacentUseUse'.

This is the minimum API the new dataflow's internals call into. The
richer legacy ESSA (refinement nodes, attribute refinements, edge
refinements) stays in 'semmle.python.essa.Essa' for legacy code.

Phase 0.6 - Comparison test:

Adds 'dataflow-new-ssa-vs-legacy/CmpTest.ql' that snapshots the
difference between definitions produced by new SSA vs legacy ESSA on
the same Python source. Baseline output records the current
'def-only-old' mismatches, grouped by category:

  * function/class/global definitions with no in-scope read (intentional;
    SSA is liveness-pruned)
  * captured / closure variables (real gap in new SSA - no
    closure-capture handling yet)
  * module variables __name__ / __package__ / $ (legacy ESSA implicit
    bindings)
  * exception 'as' bindings (depend on raise modelling)

Zero 'def-only-new' mismatches: the new SSA never produces a spurious
definition compared to legacy ESSA on this corpus.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:46 +00:00
yoff
47532ec741 Python: qualify Flow.qll's AST references with Py:: prefix
Prepares Flow.qll for co-existence with the new CFG facade by switching
'import python' to 'import python as Py' and qualifying every AST-class
reference inside Flow.qll's body. Flow.qll's own CFG types
(ControlFlowNode, BasicBlock, CallNode, NameNode, etc.) keep their
unqualified names.

This change is a no-op semantically:
  * all 24 evaluation-order tests still pass,
  * the bindings + store-load + new-CFG-SSA library tests still pass,
  * compilation produces zero errors.

The change enables a follow-up commit to swap python.qll's
'import semmle.python.Flow' for 'import semmle.python.controlflow.internal.Cfg'
without triggering name-clash errors inside Flow.qll itself. Legacy
modules that still want the legacy CFG (essa/, GuardedControlFlow,
LegacyPointsTo, objects/, pointsto/, types/, dataflow/old/) will need a
similar treatment in subsequent commits.

The qualification was applied mechanically via a script that prefixed
every reference to a known AST class. The list includes the standard
AST node types from semmle.python.{Files, Variables, Stmts, Exprs,
Class, Function, Patterns, Comprehensions} plus 'Location' / 'File' /
'Folder' / 'Container' / 'ConditionBlock' / 'Delete' / 'Load'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:46 +00:00
yoff
e74f8c9a87 Python: bring Cfg.qll's facade to API parity with Flow.qll
Adds the methods and type-narrowing overrides needed for Cfg.qll to be
a drop-in replacement for Flow.qll's CFG API surface:

  * 'override getNode()' type narrowing on all AST-shape subclasses
    (CallNode -> Py::Call, AttrNode -> Py::Attribute, ImportExprNode
    -> Py::ImportExpr, etc.). This lets callers chain methods like
    'iexpr.getNode().isRelative()' that previously failed because
    'getNode()' returned the generic AstNode.

  * 'ControlFlowNode.isBranch()' -- true and/or false successor exists.
  * 'ControlFlowNode.getAChild()' -- CFG-level child traversal via the
    AST's getAChildNode, with dominance constraint.
  * 'ControlFlowNode.strictlyReaches(other)' -- node-level reachability.
  * 'NameNode.isSelf()' -- AST-level approximation: uses the 'Variable'
    that is the first parameter of an enclosing method.
  * 'BinaryExprNode.operands(left, op, right)' + 'getAnOperand()'.
  * 'BoolExprNode.getAnOperand()'.
  * 'ForNode.getSequence()' (alias for 'getIter') and
    'ForNode.iterates(target, sequence)'.
  * 'ForNode' / 'RaiseStmtNode' type-narrowing overrides.
  * 'ExceptFlowNode.getName()' / 'ExceptGroupFlowNode.getName()'
    -- the bound 'as'-name CFG node.
  * 'DictNode.getAKey()' (only 'getAValue' was present).

These additions are independent of the dataflow-migration approach
(option 4 vs option 5). They close the API-parity gap identified
during the Option-5 investigation; with them in place, hundreds of
type-resolution errors that previously appeared when swapping Cfg for
Flow at the python.qll level go away.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:45 +00:00
yoff
2257d1eb35 Python: fix augstore for the new CFG and add store/load test
In the legacy CFG the same Python 'Name' that is the target of an
augmented assignment has two distinct CFG nodes — a load node (context
3) earlier in the basic block and a store node (context 5) later.
'augstore(load, store)' relates the pair via dominance.

The new (shared) CFG canonicalises each AST expression to a single
CFG node, so 'load' and 'store' collapse to one. The dominance-based
'augstore' from the legacy implementation no longer holds (it would
require 'load.strictlyDominates(load)'), so 'isAugLoad' / 'isAugStore'
never fired and 'isStore' missed the AugAssign target entirely.

Redefines 'augstore' as reflexive on the AugAssign target's canonical
CFG node. With this change:

  * isAugLoad / isAugStore both fire on the single canonical node.
  * isStore fires (via 'or augstore(_, this)') — matching the legacy
    classification that an augmented-assignment target is a store.
  * isLoad does not fire (excluded by 'not augstore(_, this)').

Adds 'python/ql/test/library-tests/ControlFlow/store-load/' covering
plain load/store/delete, parameters, augmented assignment, tuple
unpacking, attribute and subscript stores. The test asserts the
classification directly on the new-CFG facade.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:45 +00:00
yoff
780995cc67 Python: introduce shared-SSA adapter on the new CFG
Adds 'python/ql/lib/semmle/python/dataflow/new/internal/SsaImpl.qll', a
minimal Python SSA implementation built on the shared SSA library
('codeql.ssa.Ssa::Make<Location, Cfg, Input>'). The structure mirrors
Java's adapter at 'java/ql/lib/semmle/code/java/dataflow/internal/SsaImpl.qll'.

Key design choices:

  * 'SourceVariable' wraps 'Py::Variable'. Only variables that are read
    or deleted somewhere are tracked - write-only variables don't
    benefit from SSA construction.

  * Variable references are positional ('BasicBlock', 'int') pairs
    looked up via 'Cfg::NameNode.defines'/'.uses'/'.deletes' (which
    themselves are one-line bridges to AST-level 'Name.defines' etc.).

  * Parameter writes are not synthesised: parameter Name nodes are
    already wired into the CFG (per the earlier C#-style parameter
    extension in 'AstNodeImpl.qll'), so the regular 'variableWrite'
    path handles them at their natural CFG index.

  * Non-local / captured / global / builtin variables read in a scope
    but not written in it receive a synthetic entry definition at
    index '-1' of the scope's entry basic block. This matches Java's
    'hasEntryDef'.

  * 'del x' is modelled as a certain write at the deletion site.

Includes an inline-expectations test under
'python/ql/test/library-tests/dataflow-new-ssa/' covering:
plain parameter pass-through, simple assignment + read, reassignment
with dead-write pruning, if/else with phi insertion at the join, and
an undefined-name read (currently a known limitation - no SSA flow
without an enclosing definition).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:45 +00:00
yoff
8f0e3c8d0a Python: introduce new-CFG facade
Adds 'Cfg.qll' alongside 'AstNodeImpl.qll' in the controlflow internal
package. The facade re-exposes the same API surface as the legacy
'semmle/python/Flow.qll' (ControlFlowNode, BasicBlock, NameNode, CallNode,
AttrNode, ImportExprNode, ImportMemberNode, ImportStarNode, SubscriptNode,
CompareNode, IfExprNode, AssignmentExprNode, BinaryExprNode, BoolExprNode,
UnaryExprNode, DefinitionNode, DeletionNode, ForNode, RaiseStmtNode,
StarredNode, ExceptFlowNode, ExceptGroupFlowNode, TupleNode, ListNode,
SetNode, DictNode, IterableNode, NameConstantNode), but is implemented
on top of the new shared CFG via 'AstNodeImpl.qll'.

The variable-identity predicates ('NameNode.defines', '.uses',
'.deletes', '.isLocal', '.isNonLocal', ...) are one-line bridges to the
underlying AST predicates ('Name.defines', '.uses', '.deletes'),
mirroring the Java pattern.

Re-exports 'EntryBasicBlock' and 'dominatingEdge/2' from the shared
'BB::CfgSig' produced by 'AstNodeImpl.qll', so downstream consumers
(e.g. the SSA adapter) can wire the new CFG into other shared modules
that expect a 'CfgSig' implementation.

This facade is not yet consumed by the dataflow library — that is the
next phase.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:45 +00:00
yoff
a249d580d4 Python: test dead bindings under no-raise CFG abstraction
Adds 'dead_under_no_raise.py' to the bindings test suite, capturing the
three CPython patterns where bindings legitimately have no CFG node
because the surrounding code is unreachable under the 'no expressions
raise' abstraction:

  1. Statements after a 'try: return X; except: pass' block.
  2. The 'else:' clause of a try whose body always raises.
  3. Cache-lookup pattern 'try: return cache[k]; except: pass' followed
     by computation and store.

These bindings intentionally carry no 'cfgdefines=' annotations. If
raise modelling is later added to the CFG, the BindingsTest will surface
the new CFG nodes as unexpected results and this file will need to be
revisited.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:44 +00:00
yoff
3cf342060e Python: wire PEP 695 type parameters into the shared CFG (green)
Adds CFG coverage for the binding 'Name's introduced by PEP 695
type-parameter syntax on functions, classes, and 'type' aliases:

  def func[T](...): ...
  class Box[T]: ...
  def multi[T: int, *Ts, **P](...): ...
  type Alias[T] = ...

For each parametrised AST node, the type-parameter names (and, for
'type' aliases, the alias name itself) are added as children of the
enclosing CFG node so that 'Name.defines(v)' has a corresponding
position. Bounds and defaults are intentionally not wired (they have
no SSA-relevant semantics for our purposes).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:44 +00:00
Copilot
f17a6625f1 Python: wire match-pattern bindings into the shared CFG (green)
Adds concrete `Pattern` subclasses in `AstNodeImpl.qll` for every
`MatchPattern` AST kind, with `getChild` overrides that expose
sub-patterns and bound Names. Specifically:

- MatchCapturePattern (`case x:`) -> getVariable()
- MatchAsPattern (`case … as v:`) -> getPattern(), getAlias()
- MatchStarPattern (`case [*rest]:`) -> getTarget()
- MatchSequencePattern (`case [a, b]:`) -> getPattern(i)
- MatchClassPattern (`case Cls(p, q, k=v)`) -> getClass(), positional, keyword
- MatchMappingPattern (`case {k: v}:`) -> getMapping(i)
- MatchKeyValuePattern, MatchKeywordPattern, MatchDoubleStarPattern
- MatchOrPattern, MatchLiteralPattern, MatchValuePattern

Without these, every Name bound by a match pattern lacked a CFG node.
Removes the corresponding MISSING: annotations from match_pattern.py
(all 11 cases).

Verified: all 24 ControlFlow/evaluation-order tests still pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:44 +00:00
Copilot
89180671c8 Python: wire import-statement bindings into the shared CFG (green)
Adds `ImportStmt` and `ImportStarStmt` wrappers in `AstNodeImpl.qll`.
For each `Alias` in an import statement, both the value (module/member
expression) and the bound `asname` Name become children of the CFG node
for the import statement, in evaluation order.

Without this, every `Name` introduced by `import` / `from .. import ..`
lacked a CFG node, even though `Name.defines(v)` returns true for it on
the AST side. This was the highest-volume gap: 20,332 missing import
aliases across CPython.

Removes the corresponding MISSING: annotations from imports.py.

Verified: all 24 ControlFlow/evaluation-order tests still pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:43 +00:00
Copilot
17c6d10c66 Python: wire parameters into the shared CFG (C# pattern)
Implements `AstSig::Parameter` and `callableGetParameter(c, i)` in
`AstNodeImpl.qll`, following the C# template
(`csharp/.../ControlFlowGraph.qll:147-156`) rather than Java's
`Parameter() { none() }`.

Each Python parameter (positional, *args, keyword-only, **kwargs) now
becomes a CFG node at a stable position in the enclosing callable's
entry sequence. Defaults still evaluate at function-definition time
via `FunctionDefExpr.getDefault` / `LambdaExpr.getDefault`, so
`Parameter::getDefaultValue()` returns `none()` (the shared CFG
library calls this to model the missing-argument fallback, which
Python does not surface at the CFG level).

The bindings test now exercises parameters (the `py_expr_contexts(_, 4, ...)`
exclusion has been removed). A new `parameters.py` test case covers
positional, defaulted, vararg, kwarg, keyword-only, kitchen-sink,
method (self/cls), lambda, and PEP 570 positional-only parameters.
Several other test files were updated to annotate parameters that the
test had previously hidden (synthetic `.0` comprehension parameter,
method `self`, decorator `f`, etc.).

Verified:
- All 24 ControlFlow/evaluation-order tests still pass.
- CFG consistency query (`python/ql/consistency-queries/CfgConsistency.ql`)
  shows zero violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:43 +00:00
Copilot
bd20042636 Python: wire AnnAssign into the shared CFG (green)
Adds an `AnnAssignStmt` wrapper in `AstNodeImpl.qll` so that PEP 526
annotated assignments (`x: int = 1`, `x: int`) participate in the
control flow graph. Evaluation order follows CPython: annotation,
optional value, target binding.

Without this, `x: int = 1` had no CFG node for `x` even though
`Name.defines(v)` returns true for it on the AST side. SSA built on
the new CFG would therefore miss every annotated-assignment write.

Removes the corresponding MISSING: annotations from the CFG-binding
gap test:
- annassign.py — all four cases now green.
- match_pattern.py — class-body annotated fields (`x: int`, `y: int`).
- type_params.py — `item: T` inside class.

Verified: all 24 ControlFlow/evaluation-order tests still pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:43 +00:00
Copilot
93bd4e3b85 Python: add CFG-binding gap tests (red)
Adds inline-expectation tests for the new shared CFG implementation in
python/ql/lib/semmle/python/controlflow/internal/AstNodeImpl.qll,
covering every Python binding construct that introduces a variable.

The test files use MISSING: annotations to record bindings whose
defining Name AST node is *not* currently reachable from the new CFG.
These are the 'red' half of red-green commit pairs: subsequent commits
will extend AstNodeImpl to cover each construct and remove the
corresponding MISSING: marker.

Confirmed-broken categories:
- Import aliases (from x import a)
- Annotated assignment (x: int = 1)
- Exception handler (except E as e)
- Match patterns (case x, case [a,b], case ... as v)
- PEP 695 type params (def f[T], class C[T])

Confirmed-working (no MISSING:):
- Compound targets, with-as, comprehensions, decorated def/class,
  walrus, starred.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:42 +00:00
Copilot
9bdae5af65 Python: project via as* helpers outside characteristic predicates
Style cleanup: avoid naming newtype branch constructors (TPyStmt,
TPyExpr, TBlockStmt, TPattern, TBoolExprPair, TScope) outside the
char-preds that classify their wrappers. Method bodies and helper
predicates now use the as* projections instead:

  // Before: result = TBlockStmt(ifStmt.getBody())
  // After:  result.asStmtList() = ifStmt.getBody()

  // Before: result = TPyStmt(matchStmt.getCase(index))
  // After:  result.asStmt() = matchStmt.getCase(index)

Adds:

- AstNode.asStmtList() - the inverse of TBlockStmt(_).
- BinaryExpr.getIndex() - exposes the synthetic-pair index, used
  internally by getRightOperand to find the next pair without
  naming TBoolExprPair.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:42 +00:00
Copilot
2c43ca999b Python: use newtype-branch constructors in characteristic predicates
Style cleanup: when a class's characteristic predicate binds via a
'cast' helper like

  IfStmt() { ifStmt = this.asStmt() }

prefer naming the newtype branch directly:

  IfStmt() { this = TPyStmt(ifStmt) }

This makes the wrapped representation explicit. Apply throughout:
~30 charpreds (every Stmt/Expr leaf wrapper, plus LoopStmt, BreakStmt,
ContinueStmt, BooleanLiteral, UnaryExpr, ArithUnaryExpr, Comprehension).

Method bodies that use asStmt/asExpr to project an underlying
Python AST node (Stmt.toString, BlockStmt.getEnclosingCallable,
UnaryExpr.getOperand, etc.) keep that form - they're projections,
not classifications.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:42 +00:00
Copilot
e9f14fc044 Python: introduce TExpr union via newtype-branch alias
Mirror the TStmt refactor for the Expr hierarchy: rename the TExpr
newtype branch to TPyExpr and add

  private class TExpr = TPyExpr or TBoolExprPair;

This lets the public Expr class use TExpr directly:

  class Expr extends AstNodeImpl, TExpr { ... }

instead of

  class Expr extends AstNodeImpl {
    Expr() { this instanceof TExpr or this instanceof TBoolExprPair }
    ...
  }

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:42 +00:00
Copilot
8d4fd936f8 Python: simplify TBlockStmt char pred via exclusion list
Replace the 14-disjunct allow-list with a 2-conjunct exclusion list.
Of the 17 Py::StmtList getters in AstGenerated.qll, only Try.getHandlers()
and MatchStmt.getCases() should not be wrapped as BlockStmts (they are
iterated individually by the shared library's Try/Switch logic via
getCatch(int) and getCase(int)). All other StmtLists are imperative
block bodies.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:41 +00:00
Copilot
5e86f5b589 Python: introduce TStmt union via newtype-branch alias
Rename the TStmt newtype branch to TPyStmt, and add a private union
type alias

  private class TStmt = TPyStmt or TBlockStmt;

This lets the public Stmt class use TStmt directly in its extends
clause:

  class Stmt extends AstNodeImpl, TStmt { ... }

instead of the previous

  class Stmt extends AstNodeImpl {
    Stmt() { this instanceof TStmt or this instanceof TBlockStmt }
    ...
  }

The same pattern is used in cpp/.../TInstruction.qll and
rust/.../Synth.qll.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:41 +00:00
Copilot
a66463da2d Python: use private-abstract + final-alias pattern for AstNode
Convert AstNode from a concrete class with empty default predicates into
a private abstract class plus a final alias, matching the pattern used
in cpp/.../EdgeKind.qll and cpp/.../IRVariable.qll:

  abstract private class AstNodeImpl extends TAstNode {
    abstract string toString();
    abstract Py::Location getLocation();
    abstract Callable getEnclosingCallable();
    ...
  }

  final class AstNode = AstNodeImpl;

This makes the compiler enforce that every concrete subclass implements
toString/getLocation/getEnclosingCallable, replacing the brittle
'empty default + per-branch override' arrangement. Sister classes
inside the module now extend AstNodeImpl instead of AstNode (which is
final and cannot be extended).

The empty Parameter stub gains explicit none() overrides for the
three abstract members, since QL requires them statically even when
the class has no instances.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:41 +00:00
Copilot
e66f53d30a Python: document why Assignment subclasses are empty
Explain that the shared library's Assignment / CompoundAssignment
hierarchy extends BinaryExpr, so it cannot host Python's statement-
level assignment forms (Assign, AugAssign), and that Python has no
short-circuiting compound operators (&&=, ||=, ??=) so all
subclasses remain empty.

No behaviour change; doc comments only.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:41 +00:00
Copilot
b7fa0808e9 Python: index TBlockStmt by Py::StmtList instead of (parent, slot)
Replace the two-key TBlockStmt(Py::AstNode parent, string slot) newtype
branch with the simpler TBlockStmt(Py::StmtList sl). Each Py::StmtList
that represents an imperative block (function/class/module body, if/
while/for branch, try/except/finally body, case body, except/except*
body) becomes one BlockStmt directly. The slot string disappears;
toString just defers to Py::StmtList.toString() ('StmtList').

The newtype branch keeps an explicit characteristic predicate listing
the slots that count as block bodies. This excludes Try.getHandlers(),
which is a Py::StmtList of ExceptStmt items already iterated by the
shared library's Try logic via getCatch(int) - including it would
produce parallel CFG edges (verified: a permissive
TBlockStmt(Py::StmtList sl) version regressed CPython to 1720
multipleSuccessors and 584 deadEnds before this restriction).

Drops the getBodyStmtList helper. Caller sites now use the StmtList
accessor directly: TBlockStmt(ifStmt.getBody()),
TBlockStmt(tryStmt.getFinalbody()), etc.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:41 +00:00
Copilot
23e278e1e6 Python: unify Py::BoolExpr handling via TBoolExprPair
Previously a Py::BoolExpr appeared in two newtype branches: as TExpr(be)
(the outermost pair) and TBoolExprPair(be, i) for inner pairs of 3+
operand expressions. This forced BinaryExpr/LogicalAndExpr/LogicalOrExpr
to disjoin two cases, and the synthetic-pair handling spanned multiple
layers.

Restrict TExpr to non-BoolExpr Py::Expr, and extend TBoolExprPair to
cover every operand pair (index 0..n-2). Now every Py::BoolExpr is
represented uniformly as TBoolExprPair(_, 0) for the whole expression
and TBoolExprPair(_, i) for inner pairs.

Extend AstNode.asExpr() to also recover the underlying Py::BoolExpr
from TBoolExprPair(_, 0). This makes asExpr() the inverse of
construction: every 'result = TExpr(e)' turns into 'result.asExpr() = e',
which works uniformly for BoolExprs and non-BoolExprs alike.

Consequences:

- BinaryExpr now extends TBoolExprPair directly with a single uniform
  rule for left/right operands.
- LogicalAndExpr/LogicalOrExpr are one-line char preds via
  getBoolExpr().
- The private BoolExprPair wrapper class folds into BinaryExpr.
- 60+ leaf wrappers now read 'result.asExpr() = py_expr' instead of
  'result = TExpr(py_expr)'.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:41 +00:00
Copilot
b3f87bedda Python: merge T*AstNode wrappers into matching public classes
Five of the six per-newtype-branch wrapper classes had a natural
public class corresponding to that branch:

  TStmtAstNode      -> Stmt        (TStmt subset; BlockStmt overrides for TBlockStmt)
  TExprAstNode      -> Expr        (TExpr subset; BoolExprPair overrides for TBoolExprPair)
  TScopeAstNode     -> Callable    (= TScope exactly)
  TPatternAstNode   -> Pattern     (= TPattern exactly)
  TBlockStmtAstNode -> BlockStmt   (= TBlockStmt exactly)

Move toString/getLocation/getEnclosingCallable onto these classes and
delete the wrappers.

The sixth wrapper (TBoolExprPair) has no exact public counterpart -
BinaryExpr is broader, including TExpr-branch BoolExprs - so it
remains as a small private class, renamed BoolExprPair.

No behaviour change: all 24 NewCfg evaluation-order tests pass; all
11 shared-CFG consistency queries report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
a2d6d82f3b Python: dispatch toString/getLocation/getEnclosingCallable per branch
Replace the three big disjunctive predicates on AstNode with empty
defaults plus per-newtype-branch override classes:

  AstNode.toString()              { none() }
  AstNode.getLocation()           { none() }
  AstNode.getEnclosingCallable()  { none() }

Six private subclasses (one per newtype branch — TStmt, TExpr,
TScope, TPattern, TBoolExprPair, TBlockStmt) override these with
the branch-specific implementation. This mirrors the per-class
dispatch already used for getChild.

No behaviour change: all 24 NewCfg evaluation-order tests pass and
all 11 shared-CFG consistency queries still report 0 violations on
CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
3f6d099548 Python: adapt to new shared CFG signature
Main added two new requirements to AstSig:
- A 'Parameter' class with a 'getDefaultValue()' method, plus a
  'callableGetParameter(Callable, int)' predicate.
- A 'CallableContext' class in InputSig1, replacing the previous
  'CallableBodyPartContext'.

Add stub implementations: 'Parameter' is empty (none()) and
'callableGetParameter' returns nothing, mirroring Java's TODO. Rename
'CallableBodyPartContext = Void' to 'CallableContext = Void' in the
Python Input module.

NewCfg evaluation-order tests still pass at the 22/24 baseline; all
11 shared-CFG consistency queries still report 0 violations on
CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
414ebb9069 Python: refactor getChild into per-class OO dispatch
Replace the single ~240-line top-level getChild predicate with one
override per AST class. AstNode declares a default

  AstNode getChild(int index) { none() }

and each subclass with children overrides it (41 classes total).
The top-level predicate becomes a one-line dispatch:

  AstNode getChild(AstNode n, int index) { result = n.getChild(index) }

No behavioral change: NewCfg evaluation-order tests still pass at the
same 22/24 baseline, and all 11 shared-CFG consistency queries still
report 0 violations on CPython.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
c398f92f5d Python: include try-else in getChild for completion propagation
The shared CFG library propagates abrupt completions from child to
parent via getChild(parent, _) = child. Python's try.getElse() was
wired into normal step rules but not listed in getChild(TryStmt, ...),
so return/break/continue/raise statements occurring inside a try-else
block had no parent path and ended up as dead-end CFG nodes.

Add the else block at index -2 (alongside finally at -1). This affects
only completion propagation; the normal-flow CFG is unchanged because
TryStmt has explicit step rules.

Verified on a CPython database: all 11 shared-CFG consistency queries
now pass with 0 violations (deadEnd: 244 -> 0).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
c24e476879 Shared CFG: support for-else and while-else loops
Add two default predicates to AstSig:

  default AstNode getWhileElse(WhileStmt loop) { none() }
  default AstNode getForeachElse(ForeachStmt loop) { none() }

When defined, the explicit-step rules for While/Do and Foreach
route the loop's normal-completion exits through the else block
before reaching the after-loop node:

  - WhileStmt: after-false condition -> before-else -> after-while
    (instead of directly after-while).
  - ForeachStmt: after-collection [empty] and the LoopHeader exit
    are both routed through before-else -> after-foreach.

Python's Ast module overrides the predicates to return the
synthetic BlockStmt for the orelse slot, replacing the previous
customisations in Input::step. This eliminates parallel direct
successors emitted by the previous Python-side step additions
(verified: multipleSuccessors on a CPython database goes from
1340 to 0).

Java and C# CFG tests are unaffected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
2a04316d69 Python: compact-renumber FunctionExpr/Lambda defaults
`Args.getDefault(int)` and `Args.getKwDefault(int)` are indexed by
argument position (with gaps for args without defaults), not by
default position. The CFG `getChild` predicate for FunctionDefExpr
and LambdaExpr therefore had gaps at low indices and collisions
where defaults and kwdefaults overlapped, producing parallel
edges before the FunctionExpr.

Use `rank` to compact-renumber `getDefault(n)` and `getKwDefault(n)`
in source order. Verified on a CPython database: removes ~536
`multipleSuccessors` consistency results (1340 -> 804); the rest are
`for/else` and `while/else`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:40 +00:00
Copilot
7912e1b257 Python: collapse two-layer AstNodeImpl into a single Ast module
Merge the previous `Ast` and `AstSigImpl` modules into a single
`module Ast implements AstSig<Py::Location>`. Classes now use the
signature names (IfStmt, WhileStmt, ForeachStmt, etc.) and signature
predicates (getCondition, getThen, getElse, etc.) directly, with no
intermediate renaming layer.

Drop the TStmtListNode newtype branch entirely. Replace it with a
synthetic TBlockStmt(parent, slot) keyed by a parent AST node and a
slot label string ('body', 'orelse', 'finally'). Py::StmtList no
longer appears in the newtype; the BlockStmt class provides indexed
access to the underlying body items via getStmt(n).

All 22 of 24 evaluation-order tests still pass; the same 2
comprehension-related failures predate this refactor.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-28 21:09:39 +00:00
yoff
5746ed713f python: add consistency checks
Co-authored-by: aschackmull <aschackmull@github.com>
2026-05-28 21:09:39 +00:00
yoff
768bdb5937 Python: add pattern nodes
Co-authored-by: Copilot <copilot@github.com>
2026-05-28 21:09:39 +00:00
Taus
41b5589460 Cleanup, printCFG
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:39 +00:00
Taus
498aece892 WIP2 2026-05-28 21:09:39 +00:00
Taus
8d814e1fbf WIP 2026-05-28 21:09:39 +00:00
Taus
655f84ed0d Python: Handle dict unpacking in calls
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:39 +00:00
Taus
aaf9cc52d4 Python: Fix exception issue
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:39 +00:00
Taus
cc77f0bcfa Python: Fix match
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:39 +00:00
Taus
146a3a929d Python: Support match
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
d83d943f68 Python: More nodes
Not entirely sure about the `else:` blocks.

Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
5b1de9eacd Python: Comprehensions
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
9f93d6c902 Python: Add with
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
cc09df27ba Python: More simple statements
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
da408d7c75 Python: assignments
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
999b8f23cb Python: Attributes
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
4336b07d48 Python: Function calls
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
b8bc230a38 Python: Assert statements
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
4e3a633f14 Python: Support various literals
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
f89a773b80 Python: Ignore synthetic CFG nodes
We can only annotate the ones that correspond directly to AST nodes
anyway.

Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:38 +00:00
Taus
4583244ec6 Python: More AstNodeImpl improvements
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:37 +00:00
Taus
e66bf87f22 Python: Instantiate CFG tests with new CFG library
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:37 +00:00
Taus
6b3a790015 Python: Instantiate CFG module fully
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:37 +00:00
Taus
b5df1886ea Python: Use fields everywhere in new AST classes
Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:37 +00:00
Taus
30f28bab8d Python: First stab at shared control-flow 2026-05-28 21:09:37 +00:00
Taus
019e6f233f Python: Make CFG tests parameterised
Currently we only instantiate them with the old CFG library, but in the
future we'll want to do this with the new library as well.

Co-authored-by: yoff <yoff@github.com>
2026-05-28 21:09:37 +00:00
Taus
df6d0cad5e Python: Add ConsecutiveTimestamps test
This one is potentially a bit iffy -- it checks for a very powerful
propetry (that implies many of the other queries), but as the test
results show, it can produce false positives when there is in fact no
problem. We may want to get rid of it entirely, if it becomes too noisy.
2026-05-28 21:09:37 +00:00
Taus
6d829d6cc8 Python: Add NeverReachable test
This looks for nodes annotated with `t.never` in the test that are
reachable in the CFG. This should not happen (it messes with various
queries, e.g. the "mixed returns" query), but the test shows that in a
few particular cases (involving the `match` statement where all cases
contain `return`s), we _do_ have reachable nodes that shouldn't be.
2026-05-28 21:09:37 +00:00
Taus
661fd3156f Python: Add BasicBlockOrdering test
This one demonstrates a bug in the current CFG. In a dictionary
comprehension `{k: v for k, v in d.items()}`, we evaluate the value
before the key, which is incorrect. (A fix for this bug has been
implemented in a separate PR.)
2026-05-28 21:09:37 +00:00
Taus
cc471fd672 Python: Add some CFG-validation queries
These use the annotated, self-verifying test files to check various
consistency requirements.

Some of these may be expressing the same thing in different ways, but
it's fairly cheap to keep them around, so I have not attempted to
produce a minimal set of queries for this.
2026-05-28 21:09:37 +00:00
Taus
9a4fb5c971 Python: Add self-validating CFG tests
These tests consist of various Python constructions (hopefully a
somewhat comprehensive set) with specific timestamp annotations
scattered throughout. When the tests are run using the Python 3
interpreter, these annotations are checked and compared to the "current
timestamp" to see that they are in agreement. This is what makes the
tests "self-validating".

There are a few different kinds of annotations: the basic `t[4]` style
(meaning this is executed at timestamp 4), the `t.dead[4]` variant
(meaning this _would_ happen at timestamp 4, but it is in a dead
branch), and `t.never` (meaning this is never executed at all).

In addition to this, there is a query, MissingAnnotations, which checks
whether we have applied these annotations maximally. Many expression
nodes are not actually annotatable, so there is a sizeable list of
excluded nodes for that query.
2026-05-28 21:09:36 +00:00
2385 changed files with 29413 additions and 94985 deletions

View File

@@ -1,208 +0,0 @@
name: Update Go version
on:
workflow_dispatch:
schedule:
- cron: "0 3 * * 1" # Run weekly on Mondays at 3 AM UTC (1 = Monday)
permissions:
contents: write
pull-requests: write
jobs:
update-go-version:
name: Check and update Go version
if: github.repository == 'github/codeql'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Set up Git
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
- name: Fetch latest Go version
id: fetch-version
run: |
LATEST_GO_VERSION=$(curl -s https://go.dev/dl/?mode=json | jq -r '.[0].version')
if [ -z "$LATEST_GO_VERSION" ] || [ "$LATEST_GO_VERSION" = "null" ]; then
echo "Error: Failed to fetch latest Go version from go.dev"
exit 1
fi
echo "Latest Go version from go.dev: $LATEST_GO_VERSION"
echo "version=$LATEST_GO_VERSION" >> $GITHUB_OUTPUT
# Extract version numbers (e.g., go1.26.0 -> 1.26.0)
LATEST_VERSION_NUM=$(echo $LATEST_GO_VERSION | sed 's/^go//')
echo "version_num=$LATEST_VERSION_NUM" >> $GITHUB_OUTPUT
# Extract major.minor version (e.g., 1.26.0 -> 1.26)
LATEST_MAJOR_MINOR=$(echo $LATEST_VERSION_NUM | sed -E 's/^([0-9]+\.[0-9]+).*/\1/')
echo "major_minor=$LATEST_MAJOR_MINOR" >> $GITHUB_OUTPUT
- name: Check current Go version
id: current-version
run: |
CURRENT_VERSION=$(sed -n 's/.*go_sdk\.download(version = \"\([^\"]*\)\".*/\1/p' MODULE.bazel)
if [ -z "$CURRENT_VERSION" ]; then
echo "Error: Could not extract Go version from MODULE.bazel"
exit 1
fi
echo "Current Go version in MODULE.bazel: $CURRENT_VERSION"
echo "version=$CURRENT_VERSION" >> $GITHUB_OUTPUT
# Extract major.minor version
CURRENT_MAJOR_MINOR=$(echo $CURRENT_VERSION | sed -E 's/^([0-9]+\.[0-9]+).*/\1/')
echo "major_minor=$CURRENT_MAJOR_MINOR" >> $GITHUB_OUTPUT
- name: Compare versions
id: compare
run: |
LATEST="${{ steps.fetch-version.outputs.version_num }}"
CURRENT="${{ steps.current-version.outputs.version }}"
echo "Latest: $LATEST"
echo "Current: $CURRENT"
if [ "$LATEST" = "$CURRENT" ]; then
echo "Go version is up to date"
echo "needs_update=false" >> $GITHUB_OUTPUT
else
echo "Go version needs update from $CURRENT to $LATEST"
echo "needs_update=true" >> $GITHUB_OUTPUT
fi
- name: Update Go version in files
if: steps.compare.outputs.needs_update == 'true'
run: |
LATEST_VERSION_NUM="${{ steps.fetch-version.outputs.version_num }}"
LATEST_MAJOR_MINOR="${{ steps.fetch-version.outputs.major_minor }}"
CURRENT_VERSION="${{ steps.current-version.outputs.version }}"
CURRENT_MAJOR_MINOR="${{ steps.current-version.outputs.major_minor }}"
echo "Updating from $CURRENT_VERSION to $LATEST_VERSION_NUM"
# Escape dots in current version strings for use in sed patterns
CURRENT_VERSION_ESCAPED=$(echo "$CURRENT_VERSION" | sed 's/\./\\./g')
CURRENT_MAJOR_MINOR_ESCAPED=$(echo "$CURRENT_MAJOR_MINOR" | sed 's/\./\\./g')
# Update MODULE.bazel
sed -i "s/go_sdk\.download(version = \"$CURRENT_VERSION_ESCAPED\")/go_sdk.download(version = \"$LATEST_VERSION_NUM\")/" MODULE.bazel
if ! grep -q "go_sdk.download(version = \"$LATEST_VERSION_NUM\")" MODULE.bazel; then
echo "Error: Failed to update MODULE.bazel"
exit 1
fi
# Update go/extractor/go.mod
if ! sed -i "s/^go $CURRENT_MAJOR_MINOR_ESCAPED\$/go $LATEST_MAJOR_MINOR/" go/extractor/go.mod; then
echo "Warning: Failed to update go directive in go.mod"
fi
if ! sed -i "s/^toolchain go$CURRENT_VERSION_ESCAPED\$/toolchain go$LATEST_VERSION_NUM/" go/extractor/go.mod; then
echo "Warning: Failed to update toolchain in go.mod"
fi
# Update go/extractor/autobuilder/build-environment.go
if ! sed -i "s/var maxGoVersion = util\.NewSemVer(\"$CURRENT_MAJOR_MINOR_ESCAPED\")/var maxGoVersion = util.NewSemVer(\"$LATEST_MAJOR_MINOR\")/" go/extractor/autobuilder/build-environment.go; then
echo "Warning: Failed to update build-environment.go"
fi
# Update go/actions/test/action.yml
if ! sed -i "s/default: \"~$CURRENT_VERSION_ESCAPED\"/default: \"~$LATEST_VERSION_NUM\"/" go/actions/test/action.yml; then
echo "Warning: Failed to update action.yml"
fi
# Show what changed
git diff
- name: Check for changes
id: check-changes
if: steps.compare.outputs.needs_update == 'true'
run: |
if git diff --quiet; then
echo "No changes detected"
echo "has_changes=false" >> $GITHUB_OUTPUT
else
echo "Changes detected"
echo "has_changes=true" >> $GITHUB_OUTPUT
fi
- name: Check for existing PR
if: steps.check-changes.outputs.has_changes == 'true'
id: check-pr
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
BRANCH_NAME="workflow/go-version-update"
PR_NUMBER=$(gh pr list --head "$BRANCH_NAME" --state open --json number --jq '.[0].number')
if [ -n "$PR_NUMBER" ]; then
echo "Existing PR found: #$PR_NUMBER"
echo "pr_exists=true" >> $GITHUB_OUTPUT
echo "pr_number=$PR_NUMBER" >> $GITHUB_OUTPUT
else
echo "No existing PR found"
echo "pr_exists=false" >> $GITHUB_OUTPUT
fi
- name: Commit and push changes
if: steps.check-changes.outputs.has_changes == 'true'
run: |
BRANCH_NAME="workflow/go-version-update"
LATEST_VERSION_NUM="${{ steps.fetch-version.outputs.version_num }}"
LATEST_MAJOR_MINOR="${{ steps.fetch-version.outputs.major_minor }}"
# Create or switch to branch
git checkout -B "$BRANCH_NAME"
# Stage and commit changes
git add MODULE.bazel go/extractor/go.mod go/extractor/autobuilder/build-environment.go go/actions/test/action.yml
git commit -m "Go: Update to $LATEST_VERSION_NUM"
# Push changes
git push --force-with-lease origin "$BRANCH_NAME"
- name: Create or update PR
if: steps.check-changes.outputs.has_changes == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
BRANCH_NAME="workflow/go-version-update"
LATEST_VERSION_NUM="${{ steps.fetch-version.outputs.version_num }}"
CURRENT_VERSION="${{ steps.current-version.outputs.version }}"
PR_TITLE="Go: Update to $LATEST_VERSION_NUM"
PR_BODY=$(cat <<EOF
This PR updates Go from $CURRENT_VERSION to $LATEST_VERSION_NUM.
Updated files:
- \`MODULE.bazel\` - go_sdk.download version
- \`go/extractor/go.mod\` - go directive and toolchain
- \`go/extractor/autobuilder/build-environment.go\` - maxGoVersion (only if MAJOR.MINOR changes)
- \`go/actions/test/action.yml\` - default go-test-version
This PR was automatically created by the [Go version update workflow](https://github.com/${{ github.repository }}/blob/main/.github/workflows/go-version-update.yml).
EOF
)
if [ "${{ steps.check-pr.outputs.pr_exists }}" = "true" ]; then
echo "Updating existing PR #${{ steps.check-pr.outputs.pr_number }}"
gh pr edit "${{ steps.check-pr.outputs.pr_number }}" --title "$PR_TITLE" --body "$PR_BODY"
else
echo "Creating new PR"
gh pr create \
--title "$PR_TITLE" \
--body "$PR_BODY" \
--base main \
--head "$BRANCH_NAME" \
--label "Go"
fi

View File

@@ -2,7 +2,7 @@
* @github/code-scanning-alert-coverage * @github/code-scanning-alert-coverage
# CodeQL language libraries # CodeQL language libraries
/actions/ @github/code-scanning-alert-coverage /actions/ @github/codeql-dynamic
/cpp/ @github/codeql-c-analysis /cpp/ @github/codeql-c-analysis
/csharp/ @github/codeql-csharp /csharp/ @github/codeql-csharp
/csharp/autobuilder/Semmle.Autobuild.Cpp @github/codeql-c-extractor @github/code-scanning-language-coverage /csharp/autobuilder/Semmle.Autobuild.Cpp @github/codeql-c-extractor @github/code-scanning-language-coverage
@@ -59,5 +59,9 @@ MODULE.bazel @github/codeql-ci-reviewers
/.github/workflows/rust.yml @github/codeql-rust /.github/workflows/rust.yml @github/codeql-rust
/.github/workflows/swift.yml @github/codeql-swift /.github/workflows/swift.yml @github/codeql-swift
# Misc
/misc/scripts/accept-expected-changes-from-ci.py @RasmusWL
/misc/scripts/generate-code-scanning-query-list.py @RasmusWL
# .devcontainer # .devcontainer
/.devcontainer/ @github/codeql-ci-reviewers /.devcontainer/ @github/codeql-ci-reviewers

View File

@@ -273,7 +273,7 @@ use_repo(
) )
go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk") go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = "1.26.4") go_sdk.download(version = "1.26.0")
go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//go/extractor:go.mod") go_deps.from_file(go_mod = "//go/extractor:go.mod")

View File

@@ -1,9 +1,3 @@
## 0.4.37
### Minor Analysis Improvements
* The GitHub Actions analysis now recognizes more Bash regex checks that restrict a value to alphanumeric characters, including regexes like `^[0-9a-zA-Z]{40}([0-9a-zA-Z]{24})?$` which check for a SHA-1 or SHA-256 hash. This may reduce false positive results where command output is validated with grouped or optional alphanumeric patterns before being used.
## 0.4.36 ## 0.4.36
### Minor Analysis Improvements ### Minor Analysis Improvements

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* The GitHub Actions analysis now recognizes more Bash regex checks that restrict a value to alphanumeric characters, include regexes like `^[0-9a-zA-Z]{40}([0-9a-zA-Z]{24})?$` which check for a sha1 or sha256 hash. This may reduce false positive results where command output is validated with grouped or optional alphanumeric patterns before being used.

View File

@@ -1,4 +0,0 @@
---
category: fix
---
* The query `actions/pr-on-self-hosted-runner` was updated to the latest standard runner labels reducing false positive results.

View File

@@ -1,5 +0,0 @@
## 0.4.37
### Minor Analysis Improvements
* The GitHub Actions analysis now recognizes more Bash regex checks that restrict a value to alphanumeric characters, including regexes like `^[0-9a-zA-Z]{40}([0-9a-zA-Z]{24})?$` which check for a SHA-1 or SHA-256 hash. This may reduce false positive results where command output is validated with grouped or optional alphanumeric patterns before being used.

View File

@@ -1,2 +1,2 @@
--- ---
lastReleaseVersion: 0.4.37 lastReleaseVersion: 0.4.36

View File

@@ -1920,5 +1920,3 @@ private YamlMappingLikeNode resolveMatrixAccessPath(
else result = resolveMatrixAccessPath(newRoot, rest) else result = resolveMatrixAccessPath(newRoot, rest)
) )
} }
class Comment = YamlComment;

View File

@@ -52,12 +52,6 @@ private module YamlSig implements LibYaml::InputSig {
class ParseErrorBase extends LocatableBase, @yaml_error { class ParseErrorBase extends LocatableBase, @yaml_error {
string getMessage() { yaml_errors(this, result) } string getMessage() { yaml_errors(this, result) }
} }
class CommentBase extends LocatableBase, @yaml_comment {
string getText() { yaml_comments(this, result, _) }
override string toString() { yaml_comments(this, _, result) }
}
} }
import LibYaml::Make<YamlSig> import LibYaml::Make<YamlSig>

View File

@@ -2,12 +2,10 @@ import actions
bindingset[runner] bindingset[runner]
predicate isGithubHostedRunner(string runner) { predicate isGithubHostedRunner(string runner) {
// The list of github hosted repos: // list of github hosted repos: https://github.com/actions/runner-images/blob/main/README.md#available-images
// https://github.com/actions/runner-images/blob/main/README.md#available-images runner
// https://docs.github.com/en/enterprise-cloud@latest/actions/how-tos/write-workflows/choose-where-workflows-run/choose-the-runner-for-a-job#standard-github-hosted-runners-for-public-repositories .toLowerCase()
runner.toLowerCase().regexpMatch("^ubuntu-([0-9.]+|latest|slim)(-arm)?$") or .regexpMatch("^(ubuntu-([0-9.]+|latest)|macos-([0-9]+|latest)(-x?large)?|windows-([0-9.]+|latest))$")
runner.toLowerCase().regexpMatch("^macos-([0-9]+|latest)(-x?large|-intel)?$") or
runner.toLowerCase().regexpMatch("^windows-([0-9.]+|latest)(-vs[0-9.]+)?(-arm)?$")
} }
bindingset[runner] bindingset[runner]

View File

@@ -1,5 +1,5 @@
name: codeql/actions-all name: codeql/actions-all
version: 0.4.38-dev version: 0.4.37-dev
library: true library: true
warnOnImplicitThis: true warnOnImplicitThis: true
dependencies: dependencies:

View File

@@ -1,22 +1,3 @@
## 0.6.29
### Query Metadata Changes
* Reversed adjustment of the name of `actions/untrusted-checkout/high`, but kept the portion of the previous change for the word "trusted" to "privileged". Added a missing "a" to phrasing in `actions/untrusted-checkout/high` and `actions/untrusted-checkout/medium`.
### Major Analysis Improvements
* Adjusted `actions/untrusted-checkout/critical` to align more with other untrusted resource queries, where the alert location is the location where the artifact is obtained from (the checkout point). This aligns with the other 2 related queries. This will cause the same alerts to re-open for closed alerts of this query.
### Minor Analysis Improvements
* Altered the alert message for clarity for queries: `actions/untrusted-checkout/critical`, `actions/untrusted-checkout/high`.
* The `actions/unpinned-tag` query now recognizes 64-character SHA-256 commit hashes as properly pinned references, in addition to 40-character SHA-1 hashes.
### Bug Fixes
* Adjusted (minor) help file descriptions for queries: `actions/untrusted-checkout/critical`, `actions/untrusted-checkout/high`, `actions/untrusted-checkout/medium`. Clarified wording on a minor point, added one more listed resource and added one more recommendation for things to check.
## 0.6.28 ## 0.6.28
### Query Metadata Changes ### Query Metadata Changes

View File

@@ -1,8 +1,8 @@
/** /**
* @name Checkout of untrusted code in a non-privileged context * @name Checkout of untrusted code in a trusted context
* @description Checking out and running the build script from a fork executes untrusted code. Even in a * @description Privileged workflows have read/write access to the base repository and access to secrets.
* non-privileged workflow, this can be abused, for example to compromise self-hosted runners * By explicitly checking out and running the build script from a fork the untrusted code is running in an environment
* or to poison caches and artifacts that are later consumed by privileged workflows. * that is able to push to the base repository and to access secrets.
* @kind problem * @kind problem
* @problem.severity warning * @problem.severity warning
* @precision medium * @precision medium
@@ -20,4 +20,4 @@ from PRHeadCheckoutStep checkout
where where
// the checkout occurs in a non-privileged context // the checkout occurs in a non-privileged context
inNonPrivilegedContext(checkout) inNonPrivilegedContext(checkout)
select checkout, "Potential unsafe checkout of untrusted pull request on non-privileged workflow." select checkout, "Potential unsafe checkout of untrusted pull request on privileged workflow."

View File

@@ -0,0 +1,4 @@
---
category: majorAnalysis
---
* Adjusted `actions/untrusted-checkout/critical` to align more with other untrusted resource queries, where the alert location is the location where the artifact is obtained from (the checkout point). This aligns with the other 2 related queries. This will cause the same alerts to re-open for closed alerts of this query.

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* The `actions/unpinned-tag` query now recognizes 64-character SHA-256 commit hashes as properly pinned references, in addition to 40-character SHA-1 hashes.

View File

@@ -0,0 +1,4 @@
---
category: minorAnalysis
---
* Altered the alert message for clarity for queries: `actions/untrusted-checkout/critical`, `actions/untrusted-checkout/high`.

View File

@@ -0,0 +1,4 @@
---
category: fix
---
* Adjusted (minor) help file descriptions for queries: `actions/untrusted-checkout/critical`, `actions/untrusted-checkout/high`, `actions/untrusted-checkout/medium`. Clarified wording on in minor point, added one more listed resource and added one more recommendation for things to check.

View File

@@ -0,0 +1,4 @@
---
category: queryMetadata
---
* Reversed adjustment of the name of `actions/untrusted-checkout/high`, but kept the portion of the previous change for the word "trusted" to "privileged". Added a missing "a" to phrasing in `actions/untrusted-checkout/high` and `actions/untrusted-checkout/medium`.

View File

@@ -1,4 +0,0 @@
---
category: queryMetadata
---
* The name, description, and alert message of `actions/untrusted-checkout/medium` have been corrected to describe a non-privileged context.

View File

@@ -1,18 +0,0 @@
## 0.6.29
### Query Metadata Changes
* Reversed adjustment of the name of `actions/untrusted-checkout/high`, but kept the portion of the previous change for the word "trusted" to "privileged". Added a missing "a" to phrasing in `actions/untrusted-checkout/high` and `actions/untrusted-checkout/medium`.
### Major Analysis Improvements
* Adjusted `actions/untrusted-checkout/critical` to align more with other untrusted resource queries, where the alert location is the location where the artifact is obtained from (the checkout point). This aligns with the other 2 related queries. This will cause the same alerts to re-open for closed alerts of this query.
### Minor Analysis Improvements
* Altered the alert message for clarity for queries: `actions/untrusted-checkout/critical`, `actions/untrusted-checkout/high`.
* The `actions/unpinned-tag` query now recognizes 64-character SHA-256 commit hashes as properly pinned references, in addition to 40-character SHA-1 hashes.
### Bug Fixes
* Adjusted (minor) help file descriptions for queries: `actions/untrusted-checkout/critical`, `actions/untrusted-checkout/high`, `actions/untrusted-checkout/medium`. Clarified wording on a minor point, added one more listed resource and added one more recommendation for things to check.

View File

@@ -1,2 +1,2 @@
--- ---
lastReleaseVersion: 0.6.29 lastReleaseVersion: 0.6.28

View File

@@ -1,5 +1,5 @@
name: codeql/actions-queries name: codeql/actions-queries
version: 0.6.30-dev version: 0.6.29-dev
library: false library: false
warnOnImplicitThis: true warnOnImplicitThis: true
groups: [actions, queries] groups: [actions, queries]

View File

@@ -1,43 +0,0 @@
name: test
on:
pull_request:
jobs:
test:
strategy:
fail-fast: false
matrix:
os:
- ubuntu-latest
- ubuntu-24.04
- ubuntu-24.04-arm
- ubuntu-22.04
- ubuntu-22.04-arm
- ubuntu-26.04
- ubuntu-26.04-arm
- ubuntu-slim
- macos-26
- macos-26-xlarge
- macos-26-intel
- macos-26-large
- macos-latest-large
- macos-15-large
- macos-15
- macos-15-intel
- macos-latest
- macos-15
- macos-15-xlarge
- macos-14-large
- macos-14
- macos-14-xlarge
- windows-2025-vs2026
- windows-latest
- windows-2025
- windows-2022
- windows-11
- windows-11-arm
- windows-11-vs2026-arm
runs-on: ${{ matrix.os }}
steps:
- run: cmd

View File

@@ -1,10 +1,10 @@
| .github/workflows/artifactpoisoning81.yml:11:9:14:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/artifactpoisoning81.yml:11:9:14:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/dependabot2.yml:33:9:38:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/dependabot2.yml:33:9:38:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/mend.yml:22:9:29:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/mend.yml:22:9:29:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/poc3.yml:18:7:25:4 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/poc3.yml:18:7:25:4 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/poc.yml:30:9:36:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/poc.yml:30:9:36:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/priv_pull_request_checkout.yml:14:9:20:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/priv_pull_request_checkout.yml:14:9:20:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/test3.yml:28:9:33:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/test3.yml:28:9:33:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/test4.yml:18:7:25:4 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/test4.yml:18:7:25:4 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/test8.yml:20:9:26:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/test8.yml:20:9:26:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |
| .github/workflows/test9.yml:11:9:16:6 | Uses Step | Potential unsafe checkout of untrusted pull request on non-privileged workflow. | | .github/workflows/test9.yml:11:9:16:6 | Uses Step | Potential unsafe checkout of untrusted pull request on privileged workflow. |

View File

@@ -11,6 +11,10 @@
"java/ql/lib/semmle/code/java/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll", "java/ql/lib/semmle/code/java/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll" "csharp/ql/lib/semmle/code/csharp/dataflow/internal/rangeanalysis/SignAnalysisCommon.qll"
], ],
"Bound Java/C#": [
"java/ql/lib/semmle/code/java/dataflow/Bound.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/Bound.qll"
],
"ModulusAnalysis Java/C#": [ "ModulusAnalysis Java/C#": [
"java/ql/lib/semmle/code/java/dataflow/ModulusAnalysis.qll", "java/ql/lib/semmle/code/java/dataflow/ModulusAnalysis.qll",
"csharp/ql/lib/semmle/code/csharp/dataflow/ModulusAnalysis.qll" "csharp/ql/lib/semmle/code/csharp/dataflow/ModulusAnalysis.qll"

View File

@@ -1,2 +0,0 @@
description: Fix NameQualifier inconsistency
compatibility: full

View File

@@ -1,19 +1,3 @@
## 10.2.0
### Deprecated APIs
* The `UsingAliasTypedefType` class has been deprecated. Use `TypeAliasType` instead.
### New Features
* Added a `getOriginalTemplate` predicate to `TemplateClass`, `TemplateFunction`, `TemplateVariable`, and `AliasTemplateType`, which yields the class member template the template was generated from. The predicates only have results for templates that are members of class template instantiations.
* Added `AliasTemplateType` and `AliasTemplateInstantiationType` classes, representing C++ alias templates and their instantiations.
### Minor Analysis Improvements
* Added flow source models for `scanf_s` and related functions.
* Added a `Call` column to `LocalFlowSourceFunction::hasLocalFlowSource` and `RemoteFlowSourceFunction::hasRemoteFlowSource`. The old predicates without a `Call` column continue to be supported.
## 10.1.1 ## 10.1.1
### Minor Analysis Improvements ### Minor Analysis Improvements

View File

@@ -0,0 +1,5 @@
---
category: minorAnalysis
---
* Added flow source models for `scanf_s` and related functions.
* Added a `Call` column to `LocalFlowSourceFunction::hasLocalFlowSource` and `RemoteFlowSourceFunction::hasRemoteFlowSource`. The old predicates without a `Call` column continue to be supported.

View File

@@ -0,0 +1,4 @@
---
category: feature
---
* Added `AliasTemplateType` and `AliasTemplateInstantiationType` classes, representing C++ alias templates and their instantiations.

View File

@@ -0,0 +1,4 @@
---
category: deprecated
---
* The `UsingAliasTypedefType` class has been deprecated. Use `TypeAliasType` instead.

View File

@@ -0,0 +1,4 @@
---
category: feature
---
* Added a `getOriginalTemplate` predicate to `TemplateClass`, `TemplateFunction`, `TemplateVariable`, and `AliasTemplateType`, which yields the class member template the template was generated from. The predicates only have results for templates that are members of class template instantiations.

View File

@@ -1,15 +0,0 @@
## 10.2.0
### Deprecated APIs
* The `UsingAliasTypedefType` class has been deprecated. Use `TypeAliasType` instead.
### New Features
* Added a `getOriginalTemplate` predicate to `TemplateClass`, `TemplateFunction`, `TemplateVariable`, and `AliasTemplateType`, which yields the class member template the template was generated from. The predicates only have results for templates that are members of class template instantiations.
* Added `AliasTemplateType` and `AliasTemplateInstantiationType` classes, representing C++ alias templates and their instantiations.
### Minor Analysis Improvements
* Added flow source models for `scanf_s` and related functions.
* Added a `Call` column to `LocalFlowSourceFunction::hasLocalFlowSource` and `RemoteFlowSourceFunction::hasRemoteFlowSource`. The old predicates without a `Call` column continue to be supported.

View File

@@ -1,2 +1,2 @@
--- ---
lastReleaseVersion: 10.2.0 lastReleaseVersion: 10.1.1

View File

@@ -1,5 +1,5 @@
name: codeql/cpp-all name: codeql/cpp-all
version: 10.2.1-dev version: 10.1.2-dev
groups: cpp groups: cpp
dbscheme: semmlecode.cpp.dbscheme dbscheme: semmlecode.cpp.dbscheme
extractor: cpp extractor: cpp

View File

@@ -1071,7 +1071,7 @@ class NullPointerType extends BuiltInType {
* const float fa[40]; * const float fa[40];
* ``` * ```
*/ */
class DerivedType extends Type, NameQualifyingElement, @derivedtype { class DerivedType extends Type, @derivedtype {
override string toString() { result = this.getName() } override string toString() { result = this.getName() }
override string getName() { derivedtypes(underlyingElement(this), result, _, _) } override string getName() { derivedtypes(underlyingElement(this), result, _, _) }

View File

@@ -276,45 +276,6 @@ private predicate isClassConstructedFrom(Class c, Class templateClass) {
not c.isConstructedFrom(_) and c = templateClass not c.isConstructedFrom(_) and c = templateClass
} }
/** Gets the fully templated version of `c`. */
private Class getFullyTemplatedClassOld(Class c) {
not c.isFromUninstantiatedTemplate(_) and
isClassConstructedFrom(c, result)
}
private TemplateClass getOriginalClassTemplate(TemplateClass tc) {
result = tc.getOriginalTemplate()
or
not exists(tc.getOriginalTemplate()) and
result = tc
}
/** Gets the fully templated version of `c`. */
private Class getFullyTemplatedClassNew(Class c) {
not c.isFromUninstantiatedTemplate(_) and
exists(Class mid |
c.isConstructedFrom(mid)
or
not c.isConstructedFrom(_) and c = mid
|
result = getOriginalClassTemplate(mid)
or
not mid instanceof TemplateClass and mid = result
)
}
/** Gets the fully templated version of `c`. */
private Class getFullyTemplatedClass(Class c) {
// The `Class::getOriginalTemplate` predicate was introduced in CodeQL
// version 2.25.6 and the upgrade script leaves the
// `class_template_generated_from` extensionals empty if the database
// was generated with an older extractor. So we use the old implementation
// if the `class_template_generated_from` extensional is empty.
if class_template_generated_from(_, _)
then result = getFullyTemplatedClassNew(c)
else result = getFullyTemplatedClassOld(c)
}
/** /**
* Holds if `f` is an instantiation of a function template `templateFunc`, or * Holds if `f` is an instantiation of a function template `templateFunc`, or
* holds with `f = templateFunc` if `f` is not an instantiation of any function * holds with `f = templateFunc` if `f` is not an instantiation of any function
@@ -331,7 +292,7 @@ private predicate isFunctionConstructedFrom(Function f, Function templateFunc) {
} }
/** Gets the fully templated version of `f`. */ /** Gets the fully templated version of `f`. */
private Function getFullyTemplatedFunctionOld(Function f) { Function getFullyTemplatedFunction(Function f) {
not f.isFromUninstantiatedTemplate(_) and not f.isFromUninstantiatedTemplate(_) and
( (
exists(Class c, Class templateClass, int i | exists(Class c, Class templateClass, int i |
@@ -345,46 +306,13 @@ private Function getFullyTemplatedFunctionOld(Function f) {
) )
} }
private TemplateFunction getOriginalFunctionTemplate(TemplateFunction tf) {
result = tf.getOriginalTemplate()
or
not exists(tf.getOriginalTemplate()) and
result = tf
}
/** Gets the fully templated version of `f`. */
private Function getFullyTemplatedFunctionNew(Function f) {
not f.isFromUninstantiatedTemplate(_) and
exists(Function mid |
f.isConstructedFrom(mid)
or
not f.isConstructedFrom(_) and f = mid
|
result = getOriginalFunctionTemplate(mid)
or
not mid instanceof TemplateFunction and mid = result
)
}
/** Gets the fully templated version of `f`. */
Function getFullyTemplatedFunction(Function f) {
// The `Function::getOriginalTemplate` predicate was introduced in CodeQL
// version 2.25.6 and the upgrade script leaves the
// `function_template_generated_from` extensionals empty if the database
// was generated with an older extractor. So we use the old implementation
// if the `function_template_generated_from` extensional is empty.
if function_template_generated_from(_, _)
then result = getFullyTemplatedFunctionNew(f)
else result = getFullyTemplatedFunctionOld(f)
}
/** Prefixes `const` to `s` if `t` is const, or returns `s` otherwise. */ /** Prefixes `const` to `s` if `t` is const, or returns `s` otherwise. */
bindingset[s, t] bindingset[s, t]
private string withConst(string s, Type t) { private string withConst(string s, Type t) {
if t.isConst() then result = "const " + s else result = s if t.isConst() then result = "const " + s else result = s
} }
/** Prefixes `volatile` to `s` if `t` is volatile, or returns `s` otherwise. */ /** Prefixes `volatile` to `s` if `t` is const, or returns `s` otherwise. */
bindingset[s, t] bindingset[s, t]
private string withVolatile(string s, Type t) { private string withVolatile(string s, Type t) {
if t.isVolatile() then result = "volatile " + s else result = s if t.isVolatile() then result = "volatile " + s else result = s
@@ -562,7 +490,7 @@ pragma[nomagic]
private string getTypeNameWithoutClassTemplates(Function f, int n, int remaining) { private string getTypeNameWithoutClassTemplates(Function f, int n, int remaining) {
// If there is a declaring type then we start by expanding the function templates // If there is a declaring type then we start by expanding the function templates
exists(Class template | exists(Class template |
template = getFullyTemplatedClass(f.getDeclaringType()) and isClassConstructedFrom(f.getDeclaringType(), template) and
remaining = getNumberOfSupportedClassTemplateArguments(template) and remaining = getNumberOfSupportedClassTemplateArguments(template) and
result = getTypeNameWithoutFunctionTemplates(f, n, 0) result = getTypeNameWithoutFunctionTemplates(f, n, 0)
) )
@@ -574,7 +502,7 @@ private string getTypeNameWithoutClassTemplates(Function f, int n, int remaining
or or
exists(string mid, TypeTemplateParameter tp, Class template | exists(string mid, TypeTemplateParameter tp, Class template |
mid = getTypeNameWithoutClassTemplates(f, n, remaining + 1) and mid = getTypeNameWithoutClassTemplates(f, n, remaining + 1) and
template = getFullyTemplatedClass(f.getDeclaringType()) and isClassConstructedFrom(f.getDeclaringType(), template) and
tp = getSupportedClassTemplateArgument(template, remaining) tp = getSupportedClassTemplateArgument(template, remaining)
| |
result = mid.replaceAll(tp.getName(), "class:" + remaining.toString()) result = mid.replaceAll(tp.getName(), "class:" + remaining.toString())

View File

@@ -1430,8 +1430,7 @@ specialnamequalifyingelements(
@namequalifyingelement = @namespace @namequalifyingelement = @namespace
| @specialnamequalifyingelement | @specialnamequalifyingelement
| @usertype | @usertype
| @decltype | @decltype;
| @derivedtype;
namequalifiers( namequalifiers(
unique int id: @namequalifier, unique int id: @namequalifier,

View File

@@ -1,2 +0,0 @@
description: Fix NameQualifier inconsistency
compatibility: full

View File

@@ -1,7 +1,3 @@
## 1.6.4
No user-facing changes.
## 1.6.3 ## 1.6.3
### Minor Analysis Improvements ### Minor Analysis Improvements

View File

@@ -1,3 +0,0 @@
## 1.6.4
No user-facing changes.

View File

@@ -1,2 +1,2 @@
--- ---
lastReleaseVersion: 1.6.4 lastReleaseVersion: 1.6.3

View File

@@ -1,5 +1,5 @@
name: codeql/cpp-queries name: codeql/cpp-queries
version: 1.6.5-dev version: 1.6.4-dev
groups: groups:
- cpp - cpp
- queries - queries

View File

@@ -51,16 +51,13 @@ models
| 50 | Summary: ; ; false; ymlStepGenerated; ; ; Argument[0]; ReturnValue; taint; df-generated | | 50 | Summary: ; ; false; ymlStepGenerated; ; ; Argument[0]; ReturnValue; taint; df-generated |
| 51 | Summary: ; ; false; ymlStepManual; ; ; Argument[0]; ReturnValue; taint; manual | | 51 | Summary: ; ; false; ymlStepManual; ; ; Argument[0]; ReturnValue; taint; manual |
| 52 | Summary: ; ; false; ymlStepManual_with_body; ; ; Argument[0]; ReturnValue; taint; manual | | 52 | Summary: ; ; false; ymlStepManual_with_body; ; ; Argument[0]; ReturnValue; taint; manual |
| 53 | Summary: ; TemplateClass1; true; templateFunction2<U,V>; (U,V); ; Argument[1]; ReturnValue; value; manual | | 53 | Summary: Azure::Core::IO; BodyStream; true; Read; ; ; Argument[-1]; Argument[*0]; taint; manual |
| 54 | Summary: ; TemplateClass1<T>; false; templateFunction<U>; (T,U); ; Argument[0]; ReturnValue; value; manual | | 54 | Summary: Azure::Core::IO; BodyStream; true; ReadToCount; ; ; Argument[-1]; Argument[*0]; taint; manual |
| 55 | Summary: ; TemplateClass2<T,U>; true; function; (U,T); ; Argument[1]; ReturnValue; value; manual | | 55 | Summary: Azure::Core::IO; BodyStream; true; ReadToEnd; ; ; Argument[-1]; ReturnValue.Element; taint; manual |
| 56 | Summary: Azure::Core::IO; BodyStream; true; Read; ; ; Argument[-1]; Argument[*0]; taint; manual | | 56 | Summary: Azure; Nullable; true; Value; ; ; Argument[-1]; ReturnValue[*]; taint; manual |
| 57 | Summary: Azure::Core::IO; BodyStream; true; ReadToCount; ; ; Argument[-1]; Argument[*0]; taint; manual | | 57 | Summary: boost::asio; ; false; buffer; ; ; Argument[*0]; ReturnValue; taint; manual |
| 58 | Summary: Azure::Core::IO; BodyStream; true; ReadToEnd; ; ; Argument[-1]; ReturnValue.Element; taint; manual |
| 59 | Summary: Azure; Nullable; true; Value; ; ; Argument[-1]; ReturnValue[*]; taint; manual |
| 60 | Summary: boost::asio; ; false; buffer; ; ; Argument[*0]; ReturnValue; taint; manual |
edges edges
| asio_streams.cpp:56:18:56:23 | [summary param] *0 in buffer | asio_streams.cpp:56:18:56:23 | [summary] to write: ReturnValue in buffer | provenance | MaD:60 | | asio_streams.cpp:56:18:56:23 | [summary param] *0 in buffer | asio_streams.cpp:56:18:56:23 | [summary] to write: ReturnValue in buffer | provenance | MaD:57 |
| asio_streams.cpp:87:34:87:44 | read_until output argument | asio_streams.cpp:91:7:91:17 | recv_buffer | provenance | Src:MaD:32 | | asio_streams.cpp:87:34:87:44 | read_until output argument | asio_streams.cpp:91:7:91:17 | recv_buffer | provenance | Src:MaD:32 |
| asio_streams.cpp:87:34:87:44 | read_until output argument | asio_streams.cpp:93:29:93:39 | *recv_buffer | provenance | Src:MaD:32 Sink:MaD:2 | | asio_streams.cpp:87:34:87:44 | read_until output argument | asio_streams.cpp:93:29:93:39 | *recv_buffer | provenance | Src:MaD:32 Sink:MaD:2 |
| asio_streams.cpp:97:37:97:44 | call to source | asio_streams.cpp:98:7:98:14 | send_str | provenance | TaintFunction | | asio_streams.cpp:97:37:97:44 | call to source | asio_streams.cpp:98:7:98:14 | send_str | provenance | TaintFunction |
@@ -69,24 +66,24 @@ edges
| asio_streams.cpp:100:44:100:62 | call to buffer | asio_streams.cpp:101:7:101:17 | send_buffer | provenance | | | asio_streams.cpp:100:44:100:62 | call to buffer | asio_streams.cpp:101:7:101:17 | send_buffer | provenance | |
| asio_streams.cpp:100:44:100:62 | call to buffer | asio_streams.cpp:103:29:103:39 | *send_buffer | provenance | Sink:MaD:2 | | asio_streams.cpp:100:44:100:62 | call to buffer | asio_streams.cpp:103:29:103:39 | *send_buffer | provenance | Sink:MaD:2 |
| asio_streams.cpp:100:64:100:71 | *send_str | asio_streams.cpp:56:18:56:23 | [summary param] *0 in buffer | provenance | | | asio_streams.cpp:100:64:100:71 | *send_str | asio_streams.cpp:56:18:56:23 | [summary param] *0 in buffer | provenance | |
| asio_streams.cpp:100:64:100:71 | *send_str | asio_streams.cpp:100:44:100:62 | call to buffer | provenance | MaD:60 | | asio_streams.cpp:100:64:100:71 | *send_str | asio_streams.cpp:100:44:100:62 | call to buffer | provenance | MaD:57 |
| azure.cpp:62:10:62:14 | [summary param] this in Value | azure.cpp:62:10:62:14 | [summary] to write: ReturnValue[*] in Value | provenance | MaD:59 | | azure.cpp:62:10:62:14 | [summary param] this in Value | azure.cpp:62:10:62:14 | [summary] to write: ReturnValue[*] in Value | provenance | MaD:56 |
| azure.cpp:113:16:113:19 | [summary param] this in Read | azure.cpp:113:16:113:19 | [summary param] *0 in Read [Return] | provenance | MaD:56 | | azure.cpp:113:16:113:19 | [summary param] this in Read | azure.cpp:113:16:113:19 | [summary param] *0 in Read [Return] | provenance | MaD:53 |
| azure.cpp:114:16:114:26 | [summary param] this in ReadToCount | azure.cpp:114:16:114:26 | [summary param] *0 in ReadToCount [Return] | provenance | MaD:57 | | azure.cpp:114:16:114:26 | [summary param] this in ReadToCount | azure.cpp:114:16:114:26 | [summary param] *0 in ReadToCount [Return] | provenance | MaD:54 |
| azure.cpp:115:30:115:38 | [summary param] this in ReadToEnd | azure.cpp:115:30:115:38 | [summary] to write: ReturnValue.Element in ReadToEnd | provenance | MaD:58 | | azure.cpp:115:30:115:38 | [summary param] this in ReadToEnd | azure.cpp:115:30:115:38 | [summary] to write: ReturnValue.Element in ReadToEnd | provenance | MaD:55 |
| azure.cpp:115:30:115:38 | [summary] to write: ReturnValue.Element in ReadToEnd | azure.cpp:115:30:115:38 | [summary] to write: ReturnValue in ReadToEnd [element] | provenance | | | azure.cpp:115:30:115:38 | [summary] to write: ReturnValue.Element in ReadToEnd | azure.cpp:115:30:115:38 | [summary] to write: ReturnValue in ReadToEnd [element] | provenance | |
| azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:253:48:253:60 | *call to GetBodyStream | provenance | Src:MaD:29 | | azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:253:48:253:60 | *call to GetBodyStream | provenance | Src:MaD:29 |
| azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:257:5:257:8 | *resp | provenance | | | azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:257:5:257:8 | *resp | provenance | |
| azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:262:5:262:8 | *resp | provenance | | | azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:262:5:262:8 | *resp | provenance | |
| azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:266:38:266:41 | *resp | provenance | | | azure.cpp:253:48:253:60 | *call to GetBodyStream | azure.cpp:266:38:266:41 | *resp | provenance | |
| azure.cpp:257:5:257:8 | *resp | azure.cpp:113:16:113:19 | [summary param] this in Read | provenance | | | azure.cpp:257:5:257:8 | *resp | azure.cpp:113:16:113:19 | [summary param] this in Read | provenance | |
| azure.cpp:257:5:257:8 | *resp | azure.cpp:257:16:257:21 | Read output argument | provenance | MaD:56 | | azure.cpp:257:5:257:8 | *resp | azure.cpp:257:16:257:21 | Read output argument | provenance | MaD:53 |
| azure.cpp:257:16:257:21 | Read output argument | azure.cpp:258:10:258:16 | * ... | provenance | | | azure.cpp:257:16:257:21 | Read output argument | azure.cpp:258:10:258:16 | * ... | provenance | |
| azure.cpp:262:5:262:8 | *resp | azure.cpp:114:16:114:26 | [summary param] this in ReadToCount | provenance | | | azure.cpp:262:5:262:8 | *resp | azure.cpp:114:16:114:26 | [summary param] this in ReadToCount | provenance | |
| azure.cpp:262:5:262:8 | *resp | azure.cpp:262:23:262:28 | ReadToCount output argument | provenance | MaD:57 | | azure.cpp:262:5:262:8 | *resp | azure.cpp:262:23:262:28 | ReadToCount output argument | provenance | MaD:54 |
| azure.cpp:262:23:262:28 | ReadToCount output argument | azure.cpp:263:10:263:16 | * ... | provenance | | | azure.cpp:262:23:262:28 | ReadToCount output argument | azure.cpp:263:10:263:16 | * ... | provenance | |
| azure.cpp:266:38:266:41 | *resp | azure.cpp:115:30:115:38 | [summary param] this in ReadToEnd | provenance | | | azure.cpp:266:38:266:41 | *resp | azure.cpp:115:30:115:38 | [summary param] this in ReadToEnd | provenance | |
| azure.cpp:266:38:266:41 | *resp | azure.cpp:266:44:266:52 | call to ReadToEnd [element] | provenance | MaD:58 | | azure.cpp:266:38:266:41 | *resp | azure.cpp:266:44:266:52 | call to ReadToEnd [element] | provenance | MaD:55 |
| azure.cpp:266:44:266:52 | call to ReadToEnd [element] | azure.cpp:266:44:266:52 | call to ReadToEnd [element] | provenance | | | azure.cpp:266:44:266:52 | call to ReadToEnd [element] | azure.cpp:266:44:266:52 | call to ReadToEnd [element] | provenance | |
| azure.cpp:266:44:266:52 | call to ReadToEnd [element] | azure.cpp:267:10:267:12 | vec [element] | provenance | | | azure.cpp:266:44:266:52 | call to ReadToEnd [element] | azure.cpp:267:10:267:12 | vec [element] | provenance | |
| azure.cpp:267:10:267:12 | vec [element] | azure.cpp:267:10:267:12 | vec | provenance | | | azure.cpp:267:10:267:12 | vec [element] | azure.cpp:267:10:267:12 | vec | provenance | |
@@ -103,11 +100,11 @@ edges
| azure.cpp:281:68:281:84 | *call to ExtractBodyStream | azure.cpp:281:68:281:84 | *call to ExtractBodyStream | provenance | Src:MaD:26 | | azure.cpp:281:68:281:84 | *call to ExtractBodyStream | azure.cpp:281:68:281:84 | *call to ExtractBodyStream | provenance | Src:MaD:26 |
| azure.cpp:281:68:281:84 | *call to ExtractBodyStream | azure.cpp:282:21:282:23 | *call to get | provenance | | | azure.cpp:281:68:281:84 | *call to ExtractBodyStream | azure.cpp:282:21:282:23 | *call to get | provenance | |
| azure.cpp:282:21:282:23 | *call to get | azure.cpp:115:30:115:38 | [summary param] this in ReadToEnd | provenance | | | azure.cpp:282:21:282:23 | *call to get | azure.cpp:115:30:115:38 | [summary param] this in ReadToEnd | provenance | |
| azure.cpp:282:21:282:23 | *call to get | azure.cpp:282:28:282:36 | call to ReadToEnd [element] | provenance | MaD:58 | | azure.cpp:282:21:282:23 | *call to get | azure.cpp:282:28:282:36 | call to ReadToEnd [element] | provenance | MaD:55 |
| azure.cpp:282:28:282:36 | call to ReadToEnd [element] | azure.cpp:282:10:282:38 | call to ReadToEnd | provenance | | | azure.cpp:282:28:282:36 | call to ReadToEnd [element] | azure.cpp:282:10:282:38 | call to ReadToEnd | provenance | |
| azure.cpp:282:28:282:36 | call to ReadToEnd [element] | azure.cpp:282:28:282:36 | call to ReadToEnd [element] | provenance | | | azure.cpp:282:28:282:36 | call to ReadToEnd [element] | azure.cpp:282:28:282:36 | call to ReadToEnd [element] | provenance | |
| azure.cpp:289:24:289:56 | call to GetHeader | azure.cpp:62:10:62:14 | [summary param] this in Value | provenance | | | azure.cpp:289:24:289:56 | call to GetHeader | azure.cpp:62:10:62:14 | [summary param] this in Value | provenance | |
| azure.cpp:289:24:289:56 | call to GetHeader | azure.cpp:289:63:289:65 | call to Value | provenance | MaD:59 | | azure.cpp:289:24:289:56 | call to GetHeader | azure.cpp:289:63:289:65 | call to Value | provenance | MaD:56 |
| azure.cpp:289:32:289:40 | call to GetHeader | azure.cpp:289:24:289:56 | call to GetHeader | provenance | | | azure.cpp:289:32:289:40 | call to GetHeader | azure.cpp:289:24:289:56 | call to GetHeader | provenance | |
| azure.cpp:289:32:289:40 | call to GetHeader | azure.cpp:289:32:289:40 | call to GetHeader | provenance | Src:MaD:30 | | azure.cpp:289:32:289:40 | call to GetHeader | azure.cpp:289:32:289:40 | call to GetHeader | provenance | Src:MaD:30 |
| azure.cpp:289:63:289:65 | call to Value | azure.cpp:289:63:289:65 | call to Value | provenance | | | azure.cpp:289:63:289:65 | call to Value | azure.cpp:289:63:289:65 | call to Value | provenance | |
@@ -183,39 +180,6 @@ edges
| test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | test.cpp:119:10:119:11 | y2 | provenance | Sink:MaD:1 | | test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | test.cpp:119:10:119:11 | y2 | provenance | Sink:MaD:1 |
| test.cpp:118:44:118:44 | *x | test.cpp:111:3:111:25 | [summary param] *0 in callWithNonTypeTemplate | provenance | | | test.cpp:118:44:118:44 | *x | test.cpp:111:3:111:25 | [summary param] *0 in callWithNonTypeTemplate | provenance | |
| test.cpp:118:44:118:44 | *x | test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | provenance | MaD:48 | | test.cpp:118:44:118:44 | *x | test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | provenance | MaD:48 |
| test.cpp:125:5:125:20 | [summary param] 0 in templateFunction | test.cpp:125:5:125:20 | [summary] to write: ReturnValue in templateFunction | provenance | MaD:54 |
| test.cpp:128:5:128:21 | [summary param] 1 in templateFunction2 | test.cpp:128:5:128:21 | [summary] to write: ReturnValue in templateFunction2 | provenance | MaD:53 |
| test.cpp:133:10:133:18 | call to ymlSource | test.cpp:133:10:133:18 | call to ymlSource | provenance | Src:MaD:25 |
| test.cpp:133:10:133:18 | call to ymlSource | test.cpp:134:45:134:45 | x | provenance | |
| test.cpp:134:13:134:43 | call to templateFunction | test.cpp:134:13:134:43 | call to templateFunction | provenance | |
| test.cpp:134:13:134:43 | call to templateFunction | test.cpp:135:10:135:10 | y | provenance | Sink:MaD:1 |
| test.cpp:134:45:134:45 | x | test.cpp:125:5:125:20 | [summary param] 0 in templateFunction | provenance | |
| test.cpp:134:45:134:45 | x | test.cpp:134:13:134:43 | call to templateFunction | provenance | MaD:54 |
| test.cpp:140:4:140:11 | [summary param] 1 in function | test.cpp:140:4:140:11 | [summary] to write: ReturnValue in function | provenance | MaD:55 |
| test.cpp:140:4:140:11 | [summary param] 1 in function | test.cpp:140:4:140:11 | [summary] to write: ReturnValue in function | provenance | MaD:55 |
| test.cpp:146:10:146:18 | call to ymlSource | test.cpp:146:10:146:18 | call to ymlSource | provenance | Src:MaD:25 |
| test.cpp:146:10:146:18 | call to ymlSource | test.cpp:148:26:148:26 | x | provenance | |
| test.cpp:148:10:148:27 | call to function | test.cpp:148:10:148:27 | call to function | provenance | |
| test.cpp:148:10:148:27 | call to function | test.cpp:149:10:149:10 | z | provenance | Sink:MaD:1 |
| test.cpp:148:26:148:26 | x | test.cpp:140:4:140:11 | [summary param] 1 in function | provenance | |
| test.cpp:148:26:148:26 | x | test.cpp:148:10:148:27 | call to function | provenance | MaD:55 |
| test.cpp:155:10:155:18 | call to ymlSource | test.cpp:155:10:155:18 | call to ymlSource | provenance | Src:MaD:25 |
| test.cpp:155:10:155:18 | call to ymlSource | test.cpp:157:26:157:26 | x | provenance | |
| test.cpp:157:13:157:20 | call to function | test.cpp:157:13:157:20 | call to function | provenance | |
| test.cpp:157:13:157:20 | call to function | test.cpp:158:10:158:10 | z | provenance | Sink:MaD:1 |
| test.cpp:157:26:157:26 | x | test.cpp:140:4:140:11 | [summary param] 1 in function | provenance | |
| test.cpp:157:26:157:26 | x | test.cpp:157:13:157:20 | call to function | provenance | MaD:55 |
| test.cpp:164:34:164:34 | x | test.cpp:165:69:165:69 | x | provenance | |
| test.cpp:165:12:165:64 | call to templateFunction2 | test.cpp:164:7:164:7 | *templateFunction3 | provenance | |
| test.cpp:165:12:165:64 | call to templateFunction2 | test.cpp:165:12:165:64 | call to templateFunction2 | provenance | |
| test.cpp:165:69:165:69 | x | test.cpp:128:5:128:21 | [summary param] 1 in templateFunction2 | provenance | |
| test.cpp:165:69:165:69 | x | test.cpp:165:12:165:64 | call to templateFunction2 | provenance | MaD:53 |
| test.cpp:170:10:170:18 | call to ymlSource | test.cpp:170:10:170:18 | call to ymlSource | provenance | Src:MaD:25 |
| test.cpp:170:10:170:18 | call to ymlSource | test.cpp:172:51:172:51 | x | provenance | |
| test.cpp:172:13:172:44 | call to templateFunction3 | test.cpp:172:13:172:44 | call to templateFunction3 | provenance | |
| test.cpp:172:13:172:44 | call to templateFunction3 | test.cpp:173:10:173:10 | y | provenance | Sink:MaD:1 |
| test.cpp:172:51:172:51 | x | test.cpp:164:34:164:34 | x | provenance | |
| test.cpp:172:51:172:51 | x | test.cpp:172:13:172:44 | call to templateFunction3 | provenance | MaD:53 |
| windows.cpp:17:8:17:25 | [summary param] *0 in CommandLineToArgvA | windows.cpp:17:8:17:25 | [summary] to write: ReturnValue[**] in CommandLineToArgvA | provenance | MaD:33 | | windows.cpp:17:8:17:25 | [summary param] *0 in CommandLineToArgvA | windows.cpp:17:8:17:25 | [summary] to write: ReturnValue[**] in CommandLineToArgvA | provenance | MaD:33 |
| windows.cpp:22:15:22:29 | *call to GetCommandLineA | windows.cpp:22:15:22:29 | *call to GetCommandLineA | provenance | Src:MaD:3 | | windows.cpp:22:15:22:29 | *call to GetCommandLineA | windows.cpp:22:15:22:29 | *call to GetCommandLineA | provenance | Src:MaD:3 |
| windows.cpp:22:15:22:29 | *call to GetCommandLineA | windows.cpp:24:8:24:11 | * ... | provenance | | | windows.cpp:22:15:22:29 | *call to GetCommandLineA | windows.cpp:24:8:24:11 | * ... | provenance | |
@@ -519,43 +483,6 @@ nodes
| test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | semmle.label | call to callWithNonTypeTemplate | | test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | semmle.label | call to callWithNonTypeTemplate |
| test.cpp:118:44:118:44 | *x | semmle.label | *x | | test.cpp:118:44:118:44 | *x | semmle.label | *x |
| test.cpp:119:10:119:11 | y2 | semmle.label | y2 | | test.cpp:119:10:119:11 | y2 | semmle.label | y2 |
| test.cpp:125:5:125:20 | [summary param] 0 in templateFunction | semmle.label | [summary param] 0 in templateFunction |
| test.cpp:125:5:125:20 | [summary] to write: ReturnValue in templateFunction | semmle.label | [summary] to write: ReturnValue in templateFunction |
| test.cpp:128:5:128:21 | [summary param] 1 in templateFunction2 | semmle.label | [summary param] 1 in templateFunction2 |
| test.cpp:128:5:128:21 | [summary] to write: ReturnValue in templateFunction2 | semmle.label | [summary] to write: ReturnValue in templateFunction2 |
| test.cpp:133:10:133:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:133:10:133:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:134:13:134:43 | call to templateFunction | semmle.label | call to templateFunction |
| test.cpp:134:13:134:43 | call to templateFunction | semmle.label | call to templateFunction |
| test.cpp:134:45:134:45 | x | semmle.label | x |
| test.cpp:135:10:135:10 | y | semmle.label | y |
| test.cpp:140:4:140:11 | [summary param] 1 in function | semmle.label | [summary param] 1 in function |
| test.cpp:140:4:140:11 | [summary param] 1 in function | semmle.label | [summary param] 1 in function |
| test.cpp:140:4:140:11 | [summary] to write: ReturnValue in function | semmle.label | [summary] to write: ReturnValue in function |
| test.cpp:140:4:140:11 | [summary] to write: ReturnValue in function | semmle.label | [summary] to write: ReturnValue in function |
| test.cpp:146:10:146:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:146:10:146:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:148:10:148:27 | call to function | semmle.label | call to function |
| test.cpp:148:10:148:27 | call to function | semmle.label | call to function |
| test.cpp:148:26:148:26 | x | semmle.label | x |
| test.cpp:149:10:149:10 | z | semmle.label | z |
| test.cpp:155:10:155:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:155:10:155:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:157:13:157:20 | call to function | semmle.label | call to function |
| test.cpp:157:13:157:20 | call to function | semmle.label | call to function |
| test.cpp:157:26:157:26 | x | semmle.label | x |
| test.cpp:158:10:158:10 | z | semmle.label | z |
| test.cpp:164:7:164:7 | *templateFunction3 | semmle.label | *templateFunction3 |
| test.cpp:164:34:164:34 | x | semmle.label | x |
| test.cpp:165:12:165:64 | call to templateFunction2 | semmle.label | call to templateFunction2 |
| test.cpp:165:12:165:64 | call to templateFunction2 | semmle.label | call to templateFunction2 |
| test.cpp:165:69:165:69 | x | semmle.label | x |
| test.cpp:170:10:170:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:170:10:170:18 | call to ymlSource | semmle.label | call to ymlSource |
| test.cpp:172:13:172:44 | call to templateFunction3 | semmle.label | call to templateFunction3 |
| test.cpp:172:13:172:44 | call to templateFunction3 | semmle.label | call to templateFunction3 |
| test.cpp:172:51:172:51 | x | semmle.label | x |
| test.cpp:173:10:173:10 | y | semmle.label | y |
| windows.cpp:17:8:17:25 | [summary param] *0 in CommandLineToArgvA | semmle.label | [summary param] *0 in CommandLineToArgvA | | windows.cpp:17:8:17:25 | [summary param] *0 in CommandLineToArgvA | semmle.label | [summary param] *0 in CommandLineToArgvA |
| windows.cpp:17:8:17:25 | [summary] to write: ReturnValue[**] in CommandLineToArgvA | semmle.label | [summary] to write: ReturnValue[**] in CommandLineToArgvA | | windows.cpp:17:8:17:25 | [summary] to write: ReturnValue[**] in CommandLineToArgvA | semmle.label | [summary] to write: ReturnValue[**] in CommandLineToArgvA |
| windows.cpp:22:15:22:29 | *call to GetCommandLineA | semmle.label | *call to GetCommandLineA | | windows.cpp:22:15:22:29 | *call to GetCommandLineA | semmle.label | *call to GetCommandLineA |
@@ -761,11 +688,6 @@ subpaths
| test.cpp:25:35:25:35 | x | test.cpp:6:5:6:27 | [summary param] 0 in ymlStepManual_with_body | test.cpp:6:5:6:27 | [summary] to write: ReturnValue in ymlStepManual_with_body | test.cpp:25:11:25:33 | call to ymlStepManual_with_body | | test.cpp:25:35:25:35 | x | test.cpp:6:5:6:27 | [summary param] 0 in ymlStepManual_with_body | test.cpp:6:5:6:27 | [summary] to write: ReturnValue in ymlStepManual_with_body | test.cpp:25:11:25:33 | call to ymlStepManual_with_body |
| test.cpp:32:41:32:41 | x | test.cpp:7:47:7:52 | value2 | test.cpp:7:5:7:30 | *ymlStepGenerated_with_body | test.cpp:32:11:32:36 | call to ymlStepGenerated_with_body | | test.cpp:32:41:32:41 | x | test.cpp:7:47:7:52 | value2 | test.cpp:7:5:7:30 | *ymlStepGenerated_with_body | test.cpp:32:11:32:36 | call to ymlStepGenerated_with_body |
| test.cpp:118:44:118:44 | *x | test.cpp:111:3:111:25 | [summary param] *0 in callWithNonTypeTemplate | test.cpp:111:3:111:25 | [summary] to write: ReturnValue in callWithNonTypeTemplate | test.cpp:118:11:118:42 | call to callWithNonTypeTemplate | | test.cpp:118:44:118:44 | *x | test.cpp:111:3:111:25 | [summary param] *0 in callWithNonTypeTemplate | test.cpp:111:3:111:25 | [summary] to write: ReturnValue in callWithNonTypeTemplate | test.cpp:118:11:118:42 | call to callWithNonTypeTemplate |
| test.cpp:134:45:134:45 | x | test.cpp:125:5:125:20 | [summary param] 0 in templateFunction | test.cpp:125:5:125:20 | [summary] to write: ReturnValue in templateFunction | test.cpp:134:13:134:43 | call to templateFunction |
| test.cpp:148:26:148:26 | x | test.cpp:140:4:140:11 | [summary param] 1 in function | test.cpp:140:4:140:11 | [summary] to write: ReturnValue in function | test.cpp:148:10:148:27 | call to function |
| test.cpp:157:26:157:26 | x | test.cpp:140:4:140:11 | [summary param] 1 in function | test.cpp:140:4:140:11 | [summary] to write: ReturnValue in function | test.cpp:157:13:157:20 | call to function |
| test.cpp:165:69:165:69 | x | test.cpp:128:5:128:21 | [summary param] 1 in templateFunction2 | test.cpp:128:5:128:21 | [summary] to write: ReturnValue in templateFunction2 | test.cpp:165:12:165:64 | call to templateFunction2 |
| test.cpp:172:51:172:51 | x | test.cpp:164:34:164:34 | x | test.cpp:164:7:164:7 | *templateFunction3 | test.cpp:172:13:172:44 | call to templateFunction3 |
| windows.cpp:27:36:27:38 | *cmd | windows.cpp:17:8:17:25 | [summary param] *0 in CommandLineToArgvA | windows.cpp:17:8:17:25 | [summary] to write: ReturnValue[**] in CommandLineToArgvA | windows.cpp:27:17:27:34 | **call to CommandLineToArgvA | | windows.cpp:27:36:27:38 | *cmd | windows.cpp:17:8:17:25 | [summary param] *0 in CommandLineToArgvA | windows.cpp:17:8:17:25 | [summary] to write: ReturnValue[**] in CommandLineToArgvA | windows.cpp:27:17:27:34 | **call to CommandLineToArgvA |
| windows.cpp:537:40:537:41 | *& ... | windows.cpp:473:17:473:37 | [summary param] *1 in RtlCopyVolatileMemory | windows.cpp:473:17:473:37 | [summary param] *0 in RtlCopyVolatileMemory [Return] | windows.cpp:537:27:537:37 | RtlCopyVolatileMemory output argument | | windows.cpp:537:40:537:41 | *& ... | windows.cpp:473:17:473:37 | [summary param] *1 in RtlCopyVolatileMemory | windows.cpp:473:17:473:37 | [summary param] *0 in RtlCopyVolatileMemory [Return] | windows.cpp:537:27:537:37 | RtlCopyVolatileMemory output argument |
| windows.cpp:542:38:542:39 | *& ... | windows.cpp:479:17:479:35 | [summary param] *1 in RtlCopyDeviceMemory | windows.cpp:479:17:479:35 | [summary param] *0 in RtlCopyDeviceMemory [Return] | windows.cpp:542:25:542:35 | RtlCopyDeviceMemory output argument | | windows.cpp:542:38:542:39 | *& ... | windows.cpp:479:17:479:35 | [summary param] *1 in RtlCopyDeviceMemory | windows.cpp:479:17:479:35 | [summary param] *0 in RtlCopyDeviceMemory [Return] | windows.cpp:542:25:542:35 | RtlCopyDeviceMemory output argument |

View File

@@ -18,7 +18,4 @@ extensions:
- ["", "", False, "ymlStepManual_with_body", "", "", "Argument[0]", "ReturnValue", "taint", "manual"] - ["", "", False, "ymlStepManual_with_body", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]
- ["", "", False, "ymlStepGenerated_with_body", "", "", "Argument[0]", "ReturnValue", "taint", "df-generated"] - ["", "", False, "ymlStepGenerated_with_body", "", "", "Argument[0]", "ReturnValue", "taint", "df-generated"]
- ["", "", False, "callWithArgument", "", "", "Argument[1]", "Argument[0].Parameter[0]", "value", "manual"] - ["", "", False, "callWithArgument", "", "", "Argument[1]", "Argument[0].Parameter[0]", "value", "manual"]
- ["", "", False, "callWithNonTypeTemplate<T>", "(const T &)", "", "Argument[*0]", "ReturnValue", "value", "manual"] - ["", "", False, "callWithNonTypeTemplate<T>", "(const T &)", "", "Argument[*0]", "ReturnValue", "value", "manual"]
- ["", "TemplateClass1<T>", False, "templateFunction<U>", "(T,U)", "", "Argument[0]", "ReturnValue", "value", "manual"]
- ["", "TemplateClass1", True, "templateFunction2<U,V>", "(U,V)", "", "Argument[1]", "ReturnValue", "value", "manual"]
- ["", "TemplateClass2<T,U>", True, "function", "(U,T)", "", "Argument[1]", "ReturnValue", "value", "manual"]

View File

@@ -15,7 +15,3 @@
| test.cpp:89:11:89:11 | y | test-sink | | test.cpp:89:11:89:11 | y | test-sink |
| test.cpp:116:10:116:11 | y1 | test-sink | | test.cpp:116:10:116:11 | y1 | test-sink |
| test.cpp:119:10:119:11 | y2 | test-sink | | test.cpp:119:10:119:11 | y2 | test-sink |
| test.cpp:135:10:135:10 | y | test-sink |
| test.cpp:149:10:149:10 | z | test-sink |
| test.cpp:158:10:158:10 | z | test-sink |
| test.cpp:173:10:173:10 | y | test-sink |

View File

@@ -9,10 +9,6 @@
| test.cpp:56:8:56:16 | call to ymlSource | local | | test.cpp:56:8:56:16 | call to ymlSource | local |
| test.cpp:94:10:94:18 | call to ymlSource | local | | test.cpp:94:10:94:18 | call to ymlSource | local |
| test.cpp:114:10:114:18 | call to ymlSource | local | | test.cpp:114:10:114:18 | call to ymlSource | local |
| test.cpp:133:10:133:18 | call to ymlSource | local |
| test.cpp:146:10:146:18 | call to ymlSource | local |
| test.cpp:155:10:155:18 | call to ymlSource | local |
| test.cpp:170:10:170:18 | call to ymlSource | local |
| windows.cpp:22:15:22:29 | *call to GetCommandLineA | local | | windows.cpp:22:15:22:29 | *call to GetCommandLineA | local |
| windows.cpp:34:17:34:38 | *call to GetEnvironmentStringsA | local | | windows.cpp:34:17:34:38 | *call to GetEnvironmentStringsA | local |
| windows.cpp:39:36:39:38 | GetEnvironmentVariableA output argument | local | | windows.cpp:39:36:39:38 | GetEnvironmentVariableA output argument | local |

View File

@@ -118,57 +118,3 @@ void test_callWithNonTypeTemplate() {
int y2 = callWithNonTypeTemplate<int, 10>(x); int y2 = callWithNonTypeTemplate<int, 10>(x);
ymlSink(y2); // $ ir ymlSink(y2); // $ ir
} }
template<class T>
struct TemplateClass1 {
template<class U>
U templateFunction(T, U);
template<class U, class V>
V templateFunction2(U, V);
};
void test_template_function_in_template_class() {
TemplateClass1<int> b;
int x = ymlSource();
auto y = b.templateFunction<unsigned long>(x, 0UL);
ymlSink(y); // $ ir
}
template<class S, class T>
struct TemplateClass2 {
T function(T, S);
};
template<class V> using PartialInstantiationOfTemplateClass2 = TemplateClass2<int, V>;
void test_partial_class_instantiation() {
int x = ymlSource();
PartialInstantiationOfTemplateClass2<unsigned long> y;
int z = y.function(0UL, x);
ymlSink(z); // $ ir
}
template<class V> struct DeriveFromFromPartialTemplateInstantiation : TemplateClass2<int, V> { };
void test_inheritance() {
int x = ymlSource();
DeriveFromFromPartialTemplateInstantiation<long> y;
auto z = y.function(0L, x);
ymlSink(z); // $ ir
}
template<class T>
struct Class1 : TemplateClass1<T> {
template<class U>
int templateFunction3(U u, int x) {
return TemplateClass1<T>::template templateFunction2<U, int>(u, x);
}
};
void test_class1() {
int x = ymlSource();
Class1<int> c;
auto y = c.templateFunction3<unsigned long>(0UL, x);
ymlSink(y); // $ ir
}

View File

@@ -27383,55 +27383,54 @@ getParameterTypeName
| stl.h:91:24:91:33 | operator++ | 0 | int | | stl.h:91:24:91:33 | operator++ | 0 | int |
| stl.h:95:44:95:44 | back_inserter | 0 | func:0 & | | stl.h:95:44:95:44 | back_inserter | 0 | func:0 & |
| stl.h:95:44:95:44 | back_inserter | 0 | func:0 & | | stl.h:95:44:95:44 | back_inserter | 0 | func:0 & |
| stl.h:147:12:147:23 | basic_string | 0 | const class:2 & | | stl.h:148:3:148:14 | basic_string | 0 | const class:2 & |
| stl.h:148:3:148:14 | basic_string | 0 | const class:0 * | | stl.h:149:33:149:44 | basic_string | 0 | const class:0 * |
| stl.h:148:3:148:14 | basic_string | 1 | const class:2 & | | stl.h:149:33:149:44 | basic_string | 1 | const class:2 & |
| stl.h:149:33:149:44 | basic_string | 0 | func:0 | | stl.h:151:16:151:20 | c_str | 0 | func:0 |
| stl.h:149:33:149:44 | basic_string | 1 | func:0 | | stl.h:151:16:151:20 | c_str | 1 | func:0 |
| stl.h:149:33:149:44 | basic_string | 2 | const class:2 & | | stl.h:151:16:151:20 | c_str | 2 | const class:2 & |
| stl.h:165:8:165:16 | push_back | 0 | class:0 |
| stl.h:173:13:173:22 | operator[] | 0 | size_type | | stl.h:173:13:173:22 | operator[] | 0 | size_type |
| stl.h:175:13:175:14 | at | 0 | size_type | | stl.h:175:13:175:14 | at | 0 | size_type |
| stl.h:176:35:176:44 | operator+= | 0 | const func:0 & | | stl.h:176:35:176:44 | operator+= | 0 | size_type |
| stl.h:176:35:176:44 | operator+= | 0 | const func:0 & | | stl.h:176:35:176:44 | operator+= | 0 | size_type |
| stl.h:177:17:177:26 | operator+= | 0 | const class:0 * | | stl.h:177:17:177:26 | operator+= | 0 | const func:0 & |
| stl.h:178:17:178:22 | append | 0 | const basic_string & | | stl.h:178:17:178:22 | append | 0 | const class:0 * |
| stl.h:179:17:179:22 | append | 0 | const class:0 * | | stl.h:179:17:179:22 | append | 0 | const basic_string & |
| stl.h:180:17:180:22 | append | 0 | size_type | | stl.h:180:17:180:22 | append | 0 | const class:0 * |
| stl.h:180:17:180:22 | append | 1 | class:0 | | stl.h:181:47:181:52 | append | 0 | size_type |
| stl.h:181:47:181:52 | append | 0 | func:0 | | stl.h:181:47:181:52 | append | 1 | class:0 |
| stl.h:181:47:181:52 | append | 1 | func:0 | | stl.h:182:17:182:22 | assign | 0 | func:0 |
| stl.h:182:17:182:22 | assign | 0 | const basic_string & | | stl.h:182:17:182:22 | assign | 1 | func:0 |
| stl.h:183:17:183:22 | assign | 0 | size_type | | stl.h:183:17:183:22 | assign | 0 | const basic_string & |
| stl.h:183:17:183:22 | assign | 1 | class:0 | | stl.h:184:47:184:52 | assign | 0 | size_type |
| stl.h:184:47:184:52 | assign | 0 | func:0 | | stl.h:184:47:184:52 | assign | 1 | class:0 |
| stl.h:184:47:184:52 | assign | 1 | func:0 | | stl.h:185:17:185:22 | insert | 0 | func:0 |
| stl.h:185:17:185:22 | insert | 0 | size_type | | stl.h:185:17:185:22 | insert | 1 | func:0 |
| stl.h:185:17:185:22 | insert | 1 | const basic_string & |
| stl.h:186:17:186:22 | insert | 0 | size_type | | stl.h:186:17:186:22 | insert | 0 | size_type |
| stl.h:186:17:186:22 | insert | 1 | size_type | | stl.h:186:17:186:22 | insert | 1 | const basic_string & |
| stl.h:186:17:186:22 | insert | 2 | class:0 |
| stl.h:187:17:187:22 | insert | 0 | size_type | | stl.h:187:17:187:22 | insert | 0 | size_type |
| stl.h:187:17:187:22 | insert | 1 | const class:0 * | | stl.h:187:17:187:22 | insert | 1 | size_type |
| stl.h:188:12:188:17 | insert | 0 | const_iterator | | stl.h:187:17:187:22 | insert | 2 | class:0 |
| stl.h:188:12:188:17 | insert | 1 | size_type | | stl.h:188:12:188:17 | insert | 0 | size_type |
| stl.h:188:12:188:17 | insert | 2 | class:0 | | stl.h:188:12:188:17 | insert | 1 | const class:0 * |
| stl.h:189:42:189:47 | insert | 0 | const_iterator | | stl.h:189:42:189:47 | insert | 0 | const_iterator |
| stl.h:189:42:189:47 | insert | 1 | func:0 | | stl.h:189:42:189:47 | insert | 1 | size_type |
| stl.h:189:42:189:47 | insert | 2 | func:0 | | stl.h:189:42:189:47 | insert | 2 | class:0 |
| stl.h:190:17:190:23 | replace | 0 | size_type | | stl.h:190:17:190:23 | replace | 0 | const_iterator |
| stl.h:190:17:190:23 | replace | 1 | size_type | | stl.h:190:17:190:23 | replace | 1 | func:0 |
| stl.h:190:17:190:23 | replace | 2 | const basic_string & | | stl.h:190:17:190:23 | replace | 2 | func:0 |
| stl.h:191:17:191:23 | replace | 0 | size_type | | stl.h:191:17:191:23 | replace | 0 | size_type |
| stl.h:191:17:191:23 | replace | 1 | size_type | | stl.h:191:17:191:23 | replace | 1 | size_type |
| stl.h:191:17:191:23 | replace | 2 | size_type | | stl.h:191:17:191:23 | replace | 2 | const basic_string & |
| stl.h:191:17:191:23 | replace | 3 | class:0 | | stl.h:192:13:192:16 | copy | 0 | size_type |
| stl.h:192:13:192:16 | copy | 0 | class:0 * |
| stl.h:192:13:192:16 | copy | 1 | size_type | | stl.h:192:13:192:16 | copy | 1 | size_type |
| stl.h:192:13:192:16 | copy | 2 | size_type | | stl.h:192:13:192:16 | copy | 2 | size_type |
| stl.h:194:16:194:21 | substr | 0 | size_type | | stl.h:192:13:192:16 | copy | 3 | class:0 |
| stl.h:194:16:194:21 | substr | 1 | size_type | | stl.h:193:8:193:12 | clear | 0 | class:0 * |
| stl.h:195:8:195:11 | swap | 0 | basic_string & | | stl.h:193:8:193:12 | clear | 1 | size_type |
| stl.h:193:8:193:12 | clear | 2 | size_type |
| stl.h:195:8:195:11 | swap | 0 | size_type |
| stl.h:195:8:195:11 | swap | 1 | size_type |
| stl.h:198:94:198:102 | operator+ | 0 | const basic_string & | | stl.h:198:94:198:102 | operator+ | 0 | const basic_string & |
| stl.h:198:94:198:102 | operator+ | 1 | const basic_string & | | stl.h:198:94:198:102 | operator+ | 1 | const basic_string & |
| stl.h:199:94:199:102 | operator+ | 0 | const basic_string & | | stl.h:199:94:199:102 | operator+ | 0 | const basic_string & |

View File

@@ -1,7 +1,3 @@
| inconsistency2.cpp:3:3:3:5 | T:: | inconsistency2.cpp:3:3:3:6 | x | inconsistency2.cpp:2:20:2:20 | T |
| inconsistency2.cpp:3:3:3:11 | const s:: | inconsistency2.cpp:3:3:3:6 | x | file://:0:0:0:0 | const s |
| inconsistency.cpp:7:20:7:22 | S:: | inconsistency.cpp:7:20:7:23 | (int)... | inconsistency.cpp:4:8:4:8 | S |
| inconsistency.cpp:7:20:7:22 | S:: | inconsistency.cpp:7:20:7:23 | A | inconsistency.cpp:4:8:4:8 | S |
| name_qualifiers.cpp:29:7:29:8 | :: | name_qualifiers.cpp:29:7:29:9 | x | file://:0:0:0:0 | (global namespace) | | name_qualifiers.cpp:29:7:29:8 | :: | name_qualifiers.cpp:29:7:29:9 | x | file://:0:0:0:0 | (global namespace) |
| name_qualifiers.cpp:31:7:31:10 | N1:: | name_qualifiers.cpp:31:7:31:12 | nx | name_qualifiers.cpp:4:11:4:12 | N1 | | name_qualifiers.cpp:31:7:31:10 | N1:: | name_qualifiers.cpp:31:7:31:12 | nx | name_qualifiers.cpp:4:11:4:12 | N1 |
| name_qualifiers.cpp:34:7:34:8 | :: | name_qualifiers.cpp:34:9:34:12 | N1:: | file://:0:0:0:0 | (global namespace) | | name_qualifiers.cpp:34:7:34:8 | :: | name_qualifiers.cpp:34:9:34:12 | N1:: | file://:0:0:0:0 | (global namespace) |

View File

@@ -1,5 +1,7 @@
import cpp import cpp
from NameQualifier nq, Location l from NameQualifier nq, Location l
where l = nq.getQualifiedElement().getLocation() where
l = nq.getQualifiedElement().getLocation() and
l.getFile().getShortName() = "name_qualifiers"
select nq, nq.getQualifiedElement(), nq.getQualifyingElement() select nq, nq.getQualifiedElement(), nq.getQualifyingElement()

View File

@@ -1,8 +1,8 @@
// This file is present to test whether name-qualifying an enum constant leads to a database inconsistency. // This file is present to test whether name-qualifying an enum constant leads to a database inconsistency.
// As such, there is no QL part of the test.
struct S { enum E { A }; }; struct S { enum E { A }; };
static void f() { static int f() {
switch(0) { case S::A: break; } switch(0) { case S::A: break; }
} }

View File

@@ -1,12 +0,0 @@
namespace {
template <typename T> T f() {
T::x;
return {};
}
struct s {
static int x;
};
struct t {
s x = f<const s>();
};
}

View File

@@ -135,7 +135,7 @@ namespace Semmle.Autobuild.CSharp.Tests
if (!EnumerateFiles.TryGetValue(dir, out var str)) if (!EnumerateFiles.TryGetValue(dir, out var str))
throw new ArgumentException("Missing EnumerateFiles " + dir); throw new ArgumentException("Missing EnumerateFiles " + dir);
return str.Split("\n").Select(p => PathJoin(dir, p)); return str.Split("\n").Select(p => PathCombine(dir, p));
} }
public IDictionary<string, string> EnumerateDirectories { get; } = new Dictionary<string, string>(); public IDictionary<string, string> EnumerateDirectories { get; } = new Dictionary<string, string>();
@@ -147,7 +147,7 @@ namespace Semmle.Autobuild.CSharp.Tests
return string.IsNullOrEmpty(str) return string.IsNullOrEmpty(str)
? Enumerable.Empty<string>() ? Enumerable.Empty<string>()
: str.Split("\n").Select(p => PathJoin(dir, p)); : str.Split("\n").Select(p => PathCombine(dir, p));
} }
public bool IsWindows { get; set; } public bool IsWindows { get; set; }
@@ -170,7 +170,7 @@ namespace Semmle.Autobuild.CSharp.Tests
bool IBuildActions.IsMonoInstalled() => IsMonoInstalled; bool IBuildActions.IsMonoInstalled() => IsMonoInstalled;
public string PathJoin(params string[] parts) public string PathCombine(params string[] parts)
{ {
return string.Join(IsWindows ? '\\' : '/', parts.Where(p => !string.IsNullOrWhiteSpace(p))); return string.Join(IsWindows ? '\\' : '/', parts.Where(p => !string.IsNullOrWhiteSpace(p)));
} }

View File

@@ -109,7 +109,7 @@ namespace Semmle.Autobuild.CSharp
=> WithDotNet(builder, ensureDotNetAvailable: false, (_, env) => f(env)); => WithDotNet(builder, ensureDotNetAvailable: false, (_, env) => f(env));
private static string DotNetCommand(IBuildActions actions, string? dotNetPath) => private static string DotNetCommand(IBuildActions actions, string? dotNetPath) =>
dotNetPath is not null ? actions.PathJoin(dotNetPath, "dotnet") : "dotnet"; dotNetPath is not null ? actions.PathCombine(dotNetPath, "dotnet") : "dotnet";
private static CommandBuilder GetCleanCommand(IBuildActions actions, string? dotNetPath, IDictionary<string, string>? environment) private static CommandBuilder GetCleanCommand(IBuildActions actions, string? dotNetPath, IDictionary<string, string>? environment)
{ {

View File

@@ -158,7 +158,7 @@ namespace Semmle.Autobuild.Cpp.Tests
bool IBuildActions.IsMonoInstalled() => IsMonoInstalled; bool IBuildActions.IsMonoInstalled() => IsMonoInstalled;
string IBuildActions.PathJoin(params string[] parts) string IBuildActions.PathCombine(params string[] parts)
{ {
return string.Join(IsWindows ? '\\' : '/', parts.Where(p => !string.IsNullOrWhiteSpace(p))); return string.Join(IsWindows ? '\\' : '/', parts.Where(p => !string.IsNullOrWhiteSpace(p)));
} }

View File

@@ -108,7 +108,7 @@ namespace Semmle.Autobuild.Shared
/// </summary> /// </summary>
/// <param name="path">The relative path.</param> /// <param name="path">The relative path.</param>
/// <returns>True iff the path was found.</returns> /// <returns>True iff the path was found.</returns>
public bool HasRelativePath(string path) => HasPath(Actions.PathJoin(RootDirectory, path)); public bool HasRelativePath(string path) => HasPath(Actions.PathCombine(RootDirectory, path));
/// <summary> /// <summary>
/// List of project/solution files to build. /// List of project/solution files to build.

View File

@@ -32,7 +32,7 @@ namespace Semmle.Autobuild.Shared
yield break; yield break;
// Attempt to use vswhere to find installations of Visual Studio // Attempt to use vswhere to find installations of Visual Studio
var vswhere = actions.PathJoin(programFilesx86, "Microsoft Visual Studio", "Installer", "vswhere.exe"); var vswhere = actions.PathCombine(programFilesx86, "Microsoft Visual Studio", "Installer", "vswhere.exe");
if (actions.FileExists(vswhere)) if (actions.FileExists(vswhere))
{ {
@@ -51,14 +51,14 @@ namespace Semmle.Autobuild.Shared
if (majorVersion < 15) if (majorVersion < 15)
{ {
// Visual Studio 2015 and below // Visual Studio 2015 and below
yield return new VcVarsBatFile(actions.PathJoin(vsInstallation.InstallationPath, @"VC\vcvarsall.bat"), majorVersion); yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"VC\vcvarsall.bat"), majorVersion);
} }
else else
{ {
// Visual Studio 2017 and above // Visual Studio 2017 and above
yield return new VcVarsBatFile(actions.PathJoin(vsInstallation.InstallationPath, @"VC\Auxiliary\Build\vcvars32.bat"), majorVersion); yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"VC\Auxiliary\Build\vcvars32.bat"), majorVersion);
yield return new VcVarsBatFile(actions.PathJoin(vsInstallation.InstallationPath, @"VC\Auxiliary\Build\vcvars64.bat"), majorVersion); yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"VC\Auxiliary\Build\vcvars64.bat"), majorVersion);
yield return new VcVarsBatFile(actions.PathJoin(vsInstallation.InstallationPath, @"Common7\Tools\VsDevCmd.bat"), majorVersion); yield return new VcVarsBatFile(actions.PathCombine(vsInstallation.InstallationPath, @"Common7\Tools\VsDevCmd.bat"), majorVersion);
} }
} }
// else: Skip installation without a version // else: Skip installation without a version
@@ -68,10 +68,10 @@ namespace Semmle.Autobuild.Shared
} }
// vswhere not installed or didn't run correctly - return legacy Visual Studio versions // vswhere not installed or didn't run correctly - return legacy Visual Studio versions
yield return new VcVarsBatFile(actions.PathJoin(programFilesx86, @"Microsoft Visual Studio 14.0\VC\vcvarsall.bat"), 14); yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 14.0\VC\vcvarsall.bat"), 14);
yield return new VcVarsBatFile(actions.PathJoin(programFilesx86, @"Microsoft Visual Studio 12.0\VC\vcvarsall.bat"), 12); yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 12.0\VC\vcvarsall.bat"), 12);
yield return new VcVarsBatFile(actions.PathJoin(programFilesx86, @"Microsoft Visual Studio 11.0\VC\vcvarsall.bat"), 11); yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 11.0\VC\vcvarsall.bat"), 11);
yield return new VcVarsBatFile(actions.PathJoin(programFilesx86, @"Microsoft Visual Studio 10.0\VC\vcvarsall.bat"), 10); yield return new VcVarsBatFile(actions.PathCombine(programFilesx86, @"Microsoft Visual Studio 10.0\VC\vcvarsall.bat"), 10);
} }
/// <summary> /// <summary>

View File

@@ -60,7 +60,7 @@ namespace Semmle.Autobuild.Shared
// Use `nuget.exe` from source code repo, if present, otherwise first attempt with global // Use `nuget.exe` from source code repo, if present, otherwise first attempt with global
// `nuget` command, and if that fails, attempt to download `nuget.exe` from nuget.org // `nuget` command, and if that fails, attempt to download `nuget.exe` from nuget.org
var nuget = builder.GetFilename("nuget.exe").Select(t => t.Item1).FirstOrDefault() ?? "nuget"; var nuget = builder.GetFilename("nuget.exe").Select(t => t.Item1).FirstOrDefault() ?? "nuget";
var nugetDownloadPath = builder.Actions.PathJoin(FileUtils.GetTemporaryWorkingDirectory(builder.Actions.GetEnvironmentVariable, builder.Options.Language.UpperCaseName, out _), ".nuget", "nuget.exe"); var nugetDownloadPath = builder.Actions.PathCombine(FileUtils.GetTemporaryWorkingDirectory(builder.Actions.GetEnvironmentVariable, builder.Options.Language.UpperCaseName, out _), ".nuget", "nuget.exe");
var nugetDownloaded = false; var nugetDownloaded = false;
var ret = BuildScript.Success; var ret = BuildScript.Success;

View File

@@ -107,9 +107,8 @@ namespace Semmle.Autobuild.Shared
continue; continue;
} }
var includePath = builder.Actions.PathJoin(include.Value.Split('\\', StringSplitOptions.RemoveEmptyEntries)); var includePath = builder.Actions.PathCombine(include.Value.Split('\\', StringSplitOptions.RemoveEmptyEntries));
var path = Path.IsPathRooted(includePath) ? includePath : builder.Actions.PathJoin(DirectoryName, includePath); ret.Add(new Project<TAutobuildOptions>(builder, builder.Actions.PathCombine(DirectoryName, includePath)));
ret.Add(new Project<TAutobuildOptions>(builder, path));
} }
return ret; return ret;
}); });

View File

@@ -79,7 +79,7 @@ namespace Semmle.Autobuild.Shared
includedProjects = solution.ProjectsInOrder includedProjects = solution.ProjectsInOrder
.Where(p => p.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat) .Where(p => p.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat)
.Select(p => builder.Actions.PathJoin(DirectoryName, builder.Actions.PathJoin(p.RelativePath.Split('\\', StringSplitOptions.RemoveEmptyEntries)))) .Select(p => builder.Actions.PathCombine(DirectoryName, builder.Actions.PathCombine(p.RelativePath.Split('\\', StringSplitOptions.RemoveEmptyEntries))))
.Select(p => new Project<TAutobuildOptions>(builder, p)) .Select(p => new Project<TAutobuildOptions>(builder, p))
.ToArray(); .ToArray();
} }

View File

@@ -1,2 +0,0 @@
description: Restructure and rename types related to operations.
compatibility: full

View File

@@ -50,7 +50,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
return; return;
} }
var path = Path.Join(p, ParseFilePath(d)); var path = Path.Combine(p, ParseFilePath(d));
Paths.Add(path); Paths.Add(path);
Packages.Add(GetPackageName(p)); Packages.Add(GetPackageName(p));
} }

View File

@@ -75,7 +75,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
} }
} }
this.diagnosticsWriter = new DiagnosticsStream(Path.Join( this.diagnosticsWriter = new DiagnosticsStream(Path.Combine(
diagDirEnv ?? "", diagDirEnv ?? "",
$"dependency-manager-{DateTime.UtcNow:yyyyMMddHHmm}-{Environment.ProcessId}.jsonc")); $"dependency-manager-{DateTime.UtcNow:yyyyMMddHHmm}-{Environment.ProcessId}.jsonc"));
this.sourceDir = new DirectoryInfo(srcDir); this.sourceDir = new DirectoryInfo(srcDir);
@@ -327,7 +327,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
private void RemoveNugetPackageReference(string packagePrefix, ISet<AssemblyLookupLocation> dllLocations) private void RemoveNugetPackageReference(string packagePrefix, ISet<AssemblyLookupLocation> dllLocations)
{ {
var packageFolder = nugetPackageRestorer.PackageDirectory.DirInfo.FullName.ToLowerInvariant(); var packageFolder = nugetPackageRestorer.PackageDirectory.DirInfo.FullName.ToLowerInvariant();
var packagePathPrefix = Path.Join(packageFolder, packagePrefix.ToLowerInvariant()); var packagePathPrefix = Path.Combine(packageFolder, packagePrefix.ToLowerInvariant());
var toRemove = dllLocations.Where(s => s.Path.StartsWith(packagePathPrefix, StringComparison.InvariantCultureIgnoreCase)); var toRemove = dllLocations.Where(s => s.Path.StartsWith(packagePathPrefix, StringComparison.InvariantCultureIgnoreCase));
foreach (var path in toRemove) foreach (var path in toRemove)
{ {

View File

@@ -31,7 +31,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
} }
} }
private DotNet(ILogger logger, string? dotNetPath, TemporaryDirectory tempWorkingDirectory, DependabotProxy? dependabotProxy) : this(new DotNetCliInvoker(logger, Path.Join(dotNetPath ?? string.Empty, "dotnet"), dependabotProxy), logger, dotNetPath is null, tempWorkingDirectory) { } private DotNet(ILogger logger, string? dotNetPath, TemporaryDirectory tempWorkingDirectory, DependabotProxy? dependabotProxy) : this(new DotNetCliInvoker(logger, Path.Combine(dotNetPath ?? string.Empty, "dotnet"), dependabotProxy), logger, dotNetPath is null, tempWorkingDirectory) { }
internal static IDotNet Make(IDotNetCliInvoker dotnetCliInvoker, ILogger logger, bool runDotnetInfo) => new DotNet(dotnetCliInvoker, logger, runDotnetInfo); internal static IDotNet Make(IDotNetCliInvoker dotnetCliInvoker, ILogger logger, bool runDotnetInfo) => new DotNet(dotnetCliInvoker, logger, runDotnetInfo);
@@ -73,7 +73,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
var path = ".empty"; var path = ".empty";
if (tempWorkingDirectory != null) if (tempWorkingDirectory != null)
{ {
path = Path.Join(tempWorkingDirectory.ToString(), "emptyFakeDotnetRoot"); path = Path.Combine(tempWorkingDirectory.ToString(), "emptyFakeDotnetRoot");
Directory.CreateDirectory(path); Directory.CreateDirectory(path);
} }
@@ -303,7 +303,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
} }
else else
{ {
var dotnetInstallPath = actions.PathJoin(tempWorkingDirectory, ".dotnet", "dotnet-install.sh"); var dotnetInstallPath = actions.PathCombine(tempWorkingDirectory, ".dotnet", "dotnet-install.sh");
var downloadDotNetInstallSh = BuildScript.DownloadFile( var downloadDotNetInstallSh = BuildScript.DownloadFile(
"https://dot.net/v1/dotnet-install.sh", "https://dot.net/v1/dotnet-install.sh",
dotnetInstallPath, dotnetInstallPath,
@@ -339,7 +339,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
}; };
} }
var dotnetInfo = InfoScript(actions, actions.PathJoin(path, "dotnet"), MinimalEnvironment.ToDictionary(), logger); var dotnetInfo = InfoScript(actions, actions.PathCombine(path, "dotnet"), MinimalEnvironment.ToDictionary(), logger);
Func<string, BuildScript> getInstallAndVerify = version => Func<string, BuildScript> getInstallAndVerify = version =>
// run `dotnet --info` after install, to check that it executes successfully // run `dotnet --info` after install, to check that it executes successfully
@@ -384,7 +384,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
/// </summary> /// </summary>
public static BuildScript WithDotNet(IBuildActions actions, ILogger logger, IEnumerable<string> files, string tempWorkingDirectory, bool shouldCleanUp, bool ensureDotNetAvailable, string? version, Func<string?, BuildScript> f) public static BuildScript WithDotNet(IBuildActions actions, ILogger logger, IEnumerable<string> files, string tempWorkingDirectory, bool shouldCleanUp, bool ensureDotNetAvailable, string? version, Func<string?, BuildScript> f)
{ {
var installDir = actions.PathJoin(tempWorkingDirectory, ".dotnet"); var installDir = actions.PathCombine(tempWorkingDirectory, ".dotnet");
var installScript = DownloadDotNet(actions, logger, files, tempWorkingDirectory, shouldCleanUp, installDir, version, ensureDotNetAvailable); var installScript = DownloadDotNet(actions, logger, files, tempWorkingDirectory, shouldCleanUp, installDir, version, ensureDotNetAvailable);
return BuildScript.Bind(installScript, installed => return BuildScript.Bind(installScript, installed =>
{ {

View File

@@ -12,7 +12,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
private string FullVersion => private string FullVersion =>
version.ToString(); version.ToString();
public string FullPath => Path.Join(dir, FullVersion); public string FullPath => Path.Combine(dir, FullVersion);
/** /**
* The full path to the reference assemblies for this runtime. * The full path to the reference assemblies for this runtime.
@@ -33,7 +33,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
{ {
directories[^2] = "packs"; directories[^2] = "packs";
directories[^1] = $"{directories[^1]}.Ref"; directories[^1] = $"{directories[^1]}.Ref";
return Path.Join(string.Join(Path.DirectorySeparatorChar, directories), FullVersion, "ref"); return Path.Combine(string.Join(Path.DirectorySeparatorChar, directories), FullVersion, "ref");
} }
return null; return null;
} }

View File

@@ -0,0 +1,304 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using Semmle.Util;
namespace Semmle.Extraction.CSharp.DependencyFetching
{
/// <summary>
/// Manage the downloading of NuGet packages with nuget.exe.
/// Locates packages in a source tree and downloads all of the
/// referenced assemblies to a temp folder.
/// </summary>
internal class NugetExeWrapper : IDisposable
{
private readonly string? nugetExe;
private readonly Semmle.Util.Logging.ILogger logger;
public int PackageCount => fileProvider.PackagesConfigs.Count;
private readonly string? backupNugetConfig;
private readonly string? nugetConfigPath;
private readonly FileProvider fileProvider;
/// <summary>
/// The packages directory.
/// This will be in the user-specified or computed Temp location
/// so as to not trample the source tree.
/// </summary>
private readonly DependencyDirectory packageDirectory;
/// <summary>
/// Create the package manager for a specified source tree.
/// </summary>
public NugetExeWrapper(FileProvider fileProvider, DependencyDirectory packageDirectory, Semmle.Util.Logging.ILogger logger, Func<bool> useDefaultFeed)
{
this.fileProvider = fileProvider;
this.packageDirectory = packageDirectory;
this.logger = logger;
if (fileProvider.PackagesConfigs.Count > 0)
{
logger.LogInfo($"Found packages.config files, trying to use nuget.exe for package restore");
nugetExe = ResolveNugetExe();
if (HasNoPackageSource() && useDefaultFeed())
{
// We only modify or add a top level nuget.config file
nugetConfigPath = Path.Combine(fileProvider.SourceDir.FullName, "nuget.config");
try
{
if (File.Exists(nugetConfigPath))
{
var tempFolderPath = FileUtils.GetTemporaryWorkingDirectory(out _);
do
{
backupNugetConfig = Path.Combine(tempFolderPath, Path.GetRandomFileName());
}
while (File.Exists(backupNugetConfig));
File.Copy(nugetConfigPath, backupNugetConfig, true);
}
else
{
File.WriteAllText(nugetConfigPath,
"""
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
</packageSources>
</configuration>
""");
}
AddDefaultPackageSource(nugetConfigPath);
}
catch (Exception e)
{
logger.LogError($"Failed to add default package source to {nugetConfigPath}: {e}");
}
}
}
}
/// <summary>
/// Tries to find the location of `nuget.exe`. It looks for
/// - the environment variable specifying a location,
/// - files in the repository,
/// - tries to resolve nuget from the PATH, or
/// - downloads it if it is not found.
/// </summary>
private string ResolveNugetExe()
{
var envVarPath = Environment.GetEnvironmentVariable(EnvironmentVariableNames.NugetExePath);
if (!string.IsNullOrEmpty(envVarPath))
{
logger.LogInfo($"Using nuget.exe from environment variable: '{envVarPath}'");
return envVarPath;
}
try
{
return DownloadNugetExe(fileProvider.SourceDir.FullName);
}
catch (Exception exc)
{
logger.LogInfo($"Download of nuget.exe failed: {exc.Message}");
}
var nugetExesInRepo = fileProvider.NugetExes;
if (nugetExesInRepo.Count > 1)
{
logger.LogInfo($"Found multiple nuget.exe files in the repository: {string.Join(", ", nugetExesInRepo.OrderBy(s => s))}");
}
if (nugetExesInRepo.Count > 0)
{
var path = nugetExesInRepo.First();
logger.LogInfo($"Using nuget.exe from path '{path}'");
return path;
}
var executableName = Win32.IsWindows() ? "nuget.exe" : "nuget";
var nugetPath = FileUtils.FindProgramOnPath(executableName);
if (nugetPath is not null)
{
nugetPath = Path.Combine(nugetPath, executableName);
logger.LogInfo($"Using nuget.exe from PATH: {nugetPath}");
return nugetPath;
}
throw new Exception("Could not find or download nuget.exe.");
}
private string DownloadNugetExe(string sourceDir)
{
var directory = Path.Combine(sourceDir, ".nuget");
var nuget = Path.Combine(directory, "nuget.exe");
// Nuget.exe already exists in the .nuget directory.
if (File.Exists(nuget))
{
logger.LogInfo($"Found nuget.exe at {nuget}");
return nuget;
}
Directory.CreateDirectory(directory);
logger.LogInfo("Attempting to download nuget.exe");
FileUtils.DownloadFile(FileUtils.NugetExeUrl, nuget, logger);
logger.LogInfo($"Downloaded nuget.exe to {nuget}");
return nuget;
}
private bool RunWithMono => !Win32.IsWindows() && !string.IsNullOrEmpty(Path.GetExtension(nugetExe));
/// <summary>
/// Restore all packages in the specified packages.config file.
/// </summary>
/// <param name="packagesConfig">The packages.config file.</param>
private bool TryRestoreNugetPackage(string packagesConfig)
{
logger.LogInfo($"Restoring file \"{packagesConfig}\"...");
/* Use nuget.exe to install a package.
* Note that there is a clutch of NuGet assemblies which could be used to
* invoke this directly, which would arguably be nicer. However they are
* really unwieldy and this solution works for now.
*/
string exe, args;
if (RunWithMono)
{
exe = "mono";
args = $"\"{nugetExe}\" install -OutputDirectory \"{packageDirectory}\" \"{packagesConfig}\"";
}
else
{
exe = nugetExe!;
args = $"install -OutputDirectory \"{packageDirectory}\" \"{packagesConfig}\"";
}
var pi = new ProcessStartInfo(exe, args)
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
var threadId = Environment.CurrentManagedThreadId;
void onOut(string s) => logger.LogDebug(s, threadId);
void onError(string s) => logger.LogError(s, threadId);
var exitCode = pi.ReadOutput(out _, onOut, onError);
if (exitCode != 0)
{
logger.LogError($"Command {pi.FileName} {pi.Arguments} failed with exit code {exitCode}");
return false;
}
else
{
logger.LogInfo($"Restored file \"{packagesConfig}\"");
return true;
}
}
/// <summary>
/// Download the packages to the temp folder.
/// </summary>
public int InstallPackages()
{
return fileProvider.PackagesConfigs.Count(TryRestoreNugetPackage);
}
private bool HasNoPackageSource()
{
if (Win32.IsWindows())
{
return false;
}
try
{
logger.LogInfo("Checking if default package source is available...");
RunMonoNugetCommand("sources list -ForceEnglishOutput", out var stdout);
if (stdout.All(line => line != "No sources found."))
{
return false;
}
return true;
}
catch (Exception e)
{
logger.LogWarning($"Failed to check if default package source is added: {e}");
return false;
}
}
private void RunMonoNugetCommand(string command, out IList<string> stdout)
{
string exe, args;
if (RunWithMono)
{
exe = "mono";
args = $"\"{nugetExe}\" {command}";
}
else
{
exe = nugetExe!;
args = command;
}
var pi = new ProcessStartInfo(exe, args)
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
var threadId = Environment.CurrentManagedThreadId;
void onOut(string s) => logger.LogDebug(s, threadId);
void onError(string s) => logger.LogError(s, threadId);
pi.ReadOutput(out stdout, onOut, onError);
}
private void AddDefaultPackageSource(string nugetConfig)
{
logger.LogInfo("Adding default package source...");
RunMonoNugetCommand($"sources add -Name DefaultNugetOrg -Source {NugetPackageRestorer.PublicNugetOrgFeed} -ConfigFile \"{nugetConfig}\"", out _);
}
public void Dispose()
{
if (nugetConfigPath is null)
{
return;
}
try
{
if (backupNugetConfig is null)
{
logger.LogInfo("Removing nuget.config file");
File.Delete(nugetConfigPath);
return;
}
logger.LogInfo("Reverting nuget.config file content");
// The content of the original nuget.config file is reverted without changing the file's attributes or casing:
using (var backup = File.OpenRead(backupNugetConfig))
using (var current = File.OpenWrite(nugetConfigPath))
{
current.SetLength(0); // Truncate file
backup.CopyTo(current); // Restore original content
}
logger.LogInfo("Deleting backup nuget.config file");
File.Delete(backupNugetConfig);
}
catch (Exception exc)
{
logger.LogError($"Failed to restore original nuget.config file: {exc}");
}
}
}
}

View File

@@ -161,13 +161,13 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
reachableFeeds.UnionWith(reachableInheritedFeeds); reachableFeeds.UnionWith(reachableInheritedFeeds);
} }
using (var packagesConfigRestore = PackagesConfigRestoreFactory.Create(fileProvider, legacyPackageDirectory, logger, IsDefaultFeedReachable)) using (var nuget = new NugetExeWrapper(fileProvider, legacyPackageDirectory, logger, IsDefaultFeedReachable))
{ {
var count = packagesConfigRestore.InstallPackages(); var count = nuget.InstallPackages();
if (packagesConfigRestore.PackageCount > 0) if (nuget.PackageCount > 0)
{ {
compilationInfoContainer.CompilationInfos.Add(("packages.config files", packagesConfigRestore.PackageCount.ToString())); compilationInfoContainer.CompilationInfos.Add(("packages.config files", nuget.PackageCount.ToString()));
compilationInfoContainer.CompilationInfos.Add(("Successfully restored packages.config files", count.ToString())); compilationInfoContainer.CompilationInfos.Add(("Successfully restored packages.config files", count.ToString()));
} }
} }
@@ -209,7 +209,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
var paths = dependencies var paths = dependencies
.Paths .Paths
.Select(d => Path.Join(PackageDirectory.DirInfo.FullName, d)) .Select(d => Path.Combine(PackageDirectory.DirInfo.FullName, d))
.ToList(); .ToList();
assemblyLookupLocations.UnionWith(paths.Select(p => new AssemblyLookupLocation(p))); assemblyLookupLocations.UnionWith(paths.Select(p => new AssemblyLookupLocation(p)));
@@ -527,7 +527,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
var sb = new StringBuilder(); var sb = new StringBuilder();
fallbackNugetFeeds.ForEach((feed, index) => sb.AppendLine($"<add key=\"feed{index}\" value=\"{feed}\" />")); fallbackNugetFeeds.ForEach((feed, index) => sb.AppendLine($"<add key=\"feed{index}\" value=\"{feed}\" />"));
var nugetConfigPath = Path.Join(folderPath, "nuget.config"); var nugetConfigPath = Path.Combine(folderPath, "nuget.config");
logger.LogInfo($"Creating fallback nuget.config file {nugetConfigPath}."); logger.LogInfo($"Creating fallback nuget.config file {nugetConfigPath}.");
File.WriteAllText(nugetConfigPath, File.WriteAllText(nugetConfigPath,
$""" $"""
@@ -1052,7 +1052,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
/// </summary> /// </summary>
private static string ComputeTempDirectoryPath(string subfolderName) private static string ComputeTempDirectoryPath(string subfolderName)
{ {
return Path.Join(FileUtils.GetTemporaryWorkingDirectory(out _), subfolderName); return Path.Combine(FileUtils.GetTemporaryWorkingDirectory(out _), subfolderName);
} }
/// <summary> /// <summary>
@@ -1060,7 +1060,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
/// </summary> /// </summary>
private static string ComputeTempDirectoryPath(string srcDir, string subfolderName) private static string ComputeTempDirectoryPath(string srcDir, string subfolderName)
{ {
return Path.Join(FileUtils.GetTemporaryWorkingDirectory(out _), FileUtils.ComputeHash(srcDir), subfolderName); return Path.Combine(FileUtils.GetTemporaryWorkingDirectory(out _), FileUtils.ComputeHash(srcDir), subfolderName);
} }
} }
} }

View File

@@ -1,368 +0,0 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using Semmle.Util;
namespace Semmle.Extraction.CSharp.DependencyFetching
{
internal interface IPackagesConfigRestore : IDisposable
{
/// <summary>
/// The number of packages.config files found in the source tree.
/// </summary>
int PackageCount { get; }
/// <summary>
/// Download the packages to the temp folder.
/// </summary>
int InstallPackages();
}
/// <summary>
/// Factory for creating a package manager to restore NuGet packages referenced in packages.config files.
/// If the environment doesn't support using nuget.exe to restore packages from packages.config files, a no-op implementation is returned.
/// It is worth noting that for macOS and Linux, nuget.exe is used with mono. However, mono is being deprecated and the last GitHub images
/// to contain mono are:
/// - Ubuntu 22.04
/// - macOS 14
///
/// If the packages from the packages.config files are not restored with the packages.config restore functionality below, there is a subsequent
/// step that still may succeed in restoring the packages without the help of nuget.exe (by attempting to restore using dotnet).
/// </summary>
internal class PackagesConfigRestoreFactory
{
public static IPackagesConfigRestore Create(FileProvider fileProvider, DependencyDirectory packageDirectory, Semmle.Util.Logging.ILogger logger, Func<bool> useDefaultFeed)
{
if (SystemBuildActions.Instance.IsWindows() || SystemBuildActions.Instance.IsMonoInstalled())
{
return new NugetExeWrapper(fileProvider, packageDirectory, logger, useDefaultFeed);
}
return new NoOpPackagesConfig(fileProvider, logger);
}
/// <summary>
/// Manage the downloading of NuGet packages with nuget.exe.
/// Locates packages in a source tree and downloads all of the
/// referenced assemblies to a temp folder.
/// </summary>
private class NugetExeWrapper : IPackagesConfigRestore
{
private readonly string? nugetExe;
private readonly Semmle.Util.Logging.ILogger logger;
public int PackageCount => fileProvider.PackagesConfigs.Count;
private readonly string? backupNugetConfig;
private readonly string? nugetConfigPath;
private readonly FileProvider fileProvider;
/// <summary>
/// The packages directory.
/// This will be in the user-specified or computed Temp location
/// so as to not trample the source tree.
/// </summary>
private readonly DependencyDirectory packageDirectory;
private bool IsWindows => SystemBuildActions.Instance.IsWindows();
/// <summary>
/// Create the package manager for a specified source tree.
/// </summary>
public NugetExeWrapper(FileProvider fileProvider, DependencyDirectory packageDirectory, Semmle.Util.Logging.ILogger logger, Func<bool> useDefaultFeed)
{
this.fileProvider = fileProvider;
this.packageDirectory = packageDirectory;
this.logger = logger;
if (fileProvider.PackagesConfigs.Count > 0)
{
logger.LogInfo($"Found packages.config files, trying to use nuget.exe for package restore");
nugetExe = ResolveNugetExe();
if (!HasPackageSource() && useDefaultFeed())
{
// We only modify or add a top level nuget.config file
nugetConfigPath = Path.Join(fileProvider.SourceDir.FullName, "nuget.config");
try
{
if (File.Exists(nugetConfigPath))
{
var tempFolderPath = FileUtils.GetTemporaryWorkingDirectory(out _);
do
{
backupNugetConfig = Path.Join(tempFolderPath, Path.GetRandomFileName());
}
while (File.Exists(backupNugetConfig));
File.Copy(nugetConfigPath, backupNugetConfig, true);
}
else
{
File.WriteAllText(nugetConfigPath,
"""
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
</packageSources>
</configuration>
""");
}
AddDefaultPackageSource(nugetConfigPath);
}
catch (Exception e)
{
logger.LogError($"Failed to add default package source to {nugetConfigPath}: {e}");
}
}
}
}
/// <summary>
/// Tries to find the location of `nuget.exe`. It looks for
/// - the environment variable specifying a location,
/// - files in the repository,
/// - tries to resolve nuget from the PATH, or
/// - downloads it if it is not found.
/// </summary>
private string ResolveNugetExe()
{
var envVarPath = Environment.GetEnvironmentVariable(EnvironmentVariableNames.NugetExePath);
if (!string.IsNullOrEmpty(envVarPath))
{
logger.LogInfo($"Using nuget.exe from environment variable: '{envVarPath}'");
return envVarPath;
}
try
{
return DownloadNugetExe(fileProvider.SourceDir.FullName);
}
catch (Exception exc)
{
logger.LogInfo($"Download of nuget.exe failed: {exc.Message}");
}
var nugetExesInRepo = fileProvider.NugetExes;
if (nugetExesInRepo.Count > 1)
{
logger.LogInfo($"Found multiple nuget.exe files in the repository: {string.Join(", ", nugetExesInRepo.OrderBy(s => s))}");
}
if (nugetExesInRepo.Count > 0)
{
var path = nugetExesInRepo.First();
logger.LogInfo($"Using nuget.exe from path '{path}'");
return path;
}
var executableName = IsWindows ? "nuget.exe" : "nuget";
var nugetPath = FileUtils.FindProgramOnPath(executableName);
if (nugetPath is not null)
{
nugetPath = Path.Join(nugetPath, executableName);
logger.LogInfo($"Using nuget.exe from PATH: {nugetPath}");
return nugetPath;
}
throw new Exception("Could not find or download nuget.exe.");
}
private string DownloadNugetExe(string sourceDir)
{
var directory = Path.Join(sourceDir, ".nuget");
var nuget = Path.Join(directory, "nuget.exe");
// Nuget.exe already exists in the .nuget directory.
if (File.Exists(nuget))
{
logger.LogInfo($"Found nuget.exe at {nuget}");
return nuget;
}
Directory.CreateDirectory(directory);
logger.LogInfo("Attempting to download nuget.exe");
FileUtils.DownloadFile(FileUtils.NugetExeUrl, nuget, logger);
logger.LogInfo($"Downloaded nuget.exe to {nuget}");
return nuget;
}
private bool RunWithMono => !IsWindows && !string.IsNullOrEmpty(Path.GetExtension(nugetExe));
/// <summary>
/// Restore all packages in the specified packages.config file.
/// </summary>
/// <param name="packagesConfig">The packages.config file.</param>
private bool TryRestoreNugetPackage(string packagesConfig)
{
logger.LogInfo($"Restoring file \"{packagesConfig}\"...");
/* Use nuget.exe to install a package.
* Note that there is a clutch of NuGet assemblies which could be used to
* invoke this directly, which would arguably be nicer. However they are
* really unwieldy and this solution works for now.
*/
string exe, args;
if (RunWithMono)
{
exe = "mono";
args = $"\"{nugetExe}\" install -OutputDirectory \"{packageDirectory}\" \"{packagesConfig}\"";
}
else
{
exe = nugetExe!;
args = $"install -OutputDirectory \"{packageDirectory}\" \"{packagesConfig}\"";
}
var pi = new ProcessStartInfo(exe, args)
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
var threadId = Environment.CurrentManagedThreadId;
void onOut(string s) => logger.LogDebug(s, threadId);
void onError(string s) => logger.LogError(s, threadId);
var exitCode = pi.ReadOutput(out _, onOut, onError);
if (exitCode != 0)
{
logger.LogError($"Command {pi.FileName} {pi.Arguments} failed with exit code {exitCode}");
return false;
}
else
{
logger.LogInfo($"Restored file \"{packagesConfig}\"");
return true;
}
}
/// <summary>
/// Download the packages to the temp folder.
/// </summary>
public int InstallPackages()
{
return fileProvider.PackagesConfigs.Count(TryRestoreNugetPackage);
}
private bool HasPackageSource()
{
if (IsWindows)
{
return true;
}
try
{
logger.LogInfo("Checking if default package source is available...");
RunMonoNugetCommand("sources list -ForceEnglishOutput", out var stdout);
if (stdout.All(line => line != "No sources found."))
{
return true;
}
return false;
}
catch (Exception e)
{
logger.LogWarning($"Failed to check if default package source is added: {e}");
return true;
}
}
private void RunMonoNugetCommand(string command, out IList<string> stdout)
{
string exe, args;
if (RunWithMono)
{
exe = "mono";
args = $"\"{nugetExe}\" {command}";
}
else
{
exe = nugetExe!;
args = command;
}
var pi = new ProcessStartInfo(exe, args)
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
var threadId = Environment.CurrentManagedThreadId;
void onOut(string s) => logger.LogDebug(s, threadId);
void onError(string s) => logger.LogError(s, threadId);
pi.ReadOutput(out stdout, onOut, onError);
}
private void AddDefaultPackageSource(string nugetConfig)
{
logger.LogInfo("Adding default package source...");
RunMonoNugetCommand($"sources add -Name DefaultNugetOrg -Source {NugetPackageRestorer.PublicNugetOrgFeed} -ConfigFile \"{nugetConfig}\"", out _);
}
public void Dispose()
{
if (nugetConfigPath is null)
{
return;
}
try
{
if (backupNugetConfig is null)
{
logger.LogInfo("Removing nuget.config file");
File.Delete(nugetConfigPath);
return;
}
logger.LogInfo("Reverting nuget.config file content");
// The content of the original nuget.config file is reverted without changing the file's attributes or casing:
using (var backup = File.OpenRead(backupNugetConfig))
using (var current = File.OpenWrite(nugetConfigPath))
{
current.SetLength(0); // Truncate file
backup.CopyTo(current); // Restore original content
}
logger.LogInfo("Deleting backup nuget.config file");
File.Delete(backupNugetConfig);
}
catch (Exception exc)
{
logger.LogError($"Failed to restore original nuget.config file: {exc}");
}
}
}
private class NoOpPackagesConfig : IPackagesConfigRestore
{
private readonly Semmle.Util.Logging.ILogger logger;
private readonly FileProvider fileProvider;
public NoOpPackagesConfig(FileProvider fileProvider, Semmle.Util.Logging.ILogger logger)
{
this.fileProvider = fileProvider;
this.logger = logger;
}
public int PackageCount => fileProvider.PackagesConfigs.Count;
public int InstallPackages()
{
if (PackageCount > 0)
{
logger.LogInfo("Found packages.config files, but nuget.exe cannot be used to restore packages on this platform. Skipping restore of packages.config files.");
}
return 0;
}
public void Dispose() { }
}
}
}

View File

@@ -79,7 +79,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
var monoPath = FileUtils.FindProgramOnPath(Win32.IsWindows() ? "mono.exe" : "mono"); var monoPath = FileUtils.FindProgramOnPath(Win32.IsWindows() ? "mono.exe" : "mono");
string[] monoDirs = monoPath is not null string[] monoDirs = monoPath is not null
? [Path.GetFullPath(Path.Join(monoPath, "..", "lib", "mono")), monoPath] ? [Path.GetFullPath(Path.Combine(monoPath, "..", "lib", "mono")), monoPath]
: ["/usr/lib/mono", "/usr/local/mono", "/usr/local/bin/mono", @"C:\Program Files\Mono\lib\mono"]; : ["/usr/lib/mono", "/usr/local/mono", "/usr/local/bin/mono", @"C:\Program Files\Mono\lib\mono"];
var monoDir = monoDirs.FirstOrDefault(Directory.Exists); var monoDir = monoDirs.FirstOrDefault(Directory.Exists);

View File

@@ -63,7 +63,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
return null; return null;
} }
var path = Path.Join(version.FullPath, "Roslyn", "bincore", "csc.dll"); var path = Path.Combine(version.FullPath, "Roslyn", "bincore", "csc.dll");
logger.LogDebug($"Source generator CSC: '{path}'"); logger.LogDebug($"Source generator CSC: '{path}'");
if (!File.Exists(path)) if (!File.Exists(path))
{ {

View File

@@ -41,10 +41,10 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
.Replace('\\', '/'); // Ensure we're generating the same hash regardless of the OS .Replace('\\', '/'); // Ensure we're generating the same hash regardless of the OS
var name = FileUtils.ComputeHash($"{relativePathToCsProj}\n{this.GetType().Name}"); var name = FileUtils.ComputeHash($"{relativePathToCsProj}\n{this.GetType().Name}");
using var tempDir = new TemporaryDirectory(Path.Join(FileUtils.GetTemporaryWorkingDirectory(out _), "source-generator"), "source generator temporary", logger); using var tempDir = new TemporaryDirectory(Path.Join(FileUtils.GetTemporaryWorkingDirectory(out _), "source-generator"), "source generator temporary", logger);
var analyzerConfigPath = Path.Join(tempDir.DirInfo.FullName, $"{name}.txt"); var analyzerConfigPath = Path.Combine(tempDir.DirInfo.FullName, $"{name}.txt");
var dllPath = Path.Join(tempDir.DirInfo.FullName, $"{name}.dll"); var dllPath = Path.Combine(tempDir.DirInfo.FullName, $"{name}.dll");
var cscArgsPath = Path.Join(tempDir.DirInfo.FullName, $"{name}.rsp"); var cscArgsPath = Path.Combine(tempDir.DirInfo.FullName, $"{name}.rsp");
var outputFolder = Path.Join(targetDir, name); var outputFolder = Path.Combine(targetDir, name);
Directory.CreateDirectory(outputFolder); Directory.CreateDirectory(outputFolder);
logger.LogInfo("Producing analyzer config content."); logger.LogInfo("Producing analyzer config content.");
GenerateAnalyzerConfig(additionalFiles, csprojFile, analyzerConfigPath); GenerateAnalyzerConfig(additionalFiles, csprojFile, analyzerConfigPath);

View File

@@ -21,7 +21,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
throw new Exception("No SDK path available."); throw new Exception("No SDK path available.");
} }
SourceGeneratorFolder = Path.Join(sdkPath, "Sdks", "Microsoft.NET.Sdk.Razor", "source-generators"); SourceGeneratorFolder = Path.Combine(sdkPath, "Sdks", "Microsoft.NET.Sdk.Razor", "source-generators");
this.logger.LogInfo($"Razor source generator folder: {SourceGeneratorFolder}"); this.logger.LogInfo($"Razor source generator folder: {SourceGeneratorFolder}");
if (!Directory.Exists(SourceGeneratorFolder)) if (!Directory.Exists(SourceGeneratorFolder))
{ {

View File

@@ -50,7 +50,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
if (usings.Count > 0) if (usings.Count > 0)
{ {
var tempDir = GetTemporaryWorkingDirectory("implicitUsings"); var tempDir = GetTemporaryWorkingDirectory("implicitUsings");
var path = Path.Join(tempDir, "GlobalUsings.g.cs"); var path = Path.Combine(tempDir, "GlobalUsings.g.cs");
using (var writer = new StreamWriter(path)) using (var writer = new StreamWriter(path))
{ {
writer.WriteLine("// <auto-generated/>"); writer.WriteLine("// <auto-generated/>");

View File

@@ -32,7 +32,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
var nugetFolder = nugetPackageRestorer.TryRestore("Microsoft.CodeAnalysis.ResxSourceGenerator"); var nugetFolder = nugetPackageRestorer.TryRestore("Microsoft.CodeAnalysis.ResxSourceGenerator");
if (nugetFolder is not null) if (nugetFolder is not null)
{ {
sourceGeneratorFolder = System.IO.Path.Join(nugetFolder, "analyzers", "dotnet", "cs"); sourceGeneratorFolder = System.IO.Path.Combine(nugetFolder, "analyzers", "dotnet", "cs");
} }
} }
catch (Exception e) catch (Exception e)

View File

@@ -35,7 +35,7 @@ namespace Semmle.Extraction.CSharp.DependencyFetching
/// </summary> /// </summary>
protected string GetTemporaryWorkingDirectory(string subfolder) protected string GetTemporaryWorkingDirectory(string subfolder)
{ {
var temp = Path.Join(tempWorkingDirectory.ToString(), subfolder); var temp = Path.Combine(tempWorkingDirectory.ToString(), subfolder);
Directory.CreateDirectory(temp); Directory.CreateDirectory(temp);
return temp; return temp;

View File

@@ -1,6 +1,5 @@
using System.IO; using System.IO;
using Microsoft.CodeAnalysis; using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax; using Microsoft.CodeAnalysis.CSharp.Syntax;
using Semmle.Extraction.Kinds; using Semmle.Extraction.Kinds;
@@ -9,7 +8,7 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
internal abstract class ElementAccess : Expression<ExpressionSyntax> internal abstract class ElementAccess : Expression<ExpressionSyntax>
{ {
protected ElementAccess(ExpressionNodeInfo info, ExpressionSyntax qualifier, BracketedArgumentListSyntax argumentList) protected ElementAccess(ExpressionNodeInfo info, ExpressionSyntax qualifier, BracketedArgumentListSyntax argumentList)
: base(info.SetKind(GetKind(info.Context, info.Node, qualifier))) : base(info.SetKind(GetKind(info.Context, qualifier)))
{ {
this.qualifier = qualifier; this.qualifier = qualifier;
this.argumentList = argumentList; this.argumentList = argumentList;
@@ -18,125 +17,6 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
private readonly ExpressionSyntax qualifier; private readonly ExpressionSyntax qualifier;
private readonly BracketedArgumentListSyntax argumentList; private readonly BracketedArgumentListSyntax argumentList;
private ISymbol? GetTargetSymbol()
{
return Context.GetSymbolInfo(base.Syntax).Symbol;
}
private static void SetExprArgument(TextWriter trapFile, Expression left, Expression right)
{
trapFile.expr_argument(left, 0);
trapFile.expr_argument(right, 0);
}
private Expression MakeZeroFromEndExpression(IExpressionParentEntity parent, int child)
{
var info = new ExpressionInfo(
Context,
AnnotatedTypeSymbol.CreateNotAnnotated(Context.Compilation.GetSpecialType(SpecialType.System_Int32)),
Location,
ExprKind.INDEX,
parent,
child,
isCompilerGenerated: true,
null);
var index = new Expression(info);
MakeZeroLiteral(index, 0);
return index;
}
private Expression MakeZeroLiteral(IExpressionParentEntity parent, int child)
{
return Literal.CreateGenerated(Context, parent, child, Context.Compilation.GetSpecialType(SpecialType.System_Int32), 0, Location);
}
/// <summary>
/// It is assumed that either the input is
/// 1. A normal expression that can be used as endpoint (e.g a constant like "3").
/// 2. An index expression indicating that we should read from the end (e.g "^1").
/// </summary>
/// <param name="syntax">The syntax node representing the range endpoint.</param>
/// <param name="parent">The parent expression entity.</param>
/// <param name="child">The child index within the parent.</param>
/// <returns>An expression representing the endpoint of a range to be used in conjunction with a slice operation.</returns>
private Expression MakeFromRangeEndpoint(ExpressionSyntax syntax, IExpressionParentEntity parent, int child)
{
var info = new ExpressionNodeInfo(Context, syntax, parent, child);
return syntax.Kind() == SyntaxKind.IndexExpression
? PrefixUnary.Create(info.SetKind(ExprKind.INDEX))
: Factory.Create(info);
}
/// <summary>
/// Determines whether the given method is a slice method, which is defined as a method with
/// the name "Slice" or "Substring" and two parameters.
/// </summary>
/// <param name="method">The method symbol to check.</param>
/// <returns>True if the method is a slice method; false otherwise.</returns>
private bool IsSlice(IMethodSymbol method, out RangeExpressionSyntax? range)
{
range = null;
if (argumentList.Arguments.Count == 1)
{
range = argumentList.Arguments[0].Expression as RangeExpressionSyntax;
}
return (method.Name == "Slice" || method.Name == "Substring")
&& method.Parameters.Length == 2;
}
/// <summary>
/// Populates a slice method call based on the given range.
/// Roslyn translates indexer accesses with range expressions in the following way.
/// 1. s[a..b] -> s.Slice(a, b - a)
/// 2. s[..b] -> s.Slice(0, b)
/// 3. s[a..] -> s.Slice(a, s.Length - a)
/// 4. s[..] -> s.Slice(0, s.Length)
/// However, it is possible that both the qualifier or the index endpoints may contain method calls.
/// If we want to translate this accurately, we would need to introduce synthetic statements for qualifier and
/// the endpoints, which should then be used in the slice method call.
/// To avoid this, we translate as follows.
/// 1. s[a..b] -> s.Slice(a, b)
/// 2. s[..b] -> s.Slice(0, b)
/// 3. s[a..] -> s.Slice(a, ^0)
/// 4. s[..] -> s.Slice(0, ^0)
///
/// Even though index expressions can't technically be used in this way, they signal that we
/// could perceive ^b as "length - b".
///
/// Call arguments are only populated when a range expression is directly available in
/// the list of arguments.
/// This means that cases like below are not handled.
/// System.Range x = 1..3;
/// s[x]
/// </summary>
/// <param name="trapFile">The trap file to write to.</param>
/// <param name="slice">The slice method symbol.</param>
/// <param name="range">The range expression syntax.</param>
private void PopulateSlice(TextWriter trapFile, IMethodSymbol slice, RangeExpressionSyntax? range)
{
if (range is not null)
{
// Populate the call arguments
var left = range.LeftOperand is ExpressionSyntax lsyntax
? MakeFromRangeEndpoint(lsyntax, this, 0)
: MakeZeroLiteral(this, 0);
var right = range.RightOperand is ExpressionSyntax rsyntax
? MakeFromRangeEndpoint(rsyntax, this, 1)
: MakeZeroFromEndExpression(this, 1);
SetExprArgument(trapFile, left, right);
}
trapFile.expr_call(this, Method.Create(Context, slice));
}
protected override void PopulateExpression(TextWriter trapFile) protected override void PopulateExpression(TextWriter trapFile)
{ {
if (Kind == ExprKind.POINTER_INDIRECTION) if (Kind == ExprKind.POINTER_INDIRECTION)
@@ -150,19 +30,11 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
else else
{ {
Create(Context, qualifier, this, -1); Create(Context, qualifier, this, -1);
var target = GetTargetSymbol();
if (target is IMethodSymbol method && IsSlice(method, out var range))
{
// When an indexer on a span or string is used in conjunction with a range expression, the compiler translates
// this into a call to the "Slice" or "Substring" method.
// In this case, we want to populate a slice/substring method call instead of an indexer access.
PopulateSlice(trapFile, method, range);
return;
}
PopulateArguments(trapFile, argumentList, 0); PopulateArguments(trapFile, argumentList, 0);
if (target is IPropertySymbol { IsIndexer: true } indexer)
var symbolInfo = Context.GetSymbolInfo(base.Syntax);
if (symbolInfo.Symbol is IPropertySymbol indexer)
{ {
trapFile.expr_access(this, Indexer.Create(Context, indexer)); trapFile.expr_access(this, Indexer.Create(Context, indexer));
} }
@@ -174,11 +46,8 @@ namespace Semmle.Extraction.CSharp.Entities.Expressions
private static bool IsArray(ITypeSymbol symbol) => private static bool IsArray(ITypeSymbol symbol) =>
symbol.TypeKind == Microsoft.CodeAnalysis.TypeKind.Array || symbol.IsInlineArray(); symbol.TypeKind == Microsoft.CodeAnalysis.TypeKind.Array || symbol.IsInlineArray();
private static ExprKind GetKind(Context cx, ExpressionSyntax syntax, ExpressionSyntax qualifier) private static ExprKind GetKind(Context cx, ExpressionSyntax qualifier)
{ {
if (cx.GetSymbolInfo(syntax).Symbol is IMethodSymbol)
return ExprKind.METHOD_INVOCATION;
var qualifierType = cx.GetType(qualifier); var qualifierType = cx.GetType(qualifier);
// This is a compilation error, so make a guess and continue. // This is a compilation error, so make a guess and continue.

View File

@@ -23,9 +23,7 @@ namespace Semmle.Extraction.CSharp.Entities.Statements
} }
else if (isSpecificCatchClause) // A catch clause of the form 'catch(Ex) { ... }' else if (isSpecificCatchClause) // A catch clause of the form 'catch(Ex) { ... }'
{ {
var type = Type.Create(Context, Context.GetType(Stmt.Declaration!.Type)); trapFile.catch_type(this, Type.Create(Context, Context.GetType(Stmt.Declaration!.Type)).TypeRef, true);
trapFile.catch_type(this, type.TypeRef, true);
TypeMention.Create(Context, Stmt.Declaration!.Type, this, type);
} }
else // A catch clause of the form 'catch { ... }' else // A catch clause of the form 'catch { ... }'
{ {

View File

@@ -67,7 +67,7 @@ namespace Semmle.Extraction.CSharp
return; return;
} }
var mscorlibExists = File.Exists(Path.Join(compilerDir, "mscorlib.dll")); var mscorlibExists = File.Exists(Path.Combine(compilerDir, "mscorlib.dll"));
if (specifiedFramework is null && mscorlibExists) if (specifiedFramework is null && mscorlibExists)
{ {
@@ -107,7 +107,7 @@ namespace Semmle.Extraction.CSharp
/// <summary> /// <summary>
/// The file csc.rsp. /// The file csc.rsp.
/// </summary> /// </summary>
private string CscRsp => Path.Join(FrameworkPath, csc_rsp); private string CscRsp => Path.Combine(FrameworkPath, csc_rsp);
/// <summary> /// <summary>
/// Should we skip extraction? /// Should we skip extraction?

View File

@@ -680,7 +680,7 @@ namespace Semmle.Extraction.CSharp
{ {
try try
{ {
var fullPath = Path.GetFullPath(Path.Join(Path.GetDirectoryName(mappedFromPath)!, mappedToPath)); var fullPath = Path.GetFullPath(Path.Combine(Path.GetDirectoryName(mappedFromPath)!, mappedToPath));
ExtractionContext.Logger.LogDebug($"Found relative path in line mapping: '{mappedToPath}', interpreting it as '{fullPath}'"); ExtractionContext.Logger.LogDebug($"Found relative path in line mapping: '{mappedToPath}', interpreting it as '{fullPath}'");
mappedToPath = fullPath; mappedToPath = fullPath;

View File

@@ -159,11 +159,7 @@ namespace Semmle.Extraction.CSharp
return null; return null;
} }
var normalized = Path.DirectorySeparatorChar == '/' ? file.Replace("\\", "/") : file; return Path.GetFullPath(Path.Combine(projDir?.FullName ?? string.Empty, Path.DirectorySeparatorChar == '/' ? file.Replace("\\", "/") : file));
var path = projDir is not null && !Path.IsPathRooted(normalized)
? Path.Join(projDir.FullName, normalized)
: normalized;
return Path.GetFullPath(path);
} }
private readonly string[] references; private readonly string[] references;

View File

@@ -210,7 +210,7 @@ namespace Semmle.Extraction.CSharp
TracingAnalyser.GetOutputName(compilation, args), TracingAnalyser.GetOutputName(compilation, args),
compilation, compilation,
generatedSyntaxTrees, generatedSyntaxTrees,
Path.Join(compilationIdentifierPath, diagnosticName), Path.Combine(compilationIdentifierPath, diagnosticName),
options), options),
() => { }); () => { });
@@ -377,7 +377,7 @@ namespace Semmle.Extraction.CSharp
else else
{ {
var composed = referencePaths.Value var composed = referencePaths.Value
.Select(path => Path.Join(path, clref.Reference)) .Select(path => Path.Combine(path, clref.Reference))
.Where(path => File.Exists(path)) .Where(path => File.Exists(path))
.Select(path => analyser.PathCache.GetCanonicalPath(path)) .Select(path => analyser.PathCache.GetCanonicalPath(path))
.FirstOrDefault(); .FirstOrDefault();
@@ -559,13 +559,13 @@ namespace Semmle.Extraction.CSharp
/// Gets the path to the `csharp.log` file written to by the C# extractor. /// Gets the path to the `csharp.log` file written to by the C# extractor.
/// </summary> /// </summary>
public static string GetCSharpLogPath() => public static string GetCSharpLogPath() =>
Path.Join(GetCSharpLogDirectory(), "csharp.log"); Path.Combine(GetCSharpLogDirectory(), "csharp.log");
/// <summary> /// <summary>
/// Gets the path to a `csharp.{hash}.txt` file written to by the C# extractor. /// Gets the path to a `csharp.{hash}.txt` file written to by the C# extractor.
/// </summary> /// </summary>
public static string GetCSharpArgsLogPath(string hash) => public static string GetCSharpArgsLogPath(string hash) =>
Path.Join(GetCSharpLogDirectory(), $"csharp.{hash}.txt"); Path.Combine(GetCSharpLogDirectory(), $"csharp.{hash}.txt");
/// <summary> /// <summary>
/// Gets a list of all `csharp.{hash}.txt` files currently written to the log directory. /// Gets a list of all `csharp.{hash}.txt` files currently written to the log directory.

View File

@@ -131,7 +131,7 @@ namespace Semmle.Extraction.CSharp
return Path.ChangeExtension(entryPointFilename, ".exe"); return Path.ChangeExtension(entryPointFilename, ".exe");
} }
return Path.Join(commandLineArguments.OutputDirectory, commandLineArguments.OutputFileName); return Path.Combine(commandLineArguments.OutputDirectory, commandLineArguments.OutputFileName);
} }
private int LogDiagnostics() private int LogDiagnostics()

View File

@@ -61,7 +61,7 @@ namespace Semmle.Extraction.CSharp
* Although GetRandomFileName() is cryptographically secure, * Although GetRandomFileName() is cryptographically secure,
* there's a tiny chance the file could already exists. * there's a tiny chance the file could already exists.
*/ */
tmpFile = Path.Join(tempPath, Path.GetRandomFileName()); tmpFile = Path.Combine(tempPath, Path.GetRandomFileName());
} }
while (File.Exists(tmpFile)); while (File.Exists(tmpFile));

View File

@@ -82,13 +82,13 @@ namespace SemmleTests.Semmle.Util
[Fact] [Fact]
public void CanonicalPathMissingFile() public void CanonicalPathMissingFile()
{ {
Assert.Equal(Path.Join(Directory.GetCurrentDirectory(), "NOSUCHFILE"), cache.GetCanonicalPath("NOSUCHFILE")); Assert.Equal(Path.Combine(Directory.GetCurrentDirectory(), "NOSUCHFILE"), cache.GetCanonicalPath("NOSUCHFILE"));
} }
[Fact] [Fact]
public void CanonicalPathMissingAbsolutePath() public void CanonicalPathMissingAbsolutePath()
{ {
Assert.Equal(Path.Join(root, "no", "such", "file"), cache.GetCanonicalPath(Path.Join(root, "no", "such", "file"))); Assert.Equal(Path.Combine(root, "no", "such", "file"), cache.GetCanonicalPath(Path.Combine(root, "no", "such", "file")));
if (Win32.IsWindows()) if (Win32.IsWindows())
Assert.Equal(@"C:\Windows\no\such\file", cache.GetCanonicalPath(@"C:\windOws\no\such\file")); Assert.Equal(@"C:\Windows\no\such\file", cache.GetCanonicalPath(@"C:\windOws\no\such\file"));
@@ -97,7 +97,7 @@ namespace SemmleTests.Semmle.Util
[Fact] [Fact]
public void CanonicalPathMissingRelativePath() public void CanonicalPathMissingRelativePath()
{ {
Assert.Equal(Path.Join(Directory.GetCurrentDirectory(), "NO", "SUCH"), cache.GetCanonicalPath(Path.Join("NO", "SUCH"))); Assert.Equal(Path.Combine(Directory.GetCurrentDirectory(), "NO", "SUCH"), cache.GetCanonicalPath(Path.Combine("NO", "SUCH")));
} }
[Fact] [Fact]
@@ -125,7 +125,7 @@ namespace SemmleTests.Semmle.Util
public void CanonicalPathDots() public void CanonicalPathDots()
{ {
var abcPath = Path.GetFullPath("abc"); var abcPath = Path.GetFullPath("abc");
Assert.Equal(abcPath, cache.GetCanonicalPath(Path.Join("foo", ".", "..", "abc"))); Assert.Equal(abcPath, cache.GetCanonicalPath(Path.Combine("foo", ".", "..", "abc")));
} }
[Fact] [Fact]

View File

@@ -14,20 +14,20 @@ namespace SemmleTests.Semmle.Util
public sealed class LongPaths public sealed class LongPaths
{ {
private static readonly string tmpDir = Environment.GetEnvironmentVariable("TEST_TMPDIR") ?? Path.GetTempPath(); private static readonly string tmpDir = Environment.GetEnvironmentVariable("TEST_TMPDIR") ?? Path.GetTempPath();
private static readonly string longPathDir = Path.Join(tmpDir, "aaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbb", private static readonly string longPathDir = Path.Combine(tmpDir, "aaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"ccccccccccccccccccccccccccccccc", "ddddddddddddddddddddddddddddddddddddd", "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeee", "fffffffffffffffffffffffffffffffff", "ccccccccccccccccccccccccccccccc", "ddddddddddddddddddddddddddddddddddddd", "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeee", "fffffffffffffffffffffffffffffffff",
"ggggggggggggggggggggggggggggggggggg", "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh"); "ggggggggggggggggggggggggggggggggggg", "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh");
private static string MakeLongPath() private static string MakeLongPath()
{ {
var uniquePostfix = Guid.NewGuid().ToString("N"); var uniquePostfix = Guid.NewGuid().ToString("N");
return Path.Join(longPathDir, $"iiiiiiiiiiiiiiii{uniquePostfix}.txt"); return Path.Combine(longPathDir, $"iiiiiiiiiiiiiiii{uniquePostfix}.txt");
} }
private static string MakeShortPath() private static string MakeShortPath()
{ {
var uniquePostfix = Guid.NewGuid().ToString("N"); var uniquePostfix = Guid.NewGuid().ToString("N");
return Path.Join(tmpDir, $"test{uniquePostfix}.txt"); return Path.Combine(tmpDir, $"test{uniquePostfix}.txt");
} }
public LongPaths() public LongPaths()
@@ -62,7 +62,7 @@ namespace SemmleTests.Semmle.Util
[Fact] [Fact]
public void ParentDirectory() public void ParentDirectory()
{ {
Assert.Equal("abc", Path.GetDirectoryName(Path.Join("abc", "def"))); Assert.Equal("abc", Path.GetDirectoryName(Path.Combine("abc", "def")));
Assert.Equal(Win32.IsWindows() ? "\\" : "/", Path.GetDirectoryName($@"{Path.DirectorySeparatorChar}def")); Assert.Equal(Win32.IsWindows() ? "\\" : "/", Path.GetDirectoryName($@"{Path.DirectorySeparatorChar}def"));
Assert.Equal("", Path.GetDirectoryName(@"def")); Assert.Equal("", Path.GetDirectoryName(@"def"));

View File

@@ -137,11 +137,11 @@ namespace Semmle.Util
bool IsMonoInstalled(); bool IsMonoInstalled();
/// <summary> /// <summary>
/// Joins path segments, Path.Join(). /// Combine path segments, Path.Combine().
/// </summary> /// </summary>
/// <param name="parts">The parts of the path.</param> /// <param name="parts">The parts of the path.</param>
/// <returns>The combined path.</returns> /// <returns>The combined path.</returns>
string PathJoin(params string[] parts); string PathCombine(params string[] parts);
/// <summary> /// <summary>
/// Gets the full path for <paramref name="path"/>, Path.GetFullPath(). /// Gets the full path for <paramref name="path"/>, Path.GetFullPath().
@@ -293,7 +293,7 @@ namespace Semmle.Util
} }
} }
string IBuildActions.PathJoin(params string[] parts) => Path.Join(parts); string IBuildActions.PathCombine(params string[] parts) => Path.Combine(parts);
void IBuildActions.WriteAllText(string filename, string contents) => File.WriteAllText(filename, contents); void IBuildActions.WriteAllText(string filename, string contents) => File.WriteAllText(filename, contents);

View File

@@ -43,7 +43,7 @@ namespace Semmle.Util
var parent = Directory.GetParent(path); var parent = Directory.GetParent(path);
return parent is not null ? return parent is not null ?
Path.Join(cache.GetCanonicalPath(parent.FullName), Path.GetFileName(path)) : Path.Combine(cache.GetCanonicalPath(parent.FullName), Path.GetFileName(path)) :
path.ToUpperInvariant(); path.ToUpperInvariant();
} }
} }
@@ -138,12 +138,12 @@ namespace Semmle.Util
var entries = Directory.GetFileSystemEntries(parentPath, name); var entries = Directory.GetFileSystemEntries(parentPath, name);
return entries.Length == 1 return entries.Length == 1
? entries[0] ? entries[0]
: Path.Join(parentPath, name); : Path.Combine(parentPath, name);
} }
catch // lgtm[cs/catch-of-all-exceptions] catch // lgtm[cs/catch-of-all-exceptions]
{ {
// IO error or security error querying directory. // IO error or security error querying directory.
return Path.Join(parentPath, name); return Path.Combine(parentPath, name);
} }
} }
} }

View File

@@ -82,7 +82,7 @@ namespace Semmle.Util
{ {
exes = new[] { prog }; exes = new[] { prog };
} }
var candidates = paths?.Where(path => exes.Any(exe0 => File.Exists(Path.Join(path, exe0)))); var candidates = paths?.Where(path => exes.Any(exe0 => File.Exists(Path.Combine(path, exe0))));
return candidates?.FirstOrDefault(); return candidates?.FirstOrDefault();
} }
@@ -179,7 +179,7 @@ namespace Semmle.Util
{ {
innerpath = ConvertPathToSafeRelativePath(innerpath); innerpath = ConvertPathToSafeRelativePath(innerpath);
nested = Path.Join(outerpath, innerpath); nested = Path.Combine(outerpath, innerpath);
} }
try try
{ {
@@ -203,7 +203,7 @@ namespace Semmle.Util
{ {
var tempPath = Path.GetTempPath(); var tempPath = Path.GetTempPath();
var name = Guid.NewGuid().ToString("N").ToUpper(); var name = Guid.NewGuid().ToString("N").ToUpper();
var tempFolder = Path.Join(tempPath, "GitHub", name); var tempFolder = Path.Combine(tempPath, "GitHub", name);
Directory.CreateDirectory(tempFolder); Directory.CreateDirectory(tempFolder);
return tempFolder; return tempFolder;
}); });
@@ -231,7 +231,7 @@ namespace Semmle.Util
string outputPath; string outputPath;
do do
{ {
outputPath = Path.Join(tempFolder, Path.GetRandomFileName() + extension); outputPath = Path.Combine(tempFolder, Path.GetRandomFileName() + extension);
} }
while (File.Exists(outputPath)); while (File.Exists(outputPath));

View File

@@ -1,7 +1,3 @@
## 1.7.68
No user-facing changes.
## 1.7.67 ## 1.7.67
No user-facing changes. No user-facing changes.

View File

@@ -1,3 +0,0 @@
## 1.7.68
No user-facing changes.

View File

@@ -1,2 +1,2 @@
--- ---
lastReleaseVersion: 1.7.68 lastReleaseVersion: 1.7.67

View File

@@ -1,5 +1,5 @@
name: codeql/csharp-solorigate-all name: codeql/csharp-solorigate-all
version: 1.7.69-dev version: 1.7.68-dev
groups: groups:
- csharp - csharp
- solorigate - solorigate

View File

@@ -1,7 +1,3 @@
## 1.7.68
No user-facing changes.
## 1.7.67 ## 1.7.67
No user-facing changes. No user-facing changes.

Some files were not shown because too many files have changed in this diff Show More