Dave Bartolomeo
311ba8ea1b
Merge from main to resolve conflicts
2024-03-19 10:41:31 -04:00
Tom Hvitved
2e370e2ded
Python: Switch to shared XML.qll implementation
2024-03-19 13:17:53 +01:00
Tom Hvitved
fc55567d90
Merge pull request #15853 from hvitved/dataflow/get-location
...
Data flow: Replace `hasLocationInfo` with `getLocation`
2024-03-18 20:21:46 +01:00
github-actions[bot]
aebe9f6992
Post-release preparation for codeql-cli-2.16.5
2024-03-18 12:16:26 +00:00
github-actions[bot]
0a6243d07b
Release preparation for version 2.16.5
2024-03-18 10:14:07 +00:00
Rasmus Lerchedahl Petersen
2a0c451d2d
python: No fieldFlowBranchLimit for SummarizedCallables
...
Like https://github.com/github/codeql/pull/15689 for Ruby.
2024-03-18 10:29:36 +01:00
Rasmus Wriedt Larsen
7eb4419342
Python: Restrict type-tracking content to only be precise
...
At least for now :)
2024-03-15 10:24:57 +01:00
Rasmus Wriedt Larsen
7a3ee0f5f8
Python: Make IterableSequenceNode LocalSourceNode
...
We do this to remove the inconsistencies, and to be ready for a future
where type-tracking support content tracker of depth > 1.
It works because targets of loadSteps needs to be LocalSourceNodes
predicate loadStep(Node nodeFrom, LocalSourceNode nodeTo, Content content) {
2024-03-14 10:46:29 +01:00
Rasmus Wriedt Larsen
af8cef5b53
Python: Fixup deprecated type-tracker API
2024-03-14 10:43:28 +01:00
Rasmus Wriedt Larsen
2b09b084e0
Python: Add change-note
2024-03-14 10:43:28 +01:00
Rasmus Wriedt Larsen
92729dbbd6
Python: Support iterable unpacking in type-tracking
2024-03-14 10:42:38 +01:00
Rasmus Wriedt Larsen
dac2b57bb0
Python: type-track through dict-updates
2024-03-14 10:42:38 +01:00
Rasmus Wriedt Larsen
73fe596753
Python: type-tracking through dictionary construction
2024-03-14 10:42:38 +01:00
Rasmus Wriedt Larsen
ece8245a4b
Python: type-track through tuple content
2024-03-14 10:42:38 +01:00
Rasmus Wriedt Larsen
7721fb3331
Python: Setup shared read/store steps
2024-03-14 10:42:37 +01:00
Rasmus Wriedt Larsen
636cf611ae
Python: Allow general content in type-tracker
...
This should not result in many changes, since store/load steps are still
only implemented for attributes.
2024-03-14 10:42:37 +01:00
Rasmus Wriedt Larsen
fc8caa66c8
Python: Prepare for general content in type-tracker
...
Due to the char-pred of Content, this change should keep exactly the
same behavior as before.
2024-03-14 10:42:37 +01:00
Tom Hvitved
6c0ed28e6b
Python: Implement new data flow interface
2024-03-13 14:41:57 +01:00
Tom Hvitved
dddba3228b
Merge pull request #15867 from hvitved/dataflow/ap-limit
...
Data flow: Add `ConfigSig::accessPathLimit`
2024-03-12 14:57:51 +01:00
yoff
adbcbefaa9
Merge pull request #15551 from yoff/python/avoid-duplicate-model-inclusions
...
python: Remove `TaintStepFromSummary`
2024-03-11 13:52:20 +01:00
Tom Hvitved
da66281fef
Sync files
2024-03-11 13:02:04 +01:00
Rasmus Wriedt Larsen
42acd9c22c
Merge pull request #15695 from github/tausbn/python-add-copy-method-as-copy-step
...
Python: Add `.copy()` method call as copy step
2024-03-11 09:43:34 +01:00
Rasmus Lerchedahl Petersen
3601773856
python: support encoding lower bound
2024-03-08 14:59:28 +01:00
github-actions[bot]
dc9092c9ec
Post-release preparation for codeql-cli-2.16.4
2024-03-06 22:19:33 +00:00
github-actions[bot]
2f058ffb4d
Release preparation for version 2.16.4
2024-03-06 20:56:51 +00:00
Angela P Wen
ce31f8641a
Revert "Release preparation for version 2.16.4"
2024-03-06 12:07:33 -08:00
github-actions[bot]
661e68dab5
Release preparation for version 2.16.4
2024-03-05 18:13:58 +00:00
Angela P Wen
967963a653
Revert "Release preparation for version 2.16.4"
2024-03-05 08:53:33 -08:00
github-actions[bot]
a67218a027
Release preparation for version 2.16.4
2024-03-04 17:42:08 +00:00
Rasmus Wriedt Larsen
d99a763ef7
Python: add change-note
2024-03-01 15:24:33 +01:00
Rasmus Wriedt Larsen
eeda4355f1
Python: Fix missing DictionaryElementContent
2024-03-01 15:21:13 +01:00
Rasmus Wriedt Larsen
8079788a5f
Python: Add change-note
2024-02-28 16:44:25 +01:00
Rasmus Wriedt Larsen
cdf4dd16f0
Python: Fix module level flow for iterable unpacking
...
(and for * patterns in match)
Since `PhaseDependentFlow` uses the following predicate, that relies on
.getScope() to be present for there to be any importTimeFlow (flow at
toplevel scope), it's important that data-flow nodes implement `.getScope`.
```
private predicate isTopLevel(Node node) { node.getScope() instanceof Module }
```
By implementing getScope, we can now rely on default implementation of
`getEnclosingCallable` in DataFlow::Node:
```
/** Gets the enclosing callable of this node. */
DataFlowCallable getEnclosingCallable() { result = getCallableScope(this.getScope()) }
```
2024-02-28 16:39:08 +01:00
Tom Hvitved
62b16c0fa3
Share getFileBySourceArchiveName implementation
2024-02-23 11:25:49 +01:00
Taus
f1392712ee
Python: Add .copy() as a copy step
2024-02-22 13:09:27 +00:00
Chris Smowton
f2e04c0cb2
Merge pull request #15672 from github/post-release-prep/codeql-cli-2.16.3
...
Post-release preparation for codeql-cli-2.16.3
2024-02-20 21:59:55 +00:00
yoff
d3ee5f65db
Merge pull request #15550 from yoff/python/remove-pointsto-from-module-getAnExport
...
python: remove a use of points-to
2024-02-20 19:04:46 +01:00
github-actions[bot]
37f8fa3413
Post-release preparation for codeql-cli-2.16.3
2024-02-20 16:50:47 +00:00
github-actions[bot]
6d061fbc35
Release preparation for version 2.16.3
2024-02-20 14:26:23 +00:00
Rasmus Lerchedahl Petersen
22e72d2fed
python: Move the rewrite out to Scope.qll
2024-02-20 10:39:29 +01:00
Rasmus Lerchedahl Petersen
de727bf1b5
Revert "python: remove a use of points-to"
...
This reverts commit 5cb71ce7e5
2024-02-20 10:23:31 +01:00
Rasmus Wriedt Larsen
cbb9a64bbb
Merge pull request #15457 from RasmusWL/psycopg
...
Python: Model the `psycopg` package
2024-02-12 15:59:16 +01:00
Tom Hvitved
1ea7717714
Capture flow: Take overwrites in nested scopes into account
2024-02-09 14:49:23 +01:00
Rasmus Lerchedahl Petersen
580e68d5de
python: add support for lower bound position
2024-02-09 13:51:16 +01:00
Anders Schack-Mulligen
817aa7655f
Python: Remove redundant IncludePostUpdateFlow and PhaseDependentFlow application.
2024-02-09 11:32:08 +01:00
Dave Bartolomeo
92bd550c55
Merge pull request #15531 from github/post-release-prep/codeql-cli-2.16.2
...
Post-release preparation for codeql-cli-2.16.2
2024-02-08 05:58:17 -08:00
Rasmus Lerchedahl Petersen
45bb4a0ee5
python: remove TaintStepFromSummary
...
as it should be covered by `SummarizedCallableFromModel`
Also move things around, to look more like the Ruby code.
2024-02-08 12:48:15 +01:00
Rasmus Lerchedahl Petersen
5cb71ce7e5
python: remove a use of points-to
...
This is used by `Scope::isPublic` which in turn is called by the framework model for `setuptools`.
On my current quesry, this had a dramatic effect on the most expensive predicates:
Before
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
1m9s | 2933 | 123ms @ 422 | PointsTo::Expressions::equalityEvaluatesTo/4#ebe72212@cab7d3xr
43.1s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@8aa78a38
41.3s | 2936 | 2.1s @ 409 | PointsTo::InterProceduralPointsTo::scope_entry_value_transfer_from_earlier/4#acb2199d@cab7ddxr
30.2s | 2946 | 67ms @ 847 | PointsTo::PointsToInternal::multi_assignment_points_to/4#28782e93@cab7d0yr
29.7s | 2930 | 1.9s @ 30 | Extensions::ReModulePointToExtension.pointsTo_helper/1#a84effde@cab7dn4w
24.9s | 2933 | 84ms @ 414 | PointsTo::Expressions::inequalityEvaluatesTo/4#f0ecfab4@cab7d2xr
17.9s | 2582 | 306ms @ 31 | MRO::ClassListList.getItem/1#b6c27115#reorder_2_0_1@cab7dw6r
9.4s | 661 | 991ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
9.2s | 2738 | 26ms @ 664 | MRO::ClassList.containsSpecial/0#c967dabb#fb@cab7dg4w
8.9s | 2946 | 12ms @ 917 | PointsTo::Types::getBase/2#0ab04984@cab7du1w
7.4s | 2946 | 287ms @ 3 | PointsTo::PointsToInternal::points_to_candidate/4#0a587a42@cab7d80w
7.1s | 2934 | 14ms @ 2 | Constants::ConstantObjectInternal.attribute/3#6d9e12fc@cab7d6zr
6.8s | 2946 | 9ms @ 48 | PointsTo::InterProceduralPointsTo::callsite_points_to/4#72419c70@cab7dqxr
6.6s | 234 | 341ms @ 17 | ApiGraphs::API::Impl::rhs/3#2255afc6@a41b31w3
6.6s | 2946 | 86ms @ 5 | PointsTo::Types::six_add_metaclass/4#f926a4cb@cab7da0w
6.2s | 2930 | 341ms @ 30 | Extensions::RangeIterationVariableFact.pointsTo/3#662720c9#cpe#124@cab7di2w
5.9s | 287 | 61ms @ 4 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@cc7b56yn
5.8s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@415f35h0
5.6s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@2c1fcdq1
5.3s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@22590ca9
5.2s | 233 | 276ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@a41b30w3
5.1s | 2945 | 177ms @ 4 | PointsTo::PointsToInternal::pointsTo/4#d99f16c6@cab7dj0w
4.7s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.6s | 277 | 2.2s @ 6 | DataFlowDispatch::getCallArg/5#21589076@cc7b5vxn
4.5s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@cc05a1fv
4.3s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@cb992b2h
4.1s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@6ae639js
4s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
3.7s | | | locations_ast_234501#join_rhs@0859685o
3.7s | 10 | 1.7s @ 1 | ObjectInternal::ObjectInternal.toString/0#dispred#0b2e9429@6e8a4yh7
3.6s | 2942 | 63ms @ 94 | PointsTo::InterProceduralPointsTo::call_points_to_from_callee/4#394022a8@cab7d90w
3.6s | 232 | 213ms @ 18 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@a41b33w3
3.6s | 2933 | 7ms @ 884 | PointsTo::Types::getInheritedMetaclass/2#097d39df#bff@cab7dr1w
3.6s | 2946 | 1.3s @ 13 | PointsTo::PointsToInternal::ssa_node_refinement_points_to/4#8ea6486b@cab7dnxr
3.5s | 1319 | 387ms @ 3 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.5s | 1320 | 385ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | 4861 | 478ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.3s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@47ba63n6
3.3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@4d16e7m6
3.3s | 2946 | 28ms @ 3 | PointsTo::PointsToInternal::reachableEdge/3#d3f53c12@cab7do7w
2.9s | 233 | 110ms @ 19 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@a41b32w3
2.8s | 31 | 2.2s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#cc7b5
2.8s | 2737 | 21ms @ 444 | MRO::ClassListList.removedClassParts/4#de59b06f#reorder_2_3_4_0_1@cab7d06w
2.8s | 1322 | 462ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.8s | 2946 | 187ms @ 5 | PointsTo::Expressions::builtinCallPointsTo/5#3aa7f48b@cab7dwwr
2.8s | 2939 | 41ms @ 7 | PointsTo::PointsToInternal::use_points_to/4#ff1d0edd@cab7df0w
2.7s | 2946 | 20ms @ 92 | PointsTo::Conditionals::evaluates/5#736734b2#fbffff#reorder_5_0_2_1_3_4@cab7dp5w
2.6s | 2946 | 152ms @ 5 | Constants::callToBool/2#0b9b1e8d@cab7dn7w
2.5s | 287 | 24ms @ 4 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@cc7b53xn
2.4s | 2946 | 31ms @ 5 | PointsTo::AttributePointsTo::variableAttributePointsTo/5#60adcc49@cab7dpwr
[2024-02-08 10:44:37] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 1231.1 seconds
* Total time spent evaluating predicates: 1167.1 seconds
```
After
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
41.6s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@85aaaac1
9.2s | 661 | 905ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
7.6s | 234 | 502ms @ 19 | ApiGraphs::API::Impl::rhs/3#2255afc6@ce6d11wc
6.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@fd1dc5mi
6s | 287 | 80ms @ 113 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@925826yr
5.7s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@851052bl
5.6s | 233 | 289ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@ce6d10wc
5.4s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@f2c42d17
4.8s | 277 | 2.4s @ 6 | DataFlowDispatch::getCallArg/5#21589076@92582vxr
4.7s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@ac08e0nf
4.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@82ff50ql
4.6s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
4.3s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.2s | 232 | 249ms @ 19 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@ce6d13wc
3.8s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@0ac73425
3.6s | 1319 | 354ms @ 1 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.6s | 1320 | 381ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@9e946ea8
3.4s | 4861 | 474ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.1s | 31 | 2.5s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#92582
3s | 53 | 114ms @ 48 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@9ab38jw0
3s | 233 | 126ms @ 20 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@ce6d12wc
3s | | | locations_ast_234501#join_rhs@0859685o
3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@a2145cqf
2.8s | 234 | 206ms @ 21 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs#1@L9#ce6d1
2.8s | 1322 | 447ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.7s | 230 | 176ms @ 28 | ApiGraphs::API::Impl::MkDef#51c2f877@ce6d1w9c
2.5s | 287 | 50ms @ 112 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@925823xr
2.4s | 234 | 246ms @ 19 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs@L4#ce6d1
2.3s | | | TaintTrackingPrivate::localAdditionalTaintStep/2#a2ec8c9d@e31201hd
2.2s | 53 | 72ms @ 15 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@96b28jwo
2.2s | | | SensitiveDataSources::SensitiveDataModeling::sensitiveString/1#fdc3ad40@41f6ee2g
2s | | | DataFlowImplCommon::Cached::viableParamArg/3#4c55eddb@8f7f25oq
2s | | | Flow::ControlFlowNode.getExprChild/1#e757d179#bbf@db51e8ed
1.9s | | | project#FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8#2@e36c2dr8
1.9s | | | DataFlowPublic::Node.hasLocationInfo/5#dispred#b79d995f@6e929dfv
1.7s | 15 | 433ms @ 1 | PoorMansFunctionResolution::poorMansFunctionTracker/2#75430e01@e5202dnv
1.7s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@60d9daea
1.7s | 29 | 633ms @ 6 | _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs@L4#92582
1.5s | 233 | 79ms @ 24 | ApiGraphs::API::Impl::trackUseNode/1#1af3a9ea@ce6d16wc
1.5s | | | ApiGraphs::API::Impl::edge/3#8453bf65@1bd8a6ja
1.5s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1@5dbb806u
1.3s | 1323 | 178ms @ 13 | SsaCompute::Liveness::liveAtEntry/2#bab3ea7c@6fd4cw73
1.3s | | | SsaCompute::SsaComputeImpl::defUseRank/4#782a2f48@0f27919s
1.3s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@96b65frd
1.3s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1_10#join_rhs@c1dd43nv
1.3s | | | FlowSummaryImpl::Private::SummaryNode.toString/0#dispred#d499e234@63bd684g
1.2s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@eaebb27g
1.2s | | | _DataFlowPublic::Node#da3b6093_DataFlowPublic::Node.asExpr/0#dispred#2845197a_py_exprs#antijoin_rhs@fcd8c3kj
1.2s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@c3f634us
[2024-02-08 11:43:50] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 636.9 seconds
* Total time spent evaluating predicates: 562.4 seconds
```
2024-02-08 12:20:56 +01:00
github-actions[bot]
b5139078d0
Post-release preparation for codeql-cli-2.16.2
2024-02-06 19:22:35 +00:00
github-actions[bot]
c1b35fbf47
Release preparation for version 2.16.2
2024-02-05 17:58:57 +00:00
