Rasmus Wriedt Larsen
8079788a5f
Python: Add change-note
2024-02-28 16:44:25 +01:00
Rasmus Wriedt Larsen
cdf4dd16f0
Python: Fix module level flow for iterable unpacking
...
(and for * patterns in match)
Since `PhaseDependentFlow` uses the following predicate, that relies on
.getScope() to be present for there to be any importTimeFlow (flow at
toplevel scope), it's important that data-flow nodes implement `.getScope`.
```
private predicate isTopLevel(Node node) { node.getScope() instanceof Module }
```
By implementing getScope, we can now rely on default implementation of
`getEnclosingCallable` in DataFlow::Node:
```
/** Gets the enclosing callable of this node. */
DataFlowCallable getEnclosingCallable() { result = getCallableScope(this.getScope()) }
```
2024-02-28 16:39:08 +01:00
Tom Hvitved
62b16c0fa3
Share getFileBySourceArchiveName implementation
2024-02-23 11:25:49 +01:00
Taus
f1392712ee
Python: Add .copy() as a copy step
2024-02-22 13:09:27 +00:00
Chris Smowton
f2e04c0cb2
Merge pull request #15672 from github/post-release-prep/codeql-cli-2.16.3
...
Post-release preparation for codeql-cli-2.16.3
2024-02-20 21:59:55 +00:00
yoff
d3ee5f65db
Merge pull request #15550 from yoff/python/remove-pointsto-from-module-getAnExport
...
python: remove a use of points-to
2024-02-20 19:04:46 +01:00
github-actions[bot]
37f8fa3413
Post-release preparation for codeql-cli-2.16.3
2024-02-20 16:50:47 +00:00
github-actions[bot]
6d061fbc35
Release preparation for version 2.16.3
2024-02-20 14:26:23 +00:00
Rasmus Lerchedahl Petersen
22e72d2fed
python: Move the rewrite out to Scope.qll
2024-02-20 10:39:29 +01:00
Rasmus Lerchedahl Petersen
de727bf1b5
Revert "python: remove a use of points-to"
...
This reverts commit 5cb71ce7e5
2024-02-20 10:23:31 +01:00
Rasmus Wriedt Larsen
cbb9a64bbb
Merge pull request #15457 from RasmusWL/psycopg
...
Python: Model the `psycopg` package
2024-02-12 15:59:16 +01:00
Tom Hvitved
1ea7717714
Capture flow: Take overwrites in nested scopes into account
2024-02-09 14:49:23 +01:00
Rasmus Lerchedahl Petersen
580e68d5de
python: add support for lower bound position
2024-02-09 13:51:16 +01:00
Anders Schack-Mulligen
817aa7655f
Python: Remove redundant IncludePostUpdateFlow and PhaseDependentFlow application.
2024-02-09 11:32:08 +01:00
Dave Bartolomeo
92bd550c55
Merge pull request #15531 from github/post-release-prep/codeql-cli-2.16.2
...
Post-release preparation for codeql-cli-2.16.2
2024-02-08 05:58:17 -08:00
Rasmus Lerchedahl Petersen
45bb4a0ee5
python: remove TaintStepFromSummary
...
as it should be covered by `SummarizedCallableFromModel`
Also move things around, to look more like the Ruby code.
2024-02-08 12:48:15 +01:00
Rasmus Lerchedahl Petersen
5cb71ce7e5
python: remove a use of points-to
...
This is used by `Scope::isPublic` which in turn is called by the framework model for `setuptools`.
On my current quesry, this had a dramatic effect on the most expensive predicates:
Before
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
1m9s | 2933 | 123ms @ 422 | PointsTo::Expressions::equalityEvaluatesTo/4#ebe72212@cab7d3xr
43.1s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@8aa78a38
41.3s | 2936 | 2.1s @ 409 | PointsTo::InterProceduralPointsTo::scope_entry_value_transfer_from_earlier/4#acb2199d@cab7ddxr
30.2s | 2946 | 67ms @ 847 | PointsTo::PointsToInternal::multi_assignment_points_to/4#28782e93@cab7d0yr
29.7s | 2930 | 1.9s @ 30 | Extensions::ReModulePointToExtension.pointsTo_helper/1#a84effde@cab7dn4w
24.9s | 2933 | 84ms @ 414 | PointsTo::Expressions::inequalityEvaluatesTo/4#f0ecfab4@cab7d2xr
17.9s | 2582 | 306ms @ 31 | MRO::ClassListList.getItem/1#b6c27115#reorder_2_0_1@cab7dw6r
9.4s | 661 | 991ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
9.2s | 2738 | 26ms @ 664 | MRO::ClassList.containsSpecial/0#c967dabb#fb@cab7dg4w
8.9s | 2946 | 12ms @ 917 | PointsTo::Types::getBase/2#0ab04984@cab7du1w
7.4s | 2946 | 287ms @ 3 | PointsTo::PointsToInternal::points_to_candidate/4#0a587a42@cab7d80w
7.1s | 2934 | 14ms @ 2 | Constants::ConstantObjectInternal.attribute/3#6d9e12fc@cab7d6zr
6.8s | 2946 | 9ms @ 48 | PointsTo::InterProceduralPointsTo::callsite_points_to/4#72419c70@cab7dqxr
6.6s | 234 | 341ms @ 17 | ApiGraphs::API::Impl::rhs/3#2255afc6@a41b31w3
6.6s | 2946 | 86ms @ 5 | PointsTo::Types::six_add_metaclass/4#f926a4cb@cab7da0w
6.2s | 2930 | 341ms @ 30 | Extensions::RangeIterationVariableFact.pointsTo/3#662720c9#cpe#124@cab7di2w
5.9s | 287 | 61ms @ 4 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@cc7b56yn
5.8s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@415f35h0
5.6s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@2c1fcdq1
5.3s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@22590ca9
5.2s | 233 | 276ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@a41b30w3
5.1s | 2945 | 177ms @ 4 | PointsTo::PointsToInternal::pointsTo/4#d99f16c6@cab7dj0w
4.7s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.6s | 277 | 2.2s @ 6 | DataFlowDispatch::getCallArg/5#21589076@cc7b5vxn
4.5s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@cc05a1fv
4.3s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@cb992b2h
4.1s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@6ae639js
4s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
3.7s | | | locations_ast_234501#join_rhs@0859685o
3.7s | 10 | 1.7s @ 1 | ObjectInternal::ObjectInternal.toString/0#dispred#0b2e9429@6e8a4yh7
3.6s | 2942 | 63ms @ 94 | PointsTo::InterProceduralPointsTo::call_points_to_from_callee/4#394022a8@cab7d90w
3.6s | 232 | 213ms @ 18 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@a41b33w3
3.6s | 2933 | 7ms @ 884 | PointsTo::Types::getInheritedMetaclass/2#097d39df#bff@cab7dr1w
3.6s | 2946 | 1.3s @ 13 | PointsTo::PointsToInternal::ssa_node_refinement_points_to/4#8ea6486b@cab7dnxr
3.5s | 1319 | 387ms @ 3 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.5s | 1320 | 385ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | 4861 | 478ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.3s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@47ba63n6
3.3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@4d16e7m6
3.3s | 2946 | 28ms @ 3 | PointsTo::PointsToInternal::reachableEdge/3#d3f53c12@cab7do7w
2.9s | 233 | 110ms @ 19 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@a41b32w3
2.8s | 31 | 2.2s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#cc7b5
2.8s | 2737 | 21ms @ 444 | MRO::ClassListList.removedClassParts/4#de59b06f#reorder_2_3_4_0_1@cab7d06w
2.8s | 1322 | 462ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.8s | 2946 | 187ms @ 5 | PointsTo::Expressions::builtinCallPointsTo/5#3aa7f48b@cab7dwwr
2.8s | 2939 | 41ms @ 7 | PointsTo::PointsToInternal::use_points_to/4#ff1d0edd@cab7df0w
2.7s | 2946 | 20ms @ 92 | PointsTo::Conditionals::evaluates/5#736734b2#fbffff#reorder_5_0_2_1_3_4@cab7dp5w
2.6s | 2946 | 152ms @ 5 | Constants::callToBool/2#0b9b1e8d@cab7dn7w
2.5s | 287 | 24ms @ 4 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@cc7b53xn
2.4s | 2946 | 31ms @ 5 | PointsTo::AttributePointsTo::variableAttributePointsTo/5#60adcc49@cab7dpwr
[2024-02-08 10:44:37] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 1231.1 seconds
* Total time spent evaluating predicates: 1167.1 seconds
```
After
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
41.6s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@85aaaac1
9.2s | 661 | 905ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
7.6s | 234 | 502ms @ 19 | ApiGraphs::API::Impl::rhs/3#2255afc6@ce6d11wc
6.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@fd1dc5mi
6s | 287 | 80ms @ 113 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@925826yr
5.7s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@851052bl
5.6s | 233 | 289ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@ce6d10wc
5.4s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@f2c42d17
4.8s | 277 | 2.4s @ 6 | DataFlowDispatch::getCallArg/5#21589076@92582vxr
4.7s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@ac08e0nf
4.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@82ff50ql
4.6s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
4.3s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.2s | 232 | 249ms @ 19 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@ce6d13wc
3.8s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@0ac73425
3.6s | 1319 | 354ms @ 1 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.6s | 1320 | 381ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@9e946ea8
3.4s | 4861 | 474ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.1s | 31 | 2.5s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#92582
3s | 53 | 114ms @ 48 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@9ab38jw0
3s | 233 | 126ms @ 20 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@ce6d12wc
3s | | | locations_ast_234501#join_rhs@0859685o
3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@a2145cqf
2.8s | 234 | 206ms @ 21 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs#1@L9#ce6d1
2.8s | 1322 | 447ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.7s | 230 | 176ms @ 28 | ApiGraphs::API::Impl::MkDef#51c2f877@ce6d1w9c
2.5s | 287 | 50ms @ 112 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@925823xr
2.4s | 234 | 246ms @ 19 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs@L4#ce6d1
2.3s | | | TaintTrackingPrivate::localAdditionalTaintStep/2#a2ec8c9d@e31201hd
2.2s | 53 | 72ms @ 15 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@96b28jwo
2.2s | | | SensitiveDataSources::SensitiveDataModeling::sensitiveString/1#fdc3ad40@41f6ee2g
2s | | | DataFlowImplCommon::Cached::viableParamArg/3#4c55eddb@8f7f25oq
2s | | | Flow::ControlFlowNode.getExprChild/1#e757d179#bbf@db51e8ed
1.9s | | | project#FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8#2@e36c2dr8
1.9s | | | DataFlowPublic::Node.hasLocationInfo/5#dispred#b79d995f@6e929dfv
1.7s | 15 | 433ms @ 1 | PoorMansFunctionResolution::poorMansFunctionTracker/2#75430e01@e5202dnv
1.7s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@60d9daea
1.7s | 29 | 633ms @ 6 | _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs@L4#92582
1.5s | 233 | 79ms @ 24 | ApiGraphs::API::Impl::trackUseNode/1#1af3a9ea@ce6d16wc
1.5s | | | ApiGraphs::API::Impl::edge/3#8453bf65@1bd8a6ja
1.5s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1@5dbb806u
1.3s | 1323 | 178ms @ 13 | SsaCompute::Liveness::liveAtEntry/2#bab3ea7c@6fd4cw73
1.3s | | | SsaCompute::SsaComputeImpl::defUseRank/4#782a2f48@0f27919s
1.3s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@96b65frd
1.3s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1_10#join_rhs@c1dd43nv
1.3s | | | FlowSummaryImpl::Private::SummaryNode.toString/0#dispred#d499e234@63bd684g
1.2s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@eaebb27g
1.2s | | | _DataFlowPublic::Node#da3b6093_DataFlowPublic::Node.asExpr/0#dispred#2845197a_py_exprs#antijoin_rhs@fcd8c3kj
1.2s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@c3f634us
[2024-02-08 11:43:50] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 636.9 seconds
* Total time spent evaluating predicates: 562.4 seconds
```
2024-02-08 12:20:56 +01:00
github-actions[bot]
b5139078d0
Post-release preparation for codeql-cli-2.16.2
2024-02-06 19:22:35 +00:00
github-actions[bot]
c1b35fbf47
Release preparation for version 2.16.2
2024-02-05 17:58:57 +00:00
James Ockers
9f7f9fcc6e
Updating change-notes to reflect what will be the visible change to end users
2024-02-02 11:38:17 -08:00
James Ockers
0f1e21aa09
Adding per-language change-notes
2024-01-30 17:28:34 -08:00
James Ockers
eb5e0123d6
exclude certification from maybeCertificate() regexes
2024-01-30 13:16:18 -08:00
Rasmus Wriedt Larsen
c265c15f3f
Merge pull request #15398 from RasmusWL/html-escape
...
Python: Add `html.escape` as HTML sanitizer
2024-01-30 16:06:01 +01:00
Rasmus Wriedt Larsen
c70b32f7eb
Python: Require quote escaping for html.escape
2024-01-30 12:17:01 +01:00
Rasmus Wriedt Larsen
5867fb3d29
Python: Add change-note
2024-01-29 14:30:20 +01:00
Rasmus Wriedt Larsen
3f0dc2b022
Python: Model the psycopg package
2024-01-29 14:30:20 +01:00
yoff
391ca5d8a6
Merge pull request #15390 from Marcono1234/marcono1234/python-ascii-regex-flag
2024-01-29 14:27:50 +01:00
Marcono1234
1ad08efe08
Python: Support a (ASCII) inline regex flag
2024-01-26 22:18:49 +01:00
github-actions[bot]
d0b74c00fe
Post-release preparation for codeql-cli-2.16.1
2024-01-23 23:02:29 +00:00
github-actions[bot]
7ef611e6dc
Release preparation for version 2.16.1
2024-01-23 19:45:16 +00:00
Erik Krogh Kristensen
f1d6f56621
Merge pull request #15393 from erik-krogh/deps-jan-2024
...
All: delete outdated deprecations
2024-01-23 13:52:38 +01:00
erik-krogh
865df920f9
add change-notes
2024-01-22 19:30:57 +01:00
Rasmus Wriedt Larsen
00dc55d825
Python: Add change-note
2024-01-22 17:32:33 +01:00
Rasmus Wriedt Larsen
cbed6e861d
Python: Add html.escape as HTML sanitizer
2024-01-22 17:32:28 +01:00
Max Schaefer
17e3a45ad7
Apply suggestions from code review
...
Co-authored-by: Taus <tausbn@github.com >
2024-01-22 13:36:12 +00:00
Max Schaefer
98178458d0
Python: Add support for more URL redirect sanitisers.
...
Since some sanitisers don't handle backslashes correctly, I updated the data-flow configuration to incorporate a flow state tracking whether or not backslashes have been eliminated or converted to forward slashes.
2024-01-22 13:24:18 +00:00
erik-krogh
f60c01e3a8
Py: delete import that no longer exists
2024-01-22 09:22:50 +01:00
erik-krogh
8be7eadace
delete outdated deprecations
2024-01-22 09:11:35 +01:00
Sid Shankar
2c683c910f
Merge branch 'change/adjust-extracted-files-diagnostics' of https://github.com/sidshank/codeql into change/adjust-extracted-files-diagnostics
2024-01-17 14:32:36 +00:00
Sid Shankar
0824ab77e9
Adds change notes
2024-01-17 14:31:40 +00:00
Alexander Eyers-Taylor
934474681d
Merge pull request #15254 from github/post-release-prep/codeql-cli-2.16.0
...
Post-release preparation for codeql-cli-2.16.0
2024-01-16 14:50:40 +00:00
github-actions[bot]
57df8b92df
Post-release preparation for codeql-cli-2.16.0
2024-01-15 15:00:50 +00:00
Tom Hvitved
f90201eb56
Data flow: Remove column from mayBenefitFromCallContext
2024-01-09 11:34:43 +01:00
github-actions[bot]
a6c8cc9551
Release preparation for version 2.16.0
2024-01-08 13:11:26 +00:00
Rasmus Wriedt Larsen
95c24275f2
Merge pull request #15044 from RasmusWL/automated-subclass-models
...
Python: Automated subclass models
2024-01-05 10:43:48 +01:00
Aditya Sharad
b1803d0ac2
Merge rc/3.12 into main
2023-12-21 16:40:51 -08:00
Rasmus Lerchedahl Petersen
0f89f69555
Python: fix VariableWrite and remove unneded step
2023-12-20 15:45:18 +01:00
Rasmus Lerchedahl Petersen
215b146f06
Python: remove unused member predicate
2023-12-20 14:45:00 +01:00
Rasmus Lerchedahl Petersen
491ca3f1e6
Python: hide synthetic variable node
2023-12-20 14:42:45 +01:00
Rasmus Lerchedahl Petersen
afb3d1da6f
Python: move capture node to DataFlowPrivate
2023-12-20 14:41:17 +01:00
