Rasmus Wriedt Larsen
8079788a5f
Python: Add change-note
2024-02-28 16:44:25 +01:00
Rasmus Wriedt Larsen
cdf4dd16f0
Python: Fix module level flow for iterable unpacking
...
(and for * patterns in match)
Since `PhaseDependentFlow` uses the following predicate, that relies on
.getScope() to be present for there to be any importTimeFlow (flow at
toplevel scope), it's important that data-flow nodes implement `.getScope`.
```
private predicate isTopLevel(Node node) { node.getScope() instanceof Module }
```
By implementing getScope, we can now rely on default implementation of
`getEnclosingCallable` in DataFlow::Node:
```
/** Gets the enclosing callable of this node. */
DataFlowCallable getEnclosingCallable() { result = getCallableScope(this.getScope()) }
```
2024-02-28 16:39:08 +01:00
Rasmus Wriedt Larsen
e4699e092d
Python: Add test for iterable unpacking on module level
...
Currently doesn't work :O
2024-02-28 16:10:29 +01:00
Rasmus Lerchedahl Petersen
3eb9491cb4
python: rewrite HardcodedCredentials away from PointsTo
...
- `ModuleValue.attr` and `ClassValue.lookup` are approximated by `Function.getName`
- `ClassValue.getName` is apprximated by `Class.getName`
- `Module::named` is approximated by `Module.getName`
- `Value::named` is approximated by `Builtins::likelyBuiltin`
- `FunctionValue.getNamedArgumentForCall` is approximated by `ArgumentNode.argumentOf`
2024-02-26 17:18:40 +01:00
Rasmus Wriedt Larsen
1cfac50749
Python: Add precision to NoSQL query
...
Due to this, it was not part of any query suite :O
2024-02-26 11:23:43 +01:00
Tom Hvitved
62b16c0fa3
Share getFileBySourceArchiveName implementation
2024-02-23 11:25:49 +01:00
Taus
f1392712ee
Python: Add .copy() as a copy step
2024-02-22 13:09:27 +00:00
Taus
5125973f9b
Python: Add test case for .copy() as a copy step
2024-02-22 13:01:03 +00:00
Chris Smowton
f2e04c0cb2
Merge pull request #15672 from github/post-release-prep/codeql-cli-2.16.3
...
Post-release preparation for codeql-cli-2.16.3
2024-02-20 21:59:55 +00:00
yoff
d3ee5f65db
Merge pull request #15550 from yoff/python/remove-pointsto-from-module-getAnExport
...
python: remove a use of points-to
2024-02-20 19:04:46 +01:00
github-actions[bot]
37f8fa3413
Post-release preparation for codeql-cli-2.16.3
2024-02-20 16:50:47 +00:00
github-actions[bot]
6d061fbc35
Release preparation for version 2.16.3
2024-02-20 14:26:23 +00:00
Rasmus Lerchedahl Petersen
22e72d2fed
python: Move the rewrite out to Scope.qll
2024-02-20 10:39:29 +01:00
Rasmus Lerchedahl Petersen
de727bf1b5
Revert "python: remove a use of points-to"
...
This reverts commit 5cb71ce7e5
2024-02-20 10:23:31 +01:00
amammad
09d8a75844
Fix QLDoc issues
2024-02-14 23:31:22 +04:00
Rasmus Wriedt Larsen
eb401a205d
Python: Fix test exclusion for stdlib Python 3.12
2024-02-14 16:53:19 +01:00
Rasmus Wriedt Larsen
59014787a1
Python: Fix DataflowQueryTest
...
You're only allowed to have `result=OK` if there is a sink on that line...
2024-02-14 15:44:40 +01:00
Rasmus Wriedt Larsen
cd596f5d05
Python: Reformat test-file
...
All those newlines are not good for inline expectations
2024-02-14 15:44:06 +01:00
Rasmus Wriedt Larsen
e5bd633028
Python: Change name/id to Decompression Bomb
...
The old title/id matches how we used to write queries, but I think just
using the normal conversational name is easier for everyone :)
2024-02-14 14:54:25 +01:00
Rasmus Wriedt Larsen
69c8ef9898
Python: Use dataflow instead of taint-tracking
2024-02-14 14:52:37 +01:00
Rasmus Wriedt Larsen
ba7dd38fc9
Python: Delete duplicated file
2024-02-14 14:48:37 +01:00
Rasmus Wriedt Larsen
9ae3ea81ff
Python: Remove spurious results in stdlib
2024-02-14 14:47:28 +01:00
Rasmus Wriedt Larsen
d8fd457310
Python: Use helper predicate
...
Since the helper predicate had nice qldocs
2024-02-14 14:47:28 +01:00
Rasmus Wriedt Larsen
e7772f1062
Python: Use Unit class
2024-02-14 14:47:28 +01:00
Rasmus Wriedt Larsen
ad39b8c68b
Python: Accept .expected changes
2024-02-14 14:46:33 +01:00
Rasmus Wriedt Larsen
9399258e3b
Merge branch 'main' into amammad-python-bombs
2024-02-14 13:37:59 +01:00
Cornelius Riemenschneider
e08946ec8d
Merge pull request #15586 from github/criemen/bazel-python-language-pack
...
Python: Update BUILD.bazel files.
2024-02-13 10:19:12 +01:00
Cornelius Riemenschneider
27ebebc24b
Python: Update BUILD.bazel files.
...
This allows us to (later) build the whole python language pack with
bazel.
2024-02-12 17:10:35 +01:00
Rasmus Wriedt Larsen
cbb9a64bbb
Merge pull request #15457 from RasmusWL/psycopg
...
Python: Model the `psycopg` package
2024-02-12 15:59:16 +01:00
Tom Hvitved
1ea7717714
Capture flow: Take overwrites in nested scopes into account
2024-02-09 14:49:23 +01:00
Rasmus Lerchedahl Petersen
580e68d5de
python: add support for lower bound position
2024-02-09 13:51:16 +01:00
Anders Schack-Mulligen
088a0a54ba
Python: Add empty provenance column to expected files.
2024-02-09 11:32:08 +01:00
Anders Schack-Mulligen
817aa7655f
Python: Remove redundant IncludePostUpdateFlow and PhaseDependentFlow application.
2024-02-09 11:32:08 +01:00
Dave Bartolomeo
92bd550c55
Merge pull request #15531 from github/post-release-prep/codeql-cli-2.16.2
...
Post-release preparation for codeql-cli-2.16.2
2024-02-08 05:58:17 -08:00
Rasmus Lerchedahl Petersen
45bb4a0ee5
python: remove TaintStepFromSummary
...
as it should be covered by `SummarizedCallableFromModel`
Also move things around, to look more like the Ruby code.
2024-02-08 12:48:15 +01:00
Rasmus Lerchedahl Petersen
5cb71ce7e5
python: remove a use of points-to
...
This is used by `Scope::isPublic` which in turn is called by the framework model for `setuptools`.
On my current quesry, this had a dramatic effect on the most expensive predicates:
Before
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
1m9s | 2933 | 123ms @ 422 | PointsTo::Expressions::equalityEvaluatesTo/4#ebe72212@cab7d3xr
43.1s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@8aa78a38
41.3s | 2936 | 2.1s @ 409 | PointsTo::InterProceduralPointsTo::scope_entry_value_transfer_from_earlier/4#acb2199d@cab7ddxr
30.2s | 2946 | 67ms @ 847 | PointsTo::PointsToInternal::multi_assignment_points_to/4#28782e93@cab7d0yr
29.7s | 2930 | 1.9s @ 30 | Extensions::ReModulePointToExtension.pointsTo_helper/1#a84effde@cab7dn4w
24.9s | 2933 | 84ms @ 414 | PointsTo::Expressions::inequalityEvaluatesTo/4#f0ecfab4@cab7d2xr
17.9s | 2582 | 306ms @ 31 | MRO::ClassListList.getItem/1#b6c27115#reorder_2_0_1@cab7dw6r
9.4s | 661 | 991ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
9.2s | 2738 | 26ms @ 664 | MRO::ClassList.containsSpecial/0#c967dabb#fb@cab7dg4w
8.9s | 2946 | 12ms @ 917 | PointsTo::Types::getBase/2#0ab04984@cab7du1w
7.4s | 2946 | 287ms @ 3 | PointsTo::PointsToInternal::points_to_candidate/4#0a587a42@cab7d80w
7.1s | 2934 | 14ms @ 2 | Constants::ConstantObjectInternal.attribute/3#6d9e12fc@cab7d6zr
6.8s | 2946 | 9ms @ 48 | PointsTo::InterProceduralPointsTo::callsite_points_to/4#72419c70@cab7dqxr
6.6s | 234 | 341ms @ 17 | ApiGraphs::API::Impl::rhs/3#2255afc6@a41b31w3
6.6s | 2946 | 86ms @ 5 | PointsTo::Types::six_add_metaclass/4#f926a4cb@cab7da0w
6.2s | 2930 | 341ms @ 30 | Extensions::RangeIterationVariableFact.pointsTo/3#662720c9#cpe#124@cab7di2w
5.9s | 287 | 61ms @ 4 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@cc7b56yn
5.8s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@415f35h0
5.6s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@2c1fcdq1
5.3s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@22590ca9
5.2s | 233 | 276ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@a41b30w3
5.1s | 2945 | 177ms @ 4 | PointsTo::PointsToInternal::pointsTo/4#d99f16c6@cab7dj0w
4.7s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.6s | 277 | 2.2s @ 6 | DataFlowDispatch::getCallArg/5#21589076@cc7b5vxn
4.5s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@cc05a1fv
4.3s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@cb992b2h
4.1s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@6ae639js
4s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
3.7s | | | locations_ast_234501#join_rhs@0859685o
3.7s | 10 | 1.7s @ 1 | ObjectInternal::ObjectInternal.toString/0#dispred#0b2e9429@6e8a4yh7
3.6s | 2942 | 63ms @ 94 | PointsTo::InterProceduralPointsTo::call_points_to_from_callee/4#394022a8@cab7d90w
3.6s | 232 | 213ms @ 18 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@a41b33w3
3.6s | 2933 | 7ms @ 884 | PointsTo::Types::getInheritedMetaclass/2#097d39df#bff@cab7dr1w
3.6s | 2946 | 1.3s @ 13 | PointsTo::PointsToInternal::ssa_node_refinement_points_to/4#8ea6486b@cab7dnxr
3.5s | 1319 | 387ms @ 3 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.5s | 1320 | 385ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | 4861 | 478ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.3s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@47ba63n6
3.3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@4d16e7m6
3.3s | 2946 | 28ms @ 3 | PointsTo::PointsToInternal::reachableEdge/3#d3f53c12@cab7do7w
2.9s | 233 | 110ms @ 19 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@a41b32w3
2.8s | 31 | 2.2s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#cc7b5
2.8s | 2737 | 21ms @ 444 | MRO::ClassListList.removedClassParts/4#de59b06f#reorder_2_3_4_0_1@cab7d06w
2.8s | 1322 | 462ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.8s | 2946 | 187ms @ 5 | PointsTo::Expressions::builtinCallPointsTo/5#3aa7f48b@cab7dwwr
2.8s | 2939 | 41ms @ 7 | PointsTo::PointsToInternal::use_points_to/4#ff1d0edd@cab7df0w
2.7s | 2946 | 20ms @ 92 | PointsTo::Conditionals::evaluates/5#736734b2#fbffff#reorder_5_0_2_1_3_4@cab7dp5w
2.6s | 2946 | 152ms @ 5 | Constants::callToBool/2#0b9b1e8d@cab7dn7w
2.5s | 287 | 24ms @ 4 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@cc7b53xn
2.4s | 2946 | 31ms @ 5 | PointsTo::AttributePointsTo::variableAttributePointsTo/5#60adcc49@cab7dpwr
[2024-02-08 10:44:37] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 1231.1 seconds
* Total time spent evaluating predicates: 1167.1 seconds
```
After
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
41.6s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@85aaaac1
9.2s | 661 | 905ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
7.6s | 234 | 502ms @ 19 | ApiGraphs::API::Impl::rhs/3#2255afc6@ce6d11wc
6.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@fd1dc5mi
6s | 287 | 80ms @ 113 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@925826yr
5.7s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@851052bl
5.6s | 233 | 289ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@ce6d10wc
5.4s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@f2c42d17
4.8s | 277 | 2.4s @ 6 | DataFlowDispatch::getCallArg/5#21589076@92582vxr
4.7s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@ac08e0nf
4.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@82ff50ql
4.6s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
4.3s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.2s | 232 | 249ms @ 19 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@ce6d13wc
3.8s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@0ac73425
3.6s | 1319 | 354ms @ 1 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.6s | 1320 | 381ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@9e946ea8
3.4s | 4861 | 474ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.1s | 31 | 2.5s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#92582
3s | 53 | 114ms @ 48 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@9ab38jw0
3s | 233 | 126ms @ 20 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@ce6d12wc
3s | | | locations_ast_234501#join_rhs@0859685o
3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@a2145cqf
2.8s | 234 | 206ms @ 21 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs#1@L9#ce6d1
2.8s | 1322 | 447ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.7s | 230 | 176ms @ 28 | ApiGraphs::API::Impl::MkDef#51c2f877@ce6d1w9c
2.5s | 287 | 50ms @ 112 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@925823xr
2.4s | 234 | 246ms @ 19 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs@L4#ce6d1
2.3s | | | TaintTrackingPrivate::localAdditionalTaintStep/2#a2ec8c9d@e31201hd
2.2s | 53 | 72ms @ 15 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@96b28jwo
2.2s | | | SensitiveDataSources::SensitiveDataModeling::sensitiveString/1#fdc3ad40@41f6ee2g
2s | | | DataFlowImplCommon::Cached::viableParamArg/3#4c55eddb@8f7f25oq
2s | | | Flow::ControlFlowNode.getExprChild/1#e757d179#bbf@db51e8ed
1.9s | | | project#FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8#2@e36c2dr8
1.9s | | | DataFlowPublic::Node.hasLocationInfo/5#dispred#b79d995f@6e929dfv
1.7s | 15 | 433ms @ 1 | PoorMansFunctionResolution::poorMansFunctionTracker/2#75430e01@e5202dnv
1.7s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@60d9daea
1.7s | 29 | 633ms @ 6 | _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs@L4#92582
1.5s | 233 | 79ms @ 24 | ApiGraphs::API::Impl::trackUseNode/1#1af3a9ea@ce6d16wc
1.5s | | | ApiGraphs::API::Impl::edge/3#8453bf65@1bd8a6ja
1.5s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1@5dbb806u
1.3s | 1323 | 178ms @ 13 | SsaCompute::Liveness::liveAtEntry/2#bab3ea7c@6fd4cw73
1.3s | | | SsaCompute::SsaComputeImpl::defUseRank/4#782a2f48@0f27919s
1.3s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@96b65frd
1.3s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1_10#join_rhs@c1dd43nv
1.3s | | | FlowSummaryImpl::Private::SummaryNode.toString/0#dispred#d499e234@63bd684g
1.2s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@eaebb27g
1.2s | | | _DataFlowPublic::Node#da3b6093_DataFlowPublic::Node.asExpr/0#dispred#2845197a_py_exprs#antijoin_rhs@fcd8c3kj
1.2s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@c3f634us
[2024-02-08 11:43:50] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 636.9 seconds
* Total time spent evaluating predicates: 562.4 seconds
```
2024-02-08 12:20:56 +01:00
github-actions[bot]
b5139078d0
Post-release preparation for codeql-cli-2.16.2
2024-02-06 19:22:35 +00:00
github-actions[bot]
c1b35fbf47
Release preparation for version 2.16.2
2024-02-05 17:58:57 +00:00
James Ockers
9f7f9fcc6e
Updating change-notes to reflect what will be the visible change to end users
2024-02-02 11:38:17 -08:00
James Ockers
0f1e21aa09
Adding per-language change-notes
2024-01-30 17:28:34 -08:00
James Ockers
eb5e0123d6
exclude certification from maybeCertificate() regexes
2024-01-30 13:16:18 -08:00
Sid Shankar
f557110d9b
Merge pull request #15465 from sidshank/sidshank/rename-file-name-for-extracted-files-diagnostic
...
JS/TS/Python/Ruby: Renames diagnostic query files and tests
2024-01-30 10:19:00 -05:00
Rasmus Wriedt Larsen
c265c15f3f
Merge pull request #15398 from RasmusWL/html-escape
...
Python: Add `html.escape` as HTML sanitizer
2024-01-30 16:06:01 +01:00
Rasmus Wriedt Larsen
c70b32f7eb
Python: Require quote escaping for html.escape
2024-01-30 12:17:01 +01:00
Sid Shankar
b1d7a635f5
Renames diagnostic query files and tests
...
This commit renames the files relating to the diagnostic query that produces information on the number of files extracted. The files have been renamed from "SuccessfullExtractedFiles.*" to "ExtractedFiles.*". All related tests and test files have been renamed too.
The `@tags` and `@id` attributes of the queries have been left untouched, consistent with the `@tags` and `@id` for similar queries in other languages.
2024-01-29 20:19:20 +00:00
Rasmus Wriedt Larsen
5867fb3d29
Python: Add change-note
2024-01-29 14:30:20 +01:00
Rasmus Wriedt Larsen
3f0dc2b022
Python: Model the psycopg package
2024-01-29 14:30:20 +01:00
yoff
391ca5d8a6
Merge pull request #15390 from Marcono1234/marcono1234/python-ascii-regex-flag
2024-01-29 14:27:50 +01:00
Marcono1234
1ad08efe08
Python: Support a (ASCII) inline regex flag
2024-01-26 22:18:49 +01:00
Henry Mercer
10343dd822
Merge pull request #15416 from github/post-release-prep/codeql-cli-2.16.1
...
Post-release preparation for codeql-cli-2.16.1
2024-01-25 14:15:25 +00:00
