Rasmus Wriedt Larsen
d8fd457310
Python: Use helper predicate
...
Since the helper predicate had nice qldocs
2024-02-14 14:47:28 +01:00
Rasmus Wriedt Larsen
e7772f1062
Python: Use Unit class
2024-02-14 14:47:28 +01:00
Rasmus Wriedt Larsen
ad39b8c68b
Python: Accept .expected changes
2024-02-14 14:46:33 +01:00
Rasmus Wriedt Larsen
9399258e3b
Merge branch 'main' into amammad-python-bombs
2024-02-14 13:37:59 +01:00
Cornelius Riemenschneider
e08946ec8d
Merge pull request #15586 from github/criemen/bazel-python-language-pack
...
Python: Update BUILD.bazel files.
2024-02-13 10:19:12 +01:00
Cornelius Riemenschneider
27ebebc24b
Python: Update BUILD.bazel files.
...
This allows us to (later) build the whole python language pack with
bazel.
2024-02-12 17:10:35 +01:00
Rasmus Wriedt Larsen
cbb9a64bbb
Merge pull request #15457 from RasmusWL/psycopg
...
Python: Model the `psycopg` package
2024-02-12 15:59:16 +01:00
Tom Hvitved
1ea7717714
Capture flow: Take overwrites in nested scopes into account
2024-02-09 14:49:23 +01:00
Rasmus Lerchedahl Petersen
580e68d5de
python: add support for lower bound position
2024-02-09 13:51:16 +01:00
Anders Schack-Mulligen
088a0a54ba
Python: Add empty provenance column to expected files.
2024-02-09 11:32:08 +01:00
Anders Schack-Mulligen
817aa7655f
Python: Remove redundant IncludePostUpdateFlow and PhaseDependentFlow application.
2024-02-09 11:32:08 +01:00
Dave Bartolomeo
92bd550c55
Merge pull request #15531 from github/post-release-prep/codeql-cli-2.16.2
...
Post-release preparation for codeql-cli-2.16.2
2024-02-08 05:58:17 -08:00
Rasmus Lerchedahl Petersen
45bb4a0ee5
python: remove TaintStepFromSummary
...
as it should be covered by `SummarizedCallableFromModel`
Also move things around, to look more like the Ruby code.
2024-02-08 12:48:15 +01:00
Rasmus Lerchedahl Petersen
5cb71ce7e5
python: remove a use of points-to
...
This is used by `Scope::isPublic` which in turn is called by the framework model for `setuptools`.
On my current quesry, this had a dramatic effect on the most expensive predicates:
Before
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
1m9s | 2933 | 123ms @ 422 | PointsTo::Expressions::equalityEvaluatesTo/4#ebe72212@cab7d3xr
43.1s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@8aa78a38
41.3s | 2936 | 2.1s @ 409 | PointsTo::InterProceduralPointsTo::scope_entry_value_transfer_from_earlier/4#acb2199d@cab7ddxr
30.2s | 2946 | 67ms @ 847 | PointsTo::PointsToInternal::multi_assignment_points_to/4#28782e93@cab7d0yr
29.7s | 2930 | 1.9s @ 30 | Extensions::ReModulePointToExtension.pointsTo_helper/1#a84effde@cab7dn4w
24.9s | 2933 | 84ms @ 414 | PointsTo::Expressions::inequalityEvaluatesTo/4#f0ecfab4@cab7d2xr
17.9s | 2582 | 306ms @ 31 | MRO::ClassListList.getItem/1#b6c27115#reorder_2_0_1@cab7dw6r
9.4s | 661 | 991ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
9.2s | 2738 | 26ms @ 664 | MRO::ClassList.containsSpecial/0#c967dabb#fb@cab7dg4w
8.9s | 2946 | 12ms @ 917 | PointsTo::Types::getBase/2#0ab04984@cab7du1w
7.4s | 2946 | 287ms @ 3 | PointsTo::PointsToInternal::points_to_candidate/4#0a587a42@cab7d80w
7.1s | 2934 | 14ms @ 2 | Constants::ConstantObjectInternal.attribute/3#6d9e12fc@cab7d6zr
6.8s | 2946 | 9ms @ 48 | PointsTo::InterProceduralPointsTo::callsite_points_to/4#72419c70@cab7dqxr
6.6s | 234 | 341ms @ 17 | ApiGraphs::API::Impl::rhs/3#2255afc6@a41b31w3
6.6s | 2946 | 86ms @ 5 | PointsTo::Types::six_add_metaclass/4#f926a4cb@cab7da0w
6.2s | 2930 | 341ms @ 30 | Extensions::RangeIterationVariableFact.pointsTo/3#662720c9#cpe#124@cab7di2w
5.9s | 287 | 61ms @ 4 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@cc7b56yn
5.8s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@415f35h0
5.6s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@2c1fcdq1
5.3s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@22590ca9
5.2s | 233 | 276ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@a41b30w3
5.1s | 2945 | 177ms @ 4 | PointsTo::PointsToInternal::pointsTo/4#d99f16c6@cab7dj0w
4.7s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.6s | 277 | 2.2s @ 6 | DataFlowDispatch::getCallArg/5#21589076@cc7b5vxn
4.5s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@cc05a1fv
4.3s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@cb992b2h
4.1s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@6ae639js
4s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
3.7s | | | locations_ast_234501#join_rhs@0859685o
3.7s | 10 | 1.7s @ 1 | ObjectInternal::ObjectInternal.toString/0#dispred#0b2e9429@6e8a4yh7
3.6s | 2942 | 63ms @ 94 | PointsTo::InterProceduralPointsTo::call_points_to_from_callee/4#394022a8@cab7d90w
3.6s | 232 | 213ms @ 18 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@a41b33w3
3.6s | 2933 | 7ms @ 884 | PointsTo::Types::getInheritedMetaclass/2#097d39df#bff@cab7dr1w
3.6s | 2946 | 1.3s @ 13 | PointsTo::PointsToInternal::ssa_node_refinement_points_to/4#8ea6486b@cab7dnxr
3.5s | 1319 | 387ms @ 3 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.5s | 1320 | 385ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | 4861 | 478ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.3s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@47ba63n6
3.3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@4d16e7m6
3.3s | 2946 | 28ms @ 3 | PointsTo::PointsToInternal::reachableEdge/3#d3f53c12@cab7do7w
2.9s | 233 | 110ms @ 19 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@a41b32w3
2.8s | 31 | 2.2s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#cc7b5
2.8s | 2737 | 21ms @ 444 | MRO::ClassListList.removedClassParts/4#de59b06f#reorder_2_3_4_0_1@cab7d06w
2.8s | 1322 | 462ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.8s | 2946 | 187ms @ 5 | PointsTo::Expressions::builtinCallPointsTo/5#3aa7f48b@cab7dwwr
2.8s | 2939 | 41ms @ 7 | PointsTo::PointsToInternal::use_points_to/4#ff1d0edd@cab7df0w
2.7s | 2946 | 20ms @ 92 | PointsTo::Conditionals::evaluates/5#736734b2#fbffff#reorder_5_0_2_1_3_4@cab7dp5w
2.6s | 2946 | 152ms @ 5 | Constants::callToBool/2#0b9b1e8d@cab7dn7w
2.5s | 287 | 24ms @ 4 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@cc7b53xn
2.4s | 2946 | 31ms @ 5 | PointsTo::AttributePointsTo::variableAttributePointsTo/5#60adcc49@cab7dpwr
[2024-02-08 10:44:37] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 1231.1 seconds
* Total time spent evaluating predicates: 1167.1 seconds
```
After
```
Most expensive predicates for completed query FindUses.ql:
time | evals | max @ iter | predicate
------|-------|--------------|----------
41.6s | | | FlowSummaryImpl::Private::Steps::summaryLocalStep/3#900fb25e#ffb@85aaaac1
9.2s | 661 | 905ms @ 1 | SsaCompute::AdjacentUses::varBlockReaches/3#1824ad86@2b6af692
7.6s | 234 | 502ms @ 19 | ApiGraphs::API::Impl::rhs/3#2255afc6@ce6d11wc
6.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52_201#join_rhs@fd1dc5mi
6s | 287 | 80ms @ 113 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@925826yr
5.7s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@851052bl
5.6s | 233 | 289ms @ 21 | ApiGraphs::API::Impl::use/3#e6c88b66@ce6d10wc
5.4s | | | FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8@f2c42d17
4.8s | 277 | 2.4s @ 6 | DataFlowDispatch::getCallArg/5#21589076@92582vxr
4.7s | | | DataFlowImplCommon::Cached::viableParam/3#61239ead@ac08e0nf
4.7s | | | DataFlowImplCommon::LambdaFlow::viableParamNonLambda/3#3123cc52@82ff50ql
4.6s | | | Files::Location.toString/0#dispred#7e7e0516@b72abbo2
4.3s | | | Flow::ControlFlowNode.toString/0#dispred#e1af144b@410c23a7
4.2s | 232 | 249ms @ 19 | ApiGraphs::API::Impl::trackDefNode/2#8e3c4e6d@ce6d13wc
3.8s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#shared@0ac73425
3.6s | 1319 | 354ms @ 1 | SsaCompute::SsaDefinitions::reachesEndOfBlock/4#214bd902@fce54web
3.6s | 1320 | 381ms @ 2 | SsaCompute::SsaDefinitions::reachesEndOfBlockRec/4#63bb2cd4@fce54xeb
3.4s | | | _AstExtended::AstNode.getLocation/0#dispred#6b4dcb62_10#join_rhs_DataFlowPublic::Node.getLocation/0#__#higher_order_body@9e946ea8
3.4s | 4861 | 474ms @ 2 | SsaCompute::SsaComputeImpl::ssaDefReachesRank/4#f19c6fee@cc8515rd
3.1s | 31 | 2.5s @ 9 | _Class::Class.getAMethod/0#dispred#66416e47_DataFlowDispatch::findFunctionAccordingToMroKnownStartin__#antijoin_rhs@L6#92582
3s | 53 | 114ms @ 48 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@9ab38jw0
3s | 233 | 126ms @ 20 | ApiGraphs::API::Impl::trackUseNode/2#a0b4384d@ce6d12wc
3s | | | locations_ast_234501#join_rhs@0859685o
3s | | | DataFlowPublic::Node.toString/0#dispred#af9c307a@a2145cqf
2.8s | 234 | 206ms @ 21 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs#1@L9#ce6d1
2.8s | 1322 | 447ms @ 4 | SsaCompute::Liveness::liveAtExit/2#b6aa63f4@6fd4cx73
2.7s | 230 | 176ms @ 28 | ApiGraphs::API::Impl::MkDef#51c2f877@ce6d1w9c
2.5s | 287 | 50ms @ 112 | DataFlowDispatch::resolveClassInstanceCall/3#6e09c292@925823xr
2.4s | 234 | 246ms @ 19 | _ApiGraphs::API::Impl::MkDef#51c2f877#prev_ApiGraphs::API::Impl::trackDefNode/1#7e78e336#prev_delta___#antijoin_rhs@L4#ce6d1
2.3s | | | TaintTrackingPrivate::localAdditionalTaintStep/2#a2ec8c9d@e31201hd
2.2s | 53 | 72ms @ 15 | DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@96b28jwo
2.2s | | | SensitiveDataSources::SensitiveDataModeling::sensitiveString/1#fdc3ad40@41f6ee2g
2s | | | DataFlowImplCommon::Cached::viableParamArg/3#4c55eddb@8f7f25oq
2s | | | Flow::ControlFlowNode.getExprChild/1#e757d179#bbf@db51e8ed
1.9s | | | project#FlowSummaryImpl::Private::Steps::viableParam/4#49c13ab8#2@e36c2dr8
1.9s | | | DataFlowPublic::Node.hasLocationInfo/5#dispred#b79d995f@6e929dfv
1.7s | 15 | 433ms @ 1 | PoorMansFunctionResolution::poorMansFunctionTracker/2#75430e01@e5202dnv
1.7s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@60d9daea
1.7s | 29 | 633ms @ 6 | _Class::Class.getAMethod/0#dispred#66416e47_Function::Function.getName/0#dispred#033700ef_10#join_rh__#antijoin_rhs@L4#92582
1.5s | 233 | 79ms @ 24 | ApiGraphs::API::Impl::trackUseNode/1#1af3a9ea@ce6d16wc
1.5s | | | ApiGraphs::API::Impl::edge/3#8453bf65@1bd8a6ja
1.5s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1@5dbb806u
1.3s | 1323 | 178ms @ 13 | SsaCompute::Liveness::liveAtEntry/2#bab3ea7c@6fd4cw73
1.3s | | | SsaCompute::SsaComputeImpl::defUseRank/4#782a2f48@0f27919s
1.3s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@96b65frd
1.3s | | | ApiGraphs::API::Node.getAValueReachableFromSource/0#dispred#9a406fb1_10#join_rhs@c1dd43nv
1.3s | | | FlowSummaryImpl::Private::SummaryNode.toString/0#dispred#d499e234@63bd684g
1.2s | | | DataFlowDispatch::LibraryCallable.getACall/0#dispred#66a01171#fb@eaebb27g
1.2s | | | _DataFlowPublic::Node#da3b6093_DataFlowPublic::Node.asExpr/0#dispred#2845197a_py_exprs#antijoin_rhs@fcd8c3kj
1.2s | | | #ImportResolution::ImportResolution::allowedEssaImportStep/2#f4117c61Plus#swapped@c3f634us
[2024-02-08 11:43:50] Total evaluation times for this run:
* Wall-clock duration of evaluation run: 636.9 seconds
* Total time spent evaluating predicates: 562.4 seconds
```
2024-02-08 12:20:56 +01:00
github-actions[bot]
b5139078d0
Post-release preparation for codeql-cli-2.16.2
2024-02-06 19:22:35 +00:00
github-actions[bot]
c1b35fbf47
Release preparation for version 2.16.2
2024-02-05 17:58:57 +00:00
James Ockers
9f7f9fcc6e
Updating change-notes to reflect what will be the visible change to end users
2024-02-02 11:38:17 -08:00
James Ockers
0f1e21aa09
Adding per-language change-notes
2024-01-30 17:28:34 -08:00
James Ockers
eb5e0123d6
exclude certification from maybeCertificate() regexes
2024-01-30 13:16:18 -08:00
Sid Shankar
f557110d9b
Merge pull request #15465 from sidshank/sidshank/rename-file-name-for-extracted-files-diagnostic
...
JS/TS/Python/Ruby: Renames diagnostic query files and tests
2024-01-30 10:19:00 -05:00
Rasmus Wriedt Larsen
c265c15f3f
Merge pull request #15398 from RasmusWL/html-escape
...
Python: Add `html.escape` as HTML sanitizer
2024-01-30 16:06:01 +01:00
Rasmus Wriedt Larsen
c70b32f7eb
Python: Require quote escaping for html.escape
2024-01-30 12:17:01 +01:00
Sid Shankar
b1d7a635f5
Renames diagnostic query files and tests
...
This commit renames the files relating to the diagnostic query that produces information on the number of files extracted. The files have been renamed from "SuccessfullExtractedFiles.*" to "ExtractedFiles.*". All related tests and test files have been renamed too.
The `@tags` and `@id` attributes of the queries have been left untouched, consistent with the `@tags` and `@id` for similar queries in other languages.
2024-01-29 20:19:20 +00:00
Rasmus Wriedt Larsen
5867fb3d29
Python: Add change-note
2024-01-29 14:30:20 +01:00
Rasmus Wriedt Larsen
3f0dc2b022
Python: Model the psycopg package
2024-01-29 14:30:20 +01:00
yoff
391ca5d8a6
Merge pull request #15390 from Marcono1234/marcono1234/python-ascii-regex-flag
2024-01-29 14:27:50 +01:00
Marcono1234
1ad08efe08
Python: Support a (ASCII) inline regex flag
2024-01-26 22:18:49 +01:00
Henry Mercer
10343dd822
Merge pull request #15416 from github/post-release-prep/codeql-cli-2.16.1
...
Post-release preparation for codeql-cli-2.16.1
2024-01-25 14:15:25 +00:00
yoff
930f1b50b9
Merge pull request #15397 from github/tausbn/python-fix-deepcopy-mutable-default-fp
...
Python: Fix `deepcopy` mutable default FP
2024-01-25 10:32:58 +01:00
Taus
96b1b8e402
Python: Remove empty lines from test file
2024-01-24 12:31:23 +00:00
github-actions[bot]
d0b74c00fe
Post-release preparation for codeql-cli-2.16.1
2024-01-23 23:02:29 +00:00
github-actions[bot]
7ef611e6dc
Release preparation for version 2.16.1
2024-01-23 19:45:16 +00:00
Erik Krogh Kristensen
f1d6f56621
Merge pull request #15393 from erik-krogh/deps-jan-2024
...
All: delete outdated deprecations
2024-01-23 13:52:38 +01:00
erik-krogh
865df920f9
add change-notes
2024-01-22 19:30:57 +01:00
Rasmus Wriedt Larsen
00dc55d825
Python: Add change-note
2024-01-22 17:32:33 +01:00
Rasmus Wriedt Larsen
cbed6e861d
Python: Add html.escape as HTML sanitizer
2024-01-22 17:32:28 +01:00
Taus
d6d59377d3
Python: Fix flow through deepcopy
...
Or, more generally, any copy step, as these presumably do not preserve
object identity.
(Arguably, `copy` could still be susceptible to interior mutability, but
I think that's outside the scope of this query anyway.)
2024-01-22 15:40:30 +00:00
Taus
14c958ac4d
Python: Remove mutable default sources from inside stdlib
2024-01-22 15:23:52 +00:00
Taus
411c107660
Python: Add tests for deepcopy FPs
...
There are two issues with `deepcopy` here. Firstly, the `deepcopy` function itself
has a mutable default value in its parameter `_nil` (set to the empty list by default).
Now, this value is never actually returned from `deepcopy`, as it is only used as a
sentinel, but our analysis is not clever enough to see this. Thus, it thinks that this
mutable default is returned, and hence the result of any call to `deepcopy` is a
potential source.
To remedy this, I opted to simply exclude all sources that originate from within the
standard library. It is very unlikely for any of the sources in the standard library
to be legit.
Secondly, `deepcopy` -- by virtue of being a function that we model as preserving
values -- admits data-flow through its calls, but this is not correct for the mutable
default query, as it is here the _identity_ of the default value in question that is
important. Thus, we get spurious flow through `deepcopy` for this specific query.
2024-01-22 15:21:57 +00:00
Taus
4742481070
Python: Consolidate "mutable default" tests
...
Moves the existing tests into the `ModificationOfParameterWithDefault` subdirectory
which already contained a bunch more tests. In the process, I also removed some
duplicated test cases.
2024-01-22 13:50:33 +00:00
Max Schaefer
a4639c7ff9
Update qhelp to mention solution using urlparse.
2024-01-22 13:36:12 +00:00
Max Schaefer
17e3a45ad7
Apply suggestions from code review
...
Co-authored-by: Taus <tausbn@github.com >
2024-01-22 13:36:12 +00:00
Max Schaefer
98178458d0
Python: Add support for more URL redirect sanitisers.
...
Since some sanitisers don't handle backslashes correctly, I updated the data-flow configuration to incorporate a flow state tracking whether or not backslashes have been eliminated or converted to forward slashes.
2024-01-22 13:24:18 +00:00
erik-krogh
f60c01e3a8
Py: delete import that no longer exists
2024-01-22 09:22:50 +01:00
erik-krogh
8be7eadace
delete outdated deprecations
2024-01-22 09:11:35 +01:00
Sid Shankar
2c683c910f
Merge branch 'change/adjust-extracted-files-diagnostics' of https://github.com/sidshank/codeql into change/adjust-extracted-files-diagnostics
2024-01-17 14:32:36 +00:00
Sid Shankar
0824ab77e9
Adds change notes
2024-01-17 14:31:40 +00:00
Sid Shankar
59098be8c4
Merge branch 'main' into change/adjust-extracted-files-diagnostics
2024-01-16 21:51:41 -05:00
Alexander Eyers-Taylor
934474681d
Merge pull request #15254 from github/post-release-prep/codeql-cli-2.16.0
...
Post-release preparation for codeql-cli-2.16.0
2024-01-16 14:50:40 +00:00
github-actions[bot]
57df8b92df
Post-release preparation for codeql-cli-2.16.0
2024-01-15 15:00:50 +00:00
