Commit Graph

352 Commits

Author SHA1 Message Date
Taus
a716482c1f Python: Fix bad join in SSA
On `pritomrajkhowa/LoopBound`:

```
Definitions.ql-3:SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff ................. 4m35s
```

specifically

```
(376s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@be04e9kp after 4m58s:
388843     ~0%     {4} r1 = JOIN Essa::TPhiFunction#fff_2#join_rhs WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Rhs.2, Rhs.3
3629812090 ~1%     {7} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::variableUse#ffff ON FIRST 1 OUTPUT Lhs.0, Rhs.2, Rhs.3, Lhs.2, Lhs.3, Lhs.1, Rhs.1 'use1'
0          ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 5 OUTPUT Lhs.5, Lhs.6 'use1'
0          ~0%     {2} r4 = JOIN r3 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141     ~0%     {2} r5 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r4
                    return r5
```

Clearly we do not want to join on the variable so soon. So we unbind it
and get

```
(78s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@40e0e6uv after 434ms:
3377959 ~2%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::variableUse#ffff OUTPUT In.0, In.2, In.3, In.1 'use1'
1026855 ~2%     {4} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 3 OUTPUT Lhs.0, Rhs.3, Rhs.4, Lhs.3 'use1'
129484  ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff_1230#join_rhs ON FIRST 3 OUTPUT Rhs.3, Lhs.3 'use1'
0       ~0%     {2} r4 = JOIN r3 WITH Essa::TPhiFunction#fff_2#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'use1'
0       ~0%     {2} r5 = JOIN r4 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141  ~0%     {2} r6 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r5
                return r6
```
2021-12-07 18:19:47 +00:00
Erik Krogh Kristensen
3c59aa319e Merge pull request #7245 from erik-krogh/explicit-this-all-the-places
All langs: apply the explicit-this patch to all remaining code
2021-12-07 10:40:26 +01:00
Taus
7cd9369d91 Python: Autoformat 2021-12-07 09:29:24 +00:00
Taus
dd33f4f4d2 Python: Apply suggestions from code review
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-12-07 09:48:53 +01:00
Taus
7f44cebed7 Python: Add missing hidden flow
The easiest way to implement this was to change the definition of
`module_export` to account for chains of `import *`. We reuse the
machinery from `ImportStar.qll` for this, naturally.
2021-12-02 17:11:56 +00:00
Nick Rolfe
05415768c9 Merge remote-tracking branch 'origin/main' into nickrolfe/regexp_g_anchor 2021-12-02 12:07:13 +00:00
yoff
f10f053c36 Merge pull request #7228 from RasmusWL/fastapi-improvements
Python: FastAPI improvements
2021-12-02 12:58:53 +01:00
github-actions[bot]
87b968f337 Post-release preparation 2.7.3 2021-12-02 00:46:55 +00:00
Tom Hvitved
e410244fe0 Python: Implement ParameterPosition et al 2021-12-01 08:51:22 +01:00
github-actions[bot]
337ce65fe5 Release preparation for version 2.7.3 2021-11-30 20:39:35 +00:00
Tom Hvitved
540ecf3c21 Data flow: Sync files 2021-11-30 15:20:20 +01:00
Anders Schack-Mulligen
3e914ef2ff Dataflow: Sync. 2021-11-30 13:52:52 +01:00
Dave Bartolomeo
9f6c0991cf Catch up with recent change notes 2021-11-29 16:41:18 -05:00
Dave Bartolomeo
5ed9029143 Move change notes to correct directories 2021-11-29 16:31:11 -05:00
Dave Bartolomeo
d0dac03bad Manually bump versions 2021-11-29 14:21:08 -05:00
Dave Bartolomeo
2dfcd1dd9c Add groups property
Also removed versions from test packs
2021-11-29 14:15:53 -05:00
Rasmus Wriedt Larsen
cbd7434a7e Python: Add modeling of tempfile module 2021-11-29 15:08:36 +01:00
Rasmus Wriedt Larsen
58f92764f7 Python: Model more file access from os module 2021-11-29 14:54:02 +01:00
Rasmus Wriedt Larsen
fd23fa94a5 Python: Remove dubious fstat* modeling
These operate on file descriptors, and not on paths. file descriptors
doesn't fit into the rest of our modeling, so I would rather remove them
than to make it look like it's properly handled.

I also did not include any of the functions that work on file
descriptors when looking through all of `os`. So this keeps everything
consistent at least ;)
2021-11-29 14:54:02 +01:00
Rasmus Wriedt Larsen
e79b8f3e23 Python: Treat os.exec*, os.spawn*, and os.posix_spawn* as FileSystemAccess 2021-11-29 14:54:02 +01:00
Rasmus Wriedt Larsen
d2d5cce787 Python: Recognize keyword arguments for os.*spawn* calls 2021-11-29 14:54:02 +01:00
Rasmus Wriedt Larsen
a91208fd2c Python: Fix kwarg modeling for os.path.isdir 2021-11-29 14:54:02 +01:00
Taus
6c3aabe1df Python: Support flow through import *
Adds result for `ModuleVariableNode::getARead` corresponding to reads
that go through (chains of) `import *`.

This required a bit of a change to _which_ module variables we define.
Previously, we only included variables that were accessed elsewhere in
the same file, but now we must ensure to also include variables that may
be accessed through `import *`.
2021-11-26 13:49:08 +00:00
Taus
c3e495efe9 Python: Refactor built-ins and import * logic
Moves this from the API graphs implementation into separate files.
2021-11-26 13:49:08 +00:00
Erik Krogh Kristensen
6ff8d4de5c add all remaining explicit this 2021-11-26 13:50:10 +01:00
Anders Schack-Mulligen
00ee34c0a0 Merge pull request #7237 from hvitved/dataflow/consistency-config
Data flow: Introduce `ConsistencyConfiguration` class
2021-11-26 12:49:25 +01:00
Anders Schack-Mulligen
a06642944f Merge pull request #7232 from aschackmull/dataflow/perf
Data flow: Performance tuning
2021-11-25 15:01:01 +01:00
Tom Hvitved
6cb00992e8 Data flow: Introduce ConsistencyConfiguration class 2021-11-25 10:01:47 +01:00
Erik Krogh Kristensen
3bab8c6d1d Merge pull request #7173 from erik-krogh/getRubyInSync
JS/PY/RB: get ReDoSUtil in sync for ruby
2021-11-24 15:20:23 +01:00
Anders Schack-Mulligen
7ca3407c86 Dataflow: Sync. 2021-11-24 14:43:00 +01:00
Rasmus Wriedt Larsen
7dde52ced2 Merge pull request #7131 from RasmusWL/wsgiref.simple_server
Python: Model `wsgiref.simple_server` applications
2021-11-24 14:22:23 +01:00
Rasmus Wriedt Larsen
e2652591a5 Python: Change perf fix PoorMansFunctionResolution
Thanks @yoff, this leaves us with the following evaluation, which looks
very close to the one in the other fix (but with cleaner implementation)
-- both at 688k max tuples (although numbers are not exactly the same).

```
[2021-11-24 13:48:40] (14s) Tuple counts for PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass#ff/2@e5f05asv after 74ms:
                      47493  ~3%     {3} r1 = JOIN Class::Class::getAMethod_dispred#ff WITH py_Classes ON FIRST 1 OUTPUT Lhs.1, 0, Lhs.0
                      47335  ~0%     {2} r2 = JOIN r1 WITH AstGenerated::Function_::getArg_dispred#fff ON FIRST 2 OUTPUT Rhs.2, Lhs.2
                      46683  ~0%     {2} r3 = JOIN r2 WITH DataFlowPublic::ParameterNode::getParameter_dispred#fb_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1
                      259968 ~4%     {2} r4 = JOIN r3 WITH LocalSources::Cached::hasLocalSource#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1
                      161985 ~0%     {3} r5 = JOIN r4 WITH Attributes::AttrRef::accesses_dispred#bff_102#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1, Rhs.2
                      161985 ~2%     {3} r6 = JOIN r5 WITH Attributes::AttrRead#class#f ON FIRST 1 OUTPUT Lhs.2, Lhs.1, Lhs.0 'result'
                      688766 ~0%     {3} r7 = JOIN r6 WITH Function::Function::getName_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Rhs.1 'func', Lhs.2 'result'
                      20928  ~0%     {2} r8 = JOIN r7 WITH Class::Class::getAMethod_dispred#ff ON FIRST 2 OUTPUT Lhs.1 'func', Lhs.2 'result'
                                     return r8
```
2021-11-24 13:52:05 +01:00
Rasmus Wriedt Larsen
1411804e58 Python: Allow custom fastapi.APIRouter subclasses 2021-11-24 13:46:38 +01:00
Rasmus Wriedt Larsen
47448d9efc Python: Apply suggestions from code review
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-11-24 12:02:12 +01:00
Rasmus Wriedt Larsen
d493cfdf3a Python: Model FastAPI FileResponse as FileSystemAccess
This was an oversight from our initial FastAPI modeling work.
2021-11-24 11:44:51 +01:00
yoff
f9729bccef Merge pull request #7143 from RasmusWL/path-improvements
Python: Model `posixpath` and `os.stat`
2021-11-24 11:36:06 +01:00
Rasmus Wriedt Larsen
eaed870b31 Python: Fix performance problem in PoorMansFunctionResolution
Before these changes:

[2021-11-22 12:02:50] (8s) Tuple counts for PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass#ff/2@cbddf257 after 8.6s:
                      387565   ~0%     {3} r1 = JOIN Attributes::AttrRead#class#f WITH Attributes::AttrRef::accesses_dispred#bff ON FIRST 1 OUTPUT Rhs.2, Lhs.0 'result', Rhs.1
                      6548632  ~0%     {3} r2 = JOIN r1 WITH Function::Function::getName_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'func', Lhs.1 'result', Lhs.2
                      5640480  ~0%     {4} r3 = JOIN r2 WITH Class::Class::getAMethod_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'result', Lhs.2, Lhs.0 'func'
                      55660458 ~0%     {5} r4 = JOIN r3 WITH Class::Class::getAMethod_dispred#ff ON FIRST 1 OUTPUT Rhs.1, 0, Lhs.1 'result', Lhs.2, Lhs.3 'func'
                      55621412 ~0%     {4} r5 = JOIN r4 WITH AstGenerated::Function_::getArg_dispred#fff ON FIRST 2 OUTPUT Rhs.2, Lhs.2 'result', Lhs.3, Lhs.4 'func'
                      54467144 ~0%     {4} r6 = JOIN r5 WITH DataFlowPublic::ParameterNode::getParameter_dispred#fb_10#join_rhs ON FIRST 1 OUTPUT Lhs.2, Rhs.1, Lhs.1 'result', Lhs.3 'func'
                      20928    ~0%     {2} r7 = JOIN r6 WITH LocalSources::Cached::hasLocalSource#ff ON FIRST 2 OUTPUT Lhs.3 'func', Lhs.2 'result'
                                       return r7

With these changes:

[2021-11-22 11:54:25] (415s) Tuple counts for PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass_helper#fff/3@14db70a8 after 75ms:
                      388306 ~0%     {2} r1 = JOIN Attributes::AttrRead#class#f WITH Attributes::AttrRef::getObject_dispred#bf ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'read'
                      379420 ~4%     {2} r2 = JOIN r1 WITH LocalSources::Cached::hasLocalSource#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'read'
                      175082 ~0%     {2} r3 = JOIN r2 WITH DataFlowPublic::ParameterNode#class#fff ON FIRST 1 OUTPUT Rhs.2, Lhs.1 'read'
                      175082 ~2%     {3} r4 = JOIN r3 WITH Essa::ParameterDefinition::getParameter_dispred#ff ON FIRST 1 OUTPUT 0, Rhs.1, Lhs.1 'read'
                      166798 ~0%     {2} r5 = JOIN r4 WITH AstGenerated::Function_::getArg_dispred#fff_120#join_rhs ON FIRST 2 OUTPUT Rhs.2 'func', Lhs.2 'read'
                      162096 ~0%     {3} r6 = JOIN r5 WITH Class::Class::getAMethod_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0 'func', Rhs.1 'cls', Lhs.1 'read'
                                     return r6

[2021-11-22 11:54:25] (415s) Tuple counts for PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass_helper2#ffff/4@2b60f0s9 after 63ms:
                      162046 ~0%     {3} r1 = SCAN PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass_helper#fff OUTPUT In.2 'read', In.0 'func', In.1 'cls'
                      162046 ~0%     {3} r2 = JOIN r1 WITH Attributes::AttrRead#class#f ON FIRST 1 OUTPUT Lhs.1 'func', Lhs.2 'cls', Lhs.0 'read'
                      162046 ~1%     {3} r3 = JOIN r2 WITH py_Functions ON FIRST 1 OUTPUT Lhs.1 'cls', Lhs.2 'read', Lhs.0 'func'
                      162046 ~0%     {3} r4 = JOIN r3 WITH py_Classes ON FIRST 1 OUTPUT Lhs.1 'read', Lhs.2 'func', Lhs.0 'cls'
                      161935 ~5%     {4} r5 = JOIN r4 WITH Attributes::AttrRef::getAttributeName_dispred#bf ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'read', Lhs.1 'func', Lhs.2 'cls'
                      688526 ~1%     {4} r6 = JOIN r5 WITH Function::Function::getName_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'func', Lhs.3 'cls', Lhs.1 'read', Rhs.1 'readFunction'
                                     return r6

[2021-11-22 11:54:25] (415s) Tuple counts for PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass#ff/2@f73ae6dq after 58ms:
                      688526 ~0%     {4} r1 = SCAN PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass_helper2#ffff OUTPUT In.1, In.0, In.3 'func', In.2 'result'
                      688526 ~0%     {3} r2 = JOIN r1 WITH Class::Class::getAMethod_dispred#ff ON FIRST 2 OUTPUT Rhs.0, Lhs.2 'func', Lhs.3 'result'
                      20913  ~0%     {2} r3 = JOIN r2 WITH Class::Class::getAMethod_dispred#ff ON FIRST 2 OUTPUT Lhs.1 'func', Lhs.2 'result'
                                     return r3

We need the `pragma[only_bind_into]` in getSimpleMethodReferenceWithinClass_helper2, otherwise the tuple counts would look like, which is needlessly big.

[2021-11-22 17:14:34] (2s) Tuple counts for PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass_helper2#ffff/4@5f0505h7 after 711ms:
                      13570510 ~3%     {2} r1 = JOIN Function::Function::getName_dispred#ff_10#join_rhs WITH Attributes::AttrRef::getAttributeName_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'read', Lhs.1 'readFunction'
                      688526   ~1%     {4} r2 = JOIN r1 WITH PoorMansFunctionResolution::getSimpleMethodReferenceWithinClass_helper#fff_201#join_rhs ON FIRST 1 OUTPUT Rhs.1 'func', Rhs.2 'cls', Lhs.0 'read', Lhs.1 'readFunction'
                                       return r2
2021-11-22 17:22:39 +01:00
Rasmus Wriedt Larsen
f09f1c4c50 Python: Minor refactor in PoorMansFunctionResolution 2021-11-22 11:11:29 +01:00
Nick Rolfe
df6ba43cca Python: treat \A, \Z, \b, \B as special chars, not escapes 2021-11-19 15:49:53 +00:00
Erik Krogh Kristensen
ee858d840e get ReDoSUtil in sync for ruby 2021-11-18 16:49:34 +01:00
Erik Krogh Kristensen
1cca377e7d Merge pull request #6561 from erik-krogh/htmlReg
JS/Py/Ruby: add a bad-tag-filter query
2021-11-18 09:39:13 +01:00
Anders Schack-Mulligen
c70d384d28 Merge pull request #7045 from aschackmull/dataflow/hidden-ret-subpaths
Data flow: Support hidden return nodes in subpaths predicate
2021-11-16 15:04:51 +01:00
Rasmus Wriedt Larsen
a980f26fda Python: Model os.stat (and friends) 2021-11-16 10:45:32 +01:00
Rasmus Wriedt Larsen
9f4107d211 Python: Model posixpath, ntpath, and genericpath modules 2021-11-16 10:45:14 +01:00
Rasmus Wriedt Larsen
6b7abacc5f Merge pull request #7135 from RasmusWL/b32hexencode
Python: Model `b32hexencode`/`b32hexdecode`
2021-11-15 15:51:46 +01:00
Rasmus Wriedt Larsen
39927fa613 Python: Model b32hexencode/b32hexdecode
New in Python 3.10

See
- https://devdocs.io/python~3.10/library/base64#base64.b32hexencode
- https://devdocs.io/python~3.10/library/base64#base64.b32hexdecode
2021-11-15 15:23:49 +01:00
Rasmus Wriedt Larsen
cfdfcaa3e8 Python: Support Path.hardlink_to (new in 3.10)
See https://docs.python.org/3.10/library/pathlib.html#pathlib.Path.hardlink_to
2021-11-15 14:57:59 +01:00
Rasmus Wriedt Larsen
5d60975f65 Python: Support aiter and anext (new in 3.10)
See
- https://docs.python.org/3/whatsnew/3.10.html#other-language-changes
- https://docs.python.org/3.10/library/functions.html#aiter
- https://docs.python.org/3.10/library/functions.html#anext
2021-11-15 14:55:34 +01:00
Rasmus Wriedt Larsen
9e097f5430 Python: Improve PoorMansFunctionResolution 2021-11-15 13:40:19 +01:00
Rasmus Wriedt Larsen
6eb4525ab2 Python: Model wsgiref.simple_server applications 2021-11-15 13:34:39 +01:00