Commit Graph

29908 Commits

Author SHA1 Message Date
Mathias Vorreiter Pedersen
e0b876d2f6 Merge pull request #7102 from MathiasVP/fix-map-test
C++: Fix a testcase
2021-11-10 13:51:10 +00:00
yoff
d23a920ed4 Merge branch 'main' into python/model-aiomysql 2021-11-10 14:32:36 +01:00
Rasmus Lerchedahl Petersen
57e7bfbdba Python: model aiomysql 2021-11-10 14:29:39 +01:00
Rasmus Lerchedahl Petersen
047cff0749 Python: test aiomysql 2021-11-10 14:24:45 +01:00
Mathias Vorreiter Pedersen
ccdaf49464 C++: Fix the same bug in the test for ordered maps. 2021-11-10 13:24:27 +00:00
Erik Krogh Kristensen
55434653f5 add CWE-532 to the clear-text-logging query 2021-11-10 14:15:49 +01:00
Erik Krogh Kristensen
98da532c46 dont extract regular expressions from strings that are leaves in a string concat 2021-11-10 14:11:48 +01:00
Mathias Vorreiter Pedersen
86d78b34aa C++: Use the correct variable in the 'test'. 2021-11-10 13:04:48 +00:00
Rasmus Wriedt Larsen
de926dc2a1 Merge pull request #7085 from yoff/python/model-aiopg
Python: model aiopg
2021-11-10 13:10:30 +01:00
Rasmus Wriedt Larsen
1f90dcadf1 Python: Use backtracker for verify arg 2021-11-10 12:54:23 +01:00
Geoffrey White
2f39c64cc2 C++: Fix character in qhelp. 2021-11-10 11:23:57 +00:00
Benjamin Muskalla
0f086056a1 Merge pull request #7100 from bmuskalla/bmuskalla/ioAsFile
Java: Extract Commons IO into seperate file
2021-11-10 12:04:12 +01:00
Max Schaefer
a8c4455b20 Factor out an auxiliary predicate. 2021-11-10 10:17:59 +00:00
Rasmus Lerchedahl Petersen
92a7114b72 Python: Add API references 2021-11-10 11:06:58 +01:00
Rasmus Lerchedahl Petersen
c6d285dd2a Python: Fix test 2021-11-10 11:06:45 +01:00
yoff
a856395d56 Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-10 10:51:40 +01:00
Rasmus Wriedt Larsen
77099fe9d0 Python: Always use @id py/
The two queries in CWE-020 are used for manual evaluation (is my
understanding), and the two IDE queries should work based on their tags,
and not on the query-id.
2021-11-10 10:40:17 +01:00
Benjamin Muskalla
f9fa22c14d Removed unused import 2021-11-10 10:21:54 +01:00
Benjamin Muskalla
1a751608de Extract Commons IO into seperate file 2021-11-10 10:15:27 +01:00
Mathias Vorreiter Pedersen
e2ab1c8c5e Merge branch 'main' into use-range-analysis-in-buffer-write 2021-11-10 08:28:43 +00:00
Tony Torralba
4da1dce811 Merge pull request #7099 from github/workflow/coverage/update
Update CSV framework coverage reports
2021-11-10 08:54:11 +01:00
Erik Krogh Kristensen
ab5d9459c7 Update javascript/ql/src/Security/CWE-384/SessionFixation.qhelp
Co-authored-by: Ethan Palm <56270045+ethanpalm@users.noreply.github.com>
2021-11-10 08:24:46 +01:00
ihsinme
a0448240aa Update InsecureTemporaryFile.expected 2021-11-10 09:23:51 +03:00
ihsinme
7514fe2b45 Update test.cpp 2021-11-10 09:22:58 +03:00
ihsinme
289d58745a Update InsecureTemporaryFile.ql 2021-11-10 09:22:03 +03:00
github-actions[bot]
f5426336c3 Add changed framework coverage reports 2021-11-10 00:09:06 +00:00
Taus
33135e909a Python: Add magic to named_argument_transfer
This predicate was materialised as a _big_, _cached_ relation:

```
(169s) Tuple counts for PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs/4@38ce07 after 53.4s:
25212     ~4%     {3} r1 = SCAN Function::Function::getArgByName_dispred#fff OUTPUT In.1, In.0 'arg1', In.2 'arg2'
159751200 ~0%     {4} r2 = JOIN r1 WITH Flow::CallNode::getArgByName_dispred#fff_102#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Lhs.1 'arg1', Lhs.2 'arg2', Rhs.2 'arg3'
                  return r2
```

... However it's only used in a single place (where it is immediately
joined with the points-to relation to relate the caller and argument),
none of these joins were ever larger than 2000 tuples. This made it
pretty clear that we could gain something by pushing in that points-to
join as a bit of manual magic.

However, doing so didn't actually fix anything, since the join-orderer
then decided to join `func.getArgByName(name)` with
`call.getArgByName(name)` on `name` as the first thing (which caused a
join of the same size as above).

Unbinding didn't work, since `name` would then be an unbound `string`,
so instead I factored out relating the function, parameter, and name
thereof into its own predicate. (I could also have done this with the
call, but I would expect there to be more calls than function
definitions in general.)

Overall, this resulted in going from

```
(709s)
Definitions.ql-7:PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs ......... 53.5s
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ............................... 35.3s (456 evaluations with max 136ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#0508e8)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf .................................. 27s (100 evaluations with max 12.8s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#fc1f7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ........................................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#0508eg)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff ................................ 13.4s (505 evaluations with max 50ms in Constants::ConstantObjectInternal::attribute#ffff/4@i153#0508e5)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ....................... 12.4s (99 evaluations with max 11.8s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#fc1f
7z)
...
```

to

```
(668s)
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ................... 35.4s (456 evaluations with max 140ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#bf4328)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf ...................... 27.4s (100 evaluations with max 13.3s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#679d7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ............................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#bf432g)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff .................... 14.4s (505 evaluations with max 51ms in Constants::ConstantObjectInternal::attribute#ffff/4@i140#bf4325)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ........... 12.3s (99 evaluations with max 11.7s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#679d
7z)
...
```
2021-11-09 21:39:32 +00:00
Taus
e2f79d8516 Python: Fix several bad getScope joins
It seems the optimiser has started getting the wrong end of the stick
whenever we write `foo.getScope() = bar.getScope()` for some expressions
`foo` and `bar`.

This lead to things like

```
(196s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@5ab278 after 2m33s:
2952757013 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
495693     ~0%     {2} r2 = JOIN r1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
453589     ~0%     {2} r3 = JOIN r2 WITH Definitions::ModuleVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
                   return r3
```

and

```
(315s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@a39328 after 1m57s:
...
1785275    ~3%       {2} r24 = Definitions::ModuleVariable::global_variable_callnode#ff#shared UNION Definitions::SsaSourceVariable::getAUse_dispred#ff#shared
3008614987 ~0%       {2} r25 = JOIN r24 WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
127        ~1%       {2} r26 = JOIN r25 WITH Definitions::NonLocalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
127        ~1%       {2} r27 = JOIN r26 WITH Variables::LocalVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
...
```

(Note the timings: 2m33s and 1m57s.)

Now we have the much more reasonable

```
(38s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@c53031 after 42ms:
453589 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Definitions::ModuleVariable::scope_as_global_variable#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
               return r1
```

and

```
(46s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@4b19de after 375ms:
...
```
2021-11-09 20:54:41 +00:00
Taus
24000a50e6 Python: Fix bad join in py/redundant-assignment
A minor thing, but still best avoided:
```
Tuple counts for RedundantAssignment::pyflakes_commented_line#bb/2@0477c7 after 14.2s:
160826   ~0%     {2} r1 = SCAN py_comments OUTPUT In.2, In.1
160826   ~0%     {3} r2 = JOIN r1 WITH project#Files::Location::hasLocationInfo_dispred#bfffff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Rhs.2 'result'
160826   ~3%     {3} r3 = JOIN r2 WITH files_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'result', Lhs.1, Rhs.1 'file'
46672419 ~4%     {4} r4 = JOIN r3 WITH Files::Location::getStartLine_dispred#bf_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.2 'file', Lhs.1, Lhs.0 'result'
3951     ~0%     {4} r5 = JOIN r4 WITH Files::Location::getFile_dispred#bf ON FIRST 2 OUTPUT Lhs.2, Lhs.3 'result', Lhs.1 'file', toLower(Lhs.2)
0        ~0%     {4} r6 = SELECT r5 ON In.3 contains "pyflakes"
0        ~0%     {2} r7 = SCAN r6 OUTPUT In.2 'file', In.1 'result'
                return r7
```
2021-11-09 20:31:44 +00:00
Geoffrey White
26e9adcc34 C++: Change note. 2021-11-09 19:39:21 +00:00
Geoffrey White
ef21d1b512 C++: Add a model for curl as well. 2021-11-09 19:32:43 +00:00
Geoffrey White
6388ac5f1d C++: Add tests. 2021-11-09 18:41:57 +00:00
Geoffrey White
d2b18d952d C++: Add qhelp. 2021-11-09 18:41:56 +00:00
Geoffrey White
bd1e708c5d C++: First version of cpp/non-https-url. 2021-11-09 18:33:49 +00:00
Rasmus Wriedt Larsen
985cd1ebdb Python: Port py/request-without-cert-validation to use API graphs 2021-11-09 16:37:50 +01:00
Rasmus Wriedt Larsen
59581690fd Python: Add py/request-without-cert-validation tests 2021-11-09 16:29:57 +01:00
Rasmus Wriedt Larsen
9710aeecbf Python/C#: Add CWE-1333 to redos queries
As is already done in JS and Ruby.
2021-11-09 16:10:38 +01:00
Anders Schack-Mulligen
1efe1e0d10 Java: Improve algorithm for subtyping of parameterized types. 2021-11-09 15:49:17 +01:00
Tom Hvitved
7178a98e45 Ruby: Rename pruneUseNode{Fwd,Rev} 2021-11-09 15:16:36 +01:00
Tom Hvitved
30251740e3 Ruby: Prune nodes before computing trackUseNode 2021-11-09 15:16:36 +01:00
Tom Hvitved
8195ebf4b3 Merge pull request #7059 from hvitved/ruby/basic-store-step-postupdate
Ruby: Fix `basicStoreStep`
2021-11-09 15:16:07 +01:00
Alex Ford
556cdbaa21 ruby: QL format 2021-11-09 14:09:11 +00:00
Alex Ford
37775407a9 ruby: drop a redundant bit of documentation 2021-11-09 14:07:00 +00:00
Alex Ford
340897f262 ruby: drop unnecessary variable 2021-11-09 14:06:21 +00:00
Alex Ford
a23750a9c7 ruby: inline some predicates 2021-11-09 14:06:21 +00:00
Benjamin Muskalla
40e47c0ea3 Merge pull request #7082 from bmuskalla/filterOutputStream
Java: Model taint for `FilterOutputStream`
2021-11-09 15:06:15 +01:00
Alex Ford
c65d1d9a50 ruby: CSRFProtectionDisabled.qhelp fixes
Co-authored-by: Harry Maclean <hmac@github.com>
2021-11-09 14:05:41 +00:00
Rasmus Wriedt Larsen
f70e4fea55 Python: Add interesting path-injection FP 2021-11-09 14:53:32 +01:00
Mathias Vorreiter Pedersen
10bca3544c C++: Change 'annotate_path_to_sink' so that you now annotate a ir-path with the previous node (instead of its source). This gives a better overview of the path. 2021-11-09 13:49:12 +00:00
Benjamin Muskalla
bfe2e2e0b9 Model taint for FilterOutputStream 2021-11-09 14:21:50 +01:00