Commit Graph

29199 Commits

Author SHA1 Message Date
Taus
33135e909a Python: Add magic to named_argument_transfer
This predicate was materialised as a _big_, _cached_ relation:

```
(169s) Tuple counts for PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs/4@38ce07 after 53.4s:
25212     ~4%     {3} r1 = SCAN Function::Function::getArgByName_dispred#fff OUTPUT In.1, In.0 'arg1', In.2 'arg2'
159751200 ~0%     {4} r2 = JOIN r1 WITH Flow::CallNode::getArgByName_dispred#fff_102#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Lhs.1 'arg1', Lhs.2 'arg2', Rhs.2 'arg3'
                  return r2
```

... However it's only used in a single place (where it is immediately
joined with the points-to relation to relate the caller and argument),
none of these joins were ever larger than 2000 tuples. This made it
pretty clear that we could gain something by pushing in that points-to
join as a bit of manual magic.

However, doing so didn't actually fix anything, since the join-orderer
then decided to join `func.getArgByName(name)` with
`call.getArgByName(name)` on `name` as the first thing (which caused a
join of the same size as above).

Unbinding didn't work, since `name` would then be an unbound `string`,
so instead I factored out relating the function, parameter, and name
thereof into its own predicate. (I could also have done this with the
call, but I would expect there to be more calls than function
definitions in general.)

Overall, this resulted in going from

```
(709s)
Definitions.ql-7:PointsTo::InterProceduralPointsTo::named_argument_transfer#ffff#join_rhs ......... 53.5s
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ............................... 35.3s (456 evaluations with max 136ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#0508e8)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf .................................. 27s (100 evaluations with max 12.8s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#fc1f7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ........................................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#0508eg)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff ................................ 13.4s (505 evaluations with max 50ms in Constants::ConstantObjectInternal::attribute#ffff/4@i153#0508e5)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ....................... 12.4s (99 evaluations with max 11.8s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#fc1f
7z)
...
```

to

```
(668s)
Definitions.ql-7:Instances::InstanceObject::initializer_dispred#fbf ................... 35.4s (456 evaluations with max 140ms in Instances::InstanceObject::initializer_dispred#fbf/3@i110#bf4328)
Definitions.ql-10:DefinitionTracking::jump_to_defn_attribute#fbf ...................... 27.4s (100 evaluations with max 13.3s in DefinitionTracking::jump_to_defn_attribute#fbf/3@i1#679d7x)
Definitions.ql-7:PointsTo::PointsToInternal::pointsTo#ffff ............................ 16.1s (681 evaluations with max 2.5s in PointsTo::PointsToInternal::pointsTo#ffff/4@i4#bf432g)
Definitions.ql-7:Constants::ConstantObjectInternal::attribute#ffff .................... 14.4s (505 evaluations with max 51ms in Constants::ConstantObjectInternal::attribute#ffff/4@i140#bf4325)
Definitions.ql-10:DefinitionTracking::assignment_jump_to_defn_attribute#fbf ........... 12.3s (99 evaluations with max 11.7s in DefinitionTracking::assignment_jump_to_defn_attribute#fbf/3@i2#679d
7z)
...
```
2021-11-09 21:39:32 +00:00
Taus
e2f79d8516 Python: Fix several bad getScope joins
It seems the optimiser has started getting the wrong end of the stick
whenever we write `foo.getScope() = bar.getScope()` for some expressions
`foo` and `bar`.

This lead to things like

```
(196s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@5ab278 after 2m33s:
2952757013 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
495693     ~0%     {2} r2 = JOIN r1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
453589     ~0%     {2} r3 = JOIN r2 WITH Definitions::ModuleVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
                   return r3
```

and

```
(315s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@a39328 after 1m57s:
...
1785275    ~3%       {2} r24 = Definitions::ModuleVariable::global_variable_callnode#ff#shared UNION Definitions::SsaSourceVariable::getAUse_dispred#ff#shared
3008614987 ~0%       {2} r25 = JOIN r24 WITH Variables::Variable::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
127        ~1%       {2} r26 = JOIN r25 WITH Definitions::NonLocalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
127        ~1%       {2} r27 = JOIN r26 WITH Variables::LocalVariable#f ON FIRST 1 OUTPUT Lhs.0 'this', Lhs.1 'result'
...
```

(Note the timings: 2m33s and 1m57s.)

Now we have the much more reasonable

```
(38s) Tuple counts for Definitions::ModuleVariable::global_variable_callnode#ff/2@c53031 after 42ms:
453589 ~0%     {2} r1 = JOIN Definitions::ModuleVariable::global_variable_callnode#ff#shared WITH Definitions::ModuleVariable::scope_as_global_variable#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.1 'result'
               return r1
```

and

```
(46s) Tuple counts for Definitions::SsaSourceVariable::getAUse_dispred#ff/2@4b19de after 375ms:
...
```
2021-11-09 20:54:41 +00:00
Taus
24000a50e6 Python: Fix bad join in py/redundant-assignment
A minor thing, but still best avoided:
```
Tuple counts for RedundantAssignment::pyflakes_commented_line#bb/2@0477c7 after 14.2s:
160826   ~0%     {2} r1 = SCAN py_comments OUTPUT In.2, In.1
160826   ~0%     {3} r2 = JOIN r1 WITH project#Files::Location::hasLocationInfo_dispred#bfffff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Rhs.2 'result'
160826   ~3%     {3} r3 = JOIN r2 WITH files_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'result', Lhs.1, Rhs.1 'file'
46672419 ~4%     {4} r4 = JOIN r3 WITH Files::Location::getStartLine_dispred#bf_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.2 'file', Lhs.1, Lhs.0 'result'
3951     ~0%     {4} r5 = JOIN r4 WITH Files::Location::getFile_dispred#bf ON FIRST 2 OUTPUT Lhs.2, Lhs.3 'result', Lhs.1 'file', toLower(Lhs.2)
0        ~0%     {4} r6 = SELECT r5 ON In.3 contains "pyflakes"
0        ~0%     {2} r7 = SCAN r6 OUTPUT In.2 'file', In.1 'result'
                return r7
```
2021-11-09 20:31:44 +00:00
Geoffrey White
26e9adcc34 C++: Change note. 2021-11-09 19:39:21 +00:00
Geoffrey White
ef21d1b512 C++: Add a model for curl as well. 2021-11-09 19:32:43 +00:00
Geoffrey White
6388ac5f1d C++: Add tests. 2021-11-09 18:41:57 +00:00
Geoffrey White
d2b18d952d C++: Add qhelp. 2021-11-09 18:41:56 +00:00
Geoffrey White
bd1e708c5d C++: First version of cpp/non-https-url. 2021-11-09 18:33:49 +00:00
Rasmus Wriedt Larsen
985cd1ebdb Python: Port py/request-without-cert-validation to use API graphs 2021-11-09 16:37:50 +01:00
Rasmus Wriedt Larsen
59581690fd Python: Add py/request-without-cert-validation tests 2021-11-09 16:29:57 +01:00
Tom Hvitved
7178a98e45 Ruby: Rename pruneUseNode{Fwd,Rev} 2021-11-09 15:16:36 +01:00
Tom Hvitved
30251740e3 Ruby: Prune nodes before computing trackUseNode 2021-11-09 15:16:36 +01:00
Tom Hvitved
8195ebf4b3 Merge pull request #7059 from hvitved/ruby/basic-store-step-postupdate
Ruby: Fix `basicStoreStep`
2021-11-09 15:16:07 +01:00
Benjamin Muskalla
40e47c0ea3 Merge pull request #7082 from bmuskalla/filterOutputStream
Java: Model taint for `FilterOutputStream`
2021-11-09 15:06:15 +01:00
Rasmus Wriedt Larsen
f70e4fea55 Python: Add interesting path-injection FP 2021-11-09 14:53:32 +01:00
Benjamin Muskalla
bfe2e2e0b9 Model taint for FilterOutputStream 2021-11-09 14:21:50 +01:00
Rasmus Wriedt Larsen
1e31416049 Merge pull request #7031 from yoff/python/taint-through-with
Python: Taint through `async with`
2021-11-09 14:08:07 +01:00
Alex Ford
c708b6b76f Merge pull request #7077 from github/ruby/downgrade-hardcoded-credentials
Ruby: Downgrade `rb/hardcoded-credentials` precision from high to medium
2021-11-09 12:08:10 +00:00
Rasmus Lerchedahl Petersen
ac5a46f24f Python: split test as suggested in review 2021-11-09 13:04:52 +01:00
yoff
5f4aad40c1 Update python/ql/test/experimental/meta/InlineTaintTest.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-09 13:00:35 +01:00
Rasmus Lerchedahl Petersen
aa1541a5c3 Python: add changenote 2021-11-09 12:57:36 +01:00
Rasmus Lerchedahl Petersen
a58c47b07b Python: model aiopg.sa 2021-11-09 12:49:57 +01:00
Rasmus Lerchedahl Petersen
f53314019a Python: test aiopg.sa 2021-11-09 12:42:03 +01:00
CodeQL CI
d9d304fc13 Merge pull request #7076 from asgerf/js/tainted-path-regexp-guard2
Approved by erik-krogh
2021-11-09 03:40:37 -08:00
Rasmus Lerchedahl Petersen
cd332a75fc Python: model aiopg 2021-11-09 12:32:21 +01:00
Erik Krogh Kristensen
56a7c8b163 fix typo in change note
Co-authored-by: Asger F <asgerf@github.com>
2021-11-09 12:06:29 +01:00
Rasmus Lerchedahl Petersen
cb8f1b4593 Python: Add tests for aiopg 2021-11-09 11:49:31 +01:00
Geoffrey White
d9e02e83fe Merge pull request #6825 from MathiasVP/use-shared-ssa-in-ir-dataflow
C++: Redesign IR dataflow using the shared SSA library
2021-11-09 10:19:50 +00:00
James Fletcher
1bacce487e Merge pull request #7056 from jf205/sarif-query-help
Add new option to database analyze tutorial
2021-11-09 10:19:29 +00:00
CodeQL CI
954fd8d6f7 Merge pull request #7081 from github/revert-6924-js/skip-files-with-unsupported-encoding
Approved by esbena
2021-11-09 02:18:16 -08:00
Erik Krogh Kristensen
8727060ca7 add comment about modes of operation
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-11-09 11:15:12 +01:00
Asger Feldthaus
87aa39cef2 JS: Limited tracking of object literals with methods 2021-11-09 11:06:41 +01:00
Asger F
0c6680b2c0 Revert "JS: Skip files with unsupported file encoding" 2021-11-09 09:07:54 +00:00
ihsinme
55fe01018f Update InsecureTemporaryFile.ql 2021-11-09 09:33:33 +03:00
Asger Feldthaus
f14f9449ee JS: Use getAMatchedString instead of getConstantString 2021-11-08 15:35:35 +01:00
Asger Feldthaus
b3e64f1669 JS: Add test 2021-11-08 15:32:43 +01:00
Erik Krogh Kristensen
330c2c42b5 Merge pull request #7075 from erik-krogh/cwe297
JS: add cwe-297 to `js/disabling-certificate-validation`
2021-11-08 14:35:58 +01:00
Erik Krogh Kristensen
5cafb86c88 Merge pull request #7074 from erik-krogh/cwe942
JS: add cwe-942 to `js/cors-misconfiguration-for-credentials`
2021-11-08 14:35:53 +01:00
Rasmus Lerchedahl Petersen
3f4c2ba24e Python: Support debugging inline taint tests
The module `Conf` is created so that it can be imported
without importing the query predicates from the same file.
2021-11-08 14:08:11 +01:00
Anders Schack-Mulligen
1e0eb2f6e4 Merge pull request #7072 from aschackmull/java/nomagic-synchsetunsynchget
Java: Fix bad magic in SynchSetUnsynchGet.
2021-11-08 13:48:22 +01:00
Alex Ford
2581efc18a ruby: downgrade rb/hardcoded-credentials precision from high to medium 2021-11-08 12:32:38 +00:00
Erik Krogh Kristensen
a2175a3207 add cwe-297 to js/disabling-certificate-validation 2021-11-08 13:26:53 +01:00
Erik Krogh Kristensen
507c8addb2 add cwe-942 to js/cors-misconfiguration-for-credentials 2021-11-08 13:12:19 +01:00
Cornelius Riemenschneider
76d2665132 Merge pull request #7071 from github/criemen/simplify-csharp-tracing-config
C#: Remove macos compatibility stanzas from tracing config.
2021-11-08 13:11:44 +01:00
james
96ff2f5125 use correct type of link 2021-11-08 12:06:20 +00:00
james
dfe77f844f fix errors in debugging-data-flow-queries-using-partial-flow.rst 2021-11-08 11:59:53 +00:00
Erik Krogh Kristensen
0ab510f543 add test that requires flowToExpr 2021-11-08 12:25:45 +01:00
james
c94bfc306a improve links 2021-11-08 11:18:41 +00:00
Erik Krogh Kristensen
3d6a5263e0 improve qhelp
Co-authored-by: Esben Sparre Andreasen <esbena@github.com>
2021-11-08 12:02:39 +01:00
Mathias Vorreiter Pedersen
8e496f7121 C++: Pull in the latest changes to 'SsaImplCommon'. 2021-11-08 10:46:54 +00:00