Commit Graph

40697 Commits

Author SHA1 Message Date
Owen Mansel-Chan
9f763dd044 Move built-in models to ExternalFlow 2021-12-08 11:20:30 -05:00
Owen Mansel-Chan
d717734820 Do not allow "Argument" on its own 2021-12-08 11:20:30 -05:00
Owen Mansel-Chan
d2ca1fb2eb Address review comments #2 2021-12-08 11:20:29 -05:00
Owen Mansel-Chan
12058a2621 Fix containerStoreStep and containerReadStep 2021-12-08 11:20:29 -05:00
Owen Mansel-Chan
ab8096b717 Add tests for more content types (Element, MapKey, MapValue) 2021-12-08 11:20:28 -05:00
Owen Mansel-Chan
b7aa85b054 Address some review comments 2021-12-08 11:20:28 -05:00
Owen Mansel-Chan
f375553933 Add variadic functions test for function models 2021-12-08 11:20:27 -05:00
Owen Mansel-Chan
b75def62fe Add variadic functions test for external flow 2021-12-08 11:20:27 -05:00
Owen Mansel-Chan
d9848fe515 Add more tests for variadic functions 2021-12-08 11:20:27 -05:00
Owen Mansel-Chan
8044fb2519 Add more flow tests for external flow 2021-12-08 11:20:26 -05:00
Owen Mansel-Chan
63d997f820 (Unimportant) Fix module name for vendored stubs
This doesn't affect the test, but does mean that you can run
`go build` to check the test would build.
2021-12-08 11:20:26 -05:00
Owen Mansel-Chan
1929a1f7a7 Fix unrelated test in experimental 2021-12-08 11:20:25 -05:00
Owen Mansel-Chan
5e38f48b74 Autoformat 2021-12-08 11:20:25 -05:00
Owen Mansel-Chan
a3df3614a5 Convert completetest to an inline flow test 2021-12-08 11:20:24 -05:00
Owen Mansel-Chan
8f7a34f9cb Fix external flow tests 2021-12-08 11:20:24 -05:00
Owen Mansel-Chan
71bf834765 Fix incorrect assumption
node2 doesn't have to be a PostUpdateNode
2021-12-08 11:20:23 -05:00
Sauyon Lee
3379790686 add flow test involving CSV 2021-12-08 11:20:22 -05:00
Sauyon Lee
a632a58221 add CSV models of append 2021-12-08 11:20:22 -05:00
Sauyon Lee
070e383516 allow empty namespaces for Go 2021-12-08 11:20:21 -05:00
Owen Mansel-Chan
70c9ca5611 Update documentation in ExternalFlow.qll 2021-12-08 11:20:21 -05:00
Owen Mansel-Chan
038f951e9f Fix containerStoreStep
Update some comments as well, and change a variable name
2021-12-08 11:20:20 -05:00
Owen Mansel-Chan
be6501d8e4 Add tests for data and taint flow through arrays and var args 2021-12-08 11:20:20 -05:00
Sauyon Lee
2060731077 Add tests for external flow 2021-12-08 11:20:20 -05:00
haby0
a18aad8536 Fix one 2021-12-08 21:03:17 +08:00
Anders Schack-Mulligen
38d0bb4a60 Merge pull request #7260 from hvitved/dataflow/argument-parameter-matching
Data flow: Introduce `ParameterPosition` and `ArgumentPosition`
2021-12-08 12:49:08 +01:00
haby0
1d321c692b Refactor isMybatisXmlOrAnnotationSqlInjection 2021-12-08 18:59:55 +08:00
Alex Ford
ede1503cc6 Merge pull request #7328 from github/ruby/customizations
Ruby: add `Customizations.qll` file
2021-12-08 10:54:23 +00:00
Tom Hvitved
283173ad02 Address review comments 2021-12-08 11:26:44 +01:00
Erik Krogh Kristensen
3145e8f9b7 add upgrade script 2021-12-08 10:53:47 +01:00
Erik Krogh Kristensen
1956405d17 Merge pull request #7284 from erik-krogh/myApply-part1
JS: remove paths without unmatched returns from polynomial-redos
2021-12-08 10:46:03 +01:00
Tom Hvitved
5735bb698d Ruby: Hide desugared nodes in data-flow paths 2021-12-08 09:00:16 +01:00
yoff
0e33f730b1 Merge pull request #7329 from tausbn/tausbn/python-fix-syntax-error-locations
Python: Fix syntax error locations
2021-12-07 22:45:35 +01:00
Rasmus Wriedt Larsen
a650c56c0c Tag queries with CWE-328
CWE-328: Use of Weak Hash, see https://cwe.mitre.org/data/definitions/328.html
2021-12-07 20:54:31 +00:00
Geoffrey White
5ee9684435 C++: Change note. 2021-12-07 20:42:36 +00:00
Geoffrey White
122f6385e6 C++: Improve recognition of stdin, stdout etc. 2021-12-07 20:42:35 +00:00
Geoffrey White
6896b20dcd C++: Redesign and fix results that appear to be encrypted. 2021-12-07 20:42:13 +00:00
Alex Ford
ea7063f3c6 Ruby: make Customizations import private 2021-12-07 19:43:08 +00:00
Tom Hvitved
490872173a Data flow: Sync files 2021-12-07 20:29:18 +01:00
Tom Hvitved
07ca1c2ec0 Data flow: Adjust parameterMatch join-orders 2021-12-07 20:29:00 +01:00
Erik Krogh Kristensen
fe1a473f07 Merge pull request #138 from github/erik-krogh/bump-this
bump the severity of `ql/implicit-this`
2021-12-07 20:12:21 +01:00
Erik Krogh Kristensen
a722aeb5e3 QL: Merge pull request #138 from github/erik-krogh/bump-this
bump the severity of `ql/implicit-this`
2021-12-07 20:12:21 +01:00
Taus
e7c298d903 Python: Fix bad scope_entry_points_to join
From `pritomrajkhowa/LoopBound`:

```
Definitions.ql-7:PointsTo::PointsToInternal::scope_entry_points_to#ffff#antijoin_rhs#2 ........... 55.1s
```

specifically

```
(443s) Tuple counts for PointsTo::PointsToInternal::scope_entry_points_to#ffff#antijoin_rhs#2/3@74a7cart after 55.1s:
184070    ~0%        {3} r1 = JOIN PointsTo::PointsToInternal::scope_entry_points_to#ffff#shared#1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'arg2', Lhs.1 'arg0', Lhs.2 'arg1'
184070    ~0%        {3} r2 = STREAM DEDUP r1
919966523 ~2%        {4} r3 = JOIN r2 WITH Essa::EssaDefinition::getSourceVariable_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'arg0', Lhs.2 'arg1', Lhs.0 'arg2'
4281779   ~2293%     {3} r4 = JOIN r3 WITH Essa::EssaVariable::getScope_dispred#ff ON FIRST 2 OUTPUT Lhs.1 'arg0', Lhs.2 'arg1', Lhs.3 'arg2'
                    return r4
```

First, this is an `antijoin`, so there's likely some negation involved.
Also, there's mention of `GlobalVariable`, `getScope`, and
`getSourceVariable`, none of which appear in `scope_entry_points_to`, so
it's likely that something got inlined.

Taking a closer look at the predicates mentioned in the body, we spot
`undefined_variable` as a likely culprit.

Evaluating this predicate in isolation reveals that it's not terribly
big, so we could try just marking it with `pragma[noinline]` (I opted
for the slightly more solid `nomagic`) and see how that fares. I also
checked that `builtin_not_in_outer_scope` was similarly small, and
made that one un-inlineable as well.

The result? Well, I can't even show you. Both `scope_entry_points_to`
and `undefined_variable` are so fast that they don't appear in the
clause timing report (so they can at most take 3.5s each to evaluate, as
that is the smallest timing in the list).
2021-12-07 18:51:44 +00:00
Taus
b502ca1ea7 Python: Fix bad callsite_points_to join
From `pritomrajkhowa/LoopBound`:

```
Definitions.ql-7:PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#join_rhs#3 ........... 5m53s
```

specifically

```
(767s) Tuple counts for PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#join_rhs#3/3@f8f86764 after 5m53s:
832806293 ~0%     {4} r1 = JOIN PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared#1 WITH PointsTo::InterProceduralPointsTo::var_at_exit#fff ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'arg1', Rhs.1 'arg2', Rhs.2 'arg0'
832806293 ~0%     {3} r2 = JOIN r1 WITH Essa::TEssaNodeRefinement#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.3 'arg0', Lhs.1 'arg1', Lhs.2 'arg2'
                return r2
```

This one is a bit tricky to unpack. Where is this `shared#1` defined?

```
EVALUATE NONRECURSIVE RELATION:
SYNTHETIC PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared#1(int arg0, numbered_tuple arg1) :-
    SENTINEL PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared
    SENTINEL Definitions::EscapingAssignmentGlobalVariable#class#f
    SENTINEL Essa::TEssaNodeRefinement#ffff_03#join_rhs
    {2} r1 = JOIN PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared WITH Definitions::EscapingAssignmentGlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'arg0', Lhs.1 'arg1'
    {2} r2 = STREAM DEDUP r1
    {2} r3 = JOIN r2 WITH Essa::TEssaNodeRefinement#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.0 'arg0', Lhs.1 'arg1'
    {2} r4 = STREAM DEDUP r3
    return r4
```

Looking at `callsite_points_to`, we see a likely candidate in `srcvar`.
It is guarded with an `instanceof` check for
`EscapingAssignmentGlobalVariable` (which lines up nicely with the
sentinel on its charpred) and `getSourceVariable` is just a projection
of `TEssaNodeRefinement`.

So let's try unbinding `srcvar` to prevent an early join.

The timing is now:

```
Definitions.ql-7:PointsTo::InterProceduralPointsTo::callsite_points_to#ffff ...................... 31.3s (2554 evaluations with max 101ms in PointsTo::InterProceduralPointsTo::callsite_points_to#ffff/4@i516#581fap5w)
```
(Showing the tuple counts doesn't make sense here, since all of the
`shared` and `join_rhs` predicates have been smooshed around.)
2021-12-07 18:25:53 +00:00
Taus
a716482c1f Python: Fix bad join in SSA
On `pritomrajkhowa/LoopBound`:

```
Definitions.ql-3:SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff ................. 4m35s
```

specifically

```
(376s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@be04e9kp after 4m58s:
388843     ~0%     {4} r1 = JOIN Essa::TPhiFunction#fff_2#join_rhs WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Rhs.2, Rhs.3
3629812090 ~1%     {7} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::variableUse#ffff ON FIRST 1 OUTPUT Lhs.0, Rhs.2, Rhs.3, Lhs.2, Lhs.3, Lhs.1, Rhs.1 'use1'
0          ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 5 OUTPUT Lhs.5, Lhs.6 'use1'
0          ~0%     {2} r4 = JOIN r3 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141     ~0%     {2} r5 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r4
                    return r5
```

Clearly we do not want to join on the variable so soon. So we unbind it
and get

```
(78s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@40e0e6uv after 434ms:
3377959 ~2%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::variableUse#ffff OUTPUT In.0, In.2, In.3, In.1 'use1'
1026855 ~2%     {4} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 3 OUTPUT Lhs.0, Rhs.3, Rhs.4, Lhs.3 'use1'
129484  ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff_1230#join_rhs ON FIRST 3 OUTPUT Rhs.3, Lhs.3 'use1'
0       ~0%     {2} r4 = JOIN r3 WITH Essa::TPhiFunction#fff_2#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'use1'
0       ~0%     {2} r5 = JOIN r4 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141  ~0%     {2} r6 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r5
                return r6
```
2021-12-07 18:19:47 +00:00
Taus
59bac04d8f Python: Fix Python 2 failures 2021-12-07 18:00:46 +00:00
Taus
ffc858e34d Python: Add missing file 2021-12-07 17:29:35 +00:00
Alex Ford
bf0ecded04 Merge pull request #7326 from github/ginsbach/FixInstanceof
fix dependency cycle by removing superfluous classes
2021-12-07 17:05:26 +00:00
Taus
7437cd4d85 Python: Fix syntax error locations 2021-12-07 16:51:33 +00:00
Geoffrey White
511bee7a1a C++: Fix results that flow to/from encryption routines. 2021-12-07 15:44:18 +00:00
Alex Ford
f85a47d41f Ruby: add Customizations.qll file 2021-12-07 15:37:04 +00:00