Commit Graph

29908 Commits

Author SHA1 Message Date
Tony Torralba
c0c40cc05b Remove synthetic fields 2021-12-09 13:34:41 +01:00
Tony Torralba
3a3c7fc59e Fix stub 2021-12-09 13:34:41 +01:00
Tony Torralba
f209ff4f76 Use synthetic fields to improve taint precision 2021-12-09 13:34:39 +01:00
Tony Torralba
b7f7c5ba20 Change format of fluent models to make review easier 2021-12-09 13:33:19 +01:00
Tony Torralba
f63ffb0630 Add models for Notification builders 2021-12-09 13:33:17 +01:00
Tom Hvitved
069cf9d17f C#: Exclude stubs in GetAnOutNode.ql test 2021-12-09 13:33:14 +01:00
Tom Hvitved
cbd21edc99 C#: Override File::isFromSource in tests to exclude stubs 2021-12-09 13:33:14 +01:00
Michael Nebel
2f85735b6a C#: Use stubs instead of dll's in the dataflow global tests. 2021-12-09 13:25:11 +01:00
Tom Hvitved
69ba2e6f8c Merge pull request #7337 from michaelnebel/csharp-synthetic-field
C#: Introduce synthetic fields and use them in Task<>.
2021-12-09 13:18:44 +01:00
Michael Nebel
13347cd102 C#: Add Ql docs to synthetic fields. 2021-12-09 10:34:31 +01:00
Michael Nebel
9f4b965202 C#: Update the flow summaries produced for Task<> after introduction of synthetic fields. 2021-12-09 10:11:49 +01:00
Michael Nebel
d70d1fbf81 C#: Add support for the use of synthetic fields in flow summary CSV. 2021-12-09 10:11:48 +01:00
Michael Nebel
a43704ab43 C#: Update dataflow/global tests based on synthetic fields. 2021-12-09 10:11:48 +01:00
Michael Nebel
063398f24d C#: Use synthetic fields for Task instead of referring to private fields. 2021-12-09 10:11:48 +01:00
Michael Nebel
60f3ff8c33 C#: Introduce type for Synthetic fields. 2021-12-09 10:11:48 +01:00
haby0
8bcbf8e30f rename isMybatisCollectionTypeSqlInjection 2021-12-09 09:16:33 +08:00
Tom Hvitved
b49ca6a24c Merge pull request #7335 from hvitved/ruby/dataflow/hide-desugared-nodes
Ruby: Hide desugared nodes in data-flow paths
2021-12-08 17:39:48 +01:00
haby0
a18aad8536 Fix one 2021-12-08 21:03:17 +08:00
Anders Schack-Mulligen
38d0bb4a60 Merge pull request #7260 from hvitved/dataflow/argument-parameter-matching
Data flow: Introduce `ParameterPosition` and `ArgumentPosition`
2021-12-08 12:49:08 +01:00
haby0
1d321c692b Refactor isMybatisXmlOrAnnotationSqlInjection 2021-12-08 18:59:55 +08:00
Alex Ford
ede1503cc6 Merge pull request #7328 from github/ruby/customizations
Ruby: add `Customizations.qll` file
2021-12-08 10:54:23 +00:00
Tom Hvitved
283173ad02 Address review comments 2021-12-08 11:26:44 +01:00
Erik Krogh Kristensen
3145e8f9b7 add upgrade script 2021-12-08 10:53:47 +01:00
Erik Krogh Kristensen
1956405d17 Merge pull request #7284 from erik-krogh/myApply-part1
JS: remove paths without unmatched returns from polynomial-redos
2021-12-08 10:46:03 +01:00
Tom Hvitved
5735bb698d Ruby: Hide desugared nodes in data-flow paths 2021-12-08 09:00:16 +01:00
yoff
0e33f730b1 Merge pull request #7329 from tausbn/tausbn/python-fix-syntax-error-locations
Python: Fix syntax error locations
2021-12-07 22:45:35 +01:00
Alex Ford
ea7063f3c6 Ruby: make Customizations import private 2021-12-07 19:43:08 +00:00
Tom Hvitved
490872173a Data flow: Sync files 2021-12-07 20:29:18 +01:00
Tom Hvitved
07ca1c2ec0 Data flow: Adjust parameterMatch join-orders 2021-12-07 20:29:00 +01:00
Taus
e7c298d903 Python: Fix bad scope_entry_points_to join
From `pritomrajkhowa/LoopBound`:

```
Definitions.ql-7:PointsTo::PointsToInternal::scope_entry_points_to#ffff#antijoin_rhs#2 ........... 55.1s
```

specifically

```
(443s) Tuple counts for PointsTo::PointsToInternal::scope_entry_points_to#ffff#antijoin_rhs#2/3@74a7cart after 55.1s:
184070    ~0%        {3} r1 = JOIN PointsTo::PointsToInternal::scope_entry_points_to#ffff#shared#1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'arg2', Lhs.1 'arg0', Lhs.2 'arg1'
184070    ~0%        {3} r2 = STREAM DEDUP r1
919966523 ~2%        {4} r3 = JOIN r2 WITH Essa::EssaDefinition::getSourceVariable_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'arg0', Lhs.2 'arg1', Lhs.0 'arg2'
4281779   ~2293%     {3} r4 = JOIN r3 WITH Essa::EssaVariable::getScope_dispred#ff ON FIRST 2 OUTPUT Lhs.1 'arg0', Lhs.2 'arg1', Lhs.3 'arg2'
                    return r4
```

First, this is an `antijoin`, so there's likely some negation involved.
Also, there's mention of `GlobalVariable`, `getScope`, and
`getSourceVariable`, none of which appear in `scope_entry_points_to`, so
it's likely that something got inlined.

Taking a closer look at the predicates mentioned in the body, we spot
`undefined_variable` as a likely culprit.

Evaluating this predicate in isolation reveals that it's not terribly
big, so we could try just marking it with `pragma[noinline]` (I opted
for the slightly more solid `nomagic`) and see how that fares. I also
checked that `builtin_not_in_outer_scope` was similarly small, and
made that one un-inlineable as well.

The result? Well, I can't even show you. Both `scope_entry_points_to`
and `undefined_variable` are so fast that they don't appear in the
clause timing report (so they can at most take 3.5s each to evaluate, as
that is the smallest timing in the list).
2021-12-07 18:51:44 +00:00
Taus
b502ca1ea7 Python: Fix bad callsite_points_to join
From `pritomrajkhowa/LoopBound`:

```
Definitions.ql-7:PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#join_rhs#3 ........... 5m53s
```

specifically

```
(767s) Tuple counts for PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#join_rhs#3/3@f8f86764 after 5m53s:
832806293 ~0%     {4} r1 = JOIN PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared#1 WITH PointsTo::InterProceduralPointsTo::var_at_exit#fff ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'arg1', Rhs.1 'arg2', Rhs.2 'arg0'
832806293 ~0%     {3} r2 = JOIN r1 WITH Essa::TEssaNodeRefinement#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.3 'arg0', Lhs.1 'arg1', Lhs.2 'arg2'
                return r2
```

This one is a bit tricky to unpack. Where is this `shared#1` defined?

```
EVALUATE NONRECURSIVE RELATION:
SYNTHETIC PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared#1(int arg0, numbered_tuple arg1) :-
    SENTINEL PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared
    SENTINEL Definitions::EscapingAssignmentGlobalVariable#class#f
    SENTINEL Essa::TEssaNodeRefinement#ffff_03#join_rhs
    {2} r1 = JOIN PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared WITH Definitions::EscapingAssignmentGlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'arg0', Lhs.1 'arg1'
    {2} r2 = STREAM DEDUP r1
    {2} r3 = JOIN r2 WITH Essa::TEssaNodeRefinement#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.0 'arg0', Lhs.1 'arg1'
    {2} r4 = STREAM DEDUP r3
    return r4
```

Looking at `callsite_points_to`, we see a likely candidate in `srcvar`.
It is guarded with an `instanceof` check for
`EscapingAssignmentGlobalVariable` (which lines up nicely with the
sentinel on its charpred) and `getSourceVariable` is just a projection
of `TEssaNodeRefinement`.

So let's try unbinding `srcvar` to prevent an early join.

The timing is now:

```
Definitions.ql-7:PointsTo::InterProceduralPointsTo::callsite_points_to#ffff ...................... 31.3s (2554 evaluations with max 101ms in PointsTo::InterProceduralPointsTo::callsite_points_to#ffff/4@i516#581fap5w)
```
(Showing the tuple counts doesn't make sense here, since all of the
`shared` and `join_rhs` predicates have been smooshed around.)
2021-12-07 18:25:53 +00:00
Taus
a716482c1f Python: Fix bad join in SSA
On `pritomrajkhowa/LoopBound`:

```
Definitions.ql-3:SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff ................. 4m35s
```

specifically

```
(376s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@be04e9kp after 4m58s:
388843     ~0%     {4} r1 = JOIN Essa::TPhiFunction#fff_2#join_rhs WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Rhs.2, Rhs.3
3629812090 ~1%     {7} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::variableUse#ffff ON FIRST 1 OUTPUT Lhs.0, Rhs.2, Rhs.3, Lhs.2, Lhs.3, Lhs.1, Rhs.1 'use1'
0          ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 5 OUTPUT Lhs.5, Lhs.6 'use1'
0          ~0%     {2} r4 = JOIN r3 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141     ~0%     {2} r5 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r4
                    return r5
```

Clearly we do not want to join on the variable so soon. So we unbind it
and get

```
(78s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@40e0e6uv after 434ms:
3377959 ~2%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::variableUse#ffff OUTPUT In.0, In.2, In.3, In.1 'use1'
1026855 ~2%     {4} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 3 OUTPUT Lhs.0, Rhs.3, Rhs.4, Lhs.3 'use1'
129484  ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff_1230#join_rhs ON FIRST 3 OUTPUT Rhs.3, Lhs.3 'use1'
0       ~0%     {2} r4 = JOIN r3 WITH Essa::TPhiFunction#fff_2#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'use1'
0       ~0%     {2} r5 = JOIN r4 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141  ~0%     {2} r6 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r5
                return r6
```
2021-12-07 18:19:47 +00:00
Taus
59bac04d8f Python: Fix Python 2 failures 2021-12-07 18:00:46 +00:00
Taus
ffc858e34d Python: Add missing file 2021-12-07 17:29:35 +00:00
Alex Ford
bf0ecded04 Merge pull request #7326 from github/ginsbach/FixInstanceof
fix dependency cycle by removing superfluous classes
2021-12-07 17:05:26 +00:00
Taus
7437cd4d85 Python: Fix syntax error locations 2021-12-07 16:51:33 +00:00
Alex Ford
f85a47d41f Ruby: add Customizations.qll file 2021-12-07 15:37:04 +00:00
Tom Hvitved
a2dc505c26 Merge pull request #7317 from hvitved/ruby/param-node-refactor
Ruby: Restructure `ParameterNode(Impl)`
2021-12-07 16:29:49 +01:00
Tom Hvitved
5183290439 Merge pull request #7315 from hvitved/ruby/inline-flow-test
Ruby: Add `InlineFlowTest.qll`
2021-12-07 16:29:34 +01:00
Henry Mercer
322e39446d JS: Autoformat 2021-12-07 14:17:11 +00:00
Henry Mercer
016727d6b6 JS: Fix occasional duplicate body tokens
0e31439 introduces some occasional duplicate tokens due to duplicate AST
node attributes. The long-term fix is to update `CodeToFeatures.qll`,
but for the short-term, we update the concatenation to concatenate
unique (location, token) pairs.
2021-12-07 14:16:48 +00:00
Philip Ginsbach
b2c1b55c0c rephrase extensions as aliases 2021-12-07 13:09:25 +00:00
Tom Hvitved
b17a93eaad Merge pull request #7316 from hvitved/ruby/is-private-join
Ruby: Tweak `Method::isPrivate` join-orders
2021-12-07 13:58:19 +01:00
Tom Hvitved
4d797d6b3d Merge pull request #7324 from github/hmac/empty-else-cfg
Ruby: Include empty StmtSequences in CFG
2021-12-07 13:19:15 +01:00
Philip Ginsbach
da43984ba4 fix dependency cycle by removing superfluous classes 2021-12-07 11:59:04 +00:00
Rasmus Wriedt Larsen
ee23799a59 Merge pull request #7319 from RasmusWL/js-cwe-328
JS: Tag queries with CWE-328
2021-12-07 11:40:33 +01:00
Anders Schack-Mulligen
6c739b67fa Merge pull request #7318 from RasmusWL/java-cwe-328
Java: Tag queries with CWE-328
2021-12-07 11:39:48 +01:00
Erik Krogh Kristensen
3c59aa319e Merge pull request #7245 from erik-krogh/explicit-this-all-the-places
All langs: apply the explicit-this patch to all remaining code
2021-12-07 10:40:26 +01:00
Harry Maclean
6f42153eac Ruby: Include empty StmtSequences in CFG
Empty StmtSequences appear, for example, in the `else` branch of `if`
statements like the following:

    foo

    if cond
      bar
    else
    end

    baz

Before this change, the CFG for this code would look like this:

        foo
         │
         │
         ▼
        cond
         │
    true │
         ▼
        bar
         │
         │
         ▼
         if
         │
         │
         ▼
        baz

i.e. there is linear flow through the condition, the `then` branch, and
out of the if. This doesn't account for the possibility that the
condition is false and `bar` is not executed. After this change, the CFG
looks like this:

          foo
           │
           │
           ▼
          cond
         │    │
    true │    │ false
         ▼    │
        bar   │
         │    │
         │    │
         ▼    ▼
           if
           │
           │
           ▼
          baz

i.e. we correctly account for the `false` condition.
2021-12-07 16:01:50 +13:00
Geoffrey White
4e68a4670b Merge pull request #7322 from MathiasVP/fix-performance-of-unused-static-functions
C++: Fix performance of 'cpp/unused-static-function'.
2021-12-06 17:30:51 +00:00