Commit Graph

41812 Commits

Author SHA1 Message Date
Geoffrey White
6896b20dcd C++: Redesign and fix results that appear to be encrypted. 2021-12-07 20:42:13 +00:00
Alex Ford
ea7063f3c6 Ruby: make Customizations import private 2021-12-07 19:43:08 +00:00
Tom Hvitved
490872173a Data flow: Sync files 2021-12-07 20:29:18 +01:00
Tom Hvitved
07ca1c2ec0 Data flow: Adjust parameterMatch join-orders 2021-12-07 20:29:00 +01:00
Erik Krogh Kristensen
fe1a473f07 Merge pull request #138 from github/erik-krogh/bump-this
bump the severity of `ql/implicit-this`
2021-12-07 20:12:21 +01:00
Erik Krogh Kristensen
a722aeb5e3 QL: Merge pull request #138 from github/erik-krogh/bump-this
bump the severity of `ql/implicit-this`
2021-12-07 20:12:21 +01:00
Taus
e7c298d903 Python: Fix bad scope_entry_points_to join
From `pritomrajkhowa/LoopBound`:

```
Definitions.ql-7:PointsTo::PointsToInternal::scope_entry_points_to#ffff#antijoin_rhs#2 ........... 55.1s
```

specifically

```
(443s) Tuple counts for PointsTo::PointsToInternal::scope_entry_points_to#ffff#antijoin_rhs#2/3@74a7cart after 55.1s:
184070    ~0%        {3} r1 = JOIN PointsTo::PointsToInternal::scope_entry_points_to#ffff#shared#1 WITH Variables::GlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'arg2', Lhs.1 'arg0', Lhs.2 'arg1'
184070    ~0%        {3} r2 = STREAM DEDUP r1
919966523 ~2%        {4} r3 = JOIN r2 WITH Essa::EssaDefinition::getSourceVariable_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'arg0', Lhs.2 'arg1', Lhs.0 'arg2'
4281779   ~2293%     {3} r4 = JOIN r3 WITH Essa::EssaVariable::getScope_dispred#ff ON FIRST 2 OUTPUT Lhs.1 'arg0', Lhs.2 'arg1', Lhs.3 'arg2'
                    return r4
```

First, this is an `antijoin`, so there's likely some negation involved.
Also, there's mention of `GlobalVariable`, `getScope`, and
`getSourceVariable`, none of which appear in `scope_entry_points_to`, so
it's likely that something got inlined.

Taking a closer look at the predicates mentioned in the body, we spot
`undefined_variable` as a likely culprit.

Evaluating this predicate in isolation reveals that it's not terribly
big, so we could try just marking it with `pragma[noinline]` (I opted
for the slightly more solid `nomagic`) and see how that fares. I also
checked that `builtin_not_in_outer_scope` was similarly small, and
made that one un-inlineable as well.

The result? Well, I can't even show you. Both `scope_entry_points_to`
and `undefined_variable` are so fast that they don't appear in the
clause timing report (so they can at most take 3.5s each to evaluate, as
that is the smallest timing in the list).
2021-12-07 18:51:44 +00:00
Taus
b502ca1ea7 Python: Fix bad callsite_points_to join
From `pritomrajkhowa/LoopBound`:

```
Definitions.ql-7:PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#join_rhs#3 ........... 5m53s
```

specifically

```
(767s) Tuple counts for PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#join_rhs#3/3@f8f86764 after 5m53s:
832806293 ~0%     {4} r1 = JOIN PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared#1 WITH PointsTo::InterProceduralPointsTo::var_at_exit#fff ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'arg1', Rhs.1 'arg2', Rhs.2 'arg0'
832806293 ~0%     {3} r2 = JOIN r1 WITH Essa::TEssaNodeRefinement#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.3 'arg0', Lhs.1 'arg1', Lhs.2 'arg2'
                return r2
```

This one is a bit tricky to unpack. Where is this `shared#1` defined?

```
EVALUATE NONRECURSIVE RELATION:
SYNTHETIC PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared#1(int arg0, numbered_tuple arg1) :-
    SENTINEL PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared
    SENTINEL Definitions::EscapingAssignmentGlobalVariable#class#f
    SENTINEL Essa::TEssaNodeRefinement#ffff_03#join_rhs
    {2} r1 = JOIN PointsTo::InterProceduralPointsTo::callsite_points_to#ffff#shared WITH Definitions::EscapingAssignmentGlobalVariable#class#f ON FIRST 1 OUTPUT Lhs.0 'arg0', Lhs.1 'arg1'
    {2} r2 = STREAM DEDUP r1
    {2} r3 = JOIN r2 WITH Essa::TEssaNodeRefinement#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.0 'arg0', Lhs.1 'arg1'
    {2} r4 = STREAM DEDUP r3
    return r4
```

Looking at `callsite_points_to`, we see a likely candidate in `srcvar`.
It is guarded with an `instanceof` check for
`EscapingAssignmentGlobalVariable` (which lines up nicely with the
sentinel on its charpred) and `getSourceVariable` is just a projection
of `TEssaNodeRefinement`.

So let's try unbinding `srcvar` to prevent an early join.

The timing is now:

```
Definitions.ql-7:PointsTo::InterProceduralPointsTo::callsite_points_to#ffff ...................... 31.3s (2554 evaluations with max 101ms in PointsTo::InterProceduralPointsTo::callsite_points_to#ffff/4@i516#581fap5w)
```
(Showing the tuple counts doesn't make sense here, since all of the
`shared` and `join_rhs` predicates have been smooshed around.)
2021-12-07 18:25:53 +00:00
Taus
a716482c1f Python: Fix bad join in SSA
On `pritomrajkhowa/LoopBound`:

```
Definitions.ql-3:SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff ................. 4m35s
```

specifically

```
(376s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@be04e9kp after 4m58s:
388843     ~0%     {4} r1 = JOIN Essa::TPhiFunction#fff_2#join_rhs WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Rhs.2, Rhs.3
3629812090 ~1%     {7} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::variableUse#ffff ON FIRST 1 OUTPUT Lhs.0, Rhs.2, Rhs.3, Lhs.2, Lhs.3, Lhs.1, Rhs.1 'use1'
0          ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 5 OUTPUT Lhs.5, Lhs.6 'use1'
0          ~0%     {2} r4 = JOIN r3 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141     ~0%     {2} r5 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r4
                    return r5
```

Clearly we do not want to join on the variable so soon. So we unbind it
and get

```
(78s) Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUse#ff/2@40e0e6uv after 434ms:
3377959 ~2%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::variableUse#ffff OUTPUT In.0, In.2, In.3, In.1 'use1'
1026855 ~2%     {4} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#fffff ON FIRST 3 OUTPUT Lhs.0, Rhs.3, Rhs.4, Lhs.3 'use1'
129484  ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#ffff_1230#join_rhs ON FIRST 3 OUTPUT Rhs.3, Lhs.3 'use1'
0       ~0%     {2} r4 = JOIN r3 WITH Essa::TPhiFunction#fff_2#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'use1'
0       ~0%     {2} r5 = JOIN r4 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#ff ON FIRST 1 OUTPUT Lhs.1 'use1', Rhs.1 'use2'

897141  ~0%     {2} r6 = SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentUseUseSameVar#ff UNION r5
                return r6
```
2021-12-07 18:19:47 +00:00
Taus
59bac04d8f Python: Fix Python 2 failures 2021-12-07 18:00:46 +00:00
Taus
ffc858e34d Python: Add missing file 2021-12-07 17:29:35 +00:00
Alex Ford
bf0ecded04 Merge pull request #7326 from github/ginsbach/FixInstanceof
fix dependency cycle by removing superfluous classes
2021-12-07 17:05:26 +00:00
Taus
7437cd4d85 Python: Fix syntax error locations 2021-12-07 16:51:33 +00:00
Geoffrey White
511bee7a1a C++: Fix results that flow to/from encryption routines. 2021-12-07 15:44:18 +00:00
Alex Ford
f85a47d41f Ruby: add Customizations.qll file 2021-12-07 15:37:04 +00:00
Tom Hvitved
a2dc505c26 Merge pull request #7317 from hvitved/ruby/param-node-refactor
Ruby: Restructure `ParameterNode(Impl)`
2021-12-07 16:29:49 +01:00
Tom Hvitved
5183290439 Merge pull request #7315 from hvitved/ruby/inline-flow-test
Ruby: Add `InlineFlowTest.qll`
2021-12-07 16:29:34 +01:00
Henry Mercer
322e39446d JS: Autoformat 2021-12-07 14:17:11 +00:00
Henry Mercer
016727d6b6 JS: Fix occasional duplicate body tokens
0e31439 introduces some occasional duplicate tokens due to duplicate AST
node attributes. The long-term fix is to update `CodeToFeatures.qll`,
but for the short-term, we update the concatenation to concatenate
unique (location, token) pairs.
2021-12-07 14:16:48 +00:00
Philip Ginsbach
b2c1b55c0c rephrase extensions as aliases 2021-12-07 13:09:25 +00:00
Tom Hvitved
b17a93eaad Merge pull request #7316 from hvitved/ruby/is-private-join
Ruby: Tweak `Method::isPrivate` join-orders
2021-12-07 13:58:19 +01:00
Sauyon Lee
873f496038 Use basicLocalFlowStep instead of .getASuccessor
This prevents non-monotonic recursion through summary post-update nodes
2021-12-07 07:39:28 -05:00
Sauyon Lee
afe7edc093 Fix test output
Includes a bunch of new edges, but no new results
2021-12-07 07:39:28 -05:00
Sauyon Lee
0572c4785c Model net http sources as csv 2021-12-07 07:39:27 -05:00
Sauyon Lee
bebdb0ba53 Add RangeIndexNode 2021-12-07 07:39:27 -05:00
Sauyon Lee
3750af41d3 Add standard container steps 2021-12-07 07:39:27 -05:00
Sauyon Lee
8c4a1d2559 Consider CSV remote sources as untrusted flow sources 2021-12-07 07:39:26 -05:00
Sauyon Lee
d62f417130 Remove uses of getEnclosingCallable 2021-12-07 07:39:26 -05:00
Sauyon Lee
30ab22f5a6 Fix compilation errors with new DataFlowCallable 2021-12-07 07:39:26 -05:00
Chris Smowton
b10d5cf0b0 Broaden ReturnNode to include return nodes of summaries 2021-12-07 07:39:25 -05:00
Chris Smowton
94d9d08489 Fix DataFlow::Node::getEnclosingCallable 2021-12-07 07:39:25 -05:00
Sauyon Lee
c8a2a6356a Add summary parameter nodes 2021-12-07 07:39:25 -05:00
Sauyon Lee
4af4a11729 Make getACallee return DataFlowCallable 2021-12-07 07:39:24 -05:00
Sauyon Lee
8cba368ef5 Model archive/tar.FileInfoHeader in CSV 2021-12-07 07:39:24 -05:00
Sauyon Lee
86d3410041 Add asFunctionNode to new dataflowcallable 2021-12-07 07:39:23 -05:00
Sauyon Lee
d9383d9412 Don't use internal predicates in revel 2021-12-07 07:39:23 -05:00
Sauyon Lee
73684f483c Allow for Return[i] specifications 2021-12-07 07:39:22 -05:00
Sauyon Lee
aa747ea5ff Fix validation regexes for go 2021-12-07 07:39:22 -05:00
Sauyon Lee
0151cd4f2e Document SourceOrSinkElement 2021-12-07 07:39:22 -05:00
Sauyon Lee
0b50b7b2b1 Make DataFlowCallable either a Function or a FuncLit 2021-12-07 07:39:21 -05:00
Sauyon Lee
3ac2a50497 Update test output 2021-12-07 07:39:21 -05:00
Owen Mansel-Chan
763861bef9 Keep call to defaultTaintSanitizerGuard 2021-12-07 07:39:21 -05:00
Sauyon Lee
e41d609921 Use newtype for SourceOrSinkElement 2021-12-07 07:39:20 -05:00
Sauyon Lee
9bfe1c94b3 autoformat 2021-12-07 07:39:20 -05:00
Sauyon Lee
16371ac488 Add support for summary elements 2021-12-07 07:39:19 -05:00
Sauyon Lee
96c58b58dd Add EmptyInterfaceType 2021-12-07 07:39:19 -05:00
Sauyon Lee
26d00f1d5b Move basicLocalFlowsStep to DataFlowPrivate 2021-12-07 07:39:19 -05:00
Sauyon Lee
3098a4ef16 Qualify uses and add imports in DataFlowNodes 2021-12-07 07:39:18 -05:00
Sauyon Lee
93f2569f1d Refactor data-flow nodes 2021-12-07 07:39:18 -05:00
Sauyon Lee
9ceda08d13 Sync dataflow libraries 2021-12-07 07:39:12 -05:00