Commit Graph

31240 Commits

Author SHA1 Message Date
Tom Hvitved
118d0d9ff5 Ruby: Use "Receiver" instead of "Self" in flow summaries
Flow summaries use the "outside view", i.e., the call sites, so "receiver"
is better than "self", as the latter uses the "inside view", i.e. the callees.
2021-12-22 15:56:20 +01:00
Tom Hvitved
3a30f58f74 Address review comments 2021-12-22 15:56:20 +01:00
Tom Hvitved
400802c5ce Ruby: Add flow summaries for Array/Enumerable methods 2021-12-22 15:56:20 +01:00
Michael Nebel
748b2d2507 C#: Simplify the ForEachCapture query.
Co-authored-by: Tom Hvitved <hvitved@github.com>
2021-12-22 15:47:24 +01:00
Michael Nebel
fef6770a21 C#: Remove the callableFlow for the Add method in subtypes of System.Collections.IEnumerable. 2021-12-22 15:47:24 +01:00
Michael Nebel
51e3c582de C#: Re-write ForEachCapture test to avoid using LibraryTypeDataFlow and rely in MaD summaries instead. 2021-12-22 15:47:24 +01:00
Tom Hvitved
8c18aaae74 Ruby: Prepare for data flow through arrays 2021-12-22 15:35:34 +01:00
Tom Hvitved
27f786b41e Merge pull request #7442 from hvitved/ruby/dataflow/keyword-params
Ruby: Data flow for keyword arguments/parameters
2021-12-22 15:23:22 +01:00
Tom Hvitved
4133eb15d5 Ruby: Reintroduce old Argument[_] restriction to avoid large Cartesian product 2021-12-22 11:37:38 +01:00
Tom Hvitved
d196c77b3d Ruby: Remove some redundant overrides 2021-12-22 11:25:13 +01:00
Tom Hvitved
f5471e34f8 C#: Fix bad join-order in dispatch library
Before
```
[2021-12-22 09:46:31] (395s) Tuple counts for Dispatch::Internal::hasCallable#fff/3@258418l2 after 5m27s:
                      49000       ~0%      {2} r1 = JOIN Declaration::Declaration::getUnboundDeclaration_dispred#ff_10#join_rhs WITH project#Dispatch::Internal::DispatchMethodOrAccessorCall::getAStaticTargetExt#ff ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.0

                      31302       ~3%      {3} r2 = JOIN r1 WITH Type::ValueOrRefType::getAMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0 'c', Lhs.1 'source', Rhs.1

                      299700      ~0%      {3} r3 = JOIN r1 WITH Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0 'c', Lhs.1 'source', Rhs.1

                      16650       ~1%      {3} r4 = JOIN r1 WITH Property::Accessor::getDeclaration_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'source', Lhs.0 'c'

                      15984       ~0%      {3} r5 = JOIN r4 WITH Type::ValueOrRefType::getAMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      315684      ~1%      {3} r6 = r3 UNION r5
                      346986      ~1%      {3} r7 = r2 UNION r6

                      0           ~0%      {3} r8 = JOIN r4 WITH Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      666         ~0%      {3} r9 = JOIN r1 WITH Type::hasNonOverriddenMember#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'source', Lhs.0 'c'
                      0           ~0%      {3} r10 = JOIN r9 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable#fff#higher_order_body) ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      0           ~0%      {3} r11 = JOIN r4 WITH Type::hasNonOverriddenMember#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'source', Lhs.2 'c'
                      0           ~0%      {3} r12 = JOIN r11 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable#fff#higher_order_body#1) ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      0           ~0%      {3} r13 = r10 UNION r12
                      0           ~0%      {3} r14 = r8 UNION r13
                      346986      ~1%      {3} r15 = r7 UNION r14
                      11963234000 ~2%      {4} r16 = JOIN r15 WITH Dispatch::Internal::hasOverrider#ff ON FIRST 1 OUTPUT Lhs.2, Rhs.1 't', Lhs.1 'source', Lhs.0 'c'
                      207126      ~27%     {3} r17 = JOIN r16 WITH Unification::Gvn::Cached::getGlobalValueNumber#ff ON FIRST 2 OUTPUT Lhs.2 'source', Lhs.1 't', Lhs.3 'c'
                                           return r17
```

After
```
[2021-12-22 10:39:41] (0s) Tuple counts for Dispatch::Internal::hasCallable0#fff/3@82341e2h after 331ms:
                      93569  ~0%      {2} r1 = JOIN Type::ValueOrRefType::getAMember_dispred#fb_10#join_rhs WITH OverridableCallable::OverridableCallable#f ON FIRST 1 OUTPUT Rhs.0 'c', Lhs.1

                      511767 ~0%      {2} r2 = JOIN Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs WITH OverridableCallable::OverridableCallable#f ON FIRST 1 OUTPUT Rhs.0 'c', Lhs.1

                      35659  ~0%      {2} r3 = JOIN OverridableCallable::OverridableCallable#f WITH Property::Accessor::getDeclaration_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'c'

                      35659  ~4%      {2} r4 = JOIN r3 WITH Type::ValueOrRefType::getAMember_dispred#fb_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      547426 ~0%      {2} r5 = r2 UNION r4
                      640995 ~4%      {2} r6 = r1 UNION r5

                      74835  ~4%      {2} r7 = JOIN r3 WITH Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      32748  ~0%      {2} r8 = JOIN Type::hasNonOverriddenMember#fb_10#join_rhs WITH OverridableCallable::OverridableCallable#f ON FIRST 1 OUTPUT Lhs.1, Rhs.0 'c'
                      171228 ~0%      {2} r9 = JOIN r8 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable0#fff#higher_order_body) ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      9056   ~0%      {2} r10 = JOIN r3 WITH Type::hasNonOverriddenMember#fb_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'c'
                      23633  ~1%      {2} r11 = JOIN r10 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable0#fff#higher_order_body#1) ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      194861 ~0%      {2} r12 = r9 UNION r11
                      269696 ~0%      {2} r13 = r7 UNION r12
                      910691 ~4%      {2} r14 = r6 UNION r13
                      910691 ~2%      {3} r15 = JOIN r14 WITH Declaration::Declaration::getUnboundDeclaration_dispred#ff ON FIRST 1 OUTPUT Rhs.1 'source', Lhs.0 'c', Lhs.1
                      579872 ~2%      {3} r16 = JOIN r15 WITH project#Dispatch::Internal::DispatchMethodOrAccessorCall::getAStaticTargetExt#ff ON FIRST 1 OUTPUT Lhs.2, Lhs.1 'c', Lhs.0 'source'
                      753465 ~41%     {3} r17 = JOIN r16 WITH Unification::Gvn::Cached::getGlobalValueNumber#ff ON FIRST 1 OUTPUT Rhs.1 't', Lhs.1 'c', Lhs.2 'source'
                                      return r17

[2021-12-22 10:39:41] (0s) Tuple counts for Dispatch::Internal::hasCallable#fff/3@e44e67tv after 24ms:
                      201843 ~0%     {3} r1 = JOIN Dispatch::Internal::hasOverrider#ff WITH Dispatch::Internal::hasCallable0#fff ON FIRST 2 OUTPUT Lhs.0 't', Lhs.1 'c', Rhs.2 'source'
                                     return r1
```
2021-12-22 10:45:51 +01:00
Tamás Vajk
43b5d502b8 Merge pull request #7466 from github/workflow/coverage/update
Update CSV framework coverage reports
2021-12-22 09:05:15 +01:00
github-actions[bot]
ba7a10de68 Add changed framework coverage reports 2021-12-22 00:10:19 +00:00
Nick Rolfe
9e259b67bb Merge pull request #7305 from github/nickrolfe/user-controlled-bypass
Ruby: query to find user-controlled bypass of sensitive actions
2021-12-21 17:20:20 +00:00
Arthur Baars
a7aff11140 Merge pull request #7394 from aibaars/ruby-cfg-expr-post
Ruby: CFG: make all expressions "post-order" nodes
2021-12-21 16:36:42 +01:00
Nick Rolfe
5765f3684c Ruby: add missing qldoc comment 2021-12-21 15:29:16 +00:00
Nick Rolfe
5db80dac51 Merge remote-tracking branch 'origin/main' into nickrolfe/user-controlled-bypass 2021-12-21 15:26:08 +00:00
Michael Nebel
c138a2796f Merge pull request #7424 from michaelnebel/csharp-flow-summary-csv
C#: Flow summaries in CSV format.
2021-12-21 16:11:22 +01:00
Michael Nebel
8250fb4cf7 C#: Fixed typo in namespace. 2021-12-21 15:00:05 +01:00
Arthur Baars
a86ba3b14e Ruby: rename WhenExpr to WhenClause 2021-12-21 12:31:24 +01:00
Mathias Vorreiter Pedersen
dae5af6be8 Merge pull request #7392 from MathiasVP/fix-join-order-in-is-argument-for-parameter
C++: Fix join order in `isArgumentForParameter`
2021-12-21 09:29:32 +01:00
Tom Hvitved
f66a08155b Merge pull request #7460 from hvitved/ruby/cfg/nested-completion-non-linear-rec
Ruby: Reduce non-linear recursion in CFG completion library
2021-12-20 20:11:00 +01:00
Tom Hvitved
29cd346702 Ruby: Reduce non-linear recursion in CFG completion library
Before

```
noinline
incremental
Completion::nestedEnsureCompletion#ff(/* Completion::Completion */ Completion::TCompletion outer,
                                      int nestLevel)
:-
  (
    (
      Completion::TReturnCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TBreakCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TNextCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRedoCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRetryCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRaiseCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TExitCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    )
  ),
  exists(/* ControlFlowGraphImpl::Trees::BodyStmtTree */ cached dontcare AST::Cached::TAstNode _ |
    ControlFlowGraphImpl::Trees::BodyStmtTree::getNestLevel_dispred#ff(_,
                                                                       nestLevel)
  )
| [base_case] false()
| [delta_order]
  (
    (
      Completion::TReturnCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TBreakCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TNextCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRedoCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRetryCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRaiseCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TExitCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    )
  ),
  project#ControlFlowGraphImpl::Trees::BodyStmtTree::getNestLevel_dispred#ff(nestLevel),
  not(previous rec Completion::nestedEnsureCompletion#ff(outer, nestLevel))
.
```

After

```
noinline
Completion::nestedEnsureCompletion#ff(Completion::TCompletion outer,
                                      int nestLevel)
:-
  (
    Completion::TReturnCompletion#f(outer);
    Completion::TBreakCompletion#f(outer);
    Completion::TNextCompletion#f(outer);
    Completion::TRedoCompletion#f(outer);
    Completion::TRetryCompletion#f(outer);
    Completion::TRaiseCompletion#f(outer);
    Completion::TExitCompletion#f(outer)
  ),
  project#ControlFlowGraphImpl::Trees::BodyStmtTree::getNestLevel_dispred#ff(nestLevel)
.
```
2021-12-20 19:22:47 +01:00
Arthur Baars
6c7114804e Ruby: remove CaseExprChildMapping::getBranch 2021-12-20 19:21:36 +01:00
Arthur Baars
7644d60dae Revert "Ruby: CFG: make WhenExpr post-order"
This reverts commit cff63fa7d7.
2021-12-20 18:57:25 +01:00
Erik Krogh Kristensen
8019b52838 run the non-us patch with "modelled/modeled" 2021-12-20 17:47:15 +01:00
Erik Krogh Kristensen
4c1089fcf1 QL: add "modelled/modeled" to the ql/non-us-spelling query 2021-12-20 17:46:41 +01:00
Erik Krogh Kristensen
66c6a4d899 QL: move ql/non-us-spelling implementation to Query.qll file 2021-12-20 16:53:52 +01:00
Erik Krogh Kristensen
d17879e1f9 run the non-us patch 2021-12-20 16:24:41 +01:00
Mathias Vorreiter Pedersen
aa92fe8c90 Merge pull request #7338 from geoffw0/clrtxt2
C++: Improvements to cpp/cleartext-transmission
2021-12-20 16:05:12 +01:00
Michael Nebel
06b77eb4af C#: Re-introduce callableFlow for Add as the test test/query-tests/Language Abuse/ForeachCapture/ForeachCapture.qlref needs to be re-written before it can be removed. 2021-12-20 16:00:59 +01:00
Tom Hvitved
06575efce9 Data flow: Fix bad join-order 2021-12-20 15:44:16 +01:00
Michael Nebel
d3f2894a8e C#: Convert remaining missing parts of System.Collections.IEnumerable and sub types flow to CSV format (except for 'clearsContent'). 2021-12-20 15:33:26 +01:00
Michael Nebel
0aefb1551e C#: Convert at least System.Collection.[Generic.]ICollection flow to CSV format. 2021-12-20 15:33:26 +01:00
Michael Nebel
e9d4e38364 C#: Convert at least System.Collection.[Generic.]IList flow to CSV format. 2021-12-20 15:33:25 +01:00
Michael Nebel
44c1e3f28d C#: Re-arrange framework imports. 2021-12-20 15:33:25 +01:00
Michael Nebel
aedfc428c2 C#: Convert at least the flow summaries for System.Collections[.Generic].IDictionary and subclasses. 2021-12-20 15:33:25 +01:00
Michael Nebel
b78ec4c693 C#: Add flow summary for System.Collections.IEnumerable in CSV format. 2021-12-20 15:33:25 +01:00
Michael Nebel
20637555b5 C#: Manual cleanup of previously added IEnumerable<>.GetEnumrator flow summaries. 2021-12-20 15:33:25 +01:00
Michael Nebel
ac5b2bfa41 C#: Add flow summary for IEnumerable<T>.GetEnumerator() and update tests. 2021-12-20 15:33:25 +01:00
Michael Nebel
f93c63aa60 C#: Convert flow summaries for extension methods for subtypes of System.Collection.IEnumerable to CSV format. 2021-12-20 15:33:25 +01:00
Michael Nebel
ec4d43fed2 C#: Add missing dataflow comment in CompilerServices. 2021-12-20 15:33:25 +01:00
Erik Krogh Kristensen
2f559696e4 QL: add "modelling/modeling" to ql/non-us-spelling 2021-12-20 15:30:46 +01:00
Tom Hvitved
aa9444b16c Address review comment 2021-12-20 15:24:14 +01:00
Nick Rolfe
f18492e39b Merge pull request #7443 from github/nickrolfe/behavior
QL4QL: catch behaviour/behavior in ql/non-us-spelling
2021-12-20 13:23:53 +00:00
Erik Krogh Kristensen
9ffdfb263f Merge pull request #7441 from erik-krogh/ql-for-ql-next
QL-for-QL: Followup changes
2021-12-20 10:58:13 +01:00
Alex Ford
313e0c63fd Merge pull request #7399 from github/ruby/stdlib-logger
Ruby: Model what is written to the log from stdlib `Logger` methods
2021-12-20 09:52:29 +00:00
Erik Krogh Kristensen
8b53cca3e8 QL: use environment instead of dynamic shell script construction
Co-authored-by: Esben Sparre Andreasen <esbena@github.com>
2021-12-20 09:56:10 +01:00
Tom Hvitved
ed006d7283 Merge pull request #7231 from hvitved/csharp/dataflow/consistency-queries
C#: Enable data-flow consistency queries
2021-12-20 08:46:19 +01:00
Erik Krogh Kristensen
af47cba09a QL: fix the remaining code-scanning errors for QL-for-QL 2021-12-17 21:34:13 +01:00