Commit Graph

33884 Commits

Author SHA1 Message Date
Tom Hvitved
05e37a7465 C#: Promote existing ad-hoc consistency checks to consistency queries 2021-12-22 13:05:58 +01:00
Tom Hvitved
4133eb15d5 Ruby: Reintroduce old Argument[_] restriction to avoid large Cartesian product 2021-12-22 11:37:38 +01:00
Tom Hvitved
d196c77b3d Ruby: Remove some redundant overrides 2021-12-22 11:25:13 +01:00
Tom Hvitved
f5471e34f8 C#: Fix bad join-order in dispatch library
Before
```
[2021-12-22 09:46:31] (395s) Tuple counts for Dispatch::Internal::hasCallable#fff/3@258418l2 after 5m27s:
                      49000       ~0%      {2} r1 = JOIN Declaration::Declaration::getUnboundDeclaration_dispred#ff_10#join_rhs WITH project#Dispatch::Internal::DispatchMethodOrAccessorCall::getAStaticTargetExt#ff ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.0

                      31302       ~3%      {3} r2 = JOIN r1 WITH Type::ValueOrRefType::getAMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0 'c', Lhs.1 'source', Rhs.1

                      299700      ~0%      {3} r3 = JOIN r1 WITH Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0 'c', Lhs.1 'source', Rhs.1

                      16650       ~1%      {3} r4 = JOIN r1 WITH Property::Accessor::getDeclaration_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'source', Lhs.0 'c'

                      15984       ~0%      {3} r5 = JOIN r4 WITH Type::ValueOrRefType::getAMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      315684      ~1%      {3} r6 = r3 UNION r5
                      346986      ~1%      {3} r7 = r2 UNION r6

                      0           ~0%      {3} r8 = JOIN r4 WITH Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      666         ~0%      {3} r9 = JOIN r1 WITH Type::hasNonOverriddenMember#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'source', Lhs.0 'c'
                      0           ~0%      {3} r10 = JOIN r9 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable#fff#higher_order_body) ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      0           ~0%      {3} r11 = JOIN r4 WITH Type::hasNonOverriddenMember#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'source', Lhs.2 'c'
                      0           ~0%      {3} r12 = JOIN r11 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable#fff#higher_order_body#1) ON FIRST 1 OUTPUT Lhs.2 'c', Lhs.1 'source', Rhs.1

                      0           ~0%      {3} r13 = r10 UNION r12
                      0           ~0%      {3} r14 = r8 UNION r13
                      346986      ~1%      {3} r15 = r7 UNION r14
                      11963234000 ~2%      {4} r16 = JOIN r15 WITH Dispatch::Internal::hasOverrider#ff ON FIRST 1 OUTPUT Lhs.2, Rhs.1 't', Lhs.1 'source', Lhs.0 'c'
                      207126      ~27%     {3} r17 = JOIN r16 WITH Unification::Gvn::Cached::getGlobalValueNumber#ff ON FIRST 2 OUTPUT Lhs.2 'source', Lhs.1 't', Lhs.3 'c'
                                           return r17
```

After
```
[2021-12-22 10:39:41] (0s) Tuple counts for Dispatch::Internal::hasCallable0#fff/3@82341e2h after 331ms:
                      93569  ~0%      {2} r1 = JOIN Type::ValueOrRefType::getAMember_dispred#fb_10#join_rhs WITH OverridableCallable::OverridableCallable#f ON FIRST 1 OUTPUT Rhs.0 'c', Lhs.1

                      511767 ~0%      {2} r2 = JOIN Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs WITH OverridableCallable::OverridableCallable#f ON FIRST 1 OUTPUT Rhs.0 'c', Lhs.1

                      35659  ~0%      {2} r3 = JOIN OverridableCallable::OverridableCallable#f WITH Property::Accessor::getDeclaration_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'c'

                      35659  ~4%      {2} r4 = JOIN r3 WITH Type::ValueOrRefType::getAMember_dispred#fb_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      547426 ~0%      {2} r5 = r2 UNION r4
                      640995 ~4%      {2} r6 = r1 UNION r5

                      74835  ~4%      {2} r7 = JOIN r3 WITH Type::ValueOrRefType::hasOverriddenMember_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      32748  ~0%      {2} r8 = JOIN Type::hasNonOverriddenMember#fb_10#join_rhs WITH OverridableCallable::OverridableCallable#f ON FIRST 1 OUTPUT Lhs.1, Rhs.0 'c'
                      171228 ~0%      {2} r9 = JOIN r8 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable0#fff#higher_order_body) ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      9056   ~0%      {2} r10 = JOIN r3 WITH Type::hasNonOverriddenMember#fb_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'c'
                      23633  ~1%      {2} r11 = JOIN r10 WITH boundedFastTC(Type::ValueOrRefType::getBaseClass_dispred#ff_10#join_rhs,Dispatch::Internal::hasCallable0#fff#higher_order_body#1) ON FIRST 1 OUTPUT Lhs.1 'c', Rhs.1

                      194861 ~0%      {2} r12 = r9 UNION r11
                      269696 ~0%      {2} r13 = r7 UNION r12
                      910691 ~4%      {2} r14 = r6 UNION r13
                      910691 ~2%      {3} r15 = JOIN r14 WITH Declaration::Declaration::getUnboundDeclaration_dispred#ff ON FIRST 1 OUTPUT Rhs.1 'source', Lhs.0 'c', Lhs.1
                      579872 ~2%      {3} r16 = JOIN r15 WITH project#Dispatch::Internal::DispatchMethodOrAccessorCall::getAStaticTargetExt#ff ON FIRST 1 OUTPUT Lhs.2, Lhs.1 'c', Lhs.0 'source'
                      753465 ~41%     {3} r17 = JOIN r16 WITH Unification::Gvn::Cached::getGlobalValueNumber#ff ON FIRST 1 OUTPUT Rhs.1 't', Lhs.1 'c', Lhs.2 'source'
                                      return r17

[2021-12-22 10:39:41] (0s) Tuple counts for Dispatch::Internal::hasCallable#fff/3@e44e67tv after 24ms:
                      201843 ~0%     {3} r1 = JOIN Dispatch::Internal::hasOverrider#ff WITH Dispatch::Internal::hasCallable0#fff ON FIRST 2 OUTPUT Lhs.0 't', Lhs.1 'c', Rhs.2 'source'
                                     return r1
```
2021-12-22 10:45:51 +01:00
Tamás Vajk
43b5d502b8 Merge pull request #7466 from github/workflow/coverage/update
Update CSV framework coverage reports
2021-12-22 09:05:15 +01:00
github-actions[bot]
ba7a10de68 Add changed framework coverage reports 2021-12-22 00:10:19 +00:00
Nick Rolfe
9e259b67bb Merge pull request #7305 from github/nickrolfe/user-controlled-bypass
Ruby: query to find user-controlled bypass of sensitive actions
2021-12-21 17:20:20 +00:00
Arthur Baars
a7aff11140 Merge pull request #7394 from aibaars/ruby-cfg-expr-post
Ruby: CFG: make all expressions "post-order" nodes
2021-12-21 16:36:42 +01:00
Nick Rolfe
5765f3684c Ruby: add missing qldoc comment 2021-12-21 15:29:16 +00:00
Nick Rolfe
5db80dac51 Merge remote-tracking branch 'origin/main' into nickrolfe/user-controlled-bypass 2021-12-21 15:26:08 +00:00
Michael Nebel
c138a2796f Merge pull request #7424 from michaelnebel/csharp-flow-summary-csv
C#: Flow summaries in CSV format.
2021-12-21 16:11:22 +01:00
Michael Nebel
8250fb4cf7 C#: Fixed typo in namespace. 2021-12-21 15:00:05 +01:00
Arthur Baars
a86ba3b14e Ruby: rename WhenExpr to WhenClause 2021-12-21 12:31:24 +01:00
Mathias Vorreiter Pedersen
dae5af6be8 Merge pull request #7392 from MathiasVP/fix-join-order-in-is-argument-for-parameter
C++: Fix join order in `isArgumentForParameter`
2021-12-21 09:29:32 +01:00
Mathias Vorreiter Pedersen
5a38f81e23 C++: Accept test changes. 2021-12-21 08:08:59 +01:00
Tom Hvitved
f66a08155b Merge pull request #7460 from hvitved/ruby/cfg/nested-completion-non-linear-rec
Ruby: Reduce non-linear recursion in CFG completion library
2021-12-20 20:11:00 +01:00
Tom Hvitved
29cd346702 Ruby: Reduce non-linear recursion in CFG completion library
Before

```
noinline
incremental
Completion::nestedEnsureCompletion#ff(/* Completion::Completion */ Completion::TCompletion outer,
                                      int nestLevel)
:-
  (
    (
      Completion::TReturnCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TBreakCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TNextCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRedoCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRetryCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRaiseCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TExitCompletion#f(outer),
      rec Completion::Completion#class#f(outer)
    )
  ),
  exists(/* ControlFlowGraphImpl::Trees::BodyStmtTree */ cached dontcare AST::Cached::TAstNode _ |
    ControlFlowGraphImpl::Trees::BodyStmtTree::getNestLevel_dispred#ff(_,
                                                                       nestLevel)
  )
| [base_case] false()
| [delta_order]
  (
    (
      Completion::TReturnCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TBreakCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TNextCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRedoCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRetryCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TRaiseCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    );
    (
      Completion::TExitCompletion#f(outer),
      delta previous rec Completion::Completion#class#f(outer)
    )
  ),
  project#ControlFlowGraphImpl::Trees::BodyStmtTree::getNestLevel_dispred#ff(nestLevel),
  not(previous rec Completion::nestedEnsureCompletion#ff(outer, nestLevel))
.
```

After

```
noinline
Completion::nestedEnsureCompletion#ff(Completion::TCompletion outer,
                                      int nestLevel)
:-
  (
    Completion::TReturnCompletion#f(outer);
    Completion::TBreakCompletion#f(outer);
    Completion::TNextCompletion#f(outer);
    Completion::TRedoCompletion#f(outer);
    Completion::TRetryCompletion#f(outer);
    Completion::TRaiseCompletion#f(outer);
    Completion::TExitCompletion#f(outer)
  ),
  project#ControlFlowGraphImpl::Trees::BodyStmtTree::getNestLevel_dispred#ff(nestLevel)
.
```
2021-12-20 19:22:47 +01:00
Arthur Baars
6c7114804e Ruby: remove CaseExprChildMapping::getBranch 2021-12-20 19:21:36 +01:00
Arthur Baars
7644d60dae Revert "Ruby: CFG: make WhenExpr post-order"
This reverts commit cff63fa7d7.
2021-12-20 18:57:25 +01:00
Erik Krogh Kristensen
8019b52838 run the non-us patch with "modelled/modeled" 2021-12-20 17:47:15 +01:00
Erik Krogh Kristensen
4c1089fcf1 QL: add "modelled/modeled" to the ql/non-us-spelling query 2021-12-20 17:46:41 +01:00
Erik Krogh Kristensen
66c6a4d899 QL: move ql/non-us-spelling implementation to Query.qll file 2021-12-20 16:53:52 +01:00
Erik Krogh Kristensen
d17879e1f9 run the non-us patch 2021-12-20 16:24:41 +01:00
Mathias Vorreiter Pedersen
aa92fe8c90 Merge pull request #7338 from geoffw0/clrtxt2
C++: Improvements to cpp/cleartext-transmission
2021-12-20 16:05:12 +01:00
Michael Nebel
06b77eb4af C#: Re-introduce callableFlow for Add as the test test/query-tests/Language Abuse/ForeachCapture/ForeachCapture.qlref needs to be re-written before it can be removed. 2021-12-20 16:00:59 +01:00
Tom Hvitved
06575efce9 Data flow: Fix bad join-order 2021-12-20 15:44:16 +01:00
Michael Nebel
d3f2894a8e C#: Convert remaining missing parts of System.Collections.IEnumerable and sub types flow to CSV format (except for 'clearsContent'). 2021-12-20 15:33:26 +01:00
Michael Nebel
0aefb1551e C#: Convert at least System.Collection.[Generic.]ICollection flow to CSV format. 2021-12-20 15:33:26 +01:00
Michael Nebel
e9d4e38364 C#: Convert at least System.Collection.[Generic.]IList flow to CSV format. 2021-12-20 15:33:25 +01:00
Michael Nebel
44c1e3f28d C#: Re-arrange framework imports. 2021-12-20 15:33:25 +01:00
Michael Nebel
aedfc428c2 C#: Convert at least the flow summaries for System.Collections[.Generic].IDictionary and subclasses. 2021-12-20 15:33:25 +01:00
Michael Nebel
b78ec4c693 C#: Add flow summary for System.Collections.IEnumerable in CSV format. 2021-12-20 15:33:25 +01:00
Michael Nebel
20637555b5 C#: Manual cleanup of previously added IEnumerable<>.GetEnumrator flow summaries. 2021-12-20 15:33:25 +01:00
Michael Nebel
ac5b2bfa41 C#: Add flow summary for IEnumerable<T>.GetEnumerator() and update tests. 2021-12-20 15:33:25 +01:00
Michael Nebel
f93c63aa60 C#: Convert flow summaries for extension methods for subtypes of System.Collection.IEnumerable to CSV format. 2021-12-20 15:33:25 +01:00
Michael Nebel
ec4d43fed2 C#: Add missing dataflow comment in CompilerServices. 2021-12-20 15:33:25 +01:00
Erik Krogh Kristensen
2f559696e4 QL: add "modelling/modeling" to ql/non-us-spelling 2021-12-20 15:30:46 +01:00
Tom Hvitved
aa9444b16c Address review comment 2021-12-20 15:24:14 +01:00
Nick Rolfe
f18492e39b Merge pull request #7443 from github/nickrolfe/behavior
QL4QL: catch behaviour/behavior in ql/non-us-spelling
2021-12-20 13:23:53 +00:00
Mathias Vorreiter Pedersen
bbb936154a C++: Increase the precision of 'cpp/uncontrolled-arithmetic' to high. 2021-12-20 14:03:13 +01:00
Mathias Vorreiter Pedersen
95fa93b274 C++: Only recognize signed integers as sinks in 'cpp/uncontrolled-arithmetic' in the case of overflow. 2021-12-20 14:02:44 +01:00
Erik Krogh Kristensen
9ffdfb263f Merge pull request #7441 from erik-krogh/ql-for-ql-next
QL-for-QL: Followup changes
2021-12-20 10:58:13 +01:00
Alex Ford
313e0c63fd Merge pull request #7399 from github/ruby/stdlib-logger
Ruby: Model what is written to the log from stdlib `Logger` methods
2021-12-20 09:52:29 +00:00
Erik Krogh Kristensen
8b53cca3e8 QL: use environment instead of dynamic shell script construction
Co-authored-by: Esben Sparre Andreasen <esbena@github.com>
2021-12-20 09:56:10 +01:00
haby0
fed1d88268 Add shutil module path injection sinks 2021-12-20 16:09:06 +08:00
Tom Hvitved
ed006d7283 Merge pull request #7231 from hvitved/csharp/dataflow/consistency-queries
C#: Enable data-flow consistency queries
2021-12-20 08:46:19 +01:00
jorgectf
1f1b7a54f8 Update .expected 2021-12-19 18:58:43 +01:00
jorgectf
b6bdcd0eb8 Delete redundant exists() 2021-12-19 18:57:22 +01:00
jorgectf
98c8503ebd Fix test mismatch 2021-12-19 18:35:53 +01:00
jorgectf
f82ed8573e Model python_jwt.process_jwt 2021-12-19 18:32:14 +01:00