Commit Graph

429 Commits

Author SHA1 Message Date
Michael Nebel
76fd424795 C#: Turn isAutogenerated predicate into a predicate without result. 2022-05-09 07:30:06 +02:00
Michael Nebel
a8556f4d50 C#: Make sure that test output prints whether the summary is generated or not. 2022-05-05 13:07:22 +02:00
Michael Nebel
e416a0629a C#: Add isAutoGenerated predicate to SummarizedCallable. 2022-05-05 11:54:04 +02:00
Tom Hvitved
66a9759329 Merge pull request #8870 from hvitved/dataflow/expect-content
Data flow: Introduce `expectsContent`
2022-05-05 09:01:40 +02:00
Tom Hvitved
8e33653d25 Merge pull request #9017 from hvitved/dataflow/subpaths-perf
Data flow: Speedup `subpaths` predicate
2022-05-04 16:37:52 +02:00
Tom Hvitved
9cb63c0a5e Data flow: Sync files 2022-05-04 14:49:26 +02:00
Tom Hvitved
7f7742216c Address review comment
This reverts commit 2b4fde74bb.
2022-05-04 14:49:03 +02:00
Tom Hvitved
74e99302d6 Address review comments 2022-05-04 09:57:59 +02:00
Tom Hvitved
da72ba46d4 Data flow: Add stub expectsContent for all languages 2022-05-04 09:57:59 +02:00
Tom Hvitved
6e2e8440eb Data flow: Sync files 2022-05-04 09:57:59 +02:00
Tom Hvitved
e9c8f979f9 Data flow: Sync files 2022-05-03 11:46:51 +02:00
Tom Hvitved
2b4fde74bb Data flow: Speedup subpaths predicate
Before
```
[2022-05-02 15:47:16] (1280s) Tuple counts for DataFlowImpl::Subpaths::subpaths#656de156#ffff/4@c5f3dclb after 3m22s:
                      8389013    ~4%     {5} r1 = JOIN DataFlowImpl::Subpaths::subpaths#656de156#ffff#shared WITH DataFlowImpl::PathNode::getASuccessor#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg', Lhs.1, Lhs.2, Lhs.3, Lhs.4 'out'
                      6689751    ~0%     {4} r2 = JOIN r1 WITH DataFlowImpl::Subpaths::subpaths03#656de156#ffffff_034512#join_rhs ON FIRST 4 OUTPUT Rhs.4, Lhs.4 'out', Lhs.0 'arg', Rhs.5 'ret'

                      1513839768 ~1%     {5} r3 = JOIN r2 WITH DataFlowImpl::PathNodeImpl::getNodeEx#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'out', Lhs.2 'arg', Lhs.3 'ret', Rhs.1 'par', Lhs.3 'ret'
                      1513839768 ~1%     {5} r4 = r3 AND NOT DataFlowImpl::PathNodeImpl::isHidden#dispred#f0820431#f(Lhs.4 'ret')
                      1513839768 ~5%     {4} r5 = SCAN r4 OUTPUT In.1 'arg', In.3 'par', In.0 'out', In.4 'ret'

                      1513839768 ~2%     {4} r6 = JOIN r2 WITH DataFlowImpl::PathNodeImpl::getNodeEx#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.3 'ret', Lhs.1 'out', Lhs.2 'arg', Rhs.1 'par'
                      0          ~0%     {5} r7 = JOIN r6 WITH boundedFastTC(DataFlowImpl::Subpaths::localStepToHidden#656de156#ff_10#higher_order_body,DataFlowImpl::Subpaths::subpaths#656de156#ffff#higher_order_body) ON FIRST 1 OUTPUT Lhs.1 'out', Lhs.2 'arg', Lhs.0, Lhs.3 'par', Rhs.1 'ret'
                      0          ~0%     {5} r8 = r7 AND NOT DataFlowImpl::PathNodeImpl::isHidden#dispred#f0820431#f(Lhs.4 'ret')
                      0          ~0%     {4} r9 = SCAN r8 OUTPUT In.1 'arg', In.3 'par', In.0 'out', In.4 'ret'

                      1513839768 ~5%     {4} r10 = r5 UNION r9
                      6689751    ~0%     {4} r11 = JOIN r10 WITH DataFlowImpl::PathNode::getASuccessor#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.0 'arg', Lhs.1 'par', Lhs.3 'ret', Lhs.2 'out'
                                         return r11
```

After
```
[2022-05-03 11:44:10] (969s) Tuple counts for DataFlowImpl::Subpaths::subpaths#656de156#ffff/4@b26b969r after 11.8s:
                      8372525 ~0%     {3} r1 = JOIN DataFlowImpl::PathNode::getASuccessor#dispred#f0820431#ff_10#join_rhs WITH DataFlowImpl::PathNodeImpl::getNodeEx#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1 'arg', Rhs.1, Rhs.0
                      6673799 ~6%     {9} r2 = JOIN r1 WITH DataFlowImpl::Subpaths::subpaths03#656de156#fffffffff ON FIRST 2 OUTPUT Rhs.3, Rhs.4, Rhs.5, Rhs.7, Rhs.6, Rhs.8, Lhs.2 'par', Lhs.0 'arg', Rhs.2 'ret'

                      6637884 ~0%     {5} r3 = JOIN r2 WITH project#DataFlowImpl::pathNode#656de156#ffffffff_1234560#join_rhs ON FIRST 6 OUTPUT Lhs.6 'par', Lhs.7 'arg', Lhs.8 'ret', Rhs.6 'out', Lhs.8 'ret'

                      6637884 ~0%     {4} r4 = JOIN r2 WITH project#DataFlowImpl::pathNode#656de156#ffffffff_1234560#join_rhs ON FIRST 6 OUTPUT Rhs.6 'out', Lhs.6 'par', Lhs.7 'arg', Lhs.8 'ret'

                      51867   ~0%     {5} r5 = JOIN r4 WITH DataFlowImpl::PathNodeMid::projectToSink#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1 'par', Lhs.2 'arg', Lhs.3 'ret', Rhs.1 'out', Lhs.3 'ret'

                      6689751 ~0%     {5} r6 = r3 UNION r5
                      6689751 ~0%     {5} r7 = r6 AND NOT DataFlowImpl::PathNodeImpl::isHidden#dispred#f0820431#f(Lhs.4 'ret')
                      6689751 ~0%     {4} r8 = SCAN r7 OUTPUT In.1 'arg', In.0 'par', In.4 'ret', In.3 'out'

                      6637884 ~0%     {4} r9 = JOIN r2 WITH project#DataFlowImpl::pathNode#656de156#ffffffff_1234560#join_rhs ON FIRST 6 OUTPUT Lhs.8 'ret', Lhs.6 'par', Lhs.7 'arg', Rhs.6 'out'

                      51867   ~0%     {4} r10 = JOIN r4 WITH DataFlowImpl::PathNodeMid::projectToSink#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.3 'ret', Lhs.1 'par', Lhs.2 'arg', Rhs.1 'out'

                      6689751 ~0%     {4} r11 = r9 UNION r10
                      0       ~0%     {5} r12 = JOIN r11 WITH boundedFastTC(DataFlowImpl::Subpaths::localStepToHidden#656de156#ff_10#higher_order_body,DataFlowImpl::Subpaths::subpaths#656de156#ffff#higher_order_body) ON FIRST 1 OUTPUT Lhs.1 'par', Lhs.2 'arg', Lhs.0, Lhs.3 'out', Rhs.1 'ret'
                      0       ~0%     {5} r13 = r12 AND NOT DataFlowImpl::PathNodeImpl::isHidden#dispred#f0820431#f(Lhs.4 'ret')
                      0       ~0%     {4} r14 = SCAN r13 OUTPUT In.1 'arg', In.0 'par', In.4 'ret', In.3 'out'

                      6689751 ~0%     {4} r15 = r8 UNION r14
                                      return r15
```
2022-05-03 11:45:28 +02:00
Chuan-kai Lin
d6f0bbb816 Fix syntax errors in QL comments 2022-04-28 11:53:36 -07:00
Erik Krogh Kristensen
d389012b75 Merge branch 'main' into redundantImport 2022-04-26 14:24:51 +02:00
Anders Schack-Mulligen
59aedc2872 Merge pull request #8853 from aschackmull/dataflow/fix-join
Dataflow: Fix join-on-config producing a CP.
2022-04-26 09:52:50 +02:00
Anders Schack-Mulligen
c06efa1f42 Dataflow: Sync. 2022-04-25 13:11:04 +02:00
Anders Schack-Mulligen
40a16325a9 Minor clean-up in AccessPathSyntax. 2022-04-25 12:27:48 +02:00
Tom Hvitved
2466288656 Data flow: Simplify revFlowStore 2022-04-25 10:11:54 +02:00
Tom Hvitved
bc6ee10583 Data flow: Sync files 2022-04-22 15:10:00 +02:00
Tom Hvitved
b033f107df Merge remote-tracking branch 'upstream/main' into dataflow/interpret-read-store 2022-04-22 14:35:02 +02:00
Erik Krogh Kristensen
ff73dbc35c delete redundant imports 2022-04-22 12:55:28 +02:00
Tom Hvitved
bd09c61504 Merge pull request #8786 from hvitved/ruby/dataflow/argument-tokens
Ruby: Implement `Argument[any]` and `Argument[n..]`
2022-04-21 16:31:24 +02:00
Tom Hvitved
ea229d361c Sync files 2022-04-20 13:55:18 +02:00
Anders Schack-Mulligen
b521d64156 Dataflow: Sync. 2022-04-19 15:29:35 +02:00
Mathias Vorreiter Pedersen
91b413d59f Dataflow: Sync identical files. 2022-04-19 09:57:21 +01:00
Anders Schack-Mulligen
7beed570f2 Dataflow: Sync. 2022-04-07 13:53:48 +02:00
Tom Hvitved
4099d1318f Data flow: Tweak two join-orders
Before
```
[2022-04-06 13:19:29] (96s) Tuple counts for DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff/2@i14#aa10f2wi after 4.4s:
                      10681    ~0%     {2} r1 = SCAN DataFlowImpl2::Stage1::revFlow#7ad53399#fff#prev_delta OUTPUT In.0, In.2 'config'
                      982      ~1%     {3} r2 = JOIN r1 WITH DataFlowImpl2::readSet#7ad53399#ffff_2301#join_rhs ON FIRST 2 OUTPUT Rhs.3, Lhs.1 'config', Rhs.2
                      83691528 ~2%     {3} r3 = JOIN r2 WITH DataFlowPublic::ContentSet::getAReadContent#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1 'config', Lhs.2, Rhs.1 'c'
                      83581763 ~2%     {3} r4 = r3 AND NOT DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff#prev(Lhs.2 'c', Lhs.0 'config')
                      83581763 ~0%     {3} r5 = SCAN r4 OUTPUT In.2 'c', In.0 'config', In.1
                      0        ~0%     {3} r6 = JOIN r5 WITH DataFlowImpl2::Stage1::fwdFlowConsCand#7ad53399#ff ON FIRST 2 OUTPUT Lhs.2, Lhs.1 'config', Lhs.0 'c'
                      0        ~0%     {2} r7 = JOIN r6 WITH DataFlowImpl2::Stage1::fwdFlow#7ad53399#2#fff_02#join_rhs ON FIRST 2 OUTPUT Lhs.2 'c', Lhs.1 'config'
                                       return r7
```

After
```
[2022-04-06 13:44:38] (6s) Tuple counts for DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff/2@i14#5abbf2wn after 6ms:
                      10681  ~0%     {2} r1 = SCAN DataFlowImpl2::Stage1::revFlow#7ad53399#fff#prev_delta OUTPUT In.0, In.2 'config'
                      982    ~1%     {3} r2 = JOIN r1 WITH DataFlowImpl2::readSet#7ad53399#ffff_2301#join_rhs ON FIRST 2 OUTPUT Rhs.3, Lhs.1 'config', Rhs.2
                      109765 ~0%     {3} r3 = JOIN r2 WITH DataFlowImpl2::Stage1::fwdFlowConsCandSet#7ad53399#fff#reorder_0_2_1 ON FIRST 2 OUTPUT Lhs.1 'config', Lhs.2, Rhs.2 'c'
                      0      ~0%     {3} r4 = r3 AND NOT DataFlowImpl2::Stage1::revFlowConsCand#7ad53399#ff#prev(Lhs.2 'c', Lhs.0 'config')
                      0      ~0%     {3} r5 = SCAN r4 OUTPUT In.1, In.0 'config', In.2 'c'
                      0      ~0%     {2} r6 = JOIN r5 WITH DataFlowImpl2::Stage1::fwdFlow#7ad53399#2#fff_02#join_rhs ON FIRST 2 OUTPUT Lhs.2 'c', Lhs.1 'config'
                                     return r6
```
2022-04-06 13:52:30 +02:00
Michael Nebel
2562910b94 C#: Update Csv validation to allow sources and sink kinds to be prefixed with generated. 2022-04-05 14:25:34 +02:00
Michael Nebel
3937714f9f C#: The CaptureSummaryModels query should only produce summary models that will not be discarded at run-time. 2022-04-05 08:55:12 +02:00
Michael Nebel
784327c183 Java/Ruby: Hardcode generated flag to false. 2022-04-05 08:55:12 +02:00
Michael Nebel
de76df3988 C#: Only use generated summaries, if no handwritten model exist for a particular dataflow callable. 2022-04-05 08:55:12 +02:00
Michael Nebel
26ad4861a3 C#: Introduce parsing of the kind field. 2022-04-05 08:55:12 +02:00
Michael Nebel
f8b094ac1f C#: Only use generated flow summaries in case no handwritten summary exists. 2022-04-05 08:55:11 +02:00
Michael Nebel
4d953da480 C#: Initial steps to allow generated as a part of the kind. 2022-04-05 08:55:11 +02:00
Michael Nebel
3fe941aae2 C#: Add missing empty ext column in generated summaries. 2022-04-04 15:58:35 +02:00
Tom Hvitved
7113c1b29c C#: Implement ContentSet 2022-04-04 13:51:44 +02:00
Tom Hvitved
c4fbc618a9 Data flow: Sync files 2022-04-04 13:51:44 +02:00
Chris Smowton
28fa49dcd6 dataflow -> data-flow 2022-04-01 13:22:58 +01:00
Chris Smowton
3b0bd3bc0f Improve wording 2022-04-01 11:31:31 +01:00
Chris Smowton
99026a6071 Improve wording of isAdditionalFlow/TaintStep qldoc 2022-04-01 11:07:27 +01:00
Michael Nebel
c139850cd6 Merge pull request #8609 from michaelnebel/csharp/operatorsummaries
C#: Operator flow
2022-04-01 09:04:04 +02:00
Michael Nebel
8b08ddf7ad C#: Use callables qualified name instead of name, when printing summary like information. 2022-03-30 15:05:57 +02:00
Michael Nebel
4163078ecc C#: The qualified name of the operator should use the function name instead of the displayed name. 2022-03-30 15:05:03 +02:00
Michael Nebel
bfb206c810 C#: Let ExternalApi extend DataFlowCallable instead of Callable. 2022-03-29 14:52:57 +02:00
Michael Nebel
e9070b010b C#: Add getCall predicate to ArgumentNode. 2022-03-29 14:49:37 +02:00
Michael Nebel
5970fd9904 C#: Also include property reads in possible new sink discovery. Only include public fields and properties. 2022-03-29 11:07:57 +02:00
Michael Nebel
6804e20e4a Merge pull request #8451 from michaelnebel/csharp/modelgenerator-improvements
C#: Model generator improvements and more tests
2022-03-23 13:30:58 +01:00
Michael Nebel
b95a332ded C#: Simplify the isCollectionType predicate. 2022-03-22 11:25:14 +01:00
Michael Nebel
1d45996001 Merge pull request #8466 from michaelnebel/csharp/refactor-aspartial
C#: Refactor asPartial to allow re-use.
2022-03-22 10:54:54 +01:00
Michael Nebel
92f8a90f31 C#: Introduce a collectionType predicate. 2022-03-21 14:44:38 +01:00