Commit Graph

1100 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
c9a833fc07 Ruby: Fix performance for argumentPositionMatch
before

[2022-05-24 17:29:07] (50s) Tuple counts for DataFlowImplCommon::argumentPositionMatch#4f8df883#fff/3@03b4073c after 35.8s:
                      156250456 ~2%     {4} r1 = JOIN DataFlowDispatch::Cached::TParameterPosition#36b84300#f WITH DataFlowImplCommon::ArgNode::argumentOf#dispred#f0820431#fff CARTESIAN PRODUCT OUTPUT Rhs.2, Lhs.0 'ppos', Rhs.0, Rhs.1 'call'

                      0         ~0%     {3} r2 = JOIN r1 WITH DataFlowDispatch::Cached::TAnyKeywordArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1 'ppos', Lhs.2 'arg', Lhs.3 'call'
                      0         ~0%     {3} r3 = JOIN r2 WITH DataFlowDispatch::Cached::TKeywordParameterPosition#36b84300#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.2 'call', Lhs.1 'arg', Lhs.0 'ppos'

                      156250456 ~2%     {4} r4 = JOIN DataFlowDispatch::Cached::TParameterPosition#36b84300#f WITH DataFlowImplCommon::ArgNode::argumentOf#dispred#f0820431#fff CARTESIAN PRODUCT OUTPUT Lhs.0 'ppos', Rhs.0, Rhs.1 'call', Rhs.2

                      252424    ~0%     {4} r5 = JOIN r4 WITH DataFlowDispatch::Cached::TSelfParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.3, Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call'
                      121009    ~0%     {3} r6 = JOIN r5 WITH DataFlowDispatch::Cached::TSelfArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.3 'call', Lhs.2 'arg', Lhs.1 'ppos'

                      121009    ~0%     {3} r7 = r3 UNION r6

                      252424    ~0%     {4} r8 = JOIN r4 WITH DataFlowDispatch::Cached::TBlockParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.3, Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call'
                      11764     ~5%     {3} r9 = JOIN r8 WITH DataFlowDispatch::Cached::TBlockArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.3 'call', Lhs.2 'arg', Lhs.1 'ppos'

                      252424    ~2%     {4} r10 = JOIN r4 WITH DataFlowDispatch::Cached::TAnyKeywordParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.3, Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call'
                      20865     ~2%     {3} r11 = JOIN r10 WITH DataFlowDispatch::Cached::TKeywordArgumentPosition#36b84300#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.3 'call', Lhs.2 'arg', Lhs.1 'ppos'

                      32629     ~4%     {3} r12 = r9 UNION r11
                      153638    ~4%     {3} r13 = r7 UNION r12

                      252424    ~1%     {4} r14 = JOIN r4 WITH DataFlowDispatch::Cached::TAnyParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call', Lhs.3
                      131415    ~0%     {4} r15 = r14 AND NOT DataFlowDispatch::Cached::TSelfArgumentPosition#36b84300#f(Lhs.3)
                      131415    ~0%     {3} r16 = SCAN r15 OUTPUT In.2 'call', In.1 'arg', In.0 'ppos'

                      0         ~0%     {4} r17 = JOIN r1 WITH DataFlowDispatch::Cached::TAnyArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1 'ppos', Lhs.2 'arg', Lhs.3 'call', Lhs.0
                      0         ~0%     {4} r18 = r17 AND NOT DataFlowDispatch::Cached::TSelfParameterPosition#36b84300#f(Lhs.0 'ppos')
                      0         ~0%     {3} r19 = SCAN r18 OUTPUT In.2 'call', In.1 'arg', In.0 'ppos'

                      131415    ~0%     {3} r20 = r16 UNION r19

                      5553328   ~5%     {5} r21 = JOIN r4 WITH DataFlowDispatch::Cached::TPositionalParameterPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.3, Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call'
                      98201     ~0%     {3} r22 = JOIN r21 WITH DataFlowDispatch::Cached::TPositionalArgumentPosition#36b84300#ff ON FIRST 2 OUTPUT Lhs.4 'call', Lhs.3 'arg', Lhs.2 'ppos'

                      149435008 ~0%     {5} r23 = JOIN r4 WITH DataFlowDispatch::Cached::TKeywordParameterPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.3, Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call'
                      17930     ~3%     {3} r24 = JOIN r23 WITH DataFlowDispatch::Cached::TKeywordArgumentPosition#36b84300#ff ON FIRST 2 OUTPUT Lhs.4 'call', Lhs.3 'arg', Lhs.2 'ppos'

                      252424    ~0%     {5} r25 = JOIN r4 WITH DataFlowDispatch::Cached::TPositionalParameterLowerBoundPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.3, Lhs.0 'ppos', Lhs.1 'arg', Lhs.2 'call', Rhs.1
                      98786     ~0%     {6} r26 = JOIN r25 WITH DataFlowDispatch::Cached::TPositionalArgumentPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'ppos', Lhs.2 'arg', Lhs.3 'call', Lhs.0, Lhs.4, Rhs.1
                      98786     ~0%     {6} r27 = SELECT r26 ON In.5 >= In.4
                      98786     ~3%     {3} r28 = SCAN r27 OUTPUT In.2 'call', In.1 'arg', In.0 'ppos'

                      116716    ~0%     {3} r29 = r24 UNION r28
                      214917    ~0%     {3} r30 = r22 UNION r29
                      346332    ~0%     {3} r31 = r20 UNION r30
                      499970    ~1%     {3} r32 = r13 UNION r31
                                        return r32

now

[2022-05-24 17:26:06] (14s) Tuple counts for DataFlowImplCommon::argumentPositionMatch#4f8df883#fff/3@97d3444p after 149ms:
                      1000304 ~9%     {2} r1 = JOIN DataFlowDispatch::Cached::TParameterPosition#36b84300#f WITH DataFlowDispatch::Cached::TArgumentPosition#36b84300#f CARTESIAN PRODUCT OUTPUT Lhs.0 'ppos', Rhs.0

                      1616    ~0%     {2} r2 = JOIN r1 WITH DataFlowDispatch::Cached::TSelfParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'ppos'
                      1       ~0%     {2} r3 = JOIN r2 WITH DataFlowDispatch::Cached::TSelfArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'ppos'

                      1616    ~5%     {2} r4 = JOIN r1 WITH DataFlowDispatch::Cached::TBlockParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'ppos'
                      1       ~0%     {2} r5 = JOIN r4 WITH DataFlowDispatch::Cached::TBlockArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'ppos'

                      2       ~0%     {2} r6 = r3 UNION r5

                      1616    ~0%     {2} r7 = JOIN r1 WITH DataFlowDispatch::Cached::TAnyKeywordParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'ppos'
                      1533    ~0%     {2} r8 = JOIN r7 WITH DataFlowDispatch::Cached::TKeywordArgumentPosition#36b84300#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'ppos'

                      1000304 ~0%     {2} r9 = JOIN DataFlowDispatch::Cached::TParameterPosition#36b84300#f WITH DataFlowDispatch::Cached::TArgumentPosition#36b84300#f CARTESIAN PRODUCT OUTPUT Rhs.0, Lhs.0 'ppos'

                      619     ~0%     {2} r10 = JOIN r9 WITH DataFlowDispatch::Cached::TAnyKeywordArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1 'ppos', Lhs.0
                      592     ~0%     {2} r11 = JOIN r10 WITH DataFlowDispatch::Cached::TKeywordParameterPosition#36b84300#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'ppos'

                      2125    ~1%     {2} r12 = r8 UNION r11
                      2127    ~1%     {2} r13 = r6 UNION r12

                      1616    ~0%     {2} r14 = JOIN r1 WITH DataFlowDispatch::Cached::TAnyParameterPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.0 'ppos', Lhs.1
                      1615    ~0%     {2} r15 = r14 AND NOT DataFlowDispatch::Cached::TSelfArgumentPosition#36b84300#f(Lhs.1)
                      1615    ~2%     {2} r16 = SCAN r15 OUTPUT In.1, In.0 'ppos'

                      619     ~0%     {2} r17 = JOIN r9 WITH DataFlowDispatch::Cached::TAnyArgumentPosition#36b84300#f ON FIRST 1 OUTPUT Lhs.1 'ppos', Lhs.0
                      618     ~0%     {2} r18 = r17 AND NOT DataFlowDispatch::Cached::TSelfParameterPosition#36b84300#f(Lhs.0 'ppos')
                      618     ~0%     {2} r19 = SCAN r18 OUTPUT In.1, In.0 'ppos'

                      2233    ~0%     {2} r20 = r16 UNION r19

                      35552   ~0%     {3} r21 = JOIN r1 WITH DataFlowDispatch::Cached::TPositionalParameterPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'ppos'
                      22      ~0%     {2} r22 = JOIN r21 WITH DataFlowDispatch::Cached::TPositionalArgumentPosition#36b84300#ff ON FIRST 2 OUTPUT Lhs.1, Lhs.2 'ppos'

                      956672  ~0%     {3} r23 = JOIN r1 WITH DataFlowDispatch::Cached::TKeywordParameterPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'ppos'
                      592     ~0%     {2} r24 = JOIN r23 WITH DataFlowDispatch::Cached::TKeywordArgumentPosition#36b84300#ff ON FIRST 2 OUTPUT Lhs.1, Lhs.2 'ppos'

                      1616    ~0%     {3} r25 = JOIN r1 WITH DataFlowDispatch::Cached::TPositionalParameterLowerBoundPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'ppos', Rhs.1
                      79      ~0%     {4} r26 = JOIN r25 WITH DataFlowDispatch::Cached::TPositionalArgumentPosition#36b84300#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'ppos', Lhs.0, Lhs.2, Rhs.1
                      79      ~0%     {4} r27 = SELECT r26 ON In.3 >= In.2
                      79      ~1%     {2} r28 = SCAN r27 OUTPUT In.1, In.0 'ppos'

                      671     ~0%     {2} r29 = r24 UNION r28
                      693     ~0%     {2} r30 = r22 UNION r29
                      2926    ~0%     {2} r31 = r20 UNION r30
                      5053    ~0%     {2} r32 = r13 UNION r31
                      499970  ~6%     {3} r33 = JOIN r32 WITH DataFlowImplCommon::ArgNode::argumentOf#dispred#f0820431#fff_201#join_rhs ON FIRST 1 OUTPUT Rhs.2 'call', Rhs.1 'arg', Lhs.1 'ppos'
                                      return r33
2022-05-24 17:31:36 +02:00
Rasmus Wriedt Larsen
3afa9425ef Ruby: Add TAnyKeywordArgumentPosition and TAnyKeywordParameterPosition 2022-05-23 14:03:45 +02:00
Rasmus Wriedt Larsen
5d6fbcec64 Ruby: Autoformat 2022-05-19 16:30:12 +02:00
Rasmus Wriedt Larsen
e810ba4ef6 Ruby: Expand flowToAnyArg test 2022-05-19 16:27:04 +02:00
Rasmus Wriedt Larsen
0879b6ae12 Ruby: Fix Argument[any,any-named] handling for path component in MaD 2022-05-19 15:51:30 +02:00
Rasmus Wriedt Larsen
7784b9f879 Ruby: WIP: Make Argument[any] and any-named work
It's not fully working I think the problem is that the code below ties
up `Argument[x]` with parameter positions, and `Parameter[x]` with
argument positions. This flip might be correct for flow-summaries, but
it does NOT seem to be correct for the `path` component  in MaD.

Specifically, quick-eval for ParameterPosition does NOT include `keyword key` while
quick-eval for ArgumentPosition DOES include `keyword key`!

For the test `Foo.sinkAnyNamedArg(key: tainted) # $ MISSING: hasValueFlow=tainted`

c8be8d30b3/ruby/ql/lib/codeql/ruby/frameworks/data/internal/ApiGraphModelsSpecific.qll (L130-L133)
2022-05-19 15:51:25 +02:00
Rasmus Wriedt Larsen
df83a51e1e Ruby: Add anyNamedArg summary test 2022-05-19 15:42:41 +02:00
Rasmus Wriedt Larsen
cb6e5c24fc Ruby: Prepare for anyNamedArg summary test 2022-05-19 15:42:41 +02:00
Rasmus Wriedt Larsen
a7f627af0c Ruby: Add test for Argument[any] and any-named 2022-05-19 15:42:41 +02:00
Rasmus Wriedt Larsen
cb5ad8b775 Ruby: Don't include Argument[self] in Argument[any]
For flow-sumamries
2022-05-19 15:42:41 +02:00
Rasmus Wriedt Larsen
051754cf7e Ruby: Add test of what Argument[any] for input/output includes
and an explicit test of what `Argument[self]` includes.
2022-05-19 14:02:22 +02:00
Tom Hvitved
5e57e82997 Merge pull request #9191 from hvitved/ruby/taint-tracking-stage
Ruby: Force cached taint tracking predicates to be evaluated in data flow stage
2022-05-18 09:54:38 +02:00
Tom Hvitved
f1f96b7e5c Ruby: Force cached taint tracking predicates to be evaluated in data flow stage 2022-05-17 12:54:26 +02:00
Tom Hvitved
284357d2a0 Data flow: Do not materialize summaryArgParam 2022-05-17 12:50:01 +02:00
Mathias Vorreiter Pedersen
1280d43e36 Merge pull request #9141 from github/post-release-prep/codeql-cli-2.9.2
Post-release preparation for codeql-cli-2.9.2
2022-05-17 10:01:37 +01:00
Nick Rolfe
c518150b49 Merge pull request #9132 from github/nickrolfe/misspelling
QL for QL: generalise non-US spelling query
2022-05-16 16:03:36 +01:00
Tom Hvitved
a9f6d203cd Merge pull request #8971 from aibaars/safe-nagivation
Ruby: add safe navigation operator
2022-05-16 10:53:56 +02:00
Anders Schack-Mulligen
83f817ca45 Merge pull request #9134 from aschackmull/dataflow/perf-std-order
Dataflow: Improve standard order through easier type check elimination.
2022-05-16 10:05:17 +02:00
github-actions[bot]
b7cbd8fd75 Post-release preparation for codeql-cli-2.9.2 2022-05-12 18:21:38 +00:00
Nick Rolfe
1115227f9d Merge remote-tracking branch 'origin/main' into nickrolfe/misspelling 2022-05-12 16:10:27 +01:00
Nick Rolfe
8caad12011 Ruby: fix typos in comments 2022-05-12 16:02:20 +01:00
Anders Schack-Mulligen
8c8440a58a Merge pull request #9101 from hvitved/dataflow/include-hidden
Data flow: Add `Configuration::includeHiddenNodes()`
2022-05-12 15:36:12 +02:00
Joe Farebrother
59e400d2e0 Merge pull request #7723 from joefarebrother/redos
Java: Add ReDoS queries
2022-05-12 13:50:38 +01:00
Anders Schack-Mulligen
adb56dfa39 Dataflow: Improve standard order through easier type check elimination. 2022-05-12 14:31:38 +02:00
Mathias Vorreiter Pedersen
46f237efcb Update ruby/ql/lib/change-notes/released/0.2.1.md 2022-05-12 11:47:26 +01:00
Mathias Vorreiter Pedersen
7b8c3bdcf9 Update ruby/ql/lib/CHANGELOG.md 2022-05-12 11:47:13 +01:00
github-actions[bot]
ee9980b31c Release preparation for version 2.9.2 2022-05-12 10:17:28 +00:00
Tom Hvitved
0a7892797e Merge pull request #8938 from hvitved/ruby/with-without-mad-tokens
Ruby: Introduce `With(out)Element` MaD input tokens
2022-05-12 11:49:51 +02:00
Harry Maclean
e8972b814f Merge pull request #8635 from hmac/hmac/io-popen
Ruby: Model IO.popen
2022-05-12 21:17:55 +12:00
Tom Hvitved
46ab25b61e Merge pull request #9098 from aschackmull/dataflow/perf
Dataflow: Performance fixes
2022-05-11 20:41:48 +02:00
Anders Schack-Mulligen
4884520ee1 Dataflow: Review fix. 2022-05-11 15:40:49 +02:00
Tom Hvitved
884d3b2ff4 Ruby: Introduce With(out)Element MaD input tokens 2022-05-11 15:17:27 +02:00
Tom Hvitved
333780e635 Merge pull request #8898 from hvitved/dataflow/clear-expect-summary-components
Data flow: Introduce 'with/without content' summary components
2022-05-11 15:16:42 +02:00
Arthur Baars
e1e13b599a Fix CFG 2022-05-11 12:09:17 +02:00
Arthur Baars
d91b1891f1 Add change note 2022-05-11 12:06:08 +02:00
Arthur Baars
dbd9c1859d Add more test cases for &. operator 2022-05-11 12:06:08 +02:00
Arthur Baars
76f806159c Ruby: desugar safe navigation calls 2022-05-11 12:06:08 +02:00
Arthur Baars
c9f7568ca3 Ruby: add Call::isSafeNavigation 2022-05-11 12:06:08 +02:00
Arthur Baars
a47e429945 Merge pull request #8909 from aibaars/tree-sitter-update
Tree sitter update
2022-05-11 12:02:14 +02:00
Rasmus Wriedt Larsen
add6579385 Merge pull request #9022 from RasmusWL/ruby-fix
Ruby: Fix `isLocalSourceNode` implementation
2022-05-11 11:52:44 +02:00
Arthur Baars
907c3db5ca Address comments
Co-authored-by: Nick Rolfe <nickrolfe@github.com>
2022-05-11 09:59:42 +02:00
Tom Hvitved
712fe002b9 Data flow: Sync files 2022-05-10 12:41:10 +02:00
Tom Hvitved
bfabfc3601 Data flow: Add Configuration::includeHiddenNodes() 2022-05-10 12:40:46 +02:00
yoff
6c3e2db7fd Merge branch 'main' into python/simple-csrf 2022-05-10 10:55:28 +02:00
Anders Schack-Mulligen
f85e06c2e4 Dataflow: Sync. 2022-05-10 10:12:39 +02:00
Harry Maclean
a6cab022f6 Ruby: Add missing import 2022-05-10 17:32:01 +12:00
Harry Maclean
7b63493fa9 Ruby: Fix identification IO.open args 2022-05-10 17:32:00 +12:00
Harry Maclean
79c6dc1af0 Refactor IO/File modelling
The main goal here is to get rid of the duplicate definitions of module
`IO`, which currently exist in both `frameworks/core/IO.qll` and
`frameworks/Files.qll`.

We do this by moving the classes inside `Files::IO` to `core/IO.qll`,
but moving most of the actual definitions of those classes to an
internal module `core.internal.FileOrIO`. This means both `Files.qll`
and `IO.qll` can depend on them without leaking them to end users.
2022-05-10 17:32:00 +12:00
Harry Maclean
2d12ad6238 Ruby: Model IO.popen
This method is very similar to `Kernel.system`: it executes its
arguments as a system command in various ways.
2022-05-10 17:32:00 +12:00
Anders Schack-Mulligen
f24364d951 Merge pull request #9045 from hvitved/dataflow/subpaths-perf-take2
Data flow: Speedup `subpaths` predicate (take 2)
2022-05-09 15:39:11 +02:00