Commit Graph

41812 Commits

Author SHA1 Message Date
Aditya Sharad
a1d9228a66 Merge pull request #9831 from adityasharad/docs/supported-frameworks-changelog-links
Docs: Update supported languages page with links to CLI and pack information
2022-07-20 07:36:37 -07:00
Jeroen Ketema
694d6395d5 C++: Fix join-order problem in cpp/command-line-injection
Before on Abseil Linux:
```
Evaluated relational algebra for predicate ExecTainted::ExecState#class#91000ffb#fff@41084cm7 with tuple counts:
        40879811  ~0%    {2} r1 = SCAN DataFlowUtil::Node::getLocation#dispred#f0820431#ff OUTPUT In.1, In.0
        40879811  ~0%    {2} r2 = JOIN r1 WITH Location::Location::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1
            7527  ~3%    {3} r3 = JOIN r2 WITH ExecTainted::interestingConcatenation#91000ffb#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0
            7527  ~0%    {4} r4 = JOIN r3 WITH DataFlowUtil::Node::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.2, Lhs.1, Lhs.0, Rhs.1
            7527  ~0%    {5} r5 = JOIN r4 WITH DataFlowUtil::Node::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.2, Lhs.1, Lhs.0, Lhs.3, Rhs.1
            7527  ~0%    {6} r6 = JOIN r5 WITH DataFlowUtil::Node::getLocation#dispred#f0820431#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2, Lhs.0, Lhs.3, Lhs.4
            7527  ~0%    {3} r7 = JOIN r6 WITH Location::Location::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT ((((((("ExecState (" ++ Rhs.1) ++ " | ") ++ Lhs.4) ++ ", ") ++ Lhs.1) ++ " | ") ++ Lhs.5 ++ ")"), Lhs.3, Lhs.2
                         return r7
```

After:
```
Evaluated relational algebra for predicate ExecTainted::ExecState#class#91000ffb#fff@1ffe61ps with tuple counts:
        7527  ~0%    {3} r1 = JOIN ExecTainted::interestingConcatenation#91000ffb#ff WITH DataFlowUtil::Node::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Lhs.0, Rhs.1
        7527  ~0%    {4} r2 = JOIN r1 WITH DataFlowUtil::Node::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.0, Lhs.1, Lhs.2, Rhs.1
        7527  ~1%    {5} r3 = JOIN r2 WITH DataFlowUtil::Node::getLocation#dispred#f0820431#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0, Lhs.2, Lhs.3
        7527  ~0%    {5} r4 = JOIN r3 WITH Location::Location::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.3, Lhs.4, Rhs.1
        7527  ~4%    {6} r5 = JOIN r4 WITH DataFlowUtil::Node::getLocation#dispred#f0820431#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1, Lhs.2, Lhs.3, Lhs.4
        7527  ~0%    {3} r6 = JOIN r5 WITH Location::Location::toString#dispred#f0820431#ff ON FIRST 1 OUTPUT ((((((("ExecState (" ++ Rhs.1) ++ " | ") ++ Lhs.3) ++ ", ") ++ Lhs.5) ++ " | ") ++ Lhs.4 ++ ")"), Lhs.1, Lhs.2
                     return r6
```
2022-07-20 16:27:47 +02:00
thiggy1342
8c55a15fa6 Merge branch 'main' into experimental-manually-check-request-verb 2022-07-20 10:27:40 -04:00
thiggy1342
6f74a2609c Merge branch 'main' into experimental-strong-params 2022-07-20 10:26:49 -04:00
thiggy1342
f54fc1a88d Merge branch 'main' into add-activerecord-annotate 2022-07-20 10:26:44 -04:00
Jeroen Ketema
c2b7300709 Merge pull request #9848 from geoffw0/stringlengthconflation5
Swift: More improvements for the string length conflation query
2022-07-20 14:05:05 +02:00
Cornelius Riemenschneider
e9e5d948b3 C#: Implement proper dotnet build handling in the Lua tracing config.
For proper C# tracing, `dotnet build` needs the parameter
/p:UseSharedCompilation=false. However, we can't pass that to the other
subcommands of `dotnet`, therefore we need to figure out which subcommand
of `dotnet` is being invoked.
2022-07-20 10:11:36 +00:00
Cornelius Riemenschneider
ca819573f5 Merge pull request #9862 from github/adityasharad/codeql-cli-2.10.1-mergeback
Merge codeql-cli-2.10.1 into main
2022-07-20 10:42:34 +02:00
Paolo Tranquilli
3527897eff Swift: make type optional in TypeRepr
A type representation may not have a type in unresolved things, which
for example pop up in inactive `#if` clauses.
2022-07-20 09:13:34 +02:00
Aditya Sharad
7620a6f653 Docs: Update supported languages page with links to CLI and pack information
Include links to the CLI changelog, CLI releases, bundle releases,
pack changelogs, and pack source.

Clarify that this support information applies to the current version of
the CLI, bundle, query packs, and library packs.
2022-07-19 14:58:27 -07:00
Asger F
aa53841466 Merge pull request #9828 from github/post-release-prep/codeql-cli-2.10.1
Post-release preparation for codeql-cli-2.10.1
2022-07-19 19:49:50 +02:00
Henti Smith
018a76bb17 Merge pull request #9857 from github/henti/new_actions_predicates
Added Workflow.getName and Step.GetId
2022-07-19 16:12:54 +01:00
Henti Smith
dcc76ddf36 Apply suggestions from code review
Co-authored-by: Henry Mercer <henrymercer@github.com>
2022-07-19 15:53:12 +01:00
Henti Smith
0828474192 Added Workflow::getName and Step::GetId 2022-07-19 15:34:10 +01:00
thiggy1342
43a9b8960e Merge branch 'main' into experimental-manually-check-request-verb 2022-07-19 10:29:48 -04:00
thiggy1342
cf23d338f3 Merge branch 'main' into experimental-strong-params 2022-07-19 10:29:36 -04:00
thiggy1342
6bc2fe513d Merge branch 'main' into add-activerecord-annotate 2022-07-19 10:29:24 -04:00
Asger F
b9bdee6651 Merge branch 'main' into post-release-prep/codeql-cli-2.10.1 2022-07-19 16:24:35 +02:00
Cornelius Riemenschneider
03bf9eb166 Merge pull request #9837 from github/aeisenberg/definitions.ql
Move definitions.ql back to src
2022-07-19 14:43:10 +02:00
Taus
bfe90413e2 Merge pull request #9847 from alexet/alexet/fix-predicate-binding
Python: Fix binding incorrect predicate.
2022-07-19 13:59:13 +02:00
Arthur Baars
dcbd82907f Merge pull request #9845 from aibaars/skip-dotgit
Ruby: skip .git folder
2022-07-19 11:58:43 +02:00
Harry Maclean
ec1d1eb547 Ruby: Add change note 2022-07-19 14:33:51 +12:00
thiggy1342
962155fd61 fix changenotes 2022-07-19 00:33:04 +00:00
thiggy1342
9586259706 style tweak for checking multiple method names 2022-07-19 00:29:30 +00:00
thiggy1342
304203ad2f fix path problem output 2022-07-19 00:25:50 +00:00
Harry Maclean
7b8603c89b Ruby: Model Arel.sql 2022-07-19 11:27:15 +12:00
Taus
8c0725e8c6 Python: Fix bad join in ESSA getInput
Before:

```
Tuple counts for Essa::EssaEdgeRefinement::getInput#dispred#f0820431#ff/2@b84afc77 after 20.3s:
873421    ~0%     {3} r1 = JOIN Essa::TEssaEdgeDefinition#24e22a14#ffff_31#join_rhs WITH Essa::TEssaEdgeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'this'
181627951 ~0%     {3} r2 = JOIN r1 WITH Essa::EssaDefinition::getSourceVariable#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1, Lhs.2 'this'
873418    ~0%     {2} r3 = JOIN r2 WITH Essa::EssaDefinition::reachesEndOfBlock#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'
                return r3
```
It's perhaps not immediately obvious what's going on here (because of
the `...join_rhs` indirection), but basically we're joining together
`this` and `def` and their `getSourceVariable`, and only then actually
relating `this` and `def` through `reachesEndOfBlock`.

By unbinding `var`, we prevent this early join, which now encourages the
`reachesEndOfBlock` join to happen earlier:

```
Tuple counts for Essa::EssaEdgeRefinement::getInput#dispred#f0820431#ff/2@2d63e5lb after 2s
873421  ~0%     {2} r1 = SCAN Essa::TEssaEdgeDefinition#24e22a14#ffff OUTPUT In.3 'this', In.1
873421  ~0%     {3} r2 = JOIN r1 WITH Essa::TEssaEdgeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'this'
873421  ~0%     {3} r3 = JOIN r2 WITH Definitions::SsaSourceVariable#class#486534ab#f ON FIRST 1 OUTPUT Lhs.1, Lhs.2 'this', Lhs.0
8758877 ~0%     {3} r4 = JOIN r3 WITH Essa::EssaDefinition::reachesEndOfBlock#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.2, Lhs.1 'this'
873418  ~0%     {2} r5 = JOIN r4 WITH Essa::EssaDefinition::getSourceVariable#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'
                return r5
```
2022-07-18 20:21:39 +00:00
alexet
f9b6ca76e5 Python: Fix binding incorrect predicate. 2022-07-18 16:28:19 +01:00
thiggy1342
fc00e56058 Merge branch 'main' into experimental-manually-check-request-verb 2022-07-18 10:58:13 -04:00
Taus
bdd771989f Python: Fix bad join in syntactic_call_count
On certain databases, the evaluation of this predicate was running out
of memory due to the way the `count` aggregate was being used. Here's
an example of the tuple counts involved:

```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs/1@d2199bb8 after 1m27s:
595518502 ~521250%     {1} r1 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

26518709  ~111513%     {1} r2 = JOIN PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2 WITH Flow::CallNode::getFunction#dispred#f0820431#ff_1#join_rhs ON FIRST 1 OUTPUT Lhs.1 'arg0'

622037211 ~498045%     {1} r3 = r1 UNION r2
                        return r3
```

and a timing report that looked like this:
```
time  | evals |   max @ iter | predicate
------|-------|--------------|----------
 5m8s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#2@6d98d1nd
4m38s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#count_range@f5df1do4
3m51s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#shared#3@da3b4abf
1m58s |  7613 |  37ms @ 4609 | MRO::ClassListList::removedClassParts#f0820431#fffff#reorder_2_3_4_0_1@8155axyi
1m37s |  7613 |  33ms @ 3904 | MRO::ClassListList::bestMergeCandidate#f0820431#2#fff@8155a83w
1m27s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#antijoin_rhs@d2199bb8
 1m8s |  1825 |  63ms @ 404  | PointsTo::Expressions::equalityEvaluatesTo#741b54e2#fffff@8155aw7w
37.6s |       |              | PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs@e348fc1p
...
```

To make optimising this easier for the compiler, I moved the bodies of
the `count` aggregate into their own helper predicates (with size
linear in the number of `CallNode`s), and also factored out the many
calls to `f.getName()`.

The astute reader will notice that in writing this as a sum of `count`s
rather than a count of a disjunction, the intersection (if it exists)
will be counted twice, and so the semantics may be different. However,
since `method_call` and `function_call` require `AttrNode` and
`NameNode` functions respectively, and as these two types are disjoint,
there is no intersection, and so the semantics should be preserved.

After the change, the evaluation of `syntactic_call_count` now looks as
follows:
```
Tuple counts for PointsToContext::syntactic_call_count#cf3039a0#ff/2@662dd8s0 after 216ms:
23960  ~0%     {1} r1 = @py_scope#f AND NOT py_Functions_0#antijoin_rhs(Lhs.0 's')
23960  ~0%     {2} r2 = SCAN r1 OUTPUT In.0 's', 0

276309 ~7%     {2} r3 = SCAN @py_scope#f OUTPUT In.0 's', "__init__"
11763  ~0%     {2} r4 = JOIN r3 WITH Scope::Scope::getName#dispred#f0820431#fb ON FIRST 2 OUTPUT Lhs.0 's', 1

35723  ~0%     {2} r5 = r2 UNION r4

252349 ~0%     {2} r6 = JOIN @py_scope#f WITH Function::Function::getName#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.0 's', Rhs.1

240586 ~0%     {2} r7 = SELECT r6 ON In.1 != "__init__"

131727 ~4%     {2} r8 = r7 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r9 = SCAN r8 OUTPUT In.1, In.0 's', 0

240586 ~0%     {2} r10 = SCAN r7 OUTPUT In.1, In.0 's'

108859 ~0%     {3} r11 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1 's', Rhs.1

240586 ~0%     {3} r12 = r9 UNION r11
24100  ~0%     {2} r13 = JOIN r12 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs#1 ON FIRST 1 OUTPUT Lhs.1 's', (Rhs.1 + Lhs.2)

240586 ~0%     {2} r14 = SELECT r6 ON In.1 != "__init__"
131727 ~4%     {2} r15 = r14 AND NOT project#PointsToContext::method_call#cf3039a0#ff(Lhs.1)
131727 ~0%     {3} r16 = SCAN r15 OUTPUT In.0 's', In.1, 0

108859 ~4%     {3} r17 = JOIN r10 WITH PointsToContext::syntactic_call_count#cf3039a0#ff#join_rhs ON FIRST 1 OUTPUT Lhs.1 's', Lhs.0, Rhs.1

240586 ~4%     {3} r18 = r16 UNION r17
216486 ~2%     {3} r19 = r18 AND NOT project#PointsToContext::function_call#cf3039a0#ff(Lhs.1)
216486 ~0%     {2} r20 = SCAN r19 OUTPUT In.0 's', (0 + In.2)

240586 ~0%     {2} r21 = r13 UNION r20
276309 ~0%     {2} r22 = r5 UNION r21
                return r22
```
2022-07-18 13:58:00 +00:00
Arthur Baars
c9e5206396 Ruby: skip .git folder 2022-07-18 15:26:38 +02:00
Geoffrey White
541df9b550 Swift: Remove TODO comment. We have a test for this problem now. 2022-07-18 14:26:12 +01:00
Geoffrey White
336548f746 Swift: Improve comments. 2022-07-18 14:24:16 +01:00
Geoffrey White
9474e63faf Swift: Clean up isSink (4 - move common code out). 2022-07-18 14:24:15 +01:00
Geoffrey White
b136790efd Swift: Clean up isSink (3 - rename f -> funcDecl and move that out as well; in the other two cases this variable didn't exist, now it does). 2022-07-18 14:24:14 +01:00
Geoffrey White
0bd94a6307 Swift: Clean up isSink (2 - rename methodName -> funcName and move that out as well). 2022-07-18 14:24:13 +01:00
Geoffrey White
4854679a40 Swift: Clean up isSink (1 - move common variables to an outer exists). 2022-07-18 14:24:13 +01:00
Geoffrey White
39fb714ad1 Swift: Add test with substring declared differently. 2022-07-18 14:24:12 +01:00
Michael Nebel
6603024488 C#: Allow encryption- prefix for sinks in CsvValidation. 2022-07-18 14:32:31 +02:00
Michael Nebel
57ba0c4e5d C#: Move sinks into System.Security.Cryptography framework code. 2022-07-18 14:28:49 +02:00
Michael Nebel
c91d49a0fe C#: Add provenance column to CSV format for SymmetricAlgorithm. 2022-07-18 14:28:49 +02:00
Michael Nebel
66232a8054 C#: Fix typo. 2022-07-18 14:28:49 +02:00
Michael Nebel
a5b7e2a2e1 C#: Convert set Key of SymmetricAlgorithm to Csv sink. 2022-07-18 14:28:49 +02:00
Michael Nebel
032448041d C#: Convert CreateSymmetricKey to CSV sink. 2022-07-18 14:28:49 +02:00
Michael Nebel
1d405dba14 C#: Collapse Sink classes. 2022-07-18 14:28:49 +02:00
Michael Nebel
383ad51682 C#: Use CSV format for CreateEncryptor and CreateDecryptor sinks. 2022-07-18 14:28:49 +02:00
Michael Nebel
e6e82ef56d C#: Update test with Decrypt example. 2022-07-18 14:28:49 +02:00
Michael Nebel
52a9fb0de7 C#: Add test for decrypt. 2022-07-18 14:28:49 +02:00
Paolo Tranquilli
e1bd4a78ff Merge branch 'main' into redsun82/swift-type-repr-collapse 2022-07-18 14:05:43 +02:00
Paolo Tranquilli
10b7b1f183 Merge branch 'main' into redsun82/swift-file-label-caching 2022-07-18 14:02:57 +02:00