Commit Graph

35629 Commits

Author SHA1 Message Date
Jeroen Ketema
4a648f3c89 Fix change note items 2022-04-28 14:14:19 +02:00
github-actions[bot]
8e4cf190e9 Release preparation for version 2.9.1 2022-04-28 11:59:05 +00:00
Michael Nebel
ec316750d3 Merge pull request #8905 from michaelnebel/csharp/generatedcomment
C#: Add auto generated comment to generated models as data files.
2022-04-28 13:57:38 +02:00
Tom Hvitved
f7669815ce Address review comments 2022-04-28 13:50:26 +02:00
Alex Denisov
85918173a6 Swift: ensure the folder for trap files exists 2022-04-28 13:19:25 +02:00
Alex Denisov
9c73ae5a97 Swift: teach extractor to not produce artifacts 2022-04-28 13:18:20 +02:00
Michael Nebel
9d767b8ad8 Merge pull request #8869 from michaelnebel/csharp/frameworkcoverage
C#: Port the java FrameworkCoverage query.
2022-04-28 13:17:50 +02:00
Alex Denisov
5b75b4db79 Swift: add tracer config 2022-04-28 13:17:35 +02:00
Alex Denisov
4a03976a15 Swift: set compiler flags explicitly 2022-04-28 13:17:05 +02:00
Taus
95d235416c Python: Fix bad antijoin in getAKeyword
Before:

```
Tuple counts for Exprs::Call::getAKeyword_dispred#ff#antijoin_rhs/3@7bc202ij after 9s:
1        ~0%     {1} r1 = CONSTANT(unique int)[2]
4244385  ~2%     {1} r2 = JOIN r1 WITH py_dict_items_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0'
4244352  ~3%     {3} r3 = JOIN r2 WITH AstGenerated::Call_::getNamedArg_dispred#ffb_201#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg1', Lhs.0 'arg0', Rhs.2 'arg2'
66618690 ~3%     {5} r4 = JOIN r3 WITH AstGenerated::Call_::getNamedArg_dispred#ffb ON FIRST 1 OUTPUT Lhs.1 'arg0', Lhs.0 'arg1', Lhs.2 'arg2', Rhs.1, Rhs.2
31187133 ~0%     {5} r5 = SELECT r4 ON In.3 < In.2 'arg2'
31187133 ~1%     {5} r6 = SCAN r5 OUTPUT In.4, 0, In.0 'arg0', In.1 'arg1', In.2 'arg2'
0        ~0%     {3} r7 = JOIN r6 WITH py_dict_items ON FIRST 2 OUTPUT Lhs.2 'arg0', Lhs.3 'arg1', Lhs.4 'arg2'
                 return r7

Tuple counts for Exprs::Call::getAKeyword_dispred#ff/2@1dc9468b after 421ms:
1       ~0%     {1} r1 = CONSTANT(unique int)[2]
4244385 ~2%     {1} r2 = JOIN r1 WITH py_dict_items_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result'
4244352 ~0%     {3} r3 = JOIN r2 WITH AstGenerated::Call_::getNamedArg_dispred#ffb_201#join_rhs ON FIRST 1 OUTPUT Lhs.0 'result', Rhs.1 'this', Rhs.2
4244352 ~0%     {3} r4 = r3 AND NOT Exprs::Call::getAKeyword_dispred#ff#antijoin_rhs(Lhs.0 'result', Lhs.1 'this', Lhs.2)
4244352 ~6%     {2} r5 = SCAN r4 OUTPUT In.1 'this', In.0 'result'
                return r5
```

Oof. All that work to produce zero tuples. Luckily we can improve
matters somewhat.

Basically, there's no reason to test _all_ dictionary unpackings, since
we're only interested in a lower bound. Thus, we can use `min` instead
which is much more efficient. For convenience I factored this into its
own (private) helper predicate.

Now the tuple counts look as follows:

```
Tuple counts for Exprs::Call::getMinimumUnpackingIndex_dispred#ff#min_range/2@39b0e9sm after 1ms:
246 ~0%     {2} r1 = JOIN Keywords::DictUnpackingOrKeyword#class#f#shared WITH AstGenerated::Call_::getNamedArg_dispred#ffb_201#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Rhs.2 'arg1'
            return r1
Registering Exprs::Call::getMinimumUnpackingIndex_dispred#ff#min_range/2@39b0e9sm +  with content 9ea2f123k8necpu015v6tpsc2t1
 >>> Created relation Exprs::Call::getMinimumUnpackingIndex_dispred#ff#min_range/2@39b0e9sm with 246 rows.
Starting to evaluate predicate Exprs::Call::getMinimumUnpackingIndex_dispred#ff#min_term/3@9f4ca5g8
Tuple counts for Exprs::Call::getMinimumUnpackingIndex_dispred#ff#min_term/3@9f4ca5g8 after 0ms:
246 ~2%     {3} r1 = JOIN Keywords::DictUnpackingOrKeyword#class#f#shared WITH AstGenerated::Call_::getNamedArg_dispred#ffb_201#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Rhs.2 'arg2', Rhs.2 'arg2'
            return r1

Tuple counts for Exprs::Call::getAKeyword_dispred#ff/2@000a0alb after 906ms:
1       ~0%     {1} r1 = CONSTANT(unique int)[2]
4244385 ~2%     {1} r2 = JOIN r1 WITH py_dict_items_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result'

4244352 ~0%     {3} r3 = JOIN r2 WITH AstGenerated::Call_::getNamedArg_dispred#ffb_201#join_rhs ON FIRST 1 OUTPUT Lhs.0 'result', Rhs.1 'this', Rhs.2
4244280 ~0%     {3} r4 = r3 AND NOT Exprs::Call::getMinimumUnpackingIndex_dispred#ff_0#antijoin_rhs(Lhs.1 'this')
4244280 ~6%     {2} r5 = SCAN r4 OUTPUT In.1 'this', In.0 'result'

4244352 ~3%     {3} r6 = JOIN r2 WITH AstGenerated::Call_::getNamedArg_dispred#ffb_201#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.0 'result', Rhs.2
72      ~4%     {4} r7 = JOIN r6 WITH Exprs::Call::getMinimumUnpackingIndex_dispred#ff ON FIRST 1 OUTPUT Lhs.1 'result', Lhs.0 'this', Lhs.2, Rhs.1
72      ~4%     {4} r8 = SELECT r7 ON In.2 <= In.3
72      ~0%     {2} r9 = SCAN r8 OUTPUT In.1 'this', In.0 'result'

4244352 ~6%     {2} r10 = r5 UNION r9
                return r10
```

This is not the perfect join order (note the similarity between `r3`
and `r6`) but overall it's a win.
2022-04-28 11:11:37 +00:00
Taus
80ef09f034 Python: Fix bad join in declaredAttributeVar
Before:
```
Tuple counts for PointsTo::declaredAttributeVar#fbf/3@99d5aenq after 1.1s:
451054   ~7%     {2} r1 = SCAN variable OUTPUT In.0, In.2 'name'
1296149  ~0%     {2} r2 = JOIN r1 WITH Essa::EssaVariable::getSourceVariable_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'var', Lhs.1 'name'
12179900 ~4%     {3} r3 = JOIN r2 WITH Essa::EssaVariable::getAUse_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'name', Lhs.0 'var'
8028     ~2%     {3} r4 = JOIN r3 WITH Scope::Scope::getANormalExit_dispred#bf_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'name', Lhs.2 'var'
8028     ~2%     {3} r5 = JOIN r4 WITH Classes::PythonClassObjectInternal::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'cls', Lhs.1 'name', Lhs.2 'var'
                 return r5
```

After:
```
Tuple counts for PointsTo::declaredAttributeVar#fbf/3@cccf36hb after 4ms:
1450 ~0%     {2} r1 = SCAN Classes::PythonClassObjectInternal::getScope_dispred#ff OUTPUT In.1, In.0 'cls'
1450 ~7%     {2} r2 = JOIN r1 WITH Scope::Scope::getANormalExit_dispred#bf ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'cls'
8028 ~0%     {2} r3 = JOIN r2 WITH Essa::EssaVariable::getAUse_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'var', Lhs.1 'cls'
8028 ~0%     {3} r4 = JOIN r3 WITH Essa::EssaVariable::getSourceVariable_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'cls', Lhs.0 'var'
8028 ~2%     {3} r5 = JOIN r4 WITH variable ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.2 'name', Lhs.2 'var'
return r5
```
2022-04-28 11:11:37 +00:00
Taus
d28f9f41e8 Python: Fix bad join in import_star_read
Makes this

```
(21s) Tuple counts for DataFlowPublic::import_star_read#ff/2@fcd5e6nr after 8.5s:
9743      ~6%     {3} r1 = SCAN num#DataFlowPublic::TModuleVariableNode#fff OUTPUT In.1, In.0, In.2 'result'
9743      ~1%     {3} r2 = JOIN r1 WITH Variables::Variable::getId_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2 'result'
390808917 ~3%     {3} r3 = JOIN r2 WITH Flow::NameNode::getId_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2 'result'
307       ~0%     {2} r4 = JOIN r3 WITH ImportStar::ImportStar::importStarResolvesTo#ff ON FIRST 2 OUTPUT Lhs.0, Lhs.2 'result'
307       ~0%     {2} r5 = JOIN r4 WITH num#DataFlowPublic::TCfgNode#ff ON FIRST 1 OUTPUT Rhs.1 'n', Lhs.1 'result'
                  return r5
```

become this

```
(17s) Tuple counts for DataFlowPublic::resolved_import_star_module#fff/3@f5e84aic after 0ms:
307 ~0%     {3} r1 = JOIN ImportStar::ImportStar::importStarResolvesTo#ff WITH num#DataFlowPublic::TCfgNode#ff ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'm', Rhs.1 'n'
307 ~0%     {3} r2 = JOIN r1 WITH Flow::NameNode::getId_dispred#ff ON FIRST 1 OUTPUT Lhs.1 'm', Rhs.1 'name', Lhs.2 'n'
            return r2
(17s) Registering DataFlowPublic::resolved_import_star_module#fff/3@f5e84aic +  with content f29281ig38r98icro4ege09mrva
(17s)  >>> Created relation DataFlowPublic::resolved_import_star_module#fff/3@f5e84aic with 307 rows.
(17s) Starting to evaluate predicate DataFlowPublic::import_star_read#ff/2@57b0c06e
(17s) Tuple counts for DataFlowPublic::import_star_read#ff/2@57b0c06e after 2ms:
9743 ~0%     {3} r1 = SCAN num#DataFlowPublic::TModuleVariableNode#fff OUTPUT In.1, In.0, In.2 'result'
9743 ~0%     {3} r2 = JOIN r1 WITH Variables::Variable::getId_dispred#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Lhs.2 'result'
307  ~0%     {2} r3 = JOIN r2 WITH DataFlowPublic::resolved_import_star_module#fff ON FIRST 2 OUTPUT Rhs.2 'n', Lhs.2 'result'
             return r3
```
2022-04-28 11:11:37 +00:00
Erik Krogh Kristensen
3c07ab59a1 Merge pull request #8936 from jketema/camel-case
QL: Improve camel case query
2022-04-28 12:32:46 +02:00
yoff
4553a0913f Merge pull request #8897 from tausbn/python-fix-bad-methodcallsite-join
Python: Fix bad join in `MethodCallsiteRefinement`
2022-04-28 12:17:33 +02:00
Jeroen Ketema
b6cf536f01 QL: Fix formatting 2022-04-28 12:05:47 +02:00
Paolo Tranquilli
773ef62406 Swift: added trapgen
This checks in the trapgen script generating trap entries in C++.

The codegen suite has been slightly reorganized, moving the templates
directory up one level and chopping everything into smaller bazel
packages. Running tests is now done via
```
bazel run //swift/codegen/test
```

With respect to the PoC, the nested `codeql::trap` namespace has been
dropped in favour of a `Trap` prefix (or suffix in case of entries)
within the `codeql` namespace. Also, generated C++ code is not checked
in in git any more, and generated during build. Finally, labels get
printed in hex in the trap file.

`TrapLabel` is for the moment only default-constructible, so only one
single label is possible. `TrapArena`, that is responsible for creating
disjoint labels will come in a later commit.
2022-04-28 12:01:59 +02:00
Tony Torralba
604a5fc71f Merge pull request #8639 from atorralba/atorralba/spring-beans-improvements
Java: Improve Spring models
2022-04-28 11:59:51 +02:00
Michael Nebel
150d9ba52c Update .github/workflows/csv-coverage-metrics.yml
Co-authored-by: Henry Mercer <henry.mercer@me.com>
2022-04-28 11:57:53 +02:00
Henry Mercer
52a417b02d Merge pull request #8921 from github/dependabot/github_actions/actions/setup-python-3
Bump actions/setup-python from 2 to 3
2022-04-28 10:57:02 +01:00
Jeroen Ketema
62831e93fe QL: Add filter for NewType to camel case query 2022-04-28 11:54:42 +02:00
Henry Mercer
03c311181a Merge pull request #8922 from github/dependabot/github_actions/actions/download-artifact-3
Bump actions/download-artifact from 2 to 3
2022-04-28 10:45:49 +01:00
Jeroen Ketema
3db9d56259 QL: Improve message for camel cazse query 2022-04-28 11:41:17 +02:00
Jeroen Ketema
52fc2dac47 QL: Add camel case tests 2022-04-28 11:38:14 +02:00
Mathias Vorreiter Pedersen
2517371a37 Merge pull request #8933 from MathiasVP/revert-globals
C++: Revert #8515
2022-04-28 10:38:08 +01:00
Tom Hvitved
29f1c533a9 Ruby: Add type tracker tests for flow through keyword/positional parameters 2022-04-28 11:34:12 +02:00
Michael Nebel
c606121ae7 C#: Move autogenerated comment to file level instead of having it on each class. 2022-04-28 11:27:49 +02:00
Michael Nebel
57fc4d987f C#: Fix indentation. 2022-04-28 11:19:54 +02:00
Michael Nebel
583b9b61de C#: Add job for running the framework coverage query for C#. 2022-04-28 11:19:49 +02:00
Michael Nebel
98b2bc06ce C#: Port the java FrameworkCoverage query. 2022-04-28 11:18:12 +02:00
Tom Hvitved
db856798b9 Merge pull request #8920 from github/dependabot/github_actions/actions/setup-dotnet-2
Bump actions/setup-dotnet from 1 to 2
2022-04-28 10:47:28 +02:00
Stephan Brandauer
f4104e2b72 Merge pull request #8886 from kaeluka/add-rest-parameter-flowstep
JS: Add flow step to `...rest` parameters
2022-04-28 08:39:50 +01:00
Tony Torralba
1f1581cc97 Merge pull request #8913 from github/workflow/coverage/update
Update CSV framework coverage reports
2022-04-28 09:34:52 +02:00
Jonas Jensen
0c65e67d18 QL language reference: variables must be lowerId
To prepare for a future QL language change where variable names must
start with a lower-case letter, this commit updates the QL language
reference (including the language specification) to change the variable
name grammar from `simpleId` to `lowerId`.
2022-04-28 09:13:36 +02:00
dependabot[bot]
8c4e92d065 Bump actions/download-artifact from 2 to 3
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-28 03:32:59 +00:00
dependabot[bot]
c8fd94a830 Bump actions/setup-python from 2 to 3
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-28 03:32:57 +00:00
dependabot[bot]
6526ee797d Bump actions/setup-dotnet from 1 to 2
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 1 to 2.
- [Release notes](https://github.com/actions/setup-dotnet/releases)
- [Commits](https://github.com/actions/setup-dotnet/compare/v1...v2)

---
updated-dependencies:
- dependency-name: actions/setup-dotnet
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-28 03:32:55 +00:00
github-actions[bot]
018558b823 Add changed framework coverage reports 2022-04-28 00:18:25 +00:00
Harry Maclean
ba1d43dd42 Merge pull request #8658 from hmac/hmac/insecure-download
Ruby: Add InsecureDownload query
2022-04-28 11:07:35 +12:00
Harry Maclean
f4453f4da2 Merge pull request #8573 from hmac/hmac/missing-regexp-anchor
Ruby: Add MissingRegExpAnchor query
2022-04-28 11:06:33 +12:00
luchua-bc
590b9d8519 Standardize the query and update qldoc 2022-04-27 22:17:17 +00:00
Chris Smowton
bb049bffbd Merge pull request #8765 from artem-smotrakov/cover-jms
Java: Add flow sources and steps for RabbitMQ and JMS
2022-04-27 21:27:05 +01:00
Taus
b4a31e572f Python: Add global attribute writes 2022-04-27 16:45:00 +00:00
Taus
f71cf2e1fc Python: Add test 2022-04-27 15:48:11 +00:00
Paolo Tranquilli
f95b5853c1 Merge pull request #8788 from AlexDenisov/alexdenisov/swift-first-extractor-test
Swift: file extraction
2022-04-27 17:47:17 +02:00
Mathias Vorreiter Pedersen
dc96d55943 Merge pull request #8888 from geoffw0/xxe2
C++: Add support for createLSParser to the CWE-611 XXE query.
2022-04-27 16:24:27 +01:00
Anna Railton
00b74d8b1c Merge pull request #8895 from github/annarailton-patch-1
ATM: Update `TaintedPathInjection` -> `TaintedPath`
2022-04-27 16:15:46 +01:00
Mathias Vorreiter Pedersen
abbb7f861f Merge pull request #8904 from MathiasVP/sync-swift-schema
Swift: Sync schema after extractor changes
2022-04-27 16:14:46 +01:00
Alex Denisov
272aa594cc Swift: compiler options moved to .bazelrc 2022-04-27 17:11:16 +02:00
Mathias Vorreiter Pedersen
75c1e56bbd Revert "Merge pull request #8515 from rdmarsh2/rdmarsh2/ir-global-vars"
This reverts commit 800e4ea7df, reversing
changes made to 7ce040f331.
2022-04-27 16:04:28 +01:00
Henry Mercer
897bc2374a Merge pull request #8906 from github/henrymercer/workflow-codeowners
Add CODEOWNERS for Actions workflows
2022-04-27 15:47:11 +01:00