Commit Graph

32190 Commits

Author SHA1 Message Date
Taus
67be20f368 Python: Remove implied inequalities
Also gets rid of `inner_end`, since we're already doing `end - 1 = ...`
in the other fix (and so this is more consistent).
2022-02-04 12:46:06 +00:00
Taus
22aa4c9379 Python: Fix performance issue in charSet
Observed on `mozilla/bugbug` on the 2.8.0 CLI branch, we had the
following line in the timing report:
```
FullServerSideRequestForgery.ql-17:regex::RegexString::charSet_dispred#fff#antijoin_rhs ............... 1m13s
```

Inspecting the logs, we see the following join:

```
(644s) Tuple counts for regex::RegexString::charSet_dispred#fff#antijoin_rhs/5@f295d1bk after 1m13s:
1         ~0%         {1} r1 = CONSTANT(unique string)["]"]
2389      ~4%         {3} r2 = JOIN r1 WITH regex::RegexString::nonEscapedCharAt_dispred#fff_201#join_rhs ON FIRST 1 OUTPUT Rhs.1 'arg0', Rhs.2 'arg1', (Rhs.2 'arg1' + 1)
668873    ~0%         {6} r3 = JOIN r2 WITH regex::RegexString::char_set_start_dispred#fff ON FIRST 1 OUTPUT Lhs.0 'arg0', "]", Lhs.1 'arg1', Lhs.2 'arg2', Rhs.1 'arg3', Rhs.2 'arg4'
537501371 ~4%         {7} r4 = JOIN r3 WITH regex::RegexString::nonEscapedCharAt_dispred#fff_021#join_rhs ON FIRST 2 OUTPUT Lhs.0 'arg0', Lhs.2 'arg1', Lhs.3 'arg2', Lhs.4 'arg3', Lhs.5 'arg4', "]", Rhs.2
269085087 ~0%         {7} r5 = SELECT r4 ON In.6 > In.4 'arg4'
89583155  ~3%         {7} r6 = SELECT r5 ON In.6 < In.1 'arg1'
89583155  ~26634%     {5} r7 = SCAN r6 OUTPUT In.0 'arg0', In.1 'arg1', In.2 'arg2', In.3 'arg3', In.4 'arg4'
                    return r7
```
Now, this is problematic not just because of the large intermediary join
but also because of the large number of tuples being materialised at the
end. The culprit in this case turns out to be this bit of `charSet`:
```
not exists(int mid | this.nonEscapedCharAt(mid) = "]" | mid > inner_start and mid < inner_end)
```

Rewriting this to instead look for the minimum index at which a `]`
appears resulted in a much nicer join.

I also fixed up a similar issue surrounding the `\N` unicode escape.
Not that I think this will necessarily be relevant, but the `min`-based
solution is more robust either way.
2022-02-03 20:42:04 +00:00
Erik Krogh Kristensen
e93c46ad31 Merge pull request #7811 from erik-krogh/pyApiIpa
Python: refactor API-graph labels to an IPA type
2022-02-03 12:31:39 +01:00
Erik Krogh Kristensen
5284bbb6b3 Merge pull request #7821 from erik-krogh/upload-sarif
QL: upload sarif as part of the QL-for-QL workflow
2022-02-03 12:05:51 +01:00
Henry Mercer
224d7a7ce0 Merge pull request #7801 from github/henrymercer/js-atm-migrate-tests
JS: Migrate CodeQL tests for ML-powered queries
2022-02-03 10:17:19 +00:00
Tony Torralba
3c9b332ce0 Merge pull request #7826 from github/workflow/coverage/update
Update CSV framework coverage reports
2022-02-03 09:38:53 +01:00
Tom Hvitved
6bb71f051b Merge pull request #7791 from hvitved/dataflow/inline-local-flow-star
Data flow: Inline `local(Expr|Instruction)?(Flow|Taint)`
2022-02-03 09:02:43 +01:00
github-actions[bot]
2a9f98cce0 Add changed framework coverage reports 2022-02-03 00:10:33 +00:00
Erik Krogh Kristensen
7ef051456a upload sarif as part of the QL-for-QL workflow 2022-02-02 20:32:22 +01:00
Henry Mercer
2c17437092 JS: Run ML-powered queries tests on all PRs modifying relevant files 2022-02-02 18:11:25 +00:00
Henry Mercer
a586be956e JS: Remove versions from packs we don't intend to publish 2022-02-02 18:10:57 +00:00
Arthur Baars
6acf49d4da Merge pull request #7814 from aibaars/fix-ql-alerts
Ruby: fix all QL-QL alerts
2022-02-02 18:25:38 +01:00
Tony Torralba
4f13bf8941 Merge pull request #6492 from atorralba/atorralba/android-cleartext-storage-database
Java: Create new query Cleartext storage of sensitive information in Android databases
2022-02-02 16:23:05 +01:00
Tony Torralba
54e8ea56e8 Apply suggestions from code review
Co-authored-by: Anders Schack-Mulligen <aschackmull@users.noreply.github.com>
2022-02-02 15:44:26 +01:00
Mathias Vorreiter Pedersen
1aa32b09be Merge pull request #7802 from geoffw0/clrtxt8
C++: Recognize password struct fields.
2022-02-02 14:10:40 +00:00
Henry Mercer
7018f6ad40 JS: Add missing @id for endpoint types query 2022-02-02 13:15:15 +00:00
Henry Mercer
e6745dc63a JS: Add Action to run tests for ML-powered queries 2022-02-02 13:15:12 +00:00
Henry Mercer
422919b9d0 JS: Add generated files to .gitattributes 2022-02-02 13:15:04 +00:00
Henry Mercer
fbcb8d6857 JS: Migrate CodeQL tests for ML-powered queries 2022-02-02 13:15:04 +00:00
Arthur Baars
3b05cb621c Address comment 2022-02-02 14:11:45 +01:00
Arthur Baars
33b97f3e0c Update synchronized files 2022-02-02 13:30:45 +01:00
Arthur Baars
fdcef6225b Ruby: fix QL warnings 2022-02-02 13:29:09 +01:00
CodeQL CI
7bb11b837c Merge pull request #7788 from yoff/python/remove-library-annotation
Approved by tausbn
2022-02-02 03:51:00 -08:00
Tom Hvitved
712418e5f8 Merge pull request #7781 from hvitved/dataflow/summary-stack-bottom-less-nonlinear
Data flow: Reduce non-linear recursion in `SummaryComponentStack::bottom`
2022-02-02 10:35:53 +01:00
Benjamin Muskalla
d4c4e75bac Merge pull request #7268 from bmuskalla/modelDiffAction
Java: Produce diffs for model generator changes
2022-02-02 10:30:45 +01:00
Henry Mercer
e622e517d9 Merge pull request #7800 from github/henrymercer/js-atm-add-model-building-pack
JS: Add model building pack for ML-powered queries
2022-02-01 20:51:19 +00:00
Geoffrey White
d864af3622 C++: Change note. 2022-02-01 19:00:40 +00:00
Harry Maclean
fb00a6c61b Merge pull request #7666 from github/hmac/file-open-access
Ruby: Add File.open as a FileSystemAccess
2022-02-02 07:32:16 +13:00
Henry Mercer
14601316a5 JS: Autoformat 2022-02-01 17:08:21 +00:00
Erik Krogh Kristensen
e06f6529f1 refactor API-graph labels to an IPA type 2022-02-01 17:32:08 +01:00
Henry Mercer
368839edfc JS: Fix QLDoc style in ExtractMisclassifiedEndpointFeatures.ql 2022-02-01 15:39:15 +00:00
Arthur Baars
ea901adb3c Merge pull request #7799 from github/aibaars/fix-ruby-workflows
Ruby: use ruby specific cache key
2022-02-01 16:28:14 +01:00
Henry Mercer
db0b4fc463 JS: Add model building pack for ML-powered queries
Tests are currently still internal. They will be migrated to
`github/codeql` in a subsequent PR.
2022-02-01 15:03:26 +00:00
Arthur Baars
73d60550ce QL-QL: fix cache keys 2022-02-01 15:57:59 +01:00
Erik Krogh Kristensen
0f85a52f09 Merge pull request #7773 from erik-krogh/CWE-367
JS: add a js/file-system-race query
2022-02-01 15:36:13 +01:00
Arthur Baars
6451a71a78 Ruby: use ruby specific cache key 2022-02-01 15:18:09 +01:00
Mathias Vorreiter Pedersen
3597d80340 Merge pull request #7787 from Yonah125/main
C/C++ : Useless test
2022-02-01 14:01:27 +00:00
Nick Rolfe
5828a61fec Merge pull request #7795 from github/nickrolfe/graph_test_edge_ordering
Ruby/C#: add semmle.order attribute to edges in CFG tests
2022-02-01 13:36:15 +00:00
Erik Krogh Kristensen
a51f892a99 move dot in qhelp
Co-authored-by: Esben Sparre Andreasen <esbena@github.com>
2022-02-01 14:34:30 +01:00
Erik Krogh Kristensen
e6c90670e6 Merge pull request #7740 from erik-krogh/CWE-347
JS: promote the js/jwt-missing-verification query out of experimental
2022-02-01 13:10:35 +01:00
Rasmus Wriedt Larsen
f7a0b17ed6 Merge pull request #7687 from yoff/python/PathInjection-FlowState
python: Rewrite path injection query to use flow state
2022-02-01 11:33:37 +01:00
Mathias Vorreiter Pedersen
a59a9ba82b C++: Autoformat. 2022-02-01 08:28:53 +00:00
yoff
8df04c58e9 Merge pull request #7793 from tausbn/python-fix-bad-TPythonTuple-join-order
Python: Fix bad join order in `TPythonTuple`
2022-01-31 22:39:58 +01:00
Harry Maclean
e5b7478028 Merge pull request #7780 from github/hmac/split-tests
Ruby: Split up CI jobs
2022-02-01 09:10:01 +13:00
Nick Rolfe
990e07b986 Ruby/C#: add semmle.order attribute to edges in CFG tests 2022-01-31 20:08:24 +00:00
BACK Yonah
46c1744204 C/C++: getFullyConverted replaced by getConversion* 2022-01-31 18:51:18 +01:00
Taus
4a29095e3b Python: Fix bad join order in TPythonTuple
TL;DR: Something introduced the following bad join order:
```
(227s) Tuple counts for dom#TObject::TPythonTuple#ff/2@i2#8f58670w after 3m46s:
25000      ~0%     {2} r1 = SCAN PointsToContext::PointsToContext::appliesToScope_dispred#ff#prev_delta OUTPUT In.1, In.0 'context'
24000      ~1%     {2} r2 = JOIN r1 WITH @py_scope#f ON FIRST 1 OUTPUT Lhs.1 'context', Lhs.0
1076876712 ~6%     {3} r3 = JOIN r2 WITH Flow::TupleNode#class#f CARTESIAN PRODUCT OUTPUT Rhs.0, Lhs.0 'context', Lhs.1
870129666  ~0%     {3} r4 = JOIN r3 WITH Flow::ControlFlowNode::isLoad_dispred#f ON FIRST 1 OUTPUT Lhs.1 'context', Lhs.2, Lhs.0 'origin'
870129000  ~0%     {3} r5 = r4 AND NOT dom#TObject::TPythonTuple#ff#prev(Lhs.2 'origin', Lhs.0 'context')
870129000  ~1%     {3} r6 = SCAN r5 OUTPUT In.2 'origin', In.1, In.0 'context'
9000       ~0%     {2} r7 = JOIN r6 WITH Flow::ControlFlowNode::getScope_dispred#ff ON FIRST 2 OUTPUT Lhs.0 'origin', Lhs.2 'context'
                    return r7
```
(...the above being the tuple counts _at the point when I cancelled the
query_!)

Rewriting the code to force a join between `TupleNode#class` and
`getScope` results in the following join orders:

```
(0s) Tuple counts for TObject::scope_loads_tuplenode#ff/2@b3cf0bo5 after 13ms:
37369 ~3%     {1} r1 = JOIN Flow::TupleNode#class#f WITH Flow::ControlFlowNode::isLoad_dispred#f ON FIRST 1 OUTPUT Lhs.0 'origin'
37369 ~3%     {2} r2 = JOIN r1 WITH Flow::ControlFlowNode::getScope_dispred#ff ON FIRST 1 OUTPUT Rhs.1 's', Lhs.0 'origin'
            return r2
```
and
```
(78s) Tuple counts for dom#TObject::TPythonTuple#ff/2@i53#121c440w after 6ms:
34736 ~3%     {2} r1 = SCAN PointsToContext::PointsToContext::appliesToScope_dispred#ff#prev_delta OUTPUT In.1, In.0 'context'
7370  ~5%     {2} r2 = JOIN r1 WITH TObject::scope_loads_tuplenode#ff ON FIRST 1 OUTPUT Lhs.1 'context', Rhs.1 'origin'
7370  ~5%     {2} r3 = r2 AND NOT dom#TObject::TPythonTuple#ff#prev(Lhs.1 'origin', Lhs.0 'context')
7370  ~1%     {2} r4 = SCAN r3 OUTPUT In.1 'origin', In.0 'context'
            return r4
```
the latter being the largest iteration of `dom#TPythonTuple` throughout
the log.

No other major performance issues were observed.
2022-01-31 16:59:50 +00:00
BACK Yonah
56941dba6b C/C++ : Fixed select issue in Useless Test 2022-01-31 16:56:12 +01:00
BACK Yonah
ca2ff6f9fb C/C++: Fixing minor issues in Useless Test query 2022-01-31 16:04:56 +01:00
Tom Hvitved
5503abc73d Merge pull request #7772 from hvitved/csharp/event-accessor-event-null
C#: Guard against `AssociatedSymbol` not being an `IEventSymbol`
2022-01-31 14:52:02 +01:00