Commit Graph

9240 Commits

Author SHA1 Message Date
Ahmed Farid
5742046edf Update PossibleTimingAttackAgainstSignature.ql 2022-06-29 00:51:51 +01:00
Ahmed Farid
acbb4042df Update TimingAttack.qhelp 2022-06-29 00:51:12 +01:00
yoff
1105cd569b Merge branch 'main' into python/port-tarslip 2022-06-28 22:17:28 +02:00
yoff
6087bc6888 Merge branch 'main' into python/more-logic-tests 2022-06-28 22:16:38 +02:00
yoff
ac0c8d238f python: only clear taint on false-edge 2022-06-28 20:14:52 +00:00
Taus
38b8640582 Python: Fix bad join in RegExpBackRef::getGroup
Although this wasn't (as far as I know) causing any performance issues,
it was making the join-order badness report quite noisy, and so I
figured it was worth fixing.

Before:
```
Tuple counts for RegexTreeView::RegExpBackRef::getGroup#dispred#f0820431#ff/2@d3441d0b after 84ms:
1501195 ~3%     {2} r1 = JOIN RegexTreeView::RegExpTerm::getLiteral#dispred#f0820431#ff_10#join_rhs WITH RegexTreeView::RegExpTerm::getLiteral#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1 'result'
149     ~0%     {5} r2 = JOIN r1 WITH RegexTreeView::RegExpBackRef#class#31aac2a7#ffff ON FIRST 1 OUTPUT Rhs.1, Rhs.2, Rhs.3, Lhs.1 'result', Lhs.0 'this'
149     ~1%     {3} r3 = JOIN r2 WITH regex::RegexString::numbered_backreference#dispred#f0820431#ffff ON FIRST 3 OUTPUT Lhs.3 'result', Rhs.3, Lhs.4 'this'
4       ~0%     {2} r4 = JOIN r3 WITH RegexTreeView::RegExpGroup::getNumber#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.2 'this', Lhs.0 'result'

1501195 ~3%     {2} r5 = JOIN RegexTreeView::RegExpTerm::getLiteral#dispred#f0820431#ff_10#join_rhs WITH RegexTreeView::RegExpTerm::getLiteral#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'result', Rhs.1 'result'
42526   ~0%     {5} r6 = JOIN r5 WITH RegexTreeView::RegExpGroup#31aac2a7#ffff ON FIRST 1 OUTPUT Lhs.1 'this', Lhs.0 'result', Rhs.1, Rhs.2, Rhs.3
22      ~0%     {8} r7 = JOIN r6 WITH RegexTreeView::RegExpBackRef#class#31aac2a7#ffff ON FIRST 1 OUTPUT Lhs.2, Lhs.3, Lhs.4, Lhs.1 'result', Lhs.0 'this', Rhs.1, Rhs.2, Rhs.3
0       ~0%     {6} r8 = JOIN r7 WITH regex::RegexString::getGroupName#dispred#f0820431#ffff ON FIRST 3 OUTPUT Lhs.5, Lhs.6, Lhs.7, Rhs.3, Lhs.3 'result', Lhs.4 'this'
0       ~0%     {2} r9 = JOIN r8 WITH regex::RegexString::named_backreference#dispred#f0820431#ffff ON FIRST 4 OUTPUT Lhs.5 'this', Lhs.4 'result'

4       ~0%     {2} r10 = r4 UNION r9
                return r10
```

In this case I opted for a classical solution: tying together the
literal and number (or name) part of the backreference in order to
encourage a two-column join.

After:
```
Tuple counts for RegexTreeView::RegExpBackRef::getGroup#dispred#f0820431#ff/2@b0cc4d5n after 0ms:
898  ~1%     {3} r1 = JOIN RegexTreeView::RegExpTerm::getLiteral#dispred#f0820431#ff WITH RegexTreeView::RegExpGroup::getNumber#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Lhs.0 'result'
4    ~0%     {2} r2 = JOIN r1 WITH RegexTreeView::RegExpBackRef::hasLiteralAndNumber#f0820431#fff_120#join_rhs ON FIRST 2 OUTPUT Rhs.2 'this', Lhs.2 'result'

1110 ~0%     {5} r3 = JOIN RegexTreeView::RegExpGroup#31aac2a7#ffff WITH RegexTreeView::RegExpTerm::getLiteral#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.3, Lhs.0 'result', Rhs.1
146  ~0%     {3} r4 = JOIN r3 WITH regex::RegexString::getGroupName#dispred#f0820431#ffff ON FIRST 3 OUTPUT Lhs.4, Rhs.3, Lhs.3 'result'
0    ~0%     {2} r5 = JOIN r4 WITH RegexTreeView::RegExpBackRef::hasLiteralAndName#f0820431#fff_120#join_rhs ON FIRST 2 OUTPUT Rhs.2 'this', Lhs.2 'result'

4    ~0%     {2} r6 = r2 UNION r5
            return r6
```
2022-06-28 16:51:09 +00:00
Taus
b98c482c47 Python: Fix bad join in MRO flatten_list
This bad join was identified by the join-order-badness report, which
showed that:

py/use-of-input:MRO::flatten_list#f4eaf05f#fff#9c5fe54whnlqffdgu65vhb8uhpg# (order_500000)

calculated a whopping 212,820,108 tuples in order to produce an output of
size 55516, roughly 3833 times more effort than needed.

Here's a snippet of the slowest iteration of that predicate:
```
Tuple counts for MRO::flatten_list#f4eaf05f#fff/3@i1839#0265eb3w after 14ms:
0     ~0%     {3} r1 = JOIN MRO::need_flattening#f4eaf05f#f#prev_delta WITH MRO::ConsList#f4eaf05f#fff#reorder_2_0_1#prev ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'list', Rhs.2
0     ~0%     {3} r2 = JOIN r1 WITH MRO::ClassList::length#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.2, Lhs.1 'list', Rhs.1 'n'
0     ~0%     {3} r3 = JOIN r2 WITH MRO::ClassListList::flatten#dispred#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.1 'list', Lhs.2 'n', Rhs.1 'result'

0     ~0%     {3} r4 = SCAN MRO::ConsList#f4eaf05f#fff#prev_delta OUTPUT In.2 'list', In.0, In.1
0     ~0%     {3} r5 = JOIN r4 WITH MRO::need_flattening#f4eaf05f#f#prev ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.0 'list'
0     ~0%     {3} r6 = JOIN r5 WITH MRO::ClassList::length#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.1, Lhs.2 'list', Rhs.1 'n'
0     ~0%     {3} r7 = JOIN r6 WITH MRO::ClassListList::flatten#dispred#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.1 'list', Lhs.2 'n', Rhs.1 'result'

0     ~0%     {3} r8 = r3 UNION r7

26355 ~2%     {3} r9 = SCAN MRO::ConsList#f4eaf05f#fff#prev OUTPUT In.2 'list', In.0, In.1

0     ~0%     {3} r10 = JOIN r9 WITH MRO::need_flattening#f4eaf05f#f#prev ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.0 'list'
0     ~0%     {3} r11 = JOIN r10 WITH MRO::ClassList::length#f0820431#ff#prev_delta ON FIRST 1 OUTPUT Lhs.1, Lhs.2 'list', Rhs.1 'n'
0     ~0%     {3} r12 = JOIN r11 WITH MRO::ClassListList::flatten#dispred#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.1 'list', Lhs.2 'n', Rhs.1 'result'
...
```
(... and a bunch more lines. The same construction appears several times,
but the join order is the same each time.)

Clearly it would be better to start with whatever is in `need_flattening`,
and then do the other joins. This is what the present fix does (by
unbinding `list` in all but the `needs_flattening` call).

After the fix, the slowest iteration is as follows:

```
Tuple counts for MRO::flatten_list#f4eaf05f#fff/3@i2617#8155ab3w after 9ms:
0 ~0%     {2} r1 = SCAN MRO::need_flattening#f4eaf05f#f#prev_delta OUTPUT In.0 'list', In.0 'list'

0 ~0%     {3} r2 = JOIN r1 WITH MRO::ConsList#f4eaf05f#fff#reorder_2_0_1#prev ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'list', Rhs.2
0 ~0%     {3} r3 = JOIN r2 WITH MRO::ClassList::length#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.2, Lhs.1 'list', Rhs.1 'n'
0 ~0%     {3} r4 = JOIN r3 WITH MRO::ClassListList::flatten#dispred#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.1 'list', Lhs.2 'n', Rhs.1 'result'

1 ~0%     {2} r5 = SCAN MRO::need_flattening#f4eaf05f#f#prev OUTPUT In.0 'list', In.0 'list'

0 ~0%     {3} r6 = JOIN r5 WITH MRO::ConsList#f4eaf05f#fff#reorder_2_0_1#prev_delta ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'list', Rhs.2
0 ~0%     {3} r7 = JOIN r6 WITH MRO::ClassList::length#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.2, Lhs.1 'list', Rhs.1 'n'
0 ~0%     {3} r8 = JOIN r7 WITH MRO::ClassListList::flatten#dispred#f0820431#ff#prev ON FIRST 1 OUTPUT Lhs.1 'list', Lhs.2 'n', Rhs.1 'result'
...
```
(... and so on. The remainder is 0 tuples all the way.)

In total, we went from
```
40.6s |  7614 |  15ms @ 1839 | MRO::flatten_list#f4eaf05f#fff@0265eb3w
```
to
```
7.8s |  7614 |  11ms @ 2617 | MRO::flatten_list#f4eaf05f#fff@8155ab3w
```
2022-06-28 14:17:47 +00:00
Asger F
a522562f93 Merge pull request #9369 from asgerf/python/api-graph-api
Python: API graph renaming and documentation
2022-06-28 14:48:12 +02:00
yoff
834d2603a2 python: update use of barrier guard 2022-06-28 11:15:37 +00:00
Asger F
b3b53360ae Python: change category to deprecated because library is apparently supported anymore 2022-06-28 12:14:28 +02:00
Asger F
5dfc3c6537 Python: rename change note again 2022-06-28 12:10:26 +02:00
Asger F
d9f57e6d23 Python: rename change note file 2022-06-28 11:41:07 +02:00
Asger F
6d25fb6988 Python: add change note 2022-06-28 11:28:30 +02:00
Erik Krogh Kristensen
a343ceaf8b add suspicious-regexp-range query 2022-06-28 09:49:27 +02:00
Asger F
4c73ab2679 Apply suggestions from code review
Co-authored-by: Taus <tausbn@github.com>
2022-06-28 09:48:53 +02:00
Asger F
a033338d20 Python: Explicitly mention lack of transitive flow in asSource/asSink 2022-06-28 09:46:26 +02:00
Asger F
9b27a7cbcd Python: Dont claim that external libraries are excluded from the database 2022-06-28 09:28:26 +02:00
yoff
67b6f215dc Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2022-06-28 08:05:53 +02:00
yoff
1788507571 python: add qldoc 2022-06-27 21:00:12 +00:00
Rasmus Lerchedahl Petersen
a1fe8a5b2b python: handle not in BarrierGuard
in the program
```python
if not is_safe(path):
  return
```
the last node in the `ConditionBlock` is `not is_safe(path)`,
so it would never match "a call to is_safe".
Thus, guards inside `not` would not be part of `GuardNode`
(nor `BarrierGuard`). Now they can.
2022-06-27 20:10:47 +00:00
Rasmus Lerchedahl Petersen
882000afb3 python: not is confusing our logic
- added `is_unsafe`
- added "negated version" of two tests.
These versions do not use `not` and the analysis gets the taint right.
2022-06-27 20:10:47 +00:00
Taus
dc0f50d49a Python: Clean up variable names
Makes it more consistent with the names used in
`legalMergeCandidateNonEmpty`.
2022-06-27 19:54:09 +00:00
Taus
8fc9ce9699 Python: Fix bad join in MRO
Fixes a bad join in `list_of_linearization_of_bases_plus_bases`.

Previvously, we joined together `ConsList` and `getBase` before filtering
these out using the recursive call. Now we do the recursion first.

Co-authored-by: yoff <yoff@github.com>
2022-06-27 19:54:09 +00:00
Asger F
cc57cb8af5 Merge branch 'main' into post-release-prep/codeql-cli-2.10.0 2022-06-27 20:37:25 +02:00
root
655b9d4262 Python: Timing attack 2022-06-27 12:18:45 -04:00
Rasmus Wriedt Larsen
9e154ff4bd Merge branch 'main' into python/port-tarslip 2022-06-27 14:36:15 +02:00
Erik Krogh Kristensen
9bc12ed8fd sync review changes to other languages 2022-06-24 13:12:15 +02:00
Erik Krogh Kristensen
28ac47689f changes based on reviews 2022-06-24 13:11:46 +02:00
github-actions[bot]
d506f448ef Post-release preparation for codeql-cli-2.10.0 2022-06-24 07:36:33 +00:00
yoff
5042c804dd python: sync files and fix many small things
- but now we have non-monotonic recursion again...
2022-06-23 14:57:06 +00:00
Anders Schack-Mulligen
dc517a758e Autoformat 2022-06-23 14:44:40 +02:00
Erik Krogh Kristensen
724721c5c8 fix typo 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
22871138c6 simplify the recursion between TTrace and isReachableFromStartTuple
similar to the fix made by Shack in `ExponentialBackTracking.qll`
2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
be37763125 improve performance of process() by pruning accept states early 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
bf20b7dfc5 add change note for the ReDoS renamings 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
3bea7df45d add deprecated aliases in the old locations, and use the Query.qll pattern for js/polynomial-redos 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
13482fc97b rename ReDoSUtil to NfaUtils, and rename the "performance" folder to "regexp" 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
6b0df9bdfb refactor the concretize algorithm 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
dbeae9aefb make a parameterized module out of the RegexpMatching implementation 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
7fb3d81d2f add further normalization of char classses 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
3be4a86acd make ReDoSPruning into a parameterized module 2022-06-23 14:36:25 +02:00
Erik Krogh Kristensen
dc06e9df02 move predicates that depend on isReDoSCandidate into a ReDoSPruning module 2022-06-23 14:36:24 +02:00
Anders Schack-Mulligen
4a317a25d3 Dataflow: Sync. 2022-06-23 14:34:52 +02:00
Asger F
d94010c244 Grammar: report -> reports 2022-06-23 14:17:52 +02:00
yoff
a2851baa9f python: fix import of "merge moved" file 2022-06-23 12:05:55 +00:00
github-actions[bot]
a74051c658 Release preparation for version 2.10.0 2022-06-23 11:17:46 +00:00
Rasmus Wriedt Larsen
3248f7b423 Merge pull request #9649 from RasmusWL/certificate-modeling
Python/JS/Ruby: Ignore common words (like certain) as sensitive data source
2022-06-23 12:04:58 +02:00
yoff
140dc1a61e merge in main 2022-06-23 09:05:32 +00:00
yoff
8bf60301da python: we have hidden isParameterOf
but now allow a clear alternative
2022-06-23 08:57:50 +00:00
yoff
fe0c5d8ee5 python: make ArgumentNode publicly usable
- add `getCall`
2022-06-23 08:48:55 +00:00