Commit Graph

21651 Commits

Author SHA1 Message Date
Robert Marsh
51c4ef4f7f C++: add SSA IR test for array initializers 2019-11-06 13:32:35 -08:00
Dave Bartolomeo
a9e3bfbd11 C++/C#: Treat string literals like read-only global variables for alias purposes.
Previously, we didn't track string literals as known memory locations at all, so they all just got marked as `UnknownMemoryLocation`, just like an aribtrary read from a random pointer. This led to some confusing def-use chains, where it would look like the contents of a string literal were being written to by the side effect of an earlier function call, which of course is impossible.

To fix this, I've made two changes. First, each string literal is now given a corresponding `IRVariable` (specifically `IRStringLiteral`), since a string literal behaves more or less as a read-only global variable. Second, the `IRVariable` for each string literal is now marked `isReadOnly()`, which the alias analysis uses to determine that an arbitrary write to aliased memory will not overwrite the contents of a string literal.

I originally planned to treat all string literals with the same value as being the same memory location, since this is the usual behavior of modern compilers. However, this made implementing `IRVariable.getAST()` tricky for string literals, so I left them unpooled.
2019-11-06 13:08:28 -07:00
Robin Neatherway
7850d67a78 Remove TODO comment
I've checked Hamcrest versions 1.3, 2.0, 2.1 and 2.2
2019-11-06 17:47:02 +00:00
Geoffrey White
6c38f55e28 CPP: QLDoc protocols.qll. 2019-11-06 17:06:57 +00:00
semmle-qlci
717490b670 Merge pull request #2265 from tausbn/python-fix-unused-import-global-name-used-join-order
Approved by RasmusWL
2019-11-06 16:38:07 +00:00
Max Schaefer
54e40a8977 JavaScript: Move --html all extractor options into options file. 2019-11-06 16:30:01 +00:00
Geoffrey White
e886cf7297 CPP: 'i.e.' -> 'that is'. 2019-11-06 16:27:06 +00:00
Taus Brock-Nannestad
2b24eb2e70 Python: Fix bad join order for py/multiple-calls-to-init.
The `multiple_invocation_paths` predicate had a bad join order where
we (essentially) joined `i1` with `i2` and only then joined `i1` and `i2`
separately to reduce the number of tuples. The join coming from `i1 != i2` had
little impact, but `i1.getFunction() = multi` made a big difference (and
similarly for `i2`). I factored out the code so that these joins would be done
more eagerly. Thus, we went from

```
[2019-11-06 16:53:05] (38s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff/4@2ce75a
[2019-11-06 16:53:35] (68s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff:
                      134547    ~9%       {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<0>, I.<2>
                      235284431 ~3%       {4} r2 = JOIN r1 WITH CallGraph::TInvocation#fff AS R ON FIRST 1 OUTPUT r1.<0>, r1.<1>, R.<1>, R.<2>
                      235149884 ~3%       {4} r3 = SELECT r2 ON r2.<3> != r2.<1>
                      235149884 ~4%       {3} r4 = SCAN r3 OUTPUT r3.<1>, r3.<0>, r3.<3>
                      166753634 ~5%       {4} r5 = JOIN r4 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus#swapped AS R ON FIRST 1 OUTPUT R.<1>, r4.<2>, r4.<1>, r4.<0>
                      129778    ~0%       {4} r6 = JOIN r5 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus AS R ON FIRST 2 OUTPUT r5.<0>, r5.<3>, r5.<1>, r5.<2>
                                          return r6
[2019-11-06 16:53:35] (68s) Registering MethodCallOrder::multiple_invocation_paths#ffff + [] with content 1705dcbc08kd9aa40rp2g2e9civhv
[2019-11-06 16:53:35] (68s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff with 129778 rows and 4 columns.
```

to

```
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths_helper#ffff/4@586aec
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths_helper#ffff:
                      134547 ~0%     {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<2>, I.<0>
                      88111  ~4%     {3} r2 = JOIN r1 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus#swapped AS R ON FIRST 1 OUTPUT R.<1>, r1.<1>, r1.<0>
                      761305 ~0%     {4} r3 = JOIN r2 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus AS R ON FIRST 1 OUTPUT r2.<1>, r2.<2>, r2.<0>, R.<1>
                      673194 ~0%     {4} r4 = SELECT r3 ON r3.<3> != r3.<1>
                      673194 ~0%     {4} r5 = SCAN r4 OUTPUT r4.<2>, r4.<1>, r4.<3>, r4.<0>
                                     return r5
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths_helper#ffff + [] with content 20edaaecf25nldgp24d9c4et8m3kv
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths_helper#ffff with 673194 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs/4@9e5441
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs:
                      673194 ~0%     {4} r1 = SCAN MethodCallOrder::multiple_invocation_paths_helper#ffff AS I OUTPUT I.<2>, I.<3>, I.<0>, I.<1>
                                     return r1
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs + [] with content 2069301e655fi9mcovngg9hetfqas
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs with 673194 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff/4@2f7c34
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff:
                      134547 ~0%     {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<2>, I.<0>
                      129778 ~0%     {4} r2 = JOIN r1 WITH MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs AS R ON FIRST 2 OUTPUT R.<2>, R.<3>, r1.<0>, r1.<1>
                                     return r2
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths#ffff + [] with content 1705dcbc08kd9aa40rp2g2e9civhv
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff with 129778 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs/4@9f9146
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs:
                      129778 ~0%     {4} r1 = SCAN MethodCallOrder::multiple_invocation_paths#ffff AS I OUTPUT I.<0>, I.<3>, I.<1>, I.<2>
                                     return r1
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs + [] with content 17c3fe1fcbf6ghhdr7hiukqp41rst
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs with 129778 rows and 4 columns.
```

Execution time on `salt` went from 29.5s to somewhere below 299ms (the predicate
was not listed in the timing report).
2019-11-06 17:27:03 +01:00
Geoffrey White
81c58d5a64 CPP: Improve QLDoc comments. 2019-11-06 16:20:39 +00:00
Geoffrey White
0c3f4e530f CPP: Make some library predicates private. 2019-11-06 16:07:28 +00:00
Geoffrey White
399ac1f112 CPP: Rename 'getAssertedFalseCondition' to something less misleading. 2019-11-06 15:57:44 +00:00
Taus Brock-Nannestad
43148083eb Python: Fix bad join order for global_name_used.
As it turns out, there was a further bad join-order in the `global_name_used`
predicate. In this case, there was a common subexpression in the RA that was
being factored out and evaluated separately, producing a large number of tuples.
2019-11-06 16:37:01 +01:00
Nick Rolfe
5b00b21713 Merge pull request #2153 from matt-gretton-dann/cpp-447-support-non-type-template-parameters
RFC: C++ Support non type template parameter values
2019-11-06 15:11:34 +00:00
Rasmus Wriedt Larsen
47a292b241 Merge pull request #2264 from tausbn/python-fix-unused-import-join-order
Python: Fix bad join order in `py/unused-import`
2019-11-06 16:09:44 +01:00
Jonas Jensen
8ffd7c1055 Merge pull request #2222 from geoffw0/libraryperf
CPP: Improvements for ConditionallyInitializedVariable.ql
2019-11-06 15:54:16 +01:00
Jonas Jensen
217ecd3551 C++: Add <p> tags to split text into paragraphs
Without this, the rendered output is one big paragraph.
2019-11-06 15:50:08 +01:00
shati-patel
5d1d7ea075 Merge pull request #2257 from jf205/codeql-training-material
docs: update terminology in training material
2019-11-06 14:43:55 +00:00
Asger F
d9beb54dde Merge pull request #2102 from erik-krogh/deferredModel
JS: add Deferred model in js/use-of-returnless-function
2019-11-06 14:30:03 +00:00
Jonas Jensen
76a3db9eed Merge remote-tracking branch 'upstream/master' into ir-copy-unloaded-result 2019-11-06 15:21:22 +01:00
Taus Brock-Nannestad
b6f16dee81 Python: Fix bad join order in py/unused-import 2019-11-06 15:14:48 +01:00
semmle-qlci
f73caac88d Merge pull request #2254 from asger-semmle/for-of-propread
Approved by max-schaefer
2019-11-06 13:44:55 +00:00
Geoffrey White
f9feb05a72 CPP: Add a test of NtohlArrayNoBoundOpenSource.ql. 2019-11-06 13:36:31 +00:00
Jonas Jensen
aa841c306d C++: Use virtual dispatch in DefaultTaintTracking
This bit is only used by the compatibility code that sends flow into
parameters of functions without body.
2019-11-06 14:04:07 +01:00
Jonas Jensen
ec9ef33486 C++: IR data flow through inheritance conversions
This makes IR data flow behave more like AST data flow, and it makes IR
virtual dispatch work without further changes.
2019-11-06 14:04:07 +01:00
Jonas Jensen
49008c9ff5 C++: IR data flow local virtual dispatch
This is just good enough to cause no performance regressions and pass
the virtual-dispatch tests we have for `security.TaintTracking`. In
particular, it fixes the tests for `UncontrolledProcessOperation.ql`
when enabling `DefaultTaintTracking.qll`.
2019-11-06 14:04:02 +01:00
Max Schaefer
725059deea JavaScript: Remove --source-type module extractor options. 2019-11-06 13:01:59 +00:00
Max Schaefer
3ad5af7cef JavaScript: Move --extract-program-text extractor options into options files. 2019-11-06 13:01:55 +00:00
Max Schaefer
6b817203fd JavaScript: Move --tolerate-parse-errors extractor options into options file. 2019-11-06 13:01:28 +00:00
Max Schaefer
5681565d4a JavaScript: Move --html elements extractor options into options file. 2019-11-06 13:01:28 +00:00
Max Schaefer
8fdf6298b9 JavaScript: Remove --platform node extractor options. 2019-11-06 13:01:28 +00:00
Max Schaefer
4848372435 JavaScript: Replace --externs extractor flag with /** @externs */ comment. 2019-11-06 13:01:28 +00:00
Max Schaefer
1fa8c43a8c JavaScript: Remove a redundant extractor option. 2019-11-06 13:01:28 +00:00
Max Schaefer
79f1079460 JavaScript: Add options files with --experimental extractor options. 2019-11-06 13:01:23 +00:00
James Fletcher
2e7bd4db6f Update docs/language/ql-training/cpp/program-representation-cpp.rst
Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com>
2019-11-06 12:59:45 +00:00
Max Schaefer
a4bf361f64 JavaScript: Remove remaining --experimental extractor options. 2019-11-06 12:54:44 +00:00
james
24615f2bdb docs: further review comments 2019-11-06 12:44:45 +00:00
Erik Krogh Kristensen
19554ff6e7 change "e.g." to "for example" in qldoc 2019-11-06 13:37:54 +01:00
Erik Krogh Kristensen
dc923ef694 remove change note
Co-Authored-By: Esben Sparre Andreasen <esbena@github.com>
2019-11-06 13:28:46 +01:00
Asger F
81723ab92a JS: Update GlobalAccessPaths test 2019-11-06 11:58:06 +00:00
Asger F
3ec95881b4 Update javascript/ql/src/semmle/javascript/GlobalAccessPaths.qll
Co-Authored-By: Max Schaefer <54907921+max-schaefer@users.noreply.github.com>
2019-11-06 11:58:06 +00:00
Asger F
7e80823cb6 JS: Fix deprecated API usage 2019-11-06 11:58:06 +00:00
Asger F
b373901e11 JS: Avoid leading dot in access paths 2019-11-06 11:58:06 +00:00
Asger F
c365833731 JS: Refactor the public access path API 2019-11-06 11:58:06 +00:00
Asger F
e90516d4d8 JS: Dont use getALocalSource in fromRhs 2019-11-06 11:58:06 +00:00
Asger F
bc35f24f31 JS: Generalize access paths to arbitrary root nodes 2019-11-06 11:58:06 +00:00
Asger F
7a7a8b2b09 JS: More steps in getImmediatePredecessor 2019-11-06 11:58:06 +00:00
Max Schaefer
2bcd418c23 Merge pull request #2261 from Semmle/esbena-patch-2
Doc: Add missing `t` in `support`.
2019-11-06 09:24:30 +00:00
Esben Sparre Andreasen
21d4e5f186 Doc: Add missing t in support. 2019-11-06 10:16:43 +01:00
semmle-qlci
04f0c22f24 Merge pull request #2203 from erik-krogh/ignorePureFunction
Approved by max-schaefer, mchammer01
2019-11-06 09:09:11 +00:00
Jonas Jensen
fca7d9a391 C#: Cache IRType like for C++ 2019-11-06 07:42:34 +01:00