Commit Graph

8184 Commits

Author SHA1 Message Date
Dave Bartolomeo
2c88848d2f Merge pull request #2272 from jbj/getIRTypeForPRValue-join-order
C++/C#: Fix getIRTypeForPRValue join order
2019-11-07 12:22:39 -07:00
igfoo
c8c37c4976 Merge pull request #2271 from matt-gretton-dann/cpp-172-template-members
Template members
2019-11-07 16:30:08 +00:00
shati-patel
ec2008d57a Merge pull request #2275 from jf205/sd-4017
Learn CodeQL docs: add short note about new terminology
2019-11-07 16:12:12 +00:00
semmle-qlci
e65271dfad Merge pull request #2251 from asger-semmle/barrier-guard-improvements
Approved by esbena
2019-11-07 15:50:23 +00:00
semmle-qlci
f79c2a7630 Merge pull request #2224 from asger-semmle/access-paths-with-source-node-root
Approved by max-schaefer
2019-11-07 15:46:14 +00:00
James Fletcher
8178e3e671 Update docs/language/learn-ql/terminology-note.rst
Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com>
2019-11-07 14:40:04 +00:00
James Fletcher
d31ec56ea6 Update docs/language/learn-ql/index.rst
Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com>
2019-11-07 14:39:52 +00:00
Rasmus Wriedt Larsen
9ffb67a460 Merge pull request #2266 from tausbn/python-multiple-calls-to-init-join-order-fix
Python: Fix bad join order for `py/multiple-calls-to-init`.
2019-11-07 15:38:43 +01:00
james
db2039d8a7 docs: learn codeql terminology note 2019-11-07 14:32:05 +00:00
semmle-qlci
2b120def01 Merge pull request #2211 from hvitved/csharp/unsafe-deserialization
Approved by jf205
2019-11-07 14:16:13 +00:00
Matthew Gretton-Dann
ddf1ef8a7d C++: Add new test case for template member change
We now output literals for accesses to members of template parameters:

So for `foo` in the following example:

```
template<typename T> void bar(T& t) {
  T.foo(1)
}
```
2019-11-07 14:08:25 +00:00
Matthew Gretton-Dann
c0884e9a88 C++: Update expected results. 2019-11-07 14:08:25 +00:00
semmle-qlci
3a7f9a588d Merge pull request #2267 from max-schaefer/js/qltest-extractor-options
Approved by asger-semmle
2019-11-07 11:36:45 +00:00
shati-patel
7394d5c726 Merge pull request #2242 from felicitymay/codeql/SD-4059-markdown
Docs: update terminology in markdown files (SD-4059)
2019-11-07 11:30:32 +00:00
Jonas Jensen
6385528d5f C++/C#: Fix getIRTypeForPRValue join order
This predicate was taking 39s on a snapshot of Facebook Fizz because it
had disjuncts like this:

    43685     ~0%     {1} r34 = JOIN Type::FunctionPointerIshType#f AS L WITH Type::Type::getUnspecifiedType_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>
    43685     ~1%     {2} r35 = JOIN r34 WITH CppType::getTypeSize#ff AS R ON FIRST 1 OUTPUT R.<1>, r34.<0>
    170371500 ~2%     {2} r36 = JOIN r35 WITH IRType::IRSizedType#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r35.<1>
    43685     ~6%     {2} r37 = JOIN r36 WITH IRType::IRFunctionAddressType#class#ff AS R ON FIRST 1 OUTPUT r36.<1>, r36.<0>

Instead of fixing the joins in `getIRTypeForPRValue` itself, I've
changed the `IRType::getByteSize` predicate such that the optimiser
knows how to join with it efficiently.

The disjunct shown above now looks like this instead:

    43685  ~0%     {1} r26 = JOIN Type::FunctionPointerIshType#f AS L WITH Type::Type::getUnspecifiedType_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>
    43685  ~1%     {2} r27 = JOIN r26 WITH CppType::getTypeSize#ff AS R ON FIRST 1 OUTPUT R.<1>, r26.<0>
    43685  ~6%     {2} r28 = JOIN r27 WITH IRType::IRFunctionAddressType::getByteSize#ff_10#join_rhs AS R ON FIRST 1 OUTPUT r27.<1>, R.<1>
2019-11-07 11:48:16 +01:00
semmle-qlci
717490b670 Merge pull request #2265 from tausbn/python-fix-unused-import-global-name-used-join-order
Approved by RasmusWL
2019-11-06 16:38:07 +00:00
Max Schaefer
54e40a8977 JavaScript: Move --html all extractor options into options file. 2019-11-06 16:30:01 +00:00
Taus Brock-Nannestad
2b24eb2e70 Python: Fix bad join order for py/multiple-calls-to-init.
The `multiple_invocation_paths` predicate had a bad join order where
we (essentially) joined `i1` with `i2` and only then joined `i1` and `i2`
separately to reduce the number of tuples. The join coming from `i1 != i2` had
little impact, but `i1.getFunction() = multi` made a big difference (and
similarly for `i2`). I factored out the code so that these joins would be done
more eagerly. Thus, we went from

```
[2019-11-06 16:53:05] (38s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff/4@2ce75a
[2019-11-06 16:53:35] (68s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff:
                      134547    ~9%       {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<0>, I.<2>
                      235284431 ~3%       {4} r2 = JOIN r1 WITH CallGraph::TInvocation#fff AS R ON FIRST 1 OUTPUT r1.<0>, r1.<1>, R.<1>, R.<2>
                      235149884 ~3%       {4} r3 = SELECT r2 ON r2.<3> != r2.<1>
                      235149884 ~4%       {3} r4 = SCAN r3 OUTPUT r3.<1>, r3.<0>, r3.<3>
                      166753634 ~5%       {4} r5 = JOIN r4 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus#swapped AS R ON FIRST 1 OUTPUT R.<1>, r4.<2>, r4.<1>, r4.<0>
                      129778    ~0%       {4} r6 = JOIN r5 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus AS R ON FIRST 2 OUTPUT r5.<0>, r5.<3>, r5.<1>, r5.<2>
                                          return r6
[2019-11-06 16:53:35] (68s) Registering MethodCallOrder::multiple_invocation_paths#ffff + [] with content 1705dcbc08kd9aa40rp2g2e9civhv
[2019-11-06 16:53:35] (68s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff with 129778 rows and 4 columns.
```

to

```
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths_helper#ffff/4@586aec
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths_helper#ffff:
                      134547 ~0%     {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<2>, I.<0>
                      88111  ~4%     {3} r2 = JOIN r1 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus#swapped AS R ON FIRST 1 OUTPUT R.<1>, r1.<1>, r1.<0>
                      761305 ~0%     {4} r3 = JOIN r2 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus AS R ON FIRST 1 OUTPUT r2.<1>, r2.<2>, r2.<0>, R.<1>
                      673194 ~0%     {4} r4 = SELECT r3 ON r3.<3> != r3.<1>
                      673194 ~0%     {4} r5 = SCAN r4 OUTPUT r4.<2>, r4.<1>, r4.<3>, r4.<0>
                                     return r5
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths_helper#ffff + [] with content 20edaaecf25nldgp24d9c4et8m3kv
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths_helper#ffff with 673194 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs/4@9e5441
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs:
                      673194 ~0%     {4} r1 = SCAN MethodCallOrder::multiple_invocation_paths_helper#ffff AS I OUTPUT I.<2>, I.<3>, I.<0>, I.<1>
                                     return r1
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs + [] with content 2069301e655fi9mcovngg9hetfqas
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs with 673194 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff/4@2f7c34
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff:
                      134547 ~0%     {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<2>, I.<0>
                      129778 ~0%     {4} r2 = JOIN r1 WITH MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs AS R ON FIRST 2 OUTPUT R.<2>, R.<3>, r1.<0>, r1.<1>
                                     return r2
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths#ffff + [] with content 1705dcbc08kd9aa40rp2g2e9civhv
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff with 129778 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs/4@9f9146
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs:
                      129778 ~0%     {4} r1 = SCAN MethodCallOrder::multiple_invocation_paths#ffff AS I OUTPUT I.<0>, I.<3>, I.<1>, I.<2>
                                     return r1
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs + [] with content 17c3fe1fcbf6ghhdr7hiukqp41rst
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs with 129778 rows and 4 columns.
```

Execution time on `salt` went from 29.5s to somewhere below 299ms (the predicate
was not listed in the timing report).
2019-11-06 17:27:03 +01:00
Taus Brock-Nannestad
43148083eb Python: Fix bad join order for global_name_used.
As it turns out, there was a further bad join-order in the `global_name_used`
predicate. In this case, there was a common subexpression in the RA that was
being factored out and evaluated separately, producing a large number of tuples.
2019-11-06 16:37:01 +01:00
Nick Rolfe
5b00b21713 Merge pull request #2153 from matt-gretton-dann/cpp-447-support-non-type-template-parameters
RFC: C++ Support non type template parameter values
2019-11-06 15:11:34 +00:00
Rasmus Wriedt Larsen
47a292b241 Merge pull request #2264 from tausbn/python-fix-unused-import-join-order
Python: Fix bad join order in `py/unused-import`
2019-11-06 16:09:44 +01:00
Jonas Jensen
8ffd7c1055 Merge pull request #2222 from geoffw0/libraryperf
CPP: Improvements for ConditionallyInitializedVariable.ql
2019-11-06 15:54:16 +01:00
shati-patel
5d1d7ea075 Merge pull request #2257 from jf205/codeql-training-material
docs: update terminology in training material
2019-11-06 14:43:55 +00:00
Asger F
d9beb54dde Merge pull request #2102 from erik-krogh/deferredModel
JS: add Deferred model in js/use-of-returnless-function
2019-11-06 14:30:03 +00:00
Taus Brock-Nannestad
b6f16dee81 Python: Fix bad join order in py/unused-import 2019-11-06 15:14:48 +01:00
semmle-qlci
f73caac88d Merge pull request #2254 from asger-semmle/for-of-propread
Approved by max-schaefer
2019-11-06 13:44:55 +00:00
Max Schaefer
725059deea JavaScript: Remove --source-type module extractor options. 2019-11-06 13:01:59 +00:00
Max Schaefer
3ad5af7cef JavaScript: Move --extract-program-text extractor options into options files. 2019-11-06 13:01:55 +00:00
Max Schaefer
6b817203fd JavaScript: Move --tolerate-parse-errors extractor options into options file. 2019-11-06 13:01:28 +00:00
Max Schaefer
5681565d4a JavaScript: Move --html elements extractor options into options file. 2019-11-06 13:01:28 +00:00
Max Schaefer
8fdf6298b9 JavaScript: Remove --platform node extractor options. 2019-11-06 13:01:28 +00:00
Max Schaefer
4848372435 JavaScript: Replace --externs extractor flag with /** @externs */ comment. 2019-11-06 13:01:28 +00:00
Max Schaefer
1fa8c43a8c JavaScript: Remove a redundant extractor option. 2019-11-06 13:01:28 +00:00
Max Schaefer
79f1079460 JavaScript: Add options files with --experimental extractor options. 2019-11-06 13:01:23 +00:00
James Fletcher
2e7bd4db6f Update docs/language/ql-training/cpp/program-representation-cpp.rst
Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com>
2019-11-06 12:59:45 +00:00
Max Schaefer
a4bf361f64 JavaScript: Remove remaining --experimental extractor options. 2019-11-06 12:54:44 +00:00
james
24615f2bdb docs: further review comments 2019-11-06 12:44:45 +00:00
Erik Krogh Kristensen
19554ff6e7 change "e.g." to "for example" in qldoc 2019-11-06 13:37:54 +01:00
Erik Krogh Kristensen
dc923ef694 remove change note
Co-Authored-By: Esben Sparre Andreasen <esbena@github.com>
2019-11-06 13:28:46 +01:00
Asger F
81723ab92a JS: Update GlobalAccessPaths test 2019-11-06 11:58:06 +00:00
Asger F
3ec95881b4 Update javascript/ql/src/semmle/javascript/GlobalAccessPaths.qll
Co-Authored-By: Max Schaefer <54907921+max-schaefer@users.noreply.github.com>
2019-11-06 11:58:06 +00:00
Asger F
7e80823cb6 JS: Fix deprecated API usage 2019-11-06 11:58:06 +00:00
Asger F
b373901e11 JS: Avoid leading dot in access paths 2019-11-06 11:58:06 +00:00
Asger F
c365833731 JS: Refactor the public access path API 2019-11-06 11:58:06 +00:00
Asger F
e90516d4d8 JS: Dont use getALocalSource in fromRhs 2019-11-06 11:58:06 +00:00
Asger F
bc35f24f31 JS: Generalize access paths to arbitrary root nodes 2019-11-06 11:58:06 +00:00
Asger F
7a7a8b2b09 JS: More steps in getImmediatePredecessor 2019-11-06 11:58:06 +00:00
Max Schaefer
2bcd418c23 Merge pull request #2261 from Semmle/esbena-patch-2
Doc: Add missing `t` in `support`.
2019-11-06 09:24:30 +00:00
Esben Sparre Andreasen
21d4e5f186 Doc: Add missing t in support. 2019-11-06 10:16:43 +01:00
semmle-qlci
04f0c22f24 Merge pull request #2203 from erik-krogh/ignorePureFunction
Approved by max-schaefer, mchammer01
2019-11-06 09:09:11 +00:00