Commit Graph

19777 Commits

Author SHA1 Message Date
Dave Bartolomeo
f808dcefab Merge pull request #2277 from ian-semmle/cfg_diffs
C++: Remove tests for CFG differences
2019-11-07 12:41:40 -07:00
Dave Bartolomeo
64480c2ace Merge pull request #1999 from jbj/ir-copy-unloaded-result
C++: Make sure there's a Instruction for each Expr
2019-11-07 12:31:54 -07:00
Dave Bartolomeo
2c88848d2f Merge pull request #2272 from jbj/getIRTypeForPRValue-join-order
C++/C#: Fix getIRTypeForPRValue join order
2019-11-07 12:22:39 -07:00
Ian Lynagh
b5af4e5acd C++: Remove tests for CFG differences
Now that we have switched over, they are no longer interesting.
2019-11-07 16:32:18 +00:00
igfoo
c8c37c4976 Merge pull request #2271 from matt-gretton-dann/cpp-172-template-members
Template members
2019-11-07 16:30:08 +00:00
shati-patel
ec2008d57a Merge pull request #2275 from jf205/sd-4017
Learn CodeQL docs: add short note about new terminology
2019-11-07 16:12:12 +00:00
Asger F
812ee34bbc JS: Use Files.exists() instead 2019-11-07 15:53:29 +00:00
semmle-qlci
e65271dfad Merge pull request #2251 from asger-semmle/barrier-guard-improvements
Approved by esbena
2019-11-07 15:50:23 +00:00
semmle-qlci
f79c2a7630 Merge pull request #2224 from asger-semmle/access-paths-with-source-node-root
Approved by max-schaefer
2019-11-07 15:46:14 +00:00
James Fletcher
8178e3e671 Update docs/language/learn-ql/terminology-note.rst
Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com>
2019-11-07 14:40:04 +00:00
James Fletcher
d31ec56ea6 Update docs/language/learn-ql/index.rst
Co-Authored-By: shati-patel <42641846+shati-patel@users.noreply.github.com>
2019-11-07 14:39:52 +00:00
Rasmus Wriedt Larsen
9ffb67a460 Merge pull request #2266 from tausbn/python-multiple-calls-to-init-join-order-fix
Python: Fix bad join order for `py/multiple-calls-to-init`.
2019-11-07 15:38:43 +01:00
Asger F
8544850945 JS: Generalize StringOps::Includes to ::InclusionTest 2019-11-07 14:35:17 +00:00
james
db2039d8a7 docs: learn codeql terminology note 2019-11-07 14:32:05 +00:00
semmle-qlci
2b120def01 Merge pull request #2211 from hvitved/csharp/unsafe-deserialization
Approved by jf205
2019-11-07 14:16:13 +00:00
Matthew Gretton-Dann
ddf1ef8a7d C++: Add new test case for template member change
We now output literals for accesses to members of template parameters:

So for `foo` in the following example:

```
template<typename T> void bar(T& t) {
  T.foo(1)
}
```
2019-11-07 14:08:25 +00:00
Matthew Gretton-Dann
c0884e9a88 C++: Update expected results. 2019-11-07 14:08:25 +00:00
Erik Krogh Kristensen
0c080a82be fix expected output 2019-11-07 14:31:09 +01:00
Erik Krogh Kristensen
232e875274 add test for getEnclosingExpr 2019-11-07 14:29:31 +01:00
Erik Krogh Kristensen
e4f6f41634 add DataFlow::getEnclosingExpr to get the an Expr from a potentially reflective call 2019-11-07 14:29:31 +01:00
semmle-qlci
3a7f9a588d Merge pull request #2267 from max-schaefer/js/qltest-extractor-options
Approved by asger-semmle
2019-11-07 11:36:45 +00:00
shati-patel
7394d5c726 Merge pull request #2242 from felicitymay/codeql/SD-4059-markdown
Docs: update terminology in markdown files (SD-4059)
2019-11-07 11:30:32 +00:00
Jonas Jensen
6385528d5f C++/C#: Fix getIRTypeForPRValue join order
This predicate was taking 39s on a snapshot of Facebook Fizz because it
had disjuncts like this:

    43685     ~0%     {1} r34 = JOIN Type::FunctionPointerIshType#f AS L WITH Type::Type::getUnspecifiedType_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>
    43685     ~1%     {2} r35 = JOIN r34 WITH CppType::getTypeSize#ff AS R ON FIRST 1 OUTPUT R.<1>, r34.<0>
    170371500 ~2%     {2} r36 = JOIN r35 WITH IRType::IRSizedType#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r35.<1>
    43685     ~6%     {2} r37 = JOIN r36 WITH IRType::IRFunctionAddressType#class#ff AS R ON FIRST 1 OUTPUT r36.<1>, r36.<0>

Instead of fixing the joins in `getIRTypeForPRValue` itself, I've
changed the `IRType::getByteSize` predicate such that the optimiser
knows how to join with it efficiently.

The disjunct shown above now looks like this instead:

    43685  ~0%     {1} r26 = JOIN Type::FunctionPointerIshType#f AS L WITH Type::Type::getUnspecifiedType_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>
    43685  ~1%     {2} r27 = JOIN r26 WITH CppType::getTypeSize#ff AS R ON FIRST 1 OUTPUT R.<1>, r26.<0>
    43685  ~6%     {2} r28 = JOIN r27 WITH IRType::IRFunctionAddressType::getByteSize#ff_10#join_rhs AS R ON FIRST 1 OUTPUT r27.<1>, R.<1>
2019-11-07 11:48:16 +01:00
Max Schaefer
e314869e5c JavaScript: Classify files with names ending in _min as minified.
We already do the same for `-min` and `.min`. [Here](https://github.com/antoniogarrote/rdfstore-js/blob/master/dist/rdfstore_min.js) is a real-world example.
2019-11-07 10:33:47 +00:00
Sauyon Lee
0040c9fb4c Update links to OWASP cheat sheet 2019-11-06 20:21:47 -08:00
Robert Marsh
81ad11090e C++: uninit instr for string literal initializers 2019-11-06 13:37:03 -08:00
Robert Marsh
51c4ef4f7f C++: add SSA IR test for array initializers 2019-11-06 13:32:35 -08:00
Dave Bartolomeo
a9e3bfbd11 C++/C#: Treat string literals like read-only global variables for alias purposes.
Previously, we didn't track string literals as known memory locations at all, so they all just got marked as `UnknownMemoryLocation`, just like an aribtrary read from a random pointer. This led to some confusing def-use chains, where it would look like the contents of a string literal were being written to by the side effect of an earlier function call, which of course is impossible.

To fix this, I've made two changes. First, each string literal is now given a corresponding `IRVariable` (specifically `IRStringLiteral`), since a string literal behaves more or less as a read-only global variable. Second, the `IRVariable` for each string literal is now marked `isReadOnly()`, which the alias analysis uses to determine that an arbitrary write to aliased memory will not overwrite the contents of a string literal.

I originally planned to treat all string literals with the same value as being the same memory location, since this is the usual behavior of modern compilers. However, this made implementing `IRVariable.getAST()` tricky for string literals, so I left them unpooled.
2019-11-06 13:08:28 -07:00
Robin Neatherway
7850d67a78 Remove TODO comment
I've checked Hamcrest versions 1.3, 2.0, 2.1 and 2.2
2019-11-06 17:47:02 +00:00
Geoffrey White
6c38f55e28 CPP: QLDoc protocols.qll. 2019-11-06 17:06:57 +00:00
semmle-qlci
717490b670 Merge pull request #2265 from tausbn/python-fix-unused-import-global-name-used-join-order
Approved by RasmusWL
2019-11-06 16:38:07 +00:00
Max Schaefer
54e40a8977 JavaScript: Move --html all extractor options into options file. 2019-11-06 16:30:01 +00:00
Geoffrey White
e886cf7297 CPP: 'i.e.' -> 'that is'. 2019-11-06 16:27:06 +00:00
Taus Brock-Nannestad
2b24eb2e70 Python: Fix bad join order for py/multiple-calls-to-init.
The `multiple_invocation_paths` predicate had a bad join order where
we (essentially) joined `i1` with `i2` and only then joined `i1` and `i2`
separately to reduce the number of tuples. The join coming from `i1 != i2` had
little impact, but `i1.getFunction() = multi` made a big difference (and
similarly for `i2`). I factored out the code so that these joins would be done
more eagerly. Thus, we went from

```
[2019-11-06 16:53:05] (38s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff/4@2ce75a
[2019-11-06 16:53:35] (68s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff:
                      134547    ~9%       {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<0>, I.<2>
                      235284431 ~3%       {4} r2 = JOIN r1 WITH CallGraph::TInvocation#fff AS R ON FIRST 1 OUTPUT r1.<0>, r1.<1>, R.<1>, R.<2>
                      235149884 ~3%       {4} r3 = SELECT r2 ON r2.<3> != r2.<1>
                      235149884 ~4%       {3} r4 = SCAN r3 OUTPUT r3.<1>, r3.<0>, r3.<3>
                      166753634 ~5%       {4} r5 = JOIN r4 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus#swapped AS R ON FIRST 1 OUTPUT R.<1>, r4.<2>, r4.<1>, r4.<0>
                      129778    ~0%       {4} r6 = JOIN r5 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus AS R ON FIRST 2 OUTPUT r5.<0>, r5.<3>, r5.<1>, r5.<2>
                                          return r6
[2019-11-06 16:53:35] (68s) Registering MethodCallOrder::multiple_invocation_paths#ffff + [] with content 1705dcbc08kd9aa40rp2g2e9civhv
[2019-11-06 16:53:35] (68s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff with 129778 rows and 4 columns.
```

to

```
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths_helper#ffff/4@586aec
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths_helper#ffff:
                      134547 ~0%     {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<2>, I.<0>
                      88111  ~4%     {3} r2 = JOIN r1 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus#swapped AS R ON FIRST 1 OUTPUT R.<1>, r1.<1>, r1.<0>
                      761305 ~0%     {4} r3 = JOIN r2 WITH #CallGraph::FunctionInvocation::getACallee_dispred#ffPlus AS R ON FIRST 1 OUTPUT r2.<1>, r2.<2>, r2.<0>, R.<1>
                      673194 ~0%     {4} r4 = SELECT r3 ON r3.<3> != r3.<1>
                      673194 ~0%     {4} r5 = SCAN r4 OUTPUT r4.<2>, r4.<1>, r4.<3>, r4.<0>
                                     return r5
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths_helper#ffff + [] with content 20edaaecf25nldgp24d9c4et8m3kv
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths_helper#ffff with 673194 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs/4@9e5441
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs:
                      673194 ~0%     {4} r1 = SCAN MethodCallOrder::multiple_invocation_paths_helper#ffff AS I OUTPUT I.<2>, I.<3>, I.<0>, I.<1>
                                     return r1
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs + [] with content 2069301e655fi9mcovngg9hetfqas
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs with 673194 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff/4@2f7c34
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff:
                      134547 ~0%     {2} r1 = SCAN CallGraph::TInvocation#fff AS I OUTPUT I.<2>, I.<0>
                      129778 ~0%     {4} r2 = JOIN r1 WITH MethodCallOrder::multiple_invocation_paths_helper#ffff_2301#join_rhs AS R ON FIRST 2 OUTPUT R.<2>, R.<3>, r1.<0>, r1.<1>
                                     return r2
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths#ffff + [] with content 1705dcbc08kd9aa40rp2g2e9civhv
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff with 129778 rows and 4 columns.
[2019-11-06 17:22:22] (25s) Starting to evaluate predicate MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs/4@9f9146
[2019-11-06 17:22:22] (25s) Tuple counts for MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs:
                      129778 ~0%     {4} r1 = SCAN MethodCallOrder::multiple_invocation_paths#ffff AS I OUTPUT I.<0>, I.<3>, I.<1>, I.<2>
                                     return r1
[2019-11-06 17:22:22] (25s) Registering MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs + [] with content 17c3fe1fcbf6ghhdr7hiukqp41rst
[2019-11-06 17:22:22] (25s)  >>> Wrote relation MethodCallOrder::multiple_invocation_paths#ffff_0312#join_rhs with 129778 rows and 4 columns.
```

Execution time on `salt` went from 29.5s to somewhere below 299ms (the predicate
was not listed in the timing report).
2019-11-06 17:27:03 +01:00
Geoffrey White
81c58d5a64 CPP: Improve QLDoc comments. 2019-11-06 16:20:39 +00:00
Geoffrey White
0c3f4e530f CPP: Make some library predicates private. 2019-11-06 16:07:28 +00:00
Geoffrey White
399ac1f112 CPP: Rename 'getAssertedFalseCondition' to something less misleading. 2019-11-06 15:57:44 +00:00
Taus Brock-Nannestad
43148083eb Python: Fix bad join order for global_name_used.
As it turns out, there was a further bad join-order in the `global_name_used`
predicate. In this case, there was a common subexpression in the RA that was
being factored out and evaluated separately, producing a large number of tuples.
2019-11-06 16:37:01 +01:00
Nick Rolfe
5b00b21713 Merge pull request #2153 from matt-gretton-dann/cpp-447-support-non-type-template-parameters
RFC: C++ Support non type template parameter values
2019-11-06 15:11:34 +00:00
Rasmus Wriedt Larsen
47a292b241 Merge pull request #2264 from tausbn/python-fix-unused-import-join-order
Python: Fix bad join order in `py/unused-import`
2019-11-06 16:09:44 +01:00
Jonas Jensen
8ffd7c1055 Merge pull request #2222 from geoffw0/libraryperf
CPP: Improvements for ConditionallyInitializedVariable.ql
2019-11-06 15:54:16 +01:00
Jonas Jensen
217ecd3551 C++: Add <p> tags to split text into paragraphs
Without this, the rendered output is one big paragraph.
2019-11-06 15:50:08 +01:00
shati-patel
5d1d7ea075 Merge pull request #2257 from jf205/codeql-training-material
docs: update terminology in training material
2019-11-06 14:43:55 +00:00
Asger F
d9beb54dde Merge pull request #2102 from erik-krogh/deferredModel
JS: add Deferred model in js/use-of-returnless-function
2019-11-06 14:30:03 +00:00
Jonas Jensen
76a3db9eed Merge remote-tracking branch 'upstream/master' into ir-copy-unloaded-result 2019-11-06 15:21:22 +01:00
Taus Brock-Nannestad
b6f16dee81 Python: Fix bad join order in py/unused-import 2019-11-06 15:14:48 +01:00
semmle-qlci
f73caac88d Merge pull request #2254 from asger-semmle/for-of-propread
Approved by max-schaefer
2019-11-06 13:44:55 +00:00
Geoffrey White
f9feb05a72 CPP: Add a test of NtohlArrayNoBoundOpenSource.ql. 2019-11-06 13:36:31 +00:00
Jonas Jensen
aa841c306d C++: Use virtual dispatch in DefaultTaintTracking
This bit is only used by the compatibility code that sends flow into
parameters of functions without body.
2019-11-06 14:04:07 +01:00
Jonas Jensen
ec9ef33486 C++: IR data flow through inheritance conversions
This makes IR data flow behave more like AST data flow, and it makes IR
virtual dispatch work without further changes.
2019-11-06 14:04:07 +01:00