Commit Graph

34844 Commits

Author SHA1 Message Date
Taus
7d736952db Python: Update expected output 2022-04-26 15:49:40 +00:00
Taus
d4fc096ea8 Python: Use local flow in Pythagorean.ql
The hand-rolled notion of flow was causing some severe performance
issues (on a few databases):

```
Tuple counts for Pythagorean::square#168e234a#f#loop_invariant_prefix/2@c86989kr after 6m35s:
175000     ~5%     {2} r1 = JOIN SSA::SsaVariable::getDefinition#dispred#f0820431#ff_10#join_rhs WITH Flow::ControlFlowNode::getNode#dispred#f0820431#bf ON FIRST 1 OUTPUT Lhs.1, Rhs.1 'arg0'
174500     ~6%     {2} r2 = JOIN r1 WITH SSA::SsaVariable::getVariable#dispred#f0820431#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'arg0'
1467782500 ~5%     {3} r3 = JOIN r2 WITH AstGenerated::Name_::getVariable#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT 3, Rhs.1 'arg1', Lhs.1 'arg0'
1467553000 ~0%     {2} r4 = JOIN r3 WITH py_expr_contexts_12#join_rhs ON FIRST 2 OUTPUT Lhs.2 'arg0', Lhs.1 'arg1'
                    return r4
```

Rewriting it to use the data flow library made all of this go away. 🎉
2022-04-25 14:35:37 +00:00
Taus
b2cc91369a Python: Fix bad join in firstUse
This was what it looked like (at the point when I killed the evaluation):

```
Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#c5fa2be7#ff/2@i1#be98bwif after 1m50s:
274000     ~7%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#c5fa2be7#ffff OUTPUT In.1, In.0 'def', In.2, In.3
2731768000 ~1%     {7} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::variableSourceUse#c5fa2be7#ffff ON FIRST 1 OUTPUT Rhs.0, Lhs.2, Lhs.3, Rhs.2, Rhs.3, Rhs.1 'use', Lhs.1 'def'
178000     ~4%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#c5fa2be7#fffff ON FIRST 5 OUTPUT Lhs.6 'def', Lhs.5 'use'
                    return r3
```

And this is what it looks like now:

```
Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#c5fa2be7#ff/2@i1#f9d6ewsi after 207ms:
931353  ~2%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::AdjacentUsesImpl::variableSourceUse#c5fa2be7#ffff OUTPUT In.0, In.2, In.3, In.1 'use'
1050477 ~0%     {4} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#c5fa2be7#fffff_03412#join_rhs ON FIRST 3 OUTPUT Lhs.0, Rhs.3, Rhs.4, Lhs.3 'use'
506626  ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#c5fa2be7#ffff_1230#join_rhs ON FIRST 3 OUTPUT Rhs.3 'def', Lhs.3 'use'
                return r3
```
2022-04-25 14:33:31 +00:00
Taus
49233268a9 Python: Fix bad join in getValue
We were building essentially a CP of all control flow nodes:

```
Tuple counts for Essa::AssignmentDefinition::getValue#dispred#f0820431#ff/2@dd1f67vl after 2m45s:
733365     ~6%     {3} r1 = JOIN Essa::TEssaNodeDefinition#24e22a14#ffff_30#join_rhs WITH Essa::EssaNodeDefinition::getDefiningNode#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Rhs.0
376588     ~0%     {2} r2 = JOIN r1 WITH SsaDefinitions::SsaSource::assignment_definition#9197156e#fff ON FIRST 2 OUTPUT Lhs.2 'this', Rhs.2 'result'
376588     ~0%     {3} r3 = JOIN r2 WITH Essa::TEssaNodeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'this', Lhs.1 'result'
6965593033 ~2%     {3} r4 = JOIN r3 WITH project#SsaDefinitions::SsaSource::assignment_definition#9197156e ON FIRST 1 OUTPUT Lhs.1 'this', Rhs.1, Lhs.2 'result'
376588     ~0%     {2} r5 = JOIN r4 WITH Essa::EssaNodeDefinition::getDefiningNode#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.0 'this', Lhs.2 'result'
                    return r5
```

We first tried preventing the join on `result`, but this caused the
characteristic predicate to blow up instead. Finally, we figured just
putting the `value` part in a field would be sufficient, and this did
the trick.
2022-04-25 14:28:00 +00:00
Anders Schack-Mulligen
60eb341b49 Merge pull request #8851 from aschackmull/shared/accesspathsyntax-cleanup
Minor clean-up in AccessPathSyntax.qll
2022-04-25 13:47:22 +02:00
Mathias Vorreiter Pedersen
faaa2cedcd Merge pull request #8849 from JohnMcSandwich/main
C++: add new Windows pool allocation functions in `Allocation.qll`
2022-04-25 12:46:31 +01:00
Anders Schack-Mulligen
40a16325a9 Minor clean-up in AccessPathSyntax. 2022-04-25 12:27:48 +02:00
Anders Schack-Mulligen
cbdd4927ce Merge pull request #8582 from Marcono1234/marcono1234/JumpStmt-superclass
Java: Make `JumpStmt` a proper superclass
2022-04-25 12:22:20 +02:00
Anders Schack-Mulligen
fd2904d49c Merge pull request #8760 from Marcono1234/patch-1
Clarify `min`, `max` and `rank` documentation
2022-04-25 12:20:00 +02:00
Tom Hvitved
bffa8fa7cb Merge pull request #8641 from hvitved/dataflow/interpret-read-store
Data flow: Introduce `ContentSet`
2022-04-25 12:17:34 +02:00
Mathias Vorreiter Pedersen
516ef8d27a Update cpp/ql/lib/change-notes/2022-04-25-windows-pool-allocation-functions.md
Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>
2022-04-25 10:54:12 +01:00
Mathias Vorreiter Pedersen
12c8d9c60e C++: Add change note. 2022-04-25 10:51:49 +01:00
Anders Schack-Mulligen
b21f077e8e Update java/ql/lib/semmle/code/java/Statement.qll 2022-04-25 11:02:23 +02:00
Jeroen Ketema
ba2a884a45 Merge pull request #8818 from jketema/links
Replace `help.semmle.com` links by `codeql.github.com` links
2022-04-25 10:25:42 +02:00
JohnMcSandwich
b3dff77d1a C++: add new Windows pool allocation functions in Allocation.qll
Add:
 - ExAllocatePool2
 - ExAllocatePool3
 - ExAllocatePoolZero
2022-04-25 10:21:42 +02:00
Mathias Vorreiter Pedersen
e8b6bfbe0e Merge pull request #8813 from jketema/buffer
C++: Cover variable sized member arrays without a size in `Buffer.qll`
2022-04-25 09:20:31 +01:00
Tom Hvitved
2466288656 Data flow: Simplify revFlowStore 2022-04-25 10:11:54 +02:00
Tom Hvitved
cf0a1e748a Add change notes 2022-04-25 09:17:40 +02:00
Jeroen Ketema
79164056d1 Replace help.semmle.com links by codeql.github.com links 2022-04-22 20:42:11 +02:00
Jeroen Ketema
a4711206c8 Merge pull request #8769 from felickz/patch-1
Docs - Supported Queries - Fixing broken link
2022-04-22 18:29:27 +02:00
Mathias Vorreiter Pedersen
1e61fdde8b Merge pull request #8815 from jketema/unreachable-test-case
C++: Add non-returning function test case using `__builtin_expect`
2022-04-22 16:47:44 +01:00
Jeroen Ketema
97d4a12fb2 C++: Add non-returning function test case using __builtin_expect 2022-04-22 17:10:54 +02:00
Jeroen Ketema
dee0f09197 C++: Cover variable sized member arrays without a size in Buffer.qll
Currently the extractor incorrectly emits 0 for the array `data` below:
```
struct myStruct { // c
   ...
   char data[]; // v
};
```
This will change in the future, and  no size will be emitted anymore.
This commit makes sure `Buffer.qll` handles arrays without sizes.
2022-04-22 16:57:24 +02:00
Robert Marsh
ebdf553621 Merge pull request #8806 from dbartol/dbartol/typedefs/work
C++: Work around missing size for typedef
2022-04-22 10:09:24 -04:00
Rasmus Wriedt Larsen
3e8274ede8 Merge pull request #8812 from RasmusWL/stdlib-FileSystemAccess-improvement
Python: Minor Stdlib file system access improvement
2022-04-22 16:06:41 +02:00
Tom Hvitved
3edc72feb6 Update CaptureModels.qll 2022-04-22 15:49:17 +02:00
Tom Hvitved
bc6ee10583 Data flow: Sync files 2022-04-22 15:10:00 +02:00
Tom Hvitved
488a4ede94 Data flow: Inline getAStoreContent up-front 2022-04-22 15:09:59 +02:00
CodeQL CI
06e5962da7 Merge pull request #8791 from asgerf/js/static-accessors
Approved by erik-krogh
2022-04-22 13:39:32 +01:00
Tom Hvitved
b033f107df Merge remote-tracking branch 'upstream/main' into dataflow/interpret-read-store 2022-04-22 14:35:02 +02:00
Rasmus Wriedt Larsen
03c0366fd4 Merge branch 'main' into stdlib-FileSystemAccess-improvement 2022-04-22 14:31:31 +02:00
Erik Krogh Kristensen
789b0a46d1 Merge pull request #8578 from erik-krogh/labelNaming
JS: update `toString()` on API-graph labels.
2022-04-22 14:27:25 +02:00
Erik Krogh Kristensen
8fcbaea273 Merge branch 'main' into labelNaming 2022-04-22 13:19:44 +02:00
Erik Krogh Kristensen
c015ef6ef4 Merge pull request #8810 from erik-krogh/rubyPathgraph
Ruby: dont import the PathGraph module from Query.qll files
2022-04-22 12:02:59 +02:00
Mathias Vorreiter Pedersen
489355cdab Merge pull request #8793 from MathiasVP/exclude-internal-diagnostics
Exclude internal diagnostics from all selectors
2022-04-22 10:55:38 +01:00
Tom Hvitved
093a3879be Merge pull request #8794 from hvitved/ruby/capture-barrier-guards
Ruby: Handle captured variables in `BarrierGuard::getAGuardedNode()`
2022-04-22 11:47:36 +02:00
Erik Krogh Kristensen
a737350f27 RB: dont import the PathGraph module from Query.qll files 2022-04-22 11:46:06 +02:00
Anders Schack-Mulligen
bf921177f4 Merge pull request #8811 from erik-krogh/syncLate
Java: get tainttracking3/TaintTrackingImpl.qll in sync
2022-04-22 11:21:01 +02:00
Asger F
0187e9a3b1 Merge pull request #8808 from vovikhangcdv/doublevkay/fixing-PrototypePollutngAssignment-examples
Javascript: Fix PrototypePollutingAssignment example which is incorrect use of express leads to no result when scanning.
2022-04-22 11:18:23 +02:00
Tom Hvitved
be5363ea53 Merge pull request #8801 from hvitved/ruby/exclude-splat-in-taint-tracking
Ruby: Exclude `SplatExpr` from taint tracking
2022-04-22 11:12:05 +02:00
Rasmus Wriedt Larsen
650d57083b Python: Recognize path arguments to pathlib methods 2022-04-22 11:01:59 +02:00
Rasmus Wriedt Larsen
bcaba45202 Python: Expand pathlib tests 2022-04-22 11:01:59 +02:00
Rasmus Wriedt Larsen
059dea713d Python: Fix os.path.samefile modeling 2022-04-22 11:01:59 +02:00
Erik Krogh Kristensen
dca74a1f45 get tainttracking3/TaintTrackingImpl.qll in sync 2022-04-22 10:37:31 +02:00
Chris Smowton
d309e15072 Merge pull request #8748 from smowton/smowton/admin/dependent-dataflow-configs
Java: Avoid higher-numbered dataflow configs that depend on lower-numbered ones
2022-04-22 08:56:00 +01:00
Mathias Vorreiter Pedersen
35471ff23c Merge pull request #8809 from AlexDenisov/alexdenisov/rename-swift-db-columns
Swift: rename certain dbscheme columns
2022-04-22 08:17:37 +01:00
Mathias Vorreiter Pedersen
52dc016a7a Merge pull request #8798 from jketema/using
C++: Fix tests after extractor changes that improve `using` position accuracy
2022-04-22 08:15:34 +01:00
Alex Denisov
e85cdf2ec3 Swift: rename certain dbscheme columns 2022-04-22 08:31:09 +02:00
AlexDenisov
a5189eae9f Merge pull request #8735 from redsun82/swift-dbscheme-gen
Swift: dbscheme generator
2022-04-22 08:26:59 +02:00
Khang. Võ Vĩ
f4581ae866 fix PrototypePollutingAssignment examples 2022-04-22 11:55:45 +07:00