Commit Graph

17907 Commits

Author SHA1 Message Date
Tamas Vajk
dee1690748 C#: Remove duplicated pattern extraction logic 2020-11-12 14:34:54 +01:00
Tamas Vajk
3f150bb09b C#: Reorganize patterns 2020-11-12 13:47:38 +01:00
James Fletcher
19b974832b Merge pull request #4658 from github/main-126-merge
Merge rc/1.26 into main branch
2020-11-12 10:18:24 +00:00
james
9fc84f8061 Merge branch 'rc/1.26' into main-126-merge 2020-11-12 09:55:32 +00:00
Tamás Vajk
b5ef3bd5ec Merge pull request #4516 from tamasvajk/feature/attributes
C#: Refactor attribute extraction
2020-11-12 09:57:46 +01:00
yo-h
7c3ea0e264 Merge pull request #4602 from aschackmull/java/changenote-3812
Java: Add change note for #3812.
2020-11-11 15:14:58 -05:00
yoff
86fc9e62ef Merge pull request #4650 from RasmusWL/python-set-literal-formatting
Python: Update set literal formatting
2020-11-11 15:35:12 +01:00
CodeQL CI
f9d62adcb2 Merge pull request #4567 from asgerf/js/date-functions
Approved by erik-krogh
2020-11-11 11:17:30 +00:00
Rasmus Wriedt Larsen
611398586d Merge pull request #4649 from yoff/python-dataflow-cfgparameters
Python: Make `ParameterNode` a `CfgNode`
2020-11-11 10:22:12 +01:00
Tamas Vajk
453c97a8c3 Fix expected files of failing tests 2020-11-11 09:57:52 +01:00
Tom Hvitved
e57134e8a4 C#: Address comments 2020-11-11 09:54:51 +01:00
Tom Hvitved
d6515e7963 C#: Changes to extraction of attributes in assemblies
- Only cache on `AttributeData` and not the parent entity.
- Move `CreateGeneratedExpressionFromArgument` to `Expression.cs`.
- Restructure the various `CreateGenerated` methods so child entities are
  created inside them (and therefore no need to expose child index logic).
- Add locations to generated expressions.
- Avoid linear lookup in `CompilationUnit.cs`.
- Consolidate tests.
2020-11-11 09:54:51 +01:00
Tom Hvitved
b8d6070e00 C#: Remove redundant conjunct 2020-11-11 09:54:51 +01:00
Tom Hvitved
3048919fce C#: Introduce AssertionFailure class 2020-11-11 09:54:51 +01:00
Tom Hvitved
91dd67798a C#: Exclude assembly attribute expressions from CFG construction 2020-11-11 09:50:59 +01:00
Tamas Vajk
4b88b237b3 C#: Fix failing tests 2020-11-11 09:50:59 +01:00
Tamas Vajk
4d6ccb1ef4 C#: Add change note for attribute extraction refactoring 2020-11-11 09:46:18 +01:00
Tamas Vajk
ba730b6f02 Fix failing tests 2020-11-11 09:46:18 +01:00
Tamas Vajk
5f18840a9e C#: Extract attributes of referenced assembly 2020-11-11 09:46:18 +01:00
Tamas Vajk
cdea3afced C#: Change extraction of Attributes to have ID based on the location when location is from source 2020-11-11 09:46:18 +01:00
Tamas Vajk
c04d6f479c C#: Change attribute extraction to use AttributeData and created generated expressions when missing 2020-11-11 09:46:18 +01:00
Tamas Vajk
883795e126 C#: Add attribute test for extraction from symbols without syntax nodes 2020-11-11 09:46:18 +01:00
Tamas Vajk
c14a63488c C#: Add missing tests for attribute extraction 2020-11-11 09:46:18 +01:00
Rasmus Wriedt Larsen
9ed15732ed Python: Update set literal formatting
Now that auto-formatting rules have been updated
2020-11-11 09:38:25 +01:00
Rasmus Lerchedahl Petersen
0710963fc3 Python: update test expectations
EssaNode -> ControlFlowNode
2020-11-10 23:58:55 +01:00
Jonas Jensen
fc764db8e1 Merge pull request #4643 from nickrolfe/getFileBySourceArchiveName
Replace getEncodedFile with shared getFileBySourceArchiveName predicate
2020-11-10 17:36:29 +01:00
Mathias Vorreiter Pedersen
884554692f Merge pull request #4645 from geoffw0/oddsends2
C++: Odds and ends
2020-11-10 16:59:10 +01:00
Mathias Vorreiter Pedersen
618d9c33a1 Merge pull request #4642 from geoffw0/memberfn
C++: Improve MemberFunction.qll data flows
2020-11-10 16:54:42 +01:00
Jonas Jensen
da90717ef0 Merge pull request #4641 from aschackmull/autoformat-setliterals
C++/C#/Java/JavaScript/Python: Autoformat set literals.
2020-11-10 16:11:11 +01:00
Tom Hvitved
7e89e57f5d Merge pull request #4606 from hvitved/csharp/dataflow/ef
C#: Precise data flow for EntityFramework(Core)
2020-11-10 15:54:20 +01:00
Nick Rolfe
ac4a1f1d9b Update comment to be a QLDoc comment 2020-11-10 14:14:27 +00:00
Geoffrey White
be537a8021 C++: Move ThrowExpr, ReThrowExpr definitions to a more logical location. 2020-11-10 14:06:58 +00:00
Nick Rolfe
1e1eb7ee33 Replace getEncodedFile with shared getFileBySourceArchiveName predicate
While also making it work with paths for databases created on Windows.
2020-11-10 13:55:27 +00:00
Geoffrey White
2beb7824ec C++: Backticks. 2020-11-10 13:35:05 +00:00
Geoffrey White
599034668f C++: Fix typo. 2020-11-10 13:32:31 +00:00
Geoffrey White
f109e8ccbc C++: Remove duplicate import. 2020-11-10 13:31:47 +00:00
Geoffrey White
c4fe8df04c C++: Support data flow from parameter deref to return value. 2020-11-10 13:17:34 +00:00
Geoffrey White
4563e32e92 C++: Remove data flows we shouldn't need in MemberFunction. 2020-11-10 13:17:24 +00:00
Anders Schack-Mulligen
89ef6ea4eb C++/C#/Java/JavaScript/Python: Autoformat set literals. 2020-11-10 13:32:27 +01:00
Rasmus Lerchedahl Petersen
109d55eb25 Python: Make ParameterNode a CfgNode
Add a step from that `CfgNode` to the corresponding `EssaNode`.
The intended effect is seen in `ImpliesDataflow.expected`.
The efeect seen in other `.expected`-files is that parameter nodes
change type, that the extra steps are seen, and that flow from
`EssaVar`s is mirrored in flow from `CfgNode`s.
There is one surprise, which is the `.0` node in
`coverage/localFlow.expected`.
2020-11-10 11:35:50 +01:00
Tom Hvitved
697b0ba4c1 C#: Address review comments 2020-11-10 10:42:05 +01:00
yoff
26286e534e Merge pull request #4174 from yoff/SharedDataflow_PointsToImpliesDataflow
Python: Dataflow, Test that `pointsTo` implies data flow
merging now, will fix `self` in a later PR
2020-11-10 10:25:29 +01:00
CodeQL CI
1e048d8045 Merge pull request #4609 from asgerf/js/destructuring-export
Approved by erik-krogh
2020-11-09 15:47:00 +00:00
Anders Schack-Mulligen
31ec79819e Merge pull request #4631 from luchua-bc/java-nfe-library
Java: Factor NumberFormatException out into a library file
2020-11-09 13:50:31 +01:00
yoff
3215f50b73 Merge pull request #4632 from RasmusWL/python-move-configurations-out-of-queries
Python: move configurations out of queries
2020-11-09 13:10:33 +01:00
luchua-bc
d765c7bbb2 Update qldoc 2020-11-09 11:23:48 +00:00
Asger Feldthaus
32c5bc4836 JS: Update test case output (new flow is correct) 2020-11-09 11:19:47 +00:00
Mathias Vorreiter Pedersen
25ba6ca160 Merge pull request #4637 from jbj/temp-objects-perf-fixup
C++: pragma[noinline] parameter index predicates
2020-11-09 10:12:50 +01:00
Rasmus Wriedt Larsen
fbe51c51bb Python: Add missing QLDoc 2020-11-09 09:05:08 +01:00
Jonas Jensen
b8b1f0c617 C++: pragma[noinline] parameter index predicates
A performance regression in `definitionByReferenceNodeFromArgument#ff`
was ultimately caused by a join on parameter indexes in
`DefinitionByReferenceNode.getArgument`. Joining on numbers in QL is
always fragile, and somehow the changes in #4432 had caused the join
order here to break.

Instead of tweaking the join order in the slow predicate itself, I added
`pragma[noinline]` to one of the predicates involved in the join on
parameter indexes. This should prevent us from getting similar
performance problems in the future when we write code that joins on
parameter numbers. Joining on indexes is always risky, but it's even
more risky when one of the predicates in the join is inlined by the
compiler and expands to further joins.

I tested performance by running `CgiXss.ql` on a ChakraCore snapshot.
Tuple counts before (I interrupted execution after five minutes or so):

    (626s) Tuple counts for DataFlowUtil::definitionByReferenceNodeFromArgument#ff:
    58162      ~0%     {3} r1 = SCAN DataFlowUtil::DefinitionByReferenceNode#class#ff AS I OUTPUT I.<1>, -1, I.<0>
    26934      ~0%     {2} r2 = JOIN r1 WITH Instruction::IndexedInstruction#ff AS R ON FIRST 2 OUTPUT r1.<0>, r1.<2>
    26934      ~1%     {2} r3 = JOIN r2 WITH Instruction::SideEffectInstruction::getPrimaryInstruction_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r2.<1>
    26850      ~1%     {2} r4 = JOIN r3 WITH Instruction::CallInstruction::getThisArgumentOperand_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r3.<1>
    26850      ~0%     {2} r5 = JOIN r4 WITH Operand::Operand::getDef_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r4.<1>
    26850      ~1%     {2} r6 = JOIN r5 WITH Instruction::Instruction::getUnconvertedResultExpression_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r5.<1>
    58162      ~0%     {2} r7 = SCAN DataFlowUtil::DefinitionByReferenceNode#class#ff AS I OUTPUT I.<1>, I.<0>
    58162      ~4%     {3} r8 = JOIN r7 WITH Instruction::IndexedInstruction#ff AS R ON FIRST 1 OUTPUT R.<1>, r7.<1>, r7.<0>
    4026581120 ~0%     {4} r9 = JOIN r8 WITH Instruction::CallInstruction::getPositionalArgumentOperand_dispred#fff_102#join_rhs AS R ON FIRST 1 OUTPUT r8.<2>, R.<1>, r8.<1>, R.<2>
    31154      ~4%     {2} r10 = JOIN r9 WITH Instruction::SideEffectInstruction::getPrimaryInstruction_dispred#3#ff AS R ON FIRST 2 OUTPUT r9.<3>, r9.<2>
    31154      ~8%     {2} r11 = JOIN r10 WITH Operand::Operand::getDef_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r10.<1>
    31154      ~0%     {2} r12 = JOIN r11 WITH Instruction::Instruction::getUnconvertedResultExpression_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r11.<1>
    58004      ~0%     {2} r13 = r6 \/ r12
                       return r13

Tuple counts after:

    (0s) Tuple counts for DataFlowUtil::definitionByReferenceNodeFromArgument#ff:
    385785  ~6%     {2} r1 = SCAN DataFlowUtil::DefinitionByReferenceNode#class#ff AS I OUTPUT I.<1>, I.<0>
    385785  ~0%     {3} r2 = JOIN r1 WITH Instruction::IndexedInstruction#ff AS R ON FIRST 1 OUTPUT r1.<0>, r1.<1>, R.<1>
    385785  ~1%     {3} r3 = JOIN r2 WITH Instruction::SideEffectInstruction::getPrimaryInstruction_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r2.<2>, r2.<1>
    198736  ~4%     {2} r4 = JOIN r3 WITH Instruction::CallInstruction::getPositionalArgument#fff AS R ON FIRST 2 OUTPUT R.<2>, r3.<2>
    198736  ~0%     {2} r5 = JOIN r4 WITH Instruction::Instruction::getUnconvertedResultExpression_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r4.<1>
    385785  ~1%     {3} r6 = SCAN DataFlowUtil::DefinitionByReferenceNode#class#ff AS I OUTPUT I.<1>, -1, I.<0>
    186891  ~1%     {2} r7 = JOIN r6 WITH Instruction::IndexedInstruction#ff AS R ON FIRST 2 OUTPUT r6.<0>, r6.<2>
    186891  ~2%     {2} r8 = JOIN r7 WITH Instruction::SideEffectInstruction::getPrimaryInstruction_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r7.<1>
    183201  ~3%     {2} r9 = JOIN r8 WITH Instruction::CallInstruction::getThisArgumentOperand_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r8.<1>
    183201  ~0%     {2} r10 = JOIN r9 WITH Operand::Operand::getDef_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r9.<1>
    175449  ~8%     {2} r11 = JOIN r10 WITH Instruction::Instruction::getUnconvertedResultExpression_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r10.<1>
    374185  ~3%     {2} r12 = r5 \/ r11
                    return r12
2020-11-09 09:01:22 +01:00