Commit Graph

33347 Commits

Author SHA1 Message Date
Henry Mercer
7f61738a23 Use US English spelling 2022-01-12 13:07:09 +00:00
Henry Mercer
6e37a65e84 Remove CodeToFeatures AST library 2022-01-12 12:47:28 +00:00
Henry Mercer
957e34d8a7 Make function body features library independent of CodeToFeatures AST 2022-01-12 12:47:28 +00:00
Henry Mercer
9e50ce873d Move function body features into their own file 2022-01-12 12:47:28 +00:00
Henry Mercer
865fb5d0ef Migrate representative entity -> representative function 2022-01-12 12:47:27 +00:00
Henry Mercer
0e5b493d0e Remove CodeToFeatures AST consistency checks
We no longer use the `CodeToFeatures` AST, therefore these checks are
defunct.
2022-01-12 12:47:27 +00:00
Henry Mercer
387829bbb4 Extract body tokens from the JS AST, not the CodeToFeatures AST 2022-01-12 12:47:25 +00:00
Henry Mercer
3ef69763a7 Merge pull request #7567 from github/henrymercer/atm-body-tokens-perf-opt
ATM: Optimize body tokens by pushing in size restriction
2022-01-12 12:45:27 +00:00
Tamás Vajk
9065a7f320 Merge pull request #7573 from tamasvajk/fix/java-field-decl-tostr
Java: Fix toString on field declarations with single field
2022-01-12 13:03:16 +01:00
Tony Torralba
8a80e02861 Merge pull request #7574 from pwntester/improve_strings_qll
Add models for AbstractStringBuilder.substring,subsequence,getChars
2022-01-12 12:01:28 +01:00
Tony Torralba
c2105e506b Added test cases 2022-01-12 11:06:58 +01:00
Alvaro Muñoz Sanchez
715d372572 Add models for AbstractStringBuilder.substring,subsequence,getChars 2022-01-12 10:54:27 +01:00
Anders Schack-Mulligen
c6a9b2b6ff Merge pull request #7572 from github/workflow/coverage/update
Update CSV framework coverage reports
2022-01-12 09:39:14 +01:00
Mathias Vorreiter Pedersen
9e51908b02 Merge pull request #7551 from MathiasVP/fix-join-orders-in-unsigned-difference-expr-query
C++: Fix join orders in `cpp/unsigned-difference-expression-compared-zero`
codeql-cli/v2.7.5
2022-01-12 08:29:03 +00:00
Tamas Vajk
b9e0310aa2 Java: Fix toString on field declarations with single field 2022-01-12 09:22:16 +01:00
Michael Nebel
f17c110f51 Merge pull request #7562 from michaelnebel/csharp/record-seal-tostring
C#: Record types are allowed to seal ToString (test only).
2022-01-12 08:08:32 +01:00
luchua-bc
263dbd33f6 Optimize the query 2022-01-12 02:33:17 +00:00
github-actions[bot]
c79e8ab440 Add changed framework coverage reports 2022-01-12 00:10:48 +00:00
Andrew Eisenberg
e4eb2c2a59 Update docs on the output of resolve qlpacks
The output has changed and there are no more upgrades
packs. There are also other changes included here.
2022-01-11 15:54:53 -08:00
Andrew Eisenberg
da4f1d86aa Merge pull request #7355 from github/aeisenberg/remove-upgrades
Move upgrades into standard library packs
2022-01-11 14:09:10 -08:00
Andrew Eisenberg
07228672df Merge branch 'main' into aeisenberg/remove-upgrades 2022-01-11 11:25:27 -08:00
Mathias Vorreiter Pedersen
c45127fdd6 Merge pull request #7541 from github/rdmarsh2/dataflow-ipa-params
C++: Use an IPA type rather than negative indexes for argument/parameter matching in data flow
2022-01-11 16:52:13 +00:00
Tony Torralba
7b0d9ea525 Merge pull request #7054 from atorralba/atorralba/promote-log-injection
Java: Promote Log Injection from experimental
2022-01-11 17:26:18 +01:00
Henry Mercer
3f70476c87 ATM: Optimize body tokens by pushing in size limit
Pushing the restriction to 256 tokens into the `bodyTokens` predicate
means we avoid this predicate blowing up due to very large functions.

This results in a runtime improvement from 1800s+ to 294s as measured
on a problematic repo on my machine (I didn't wait for the query to
finish running).
2022-01-11 16:16:54 +00:00
Tony Torralba
1030ff7063 Update java/ql/src/Security/CWE/CWE-117/LogInjection.ql 2022-01-11 16:25:32 +01:00
Tony Torralba
4aacba8594 Merge pull request #6468 from atorralba/atorralba/promote-cleartext-sharedprefs
Java: Promote Cleartext storage of sensitive information using SharedPreferences from experimental
2022-01-11 16:23:53 +01:00
Benjamin Muskalla
426f3117d6 Clarify model names and escape variables 2022-01-11 15:58:21 +01:00
Tony Torralba
394c4a9ee0 Remove unused code 2022-01-11 14:50:48 +01:00
Mathias Vorreiter Pedersen
b3a7090068 C++: Fix join in reachesRefParameter by joining with 'getEnd' instead
of 'getANode'.

Before:

Tuple counts for FlowVar::FlowVar::reachesRefParameter_dispred#ff/2@956ac39i after 229ms:
  24806   ~1%     {2} r1 = JOIN FlowVar::FlowVar_internal::parameterIsNonConstReference#f WITH Parameter::Parameter::getFunction_dispred#ff ON FIRST 1 OUTPUT Lhs.0 'p', Rhs.1
  56985   ~3%     {3} r2 = JOIN r1 WITH num#FlowVar::FlowVar_internal::TBlockVar#fff_12#join_rhs ON FIRST 1 OUTPUT Rhs.1 'this', Lhs.0 'p', Lhs.1
  2384489 ~4%     {4} r3 = JOIN r2 WITH FlowVar::FlowVar_internal::getAReachedBlockVarSBB#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.2, Lhs.1 'p', Lhs.0 'this'
  49457   ~0%     {2} r4 = JOIN r3 WITH SubBasicBlocks::SubBasicBlock::getANode_dispred#fb ON FIRST 2 OUTPUT Lhs.3 'this', Lhs.2 'p'
                  return r4

After:

Tuple counts for FlowVar::FlowVar::reachesRefParameter_dispred#ff/2@46f8bfn7 after 32ms:
  24806 ~1%     {2} r1 = JOIN FlowVar::FlowVar_internal::parameterIsNonConstReference#f WITH Parameter::Parameter::getFunction_dispred#ff ON FIRST 1 OUTPUT Lhs.0 'p', Rhs.1
  56985 ~1%     {3} r2 = JOIN r1 WITH num#FlowVar::FlowVar_internal::TBlockVar#fff_12#join_rhs ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'p', Rhs.1 'this'
  56985 ~1%     {3} r3 = JOIN r2 WITH SubBasicBlocks::SubBasicBlock::getEnd_dispred#fb_10#join_rhs ON FIRST 1 OUTPUT Lhs.2 'this', Rhs.1, Lhs.1 'p'
  49457 ~0%     {2} r4 = JOIN r3 WITH FlowVar::FlowVar_internal::getAReachedBlockVarSBB#ff ON FIRST 2 OUTPUT Lhs.0 'this', Lhs.2 'p'
                return r4
2022-01-11 13:48:20 +00:00
Michael Nebel
77763d7ee5 Merge pull request #7559 from michaelnebel/csharp/const-interpolatedstring
C#: Constant string interpolation (test only).
2022-01-11 14:01:55 +01:00
Michael Nebel
56bc3db46a C#: Add test case for sealed ToString modifier on a record type. 2022-01-11 13:58:43 +01:00
Michael Nebel
ae5d3a1ccb C#: Add example of sealing ToString on a record type. 2022-01-11 13:57:29 +01:00
Anders Schack-Mulligen
fdb4851521 Java: A few perf fixes for getASupertype*(). 2022-01-11 13:33:54 +01:00
Tony Torralba
50caf7d8dc Move change note to new location and remove import
Co-authored-by: Anders Schack-Mulligen <aschackmull@users.noreply.github.com>
2022-01-11 12:24:44 +01:00
Tony Torralba
b9e32208ee Move change note to new location 2022-01-11 12:23:16 +01:00
Michael Nebel
1d8f8f79bb C#: Add const interpolated string test case. 2022-01-11 12:02:07 +01:00
Michael Nebel
5b89f0e0b8 C#: Add example of const interpolated string. 2022-01-11 12:01:40 +01:00
Benjamin Muskalla
49d2fbfb5f Fixed slug references and PR skips 2022-01-11 11:47:28 +01:00
Stephan Brandauer
132e0bf4b7 add database accesses as additional (heuristic) remote flow sources 2022-01-11 11:38:41 +01:00
Sebastian Bauersfeld
e2a9ced691 Java: Pass taint through Apache's StringEscapeUtils.escapeJson() method. 2022-01-11 15:49:44 +07:00
Sebastian Bauersfeld
f36ee95128 Java: Pass taint through Spring's AbstractMessageSource.getMessage() methods. 2022-01-11 15:48:29 +07:00
Anders Schack-Mulligen
2a36744deb Merge pull request #7552 from smowton/smowton/fix/local-parameterized-classes
Note that parameterizations of local classes are themselves local
2022-01-11 09:36:15 +01:00
Alex Ford
b9ed8ed416 Merge pull request #7553 from github/revert-7498-dependabot/cargo/ruby/generator/clap-3.0
Ruby: Revert "Update clap requirement from 2.33 to 3.0 in /ruby/generator"
2022-01-10 19:36:40 +00:00
Alex Ford
17e5b9cffa Revert "Update clap requirement from 2.33 to 3.0 in /ruby/generator" 2022-01-10 18:21:04 +00:00
Chris Smowton
e352a4b994 Note that parameterizations of local classes are themselves local
Previously `LocalClass` itself would match `.isLocal()` whereas `LocalClass<Param>` would not. Rather than require each individual user to check for `.getSourceDeclaration().isLocal()`, let's note that the specializations themselves are local.
2022-01-10 18:19:31 +00:00
Robert Marsh
fe355a0bc9 C++: update test comments 2022-01-10 12:38:08 -05:00
Mathias Vorreiter Pedersen
2a02ce137a C++: Fix join orders in 'exprIsSubLeftOrLess'.
Before:

Tuple counts for UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff/2@i3#a5071w3a after 24s:
  304220    ~2%      {2} r1 = JOIN UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff#prev_delta WITH Expr::BinaryOperation#class#f#join_rhs ON FIRST 1 OUTPUT Lhs.1, Rhs.0 'sub'

  190061335 ~24%     {2} r2 = JOIN r1 WITH DataFlowUtil::localFlowStep#ff ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n'

  3956      ~0%      {2} r3 = JOIN r1 WITH DataFlowUtil::localFlowStep#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n'

  407983    ~1%      {2} r4 = JOIN Expr::BinaryOperation#class#f#join_rhs WITH UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff#prev ON FIRST 1 OUTPUT Rhs.1 'n', Lhs.0 'sub'
  380823    ~0%      {2} r5 = JOIN r4 WITH DataFlowUtil::TExprNode#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1
  0         ~0%      {2} r6 = JOIN r5 WITH UnsignedDifferenceExpressionComparedZero::isGuarded#fff#prev_delta ON FIRST 2 OUTPUT Rhs.2, Lhs.0 'sub'
  0         ~0%      {2} r7 = JOIN r6 WITH DataFlowUtil::TExprNode#ff ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n'

  3956      ~0%      {2} r8 = r3 UNION r7
  190065291 ~24%     {2} r9 = r2 UNION r8
  ...

After:

Tuple counts for UnsignedDifferenceExpressionComparedZero::interestingSubExpr#f/1@654e29g3 after 228ms:
  370 ~2%     {2} r1 = ComparisonOperation::RelationalOperation::getGreaterOperand_dispred#fb AND NOT Exclusions::isFromMacroDefinition#b(Lhs.1 'sub')
  370 ~0%     {2} r2 = SCAN r1 OUTPUT In.1 'sub', In.0
  370 ~3%     {3} r3 = JOIN r2 WITH Expr::Expr::getFullyConverted_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 'sub'
  210 ~1%     {2} r4 = JOIN r3 WITH SimpleRangeAnalysis::SimpleRangeAnalysisCached::exprMightOverflowNegatively#f ON FIRST 1 OUTPUT Lhs.2 'sub', Lhs.1
  210 ~0%     {3} r5 = JOIN r4 WITH Expr::Expr::getFullyConverted_dispred#ff ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'sub', Rhs.1
  210 ~1%     {3} r6 = JOIN r5 WITH ComparisonOperation::RelationalOperation::getLesserOperand_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'sub', Lhs.2
  59  ~2%     {4} r7 = JOIN r6 WITH Expr::Expr::getValue_dispred#ff ON FIRST 1 OUTPUT Lhs.1 'sub', Lhs.2, Rhs.1, toInt(Rhs.1)
  17  ~0%     {4} r8 = SELECT r7 ON In.3 = 0
  17  ~0%     {2} r9 = SCAN r8 OUTPUT In.1, In.0 'sub'
  8   ~0%     {2} r10 = JOIN r9 WITH Expr::Expr::getUnspecifiedType_dispred#bb ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'sub'
  8   ~0%     {1} r11 = JOIN r10 WITH Type::IntegralType::isUnsigned_dispred#f ON FIRST 1 OUTPUT Lhs.1 'sub'
              return r11

Tuple counts for UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff/2@i2#61800weu after 1ms:
  8  ~0%      {2} r1 = JOIN UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff#prev_delta WITH UnsignedDifferenceExpressionComparedZero::interestingSubExpr#f ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'sub'

  0  ~0%      {2} r2 = JOIN r1 WITH DataFlowUtil::localFlowStep#ff ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n'

  1  ~0%      {2} r3 = JOIN r1 WITH DataFlowUtil::localFlowStep#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n'

  0  ~0%      {3} r4 = JOIN UnsignedDifferenceExpressionComparedZero::isGuarded#fff#prev_delta WITH UnsignedDifferenceExpressionComparedZero::interestingSubExpr#f ON FIRST 1 OUTPUT Lhs.1, Lhs.0 'sub', Lhs.2
  0  ~0%      {3} r5 = JOIN r4 WITH DataFlowUtil::TExprNode#ff ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n', Lhs.2
  0  ~0%      {2} r6 = JOIN r5 WITH UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff#prev ON FIRST 2 OUTPUT Lhs.2, Lhs.0 'sub'
  0  ~0%      {2} r7 = JOIN r6 WITH DataFlowUtil::TExprNode#ff ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1 'n'

  1  ~0%      {2} r8 = r3 UNION r7
  1  ~0%      {2} r9 = r2 UNION r8
  ...
2022-01-10 17:28:14 +00:00
Mathias Vorreiter Pedersen
f2d6bcd767 C++: Fix join order in 'isGuarded'.
Before:

Tuple counts for UnsignedDifferenceExpressionComparedZero::isGuarded#bff/3@ec24001m after 1.7s:
  97431    ~0%     {2} r1 = JOIN UnsignedDifferenceExpressionComparedZero::isGuarded#bff#join_rhs WITH project#BasicBlocks::Cached::basic_block_member ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'sub'
  11809769 ~1%     {2} r2 = JOIN r1 WITH Guards::GuardCondition::controls_dispred#fff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1 'sub', Rhs.1
  11809769 ~0%     {4} r3 = JOIN r2 WITH project#BasicBlocks::Cached::basic_block_member ON FIRST 1 OUTPUT Lhs.1, Rhs.1, false, Lhs.0 'sub'
  629277   ~4%     {7} r4 = JOIN r3 WITH Guards::GuardCondition::ensuresLt_dispred#ffffff_045123#join_rhs ON FIRST 3 OUTPUT Lhs.3 'sub', Lhs.0, Lhs.1, false, Rhs.3 'left', Rhs.4 'right', Rhs.5
  628120   ~4%     {7} r5 = SELECT r4 ON In.6 >= 0
  628120   ~1%     {3} r6 = SCAN r5 OUTPUT In.0 'sub', In.4 'left', In.5 'right'
                    return r6

After:

Tuple counts for UnsignedDifferenceExpressionComparedZero::isGuarded#fff/3@i2#a5071x3a after 392ms:
  103763 ~0%     {2} r1 = SCAN UnsignedDifferenceExpressionComparedZero::exprIsSubLeftOrLess#ff#prev_delta OUTPUT In.0 'sub', 26
  103763 ~0%     {1} r2 = JOIN r1 WITH exprs ON FIRST 2 OUTPUT Lhs.0 'sub'
  97431  ~0%     {3} r3 = JOIN r2 WITH project#BasicBlocks::Cached::basic_block_member ON FIRST 1 OUTPUT Rhs.1, false, Lhs.0 'sub'
  629277 ~0%     {7} r4 = JOIN r3 WITH Guards::GuardCondition::ensuresLt_dispred#ffffff_450123#join_rhs ON FIRST 2 OUTPUT Lhs.2 'sub', Lhs.0, false, Rhs.2, Rhs.3 'left', Rhs.4 'right', Rhs.5
  628120 ~0%     {7} r5 = SELECT r4 ON In.6 >= 0
  628120 ~1%     {6} r6 = SCAN r5 OUTPUT In.0 'sub', In.1, In.3, In.4 'left', In.5 'right', In.6
  628120 ~1%     {6} r7 = r6 AND NOT UnsignedDifferenceExpressionComparedZero::isGuarded#fff#prev(Lhs.0 'sub', Lhs.3 'left', Lhs.4 'right')
  628120 ~0%     {5} r8 = SCAN r7 OUTPUT In.2, In.1, In.0 'sub', In.3 'left', In.4 'right'
  628120 ~1%     {3} r9 = JOIN r8 WITH Guards::GuardCondition::controls_dispred#fff ON FIRST 2 OUTPUT Lhs.2 'sub', Lhs.3 'left', Lhs.4 'right'
                  return r9
2022-01-10 17:03:40 +00:00
Tony Torralba
fbebf5e953 Move change note to new location 2022-01-10 17:27:02 +01:00
Tony Torralba
0e738622df Merge branch 'main' into atorralba/promote-log-injection 2022-01-10 17:24:25 +01:00