Commit Graph

5503 Commits

Author SHA1 Message Date
Taus
80ef09f034 Python: Fix bad join in declaredAttributeVar
Before:
```
Tuple counts for PointsTo::declaredAttributeVar#fbf/3@99d5aenq after 1.1s:
451054   ~7%     {2} r1 = SCAN variable OUTPUT In.0, In.2 'name'
1296149  ~0%     {2} r2 = JOIN r1 WITH Essa::EssaVariable::getSourceVariable_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'var', Lhs.1 'name'
12179900 ~4%     {3} r3 = JOIN r2 WITH Essa::EssaVariable::getAUse_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'name', Lhs.0 'var'
8028     ~2%     {3} r4 = JOIN r3 WITH Scope::Scope::getANormalExit_dispred#bf_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'name', Lhs.2 'var'
8028     ~2%     {3} r5 = JOIN r4 WITH Classes::PythonClassObjectInternal::getScope_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'cls', Lhs.1 'name', Lhs.2 'var'
                 return r5
```

After:
```
Tuple counts for PointsTo::declaredAttributeVar#fbf/3@cccf36hb after 4ms:
1450 ~0%     {2} r1 = SCAN Classes::PythonClassObjectInternal::getScope_dispred#ff OUTPUT In.1, In.0 'cls'
1450 ~7%     {2} r2 = JOIN r1 WITH Scope::Scope::getANormalExit_dispred#bf ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'cls'
8028 ~0%     {2} r3 = JOIN r2 WITH Essa::EssaVariable::getAUse_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1 'var', Lhs.1 'cls'
8028 ~0%     {3} r4 = JOIN r3 WITH Essa::EssaVariable::getSourceVariable_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'cls', Lhs.0 'var'
8028 ~2%     {3} r5 = JOIN r4 WITH variable ON FIRST 1 OUTPUT Lhs.1 'cls', Rhs.2 'name', Lhs.2 'var'
return r5
```
2022-04-28 11:11:37 +00:00
Taus
d28f9f41e8 Python: Fix bad join in import_star_read
Makes this

```
(21s) Tuple counts for DataFlowPublic::import_star_read#ff/2@fcd5e6nr after 8.5s:
9743      ~6%     {3} r1 = SCAN num#DataFlowPublic::TModuleVariableNode#fff OUTPUT In.1, In.0, In.2 'result'
9743      ~1%     {3} r2 = JOIN r1 WITH Variables::Variable::getId_dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2 'result'
390808917 ~3%     {3} r3 = JOIN r2 WITH Flow::NameNode::getId_dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2 'result'
307       ~0%     {2} r4 = JOIN r3 WITH ImportStar::ImportStar::importStarResolvesTo#ff ON FIRST 2 OUTPUT Lhs.0, Lhs.2 'result'
307       ~0%     {2} r5 = JOIN r4 WITH num#DataFlowPublic::TCfgNode#ff ON FIRST 1 OUTPUT Rhs.1 'n', Lhs.1 'result'
                  return r5
```

become this

```
(17s) Tuple counts for DataFlowPublic::resolved_import_star_module#fff/3@f5e84aic after 0ms:
307 ~0%     {3} r1 = JOIN ImportStar::ImportStar::importStarResolvesTo#ff WITH num#DataFlowPublic::TCfgNode#ff ON FIRST 1 OUTPUT Lhs.0, Lhs.1 'm', Rhs.1 'n'
307 ~0%     {3} r2 = JOIN r1 WITH Flow::NameNode::getId_dispred#ff ON FIRST 1 OUTPUT Lhs.1 'm', Rhs.1 'name', Lhs.2 'n'
            return r2
(17s) Registering DataFlowPublic::resolved_import_star_module#fff/3@f5e84aic +  with content f29281ig38r98icro4ege09mrva
(17s)  >>> Created relation DataFlowPublic::resolved_import_star_module#fff/3@f5e84aic with 307 rows.
(17s) Starting to evaluate predicate DataFlowPublic::import_star_read#ff/2@57b0c06e
(17s) Tuple counts for DataFlowPublic::import_star_read#ff/2@57b0c06e after 2ms:
9743 ~0%     {3} r1 = SCAN num#DataFlowPublic::TModuleVariableNode#fff OUTPUT In.1, In.0, In.2 'result'
9743 ~0%     {3} r2 = JOIN r1 WITH Variables::Variable::getId_dispred#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Lhs.2 'result'
307  ~0%     {2} r3 = JOIN r2 WITH DataFlowPublic::resolved_import_star_module#fff ON FIRST 2 OUTPUT Rhs.2 'n', Lhs.2 'result'
             return r3
```
2022-04-28 11:11:37 +00:00
yoff
4553a0913f Merge pull request #8897 from tausbn/python-fix-bad-methodcallsite-join
Python: Fix bad join in `MethodCallsiteRefinement`
2022-04-28 12:17:33 +02:00
Taus
b4a31e572f Python: Add global attribute writes 2022-04-27 16:45:00 +00:00
Taus
f71cf2e1fc Python: Add test 2022-04-27 15:48:11 +00:00
yoff
39753d5a0b Merge pull request #8693 from erik-krogh/pyApi
PY: more API-graphs refactorings
2022-04-27 13:19:50 +02:00
Taus
d3a05b8b7e Python: Fix bad join in MethodCallsiteRefinement
Observed on `FreeCAD/FreeCAD`:

```
Tuple counts for Essa::MethodCallsiteRefinement#24e22a14#f/1@274967ic after 34.5s:
638284     ~0%     {2} r1 = SCAN Essa::TEssaNodeRefinement#24e22a14#ffff OUTPUT In.0, In.3 'this'
636521     ~0%     {2} r2 = r1 AND NOT Essa::SingleSuccessorGuard#class#24e22a14#f(Lhs.1 'this')
1579493668 ~0%     {2} r3 = JOIN r2 WITH SsaDefinitions::SsaSource::method_call_refinement#9197156e#fff ON FIRST 1 OUTPUT Lhs.1 'this', Rhs.2
266673     ~3%     {1} r4 = JOIN r3 WITH Essa::EssaNodeRefinement::getDefiningNode#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.0 'this'
                    return r4
```

After a bit of unbinding, we have:

```
Tuple counts for Essa::MethodCallsiteRefinement#24e22a14#f/1@d73d8e27 after 66ms:
215168 ~1%     {2} r1 = SCAN Definitions::SsaSourceVariable#class#486534ab#f OUTPUT In.0, In.0
283965 ~2%     {2} r2 = JOIN r1 WITH SsaDefinitions::SsaSource::method_call_refinement#9197156e#fff ON FIRST 1 OUTPUT Rhs.2, Lhs.1
401274 ~0%     {2} r3 = JOIN r2 WITH Essa::EssaNodeRefinement::getDefiningNode#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Rhs.1 'this'
266671 ~2%     {1} r4 = JOIN r3 WITH Essa::TEssaNodeRefinement#24e22a14#ffff_03#join_rhs ON FIRST 2 OUTPUT Lhs.1 'this'
266671 ~2%     {1} r5 = r4 AND NOT Essa::SingleSuccessorGuard#class#24e22a14#f(Lhs.0 'this')
                return r5
```
(I'm somewhat confused about the slight difference in tuples, but it's
probably just because the compiler moved some stuff around.)
2022-04-27 11:13:37 +00:00
Erik Krogh Kristensen
e1c7d369be Merge pull request #8796 from erik-krogh/redundantImport
Remove redundant imports
2022-04-27 12:39:51 +02:00
yoff
9d774463f5 Merge pull request #8859 from tausbn/python-fix-bad-essa-joins
Python: Fix a bunch of bad joins
2022-04-27 12:27:50 +02:00
Taus
7d736952db Python: Update expected output 2022-04-26 15:49:40 +00:00
Erik Krogh Kristensen
7dba2b5868 PY: revert deletion of redundant-import in ClientSuppliedIpUsedInSecurityCheckLib.qll 2022-04-26 14:51:21 +02:00
Erik Krogh Kristensen
d389012b75 Merge branch 'main' into redundantImport 2022-04-26 14:24:51 +02:00
yoff
76f2eca1ee Merge pull request #8560 from erik-krogh/movePolyTest
PY: move the polynomialbacktracking-test to the test folder
2022-04-26 14:21:30 +02:00
Anders Schack-Mulligen
59aedc2872 Merge pull request #8853 from aschackmull/dataflow/fix-join
Dataflow: Fix join-on-config producing a CP.
2022-04-26 09:52:50 +02:00
Mathias Vorreiter Pedersen
aca4c8727f Merge pull request #8802 from github/post-release-prep/codeql-cli-2.9.0
Post-release preparation for codeql-cli-2.9.0
2022-04-25 22:52:55 +01:00
Taus
d4fc096ea8 Python: Use local flow in Pythagorean.ql
The hand-rolled notion of flow was causing some severe performance
issues (on a few databases):

```
Tuple counts for Pythagorean::square#168e234a#f#loop_invariant_prefix/2@c86989kr after 6m35s:
175000     ~5%     {2} r1 = JOIN SSA::SsaVariable::getDefinition#dispred#f0820431#ff_10#join_rhs WITH Flow::ControlFlowNode::getNode#dispred#f0820431#bf ON FIRST 1 OUTPUT Lhs.1, Rhs.1 'arg0'
174500     ~6%     {2} r2 = JOIN r1 WITH SSA::SsaVariable::getVariable#dispred#f0820431#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'arg0'
1467782500 ~5%     {3} r3 = JOIN r2 WITH AstGenerated::Name_::getVariable#dispred#f0820431#ff_10#join_rhs ON FIRST 1 OUTPUT 3, Rhs.1 'arg1', Lhs.1 'arg0'
1467553000 ~0%     {2} r4 = JOIN r3 WITH py_expr_contexts_12#join_rhs ON FIRST 2 OUTPUT Lhs.2 'arg0', Lhs.1 'arg1'
                    return r4
```

Rewriting it to use the data flow library made all of this go away. 🎉
2022-04-25 14:35:37 +00:00
Taus
b2cc91369a Python: Fix bad join in firstUse
This was what it looked like (at the point when I killed the evaluation):

```
Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#c5fa2be7#ff/2@i1#be98bwif after 1m50s:
274000     ~7%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#c5fa2be7#ffff OUTPUT In.1, In.0 'def', In.2, In.3
2731768000 ~1%     {7} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::variableSourceUse#c5fa2be7#ffff ON FIRST 1 OUTPUT Rhs.0, Lhs.2, Lhs.3, Rhs.2, Rhs.3, Rhs.1 'use', Lhs.1 'def'
178000     ~4%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#c5fa2be7#fffff ON FIRST 5 OUTPUT Lhs.6 'def', Lhs.5 'use'
                    return r3
```

And this is what it looks like now:

```
Tuple counts for SsaCompute::SsaComputeImpl::AdjacentUsesImpl::firstUse#c5fa2be7#ff/2@i1#f9d6ewsi after 207ms:
931353  ~2%     {4} r1 = SCAN SsaCompute::SsaComputeImpl::AdjacentUsesImpl::variableSourceUse#c5fa2be7#ffff OUTPUT In.0, In.2, In.3, In.1 'use'
1050477 ~0%     {4} r2 = JOIN r1 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::adjacentVarRefs#c5fa2be7#fffff_03412#join_rhs ON FIRST 3 OUTPUT Lhs.0, Rhs.3, Rhs.4, Lhs.3 'use'
506626  ~0%     {2} r3 = JOIN r2 WITH SsaCompute::SsaComputeImpl::AdjacentUsesImpl::definesAt#c5fa2be7#ffff_1230#join_rhs ON FIRST 3 OUTPUT Rhs.3 'def', Lhs.3 'use'
                return r3
```
2022-04-25 14:33:31 +00:00
Taus
49233268a9 Python: Fix bad join in getValue
We were building essentially a CP of all control flow nodes:

```
Tuple counts for Essa::AssignmentDefinition::getValue#dispred#f0820431#ff/2@dd1f67vl after 2m45s:
733365     ~6%     {3} r1 = JOIN Essa::TEssaNodeDefinition#24e22a14#ffff_30#join_rhs WITH Essa::EssaNodeDefinition::getDefiningNode#dispred#f0820431#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Rhs.0
376588     ~0%     {2} r2 = JOIN r1 WITH SsaDefinitions::SsaSource::assignment_definition#9197156e#fff ON FIRST 2 OUTPUT Lhs.2 'this', Rhs.2 'result'
376588     ~0%     {3} r3 = JOIN r2 WITH Essa::TEssaNodeDefinition#24e22a14#ffff_30#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.0 'this', Lhs.1 'result'
6965593033 ~2%     {3} r4 = JOIN r3 WITH project#SsaDefinitions::SsaSource::assignment_definition#9197156e ON FIRST 1 OUTPUT Lhs.1 'this', Rhs.1, Lhs.2 'result'
376588     ~0%     {2} r5 = JOIN r4 WITH Essa::EssaNodeDefinition::getDefiningNode#dispred#f0820431#ff ON FIRST 2 OUTPUT Lhs.0 'this', Lhs.2 'result'
                    return r5
```

We first tried preventing the join on `result`, but this caused the
characteristic predicate to blow up instead. Finally, we figured just
putting the `value` part in a field would be sufficient, and this did
the trick.
2022-04-25 14:28:00 +00:00
Anders Schack-Mulligen
c06efa1f42 Dataflow: Sync. 2022-04-25 13:11:04 +02:00
Tom Hvitved
bffa8fa7cb Merge pull request #8641 from hvitved/dataflow/interpret-read-store
Data flow: Introduce `ContentSet`
2022-04-25 12:17:34 +02:00
Tom Hvitved
2466288656 Data flow: Simplify revFlowStore 2022-04-25 10:11:54 +02:00
Tom Hvitved
cf0a1e748a Add change notes 2022-04-25 09:17:40 +02:00
Jeroen Ketema
79164056d1 Replace help.semmle.com links by codeql.github.com links 2022-04-22 20:42:11 +02:00
Erik Krogh Kristensen
45080e7777 PY: add missing qldoc 2022-04-22 15:30:31 +02:00
Erik Krogh Kristensen
acac8919b3 PY: update expected output for deprecation warning in test file 2022-04-22 15:28:31 +02:00
Tom Hvitved
bc6ee10583 Data flow: Sync files 2022-04-22 15:10:00 +02:00
Tom Hvitved
b033f107df Merge remote-tracking branch 'upstream/main' into dataflow/interpret-read-store 2022-04-22 14:35:02 +02:00
Rasmus Wriedt Larsen
03c0366fd4 Merge branch 'main' into stdlib-FileSystemAccess-improvement 2022-04-22 14:31:31 +02:00
Erik Krogh Kristensen
8fcbaea273 Merge branch 'main' into labelNaming 2022-04-22 13:19:44 +02:00
Erik Krogh Kristensen
ff73dbc35c delete redundant imports 2022-04-22 12:55:28 +02:00
Erik Krogh Kristensen
a96489b23d delete duplicate imports 2022-04-22 12:41:30 +02:00
Rasmus Wriedt Larsen
650d57083b Python: Recognize path arguments to pathlib methods 2022-04-22 11:01:59 +02:00
Rasmus Wriedt Larsen
bcaba45202 Python: Expand pathlib tests 2022-04-22 11:01:59 +02:00
Rasmus Wriedt Larsen
059dea713d Python: Fix os.path.samefile modeling 2022-04-22 11:01:59 +02:00
github-actions[bot]
1aecfc67c2 Post-release preparation for codeql-cli-2.9.0 2022-04-21 19:22:19 +00:00
Dave Bartolomeo
b224f81e24 Fix formatting in change log 2022-04-21 10:59:38 -04:00
Dave Bartolomeo
fb710cd944 Fix formatting in change log 2022-04-21 10:59:03 -04:00
github-actions[bot]
eeaf233c29 Release preparation for version 2.9.0 2022-04-21 14:49:00 +00:00
Erik Krogh Kristensen
aec8413487 PY: mention newtype constructors in API graph label classes 2022-04-20 18:38:44 +02:00
Anders Schack-Mulligen
677c436e99 Merge pull request #8703 from aschackmull/dataflow/revert-state-in-out-barriers
Dataflow: Revert support for flow-state based in-/out-barriers
2022-04-20 14:54:02 +02:00
Rasmus Wriedt Larsen
bb6969a175 Merge branch 'main' into promote-xxe 2022-04-20 13:42:02 +02:00
Rasmus Wriedt Larsen
888a38c060 Python: Add change-note 2022-04-20 11:46:09 +02:00
Rasmus Wriedt Larsen
d70f247001 Python: More private import python 2022-04-20 11:42:13 +02:00
Rasmus Wriedt Larsen
084c8eb22e Python: Don't re-export python under DataFlow:: 2022-04-20 11:42:10 +02:00
Rasmus Wriedt Larsen
5dbbd17bb2 Python: Add test to ensure we keep DataFlow imports clean
Currently we're not in a good state :(
2022-04-20 11:41:01 +02:00
yoff
0c7130602a Merge pull request #8731 from RasmusWL/delete-old-readme
Python: Delete old dataflow readme
2022-04-20 10:36:12 +02:00
yoff
a66153d73e Merge pull request #8733 from RasmusWL/split-dataflow-private
Python: Split `DataFlowPrivate`
2022-04-20 10:21:05 +02:00
Anders Schack-Mulligen
48fbbf2531 Dataflow: Add change notes. 2022-04-19 15:29:35 +02:00
Anders Schack-Mulligen
b521d64156 Dataflow: Sync. 2022-04-19 15:29:35 +02:00
Mathias Vorreiter Pedersen
91b413d59f Dataflow: Sync identical files. 2022-04-19 09:57:21 +01:00