Commit Graph

68274 Commits

Author SHA1 Message Date
Taus
446dbf67cc Python: Fix bad join in getImmediateModuleReference
The "most expensive predicates" report had the following line on a
certain database:

```
1m15s |    11 |   37s @ 4    | ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0@12bb4xdo
```

Investigating further revealed the following bad joins

```
(388s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i4#b2325xoe after 36.5s:
0         ~0%        {2} r1 = JOIN `ImportResolution::ImportResolution::sys_modules_module_with_name/1#134529bf#prev_delta` WITH `ImportResolution::ImportResolution::getReferenceToModuleName/1#bc5da225` ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1 'm'

74884348  ~0%        {3} r2 = JOIN `ImportResolution::ImportResolution::getModuleReference/1#28368ea4#prev_delta` WITH `ImportResolution::ImportResolution::potential_module_export/2#19340171` ON FIRST 1 OUTPUT Lhs.1, Rhs.1, Lhs.0
5221604   ~0%        {3}    | JOIN WITH `Attributes::AttrRef.accesses/2#dispred#31929f12_120#join_rhs` ON FIRST 2 OUTPUT Rhs.2 'result', Lhs.2, Lhs.1
5219926   ~2%        {3}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.1, Lhs.2, Lhs.0 'result'
5300880   ~1%        {2}    | JOIN WITH `ImportResolution::ImportResolution::module_export/3#f2fc6a2a` ON FIRST 2 OUTPUT Rhs.2, Lhs.2 'result'
42211     ~5%        {2}    | JOIN WITH `ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0#prev` ON FIRST 1 OUTPUT Lhs.1 'result', Rhs.1 'm'

957042    ~4%        {3} r3 = JOIN `ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0#prev_delta` WITH `ImportResolution::ImportResolution::module_export/3#f2fc6a2a_201#join_rhs` ON FIRST 1 OUTPUT Rhs.1, Rhs.2, Lhs.1 'm'
957035    ~0%        {3}    | JOIN WITH `ImportResolution::ImportResolution::potential_module_export/2#19340171` ON FIRST 2 OUTPUT Lhs.1, Lhs.2 'm', Lhs.0
236753257 ~1%        {4}    | JOIN WITH `Attributes::AttrRef.accesses/2#dispred#31929f12_201#join_rhs` ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1 'm', Lhs.2, Rhs.2
199557145 ~2%        {4}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.2, Lhs.3, Lhs.1 'm', Lhs.0 'result'
1         ~0%        {2}    | JOIN WITH `ImportResolution::ImportResolution::getModuleReference/1#28368ea4#prev` ON FIRST 2 OUTPUT Lhs.3 'result', Lhs.2 'm'

15199013  ~1951%     {2} r4 = JOIN `ImportResolution::ImportResolution::getModuleReference/1#28368ea4#prev_delta` WITH `Module::Module.getPackageName/0#dispred#bb0c3872` ON FIRST 1 OUTPUT Lhs.1, Rhs.1
14707604  ~2136%     {3}    | JOIN WITH `Attributes::AttrRef.accesses/2#dispred#31929f12_102#join_rhs` ON FIRST 1 OUTPUT Rhs.1 'result', Lhs.1, Rhs.2

14623588  ~2190%     {4} r5 = JOIN r4 WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT _, Lhs.0 'result', Lhs.1, Lhs.2
14623588  ~2058%     {2}    | REWRITE WITH Tmp.0 := ".", Out.0 := (In.2 ++ Tmp.0 ++ In.3) KEEPING 2

14623588  ~2139%     {5} r6 = JOIN r4 WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT _, Lhs.0 'result', Lhs.1, Lhs.2, _
14623588  ~2092%     {2}    | REWRITE WITH Tmp.0 := ".", Tmp.0 := (In.2 ++ Tmp.0 ++ In.3), Tmp.4 := ".__init__", Out.0 := (Tmp.0 ++ Tmp.4) KEEPING 2

29247176  ~2099%     {2} r7 = r5 UNION r6
199786001 ~6922%     {2}    | JOIN WITH `Module::isPreferredModuleForName/2#5fb427f9_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1, Lhs.1 'result'
199756923 ~7024%     {2}    | JOIN WITH `Module::Module.getFile/0#dispred#53eb9b1b_10#join_rhs` ON FIRST 1 OUTPUT Lhs.1 'result', Rhs.1 'm'

199799135 ~6954%     {2} r8 = r1 UNION r2 UNION r3 UNION r7
199793992 ~6954%     {2}    | AND NOT `ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0#prev`(FIRST 2)
                     return r8
```

Clearly, waiting to joining with `getModuleReference` last is not
healthy. To fix this, I opted to simply create a helper predicate for
the `accesses` construct.

After this change, here are the iteration timings

```
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i1#74f41yqa after 1.2s:
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i1#8a053ys7 after 1.3s:
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i2#74f41yqa after 20ms:
(327s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i2#8a053ys7 after 20ms:
(337s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i4#74f41yqa after 8.5s:
(341s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i4#8a053ys7 after 3.2s:
(346s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i5#74f41yqa after 7.2s:
(349s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i6#74f41yqa after 3ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i5#8a053ys7 after 10s:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i8#74f41yqa after 37ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i9#74f41yqa after 0ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i10#74f41yqa after 0ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i11#74f41yqa after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i12#74f41yqa after 1ms:
(353s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i6#8a053ys7 after 1ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i8#8a053ys7 after 7ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i9#8a053ys7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i10#8a053ys7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i11#8a053ys7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::getImmediateModuleReference/1#3553e6c0#reorder_1_0/2@i12#8a053ys7 after 0ms:
```

And the helper predicate itself is also quick to evaluate:

```
(327s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i1#74f41xqa after 0ms:
(327s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i1#8a053xs7 after 0ms:
(329s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i3#74f41xqa after 99ms:
(337s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i3#8a053xs7 after 98ms:
(338s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i4#74f41xqa after 679ms:
(341s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i4#8a053xs7 after 400ms:
(346s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i5#74f41xqa after 1ms:
(349s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i6#74f41xqa after 22ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i5#8a053xs7 after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i7#74f41xqa after 1.4s:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i8#74f41xqa after 8ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i9#74f41xqa after 0ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i10#74f41xqa after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i11#74f41xqa after 1ms:
(352s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i12#74f41xqa after 1ms:
(353s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i13#74f41xqa after 806ms:
(353s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i6#8a053xs7 after 7ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i7#8a053xs7 after 870ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i8#8a053xs7 after 2ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i9#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i10#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i11#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i12#8a053xs7 after 0ms:
(354s) Tuple counts for ImportResolution::ImportResolution::module_reference_accesses/3#8f45b418#reorder_1_2_0/3@i13#8a053xs7 after 276ms:
```

(I note that we appear to be evaluating this code twice, which is a bit
worrying. I'll leave that investigaton for later.)
2024-07-01 12:53:04 +00:00
Taus
d9b337cb2c Merge pull request #16804 from github/tausbn/python-fix-bad-join-in-dataflow-dispatch
Python: Fix bad join in `DataFlowDispatch`
2024-07-01 13:14:28 +02:00
Arthur Baars
c6d02e4909 Merge pull request #16878 from github/aibaars/merge-3.14
Merge rc/3.14 into main
2024-07-01 11:04:57 +02:00
Owen Mansel-Chan
0db4110449 Merge pull request #16879 from github/workflow/coverage/update
Update CSV framework coverage reports
2024-06-30 13:59:16 +01:00
github-actions[bot]
26194eb65f Add changed framework coverage reports 2024-06-30 00:19:16 +00:00
Owen Mansel-Chan
390fecbef6 Merge pull request #16866 from owen-mc/go/add-go-jose-to-library-coverage-frameworks
Go: Add Go JOSE to library coverage frameworks
2024-06-29 07:30:36 +01:00
Arthur Baars
b12b33c8f9 Merge remote-tracking branch 'upstream/main' into 'rc/3.14' 2024-06-28 19:50:35 +02:00
Jeroen Ketema
a85f73972c Merge pull request #16870 from jketema/preproc-more
C++: Update test expectations
2024-06-28 19:01:12 +02:00
Jami
42925b56e3 Merge pull request #15921 from jcogs33/jcogs33/unsafe-url-forward-promotion-resource-and-file-methods
Java: add models for some resource-related methods
2024-06-28 08:05:50 -04:00
Erik Krogh Kristensen
60811116ab Merge pull request #16332 from erik-krogh/ts55
JS: upgrade TypeScript to 5.5
2024-06-28 13:59:52 +02:00
Tamás Vajk
1cf5e89b96 Merge pull request #16747 from tamasvajk/buildless/binary-log-extractor-2
C#: Add binlog support to buildless with source generator support
2024-06-28 12:39:54 +02:00
Erik Krogh Kristensen
fd3089ee5a Merge pull request #14342 from maikypedia/maikypedia/javascript-cors
JS: Add Permissive CORS query (CWE-942)
2024-06-28 11:27:37 +02:00
Mathias Vorreiter Pedersen
72caadb7db Merge pull request #16869 from hvitved/shared/cfg/mermaid-output
Shared: Generate mermaid output in `View CFG` query
2024-06-28 10:21:02 +01:00
Tom Hvitved
e1c1314824 Shared: Generate mermaid output in View CFG query 2024-06-28 09:45:22 +02:00
Tamas Vajk
4db586f17d Add TSP diagnostic for binlog usage 2024-06-28 09:40:41 +02:00
Tamas Vajk
c8927447f5 Record diagnostics in binlog extraction test 2024-06-28 09:40:13 +02:00
Jami Cogswell
85a1e1a972 Java: update change note date 2024-06-27 22:11:01 -04:00
Jami Cogswell
be565288f2 Java: update more test cases due to shifted alert provenance line numbers 2024-06-27 22:08:38 -04:00
Jami Cogswell
c73af7f789 Java: update some test cases due to shifted alert provenance line numbers 2024-06-27 21:07:35 -04:00
Maiky
d0cf2a978c Merge branch 'main' into maikypedia/javascript-cors 2024-06-27 20:24:42 +02:00
Jeroen Ketema
8defd27b49 C++: Update test expectations 2024-06-27 17:45:33 +02:00
yoff
40b7534210 Merge pull request #16860 from yoff/python/MaD-doc-fixup
Python: fix typo pointed out in review but missed by me
2024-06-27 17:02:39 +02:00
Owen Mansel-Chan
98b2d1f2d7 Add Go JOSE to library coverage frameworks 2024-06-27 14:52:08 +01:00
Owen Mansel-Chan
5d7a6e3836 Merge pull request #16851 from github/workflow/coverage/update
Update CSV framework coverage reports
2024-06-27 13:57:30 +01:00
Tamas Vajk
1e2d1ef715 Use dedicated API to get generated syntax trees 2024-06-27 14:49:29 +02:00
Tamas Vajk
0b41d5121a Compute unique identifier (folder path) for each compilation 2024-06-27 14:49:24 +02:00
Tamas Vajk
1ae40c95b1 Change binlog test to have colliding generated file locations 2024-06-27 14:49:20 +02:00
Tamas Vajk
b22f5f557a Fix failed extraction handling 2024-06-27 14:49:16 +02:00
Tamas Vajk
fb0520c74a C#: Adjust trap location, database ID and archiving of generated sources 2024-06-27 14:49:10 +02:00
Tamas Vajk
dcd84f47a4 Improve source archiving to handle non-existent files 2024-06-27 14:49:06 +02:00
Tamas Vajk
31ad195dc3 C#: Add binlog support to buildless with source generator support 2024-06-27 14:48:54 +02:00
Tamás Vajk
b2d2f2d0d8 Merge pull request #16858 from tamasvajk/fix/solution
C#: Fix solution file
2024-06-27 14:31:15 +02:00
Owen Mansel-Chan
50cc720117 Merge branch 'main' into workflow/coverage/update 2024-06-27 10:51:23 +01:00
Rasmus Lerchedahl Petersen
da03237b32 Python: fix typo pointed out in review but missed by me 2024-06-27 11:21:28 +02:00
Tamás Vajk
6c727b1e7d Merge pull request #16857 from tamasvajk/feature/stringformat
C#: Change `string.Format` calls to interpolated strings
2024-06-27 10:55:21 +02:00
Tamas Vajk
3e20d908c4 C#: Fix solution file 2024-06-27 10:53:39 +02:00
Chris Smowton
c5678ad156 Merge pull request #16847 from smowton/smowton/admin/java-diagnostic-test-expectations
Java: Adjust test expectations for Java diagnostic severity changes
2024-06-27 09:53:25 +01:00
Tamas Vajk
0c34b4535a C#: Change string.Format calls to interpolated strings 2024-06-27 09:20:50 +02:00
Tamás Vajk
4a98436884 Merge pull request #16855 from tamasvajk/fix/integration-test
C#: Fix integration test expected file
2024-06-27 08:52:05 +02:00
Tamas Vajk
6b6d705b4f C#: Fix integration test expected file 2024-06-27 08:14:37 +02:00
Jami
901245ae3d Merge branch 'main' into jcogs33/unsafe-url-forward-promotion-resource-and-file-methods 2024-06-26 21:57:07 -04:00
github-actions[bot]
0a8c9da0ac Add changed framework coverage reports 2024-06-27 00:17:25 +00:00
Chris Smowton
f2cbf08d15 Adjust test expectations 2024-06-26 17:37:48 +01:00
Owen Mansel-Chan
272132a09a Merge pull request #16799 from owen-mc/go/mad/match-all-package-versions
Go: Make models-as-data package column match any version without "$ANYVERSION"
2024-06-26 13:46:49 +01:00
Anders Schack-Mulligen
9d8ee99c1c Merge pull request #16806 from aschackmull/dataflow/debug-stages
Dataflow: Add path-problem view of intermediate stages for debug purposes.
2024-06-26 12:53:12 +02:00
Michael Nebel
e1f65d1f8b Merge pull request #16836 from michaelnebel/csharp/bestlocation
C#: Be more consistent when picking between locations.
2024-06-26 12:46:50 +02:00
Cornelius Riemenschneider
c4cc30fb7a Merge pull request #16839 from github/criemen/bazel-721
Bump to bazel 7.2.1.
2024-06-26 11:26:19 +02:00
Paolo Tranquilli
53a7d823ec Merge pull request #16841 from github/redsun82/kotlin
Kotlin: exclude `KotlinExtractorDbScheme.kt` generated by hand
2024-06-26 11:18:44 +02:00
Tamás Vajk
81f4786643 Merge pull request #16832 from tamasvajk/feature/update-dependencies
C#: Update (some) nuget dependencies
2024-06-26 11:12:26 +02:00
Michael Nebel
e258d9fa74 C#: Use the first best location from the list of locations. 2024-06-26 11:04:38 +02:00