Commit Graph

95 Commits

Author SHA1 Message Date
Jonas Jensen
3ffda2f341 Shared: Overhaul the AlertFiltering QLDoc
The documentation is now up-to-date with the new and more relaxed rules
that allow overapproximating the results. I have also attempted to make
a clearer distinction between the requirements of the specification and
the behaviour of the implementation.
2025-07-09 14:32:18 +02:00
Jonas Jensen
0d7a842e2f Shared: improve documentation in AlertFiltering 2025-07-09 09:43:49 +02:00
Jonas Jensen
f1e9f0e323 Shared: improve join order in filterByLocation
It's better to join with the range expression first since that will only
multiply tuple counts by the number of lines in an average source/sink.
Joining with `restrictAlertsToStartLine` first would multiply tuple
counts by the number of sources/sinks in a given file.
2025-07-09 09:24:26 +02:00
Asger F
4a2d795076 Shared: Make approximate location filtering the default behaviour 2025-07-02 14:41:02 +02:00
Asger F
8b345518f4 Shared: Add approximate version of getASelected{Source,Sink}Location 2025-07-02 14:39:39 +02:00
Asger F
d1b4172486 Shared: Factor out some helper predicates in alert filtering 2025-07-02 14:39:37 +02:00
Kasper Svendsen
c207cfdeb7 Overlay: Add overlay annotations to Java & shared libraries 2025-06-24 10:25:06 +02:00
Tom Hvitved
aa0fc05df8 Rust: Remove external locations in tests using post-processing 2025-06-04 13:07:43 +02:00
Asger F
2ce01bfb9a Add Folder::Resolve as a generalisation of Folder::Append 2025-04-29 09:42:23 +02:00
Asger F
eb059969e3 Move getAChildContainer one scope up 2025-04-29 09:42:22 +02:00
Michael Nebel
617f4729d8 Shared: Match line information on Alert and Sink locations. 2025-04-23 12:35:17 +02:00
Tom Hvitved
ae5ac11387 Shared: Fix join in FileSystem.qll
Before
```
Evaluated relational algebra for predicate FileSystem::Folder::Append<PathResolution::shouldAppend>::appendStep/3#bed54f6d@d2a7eeoa on iteration 2 running pipeline standard with tuple counts:
         120   ~2%    {4} r1 = SCAN `FileSystem::Folder::Append<PathResolution::shouldAppend>::appendStep/3#bed54f6d#prev_delta` OUTPUT In.1, In.0, In.2, In.3

           0   ~0%    {6} r2 = JOIN r1 WITH `_FileSystem::Folder::Append<PathResolution::shouldAppend>::getComponent/2#a6e36a04#join_rhs#1` ON FIRST 1 OUTPUT Lhs.0, Rhs.1, Lhs.1, Lhs.2, Lhs.3, _
                      {6}    | REWRITE WITH Tmp.5 := 1, Out.5 := (InOut.1 - Tmp.5), TEST Out.5 = InOut.3
           0   ~0%    {4}    | SCAN OUTPUT In.2, In.0, In.1, In.4

          91   ~3%    {6} r3 = JOIN r1 WITH `_FileSystem::Folder::Append<PathResolution::shouldAppend>::getComponent/2#a6e36a04#join_rhs` ON FIRST 1 OUTPUT Lhs.0, Rhs.1, Lhs.1, Lhs.2, Lhs.3, _
                      {6}    | REWRITE WITH Tmp.5 := 1, Out.5 := (InOut.1 - Tmp.5), TEST Out.5 = InOut.3
          81   ~0%    {4}    | SCAN OUTPUT In.4, In.0, In.1, In.2
          81   ~3%    {4}    | JOIN WITH containerparent_10#join_rhs ON FIRST 1 OUTPUT Lhs.3, Lhs.1, Lhs.2, Rhs.1

         269   ~0%    {7} r4 = JOIN r1 WITH `_FileSystem::Folder::Append<PathResolution::shouldAppend>::getComponent/2#a6e36a04#join_rhs#2` ON FIRST 1 OUTPUT Lhs.0, Rhs.1, Rhs.2, Lhs.1, Lhs.2, Lhs.3, _
                      {7}    | REWRITE WITH Tmp.6 := 1, Out.6 := (InOut.1 - Tmp.6), TEST Out.6 = InOut.4
          39   ~1%    {5}    | SCAN OUTPUT In.5, In.0, In.1, In.2, In.3
        1295   ~0%    {6}    | JOIN WITH containerparent ON FIRST 1 OUTPUT Rhs.1, Lhs.3, _, Lhs.1, Lhs.2, Lhs.4
        1295   ~0%    {6}    | REWRITE WITH Out.2 := 1
          34   ~1%    {4}    | JOIN WITH `cached_FileSystem::Container.splitAbsolutePath/2#dispred#dc97b0cc` ON FIRST 3 OUTPUT Lhs.5, Lhs.3, Lhs.4, Lhs.0

         115   ~0%    {4} r5 = r2 UNION r3 UNION r4
         115   ~0%    {4}    | AND NOT `FileSystem::Folder::Append<PathResolution::shouldAppend>::appendStep/3#bed54f6d#prev`(FIRST 4)
                      return r5
```

After
```
Evaluated relational algebra for predicate FileSystem::Folder::Append<PathResolution::shouldAppend>::appendStep/3#bed54f6d@4fb6e6v7 on iteration 2 running pipeline standard with tuple counts:
        120   ~0%    {4} r1 = SCAN `FileSystem::Folder::Append<PathResolution::shouldAppend>::appendStep/3#bed54f6d#prev_delta` OUTPUT In.1, In.0, In.2, In.3

          0   ~0%    {6} r2 = JOIN r1 WITH `_FileSystem::Folder::Append<PathResolution::shouldAppend>::getComponent/2#a6e36a04#join_rhs#1` ON FIRST 1 OUTPUT Lhs.0, Rhs.1, Lhs.1, Lhs.2, Lhs.3, _
                     {6}    | REWRITE WITH Tmp.5 := 1, Out.5 := (InOut.1 - Tmp.5), TEST Out.5 = InOut.3
          0   ~0%    {4}    | SCAN OUTPUT In.2, In.0, In.1, In.4

         91   ~0%    {6} r3 = JOIN r1 WITH `_FileSystem::Folder::Append<PathResolution::shouldAppend>::getComponent/2#a6e36a04#join_rhs` ON FIRST 1 OUTPUT Lhs.0, Rhs.1, Lhs.1, Lhs.2, Lhs.3, _
                     {6}    | REWRITE WITH Tmp.5 := 1, Out.5 := (InOut.1 - Tmp.5), TEST Out.5 = InOut.3
         81   ~0%    {4}    | SCAN OUTPUT In.4, In.0, In.1, In.2
         81   ~5%    {4}    | JOIN WITH containerparent_10#join_rhs ON FIRST 1 OUTPUT Lhs.3, Lhs.1, Lhs.2, Rhs.1

        269   ~0%    {7} r4 = JOIN r1 WITH `_FileSystem::Folder::Append<PathResolution::shouldAppend>::getComponent/2#a6e36a04#join_rhs#2` ON FIRST 1 OUTPUT Lhs.0, Rhs.1, Rhs.2, Lhs.1, Lhs.2, Lhs.3, _
                     {7}    | REWRITE WITH Tmp.6 := 1, Out.6 := (InOut.1 - Tmp.6), TEST Out.6 = InOut.4
         39   ~3%    {5}    | SCAN OUTPUT In.5, In.2, In.0, In.1, In.3
         34   ~0%    {4}    | JOIN WITH `FileSystem::Folder::Append<PathResolution::shouldAppend>::getAChildContainer/2#2e91feca` ON FIRST 2 OUTPUT Lhs.4, Lhs.2, Lhs.3, Rhs.2

        115   ~2%    {4} r5 = r2 UNION r3 UNION r4
        115   ~2%    {4}    | AND NOT `FileSystem::Folder::Append<PathResolution::shouldAppend>::appendStep/3#bed54f6d#prev`(FIRST 4)
                     return r5
```
2025-04-22 12:16:59 +02:00
Tom Hvitved
52401aaa73 Address review comments 2025-04-09 17:19:25 +02:00
Tom Hvitved
13f4a6afa6 Rust: Handle path attributes in path resolution 2025-04-07 15:24:17 +02:00
Asger F
c49ffa01ee JS: Enable post-processed inline expectations for query predicates 2025-04-03 13:24:17 +02:00
Asger F
6e53ebed47 Ensure total ranking order to avoid ambiguous results 2025-02-26 09:34:06 +01:00
Asger F
bb8f4529bf Fix bug when RelatedLocation was used with a query ID 2025-02-25 14:52:32 +01:00
Asger F
694f01ab78 Fix column count and add clarifying comment 2025-02-25 11:57:01 +01:00
Asger F
e2fe74ccd6 JS: Add support for RelatedLocation tags 2025-02-21 14:44:45 +01:00
Asger F
654c6bfec7 Merge pull request #18735 from asgerf/inline-test-non-location
Test: Support arbitrary locations in inline test post-processor
2025-02-12 10:30:50 +01:00
Asger F
c306f44589 Remove override of final predicate 2025-02-11 17:07:09 +01:00
Asger F
80e79b11f7 Apply suggestions from code review
Co-authored-by: Tom Hvitved <hvitved@github.com>
2025-02-11 16:53:13 +01:00
Asger F
967c0860f9 Test: support queries that don't select a Location 2025-02-11 12:58:49 +01:00
Tom Hvitved
89502d63e5 Rust: Implement database quality telemetry query 2025-02-06 10:46:48 +01:00
Tom Hvitved
fc04ad1ef0 Test: Remove location parsing 2025-02-04 09:34:33 +01:00
Asger F
28472ae12f Test: Don't expect 'Source' tag when source and alert are on same line
Previously the Source tag was required if the source and alert did not
have the exact same location. This relaxes the restriction to being on
the same line.

Note that in order to be "on the same line" both start and end lines
have to match.

It's still possible for a given line to expect both Alert and Source
tags, in case the source pairs up with another alert on a different
line.
2025-02-03 11:31:02 +01:00
Chuan-kai Lin
96caa686fc AlertFiltering: add restrictAlertsToExactLocation
This commit introduces a new extensible predicate
restrictAlertsToExactLocation, which is similar to the existing
restrictAlertsTo predicate but matches alert locations exactly.
2025-01-29 07:50:45 -08:00
Chuan-kai Lin
b9b9394259 AlertFiltering: allow multiple filtering predicates
This commit rephrases the documentation for the restrictAlertsTo
predicate and renames the predicate columns for clarity. The new
documentation should be equivalent to the old documentation, except
allowing for the possibility that there may be multiple alert filtering
predicates.
2025-01-28 07:51:45 -08:00
Asger F
ce8912ddcc Test: Handle 'problems' result set as an alias for '#select' 2025-01-10 14:18:21 +01:00
Jeroen Ketema
10592bb1c4 Merge pull request #18192 from jketema/inline-rm
Remove deprecated `InlineExpectationsTest` class-based API
2024-12-04 11:34:39 +01:00
Chuan-kai Lin
63c8769323 AlertFiltering: prohibit partial filtering
This documentation-only commit clarifies that a query should either
ignore restrictAlertsTo completely or apply restrictAlertsTo filtering
to all alerts.  This update eliminates the ambiguity on whether a query
may choose to apply restrictAlertsTo filtering to only some alerts but
not others (it may not).
2024-12-03 12:26:35 -08:00
Jeroen Ketema
8271ad60c1 Remove deprecated InlineExpectationsTest class-based API 2024-12-03 19:18:37 +01:00
Tom Hvitved
143d7e2084 Rust: Use extended canonical paths to resolve calls in data flow 2024-11-22 13:23:39 +01:00
Tom Hvitved
42e0d7ce10 Util: Refactor DenseRank implementation 2024-11-20 13:09:59 +01:00
Tom Hvitved
495c92df38 Shared: Also take query ID into account in PathProblemSourceTestInput 2024-10-30 10:54:40 +01:00
Tom Hvitved
e7a3e6bfed Shared: Post-processing query for inline test expectations 2024-10-29 13:35:29 +01:00
Tom Hvitved
7c4d5981dd Shared: Add missing spaces in inline test expectation output 2024-10-25 13:23:03 +02:00
Tom Hvitved
4df9cd88ef Merge pull request #17658 from hvitved/shared/cfg-conditional-splitting
Shared `ConditionalCompletionSplitting` implementation
2024-10-10 13:21:38 +02:00
Tom Hvitved
aa7215bbd5 Address review comments 2024-10-09 17:22:16 +02:00
Tom Hvitved
3d95369608 Shared ConditionalSplitting implementation 2024-10-09 11:02:13 +02:00
Tom Hvitved
9952997b48 Shared: Add DenseRank library 2024-10-09 10:16:59 +02:00
Chuan-kai Lin
ff78bebf19 Shared support for alert filtering 2024-09-11 13:18:26 -07:00
Anders Schack-Mulligen
da5abc8321 Dataflow: Replace MakeSets with QlBuiltins::InternSets. 2024-07-15 13:35:57 +02:00
Anders Schack-Mulligen
4ff37cc7b6 Dataflow: Address review comments. 2024-05-27 11:01:52 +02:00
Anders Schack-Mulligen
972b81bbd1 Util: Allow best-effort total orders with a reasonable fallback. 2024-05-27 11:01:52 +02:00
Anders Schack-Mulligen
486eaad566 Shared: Add MakeSets module. 2024-05-27 11:01:51 +02:00
Tom Hvitved
62b16c0fa3 Share getFileBySourceArchiveName implementation 2024-02-23 11:25:49 +01:00
Tom Hvitved
5a426d1800 Data flow: Use Boolean class 2023-12-14 09:04:16 +01:00
Joe Farebrother
aa418dc7d0 Add more line breaks in qldoc 2023-10-17 12:51:22 +01:00
Joe Farebrother
05162c68ec Fix typo 2023-10-16 13:43:06 +01:00