Commit Graph

2719 Commits

Author SHA1 Message Date
Nick Rolfe
003d208574 JS: do fewer regexp matches in SensitiveActions 2024-04-23 15:31:38 +01:00
Anders Schack-Mulligen
b2f09949df Merge pull request #15599 from aschackmull/dataflow/fieldflowbranchlimit-v2
Dataflow: update fieldFlowBranchLimit semantics
2024-04-23 10:08:05 +02:00
Rasmus Wriedt Larsen
e0e405bb31 Python: replace dataflow-test location in files 2024-04-23 09:40:59 +02:00
Joe Farebrother
f85ee38e04 Add instance taint steps for requests 2024-04-22 16:03:39 +01:00
Joe Farebrother
88e3227ed0 Add pyramid models 2024-04-22 13:27:18 +01:00
Taus
81246cd41a Python: Add missing QLDoc for isUnicode 2024-04-22 12:08:53 +00:00
Taus
bab461ffd1 Python: Add change note 2024-04-22 12:00:09 +00:00
Taus
d51fcd4f2a Python: Change Str to StringLiteral
As far as I can tell, this was the only occurrence of `Str` as a type
throughout the entire library.
2024-04-22 12:00:09 +00:00
Taus
b484aee39e Python: Autoformat everything
Of course, `StringLiteral` being much longer than `StrConst` meant a
bunch of files changed formatting.
2024-04-22 12:00:09 +00:00
Taus
1c68c987b0 Python: Change all remaining occurrences of StrConst
Done using
```
git grep StrConst | xargs sed -i 's/StrConst/StringLiteral/g'
```
2024-04-22 12:00:09 +00:00
Taus
f6487d7b13 Python: Rename StrConst to StringLiteral
Does a few things:
- Renames `StrConst` to `StringLiteral`, and deprecates the former.
- Also deprecates `Str`.
- Adds an override of `StringLiteral::toString` making it output
`"StringLiteral"` rather than the inherited `"Str"`. This ensures that
the AST viewer shows these nodes as the former type, not the latter.

There are a large number of uses of `StrConst` in the codebase. These
will be fixed in a later commit.
2024-04-22 12:00:09 +00:00
Asger F
decd576a6b Merge pull request #15386 from asgerf/js/graph-export
JS: Add library for exporting graphs as type models
2024-04-18 11:56:17 +02:00
Alexander Eyers-Taylor
da3fa22cbd Merge pull request #16228 from github/post-release-prep/codeql-cli-2.17.1
Post-release preparation for codeql-cli-2.17.1
2024-04-17 11:24:34 +01:00
Asger F
3335d48154 Sync files 2024-04-16 20:26:41 +02:00
Asger F
be64daf265 Merge branch 'main' into js/graph-export 2024-04-16 20:23:33 +02:00
Cornelius Riemenschneider
6ba27dc863 Upgrade rules_pkg to 0.10.1. 2024-04-16 16:29:56 +02:00
github-actions[bot]
622e176a16 Post-release preparation for codeql-cli-2.17.1 2024-04-16 14:21:32 +00:00
github-actions[bot]
9bfe4ea90a Release preparation for version 2.17.1 2024-04-15 17:34:47 +00:00
Anders Schack-Mulligen
2f0987e980 Dataflow: Add dummy DataFlowSecondLevelScope implementations.
These could be an empty type, but Unit was available and it probably
doesn't matter.
2024-04-15 15:16:30 +02:00
Tom Hvitved
e7dc120456 Add deprecation comments 2024-04-12 13:40:15 +02:00
Tom Hvitved
ceb5b4c56e Python: No longer use models-as-data CSV interface 2024-04-12 13:40:15 +02:00
Tom Hvitved
fdb77457b3 Sync files 2024-04-12 13:40:14 +02:00
Anders Schack-Mulligen
a8fc100108 Python: Add alert provenance plumbing. 2024-04-12 09:20:08 +02:00
Anders Schack-Mulligen
eafc0075fd Legacy dataflow: Sync. 2024-04-12 09:19:54 +02:00
Rasmus Wriedt Larsen
3db560158a Merge pull request #16169 from RasmusWL/mad-remoteflowsource
Python: Fix `RemoteFlowSourceFromCsv`
2024-04-10 13:06:42 +02:00
Rasmus Wriedt Larsen
4fed3cf12d Python: Fix RemoteFlowSourceFromCsv 2024-04-10 11:31:34 +02:00
Asger F
f5355cfa98 Dynamic: Sync ApiGraphModels.qll 2024-04-09 14:37:20 +02:00
yoff
1048cf7c5e Merge pull request #15711 from RasmusWL/tt-content
Python: Add type tracking for content
2024-04-09 10:37:43 +02:00
Josh Brown
30bf0a7c8b manual merge 2024-04-05 15:30:56 -07:00
Sylwia Budzynska
112992585a Add change note 2024-04-05 14:56:06 +02:00
Sylwia Budzynska
84d69566c9 Fix decorator QLdoc 2024-04-05 14:51:30 +02:00
Sylwia Budzynska
ca7789d73c Fix QLdoc 2024-04-05 14:40:17 +02:00
Sylwia Budzynska
bed0d5678d Add Gradio models 2024-04-05 14:14:21 +02:00
Tom Hvitved
1dc13cc169 Merge pull request #15923 from hvitved/shared-xml-impl
Properly shared `XML.qll` implementation
2024-04-03 11:39:50 +02:00
github-actions[bot]
c511de2eae Release preparation for version 2.17.0 2024-04-02 17:15:41 +00:00
Chuan-kai Lin
6c649c898e Revert "Release preparation for version 2.17.0" 2024-04-02 10:06:20 -07:00
Rasmus Wriedt Larsen
a22b9947c0 Python: Revert IterableSequenceNode as LocalSourceNode
When looking things over a bit more, we could actually exclude the steps
that would never be used instead. A much more involved solution, but
more performance oriented and clear in terms of what is supported (at
least until we start supporting type-tracking with more than depth 1
access-path, if that ever happens)
2024-04-02 16:51:00 +02:00
Rasmus Wriedt Larsen
8707a63edb Python: Add comments around storeStepCommon 2024-04-02 13:26:26 +02:00
Rasmus Wriedt Larsen
20202aba90 Python: Deprecate AttributeName 2024-04-02 13:21:46 +02:00
github-actions[bot]
8e61c6625b Post-release preparation for codeql-cli-2.17.0 2024-04-01 15:27:42 +00:00
github-actions[bot]
ec97d9a304 Release preparation for version 2.17.0 2024-04-01 13:46:57 +00:00
Dilan
3325cb9ec6 Merge tag 'codeql-cli/latest'
Compatible with the latest released version of the CodeQL CLI
2024-03-26 12:59:35 +00:00
Henry Mercer
0646744928 Merge branch 'main' into henrymercer/merge-back-rc-3.13 2024-03-26 12:59:12 +00:00
github-actions[bot]
f67b5f9158 Post-release preparation for codeql-cli-2.16.6 2024-03-25 18:17:15 +00:00
github-actions[bot]
71ab804274 Release preparation for version 2.16.6 2024-03-25 16:58:08 +00:00
Rasmus Wriedt Larsen
69f6e1e263 Merge pull request #16010 from RasmusWL/perf
Python: Two small join-order fixes
2024-03-22 11:36:17 +01:00
Arthur Baars
c219b1a3c7 Merge pull request #16013 from github/rc/3.13
Merge rc/3.13 into main
2024-03-21 16:04:58 +01:00
Rasmus Wriedt Larsen
93f940aa9c Python: Join-order improvement for DataFlowDispatch::TrackAttrReadInput
I was surprised to see that this predicate actually gets evaluated 3 times

- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@c15596yu was evaluated in 74 iterations totaling 165ms (delta sizes total: 113119).
- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@3459ejws was evaluated in 30 iterations totaling 76ms (delta sizes total: 32555).
- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@5ac22jwq was evaluated in 30 iterations totaling 108ms (delta sizes total: 32555).

It does however fit with it being used in exactly 3 places: https://github.com/search?q=repo%3Agithub%2Fcodeql+%2FattrReadTracker%5C%28%2F&type=code -- so I assume it's because each use forces a new evaluation. Although that's something we could look into solving, for now I'm just trying to fix the join-order.

Initial

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@3459ejws was evaluated in 30 iterations totaling 76ms (delta sizes total: 32555).
        7068090   ~0%    {2} r1 = SCAN Attributes::AttrRead#class#f6c3f431 OUTPUT In.0, In.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
        3901178   ~5%    {2}    | SCAN OUTPUT In.1, In.1
        3901178   ~0%    {3}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97` ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1

          13615   ~1%    {2} r2 = JOIN r1 WITH `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

             94   ~2%    {2} r3 = JOIN r1 WITH `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          18846   ~1%    {2} r4 = JOIN r1 WITH `DataFlowDispatch::classInstanceTracker/1#d73ecef4#prev_delta_1#join_rhs` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          32555   ~1%    {2} r5 = r2 UNION r3 UNION r4
                         return r5
```

==>

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@f2517jwq was evaluated in 30 iterations totaling 12ms (delta sizes total: 32704).
        186719  ~121%    {1} r1 = SCAN `DataFlowDispatch::classInstanceTracker/1#d73ecef4#prev_delta` OUTPUT In.1

        164342  ~158%    {1} r2 = SCAN `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` OUTPUT In.0

            96    ~0%    {1} r3 = SCAN `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` OUTPUT In.0

        351157   ~80%    {1} r4 = r1 UNION r2 UNION r3
         88074   ~14%    {1}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1
         41789   ~18%    {2}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.0, Lhs.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
         32883    ~2%    {2}    | SCAN OUTPUT In.1, In.1
                         return r4
```

AND

initial

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@c15596yu was evaluated in 74 iterations totaling 165ms (delta sizes total: 113119).
        17434622   ~0%    {2} r1 = SCAN Attributes::AttrRead#class#f6c3f431 OUTPUT In.0, In.0
                          {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
         9483976   ~4%    {2}    | SCAN OUTPUT In.1, In.1
         9483976   ~0%    {3}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97` ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1

           19258   ~1%    {2} r2 = JOIN r1 WITH `DataFlowDispatch::classInstanceTracker/1#d73ecef4#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

            1654   ~1%    {2} r3 = JOIN r1 WITH `DataFlowDispatch::superCallNoArgumentTracker/1#0a2e8a06#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

            1314   ~4%    {2} r4 = JOIN r1 WITH `DataFlowDispatch::clsArgumentTracker/1#47339327#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

              94   ~2%    {2} r5 = JOIN r1 WITH `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

           77217   ~0%    {2} r6 = JOIN r1 WITH `DataFlowDispatch::selfTracker/1#f157aa27#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

           13632   ~1%    {2} r7 = JOIN r1 WITH `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          113169   ~0%    {2} r8 = r2 UNION r3 UNION r4 UNION r5 UNION r6 UNION r7
                          return r8
```
==>

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@d732e6yt was evaluated in 74 iterations totaling 31ms (delta sizes total: 113129).
        186719  ~150%    {1} r1 = SCAN `DataFlowDispatch::classInstanceTracker/1#d73ecef4#reorder_1_0#prev_delta` OUTPUT In.0

          1669    ~0%    {1} r2 = SCAN `DataFlowDispatch::superCallNoArgumentTracker/1#0a2e8a06#reorder_1_0#prev_delta` OUTPUT In.0

          3425   ~15%    {1} r3 = SCAN `DataFlowDispatch::clsArgumentTracker/1#47339327#prev_delta` OUTPUT In.1

            96    ~0%    {1} r4 = SCAN `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` OUTPUT In.0

        123310    ~0%    {1} r5 = SCAN `DataFlowDispatch::selfTracker/1#f157aa27#reorder_1_0#prev_delta` OUTPUT In.0

        164342  ~581%    {1} r6 = SCAN `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` OUTPUT In.0

        479561   ~94%    {1} r7 = r1 UNION r2 UNION r3 UNION r4 UNION r5 UNION r6
        169424    ~2%    {1}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1
        116290    ~0%    {2}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.0, Lhs.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
        113160    ~0%    {2}    | SCAN OUTPUT In.1, In.1
                         return r7
```
2024-03-21 15:55:58 +01:00
Rasmus Wriedt Larsen
bfa8515b28 Python: Apply suggestions from code review
Co-authored-by: Anders Schack-Mulligen <aschackmull@users.noreply.github.com>
2024-03-21 14:51:45 +01:00
Dilan
955fd2cc5a Merge tag 'codeql-cli/latest'
Compatible with the latest released version of the CodeQL CLI
2024-03-21 13:00:21 +00:00