Commit Graph

9236 Commits

Author SHA1 Message Date
Taus
8ec414d454 Python: Add copy of internal Python 3 tests
Again, mostly extractor tests, and a single library test.
2024-04-15 12:30:01 +00:00
Taus
b22b4c5b85 Python: Add copy of internal Python 2 tests
These are mostly extractor tests, which now live in `2/extractor-tests`,
and a single library tests that was added to the existing directory.
2024-04-15 12:26:45 +00:00
Taus
e45f6e9b24 Python: Add copy of extractor tests
These get to live next to the existing library and query tests, and are
run as part of both the Python 2 and Python 3 language tests.
2024-04-15 12:22:07 +00:00
Tom Hvitved
e7dc120456 Add deprecation comments 2024-04-12 13:40:15 +02:00
Tom Hvitved
ceb5b4c56e Python: No longer use models-as-data CSV interface 2024-04-12 13:40:15 +02:00
Tom Hvitved
fdb77457b3 Sync files 2024-04-12 13:40:14 +02:00
Anders Schack-Mulligen
cbcdf1f8b5 Python: Update expected output (interesting). 2024-04-12 09:20:33 +02:00
Anders Schack-Mulligen
bfcfedab8c Python: Update expected output (uninteresting). 2024-04-12 09:20:30 +02:00
Anders Schack-Mulligen
a8fc100108 Python: Add alert provenance plumbing. 2024-04-12 09:20:08 +02:00
Anders Schack-Mulligen
eafc0075fd Legacy dataflow: Sync. 2024-04-12 09:19:54 +02:00
Sid Shankar
e33c5706f8 Modifies check for py launcher
This commit modifies the check for the "py" launcher on windows. We now look for the launcher only if the python_executable_name extractor option is not specified.
2024-04-11 12:59:41 -04:00
Rasmus Wriedt Larsen
c4e674b8d2 Merge pull request #16173 from RasmusWL/remove-lib-stubs
Python: Remove deprecated stubs for points-to tests
2024-04-10 17:12:16 +02:00
Rasmus Wriedt Larsen
9615e2ded9 Python: Remove deprecated stubs for points-to tests
I grep'ed through all our options files, and couldn't find any tests
that relies on these anymore 👍
2024-04-10 13:12:36 +02:00
Rasmus Wriedt Larsen
78ca691912 Python: remove deprecated points-to test for zope 2024-04-10 13:12:17 +02:00
Rasmus Wriedt Larsen
3db560158a Merge pull request #16169 from RasmusWL/mad-remoteflowsource
Python: Fix `RemoteFlowSourceFromCsv`
2024-04-10 13:06:42 +02:00
Rasmus Wriedt Larsen
4fed3cf12d Python: Fix RemoteFlowSourceFromCsv 2024-04-10 11:31:34 +02:00
Rasmus Wriedt Larsen
6f1a9d4574 Merge pull request #16159 from RasmusWL/fix-integration-tests
Python: Fixup integration tests after no dep inst
2024-04-09 15:08:20 +02:00
Rasmus Wriedt Larsen
6ce38be3cc Merge pull request #16112 from github/tausbn/python-various-extractor-fixups
Python: Various extractor fixups
2024-04-09 14:46:23 +02:00
Asger F
f5355cfa98 Dynamic: Sync ApiGraphModels.qll 2024-04-09 14:37:20 +02:00
Rasmus Wriedt Larsen
e9e7ccddce Python: delete force-enable-library-extraction integration test 2024-04-09 14:02:34 +02:00
Rasmus Wriedt Larsen
a0d6324f68 Python: Fix ignore-venv integration test
Now that we no longer support the fallback option
(https://github.com/github/codeql/pull/16127)
2024-04-09 14:01:10 +02:00
Rasmus Wriedt Larsen
bb4952f557 Revert "Python: Disable failing integration tests"
This reverts commit 8c2455fc11.
2024-04-09 14:00:25 +02:00
Taus
8c2455fc11 Python: Disable failing integration tests
These failures were likely caused by
https://github.com/github/codeql/pull/16127

My guess is that they can probably be deleted altogether, but as the
failures are blocking other development, I have opted to simply disable
them for the time being.
2024-04-09 10:49:30 +00:00
yoff
1048cf7c5e Merge pull request #15711 from RasmusWL/tt-content
Python: Add type tracking for content
2024-04-09 10:37:43 +02:00
Sylwia Budzynska
5d946586b8 Add tests 2024-04-08 15:39:54 +02:00
Sylwia Budzynska
112992585a Add change note 2024-04-05 14:56:06 +02:00
Sylwia Budzynska
84d69566c9 Fix decorator QLdoc 2024-04-05 14:51:30 +02:00
Sylwia Budzynska
ca7789d73c Fix QLdoc 2024-04-05 14:40:17 +02:00
Sylwia Budzynska
bed0d5678d Add Gradio models 2024-04-05 14:14:21 +02:00
Taus
ef9f99b3be Python: Remove unparse.py 2024-04-05 12:30:40 +02:00
Taus
599f573a4a Python: Preserve comments and docstrings in extractor 2024-04-05 12:30:40 +02:00
Taus
752d28c1b9 Python: Update repinning instructions
This aligns us better with the corresponding instructions for
the Ruby extractor.
2024-04-05 12:30:40 +02:00
Taus
7bec41096c Python: Rename tsg-build target to tsp-build
The latter makes more sense, as it's actually building
`tree-sitter-python`.
2024-04-05 12:30:40 +02:00
Rasmus Wriedt Larsen
4faff83aa0 Python: Extractor: Remove dependency installation fallback 2024-04-04 16:49:55 +02:00
Tom Hvitved
1dc13cc169 Merge pull request #15923 from hvitved/shared-xml-impl
Properly shared `XML.qll` implementation
2024-04-03 11:39:50 +02:00
Rasmus Wriedt Larsen
a22b9947c0 Python: Revert IterableSequenceNode as LocalSourceNode
When looking things over a bit more, we could actually exclude the steps
that would never be used instead. A much more involved solution, but
more performance oriented and clear in terms of what is supported (at
least until we start supporting type-tracking with more than depth 1
access-path, if that ever happens)
2024-04-02 16:51:00 +02:00
Rasmus Wriedt Larsen
8707a63edb Python: Add comments around storeStepCommon 2024-04-02 13:26:26 +02:00
Rasmus Wriedt Larsen
20202aba90 Python: Deprecate AttributeName 2024-04-02 13:21:46 +02:00
github-actions[bot]
8e61c6625b Post-release preparation for codeql-cli-2.17.0 2024-04-01 15:27:42 +00:00
github-actions[bot]
ec97d9a304 Release preparation for version 2.17.0 2024-04-01 13:46:57 +00:00
Henry Mercer
0646744928 Merge branch 'main' into henrymercer/merge-back-rc-3.13 2024-03-26 12:59:12 +00:00
github-actions[bot]
f67b5f9158 Post-release preparation for codeql-cli-2.16.6 2024-03-25 18:17:15 +00:00
github-actions[bot]
71ab804274 Release preparation for version 2.16.6 2024-03-25 16:58:08 +00:00
Rasmus Wriedt Larsen
d516db6abc Merge pull request #15903 from yoff/python/test-MaD-keyword-argument
Python: test MaD syntax for keyword argument
2024-03-25 15:51:49 +01:00
Rasmus Wriedt Larsen
69f6e1e263 Merge pull request #16010 from RasmusWL/perf
Python: Two small join-order fixes
2024-03-22 11:36:17 +01:00
yoff
c520cb6d58 Merge branch 'main' into python/test-MaD-keyword-argument 2024-03-22 10:56:08 +01:00
Rasmus Lerchedahl Petersen
eef60c9ad2 python: add test for "ReturnValue.TupleElement[0,1]"
also synchronise files
2024-03-22 10:54:12 +01:00
Arthur Baars
c219b1a3c7 Merge pull request #16013 from github/rc/3.13
Merge rc/3.13 into main
2024-03-21 16:04:58 +01:00
Rasmus Wriedt Larsen
93f940aa9c Python: Join-order improvement for DataFlowDispatch::TrackAttrReadInput
I was surprised to see that this predicate actually gets evaluated 3 times

- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@c15596yu was evaluated in 74 iterations totaling 165ms (delta sizes total: 113119).
- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@3459ejws was evaluated in 30 iterations totaling 76ms (delta sizes total: 32555).
- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@5ac22jwq was evaluated in 30 iterations totaling 108ms (delta sizes total: 32555).

It does however fit with it being used in exactly 3 places: https://github.com/search?q=repo%3Agithub%2Fcodeql+%2FattrReadTracker%5C%28%2F&type=code -- so I assume it's because each use forces a new evaluation. Although that's something we could look into solving, for now I'm just trying to fix the join-order.

Initial

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@3459ejws was evaluated in 30 iterations totaling 76ms (delta sizes total: 32555).
        7068090   ~0%    {2} r1 = SCAN Attributes::AttrRead#class#f6c3f431 OUTPUT In.0, In.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
        3901178   ~5%    {2}    | SCAN OUTPUT In.1, In.1
        3901178   ~0%    {3}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97` ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1

          13615   ~1%    {2} r2 = JOIN r1 WITH `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

             94   ~2%    {2} r3 = JOIN r1 WITH `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          18846   ~1%    {2} r4 = JOIN r1 WITH `DataFlowDispatch::classInstanceTracker/1#d73ecef4#prev_delta_1#join_rhs` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          32555   ~1%    {2} r5 = r2 UNION r3 UNION r4
                         return r5
```

==>

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@f2517jwq was evaluated in 30 iterations totaling 12ms (delta sizes total: 32704).
        186719  ~121%    {1} r1 = SCAN `DataFlowDispatch::classInstanceTracker/1#d73ecef4#prev_delta` OUTPUT In.1

        164342  ~158%    {1} r2 = SCAN `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` OUTPUT In.0

            96    ~0%    {1} r3 = SCAN `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` OUTPUT In.0

        351157   ~80%    {1} r4 = r1 UNION r2 UNION r3
         88074   ~14%    {1}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1
         41789   ~18%    {2}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.0, Lhs.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
         32883    ~2%    {2}    | SCAN OUTPUT In.1, In.1
                         return r4
```

AND

initial

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@c15596yu was evaluated in 74 iterations totaling 165ms (delta sizes total: 113119).
        17434622   ~0%    {2} r1 = SCAN Attributes::AttrRead#class#f6c3f431 OUTPUT In.0, In.0
                          {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
         9483976   ~4%    {2}    | SCAN OUTPUT In.1, In.1
         9483976   ~0%    {3}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97` ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1

           19258   ~1%    {2} r2 = JOIN r1 WITH `DataFlowDispatch::classInstanceTracker/1#d73ecef4#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

            1654   ~1%    {2} r3 = JOIN r1 WITH `DataFlowDispatch::superCallNoArgumentTracker/1#0a2e8a06#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

            1314   ~4%    {2} r4 = JOIN r1 WITH `DataFlowDispatch::clsArgumentTracker/1#47339327#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

              94   ~2%    {2} r5 = JOIN r1 WITH `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

           77217   ~0%    {2} r6 = JOIN r1 WITH `DataFlowDispatch::selfTracker/1#f157aa27#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

           13632   ~1%    {2} r7 = JOIN r1 WITH `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          113169   ~0%    {2} r8 = r2 UNION r3 UNION r4 UNION r5 UNION r6 UNION r7
                          return r8
```
==>

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@d732e6yt was evaluated in 74 iterations totaling 31ms (delta sizes total: 113129).
        186719  ~150%    {1} r1 = SCAN `DataFlowDispatch::classInstanceTracker/1#d73ecef4#reorder_1_0#prev_delta` OUTPUT In.0

          1669    ~0%    {1} r2 = SCAN `DataFlowDispatch::superCallNoArgumentTracker/1#0a2e8a06#reorder_1_0#prev_delta` OUTPUT In.0

          3425   ~15%    {1} r3 = SCAN `DataFlowDispatch::clsArgumentTracker/1#47339327#prev_delta` OUTPUT In.1

            96    ~0%    {1} r4 = SCAN `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` OUTPUT In.0

        123310    ~0%    {1} r5 = SCAN `DataFlowDispatch::selfTracker/1#f157aa27#reorder_1_0#prev_delta` OUTPUT In.0

        164342  ~581%    {1} r6 = SCAN `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` OUTPUT In.0

        479561   ~94%    {1} r7 = r1 UNION r2 UNION r3 UNION r4 UNION r5 UNION r6
        169424    ~2%    {1}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1
        116290    ~0%    {2}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.0, Lhs.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
        113160    ~0%    {2}    | SCAN OUTPUT In.1, In.1
                         return r7
```
2024-03-21 15:55:58 +01:00
Rasmus Wriedt Larsen
bfa8515b28 Python: Apply suggestions from code review
Co-authored-by: Anders Schack-Mulligen <aschackmull@users.noreply.github.com>
2024-03-21 14:51:45 +01:00