Commit Graph

8437 Commits

Author SHA1 Message Date
Taus
bf8307f78d Python: Fix flags test
In 25cb52aa52 I accidentally introduced an extra `not` where none existed before. Whoops!
2024-04-16 21:17:34 +02:00
Taus
48e367cb11 Python: Remove test with CRLF line endings
These were causing `git` to behave strangely, leaving files that were
impossible to reset. In the future we should probably generate these
problematic test files on the fly, so that they don't have to exist in
the repo, but in the short run, it's easier to just remove them so as to
not block other users of the repo.
2024-04-16 17:18:20 +00:00
Cornelius Riemenschneider
4cb0695646 Merge pull request #16229 from github/criemen/rules-pkg
Upgrade rules_pkg to 0.10.1.
2024-04-16 18:03:00 +02:00
Taus
c647f30576 Merge pull request #16212 from github/tausbn/python-add-copy-of-internal-tests
Python: Add copy of internal tests
2024-04-16 16:32:31 +02:00
Cornelius Riemenschneider
6ba27dc863 Upgrade rules_pkg to 0.10.1. 2024-04-16 16:29:56 +02:00
Taus
82057e2e46 Python: Autoformat tests
I guess these were never considered back when we switched to
autoformatting everything.
2024-04-15 15:27:21 +00:00
Taus
25cb52aa52 Python: Fix inefficient string comparison 2024-04-15 15:23:07 +00:00
Taus
8ec414d454 Python: Add copy of internal Python 3 tests
Again, mostly extractor tests, and a single library test.
2024-04-15 12:30:01 +00:00
Taus
b22b4c5b85 Python: Add copy of internal Python 2 tests
These are mostly extractor tests, which now live in `2/extractor-tests`,
and a single library tests that was added to the existing directory.
2024-04-15 12:26:45 +00:00
Taus
e45f6e9b24 Python: Add copy of extractor tests
These get to live next to the existing library and query tests, and are
run as part of both the Python 2 and Python 3 language tests.
2024-04-15 12:22:07 +00:00
Tom Hvitved
e7dc120456 Add deprecation comments 2024-04-12 13:40:15 +02:00
Tom Hvitved
ceb5b4c56e Python: No longer use models-as-data CSV interface 2024-04-12 13:40:15 +02:00
Tom Hvitved
fdb77457b3 Sync files 2024-04-12 13:40:14 +02:00
Anders Schack-Mulligen
cbcdf1f8b5 Python: Update expected output (interesting). 2024-04-12 09:20:33 +02:00
Anders Schack-Mulligen
bfcfedab8c Python: Update expected output (uninteresting). 2024-04-12 09:20:30 +02:00
Anders Schack-Mulligen
a8fc100108 Python: Add alert provenance plumbing. 2024-04-12 09:20:08 +02:00
Anders Schack-Mulligen
eafc0075fd Legacy dataflow: Sync. 2024-04-12 09:19:54 +02:00
Sid Shankar
e33c5706f8 Modifies check for py launcher
This commit modifies the check for the "py" launcher on windows. We now look for the launcher only if the python_executable_name extractor option is not specified.
2024-04-11 12:59:41 -04:00
Rasmus Wriedt Larsen
c4e674b8d2 Merge pull request #16173 from RasmusWL/remove-lib-stubs
Python: Remove deprecated stubs for points-to tests
2024-04-10 17:12:16 +02:00
Rasmus Wriedt Larsen
9615e2ded9 Python: Remove deprecated stubs for points-to tests
I grep'ed through all our options files, and couldn't find any tests
that relies on these anymore 👍
2024-04-10 13:12:36 +02:00
Rasmus Wriedt Larsen
78ca691912 Python: remove deprecated points-to test for zope 2024-04-10 13:12:17 +02:00
Rasmus Wriedt Larsen
3db560158a Merge pull request #16169 from RasmusWL/mad-remoteflowsource
Python: Fix `RemoteFlowSourceFromCsv`
2024-04-10 13:06:42 +02:00
Rasmus Wriedt Larsen
4fed3cf12d Python: Fix RemoteFlowSourceFromCsv 2024-04-10 11:31:34 +02:00
Rasmus Wriedt Larsen
6f1a9d4574 Merge pull request #16159 from RasmusWL/fix-integration-tests
Python: Fixup integration tests after no dep inst
2024-04-09 15:08:20 +02:00
Rasmus Wriedt Larsen
6ce38be3cc Merge pull request #16112 from github/tausbn/python-various-extractor-fixups
Python: Various extractor fixups
2024-04-09 14:46:23 +02:00
Rasmus Wriedt Larsen
e9e7ccddce Python: delete force-enable-library-extraction integration test 2024-04-09 14:02:34 +02:00
Rasmus Wriedt Larsen
a0d6324f68 Python: Fix ignore-venv integration test
Now that we no longer support the fallback option
(https://github.com/github/codeql/pull/16127)
2024-04-09 14:01:10 +02:00
Rasmus Wriedt Larsen
bb4952f557 Revert "Python: Disable failing integration tests"
This reverts commit 8c2455fc11.
2024-04-09 14:00:25 +02:00
Taus
8c2455fc11 Python: Disable failing integration tests
These failures were likely caused by
https://github.com/github/codeql/pull/16127

My guess is that they can probably be deleted altogether, but as the
failures are blocking other development, I have opted to simply disable
them for the time being.
2024-04-09 10:49:30 +00:00
yoff
1048cf7c5e Merge pull request #15711 from RasmusWL/tt-content
Python: Add type tracking for content
2024-04-09 10:37:43 +02:00
Taus
ef9f99b3be Python: Remove unparse.py 2024-04-05 12:30:40 +02:00
Taus
599f573a4a Python: Preserve comments and docstrings in extractor 2024-04-05 12:30:40 +02:00
Taus
752d28c1b9 Python: Update repinning instructions
This aligns us better with the corresponding instructions for
the Ruby extractor.
2024-04-05 12:30:40 +02:00
Taus
7bec41096c Python: Rename tsg-build target to tsp-build
The latter makes more sense, as it's actually building
`tree-sitter-python`.
2024-04-05 12:30:40 +02:00
Rasmus Wriedt Larsen
4faff83aa0 Python: Extractor: Remove dependency installation fallback 2024-04-04 16:49:55 +02:00
Tom Hvitved
1dc13cc169 Merge pull request #15923 from hvitved/shared-xml-impl
Properly shared `XML.qll` implementation
2024-04-03 11:39:50 +02:00
Rasmus Wriedt Larsen
a22b9947c0 Python: Revert IterableSequenceNode as LocalSourceNode
When looking things over a bit more, we could actually exclude the steps
that would never be used instead. A much more involved solution, but
more performance oriented and clear in terms of what is supported (at
least until we start supporting type-tracking with more than depth 1
access-path, if that ever happens)
2024-04-02 16:51:00 +02:00
Rasmus Wriedt Larsen
8707a63edb Python: Add comments around storeStepCommon 2024-04-02 13:26:26 +02:00
Rasmus Wriedt Larsen
20202aba90 Python: Deprecate AttributeName 2024-04-02 13:21:46 +02:00
github-actions[bot]
8e61c6625b Post-release preparation for codeql-cli-2.17.0 2024-04-01 15:27:42 +00:00
github-actions[bot]
ec97d9a304 Release preparation for version 2.17.0 2024-04-01 13:46:57 +00:00
Henry Mercer
0646744928 Merge branch 'main' into henrymercer/merge-back-rc-3.13 2024-03-26 12:59:12 +00:00
github-actions[bot]
f67b5f9158 Post-release preparation for codeql-cli-2.16.6 2024-03-25 18:17:15 +00:00
github-actions[bot]
71ab804274 Release preparation for version 2.16.6 2024-03-25 16:58:08 +00:00
Rasmus Wriedt Larsen
d516db6abc Merge pull request #15903 from yoff/python/test-MaD-keyword-argument
Python: test MaD syntax for keyword argument
2024-03-25 15:51:49 +01:00
Rasmus Wriedt Larsen
69f6e1e263 Merge pull request #16010 from RasmusWL/perf
Python: Two small join-order fixes
2024-03-22 11:36:17 +01:00
yoff
c520cb6d58 Merge branch 'main' into python/test-MaD-keyword-argument 2024-03-22 10:56:08 +01:00
Rasmus Lerchedahl Petersen
eef60c9ad2 python: add test for "ReturnValue.TupleElement[0,1]"
also synchronise files
2024-03-22 10:54:12 +01:00
Arthur Baars
c219b1a3c7 Merge pull request #16013 from github/rc/3.13
Merge rc/3.13 into main
2024-03-21 16:04:58 +01:00
Rasmus Wriedt Larsen
93f940aa9c Python: Join-order improvement for DataFlowDispatch::TrackAttrReadInput
I was surprised to see that this predicate actually gets evaluated 3 times

- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@c15596yu was evaluated in 74 iterations totaling 165ms (delta sizes total: 113119).
- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@3459ejws was evaluated in 30 iterations totaling 76ms (delta sizes total: 32555).
- Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@5ac22jwq was evaluated in 30 iterations totaling 108ms (delta sizes total: 32555).

It does however fit with it being used in exactly 3 places: https://github.com/search?q=repo%3Agithub%2Fcodeql+%2FattrReadTracker%5C%28%2F&type=code -- so I assume it's because each use forces a new evaluation. Although that's something we could look into solving, for now I'm just trying to fix the join-order.

Initial

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@3459ejws was evaluated in 30 iterations totaling 76ms (delta sizes total: 32555).
        7068090   ~0%    {2} r1 = SCAN Attributes::AttrRead#class#f6c3f431 OUTPUT In.0, In.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
        3901178   ~5%    {2}    | SCAN OUTPUT In.1, In.1
        3901178   ~0%    {3}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97` ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1

          13615   ~1%    {2} r2 = JOIN r1 WITH `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

             94   ~2%    {2} r3 = JOIN r1 WITH `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          18846   ~1%    {2} r4 = JOIN r1 WITH `DataFlowDispatch::classInstanceTracker/1#d73ecef4#prev_delta_1#join_rhs` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          32555   ~1%    {2} r5 = r2 UNION r3 UNION r4
                         return r5
```

==>

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@f2517jwq was evaluated in 30 iterations totaling 12ms (delta sizes total: 32704).
        186719  ~121%    {1} r1 = SCAN `DataFlowDispatch::classInstanceTracker/1#d73ecef4#prev_delta` OUTPUT In.1

        164342  ~158%    {1} r2 = SCAN `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` OUTPUT In.0

            96    ~0%    {1} r3 = SCAN `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` OUTPUT In.0

        351157   ~80%    {1} r4 = r1 UNION r2 UNION r3
         88074   ~14%    {1}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1
         41789   ~18%    {2}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.0, Lhs.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
         32883    ~2%    {2}    | SCAN OUTPUT In.1, In.1
                         return r4
```

AND

initial

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@c15596yu was evaluated in 74 iterations totaling 165ms (delta sizes total: 113119).
        17434622   ~0%    {2} r1 = SCAN Attributes::AttrRead#class#f6c3f431 OUTPUT In.0, In.0
                          {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
         9483976   ~4%    {2}    | SCAN OUTPUT In.1, In.1
         9483976   ~0%    {3}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97` ON FIRST 1 OUTPUT Rhs.1, Lhs.0, Lhs.1

           19258   ~1%    {2} r2 = JOIN r1 WITH `DataFlowDispatch::classInstanceTracker/1#d73ecef4#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

            1654   ~1%    {2} r3 = JOIN r1 WITH `DataFlowDispatch::superCallNoArgumentTracker/1#0a2e8a06#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

            1314   ~4%    {2} r4 = JOIN r1 WITH `DataFlowDispatch::clsArgumentTracker/1#47339327#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

              94   ~2%    {2} r5 = JOIN r1 WITH `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

           77217   ~0%    {2} r6 = JOIN r1 WITH `DataFlowDispatch::selfTracker/1#f157aa27#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

           13632   ~1%    {2} r7 = JOIN r1 WITH `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` ON FIRST 1 OUTPUT Lhs.1, Lhs.2

          113169   ~0%    {2} r8 = r2 UNION r3 UNION r4 UNION r5 UNION r6 UNION r7
                          return r8
```
==>

```
Pipeline standard for DataFlowDispatch::TrackAttrReadInput::start/2#67f26627@d732e6yt was evaluated in 74 iterations totaling 31ms (delta sizes total: 113129).
        186719  ~150%    {1} r1 = SCAN `DataFlowDispatch::classInstanceTracker/1#d73ecef4#reorder_1_0#prev_delta` OUTPUT In.0

          1669    ~0%    {1} r2 = SCAN `DataFlowDispatch::superCallNoArgumentTracker/1#0a2e8a06#reorder_1_0#prev_delta` OUTPUT In.0

          3425   ~15%    {1} r3 = SCAN `DataFlowDispatch::clsArgumentTracker/1#47339327#prev_delta` OUTPUT In.1

            96    ~0%    {1} r4 = SCAN `DataFlowDispatch::superCallTwoArgumentTracker/2#d18be99f#reorder_2_0_1#prev_delta` OUTPUT In.0

        123310    ~0%    {1} r5 = SCAN `DataFlowDispatch::selfTracker/1#f157aa27#reorder_1_0#prev_delta` OUTPUT In.0

        164342  ~581%    {1} r6 = SCAN `DataFlowDispatch::classTracker/1#d11f2237#reorder_1_0#prev_delta` OUTPUT In.0

        479561   ~94%    {1} r7 = r1 UNION r2 UNION r3 UNION r4 UNION r5 UNION r6
        169424    ~2%    {1}    | JOIN WITH `Attributes::AttrRef.getObject/0#dispred#d7cd0a97_10#join_rhs` ON FIRST 1 OUTPUT Rhs.1
        116290    ~0%    {2}    | JOIN WITH Attributes::AttrRead#class#f6c3f431 ON FIRST 1 OUTPUT Lhs.0, Lhs.0
                         {2}    | AND NOT `DataFlowDispatch::TrackAttrReadInput::start/2#67f26627#prev`(FIRST 2)
        113160    ~0%    {2}    | SCAN OUTPUT In.1, In.1
                         return r7
```
2024-03-21 15:55:58 +01:00