Commit Graph

2899 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
5bee44dcfe Python: add change-note 2023-11-07 11:27:11 +01:00
Rasmus Wriedt Larsen
6568332e3d Python: Add basic flow for class attributes 2023-11-07 11:23:42 +01:00
Geoffrey White
e8a466a02c Update dead link. 2023-11-07 09:26:07 +00:00
amammad
ad756d59c8 put new frameworks in Frameworks.qll and fix some mistakes of Baize 2023-11-06 19:17:50 +01:00
amammad
637c52d10a separate each new FileSystemAccess packages. 2023-11-06 19:03:55 +01:00
Taus
f67c68da9a Python: Make TypeParameter extend AstNode
With `AstNode` defined as a union of other classes, we don't get this for free.

(Compare with `DictItem`, which is in a similar situation.)
2023-11-06 13:50:55 +00:00
Taus
878299823c Python: Add up-/downgrade scripts
In the upgrade direction, we simply do nothing.

In the downgrade direction, we remove the two new relations, and
also any `Stmt` nodes corresponding to `TypeAlias` nodes.
2023-11-06 13:50:55 +00:00
Taus
9cd1e0e546 Python: Add stats for new relations 2023-11-06 13:50:55 +00:00
Taus
e8209a6a10 Python: Fix missing override compilation error 2023-11-06 13:50:55 +00:00
Taus
2e77b8d3c2 Python: Add wrapper classes around the newly added AST nodes 2023-11-06 13:50:55 +00:00
Taus
c397f707a1 Python: Add automatically generated files
For these, I opted for a placement that would cause as few changes to the
dbscheme as possible. This puts the new `type_parameters` fields as the
last field on function and class definitions.
2023-11-06 13:50:55 +00:00
Rasmus Wriedt Larsen
f6ae7523b5 Merge pull request #14591 from RasmusWL/minor-stringpool-improvement
Python: Minor cleanup for string pool interaction
2023-11-06 13:13:27 +01:00
Rasmus Wriedt Larsen
92b13c4259 Merge branch 'main' into amammad-python-FileSystemAccess 2023-11-06 11:30:09 +01:00
yoff
fd757b0089 Update python/ql/lib/semmle/python/dataflow/new/internal/DataFlowPrivate.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2023-11-02 09:31:28 +01:00
Rasmus Lerchedahl Petersen
58bf70d61b Python: filter self steps from use-use flow
Factor out use-use flow in order to do this.
Also improve names and comments.

I also wanted to change the types in `difinitionFlowStep`, but
that broke the module instantiation.
2023-11-02 09:31:28 +01:00
yoff
c26c68c286 Merge pull request #14617 from yoff/python/module-for-import-time-flow
Python: module for import time flow
2023-11-02 09:28:51 +01:00
Tom Hvitved
3c86aad16d Merge pull request #14628 from hvitved/ruby/type-tracking-store-post-update
Ruby: Summarized type-tracking stores should target post-update nodes
2023-11-01 13:54:21 +01:00
Tom Hvitved
0c5b528d54 Address review comments 2023-11-01 11:32:57 +01:00
Rasmus Lerchedahl Petersen
f7a8a8ae19 Python: Fix QL alert 2023-11-01 09:24:59 +01:00
Rasmus Lerchedahl Petersen
38b811b050 Python: Separate -> PhaseDependentFlow 2023-10-31 21:50:33 +01:00
Rasmus Lerchedahl Petersen
7f6ae8b2ab Python: improve readability 2023-10-31 14:52:37 +01:00
yoff
f76cde36bb Apply suggestions from code review
Co-authored-by: Taus <tausbn@github.com>
2023-10-31 14:12:24 +01:00
Chris Smowton
79e1aa0498 Merge pull request #14634 from github/post-release-prep/codeql-cli-2.15.2
Post-release preparation for codeql-cli-2.15.2
2023-10-31 10:24:53 +00:00
github-actions[bot]
2b939fdf08 Post-release preparation for codeql-cli-2.15.2 2023-10-30 16:06:51 +00:00
Harry Maclean
083be305e1 Shared: Add neutralModel extensible predicate
The neutralModel extensible predicate already exists in Java and C#, so
this change brings the dynamic languages more in line with static
languages. The Model Editor uses this predicate to mark endpoints as
"not interesting" from a data flow perspective.
2023-10-30 11:31:57 +00:00
github-actions[bot]
4641990021 Release preparation for version 2.15.2 2023-10-30 11:05:53 +00:00
Tom Hvitved
bf3fb685ad Python: Sync in changes to SummaryTypeTracker.qll 2023-10-30 10:52:07 +01:00
Rasmus Lerchedahl Petersen
1f8a51cffe Python: add module for post-update flow
The logic for adding flow out of post-update-nodes
was also duplicated, so I added a module for that also.

The two modules are now only used together. I did not
fuse them, though, as I think they may not be used
together in the case of use-use-flow.
2023-10-27 15:07:49 +02:00
Rasmus Lerchedahl Petersen
e745df6478 Python: module for import time flow
The logic for separating local flow into _import time_
and _runtime_ was duplicated a few times.
Create a module for it instead, and add a good qldoc.
2023-10-27 15:07:49 +02:00
yoff
867a39083e Merge pull request #14114 from yoff/python/allow-namespace-packages
Python: Allow namespace packages
2023-10-26 16:56:05 +02:00
Rasmus Lerchedahl Petersen
dcc778520a Python: refactor code
Also add explanatory comment.

Co-authored-by: Taus <tausbn@github.com>
2023-10-26 15:00:02 +02:00
Rasmus Lerchedahl Petersen
50041f07a3 Python: fix comment 2023-10-26 14:28:00 +02:00
Max Schaefer
3939167ba2 Include more details in the message for py/weak-cryptographic-algorithm.
Specifically, we add a link to the location where the cryptographic algorithm is configured, which can be far away from its use.
2023-10-26 11:28:09 +01:00
Rasmus Wriedt Larsen
be494b73d4 Python: Minor cleanup for string pool interaction
This doesn't really seem to impact performance, is just a bit cleaner way to write the QL code.

In the worst case, since we do string concat with empty string, we would need to load string + attempt to store new string in string pool, which is useless since we can surely just reuse the integer ID.

-- Previously

Evaluated non-recursive predicate ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0#join_rhs#CSE@5fdc6eia in 1ms (size: 2).
Evaluated relational algebra for predicate ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0#join_rhs#CSE@5fdc6eia with tuple counts:
         1   ~0%    {1} r1 = CONSTANT(unique string)[""]

         1   ~0%    {1} r2 = CONSTANT(unique string)[".__init__"]

         2   ~0%    {1} r3 = r1 UNION r2
                    return r3

Evaluated recursive predicate ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0@b4f54xlv in 72ms on iteration 1 (delta size: 26339).
Evaluated relational algebra for predicate ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0@b4f54xlv on iteration 1 running pipeline base with tuple counts:
         6382   ~0%    {3} r1 = JOIN ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0#join_rhs#CSE WITH Module#68efc075::Module::getFile#0#dispred#ff CARTESIAN PRODUCT OUTPUT Rhs.1, Lhs.0, Rhs.0
         6066   ~2%    {4} r2 = JOIN r1 WITH Module#68efc075::isPreferredModuleForName#2#ff ON FIRST 1 OUTPUT Lhs.2, _, Lhs.1, Rhs.1
                       {2} r3 = REWRITE r2 WITH Tmp.1 := "", Out.1 := InverseAppend(Tmp.1,In.2,In.3) KEEPING 2
         3721   ~0%    {2} r4 = SCAN r3 OUTPUT In.1, In.0
        25945   ~2%    {2} r5 = JOIN r4 WITH ImportResolution#f829f0f0::ImportResolution::getReferenceToModuleName#1#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1

           38   ~4%    {3} r6 = SCAN SsaDefinitions#9197156e::SsaSource::init_module_submodule_defn#2#ff OUTPUT In.1, _, In.0
           38   ~1%    {3} r7 = REWRITE r6 WITH Out.1 := -1
           38   ~1%    {2} r8 = JOIN r7 WITH py_scope_flow_021#join_rhs ON FIRST 2 OUTPUT Rhs.2, Lhs.2
           38   ~0%    {2} r9 = JOIN r8 WITH Module#68efc075::Module::getPackageName#0#dispred#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1
          394   ~2%    {2} r10 = JOIN r9 WITH Essa#24e22a14::EssaVariable::getSourceVariable#0#dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1
          394   ~0%    {3} r11 = JOIN r10 WITH DataFlowPublic#c4b6ce7e::EssaNode#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1, Rhs.1

          394   ~0%    {5} r12 = JOIN r11 WITH Essa#24e22a14::EssaVariable::getName#0#dispred#ff ON FIRST 1 OUTPUT _, Lhs.2, Lhs.1, Rhs.1, _
          394   ~0%    {2} r13 = REWRITE r12 WITH Tmp.0 := ".", Tmp.4 := "", Out.0 := (In.2 ++ Tmp.0 ++ In.3 ++ Tmp.4) KEEPING 2

          394   ~0%    {5} r14 = JOIN r11 WITH Essa#24e22a14::EssaVariable::getName#0#dispred#ff ON FIRST 1 OUTPUT _, Lhs.2, Lhs.1, Rhs.1, _
          394   ~1%    {2} r15 = REWRITE r14 WITH Tmp.0 := ".", Tmp.4 := ".__init__", Out.0 := (In.2 ++ Tmp.0 ++ In.3 ++ Tmp.4) KEEPING 2

          788   ~3%    {2} r16 = r13 UNION r15
          401   ~2%    {2} r17 = JOIN r16 WITH Module#68efc075::isPreferredModuleForName#2#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1
          394   ~1%    {2} r18 = JOIN r17 WITH Module#68efc075::Module::getFile#0#dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Rhs.1

        26339   ~1%    {2} r19 = r5 UNION r18
                       return r19

-- Now

Evaluated recursive predicate ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0@409a0xhe in 8ms on iteration 1 (delta size: 26339).
Evaluated relational algebra for predicate ImportResolution#f829f0f0::ImportResolution::getImmediateModuleReference#1#ff#reorder_1_0@409a0xhe on iteration 1 running pipeline base with tuple counts:
         3033   ~0%    {2} r1 = JOIN Module#68efc075::Module::getFile#0#dispred#ff_10#join_rhs WITH Module#68efc075::isPreferredModuleForName#2#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1

         3033   ~0%    {4} r2 = JOIN Module#68efc075::Module::getFile#0#dispred#ff_10#join_rhs WITH Module#68efc075::isPreferredModuleForName#2#ff ON FIRST 1 OUTPUT Lhs.1, _, Rhs.1, _
                       {2} r3 = REWRITE r2 WITH Tmp.1 := "", Tmp.3 := ".__init__", Out.1 := InverseAppend(Tmp.1,Tmp.3,In.2) KEEPING 2
          688   ~0%    {2} r4 = SCAN r3 OUTPUT In.1, In.0

         3721   ~0%    {2} r5 = r1 UNION r4
        25945   ~2%    {2} r6 = JOIN r5 WITH ImportResolution#f829f0f0::ImportResolution::getReferenceToModuleName#1#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1

           38   ~4%    {3} r7 = SCAN SsaDefinitions#9197156e::SsaSource::init_module_submodule_defn#2#ff OUTPUT In.1, _, In.0
           38   ~1%    {3} r8 = REWRITE r7 WITH Out.1 := -1
           38   ~1%    {2} r9 = JOIN r8 WITH py_scope_flow_021#join_rhs ON FIRST 2 OUTPUT Rhs.2, Lhs.2
           38   ~0%    {2} r10 = JOIN r9 WITH Module#68efc075::Module::getPackageName#0#dispred#ff ON FIRST 1 OUTPUT Lhs.1, Rhs.1
          394   ~2%    {2} r11 = JOIN r10 WITH Essa#24e22a14::EssaVariable::getSourceVariable#0#dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1
          394   ~0%    {3} r12 = JOIN r11 WITH DataFlowPublic#c4b6ce7e::EssaNode#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.0, Lhs.1, Rhs.1

          394   ~2%    {4} r13 = JOIN r12 WITH Essa#24e22a14::EssaVariable::getName#0#dispred#ff ON FIRST 1 OUTPUT _, Lhs.2, Lhs.1, Rhs.1
          394   ~0%    {2} r14 = REWRITE r13 WITH Tmp.0 := ".", Out.0 := (In.2 ++ Tmp.0 ++ In.3) KEEPING 2

          394   ~0%    {5} r15 = JOIN r12 WITH Essa#24e22a14::EssaVariable::getName#0#dispred#ff ON FIRST 1 OUTPUT _, Lhs.2, Lhs.1, Rhs.1, _
          394   ~5%    {2} r16 = REWRITE r15 WITH Tmp.0 := ".", Tmp.0 := (In.2 ++ Tmp.0 ++ In.3), Tmp.4 := ".__init__", Out.0 := (Tmp.0 ++ Tmp.4) KEEPING 2

          788   ~0%    {2} r17 = r14 UNION r16
          401   ~1%    {2} r18 = JOIN r17 WITH Module#68efc075::isPreferredModuleForName#2#ff_10#join_rhs ON FIRST 1 OUTPUT Rhs.1, Lhs.1
          394   ~1%    {2} r19 = JOIN r18 WITH Module#68efc075::Module::getFile#0#dispred#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Rhs.1

        26339   ~0%    {2} r20 = r6 UNION r19
                       return r20
2023-10-25 14:53:51 +01:00
Rasmus Wriedt Larsen
e8f548ab52 Python: Model routed parameter flow to *args and **kwargs in Django + rest framework 2023-10-23 17:18:22 +02:00
amammad
a0e2e1ef21 add to Frameworks.qll 2023-10-21 19:49:27 +02:00
amammad
1fe565a46f cherrypy framework file system access Sinks are added 2023-10-21 19:47:30 +02:00
Mathew Payne
a24e168ec0 Merge branch 'main' into py-restframework 2023-10-20 11:39:07 +01:00
Dave Bartolomeo
712f7758cf Merge branch 'main' into post-release-prep/codeql-cli-2.15.1 2023-10-19 12:14:07 -04:00
github-actions[bot]
8dcd8b9e5b Post-release preparation for codeql-cli-2.15.1 2023-10-17 20:24:00 +00:00
Rasmus Wriedt Larsen
62a992473f Python: Update QLDoc 2023-10-17 10:09:19 +02:00
Rasmus Wriedt Larsen
2bf4c32433 Python: Add syntactic support for yield in contextlib.contextmanager 2023-10-17 09:51:20 +02:00
github-actions[bot]
3b3c036626 Release preparation for version 2.15.1 2023-10-16 17:49:39 +00:00
Henry Mercer
1a370bfbbe Merge pull request #14443 from github/post-release-prep/codeql-cli-2.15.0
Post-release preparation for codeql-cli-2.15.0
2023-10-11 17:39:04 +01:00
github-actions[bot]
ae6af17c74 Post-release preparation for codeql-cli-2.15.0 2023-10-11 14:19:20 +00:00
Rasmus Wriedt Larsen
68d00a829e Merge pull request #14430 from RasmusWL/api-graph-import-star
Python: Better allow `import *` to work with API graphs
2023-10-11 10:03:46 +02:00
Rasmus Wriedt Larsen
ee75b104eb Python: Add change-note 2023-10-10 17:45:11 +02:00
Rasmus Wriedt Larsen
72d0dcdaba Python: Workaround for module level items from import * not being LocalSourceNodes 2023-10-10 17:45:11 +02:00
yoff
f1266a3e81 Merge pull request #14417 from github/tausbn/python-add-flow-for-assignment-expressions 2023-10-10 17:09:20 +02:00
Rasmus Wriedt Larsen
2d947a4f53 Merge pull request #13781 from maikypedia/maikypedia/python-unsafe-deserialization
Python: Add unsafe deserialization sinks (CWE-502)
2023-10-10 13:30:38 +02:00