Commit Graph

3953 Commits

Author SHA1 Message Date
yoff
391ca5d8a6 Merge pull request #15390 from Marcono1234/marcono1234/python-ascii-regex-flag 2024-01-29 14:27:50 +01:00
Marcono1234
1ad08efe08 Python: Support a (ASCII) inline regex flag 2024-01-26 22:18:49 +01:00
yoff
930f1b50b9 Merge pull request #15397 from github/tausbn/python-fix-deepcopy-mutable-default-fp
Python: Fix `deepcopy` mutable default FP
2024-01-25 10:32:58 +01:00
Taus
96b1b8e402 Python: Remove empty lines from test file 2024-01-24 12:31:23 +00:00
Erik Krogh Kristensen
f1d6f56621 Merge pull request #15393 from erik-krogh/deps-jan-2024
All: delete outdated deprecations
2024-01-23 13:52:38 +01:00
Rasmus Wriedt Larsen
cbed6e861d Python: Add html.escape as HTML sanitizer 2024-01-22 17:32:28 +01:00
Taus
d6d59377d3 Python: Fix flow through deepcopy
Or, more generally, any copy step, as these presumably do not preserve
object identity.

(Arguably, `copy` could still be susceptible to interior mutability, but
I think that's outside the scope of this query anyway.)
2024-01-22 15:40:30 +00:00
Taus
14c958ac4d Python: Remove mutable default sources from inside stdlib 2024-01-22 15:23:52 +00:00
Taus
411c107660 Python: Add tests for deepcopy FPs
There are two issues with `deepcopy` here. Firstly, the `deepcopy` function itself
has a mutable default value in its parameter `_nil` (set to the empty list by default).

Now, this value is never actually returned from `deepcopy`, as it is only used as a
sentinel, but our analysis is not clever enough to see this. Thus, it thinks that this
mutable default is returned, and hence the result of any call to `deepcopy` is a
potential source.

To remedy this, I opted to simply exclude all sources that originate from within the
standard library. It is very unlikely for any of the sources in the standard library
to be legit.

Secondly, `deepcopy` -- by virtue of being a function that we model as preserving
values -- admits data-flow through its calls, but this is not correct for the mutable
default query, as it is here the _identity_ of the default value in question that is
important. Thus, we get spurious flow through `deepcopy` for this specific query.
2024-01-22 15:21:57 +00:00
Taus
4742481070 Python: Consolidate "mutable default" tests
Moves the existing tests into the `ModificationOfParameterWithDefault` subdirectory
which already contained a bunch more tests. In the process, I also removed some
duplicated test cases.
2024-01-22 13:50:33 +00:00
Max Schaefer
98178458d0 Python: Add support for more URL redirect sanitisers.
Since some sanitisers don't handle backslashes correctly, I updated the data-flow configuration to incorporate a flow state tracking whether or not backslashes have been eliminated or converted to forward slashes.
2024-01-22 13:24:18 +00:00
erik-krogh
8be7eadace delete outdated deprecations 2024-01-22 09:11:35 +01:00
Sid Shankar
59098be8c4 Merge branch 'main' into change/adjust-extracted-files-diagnostics 2024-01-16 21:51:41 -05:00
Sid Shankar
fb660b8f05 Py: Report any extracted file as successfully extracted 2024-01-08 22:20:51 +00:00
Rasmus Wriedt Larsen
95c24275f2 Merge pull request #15044 from RasmusWL/automated-subclass-models
Python: Automated subclass models
2024-01-05 10:43:48 +01:00
Rasmus Lerchedahl Petersen
da4aef80e9 Revert "Python: make it a real consistency check"
This reverts commit 45411f4a93.
2023-12-20 16:15:17 +01:00
Rasmus Lerchedahl Petersen
d039ceb689 Python: add test for fields 2023-12-20 15:23:45 +01:00
Rasmus Lerchedahl Petersen
706e9dc896 Python: fix compilation 2023-12-20 15:23:33 +01:00
Rasmus Lerchedahl Petersen
45411f4a93 Python: make it a real consistency check 2023-12-20 14:53:37 +01:00
Rasmus Lerchedahl Petersen
f8417b0dd8 Merge branch 'main' of https://github.com/github/codeql into python/captured-variables-basic 2023-12-20 13:16:42 +01:00
Rasmus Lerchedahl Petersen
169d7a3c98 Python: Add scope entry definition nodes
otherwise we confuse captured variables
in the single scope entry cfg node. Now
we have one for each defined variable.
2023-12-20 12:09:00 +01:00
Rasmus Lerchedahl Petersen
3b7e29bed6 Python: add test for crosstalk 2023-12-20 12:08:05 +01:00
Rasmus Wriedt Larsen
72687e0368 Merge branch 'main' into automated-subclass-models 2023-12-19 17:08:25 +01:00
Rasmus Wriedt Larsen
3e878f5a0b Python: Model django response subclass relationship 2023-12-19 17:07:02 +01:00
Rasmus Wriedt Larsen
abe6f1639a Python: Add example of models subclassing problem
In reality, we only want to model this as a `rest_framework.response.Response`, since our .qll modeling is more precise for rest-framework responses than if we also modeled it as a basic django http response. (specifically, that default mime-type handling is way different).
2023-12-19 17:07:02 +01:00
Rasmus Wriedt Larsen
933938d926 Python: Make rest_framework tests runnable again 2023-12-19 17:07:01 +01:00
Rasmus Wriedt Larsen
0fe29b6a86 Python: Recover subclass finder .expected after cherry picking commits from https://github.com/github/codeql/pull/15030 2023-12-19 17:07:01 +01:00
Rasmus Lerchedahl Petersen
75f9eeb4e9 Python: adjust test expectations
mostly removing of nodes from the graph.
One result lost:
```
check("submodule.submodule_attr", submodule.submodule_attr, "submodule_attr", globals()) #$ MISSING:prints=submodule_attr
```
2023-12-19 17:07:01 +01:00
Rasmus Wriedt Larsen
e050f2e998 Python: Adjust subclass finder to no ESSA nodes
But the new test results looks very strange indeed!
2023-12-19 17:07:01 +01:00
Rasmus Wriedt Larsen
2305d55967 Merge pull request #15101 from yoff/python/update-InlineTaintTestPaths-to-new-api
Python: update to new API update is in a comment, so compilation never failed in CI.
2023-12-19 11:10:55 +01:00
yoff
a60c52b8b7 Merge branch 'main' into python/captured-variables-basic 2023-12-18 23:44:46 +01:00
Rasmus Lerchedahl Petersen
78c484faab Python: remove support for capturing callbacks
This will be added in a follow-up PR instead.
2023-12-18 23:24:57 +01:00
Rasmus Lerchedahl Petersen
64655a0ea8 Python: Use enw class name 2023-12-16 01:36:46 +01:00
Rasmus Lerchedahl Petersen
1ee11ae7af Merge branch 'main' of https://github.com/github/codeql into python/captured-variables-basic 2023-12-15 14:31:57 +01:00
Rasmus Lerchedahl Petersen
abd544d96c Python: consistency failure gone 2023-12-15 00:38:58 +01:00
Tom Hvitved
3b1146bf98 Python: Adopt shared type tracking library 2023-12-14 13:22:44 +01:00
Tom Hvitved
c8b4a215bc Merge pull request #14573 from hvitved/flow-summary-impl-param
Move `FlowSummaryImpl.qll` to `dataflow` pack
2023-12-14 12:24:15 +01:00
Rasmus Lerchedahl Petersen
0b6d47b8bc Python: update to new API
update is in a comment, so compilation
never failed in CI.
2023-12-14 11:56:05 +01:00
Rasmus Lerchedahl Petersen
2a5736e73d Python: add consistency exception
this must have been lost in my
clean-up rebase.
2023-12-14 11:50:09 +01:00
Rasmus Lerchedahl Petersen
479d81fb75 Python: fix nonlocal captured variables
This depends on the extractor fix
2023-12-14 10:37:27 +01:00
Rasmus Lerchedahl Petersen
17a0029585 Python: support callbacks to library calls
TODO:
The member predicate `LibraryLambdaMethod::getACall` is
currently too permissive.
Ideally, we would have `libraryCallHasLambdaArg`
as in Ruby. But even a more precise
`libraryCall` predicate might be fine.
2023-12-14 10:27:15 +01:00
Rasmus Lerchedahl Petersen
7565873e83 Python: test callbacks to library calls 2023-12-14 10:27:15 +01:00
Rasmus Lerchedahl Petersen
797deebcdd Python: exclude CaptureNodes 2023-12-14 10:27:15 +01:00
Rasmus Lerchedahl Petersen
c054ba6a97 python: instantiate module for variable capture
This provides variable capture in standard situations:
- nested functions
- lambdas
There are some deficiencies:
- we do not yet handle objects capturing variables.
- we do not handle variables captured via the `nonlocal` keyword.
  This should be solved at the AST level, though, and then it
  should "just work".

There are still inconsistencies in the case where
a `SynthesizedCaptureNode` has a comprehensions
as its enclosing callable. In this case,
`TFunction(cn.getEnclosingCallable())` is not
defined and so getEnclosingCallable does not exist
for the `CaptureNode`.
2023-12-14 10:25:39 +01:00
Rasmus Lerchedahl Petersen
6db55cd12f Python: add missing annotation 2023-12-14 10:20:49 +01:00
Rasmus Wriedt Larsen
419130be21 Merge pull request #15030 from yoff/python/remove-module-entry-definitions
Python: Remove control flow nodes for module entry definitions from the dataflow graph.
2023-12-11 11:40:17 +01:00
Tom Hvitved
faaa558ed9 Python: Use FlowSummaryImpl from dataflow pack 2023-12-10 11:25:44 +01:00
Anders Schack-Mulligen
64eb4ff753 Merge pull request #14983 from aschackmull/dataflow/deprecate-old-api
Data Flow: Deprecate old data flow api.
2023-12-08 14:27:25 +01:00
Rasmus Wriedt Larsen
6ef9a2b11e Python: Fix problem if import is used
I fixed it in both predicates... I think we might still be able to remove
`newDirectAlias` -- but with it being better, it will allow us to better test if `newImportAlias` actually cover everything we need!
2023-12-08 11:27:52 +01:00
Rasmus Wriedt Larsen
fcdc8102e2 Python: Add test highlight problem is import is used :O 2023-12-08 11:27:52 +01:00