Commit Graph

4008 Commits

Author SHA1 Message Date
Sid Shankar
f557110d9b Merge pull request #15465 from sidshank/sidshank/rename-file-name-for-extracted-files-diagnostic
JS/TS/Python/Ruby: Renames diagnostic query files and tests
2024-01-30 10:19:00 -05:00
Rasmus Wriedt Larsen
c265c15f3f Merge pull request #15398 from RasmusWL/html-escape
Python: Add `html.escape` as HTML sanitizer
2024-01-30 16:06:01 +01:00
Rasmus Wriedt Larsen
c70b32f7eb Python: Require quote escaping for html.escape 2024-01-30 12:17:01 +01:00
Sid Shankar
b1d7a635f5 Renames diagnostic query files and tests
This commit renames the files relating to the diagnostic query that produces information on the number of files extracted. The files have been renamed from "SuccessfullExtractedFiles.*" to "ExtractedFiles.*". All related tests and test files have been renamed too.

The `@tags` and `@id` attributes of the queries have been left untouched, consistent with the `@tags` and `@id` for similar queries in other languages.
2024-01-29 20:19:20 +00:00
Rasmus Wriedt Larsen
3f0dc2b022 Python: Model the psycopg package 2024-01-29 14:30:20 +01:00
yoff
391ca5d8a6 Merge pull request #15390 from Marcono1234/marcono1234/python-ascii-regex-flag 2024-01-29 14:27:50 +01:00
Marcono1234
1ad08efe08 Python: Support a (ASCII) inline regex flag 2024-01-26 22:18:49 +01:00
yoff
930f1b50b9 Merge pull request #15397 from github/tausbn/python-fix-deepcopy-mutable-default-fp
Python: Fix `deepcopy` mutable default FP
2024-01-25 10:32:58 +01:00
Taus
96b1b8e402 Python: Remove empty lines from test file 2024-01-24 12:31:23 +00:00
Erik Krogh Kristensen
f1d6f56621 Merge pull request #15393 from erik-krogh/deps-jan-2024
All: delete outdated deprecations
2024-01-23 13:52:38 +01:00
Rasmus Wriedt Larsen
cbed6e861d Python: Add html.escape as HTML sanitizer 2024-01-22 17:32:28 +01:00
Taus
d6d59377d3 Python: Fix flow through deepcopy
Or, more generally, any copy step, as these presumably do not preserve
object identity.

(Arguably, `copy` could still be susceptible to interior mutability, but
I think that's outside the scope of this query anyway.)
2024-01-22 15:40:30 +00:00
Taus
14c958ac4d Python: Remove mutable default sources from inside stdlib 2024-01-22 15:23:52 +00:00
Taus
411c107660 Python: Add tests for deepcopy FPs
There are two issues with `deepcopy` here. Firstly, the `deepcopy` function itself
has a mutable default value in its parameter `_nil` (set to the empty list by default).

Now, this value is never actually returned from `deepcopy`, as it is only used as a
sentinel, but our analysis is not clever enough to see this. Thus, it thinks that this
mutable default is returned, and hence the result of any call to `deepcopy` is a
potential source.

To remedy this, I opted to simply exclude all sources that originate from within the
standard library. It is very unlikely for any of the sources in the standard library
to be legit.

Secondly, `deepcopy` -- by virtue of being a function that we model as preserving
values -- admits data-flow through its calls, but this is not correct for the mutable
default query, as it is here the _identity_ of the default value in question that is
important. Thus, we get spurious flow through `deepcopy` for this specific query.
2024-01-22 15:21:57 +00:00
Taus
4742481070 Python: Consolidate "mutable default" tests
Moves the existing tests into the `ModificationOfParameterWithDefault` subdirectory
which already contained a bunch more tests. In the process, I also removed some
duplicated test cases.
2024-01-22 13:50:33 +00:00
Max Schaefer
98178458d0 Python: Add support for more URL redirect sanitisers.
Since some sanitisers don't handle backslashes correctly, I updated the data-flow configuration to incorporate a flow state tracking whether or not backslashes have been eliminated or converted to forward slashes.
2024-01-22 13:24:18 +00:00
erik-krogh
8be7eadace delete outdated deprecations 2024-01-22 09:11:35 +01:00
Sid Shankar
59098be8c4 Merge branch 'main' into change/adjust-extracted-files-diagnostics 2024-01-16 21:51:41 -05:00
Sid Shankar
fb660b8f05 Py: Report any extracted file as successfully extracted 2024-01-08 22:20:51 +00:00
Rasmus Wriedt Larsen
95c24275f2 Merge pull request #15044 from RasmusWL/automated-subclass-models
Python: Automated subclass models
2024-01-05 10:43:48 +01:00
Rasmus Lerchedahl Petersen
da4aef80e9 Revert "Python: make it a real consistency check"
This reverts commit 45411f4a93.
2023-12-20 16:15:17 +01:00
Rasmus Lerchedahl Petersen
d039ceb689 Python: add test for fields 2023-12-20 15:23:45 +01:00
Rasmus Lerchedahl Petersen
706e9dc896 Python: fix compilation 2023-12-20 15:23:33 +01:00
Rasmus Lerchedahl Petersen
45411f4a93 Python: make it a real consistency check 2023-12-20 14:53:37 +01:00
Rasmus Lerchedahl Petersen
f8417b0dd8 Merge branch 'main' of https://github.com/github/codeql into python/captured-variables-basic 2023-12-20 13:16:42 +01:00
Rasmus Lerchedahl Petersen
169d7a3c98 Python: Add scope entry definition nodes
otherwise we confuse captured variables
in the single scope entry cfg node. Now
we have one for each defined variable.
2023-12-20 12:09:00 +01:00
Rasmus Lerchedahl Petersen
3b7e29bed6 Python: add test for crosstalk 2023-12-20 12:08:05 +01:00
Rasmus Wriedt Larsen
72687e0368 Merge branch 'main' into automated-subclass-models 2023-12-19 17:08:25 +01:00
Rasmus Wriedt Larsen
3e878f5a0b Python: Model django response subclass relationship 2023-12-19 17:07:02 +01:00
Rasmus Wriedt Larsen
abe6f1639a Python: Add example of models subclassing problem
In reality, we only want to model this as a `rest_framework.response.Response`, since our .qll modeling is more precise for rest-framework responses than if we also modeled it as a basic django http response. (specifically, that default mime-type handling is way different).
2023-12-19 17:07:02 +01:00
Rasmus Wriedt Larsen
933938d926 Python: Make rest_framework tests runnable again 2023-12-19 17:07:01 +01:00
Rasmus Wriedt Larsen
0fe29b6a86 Python: Recover subclass finder .expected after cherry picking commits from https://github.com/github/codeql/pull/15030 2023-12-19 17:07:01 +01:00
Rasmus Lerchedahl Petersen
75f9eeb4e9 Python: adjust test expectations
mostly removing of nodes from the graph.
One result lost:
```
check("submodule.submodule_attr", submodule.submodule_attr, "submodule_attr", globals()) #$ MISSING:prints=submodule_attr
```
2023-12-19 17:07:01 +01:00
Rasmus Wriedt Larsen
e050f2e998 Python: Adjust subclass finder to no ESSA nodes
But the new test results looks very strange indeed!
2023-12-19 17:07:01 +01:00
Rasmus Wriedt Larsen
2305d55967 Merge pull request #15101 from yoff/python/update-InlineTaintTestPaths-to-new-api
Python: update to new API update is in a comment, so compilation never failed in CI.
2023-12-19 11:10:55 +01:00
yoff
a60c52b8b7 Merge branch 'main' into python/captured-variables-basic 2023-12-18 23:44:46 +01:00
Rasmus Lerchedahl Petersen
78c484faab Python: remove support for capturing callbacks
This will be added in a follow-up PR instead.
2023-12-18 23:24:57 +01:00
Rasmus Lerchedahl Petersen
64655a0ea8 Python: Use enw class name 2023-12-16 01:36:46 +01:00
Rasmus Lerchedahl Petersen
1ee11ae7af Merge branch 'main' of https://github.com/github/codeql into python/captured-variables-basic 2023-12-15 14:31:57 +01:00
Rasmus Lerchedahl Petersen
abd544d96c Python: consistency failure gone 2023-12-15 00:38:58 +01:00
Tom Hvitved
3b1146bf98 Python: Adopt shared type tracking library 2023-12-14 13:22:44 +01:00
Tom Hvitved
c8b4a215bc Merge pull request #14573 from hvitved/flow-summary-impl-param
Move `FlowSummaryImpl.qll` to `dataflow` pack
2023-12-14 12:24:15 +01:00
Rasmus Lerchedahl Petersen
0b6d47b8bc Python: update to new API
update is in a comment, so compilation
never failed in CI.
2023-12-14 11:56:05 +01:00
Rasmus Lerchedahl Petersen
2a5736e73d Python: add consistency exception
this must have been lost in my
clean-up rebase.
2023-12-14 11:50:09 +01:00
Rasmus Lerchedahl Petersen
479d81fb75 Python: fix nonlocal captured variables
This depends on the extractor fix
2023-12-14 10:37:27 +01:00
Rasmus Lerchedahl Petersen
17a0029585 Python: support callbacks to library calls
TODO:
The member predicate `LibraryLambdaMethod::getACall` is
currently too permissive.
Ideally, we would have `libraryCallHasLambdaArg`
as in Ruby. But even a more precise
`libraryCall` predicate might be fine.
2023-12-14 10:27:15 +01:00
Rasmus Lerchedahl Petersen
7565873e83 Python: test callbacks to library calls 2023-12-14 10:27:15 +01:00
Rasmus Lerchedahl Petersen
797deebcdd Python: exclude CaptureNodes 2023-12-14 10:27:15 +01:00
Rasmus Lerchedahl Petersen
c054ba6a97 python: instantiate module for variable capture
This provides variable capture in standard situations:
- nested functions
- lambdas
There are some deficiencies:
- we do not yet handle objects capturing variables.
- we do not handle variables captured via the `nonlocal` keyword.
  This should be solved at the AST level, though, and then it
  should "just work".

There are still inconsistencies in the case where
a `SynthesizedCaptureNode` has a comprehensions
as its enclosing callable. In this case,
`TFunction(cn.getEnclosingCallable())` is not
defined and so getEnclosingCallable does not exist
for the `CaptureNode`.
2023-12-14 10:25:39 +01:00
Rasmus Lerchedahl Petersen
6db55cd12f Python: add missing annotation 2023-12-14 10:20:49 +01:00