Commit Graph

2201 Commits

Author SHA1 Message Date
Calum Grant
c26d05b1d5 Merge pull request #5532 from RasmusWL/python-cleanup
Python: Delete filter queries, code duplication library, and precision tag from metric queries
2021-03-29 17:16:43 +01:00
Rasmus Wriedt Larsen
92e0e195a4 Revert "Merge pull request #5506 from tausbn/python-allow-absolute-imports-from-source-directory"
This reverts commit 8d15680af4, reversing
changes made to 63831cc62b.

This PR caused performance problems, so reverting now to clear up immediate
problems.
2021-03-27 18:08:20 +01:00
Rasmus Lerchedahl Petersen
1be2be843d Python: update test expectations 2021-03-26 13:08:23 +01:00
Rasmus Lerchedahl Petersen
c93e0c08fd Merge branch 'python-port-insecure-protocol' of github.com:yoff/codeql into python-port-insecure-protocol 2021-03-26 00:26:33 +01:00
yoff
54dad57cf4 Update python/ql/test/query-tests/Security/CWE-327/pyOpenSSL_fluent.py
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-03-26 00:25:40 +01:00
Rasmus Lerchedahl Petersen
2b257318f1 Python: more precise comment 2021-03-25 23:22:24 +01:00
yoff
208d5157fa Merge pull request #5500 from RasmusWL/django-forms
Python: Model RemoteFlowSources on Django forms/fields
2021-03-25 20:43:19 +01:00
Rasmus Wriedt Larsen
bd4934380a Python: Remove code duplication library 2021-03-25 15:27:55 +01:00
yoff
8d15680af4 Merge pull request #5506 from tausbn/python-allow-absolute-imports-from-source-directory
Python: Allow absolute imports in directories with scripts
2021-03-24 14:42:14 +01:00
Rasmus Wriedt Larsen
70974ea197 Python: Fix grammar in QLDoc
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-03-24 14:06:06 +01:00
Taus Brock-Nannestad
8d30ee5c3c Python: Include unmarked Python file in snapshot
Sadly, it seems we're not interpreting this as Python code, even if we
explicitly ask to have it included.
2021-03-24 14:01:13 +01:00
Taus Brock-Nannestad
6d86239929 Python: Test all cases
Note that the test in `no_py_extension` isn't complete, since we're
not extracting the `main` file there.
2021-03-24 13:15:59 +01:00
Taus Brock-Nannestad
17d1768259 Python: Allow absolute imports in directories with scripts
Fixes the import logic to account for absolute imports.

We do this by classifying which files and folders may serve as the
entry point for execution, based on a few simple heuristics. If the
file `module.py` is in the same folder as a file `main.py` that may be
executed directly, then we allow `module` to be a valid name for
`module.py` so that `import module` will work as expected.
2021-03-23 18:32:17 +01:00
Taus Brock-Nannestad
4289e358bf Python: Add module import test case
This one will require some explanation...

First, the file structure. This commit adds a test consisting
representing a few different kinds of imports.

- Absolute imports, from `module.py` to `main.py` when the latter is
  executed directly.
- A package (contained in the `package` folder)
- A namespace package (contained in the `namespace_package` folder)

All of these are inside a folder called `code` for reasons I will
detail later.

The file `main.py` is identified as a script, by the presence of the
`!#` comment in its first line.

The files themselves are executable, and `python3 main.py` will print
out all modules in the order they are imported.

The test itself is very simple. It simply lists all modules and their
corresponding names. As is plainly visible, without modification we
only pick up `package` and its component modules as having names. This
is the bit that needs to be fixed.

Convincing the test runner to extract this test in a way that mimics
reality is, unfortunately, a bit complicated. By default, the test
runner itself includes any Python files in the test directory as
modules in the invocation of the extractor, and so we must hide
everything in the `code` subdirectory.

Secondly, a `--path` argument (set to the test directory) is
automatically added, and this would also interfere with extraction,
and hence we must prevent this. Luckily, if we supply our own `--path`
argument -- even if it doesn't make any sense -- then the other
argument is left out.

Finally, we must actually tell the extractor to extract the files (or
it would just happily pass the test with zero files extracted), so the
`-R .` argument ensures that we recurse over the files in the test
directory after all.
2021-03-23 18:21:58 +01:00
Rasmus Wriedt Larsen
1f5e52e822 Python: Cleanup "first" type-tracking predicate to be private
Since it's exposed nicely in the version that doesn't have a
`DataFlow::TypeTracker` parameter, these should be private.

Also found one instance where I had accidentially used DataFlow::Node instead of
LocalSourceNode
2021-03-23 16:40:56 +01:00
Rasmus Wriedt Larsen
8d0f6086af Python: Model django forms/fields
I'm not feeling 100% confident about `SelfRefMixin`, but since I needed it for
both DjangoViewClass and DjangoFormClass, I wanted to avoid copy-pasting this
code around. However, I'm not so opitimistic about it that I want to add it to a
sharable utility qll file :D
2021-03-23 13:57:38 +01:00
Rasmus Wriedt Larsen
c8a6e837b5 Python: Model QuerySet chains in django 2021-03-22 14:38:54 +01:00
Rasmus Wriedt Larsen
3a83ecf067 Python: Add test for taint in django forms/fields 2021-03-22 10:03:32 +01:00
Rasmus Wriedt Larsen
701b935564 Python: Add example of QuerySet chain (django) 2021-03-22 00:57:43 +01:00
yoff
164b383fda Update python/ql/test/query-tests/Security/CWE-327/pyOpenSSL_fluent.py
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-03-19 19:12:13 +01:00
Rasmus Wriedt Larsen
d9079e34e3 Python: Move framework tests out of experimental
Since they are not experimental anymore 😄
2021-03-19 15:51:54 +01:00
yoff
37036b5e76 Merge pull request #5437 from RasmusWL/small-pyyaml-improvements
Python: Small PyYAML improvements
2021-03-19 11:15:49 +01:00
Rasmus Wriedt Larsen
7543f10593 Python: Reorganize PyYAML tests a bit 2021-03-19 09:53:25 +01:00
Rasmus Lerchedahl Petersen
e0e6d5724e Merge branch 'main' of github.com:github/codeql into python-port-insecure-protocol 2021-03-18 23:34:53 +01:00
yoff
746e9948b0 Merge pull request #5075 from RasmusWL/crypto
Python: Port py/weak-crypto-key to use type-tracking
2021-03-18 20:53:28 +01:00
Rasmus Wriedt Larsen
42b2c3ed52 Python: Model C-based loaders for PyYAML
Not really that important. But easy to do while I was working on this library.
2021-03-18 11:55:01 +01:00
Rasmus Wriedt Larsen
54e6f51512 Python: Add example of C-based PyYAML loaders
```
In [6]: yaml.load("!!python/object/new:os.system [echo EXPLOIT!]", yaml.CLoader)
EXPLOIT!
Out[6]: 0
```
2021-03-18 11:50:59 +01:00
Rasmus Wriedt Larsen
14e9bda5de Python: Refactor PyYAML tests a bit 2021-03-18 11:39:47 +01:00
CodeQL CI
1d9f8c2d37 Merge pull request #5427 from RasmusWL/use-new-builtin-modeling
Approved by yoff
2021-03-17 09:07:36 -07:00
Rasmus Wriedt Larsen
d52d328587 Python: Use new API::builtin in stdlib modeling 2021-03-17 15:50:01 +01:00
Rasmus Wriedt Larsen
315127d888 Python: Also test py/insecure-default-protocol on Python 3 2021-03-17 14:53:36 +01:00
Rasmus Wriedt Larsen
fbbec5d2b9 Merge pull request #5118 from yoff/python-port-stacktrace-exosure
Python: Port stack trace exposure
2021-03-16 14:52:44 +01:00
Rasmus Wriedt Larsen
50978364a6 Merge pull request #5246 from yoff/python-port-insecure-default-protocol
Python: Port insecure default protocol
2021-03-16 14:30:19 +01:00
Rasmus Lerchedahl Petersen
6fff746b16 Merge branch 'main' of github.com:github/codeql into python-port-insecure-protocol 2021-03-15 17:37:28 +01:00
Rasmus Lerchedahl Petersen
514a69c47a Python: Support ssl.PROTOCOL_TLS_SERVER
and `ssl.PROTOCOL_TLS_CLIENT`
2021-03-15 17:30:01 +01:00
Rasmus Lerchedahl Petersen
87f3ba2684 Python: add tests for ssl.PROTOCOL_TLS_SERVER
and `ssl.PROTOCOL_TLS_CLIENT`
2021-03-15 17:24:39 +01:00
Rasmus Lerchedahl Petersen
731f4559b4 Python: update test expectations 2021-03-15 17:23:58 +01:00
Rasmus Lerchedahl Petersen
4094b18407 Python: Clean up tests 2021-03-15 16:28:08 +01:00
yoff
a760ed8c55 Merge pull request #5388 from tausbn/python-api-graph-builtins
Python: Support built-ins in API graphs
2021-03-12 17:45:59 +01:00
Taus
dfc0e9b906 Merge pull request #5243 from RasmusWL/port-bind-to-all-interfaces
Python: Port py/bind-socket-all-network-interfaces query
2021-03-12 16:04:19 +01:00
Taus Brock-Nannestad
978200e2ad Python: Distinguish between Python 2 and 3
Also moves the filtering on `name` to before the big disjunction in
`MkModuleImport`.
2021-03-12 12:35:23 +01:00
Taus Brock-Nannestad
c7b2b719cf Python: Support builtins in API graphs 2021-03-11 23:03:18 +01:00
Rasmus Lerchedahl Petersen
cc9a938054 InlineExpectationTest: clarify the nedd for an
empty `.expected` file
2021-03-08 09:18:47 +01:00
Rasmus Lerchedahl Petersen
7142ddcb25 Python: add taint step for __traceback__ 2021-03-08 08:13:07 +01:00
Rasmus Lerchedahl Petersen
b36e0d0be7 Python: target SSA variable rather than Cfg node
also add "INTERNAL: Do not use."
also give test functions different names
2021-03-08 08:04:42 +01:00
Rasmus Lerchedahl Petersen
296297915c Python: add test for __traceback__ 2021-03-07 17:50:28 +01:00
yoff
d17246ce2b Merge pull request #5255 from RasmusWL/port-flask-debug
Python: port py/flask-debug query
2021-03-05 09:39:14 +01:00
Rasmus Lerchedahl Petersen
7d556b354d Python: Update test annotation and expectation 2021-03-05 09:16:35 +01:00
Taus
d2ed216670 Merge pull request #5280 from RasmusWL/highlight-tornado-uri
Python: Highlight how request.uri works in Tornado
2021-03-04 23:31:02 +01:00
Rasmus Lerchedahl Petersen
9f8a028dfc Python: add .expected-file 2021-03-04 00:12:34 +01:00