Commit Graph

689 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
d388dd547e Python: Model HTTPMessage from Stdlib 2021-07-22 10:43:18 +02:00
Rasmus Wriedt Larsen
dac71ded9d Python: Add Authorization modeling in Flask 2021-07-22 10:43:18 +02:00
Rasmus Wriedt Larsen
133632119d Python: Model werkzeug Headers
Also removed a misleading comment link to method on wrong class :D
2021-07-22 10:43:18 +02:00
Rasmus Wriedt Larsen
4d9c86a252 Python: Model Werkzeug FileStorage.save as FileSystemAccess 2021-07-22 10:43:18 +02:00
Rasmus Wriedt Larsen
9cb4899c5c Python: Add FileStorage modeling in Flask 2021-07-22 10:43:18 +02:00
Rasmus Wriedt Larsen
04190ea308 Python: Add file-like modeling to werkzeug FileStorage 2021-07-22 10:43:18 +02:00
Rasmus Wriedt Larsen
4f4dec50f2 Python: Model ResovlerMatch in Django
Like before, omitted ClassInstantiation
2021-07-22 10:43:13 +02:00
Rasmus Wriedt Larsen
7dc6518350 Python: Add FileLikeObject modeling
Such that the result of `request.FILES["key"].file.read()` is tainted
2021-07-21 16:35:09 +02:00
Rasmus Wriedt Larsen
18c0d13efd Python: Model most of UploadedFile in Django 2021-07-21 16:35:09 +02:00
Rasmus Wriedt Larsen
5ec5557203 Python: Model MultiValueDict in Django 2021-07-21 16:35:09 +02:00
Rasmus Wriedt Larsen
51b543c67c Python: Model taint for django request methods 2021-07-21 16:35:09 +02:00
Taus
55d822cc56 Python: Add TypeTrackingNode
Splits `ModuleVariableNode` away from `LocalSourceNode`, instead
creating a class `TypeTrackingNode` that encapsulates both of these.

This means we no longer have module variable nodes as part of
`LocalSourceNode` (which is good, since they have no "local" aspect to
them), and hence we can have `LocalSourceNode` inherit directly from
`ExprNode` (which makes the API a bit nicer).

Unfortunately these are breaking changes, so we can't actually fulfil
the above two desiderata until the `track` and `backtrack` methods on
`LocalSourceNode` have been fully deprecated. For this reason, we
preserve the present implementation of `LocalSourceNode`, and instead
lay the foundation for switching over in the future, by deprecating
`track` and `backtrack` on `LocalSourceNode`.
2021-07-02 18:00:33 +00:00
Taus
e4af14638b Merge pull request #6175 from yoff/python-port-ReDoS
Python: port ReDoS queries from Javascript
2021-06-30 16:26:07 +02:00
yoff
6a77b890af Merge pull request #6155 from RasmusWL/port-cleartext-queries
Python: Port cleartext queries
2021-06-30 15:52:34 +02:00
Rasmus Lerchedahl Petersen
651f8abba0 Python: Avoid multiple results for toString 2021-06-30 14:39:49 +02:00
Rasmus Wriedt Larsen
c2708176b1 Python: Support %-style formatting for MarkupSafe 2021-06-30 14:15:41 +02:00
Rasmus Wriedt Larsen
0a4efd0e86 Python: Add %-style formatting tests for MarkupSafe 2021-06-30 14:13:59 +02:00
Rasmus Wriedt Larsen
075953860b Merge branch 'main' into markupsafe-modeling 2021-06-30 13:55:08 +02:00
yoff
c19522e921 Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2021-06-30 11:49:45 +02:00
Rasmus Lerchedahl Petersen
fbfe415162 Python: Limit test files 2021-06-29 11:18:24 +02:00
Rasmus Lerchedahl Petersen
21007d21f4 Python: track if qualifiers allow unbounded
repeats. This in preparation for ReDoS
2021-06-28 17:04:48 +02:00
Rasmus Lerchedahl Petersen
74ca1d00b9 Python: More precise regex parsing 2021-06-28 17:04:48 +02:00
Rasmus Lerchedahl Petersen
e5f07cc4d3 Python: inline test of regex components
- Added naive implementation of `charRange` so the test can run.
- Made predicates public as needed.
2021-06-28 17:04:48 +02:00
Rasmus Wriedt Larsen
97571e0b4f Python: Add modeling of peewee 2021-06-25 17:50:59 +02:00
Rasmus Wriedt Larsen
c476c89de5 Python: Add tests for peewee 2021-06-25 16:08:57 +02:00
Rasmus Wriedt Larsen
68cfeb0b5c Python: Model logging from the logging module 2021-06-25 14:26:35 +02:00
Rasmus Wriedt Larsen
36c9ceb13b Python: Add Logging concept 2021-06-25 14:26:35 +02:00
Rasmus Wriedt Larsen
7017beca47 Python: Model CookieWrite for twisted
Had to split the call to `request.cookies.append` since inline
expectation tests didn't like the expectation that contained `=` :(
2021-06-24 17:34:43 +02:00
Rasmus Wriedt Larsen
4606444b85 Python: Model CookieWrite for flask 2021-06-24 17:34:43 +02:00
Rasmus Wriedt Larsen
65c526df86 Python: Model CookieWrite for tornado 2021-06-24 17:34:43 +02:00
Rasmus Wriedt Larsen
9340d658a4 Python: Model CookieWrite for django 2021-06-24 17:34:43 +02:00
Rasmus Wriedt Larsen
226425e831 Python: Model CookieWrite for aiohttp 2021-06-24 17:34:43 +02:00
Rasmus Wriedt Larsen
e1af1f11ee Python: Add HTTP::Server::CookieWrite concept
along with tests, but no implementations (to ease reviewing).

---

I've put quite some thinking into what to call our concept for this.

[JS has `CookieDefinition`](581f4ed757/javascript/ql/src/semmle/javascript/frameworks/HTTP.qll (L148-L187)), but I couldn't find a matching concept in any other languages.

We used to call this [`CookieSet`](f07a7bf8cf/python/ql/src/semmle/python/web/Http.qll (L76)) (and had a corresponding `CookieGet`).

But for headers, [Go calls this `HeaderWrite`](cd1e14ed09/ql/src/semmle/go/concepts/HTTP.qll (L97-L131)) and [JS calls this `HeaderDefinition`](581f4ed757/javascript/ql/src/semmle/javascript/frameworks/HTTP.qll (L23-L46))

I think it would be really cool if we have a naming scheme that means the name for getting the value of a header on a incoming request is obvious. I think `HeaderWrite`/`HeaderRead` fulfils this best. We could go with `HeaderSet`/`HeaderGet`, but they feel a bit too vague to me. For me, I'm so used to talking about def-use, that I would immediately go for `HeaderDefinition` and `HeaderUse`, which could work, but is kinda strange.

So in the end that means I went with `CookieWrite`, since that allows using a consistent naming scheme for the future :)
2021-06-24 17:34:43 +02:00
Rasmus Wriedt Larsen
686638a65f Merge pull request #6049 from RasmusWL/jmespath
Python: Add modeling of `jmespath`
2021-06-24 11:13:19 +02:00
CodeQL CI
469e709113 Merge pull request #6055 from RasmusWL/rsa-modeling
Approved by yoff
2021-06-23 08:35:25 -07:00
Rasmus Wriedt Larsen
447099a1df Python: Update jmespath tests 2021-06-23 13:32:19 +02:00
Rasmus Wriedt Larsen
902b450b12 Python: Also model pathlib.Path().open().write()
And this transition to type-trackers also helped fix the missing path
through function calls 👍
2021-06-23 10:50:04 +02:00
Rasmus Wriedt Larsen
39ec8701ca Python: Add FileSystemWriteAccess concept
I made `FileSystemWriteAccess` be a subclass of `FileSystemAccess` (like in [JS](64001cc02c/javascript/ql/src/semmle/javascript/Concepts.qll (L68-L74))), but then I started wondering about how I could  give a good result for `getAPathArgument`, and what would a good result even be? The argument to the `open` call, or the object that the `write` method is called on? I can't see how doing either of these enables us to do anything useful...

So I looked closer at how JS uses `FileSystemWriteAccess`:

1. as sink for zip-slip: 7c51dff0f7/javascript/ql/src/semmle/javascript/security/dataflow/ZipSlipCustomizations.qll (L121)
2. as sink for downloading unsafe files (identified through their extension) through non-secure connections: 89ef6ea4eb/javascript/ql/src/semmle/javascript/security/dataflow/InsecureDownloadCustomizations.qll (L134-L150)
3. as sink for writing untrusted data to a local file  93b1e59d62/javascript/ql/src/semmle/javascript/security/dataflow/HttpToFileAccessCustomizations.qll (L43-L46)

for the 2 first sinks, it's important that `getAPathArgument` has a proper result... so that solves the problem, and highlights that it _can_ be important to give proper results for `getAPathArgument` (if possible).

So I'm trying to do best effort for `f = open(...); f.write(...)`, but with this current code we won't always be able to give a result (as highlighted by the tests). It will also be the case that there are multiple `FileSystemAccess` with the same path-argument, which could be a little strange.

overall, I'm not super confident about the way this new concept and implementation turned out, but it also seems like the best I could come up with right now...

The obvious alternative solution is to NOT make `FileSystemWriteAccess` a subclass of `FileSystemAccess`, but I'm not very tempted to go down this path, given the examples of this being useful above, and just the general notion that we should be able to model writes as being a specialized kind of `FileSystemAccess`.
2021-06-23 10:50:04 +02:00
Rasmus Wriedt Larsen
6a6d6fbe92 Python: Add leading space in some inline tests 2021-06-23 10:50:04 +02:00
Rasmus Wriedt Larsen
13609b2888 Python: Move pathlib tests to Python 3 only tests 2021-06-23 10:50:04 +02:00
Rasmus Wriedt Larsen
1c48aca630 Merge branch 'main' into jmespath 2021-06-21 15:26:45 +02:00
yoff
baf8d0a990 Merge pull request #6045 from RasmusWL/twisted
Python: Model twisted
2021-06-21 14:52:57 +02:00
Rasmus Wriedt Larsen
8208aebd7e Python: Apply suggestions from code review
Co-authored-by: yoff <lerchedahl@gmail.com>
2021-06-21 10:43:25 +02:00
Rasmus Wriedt Larsen
498703fc81 Python: Escaping only valid with both input/output defined
Problematic part is

```codeql
  /** A escape from string format with `markupsafe.Markup` as the format string. */
  private class MarkupEscapeFromStringFormat extends MarkupSafeEscape, Markup::StringFormat {
    override DataFlow::Node getAnInput() {
      result in [this.getArg(_), this.getArgByName(_)] and
      not result = Markup::instance()
    }

    override DataFlow::Node getOutput() { result = this }
  }
```

since the char-pred still holds even if `getAnInput` has no results...

I will say that doing it this way feels kinda dirty, and we _could_ fix
this by including the logic in `getAnInput` in the char-pred as well.
But as I see it, that would just lead to a lot of code duplication,
which isn't very nice.
2021-06-16 19:09:00 +02:00
Rasmus Wriedt Larsen
6539df6422 Python: Add ConceptsTest for MarkupSafe 2021-06-16 19:09:00 +02:00
Rasmus Wriedt Larsen
14de3bffb7 Python: Model MarkupSafe PyPI package
Since expectation tests had so many changes from ConceptsTest, I'm going
to do the changes for that on in a separate commit. The important part
is the changes to taint-tracking, which is highlighted in this commit.
2021-06-16 19:09:00 +02:00
Rasmus Wriedt Larsen
d18b9a2704 Python: Add markupsafe tests 2021-06-16 19:09:00 +02:00
Rasmus Wriedt Larsen
156b10cb59 Merge branch 'main' into promote-clickhouse 2021-06-15 11:30:19 +02:00
Rasmus Wriedt Larsen
53f7633662 Python: Model await request.post() as MultiDictProxy
as highlight as being quite easy to do by @yoff 👍
2021-06-11 14:53:30 +02:00
Rasmus Wriedt Larsen
df67028a1d Python: Model aiohttp.StreamReader 2021-06-11 12:06:53 +02:00