Commit Graph

2529 Commits

Author SHA1 Message Date
Taus
f4f47bd5ed Merge pull request #4236 from RasmusWL/python-experimental-taint-sanitizers
Python: Expand on taint sanitizer tests
2020-09-09 17:51:24 +02:00
Rasmus Wriedt Larsen
b8e057f7ad Python: isSanitizerGuard test is future work 2020-09-09 15:57:53 +02:00
Rasmus Lerchedahl Petersen
b1567827a0 Python: Repair flow out of post-update nodes 2020-09-09 15:52:07 +02:00
Rasmus Wriedt Larsen
2172fb6e65 Dataflow: s/data flow/taint propagation/ in QLDoc for sanitizers 2020-09-09 14:30:33 +02:00
Rasmus Wriedt Larsen
d90f0be2c4 Dataflow: defaultTaintBarrier => defaultTaintSanitizer
Just keeping things a bit more consistent :)
2020-09-09 14:11:56 +02:00
Rasmus Wriedt Larsen
ab8cc23ce7 Python: Expand on taint sanitizer tests
Most interesting to look at the custom sanitizers. Once we have use-use flow, we
should handle this case:

```
s = TAINTED_STRING
emulated_authentication_check(s)
ensure_not_tainted(s)
```
2020-09-09 13:57:25 +02:00
Rasmus Lerchedahl Petersen
9e59d79a72 Python: Repair flow from pre-update nodes 2020-09-09 13:51:24 +02:00
Rasmus Lerchedahl Petersen
ce7f82ddc6 Python: Add def-use jump-steps 2020-09-09 13:27:14 +02:00
Rasmus Lerchedahl Petersen
c661f43316 Python: Port use-use implementation from Java 2020-09-09 12:19:40 +02:00
CodeQL CI
22b3b0a5f1 Merge pull request #3953 from RasmusWL/python-more-call-graph-tracing
Approved by tausbn
2020-09-07 17:34:14 +01:00
Rasmus Wriedt Larsen
61998afc56 Python: Remove unnecessary comment
Was introduced in 5d031d7abe when I actually fixed
the loop variable capture problem.
2020-09-07 15:06:07 +02:00
Rasmus Wriedt Larsen
fb37330f5e Python: Fix grammar
Co-authored-by: Taus <tausbn@github.com>
2020-09-07 14:59:07 +02:00
yoff
2a70da4da6 Merge pull request #4210 from tausbn/python-remove-spurious-global-flow
Python: Remove implicit uses from `essaFlowStep`
2020-09-07 10:16:18 +02:00
yoff
ae9f58489d Merge pull request #4159 from RasmusWL/python-port-dataflow-tests
Python: port dataflow tests
2020-09-07 09:54:12 +02:00
Taus Brock-Nannestad
266365d0b6 Python: Update strange-essaflow test 2020-09-04 16:38:59 +02:00
Taus Brock-Nannestad
260763a748 Python: Update consistency test results 2020-09-04 16:35:49 +02:00
Taus Brock-Nannestad
df1448cfb2 Merge branch 'main' into python-remove-spurious-global-flow 2020-09-04 16:28:03 +02:00
Taus
5ffc959e32 Merge pull request #4211 from RasmusWL/python-strange-essaflow
Python: Add example of strange DataFlow::jumpStep
2020-09-04 15:47:22 +02:00
Rasmus Wriedt Larsen
720e8c432e Python: Update comment for validTest expectations 2020-09-04 15:08:57 +02:00
Rasmus Wriedt Larsen
cf57afd102 Python: Add example of strange DataFlow::jumpStep
The example code is just copied from command injection tests, that is not too
important. The important part is that `jumpStep` says there is flow from the
import of `os` to `app.route()` :O
2020-09-04 14:39:16 +02:00
Taus
59c7907ee4 Merge pull request #4207 from RasmusWL/python-typetracker-small-fixes
Python: Small fixes for TypeTracker
2020-09-04 14:30:10 +02:00
Taus Brock-Nannestad
98266ad5da Python: Remove implicit uses from essaFlowStep 2020-09-04 14:22:43 +02:00
Rasmus Wriedt Larsen
6aae75799e Python: Fix import in type tracking test
Fixes 7855576a6
2020-09-04 13:36:25 +02:00
CodeQL CI
fd715a5b66 Merge pull request #4179 from RasmusWL/python-tainttracking-ala-go
Approved by tausbn, yoff
2020-09-04 12:20:12 +01:00
yoff
7a00fbc654 Merge pull request #4154 from RasmusWL/python-more-complete-dataflow-tests
Python more complete dataflow tests
2020-09-04 11:35:24 +02:00
Rasmus Wriedt Larsen
f12fa52e22 Python: Update inline example for TypeTracker usage 2020-09-04 11:11:30 +02:00
Rasmus Wriedt Larsen
189c94f9e3 Python: Add TypeTracker::end()
Copied from JS
2020-09-04 11:10:10 +02:00
Rasmus Wriedt Larsen
7855576a69 Python: TypeTracker only exposes its own interface
This is especially important if the TypeTracker needs to be publicly imported by
DataFlowPublic.
2020-09-04 10:58:20 +02:00
Rasmus Wriedt Larsen
29bf98ad26 Python: Fix CUSTOM_SOURCE dataflow regression test 2020-09-03 15:03:53 +02:00
Rasmus Wriedt Larsen
febbe1229a Merge branch 'main' into python-more-complete-dataflow-tests 2020-09-03 14:58:20 +02:00
Rasmus Lerchedahl Petersen
aad51af4ce Python: use concrete iterable source 2020-09-03 11:25:41 +02:00
Rasmus Wriedt Larsen
b958c3b833 Python: Update comment for test8 2020-09-03 11:13:32 +02:00
Rasmus Wriedt Larsen
bf34b07605 Python: Add a few taint tests for default sanitizer
specifically the ones removes from dataflow tests in https://github.com/yoff/codeql/pull/1
2020-09-02 16:56:05 +02:00
Taus
8e86d56bce Merge pull request #4189 from RasmusWL/python-experimental-file-structure
Python: Move files in experimental dirs to be consistent
2020-09-02 16:34:35 +02:00
Arthur Baars
90f013d74f Merge pull request #4176 from aibaars/missing-qhelp
Add missing QHelp files
2020-09-02 16:12:42 +02:00
Rasmus Wriedt Larsen
4387d106aa Python: Fix formatting (last time, promise) 2020-09-02 15:36:50 +02:00
Rasmus Wriedt Larsen
8aab0c8be7 Python: Fix .qlref for experimental security tests 2020-09-02 15:35:50 +02:00
Rasmus Wriedt Larsen
bf3a266f58 Python: dataflow regression tests: remove taint tracking tests
they will be reintroduced in an other PR
2020-09-02 13:51:00 +02:00
Rasmus Wriedt Larsen
552637a446 Python: dataflow regression tests: fix flow_in_iteration 2020-09-02 13:50:24 +02:00
Rasmus Wriedt Larsen
4977790617 Python: dataflow regression tests: fix source2 2020-09-02 13:49:56 +02:00
Rasmus Wriedt Larsen
34c5da563e Python: Move files in experiemntal dirs to be consistent
Except for dataflow (where we have a lot of changes, and I don't want to
introduce lots of merge conflicts right now).
2020-09-02 13:39:01 +02:00
Rasmus Wriedt Larsen
9c8b829d65 Python: Fix formatting 2020-09-02 13:27:35 +02:00
Calum Grant
29b3759655 Merge pull request #3961 from tausbn/python-add-typetracker
Python: Add type tracker and step summary implementation.
2020-09-02 09:42:14 +01:00
Rasmus Wriedt Larsen
ab06c459f4 Python: Make validTest error on empty output again
I accidentially disabled that when introducing the ability to handle more than
one OK.
2020-09-01 14:42:11 +02:00
Rasmus Wriedt Larsen
0cc018fec0 Python: Taint tracking setup alá Go
\## TaintFlow sources

The class `RemoteFlowSource` is very similarly defined as the other languages [C++](ac22e7950c/cpp/ql/src/semmle/code/cpp/security/FlowSources.qll), [Java](6de612a566/java/ql/src/semmle/code/java/dataflow/FlowSources.qll), [C#](fddbce0b7b/csharp/ql/src/semmle/code/csharp/security/dataflow/flowsources/Remote.qll), [JS](78334af354/javascript/ql/src/semmle/javascript/security/dataflow/RemoteFlowSources.qll), and [Go](24b3133e0c/ql/src/semmle/go/security/FlowSources.qll). There are some minor differences:

- Java/C++ defines the class in `FlowSources.qll`
- C# uses `csharp/ql/src/semmle/code/csharp/security/dataflow/flowsources/Remote.qll`, and provide `StoredFlowSource` and `LocalFlowSource` in separate classes.
- JS uses `RemoteFlowSources.qll`.
- JS defines additional predicate `RemoteFlowSource.isUserControlledObject`
- Go uses the class name `UntrustedFlowSource`, but still defined in `ql/src/semmle/go/security/FlowSources.qll`
- Go uses the `::Range` pattern to allow both extensibility and refinement

The big difference is how a RemoteFlowSource is specified:

- Java and C# have all subclasses of `RemoteFlowSource` defined in the same file
- Go and JS defines subclasses for frameworks in the actual framework `.qll` file, and all frameworks are transitively imported by `import go` or `import javascript` (so subclasses are always in scope).
- C++ uses class `RemoteFlowFunction` to do all the heavy lifting (and its subclasses are transitively imported).

\### What we will do

Use file `RemoteFlowSource.qll`, define subclasses in framework library classes.

_Why? Personally I really like it, Go/JS is already doing it, and Tom expressed a preference for doing the same for C# (although that is not what they are doing today)._

Jonas gave this advice:
> Whether you split the definitions between multiple files or keep them all in one file, the property you want is that all definitions are included when the abstract class is included. Otherwise you can get unexpected results via transitive includes.

We will make imports of all frameworks in the same file that defines `RemoteFlowSource`, as it seems to be the least intrusive change. If that turns out to be a problem, we can also move them to `python.qll` (the other way is not so easy).

\## TaintFlow sinks

[JS](473787a426/javascript/ql/src/semmle/javascript/Concepts.qll) and [Go](ecff1e6a16/ql/src/semmle/go/Concepts.qll) defines abstract base classes for interesting sinks in `Concepts.qll` (and all uses the `::Range` pattern in Go).

I really like this idea, since it allows multiple queries to reuse the same sink definitions, and it makes it _easy_ to discover what default sinks are available.

Personally I'm not 100% on board with the naming, but I don't have any good reason to change the naming convention.

\## Framework modeling

Following the model from Go ([example](https://github.com/github/codeql-go/blob/main/ql/src/semmle/go/frameworks/Gin.qll)), I propose that we make every definition in a framework modeling `private`. This allows some greater flexibility in changing our modeling, since we don't need to think about keeping deprecated versions around for a whole year.

It _does_ have the downside that someone writing a query can't reuse the classes/predicates for a framework, but it didn't seem to be too big of a concern. If we need to provide access, we can always make the definitions non-private (the other way is not so easy).

\## Customizations

Also introduced `Customizations.qll` like in JS/Java/Go (to replace `site.qll`)
2020-09-01 14:37:11 +02:00
Taus Brock-Nannestad
6a96c53d15 Python: Add missing getNode invocation 2020-09-01 14:04:31 +02:00
Rasmus Lerchedahl Petersen
8b13a429b7 Python: Address review comments 2020-09-01 14:00:41 +02:00
Taus Brock-Nannestad
26d14aba98 Python: Use nodeFrom/nodeTo instead of pred/succ 2020-09-01 14:00:30 +02:00
Arthur Baars
aedfa47cb4 Add missing QHelp files 2020-09-01 12:46:57 +02:00
Rasmus Wriedt Larsen
c5e3333d10 Python: Update expected tests after last commit
I'm pushing too fast it seems
2020-09-01 12:01:34 +02:00