Commit Graph

1781 Commits

Author SHA1 Message Date
Arthur Baars
90f013d74f Merge pull request #4176 from aibaars/missing-qhelp
Add missing QHelp files
2020-09-02 16:12:42 +02:00
Rasmus Wriedt Larsen
4387d106aa Python: Fix formatting (last time, promise) 2020-09-02 15:36:50 +02:00
Rasmus Wriedt Larsen
34c5da563e Python: Move files in experiemntal dirs to be consistent
Except for dataflow (where we have a lot of changes, and I don't want to
introduce lots of merge conflicts right now).
2020-09-02 13:39:01 +02:00
Rasmus Wriedt Larsen
9c8b829d65 Python: Fix formatting 2020-09-02 13:27:35 +02:00
Calum Grant
29b3759655 Merge pull request #3961 from tausbn/python-add-typetracker
Python: Add type tracker and step summary implementation.
2020-09-02 09:42:14 +01:00
Rasmus Wriedt Larsen
0cc018fec0 Python: Taint tracking setup alá Go
\## TaintFlow sources

The class `RemoteFlowSource` is very similarly defined as the other languages [C++](ac22e7950c/cpp/ql/src/semmle/code/cpp/security/FlowSources.qll), [Java](6de612a566/java/ql/src/semmle/code/java/dataflow/FlowSources.qll), [C#](fddbce0b7b/csharp/ql/src/semmle/code/csharp/security/dataflow/flowsources/Remote.qll), [JS](78334af354/javascript/ql/src/semmle/javascript/security/dataflow/RemoteFlowSources.qll), and [Go](24b3133e0c/ql/src/semmle/go/security/FlowSources.qll). There are some minor differences:

- Java/C++ defines the class in `FlowSources.qll`
- C# uses `csharp/ql/src/semmle/code/csharp/security/dataflow/flowsources/Remote.qll`, and provide `StoredFlowSource` and `LocalFlowSource` in separate classes.
- JS uses `RemoteFlowSources.qll`.
- JS defines additional predicate `RemoteFlowSource.isUserControlledObject`
- Go uses the class name `UntrustedFlowSource`, but still defined in `ql/src/semmle/go/security/FlowSources.qll`
- Go uses the `::Range` pattern to allow both extensibility and refinement

The big difference is how a RemoteFlowSource is specified:

- Java and C# have all subclasses of `RemoteFlowSource` defined in the same file
- Go and JS defines subclasses for frameworks in the actual framework `.qll` file, and all frameworks are transitively imported by `import go` or `import javascript` (so subclasses are always in scope).
- C++ uses class `RemoteFlowFunction` to do all the heavy lifting (and its subclasses are transitively imported).

\### What we will do

Use file `RemoteFlowSource.qll`, define subclasses in framework library classes.

_Why? Personally I really like it, Go/JS is already doing it, and Tom expressed a preference for doing the same for C# (although that is not what they are doing today)._

Jonas gave this advice:
> Whether you split the definitions between multiple files or keep them all in one file, the property you want is that all definitions are included when the abstract class is included. Otherwise you can get unexpected results via transitive includes.

We will make imports of all frameworks in the same file that defines `RemoteFlowSource`, as it seems to be the least intrusive change. If that turns out to be a problem, we can also move them to `python.qll` (the other way is not so easy).

\## TaintFlow sinks

[JS](473787a426/javascript/ql/src/semmle/javascript/Concepts.qll) and [Go](ecff1e6a16/ql/src/semmle/go/Concepts.qll) defines abstract base classes for interesting sinks in `Concepts.qll` (and all uses the `::Range` pattern in Go).

I really like this idea, since it allows multiple queries to reuse the same sink definitions, and it makes it _easy_ to discover what default sinks are available.

Personally I'm not 100% on board with the naming, but I don't have any good reason to change the naming convention.

\## Framework modeling

Following the model from Go ([example](https://github.com/github/codeql-go/blob/main/ql/src/semmle/go/frameworks/Gin.qll)), I propose that we make every definition in a framework modeling `private`. This allows some greater flexibility in changing our modeling, since we don't need to think about keeping deprecated versions around for a whole year.

It _does_ have the downside that someone writing a query can't reuse the classes/predicates for a framework, but it didn't seem to be too big of a concern. If we need to provide access, we can always make the definitions non-private (the other way is not so easy).

\## Customizations

Also introduced `Customizations.qll` like in JS/Java/Go (to replace `site.qll`)
2020-09-01 14:37:11 +02:00
Taus Brock-Nannestad
6a96c53d15 Python: Add missing getNode invocation 2020-09-01 14:04:31 +02:00
Taus Brock-Nannestad
26d14aba98 Python: Use nodeFrom/nodeTo instead of pred/succ 2020-09-01 14:00:30 +02:00
Arthur Baars
aedfa47cb4 Add missing QHelp files 2020-09-01 12:46:57 +02:00
Rasmus Wriedt Larsen
cda88a5e64 Python: Refactor: use DataFlow::Node.asExpr() 2020-09-01 11:53:06 +02:00
Rasmus Wriedt Larsen
ddc55a18cf Python: Fix taint handling of copy.deepcopy
(test results didn't change)

Thanks @yoff 👍
2020-09-01 11:50:46 +02:00
Taus Brock-Nannestad
ec64606d5a Python: Remove CopyStep branch type 2020-08-31 17:23:02 +02:00
Taus Brock-Nannestad
eb6443df21 Merge branch 'python-add-typetracker' of github.com:tausbn/ql into python-add-typetracker 2020-08-31 17:22:13 +02:00
Taus
8e1f99af99 Python: Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-08-31 17:20:12 +02:00
Taus Brock-Nannestad
06103f4ff2 Python: Consistently use attribute/attr 2020-08-31 17:16:31 +02:00
Rasmus Wriedt Larsen
cf2eacd7a6 Python: Adjust additional taint after PostUpdateNode addition
Still no results though :(
2020-08-31 14:59:29 +02:00
Rasmus Wriedt Larsen
4e73abc254 Merge branch 'main' into python-more-additional-taint-steps 2020-08-31 14:34:42 +02:00
Taus Brock-Nannestad
5d853e840a Merge branch 'main' into python-add-typetracker 2020-08-28 19:59:58 +02:00
Taus Brock-Nannestad
fbe8b64dd4 Python: Add support for attribute reads and writes 2020-08-28 19:55:14 +02:00
Rasmus Lerchedahl Petersen
6b8d9f2a77 Merge branch 'main' of github.com:github/codeql into SharedDataflow_PostUpdateNodes 2020-08-28 13:01:14 +02:00
Rasmus Lerchedahl Petersen
9503c5d8bb Python: Add post-update nodes 2020-08-28 12:59:11 +02:00
Rasmus Wriedt Larsen
496d856c48 Python: Reformualte explanation of experience from JS 2020-08-28 10:49:33 +02:00
Taus
1206ff5889 Merge pull request #4150 from RasmusWL/python-dataflow-private-import
Python: Make import of python private in shared dataflow
2020-08-27 18:05:55 +02:00
Taus Brock-Nannestad
7112aa2e9a Merge branch 'main' into python-add-typetracker 2020-08-27 17:05:26 +02:00
Taus
e7322d114f Merge pull request #4077 from yoff/MagicMethods
Python: Add support for magic methods
2020-08-27 13:20:56 +02:00
Taus
d3175a7899 Merge pull request #4110 from yoff/SharedDataflow_ParsimoniousFlowNodes
Python: Shared dataflow, parsimonious flow nodes
2020-08-27 13:19:23 +02:00
Rasmus Wriedt Larsen
909bff2313 Python: Make import of python private in shared dataflow 2020-08-27 11:48:56 +02:00
Rasmus Wriedt Larsen
d0081dfbfa Python: Attempt at taint step for list.append/set.add 2020-08-27 10:57:07 +02:00
Esben Sparre Andreasen
67278d9c93 Merge pull request #4141 from esbena/js/clarify-sanitization
JS: make sanitization a "common" technique rather than "important"
2020-08-27 08:08:17 +02:00
Rasmus Wriedt Larsen
bd21fc5601 Python: Autoformat 2020-08-26 20:37:48 +02:00
Rasmus Wriedt Larsen
423139bc22 Python: Add additional taint steps for iterable-unpacking 2020-08-26 20:21:15 +02:00
Rasmus Wriedt Larsen
afb160fbbb Python: Add additional taint steps for for-iteration 2020-08-26 20:18:31 +02:00
Rasmus Wriedt Larsen
e2a89aa296 Python: Add additional taint steps for copy
deepcopy was already handled somehow, don't really know how :D
2020-08-26 19:39:38 +02:00
Rasmus Wriedt Larsen
b974dadca1 Python: Add additional taint steps for containers 2020-08-26 19:39:37 +02:00
Rasmus Wriedt Larsen
32f9d30136 Python: Add syntactic taint steps for json methods 2020-08-26 19:39:36 +02:00
Rasmus Lerchedahl Petersen
bf6211f639 Merge branch 'main' of github.com:github/codeql into SharedDataflow_ParsimoniousFlowNodes 2020-08-26 17:50:17 +02:00
Rasmus Lerchedahl Petersen
6c173047e6 Merge branch 'MagicMethods' of github.com:yoff/codeql into MagicMethods 2020-08-26 17:43:27 +02:00
Rasmus Lerchedahl Petersen
47e35c530d Merge branch 'main' of github.com:github/codeql into MagicMethods 2020-08-26 17:42:44 +02:00
Taus
b1946c60dd Merge pull request #4127 from RasmusWL/python-tainttracking-fstring
Python: Handle f-strings in (current) taint tracking
2020-08-26 16:06:01 +02:00
Esben Sparre Andreasen
89305865d0 JS: make sanitization a "common" technique rather than "important" 2020-08-26 15:41:54 +02:00
Rasmus Lerchedahl Petersen
fae915bbb5 Python: QL doc 2020-08-25 21:02:17 +02:00
Rasmus Lerchedahl Petersen
551ae42fb9 Merge branch 'main' of github.com:github/codeql into SharedDataflow_NestedComprehensions 2020-08-25 15:45:20 +02:00
Rasmus Lerchedahl Petersen
d67f57a0bb Python: Remove dead code 2020-08-25 15:39:37 +02:00
Taus
000fa33d54 Merge pull request #4013 from yoff/SharedDataflow_SequenceFlow
Python: Shared dataflow: Content flow
2020-08-25 15:38:14 +02:00
Rasmus Lerchedahl Petersen
56b78a664e Python: Store step for generators 2020-08-25 15:36:26 +02:00
Rasmus Lerchedahl Petersen
ecf3928ed1 Python: Handle comprehensions with multiple fors 2020-08-25 15:21:08 +02:00
Rasmus Wriedt Larsen
cf121cc4d0 Python: TaintTracking: stringMethods => stringManipualtion 2020-08-25 13:05:27 +02:00
Rasmus Lerchedahl Petersen
1cdb6be531 Merge branch 'main' of github.com:github/codeql into SharedDataflow_NestedComprehensions 2020-08-25 13:05:13 +02:00
Rasmus Wriedt Larsen
238e0845aa Python: Minor refactoring 2020-08-25 12:50:41 +02:00
Rasmus Wriedt Larsen
0439b83c60 Python: Taint when using unicode 2020-08-25 12:50:32 +02:00