Commit Graph

2489 Commits

Author SHA1 Message Date
Rasmus Lerchedahl Petersen
17273dd27e Python: Add FileSystemAccess open 2020-10-21 22:01:49 +02:00
Rasmus Lerchedahl Petersen
4570c29a11 Python: port query 2020-10-21 21:40:42 +02:00
Rasmus Lerchedahl Petersen
eb5ed23354 Python: Add TaintTracking2 2020-10-21 21:39:50 +02:00
Rasmus Lerchedahl Petersen
da77cbb3d0 Python: concepts PathCheck and PathNormalization
Should they be in a module?
2020-10-21 21:37:43 +02:00
Rasmus Wriedt Larsen
278c071fe6 Python: Modernise werkzeug FileStorage modeling 2020-10-21 20:36:40 +02:00
Rasmus Wriedt Larsen
d0fdb542e5 Python: Modernise werkzeug MultiDict modeling 2020-10-21 20:30:20 +02:00
Rasmus Wriedt Larsen
b6bd70a5da Python: Modernise flask library modeling
Two interesting things happened while doing this:

1. I found out that you can't use the same name to define a submodule as any
parent module. So we need give unique names to the top-level module, and the
module for modeling the `flask.Flask` class. I randomly choose a new name for
the top-level module to get things moving (and not be stuck in bikeshedding
forever).

2. With this new setup, I wanted to expose the `route` and `add_url_rule`
methods on instances of `flask.Flask`. It wasn't quite obvious how to do so. I
simply lumped them next to `classRef()` and `instance()`, without too much
care. I did consider putting them inside a `instance` module, which would allow
you to access them by `flask::Flask::instance::route()`, but I wasn't quite
sure, and just did something easy to get moving.
2020-10-21 20:30:14 +02:00
Rasmus Wriedt Larsen
62d665ecb3 Python: Fix shared QLDoc for InstanceSource 2020-10-21 19:55:40 +02:00
Rasmus Wriedt Larsen
047a326183 Python: Remove reference to old PR
These have been added to internal tracking issue instead
2020-10-21 19:53:25 +02:00
Rasmus Lerchedahl Petersen
2e8cbbd866 Python: Add concept FileSystemAccess 2020-10-21 17:35:21 +02:00
Rasmus Wriedt Larsen
a6abee9b3a Merge pull request #4476 from yoff/python-port-sql-injection
Python: Port SqlInjection
2020-10-21 15:55:19 +02:00
yoff
ea4ea6b3e6 Merge pull request #4529 from tausbn/python-remove-cartesian-product-in-tkwoverflownode
Python: Remove bad join in `getCallableScope`
2020-10-21 15:36:38 +02:00
Rasmus Lerchedahl Petersen
060481053a Python: Add note about incompleteness
I was going to do this in an issue, but it makes sense
to have it in the code. We could still add an issue as well.
2020-10-21 15:15:19 +02:00
Rasmus Lerchedahl Petersen
c57c798bfa Python: Add TODO 2020-10-21 15:10:40 +02:00
yoff
ee5221abb4 Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-10-21 15:08:16 +02:00
Taus Brock-Nannestad
31c169daac Python: Remove bad join in getCallableScope
Clause timing report had this suspicious entry

```
CommandInjection.ql-12:DataFlowPublic::Node::getCallableScope#bbf .................. 7.2s
    (4 evaluations with max 6.4s in DataFlowPublic::Node::getCallableScope#bbf/3@i3#119d7b)
```

which indeed was a bad join:
```
Tuple counts for DataFlowPublic::Node::getCallableScope#bbf:
293509   ~2%     {3} r1 = JOIN DataFlowPublic::Node::getCallableScope#bbf#prev_delta AS L WITH DataFlowPublic::TNode#f AS R ON FIRST 1 OUTPUT L.<1>, L.<0>, L.<2>
22337162 ~0%     {3} r2 = JOIN r1 WITH Scope::Scope::getEnclosingScope_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT r1.<1>, r1.<2>, R.<1>
22337162 ~0%     {3} r3 = r2 AND NOT DataFlowPublic::Node::getCallableScope#bbf#prev AS R(r2.<0>, r2.<2>, r2.<1>)
22337162 ~0%     {3} r4 = SCAN r3 OUTPUT r3.<0>, r3.<2>, r3.<1>
722      ~1%     {3} r5 = JOIN r4 WITH m#DataFlowPublic::Node::getCallableScope#bbf AS R ON FIRST 2 OUTPUT r4.<0>, r4.<1>, r4.<2>
722      ~1%     {3} r6 = JOIN r5 WITH m#DataFlowPublic::Node::getCallableScope#bbf AS R ON FIRST 2 OUTPUT r5.<0>, r5.<2>, r5.<1>
722      ~1%     {3} r7 = r6 AND NOT project#DataFlowPrivate::DataFlowCallable::getScope_dispred#ff AS R(r6.<2>)
722      ~1%     {3} r8 = SCAN r7 OUTPUT r7.<0>, r7.<2>, r7.<1>
                 return r8
```

In this case, the join went away by simply moving the helper predicate
out of the class it was situated in (and since it doesn't mention
`this`, it didn't really belong there in the first place).

Result:
```
DataFlowPublic.qll-8:DataFlowPublic::getCallableScope#ff ........................... 26ms
    (4 evaluations with max 15ms in DataFlowPublic::getCallableScope#ff/2@i3#709a9e)
```
2020-10-21 14:45:05 +02:00
yoff
9bd808c924 Merge pull request #4528 from tausbn/python-remove-cartesian-product-in-tkwoverflownode
Python: Prevent early join on `argName` in `getArg`
2020-10-21 14:43:39 +02:00
Rasmus Lerchedahl Petersen
53ff1a32c1 Merge branch 'main' of github.com:github/codeql into python-port-sql-injection 2020-10-21 14:38:02 +02:00
Rasmus Lerchedahl Petersen
77d4cbc0df Python: Only allow unsafe positional args to extra 2020-10-21 14:21:36 +02:00
Rasmus Lerchedahl Petersen
3a416bce2d Python: Move test annotation 2020-10-21 14:18:16 +02:00
Rasmus Lerchedahl Petersen
4571b3188c Python: Fix false negative 2020-10-21 14:16:35 +02:00
Rasmus Lerchedahl Petersen
03c62fd267 Python: Fix typo in test case 2020-10-21 14:03:46 +02:00
yoff
75357727c4 Merge pull request #4490 from RasmusWL/python-model-django-sources
Python: model Django HttpRequest as RemoteFlowSource
2020-10-21 13:46:51 +02:00
Rasmus Lerchedahl Petersen
7087522ed9 Python: Move doc reference 2020-10-21 13:30:38 +02:00
Rasmus Lerchedahl Petersen
e49c7d64bd Python: test for keyword arguments to extra 2020-10-21 13:28:12 +02:00
Rasmus Lerchedahl Petersen
d249b51a5e Python: Add test-case for indirect RawSQL 2020-10-21 13:23:19 +02:00
Taus Brock-Nannestad
eb3333c0ce Python: Prevent early join on argName in getArg 2020-10-21 13:23:09 +02:00
Rasmus Lerchedahl Petersen
e51543ea79 Python: allow any positional argument in annotate 2020-10-21 13:17:20 +02:00
Rasmus Lerchedahl Petersen
2c855c739b Python: Ignore relatively safe arguments of extra 2020-10-21 13:15:46 +02:00
Rasmus Lerchedahl Petersen
6805fb63cc Python: Use modern pattern for RawSQL class 2020-10-21 13:14:32 +02:00
Rasmus Lerchedahl Petersen
e44247bb00 Python: Add links to function docs 2020-10-21 13:03:14 +02:00
yoff
8e7e107365 Apply suggestions from code review
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-10-21 12:55:38 +02:00
Rasmus Wriedt Larsen
5874a7b422 Merge pull request #4488 from yoff/SharedDataflow_ArgumentPassingTests
Python: Shared dataflow, argument passing tests
2020-10-21 12:01:37 +02:00
CodeQL CI
eaed93fa7d Merge pull request #4513 from RasmusWL/python-model-fabric
Approved by yoff
2020-10-21 01:58:19 -07:00
Rasmus Lerchedahl Petersen
9ee5a01d7e Python: Reword comment on isBarrierIn 2020-10-21 10:30:40 +02:00
Rasmus Wriedt Larsen
b0af0b94d0 Python: Fix grammar
Co-authored-by: yoff <lerchedahl@gmail.com>
2020-10-21 09:58:37 +02:00
Erik Krogh Kristensen
e18cf08d99 documentation changes based on review 2020-10-21 09:45:16 +02:00
Erik Krogh Kristensen
c1dba2ee9f add a few shouldPrint calls to improve performance 2020-10-21 09:37:53 +02:00
Rasmus Lerchedahl Petersen
90d0cff384 Python: Use flask routing 2020-10-21 00:30:16 +02:00
Rasmus Lerchedahl Petersen
383d846396 Python: address review
- smooth out future merge
- keyword argument for execute
2020-10-21 00:15:05 +02:00
Rasmus Lerchedahl Petersen
e1dfbc0486 Python: address review 2020-10-20 23:59:44 +02:00
Erik Krogh Kristensen
3306b59a14 Update python/ql/src/semmle/python/PrintAst.qll
Co-authored-by: yoff <lerchedahl@gmail.com>
2020-10-20 23:19:47 +02:00
yoff
01845d1278 Update python/ql/src/experimental/semmle/python/frameworks/Django.qll
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-10-20 21:43:15 +02:00
Rasmus Wriedt Larsen
6053157b09 Python: Fix QLDoc to not mention sudo method on fabric Group 2020-10-20 17:30:09 +02:00
Taus Brock-Nannestad
0e2ec0dfb4 Python: Remove cartesian product in TKwOverflowNode
With CP:

(0s) Tuple counts for dom#DataFlowPublic::TKwOverflowNode#ff:
1209    ~0%       {2} r1 = JOIN project#AstGenerated::Function_::getKwarg_dispred#ff AS L WITH ObjectAPI::CallableValue::getScope_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, L.<0>
4329    ~0%       {3} r2 = JOIN r1 WITH DataFlowPrivate::ArgumentPassing::connects#bb_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r1.<1>, r1.<0>
7819    ~2%       {4} r3 = JOIN r2 WITH Flow::CallNode::getArgByName_dispred#fff AS R ON FIRST 1 OUTPUT r2.<1>, r2.<2>, r2.<0>, R.<1>
7114    ~1%       {4} r4 = r3 AND NOT Function::Function::getArgByName_dispred#fff_01#antijoin_rhs AS R(r3.<0>, r3.<3>)
7114    ~76%      {2} r5 = SCAN r4 OUTPUT r4.<2>, r4.<1>
1123    ~0%       {1} r6 = JOIN project#Exprs::Call::getKwargs_dispred#ff AS L WITH py_flow_bb_node_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>
1123    ~0%       {1} r7 = JOIN r6 WITH Flow::CallNode#class#f AS R ON FIRST 1 OUTPUT r6.<0>
1357707 ~0%       {2} r8 = JOIN r7 WITH project#AstGenerated::Function_::getKwarg_dispred#ff AS R CARTESIAN PRODUCT OUTPUT R.<0>, r7.<0>
1357707 ~0%       {2} r9 = JOIN r8 WITH ObjectAPI::CallableValue::getScope_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT r8.<1>, R.<1>
1364821 ~0%       {2} r10 = r5 \/ r9
                  return r10

Without CP:

(13s) Tuple counts for dom#DataFlowPublic::TKwOverflowNode#ff:
1209    ~0%       {2} r1 = JOIN project#AstGenerated::Function_::getKwarg_dispred#ff AS L WITH ObjectAPI::CallableValue::getScope_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, L.<0>
19175   ~4%       {3} r2 = JOIN r1 WITH DataFlowPrivate::ArgumentPassing::connects#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r1.<1>, r1.<0>
7819    ~2%       {4} r3 = JOIN r2 WITH Flow::CallNode::getArgByName_dispred#fff AS R ON FIRST 1 OUTPUT r2.<1>, r2.<2>, r2.<0>, R.<1>
7114    ~1%       {4} r4 = r3 AND NOT Function::Function::getArgByName_dispred#fff_01#antijoin_rhs AS R(r3.<0>, r3.<3>)
7114    ~76%      {2} r5 = SCAN r4 OUTPUT r4.<2>, r4.<1>
1123    ~0%       {1} r6 = JOIN project#Exprs::Call::getKwargs_dispred#ff AS L WITH py_flow_bb_node_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>
574     ~0%       {2} r7 = JOIN r6 WITH DataFlowPrivate::ArgumentPassing::connects#ff AS R ON FIRST 1 OUTPUT R.<1>, r6.<0>
524     ~1%       {3} r8 = JOIN r7 WITH ObjectAPI::CallableValue::getScope_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r7.<1>, r7.<0>
291     ~0%       {2} r9 = JOIN r8 WITH project#AstGenerated::Function_::getKwarg_dispred#ff AS R ON FIRST 1 OUTPUT r8.<1>, r8.<2>
7405    ~72%      {2} r10 = r5 \/ r9
                  return r10
2020-10-20 17:29:26 +02:00
Rasmus Wriedt Larsen
cd116fc5f2 Python: Fix grammar in QLDoc
Co-authored-by: yoff <lerchedahl@gmail.com>
2020-10-20 17:26:33 +02:00
Rasmus Wriedt Larsen
6920f3012c Python: Django route handlers in different file now works
Fixed by https://github.com/github/codeql/pull/4514
2020-10-20 15:41:14 +02:00
Rasmus Wriedt Larsen
c8441dc4fb Merge branch 'main' into python-model-django-sources 2020-10-20 15:38:20 +02:00
yoff
17155b64f5 Merge pull request #4514 from tausbn/python-add-module-boundary-flow-steps
Python: Add module boundary flow steps
2020-10-20 14:36:10 +02:00
Rasmus Wriedt Larsen
80adbdfbc1 Python: Mark unhandled django route handlers with f-:
That is playing more nicely with the expected usage of the inline-tests.
2020-10-20 13:44:34 +02:00