Commit Graph

2505 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
b2ce0fcb72 Python: Add post-update nodes to args of unresolved calls
Besides solving the problem with `setattr`, it also solved some old
problems with json library modeling (yay).
2022-02-04 11:51:53 +01:00
Erik Krogh Kristensen
5e23da813f rename named-parameters to keyword-parameters 2022-02-03 23:10:39 +01:00
Erik Krogh Kristensen
3801a158a8 remove module exporst nodes from API graphs 2022-02-03 23:10:39 +01:00
Erik Krogh Kristensen
3be3da2eb6 add recursive API-graph test 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
ef5818e243 support import * in ApiGraphs 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
16774ba285 add support for named parameters in API graphs 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
095c73f1fe redo the ApiGraph testing framework 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
66fd43fc3b add def edge for function returns 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
d8eea7ba4c property writes are def nodes 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
a908b219e9 more backtracking of def nodes, and lots of tests 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
038b032a43 get basic module exports to work in API-graphs 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
df9efbe778 get mimimal def nodes to work in python 2022-02-03 23:10:38 +01:00
Erik Krogh Kristensen
52ca0d168b move API-graph tests out of the experimental test folder 2022-02-03 23:10:37 +01:00
Harry Maclean
ab7fd89653 Merge pull request #7663 from github/hmac/api-graph-subclass
Ruby: Add basic subclassing support to API Graphs
2022-02-04 10:19:07 +13:00
Rasmus Wriedt Larsen
5cd08b8e8c Python: Ignore .isAbsent() from ClassCall
This means that DataFlowCall is only for resolvable calls, which might not seem
like a big thing in itself, but enables the next commit to actually work :P
2022-02-03 14:58:30 +01:00
Rasmus Wriedt Larsen
a5c2341204 Python: Add simple test of DataFlowCall
Notice the strange thing with treating `mypkg.foo(42)` as a ClassCall,
but completely ignoring `mypkg.subpkg.bar(43)` -- due to having the two
`ClassValue`s:

- `Missing module attribute mypkg.foo`
- `Missing module attribute mypkg.subpkg`

But not `Missing module attribute mypkg.subpkg` with the current import
structure.
2022-02-03 14:58:30 +01:00
Rasmus Wriedt Larsen
49b5d60229 Python: Use AttrRead/AttrWrite for attr read/store steps
Note that this doesn't actually add the desired flow from setattr, due
to missing post-update note. This will be fixed in later commit.
2022-02-03 14:58:30 +01:00
Rasmus Wriedt Larsen
5774459dfb Python: restrict AttrRead with AttrNode.isLoad() 2022-02-03 14:58:23 +01:00
Arthur Baars
33b97f3e0c Update synchronized files 2022-02-02 13:30:45 +01:00
Rasmus Wriedt Larsen
fb6b8eb394 Python: Add simple test of AttrRead/AttrWrite 2022-02-02 11:19:35 +01:00
Rasmus Wriedt Larsen
51bc6dcf7e Python: Add attributeClearStep 2022-02-02 11:19:35 +01:00
Rasmus Wriedt Larsen
d2b72a7547 Python: Expand fieldflow tests 2022-02-02 11:19:31 +01:00
Rasmus Wriedt Larsen
f6215f2300 Python: Refactor field-flow test 2022-02-01 17:59:03 +01:00
Rasmus Wriedt Larsen
cc4fe38fbd Python: Delete dedicated argumentRouting<N> tests
I feel like they don't bring any value anymore, since we have the nice
inline expectation tests. If I'm wrong, happy to revert this commit
though.
2022-02-01 17:51:33 +01:00
Rasmus Wriedt Larsen
54f53c828e Python: Refactor argumentRoutingTest.ql to be more generic
I checked to see that the tests still works. If I deleted the `arg5`
annotation, it got failures:

```diff
diff --git a/python/ql/test/experimental/dataflow/coverage/argumentPassing.py b/python/ql/test/experimental/dataflow/coverage/argumentPassing.py
index e218bdde9b..71816c1e01 100644
--- a/python/ql/test/experimental/dataflow/coverage/argumentPassing.py
+++ b/python/ql/test/experimental/dataflow/coverage/argumentPassing.py
@@ -46,7 +46,7 @@ def argument_passing(
     c,
     d=arg4,  #$ arg4 func=argument_passing
     *,
-    e=arg5,  #$ arg5 func=argument_passing
+    e=arg5,
     f,
     **g,
 ):
diff --git a/python/ql/test/experimental/dataflow/coverage/argumentRoutingTest.expected b/python/ql/test/experimental/dataflow/coverage/argumentRoutingTest.expected
index e69de29bb2..22037a40c3 100644
--- a/python/ql/test/experimental/dataflow/coverage/argumentRoutingTest.expected
+++ b/python/ql/test/experimental/dataflow/coverage/argumentRoutingTest.expected
@@ -0,0 +1,2 @@
+| argumentPassing.py:49:7:49:10 | ControlFlowNode for arg5 | Unexpected result: arg5= |
+| argumentPassing.py:49:7:49:10 | ControlFlowNode for arg5 | Unexpected result: func=argument_passing |
```
2022-02-01 17:50:06 +01:00
Rasmus Wriedt Larsen
76f3d74fed Python: Remove extra whitespace from argumentPassing.py 2022-02-01 17:48:16 +01:00
Rasmus Wriedt Larsen
5ee755db09 Python: Require MISSING: flow annotations for normal data-flow tests
I had to rewrite the SINK1-SINK7 definitions, since this new requirement
complained that we had to add this `MISSING: flow` annotation :D

Doing this implementation also revealed that there was a bug, since I
did not compare files when checking for these `MISSING:` annotations. So
fixed that up in the implementation for inline taint tests as well.

(extra whitespace in argumentPassing.py to avoid changing line numbers
for other tests)
2022-02-01 17:46:53 +01:00
Rasmus Wriedt Larsen
2bc4a60496 Python: Unify normal dataflow test setup
I went with NormalDataflowTest to signify that if you don't know what
you're looking for, this is probably the one. I did not want to just
call it DataflowTest, since that becomes a big vague when there are also
`FlowTest.qll` and `MaximalFlowTest.qll` -- I'm open to renaming this
though 👍
2022-02-01 17:31:31 +01:00
Rasmus Wriedt Larsen
41319607a9 Python: Use InlineExpectationsTest for field-flow tests
I deleted the old tests, so it's very clear what tests to look for
2022-02-01 17:31:31 +01:00
Rasmus Wriedt Larsen
d6f415bae2 Python: Run match tests if Python 3.10 or newer
Also fixes a bug in the tests
2022-02-01 17:31:31 +01:00
Rasmus Wriedt Larsen
a4bb0cc5d8 Python: Run tests for fieldflow/test.py 2022-02-01 15:32:07 +01:00
Rasmus Wriedt Larsen
1390f034f3 Python: Delete duplicated tests
All the same tests are present in `fieldflow/test.py`
2022-02-01 15:31:30 +01:00
Rasmus Wriedt Larsen
1394b38032 Python: Improve customSanitizer tests
Before we didn't show how we treated the value _after_ the check. But we
do actually handle this nicely 💪
2022-02-01 15:09:29 +01:00
Rasmus Lerchedahl Petersen
c2cd58edc4 python: rewrite to separate configurations
source nodes get duplicated, so perhaps flow states
are actually better for performance?
2022-02-01 14:36:11 +01:00
Rasmus Wriedt Larsen
f7a0b17ed6 Merge pull request #7687 from yoff/python/PathInjection-FlowState
python: Rewrite path injection query to use flow state
2022-02-01 11:33:37 +01:00
jorgectf
080775c873 Merge branch 'jorgectf/python/deserialization' of https://github.com/jorgectf/codeql into jorgectf/python/deserialization 2022-01-31 17:48:47 +01:00
Jorge
a1f8acc9bb Merge branch 'github:main' into jorgectf/python/deserialization 2022-01-31 17:48:35 +01:00
Rasmus Lerchedahl Petersen
20d54543fd python: move log injection out of experimental
- move from custom concept `LogOutput` to standard concept `Logging`
- remove `Log.qll` from experimental frameworks
  - fold models into standard models (naively for now)
    - stdlib:
      - make Logger module public
      - broaden definition of instance
      - add `extra` keyword as possible source
   - flak: add app.logger as logger instance
   - django: `add django.utils.log.request_logger` as logger instance
     (should we add the rest?)
- remove LogOutput from experimental concepts
2022-01-31 11:27:55 +01:00
Rasmus Lerchedahl Petersen
0c3bce1415 python: deprecation
I am slightly concerned that the test now generates many more
intermediate results. I suppose that maes the analysis heavy.
Should the new library get a new name instead, so the old code
does not get evaluated?
2022-01-31 08:32:24 +01:00
Rasmus Wriedt Larsen
f962d8e72c Python: Move test to correct location 2022-01-28 11:33:21 +01:00
Rasmus Lerchedahl Petersen
ab43f041c3 python: rename files 2022-01-28 11:00:17 +01:00
Rasmus Lerchedahl Petersen
4c3c4deb34 python: Move over query and tests 2022-01-28 09:19:11 +01:00
Rasmus Wriedt Larsen
4338c06b0d Python: Support Django FileField.upload_to 2022-01-27 17:20:16 +01:00
Rasmus Lerchedahl Petersen
b93c04bb79 python: Add reverse flow in some patterns
Particularly in value and literal patterns.
This is getting a little bit into the guards aspect of matching.
We could similarly add reverse flow in terms of
sub-patterns storing to a sequence pattern,
a flow step from alternatives to an-or-pattern, etc..
It does not seem too likely that sources are embedded in patterns
to begin with, but for secrets perhaps?

It is illustrated by the literal test. The value test still fails.
I believe we miss flow in general from the static attribute.
2022-01-27 15:20:23 +01:00
Rasmus Lerchedahl Petersen
47af3a69a5 Merge branch 'main' of github.com:github/codeql into python/support-match 2022-01-26 11:39:46 +01:00
Harry Maclean
517f2d0823 Add optional results to InlineExpectationsTest
The idea behind optional results is that there may be instances where
each line of source code has many results and you don't want to annotate
all of them, but you still want to ensure that any annotations you do
have are correct.

This change makes that possible by exposing a new predicate
`hasOptionalResult`, which has the same signature as `hasResult`.

Results produced by `hasOptionalResult` will be matched against any
annotations, but the lack of a matching annotation will not cause a
failure.

We will use this in the inline tests for the API edge getASubclass,
because for each API path that uses getASubclass there is always a
shorter path that does not use it, and thus we can't use the normal
shortest-path matching approach that works for other API Graph tests.
2022-01-25 16:41:49 +13:00
Rasmus Wriedt Larsen
301318020f Merge pull request #7455 from haby0/py/add-shutil-module-path-injection-sinks
Python: Add shutil module sinks for path injection query
2022-01-24 20:06:36 +01:00
Rasmus Lerchedahl Petersen
9aa4c4a6a7 python: Add missing input
also update test expectation
2022-01-21 13:55:33 +01:00
Erik Krogh Kristensen
ddfc3bc00f use set literals instead of big disjunctions 2022-01-21 11:46:33 +01:00
Rasmus Lerchedahl Petersen
a5bc5373d0 python: Rewrite path injection to use flow state
This removes the FP cause by chaining
This PR also removes `ChainedConfigs12.qll`,
as we hope to solve future problems via flow states.
2022-01-21 09:26:48 +01:00