Commit Graph

3069 Commits

Author SHA1 Message Date
Rasmus Lerchedahl Petersen
4fe2576b9a Python: start modernizing routing tests 2020-11-12 22:43:34 +01:00
Rasmus Lerchedahl Petersen
823ed447df Python: Add new-style tests
should perhaps move `LocalFlowStepTest` and `MaximalFlowStep` into where
 they are referenced (they did not seem too reusable after all).
Should also add argument tests in the same way.
2020-11-12 21:57:29 +01:00
yoff
86fc9e62ef Merge pull request #4650 from RasmusWL/python-set-literal-formatting
Python: Update set literal formatting
2020-11-11 15:35:12 +01:00
Rasmus Wriedt Larsen
611398586d Merge pull request #4649 from yoff/python-dataflow-cfgparameters
Python: Make `ParameterNode` a `CfgNode`
2020-11-11 10:22:12 +01:00
Rasmus Wriedt Larsen
9ed15732ed Python: Update set literal formatting
Now that auto-formatting rules have been updated
2020-11-11 09:38:25 +01:00
Rasmus Lerchedahl Petersen
0710963fc3 Python: update test expectations
EssaNode -> ControlFlowNode
2020-11-10 23:58:55 +01:00
Jonas Jensen
fc764db8e1 Merge pull request #4643 from nickrolfe/getFileBySourceArchiveName
Replace getEncodedFile with shared getFileBySourceArchiveName predicate
2020-11-10 17:36:29 +01:00
Nick Rolfe
ac4a1f1d9b Update comment to be a QLDoc comment 2020-11-10 14:14:27 +00:00
Nick Rolfe
1e1eb7ee33 Replace getEncodedFile with shared getFileBySourceArchiveName predicate
While also making it work with paths for databases created on Windows.
2020-11-10 13:55:27 +00:00
Anders Schack-Mulligen
89ef6ea4eb C++/C#/Java/JavaScript/Python: Autoformat set literals. 2020-11-10 13:32:27 +01:00
Rasmus Lerchedahl Petersen
109d55eb25 Python: Make ParameterNode a CfgNode
Add a step from that `CfgNode` to the corresponding `EssaNode`.
The intended effect is seen in `ImpliesDataflow.expected`.
The efeect seen in other `.expected`-files is that parameter nodes
change type, that the extra steps are seen, and that flow from
`EssaVar`s is mirrored in flow from `CfgNode`s.
There is one surprise, which is the `.0` node in
`coverage/localFlow.expected`.
2020-11-10 11:35:50 +01:00
yoff
26286e534e Merge pull request #4174 from yoff/SharedDataflow_PointsToImpliesDataflow
Python: Dataflow, Test that `pointsTo` implies data flow
merging now, will fix `self` in a later PR
2020-11-10 10:25:29 +01:00
Rasmus Wriedt Larsen
fbe51c51bb Python: Add missing QLDoc 2020-11-09 09:05:08 +01:00
Rasmus Wriedt Larsen
ed0e4f8425 Python: reasoning about => detecting
Co-authored-by: yoff <lerchedahl@gmail.com>
2020-11-09 09:01:04 +01:00
Taus
a9149b7e47 Python: Update python/ql/src/semmle/python/dataflow/new/internal/DataFlowPrivate.qll
Co-authored-by: yoff <lerchedahl@gmail.com>
2020-11-06 17:15:58 +01:00
Taus Brock-Nannestad
5a9cc0861c Merge branch 'main' into python-add-source-nodes 2020-11-06 17:12:41 +01:00
yoff
45317bcec9 Update python/ql/test/library-tests/PointsTo/new/code/w_function_values.py
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-11-06 15:03:20 +01:00
Rasmus Wriedt Larsen
9ebe59d393 Python: Move UnsafeDeserialization configuration to own file 2020-11-06 14:27:37 +01:00
Rasmus Wriedt Larsen
d38c48d2c8 Python: Move ReflectedXSS configuration to own file 2020-11-06 14:24:31 +01:00
Rasmus Wriedt Larsen
1897a0d59a Python: Move PathInjection configuration to own file
This one required a bit more thought, but ended up pretty nicely. Had to write
some QLDoc, but I think it turned out OK.
2020-11-06 14:21:23 +01:00
Rasmus Wriedt Larsen
0c6bd8401a Python: Move SqlInjection configuration to own file 2020-11-06 14:09:46 +01:00
Rasmus Wriedt Larsen
6299b73a46 Python: Move CommandInjection configuration to own file 2020-11-06 14:07:06 +01:00
Rasmus Wriedt Larsen
7c04c59456 Python: Move CodeInjection configuration to own file
This makes it easy to extend the sources/sinks of the configuration and re-run
the query from the query console on LGTM.com.

File location in `semmle.<lang>.security.dataflow.<QueryName>.qll` is matching
what we currently do in other languages (JS and C# sampled).

I did not follow the pattern in other languages for wrapping all the code in a
`module CodeInjection`, since I didn't understand the value in doing so -- I
would like confirmation from the other teams if we _should_ actually do that,
before merging.
2020-11-06 13:58:06 +01:00
Rasmus Lerchedahl Petersen
fe186bf854 Python: Add test 2020-11-06 13:30:11 +01:00
Rasmus Lerchedahl Petersen
64b9e9150e Python: only show results in extracted files 2020-11-06 12:01:16 +01:00
Taus Brock-Nannestad
7c58b28e36 Python: Write DataFlow::update more succinctly
This has no impact on performance, but it cleans up the code a bit,
and (hopefully) makes it more readable.
2020-11-05 16:47:41 +01:00
Taus Brock-Nannestad
bae4acabb1 Python: Fix bad join in StrConst::isUnicode
Also fixes a bug ("`B`" was not recognised as a bytestring prefix).

The basic idea behind this fix is that the set of possible prefixes is
fairly small, so it's easier just to precompute them, and then join
them with the entire prefix of the string in question (rather than
look at each string in isolation, get its prefix, and _then_ check
whether it looks like it's a unicode string prefix, which essentially
is what the code did before).
2020-11-05 16:45:27 +01:00
Taus Brock-Nannestad
1251bc57f5 Python: Fix bad join in TObject::literal_instantiation
Here, `context.appliesTo(n)` was being distributed across all of the
disjuncts, which caused poor performance.

The new helper predicate, `literal_node_class` should be fairly small,
since it only applies to a subset of `ControlFlowNode`s, and only
assigns a limited set of `ClassObjectInternal`s to these nodes.
2020-11-05 16:40:29 +01:00
Taus Brock-Nannestad
35a63e2411 Python: Fix bad join in regex::used_as_regex
Since the number of relevant attributes in the `re` module is fairly
small, it made sense to factor this out in a separate predicate, and
the join order also became more sensible.
2020-11-05 16:33:59 +01:00
Taus Brock-Nannestad
035e747ad5 Python: Fix slow use of regexCapture in Builtin::strValue
This is only _really_ expensive when there are a _lot_ of strings in
the database, but for this case, where we're always extracting the
same substring of the string, it's easier -- and faster -- to just
make a substring operation directly.
2020-11-05 16:33:33 +01:00
Taus Brock-Nannestad
83ba8c9bf5 Python: Add LocalSourceNode and flowsTo
This fixes the major performance problem with type tracking on
some (pathological) databases.

The interface could probably be improved a bit. In particular, I'm
thinking that we might want to have `DataFlow::exprNode` return a
`LocalSourceNode` so that a cast isn't necessary in order to use
`flowsTo`.

I have added two `cached` annotations. The one on `flowsTo` is
crucial, as performance regresses without it. The one on
`simpleLocalFlowStep` may not be needed, but Java has a similar
annotation, and to me it makes sense to have this relation cached.
2020-11-05 16:26:03 +01:00
Rasmus Lerchedahl Petersen
6cecd3ba83 Python: Move and rename query 2020-11-05 11:49:39 +01:00
yoff
79fcf598f3 Merge pull request #4608 from RasmusWL/patch-1
Python: Remove unnecessary cached annotation from adjacentRefUse
2020-11-04 16:08:30 +01:00
Rasmus Lerchedahl Petersen
38b2bb2828 Python: Add testfile with regressions 2020-11-04 15:55:59 +01:00
Rasmus Lerchedahl Petersen
6df3b8d524 Python: Update query and expectation 2020-11-04 15:17:38 +01:00
Rasmus Wriedt Larsen
31247739d7 Python: Remove unnecessary cached annotation from adjacentRefUse
As discussed in https://github.com/github/codeql/pull/4544#pullrequestreview-516575676
2020-11-04 15:16:08 +01:00
Rasmus Lerchedahl Petersen
9baa7b73da Merge branch 'main' of github.com:github/codeql into SharedDataflow_PointsToImpliesDataflow 2020-11-04 15:05:59 +01:00
yoff
62cb4ec974 Merge pull request #4605 from RasmusWL/python-fix-django-response-modeling
Python: fix django response modeling
2020-11-04 15:00:52 +01:00
Rasmus Wriedt Larsen
5cf8285717 Python: Fix default mimetype for django FileResponse 2020-11-04 12:28:51 +01:00
Rasmus Wriedt Larsen
826aedeb85 Python: Remove resolved TODO 2020-11-04 12:17:31 +01:00
Rasmus Wriedt Larsen
353505ec6c Python: Handle content of Django redirects correctly 2020-11-04 12:10:58 +01:00
Taus
180373c41d Merge pull request #4597 from yoff/python-fix-ql-doc
Python: Fix ql doc
2020-11-04 11:37:32 +01:00
Rasmus Wriedt Larsen
92dc7dc2f3 Python: Use mimetype instead of content-type in django modeling
This enables the XSS query to actually find results from django responses.
2020-11-04 11:34:20 +01:00
Anders Schack-Mulligen
92494441a7 Merge pull request #4554 from aschackmull/dataflow/reverse-partial
Dataflow: Add support reverse partial flow exploration.
2020-11-03 15:34:30 +01:00
Rasmus Lerchedahl Petersen
1023b239e4 Python: Simplify doc 2020-11-03 12:10:00 +01:00
yoff
d6a33a1253 Apply suggestions from code review
Co-authored-by: Taus <tausbn@github.com>
Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>
2020-11-03 12:04:43 +01:00
yoff
104ff5d217 Merge pull request #4596 from RasmusWL/python-import-customizations
Python: Import Customizations into python
2020-11-03 11:49:51 +01:00
Rasmus Lerchedahl Petersen
b71ea40dbd Python: QL doc for Werkzeug 2020-11-03 11:44:48 +01:00
Rasmus Lerchedahl Petersen
1773cc3a38 Python: QL doc for MySQLdb 2020-11-03 11:39:28 +01:00
Rasmus Lerchedahl Petersen
01783acca6 Python: QL doc for RemoteFlowSources 2020-11-03 11:37:34 +01:00