Commit Graph

9543 Commits

Author SHA1 Message Date
Rasmus Wriedt Larsen
9f4088413a Python: Modernise py/local-shadows-builtin
+ moved `scope instanceof Function` so it makes more sense :)
2019-12-20 15:05:49 +01:00
Tom Hvitved
29cd6a9e30 Sync XML.qll 2019-12-19 10:29:30 +01:00
Taus
52d231c219 Merge pull request #2469 from RasmusWL/python-modernise-twisted-library
Python: modernise twisted library
2019-12-18 13:55:50 +01:00
Taus
eb6feeeaf8 Merge pull request #2482 from RasmusWL/python-include-zope-web-tests
Python: include zope web tests from internal repo
2019-12-18 13:55:23 +01:00
Rasmus Wriedt Larsen
48f873e3d9 Python: Add getAReturnedNode to PythonFunctionValue 2019-12-18 12:00:43 +01:00
Rasmus Wriedt Larsen
582ef6cec9 Python: Restructure logic in Twisted.qll 2019-12-18 10:42:39 +01:00
Rasmus Wriedt Larsen
9942c3fd8b Python: Autoformat twisted library 2019-12-18 10:42:39 +01:00
Rasmus Wriedt Larsen
ac55e6aba6 Python: Modernise twisted library 2019-12-18 10:42:39 +01:00
Rasmus Wriedt Larsen
4e3c183676 Python: Adapt twisted tests so they pass 2019-12-18 10:42:39 +01:00
Rasmus Wriedt Larsen
6011cb74f8 Python: Add twisted tests from internal repo 2019-12-18 10:42:39 +01:00
Rasmus Wriedt Larsen
8b5d6ae2cf Python: Modernise zope web tests 2019-12-17 17:42:03 +01:00
Rasmus Wriedt Larsen
e257ba40c4 Python: Make zope web tests pass 2019-12-17 17:42:03 +01:00
Taus Brock-Nannestad
1d94f6d303 Python: Fix several bad join orders.
Performance on `taers232c/GAMADV-X` (which exhibited pathological behaviour in
the most recent dist upgrade) went from ~670s to ~313s on
`py/hardcoded-credentials`.

There are still a few tuple counts in the 10-100 million range, but this commit
takes care of all of the ones that numbered in the billions. (A single tuple
count in the 100-1000 million range remains, but it appears to be less critical,
taking only two seconds to calculate.)
2019-12-17 17:19:49 +01:00
Max Schaefer
a17b615ae5 C++/Python: Deprecate XMLFile.getPath and XMLFile.getFolder.
Both can be expressed using predicates inherited from `File`.
2019-12-17 10:15:43 +00:00
Max Schaefer
47c1fc7358 C++/Python: Fix XMLFile.getPath and XMLFile.getFolder.
Previously, the former returned the file's stem (that is, basename without extension), and the latter never held.
2019-12-17 10:15:43 +00:00
Max Schaefer
bf30f9cdd2 Python: Remove use of deprecated predicate. 2019-12-17 10:15:43 +00:00
Max Schaefer
ef453db225 Python: Adjust XMLParent.getName to match other languages. 2019-12-17 10:15:43 +00:00
Max Schaefer
3068a89ab2 Python: Adjust implementation of allCharactersString to match other languages. 2019-12-17 10:15:43 +00:00
Max Schaefer
923e36ba4f C++/Java/JavaScript/Python: Make qldoc consistent. 2019-12-17 10:15:43 +00:00
Max Schaefer
a2fe678464 C++/Java/JavaScript/Python: Unify imports in XML.qll. 2019-12-17 10:15:43 +00:00
Max Schaefer
49abffa477 Python: Autoformat XML.qll. 2019-12-17 10:15:43 +00:00
Rasmus Wriedt Larsen
6a73479ea3 Python: Modernise py/loop-variable-capture 2019-12-16 13:36:59 +01:00
Rasmus Wriedt Larsen
9004a8e71f Python: Fix characteristic predicate of XMLParent
Like in https://github.com/Semmle/ql/pull/2520
2019-12-16 13:32:28 +01:00
Rasmus Wriedt Larsen
e3b502085b Merge pull request #2515 from tausbn/python-fix-bad-join-order-in-statement-no-effect
Python: Fix bad join order in `py/ineffectual-statement`.
2019-12-12 13:41:18 +00:00
Taus Brock-Nannestad
d5cc42e34c Python: Fix bad join order in py/ineffectual-statement.
This used to take 30s on `cpython`.
```
Tuple counts for StatementNoEffect::side_effecting_binary#f:
46522     ~0%     {2} r1 = ClassObject::ClassObject::hasAttribute_dispred#fb AS L AND NOT StatementNoEffect::side_effecting_binary#f#antijoin_rhs AS R(L.<0>, L.<1>)
46522     ~2%     {2} r2 = SCAN r1 OUTPUT r1.<1>, r1.<0>
950960    ~2%     {2} r3 = JOIN r2 WITH Operations::Operator::getSpecialMethodName_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r2.<1>
950960    ~2%     {2} r4 = JOIN r3 WITH py_operators AS R ON FIRST 1 OUTPUT R.<2>, r3.<1>
950960    ~0%     {3} r5 = JOIN r4 WITH AstGenerated::BinaryExpr_::getLeft_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r4.<1>, r4.<0>
122934382 ~0%     {2} r6 = JOIN r2 WITH Operations::Cmpop::getSpecialMethodName_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r2.<1>
122934382 ~3%     {3} r7 = JOIN r6 WITH project#Operations::Compare::compares_dispred#ffff#3_201#join_rhs AS R ON FIRST 1 OUTPUT R.<2>, r6.<1>, R.<1>
123885342 ~3%     {3} r8 = r5 \/ r7
300       ~8%     {1} r9 = JOIN r8 WITH project#Exprs::Expr::refersTo_dispred#ffff AS R ON FIRST 2 OUTPUT r8.<2>
                  return r9
```
With this commit, it takes a few milliseconds.
2019-12-11 17:58:30 +01:00
Henning Makholm
073563a19b Python tests: explicitly specify --lang2 for python2 tests
This allows them to work with the `LegacyQltLanguage.PYTHON3` extraction recipe.
2019-12-07 02:38:02 +01:00
Rasmus Wriedt Larsen
387ab52855 Python: Add zope web tests from internal repo 2019-12-02 14:38:03 +01:00
Jonas Jensen
5b24b1efc3 Merge remote-tracking branch 'upstream/rc/1.23' into mergeback-20191202
Conflicts solved:
	javascript/extractor/src/com/semmle/js/extractor/Main.java
	javascript/ql/test/query-tests/Statements/UseOfReturnlessFunction/tst.js
2019-12-02 09:57:34 +01:00
Taus
20513561a0 Merge pull request #2459 from RasmusWL/python-modernise-TurboGears-library
Python: modernise TurboGears library
2019-11-28 14:36:01 +01:00
semmle-qlci
2b0eef3b14 Merge pull request #2448 from tausbn/python-use-import-python-consistently
Approved by RasmusWL
2019-11-28 12:47:00 +00:00
Rasmus Wriedt Larsen
44cc9dd0be Python: Add TurboGears templating example 2019-11-27 15:07:32 +01:00
Rasmus Wriedt Larsen
b526421072 Python: Autoformat TurboGears library 2019-11-27 14:19:51 +01:00
Rasmus Wriedt Larsen
9ef270fc92 Python: Modernise TurboGears library 2019-11-27 14:19:04 +01:00
Rasmus Wriedt Larsen
3e5e14a14b Merge pull request #2431 from tausbn/python-cyclic-import-future-annotations
Python: Account for non-evaluation of annotations in cyclic imports.
2019-11-27 13:31:53 +01:00
Taus Brock-Nannestad
118275e948 Python: Use import python consistently in all queries. 2019-11-26 17:37:09 +01:00
Taus
e585f1ba85 Merge pull request #1332 from lcartey/python/change-kind
Python: Update filter queries to have appropriate kind
2019-11-26 17:15:38 +01:00
Taus Brock-Nannestad
036e0f75c8 Python: Account for non-evaluation of annotations in cyclic imports.
Should fix #2426.

Essentially, we disregard expressions used inside annotations, if these
annotations occur in a file that has `from __future__ import annotations`, as
this prevents the annotations from being evaluated.
2019-11-25 15:32:52 +01:00
Rasmus Wriedt Larsen
0f91139055 Merge pull request #2419 from tausbn/python-fix-use-of-input-fp
Python: Fix false positive for `py/use-of-input`.
2019-11-25 12:08:39 +01:00
Rebecca Valentine
a8204385c3 Adds fix for __init_subclass__ bug. (#2390)
* Adds fix for __init_subclass__ bug.

* Adds test case.

* Move test on name.

I think it makes more sense here, alongside the other "special" method names.
2019-11-24 12:18:17 +01:00
Taus Brock-Nannestad
67647bda66 Python: Fix false positive for py/use-of-input.
Fixes #1969.

The points-to analysis does not know that the assignment `input = raw_input`
cannot fail under Python 2, and so there are two possible values that `input`
could point-to after exiting the exception handler: the built-in `input`, or the
built-in `raw_input`. In the latter case we do not want to report the alert, and
so adding a check that the given function does not point-to the built-in
`raw_input` suffices.
2019-11-22 16:46:20 +01:00
Rasmus Wriedt Larsen
46b6e6d722 Merge pull request #2409 from tausbn/python-typing-forward-reference-fp
Python: Support forward references inside return type annotations.
2019-11-22 11:18:04 +01:00
Rasmus Wriedt Larsen
536c211a73 Merge pull request #2401 from tausbn/python-fix-non-iterable-class-confusion-fp
Python: Fix false positive in `py/non-iterator-in-for-loop`
2019-11-22 11:15:16 +01:00
Taus Brock-Nannestad
033524ce63 Python: Support forward references inside return type annotations.
Should fix #2407.

Also allows for the string containing the forward reference to appear inside a
subexpression of the type annotation.
2019-11-21 15:37:32 +01:00
Taus Brock-Nannestad
9fda4ab480 Python: Fix false positive in py/non-iterator-in-for-loop
Should fix #1833, #2137, and #2187.

Internally, comprehensions are (at present) elaborated into local functions and
iterators as described in [PEP-289](https://www.python.org/dev/peps/pep-0289/).
That is, something like:

```
g = (x**2 for x in range(10))
```

becomes something akin to

```
def __gen(exp):
    for x in exp:
        yield x**2
g = __gen(iter(range(10)))
```

In the context of the top-level of a class, this means `__gen` looks as if it is
a method of the class, and in particular `exp` looks like it's the `self`
argument of this method, which leads the points-to analysis to think that `exp`
is an instance of the surrounding class itself.

The fix in this case is pretty simple: we look for occurrences of `exp` (in fact
called `.0` internally -- carefully chosen to _not_ be a valid Python
identifier) and explicitly exclude this parameter from being classified as a
`self` parameter.
2019-11-21 11:49:29 +01:00
Taus Brock-Nannestad
c525ab325f Python: Add toString information for all classes without such.
Having `toString()` defined to be `none()` is a major headache when debugging,
as `toString`-less results are silently elided. This PR puts dummy `toString`s
in place of the `none()`s.

(I am mostly creating this to see if it impacts our tests and/or the
performance. If not, we may as well merge it.)
2019-11-20 14:47:20 +01:00
Rasmus Wriedt Larsen
b39bcde31c Merge pull request #2375 from tausbn/python-fix-mutable-value-type-coercion-fp
Python: Don't report mutable parameters that are in fact immutable.
2019-11-19 13:26:23 +01:00
Rasmus Wriedt Larsen
231414ceaf Merge pull request #2374 from tausbn/python-fix-mappingproxytype-fp
Python: Fix non-container FP relating to `MappingProxyType`.
2019-11-19 13:13:26 +01:00
Taus
4c700882b6 Merge pull request #2190 from RasmusWL/python-modernise-tornado-library
Python: modernise tornado library
2019-11-19 09:36:30 +01:00
Taus Brock-Nannestad
3491d90b1e Python: Apply auto-format. 2019-11-18 16:50:32 +01:00
Taus Brock-Nannestad
1385f3c018 Python: Fix non-container FP relating to MappingProxyType.
Fixes #2307.

Also modernises the query to use the `Value` API.
2019-11-18 16:50:32 +01:00