Commit Graph

3049 Commits

Author SHA1 Message Date
Taus
f93b68d4dc Python: Get rid of _attr methods 2021-04-13 12:25:38 +00:00
Taus
98d936d8b3 Python: Tornado cleanup using API graphs
I wasn't able to roll out API graphs as widely in Tornado as I had
hoped, since we're lacking the "def" part. This means most of the
`InstanceSource` machinery will have to stay.
2021-04-13 12:25:38 +00:00
Taus
fda750ef26 Merge pull request #5642 from tausbn/python-use-api-graphs-in-stdlib
Python: Use API graphs in `Stdlib.qll`
2021-04-12 18:05:38 +02:00
Taus
6d4ddc0329 Merge pull request #5614 from tausbn/python-allow-absolute-imports-from-source-directory
Python: Allow absolute imports from source directory
2021-04-12 18:02:00 +02:00
CodeQL CI
bc56d16c18 Merge pull request #5485 from RasmusWL/django-queryset-chains
Approved by tausbn
2021-04-12 08:49:31 -07:00
Tom Hvitved
7d2a60e910 Merge pull request #5640 from hvitved/dataflow/path-step-perf
Data flow: Prevent bad join-order in `pathStep`
2021-04-12 14:40:46 +02:00
Rasmus Wriedt Larsen
364d48948f Merge pull request #3810 from dilanbhalla/syntaxpython
Python: Function/Class Naming Convention (Syntax)
2021-04-12 10:42:17 +02:00
Taus
10be2735ec Python: Get rid of _attr predicates
Also changes all `CfgNode`s representing calls to `CallCfgNode`s.
2021-04-10 12:12:18 +00:00
Taus
720fbaf301 Python: Fix test error.
Somehow, having to type "Node" all day long made me turn "json" into
"node"...

Also removes some bits that weren't needed after all.
2021-04-09 19:04:49 +00:00
Taus
cc4827600b Python: Use API graphs in Stdlib.qll
Eliminates _almost_ all of the bespoke type trackers found here. The
ones that remain do not fit easily inside the framework of API graphs
(at least, not yet), and I did not see any easy ways to clean them up.
They have, however, been rewritten to use `LocalSourceNode` internally,
which was the primary goal of this exercise.

I'm sure we could also clean up many of the inner modules given the more
lean presentation we have now, but this can wait for a different PR.
2021-04-09 17:11:47 +00:00
Tom Hvitved
f130616369 Data flow: Make getLocalCc private again 2021-04-09 16:22:58 +02:00
Taus
d2b874f217 Python: Use API graphs in PEP249 support
Because the replacement extension point now extends `API::Node`, I
modified the `toString` method of the latter to have an empty body.
The alternative would be to require everyone to provide a `toString`
predicate for their extensions, but seeing as these will usually be
pointing to already existing API graph nodes, this seems silly.

(This may be the reason why the equivalent method in the JS libs has
such an implementation.)
2021-04-09 14:19:00 +00:00
Tom Hvitved
6874b8d4b3 Data flow: Prevent bad join-order in pathStep 2021-04-09 14:24:47 +02:00
Taus
affdedd840 Python: Add missing builtins to API::builtin
We were missing out on `None`, `True`, and `False` as these do not
appear as actual attributes of the `builtins` module in Python 3
(because they are elevated to the status of keywords there)

The simple solution, then, is to just always include them directly.
2021-04-09 12:02:07 +00:00
Dilan
d73ba13b28 autoformat fix 2021-04-08 11:41:58 -07:00
Rasmus Wriedt Larsen
c738f387b1 Merge pull request #5624 from tausbn/python-make-callcfgnode-a-localsourcenode
Python: Improve `CallCfgNode` interface
2021-04-08 13:38:24 +02:00
Taus
cf5f760ecd Merge pull request #5582 from RasmusWL/all-tuple
Python: Add support for `__all__` assigned to tuple
2021-04-08 13:03:27 +02:00
Rasmus Wriedt Larsen
83477439a1 Python: Make django views/fields/forms class modeling extensible
This also requires that we make this part of the modeling public, which I guess
is step we want to take eventually anyway!

I'm not quite sure whether the modules `Django::Views` and `Django::Forms` are
actually helpful, or whether we should just have their modules available as
`Django::View`, `Django::Form`, and `Django::Field`...
2021-04-08 12:45:37 +02:00
Rasmus Wriedt Larsen
b7483a5394 Python: Add modeledSubclassRef for Django views/fields/forms 2021-04-08 12:45:36 +02:00
Rasmus Wriedt Larsen
322bdcb703 Python: Port Django view modeling to API graphs 2021-04-08 12:45:35 +02:00
Rasmus Wriedt Larsen
8ce5c46e05 Python: Minor refactor
modName/clsName _is_ shorter, but also looks way worse :D
2021-04-08 12:45:34 +02:00
Dilan
675de07c3e autoformat ql 2021-04-07 15:04:18 -07:00
Taus
903f364dab Python: Improve CallCfgNode interface
Call nodes are always local sources (specifically sources of the return
value of the call), and so inheriting from `LocalSourceNode` will have
no effect on results, but _should_ make it a bit more smooth to use the
API.
2021-04-07 13:31:12 +00:00
Taus
6c69c1aeeb Python: Minor cleanup 2021-04-07 10:47:21 +00:00
Taus
b44db460f6 Python: Only track modules that are imported 2021-04-06 19:55:43 +00:00
Taus Brock-Nannestad
8e11abca40 Revert "Merge pull request #5552 from RasmusWL/revert-import-change"
This reverts commit 49d1937dc4, reversing
changes made to d4877a9038.
2021-04-06 17:39:41 +02:00
Rasmus Wriedt Larsen
bc49bc7095 Python: Add variable with underscore to __all__ tests 2021-04-06 11:54:25 +02:00
Rasmus Wriedt Larsen
224d3790b5 Python: Highlight all_indirect.py is not super important
At least not in my mind
2021-04-06 11:50:04 +02:00
Rasmus Wriedt Larsen
b11703cc74 Python: all_dybamic2 => all_indirect 2021-04-06 11:49:55 +02:00
Rasmus Lerchedahl Petersen
c777f1d8d7 Merge branch 'main' of github.com:github/codeql into python-api-enhancements 2021-04-06 09:31:26 +02:00
yoff
a23d8deb10 Merge pull request #5483 from RasmusWL/minor-fixup-django
Python: Better text for getSourceType in Django
2021-04-06 08:30:58 +02:00
Rasmus Wriedt Larsen
95ac2c8edd Python: Add another dynamic __all__ test 2021-03-31 17:31:55 +02:00
Rasmus Wriedt Larsen
ab3edf37d7 Python: Handle __all__ assigned to a tuple
Examples where this is used in real code:

- 76c0b32f82/django/core/files/temp.py (L24)
- 76c0b32f82/django/contrib/gis/gdal/__init__.py (L44-L49)
2021-03-31 17:25:19 +02:00
Rasmus Wriedt Larsen
43306f4700 Python: Add tests for Module.declaredInAll 2021-03-31 17:24:17 +02:00
Rasmus Wriedt Larsen
51c27de049 Merge branch 'main' into revert-import-change 2021-03-30 21:51:53 +02:00
Calum Grant
c26d05b1d5 Merge pull request #5532 from RasmusWL/python-cleanup
Python: Delete filter queries, code duplication library, and precision tag from metric queries
2021-03-29 17:16:43 +01:00
Rasmus Wriedt Larsen
96a66fa4ee Python: Apply suggestions from code review 2021-03-29 17:02:56 +02:00
CodeQL CI
3613ceb07f Merge pull request #5535 from tausbn/python-prevent-bad-TCs
Approved by yoff
2021-03-29 12:03:08 +01:00
Rasmus Wriedt Larsen
92e0e195a4 Revert "Merge pull request #5506 from tausbn/python-allow-absolute-imports-from-source-directory"
This reverts commit 8d15680af4, reversing
changes made to 63831cc62b.

This PR caused performance problems, so reverting now to clear up immediate
problems.
2021-03-27 18:08:20 +01:00
Taus Brock-Nannestad
f17bbd9982 Python: Fix another bad TC.
This one is a bit awkward, since the previous version was supposed to
improve indexing. Unfortunately this is vastly outweighed by the slow
convergence of the TC. Right now we pay the cost of inverting the
`hasFlowSource` relation, but this is still cheaper.
2021-03-26 16:38:13 +01:00
yoff
208d5157fa Merge pull request #5500 from RasmusWL/django-forms
Python: Model RemoteFlowSources on Django forms/fields
2021-03-25 20:43:19 +01:00
Taus Brock-Nannestad
c2f112cb92 Python: Filter _before_ the cartesian product
It's always a sad thing to see a good plan go wrong:

86860032 ~0%      {4} r26 = JOIN r19 WITH DataFlowPublic::TupleElementContent#class#ff CARTESIAN PRODUCT OUTPUT Lhs.0 'nodeFrom', Lhs.1 'nodeTo', Rhs.0, Rhs.1
129256   ~3%      {4} r27 = SELECT r26 ON In.3 <= 7
129256   ~0%      {3} r28 = SCAN r27 OUTPUT In.0 'nodeFrom', In.2 'c', In.1 'nodeTo'

Happily, now it looks like this:

129256  ~0%      {3} r20 = JOIN r19 WITH DataFlowPrivate::small_tuple#f CARTESIAN PRODUCT OUTPUT Lhs.0 'nodeFrom', Rhs.0, Lhs.1 'nodeTo'
2021-03-25 19:06:05 +01:00
Taus Brock-Nannestad
8734df334b Python: Slight cleanup 2021-03-25 18:35:16 +01:00
Taus Brock-Nannestad
229250dc54 Python: Limit size of TupleElementContent
A more principled approach is possible here, but in the short term
this will prevent an explosion.

For reference, openstack/cinder has roughly 19000 `ForTarget`s and
tuples of size up to 5300, and we were calculating the cartesian
product of these.
2021-03-25 18:28:49 +01:00
yoff
716e0f1404 Merge pull request #5517 from tausbn/python-prevent-potentially-bad-join-order
Python: Prevent potentially bad join order
2021-03-25 18:14:47 +01:00
Taus Brock-Nannestad
dbef36cbbb Python: Prevent bad TC and add a bit of caching
Using `simpleLocalFlowStep+` with the first argument specialised to
`CfgNode` was causing the compiler to turn this into a very slowly
converging manual TC computation.

Instead, we use `simpleLocalFlowStep*` (which is fast) and then join
that with a single step from any `CfgNode`. This should amount to the
same thing.

I also noticed that the charpred for `LocalSourceNode` was getting
recomputed a lot, so this is now cached. (The recomputation was
especially bad since it relied on `simpleLocalFlowStep+`, but anyway
it's a good idea not to recompute this.)
2021-03-25 17:28:37 +01:00
Rasmus Wriedt Larsen
9abe02f419 Python: Fix query metadata for old queries that have been ported
I'm not sure even I want to keep these around much longer. They seem to be
causing more problem than they are doing good.
2021-03-25 16:01:56 +01:00
Rasmus Wriedt Larsen
bd4934380a Python: Remove code duplication library 2021-03-25 15:27:55 +01:00
Rasmus Wriedt Larsen
09fbf480db Python: Remove precision tag from metric queries 2021-03-25 15:06:47 +01:00
Rasmus Wriedt Larsen
e3b2e0a1de Python: Delete filter queries 2021-03-25 15:06:46 +01:00