Technically not always true, but my assumption is that +90% of the time
that's what it will be used for, so while we could be more precise by
adding a taint-step from the `input` part of the construction, I'm not
sure it's worth it in this case.
Furthermore, doing so would break with the current way we model
threat-model sources, and how sources are generally modeled in JS... so
for a very pretty setup it would require changing all the other `file`
threat-model sources to start at the constructors such as
`fs.createReadStream()` and have taint-propagation steps towards the
actual use (like we do in Python)...
I couldn't see an easy path forwards for doing this while keeping the
Concepts integration, so I opted for the simpler solution here.
You could argue that proper modeling be done in the same way as
`NodeJSFileSystemAccessRead` is done for the callback based `fs` API (in
NodeJSLib.qll). However, that work is straying from the core goals I'm
working towards right now, so I'll argue that "perfect is the enemy of
good", and leave this as is for now.
Does a bunch of things, unfortunately all in the same place, so my
apologies in advance for a slightly complicated commit.
As for the changes themselves, this commit
- Adds timers for the old and new parsers. This means we get the overall
time spent on these parts of the extractor if the extractor is run with
`DEBUG` output shown.
- Adds logging information (at the `DEBUG` level) to show which
invocations of the parsers happen when, and whether they succeed or not.
- Adds support for using an environment variable named
`CODEQL_PYTHON_DISABLE_OLD_PARSER` to disable using the old parser
entirely. This makes it easier to test the new parser in isolation.
- Fixes a bug where we did not check whether a parse with the new parser
had already succeeded, and so would do a superfluous second parse.
If:
* the text for a file_id is not found (likely non-utf data in file)
* path does not appear in Vfs, in which case we fall back on loading the file from disk with no "semantics" available
