Commit Graph

578 Commits

Author SHA1 Message Date
Asger F
a3c7b631f4 JS: Extract type source text with substring 2019-07-01 11:29:55 +01:00
Asger F
edd96b056e JS: Remove redundant source field 2019-07-01 11:29:55 +01:00
Asger F
9403834af5 JS: Include leading star in parsed source 2019-07-01 11:29:55 +01:00
Asger F
9b4bdaecce JS: Remove unneeded replace call 2019-07-01 11:29:55 +01:00
Max Schaefer
a417884173 JavaScript: Fix potential null-pointer exception in YAML extractor.
`ScalarEvent.getStyle()` is documented as returning `null` for plain
scalars, so we need to handle that specially (cf
https://github.com/Semmle/ql/blob/master/javascript/ql/src/semmle/javascript/YAML.qll#L100
for the corresponding code in the library, which expects plain style to
be encoded as zero).
2019-06-23 21:56:02 +02:00
Esben Sparre Andreasen
6885b5cf1f JS: fix yaml StringIndexOutOfBoundsException 2019-06-21 15:18:56 +02:00
Max Schaefer
38a38ab780 JavaScript: Make autobuilder fail if no JS/TS code was seen.
In particular, the autobuilder will no longer succeed for projects that
contain HTML or YAML files but no JS/TS code. Further down the line,
this prevents LGTM.com from classifying such projects as "JavaScript"
projects.
2019-05-28 14:43:59 +01:00
semmle-qlci
9fb61d5f26 Merge pull request #1371 from xiemaisi/js/index-xml
Approved by asger-semmle
2019-05-28 14:30:43 +01:00
semmle-qlci
ead59baa0e Merge pull request #1369 from xiemaisi/js/fix-autobuild-test
Approved by asger-semmle
2019-05-28 12:27:17 +01:00
Max Schaefer
7f8f126338 JavaScript: Add support for XML extraction. 2019-05-28 09:44:24 +01:00
Max Schaefer
4992970181 JavaScript: Fix an auto-build test. 2019-05-28 09:44:24 +01:00
Max Schaefer
9ec366cf88 JavaScript: Move support for optional catch to ES2019. 2019-05-10 08:27:25 +01:00
Max Schaefer
d93d68d7f5 JavaScript: Add parser support for Unicode newlines in string literals. 2019-05-10 08:16:20 +01:00
Max Schaefer
86a7fa3abb JavaScript: Make extractor default to ECMA 2019.
Also introduces an enum constant for ECMA 2020 (not used anywhere yet).
2019-05-10 08:09:46 +01:00
Asger F
e9fcb670ff JS: Provide source locations for JSDocTypeExpr 2019-04-26 16:56:04 +01:00
Aditya Sharad
4121e7245b TS extractor: Allow the Node.js runtime to be configured via environment variables.
`SEMMLE_TYPESCRIPT_NODE_RUNTIME` can be used to provide the path to the Node.js runtime executable.
If this is omitted, the extractor defaults to the current behaviour of looking for `node` on the PATH.

`SEMMLE_TYPESCRIPT_NODE_RUNTIME_EXTRA_ARGS` can be used to provide additional arguments to the
Node.js runtime. These are passed first, before the arguments supplied by the extractor.

These changes are designed to allow TypeScript extraction in controlled customer environments where
we cannot control the PATH, or must use custom Node.js executables with certain arguments set.
2019-04-23 15:04:14 -07:00
Asger F
b6ea121808 TS: Make full TS extraction the default in AutoBuild 2019-04-15 12:11:05 +01:00
semmle-qlci
ed5fd96603 Merge pull request #1227 from asger-semmle/typescript3.4
Approved by xiemaisi
2019-04-11 10:39:57 +01:00
Max Schaefer
f22cb186e3 JavaScript: Teach AutoBuilder to extract YAML files by default. 2019-04-10 18:47:06 -07:00
Asger F
ef7bde472b TS: Fix javadoc 2019-04-10 15:33:39 +01:00
Asger F
c1c7ebfc48 TS: Support const type assertions 2019-04-10 12:54:42 +01:00
Asger F
d5ae69d40a TS: Support readonly type expressions 2019-04-10 12:26:46 +01:00
Asger F
11f460c6a3 JS: Generalize KeyofTypeExpr to UnaryTypeExpr 2019-04-10 12:21:28 +01:00
Max Schaefer
8c460ae385 Merge remote-tracking branch 'upstream/master' into rc/1.20-merge-master
Conflict in `javascript/extractor/src/com/semmle/js/extractor/Main.java` resolved
in favour of `master`.
2019-03-21 14:46:29 +00:00
Max Schaefer
1835028b93 JavaScript: Show character code when reporting unexpected character. 2019-03-21 10:44:49 +00:00
Asger F
aaa8bfb874 TS: allow namespace imports as types 2019-03-20 10:09:18 +00:00
Max Schaefer
03a2204c30 JavaScript: Improve support for dynamic imports.
`yield import(...)` previously caused a syntax error, now it is parsed
correctly.

`parseYield` is the only place where the value of `startsExpr` matters,
so this change should not affect anything else.
2019-03-13 14:40:12 +00:00
semmle-qlci
13c6f55a2e Merge pull request #1085 from asger-semmle/extract-symbol
Approved by xiemaisi
2019-03-12 14:07:17 +00:00
Asger F
f2ec35c334 TS: remove unused import 2019-03-12 11:35:59 +00:00
Asger F
824a50d14e TS: fix extraction of symbols with unusual names 2019-03-12 11:24:16 +00:00
Asger F
957db8374c JS: Bump extractor version string 2019-03-11 16:53:52 +00:00
Asger F
146909ba13 TS: fix missing this type in method declarations 2019-03-11 16:53:25 +00:00
Asger F
d7f0099426 TS: bump extractor version 2019-03-11 11:26:46 +00:00
Max Schaefer
832dff54e8 JavaScript: Remove a @link in Javadoc.
Javadoc claims not to be able to resolve this link, while Eclipse manages to do so without any problems, failing an internal PR check.

It's only in a test, though, so I just removed it.
2019-03-05 17:02:44 +00:00
Max Schaefer
b1033b079f JavaScript: Make configuration of parallel extraction consistent with parallel evaluation.
Just like parallel evaluation, the number of extractor threads is now determined by the `LGTM_THREADS` environment variable, and defaults to one.
2019-03-05 10:06:32 +00:00
Max Schaefer
c4fa29dd0f JavaScript: Autoformat extractor sources using google-java-format.
No special settings; command:

  find javascript/extractor/src -name "*.java" | xargs java -jar /path/to/google-java-format-1.7-all-deps.jar --replace
2019-02-28 14:30:06 +00:00
Asger F
5478e0da62 Merge pull request #998 from xiemaisi/js/autobuild-file-types
JavaScript: Make file types customisable in AutoBuild.
2019-02-28 15:26:35 +01:00
Max Schaefer
9d77619afc JavaScript: Make file types customisable in AutoBuild.
Every once in a while we encounter projects using some custom file extension for files that we could in principle extract, but since the extractor doesn't know about the extension the files are skipped.

To handle this, the legacy extractor has a `--file-type` option that one can use to specify a file type to use for all files in that particular extraction. So far, `AutoBuild` has nothing of the sort.

This PR proposes to introduce an environment variable `LGTM_INDEX_FILETYPES` to allow a similar customisation. In the fullness of time, this variable would be set through `lgtm.yml` in the usual way, but for now it is undocumented and for internal use only.

Specifically, `LGTM_INDEX_FILETYPES` is a newline-separated list of ".extension:filetype" pairs, specifying that files with the given `.extension` should be extracted as type `filetype`, where
`filetype` is one of `js`, `html`, `json`, `typescript` or `yaml`.

For example, `.jsm:js` causes all `.jsm` files to be extracted as JavaScript.

This can also be used to override default file types: for example, by specifying `.js:typescript` all JavaScript files will be extracted as TypeScript.
2019-02-27 12:02:01 +00:00
Max Schaefer
2ed37903d8 JavaScript: Include list of relevant environment variables in Javadoc for AutoBuild. 2019-02-27 11:54:59 +00:00
Max Schaefer
6ecdb0edd5 JavaScript: Allow first expression in array literal to be an in expression. 2019-02-27 08:58:28 +00:00
Max Schaefer
d4dbe3bfb6 JavaScript: Back out parsing of qualified XML identifiers.
Their syntax conflicts with the proposed function-bind operator, which is more important to support.
2019-02-24 21:30:59 +00:00
Max Schaefer
7491b5ea53 JavaScript: Add a comment. 2019-02-24 21:02:12 +00:00
Max Schaefer
f726125b71 JavaScript: Restrict E4X processing instruction disambiguation to the <?xml ...?> case. 2019-02-24 20:56:43 +00:00
Max Schaefer
cc216ad250 JavaScript: Buffer recoverable syntax errors during speculative parsing.
Analogous to how we buffer tokens, we need to delay reporting these errors until we have committed to a parse.
2019-02-24 20:45:41 +00:00
Max Schaefer
c7e428eb27 JavaScript: Handle E4X/Flow lexical ambiguity. 2019-02-24 20:45:41 +00:00
Max Schaefer
d6deefed86 JavaScript: Accept CDATA in E4X content. 2019-02-24 20:45:41 +00:00
Max Schaefer
81b86d9a0f JavaScript: Skip XML processing instructions in E4X content. 2019-02-24 20:45:41 +00:00
Max Schaefer
5a89024507 JavaScript: Be more lenient about keywords used as identifiers. 2019-02-24 20:45:41 +00:00
Max Schaefer
dbbb961b48 JavaScript: Accept let expressions with an object literal as their body. 2019-02-24 20:45:41 +00:00
Max Schaefer
63ed569724 JavaScript: Recover from missing initializers in const/destructuring declarations. 2019-02-24 20:45:41 +00:00