Commit Graph

88 Commits

Author SHA1 Message Date
Nick Rolfe
143256e673 Fix filenames in source archives 2021-09-20 22:17:45 +01:00
Nick Rolfe
c30c7b380d Replace if let with match. 2021-09-20 12:22:55 +01:00
Nick Rolfe
0936c4cd7b Fix trap extension for source files without extensions
We were writing files with names like `Gemfile..trap.gz`. Now fixed to
`Gemfile.trap.gz`.
2021-09-20 12:11:00 +01:00
Tom Hvitved
1393dc9eb4 Update extractor/src/main.rs
Co-authored-by: Nick Rolfe <nickrolfe@github.com>
2021-09-20 12:50:24 +02:00
Tom Hvitved
58d06715fc Extract a special empty location 2021-09-20 09:52:26 +02:00
Nick Rolfe
cf72bada3d Fix 'unused borrow that must be used' warnings.
I don't remember seeing this warning before upgrading to Rust 1.55
2021-09-09 17:03:10 +01:00
Tom Hvitved
7f03b87142 Drop redundant columns from files and folders relations 2021-08-31 12:16:26 +02:00
Arthur Baars
c568162256 Use a single TrapWriter
The output of two distinct TrapWriters should not be written to the
same TRAP file because this causes name clashes between TRAP labels.
2021-07-29 12:50:27 +02:00
Arthur Baars
fcf2d4cbd2 Apply suggestions from code review
Co-authored-by: Nick Rolfe <nickrolfe@github.com>
2021-07-29 09:02:57 +02:00
Arthur Baars
38eb6c112f Add ERB extraction 2021-07-27 18:43:51 +02:00
Arthur Baars
2e10f8f054 Prefix dbscheme entries with language name 2021-07-27 18:17:19 +02:00
Nick Rolfe
ab72b4e9e7 Use hexadecimal encoding for TRAP labels 2021-06-17 16:16:32 +01:00
Nick Rolfe
ed93233917 Remove unnecessary spaces in TRAP output 2021-06-17 16:16:06 +01:00
Nick Rolfe
1388d82f1d Escape field names with table storage 2021-06-01 13:32:13 +01:00
Arthur Baars
73aae5dfd9 Use num_cpus-1 threads by default 2021-05-25 09:28:49 +02:00
Alex Ford
11949c6b77 Merge pull request #176 from github/diagnostics-entries
Start writing diagnostics to the DB, and some basic summary/diagnostics queries
2021-05-13 14:31:01 +01:00
Alex Ford
277a6a020a diagnostics: use debug rather than hidden terminology, and leave gaps for other severities 2021-05-13 13:44:10 +01:00
Alex Ford
11376bc411 note that severity 3 corresponds to an error diagnostic level 2021-05-12 16:39:51 +01:00
Alex Ford
9663b74e12 use severity level 3 to indicate an extraction error for a file 2021-05-11 19:23:05 +01:00
Alex Ford
8ab95324eb dedupe some error reporting code 2021-05-11 14:09:10 +01:00
Alex Ford
0f3168f293 record more parse errors 2021-05-10 21:23:24 +01:00
Alex Ford
269ae8331b record 'unknown table type' extraction errors 2021-05-07 17:56:50 +01:00
Alex Ford
272aec27f2 clean up the parse_error writing code 2021-05-07 00:15:09 +01:00
Alex Ford
3a1dff1c95 start writing diagnostics entries for parse errors 2021-05-06 23:09:43 +01:00
Nick Rolfe
c37f390efc Reserve more capacity for escaped key 2021-05-05 13:21:16 +01:00
Nick Rolfe
99ae17de03 Avoid copying key when it doesn't need escaping 2021-05-05 12:54:23 +01:00
Nick Rolfe
d2d5f31599 Escape keys for files and folders 2021-05-04 16:52:35 +01:00
Nick Rolfe
9b96bc32cc Add ast_node_parent relation 2021-03-08 18:25:37 +00:00
Arthur Baars
846173732b Add newline at the end of each code region in an ERB file 2021-02-05 09:49:25 +01:00
Arthur Baars
9d974bd56d Extract ERB files 2021-02-05 09:48:54 +01:00
Nick Rolfe
1d3f06aca1 Simplify propagation of errors 2021-01-07 11:11:15 +00:00
Nick Rolfe
92c78e2b2d Simplify num_codeql_threads function slightly 2021-01-07 11:10:43 +00:00
Nick Rolfe
bb2bdc01b5 Have the extract function create the TS parser object 2021-01-07 10:56:23 +00:00
Nick Rolfe
bf4eac5113 Parallelize extraction
Use the Rayon library to do parallel iteration over the file list. The
number of threads used respects the CODEQL_THREADS environment variable.
2021-01-06 18:22:27 +00:00
Nick Rolfe
282d20d766 Remove redundant field on ChildNode struct 2020-12-16 20:57:06 +00:00
Nick Rolfe
f5282edfc1 Simplifications based on PR feedback 2020-12-16 17:54:40 +00:00
Nick Rolfe
3f5eab04b5 Create disjoint db types for different operators 2020-12-15 15:22:33 +00:00
Arthur Baars
735aec9d34 Ensure top-level nodes have distinct parent_index values 2020-11-25 13:48:25 +01:00
Arthur Baars
89953fd87c Add parent_index field to @astnode 2020-11-25 13:48:25 +01:00
Arthur Baars
b72db8b6f1 Add parent field to AstNode 2020-11-25 13:48:25 +01:00
Nick Rolfe
1a9663ff7d Replace single-branch match with if let 2020-11-16 18:43:54 +00:00
Nick Rolfe
68c97a2d13 Use .. to ignore fields
Co-authored-by: Arthur Baars <aibaars@github.com>
2020-11-16 18:41:18 +00:00
Nick Rolfe
bbe7c70d34 more refactoring of names 2020-11-16 17:54:16 +00:00
Nick Rolfe
83a0e5fea6 Refactor to move naming decisions to shared library 2020-11-16 17:54:14 +00:00
Arthur Baars
c565f323f6 Don't register extra tokens as children of the parent node 2020-11-05 12:53:58 +01:00
Arthur Baars
86aa05e3cb Address comments 2020-11-04 14:49:47 +01:00
Arthur Baars
c3e8d85f0b Tolerate tokens containing invalid UTF-8 2020-11-04 14:46:31 +01:00
Arthur Baars
053c9f60a4 Store tokens in a separate table 2020-11-04 13:35:24 +01:00
Arthur Baars
65c1f2c359 Merge pull request #20 from github/aibaars/extract-extra
Extract 'extra' nodes and their subtrees
2020-11-03 13:45:33 +01:00
Arthur Baars
dc3459de8e Extract 'extra' nodes and their subtrees 2020-11-03 10:03:11 +01:00