Compare commits

...

82 Commits

Author SHA1 Message Date
shati-patel
6d64c8f031 v1.5.10
Some checks failed
Release / Release (push) Has been cancelled
Release / Publish to VS Code Marketplace (push) Has been cancelled
Release / Publish to Open VSX Registry (push) Has been cancelled
2022-01-25 16:19:57 +00:00
Charis Kyriakou
1216fce853 Download and process analyses results (#1089) 2022-01-25 08:28:53 +00:00
dependabot[bot]
c598306f49 Bump node-fetch from 2.6.1 to 2.6.7 in /extensions/ql-vscode
Bumps [node-fetch](https://github.com/node-fetch/node-fetch) from 2.6.1 to 2.6.7.
- [Release notes](https://github.com/node-fetch/node-fetch/releases)
- [Changelog](https://github.com/node-fetch/node-fetch/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/node-fetch/node-fetch/compare/v2.6.1...v2.6.7)

---
updated-dependencies:
- dependency-name: node-fetch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-24 13:56:26 -08:00
Shati Patel
4f8d6e310c Bump CLI version for integration tests 2022-01-24 11:56:10 +00:00
Shati Patel
894eb7046e Make step for maintainers only 2022-01-20 09:32:07 +00:00
shati-patel
3d6515e807 Update documentation step in PR template 2022-01-20 09:32:07 +00:00
shati-patel
068d461c14 Update progress bar for "install pack dependencies" 2022-01-20 09:15:35 +00:00
shati-patel
8e20d01b4e Sleep earlier 2022-01-19 20:46:33 +00:00
shati-patel
8aaa2492f2 Wait a few seconds before monitoring remote query run 2022-01-19 20:46:33 +00:00
Shati Patel
c9a649f974 Update extensions/ql-vscode/CHANGELOG.md
Co-authored-by: Aditya Sharad <6874315+adityasharad@users.noreply.github.com>
2022-01-19 20:43:25 +00:00
shati-patel
f07d9cff9b Update wording to be more clear 2022-01-19 20:43:25 +00:00
shati-patel
b7bfd9ea85 Add CLI version constraint for packaging 2022-01-19 20:43:25 +00:00
shati-patel
25f0e3ccab Add separate tests for valid/invalid pack install 2022-01-19 20:43:25 +00:00
shati-patel
e19addec60 Catch error in tests 2022-01-19 20:43:25 +00:00
shati-patel
a5bc25e211 Fix import + throw error 2022-01-19 20:43:25 +00:00
shati-patel
c90659fd92 First attempt at tests 2022-01-19 20:43:25 +00:00
shati-patel
30b7fe7472 Update changelog 2022-01-19 20:43:25 +00:00
shati-patel
d54fbdf4e6 Address review comments
1. Hard-code more common query packs
2. Correctly resolve workspace packs
3. Only install workspace packs
2022-01-19 20:43:25 +00:00
shati-patel
6d7b02583d Add "pack install" and "pack download" commands 2022-01-19 20:43:25 +00:00
shati-patel
51906cbbda Update dependencies in integration test runner 2022-01-19 16:16:53 +00:00
Shati Patel
d3da9d30f4 Make sure extension dependencies are installed 2022-01-19 16:16:53 +00:00
Charis Kyriakou
9b9a0cb64a Split download actions in remote queries view (#1083) 2022-01-19 09:41:04 +00:00
Andrew Eisenberg
1dde5af591 Bump CLI version to run integration tests against 2022-01-18 08:41:00 -08:00
Charis Kyriakou
4312d35743 Added paging to the listWorkflowRunArtifacts call (#1082) 2022-01-18 16:24:25 +00:00
Charis Kyriakou
2dcdbcbd32 Break remote queries view into more components (#1079) 2022-01-18 15:46:45 +00:00
Charis Kyriakou
e8e50c4381 Extract base react components (#1078) 2022-01-17 19:41:33 +00:00
Charis Kyriakou
0e6d85374f Rename analysis result to analysis summary (#1074) 2022-01-10 11:57:02 +00:00
Angela P Wen
54789613dc Merge pull request #1071 from angelapwen/standardize-integration-args
Standardize integration test args in VSCode debugger
2022-01-07 12:00:20 -08:00
Angela P Wen
43b3f72a41 Clarify instructions in comment 2022-01-07 11:51:44 -08:00
Angela P Wen
13742a4e9e Match integration test args with run-integration-tests.ts 2022-01-07 11:25:48 -08:00
Charis Kyriakou
6bd7f0ae12 Add helper command for working on the Remote Queries Results view (#1069) 2022-01-07 13:39:07 +00:00
shati-patel
fc51b336fa Update changelog 2022-01-06 19:34:29 +00:00
shati-patel
df16d1ab1d Results view: Don't reopen webview if it's already visible 2022-01-06 19:34:29 +00:00
Dominik Bamberger
b661b2be97 Update Docs ping in issue template (#1061)
* Update Docs ping in issue template

* Update .github/pull_request_template.md

Co-authored-by: Aditya Sharad <6874315+adityasharad@users.noreply.github.com>

Co-authored-by: Aditya Sharad <6874315+adityasharad@users.noreply.github.com>
2021-12-22 08:59:25 -08:00
Andrew Eisenberg
2d39bee416 Ensure all tests are run 2021-12-17 13:22:20 -08:00
Andrew Eisenberg
56eeb1badb Delete output folder before building 2021-12-17 13:09:02 -08:00
shati-patel
d547f81a55 Bump version to v1.5.10 2021-12-17 15:36:16 +00:00
Charis Kyriakou
e1b35cdbbc Fix CSS file paths
Some checks failed
Release / Release (push) Has been cancelled
Release / Publish to VS Code Marketplace (push) Has been cancelled
Release / Publish to Open VSX Registry (push) Has been cancelled
2021-12-17 15:14:08 +00:00
shati-patel
c01704b8aa v1.5.9 2021-12-17 12:04:27 +00:00
shati-patel
5a19042fc8 Update changelog 2021-12-16 17:58:55 +00:00
shati-patel
bdf8c0b9c2 Add setting to enable/disable Quick Eval codelens 2021-12-16 17:58:55 +00:00
Shati Patel
bc08cbe74f Tidy up and add test for getting query metadata (#1050)
* Move/rename query metadata function

* Add test for `tryGetQueryMetadata`

* Split into two tests
2021-12-15 20:11:59 +00:00
Andrew Eisenberg
6e2e72a500 Be nicer about where to open the results webview (#1037)
* Be nicer about where to open the results webview

Currently, the webview _always_ opens next to the currently active
editor. This is a pain if you already have 2 columns open since this
means that the webview will open in a third column, which is rarely
what you want.

This change uses a more sophisticated approach to opening the webview:

1. If there is only one column, open webview to the right of it
2. If there are multiple columns and the active editor is _not_ the
   last column, open to the right of the active editor
3. Otherwise open in the first column.

This will avoid opening a new column unless there is only one column
open right now.

There is no native API that vscode exposed to compare column locations,
so this uses the `ViewColumn` api is a slightly non-standard way.

A limitation is that if the last column is empty and the active editor
is to the left of it, then the webview will not be opened there (which
would be nice). Instead, it will be opened in column 1.

Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>
2021-12-15 19:33:53 +00:00
shati-patel
d0953fb63c Remote queries: Get query name from metadata (if possible) 2021-12-15 11:00:41 +00:00
Charis Kyriakou
4dbd15c66d Remote queries: No results view (#1048) 2021-12-15 08:55:00 +00:00
Charis Kyriakou
e9e41e07d1 Implement download behaviour in remote queries view (#1046) 2021-12-15 08:34:34 +00:00
Andrew Eisenberg
b435df4682 Fix type in comment
Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>
2021-12-14 09:39:43 -08:00
Andrew Eisenberg
a3bf9f1c71 Handle different dependencies in remote queries tests
Starting in CLI 2.7.5, there will no longer be any
`codeql/javascript-upgrades` pack. Change the test so that it passes
using both old and new packs.
2021-12-14 09:39:43 -08:00
shati-patel
72ff828b57 Style link text + tidy up functions 2021-12-14 12:13:20 +00:00
shati-patel
b7f86ae7a9 Display query text in "virtual" (readonly) file 2021-12-14 12:13:20 +00:00
shati-patel
3c73390a44 Save query text in a temporary file 2021-12-14 12:13:20 +00:00
shati-patel
7117faa92b Rename properties and handle missing files 2021-12-14 12:13:20 +00:00
shati-patel
4257555c88 Remote queries: Open query file/text from webview 2021-12-14 12:13:20 +00:00
Angela P Wen
33b1465ccc Docs: add clarification on directory for running tests via CLI 2021-12-10 12:29:15 -08:00
Andrew Eisenberg
c8ed8b2591 Add code lens for quick evaluation (#1035)
* Add code lens for quick eval command

* Ensure commented out predicates do not have code lens

* Improve conditional check for commented out predicate  detection

* Refactor regex

* Move comment check to eliminate evaluating regex more than once

Co-authored-by: marcnjaramillo <mnj.webdeveloper@gmail.com>
2021-12-10 19:17:21 +00:00
Andrew Eisenberg
58f4a82616 Update changelog 2021-12-10 07:50:08 -08:00
Andrew Eisenberg
d5f0a659af Avoid showing the alert option in the drop down
Only show it when there really is an alert table to see.
2021-12-10 07:50:08 -08:00
Charis Kyriakou
60c977bff9 Move GitHub actions code to separate module (#1044) 2021-12-10 13:59:20 +00:00
Andrew Eisenberg
73f1beac6a Bump cli version for integration tests 2021-12-09 13:50:41 -08:00
Charis Kyriakou
6195c6552f Made the repo list in the remote query view expandable (#1039) 2021-12-09 10:38:18 +00:00
Charis Kyriakou
e365744dbc Monitor remote query run and render results (#1033) 2021-12-09 10:05:51 +00:00
Andrew Eisenberg
68f566dd1a Pass --old-eval-stats to query server
This is in preparation of supporting structured query logs.
When passing this option, use the old format of query logs.
Later, when we want to add support for structured query
logs, we can add remove this option.
2021-12-07 07:53:58 -08:00
Charis Kyriakou
bf350779c9 Merge pull request #1032 from github/remote-query-submission-result
Expose remote query submission result
2021-12-06 09:28:14 +00:00
Charis Kyriakou
07329c9ea5 Expose remote query submission result 2021-12-03 16:16:48 +00:00
Shati Patel
7e6483490a Merge remote queries webview outline into main (#1027) 2021-12-03 10:48:54 +00:00
shati-patel
749565828d Bump version to v1.5.9 2021-12-02 14:27:37 +00:00
shati-patel
ff751cc877 v1.5.8
Some checks failed
Release / Release (push) Has been cancelled
Release / Publish to VS Code Marketplace (push) Has been cancelled
Release / Publish to Open VSX Registry (push) Has been cancelled
2021-12-02 11:26:36 +00:00
Robin Neatherway
d7ba941803 Merge pull request #1009 from github/aeisenberg/remote-nested-queries
Remote queries: Handle nested queries
2021-12-01 19:24:10 +00:00
Andrew Eisenberg
e58201e24b Ensure server uses a well-known query pack name 2021-12-01 10:53:51 -08:00
Henry Mercer
81e60286f2 Require canary for loading models from packs 2021-12-01 09:40:06 +00:00
Henry Mercer
8e156d69d7 Apply suggestions from code review
Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>
2021-12-01 09:40:06 +00:00
Henry Mercer
dfcaa27235 Update lockfile 2021-12-01 09:40:06 +00:00
Henry Mercer
ed0553c6b6 Gate loading ML models behind a hidden setting 2021-12-01 09:40:06 +00:00
Henry Mercer
84ecbfc7a1 Resolve ML models and pass them to the queryserver 2021-12-01 09:40:06 +00:00
Andrew Eisenberg
e13349ceb0 Update changelog 2021-11-29 11:16:49 -08:00
Andrew Eisenberg
a1bcb7519f Ensure src.zip is prioritized over src folder
Fixes a bug where legacy databases with both unzipped and zipped sources
were incorrectly being loaded with the src folder.
2021-11-29 11:16:49 -08:00
Andrew Eisenberg
b481441052 Emit more relevant error message when failing to add source folder (#1021)
* Emit more relevant error message when failing to add source folder

Fixes #1020

* Update changelog

* Clarify changelog and error message

Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>

Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>
2021-11-29 16:28:07 +00:00
github-actions[bot]
6a1d1a492e Bump version to v1.5.8 (#1017)
Co-authored-by: aeisenberg <aeisenberg@users.noreply.github.com>
2021-11-23 21:13:41 +00:00
Andrew Eisenberg
a9b9502dbd Fix failing tests on windows
1. Acknowledge that the CLI has a bug for path serialization on <=2.7.2.
   Avoid testing the query path on that version.
2. Fix calculation of root path on windows.
2021-11-22 12:53:52 -08:00
Andrew Eisenberg
d9c5ecf462 Fix failing test and remove changelog note 2021-11-19 14:32:38 -08:00
Andrew Eisenberg
4c83805030 Update changelog 2021-11-18 18:13:29 -08:00
Andrew Eisenberg
742bca1cf5 Remote queries: Handle nested queries
This change allows remote queries to run a query from a directory that
is not in the root of the qlpack.

The change is the following:

1. walk up the directory hierarchy to check for a non-local qlpack.yml
2. Copy over the files as before, but keep track of the relative
   location of the query compared to the location of the qlpack.yml.
3. Change the defaultSuite of the qlpack.yml so that _only_ this query
   is run as part of the default query.

Also, this adds a new integration test to ensure the nested query is
packaged appropriately.
2021-11-18 15:27:29 -08:00
70 changed files with 2809 additions and 195 deletions

View File

@@ -9,4 +9,4 @@ Replace this with a description of the changes your pull request makes.
- [ ] [CHANGELOG.md](https://github.com/github/vscode-codeql/blob/main/extensions/ql-vscode/CHANGELOG.md) has been updated to incorporate all user visible changes made by this pull request.
- [ ] Issues have been created for any UI or other user-facing changes made by this pull request.
- [ ] `@github/docs-content-codeql` has been cc'd in all issues for UI or other user-facing changes made by this pull request.
- [ ] _[Maintainers only]_ If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the [github/codeql](https://github.com/github/codeql/tree/main/docs/codeql/codeql-for-visual-studio-code) repo and add the `ready-for-doc-review` label there.

View File

@@ -135,7 +135,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
version: ['v2.3.3', 'v2.4.6', 'v2.5.9', 'v2.6.3', 'v2.7.2', 'nightly']
version: ['v2.3.3', 'v2.4.6', 'v2.5.9', 'v2.6.3', 'v2.7.6', 'nightly']
env:
CLI_VERSION: ${{ matrix.version }}
NIGHTLY_URL: ${{ needs.find-nightly.outputs.url }}

16
.vscode/launch.json vendored
View File

@@ -59,7 +59,9 @@
"runtimeExecutable": "${execPath}",
"args": [
"--extensionDevelopmentPath=${workspaceRoot}/extensions/ql-vscode",
"--extensionTestsPath=${workspaceRoot}/extensions/ql-vscode/out/vscode-tests/no-workspace/index"
"--extensionTestsPath=${workspaceRoot}/extensions/ql-vscode/out/vscode-tests/no-workspace/index",
"--disable-extensions",
"--disable-gpu"
],
"stopOnEntry": false,
"sourceMaps": true,
@@ -75,6 +77,8 @@
"args": [
"--extensionDevelopmentPath=${workspaceRoot}/extensions/ql-vscode",
"--extensionTestsPath=${workspaceRoot}/extensions/ql-vscode/out/vscode-tests/minimal-workspace/index",
"--disable-extensions",
"--disable-gpu",
"${workspaceRoot}/extensions/ql-vscode/test/data"
],
"stopOnEntry": false,
@@ -91,8 +95,16 @@
"args": [
"--extensionDevelopmentPath=${workspaceRoot}/extensions/ql-vscode",
"--extensionTestsPath=${workspaceRoot}/extensions/ql-vscode/out/vscode-tests/cli-integration/index",
"--disable-gpu",
"--disable-extension",
"eamodio.gitlens",
"--disable-extension",
"github.codespaces",
"--disable-extension",
"github.copilot",
"${workspaceRoot}/extensions/ql-vscode/src/vscode-tests/cli-integration/data",
// Add a path to a checked out instance of the codeql repository so the libraries are
// Uncomment the last line and modify the path to a checked out
// instance of the codeql repository so the libraries are
// available in the workspace for the tests.
// "${workspaceRoot}/../codeql"
],

View File

@@ -81,7 +81,7 @@ You can use VS Code to debug the extension without explicitly installing it. Jus
Unit tests and many integration tests do not require a copy of the CodeQL CLI.
Outside of vscode, run:
Outside of vscode, in the `extensions/ql-vscode` directory, run:
```shell
npm run test && npm run integration

View File

@@ -1,5 +1,21 @@
# CodeQL for Visual Studio Code: Changelog
## 1.5.10 - 25 January 2022
- Fix a bug where the results view moved column even when it was already visible. [#1070](https://github.com/github/vscode-codeql/pull/1070)
- Add packaging-related commands. _CodeQL: Download Packs_ downloads query packs from the package registry that can be run locally, and _CodeQL: Install Pack Dependencies_ installs dependencies for packs in your workspace. [#1076](https://github.com/github/vscode-codeql/pull/1076)
## 1.5.9 - 17 December 2021
- Avoid creating a third column when opening the results view. The results view will always open to the right of the active editor, unless the active editor is in the rightmost editor column. In that case open in the leftmost column. [#1037](https://github.com/github/vscode-codeql/pull/1037)
- Add a CodeLens to make the Quick Evaluation command more accessible. Click the `Quick Evaluation` prompt above a predicate definition in the editor to evaluate that predicate on its own. You can enable/disable this feature in the `codeQL.runningQueries.quickEvalCodelens` setting. [#1035](https://github.com/github/vscode-codeql/pull/1035) & [#1052](https://github.com/github/vscode-codeql/pull/1052)
- Fix a bug where the _Alerts_ option would show in the results view even if there is no alerts table available. [#1038](https://github.com/github/vscode-codeql/pull/1038)
## 1.5.8 - 2 December 2021
- Emit a more explicit error message when a user tries to add a database with an unzipped source folder to the workspace. [#1021](https://github.com/github/vscode-codeql/pull/1021)
- Ensure `src.zip` archives are used as the canonical source instead of `src` folders when importing databases. [#1025](https://github.com/github/vscode-codeql/pull/1025)
## 1.5.7 - 23 November 2021
- Fix the _CodeQL: Open Referenced File_ command for Windows systems. [#979](https://github.com/github/vscode-codeql/pull/979)

View File

@@ -1,5 +1,5 @@
import * as gulp from 'gulp';
import { compileTypeScript, watchTypeScript, copyViewCss } from './typescript';
import { compileTypeScript, watchTypeScript, copyViewCss, cleanOutput } from './typescript';
import { compileTextMateGrammar } from './textmate';
import { copyTestData } from './tests';
import { compileView } from './webpack';
@@ -7,9 +7,12 @@ import { packageExtension } from './package';
import { injectAppInsightsKey } from './appInsights';
export const buildWithoutPackage =
gulp.parallel(
compileTypeScript, compileTextMateGrammar, compileView, copyTestData, copyViewCss
gulp.series(
cleanOutput,
gulp.parallel(
compileTypeScript, compileTextMateGrammar, compileView, copyTestData, copyViewCss
)
);
export { compileTextMateGrammar, watchTypeScript, compileTypeScript, copyTestData, injectAppInsightsKey };
export { cleanOutput, compileTextMateGrammar, watchTypeScript, compileTypeScript, copyTestData, injectAppInsightsKey };
export default gulp.series(buildWithoutPackage, injectAppInsightsKey, packageExtension);

View File

@@ -2,6 +2,7 @@ import * as colors from 'ansi-colors';
import * as gulp from 'gulp';
import * as sourcemaps from 'gulp-sourcemaps';
import * as ts from 'gulp-typescript';
import * as del from 'del';
function goodReporter(): ts.reporter.Reporter {
return {
@@ -20,6 +21,10 @@ function goodReporter(): ts.reporter.Reporter {
const tsProject = ts.createProject('tsconfig.json');
export function cleanOutput() {
return tsProject.projectDirectory ? del(tsProject.projectDirectory + '/out/*') : Promise.resolve();
}
export function compileTypeScript() {
return tsProject.src()
.pipe(sourcemaps.init())
@@ -37,6 +42,6 @@ export function watchTypeScript() {
/** Copy CSS files for the results view into the output directory. */
export function copyViewCss() {
return gulp.src('src/view/*.css')
return gulp.src('src/**/view/*.css')
.pipe(gulp.dest('out'));
}

View File

@@ -6,6 +6,7 @@ export const config: webpack.Configuration = {
entry: {
resultsView: './src/view/results.tsx',
compareView: './src/compare/view/Compare.tsx',
remoteQueriesView: './src/remote-queries/view/RemoteQueries.tsx',
},
output: {
path: path.resolve(__dirname, '..', 'out'),

View File

@@ -1,12 +1,12 @@
{
"name": "vscode-codeql",
"version": "1.5.7",
"version": "1.5.10",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "vscode-codeql",
"version": "1.5.7",
"version": "1.5.10",
"license": "MIT",
"dependencies": {
"@octokit/rest": "^18.5.6",
@@ -16,7 +16,7 @@
"glob-promise": "^3.4.0",
"js-yaml": "^3.14.0",
"minimist": "~1.2.5",
"node-fetch": "~2.6.0",
"node-fetch": "~2.6.7",
"path-browserify": "^1.0.1",
"react": "^16.8.6",
"react-dom": "^16.8.6",
@@ -40,6 +40,7 @@
"@types/chai-as-promised": "~7.1.2",
"@types/child-process-promise": "^2.2.1",
"@types/classnames": "~2.2.9",
"@types/del": "^4.0.0",
"@types/fs-extra": "^9.0.6",
"@types/glob": "^7.1.1",
"@types/google-protobuf": "^3.2.7",
@@ -73,6 +74,7 @@
"chai": "^4.2.0",
"chai-as-promised": "~7.1.1",
"css-loader": "~3.1.0",
"del": "^6.0.0",
"eslint": "~6.8.0",
"eslint-plugin-react": "~7.19.0",
"glob": "^7.1.4",
@@ -503,6 +505,16 @@
"integrity": "sha512-rr+OQyAjxze7GgWrSaJwydHStIhHq2lvY3BOC2Mj7KnzI7XK0Uw1TOOdI9lDoajEbSWLiYgoo4f1R51erQfhPQ==",
"dev": true
},
"node_modules/@types/del": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@types/del/-/del-4.0.0.tgz",
"integrity": "sha512-LDE5atstX5kKnTobWknpmGHC52DH/tp8pIVsD2SSxaOFqW3AQr0EpdzYIfkZ331xe7l9Vn9NlJsBG6viU3mjBA==",
"deprecated": "This is a stub types definition. del provides its own type definitions, so you do not need this installed.",
"dev": true,
"dependencies": {
"del": "*"
}
},
"node_modules/@types/eslint": {
"version": "7.2.7",
"resolved": "https://registry.npmjs.org/@types/eslint/-/eslint-7.2.7.tgz",
@@ -3336,6 +3348,43 @@
"node": ">=0.10.0"
}
},
"node_modules/del": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/del/-/del-6.0.0.tgz",
"integrity": "sha512-1shh9DQ23L16oXSZKB2JxpL7iMy2E0S9d517ptA1P8iw0alkPtQcrKH7ru31rYtKwF499HkTu+DRzq3TCKDFRQ==",
"dev": true,
"dependencies": {
"globby": "^11.0.1",
"graceful-fs": "^4.2.4",
"is-glob": "^4.0.1",
"is-path-cwd": "^2.2.0",
"is-path-inside": "^3.0.2",
"p-map": "^4.0.0",
"rimraf": "^3.0.2",
"slash": "^3.0.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/del/node_modules/rimraf": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz",
"integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==",
"dev": true,
"dependencies": {
"glob": "^7.1.3"
},
"bin": {
"rimraf": "bin.js"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/delayed-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
@@ -5886,6 +5935,24 @@
"integrity": "sha1-iVJojF7C/9awPsyF52ngKQMINHA=",
"dev": true
},
"node_modules/is-path-cwd": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/is-path-cwd/-/is-path-cwd-2.2.0.tgz",
"integrity": "sha512-w942bTcih8fdJPJmQHFzkS76NEP8Kzzvmw92cXsazb8intwLqPibPPdXf4ANdKV3rYMuuQYGIWtvz9JilB3NFQ==",
"dev": true,
"engines": {
"node": ">=6"
}
},
"node_modules/is-path-inside": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/is-path-inside/-/is-path-inside-3.0.3.tgz",
"integrity": "sha512-Fd4gABb+ycGAmKou8eMftCupSir5lRxqf4aD/vd0cD2qc4HL07OjCeuHMr8Ro4CoMaeCKDB0/ECBOVWjTwUvPQ==",
"dev": true,
"engines": {
"node": ">=8"
}
},
"node_modules/is-plain-obj": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-2.1.0.tgz",
@@ -7611,11 +7678,22 @@
}
},
"node_modules/node-fetch": {
"version": "2.6.1",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.1.tgz",
"integrity": "sha512-V4aYg89jEoVRxRb2fJdAg8FHvI7cEyYdVAh94HH0UIK8oJxUfkjlDQN9RbMx+bEjP7+ggMiFRprSti032Oipxw==",
"version": "2.6.7",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.7.tgz",
"integrity": "sha512-ZjMPFEfVx5j+y2yF35Kzx5sF7kDzxuDj6ziH4FFbOp87zKDZNx8yExJIb05OGF4Nlt9IHFIMBkRl41VdvcNdbQ==",
"dependencies": {
"whatwg-url": "^5.0.0"
},
"engines": {
"node": "4.x || >=6.0.0"
},
"peerDependencies": {
"encoding": "^0.1.0"
},
"peerDependenciesMeta": {
"encoding": {
"optional": true
}
}
},
"node_modules/node-releases": {
@@ -10373,6 +10451,11 @@
"xtend": "~4.0.1"
}
},
"node_modules/tr46": {
"version": "0.0.3",
"resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz",
"integrity": "sha1-gYT9NH2snNwYWZLzpmIuFLnZq2o="
},
"node_modules/traverse": {
"version": "0.3.9",
"resolved": "https://registry.npmjs.org/traverse/-/traverse-0.3.9.tgz",
@@ -11235,6 +11318,11 @@
"node": ">=10.13.0"
}
},
"node_modules/webidl-conversions": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
"integrity": "sha1-JFNCdeKnvGvnvIZhHMFq4KVlSHE="
},
"node_modules/webpack": {
"version": "5.28.0",
"resolved": "https://registry.npmjs.org/webpack/-/webpack-5.28.0.tgz",
@@ -11546,6 +11634,15 @@
"node": ">=6"
}
},
"node_modules/whatwg-url": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
"integrity": "sha1-lmRU6HZUYuN2RNNib2dCzotwll0=",
"dependencies": {
"tr46": "~0.0.3",
"webidl-conversions": "^3.0.0"
}
},
"node_modules/which": {
"version": "1.3.1",
"resolved": "https://registry.npmjs.org/which/-/which-1.3.1.tgz",
@@ -12253,6 +12350,15 @@
"integrity": "sha512-rr+OQyAjxze7GgWrSaJwydHStIhHq2lvY3BOC2Mj7KnzI7XK0Uw1TOOdI9lDoajEbSWLiYgoo4f1R51erQfhPQ==",
"dev": true
},
"@types/del": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@types/del/-/del-4.0.0.tgz",
"integrity": "sha512-LDE5atstX5kKnTobWknpmGHC52DH/tp8pIVsD2SSxaOFqW3AQr0EpdzYIfkZ331xe7l9Vn9NlJsBG6viU3mjBA==",
"dev": true,
"requires": {
"del": "*"
}
},
"@types/eslint": {
"version": "7.2.7",
"resolved": "https://registry.npmjs.org/@types/eslint/-/eslint-7.2.7.tgz",
@@ -14609,6 +14715,33 @@
}
}
},
"del": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/del/-/del-6.0.0.tgz",
"integrity": "sha512-1shh9DQ23L16oXSZKB2JxpL7iMy2E0S9d517ptA1P8iw0alkPtQcrKH7ru31rYtKwF499HkTu+DRzq3TCKDFRQ==",
"dev": true,
"requires": {
"globby": "^11.0.1",
"graceful-fs": "^4.2.4",
"is-glob": "^4.0.1",
"is-path-cwd": "^2.2.0",
"is-path-inside": "^3.0.2",
"p-map": "^4.0.0",
"rimraf": "^3.0.2",
"slash": "^3.0.0"
},
"dependencies": {
"rimraf": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz",
"integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==",
"dev": true,
"requires": {
"glob": "^7.1.3"
}
}
}
},
"delayed-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
@@ -16684,6 +16817,18 @@
"integrity": "sha1-iVJojF7C/9awPsyF52ngKQMINHA=",
"dev": true
},
"is-path-cwd": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/is-path-cwd/-/is-path-cwd-2.2.0.tgz",
"integrity": "sha512-w942bTcih8fdJPJmQHFzkS76NEP8Kzzvmw92cXsazb8intwLqPibPPdXf4ANdKV3rYMuuQYGIWtvz9JilB3NFQ==",
"dev": true
},
"is-path-inside": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/is-path-inside/-/is-path-inside-3.0.3.tgz",
"integrity": "sha512-Fd4gABb+ycGAmKou8eMftCupSir5lRxqf4aD/vd0cD2qc4HL07OjCeuHMr8Ro4CoMaeCKDB0/ECBOVWjTwUvPQ==",
"dev": true
},
"is-plain-obj": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-2.1.0.tgz",
@@ -18055,9 +18200,12 @@
}
},
"node-fetch": {
"version": "2.6.1",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.1.tgz",
"integrity": "sha512-V4aYg89jEoVRxRb2fJdAg8FHvI7cEyYdVAh94HH0UIK8oJxUfkjlDQN9RbMx+bEjP7+ggMiFRprSti032Oipxw=="
"version": "2.6.7",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.7.tgz",
"integrity": "sha512-ZjMPFEfVx5j+y2yF35Kzx5sF7kDzxuDj6ziH4FFbOp87zKDZNx8yExJIb05OGF4Nlt9IHFIMBkRl41VdvcNdbQ==",
"requires": {
"whatwg-url": "^5.0.0"
}
},
"node-releases": {
"version": "1.1.71",
@@ -20275,6 +20423,11 @@
}
}
},
"tr46": {
"version": "0.0.3",
"resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz",
"integrity": "sha1-gYT9NH2snNwYWZLzpmIuFLnZq2o="
},
"traverse": {
"version": "0.3.9",
"resolved": "https://registry.npmjs.org/traverse/-/traverse-0.3.9.tgz",
@@ -20959,6 +21112,11 @@
"graceful-fs": "^4.1.2"
}
},
"webidl-conversions": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
"integrity": "sha1-JFNCdeKnvGvnvIZhHMFq4KVlSHE="
},
"webpack": {
"version": "5.28.0",
"resolved": "https://registry.npmjs.org/webpack/-/webpack-5.28.0.tgz",
@@ -21168,6 +21326,15 @@
}
}
},
"whatwg-url": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
"integrity": "sha1-lmRU6HZUYuN2RNNib2dCzotwll0=",
"requires": {
"tr46": "~0.0.3",
"webidl-conversions": "^3.0.0"
}
},
"which": {
"version": "1.3.1",
"resolved": "https://registry.npmjs.org/which/-/which-1.3.1.tgz",

View File

@@ -4,7 +4,7 @@
"description": "CodeQL for Visual Studio Code",
"author": "GitHub",
"private": true,
"version": "1.5.7",
"version": "1.5.10",
"publisher": "GitHub",
"license": "MIT",
"icon": "media/VS-marketplace-CodeQL-icon.png",
@@ -19,7 +19,8 @@
"Programming Languages"
],
"extensionDependencies": [
"hbenl.vscode-test-explorer"
"hbenl.vscode-test-explorer",
"ms-vscode.test-adapter-converter"
],
"capabilities": {
"untrustedWorkspaces": {
@@ -206,6 +207,11 @@
"default": null,
"description": "Path to a directory where the CodeQL extension should store query server logs. If empty, the extension stores logs in a temporary workspace folder and deletes the contents after each run."
},
"codeQL.runningQueries.quickEvalCodelens": {
"type": "boolean",
"default": true,
"description": "Enable the 'Quick Evaluation' CodeLens."
},
"codeQL.resultsDisplay.pageSize": {
"type": "integer",
"default": 200,
@@ -284,6 +290,10 @@
"command": "codeQL.runRemoteQuery",
"title": "CodeQL: Run Remote Query"
},
{
"command": "codeQL.showFakeRemoteQueryResults",
"title": "CodeQL: [Internal] Show fake remote query results"
},
{
"command": "codeQL.runQueries",
"title": "CodeQL: Run Queries in Selected Files"
@@ -364,6 +374,14 @@
"command": "codeQL.clearCache",
"title": "CodeQL: Clear Cache"
},
{
"command": "codeQL.installPackDependencies",
"title": "CodeQL: Install Pack Dependencies"
},
{
"command": "codeQL.downloadPacks",
"title": "CodeQL: Download Packs"
},
{
"command": "codeQLDatabases.setCurrentDatabase",
"title": "Set Current Database"
@@ -746,6 +764,10 @@
"command": "codeQL.runRemoteQuery",
"when": "config.codeQL.canary && editorLangId == ql && resourceExtname == .ql"
},
{
"command": "codeQL.showFakeRemoteQueryResults",
"when": "config.codeQL.canary"
},
{
"command": "codeQL.runQueries",
"when": "false"
@@ -987,7 +1009,7 @@
"glob-promise": "^3.4.0",
"js-yaml": "^3.14.0",
"minimist": "~1.2.5",
"node-fetch": "~2.6.0",
"node-fetch": "~2.6.7",
"path-browserify": "^1.0.1",
"react": "^16.8.6",
"react-dom": "^16.8.6",
@@ -1011,6 +1033,7 @@
"@types/chai-as-promised": "~7.1.2",
"@types/child-process-promise": "^2.2.1",
"@types/classnames": "~2.2.9",
"@types/del": "^4.0.0",
"@types/fs-extra": "^9.0.6",
"@types/glob": "^7.1.1",
"@types/google-protobuf": "^3.2.7",
@@ -1044,6 +1067,7 @@
"chai": "^4.2.0",
"chai-as-promised": "~7.1.1",
"css-loader": "~3.1.0",
"del": "^6.0.0",
"eslint": "~6.8.0",
"eslint-plugin-react": "~7.19.0",
"glob": "^7.1.4",

View File

@@ -87,6 +87,15 @@ export type QlpacksInfo = { [name: string]: string[] };
*/
export type LanguagesInfo = { [name: string]: string[] };
/** Information about an ML model, as resolved by `codeql resolve ml-models`. */
export type MlModelInfo = {
checksum: string;
path: string;
};
/** The expected output of `codeql resolve ml-models`. */
export type MlModelsInfo = { models: MlModelInfo[] };
/**
* The expected output of `codeql resolve qlref`.
*/
@@ -160,7 +169,7 @@ export class CodeQLCliServer implements Disposable {
/** Version of current cli, lazily computed by the `getVersion()` method */
private _version: SemVer | undefined;
/**
/**
* The languages supported by the current version of the CLI, computed by `getSupportedLanguages()`.
*/
private _supportedLanguages: string[] | undefined;
@@ -584,6 +593,12 @@ export class CodeQLCliServer implements Disposable {
return await this.runJsonCodeQlCliCommand<QueryMetadata>(['resolve', 'metadata'], [queryPath], 'Resolving query metadata');
}
/** Resolves the ML models that should be available when evaluating a query. */
async resolveMlModels(additionalPacks: string[]): Promise<MlModelsInfo> {
return await this.runJsonCodeQlCliCommand<MlModelsInfo>(['resolve', 'ml-models'], ['--additional-packs',
additionalPacks.join(path.delimiter)], 'Resolving ML models', false);
}
/**
* Gets the RAM setting for the query server.
* @param queryMemoryMb The maximum amount of RAM to use, in MB.
@@ -621,16 +636,16 @@ export class CodeQLCliServer implements Disposable {
return await this.runCodeQlCliCommand(['database', 'unbundle'], subcommandArgs, `Extracting ${archivePath} to directory ${target}`);
}
/**
* Uses a .qhelp file to generate Query Help documentation in a specified format.
* @param pathToQhelp The path to the .qhelp file
* @param format The format in which the query help should be generated {@link https://codeql.github.com/docs/codeql-cli/manual/generate-query-help/#cmdoption-codeql-generate-query-help-format}
* @param outputDirectory The output directory for the generated file
*/
async generateQueryHelp(pathToQhelp:string, outputDirectory?: string): Promise<string> {
async generateQueryHelp(pathToQhelp: string, outputDirectory?: string): Promise<string> {
const subcommandArgs = ['--format=markdown'];
if(outputDirectory) subcommandArgs.push('--output', outputDirectory);
if (outputDirectory) subcommandArgs.push('--output', outputDirectory);
subcommandArgs.push(pathToQhelp);
return await this.runCodeQlCliCommand(['generate', 'query-help'], subcommandArgs, `Generating qhelp in markdown format at ${outputDirectory}`);
@@ -791,7 +806,7 @@ export class CodeQLCliServer implements Disposable {
/**
* Gets the list of available languages. Refines the result of `resolveLanguages()`, by excluding
* extra things like "xml" and "properties".
*
*
* @returns An array of languages that are supported by the current version of the CodeQL CLI.
*/
public async getSupportedLanguages(): Promise<string[]> {
@@ -830,6 +845,14 @@ export class CodeQLCliServer implements Disposable {
);
}
/**
* Downloads a specified pack.
* @param packs The `<package-scope/name[@version]>` of the packs to download.
*/
async packDownload(packs: string[]) {
return this.runJsonCodeQlCliCommand(['pack', 'download'], packs, 'Downloading packs');
}
async packInstall(dir: string) {
return this.runJsonCodeQlCliCommand(['pack', 'install'], [dir], 'Installing pack dependencies');
}
@@ -1166,6 +1189,21 @@ export class CliVersionConstraint {
*/
public static CLI_VERSION_REMOTE_QUERIES = new SemVer('2.6.3');
/**
* CLI version where the `resolve ml-models` subcommand was introduced.
*/
public static CLI_VERSION_WITH_RESOLVE_ML_MODELS = new SemVer('2.7.3');
/**
* CLI version where the `--old-eval-stats` option to the query server was introduced.
*/
public static CLI_VERSION_WITH_OLD_EVAL_STATS = new SemVer('2.7.4');
/**
* CLI version where packaging was introduced.
*/
public static CLI_VERSION_WITH_PACKAGING = new SemVer('2.6.0');
constructor(private readonly cli: CodeQLCliServer) {
/**/
}
@@ -1210,4 +1248,15 @@ export class CliVersionConstraint {
return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_REMOTE_QUERIES);
}
async supportsResolveMlModels() {
return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_RESOLVE_ML_MODELS);
}
async supportsOldEvalStats() {
return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_OLD_EVAL_STATS);
}
async supportsPackaging() {
return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_PACKAGING);
}
}

View File

@@ -135,13 +135,13 @@ export class CompareInterfaceManager extends DisposableObject {
);
const stylesheetPathOnDisk = Uri.file(
ctx.asAbsolutePath('out/resultsView.css')
ctx.asAbsolutePath('out/view/resultsView.css')
);
panel.webview.html = getHtmlForWebview(
panel.webview,
scriptPathOnDisk,
stylesheetPathOnDisk
[stylesheetPathOnDisk]
);
panel.webview.onDidReceiveMessage(
async (e) => this.handleMsgFromView(e),

View File

@@ -275,6 +275,16 @@ export class CliConfigListener extends ConfigListener implements CliConfig {
}
}
/**
* Whether to enable CodeLens for the 'Quick Evaluation' command.
*/
const QUICK_EVAL_CODELENS_SETTING = new Setting('quickEvalCodelens', RUNNING_QUERIES_SETTING);
export function isQuickEvalCodelensEnabled() {
return QUICK_EVAL_CODELENS_SETTING.getValue<boolean>();
}
// Enable experimental features
/**
@@ -333,3 +343,15 @@ export function getRemoteControllerRepo(): string | undefined {
export async function setRemoteControllerRepo(repo: string | undefined) {
await REMOTE_CONTROLLER_REPO.updateValue(repo, ConfigurationTarget.Global);
}
/**
* Whether to insecurely load ML models from CodeQL packs.
*
* This setting is for internal users only.
*/
const SHOULD_INSECURELY_LOAD_MODELS_FROM_PACKS =
new Setting('shouldInsecurelyLoadModelsFromPacks', RUNNING_QUERIES_SETTING);
export function shouldInsecurelyLoadMlModelsFromPacks(): boolean {
return SHOULD_INSECURELY_LOAD_MODELS_FROM_PACKS.getValue<boolean>();
}

View File

@@ -121,20 +121,21 @@ async function findDataset(parentDirectory: string): Promise<vscode.Uri> {
return vscode.Uri.file(dbAbsolutePath);
}
async function findSourceArchive(
// exported for testing
export async function findSourceArchive(
databasePath: string, silent = false
): Promise<vscode.Uri | undefined> {
const relativePaths = ['src', 'output/src_archive'];
for (const relativePath of relativePaths) {
const basePath = path.join(databasePath, relativePath);
const zipPath = basePath + '.zip';
if (await fs.pathExists(basePath)) {
return vscode.Uri.file(basePath);
} else if (await fs.pathExists(zipPath)) {
// Prefer using a zip archive over a directory.
if (await fs.pathExists(zipPath)) {
return encodeArchiveBasePath(zipPath);
} else if (await fs.pathExists(basePath)) {
return vscode.Uri.file(basePath);
}
}
if (!silent) {
@@ -161,7 +162,6 @@ async function resolveDatabase(
datasetUri,
sourceArchiveUri
};
}
/** Gets the relative paths of all `.dbscheme` files in the given directory. */
@@ -258,7 +258,7 @@ export interface DatabaseItem {
* Returns the root uri of the virtual filesystem for this database's source archive,
* as displayed in the filesystem explorer.
*/
getSourceArchiveExplorerUri(): vscode.Uri | undefined;
getSourceArchiveExplorerUri(): vscode.Uri;
/**
* Holds if `uri` belongs to this database's source archive.
@@ -274,6 +274,11 @@ export interface DatabaseItem {
* Gets the state of this database, to be persisted in the workspace state.
*/
getPersistedState(): PersistedDatabaseItem;
/**
* Verifies that this database item has a zipped source folder. Returns an error message if it does not.
*/
verifyZippedSources(): string | undefined;
}
export enum DatabaseEventKind {
@@ -459,13 +464,26 @@ export class DatabaseItemImpl implements DatabaseItem {
/**
* Returns the root uri of the virtual filesystem for this database's source archive.
*/
public getSourceArchiveExplorerUri(): vscode.Uri | undefined {
public getSourceArchiveExplorerUri(): vscode.Uri {
const sourceArchive = this.sourceArchive;
if (sourceArchive === undefined || !sourceArchive.fsPath.endsWith('.zip'))
return undefined;
if (sourceArchive === undefined || !sourceArchive.fsPath.endsWith('.zip')) {
throw new Error(this.verifyZippedSources());
}
return encodeArchiveBasePath(sourceArchive.fsPath);
}
public verifyZippedSources(): string | undefined {
const sourceArchive = this.sourceArchive;
if (sourceArchive === undefined) {
return `${this.name} has no source archive.`;
}
if (!sourceArchive.fsPath.endsWith('.zip')) {
return `${this.name} has a source folder that is unzipped.`;
}
return;
}
/**
* Holds if `uri` belongs to this database's source archive.
*/
@@ -603,26 +621,28 @@ export class DatabaseManager extends DisposableObject {
// This is undesirable, as we might be adding and removing many
// workspace folders as the user adds and removes databases.
const end = (vscode.workspace.workspaceFolders || []).length;
const msg = item.verifyZippedSources();
if (msg) {
void logger.log(`Could not add source folder because ${msg}`);
return;
}
const uri = item.getSourceArchiveExplorerUri();
if (uri === undefined) {
void logger.log(`Couldn't obtain file explorer uri for ${item.name}`);
}
else {
void logger.log(`Adding workspace folder for ${item.name} source archive at index ${end}`);
if ((vscode.workspace.workspaceFolders || []).length < 2) {
// Adding this workspace folder makes the workspace
// multi-root, which may surprise the user. Let them know
// we're doing this.
void vscode.window.showInformationMessage(`Adding workspace folder for source archive of database ${item.name}.`);
}
vscode.workspace.updateWorkspaceFolders(end, 0, {
name: `[${item.name} source archive]`,
uri,
});
// vscode api documentation says we must to wait for this event
// between multiple `updateWorkspaceFolders` calls.
await eventFired(vscode.workspace.onDidChangeWorkspaceFolders);
void logger.log(`Adding workspace folder for ${item.name} source archive at index ${end}`);
if ((vscode.workspace.workspaceFolders || []).length < 2) {
// Adding this workspace folder makes the workspace
// multi-root, which may surprise the user. Let them know
// we're doing this.
void vscode.window.showInformationMessage(`Adding workspace folder for source archive of database ${item.name}.`);
}
vscode.workspace.updateWorkspaceFolders(end, 0, {
name: `[${item.name} source archive]`,
uri,
});
// vscode api documentation says we must to wait for this event
// between multiple `updateWorkspaceFolders` calls.
await eventFired(vscode.workspace.onDidChangeWorkspaceFolders);
}
}
@@ -732,7 +752,7 @@ export class DatabaseManager extends DisposableObject {
this.updatePersistedCurrentDatabaseItem();
await vscode.commands.executeCommand('setContext', 'codeQL.currentDatabaseItem', item?.name);
this._onDidChangeCurrentDatabaseItem.fire({
item,
kind: DatabaseEventKind.Change

View File

@@ -11,7 +11,10 @@ import {
window as Window,
env,
window,
QuickPickItem
QuickPickItem,
Range,
workspace,
ProviderResult
} from 'vscode';
import { LanguageClient } from 'vscode-languageclient';
import * as os from 'os';
@@ -21,6 +24,7 @@ import { testExplorerExtensionId, TestHub } from 'vscode-test-adapter-api';
import { AstViewer } from './astViewer';
import * as archiveFilesystemProvider from './archive-filesystem-provider';
import QuickEvalCodeLensProvider from './quickEvalCodeLensProvider';
import { CodeQLCliServer, CliVersionConstraint } from './cli';
import {
CliConfigListener,
@@ -74,7 +78,13 @@ import {
import { CodeQlStatusBarHandler } from './status-bar';
import { Credentials } from './authentication';
import { runRemoteQuery } from './run-remote-query';
import { RemoteQueriesManager } from './remote-queries/remote-queries-manager';
import { RemoteQuery } from './remote-queries/remote-query';
import { URLSearchParams } from 'url';
import { RemoteQueriesInterfaceManager } from './remote-queries/remote-queries-interface';
import { sampleRemoteQuery, sampleRemoteQueryResult } from './remote-queries/sample-data';
import { handleDownloadPacks, handleInstallPackDependencies } from './packaging';
import { AnalysesResultsManager } from './remote-queries/analyses-results-manager';
/**
* extension.ts
@@ -155,6 +165,7 @@ export interface CodeQLExtensionInterface {
* @returns CodeQLExtensionInterface
*/
export async function activate(ctx: ExtensionContext): Promise<CodeQLExtensionInterface | Record<string, never>> {
void logger.log(`Starting ${extensionId} extension`);
if (extension === undefined) {
throw new Error(`Can't find extension ${extensionId}`);
@@ -165,6 +176,9 @@ export async function activate(ctx: ExtensionContext): Promise<CodeQLExtensionIn
await initializeTelemetry(extension, ctx);
languageSupport.install();
const codelensProvider = new QuickEvalCodeLensProvider();
languages.registerCodeLensProvider({ scheme: 'file', language: 'ql' }, codelensProvider);
ctx.subscriptions.push(distributionConfigListener);
const codeQlVersionRange = DEFAULT_DISTRIBUTION_VERSION_RANGE;
const distributionManager = new DistributionManager(distributionConfigListener, codeQlVersionRange, ctx);
@@ -470,6 +484,7 @@ async function activateWithInstalledDistribution(
progress: ProgressCallback,
token: CancellationToken,
databaseItem: DatabaseItem | undefined,
range?: Range
): Promise<void> {
if (qs !== undefined) {
// If no databaseItem is specified, use the database currently selected in the Databases UI
@@ -484,7 +499,9 @@ async function activateWithInstalledDistribution(
quickEval,
selectedQuery,
progress,
token
token,
undefined,
range
);
const item = qhm.buildCompletedQuery(info);
await showResultsForCompletedQuery(item, WebviewReveal.NotForced);
@@ -501,23 +518,23 @@ async function activateWithInstalledDistribution(
selectedQuery: Uri
): Promise<void> {
// selectedQuery is unpopulated when executing through the command palette
const pathToQhelp = selectedQuery ? selectedQuery.fsPath : window.activeTextEditor?.document.uri.fsPath;
if(pathToQhelp) {
const pathToQhelp = selectedQuery ? selectedQuery.fsPath : window.activeTextEditor?.document.uri.fsPath;
if (pathToQhelp) {
// Create temporary directory
const relativePathToMd = path.basename(pathToQhelp, '.qhelp') + '.md';
const absolutePathToMd = path.join(qhelpTmpDir.name, relativePathToMd);
const uri = Uri.file(absolutePathToMd);
try {
await cliServer.generateQueryHelp(pathToQhelp , absolutePathToMd);
await cliServer.generateQueryHelp(pathToQhelp, absolutePathToMd);
await commands.executeCommand('markdown.showPreviewToSide', uri);
} catch (err) {
const errorMessage = err.message.includes('Generating qhelp in markdown') ? (
const errorMessage = err.message.includes('Generating qhelp in markdown') ? (
`Could not generate markdown from ${pathToQhelp}: Bad formatting in .qhelp file.`
) : `Could not open a preview of the generated file (${absolutePathToMd}).`;
void helpers.showAndLogErrorMessage(errorMessage, { fullMessage: `${errorMessage}\n${err}` });
}
}
}
async function openReferencedFile(
@@ -732,6 +749,22 @@ async function activateWithInstalledDistribution(
cancellable: true
})
);
ctx.subscriptions.push(
commandRunnerWithProgress(
'codeQL.codeLensQuickEval',
async (
progress: ProgressCallback,
token: CancellationToken,
uri: Uri,
range: Range
) => await compileAndRunQuery(true, uri, progress, token, undefined, range),
{
title: 'Running query',
cancellable: true
})
);
ctx.subscriptions.push(
commandRunnerWithProgress('codeQL.quickQuery', async (
progress: ProgressCallback,
@@ -743,6 +776,12 @@ async function activateWithInstalledDistribution(
}
)
);
void logger.log('Initializing remote queries interface.');
const rqm = new RemoteQueriesManager(ctx, logger, cliServer);
registerRemoteQueryTextProvider();
// The "runRemoteQuery" command is internal-only.
ctx.subscriptions.push(
commandRunnerWithProgress('codeQL.runRemoteQuery', async (
@@ -756,8 +795,11 @@ async function activateWithInstalledDistribution(
step: 0,
message: 'Getting credentials'
});
const credentials = await Credentials.initialize(ctx);
await runRemoteQuery(cliServer, credentials, uri || window.activeTextEditor?.document.uri, false, progress, token);
await rqm.runRemoteQuery(
uri || window.activeTextEditor?.document.uri,
progress,
token
);
} else {
throw new Error('Remote queries require the CodeQL Canary version to run.');
}
@@ -766,6 +808,21 @@ async function activateWithInstalledDistribution(
cancellable: true
})
);
ctx.subscriptions.push(
commandRunner('codeQL.monitorRemoteQuery', async (
query: RemoteQuery,
token: CancellationToken) => {
await rqm.monitorRemoteQuery(query, token);
}));
ctx.subscriptions.push(
commandRunner('codeQL.showFakeRemoteQueryResults', async () => {
const analysisResultsManager = new AnalysesResultsManager(ctx, logger);
const rqim = new RemoteQueriesInterfaceManager(ctx, logger, analysisResultsManager);
await rqim.showResults(sampleRemoteQuery, sampleRemoteQueryResult);
}));
ctx.subscriptions.push(
commandRunner(
'codeQL.openReferencedFile',
@@ -868,6 +925,26 @@ async function activateWithInstalledDistribution(
}
}));
ctx.subscriptions.push(
commandRunnerWithProgress('codeQL.installPackDependencies', async (
progress: ProgressCallback
) =>
await handleInstallPackDependencies(cliServer, progress),
{
title: 'Installing pack dependencies',
}
));
ctx.subscriptions.push(
commandRunnerWithProgress('codeQL.downloadPacks', async (
progress: ProgressCallback
) =>
await handleDownloadPacks(cliServer, progress),
{
title: 'Downloading packs',
}
));
commands.registerCommand('codeQL.showLogs', () => {
logger.show();
});
@@ -939,3 +1016,20 @@ async function initializeLogging(ctx: ExtensionContext): Promise<void> {
}
const checkForUpdatesCommand = 'codeQL.checkForUpdatesToCLI';
/**
* This text provider lets us open readonly files in the editor.
*
* TODO: Consolidate this with the 'codeql' text provider in query-history.ts.
*/
function registerRemoteQueryTextProvider() {
workspace.registerTextDocumentContentProvider('remote-query', {
provideTextDocumentContent(
uri: Uri
): ProviderResult<string> {
const params = new URLSearchParams(uri.query);
return params.get('queryText');
},
});
}

View File

@@ -12,6 +12,7 @@ import {
import { CodeQLCliServer, QlpacksInfo } from './cli';
import { UserCancellationException } from './commandRunner';
import { logger } from './logging';
import { QueryMetadata } from './pure/interface-types';
/**
* Show an error message and log it to the console
@@ -516,3 +517,19 @@ export async function askForLanguage(cliServer: CodeQLCliServer, throwOnEmpty =
}
return language;
}
/**
* Gets metadata for a query, if it exists.
* @param cliServer The CLI server.
* @param queryPath The path to the query.
* @returns A promise that resolves to the query metadata, if available.
*/
export async function tryGetQueryMetadata(cliServer: CodeQLCliServer, queryPath: string): Promise<QueryMetadata | undefined> {
try {
return await cliServer.resolveMetadata(queryPath);
} catch (e) {
// Ignore errors and provide no metadata.
void logger.log(`Couldn't resolve metadata for ${queryPath}: ${e}`);
return;
}
}

View File

@@ -1,4 +1,5 @@
import * as crypto from 'crypto';
import * as os from 'os';
import {
Uri,
Location,
@@ -117,13 +118,19 @@ export function tryResolveLocation(
export function getHtmlForWebview(
webview: Webview,
scriptUriOnDisk: Uri,
stylesheetUriOnDisk: Uri
stylesheetUrisOnDisk: Uri[],
): string {
// Convert the on-disk URIs into webview URIs.
const scriptWebviewUri = webview.asWebviewUri(scriptUriOnDisk);
const stylesheetWebviewUri = webview.asWebviewUri(stylesheetUriOnDisk);
const stylesheetWebviewUris = stylesheetUrisOnDisk.map(stylesheetUriOnDisk =>
webview.asWebviewUri(stylesheetUriOnDisk));
// Use a nonce in the content security policy to uniquely identify the above resources.
const nonce = getNonce();
const stylesheetsHtmlLines = stylesheetWebviewUris.map(stylesheetWebviewUri =>
`<link nonce="${nonce}" rel="stylesheet" href="${stylesheetWebviewUri}">`);
/*
* Content security policy:
* default-src: allow nothing by default.
@@ -137,7 +144,7 @@ export function getHtmlForWebview(
<head>
<meta http-equiv="Content-Security-Policy"
content="default-src 'none'; script-src 'nonce-${nonce}'; style-src 'nonce-${nonce}'; connect-src ${webview.cspSource};">
<link nonce="${nonce}" rel="stylesheet" href="${stylesheetWebviewUri}">
${stylesheetsHtmlLines.join(` ${os.EOL}`)}
</head>
<body>
<div id=root>

View File

@@ -160,10 +160,11 @@ export class InterfaceManager extends DisposableObject {
getPanel(): vscode.WebviewPanel {
if (this._panel == undefined) {
const { ctx } = this;
const webViewColumn = this.chooseColumnForWebview();
const panel = (this._panel = Window.createWebviewPanel(
'resultsView', // internal name
'CodeQL Query Results', // user-visible name
{ viewColumn: vscode.ViewColumn.Beside, preserveFocus: true },
{ viewColumn: webViewColumn, preserveFocus: true },
{
enableScripts: true,
enableFindWidget: true,
@@ -187,12 +188,12 @@ export class InterfaceManager extends DisposableObject {
ctx.asAbsolutePath('out/resultsView.js')
);
const stylesheetPathOnDisk = vscode.Uri.file(
ctx.asAbsolutePath('out/resultsView.css')
ctx.asAbsolutePath('out/view/resultsView.css')
);
panel.webview.html = getHtmlForWebview(
panel.webview,
scriptPathOnDisk,
stylesheetPathOnDisk
[stylesheetPathOnDisk]
);
panel.webview.onDidReceiveMessage(
async (e) => this.handleMsgFromView(e),
@@ -203,6 +204,29 @@ export class InterfaceManager extends DisposableObject {
return this._panel;
}
/**
* Choose where to open the webview.
*
* If there is a single view column, then open beside it.
* If there are multiple view columns, then open beside the active column,
* unless the active editor is the last column. In this case, open in the first column.
*
* The goal is to avoid opening new columns when there already are two columns open.
*/
private chooseColumnForWebview(): vscode.ViewColumn {
// This is not a great way to determine the number of view columns, but I
// can't find a vscode API that does it any better.
// Here, iterate through all the visible editors and determine the max view column.
// This won't work if the largest view column is empty.
const colCount = Window.visibleTextEditors.reduce((maxVal, editor) =>
Math.max(maxVal, Number.parseInt(editor.viewColumn?.toFixed() || '0', 10)), 0);
if (colCount <= 1) {
return vscode.ViewColumn.Beside;
}
const activeViewColumnNum = Number.parseInt(Window.activeTextEditor?.viewColumn?.toFixed() || '0', 10);
return activeViewColumnNum === colCount ? vscode.ViewColumn.One : vscode.ViewColumn.Beside;
}
private async changeInterpretedSortState(
sortState: InterpretedResultsSortState | undefined
): Promise<void> {
@@ -355,27 +379,29 @@ export class InterfaceManager extends DisposableObject {
const panel = this.getPanel();
await this.waitForPanelLoaded();
if (forceReveal === WebviewReveal.Forced) {
panel.reveal(undefined, true);
} else if (!panel.visible) {
// The results panel exists, (`.getPanel()` guarantees it) but
// is not visible; it's in a not-currently-viewed tab. Show a
// more asynchronous message to not so abruptly interrupt
// user's workflow by immediately revealing the panel.
const showButton = 'View Results';
const queryName = results.queryName;
const resultPromise = vscode.window.showInformationMessage(
`Finished running query ${queryName.length > 0 ? ` "${queryName}"` : ''
}.`,
showButton
);
// Address this click asynchronously so we still update the
// query history immediately.
void resultPromise.then((result) => {
if (result === showButton) {
panel.reveal();
}
});
if (!panel.visible) {
if (forceReveal === WebviewReveal.Forced) {
panel.reveal(undefined, true);
} else {
// The results panel exists, (`.getPanel()` guarantees it) but
// is not visible; it's in a not-currently-viewed tab. Show a
// more asynchronous message to not so abruptly interrupt
// user's workflow by immediately revealing the panel.
const showButton = 'View Results';
const queryName = results.queryName;
const resultPromise = vscode.window.showInformationMessage(
`Finished running query ${queryName.length > 0 ? ` "${queryName}"` : ''
}.`,
showButton
);
// Address this click asynchronously so we still update the
// query history immediately.
void resultPromise.then((result) => {
if (result === showButton) {
panel.reveal();
}
});
}
}
// Note that the resultSetSchemas will return offsets for the default (unsorted) page,

View File

@@ -0,0 +1,146 @@
import { CliVersionConstraint, CodeQLCliServer } from './cli';
import {
getOnDiskWorkspaceFolders,
showAndLogErrorMessage,
showAndLogInformationMessage,
} from './helpers';
import { QuickPickItem, window } from 'vscode';
import { ProgressCallback, UserCancellationException } from './commandRunner';
import { logger } from './logging';
const QUERY_PACKS = [
'codeql/cpp-queries',
'codeql/csharp-queries',
'codeql/go-queries',
'codeql/java-queries',
'codeql/javascript-queries',
'codeql/python-queries',
'codeql/ruby-queries',
'codeql/csharp-solorigate-queries',
'codeql/javascript-experimental-atm-queries',
];
/**
* Prompts user to choose packs to download, and downloads them.
*
* @param cliServer The CLI server.
* @param progress A progress callback.
*/
export async function handleDownloadPacks(
cliServer: CodeQLCliServer,
progress: ProgressCallback,
): Promise<void> {
if (!(await cliServer.cliConstraints.supportsPackaging())) {
throw new Error(`Packaging commands are not supported by this version of CodeQL. Please upgrade to v${CliVersionConstraint.CLI_VERSION_WITH_PACKAGING
} or later.`);
}
progress({
message: 'Choose packs to download',
step: 1,
maxStep: 2,
});
let packsToDownload: string[] = [];
const queryPackOption = 'Download all core query packs';
const customPackOption = 'Download custom specified pack';
const quickpick = await window.showQuickPick(
[queryPackOption, customPackOption],
{ ignoreFocusOut: true }
);
if (quickpick === queryPackOption) {
packsToDownload = QUERY_PACKS;
} else if (quickpick === customPackOption) {
const customPack = await window.showInputBox({
prompt:
'Enter the <package-scope/name[@version]> of the pack to download',
ignoreFocusOut: true,
});
if (customPack) {
packsToDownload.push(customPack);
} else {
throw new UserCancellationException('No pack specified.');
}
}
if (packsToDownload?.length > 0) {
progress({
message: 'Downloading packs. This may take a few minutes.',
step: 2,
maxStep: 2,
});
try {
await cliServer.packDownload(packsToDownload);
void showAndLogInformationMessage('Finished downloading packs.');
} catch (error) {
void showAndLogErrorMessage(
'Unable to download all packs. See log for more details.'
);
}
}
}
interface QLPackQuickPickItem extends QuickPickItem {
packRootDir: string[];
}
/**
* Prompts user to choose packs to install, and installs them.
*
* @param cliServer The CLI server.
* @param progress A progress callback.
*/
export async function handleInstallPackDependencies(
cliServer: CodeQLCliServer,
progress: ProgressCallback,
): Promise<void> {
if (!(await cliServer.cliConstraints.supportsPackaging())) {
throw new Error(`Packaging commands are not supported by this version of CodeQL. Please upgrade to v${CliVersionConstraint.CLI_VERSION_WITH_PACKAGING
} or later.`);
}
progress({
message: 'Choose packs to install dependencies for',
step: 1,
maxStep: 2,
});
const workspacePacks = await cliServer.resolveQlpacks(getOnDiskWorkspaceFolders());
const quickPickItems = Object.entries(workspacePacks).map<QLPackQuickPickItem>(([key, value]) => ({
label: key,
packRootDir: value,
}));
const packsToInstall = await window.showQuickPick(quickPickItems, {
placeHolder: 'Select packs to install dependencies for',
canPickMany: true,
ignoreFocusOut: true,
});
const numberOfPacks = packsToInstall?.length || 0;
if (packsToInstall && numberOfPacks > 0) {
const failedPacks = [];
const errors = [];
// Start at 1 because we already have the first step
let count = 1;
for (const pack of packsToInstall) {
count++;
progress({
message: `Installing dependencies for ${pack.label}`,
step: count,
maxStep: numberOfPacks + 1,
});
try {
for (const dir of pack.packRootDir) {
await cliServer.packInstall(dir);
}
} catch (error) {
failedPacks.push(pack.label);
errors.push(error);
}
}
if (failedPacks.length > 0) {
void logger.log(`Errors:\n${errors.join('\n')}`);
throw new Error(
`Unable to install pack dependencies for: ${failedPacks.join(', ')}. See log for more details.`
);
} else {
void showAndLogInformationMessage('Finished installing pack dependencies.');
}
} else {
throw new UserCancellationException('No packs selected.');
}
}

View File

@@ -1,4 +1,6 @@
import * as sarif from 'sarif';
import { AnalysisResults } from '../remote-queries/shared/analysis-result';
import { AnalysisSummary, RemoteQueryResult } from '../remote-queries/shared/remote-query-result';
import { RawResultSet, ResultRow, ResultSetSchema, Column, ResolvableLocationValue } from './bqrs-cli-types';
/**
@@ -180,6 +182,11 @@ export interface OpenFileMsg {
filePath: string;
}
export interface OpenVirtualFileMsg {
t: 'openVirtualFile';
queryText: string;
}
/**
* Message from the results view to toggle the display of
* query diagnostics.
@@ -364,3 +371,44 @@ export interface ParsedResultSets {
resultSetNames: string[];
resultSet: ResultSet;
}
export type FromRemoteQueriesMessage =
| RemoteQueryLoadedMessage
| RemoteQueryErrorMessage
| OpenFileMsg
| OpenVirtualFileMsg
| RemoteQueryDownloadAnalysisResultsMessage
| RemoteQueryDownloadAllAnalysesResultsMessage;
export type ToRemoteQueriesMessage =
| SetRemoteQueryResultMessage
| SetAnalysesResultsMessage;
export interface RemoteQueryLoadedMessage {
t: 'remoteQueryLoaded';
}
export interface SetRemoteQueryResultMessage {
t: 'setRemoteQueryResult';
queryResult: RemoteQueryResult
}
export interface SetAnalysesResultsMessage {
t: 'setAnalysesResults';
analysesResults: AnalysisResults[];
}
export interface RemoteQueryErrorMessage {
t: 'remoteQueryError';
error: string;
}
export interface RemoteQueryDownloadAnalysisResultsMessage {
t: 'remoteQueryDownloadAnalysisResults';
analysisSummary: AnalysisSummary
}
export interface RemoteQueryDownloadAllAnalysesResultsMessage {
t: 'remoteQueryDownloadAllAnalysesResults';
analysisSummaries: AnalysisSummary[];
}

View File

@@ -711,6 +711,11 @@ export interface EvaluateQueriesParams {
export type TemplateDefinitions = { [key: string]: TemplateSource }
export interface MlModel {
/** A URI pointing to the root directory of the model. */
uri: string;
}
/**
* A single query that should be run
*/
@@ -744,6 +749,11 @@ export interface QueryToRun {
* map should be set to the empty set or give an error.
*/
allowUnknownTemplates: boolean;
/**
* The list of ML models that should be made available
* when evaluating the query.
*/
availableMlModels?: MlModel[];
}
/**

View File

@@ -32,7 +32,7 @@ export type QueryHistoryItemOptions = {
isQuickQuery?: boolean;
};
const SHOW_QUERY_TEXT_MSG = `\
export const SHOW_QUERY_TEXT_MSG = `\
////////////////////////////////////////////////////////////////////////////////////
// This is the text of the entire query file when it was executed for this query //
// run. The text or dependent libraries may have changed since then. //

View File

@@ -167,6 +167,10 @@ export class QueryServerClient extends DisposableObject {
args.push('--require-db-registration');
}
if (await this.cliServer.cliConstraints.supportsOldEvalStats()) {
args.push('--old-eval-stats');
}
if (this.config.debug) {
args.push('--debug', '--tuple-counting');
}

View File

@@ -0,0 +1,46 @@
import {
CodeLensProvider,
TextDocument,
CodeLens,
Command,
Range
} from 'vscode';
import { isQuickEvalCodelensEnabled } from './config';
class QuickEvalCodeLensProvider implements CodeLensProvider {
async provideCodeLenses(document: TextDocument): Promise<CodeLens[]> {
const codeLenses: CodeLens[] = [];
if (isQuickEvalCodelensEnabled()) {
for (let index = 0; index < document.lineCount; index++) {
const textLine = document.lineAt(index);
// Match a predicate signature, including predicate name, parameter list, and opening brace.
// This currently does not match predicates that span multiple lines.
const regex = new RegExp(/(\w+)\s*\([^()]*\)\s*\{/);
const matches = textLine.text.match(regex);
// Make sure that a code lens is not generated for any predicate that is commented out.
if (matches && !(/^\s*\/\//).test(textLine.text)) {
const range: Range = new Range(
textLine.range.start.line, matches.index!,
textLine.range.end.line, matches.index! + 1
);
const command: Command = {
command: 'codeQL.codeLensQuickEval',
title: `Quick Evaluation: ${matches[1]}`,
arguments: [document.uri, range]
};
const codeLens = new CodeLens(range, command);
codeLenses.push(codeLens);
}
}
}
return codeLenses;
}
}
export default QuickEvalCodeLensProvider;

View File

@@ -0,0 +1,94 @@
import { ExtensionContext } from 'vscode';
import { Credentials } from '../authentication';
import { Logger } from '../logging';
import { downloadArtifactFromLink } from './gh-actions-api-client';
import * as path from 'path';
import * as fs from 'fs-extra';
import { AnalysisSummary } from './shared/remote-query-result';
import * as sarif from 'sarif';
import { AnalysisResults, QueryResult } from './shared/analysis-result';
export class AnalysesResultsManager {
// Store for the results of various analyses for a single remote query.
private readonly analysesResults: AnalysisResults[];
constructor(
private readonly ctx: ExtensionContext,
private readonly logger: Logger,
) {
this.analysesResults = [];
}
public async downloadAnalysisResults(
analysisSummary: AnalysisSummary,
): Promise<void> {
if (this.analysesResults.some(x => x.nwo === analysisSummary.nwo)) {
// We already have the results for this analysis, don't download again.
return;
}
const credentials = await Credentials.initialize(this.ctx);
void this.logger.log(`Downloading and processing results for ${analysisSummary.nwo}`);
await this.downloadSingleAnalysisResults(analysisSummary, credentials);
}
public async downloadAllResults(
analysisSummaries: AnalysisSummary[],
): Promise<void> {
const credentials = await Credentials.initialize(this.ctx);
void this.logger.log('Downloading and processing all results');
for (const analysis of analysisSummaries) {
await this.downloadSingleAnalysisResults(analysis, credentials);
}
}
public getAnalysesResults(): AnalysisResults[] {
return [...this.analysesResults];
}
private async downloadSingleAnalysisResults(
analysis: AnalysisSummary,
credentials: Credentials
): Promise<void> {
const artifactPath = await downloadArtifactFromLink(credentials, analysis.downloadLink);
let analysisResults: AnalysisResults;
if (path.extname(artifactPath) === '.sarif') {
const queryResults = await this.readResults(artifactPath);
analysisResults = { nwo: analysis.nwo, results: queryResults };
} else {
void this.logger.log('Cannot download results. Only alert and path queries are fully supported.');
analysisResults = { nwo: analysis.nwo, results: [] };
}
this.analysesResults.push(analysisResults);
}
private async readResults(filePath: string): Promise<QueryResult[]> {
const queryResults: QueryResult[] = [];
const sarifContents = await fs.readFile(filePath, 'utf8');
const sarifLog = JSON.parse(sarifContents) as sarif.Log;
// Read the sarif file and extract information that we want to display
// in the UI. For now we're only getting the message texts but we'll gradually
// extract more information based on the UX we want to build.
sarifLog.runs?.forEach(run => {
run?.results?.forEach(result => {
if (result?.message?.text) {
queryResults.push({
message: result.message.text
});
}
});
});
return queryResults;
}
}

View File

@@ -0,0 +1,20 @@
/**
* Represents a link to an artifact to be downloaded.
*/
export interface DownloadLink {
/**
* A unique id of the artifact being downloaded.
*/
id: string;
/**
* The URL path to use against the GitHub API to download the
* linked artifact.
*/
urlPath: string;
/**
* An optional path to follow inside the downloaded archive containing the artifact.
*/
innerFilePath?: string;
}

View File

@@ -0,0 +1,263 @@
import * as unzipper from 'unzipper';
import * as path from 'path';
import * as fs from 'fs-extra';
import { showAndLogWarningMessage } from '../helpers';
import { Credentials } from '../authentication';
import { logger } from '../logging';
import { tmpDir } from '../run-queries';
import { RemoteQueryWorkflowResult } from './remote-query-workflow-result';
import { DownloadLink } from './download-link';
import { RemoteQuery } from './remote-query';
import { RemoteQueryResultIndex, RemoteQueryResultIndexItem } from './remote-query-result-index';
interface ApiResultIndexItem {
nwo: string;
id: string;
results_count: number;
bqrs_file_size: number;
sarif_file_size?: number;
}
export async function getRemoteQueryIndex(
credentials: Credentials,
remoteQuery: RemoteQuery
): Promise<RemoteQueryResultIndex | undefined> {
const controllerRepo = remoteQuery.controllerRepository;
const owner = controllerRepo.owner;
const repoName = controllerRepo.name;
const workflowRunId = remoteQuery.actionsWorkflowRunId;
const workflowUri = `https://github.com/${owner}/${repoName}/actions/runs/${workflowRunId}`;
const artifactsUrlPath = `/repos/${owner}/${repoName}/actions/artifacts`;
const artifactList = await listWorkflowRunArtifacts(credentials, owner, repoName, workflowRunId);
const resultIndexArtifactId = getArtifactIDfromName('result-index', workflowUri, artifactList);
const resultIndexItems = await getResultIndexItems(credentials, owner, repoName, resultIndexArtifactId);
const allResultsArtifactId = getArtifactIDfromName('all-results', workflowUri, artifactList);
const items = resultIndexItems.map(item => {
const artifactId = getArtifactIDfromName(item.id, workflowUri, artifactList);
return {
id: item.id.toString(),
artifactId: artifactId,
nwo: item.nwo,
resultCount: item.results_count,
bqrsFileSize: item.bqrs_file_size,
sarifFileSize: item.sarif_file_size,
} as RemoteQueryResultIndexItem;
});
return {
allResultsArtifactId,
artifactsUrlPath,
items,
};
}
export async function downloadArtifactFromLink(
credentials: Credentials,
downloadLink: DownloadLink
): Promise<string> {
const octokit = await credentials.getOctokit();
// Download the zipped artifact.
const response = await octokit.request(`GET ${downloadLink.urlPath}/zip`, {});
const zipFilePath = path.join(tmpDir.name, `${downloadLink.id}.zip`);
await saveFile(`${zipFilePath}`, response.data as ArrayBuffer);
// Extract the zipped artifact.
const extractedPath = path.join(tmpDir.name, downloadLink.id);
await unzipFile(zipFilePath, extractedPath);
return downloadLink.innerFilePath
? path.join(extractedPath, downloadLink.innerFilePath)
: extractedPath;
}
/**
* Downloads the result index artifact and extracts the result index items.
* @param credentials Credentials for authenticating to the GitHub API.
* @param owner
* @param repo
* @param workflowRunId The ID of the workflow run to get the result index for.
* @returns An object containing the result index.
*/
async function getResultIndexItems(
credentials: Credentials,
owner: string,
repo: string,
artifactId: number
): Promise<ApiResultIndexItem[]> {
const artifactPath = await downloadArtifact(credentials, owner, repo, artifactId);
const indexFilePath = path.join(artifactPath, 'index.json');
if (!(await fs.pathExists(indexFilePath))) {
void showAndLogWarningMessage('Could not find an `index.json` file in the result artifact.');
return [];
}
const resultIndex = await fs.readFile(path.join(artifactPath, 'index.json'), 'utf8');
try {
return JSON.parse(resultIndex);
} catch (error) {
throw new Error(`Invalid result index file: ${error}`);
}
}
/**
* Gets the status of a workflow run.
* @param credentials Credentials for authenticating to the GitHub API.
* @param owner
* @param repo
* @param workflowRunId The ID of the workflow run to get the result index for.
* @returns The workflow run status.
*/
export async function getWorkflowStatus(
credentials: Credentials,
owner: string,
repo: string,
workflowRunId: number): Promise<RemoteQueryWorkflowResult> {
const octokit = await credentials.getOctokit();
const workflowRun = await octokit.rest.actions.getWorkflowRun({
owner,
repo,
run_id: workflowRunId
});
if (workflowRun.data.status === 'completed') {
if (workflowRun.data.conclusion === 'success') {
return { status: 'CompletedSuccessfully' };
} else {
const error = getWorkflowError(workflowRun.data.conclusion);
return { status: 'CompletedUnsuccessfully', error };
}
}
return { status: 'InProgress' };
}
/**
* Lists the workflow run artifacts for the given workflow run ID.
* @param credentials Credentials for authenticating to the GitHub API.
* @param owner
* @param repo
* @param workflowRunId The ID of the workflow run to list artifacts for.
* @returns An array of artifact details (including artifact name and ID).
*/
async function listWorkflowRunArtifacts(
credentials: Credentials,
owner: string,
repo: string,
workflowRunId: number
) {
const octokit = await credentials.getOctokit();
// There are limits on the number of artifacts that are returned by the API
// so we use paging to make sure we retrieve all of them.
let morePages = true;
let pageNum = 1;
const allArtifacts = [];
while (morePages) {
const response = await octokit.rest.actions.listWorkflowRunArtifacts({
owner,
repo,
run_id: workflowRunId,
per_page: 100,
page: pageNum
});
allArtifacts.push(...response.data.artifacts);
pageNum++;
if (response.data.artifacts.length < 100) {
morePages = false;
}
}
return allArtifacts;
}
/**
* @param artifactName The artifact name, as a string.
* @param artifacts An array of artifact details (from the "list workflow run artifacts" API response).
* @returns The artifact ID corresponding to the given artifact name.
*/
function getArtifactIDfromName(
artifactName: string,
workflowUri: string,
artifacts: Array<{ id: number, name: string }>
): number {
const artifact = artifacts.find(a => a.name === artifactName);
if (!artifact) {
const errorMessage =
`Could not find artifact with name ${artifactName} in workflow ${workflowUri}.
Please check whether the workflow run has successfully completed.`;
throw Error(errorMessage);
}
return artifact?.id;
}
/**
* Downloads an artifact from a workflow run.
* @param credentials Credentials for authenticating to the GitHub API.
* @param owner
* @param repo
* @param artifactId The ID of the artifact to download.
* @returns The path to the enclosing directory of the unzipped artifact.
*/
async function downloadArtifact(
credentials: Credentials,
owner: string,
repo: string,
artifactId: number
): Promise<string> {
const octokit = await credentials.getOctokit();
const response = await octokit.rest.actions.downloadArtifact({
owner,
repo,
artifact_id: artifactId,
archive_format: 'zip',
});
const artifactPath = path.join(tmpDir.name, `${artifactId}`);
await saveFile(`${artifactPath}.zip`, response.data as ArrayBuffer);
await unzipFile(`${artifactPath}.zip`, artifactPath);
return artifactPath;
}
async function saveFile(filePath: string, data: ArrayBuffer): Promise<void> {
void logger.log(`Saving file to ${filePath}`);
await fs.writeFile(filePath, Buffer.from(data));
}
async function unzipFile(sourcePath: string, destinationPath: string) {
void logger.log(`Unzipping file to ${destinationPath}`);
const file = await unzipper.Open.file(sourcePath);
await file.extract({ path: destinationPath });
}
function getWorkflowError(conclusion: string | null): string {
if (!conclusion) {
return 'Workflow finished without a conclusion';
}
if (conclusion === 'cancelled') {
return 'The remote query execution was cancelled.';
}
if (conclusion === 'timed_out') {
return 'The remote query execution timed out.';
}
if (conclusion === 'failure') {
// TODO: Get the actual error from the workflow or potentially
// from an artifact from the action itself.
return 'The remote query execution has failed.';
}
return `Unexpected query execution conclusion: ${conclusion}`;
}

View File

@@ -0,0 +1,284 @@
import {
WebviewPanel,
ExtensionContext,
window as Window,
ViewColumn,
Uri,
workspace,
} from 'vscode';
import * as path from 'path';
import { tmpDir } from '../run-queries';
import {
ToRemoteQueriesMessage,
FromRemoteQueriesMessage,
RemoteQueryDownloadAnalysisResultsMessage,
RemoteQueryDownloadAllAnalysesResultsMessage,
} from '../pure/interface-types';
import { Logger } from '../logging';
import { getHtmlForWebview } from '../interface-utils';
import { assertNever } from '../pure/helpers-pure';
import { AnalysisSummary, RemoteQueryResult } from './remote-query-result';
import { RemoteQuery } from './remote-query';
import { RemoteQueryResult as RemoteQueryResultViewModel } from './shared/remote-query-result';
import { AnalysisSummary as AnalysisResultViewModel } from './shared/remote-query-result';
import { showAndLogWarningMessage } from '../helpers';
import { URLSearchParams } from 'url';
import { SHOW_QUERY_TEXT_MSG } from '../query-history';
import { AnalysesResultsManager } from './analyses-results-manager';
import { AnalysisResults } from './shared/analysis-result';
export class RemoteQueriesInterfaceManager {
private panel: WebviewPanel | undefined;
private panelLoaded = false;
private panelLoadedCallBacks: (() => void)[] = [];
constructor(
private readonly ctx: ExtensionContext,
private readonly logger: Logger,
private readonly analysesResultsManager: AnalysesResultsManager
) {
this.panelLoadedCallBacks.push(() => {
void logger.log('Remote queries view loaded');
});
}
async showResults(query: RemoteQuery, queryResult: RemoteQueryResult) {
this.getPanel().reveal(undefined, true);
await this.waitForPanelLoaded();
await this.postMessage({
t: 'setRemoteQueryResult',
queryResult: this.buildViewModel(query, queryResult)
});
}
/**
* Builds up a model tailored to the view based on the query and result domain entities.
* The data is cleaned up, sorted where necessary, and transformed to a format that
* the view model can use.
* @param query Information about the query that was run.
* @param queryResult The result of the query.
* @returns A fully created view model.
*/
private buildViewModel(query: RemoteQuery, queryResult: RemoteQueryResult): RemoteQueryResultViewModel {
const queryFileName = path.basename(query.queryFilePath);
const totalResultCount = queryResult.analysisSummaries.reduce((acc, cur) => acc + cur.resultCount, 0);
const executionDuration = this.getDuration(queryResult.executionEndTime, query.executionStartTime);
const analysisSummaries = this.buildAnalysisSummaries(queryResult.analysisSummaries);
const affectedRepositories = queryResult.analysisSummaries.filter(r => r.resultCount > 0);
return {
queryTitle: query.queryName,
queryFileName: queryFileName,
queryFilePath: query.queryFilePath,
queryText: query.queryText,
totalRepositoryCount: query.repositories.length,
affectedRepositoryCount: affectedRepositories.length,
totalResultCount: totalResultCount,
executionTimestamp: this.formatDate(query.executionStartTime),
executionDuration: executionDuration,
downloadLink: queryResult.allResultsDownloadLink,
analysisSummaries: analysisSummaries
};
}
getPanel(): WebviewPanel {
if (this.panel == undefined) {
const { ctx } = this;
const panel = (this.panel = Window.createWebviewPanel(
'remoteQueriesView',
'Remote Query Results',
{ viewColumn: ViewColumn.Active, preserveFocus: true },
{
enableScripts: true,
enableFindWidget: true,
retainContextWhenHidden: true,
localResourceRoots: [
Uri.file(tmpDir.name),
Uri.file(path.join(this.ctx.extensionPath, 'out')),
],
}
));
this.panel.onDidDispose(
() => {
this.panel = undefined;
},
null,
ctx.subscriptions
);
const scriptPathOnDisk = Uri.file(
ctx.asAbsolutePath('out/remoteQueriesView.js')
);
const baseStylesheetUriOnDisk = Uri.file(
ctx.asAbsolutePath('out/remote-queries/view/baseStyles.css')
);
const stylesheetPathOnDisk = Uri.file(
ctx.asAbsolutePath('out/remote-queries/view/remoteQueries.css')
);
panel.webview.html = getHtmlForWebview(
panel.webview,
scriptPathOnDisk,
[baseStylesheetUriOnDisk, stylesheetPathOnDisk]
);
panel.webview.onDidReceiveMessage(
async (e) => this.handleMsgFromView(e),
undefined,
ctx.subscriptions
);
}
return this.panel;
}
private waitForPanelLoaded(): Promise<void> {
return new Promise((resolve) => {
if (this.panelLoaded) {
resolve();
} else {
this.panelLoadedCallBacks.push(resolve);
}
});
}
private async openFile(filePath: string) {
try {
const textDocument = await workspace.openTextDocument(filePath);
await Window.showTextDocument(textDocument, ViewColumn.One);
} catch (error) {
void showAndLogWarningMessage(`Could not open file: ${filePath}`);
}
}
private async openVirtualFile(text: string) {
try {
const params = new URLSearchParams({
queryText: encodeURIComponent(SHOW_QUERY_TEXT_MSG + text)
});
const uri = Uri.parse(
`remote-query:query-text.ql?${params.toString()}`,
true
);
const doc = await workspace.openTextDocument(uri);
await Window.showTextDocument(doc, { preview: false });
} catch (error) {
void showAndLogWarningMessage('Could not open query text');
}
}
private async handleMsgFromView(
msg: FromRemoteQueriesMessage
): Promise<void> {
switch (msg.t) {
case 'remoteQueryLoaded':
this.panelLoaded = true;
this.panelLoadedCallBacks.forEach((cb) => cb());
this.panelLoadedCallBacks = [];
break;
case 'remoteQueryError':
void this.logger.log(
`Remote query error: ${msg.error}`
);
break;
case 'openFile':
await this.openFile(msg.filePath);
break;
case 'openVirtualFile':
await this.openVirtualFile(msg.queryText);
break;
case 'remoteQueryDownloadAnalysisResults':
await this.downloadAnalysisResults(msg);
break;
case 'remoteQueryDownloadAllAnalysesResults':
await this.downloadAllAnalysesResults(msg);
break;
default:
assertNever(msg);
}
}
private async downloadAnalysisResults(msg: RemoteQueryDownloadAnalysisResultsMessage): Promise<void> {
await this.analysesResultsManager.downloadAnalysisResults(msg.analysisSummary);
await this.setAnalysisResults(this.analysesResultsManager.getAnalysesResults());
}
private async downloadAllAnalysesResults(msg: RemoteQueryDownloadAllAnalysesResultsMessage): Promise<void> {
await this.analysesResultsManager.downloadAllResults(msg.analysisSummaries);
await this.setAnalysisResults(this.analysesResultsManager.getAnalysesResults());
}
private async setAnalysisResults(analysesResults: AnalysisResults[]): Promise<void> {
await this.postMessage({
t: 'setAnalysesResults',
analysesResults: analysesResults
});
}
private postMessage(msg: ToRemoteQueriesMessage): Thenable<boolean> {
return this.getPanel().webview.postMessage(msg);
}
private getDuration(startTime: Date, endTime: Date): string {
const diffInMs = startTime.getTime() - endTime.getTime();
return this.formatDuration(diffInMs);
}
private formatDuration(ms: number): string {
const seconds = ms / 1000;
const minutes = seconds / 60;
const hours = minutes / 60;
const days = hours / 24;
if (days > 1) {
return `${days.toFixed(2)} days`;
} else if (hours > 1) {
return `${hours.toFixed(2)} hours`;
} else if (minutes > 1) {
return `${minutes.toFixed(2)} minutes`;
} else {
return `${seconds.toFixed(2)} seconds`;
}
}
private formatDate = (d: Date): string => {
const datePart = d.toLocaleDateString(undefined, { day: 'numeric', month: 'short' });
const timePart = d.toLocaleTimeString(undefined, { hour: 'numeric', minute: 'numeric', hour12: true });
return `${datePart} at ${timePart}`;
};
private formatFileSize(bytes: number): string {
const kb = bytes / 1024;
const mb = kb / 1024;
const gb = mb / 1024;
if (bytes < 1024) {
return `${bytes} bytes`;
} else if (kb < 1024) {
return `${kb.toFixed(2)} KB`;
} else if (mb < 1024) {
return `${mb.toFixed(2)} MB`;
} else {
return `${gb.toFixed(2)} GB`;
}
}
/**
* Builds up a list of analysis summaries, in a data structure tailored to the view.
* @param analysisSummaries The summaries of a specific analyses.
* @returns A fully created view model.
*/
private buildAnalysisSummaries(analysisSummaries: AnalysisSummary[]): AnalysisResultViewModel[] {
const filteredAnalysisSummaries = analysisSummaries.filter(r => r.resultCount > 0);
const sortedAnalysisSummaries = filteredAnalysisSummaries.sort((a, b) => b.resultCount - a.resultCount);
return sortedAnalysisSummaries.map((analysisResult) => ({
nwo: analysisResult.nwo,
resultCount: analysisResult.resultCount,
downloadLink: analysisResult.downloadLink,
fileSize: this.formatFileSize(analysisResult.fileSizeInBytes)
}));
}
}

View File

@@ -0,0 +1,105 @@
import { CancellationToken, commands, ExtensionContext, Uri, window } from 'vscode';
import { Credentials } from '../authentication';
import { CodeQLCliServer } from '../cli';
import { ProgressCallback } from '../commandRunner';
import { showAndLogErrorMessage, showInformationMessageWithAction } from '../helpers';
import { Logger } from '../logging';
import { runRemoteQuery } from './run-remote-query';
import { RemoteQueriesInterfaceManager } from './remote-queries-interface';
import { RemoteQuery } from './remote-query';
import { RemoteQueriesMonitor } from './remote-queries-monitor';
import { getRemoteQueryIndex } from './gh-actions-api-client';
import { RemoteQueryResultIndex } from './remote-query-result-index';
import { RemoteQueryResult } from './remote-query-result';
import { DownloadLink } from './download-link';
import { AnalysesResultsManager } from './analyses-results-manager';
export class RemoteQueriesManager {
private readonly remoteQueriesMonitor: RemoteQueriesMonitor;
private readonly analysesResultsManager: AnalysesResultsManager;
constructor(
private readonly ctx: ExtensionContext,
private readonly logger: Logger,
private readonly cliServer: CodeQLCliServer
) {
this.analysesResultsManager = new AnalysesResultsManager(ctx, logger);
this.remoteQueriesMonitor = new RemoteQueriesMonitor(ctx, logger);
}
public async runRemoteQuery(
uri: Uri | undefined,
progress: ProgressCallback,
token: CancellationToken
): Promise<void> {
const credentials = await Credentials.initialize(this.ctx);
const querySubmission = await runRemoteQuery(
this.cliServer,
credentials, uri || window.activeTextEditor?.document.uri,
false,
progress,
token);
if (querySubmission && querySubmission.query) {
void commands.executeCommand('codeQL.monitorRemoteQuery', querySubmission.query);
}
}
public async monitorRemoteQuery(
query: RemoteQuery,
cancellationToken: CancellationToken
): Promise<void> {
const credentials = await Credentials.initialize(this.ctx);
const queryResult = await this.remoteQueriesMonitor.monitorQuery(query, cancellationToken);
const executionEndTime = new Date();
if (queryResult.status === 'CompletedSuccessfully') {
const resultIndex = await getRemoteQueryIndex(credentials, query);
if (!resultIndex) {
void showAndLogErrorMessage(`There was an issue retrieving the result for the query ${query.queryName}`);
return;
}
const queryResult = this.mapQueryResult(executionEndTime, resultIndex);
const totalResultCount = queryResult.analysisSummaries.reduce((acc, cur) => acc + cur.resultCount, 0);
const message = `Query "${query.queryName}" run on ${query.repositories.length} repositories and returned ${totalResultCount} results`;
const shouldOpenView = await showInformationMessageWithAction(message, 'View');
if (shouldOpenView) {
const rqim = new RemoteQueriesInterfaceManager(this.ctx, this.logger, this.analysesResultsManager);
await rqim.showResults(query, queryResult);
}
} else if (queryResult.status === 'CompletedUnsuccessfully') {
await showAndLogErrorMessage(`Remote query execution failed. Error: ${queryResult.error}`);
return;
} else if (queryResult.status === 'Cancelled') {
await showAndLogErrorMessage('Remote query monitoring was cancelled');
}
}
private mapQueryResult(executionEndTime: Date, resultIndex: RemoteQueryResultIndex): RemoteQueryResult {
const analysisSummaries = resultIndex.items.map(item => ({
nwo: item.nwo,
resultCount: item.resultCount,
fileSizeInBytes: item.sarifFileSize ? item.sarifFileSize : item.bqrsFileSize,
downloadLink: {
id: item.artifactId.toString(),
urlPath: `${resultIndex.artifactsUrlPath}/${item.artifactId}`,
innerFilePath: item.sarifFileSize ? 'results.sarif' : 'results.bqrs'
} as DownloadLink
}));
return {
executionEndTime,
analysisSummaries,
allResultsDownloadLink: {
id: resultIndex.allResultsArtifactId.toString(),
urlPath: `${resultIndex.artifactsUrlPath}/${resultIndex.allResultsArtifactId}`
}
};
}
}

View File

@@ -0,0 +1,61 @@
import * as vscode from 'vscode';
import { Credentials } from '../authentication';
import { Logger } from '../logging';
import { getWorkflowStatus } from './gh-actions-api-client';
import { RemoteQuery } from './remote-query';
import { RemoteQueryWorkflowResult } from './remote-query-workflow-result';
export class RemoteQueriesMonitor {
// With a sleep of 5 seconds, the maximum number of attempts takes
// us to just over 2 days worth of monitoring.
private static readonly maxAttemptCount = 17280;
private static readonly sleepTime = 5000;
constructor(
private readonly extensionContext: vscode.ExtensionContext,
private readonly logger: Logger
) {
}
public async monitorQuery(
remoteQuery: RemoteQuery,
cancellationToken: vscode.CancellationToken
): Promise<RemoteQueryWorkflowResult> {
const credentials = await Credentials.initialize(this.extensionContext);
if (!credentials) {
throw Error('Error authenticating with GitHub');
}
let attemptCount = 0;
while (attemptCount <= RemoteQueriesMonitor.maxAttemptCount) {
await this.sleep(RemoteQueriesMonitor.sleepTime);
if (cancellationToken && cancellationToken.isCancellationRequested) {
return { status: 'Cancelled' };
}
const workflowStatus = await getWorkflowStatus(
credentials,
remoteQuery.controllerRepository.owner,
remoteQuery.controllerRepository.name,
remoteQuery.actionsWorkflowRunId);
if (workflowStatus.status !== 'InProgress') {
return workflowStatus;
}
attemptCount++;
}
void this.logger.log('Remote query monitoring timed out after 2 days');
return { status: 'Cancelled' };
}
private async sleep(ms: number) {
return new Promise(resolve => setTimeout(resolve, ms));
}
}

View File

@@ -0,0 +1,14 @@
export interface RemoteQueryResultIndex {
artifactsUrlPath: string;
allResultsArtifactId: number;
items: RemoteQueryResultIndexItem[];
}
export interface RemoteQueryResultIndexItem {
id: string;
artifactId: number;
nwo: string;
resultCount: number;
bqrsFileSize: number;
sarifFileSize?: number;
}

View File

@@ -0,0 +1,14 @@
import { DownloadLink } from './download-link';
export interface RemoteQueryResult {
executionEndTime: Date;
analysisSummaries: AnalysisSummary[];
allResultsDownloadLink: DownloadLink;
}
export interface AnalysisSummary {
nwo: string,
resultCount: number,
downloadLink: DownloadLink,
fileSizeInBytes: number
}

View File

@@ -0,0 +1,6 @@
import { RemoteQuery } from './remote-query';
export interface RemoteQuerySubmissionResult {
queryDirPath?: string;
query?: RemoteQuery;
}

View File

@@ -0,0 +1,10 @@
export type RemoteQueryWorkflowStatus =
| 'InProgress'
| 'CompletedSuccessfully'
| 'CompletedUnsuccessfully'
| 'Cancelled';
export interface RemoteQueryWorkflowResult {
status: RemoteQueryWorkflowStatus;
error?: string;
}

View File

@@ -0,0 +1,11 @@
import { Repository } from './repository';
export interface RemoteQuery {
queryName: string;
queryFilePath: string;
queryText: string;
controllerRepository: Repository;
repositories: Repository[];
executionStartTime: Date;
actionsWorkflowRunId: number;
}

View File

@@ -0,0 +1,4 @@
export interface Repository {
owner: string;
name: string;
}

View File

@@ -3,14 +3,25 @@ import * as path from 'path';
import * as yaml from 'js-yaml';
import * as fs from 'fs-extra';
import * as tmp from 'tmp-promise';
import { askForLanguage, findLanguage, getOnDiskWorkspaceFolders, showAndLogErrorMessage, showAndLogInformationMessage, showInformationMessageWithAction } from './helpers';
import { Credentials } from './authentication';
import * as cli from './cli';
import { logger } from './logging';
import { getRemoteControllerRepo, getRemoteRepositoryLists, setRemoteControllerRepo } from './config';
import { tmpDir } from './run-queries';
import { ProgressCallback, UserCancellationException } from './commandRunner';
import {
askForLanguage,
findLanguage,
getOnDiskWorkspaceFolders,
showAndLogErrorMessage,
showAndLogInformationMessage,
showInformationMessageWithAction,
tryGetQueryMetadata
} from '../helpers';
import { Credentials } from '../authentication';
import * as cli from '../cli';
import { logger } from '../logging';
import { getRemoteControllerRepo, getRemoteRepositoryLists, setRemoteControllerRepo } from '../config';
import { tmpDir } from '../run-queries';
import { ProgressCallback, UserCancellationException } from '../commandRunner';
import { OctokitResponse } from '@octokit/types/dist-types';
import { RemoteQuery } from './remote-query';
import { RemoteQuerySubmissionResult } from './remote-query-submission-result';
import { QueryMetadata } from '../pure/interface-types';
interface Config {
repositories: string[];
@@ -18,6 +29,13 @@ interface Config {
language?: string;
}
export interface QlPack {
name: string;
version: string;
dependencies: { [key: string]: string };
defaultSuite?: Record<string, unknown>[];
defaultSuiteFile?: string;
}
interface RepoListQuickPickItem extends QuickPickItem {
repoList: string[];
}
@@ -33,6 +51,11 @@ interface QueriesResponse {
*/
const REPO_REGEX = /^(?:[a-zA-Z0-9]+-)*[a-zA-Z0-9]+\/[a-zA-Z0-9-_]+$/;
/**
* Well-known names for the query pack used by the server.
*/
const QUERY_PACK_NAME = 'codeql-remote/query';
/**
* Gets the repositories to run the query against.
*/
@@ -89,13 +112,9 @@ async function generateQueryPack(cliServer: cli.CodeQLCliServer, queryFile: stri
base64Pack: string,
language: string
}> {
const originalPackRoot = path.dirname(queryFile);
// TODO this assumes that the qlpack.yml is in the same directory as the query file, but in reality,
// the file could be in a parent directory.
const targetQueryFileName = path.join(queryPackDir, path.basename(queryFile));
// the server is expecting the query file to be named `query.ql`. Rename it here.
const renamedQueryFile = path.join(queryPackDir, 'query.ql');
const originalPackRoot = await findPackRoot(queryFile);
const packRelativePath = path.relative(originalPackRoot, queryFile);
const targetQueryFileName = path.join(queryPackDir, packRelativePath);
let language: string | undefined;
if (await fs.pathExists(path.join(originalPackRoot, 'qlpack.yml'))) {
@@ -125,9 +144,6 @@ async function generateQueryPack(cliServer: cli.CodeQLCliServer, queryFile: stri
})
});
// ensure the qlpack.yml has a valid name
await ensureQueryPackName(queryPackDir);
void logger.log(`Copied ${copiedCount} files to ${queryPackDir}`);
language = await findLanguage(cliServer, Uri.file(targetQueryFileName));
@@ -138,13 +154,11 @@ async function generateQueryPack(cliServer: cli.CodeQLCliServer, queryFile: stri
// copy only the query file to the query pack directory
// and generate a synthetic query pack
// TODO this has a limitation that query packs inside of a workspace will not resolve its peer dependencies.
// Something to work on later. For now, we will only support query packs that are not in a workspace.
void logger.log(`Copying ${queryFile} to ${queryPackDir}`);
await fs.copy(queryFile, targetQueryFileName);
void logger.log('Generating synthetic query pack');
const syntheticQueryPack = {
name: 'codeql-remote/query',
name: QUERY_PACK_NAME,
version: '0.0.0',
dependencies: {
[`codeql/${language}-all`]: '*',
@@ -156,7 +170,7 @@ async function generateQueryPack(cliServer: cli.CodeQLCliServer, queryFile: stri
throw new UserCancellationException('Could not determine language.');
}
await fs.rename(targetQueryFileName, renamedQueryFile);
await ensureNameAndSuite(queryPackDir, packRelativePath);
const bundlePath = await getPackedBundlePath(queryPackDir);
void logger.log(`Compiling and bundling query pack from ${queryPackDir} to ${bundlePath}. (This may take a while.)`);
@@ -170,23 +184,24 @@ async function generateQueryPack(cliServer: cli.CodeQLCliServer, queryFile: stri
};
}
/**
* Ensure that the qlpack.yml has a valid name. For local purposes,
* Anonymous packs and names that are not prefixed by a scope (ie `<foo>/`)
* are sufficient. But in order to create a pack, the name must be prefixed.
*
* @param queryPackDir the directory containing the query pack.
*/
async function ensureQueryPackName(queryPackDir: string) {
const pack = yaml.safeLoad(await fs.readFile(path.join(queryPackDir, 'qlpack.yml'), 'utf8')) as { name: string; };
if (!pack.name || !pack.name.includes('/')) {
if (!pack.name) {
pack.name = 'codeql-remote/query';
} else if (!pack.name.includes('/')) {
pack.name = `codeql-remote/${pack.name}`;
async function findPackRoot(queryFile: string): Promise<string> {
// recursively find the directory containing qlpack.yml
let dir = path.dirname(queryFile);
while (!(await fs.pathExists(path.join(dir, 'qlpack.yml')))) {
dir = path.dirname(dir);
if (isFileSystemRoot(dir)) {
// there is no qlpack.yml in this direcory or any parent directory.
// just use the query file's directory as the pack root.
return path.dirname(queryFile);
}
await fs.writeFile(path.join(queryPackDir, 'qlpack.yml'), yaml.safeDump(pack));
}
return dir;
}
function isFileSystemRoot(dir: string): boolean {
const pathObj = path.parse(dir);
return pathObj.root === dir && pathObj.base === '';
}
async function createRemoteQueriesTempDirectory() {
@@ -211,7 +226,7 @@ export async function runRemoteQuery(
dryRun: boolean,
progress: ProgressCallback,
token: CancellationToken
): Promise<void | string> {
): Promise<void | RemoteQuerySubmissionResult> {
if (!(await cliServer.cliConstraints.supportsRemoteQueries())) {
throw new Error(`Remote queries are not supported by this version of CodeQL. Please upgrade to v${cli.CliVersionConstraint.CLI_VERSION_REMOTE_QUERIES
} or later.`);
@@ -315,13 +330,21 @@ export async function runRemoteQuery(
message: 'Sending request'
});
await runRemoteQueriesApiRequest(credentials, ref, language, repositories, owner, repo, base64Pack, dryRun);
const workflowRunId = await runRemoteQueriesApiRequest(credentials, ref, language, repositories, owner, repo, base64Pack, dryRun);
const queryStartTime = new Date();
const queryMetadata = await tryGetQueryMetadata(cliServer, queryFile);
if (dryRun) {
return remoteQueryDir.path;
return { queryDirPath: remoteQueryDir.path };
} else {
if (!workflowRunId) {
return;
}
const remoteQuery = await buildRemoteQueryEntity(repositories, queryFile, queryMetadata, owner, repo, queryStartTime, workflowRunId);
// don't return the path because it has been deleted
return;
return { query: remoteQuery };
}
} finally {
@@ -343,8 +366,7 @@ async function runRemoteQueriesApiRequest(
repo: string,
queryPackBase64: string,
dryRun = false
): Promise<void> {
): Promise<void | number> {
if (dryRun) {
void showAndLogInformationMessage('[DRY RUN] Would have sent request. See extension log for the payload.');
void logger.log(JSON.stringify({ ref, language, repositories, owner, repo, queryPackBase64: queryPackBase64.substring(0, 100) + '... ' + queryPackBase64.length + ' bytes' }));
@@ -366,10 +388,11 @@ async function runRemoteQueriesApiRequest(
}
}
);
void showAndLogInformationMessage(`Successfully scheduled runs. [Click here to see the progress](https://github.com/${owner}/${repo}/actions/runs/${response.data.workflow_run_id}).`);
const workflowRunId = response.data.workflow_run_id;
void showAndLogInformationMessage(`Successfully scheduled runs. [Click here to see the progress](https://github.com/${owner}/${repo}/actions/runs/${workflowRunId}).`);
return workflowRunId;
} catch (error) {
await attemptRerun(error, credentials, ref, language, repositories, owner, repo, queryPackBase64, dryRun);
return await attemptRerun(error, credentials, ref, language, repositories, owner, repo, queryPackBase64, dryRun);
}
}
@@ -407,9 +430,66 @@ export async function attemptRerun(
if (rerunQuery) {
const validRepositories = repositories.filter(r => !invalidRepos.includes(r) && !reposWithoutDbUploads.includes(r));
void logger.log(`Rerunning query on set of valid repositories: ${JSON.stringify(validRepositories)}`);
await runRemoteQueriesApiRequest(credentials, ref, language, validRepositories, owner, repo, queryPackBase64, dryRun);
return await runRemoteQueriesApiRequest(credentials, ref, language, validRepositories, owner, repo, queryPackBase64, dryRun);
}
} else {
void showAndLogErrorMessage(error);
}
}
/**
* Updates the default suite of the query pack. This is used to ensure
* only the specified query is run.
*
* Also, ensure the query pack name is set to the name expected by the server.
*
* @param queryPackDir The directory containing the query pack
* @param packRelativePath The relative path to the query pack from the root of the query pack
*/
async function ensureNameAndSuite(queryPackDir: string, packRelativePath: string): Promise<void> {
const packPath = path.join(queryPackDir, 'qlpack.yml');
const qlpack = yaml.safeLoad(await fs.readFile(packPath, 'utf8')) as QlPack;
delete qlpack.defaultSuiteFile;
qlpack.name = QUERY_PACK_NAME;
qlpack.defaultSuite = [{
description: 'Query suite for remote query'
}, {
query: packRelativePath.replace(/\\/g, '/')
}];
await fs.writeFile(packPath, yaml.safeDump(qlpack));
}
async function buildRemoteQueryEntity(
repositories: string[],
queryFilePath: string,
queryMetadata: QueryMetadata | undefined,
controllerRepoOwner: string,
controllerRepoName: string,
queryStartTime: Date,
workflowRunId: number
): Promise<RemoteQuery> {
// The query name is either the name as specified in the query metadata, or the file name.
const queryName = queryMetadata?.name ?? path.basename(queryFilePath);
const queryRepos = repositories.map(r => {
const [owner, repo] = r.split('/');
return { owner: owner, name: repo };
});
const queryText = await fs.readFile(queryFilePath, 'utf8');
return {
queryName,
queryFilePath,
queryText,
controllerRepository: {
owner: controllerRepoOwner,
name: controllerRepoName,
},
repositories: queryRepos,
executionStartTime: queryStartTime,
actionsWorkflowRunId: workflowRunId
};
}

View File

@@ -0,0 +1,86 @@
import { RemoteQuery } from './remote-query';
import { RemoteQueryResult } from './remote-query-result';
export const sampleRemoteQuery: RemoteQuery = {
queryName: 'Inefficient regular expression',
queryFilePath: '/Users/foo/dev/vscode-codeql-starter/ql/javascript/ql/src/Performance/ReDoS.ql',
queryText: '/**\n * @name Inefficient regular expression\n * @description A regular expression that requires exponential time to match certain inputs\n * can be a performance bottleneck, and may be vulnerable to denial-of-service\n * attacks.\n * @kind problem\n * @problem.severity error\n * @security-severity 7.5\n * @precision high\n * @id js/redos\n * @tags security\n * external/cwe/cwe-1333\n * external/cwe/cwe-730\n * external/cwe/cwe-400\n */\n\nimport javascript\nimport semmle.javascript.security.performance.ReDoSUtil\nimport semmle.javascript.security.performance.ExponentialBackTracking\n\nfrom RegExpTerm t, string pump, State s, string prefixMsg\nwhere hasReDoSResult(t, pump, s, prefixMsg)\nselect t,\n "This part of the regular expression may cause exponential backtracking on strings " + prefixMsg +\n "containing many repetitions of \'" + pump + "\'."\n',
controllerRepository: {
owner: 'big-corp',
name: 'controller-repo'
},
repositories: [
{
owner: 'big-corp',
name: 'repo1'
},
{
owner: 'big-corp',
name: 'repo2'
},
{
owner: 'big-corp',
name: 'repo3'
},
{
owner: 'big-corp',
name: 'repo4'
},
{
owner: 'big-corp',
name: 'repo5'
}
],
executionStartTime: new Date('2022-01-06T17:02:15.026Z'),
actionsWorkflowRunId: 1662757118
};
export const sampleRemoteQueryResult: RemoteQueryResult = {
executionEndTime: new Date('2022-01-06T17:04:37.026Z'),
allResultsDownloadLink: {
id: '137697018',
urlPath: '/repos/big-corp/controller-repo/actions/artifacts/137697018'
},
analysisSummaries: [
{
nwo: 'big-corp/repo1',
resultCount: 85,
fileSizeInBytes: 14123,
downloadLink: {
id: '137697017',
urlPath: '/repos/big-corp/controller-repo/actions/artifacts/137697017',
innerFilePath: 'results.sarif'
}
},
{
nwo: 'big-corp/repo2',
resultCount: 20,
fileSizeInBytes: 8698,
downloadLink: {
id: '137697018',
urlPath: '/repos/big-corp/controller-repo/actions/artifacts/137697018',
innerFilePath: 'results.sarif'
}
},
{
nwo: 'big-corp/repo3',
resultCount: 8,
fileSizeInBytes: 4123,
downloadLink: {
id: '137697019',
urlPath: '/repos/big-corp/controller-repo/actions/artifacts/137697019',
innerFilePath: 'results.sarif'
}
},
{
nwo: 'big-corp/repo4',
resultCount: 3,
fileSizeInBytes: 3313,
downloadLink: {
id: '137697020',
urlPath: '/repos/big-corp/controller-repo/actions/artifacts/137697020',
innerFilePath: 'results.sarif'
}
}
]
};

View File

@@ -0,0 +1,8 @@
export interface AnalysisResults {
nwo: string;
results: QueryResult[];
}
export interface QueryResult {
message?: string;
}

View File

@@ -0,0 +1,22 @@
import { DownloadLink } from '../download-link';
export interface RemoteQueryResult {
queryTitle: string;
queryFileName: string;
queryFilePath: string;
queryText: string;
totalRepositoryCount: number;
affectedRepositoryCount: number;
totalResultCount: number;
executionTimestamp: string;
executionDuration: string;
downloadLink: DownloadLink;
analysisSummaries: AnalysisSummary[]
}
export interface AnalysisSummary {
nwo: string,
resultCount: number,
downloadLink: DownloadLink,
fileSize: string,
}

View File

@@ -0,0 +1,13 @@
module.exports = {
env: {
browser: true
},
extends: [
"plugin:react/recommended"
],
settings: {
react: {
version: 'detect'
}
}
}

View File

@@ -0,0 +1,9 @@
import * as React from 'react';
const Badge = ({ text }: { text: string }) => (
<span className="vscode-codeql__badge-container">
<span className="vscode-codeql__badge">{text}</span>
</span>
);
export default Badge;

View File

@@ -0,0 +1,11 @@
import * as React from 'react';
import * as octicons from '../../view/octicons';
const DownloadButton = ({ text, onClick }: { text: string, onClick: () => void }) => (
<a className="vscode-codeql__download-button"
onClick={onClick}>
{octicons.download}{text}
</a>
);
export default DownloadButton;

View File

@@ -0,0 +1,223 @@
import * as React from 'react';
import { useEffect, useState } from 'react';
import * as Rdom from 'react-dom';
import { ToRemoteQueriesMessage } from '../../pure/interface-types';
import { AnalysisSummary, RemoteQueryResult } from '../shared/remote-query-result';
import * as octicons from '../../view/octicons';
import { vscode } from '../../view/vscode-api';
import SectionTitle from './SectionTitle';
import VerticalSpace from './VerticalSpace';
import Badge from './Badge';
import ViewTitle from './ViewTitle';
import DownloadButton from './DownloadButton';
import { AnalysisResults } from '../shared/analysis-result';
const numOfReposInContractedMode = 10;
const emptyQueryResult: RemoteQueryResult = {
queryTitle: '',
queryFileName: '',
queryFilePath: '',
queryText: '',
totalRepositoryCount: 0,
affectedRepositoryCount: 0,
totalResultCount: 0,
executionTimestamp: '',
executionDuration: '',
downloadLink: {
id: '',
urlPath: '',
},
analysisSummaries: []
};
const downloadAnalysisResults = (analysisSummary: AnalysisSummary) => {
vscode.postMessage({
t: 'remoteQueryDownloadAnalysisResults',
analysisSummary
});
};
const downloadAllAnalysesResults = (query: RemoteQueryResult) => {
vscode.postMessage({
t: 'remoteQueryDownloadAllAnalysesResults',
analysisSummaries: query.analysisSummaries
});
};
const openQueryFile = (queryResult: RemoteQueryResult) => {
vscode.postMessage({
t: 'openFile',
filePath: queryResult.queryFilePath
});
};
const openQueryTextVirtualFile = (queryResult: RemoteQueryResult) => {
vscode.postMessage({
t: 'openVirtualFile',
queryText: queryResult.queryText
});
};
const QueryInfo = (queryResult: RemoteQueryResult) => (
<>
<VerticalSpace />
{queryResult.totalResultCount} results in {queryResult.totalRepositoryCount} repositories
({queryResult.executionDuration}), {queryResult.executionTimestamp}
<VerticalSpace />
<span className="vscode-codeql__query-file">{octicons.file}
<a className="vscode-codeql__query-file-link" href="#" onClick={() => openQueryFile(queryResult)}>
{queryResult.queryFileName}
</a>
</span>
<span>{octicons.codeSquare}
<a className="vscode-codeql__query-file-link" href="#" onClick={() => openQueryTextVirtualFile(queryResult)}>
query
</a>
</span>
</>
);
const SummaryTitleWithResults = (queryResult: RemoteQueryResult) => (
<div className="vscode-codeql__query-summary-container">
<SectionTitle text={`Repositories with results (${queryResult.affectedRepositoryCount}):`} />
<DownloadButton
text="Download all"
onClick={() => downloadAllAnalysesResults(queryResult)} />
</div>
);
const SummaryTitleNoResults = () => (
<div className="vscode-codeql__query-summary-container">
<SectionTitle text="No results found" />
</div>
);
const SummaryItem = (props: AnalysisSummary) => (
<span>
<span className="vscode-codeql__analysis-item">{octicons.repo}</span>
<span className="vscode-codeql__analysis-item">{props.nwo}</span>
<span className="vscode-codeql__analysis-item"><Badge text={props.resultCount.toString()} /></span>
<span className="vscode-codeql__analysis-item">
<DownloadButton
text={props.fileSize}
onClick={() => downloadAnalysisResults(props)} />
</span>
</span>
);
const Summary = (queryResult: RemoteQueryResult) => {
const [repoListExpanded, setRepoListExpanded] = useState(false);
const numOfReposToShow = repoListExpanded ? queryResult.analysisSummaries.length : numOfReposInContractedMode;
return (
<>
{
queryResult.affectedRepositoryCount === 0
? <SummaryTitleNoResults />
: <SummaryTitleWithResults {...queryResult} />
}
<ul className="vscode-codeql__analysis-summaries-list">
{queryResult.analysisSummaries.slice(0, numOfReposToShow).map((summary, i) =>
<li key={summary.nwo} className="vscode-codeql__analysis-summaries-list-item">
<SummaryItem {...summary} />
</li>
)}
</ul>
{
queryResult.analysisSummaries.length > numOfReposInContractedMode &&
<button className="vscode-codeql__expand-button" onClick={() => setRepoListExpanded(!repoListExpanded)}>
{repoListExpanded ? (<span>View less</span>) : (<span>View all</span>)}
</button>
}
</>
);
};
const AnalysesResultsTitle = ({ totalAnalysesResults, totalResults }: { totalAnalysesResults: number, totalResults: number }) => {
if (totalAnalysesResults === totalResults) {
return <SectionTitle text={`${totalAnalysesResults} results`} />;
}
return <SectionTitle text={`${totalAnalysesResults}/${totalResults} results`} />;
};
const AnalysesResultsDescription = ({ totalAnalysesResults, totalResults }: { totalAnalysesResults: number, totalResults: number }) => {
if (totalAnalysesResults < totalResults) {
return <>
<VerticalSpace />
Some results haven&apos;t been downloaded automatically because of their size or because enough were downloaded already.
Download them manually from the list above if you want to see them here.
</>;
}
return <></>;
};
const AnalysesResults = ({ analysesResults, totalResults }: { analysesResults: AnalysisResults[], totalResults: number }) => {
const totalAnalysesResults = analysesResults.reduce((acc, curr) => acc + curr.results.length, 0);
if (totalResults === 0) {
return <></>;
}
return (
<>
<VerticalSpace />
<VerticalSpace />
<AnalysesResultsTitle
totalAnalysesResults={totalAnalysesResults}
totalResults={totalResults} />
<AnalysesResultsDescription
totalAnalysesResults={totalAnalysesResults}
totalResults={totalResults} />
</>
);
};
export function RemoteQueries(): JSX.Element {
const [queryResult, setQueryResult] = useState<RemoteQueryResult>(emptyQueryResult);
const [analysesResults, setAnalysesResults] = useState<AnalysisResults[]>([]);
useEffect(() => {
window.addEventListener('message', (evt: MessageEvent) => {
if (evt.origin === window.origin) {
const msg: ToRemoteQueriesMessage = evt.data;
if (msg.t === 'setRemoteQueryResult') {
setQueryResult(msg.queryResult);
} else if (msg.t === 'setAnalysesResults') {
setAnalysesResults(msg.analysesResults);
}
} else {
// sanitize origin
const origin = evt.origin.replace(/\n|\r/g, '');
console.error(`Invalid event origin ${origin}`);
}
});
});
if (!queryResult) {
return <div>Waiting for results to load.</div>;
}
try {
return <div>
<ViewTitle title={queryResult.queryTitle} />
<QueryInfo {...queryResult} />
<Summary {...queryResult} />
<AnalysesResults analysesResults={analysesResults} totalResults={queryResult.totalResultCount} />
</div>;
} catch (err) {
console.error(err);
return <div>There was an error displaying the view.</div>;
}
}
Rdom.render(
<RemoteQueries />,
document.getElementById('root'),
// Post a message to the extension when fully loaded.
() => vscode.postMessage({ t: 'remoteQueryLoaded' })
);

View File

@@ -0,0 +1,7 @@
import * as React from 'react';
const SectionTitle = ({ text }: { text: string }) => (
<h2 className="vscode-codeql__section-title">{text}</h2>
);
export default SectionTitle;

View File

@@ -0,0 +1,7 @@
import * as React from 'react';
const VerticalSpace = () => (
<div className="vscode-codeql__vertical-space" />
);
export default VerticalSpace;

View File

@@ -0,0 +1,7 @@
import * as React from 'react';
const ViewTitle = ({ title }: { title: string }) => (
<h1 className="vscode-codeql__view-title">{title}</h1>
);
export default ViewTitle;

View File

@@ -0,0 +1,75 @@
body {
font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial,
sans-serif, Apple Color Emoji, Segoe UI Emoji;
}
/* -------------------------------------------------------------------------- */
/* SectionTitle component */
/* -------------------------------------------------------------------------- */
.vscode-codeql__section-title {
font-size: medium;
font-weight: 500;
padding: 0 0.5em 0 0;
margin: 0;
display: inline-block;
vertical-align: middle;
}
/* -------------------------------------------------------------------------- */
/* ViewTitle component */
/* -------------------------------------------------------------------------- */
.vscode-codeql__view-title {
font-size: large;
margin-bottom: 0.5em;
font-weight: 500;
}
/* -------------------------------------------------------------------------- */
/* VerticalSpace component */
/* -------------------------------------------------------------------------- */
.vscode-codeql__vertical-space {
flex: 0 0 auto;
height: 0.5rem;
}
/* -------------------------------------------------------------------------- */
/* Badge component */
/* -------------------------------------------------------------------------- */
.vscode-codeql__badge-container {
justify-content: center;
align-items: center;
min-height: 100vh;
padding-left: 0.2em;
}
.vscode-codeql__badge {
display: inline-block;
min-width: 1.5em;
padding: 0.3em;
border-radius: 35%;
font-size: x-small;
text-align: center;
background: var(--vscode-badge-background);
color: var(--vscode-badge-foreground);
border-color: var(--vscode-badge-background);
}
/* -------------------------------------------------------------------------- */
/* DownloadButton component */
/* -------------------------------------------------------------------------- */
.vscode-codeql__download-button {
display: inline-block;
font-size: x-small;
text-decoration: none;
cursor: pointer;
vertical-align: middle;
}
.vscode-codeql__download-button svg {
fill: var(--vscode-textLink-foreground);
}

View File

@@ -0,0 +1,52 @@
.octicon {
fill: var(--vscode-editor-foreground);
height: 1.2em;
width: 1.2em;
vertical-align: middle;
display: inline-block;
}
.octicon-light {
opacity: 0.6;
}
.vscode-codeql__query-file {
padding-right: 1em;
}
.vscode-codeql__query-file-link {
text-decoration: none;
padding-left: 0.3em;
color: var(--vscode-editor-foreground);
}
.vscode-codeql__query-file-link:hover {
color: var(--vscode-editor-foreground);
}
.vscode-codeql__query-summary-container {
padding-top: 1.5em;
}
.vscode-codeql__analysis-summaries-list {
list-style-type: none;
margin: 0;
padding: 0.5em 0 0 0;
}
.vscode-codeql__analysis-summaries-list-item {
margin-top: 0.5em;
}
.vscode-codeql__analysis-item {
padding-right: 0.1em;
}
.vscode-codeql__expand-button {
background: none;
color: var(--vscode-textLink-foreground);
border: none;
cursor: pointer;
padding-top: 1em;
font-size: x-small;
}

View File

@@ -0,0 +1,18 @@
{
"compilerOptions": {
"module": "esnext",
"moduleResolution": "node",
"target": "es6",
"outDir": "out",
"lib": ["es6", "dom"],
"jsx": "react",
"sourceMap": true,
"rootDir": "..",
"strict": true,
"noUnusedLocals": true,
"noImplicitReturns": true,
"noFallthroughCasesInSwitch": true,
"experimentalDecorators": true
},
"exclude": ["node_modules"]
}

View File

@@ -5,17 +5,19 @@ import * as tmp from 'tmp-promise';
import {
CancellationToken,
ConfigurationTarget,
Range,
TextDocument,
TextEditor,
Uri,
window
window,
workspace
} from 'vscode';
import { ErrorCodes, ResponseError } from 'vscode-languageclient';
import * as cli from './cli';
import * as config from './config';
import { DatabaseItem } from './databases';
import { getOnDiskWorkspaceFolders, showAndLogErrorMessage } from './helpers';
import { getOnDiskWorkspaceFolders, showAndLogErrorMessage, tryGetQueryMetadata } from './helpers';
import { ProgressCallback, UserCancellationException } from './commandRunner';
import { DatabaseInfo, QueryMetadata, ResultsPaths } from './pure/interface-types';
import { logger } from './logging';
@@ -86,6 +88,7 @@ export class QueryInfo {
async run(
qs: qsClient.QueryServerClient,
upgradeQlo: string | undefined,
availableMlModels: cli.MlModelInfo[],
progress: ProgressCallback,
token: CancellationToken,
): Promise<messages.EvaluationResult> {
@@ -93,12 +96,15 @@ export class QueryInfo {
const callbackId = qs.registerCallback(res => { result = res; });
const availableMlModelUris: messages.MlModel[] = availableMlModels.map(model => ({ uri: Uri.file(model.path).toString(true) }));
const queryToRun: messages.QueryToRun = {
resultsPath: this.resultsPaths.resultsPath,
qlo: Uri.file(this.compiledQueryPath).toString(),
compiledUpgrade: upgradeQlo && Uri.file(upgradeQlo).toString(),
allowUnknownTemplates: true,
templateValues: this.templates,
availableMlModels: availableMlModelUris,
id: callbackId,
timeoutSecs: qs.config.timeoutSecs,
};
@@ -327,9 +333,10 @@ async function convertToQlPath(filePath: string): Promise<string> {
/** Gets the selected position within the given editor. */
async function getSelectedPosition(editor: TextEditor): Promise<messages.Position> {
const pos = editor.selection.start;
const posEnd = editor.selection.end;
async function getSelectedPosition(editor: TextEditor, range?: Range): Promise<messages.Position> {
const selectedRange = range || editor.selection;
const pos = selectedRange.start;
const posEnd = selectedRange.end;
// Convert from 0-based to 1-based line and column numbers.
return {
fileName: await convertToQlPath(editor.document.fileName),
@@ -485,7 +492,7 @@ type SelectedQuery = {
* @param selectedResourceUri The selected resource when the command was run.
* @param quickEval Whether the command being run is `Quick Evaluation`.
*/
export async function determineSelectedQuery(selectedResourceUri: Uri | undefined, quickEval: boolean): Promise<SelectedQuery> {
export async function determineSelectedQuery(selectedResourceUri: Uri | undefined, quickEval: boolean, range?: Range): Promise<SelectedQuery> {
const editor = window.activeTextEditor;
// Choose which QL file to use.
@@ -539,7 +546,7 @@ export async function determineSelectedQuery(selectedResourceUri: Uri | undefine
// Report an error if we end up in this (hopefully unlikely) situation.
throw new Error('The selected resource for quick evaluation should match the active editor.');
}
quickEvalPosition = await getSelectedPosition(editor);
quickEvalPosition = await getSelectedPosition(editor, range);
quickEvalText = editor.document.getText(editor.selection);
}
@@ -555,13 +562,14 @@ export async function compileAndRunQueryAgainstDatabase(
progress: ProgressCallback,
token: CancellationToken,
templates?: messages.TemplateDefinitions,
range?: Range
): Promise<QueryWithResults> {
if (!db.contents || !db.contents.dbSchemeUri) {
throw new Error(`Database ${db.databaseUri} does not have a CodeQL database scheme.`);
}
// Determine which query to run, based on the selection and the active editor.
const { queryPath, quickEvalPosition, quickEvalText } = await determineSelectedQuery(selectedQueryUri, quickEval);
const { queryPath, quickEvalPosition, quickEvalText } = await determineSelectedQuery(selectedQueryUri, quickEval, range);
const historyItemOptions: QueryHistoryItemOptions = {};
historyItemOptions.isQuickQuery === isQuickQueryPath(queryPath);
@@ -604,12 +612,24 @@ export async function compileAndRunQueryAgainstDatabase(
};
// Read the query metadata if possible, to use in the UI.
let metadata: QueryMetadata | undefined;
try {
metadata = await cliServer.resolveMetadata(qlProgram.queryPath);
} catch (e) {
// Ignore errors and provide no metadata.
void logger.log(`Couldn't resolve metadata for ${qlProgram.queryPath}: ${e}`);
const metadata = await tryGetQueryMetadata(cliServer, qlProgram.queryPath);
let availableMlModels: cli.MlModelInfo[] = [];
// The `capabilities.untrustedWorkspaces.restrictedConfigurations` entry in package.json doesn't
// work with hidden settings, so we manually check that the workspace is trusted before looking at
// whether the `shouldInsecurelyLoadMlModelsFromPacks` setting is enabled.
if (workspace.isTrusted &&
config.isCanary() &&
config.shouldInsecurelyLoadMlModelsFromPacks() &&
await cliServer.cliConstraints.supportsResolveMlModels()) {
try {
availableMlModels = (await cliServer.resolveMlModels(diskWorkspaceFolders)).models;
void logger.log(`Found available ML models at the following paths: ${availableMlModels.map(x => `'${x.path}'`).join(', ')}.`);
} catch (e) {
const message = `Couldn't resolve available ML models for ${qlProgram.queryPath}. Running the ` +
`query without any ML models: ${e}.`;
void showAndLogErrorMessage(message);
}
}
const query = new QueryInfo(qlProgram, db, packConfig.dbscheme, quickEvalPosition, metadata, templates);
@@ -634,7 +654,7 @@ export async function compileAndRunQueryAgainstDatabase(
}
if (errors.length === 0) {
const result = await query.run(qs, upgradeQlo, progress, token);
const result = await query.run(qs, upgradeQlo, availableMlModels, progress, token);
if (result.resultType !== messages.QueryResultType.SUCCESS) {
const message = result.message || 'Failed to run query';
void logger.log(message);

View File

@@ -20,3 +20,23 @@ export const listUnordered = <svg className="octicon octicon-light" width="16" h
export const info = <svg className="octicon octicon-light" width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg" >
<path fillRule="evenodd" clipRule="evenodd" d="M8.568 1.03a6.8 6.8 0 0 1 4.192 2.02 7.06 7.06 0 0 1 .46 9.39 6.85 6.85 0 0 1-8.58 1.74 7 7 0 0 1-3.12-3.5 7.12 7.12 0 0 1-.23-4.71 7 7 0 0 1 2.77-3.79 6.8 6.8 0 0 1 4.508-1.15zm.472 12.85a5.89 5.89 0 0 0 3.41-2.07 6.07 6.07 0 0 0-.4-8.06 5.82 5.82 0 0 0-7.43-.74 6.06 6.06 0 0 0 .5 10.29 5.81 5.81 0 0 0 3.92.58zM8.51 7h-1v4h1V7zm0-2h-1v1h1V5z" />
</svg>;
/**
* The icons below come from https://primer.style/octicons/
*/
export const file = <svg className="octicon octicon-light" width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fillRule="evenodd" d="M3.75 1.5a.25.25 0 00-.25.25v11.5c0 .138.112.25.25.25h8.5a.25.25 0 00.25-.25V6H9.75A1.75 1.75 0 018 4.25V1.5H3.75zm5.75.56v2.19c0 .138.112.25.25.25h2.19L9.5 2.06zM2 1.75C2 .784 2.784 0 3.75 0h5.086c.464 0 .909.184 1.237.513l3.414 3.414c.329.328.513.773.513 1.237v8.086A1.75 1.75 0 0112.25 15h-8.5A1.75 1.75 0 012 13.25V1.75z"></path>
</svg>;
export const codeSquare = <svg className="octicon octicon-light" width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fillRule="evenodd" d="M1.75 1.5a.25.25 0 00-.25.25v12.5c0 .138.112.25.25.25h12.5a.25.25 0 00.25-.25V1.75a.25.25 0 00-.25-.25H1.75zM0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v12.5A1.75 1.75 0 0114.25 16H1.75A1.75 1.75 0 010 14.25V1.75zm9.22 3.72a.75.75 0 000 1.06L10.69 8 9.22 9.47a.75.75 0 101.06 1.06l2-2a.75.75 0 000-1.06l-2-2a.75.75 0 00-1.06 0zM6.78 6.53a.75.75 0 00-1.06-1.06l-2 2a.75.75 0 000 1.06l2 2a.75.75 0 101.06-1.06L5.31 8l1.47-1.47z"></path>
</svg>;
export const repo = <svg className="octicon octicon-light" width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fillRule="evenodd" d="M2 2.5A2.5 2.5 0 014.5 0h8.75a.75.75 0 01.75.75v12.5a.75.75 0 01-.75.75h-2.5a.75.75 0 110-1.5h1.75v-2h-8a1 1 0 00-.714 1.7.75.75 0 01-1.072 1.05A2.495 2.495 0 012 11.5v-9zm10.5-1V9h-8c-.356 0-.694.074-1 .208V2.5a1 1 0 011-1h8zM5 12.25v3.25a.25.25 0 00.4.2l1.45-1.087a.25.25 0 01.3 0L8.6 15.7a.25.25 0 00.4-.2v-3.25a.25.25 0 00-.25-.25h-3.5a.25.25 0 00-.25.25z"></path>
</svg>;
export const download = <svg className="octicon octicon-light" width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fillRule="evenodd" d="M7.47 10.78a.75.75 0 001.06 0l3.75-3.75a.75.75 0 00-1.06-1.06L8.75 8.44V1.75a.75.75 0 00-1.5 0v6.69L4.78 5.97a.75.75 0 00-1.06 1.06l3.75 3.75zM3.75 13a.75.75 0 000 1.5h8.5a.75.75 0 000-1.5h-8.5z"></path>
</svg>;

View File

@@ -106,7 +106,9 @@ export class ResultTables
}
private getResultSetNames(): string[] {
return this.props.parsedResultSets.resultSetNames.concat([ALERTS_TABLE_NAME]);
return this.props.interpretation
? this.props.parsedResultSets.resultSetNames.concat([ALERTS_TABLE_NAME])
: this.props.parsedResultSets.resultSetNames;
}
constructor(props: ResultTablesProps) {

View File

@@ -1,10 +1,10 @@
import { FromCompareViewMessage, FromResultsViewMsg } from '../pure/interface-types';
import { FromCompareViewMessage, FromRemoteQueriesMessage, FromResultsViewMsg } from '../pure/interface-types';
export interface VsCodeApi {
/**
* Post message back to vscode extension.
*/
postMessage(msg: FromResultsViewMsg | FromCompareViewMessage): void;
postMessage(msg: FromResultsViewMsg | FromCompareViewMessage | FromRemoteQueriesMessage): void;
}
declare const acquireVsCodeApi: () => VsCodeApi;

View File

@@ -0,0 +1,4 @@
name: foo/bar
version: 0.0.0
dependencies:
foo/baz: '*'

View File

@@ -0,0 +1,2 @@
// This file should not be included the remote query pack.
select 1

View File

@@ -0,0 +1,3 @@
int number() {
result = 1
}

View File

@@ -0,0 +1,4 @@
name: github/remote-query-pack
version: 0.0.0
dependencies:
codeql/javascript-all: '*'

View File

@@ -0,0 +1,4 @@
import javascript
import otherfolder.lib
select number()

View File

@@ -1,5 +1,4 @@
name: github/remote-query-pack
version: 0.0.0
extractor: javascript
dependencies:
codeql/javascript-all: '*'

View File

@@ -1,3 +1,10 @@
/**
* @name This is the name
* @kind problem
* @problem.severity warning
* @id javascript/example/test-query
*/
import javascript
select 1

View File

@@ -0,0 +1,42 @@
import * as path from 'path';
import { extensions } from 'vscode';
import 'mocha';
import { CodeQLCliServer } from '../../cli';
import { CodeQLExtensionInterface } from '../../extension';
import { tryGetQueryMetadata } from '../../helpers';
import { expect } from 'chai';
describe('helpers (with CLI)', function() {
const baseDir = path.join(__dirname, '../../../src/vscode-tests/cli-integration');
// up to 3 minutes per test
this.timeout(3 * 60 * 1000);
let cli: CodeQLCliServer;
beforeEach(async () => {
const extension = await extensions.getExtension<CodeQLExtensionInterface | Record<string, never>>('GitHub.vscode-codeql')!.activate();
if ('cliServer' in extension) {
cli = extension.cliServer;
} else {
throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
}
});
it('should get query metadata when available', async () => {
// Query with metadata
const metadata = await tryGetQueryMetadata(cli, path.join(baseDir, 'data', 'simple-javascript-query.ql'));
expect(metadata!.name).to.equal('This is the name');
expect(metadata!.kind).to.equal('problem');
expect(metadata!.id).to.equal('javascript/example/test-query');
});
it('should handle query with no metadata', async () => {
// Query with empty metadata
const noMetadata = await tryGetQueryMetadata(cli, path.join(baseDir, 'data', 'simple-query.ql'));
expect(noMetadata).to.deep.equal({});
});
});

View File

@@ -0,0 +1,128 @@
import * as sinon from 'sinon';
import { extensions, window } from 'vscode';
import 'mocha';
import * as path from 'path';
import * as pq from 'proxyquire';
import { CliVersionConstraint, CodeQLCliServer } from '../../cli';
import { CodeQLExtensionInterface } from '../../extension';
import { expect } from 'chai';
const proxyquire = pq.noPreserveCache();
describe('Packaging commands', function() {
let sandbox: sinon.SinonSandbox;
// up to 3 minutes per test
this.timeout(3 * 60 * 1000);
let cli: CodeQLCliServer;
let progress: sinon.SinonSpy;
let quickPickSpy: sinon.SinonStub;
let inputBoxSpy: sinon.SinonStub;
let showAndLogErrorMessageSpy: sinon.SinonStub;
let showAndLogInformationMessageSpy: sinon.SinonStub;
let mod: any;
beforeEach(async function() {
sandbox = sinon.createSandbox();
const extension = await extensions
.getExtension<CodeQLExtensionInterface | Record<string, never>>(
'GitHub.vscode-codeql'
)!
.activate();
if ('cliServer' in extension) {
cli = extension.cliServer;
} else {
throw new Error(
'Extension not initialized. Make sure cli is downloaded and installed properly.'
);
}
if (!(await cli.cliConstraints.supportsPackaging())) {
console.log(`Packaging commands are not supported on CodeQL CLI v${CliVersionConstraint.CLI_VERSION_WITH_PACKAGING
}. Skipping this test.`);
this.skip();
}
progress = sandbox.spy();
quickPickSpy = sandbox.stub(window, 'showQuickPick');
inputBoxSpy = sandbox.stub(window, 'showInputBox');
showAndLogErrorMessageSpy = sandbox.stub();
showAndLogInformationMessageSpy = sandbox.stub();
mod = proxyquire('../../packaging', {
'./helpers': {
showAndLogErrorMessage: showAndLogErrorMessageSpy,
showAndLogInformationMessage: showAndLogInformationMessageSpy,
},
});
});
afterEach(() => {
sandbox.restore();
});
it('should download all core query packs', async () => {
quickPickSpy.resolves('Download all core query packs');
await mod.handleDownloadPacks(cli, progress);
expect(showAndLogInformationMessageSpy.firstCall.args[0]).to.contain(
'Finished downloading packs.'
);
});
it('should download valid user-specified pack', async () => {
quickPickSpy.resolves('Download custom specified pack');
inputBoxSpy.resolves('codeql/csharp-solorigate-queries');
await mod.handleDownloadPacks(cli, progress);
expect(showAndLogInformationMessageSpy.firstCall.args[0]).to.contain(
'Finished downloading packs.'
);
});
it('should show error when downloading invalid user-specified pack', async () => {
quickPickSpy.resolves('Download custom specified pack');
inputBoxSpy.resolves('foo/not-a-real-pack@0.0.1');
await mod.handleDownloadPacks(cli, progress);
expect(showAndLogErrorMessageSpy.firstCall.args[0]).to.contain(
'Unable to download all packs.'
);
});
it('should install valid workspace pack', async () => {
const rootDir = path.join(__dirname, '../../../src/vscode-tests/cli-integration/data');
quickPickSpy.resolves([
{
label: 'integration-test-queries-javascript',
packRootDir: [rootDir],
},
]);
await mod.handleInstallPackDependencies(cli, progress);
expect(showAndLogInformationMessageSpy.firstCall.args[0]).to.contain(
'Finished installing pack dependencies.'
);
});
it('should throw an error when installing invalid workspace pack', async () => {
const rootDir = path.join(__dirname, '../../../src/vscode-tests/cli-integration/data-invalid-pack');
quickPickSpy.resolves([
{
label: 'foo/bar',
packRootDir: [rootDir],
},
]);
try {
// expect this to throw an error
await mod.handleInstallPackDependencies(cli, progress);
// This line should not be reached
expect(true).to.be.false;
} catch (error) {
expect(error.message).to.contain('Unable to install pack dependencies');
}
});
});

View File

@@ -4,14 +4,16 @@ import * as sinon from 'sinon';
import { CancellationToken, extensions, QuickPickItem, Uri, window } from 'vscode';
import 'mocha';
import * as fs from 'fs-extra';
import * as os from 'os';
import * as yaml from 'js-yaml';
import { runRemoteQuery } from '../../run-remote-query';
import { QlPack, runRemoteQuery } from '../../remote-queries/run-remote-query';
import { Credentials } from '../../authentication';
import { CliVersionConstraint, CodeQLCliServer } from '../../cli';
import { CodeQLExtensionInterface } from '../../extension';
import { setRemoteControllerRepo, setRemoteRepositoryLists } from '../../config';
import { UserCancellationException } from '../../commandRunner';
import { lte } from 'semver';
describe('Remote queries', function() {
const baseDir = path.join(__dirname, '../../../src/vscode-tests/cli-integration');
@@ -66,7 +68,9 @@ describe('Remote queries', function() {
it('should run a remote query that is part of a qlpack', async () => {
const fileUri = getFile('data-remote-qlpack/in-pack.ql');
const queryPackRootDir = (await runRemoteQuery(cli, credentials, fileUri, true, progress, token))!;
const querySubmissionResult = await runRemoteQuery(cli, credentials, fileUri, true, progress, token);
expect(querySubmissionResult).to.be.ok;
const queryPackRootDir = querySubmissionResult!.queryDirPath!;
printDirectoryContents(queryPackRootDir);
// to retrieve the list of repositories
@@ -80,8 +84,7 @@ describe('Remote queries', function() {
const queryPackDir = path.join(queryPackRootDir, 'query-pack');
printDirectoryContents(queryPackDir);
// in-pack.ql renamed to query.ql
expect(fs.existsSync(path.join(queryPackDir, 'query.ql'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'in-pack.ql'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'lib.qll'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'qlpack.yml'))).to.be.true;
@@ -96,7 +99,7 @@ describe('Remote queries', function() {
const compiledPackDir = path.join(queryPackDir, '.codeql/pack/github/remote-query-pack/0.0.0/');
printDirectoryContents(compiledPackDir);
expect(fs.existsSync(path.join(compiledPackDir, 'query.ql'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'in-pack.ql'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'lib.qll'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'qlpack.yml'))).to.be.true;
// depending on the cli version, we should have one of these files
@@ -105,17 +108,24 @@ describe('Remote queries', function() {
fs.existsSync(path.join(compiledPackDir, 'codeql-pack.lock.yml'))
).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'not-in-pack.ql'))).to.be.false;
verifyQlPack(path.join(compiledPackDir, 'qlpack.yml'), 'in-pack.ql', '0.0.0', await pathSerializationBroken());
// dependencies
const libraryDir = path.join(compiledPackDir, '.codeql/libraries/codeql');
const packNames = fs.readdirSync(libraryDir).sort();
expect(packNames).to.deep.equal(['javascript-all', 'javascript-upgrades']);
// check dependencies.
// 2.7.4 and earlier have ['javascript-all', 'javascript-upgrades']
// later only have ['javascript-all']. ensure this test can handle either
expect(packNames.length).to.be.lessThan(3).and.greaterThan(0);
expect(packNames[0]).to.deep.equal('javascript-all');
});
it('should run a remote query that is not part of a qlpack', async () => {
const fileUri = getFile('data-remote-no-qlpack/in-pack.ql');
const queryPackRootDir = (await runRemoteQuery(cli, credentials, fileUri, true, progress, token))!;
const querySubmissionResult = await runRemoteQuery(cli, credentials, fileUri, true, progress, token);
expect(querySubmissionResult).to.be.ok;
const queryPackRootDir = querySubmissionResult!.queryDirPath!;
// to retrieve the list of repositories
// and a second time to ask for the language
@@ -129,8 +139,8 @@ describe('Remote queries', function() {
const queryPackDir = path.join(queryPackRootDir, 'query-pack');
printDirectoryContents(queryPackDir);
// in-pack.ql renamed to query.ql
expect(fs.existsSync(path.join(queryPackDir, 'query.ql'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'in-pack.ql'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'qlpack.yml'))).to.be.true;
// depending on the cli version, we should have one of these files
expect(
@@ -143,8 +153,10 @@ describe('Remote queries', function() {
// the compiled pack
const compiledPackDir = path.join(queryPackDir, '.codeql/pack/codeql-remote/query/0.0.0/');
printDirectoryContents(compiledPackDir);
expect(fs.existsSync(path.join(compiledPackDir, 'query.ql'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'in-pack.ql'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'qlpack.yml'))).to.be.true;
verifyQlPack(path.join(compiledPackDir, 'qlpack.yml'), 'in-pack.ql', '0.0.0', await pathSerializationBroken());
// depending on the cli version, we should have one of these files
expect(
fs.existsSync(path.join(compiledPackDir, 'qlpack.lock.yml')) ||
@@ -158,11 +170,75 @@ describe('Remote queries', function() {
expect(qlpackContents.version).to.equal('0.0.0');
expect(qlpackContents.dependencies?.['codeql/javascript-all']).to.equal('*');
// dependencies
const libraryDir = path.join(compiledPackDir, '.codeql/libraries/codeql');
printDirectoryContents(libraryDir);
const packNames = fs.readdirSync(libraryDir).sort();
expect(packNames).to.deep.equal(['javascript-all', 'javascript-upgrades']);
// check dependencies.
// 2.7.4 and earlier have ['javascript-all', 'javascript-upgrades']
// later only have ['javascript-all']. ensure this test can handle either
expect(packNames.length).to.be.lessThan(3).and.greaterThan(0);
expect(packNames[0]).to.deep.equal('javascript-all');
});
it('should run a remote query that is nested inside a qlpack', async () => {
const fileUri = getFile('data-remote-qlpack-nested/subfolder/in-pack.ql');
const querySubmissionResult = await runRemoteQuery(cli, credentials, fileUri, true, progress, token);
expect(querySubmissionResult).to.be.ok;
const queryPackRootDir = querySubmissionResult!.queryDirPath!;
// to retrieve the list of repositories
expect(showQuickPickSpy).to.have.been.calledOnce;
// check a few files that we know should exist and others that we know should not
// the tarball to deliver to the server
printDirectoryContents(queryPackRootDir);
expect(fs.readdirSync(queryPackRootDir).find(f => f.startsWith('qlpack-') && f.endsWith('-generated.tgz'))).not.to.be.undefined;
const queryPackDir = path.join(queryPackRootDir, 'query-pack');
printDirectoryContents(queryPackDir);
expect(fs.existsSync(path.join(queryPackDir, 'subfolder/in-pack.ql'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'qlpack.yml'))).to.be.true;
// depending on the cli version, we should have one of these files
expect(
fs.existsSync(path.join(queryPackDir, 'qlpack.lock.yml')) ||
fs.existsSync(path.join(queryPackDir, 'codeql-pack.lock.yml'))
).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'otherfolder/lib.qll'))).to.be.true;
expect(fs.existsSync(path.join(queryPackDir, 'not-in-pack.ql'))).to.be.false;
// the compiled pack
const compiledPackDir = path.join(queryPackDir, '.codeql/pack/github/remote-query-pack/0.0.0/');
printDirectoryContents(compiledPackDir);
expect(fs.existsSync(path.join(compiledPackDir, 'otherfolder/lib.qll'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'subfolder/in-pack.ql'))).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'qlpack.yml'))).to.be.true;
verifyQlPack(path.join(compiledPackDir, 'qlpack.yml'), 'subfolder/in-pack.ql', '0.0.0', await pathSerializationBroken());
// depending on the cli version, we should have one of these files
expect(
fs.existsSync(path.join(compiledPackDir, 'qlpack.lock.yml')) ||
fs.existsSync(path.join(compiledPackDir, 'codeql-pack.lock.yml'))
).to.be.true;
expect(fs.existsSync(path.join(compiledPackDir, 'not-in-pack.ql'))).to.be.false;
// should have generated a correct qlpack file
const qlpackContents: any = yaml.safeLoad(fs.readFileSync(path.join(compiledPackDir, 'qlpack.yml'), 'utf8'));
expect(qlpackContents.name).to.equal('codeql-remote/query');
expect(qlpackContents.version).to.equal('0.0.0');
expect(qlpackContents.dependencies?.['codeql/javascript-all']).to.equal('*');
const libraryDir = path.join(compiledPackDir, '.codeql/libraries/codeql');
printDirectoryContents(libraryDir);
const packNames = fs.readdirSync(libraryDir).sort();
// check dependencies.
// 2.7.4 and earlier have ['javascript-all', 'javascript-upgrades']
// later only have ['javascript-all']. ensure this test can handle either
expect(packNames.length).to.be.lessThan(3).and.greaterThan(0);
expect(packNames[0]).to.deep.equal('javascript-all');
});
it('should cancel a run before uploading', async () => {
@@ -180,6 +256,41 @@ describe('Remote queries', function() {
}
});
function verifyQlPack(qlpackPath: string, queryPath: string, packVersion: string, pathSerializationBroken: boolean) {
const qlPack = yaml.safeLoad(fs.readFileSync(qlpackPath, 'utf8')) as QlPack;
if (pathSerializationBroken) {
// the path serialization is broken, so we force it to be the path in the pack to be same as the query path
qlPack.defaultSuite![1].query = queryPath;
}
// don't check the build metadata since it is variable
delete (qlPack as any).buildMetadata;
expect(qlPack).to.deep.equal({
name: 'codeql-remote/query',
version: packVersion,
dependencies: {
'codeql/javascript-all': '*',
},
library: false,
defaultSuite: [{
description: 'Query suite for remote query'
}, {
query: queryPath
}]
});
}
/**
* In version 2.7.2 and earlier, relative paths were not serialized correctly inside the qlpack.yml file.
* So, ignore part of the test for these versions.
*
* @returns true if path serialization is broken in this run
*/
async function pathSerializationBroken() {
return lte((await cli.getVersion()), '2.7.2') && os.platform() === 'win32';
}
function getFile(file: string): Uri {
return Uri.file(path.join(baseDir, file));
}

View File

@@ -44,7 +44,7 @@ const _10MB = _1MB * 10;
// CLI version to test. Hard code the latest as default. And be sure
// to update the env if it is not otherwise set.
const CLI_VERSION = process.env.CLI_VERSION || 'v2.7.2';
const CLI_VERSION = process.env.CLI_VERSION || 'v2.7.6';
process.env.CLI_VERSION = CLI_VERSION;
// Base dir where CLIs will be downloaded into

View File

@@ -12,7 +12,8 @@ import {
DatabaseManager,
DatabaseItemImpl,
DatabaseContents,
FullDatabaseOptions
FullDatabaseOptions,
findSourceArchive
} from '../../databases';
import { Logger } from '../../logging';
import { QueryServerClient } from '../../queryserver-client';
@@ -179,7 +180,7 @@ describe('databases', () => {
expect(spy).to.have.been.calledWith(mockEvent);
});
it('should add a database item source archive', async function () {
it('should add a database item source archive', async function() {
const mockDbItem = createMockDB();
mockDbItem.name = 'xxx';
await (databaseManager as any).addDatabaseSourceArchiveFolder(mockDbItem);
@@ -478,6 +479,49 @@ describe('databases', () => {
const db = createMockDB(sourceLocationUri(), Uri.file('/path/to/dir/dir.testproj'));
expect(await db.isAffectedByTest('/path/to/test.ql')).to.false;
});
});
describe('findSourceArchive', function() {
// not sure why, but some of these tests take more than two seconds to run.
this.timeout(5000);
['src', 'output/src_archive'].forEach(name => {
it(`should find source folder in ${name}`, async () => {
const uri = Uri.file(path.join(dir.name, name));
fs.createFileSync(path.join(uri.fsPath, 'hucairz.txt'));
const srcUri = await findSourceArchive(dir.name);
expect(srcUri!.fsPath).to.eq(uri.fsPath);
});
it(`should find source archive in ${name}.zip`, async () => {
const uri = Uri.file(path.join(dir.name, name + '.zip'));
fs.createFileSync(uri.fsPath);
const srcUri = await findSourceArchive(dir.name);
expect(srcUri!.fsPath).to.eq(uri.fsPath);
});
it(`should prioritize ${name}.zip over ${name}`, async () => {
const uri = Uri.file(path.join(dir.name, name + '.zip'));
fs.createFileSync(uri.fsPath);
const uriFolder = Uri.file(path.join(dir.name, name));
fs.createFileSync(path.join(uriFolder.fsPath, 'hucairz.txt'));
const srcUri = await findSourceArchive(dir.name);
expect(srcUri!.fsPath).to.eq(uri.fsPath);
});
});
it('should prioritize src over output/src_archive', async () => {
const uriSrc = Uri.file(path.join(dir.name, 'src.zip'));
fs.createFileSync(uriSrc.fsPath);
const uriSrcArchive = Uri.file(path.join(dir.name, 'src.zip'));
fs.createFileSync(uriSrcArchive.fsPath);
const resultUri = await findSourceArchive(dir.name);
expect(resultUri!.fsPath).to.eq(uriSrc.fsPath);
});
});
function createMockDB(

View File

@@ -25,11 +25,11 @@ describe('run-remote-query', function() {
showInputBoxSpy = sandbox.stub(window, 'showInputBox');
getRemoteRepositoryListsSpy = sandbox.stub();
showAndLogErrorMessageSpy = sandbox.stub();
mod = proxyquire('../../run-remote-query', {
'./config': {
mod = proxyquire('../../remote-queries/run-remote-query', {
'../config': {
getRemoteRepositoryLists: getRemoteRepositoryListsSpy
},
'./helpers': {
'../helpers': {
showAndLogErrorMessage: showAndLogErrorMessageSpy
},
});
@@ -134,12 +134,12 @@ describe('run-remote-query', function() {
logSpy = sandbox.stub();
showAndLogErrorMessageSpy = sandbox.stub();
showInformationMessageWithActionSpy = sandbox.stub();
mod = proxyquire('../../run-remote-query', {
'./helpers': {
mod = proxyquire('../../remote-queries/run-remote-query', {
'../helpers': {
showAndLogErrorMessage: showAndLogErrorMessageSpy,
showInformationMessageWithAction: showInformationMessageWithActionSpy
},
'./logging': {
'../logging': {
'logger': {
log: logSpy
}

View File

@@ -7,7 +7,7 @@ import { sarifParser } from '../../sarif-parser';
chai.use(chaiAsPromised);
const expect = chai.expect;
describe.only('sarif parser', function() {
describe('sarif parser', function() {
const sarifDir = path.join(__dirname, 'data/sarif');
it('should parse a valid SARIF file', async () => {
const result = await sarifParser(path.join(sarifDir, 'validSarif.sarif'));
@@ -22,4 +22,4 @@ describe.only('sarif parser', function() {
const result = await sarifParser(path.join(sarifDir, 'emptyResultsSarif.sarif'));
expect(result.runs[0].results).to.be.empty;
});
});
});

View File

@@ -80,10 +80,19 @@ async function main() {
if (dirs.includes(TestDir.CliIntegration)) {
console.log('Installing required extensions');
const cliPath = resolveCliPathFromVSCodeExecutablePath(vscodeExecutablePath);
cp.spawnSync(cliPath, ['--install-extension', 'hbenl.vscode-test-explorer'], {
encoding: 'utf-8',
stdio: 'inherit'
});
cp.spawnSync(
cliPath,
[
'--install-extension',
'hbenl.vscode-test-explorer',
'--install-extension',
'ms-vscode.test-adapter-converter',
],
{
encoding: 'utf-8',
stdio: 'inherit',
}
);
}
console.log(`Running integration tests in these directories: ${dirs}`);

6
package-lock.json generated
View File

@@ -1,6 +0,0 @@
{
"name": "vscode-codeql",
"lockfileVersion": 2,
"requires": true,
"packages": {}
}