v1.3.10

Clarify how to run CLI tests locally
Also, remove an errant `only`, which was preventing some tests from running.
2021-01-20 14:15:45 -08:00 · 2021-01-20 13:05:53 -08:00 · 2021-01-20 13:05:53 -08:00 · 2021-01-20 13:05:53 -08:00 · 2021-01-19 12:51:12 -08:00 · 2021-01-19 06:51:18 -08:00
66 changed files with 2940 additions and 1449 deletions
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -20,17 +20,17 @@ jobs:

      - uses: actions/setup-node@v1
        with:
-          node-version: '10.18.1'
+          node-version: '14.14.0'

      - name: Install dependencies
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          npm install
        shell: bash

      - name: Build
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          npm run build
        shell: bash

@@ -61,24 +61,23 @@ jobs:

      - uses: actions/setup-node@v1
        with:
-          node-version: '10.18.1'
+          node-version: '14.14.0'

-      # We have to build the dependencies in `lib` before running any tests.
      - name: Install dependencies
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          npm install
        shell: bash

      - name: Build
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          npm run build
        shell: bash

      - name: Lint
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          npm run lint

      - name: Install CodeQL
@@ -91,27 +90,76 @@ jobs:
        shell: bash

      - name: Run unit tests (Linux)
+        working-directory: extensions/ql-vscode
        if: matrix.os == 'ubuntu-latest'
        run: |
-          cd extensions/ql-vscode
          CODEQL_PATH=$GITHUB_WORKSPACE/codeql-home/codeql/codeql npm run test

      - name: Run unit tests (Windows)
        if: matrix.os == 'windows-latest'
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          $env:CODEQL_PATH=$(Join-Path $env:GITHUB_WORKSPACE -ChildPath 'codeql-home/codeql/codeql.exe')
          npm run test

      - name: Run integration tests (Linux)
        if: matrix.os == 'ubuntu-latest'
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          sudo apt-get install xvfb
          /usr/bin/xvfb-run npm run integration

      - name: Run integration tests (Windows)
        if: matrix.os == 'windows-latest'
+        working-directory: extensions/ql-vscode
        run: |
-          cd extensions/ql-vscode
          npm run integration
+
+  cli-test:
+    name: CLI Test
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        version: ['v2.2.6', 'v2.3.3', 'v2.4.0']
+    env:
+      CLI_VERSION: ${{ matrix.version }}
+      TEST_CODEQL_PATH: '${{ github.workspace }}/codeql'
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - uses: actions/setup-node@v1
+        with:
+          node-version: '14.14.0'
+
+      - name: Install dependencies
+        working-directory: extensions/ql-vscode
+        run: |
+          npm install
+        shell: bash
+
+      - name: Build
+        working-directory: extensions/ql-vscode
+        run: |
+          npm run build
+        shell: bash
+
+      - name: Checkout QL
+        uses: actions/checkout@v2
+        with:
+          repository: github/codeql
+          path: codeql
+
+      - name: Run CLI tests (Linux)
+        working-directory: extensions/ql-vscode
+        if: matrix.os == 'ubuntu-latest'
+        run: |
+          /usr/bin/xvfb-run npm run cli-integration
+
+      - name: Run CLI tests (Windows)
+        working-directory: extensions/ql-vscode
+        if: matrix.os == 'windows-latest'
+        run: |
+          npm run cli-integration
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -32,7 +32,7 @@ jobs:
      - name: Install dependencies
        run: |
          cd extensions/ql-vscode
-          npm install
+          npm ci
        shell: bash

      - name: Build
@@ -55,9 +55,6 @@ jobs:
          REF_NAME="$(echo ${{ github.ref }} | sed -e 's:^refs/tags/::' | sed -e 's:/:-:g')"
          echo "::set-output name=ref_name::$REF_NAME"

-      # Uploading artifacts is not necessary to create a release.
-      # This is just in case the release itself fails and we want to access the built artifacts from Actions.
-      # TODO Remove if not useful.
      - name: Upload artifacts
        uses: actions/upload-artifact@v2
        with:
@@ -126,3 +123,40 @@ jobs:
          body: This PR was automatically generated by the GitHub Actions release workflow in this repository.
          branch: ${{ format('version/bump-to-{0}', steps.bump-patch-version.outputs.next_version) }}
          base: main
+
+  vscode-publish:
+    name: Publish to VS Code Marketplace
+    needs: build
+    environment: publish-vscode-marketplace
+    runs-on: ubuntu-latest
+    env:
+      VSCE_TOKEN: ${{ secrets.VSCE_TOKEN }}
+    steps:
+      - name: Download artifact
+        uses: actions/download-artifact@v2
+        with:
+          name: vscode-codeql-extension
+
+      - name: Publish to Registry
+        run: |
+          npx vsce publish -p $VSCE_TOKEN --packagePath *.vsix || \
+            echo "Failed to publish to VS Code Marketplace. \
+                  If this was an authentication problem, please make sure the \
+                  auth token hasn't expired."
+
+  open-vsx-publish:
+    name: Publish to Open VSX Registry
+    needs: build
+    environment: publish-open-vsx
+    runs-on: ubuntu-latest
+    env:
+      OPEN_VSX_TOKEN: ${{ secrets.OPEN_VSX_TOKEN }}
+    steps:
+      - name: Download artifact
+        uses: actions/download-artifact@v2
+        with:
+          name: vscode-codeql-extension
+
+      - name: Publish to Registry
+        run: |
+          npx ovsx publish -p $OPEN_VSX_TOKEN *.vsix
--- a/.gitignore
+++ b/.gitignore
@@ -4,6 +4,7 @@
 # Generated files
 /dist/
 out/
+build/
 server/
 node_modules/
 gen/
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@ -8,7 +8,9 @@
      "request": "launch",
      "runtimeExecutable": "${execPath}",
      "args": [
-        "--extensionDevelopmentPath=${workspaceRoot}/extensions/ql-vscode"
+        "--extensionDevelopmentPath=${workspaceRoot}/extensions/ql-vscode",
+        // Add a reference to a workspace to open. Eg-
+        // "${workspaceRoot}/../vscode-codeql-starter/vscode-codeql-starter.code-workspace"
      ],
      "stopOnEntry": false,
      "sourceMaps": true,
@@ -77,6 +79,25 @@
      "outFiles": [
        "${workspaceRoot}/extensions/ql-vscode/out/**/*.js",
      ],
+    },
+    {
+      "name": "Launch Integration Tests - With CLI",
+      "type": "extensionHost",
+      "request": "launch",
+      "runtimeExecutable": "${execPath}",
+      "args": [
+        "--extensionDevelopmentPath=${workspaceRoot}/extensions/ql-vscode",
+        "--extensionTestsPath=${workspaceRoot}/extensions/ql-vscode/out/vscode-tests/cli-integration/index",
+        "${workspaceRoot}/extensions/ql-vscode/src/vscode-tests/cli-integration/data",
+        // Add a path to a checked out instance of the codeql repository so the libraries are
+        // available in the workspace for the tests.
+        // "${workspaceRoot}/../codeql"
+      ],
+      "stopOnEntry": false,
+      "sourceMaps": true,
+      "outFiles": [
+        "${workspaceRoot}/extensions/ql-vscode/out/**/*.js",
+      ],
    }
  ]
 }
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -114,14 +114,32 @@ Alternatively, you can run the tests inside of vscode. There are several vscode
 1. Download the VSIX from the draft GitHub release at the top of [the releases page](https://github.com/github/vscode-codeql/releases) that is created when the release build finishes.
 1. Unzip the `.vsix` and inspect its `package.json` to make sure the version is what you expect,
   or look at the source if there's any doubt the right code is being shipped.
-1. Log into the [Visual Studio Marketplace](https://marketplace.visualstudio.com/manage/publishers/github).
-1. Click the `...` menu in the CodeQL row and click **Update**.
-1. Drag the `.vsix` file you downloaded from the GitHub release into the Marketplace and click **Upload**.
-1. Go to the draft GitHub release, click 'Edit', add some summary description, and publish it.
+1. Go to the actions tab of the vscode-codeql repository and select the [Release workflow](https://github.com/github/vscode-codeql/actions?query=workflow%3ARelease).
+    - If there is an authentication failure when publishing, be sure to check that the authentication keys haven't expired. See below.
+1. Approve the deployments of the correct Release workflow. This will automatically publish to Open VSX and VS Code Marketplace.
+1. Go to the draft GitHub release in [the releases tab of the repository](https://github.com/github/vscode-codeql/releases), click 'Edit', add some summary description, and publish it.
 1. Confirm the new release is marked as the latest release at <https://github.com/github/vscode-codeql/releases>.
 1. If documentation changes need to be published, notify documentation team that release has been made.
 1. Review and merge the version bump PR that is automatically created by Actions.

+## Secrets and authentication for publishing
+
+Repository administrators, will need to manage the authentication keys for publishing to the VS Code marketplace and Open VSX. Each requires an authentication token. The VS Code marketplace token expires yearly.
+
+To regenerate the Open VSX token:
+
+1. Log in to the [user settings page on Open VSX](https://open-vsx.org/user-settings/namespaces).
+1. Make sure you are a member of the GitHub namespace.
+1. Go to the [Access Tokens](https://open-vsx.org/user-settings/tokens) page and generate a new token.
+1. Update the secret in the `publish-open-vsx` environment in the project settings.
+
+To regenerate the VSCode Marketplace token:
+
+1. Follow the instructions on [getting a PAT for Azure DevOps](https://code.visualstudio.com/api/working-with-extensions/publishing-extension#get-a-personal-access-token).
+1. Update the secret in the `publish-vscode-marketplace` environment in the project settings.
+
+Not that Azure DevOps PATs expire yearly and must be regenerated.
+
 ## Resources

 * [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
--- a/extensions/ql-vscode/CHANGELOG.md
+++ b/extensions/ql-vscode/CHANGELOG.md
@@ -1,5 +1,27 @@
 # CodeQL for Visual Studio Code: Changelog

+## 1.3.10 - 20 January 2021
+
+- Include the full stack in error log messages to help with debugging. [#726](https://github.com/github/vscode-codeql/pull/726)
+
+## 1.3.9 - 12 January 2021
+
+- No changes visible to end users.
+
+## 1.3.8 - 17 December 2020
+
+- Ensure databases are unlocked when removing them from the workspace. This will ensure that after a database is removed from VS Code, queries can be run on it from the command line without restarting the IDE. Requires CodeQL CLI 2.4.1 or later. [#681](https://github.com/github/vscode-codeql/pull/681)
+- Fix bug when removing databases where sometimes the source folder would not also be removed from the workspace or the database files would not be deleted from the workspace storage location. [#692](https://github.com/github/vscode-codeql/pull/692)
+- Query results with no string representation will now be displayed with placeholder text in query results. Previously, they were omitted. [#694](https://github.com/github/vscode-codeql/pull/694)
+- Add a label for the language of a database in the databases view. This will only take effect for new databases created with the CodeQL CLI v2.4.1 or later. [#697](https://github.com/github/vscode-codeql/pull/697)
+- Add clearer error message when running a query using a missing or invalid qlpack. [#702](https://github.com/github/vscode-codeql/pull/702)
+- Add clearer error message when trying to run a command from the query history view if no item in the history is selected. [#702](https://github.com/github/vscode-codeql/pull/702)
+- Fix a bug where it is not possible to download some database archives. This fix specifically addresses large archives and archives whose central directories do not align with file headers. [#700](https://github.com/github/vscode-codeql/pull/700)
+- Avoid error dialogs when QL test discovery or database cleanup encounters a missing directory. [#706](https://github.com/github/vscode-codeql/pull/706)
+- Add descriptive text and a link in the results view. [#711](https://github.com/github/vscode-codeql/pull/711)
+- Fix the _Set Label_ command in the query history view. [#710](https://github.com/github/vscode-codeql/pull/710)
+- Add the _CodeQL: View AST_ command to the right-click context menu when a source file in a database source archive is open in the editor. [#712](https://github.com/github/vscode-codeql/pull/712)
+
 ## 1.3.7 - 24 November 2020

 - Editors opened by navigating from the results view are no longer opened in _preview mode_. Now they are opened as a persistent editor. [#630](https://github.com/github/vscode-codeql/pull/630)
--- a/extensions/ql-vscode/gulpfile.ts/index.ts
+++ b/extensions/ql-vscode/gulpfile.ts/index.ts
@@ -6,5 +6,5 @@ import { compileView } from './webpack';
 import { packageExtension } from './package';

 export const buildWithoutPackage = gulp.parallel(compileTypeScript, compileTextMateGrammar, compileView, copyTestData, copyViewCss);
-export { compileTextMateGrammar, watchTypeScript, compileTypeScript };
+export { compileTextMateGrammar, watchTypeScript, compileTypeScript, copyTestData };
 exports.default = gulp.series(exports.buildWithoutPackage, packageExtension);
--- a/extensions/ql-vscode/gulpfile.ts/tests.ts
+++ b/extensions/ql-vscode/gulpfile.ts/tests.ts
@@ -1,6 +1,17 @@
 import * as gulp from 'gulp';

 export function copyTestData() {
+  copyNoWorkspaceData();
+  copyCliIntegrationData();
+  return Promise.resolve();
+}
+
+function copyNoWorkspaceData() {
  return gulp.src('src/vscode-tests/no-workspace/data/**/*')
    .pipe(gulp.dest('out/vscode-tests/no-workspace/data'));
 }
+
+function copyCliIntegrationData() {
+  return gulp.src('src/vscode-tests/cli-integration/data/**/*')
+    .pipe(gulp.dest('out/vscode-tests/cli-integration/data'));
+}
--- a/extensions/ql-vscode/package-lock.json
+++ b/extensions/ql-vscode/package-lock.json
@@ -1,6 +1,6 @@
 {
  "name": "vscode-codeql",
-  "version": "1.3.7",
+  "version": "1.3.9",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
@@ -217,9 +217,9 @@
      "dev": true
    },
    "@types/fs-extra": {
-      "version": "9.0.3",
-      "resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-9.0.3.tgz",
-      "integrity": "sha512-NKdGoXLTFTRED3ENcfCsH8+ekV4gbsysanx2OPbstXVV6fZMgUCqTxubs6I9r7pbOJbFgVq1rpFtLURjKCZWUw==",
+      "version": "9.0.6",
+      "resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-9.0.6.tgz",
+      "integrity": "sha512-ecNRHw4clCkowNOBJH1e77nvbPxHYnWIXMv1IAoG/9+MYGkgoyr3Ppxr7XYFNL41V422EDhyV4/4SSK8L2mlig==",
      "dev": true,
      "requires": {
        "@types/node": "*"
@@ -304,9 +304,9 @@
      "integrity": "sha512-tHq6qdbT9U1IRSGf14CL0pUlULksvY9OZ+5eEgl1N7t+OA3tGvNpxJCzuKQlsNgCVwbAs670L1vcVQi8j9HjnA=="
    },
    "@types/mocha": {
-      "version": "8.0.3",
-      "resolved": "https://registry.npmjs.org/@types/mocha/-/mocha-8.0.3.tgz",
-      "integrity": "sha512-vyxR57nv8NfcU0GZu8EUXZLTbCMupIUwy95LJ6lllN+JRPG25CwMHoB1q5xKh8YKhQnHYRAn4yW2yuHbf/5xgg==",
+      "version": "8.0.4",
+      "resolved": "https://registry.npmjs.org/@types/mocha/-/mocha-8.0.4.tgz",
+      "integrity": "sha512-M4BwiTJjHmLq6kjON7ZoI2JMlBvpY3BYSdiP6s/qCT3jb1s9/DeJF0JELpAxiVSIxXDzfNKe+r7yedMIoLbknQ==",
      "dev": true
    },
    "@types/node": {
@@ -621,6 +621,12 @@
        }
      }
    },
+    "@ungap/promise-all-settled": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/@ungap/promise-all-settled/-/promise-all-settled-1.1.2.tgz",
+      "integrity": "sha512-sL/cEvJWAnClXw0wHk85/2L0G6Sj8UB0Ctc1TEMbKSsmpRosqhwj9gWgFRZSrBr2f9tiXISwNhCPmlfqUqyb9Q==",
+      "dev": true
+    },
    "@webassemblyjs/ast": {
      "version": "1.9.0",
      "resolved": "https://registry.npmjs.org/@webassemblyjs/ast/-/ast-1.9.0.tgz",
@@ -1134,18 +1140,6 @@
      "integrity": "sha1-qJS3XUvE9s1nnvMkSp/Y9Gri1Cg=",
      "dev": true
    },
-    "array.prototype.map": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/array.prototype.map/-/array.prototype.map-1.0.2.tgz",
-      "integrity": "sha512-Az3OYxgsa1g7xDYp86l0nnN4bcmuEITGe1rbdEBVkrqkzMgDcbdQ2R7r41pNzti+4NMces3H8gMmuioZUilLgw==",
-      "dev": true,
-      "requires": {
-        "define-properties": "^1.1.3",
-        "es-abstract": "^1.17.0-next.1",
-        "es-array-method-boxes-properly": "^1.0.0",
-        "is-string": "^1.0.4"
-      }
-    },
    "asn1.js": {
      "version": "4.10.1",
      "resolved": "https://registry.npmjs.org/asn1.js/-/asn1.js-4.10.1.tgz",
@@ -2842,35 +2836,6 @@
        "string.prototype.trimstart": "^1.0.1"
      }
    },
-    "es-array-method-boxes-properly": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/es-array-method-boxes-properly/-/es-array-method-boxes-properly-1.0.0.tgz",
-      "integrity": "sha512-wd6JXUmyHmt8T5a2xreUwKcGPq6f1f+WwIJkijUqiGcJz1qqnZgP6XIK+QyIWU5lT7imeNxUll48bziG+TSYcA==",
-      "dev": true
-    },
-    "es-get-iterator": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/es-get-iterator/-/es-get-iterator-1.1.0.tgz",
-      "integrity": "sha512-UfrmHuWQlNMTs35e1ypnvikg6jCz3SK8v8ImvmDsh36fCVUR1MqoFDiyn0/k52C8NqO3YsO8Oe0azeesNuqSsQ==",
-      "dev": true,
-      "requires": {
-        "es-abstract": "^1.17.4",
-        "has-symbols": "^1.0.1",
-        "is-arguments": "^1.0.4",
-        "is-map": "^2.0.1",
-        "is-set": "^2.0.1",
-        "is-string": "^1.0.5",
-        "isarray": "^2.0.5"
-      },
-      "dependencies": {
-        "isarray": {
-          "version": "2.0.5",
-          "resolved": "https://registry.npmjs.org/isarray/-/isarray-2.0.5.tgz",
-          "integrity": "sha512-xHjhDr3cNBK0BzdUJSPXZntQUx/mwMS5Rw4A7lPJ90XGAO6ISP/ePDNuo0vhqOZU+UD5JoodwCAAoZQd3FeAKw==",
-          "dev": true
-        }
-      }
-    },
    "es-to-primitive": {
      "version": "1.2.1",
      "resolved": "https://registry.npmjs.org/es-to-primitive/-/es-to-primitive-1.2.1.tgz",
@@ -3648,21 +3613,10 @@
      "dev": true
    },
    "flat": {
-      "version": "4.1.0",
-      "resolved": "https://registry.npmjs.org/flat/-/flat-4.1.0.tgz",
-      "integrity": "sha512-Px/TiLIznH7gEDlPXcUD4KnBusa6kR6ayRUVcnEAbreRIuhkqow/mun59BuRXwoYk7ZQOLW1ZM05ilIvK38hFw==",
-      "dev": true,
-      "requires": {
-        "is-buffer": "~2.0.3"
-      },
-      "dependencies": {
-        "is-buffer": {
-          "version": "2.0.4",
-          "resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-2.0.4.tgz",
-          "integrity": "sha512-Kq1rokWXOPXWuaMAqZiJW4XxsmD9zGx9q4aePabbn3qCRGedtH7Cm+zV8WETitMfu1wdh+Rvd6w5egwSngUX2A==",
-          "dev": true
-        }
-      }
+      "version": "5.0.2",
+      "resolved": "https://registry.npmjs.org/flat/-/flat-5.0.2.tgz",
+      "integrity": "sha512-b6suED+5/3rTpUBdG1gupIl8MPFCAMA0QXwmljLhvCUKcUvdE4gWky9zpuGCcXHOsz4J9wPGNWq6OKpmIzz3hQ==",
+      "dev": true
    },
    "flat-cache": {
      "version": "2.0.1",
@@ -4558,9 +4512,9 @@
      "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
    },
    "ini": {
-      "version": "1.3.5",
-      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.5.tgz",
-      "integrity": "sha512-RZY5huIKCMRWDUqZlEi72f/lmXKMvuszcMBduliQ3nnWbx9X/ZBQO7DijMEYS9EhHBb2qacRUMtC7svLwe0lcw==",
+      "version": "1.3.8",
+      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
+      "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
      "dev": true
    },
    "inquirer": {
@@ -4698,12 +4652,6 @@
        }
      }
    },
-    "is-arguments": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/is-arguments/-/is-arguments-1.0.4.tgz",
-      "integrity": "sha512-xPh0Rmt8NE65sNzvyUmWgI1tz3mKq74lGA0mL8LYZcoIzKOzDh6HmrYm3d18k60nHerC8A9Km8kYu87zfSFnLA==",
-      "dev": true
-    },
    "is-arrayish": {
      "version": "0.2.1",
      "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.2.1.tgz",
@@ -4803,12 +4751,6 @@
        "is-extglob": "^2.1.1"
      }
    },
-    "is-map": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/is-map/-/is-map-2.0.1.tgz",
-      "integrity": "sha512-T/S49scO8plUiAOA2DBTBG3JHpn1yiw0kRp6dgiZ0v2/6twi5eiB0rHtHFH9ZIrvlWc6+4O+m4zg5+Z833aXgw==",
-      "dev": true
-    },
    "is-negated-glob": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/is-negated-glob/-/is-negated-glob-1.0.0.tgz",
@@ -4848,9 +4790,9 @@
      "dev": true
    },
    "is-plain-obj": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-1.1.0.tgz",
-      "integrity": "sha1-caUMhCnfync8kqOQpKA7OfzVHT4=",
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-2.1.0.tgz",
+      "integrity": "sha512-YWnfyRwxL/+SsrWYfOpUtz5b3YD+nyfkHvjbcanzk8zgyO4ASD67uVMRt8k5bM4lLMDnXfriRhOpemw+NfT1eA==",
      "dev": true
    },
    "is-plain-object": {
@@ -4892,12 +4834,6 @@
        "is-unc-path": "^1.0.0"
      }
    },
-    "is-set": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/is-set/-/is-set-2.0.1.tgz",
-      "integrity": "sha512-eJEzOtVyenDs1TMzSQ3kU3K+E0GUS9sno+F0OBT97xsgcJsF9nXMBtkT9/kut5JEpM7oL7X/0qxR17K3mcwIAA==",
-      "dev": true
-    },
    "is-stream": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-2.0.0.tgz",
@@ -4968,22 +4904,6 @@
      "integrity": "sha1-TkMekrEalzFjaqH5yNHMvP2reN8=",
      "dev": true
    },
-    "iterate-iterator": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/iterate-iterator/-/iterate-iterator-1.0.1.tgz",
-      "integrity": "sha512-3Q6tudGN05kbkDQDI4CqjaBf4qf85w6W6GnuZDtUVYwKgtC1q8yxYX7CZed7N+tLzQqS6roujWvszf13T+n9aw==",
-      "dev": true
-    },
-    "iterate-value": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/iterate-value/-/iterate-value-1.0.2.tgz",
-      "integrity": "sha512-A6fMAio4D2ot2r/TYzr4yUWrmwNdsN5xL7+HUiyACE4DXm+q8HtPcnFTp+NnW3k4N05tZ7FVYFFb2CR13NxyHQ==",
-      "dev": true,
-      "requires": {
-        "es-get-iterator": "^1.0.2",
-        "iterate-iterator": "^1.0.1"
-      }
-    },
    "js-tokens": {
      "version": "4.0.0",
      "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
@@ -5934,15 +5854,16 @@
      }
    },
    "mocha": {
-      "version": "8.1.3",
-      "resolved": "https://registry.npmjs.org/mocha/-/mocha-8.1.3.tgz",
-      "integrity": "sha512-ZbaYib4hT4PpF4bdSO2DohooKXIn4lDeiYqB+vTmCdr6l2woW0b6H3pf5x4sM5nwQMru9RvjjHYWVGltR50ZBw==",
+      "version": "8.2.1",
+      "resolved": "https://registry.npmjs.org/mocha/-/mocha-8.2.1.tgz",
+      "integrity": "sha512-cuLBVfyFfFqbNR0uUKbDGXKGk+UDFe6aR4os78XIrMQpZl/nv7JYHcvP5MFIAb374b2zFXsdgEGwmzMtP0Xg8w==",
      "dev": true,
      "requires": {
+        "@ungap/promise-all-settled": "1.1.2",
        "ansi-colors": "4.1.1",
        "browser-stdout": "1.3.1",
-        "chokidar": "3.4.2",
-        "debug": "4.1.1",
+        "chokidar": "3.4.3",
+        "debug": "4.2.0",
        "diff": "4.0.2",
        "escape-string-regexp": "4.0.0",
        "find-up": "5.0.0",
@@ -5953,17 +5874,16 @@
        "log-symbols": "4.0.0",
        "minimatch": "3.0.4",
        "ms": "2.1.2",
-        "object.assign": "4.1.0",
-        "promise.allsettled": "1.0.2",
-        "serialize-javascript": "4.0.0",
-        "strip-json-comments": "3.0.1",
-        "supports-color": "7.1.0",
+        "nanoid": "3.1.12",
+        "serialize-javascript": "5.0.1",
+        "strip-json-comments": "3.1.1",
+        "supports-color": "7.2.0",
        "which": "2.0.2",
        "wide-align": "1.1.3",
-        "workerpool": "6.0.0",
+        "workerpool": "6.0.2",
        "yargs": "13.3.2",
        "yargs-parser": "13.1.2",
-        "yargs-unparser": "1.6.1"
+        "yargs-unparser": "2.0.0"
      },
      "dependencies": {
        "anymatch": {
@@ -5992,9 +5912,9 @@
          }
        },
        "chokidar": {
-          "version": "3.4.2",
-          "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.4.2.tgz",
-          "integrity": "sha512-IZHaDeBeI+sZJRX7lGcXsdzgvZqKv6sECqsbErJA4mHWfpRrD8B97kSFN4cQz6nGBGiuFia1MKR4d6c1o8Cv7A==",
+          "version": "3.4.3",
+          "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.4.3.tgz",
+          "integrity": "sha512-DtM3g7juCXQxFVSNPNByEC2+NImtBuxQQvWlHunpJIS5Ocr0lG306cC7FCi7cEA0fzmybPUIl4txBIobk1gGOQ==",
          "dev": true,
          "requires": {
            "anymatch": "~3.1.1",
@@ -6004,7 +5924,7 @@
            "is-binary-path": "~2.1.0",
            "is-glob": "~4.0.1",
            "normalize-path": "~3.0.0",
-            "readdirp": "~3.4.0"
+            "readdirp": "~3.5.0"
          }
        },
        "cliui": {
@@ -6019,12 +5939,12 @@
          }
        },
        "debug": {
-          "version": "4.1.1",
-          "resolved": "https://registry.npmjs.org/debug/-/debug-4.1.1.tgz",
-          "integrity": "sha512-pYAIzeRo8J6KPEaJ0VWOh5Pzkbw/RetuzehGM7QRRX5he4fPHx2rdKMB256ehJCkX+XRQm16eZLqLNS8RSZXZw==",
+          "version": "4.2.0",
+          "resolved": "https://registry.npmjs.org/debug/-/debug-4.2.0.tgz",
+          "integrity": "sha512-IX2ncY78vDTjZMFUdmsvIRFY2Cf4FnD0wRs+nQwJU8Lu99/tPFdb0VybiiMTPe3I6rQmwsqQqRBvxU+bZ/I8sg==",
          "dev": true,
          "requires": {
-            "ms": "^2.1.1"
+            "ms": "2.1.2"
          }
        },
        "emoji-regex": {
@@ -6123,12 +6043,12 @@
          "dev": true
        },
        "p-limit": {
-          "version": "3.0.2",
-          "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-3.0.2.tgz",
-          "integrity": "sha512-iwqZSOoWIW+Ew4kAGUlN16J4M7OB3ysMLSZtnhmqx7njIHFPlxWBX8xo3lVTyFVq6mI/lL9qt2IsN1sHwaxJkg==",
+          "version": "3.1.0",
+          "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-3.1.0.tgz",
+          "integrity": "sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==",
          "dev": true,
          "requires": {
-            "p-try": "^2.0.0"
+            "yocto-queue": "^0.1.0"
          }
        },
        "p-locate": {
@@ -6147,9 +6067,9 @@
          "dev": true
        },
        "readdirp": {
-          "version": "3.4.0",
-          "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.4.0.tgz",
-          "integrity": "sha512-0xe001vZBnJEK+uKcj8qOhyAKPzIT+gStxWr3LCB0DwcXR5NZJ3IaC+yGnHCYzB/S7ov3m3EEbZI2zeNvX+hGQ==",
+          "version": "3.5.0",
+          "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.5.0.tgz",
+          "integrity": "sha512-cMhu7c/8rdhkHXWsY+osBhfSy0JikwpHK/5+imo+LpeasTF8ouErHrlYkwT0++njiyuDvc7OFY5T3ukvZ8qmFQ==",
          "dev": true,
          "requires": {
            "picomatch": "^2.2.1"
@@ -6162,9 +6082,9 @@
          "dev": true
        },
        "serialize-javascript": {
-          "version": "4.0.0",
-          "resolved": "https://registry.npmjs.org/serialize-javascript/-/serialize-javascript-4.0.0.tgz",
-          "integrity": "sha512-GaNA54380uFefWghODBWEGisLZFj00nS5ACs6yHa9nLqlLpVLO8ChDGeKRjZnV4Nh4n0Qi7nhYZD/9fCPzEqkw==",
+          "version": "5.0.1",
+          "resolved": "https://registry.npmjs.org/serialize-javascript/-/serialize-javascript-5.0.1.tgz",
+          "integrity": "sha512-SaaNal9imEO737H2c05Og0/8LUXG7EnsZyMa8MzkmuHoELfT6txuj0cMqRj6zfPKnmQ1yasR4PCJc8x+M4JSPA==",
          "dev": true,
          "requires": {
            "randombytes": "^2.1.0"
@@ -6181,16 +6101,10 @@
            "strip-ansi": "^5.1.0"
          }
        },
-        "strip-json-comments": {
-          "version": "3.0.1",
-          "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.0.1.tgz",
-          "integrity": "sha512-VTyMAUfdm047mwKl+u79WIdrZxtFtn+nBxHeb844XBQ9uMNTuTHdx2hc5RiAJYqwTj3wc/xe5HLSdJSkJ+WfZw==",
-          "dev": true
-        },
        "supports-color": {
-          "version": "7.1.0",
-          "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.1.0.tgz",
-          "integrity": "sha512-oRSIpR8pxT1Wr2FquTNnGet79b3BWljqOuoW/h4oBhxJ/HUbX5nX6JSruTkvXDCFMwDPvsaTTbvMLKZWSy0R5g==",
+          "version": "7.2.0",
+          "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
+          "integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==",
          "dev": true,
          "requires": {
            "has-flag": "^4.0.0"
@@ -6299,16 +6213,6 @@
              "dev": true
            }
          }
-        },
-        "yargs-parser": {
-          "version": "13.1.2",
-          "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-13.1.2.tgz",
-          "integrity": "sha512-3lbsNRf/j+A4QuSZfDRA7HRSfWrzO0YjqTJd5kjAq37Zep1CEgaYmrH9Q3GwPiB9cHyd1Y1UwggGhJGoxipbzg==",
-          "dev": true,
-          "requires": {
-            "camelcase": "^5.0.0",
-            "decamelize": "^1.2.0"
-          }
        }
      }
    },
@@ -6363,6 +6267,12 @@
      "dev": true,
      "optional": true
    },
+    "nanoid": {
+      "version": "3.1.12",
+      "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.1.12.tgz",
+      "integrity": "sha512-1qstj9z5+x491jfiC4Nelk+f8XBad7LN20PmyWINJEMRSf3wcAjAWysw1qaA8z6NSKe2sjq1hRSDpBH5paCb6A==",
+      "dev": true
+    },
    "nanomatch": {
      "version": "1.2.13",
      "resolved": "https://registry.npmjs.org/nanomatch/-/nanomatch-1.2.13.tgz",
@@ -7335,19 +7245,6 @@
      "resolved": "https://registry.npmjs.org/promise-polyfill/-/promise-polyfill-6.1.0.tgz",
      "integrity": "sha1-36lpQ+qcEh/KTem1hoyznTRy4Fc="
    },
-    "promise.allsettled": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/promise.allsettled/-/promise.allsettled-1.0.2.tgz",
-      "integrity": "sha512-UpcYW5S1RaNKT6pd+s9jp9K9rlQge1UXKskec0j6Mmuq7UJCvlS2J2/s/yuPN8ehftf9HXMxWlKiPbGGUzpoRg==",
-      "dev": true,
-      "requires": {
-        "array.prototype.map": "^1.0.1",
-        "define-properties": "^1.1.3",
-        "es-abstract": "^1.17.0-next.1",
-        "function-bind": "^1.1.1",
-        "iterate-value": "^1.0.0"
-      }
-    },
    "prop-types": {
      "version": "15.7.2",
      "resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.7.2.tgz",
@@ -9871,9 +9768,9 @@
      }
    },
    "workerpool": {
-      "version": "6.0.0",
-      "resolved": "https://registry.npmjs.org/workerpool/-/workerpool-6.0.0.tgz",
-      "integrity": "sha512-fU2OcNA/GVAJLLyKUoHkAgIhKb0JoCpSjLC/G2vYKxUjVmQwGbRVeoPJ1a8U4pnVofz4AQV5Y/NEw8oKqxEBtA==",
+      "version": "6.0.2",
+      "resolved": "https://registry.npmjs.org/workerpool/-/workerpool-6.0.2.tgz",
+      "integrity": "sha512-DSNyvOpFKrNusaaUwk+ej6cBj1bmhLcBfj80elGk+ZIo5JSkq+unB1dLKEOcNfJDZgjGICfhQ0Q5TbP0PvF4+Q==",
      "dev": true
    },
    "wrap-ansi": {
@@ -10044,167 +9941,38 @@
      }
    },
    "yargs-parser": {
-      "version": "5.0.0-security.0",
-      "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-5.0.0-security.0.tgz",
-      "integrity": "sha512-T69y4Ps64LNesYxeYGYPvfoMTt/7y1XtfpIslUeK4um+9Hu7hlGoRtaDLvdXb7+/tfq4opVa2HRY5xGip022rQ==",
+      "version": "13.1.2",
+      "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-13.1.2.tgz",
+      "integrity": "sha512-3lbsNRf/j+A4QuSZfDRA7HRSfWrzO0YjqTJd5kjAq37Zep1CEgaYmrH9Q3GwPiB9cHyd1Y1UwggGhJGoxipbzg==",
      "dev": true,
      "requires": {
-        "camelcase": "^3.0.0",
-        "object.assign": "^4.1.0"
-      },
-      "dependencies": {
-        "camelcase": {
-          "version": "3.0.0",
-          "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-3.0.0.tgz",
-          "integrity": "sha1-MvxLn82vhF/N9+c7uXysImHwqwo=",
-          "dev": true
-        }
+        "camelcase": "^5.0.0",
+        "decamelize": "^1.2.0"
      }
    },
    "yargs-unparser": {
-      "version": "1.6.1",
-      "resolved": "https://registry.npmjs.org/yargs-unparser/-/yargs-unparser-1.6.1.tgz",
-      "integrity": "sha512-qZV14lK9MWsGCmcr7u5oXGH0dbGqZAIxTDrWXZDo5zUr6b6iUmelNKO6x6R1dQT24AH3LgRxJpr8meWy2unolA==",
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/yargs-unparser/-/yargs-unparser-2.0.0.tgz",
+      "integrity": "sha512-7pRTIA9Qc1caZ0bZ6RYRGbHJthJWuakf+WmHK0rVeLkNrrGhfoabBNdue6kdINI6r4if7ocq9aD/n7xwKOdzOA==",
      "dev": true,
      "requires": {
-        "camelcase": "^5.3.1",
-        "decamelize": "^1.2.0",
-        "flat": "^4.1.0",
-        "is-plain-obj": "^1.1.0",
-        "yargs": "^14.2.3"
+        "camelcase": "^6.0.0",
+        "decamelize": "^4.0.0",
+        "flat": "^5.0.2",
+        "is-plain-obj": "^2.1.0"
      },
      "dependencies": {
-        "cliui": {
-          "version": "5.0.0",
-          "resolved": "https://registry.npmjs.org/cliui/-/cliui-5.0.0.tgz",
-          "integrity": "sha512-PYeGSEmmHM6zvoef2w8TPzlrnNpXIjTipYK780YswmIP9vjxmd6Y2a3CB2Ks6/AU8NHjZugXvo8w3oWM2qnwXA==",
-          "dev": true,
-          "requires": {
-            "string-width": "^3.1.0",
-            "strip-ansi": "^5.2.0",
-            "wrap-ansi": "^5.1.0"
-          }
-        },
-        "emoji-regex": {
-          "version": "7.0.3",
-          "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-7.0.3.tgz",
-          "integrity": "sha512-CwBLREIQ7LvYFB0WyRvwhq5N5qPhc6PMjD6bYggFlI5YyDgl+0vxq5VHbMOFqLg7hfWzmu8T5Z1QofhmTIhItA==",
+        "camelcase": {
+          "version": "6.2.0",
+          "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-6.2.0.tgz",
+          "integrity": "sha512-c7wVvbw3f37nuobQNtgsgG9POC9qMbNuMQmTCqZv23b6MIz0fcYpBiOlv9gEN/hdLdnZTDQhg6e9Dq5M1vKvfg==",
          "dev": true
        },
-        "find-up": {
-          "version": "3.0.0",
-          "resolved": "https://registry.npmjs.org/find-up/-/find-up-3.0.0.tgz",
-          "integrity": "sha512-1yD6RmLI1XBfxugvORwlck6f75tYL+iR0jqwsOrOxMZyGYqUuDhJ0l4AXdO1iX/FTs9cBAMEk1gWSEx1kSbylg==",
-          "dev": true,
-          "requires": {
-            "locate-path": "^3.0.0"
-          }
-        },
-        "get-caller-file": {
-          "version": "2.0.5",
-          "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
-          "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==",
-          "dev": true
-        },
-        "is-fullwidth-code-point": {
-          "version": "2.0.0",
-          "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-2.0.0.tgz",
-          "integrity": "sha1-o7MKXE8ZkYMWeqq5O+764937ZU8=",
-          "dev": true
-        },
-        "locate-path": {
-          "version": "3.0.0",
-          "resolved": "https://registry.npmjs.org/locate-path/-/locate-path-3.0.0.tgz",
-          "integrity": "sha512-7AO748wWnIhNqAuaty2ZWHkQHRSNfPVIsPIfwEOWO22AmaoVrWavlOcMR5nzTLNYvp36X220/maaRsrec1G65A==",
-          "dev": true,
-          "requires": {
-            "p-locate": "^3.0.0",
-            "path-exists": "^3.0.0"
-          }
-        },
-        "p-locate": {
-          "version": "3.0.0",
-          "resolved": "https://registry.npmjs.org/p-locate/-/p-locate-3.0.0.tgz",
-          "integrity": "sha512-x+12w/To+4GFfgJhBEpiDcLozRJGegY+Ei7/z0tSLkMmxGZNybVMSfWj9aJn8Z5Fc7dBUNJOOVgPv2H7IwulSQ==",
-          "dev": true,
-          "requires": {
-            "p-limit": "^2.0.0"
-          }
-        },
-        "path-exists": {
-          "version": "3.0.0",
-          "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-3.0.0.tgz",
-          "integrity": "sha1-zg6+ql94yxiSXqfYENe1mwEP1RU=",
-          "dev": true
-        },
-        "require-main-filename": {
-          "version": "2.0.0",
-          "resolved": "https://registry.npmjs.org/require-main-filename/-/require-main-filename-2.0.0.tgz",
-          "integrity": "sha512-NKN5kMDylKuldxYLSUfrbo5Tuzh4hd+2E8NPPX02mZtn1VuREQToYe/ZdlJy+J3uCpfaiGF05e7B8W0iXbQHmg==",
-          "dev": true
-        },
-        "string-width": {
-          "version": "3.1.0",
-          "resolved": "https://registry.npmjs.org/string-width/-/string-width-3.1.0.tgz",
-          "integrity": "sha512-vafcv6KjVZKSgz06oM/H6GDBrAtz8vdhQakGjFIvNrHA6y3HCF1CInLy+QLq8dTJPQ1b+KDUqDFctkdRW44e1w==",
-          "dev": true,
-          "requires": {
-            "emoji-regex": "^7.0.1",
-            "is-fullwidth-code-point": "^2.0.0",
-            "strip-ansi": "^5.1.0"
-          }
-        },
-        "which-module": {
-          "version": "2.0.0",
-          "resolved": "https://registry.npmjs.org/which-module/-/which-module-2.0.0.tgz",
-          "integrity": "sha1-2e8H3Od7mQK4o6j6SzHD4/fm6Ho=",
-          "dev": true
-        },
-        "wrap-ansi": {
-          "version": "5.1.0",
-          "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-5.1.0.tgz",
-          "integrity": "sha512-QC1/iN/2/RPVJ5jYK8BGttj5z83LmSKmvbvrXPNCLZSEb32KKVDJDl/MOt2N01qU2H/FkzEa9PKto1BqDjtd7Q==",
-          "dev": true,
-          "requires": {
-            "ansi-styles": "^3.2.0",
-            "string-width": "^3.0.0",
-            "strip-ansi": "^5.0.0"
-          }
-        },
-        "y18n": {
+        "decamelize": {
          "version": "4.0.0",
-          "resolved": "https://registry.npmjs.org/y18n/-/y18n-4.0.0.tgz",
-          "integrity": "sha512-r9S/ZyXu/Xu9q1tYlpsLIsa3EeLXXk0VwlxqTcFRfg9EhMW+17kbt9G0NrgCmhGb5vT2hyhJZLfDGx+7+5Uj/w==",
+          "resolved": "https://registry.npmjs.org/decamelize/-/decamelize-4.0.0.tgz",
+          "integrity": "sha512-9iE1PgSik9HeIIw2JO94IidnE3eBoQrFJ3w7sFuzSX4DpmZ3v5sZpUiV5Swcf6mQEF+Y0ru8Neo+p+nyh2J+hQ==",
          "dev": true
-        },
-        "yargs": {
-          "version": "14.2.3",
-          "resolved": "https://registry.npmjs.org/yargs/-/yargs-14.2.3.tgz",
-          "integrity": "sha512-ZbotRWhF+lkjijC/VhmOT9wSgyBQ7+zr13+YLkhfsSiTriYsMzkTUFP18pFhWwBeMa5gUc1MzbhrO6/VB7c9Xg==",
-          "dev": true,
-          "requires": {
-            "cliui": "^5.0.0",
-            "decamelize": "^1.2.0",
-            "find-up": "^3.0.0",
-            "get-caller-file": "^2.0.1",
-            "require-directory": "^2.1.1",
-            "require-main-filename": "^2.0.0",
-            "set-blocking": "^2.0.0",
-            "string-width": "^3.0.0",
-            "which-module": "^2.0.0",
-            "y18n": "^4.0.0",
-            "yargs-parser": "^15.0.1"
-          }
-        },
-        "yargs-parser": {
-          "version": "15.0.1",
-          "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-15.0.1.tgz",
-          "integrity": "sha512-0OAMV2mAZQrs3FkNpDQcBk1x5HXb8X4twADss4S0Iuk+2dGnLOE/fRHrsYm542GduMveyA77OF4wrNJuanRCWw==",
-          "dev": true,
-          "requires": {
-            "camelcase": "^5.0.0",
-            "decamelize": "^1.2.0"
-          }
        }
      }
    },
@@ -10233,6 +10001,12 @@
      "integrity": "sha512-Ux4ygGWsu2c7isFWe8Yu1YluJmqVhxqK2cLXNQA5AcC3QfbGNpM7fu0Y8b/z16pXLnFxZYvWhd3fhBY9DLmC6Q==",
      "dev": true
    },
+    "yocto-queue": {
+      "version": "0.1.0",
+      "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz",
+      "integrity": "sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==",
+      "dev": true
+    },
    "zip-a-folder": {
      "version": "0.0.12",
      "resolved": "https://registry.npmjs.org/zip-a-folder/-/zip-a-folder-0.0.12.tgz",
--- a/extensions/ql-vscode/package.json
+++ b/extensions/ql-vscode/package.json
@@ -4,7 +4,7 @@
  "description": "CodeQL for Visual Studio Code",
  "author": "GitHub",
  "private": true,
-  "version": "1.3.7",
+  "version": "1.3.10",
  "publisher": "GitHub",
  "license": "MIT",
  "icon": "media/VS-marketplace-CodeQL-icon.png",
@@ -304,7 +304,7 @@
      },
      {
        "command": "codeQLQueryHistory.openQuery",
-        "title": "Open Query",
+        "title": "Open the query that produced these results",
        "icon": {
          "light": "media/light/edit.svg",
          "dark": "media/dark/edit.svg"
@@ -526,7 +526,8 @@
        },
        {
          "command": "codeQL.runQueries",
-          "group": "9_qlCommands"
+          "group": "9_qlCommands",
+          "when": "resourceScheme != codeql-zip-archive"
        }
      ],
      "commandPalette": [
@@ -656,6 +657,10 @@
          "command": "codeQL.runQuery",
          "when": "editorLangId == ql && resourceExtname == .ql"
        },
+        {
+          "command": "codeQL.viewAst",
+          "when": "resourceScheme == codeql-zip-archive"
+        },
        {
          "command": "codeQL.quickEval",
          "when": "editorLangId == ql"
@@ -690,7 +695,7 @@
    "viewsWelcome": [
      {
        "view": "codeQLAstViewer",
-        "contents": "Run the 'CodeQL: View AST' command on an open source file from a Code QL database.\n[View AST](command:codeQL.viewAst)"
+        "contents": "Run the 'CodeQL: View AST' command on an open source file from a CodeQL database.\n[View AST](command:codeQL.viewAst)"
      },
      {
        "view": "codeQLQueryHistory",
@@ -698,7 +703,7 @@
      },
      {
        "view": "codeQLDatabases",
-        "contents": "Add a Code QL database:\n[From a folder](command:codeQLDatabases.chooseDatabaseFolder)\n[From an archive](command:codeQLDatabases.chooseDatabaseArchive)\n[From a URL (as a zip file)](command:codeQLDatabases.chooseDatabaseInternet)\n[From LGTM](command:codeQLDatabases.chooseDatabaseLgtm)"
+        "contents": "Add a CodeQL database:\n[From a folder](command:codeQLDatabases.chooseDatabaseFolder)\n[From an archive](command:codeQLDatabases.chooseDatabaseArchive)\n[From a URL (as a zip file)](command:codeQLDatabases.chooseDatabaseInternet)\n[From LGTM](command:codeQLDatabases.chooseDatabaseLgtm)"
      }
    ]
  },
@@ -708,7 +713,8 @@
    "watch:extension": "tsc --watch",
    "test": "mocha --exit -r ts-node/register test/pure-tests/**/*.ts",
    "preintegration": "rm -rf ./out/vscode-tests && gulp",
-    "integration": "node ./out/vscode-tests/run-integration-tests.js",
+    "integration": "node ./out/vscode-tests/run-integration-tests.js no-workspace,minimal-workspace",
+    "cli-integration": "npm run preintegration && node ./out/vscode-tests/run-integration-tests.js cli-integration",
    "update-vscode": "node ./node_modules/vscode/bin/install",
    "format": "tsfmt -r && eslint src test --ext .ts,.tsx --fix",
    "lint": "eslint src test --ext .ts,.tsx --max-warnings=0",
@@ -740,14 +746,14 @@
    "@types/chai-as-promised": "~7.1.2",
    "@types/child-process-promise": "^2.2.1",
    "@types/classnames": "~2.2.9",
-    "@types/fs-extra": "^9.0.3",
+    "@types/fs-extra": "^9.0.6",
    "@types/glob": "^7.1.1",
    "@types/google-protobuf": "^3.2.7",
    "@types/gulp": "^4.0.6",
    "@types/gulp-sourcemaps": "0.0.32",
    "@types/js-yaml": "^3.12.5",
    "@types/jszip": "~3.1.6",
-    "@types/mocha": "~8.0.3",
+    "@types/mocha": "^8.0.4",
    "@types/node": "^12.14.1",
    "@types/node-fetch": "~2.5.2",
    "@types/proxyquire": "~1.3.28",
@@ -778,7 +784,7 @@
    "husky": "~4.2.5",
    "jsonc-parser": "^2.3.0",
    "lint-staged": "~10.2.2",
-    "mocha": "~8.1.3",
+    "mocha": "^8.2.1",
    "mocha-sinon": "~2.1.0",
    "npm-run-all": "^4.1.5",
    "prettier": "~2.0.5",
--- a/extensions/ql-vscode/src/astViewer.ts
+++ b/extensions/ql-vscode/src/astViewer.ts
@@ -18,7 +18,7 @@ import { DatabaseItem } from './databases';
 import { UrlValue, BqrsId } from './pure/bqrs-cli-types';
 import { showLocation } from './interface-utils';
 import { isStringLoc, isWholeFileLoc, isLineColumnLoc } from './pure/bqrs-utils';
-import { commandRunner } from './helpers';
+import { commandRunner } from './commandRunner';
 import { DisposableObject } from './vscode-utils/disposable-object';

 export interface AstItem {
@@ -40,7 +40,7 @@ class AstViewerDataProvider extends DisposableObject implements TreeDataProvider
  public db: DatabaseItem | undefined;

  private _onDidChangeTreeData =
-    new EventEmitter<AstItem | undefined>();
+    this.push(new EventEmitter<AstItem | undefined>());
  readonly onDidChangeTreeData: Event<AstItem | undefined> =
    this._onDidChangeTreeData.event;

--- a/extensions/ql-vscode/src/cli.ts
+++ b/extensions/ql-vscode/src/cli.ts
@@ -50,6 +50,7 @@ export interface DbInfo {
  sourceArchiveRoot: string;
  datasetFolder: string;
  logsFolder: string;
+  languages: string[];
 }

 /**
@@ -123,6 +124,11 @@ export class CodeQLCliServer implements Disposable {
   */
  private static CLI_VERSION_WITH_DECOMPILE_KIND_DIL = new SemVer('2.3.0');

+  /**
+   * CLI version where languages are exposed during a `codeql resolve database` command.
+   */
+  private static CLI_VERSION_WITH_LANGUAGE = new SemVer('2.4.1');
+
  /** The process for the cli server, or undefined if one doesn't exist yet */
  process?: child_process.ChildProcessWithoutNullStreams;
  /** Queue of future commands*/
@@ -133,15 +139,20 @@ export class CodeQLCliServer implements Disposable {
  nullBuffer: Buffer;

  /** Version of current cli, lazily computed by the `getVersion()` method */
-  _version: SemVer | undefined;
+  private _version: SemVer | undefined;

  /** Path to current codeQL executable, or undefined if not running yet. */
  codeQlPath: string | undefined;

+  /**
+   * When set to true, ignore some modal popups and assume user has clicked "yes".
+   */
+  public quiet = false;
+
  constructor(
    private distributionProvider: DistributionProvider,
    private cliConfig: CliConfig,
-    private logger: Logger,
+    private logger: Logger
  ) {
    this.commandQueue = [];
    this.commandInProcess = false;
@@ -256,7 +267,7 @@ export class CodeQLCliServer implements Disposable {
      const argsString = args.join(' ');
      this.logger.log(`${description} using CodeQL CLI: ${argsString}...`);
      try {
-        await new Promise((resolve, reject) => {
+        await new Promise<void>((resolve, reject) => {
          // Start listening to stdout
          process.stdout.addListener('data', (newData: Buffer) => {
            stdoutBuffers.push(newData);
@@ -689,7 +700,7 @@ export class CodeQLCliServer implements Disposable {
  }

  async generateDil(qloFile: string, outFile: string): Promise<void> {
-    const extraArgs = (await this.getVersion()).compare(CodeQLCliServer.CLI_VERSION_WITH_DECOMPILE_KIND_DIL) >= 0
+    const extraArgs = await this.supportsDecompileDil()
      ? ['--kind', 'dil', '-o', outFile, qloFile]
      : ['-o', outFile, qloFile];
    await this.runCodeQlCliCommand(
@@ -699,13 +710,21 @@ export class CodeQLCliServer implements Disposable {
    );
  }

-  private async getVersion() {
+  public async getVersion() {
    if (!this._version) {
      this._version = await this.refreshVersion();
    }
    return this._version;
  }

+  private async supportsDecompileDil() {
+    return (await this.getVersion()).compare(CodeQLCliServer.CLI_VERSION_WITH_DECOMPILE_KIND_DIL) >= 0;
+  }
+
+  public async supportsLanguageName() {
+    return (await this.getVersion()).compare(CodeQLCliServer.CLI_VERSION_WITH_LANGUAGE) >= 0;
+  }
+
  private async refreshVersion() {
    const distribution = await this.distributionProvider.getDistribution();
    switch (distribution.kind) {
--- a/extensions/ql-vscode/src/commandRunner.ts
+++ b/extensions/ql-vscode/src/commandRunner.ts
@@ -0,0 +1,227 @@
+import {
+  CancellationToken,
+  ProgressOptions,
+  window as Window,
+  commands,
+  Disposable,
+  ProgressLocation
+} from 'vscode';
+import { showAndLogErrorMessage, showAndLogWarningMessage } from './helpers';
+import { logger } from './logging';
+
+export class UserCancellationException extends Error {
+  /**
+   * @param message The error message
+   * @param silent If silent is true, then this exception will avoid showing a warning message to the user.
+   */
+  constructor(message?: string, public readonly silent = false) {
+    super(message);
+  }
+}
+
+export interface ProgressUpdate {
+  /**
+   * The current step
+   */
+  step: number;
+  /**
+   * The maximum step. This *should* be constant for a single job.
+   */
+  maxStep: number;
+  /**
+   * The current progress message
+   */
+  message: string;
+}
+
+export type ProgressCallback = (p: ProgressUpdate) => void;
+
+/**
+ * A task that handles command invocations from `commandRunner`
+ * and includes a progress monitor.
+ *
+ *
+ * Arguments passed to the command handler are passed along,
+ * untouched to this `ProgressTask` instance.
+ *
+ * @param progress a progress handler function. Call this
+ * function with a `ProgressUpdate` instance in order to
+ * denote some progress being achieved on this task.
+ * @param token a cencellation token
+ * @param args arguments passed to this task passed on from
+ * `commands.registerCommand`.
+ */
+export type ProgressTask<R> = (
+  progress: ProgressCallback,
+  token: CancellationToken,
+  ...args: any[]
+) => Thenable<R>;
+
+/**
+ * A task that handles command invocations from `commandRunner`.
+ * Arguments passed to the command handler are passed along,
+ * untouched to this `NoProgressTask` instance.
+ *
+ * @param args arguments passed to this task passed on from
+ * `commands.registerCommand`.
+ */
+type NoProgressTask = ((...args: any[]) => Promise<any>);
+
+/**
+ * This mediates between the kind of progress callbacks we want to
+ * write (where we *set* current progress position and give
+ * `maxSteps`) and the kind vscode progress api expects us to write
+ * (which increment progress by a certain amount out of 100%).
+ *
+ * Where possible, the `commandRunner` function below should be used
+ * instead of this function. The commandRunner is meant for wrapping
+ * top-level commands and provides error handling and other support
+ * automatically.
+ *
+ * Only use this function if you need a progress monitor and the
+ * control flow does not always come from a command (eg- during
+ * extension activation, or from an internal language server
+ * request).
+ */
+export function withProgress<R>(
+  options: ProgressOptions,
+  task: ProgressTask<R>,
+  ...args: any[]
+): Thenable<R> {
+  let progressAchieved = 0;
+  return Window.withProgress(options,
+    (progress, token) => {
+      return task(p => {
+        const { message, step, maxStep } = p;
+        const increment = 100 * (step - progressAchieved) / maxStep;
+        progressAchieved = step;
+        progress.report({ message, increment });
+      }, token, ...args);
+    });
+}
+
+/**
+ * A generic wrapper for command registration. This wrapper adds uniform error handling for commands.
+ *
+ * In this variant of the command runner, no progress monitor is used.
+ *
+ * @param commandId The ID of the command to register.
+ * @param task The task to run. It is passed directly to `commands.registerCommand`. Any
+ * arguments to the command handler are passed on to the task.
+ */
+export function commandRunner(
+  commandId: string,
+  task: NoProgressTask,
+): Disposable {
+  return commands.registerCommand(commandId, async (...args: any[]) => {
+    try {
+      return await task(...args);
+    } catch (e) {
+      const errorMessage = `${e.message || e} (${commandId})`;
+      if (e instanceof UserCancellationException) {
+        // User has cancelled this action manually
+        if (e.silent) {
+          logger.log(errorMessage);
+        } else {
+          showAndLogWarningMessage(errorMessage);
+        }
+      } else {
+        // Include the full stack in the error log only.
+        const fullMessage = e.stack
+          ? `${errorMessage}\n${e.stack}`
+          : errorMessage;
+        showAndLogErrorMessage(errorMessage, {
+          fullMessage
+        });
+      }
+      return undefined;
+    }
+  });
+}
+
+/**
+ * A generic wrapper for command registration.  This wrapper adds uniform error handling,
+ * progress monitoring, and cancellation for commands.
+ *
+ * @param commandId The ID of the command to register.
+ * @param task The task to run. It is passed directly to `commands.registerCommand`. Any
+ * arguments to the command handler are passed on to the task after the progress callback
+ * and cancellation token.
+ * @param progressOptions Progress options to be sent to the progress monitor.
+ */
+export function commandRunnerWithProgress<R>(
+  commandId: string,
+  task: ProgressTask<R>,
+  progressOptions: Partial<ProgressOptions>
+): Disposable {
+  return commands.registerCommand(commandId, async (...args: any[]) => {
+    const progressOptionsWithDefaults = {
+      location: ProgressLocation.Notification,
+      ...progressOptions
+    };
+    try {
+      return await withProgress(progressOptionsWithDefaults, task, ...args);
+    } catch (e) {
+      const errorMessage = `${e.message || e} (${commandId})`;
+      if (e instanceof UserCancellationException) {
+        // User has cancelled this action manually
+        if (e.silent) {
+          logger.log(errorMessage);
+        } else {
+          showAndLogWarningMessage(errorMessage);
+        }
+      } else {
+        // Include the full stack in the error log only.
+        const fullMessage = e.stack
+          ? `${errorMessage}\n${e.stack}`
+          : errorMessage;
+        showAndLogErrorMessage(errorMessage, {
+          fullMessage
+        });
+      }
+      return undefined;
+    }
+  });
+}
+
+/**
+ * Displays a progress monitor that indicates how much progess has been made
+ * reading from a stream.
+ *
+ * @param readable The stream to read progress from
+ * @param messagePrefix A prefix for displaying the message
+ * @param totalNumBytes Total number of bytes in this stream
+ * @param progress The progress callback used to set messages
+ */
+export function reportStreamProgress(
+  readable: NodeJS.ReadableStream,
+  messagePrefix: string,
+  totalNumBytes?: number,
+  progress?: ProgressCallback
+) {
+  if (progress && totalNumBytes) {
+    let numBytesDownloaded = 0;
+    const bytesToDisplayMB = (numBytes: number): string => `${(numBytes / (1024 * 1024)).toFixed(1)} MB`;
+    const updateProgress = () => {
+      progress({
+        step: numBytesDownloaded,
+        maxStep: totalNumBytes,
+        message: `${messagePrefix} [${bytesToDisplayMB(numBytesDownloaded)} of ${bytesToDisplayMB(totalNumBytes)}]`,
+      });
+    };
+
+    // Display the progress straight away rather than waiting for the first chunk.
+    updateProgress();
+
+    readable.on('data', data => {
+      numBytesDownloaded += data.length;
+      updateProgress();
+    });
+  } else if (progress) {
+    progress({
+      step: 1,
+      maxStep: 2,
+      message: `${messagePrefix} (Size unknown)`,
+    });
+  }
+}
--- a/extensions/ql-vscode/src/config.ts
+++ b/extensions/ql-vscode/src/config.ts
@@ -52,7 +52,8 @@ const QUERY_HISTORY_FORMAT_SETTING = new Setting('format', QUERY_HISTORY_SETTING
 const DISTRIBUTION_CHANGE_SETTINGS = [CUSTOM_CODEQL_PATH_SETTING, INCLUDE_PRERELEASE_SETTING, PERSONAL_ACCESS_TOKEN_SETTING];

 export interface DistributionConfig {
-  customCodeQlPath?: string;
+  readonly customCodeQlPath?: string;
+  updateCustomCodeQlPath: (newPath: string | undefined) => Promise<void>;
  includePrerelease: boolean;
  personalAccessToken?: string;
  ownerName?: string;
@@ -149,6 +150,10 @@ export class DistributionConfigListener extends ConfigListener implements Distri
    return PERSONAL_ACCESS_TOKEN_SETTING.getValue() || undefined;
  }

+  public async updateCustomCodeQlPath(newPath: string | undefined) {
+    await CUSTOM_CODEQL_PATH_SETTING.updateValue(newPath, ConfigurationTarget.Global);
+  }
+
  protected handleDidChangeConfiguration(e: ConfigurationChangeEvent): void {
    this.handleDidChangeConfigurationForRelevantSettings(DISTRIBUTION_CHANGE_SETTINGS, e);
  }
--- a/extensions/ql-vscode/src/contextual/locationFinder.ts
+++ b/extensions/ql-vscode/src/contextual/locationFinder.ts
@@ -8,7 +8,7 @@ import fileRangeFromURI from './fileRangeFromURI';
 import * as messages from '../pure/messages';
 import { QueryServerClient } from '../queryserver-client';
 import { QueryWithResults, compileAndRunQueryAgainstDatabase } from '../run-queries';
-import { ProgressCallback } from '../helpers';
+import { ProgressCallback } from '../commandRunner';
 import { KeyType } from './keyType';
 import { qlpackOfDatabase, resolveQueries } from './queryResolver';

@@ -52,9 +52,6 @@ export async function getLocationsForUriString(
  }

  const qlpack = await qlpackOfDatabase(cli, db);
-  if (qlpack === undefined) {
-    throw new Error('Can\'t infer qlpack from database source archive');
-  }
  const templates = createTemplates(uri.pathWithinSourceArchive);

  const links: FullLocationLink[] = [];
--- a/extensions/ql-vscode/src/contextual/queryResolver.ts
+++ b/extensions/ql-vscode/src/contextual/queryResolver.ts
@@ -12,12 +12,13 @@ import {
 import { CodeQLCliServer } from '../cli';
 import { DatabaseItem } from '../databases';

-export async function qlpackOfDatabase(cli: CodeQLCliServer, db: DatabaseItem): Promise<string | undefined> {
-  if (db.contents === undefined)
-    return undefined;
+export async function qlpackOfDatabase(cli: CodeQLCliServer, db: DatabaseItem): Promise<string> {
+  if (db.contents === undefined) {
+    throw new Error('Database is invalid and cannot infer QLPack.');
+  }
  const datasetPath = db.contents.datasetUri.fsPath;
-  const { qlpack } = await helpers.resolveDatasetFolder(cli, datasetPath);
-  return qlpack;
+  const dbscheme = await helpers.getPrimaryDbscheme(datasetPath);
+  return await helpers.getQlPackForDbscheme(cli, dbscheme);
 }


--- a/extensions/ql-vscode/src/contextual/templateProvider.ts
+++ b/extensions/ql-vscode/src/contextual/templateProvider.ts
@@ -3,7 +3,8 @@ import * as vscode from 'vscode';
 import { decodeSourceArchiveUri, encodeArchiveBasePath, zipArchiveScheme } from '../archive-filesystem-provider';
 import { CodeQLCliServer } from '../cli';
 import { DatabaseManager } from '../databases';
-import { CachedOperation, ProgressCallback, withProgress } from '../helpers';
+import { CachedOperation } from '../helpers';
+import { ProgressCallback, withProgress } from '../commandRunner';
 import * as messages from '../pure/messages';
 import { QueryServerClient } from '../queryserver-client';
 import { compileAndRunQueryAgainstDatabase, QueryWithResults } from '../run-queries';
@@ -156,9 +157,6 @@ export class TemplatePrintAstProvider {
    }

    const qlpack = await qlpackOfDatabase(this.cli, db);
-    if (!qlpack) {
-      throw new Error('Can\'t infer qlpack from database source archive');
-    }
    const queries = await resolveQueries(this.cli, qlpack, KeyType.PrintAstQuery);
    if (queries.length > 1) {
      throw new Error('Found multiple Print AST queries. Can\'t continue');
--- a/extensions/ql-vscode/src/databaseFetcher.ts
+++ b/extensions/ql-vscode/src/databaseFetcher.ts
@@ -12,22 +12,26 @@ import * as path from 'path';

 import { DatabaseManager, DatabaseItem } from './databases';
 import {
-  ProgressCallback,
  showAndLogInformationMessage,
 } from './helpers';
+import {
+  reportStreamProgress,
+  ProgressCallback,
+} from './commandRunner';
 import { logger } from './logging';
+import { tmpDir } from './run-queries';

 /**
 * Prompts a user to fetch a database from a remote location. Database is assumed to be an archive file.
 *
- * @param databasesManager the DatabaseManager
+ * @param databaseManager the DatabaseManager
 * @param storagePath where to store the unzipped database.
 */
 export async function promptImportInternetDatabase(
-  databasesManager: DatabaseManager,
+  databaseManager: DatabaseManager,
  storagePath: string,
  progress: ProgressCallback,
-  _: CancellationToken,
+  token: CancellationToken,
 ): Promise<DatabaseItem | undefined> {
  const databaseUrl = await window.showInputBox({
    prompt: 'Enter URL of zipfile of database to download',
@@ -40,9 +44,10 @@ export async function promptImportInternetDatabase(

  const item = await databaseArchiveFetcher(
    databaseUrl,
-    databasesManager,
+    databaseManager,
    storagePath,
-    progress
+    progress,
+    token
  );

  if (item) {
@@ -58,14 +63,14 @@ export async function promptImportInternetDatabase(
 * User enters a project url and then the user is asked which language
 * to download (if there is more than one)
 *
- * @param databasesManager the DatabaseManager
+ * @param databaseManager the DatabaseManager
 * @param storagePath where to store the unzipped database.
 */
 export async function promptImportLgtmDatabase(
-  databasesManager: DatabaseManager,
+  databaseManager: DatabaseManager,
  storagePath: string,
  progress: ProgressCallback,
-  _: CancellationToken
+  token: CancellationToken
 ): Promise<DatabaseItem | undefined> {
  const lgtmUrl = await window.showInputBox({
    prompt:
@@ -80,9 +85,10 @@ export async function promptImportLgtmDatabase(
    if (databaseUrl) {
      const item = await databaseArchiveFetcher(
        databaseUrl,
-        databasesManager,
+        databaseManager,
        storagePath,
-        progress
+        progress,
+        token
      );
      if (item) {
        commands.executeCommand('codeQLDatabases.focus');
@@ -100,22 +106,23 @@ export async function promptImportLgtmDatabase(
 * Imports a database from a local archive.
 *
 * @param databaseUrl the file url of the archive to import
- * @param databasesManager the DatabaseManager
+ * @param databaseManager the DatabaseManager
 * @param storagePath where to store the unzipped database.
 */
 export async function importArchiveDatabase(
  databaseUrl: string,
-  databasesManager: DatabaseManager,
+  databaseManager: DatabaseManager,
  storagePath: string,
  progress: ProgressCallback,
-  _: CancellationToken,
+  token: CancellationToken,
 ): Promise<DatabaseItem | undefined> {
  try {
    const item = await databaseArchiveFetcher(
      databaseUrl,
-      databasesManager,
+      databaseManager,
      storagePath,
-      progress
+      progress,
+      token
    );
    if (item) {
      commands.executeCommand('codeQLDatabases.focus');
@@ -137,17 +144,19 @@ export async function importArchiveDatabase(
 * or in the local filesystem.
 *
 * @param databaseUrl URL from which to grab the database
- * @param databasesManager the DatabaseManager
+ * @param databaseManager the DatabaseManager
 * @param storagePath where to store the unzipped database.
- * @param progressCallback optional callback to send progress messages to
+ * @param progress callback to send progress messages to
+ * @param token cancellation token
 */
 async function databaseArchiveFetcher(
  databaseUrl: string,
-  databasesManager: DatabaseManager,
+  databaseManager: DatabaseManager,
  storagePath: string,
-  progressCallback?: ProgressCallback
+  progress: ProgressCallback,
+  token: CancellationToken
 ): Promise<DatabaseItem> {
-  progressCallback?.({
+  progress({
    message: 'Getting database',
    step: 1,
    maxStep: 4,
@@ -159,12 +168,12 @@ async function databaseArchiveFetcher(
  const unzipPath = await getStorageFolder(storagePath, databaseUrl);

  if (isFile(databaseUrl)) {
-    await readAndUnzip(databaseUrl, unzipPath);
+    await readAndUnzip(databaseUrl, unzipPath, progress);
  } else {
-    await fetchAndUnzip(databaseUrl, unzipPath, progressCallback);
+    await fetchAndUnzip(databaseUrl, unzipPath, progress);
  }

-  progressCallback?.({
+  progress({
    message: 'Opening database',
    step: 3,
    maxStep: 4,
@@ -177,15 +186,15 @@ async function databaseArchiveFetcher(
    'codeql-database.yml'
  );
  if (dbPath) {
-    progressCallback?.({
+    progress({
      message: 'Validating and fixing source location',
      step: 4,
      maxStep: 4,
    });
    await ensureZippedSourceLocation(dbPath);

-    const item = await databasesManager.openDatabase(Uri.file(dbPath));
-    databasesManager.setCurrentDatabaseItem(item);
+    const item = await databaseManager.openDatabase(progress, token, Uri.file(dbPath));
+    await databaseManager.setCurrentDatabaseItem(item);
    return item;
  } else {
    throw new Error('Database not found in archive.');
@@ -232,48 +241,67 @@ function validateHttpsUrl(databaseUrl: string) {
  }
 }

-async function readAndUnzip(databaseUrl: string, unzipPath: string) {
-  const databaseFile = Uri.parse(databaseUrl).fsPath;
-  const directory = await unzipper.Open.file(databaseFile);
+async function readAndUnzip(
+  zipUrl: string,
+  unzipPath: string,
+  progress?: ProgressCallback
+) {
+  // TODO: Providing progress as the file is unzipped is currently blocked
+  // on https://github.com/ZJONSSON/node-unzipper/issues/222
+  const zipFile = Uri.parse(zipUrl).fsPath;
+  progress?.({
+    maxStep: 10,
+    step: 9,
+    message: `Unzipping into ${path.basename(unzipPath)}`
+  });
+  // Must get the zip central directory since streaming the
+  // zip contents may not have correct local file headers.
+  // Instead, we can only rely on the central directory.
+  const directory = await unzipper.Open.file(zipFile);
  await directory.extract({ path: unzipPath });
 }

 async function fetchAndUnzip(
  databaseUrl: string,
  unzipPath: string,
-  progressCallback?: ProgressCallback
+  progress?: ProgressCallback
 ) {
-  const response = await fetch(databaseUrl);
+  // Although it is possible to download and stream directly to an unzipped directory,
+  // we need to avoid this for two reasons. The central directory is located at the
+  // end of the zip file. It is the source of truth of the content locations. Individual
+  // file headers may be incorrect. Additionally, saving to file first will reduce memory
+  // pressure compared with unzipping while downloading the archive.

-  await checkForFailingResponse(response);
+  const archivePath = path.join(tmpDir.name, `archive-${Date.now()}.zip`);

-  const unzipStream = unzipper.Extract({
-    path: unzipPath,
-  });
-
-  progressCallback?.({
+  progress?.({
    maxStep: 3,
-    message: 'Unzipping database',
-    step: 2,
-  });
-  await new Promise((resolve, reject) => {
-    const handler = (err: Error) => {
-      if (err.message.startsWith('invalid signature')) {
-        reject(new Error('Not a valid archive.'));
-      } else {
-        reject(err);
-      }
-    };
-    response.body.on('error', handler);
-    unzipStream.on('error', handler);
-    unzipStream.on('close', resolve);
-    response.body.pipe(unzipStream);
+    message: 'Downloading database',
+    step: 1,
  });
+
+  const response = await checkForFailingResponse(await fetch(databaseUrl));
+  const archiveFileStream = fs.createWriteStream(archivePath);
+
+  const contentLength = response.headers.get('content-length');
+  const totalNumBytes = contentLength ? parseInt(contentLength, 10) : undefined;
+  reportStreamProgress(response.body, 'Downloading database', totalNumBytes, progress);
+
+  await new Promise((resolve, reject) =>
+    response.body.pipe(archiveFileStream)
+      .on('finish', resolve)
+      .on('error', reject)
+  );
+
+  await readAndUnzip(Uri.file(archivePath).toString(true), unzipPath, progress);
+
+  // remove archivePath eagerly since these archives can be large.
+  await fs.remove(archivePath);
 }

-async function checkForFailingResponse(response: Response): Promise<void | never> {
+async function checkForFailingResponse(response: Response): Promise<Response | never> {
  if (response.ok) {
-    return;
+    return response;
  }

  // An error downloading the database. Attempt to extract the resaon behind it.
--- a/extensions/ql-vscode/src/databases-ui.ts
+++ b/extensions/ql-vscode/src/databases-ui.ts
@@ -18,15 +18,17 @@ import {
  DatabaseItem,
  DatabaseManager,
  getUpgradesDirectories,
-  isLikelyDatabaseRoot,
-  isLikelyDbLanguageFolder,
 } from './databases';
 import {
  commandRunner,
  commandRunnerWithProgress,
-  getOnDiskWorkspaceFolders,
  ProgressCallback,
-  showAndLogErrorMessage
+} from './commandRunner';
+import {
+  getOnDiskWorkspaceFolders,
+  showAndLogErrorMessage,
+  isLikelyDatabaseRoot,
+  isLikelyDbLanguageFolder
 } from './helpers';
 import { logger } from './logging';
 import { clearCacheInDatabase } from './run-queries';
@@ -81,7 +83,7 @@ class DatabaseTreeDataProvider extends DisposableObject
  implements TreeDataProvider<DatabaseItem> {
  private _sortOrder = SortOrder.NameAsc;

-  private readonly _onDidChangeTreeData = new EventEmitter<DatabaseItem | undefined>();
+  private readonly _onDidChangeTreeData = this.push(new EventEmitter<DatabaseItem | undefined>());
  private currentDatabaseItem: DatabaseItem | undefined;

  constructor(
@@ -143,6 +145,7 @@ class DatabaseTreeDataProvider extends DisposableObject
      );
    }
    item.tooltip = element.databaseUri.fsPath;
+    item.description = element.language;
    return item;
  }

@@ -318,9 +321,13 @@ export class DatabaseUI extends DisposableObject {
      )
    );
    this.push(
-      commandRunner(
+      commandRunnerWithProgress(
        'codeQLDatabases.removeDatabase',
-        this.handleRemoveDatabase
+        this.handleRemoveDatabase,
+        {
+          title: 'Removing database',
+          cancellable: false
+        }
      )
    );
    this.push(
@@ -373,7 +380,17 @@ export class DatabaseUI extends DisposableObject {

  handleRemoveOrphanedDatabases = async (): Promise<void> => {
    logger.log('Removing orphaned databases from workspace storage.');
-    let dbDirs =
+    let dbDirs = undefined;
+
+    if (
+      !(await fs.pathExists(this.storagePath) ||
+        !(await fs.stat(this.storagePath)).isDirectory())
+    ) {
+      logger.log('Missing or invalid storage directory. Not trying to remove orphaned databases.');
+      return;
+    }
+
+    dbDirs =
      // read directory
      (await fs.readdir(this.storagePath, { withFileTypes: true }))
        // remove non-directories
@@ -580,7 +597,7 @@ export class DatabaseUI extends DisposableObject {
          token
        );
      } else {
-        await this.setCurrentDatabase(uri);
+        await this.setCurrentDatabase(progress, token, uri);
      }
    } catch (e) {
      // rethrow and let this be handled by default error handling.
@@ -593,15 +610,17 @@ export class DatabaseUI extends DisposableObject {
  };

  private handleRemoveDatabase = async (
+    progress: ProgressCallback,
+    token: CancellationToken,
    databaseItem: DatabaseItem,
    multiSelect: DatabaseItem[] | undefined
  ): Promise<void> => {
    if (multiSelect?.length) {
-      multiSelect.forEach((dbItem) =>
-        this.databaseManager.removeDatabaseItem(dbItem)
-      );
+      await Promise.all(multiSelect.map((dbItem) =>
+        this.databaseManager.removeDatabaseItem(progress, token, dbItem)
+      ));
    } else {
-      this.databaseManager.removeDatabaseItem(databaseItem);
+      await this.databaseManager.removeDatabaseItem(progress, token, databaseItem);
    }
  };

@@ -651,11 +670,13 @@ export class DatabaseUI extends DisposableObject {
  }

  private async setCurrentDatabase(
+    progress: ProgressCallback,
+    token: CancellationToken,
    uri: Uri
  ): Promise<DatabaseItem | undefined> {
    let databaseItem = this.databaseManager.findDatabaseItem(uri);
    if (databaseItem === undefined) {
-      databaseItem = await this.databaseManager.openDatabase(uri);
+      databaseItem = await this.databaseManager.openDatabase(progress, token, uri);
    }
    await this.databaseManager.setCurrentDatabaseItem(databaseItem);

@@ -680,7 +701,7 @@ export class DatabaseUI extends DisposableObject {
    if (byFolder) {
      const fixedUri = await this.fixDbUri(uri);
      // we are selecting a database folder
-      return await this.setCurrentDatabase(fixedUri);
+      return await this.setCurrentDatabase(progress, token, fixedUri);
    } else {
      // we are selecting a database archive. Must unzip into a workspace-controlled area
      // before importing.
--- a/extensions/ql-vscode/src/databases.ts
+++ b/extensions/ql-vscode/src/databases.ts
@@ -4,11 +4,21 @@ import * as path from 'path';
 import * as vscode from 'vscode';
 import * as cli from './cli';
 import { ExtensionContext } from 'vscode';
-import { showAndLogErrorMessage, showAndLogWarningMessage, showAndLogInformationMessage } from './helpers';
+import {
+  showAndLogErrorMessage,
+  showAndLogWarningMessage,
+  showAndLogInformationMessage,
+  isLikelyDatabaseRoot,
+} from './helpers';
+import {
+  ProgressCallback,
+  withProgress
+} from './commandRunner';
 import { zipArchiveScheme, encodeArchiveBasePath, decodeSourceArchiveUri, encodeSourceArchiveUri } from './archive-filesystem-provider';
 import { DisposableObject } from './vscode-utils/disposable-object';
-import { QueryServerConfig } from './config';
 import { Logger, logger } from './logging';
+import { registerDatabases, Dataset, deregisterDatabases } from './pure/messages';
+import { QueryServerClient } from './queryserver-client';

 /**
 * databases.ts
@@ -36,11 +46,13 @@ export interface DatabaseOptions {
  displayName?: string;
  ignoreSourceArchive?: boolean;
  dateAdded?: number | undefined;
+  language?: string;
 }

-interface FullDatabaseOptions extends DatabaseOptions {
+export interface FullDatabaseOptions extends DatabaseOptions {
  ignoreSourceArchive: boolean;
  dateAdded: number | undefined;
+  language: string | undefined;
 }

 interface PersistedDatabaseItem {
@@ -193,6 +205,9 @@ export interface DatabaseItem {
  readonly databaseUri: vscode.Uri;
  /** The name of the database to be displayed in the UI */
  name: string;
+
+  /** The primary language of the database or empty string if unknown */
+  readonly language: string;
  /** The URI of the database's source archive, or `undefined` if no source archive is to be used. */
  readonly sourceArchive: vscode.Uri | undefined;
  /**
@@ -249,6 +264,11 @@ export interface DatabaseItem {
   * Holds if `uri` belongs to this database's source archive.
   */
  belongsToSourceArchiveExplorerUri(uri: vscode.Uri): boolean;
+
+  /**
+   * Gets the state of this database, to be persisted in the workspace state.
+   */
+  getPersistedState(): PersistedDatabaseItem;
 }

 export enum DatabaseEventKind {
@@ -427,6 +447,10 @@ export class DatabaseItemImpl implements DatabaseItem {
    return dbInfo.datasetFolder;
  }

+  public get language() {
+    return this.options.language || '';
+  }
+
  /**
   * Returns the root uri of the virtual filesystem for this database's source archive.
   */
@@ -479,12 +503,13 @@ export class DatabaseManager extends DisposableObject {
  private readonly _onDidChangeCurrentDatabaseItem = this.push(new vscode.EventEmitter<DatabaseChangedEvent>());
  readonly onDidChangeCurrentDatabaseItem = this._onDidChangeCurrentDatabaseItem.event;

-  private readonly _databaseItems: DatabaseItemImpl[] = [];
+  private readonly _databaseItems: DatabaseItem[] = [];
  private _currentDatabaseItem: DatabaseItem | undefined = undefined;

  constructor(
-    private ctx: ExtensionContext,
-    public config: QueryServerConfig,
+    private readonly ctx: ExtensionContext,
+    private readonly qs: QueryServerClient,
+    private readonly cli: cli.CodeQLCliServer,
    public logger: Logger
  ) {
    super();
@@ -493,23 +518,25 @@ export class DatabaseManager extends DisposableObject {
  }

  public async openDatabase(
-    uri: vscode.Uri, options?: DatabaseOptions
+    progress: ProgressCallback,
+    token: vscode.CancellationToken,
+    uri: vscode.Uri,
  ): Promise<DatabaseItem> {
-
    const contents = await resolveDatabaseContents(uri);
-    const realOptions = options || {};
    // Ignore the source archive for QLTest databases by default.
    const isQLTestDatabase = path.extname(uri.fsPath) === '.testproj';
    const fullOptions: FullDatabaseOptions = {
-      ignoreSourceArchive: (realOptions.ignoreSourceArchive !== undefined) ?
-        realOptions.ignoreSourceArchive : isQLTestDatabase,
-      displayName: realOptions.displayName,
-      dateAdded: realOptions.dateAdded || Date.now()
+      ignoreSourceArchive: isQLTestDatabase,
+      // displayName is only set if a user explicitly renames a database
+      displayName: undefined,
+      dateAdded: Date.now(),
+      language: await this.getPrimaryLanguage(uri.fsPath)
    };
    const databaseItem = new DatabaseItemImpl(uri, contents, fullOptions, (event) => {
      this._onDidChangeDatabaseItem.fire(event);
    });
-    await this.addDatabaseItem(databaseItem);
+
+    await this.addDatabaseItem(progress, token, databaseItem);
    await this.addDatabaseSourceArchiveFolder(databaseItem);

    return databaseItem;
@@ -555,12 +582,15 @@ export class DatabaseManager extends DisposableObject {
  }

  private async createDatabaseItemFromPersistedState(
+    progress: ProgressCallback,
+    token: vscode.CancellationToken,
    state: PersistedDatabaseItem
  ): Promise<DatabaseItem> {

    let displayName: string | undefined = undefined;
    let ignoreSourceArchive = false;
    let dateAdded = undefined;
+    let language = undefined;
    if (state.options) {
      if (typeof state.options.displayName === 'string') {
        displayName = state.options.displayName;
@@ -571,43 +601,69 @@ export class DatabaseManager extends DisposableObject {
      if (typeof state.options.dateAdded === 'number') {
        dateAdded = state.options.dateAdded;
      }
+      language = state.options.language;
    }
+
+    const dbBaseUri = vscode.Uri.parse(state.uri, true);
+    if (language === undefined) {
+      // we haven't been successful yet at getting the language. try again
+      language = await this.getPrimaryLanguage(dbBaseUri.fsPath);
+    }
+
    const fullOptions: FullDatabaseOptions = {
      ignoreSourceArchive,
      displayName,
-      dateAdded
+      dateAdded,
+      language
    };
-    const item = new DatabaseItemImpl(vscode.Uri.parse(state.uri, true), undefined, fullOptions,
+    const item = new DatabaseItemImpl(dbBaseUri, undefined, fullOptions,
      (event) => {
        this._onDidChangeDatabaseItem.fire(event);
      });
-    await this.addDatabaseItem(item);

+    await this.addDatabaseItem(progress, token, item);
    return item;
  }

  private async loadPersistedState(): Promise<void> {
-    const currentDatabaseUri = this.ctx.workspaceState.get<string>(CURRENT_DB);
-    const databases = this.ctx.workspaceState.get<PersistedDatabaseItem[]>(DB_LIST, []);
-
-    try {
-      for (const database of databases) {
-        const databaseItem = await this.createDatabaseItemFromPersistedState(database);
+    return withProgress({
+      location: vscode.ProgressLocation.Notification
+    },
+      async (progress, token) => {
+        const currentDatabaseUri = this.ctx.workspaceState.get<string>(CURRENT_DB);
+        const databases = this.ctx.workspaceState.get<PersistedDatabaseItem[]>(DB_LIST, []);
+        let step = 0;
+        progress({
+          maxStep: databases.length,
+          message: 'Loading persisted databases',
+          step
+        });
        try {
-          await databaseItem.refresh();
-          if (currentDatabaseUri === database.uri) {
-            this.setCurrentDatabaseItem(databaseItem, true);
+          for (const database of databases) {
+            progress({
+              maxStep: databases.length,
+              message: `Loading ${database.options?.displayName || 'databases'}`,
+              step: ++step
+            });
+
+            const databaseItem = await this.createDatabaseItemFromPersistedState(progress, token, database);
+            try {
+              await databaseItem.refresh();
+              await this.registerDatabase(progress, token, databaseItem);
+              if (currentDatabaseUri === database.uri) {
+                this.setCurrentDatabaseItem(databaseItem, true);
+              }
+            }
+            catch (e) {
+              // When loading from persisted state, leave invalid databases in the list. They will be
+              // marked as invalid, and cannot be set as the current database.
+            }
          }
+        } catch (e) {
+          // database list had an unexpected type - nothing to be done?
+          showAndLogErrorMessage(`Database list loading failed: ${e.message}`);
        }
-        catch (e) {
-          // When loading from persisted state, leave invalid databases in the list. They will be
-          // marked as invalid, and cannot be set as the current database.
-        }
-      }
-    } catch (e) {
-      // database list had an unexpected type - nothing to be done?
-      showAndLogErrorMessage(`Database list loading failed: ${e.message}`);
-    }
+      });
  }

  public get databaseItems(): readonly DatabaseItem[] {
@@ -618,8 +674,10 @@ export class DatabaseManager extends DisposableObject {
    return this._currentDatabaseItem;
  }

-  public async setCurrentDatabaseItem(item: DatabaseItem | undefined,
-    skipRefresh = false): Promise<void> {
+  public async setCurrentDatabaseItem(
+    item: DatabaseItem | undefined,
+    skipRefresh = false
+  ): Promise<void> {

    if (!skipRefresh && (item !== undefined)) {
      await item.refresh();  // Will throw on invalid database.
@@ -627,6 +685,7 @@ export class DatabaseManager extends DisposableObject {
    if (this._currentDatabaseItem !== item) {
      this._currentDatabaseItem = item;
      this.updatePersistedCurrentDatabaseItem();
+
      this._onDidChangeCurrentDatabaseItem.fire({
        item,
        kind: DatabaseEventKind.Change
@@ -653,9 +712,20 @@ export class DatabaseManager extends DisposableObject {
    return this._databaseItems.find(item => item.sourceArchive && item.sourceArchive.toString(true) === uriString);
  }

-  private async addDatabaseItem(item: DatabaseItemImpl) {
+  private async addDatabaseItem(
+    progress: ProgressCallback,
+    token: vscode.CancellationToken,
+    item: DatabaseItem
+  ) {
    this._databaseItems.push(item);
    this.updatePersistedDatabaseList();
+
+    // Add this database item to the allow-list
+    // Database items reconstituted from persisted state
+    // will not have their contents yet.
+    if (item.contents?.datasetUri) {
+      await this.registerDatabase(progress, token, item);
+    }
    // note that we use undefined as the item in order to reset the entire tree
    this._onDidChangeDatabaseItem.fire({
      item: undefined,
@@ -673,9 +743,14 @@ export class DatabaseManager extends DisposableObject {
    });
  }

-  public removeDatabaseItem(item: DatabaseItem) {
-    if (this._currentDatabaseItem == item)
+  public async removeDatabaseItem(
+    progress: ProgressCallback,
+    token: vscode.CancellationToken,
+    item: DatabaseItem
+  ) {
+    if (this._currentDatabaseItem == item) {
      this._currentDatabaseItem = undefined;
+    }
    const index = this.databaseItems.findIndex(searchItem => searchItem === item);
    if (index >= 0) {
      this._databaseItems.splice(index, 1);
@@ -683,8 +758,10 @@ export class DatabaseManager extends DisposableObject {
    this.updatePersistedDatabaseList();

    // Delete folder from workspace, if it is still there
-    const folderIndex = (vscode.workspace.workspaceFolders || []).findIndex(folder => item.belongsToSourceArchiveExplorerUri(folder.uri));
-    if (index >= 0) {
+    const folderIndex = (vscode.workspace.workspaceFolders || []).findIndex(
+      folder => item.belongsToSourceArchiveExplorerUri(folder.uri)
+    );
+    if (folderIndex >= 0) {
      logger.log(`Removing workspace folder at index ${folderIndex}`);
      vscode.workspace.updateWorkspaceFolders(folderIndex, 1);
    }
@@ -692,11 +769,14 @@ export class DatabaseManager extends DisposableObject {
    // Delete folder from file system only if it is controlled by the extension
    if (this.isExtensionControlledLocation(item.databaseUri)) {
      logger.log('Deleting database from filesystem.');
-      fs.remove(item.databaseUri.path).then(
-        () => logger.log(`Deleted '${item.databaseUri.path}'`),
-        e => logger.log(`Failed to delete '${item.databaseUri.path}'. Reason: ${e.message}`));
+      fs.remove(item.databaseUri.fsPath).then(
+        () => logger.log(`Deleted '${item.databaseUri.fsPath}'`),
+        e => logger.log(`Failed to delete '${item.databaseUri.fsPath}'. Reason: ${e.message}`));
    }

+    // Remove this database item from the allow-list
+    await this.deregisterDatabase(progress, token, item);
+
    // note that we use undefined as the item in order to reset the entire tree
    this._onDidChangeDatabaseItem.fire({
      item: undefined,
@@ -704,6 +784,34 @@ export class DatabaseManager extends DisposableObject {
    });
  }

+  private async deregisterDatabase(
+    progress: ProgressCallback,
+    token: vscode.CancellationToken,
+    dbItem: DatabaseItem,
+  ) {
+    if (dbItem.contents && (await this.qs.supportsDatabaseRegistration())) {
+      const databases: Dataset[] = [{
+        dbDir: dbItem.contents.datasetUri.fsPath,
+        workingSet: 'default'
+      }];
+      await this.qs.sendRequest(deregisterDatabases, { databases }, token, progress);
+    }
+  }
+
+  private async registerDatabase(
+    progress: ProgressCallback,
+    token: vscode.CancellationToken,
+    dbItem: DatabaseItem,
+  ) {
+    if (dbItem.contents && (await this.qs.supportsDatabaseRegistration())) {
+      const databases: Dataset[] = [{
+        dbDir: dbItem.contents.datasetUri.fsPath,
+        workingSet: 'default'
+      }];
+      await this.qs.sendRequest(registerDatabases, { databases }, token, progress);
+    }
+  }
+
  private updatePersistedCurrentDatabaseItem(): void {
    this.ctx.workspaceState.update(CURRENT_DB, this._currentDatabaseItem ?
      this._currentDatabaseItem.databaseUri.toString(true) : undefined);
@@ -715,7 +823,24 @@ export class DatabaseManager extends DisposableObject {

  private isExtensionControlledLocation(uri: vscode.Uri) {
    const storagePath = this.ctx.storagePath || this.ctx.globalStoragePath;
-    return uri.path.startsWith(storagePath);
+    // the uri.fsPath function on windows returns a lowercase drive letter,
+    // but storagePath will have an uppercase drive letter. Be sure to compare
+    // URIs to URIs only
+    if (storagePath) {
+      return uri.fsPath.startsWith(vscode.Uri.file(storagePath).fsPath);
+    }
+    return false;
+  }
+
+  private async getPrimaryLanguage(dbPath: string) {
+    if (!(await this.cli.supportsLanguageName())) {
+      // return undefined so that we recalculate on restart until the cli is at a version that
+      // supports this feature. This recalculation is cheap since we avoid calling into the cli
+      // unless we know it can return the langauges property.
+      return undefined;
+    }
+    const dbInfo = await this.cli.resolveDatabase(dbPath);
+    return dbInfo.languages?.[0] || '';
  }
 }

@@ -728,23 +853,3 @@ export function getUpgradesDirectories(scripts: string[]): vscode.Uri[] {
  const uniqueParentDirs = new Set(parentDirs);
  return Array.from(uniqueParentDirs).map(filePath => vscode.Uri.file(filePath));
 }
-
-
-// TODO: Get the list of supported languages from a list that will be auto-updated.
-
-export async function isLikelyDatabaseRoot(fsPath: string) {
-  const [a, b, c] = (await Promise.all([
-    // databases can have either .dbinfo or codeql-database.yml.
-    fs.pathExists(path.join(fsPath, '.dbinfo')),
-    fs.pathExists(path.join(fsPath, 'codeql-database.yml')),
-
-    // they *must* have a db-language folder
-    (await fs.readdir(fsPath)).some(isLikelyDbLanguageFolder)
-  ]));
-
-  return (a || b) && c;
-}
-
-export function isLikelyDbLanguageFolder(dbPath: string) {
-  return !!path.basename(dbPath).startsWith('db-');
-}
--- a/extensions/ql-vscode/src/discovery.ts
+++ b/extensions/ql-vscode/src/discovery.ts
@@ -1,5 +1,5 @@
 import { DisposableObject } from './vscode-utils/disposable-object';
-import { showAndLogErrorMessage } from './helpers';
+import { logger } from './logging';

 /**
 * Base class for "discovery" operations, which scan the file system to find specific kinds of
@@ -62,7 +62,7 @@ export abstract class Discovery<T> extends DisposableObject {
    });

    discoveryPromise.catch(err => {
-      showAndLogErrorMessage(`${this.name} failed. Reason: ${err.message}`);
+      logger.log(`${this.name} failed. Reason: ${err.message}`);
    });

    discoveryPromise.finally(() => {
--- a/extensions/ql-vscode/src/distribution.ts
+++ b/extensions/ql-vscode/src/distribution.ts
@@ -7,10 +7,15 @@ import * as unzipper from 'unzipper';
 import * as url from 'url';
 import { ExtensionContext, Event } from 'vscode';
 import { DistributionConfig } from './config';
-import { InvocationRateLimiter, InvocationRateLimiterResultKind, showAndLogErrorMessage } from './helpers';
+import {
+  InvocationRateLimiter,
+  InvocationRateLimiterResultKind,
+  showAndLogErrorMessage,
+  showAndLogWarningMessage
+} from './helpers';
 import { logger } from './logging';
-import * as helpers from './helpers';
 import { getCodeQlCliVersion } from './cli-version';
+import { ProgressCallback, reportStreamProgress } from './commandRunner';

 /**
 * distribution.ts
@@ -49,16 +54,36 @@ export interface DistributionProvider {
 }

 export class DistributionManager implements DistributionProvider {
-  constructor(extensionContext: ExtensionContext, config: DistributionConfig, versionRange: semver.Range) {
-    this._config = config;
-    this._extensionSpecificDistributionManager = new ExtensionSpecificDistributionManager(extensionContext, config, versionRange);
+
+  /**
+   * Get the name of the codeql cli installation we prefer to install, based on our current platform.
+   */
+  public static getRequiredAssetName(): string {
+    switch (os.platform()) {
+      case 'linux':
+        return 'codeql-linux64.zip';
+      case 'darwin':
+        return 'codeql-osx64.zip';
+      case 'win32':
+        return 'codeql-win64.zip';
+      default:
+        return 'codeql.zip';
+    }
+  }
+
+  constructor(
+    public readonly config: DistributionConfig,
+    private readonly versionRange: semver.Range,
+    extensionContext: ExtensionContext
+  ) {
    this._onDidChangeDistribution = config.onDidChangeConfiguration;
-    this._updateCheckRateLimiter = new InvocationRateLimiter(
+    this.extensionSpecificDistributionManager =
+      new ExtensionSpecificDistributionManager(config, versionRange, extensionContext);
+    this.updateCheckRateLimiter = new InvocationRateLimiter(
      extensionContext,
      'extensionSpecificDistributionUpdateCheck',
-      () => this._extensionSpecificDistributionManager.checkForUpdatesToDistribution()
+      () => this.extensionSpecificDistributionManager.checkForUpdatesToDistribution()
    );
-    this._versionRange = versionRange;
  }

  /**
@@ -95,9 +120,9 @@ export class DistributionManager implements DistributionProvider {
     * - If the user is using an extension-managed CLI, then prereleases are only accepted when the
     * includePrerelease config option is set.
     */
-    const includePrerelease = distribution.kind !== DistributionKind.ExtensionManaged || this._config.includePrerelease;
+    const includePrerelease = distribution.kind !== DistributionKind.ExtensionManaged || this.config.includePrerelease;

-    if (!semver.satisfies(version, this._versionRange, { includePrerelease })) {
+    if (!semver.satisfies(version, this.versionRange, { includePrerelease })) {
      return {
        distribution,
        kind: FindDistributionResultKind.IncompatibleDistribution,
@@ -126,9 +151,9 @@ export class DistributionManager implements DistributionProvider {
   */
  async getDistributionWithoutVersionCheck(): Promise<Distribution | undefined> {
    // Check config setting, then extension specific distribution, then PATH.
-    if (this._config.customCodeQlPath) {
-      if (!await fs.pathExists(this._config.customCodeQlPath)) {
-        showAndLogErrorMessage(`The CodeQL executable path is specified as "${this._config.customCodeQlPath}" ` +
+    if (this.config.customCodeQlPath) {
+      if (!await fs.pathExists(this.config.customCodeQlPath)) {
+        showAndLogErrorMessage(`The CodeQL executable path is specified as "${this.config.customCodeQlPath}" ` +
          'by a configuration setting, but a CodeQL executable could not be found at that path. Please check ' +
          'that a CodeQL executable exists at the specified path or remove the setting.');
        return undefined;
@@ -137,18 +162,18 @@ export class DistributionManager implements DistributionProvider {
      // emit a warning if using a deprecated launcher and a non-deprecated launcher exists
      if (
        deprecatedCodeQlLauncherName() &&
-        this._config.customCodeQlPath.endsWith(deprecatedCodeQlLauncherName()!) &&
+        this.config.customCodeQlPath.endsWith(deprecatedCodeQlLauncherName()!) &&
        await this.hasNewLauncherName()
      ) {
        warnDeprecatedLauncher();
      }
      return {
-        codeQlPath: this._config.customCodeQlPath,
+        codeQlPath: this.config.customCodeQlPath,
        kind: DistributionKind.CustomPathConfig
      };
    }

-    const extensionSpecificCodeQlPath = await this._extensionSpecificDistributionManager.getCodeQlPathWithoutVersionCheck();
+    const extensionSpecificCodeQlPath = await this.extensionSpecificDistributionManager.getCodeQlPathWithoutVersionCheck();
    if (extensionSpecificCodeQlPath !== undefined) {
      return {
        codeQlPath: extensionSpecificCodeQlPath,
@@ -181,12 +206,12 @@ export class DistributionManager implements DistributionProvider {
  public async checkForUpdatesToExtensionManagedDistribution(
    minSecondsSinceLastUpdateCheck: number): Promise<DistributionUpdateCheckResult> {
    const distribution = await this.getDistributionWithoutVersionCheck();
-    const extensionManagedCodeQlPath = await this._extensionSpecificDistributionManager.getCodeQlPathWithoutVersionCheck();
+    const extensionManagedCodeQlPath = await this.extensionSpecificDistributionManager.getCodeQlPathWithoutVersionCheck();
    if (distribution?.codeQlPath !== extensionManagedCodeQlPath) {
      // A distribution is present but it isn't managed by the extension.
      return createInvalidLocationResult();
    }
-    const updateCheckResult = await this._updateCheckRateLimiter.invokeFunctionIfIntervalElapsed(minSecondsSinceLastUpdateCheck);
+    const updateCheckResult = await this.updateCheckRateLimiter.invokeFunctionIfIntervalElapsed(minSecondsSinceLastUpdateCheck);
    switch (updateCheckResult.kind) {
      case InvocationRateLimiterResultKind.Invoked:
        return updateCheckResult.result;
@@ -201,8 +226,8 @@ export class DistributionManager implements DistributionProvider {
   * Returns a failed promise if an unexpected error occurs during installation.
   */
  public installExtensionManagedDistributionRelease(release: Release,
-    progressCallback?: helpers.ProgressCallback): Promise<void> {
-    return this._extensionSpecificDistributionManager.installDistributionRelease(release, progressCallback);
+    progressCallback?: ProgressCallback): Promise<void> {
+    return this.extensionSpecificDistributionManager!.installDistributionRelease(release, progressCallback);
  }

  public get onDidChangeDistribution(): Event<void> | undefined {
@@ -215,27 +240,27 @@ export class DistributionManager implements DistributionProvider {
   * installation. False otherwise.
   */
  private async hasNewLauncherName(): Promise<boolean> {
-    if (!this._config.customCodeQlPath) {
+    if (!this.config.customCodeQlPath) {
      // not managed externally
      return false;
    }
-    const dir = path.dirname(this._config.customCodeQlPath);
+    const dir = path.dirname(this.config.customCodeQlPath);
    const newLaunderPath = path.join(dir, codeQlLauncherName());
    return await fs.pathExists(newLaunderPath);
  }

-  private readonly _config: DistributionConfig;
-  private readonly _extensionSpecificDistributionManager: ExtensionSpecificDistributionManager;
-  private readonly _updateCheckRateLimiter: InvocationRateLimiter<DistributionUpdateCheckResult>;
+  private readonly extensionSpecificDistributionManager: ExtensionSpecificDistributionManager;
+  private readonly updateCheckRateLimiter: InvocationRateLimiter<DistributionUpdateCheckResult>;
  private readonly _onDidChangeDistribution: Event<void> | undefined;
-  private readonly _versionRange: semver.Range;
 }

 class ExtensionSpecificDistributionManager {
-  constructor(extensionContext: ExtensionContext, config: DistributionConfig, versionRange: semver.Range) {
-    this._extensionContext = extensionContext;
-    this._config = config;
-    this._versionRange = versionRange;
+  constructor(
+    private readonly config: DistributionConfig,
+    private readonly versionRange: semver.Range,
+    private readonly extensionContext: ExtensionContext
+  ) {
+    /**/
  }

  public async getCodeQlPathWithoutVersionCheck(): Promise<string | undefined> {
@@ -283,14 +308,14 @@ class ExtensionSpecificDistributionManager {
   * Returns a failed promise if an unexpected error occurs during installation.
   */
  public async installDistributionRelease(release: Release,
-    progressCallback?: helpers.ProgressCallback): Promise<void> {
+    progressCallback?: ProgressCallback): Promise<void> {
    await this.downloadDistribution(release, progressCallback);
    // Store the installed release within the global extension state.
    this.storeInstalledRelease(release);
  }

  private async downloadDistribution(release: Release,
-    progressCallback?: helpers.ProgressCallback): Promise<void> {
+    progressCallback?: ProgressCallback): Promise<void> {
    try {
      await this.removeDistribution();
    } catch (e) {
@@ -299,7 +324,7 @@ class ExtensionSpecificDistributionManager {
    }

    // Filter assets to the unique one that we require.
-    const requiredAssetName = this.getRequiredAssetName();
+    const requiredAssetName = DistributionManager.getRequiredAssetName();
    const assets = release.assets.filter(asset => asset.name === requiredAssetName);
    if (assets.length === 0) {
      throw new Error(`Invariant violation: chose a release to install that didn't have ${requiredAssetName}`);
@@ -317,27 +342,8 @@ class ExtensionSpecificDistributionManager {
      const archiveFile = fs.createWriteStream(archivePath);

      const contentLength = assetStream.headers.get('content-length');
-      let numBytesDownloaded = 0;
-
-      if (progressCallback && contentLength !== null) {
-        const totalNumBytes = parseInt(contentLength, 10);
-        const bytesToDisplayMB = (numBytes: number): string => `${(numBytes / (1024 * 1024)).toFixed(1)} MB`;
-        const updateProgress = (): void => {
-          progressCallback({
-            step: numBytesDownloaded,
-            maxStep: totalNumBytes,
-            message: `Downloading CodeQL CLI… [${bytesToDisplayMB(numBytesDownloaded)} of ${bytesToDisplayMB(totalNumBytes)}]`,
-          });
-        };
-
-        // Display the progress straight away rather than waiting for the first chunk.
-        updateProgress();
-
-        assetStream.body.on('data', data => {
-          numBytesDownloaded += data.length;
-          updateProgress();
-        });
-      }
+      const totalNumBytes = contentLength ? parseInt(contentLength, 10) : undefined;
+      reportStreamProgress(assetStream.body, 'Downloading CodeQL CLI…', totalNumBytes, progressCallback);

      await new Promise((resolve, reject) =>
        assetStream.body.pipe(archiveFile)
@@ -366,22 +372,12 @@ class ExtensionSpecificDistributionManager {
    }
  }

-  /**
-   * Get the name of the codeql cli installation we prefer to install, based on our current platform.
-   */
-  private getRequiredAssetName(): string {
-    if (os.platform() === 'linux') return 'codeql-linux64.zip';
-    if (os.platform() === 'darwin') return 'codeql-osx64.zip';
-    if (os.platform() === 'win32') return 'codeql-win64.zip';
-    return 'codeql.zip';
-  }
-
  private async getLatestRelease(): Promise<Release> {
-    const requiredAssetName = this.getRequiredAssetName();
+    const requiredAssetName = DistributionManager.getRequiredAssetName();
    logger.log(`Searching for latest release including ${requiredAssetName}.`);
    return this.createReleasesApiConsumer().getLatestRelease(
-      this._versionRange,
-      this._config.includePrerelease,
+      this.versionRange,
+      this.config.includePrerelease,
      release => {
        const matchingAssets = release.assets.filter(asset => asset.name === requiredAssetName);
        if (matchingAssets.length === 0) {
@@ -399,23 +395,23 @@ class ExtensionSpecificDistributionManager {
  }

  private createReleasesApiConsumer(): ReleasesApiConsumer {
-    const ownerName = this._config.ownerName ? this._config.ownerName : DEFAULT_DISTRIBUTION_OWNER_NAME;
-    const repositoryName = this._config.repositoryName ? this._config.repositoryName : DEFAULT_DISTRIBUTION_REPOSITORY_NAME;
-    return new ReleasesApiConsumer(ownerName, repositoryName, this._config.personalAccessToken);
+    const ownerName = this.config.ownerName ? this.config.ownerName : DEFAULT_DISTRIBUTION_OWNER_NAME;
+    const repositoryName = this.config.repositoryName ? this.config.repositoryName : DEFAULT_DISTRIBUTION_REPOSITORY_NAME;
+    return new ReleasesApiConsumer(ownerName, repositoryName, this.config.personalAccessToken);
  }

  private async bumpDistributionFolderIndex(): Promise<void> {
-    const index = this._extensionContext.globalState.get(
+    const index = this.extensionContext.globalState.get(
      ExtensionSpecificDistributionManager._currentDistributionFolderIndexStateKey, 0);
-    await this._extensionContext.globalState.update(
+    await this.extensionContext.globalState.update(
      ExtensionSpecificDistributionManager._currentDistributionFolderIndexStateKey, index + 1);
  }

  private getDistributionStoragePath(): string {
    // Use an empty string for the initial distribution for backwards compatibility.
-    const distributionFolderIndex = this._extensionContext.globalState.get(
+    const distributionFolderIndex = this.extensionContext.globalState.get(
      ExtensionSpecificDistributionManager._currentDistributionFolderIndexStateKey, 0) || '';
-    return path.join(this._extensionContext.globalStoragePath,
+    return path.join(this.extensionContext.globalStoragePath,
      ExtensionSpecificDistributionManager._currentDistributionFolderBaseName + distributionFolderIndex);
  }

@@ -425,17 +421,13 @@ class ExtensionSpecificDistributionManager {
  }

  private getInstalledRelease(): Release | undefined {
-    return this._extensionContext.globalState.get(ExtensionSpecificDistributionManager._installedReleaseStateKey);
+    return this.extensionContext.globalState.get(ExtensionSpecificDistributionManager._installedReleaseStateKey);
  }

  private async storeInstalledRelease(release: Release | undefined): Promise<void> {
-    await this._extensionContext.globalState.update(ExtensionSpecificDistributionManager._installedReleaseStateKey, release);
+    await this.extensionContext.globalState.update(ExtensionSpecificDistributionManager._installedReleaseStateKey, release);
  }

-  private readonly _config: DistributionConfig;
-  private readonly _extensionContext: ExtensionContext;
-  private readonly _versionRange: semver.Range;
-
  private static readonly _currentDistributionFolderBaseName = 'distribution';
  private static readonly _currentDistributionFolderIndexStateKey = 'distributionFolderIndex';
  private static readonly _installedReleaseStateKey = 'distributionRelease';
@@ -576,7 +568,7 @@ export async function extractZipArchive(archivePath: string, outPath: string): P
  }));
 }

-function codeQlLauncherName(): string {
+export function codeQlLauncherName(): string {
  return (os.platform() === 'win32') ? 'codeql.exe' : 'codeql';
 }

@@ -720,7 +712,9 @@ export async function getExecutableFromDirectory(directory: string, warnWhenNotF
 }

 function warnDeprecatedLauncher() {
-  helpers.showAndLogWarningMessage(
+
+  showAndLogWarningMessage(
+
    `The "${deprecatedCodeQlLauncherName()!}" launcher has been deprecated and will be removed in a future version. ` +
    `Please use "${codeQlLauncherName()}" instead. It is recommended to update to the latest CodeQL binaries.`
  );
--- a/extensions/ql-vscode/src/extension.ts
+++ b/extensions/ql-vscode/src/extension.ts
@@ -45,6 +45,7 @@ import {
  GithubRateLimitedError
 } from './distribution';
 import * as helpers from './helpers';
+import { commandRunner, commandRunnerWithProgress, ProgressCallback, ProgressUpdate, withProgress } from './commandRunner';
 import { assertNever } from './pure/helpers-pure';
 import { spawnIdeServer } from './ide-server';
 import { InterfaceManager } from './interface';
@@ -108,21 +109,45 @@ function registerErrorStubs(excludedCommands: string[], stubGenerator: (command:

  stubbedCommands.forEach(command => {
    if (excludedCommands.indexOf(command) === -1) {
-      errorStubs.push(helpers.commandRunner(command, stubGenerator(command)));
+      errorStubs.push(commandRunner(command, stubGenerator(command)));
    }
  });
 }

-export async function activate(ctx: ExtensionContext): Promise<void> {
+/**
+ * The publicly available interface for this extension. This is to
+ * be used in our tests.
+ */
+export interface CodeQLExtensionInterface {
+  readonly ctx: ExtensionContext;
+  readonly cliServer: CodeQLCliServer;
+  readonly qs: qsClient.QueryServerClient;
+  readonly distributionManager: DistributionManager;
+  readonly databaseManager: DatabaseManager;
+  readonly databaseUI: DatabaseUI;
+  readonly dispose: () => void;
+}
+
+/**
+ * Returns the CodeQLExtensionInterface, or an empty object if the interface is not
+ * available afer activation is complete. This will happen if there is no cli
+ * installed when the extension starts. Downloading and installing the cli
+ * will happen at a later time.
+ *
+ * @param ctx The extension context
+ *
+ * @returns CodeQLExtensionInterface
+ */
+export async function activate(ctx: ExtensionContext): Promise<CodeQLExtensionInterface | {}> {
  logger.log('Starting CodeQL extension');

+  const distributionConfigListener = new DistributionConfigListener();
  initializeLogging(ctx);
  languageSupport.install();

-  const distributionConfigListener = new DistributionConfigListener();
  ctx.subscriptions.push(distributionConfigListener);
  const codeQlVersionRange = DEFAULT_DISTRIBUTION_VERSION_RANGE;
-  const distributionManager = new DistributionManager(ctx, distributionConfigListener, codeQlVersionRange);
+  const distributionManager = new DistributionManager(distributionConfigListener, codeQlVersionRange, ctx);

  const shouldUpdateOnNextActivationKey = 'shouldUpdateOnNextActivation';

@@ -170,7 +195,7 @@ export async function activate(ctx: ExtensionContext): Promise<void> {
            location: ProgressLocation.Notification,
          };

-          await helpers.withProgress(progressOptions, progress =>
+          await withProgress(progressOptions, progress =>
            distributionManager.installExtensionManagedDistributionRelease(result.updatedRelease, progress));

          await ctx.globalState.update(shouldUpdateOnNextActivationKey, false);
@@ -253,14 +278,14 @@ export async function activate(ctx: ExtensionContext): Promise<void> {
    return result;
  }

-  async function installOrUpdateThenTryActivate(config: DistributionUpdateConfig): Promise<void> {
+  async function installOrUpdateThenTryActivate(config: DistributionUpdateConfig): Promise<CodeQLExtensionInterface | {}> {
    await installOrUpdateDistribution(config);

    // Display the warnings even if the extension has already activated.
    const distributionResult = await getDistributionDisplayingDistributionWarnings();
-
+    let extensionInterface: CodeQLExtensionInterface | {} = {};
    if (!beganMainExtensionActivation && distributionResult.kind !== FindDistributionResultKind.NoDistribution) {
-      await activateWithInstalledDistribution(ctx, distributionManager);
+      extensionInterface = await activateWithInstalledDistribution(ctx, distributionManager);
    } else if (distributionResult.kind === FindDistributionResultKind.NoDistribution) {
      registerErrorStubs([checkForUpdatesCommand], command => async () => {
        const installActionName = 'Install CodeQL CLI';
@@ -268,7 +293,7 @@ export async function activate(ctx: ExtensionContext): Promise<void> {
          items: [installActionName]
        });
        if (chosenAction === installActionName) {
-          installOrUpdateThenTryActivate({
+          await installOrUpdateThenTryActivate({
            isUserInitiated: true,
            shouldDisplayMessageWhenNoUpdates: false,
            allowAutoUpdating: true
@@ -276,6 +301,7 @@ export async function activate(ctx: ExtensionContext): Promise<void> {
        }
      });
    }
+    return extensionInterface;
  }

  ctx.subscriptions.push(distributionConfigListener.onDidChangeConfiguration(() => installOrUpdateThenTryActivate({
@@ -283,13 +309,13 @@ export async function activate(ctx: ExtensionContext): Promise<void> {
    shouldDisplayMessageWhenNoUpdates: false,
    allowAutoUpdating: true
  })));
-  ctx.subscriptions.push(helpers.commandRunner(checkForUpdatesCommand, () => installOrUpdateThenTryActivate({
+  ctx.subscriptions.push(commandRunner(checkForUpdatesCommand, () => installOrUpdateThenTryActivate({
    isUserInitiated: true,
    shouldDisplayMessageWhenNoUpdates: true,
    allowAutoUpdating: true
  })));

-  await installOrUpdateThenTryActivate({
+  return await installOrUpdateThenTryActivate({
    isUserInitiated: !!ctx.globalState.get(shouldUpdateOnNextActivationKey),
    shouldDisplayMessageWhenNoUpdates: false,

@@ -302,7 +328,7 @@ export async function activate(ctx: ExtensionContext): Promise<void> {
 async function activateWithInstalledDistribution(
  ctx: ExtensionContext,
  distributionManager: DistributionManager
-): Promise<void> {
+): Promise<CodeQLExtensionInterface> {
  beganMainExtensionActivation = true;
  // Remove any error stubs command handlers left over from first part
  // of activation.
@@ -339,7 +365,7 @@ async function activateWithInstalledDistribution(
  await qs.startQueryServer();

  logger.log('Initializing database manager.');
-  const dbm = new DatabaseManager(ctx, qlConfigurationListener, logger);
+  const dbm = new DatabaseManager(ctx, qs, cliServer, logger);
  ctx.subscriptions.push(dbm);
  logger.log('Initializing database panel.');
  const databaseUI = new DatabaseUI(
@@ -354,6 +380,7 @@ async function activateWithInstalledDistribution(

  logger.log('Initializing query history manager.');
  const queryHistoryConfigurationListener = new QueryHistoryConfigListener();
+  ctx.subscriptions.push(queryHistoryConfigurationListener);
  const showResults = async (item: CompletedQuery) =>
    showResultsForCompletedQuery(item, WebviewReveal.Forced);

@@ -404,7 +431,7 @@ async function activateWithInstalledDistribution(
  async function compileAndRunQuery(
    quickEval: boolean,
    selectedQuery: Uri | undefined,
-    progress: helpers.ProgressCallback,
+    progress: ProgressCallback,
    token: CancellationToken,
  ): Promise<void> {
    if (qs !== undefined) {
@@ -426,7 +453,7 @@ async function activateWithInstalledDistribution(
      // The call to showResults potentially creates SARIF file;
      // Update the tree item context value to allow viewing that
      // SARIF file from context menu.
-      await qhm.updateTreeItemContextValue(item);
+      await qhm.refreshTreeView(item);
    }
  }

@@ -465,10 +492,10 @@ async function activateWithInstalledDistribution(

  logger.log('Registering top-level command palette commands.');
  ctx.subscriptions.push(
-    helpers.commandRunnerWithProgress(
+    commandRunnerWithProgress(
      'codeQL.runQuery',
      async (
-        progress: helpers.ProgressCallback,
+        progress: ProgressCallback,
        token: CancellationToken,
        uri: Uri | undefined
      ) => await compileAndRunQuery(false, uri, progress, token),
@@ -479,10 +506,10 @@ async function activateWithInstalledDistribution(
    )
  );
  ctx.subscriptions.push(
-    helpers.commandRunnerWithProgress(
+    commandRunnerWithProgress(
      'codeQL.runQueries',
      async (
-        progress: helpers.ProgressCallback,
+        progress: ProgressCallback,
        token: CancellationToken,
        _: Uri | undefined,
        multi: Uri[]
@@ -507,7 +534,7 @@ async function activateWithInstalledDistribution(

        // Use a wrapped progress so that messages appear with the queries remaining in it.
        let queriesRemaining = queryUris.length;
-        function wrappedProgress(update: helpers.ProgressUpdate) {
+        function wrappedProgress(update: ProgressUpdate) {
          const message = queriesRemaining > 1
            ? `${queriesRemaining} remaining. ${update.message}`
            : update.message;
@@ -543,10 +570,10 @@ async function activateWithInstalledDistribution(
      })
  );
  ctx.subscriptions.push(
-    helpers.commandRunnerWithProgress(
+    commandRunnerWithProgress(
      'codeQL.quickEval',
      async (
-        progress: helpers.ProgressCallback,
+        progress: ProgressCallback,
        token: CancellationToken,
        uri: Uri | undefined
      ) => await compileAndRunQuery(true, uri, progress, token),
@@ -556,8 +583,8 @@ async function activateWithInstalledDistribution(
      })
  );
  ctx.subscriptions.push(
-    helpers.commandRunnerWithProgress('codeQL.quickQuery', async (
-      progress: helpers.ProgressCallback,
+    commandRunnerWithProgress('codeQL.quickQuery', async (
+      progress: ProgressCallback,
      token: CancellationToken
    ) =>
      displayQuickQuery(ctx, cliServer, databaseUI, progress, token),
@@ -568,7 +595,7 @@ async function activateWithInstalledDistribution(
  );

  ctx.subscriptions.push(
-    helpers.commandRunner('codeQL.restartQueryServer', async () => {
+    commandRunner('codeQL.restartQueryServer', async () => {
      await qs.restartQueryServer();
      helpers.showAndLogInformationMessage('CodeQL Query Server restarted.', {
        outputLogger: queryServerLogger,
@@ -576,24 +603,24 @@ async function activateWithInstalledDistribution(
    })
  );
  ctx.subscriptions.push(
-    helpers.commandRunner('codeQL.chooseDatabaseFolder', (
-      progress: helpers.ProgressCallback,
+    commandRunner('codeQL.chooseDatabaseFolder', (
+      progress: ProgressCallback,
      token: CancellationToken
    ) =>
      databaseUI.handleChooseDatabaseFolder(progress, token)
    )
  );
  ctx.subscriptions.push(
-    helpers.commandRunner('codeQL.chooseDatabaseArchive', (
-      progress: helpers.ProgressCallback,
+    commandRunner('codeQL.chooseDatabaseArchive', (
+      progress: ProgressCallback,
      token: CancellationToken
    ) =>
      databaseUI.handleChooseDatabaseArchive(progress, token)
    )
  );
  ctx.subscriptions.push(
-    helpers.commandRunnerWithProgress('codeQL.chooseDatabaseLgtm', (
-      progress: helpers.ProgressCallback,
+    commandRunnerWithProgress('codeQL.chooseDatabaseLgtm', (
+      progress: ProgressCallback,
      token: CancellationToken
    ) =>
      databaseUI.handleChooseDatabaseLgtm(progress, token),
@@ -602,8 +629,8 @@ async function activateWithInstalledDistribution(
      })
  );
  ctx.subscriptions.push(
-    helpers.commandRunnerWithProgress('codeQL.chooseDatabaseInternet', (
-      progress: helpers.ProgressCallback,
+    commandRunnerWithProgress('codeQL.chooseDatabaseInternet', (
+      progress: ProgressCallback,
      token: CancellationToken
    ) =>
      databaseUI.handleChooseDatabaseInternet(progress, token),
@@ -630,8 +657,8 @@ async function activateWithInstalledDistribution(

  const astViewer = new AstViewer();
  ctx.subscriptions.push(astViewer);
-  ctx.subscriptions.push(helpers.commandRunnerWithProgress('codeQL.viewAst', async (
-    progress: helpers.ProgressCallback,
+  ctx.subscriptions.push(commandRunnerWithProgress('codeQL.viewAst', async (
+    progress: ProgressCallback,
    token: CancellationToken
  ) => {
    const ast = await new TemplatePrintAstProvider(cliServer, qs, dbm, progress, token)
@@ -647,6 +674,18 @@ async function activateWithInstalledDistribution(
  commands.executeCommand('codeQLDatabases.removeOrphanedDatabases');

  logger.log('Successfully finished extension initialization.');
+
+  return {
+    ctx,
+    cliServer,
+    qs,
+    distributionManager,
+    databaseManager: dbm,
+    databaseUI,
+    dispose: () => {
+      ctx.subscriptions.forEach(d => d.dispose());
+    }
+  };
 }

 function getContextStoragePath(ctx: ExtensionContext) {
--- a/extensions/ql-vscode/src/helpers.ts
+++ b/extensions/ql-vscode/src/helpers.ts
@@ -3,191 +3,31 @@ import * as glob from 'glob-promise';
 import * as yaml from 'js-yaml';
 import * as path from 'path';
 import {
-  CancellationToken,
  ExtensionContext,
-  ProgressOptions,
  window as Window,
-  workspace,
-  commands,
-  Disposable,
-  ProgressLocation
+  workspace
 } from 'vscode';
 import { CodeQLCliServer } from './cli';
 import { logger } from './logging';

-export class UserCancellationException extends Error {
-  /**
-   * @param message The error message
-   * @param silent If silent is true, then this exception will avoid showing a warning message to the user.
-   */
-  constructor(message?: string, public readonly silent = false) {
-    super(message);
-  }
-}
-
-export interface ProgressUpdate {
-  /**
-   * The current step
-   */
-  step: number;
-  /**
-   * The maximum step. This *should* be constant for a single job.
-   */
-  maxStep: number;
-  /**
-   * The current progress message
-   */
-  message: string;
-}
-
-export type ProgressCallback = (p: ProgressUpdate) => void;
-
-/**
- * A task that handles command invocations from `commandRunner`
- * and includes a progress monitor.
- *
- *
- * Arguments passed to the command handler are passed along,
- * untouched to this `ProgressTask` instance.
- *
- * @param progress a progress handler function. Call this
- * function with a `ProgressUpdate` instance in order to
- * denote some progress being achieved on this task.
- * @param token a cencellation token
- * @param args arguments passed to this task passed on from
- * `commands.registerCommand`.
- */
-export type ProgressTask<R> = (
-  progress: ProgressCallback,
-  token: CancellationToken,
-  ...args: any[]
-) => Thenable<R>;
-
-/**
- * A task that handles command invocations from `commandRunner`.
- * Arguments passed to the command handler are passed along,
- * untouched to this `NoProgressTask` instance.
- *
- * @param args arguments passed to this task passed on from
- * `commands.registerCommand`.
- */
-type NoProgressTask = ((...args: any[]) => Promise<any>);
-
-/**
- * This mediates between the kind of progress callbacks we want to
- * write (where we *set* current progress position and give
- * `maxSteps`) and the kind vscode progress api expects us to write
- * (which increment progress by a certain amount out of 100%).
- *
- * Where possible, the `commandRunner` function below should be used
- * instead of this function. The commandRunner is meant for wrapping
- * top-level commands and provides error handling and other support
- * automatically.
- *
- * Only use this function if you need a progress monitor and the
- * control flow does not always come from a command (eg- during
- * extension activation, or from an internal language server
- * request).
- */
-export function withProgress<R>(
-  options: ProgressOptions,
-  task: ProgressTask<R>,
-  ...args: any[]
-): Thenable<R> {
-  let progressAchieved = 0;
-  return Window.withProgress(options,
-    (progress, token) => {
-      return task(p => {
-        const { message, step, maxStep } = p;
-        const increment = 100 * (step - progressAchieved) / maxStep;
-        progressAchieved = step;
-        progress.report({ message, increment });
-      }, token, ...args);
-    });
-}
-
-/**
- * A generic wrapper for command registration. This wrapper adds uniform error handling for commands.
- *
- * In this variant of the command runner, no progress monitor is used.
- *
- * @param commandId The ID of the command to register.
- * @param task The task to run. It is passed directly to `commands.registerCommand`. Any
- * arguments to the command handler are passed on to the task.
- */
-export function commandRunner(
-  commandId: string,
-  task: NoProgressTask,
-): Disposable {
-  return commands.registerCommand(commandId, async (...args: any[]) => {
-    try {
-      await task(...args);
-    } catch (e) {
-      if (e instanceof UserCancellationException) {
-        // User has cancelled this action manually
-        if (e.silent) {
-          logger.log(e.message);
-        } else {
-          showAndLogWarningMessage(e.message);
-        }
-      } else {
-        showAndLogErrorMessage(e.message || e);
-      }
-    }
-  });
-}
-
-/**
- * A generic wrapper for command registration.  This wrapper adds uniform error handling,
- * progress monitoring, and cancellation for commands.
- *
- * @param commandId The ID of the command to register.
- * @param task The task to run. It is passed directly to `commands.registerCommand`. Any
- * arguments to the command handler are passed on to the task after the progress callback
- * and cancellation token.
- * @param progressOptions Progress options to be sent to the progress monitor.
- */
-export function commandRunnerWithProgress<R>(
-  commandId: string,
-  task: ProgressTask<R>,
-  progressOptions: Partial<ProgressOptions>
-): Disposable {
-  return commands.registerCommand(commandId, async (...args: any[]) => {
-    const progressOptionsWithDefaults = {
-      location: ProgressLocation.Notification,
-      ...progressOptions
-    };
-    try {
-      await withProgress(progressOptionsWithDefaults, task, ...args);
-    } catch (e) {
-      if (e instanceof UserCancellationException) {
-        // User has cancelled this action manually
-        if (e.silent) {
-          logger.log(e.message);
-        } else {
-          showAndLogWarningMessage(e.message);
-        }
-      } else {
-        showAndLogErrorMessage(e.message || e);
-      }
-    }
-  });
-}
-
 /**
 * Show an error message and log it to the console
 *
 * @param message The message to show.
 * @param options.outputLogger The output logger that will receive the message
 * @param options.items A set of items that will be rendered as actions in the message.
+ * @param options.fullMessage An alternate message that is added to the log, but not displayed
+ *                           in the popup. This is useful for adding extra detail to the logs
+ *                           that would be too noisy for the popup.
 *
 * @return A promise that resolves to the selected item or undefined when being dismissed.
 */
 export async function showAndLogErrorMessage(message: string, {
  outputLogger = logger,
-  items = [] as string[]
+  items = [] as string[],
+  fullMessage = undefined as (string | undefined)
 } = {}): Promise<string | undefined> {
-  return internalShowAndLog(message, items, outputLogger, Window.showErrorMessage);
+  return internalShowAndLog(message, items, outputLogger, Window.showErrorMessage, fullMessage);
 }
 /**
 * Show a warning message and log it to the console
@@ -222,10 +62,15 @@ export async function showAndLogInformationMessage(message: string, {

 type ShowMessageFn = (message: string, ...items: string[]) => Thenable<string | undefined>;

-async function internalShowAndLog(message: string, items: string[], outputLogger = logger,
-  fn: ShowMessageFn): Promise<string | undefined> {
+async function internalShowAndLog(
+  message: string,
+  items: string[],
+  outputLogger = logger,
+  fn: ShowMessageFn,
+  fullMessage?: string
+): Promise<string | undefined> {
  const label = 'Show Log';
-  outputLogger.log(message);
+  outputLogger.log(fullMessage || message);
  const result = await fn(message, label, ...items);
  if (result === label) {
    outputLogger.show();
@@ -358,12 +203,6 @@ function createRateLimitedResult(): RateLimitedResult {
  };
 }

-
-export type DatasetFolderInfo = {
-  dbscheme: string;
-  qlpack: string;
-}
-
 export async function getQlPackForDbscheme(cliServer: CodeQLCliServer, dbschemePath: string): Promise<string> {
  const qlpacks = await cliServer.resolveQlpacks(getOnDiskWorkspaceFolders());
  const packs: { packDir: string | undefined; packName: string }[] =
@@ -391,7 +230,7 @@ export async function getQlPackForDbscheme(cliServer: CodeQLCliServer, dbschemeP
  throw new Error(`Could not find qlpack file for dbscheme ${dbschemePath}`);
 }

-export async function resolveDatasetFolder(cliServer: CodeQLCliServer, datasetFolder: string): Promise<DatasetFolderInfo> {
+export async function getPrimaryDbscheme(datasetFolder: string): Promise<string> {
  const dbschemes = await glob(path.join(datasetFolder, '*.dbscheme'));

  if (dbschemes.length < 1) {
@@ -400,12 +239,11 @@ export async function resolveDatasetFolder(cliServer: CodeQLCliServer, datasetFo

  dbschemes.sort();
  const dbscheme = dbschemes[0];
+
  if (dbschemes.length > 1) {
    Window.showErrorMessage(`Found multiple dbschemes in ${datasetFolder} during quick query; arbitrarily choosing the first, ${dbscheme}, to decide what library to use.`);
  }
-
-  const qlpack = await getQlPackForDbscheme(cliServer, dbscheme);
-  return { dbscheme, qlpack };
+  return dbscheme;
 }

 /**
@@ -464,3 +302,74 @@ export class CachedOperation<U> {
    }
  }
 }
+
+
+
+/**
+ * The following functions al heuristically determine metadata about databases.
+ */
+
+/**
+ * Note that this heuristic is only being used for backwards compatibility with
+ * CLI versions before the langauge name was introduced to dbInfo. Features
+ * that do not require backwards compatibility should call
+ * `cli.CodeQLCliServer.resolveDatabase` and use the first entry in the
+ * `languages` property.
+ *
+ * @see cli.CodeQLCliServer.supportsLanguageName
+ * @see cli.CodeQLCliServer.resolveDatabase
+ */
+const dbSchemeToLanguage = {
+  'semmlecode.javascript.dbscheme': 'javascript',
+  'semmlecode.cpp.dbscheme': 'cpp',
+  'semmlecode.dbscheme': 'java',
+  'semmlecode.python.dbscheme': 'python',
+  'semmlecode.csharp.dbscheme': 'csharp',
+  'go.dbscheme': 'go'
+};
+
+/**
+ * Returns the initial contents for an empty query, based on the language of the selected
+ * databse.
+ *
+ * First try to use the given language name. If that doesn't exist, try to infer it based on
+ * dbscheme. Otherwise return no import statement.
+ *
+ * @param language the database language or empty string if unknown
+ * @param dbscheme path to the dbscheme file
+ *
+ * @returns an import and empty select statement appropriate for the selected language
+ */
+export function getInitialQueryContents(language: string, dbscheme: string) {
+  if (!language) {
+    const dbschemeBase = path.basename(dbscheme) as keyof typeof dbSchemeToLanguage;
+    language = dbSchemeToLanguage[dbschemeBase];
+  }
+
+  return language
+    ? `import ${language}\n\nselect ""`
+    : 'select ""';
+}
+
+/**
+ * Heuristically determines if the directory passed in corresponds
+ * to a database root.
+ *
+ * @param maybeRoot
+ */
+export async function isLikelyDatabaseRoot(maybeRoot: string) {
+  const [a, b, c] = (await Promise.all([
+    // databases can have either .dbinfo or codeql-database.yml.
+    fs.pathExists(path.join(maybeRoot, '.dbinfo')),
+    fs.pathExists(path.join(maybeRoot, 'codeql-database.yml')),
+
+    // they *must* have a db-{language} folder
+    glob('db-*/', { cwd: maybeRoot })
+  ]));
+
+  return !!((a || b) && c);
+}
+
+export function isLikelyDbLanguageFolder(dbPath: string) {
+  return !!path.basename(dbPath).startsWith('db-');
+}
--- a/extensions/ql-vscode/src/interface.ts
+++ b/extensions/ql-vscode/src/interface.ts
@@ -30,7 +30,7 @@ import {
  RawResultsSortState,
 } from './pure/interface-types';
 import { Logger } from './logging';
-import { commandRunner } from './helpers';
+import { commandRunner } from './commandRunner';
 import * as messages from './pure/messages';
 import { CompletedQuery, interpretResults } from './query-results';
 import { QueryInfo, tmpDir } from './run-queries';
@@ -286,6 +286,9 @@ export class InterfaceManager extends DisposableObject {
          );
        }
        break;
+      case 'openFile':
+        await this.openFile(msg.filePath);
+        break;
      default:
        assertNever(msg);
    }
@@ -413,6 +416,8 @@ export class InterfaceManager extends DisposableObject {
      database: results.database,
      shouldKeepOldResultsWhileRendering,
      metadata: results.query.metadata,
+      queryName: results.toString(),
+      queryPath: results.query.program.queryPath
    });
  }

@@ -444,6 +449,8 @@ export class InterfaceManager extends DisposableObject {
      resultSetNames,
      pageSize: PAGE_SIZE.getValue(),
      numPages: numInterpretedPages(this._interpretation),
+      queryName: this._displayedQuery.toString(),
+      queryPath: this._displayedQuery.query.program.queryPath
    });
  }

@@ -456,6 +463,11 @@ export class InterfaceManager extends DisposableObject {
    return schemas['result-sets'];
  }

+  public async openFile(filePath: string) {
+    const textDocument = await vscode.workspace.openTextDocument(filePath);
+    await vscode.window.showTextDocument(textDocument, vscode.ViewColumn.One);
+  }
+
  /**
   * Show a page of raw results from the chosen table.
   */
@@ -517,6 +529,8 @@ export class InterfaceManager extends DisposableObject {
      database: results.database,
      shouldKeepOldResultsWhileRendering: false,
      metadata: results.query.metadata,
+      queryName: results.toString(),
+      queryPath: results.query.program.queryPath
    });
  }

@@ -534,8 +548,9 @@ export class InterfaceManager extends DisposableObject {
      sourceInfo
    );
    sarif.runs.forEach(run => {
-      if (run.results !== undefined)
+      if (run.results !== undefined) {
        sortInterpretedResults(run.results, sortState);
+      }
    });

    const numTotalResults = (() => {
--- a/extensions/ql-vscode/src/pure/helpers-pure.ts
+++ b/extensions/ql-vscode/src/pure/helpers-pure.ts
@@ -2,7 +2,7 @@
 * helpers-pure.ts
 * ------------
 *
- * Helper functions that don't depend on vscode and therefore can be used by the front-end and pure unit tests.
+ * Helper functions that don't depend on vscode or the CLI and therefore can be used by the front-end and pure unit tests.
 */

 /**
--- a/extensions/ql-vscode/src/pure/interface-types.ts
+++ b/extensions/ql-vscode/src/pure/interface-types.ts
@@ -88,6 +88,8 @@ export interface SetStateMsg {
  interpretation: undefined | Interpretation;
  database: DatabaseInfo;
  metadata?: QueryMetadata;
+  queryName: string;
+  queryPath: string;
  /**
   * Whether to keep displaying the old results while rendering the new results.
   *
@@ -116,6 +118,8 @@ export interface ShowInterpretedPageMsg {
  numPages: number;
  pageSize: number;
  resultSetNames: string[];
+  queryName: string;
+  queryPath: string;
 }

 /** Advance to the next or previous path no in the path viewer */
@@ -153,7 +157,8 @@ export type FromResultsViewMsg =
  | ChangeRawResultsSortMsg
  | ChangeInterpretedResultsSortMsg
  | ResultViewLoaded
-  | ChangePage;
+  | ChangePage
+  | OpenFileMsg;

 /**
 * Message from the results view to open a database source
@@ -165,6 +170,14 @@ export interface ViewSourceFileMsg {
  databaseUri: string;
 }

+/**
+ * Message from the results view to open a file in an editor.
+ */
+export interface OpenFileMsg {
+  t: 'openFile';
+  /* Full path to the file to open. */
+  filePath: string;
+}

 /**
 * Message from the results view to toggle the display of
--- a/extensions/ql-vscode/src/pure/messages.ts
+++ b/extensions/ql-vscode/src/pure/messages.ts
@@ -150,6 +150,11 @@ export interface CompilationOptions {
   * Whether to disable toString values in the results.
   */
  noComputeToString: boolean;
+  /**
+   * Whether to ensure that elements that do not have a displayString
+   * get reported anyway. Useful for universal compilation options.
+   */
+  computeDefaultStrings: boolean;
 }

 /**
@@ -380,8 +385,8 @@ export namespace ResultColumnKind {
   */
  export const BOOLEAN = 3;
  /**
- * A column of type `date`
- */
+   * A column of type `date`
+   */
  export const DATE = 4;
  /**
   * A column of a non-primitive type
@@ -401,6 +406,11 @@ export interface CompileUpgradeParams {
   * A directory to store parts of the compiled upgrade
   */
  upgradeTempDir: string;
+  /**
+   * Enable single file upgrades, set to true to allow
+   * using single file upgrades.
+   */
+  singleFileUpgrades: true;
 }

 /**
@@ -487,10 +497,13 @@ export interface UpgradeDescription {
  newSha: string;
 }

+
+export type CompiledUpgrades = MultiFileCompiledUpgrades | SingleFileCompiledUpgrade
+
 /**
- * A compiled upgrade.
+ * The parts shared by all compiled upgrades
 */
-export interface CompiledUpgrades {
+interface CompiledUpgradesBase {
  /**
  * The initial sha of the dbscheme to upgrade from
  */
@@ -499,14 +512,46 @@ export interface CompiledUpgrades {
   * The path to the new dataset statistics
   */
  newStatsPath: string;
+  /**
+   * The sha of the target dataset.
+   */
+  targetSha: string;
+}
+
+
+/**
+ * A compiled upgrade.
+ * The upgrade is spread among multiple files.
+ */
+interface MultiFileCompiledUpgrades extends CompiledUpgradesBase {
+  /**
+   * The path to the new dataset dbscheme
+   */
+  newDbscheme: string;
  /**
   * The steps in the upgrade path
   */
  scripts: CompiledUpgradeScript[];
  /**
-   * The sha of the target dataset.
+   * Will never exist in an old result
   */
-  targetSha: string;
+  compiledUpgradeFile?: never;
+}
+
+
+/**
+ * A compiled upgrade.
+ * The upgrade is in a single file.
+ */
+export interface SingleFileCompiledUpgrade extends CompiledUpgradesBase {
+  /**
+   * The steps in the upgrade path
+   */
+  descriptions: UpgradeDescription[];
+  /**
+   * A path to a file containing the upgrade
+   */
+  compiledUpgradeFile: string;
 }

 /**
@@ -837,7 +882,6 @@ export interface RunUpgradeParams {
  toRun: CompiledUpgrades;
 }

-
 /**
 * The result of running an upgrade
 */
@@ -857,6 +901,21 @@ export interface RunUpgradeResult {
  finalSha: string;
 }

+export interface RegisterDatabasesParams {
+  databases: Dataset[];
+}
+
+export interface DeregisterDatabasesParams {
+  databases: Dataset[];
+}
+
+export type RegisterDatabasesResult = {
+  registeredDatabases: Dataset[];
+};
+
+export type DeregisterDatabasesResult = {
+  registeredDatabases: Dataset[];
+};

 /**
 * Type for any action that could have progress messages.
@@ -934,6 +993,20 @@ export const runQueries = new rpc.RequestType<WithProgressId<EvaluateQueriesPara
 */
 export const runUpgrade = new rpc.RequestType<WithProgressId<RunUpgradeParams>, RunUpgradeResult, void, void>('evaluation/runUpgrade');

+export const registerDatabases = new rpc.RequestType<
+  WithProgressId<RegisterDatabasesParams>,
+  RegisterDatabasesResult,
+  void,
+  void
+>('evaluation/registerDatabases');
+
+export const deregisterDatabases = new rpc.RequestType<
+  WithProgressId<DeregisterDatabasesParams>,
+  DeregisterDatabasesResult,
+  void,
+  void
+>('evaluation/deregisterDatabases');
+
 /**
 * Request returned to the client to notify completion of a query.
 * The full runQueries job is completed when all queries are acknowledged.
--- a/extensions/ql-vscode/src/qltest-discovery.ts
+++ b/extensions/ql-vscode/src/qltest-discovery.ts
@@ -3,6 +3,7 @@ import { Discovery } from './discovery';
 import { EventEmitter, Event, Uri, RelativePattern, WorkspaceFolder, env } from 'vscode';
 import { MultiFileSystemWatcher } from './vscode-utils/multi-file-system-watcher';
 import { CodeQLCliServer } from './cli';
+import * as fs from 'fs-extra';

 /**
 * A node in the tree of tests. This will be either a `QLTestDirectory` or a `QLTestFile`.
@@ -180,19 +181,21 @@ export class QLTestDiscovery extends Discovery<QLTestDiscoveryResults> {
  private async discoverTests(): Promise<QLTestDirectory> {
    const fullPath = this.workspaceFolder.uri.fsPath;
    const name = this.workspaceFolder.name;
-    const resolvedTests = (await this.cliServer.resolveTests(fullPath))
-      .filter((testPath) => !QLTestDiscovery.ignoreTestPath(testPath));
-
    const rootDirectory = new QLTestDirectory(fullPath, name);
-    for (const testPath of resolvedTests) {
-      const relativePath = path.normalize(path.relative(fullPath, testPath));
-      const dirName = path.dirname(relativePath);
-      const parentDirectory = rootDirectory.createDirectory(dirName);
-      parentDirectory.addChild(new QLTestFile(testPath, path.basename(testPath)));
+
+    // Don't try discovery on workspace folders that don't exist on the filesystem
+    if ((await fs.pathExists(fullPath))) {
+      const resolvedTests = (await this.cliServer.resolveTests(fullPath))
+        .filter((testPath) => !QLTestDiscovery.ignoreTestPath(testPath));
+      for (const testPath of resolvedTests) {
+        const relativePath = path.normalize(path.relative(fullPath, testPath));
+        const dirName = path.dirname(relativePath);
+        const parentDirectory = rootDirectory.createDirectory(dirName);
+        parentDirectory.addChild(new QLTestFile(testPath, path.basename(testPath)));
+      }
+
+      rootDirectory.finish();
    }
-
-    rootDirectory.finish();
-
    return rootDirectory;
  }

--- a/extensions/ql-vscode/src/query-history.ts
+++ b/extensions/ql-vscode/src/query-history.ts
@@ -4,7 +4,15 @@ import { window as Window } from 'vscode';
 import { CompletedQuery } from './query-results';
 import { QueryHistoryConfig } from './config';
 import { QueryWithResults } from './run-queries';
-import * as helpers from './helpers';
+import {
+  showAndLogErrorMessage,
+  showAndLogInformationMessage,
+  showAndLogWarningMessage,
+  showBinaryChoiceDialog
+} from './helpers';
+import {
+  commandRunner
+} from './commandRunner';
 import { logger } from './logging';
 import { URLSearchParams } from 'url';
 import { QueryServerClient } from './queryserver-client';
@@ -52,23 +60,12 @@ const SHOW_QUERY_TEXT_QUICK_EVAL_MSG = `\
 */
 const FAILED_QUERY_HISTORY_ITEM_ICON = 'media/red-x.svg';

-interface QueryHistoryDataProvider extends vscode.TreeDataProvider<CompletedQuery> {
-  updateTreeItemContextValue(element: CompletedQuery): Promise<void>;
-}
-
 /**
 * Tree data provider for the query history view.
 */
-class HistoryTreeDataProvider implements QueryHistoryDataProvider {
-  /**
-   * XXX: This idiom for how to get a `.fire()`-able event emitter was
-   * cargo culted from another vscode extension. It seems rather
-   * involved and I hope there's something better that can be done
-   * instead.
-   */
-  private _onDidChangeTreeData: vscode.EventEmitter<
-    CompletedQuery | undefined
-  > = new vscode.EventEmitter<CompletedQuery | undefined>();
+export class HistoryTreeDataProvider extends DisposableObject {
+  private _onDidChangeTreeData = super.push(new vscode.EventEmitter<CompletedQuery | undefined>());
+
  readonly onDidChangeTreeData: vscode.Event<CompletedQuery | undefined> = this
    ._onDidChangeTreeData.event;

@@ -82,43 +79,35 @@ class HistoryTreeDataProvider implements QueryHistoryDataProvider {
  private current: CompletedQuery | undefined;

  constructor(extensionPath: string) {
+    super();
    this.failedIconPath = path.join(
      extensionPath,
      FAILED_QUERY_HISTORY_ITEM_ICON
    );
  }

-  async updateTreeItemContextValue(element: CompletedQuery): Promise<void> {
-    // Mark this query history item according to whether it has a
-    // SARIF file so that we can make context menu items conditionally
-    // available.
-    const hasResults = await element.query.hasInterpretedResults();
-    element.treeItem!.contextValue = hasResults
-      ? 'interpretedResultsItem'
-      : 'rawResultsItem';
-    this.refresh();
-  }
-
  async getTreeItem(element: CompletedQuery): Promise<vscode.TreeItem> {
-    if (element.treeItem !== undefined)
-      return element.treeItem;
+    const treeItem = new vscode.TreeItem(element.toString());

-    const it = new vscode.TreeItem(element.toString());
-
-    it.command = {
+    treeItem.command = {
      title: 'Query History Item',
      command: 'codeQLQueryHistory.itemClicked',
      arguments: [element],
    };

-    element.treeItem = it;
-    this.updateTreeItemContextValue(element);
+    // Mark this query history item according to whether it has a
+    // SARIF file so that we can make context menu items conditionally
+    // available.
+    const hasResults = await element.query.hasInterpretedResults();
+    treeItem.contextValue = hasResults
+      ? 'interpretedResultsItem'
+      : 'rawResultsItem';

    if (!element.didRunSuccessfully) {
-      it.iconPath = this.failedIconPath;
+      treeItem.iconPath = this.failedIconPath;
    }

-    return it;
+    return treeItem;
  }

  getChildren(
@@ -135,7 +124,7 @@ class HistoryTreeDataProvider implements QueryHistoryDataProvider {
    return this.current;
  }

-  push(item: CompletedQuery): void {
+  pushQuery(item: CompletedQuery): void {
    this.current = item;
    this.history.push(item);
    this.refresh();
@@ -163,8 +152,8 @@ class HistoryTreeDataProvider implements QueryHistoryDataProvider {
    return this.history;
  }

-  refresh() {
-    this._onDidChangeTreeData.fire(undefined);
+  refresh(completedQuery?: CompletedQuery) {
+    this._onDidChangeTreeData.fire(completedQuery);
  }

  find(queryId: number): CompletedQuery | undefined {
@@ -178,6 +167,7 @@ class HistoryTreeDataProvider implements QueryHistoryDataProvider {
 */
 const DOUBLE_CLICK_TIME = 500;

+const NO_QUERY_SELECTED = 'No query selected. Select a query history item you have already run and try again.';
 export class QueryHistoryManager extends DisposableObject {
  treeDataProvider: HistoryTreeDataProvider;
  treeView: vscode.TreeView<CompletedQuery>;
@@ -204,6 +194,7 @@ export class QueryHistoryManager extends DisposableObject {
      canSelectMany: true,
    });
    this.push(this.treeView);
+    this.push(treeDataProvider);

    // Lazily update the tree view selection due to limitations of TreeView API (see
    // `updateTreeViewSelectionIfVisible` doc for details)
@@ -224,55 +215,55 @@ export class QueryHistoryManager extends DisposableObject {

    logger.log('Registering query history panel commands.');
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.openQuery',
        this.handleOpenQuery.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.removeHistoryItem',
        this.handleRemoveHistoryItem.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.setLabel',
        this.handleSetLabel.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.compareWith',
        this.handleCompareWith.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.showQueryLog',
        this.handleShowQueryLog.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.showQueryText',
        this.handleShowQueryText.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.viewSarif',
        this.handleViewSarif.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.viewDil',
        this.handleViewDil.bind(this)
      )
    );
    this.push(
-      helpers.commandRunner(
+      commandRunner(
        'codeQLQueryHistory.itemClicked',
        async (item: CompletedQuery) => {
          return this.handleItemClicked(item, [item]);
@@ -315,6 +306,10 @@ export class QueryHistoryManager extends DisposableObject {
      return;
    }

+    if (!finalSingleItem) {
+      throw new Error(NO_QUERY_SELECTED);
+    }
+
    const textDocument = await vscode.workspace.openTextDocument(
      vscode.Uri.file(finalSingleItem.query.program.queryPath)
    );
@@ -367,11 +362,9 @@ export class QueryHistoryManager extends DisposableObject {
    });
    // undefined response means the user cancelled the dialog; don't change anything
    if (response !== undefined) {
-      if (response === '')
-        // Interpret empty string response as 'go back to using default'
-        singleItem.options.label = undefined;
-      else singleItem.options.label = response;
-      this.treeDataProvider.refresh();
+      // Interpret empty string response as 'go back to using default'
+      singleItem.options.label = response === '' ? undefined : response;
+      this.treeDataProvider.refresh(singleItem);
    }
  }

@@ -391,7 +384,7 @@ export class QueryHistoryManager extends DisposableObject {
        this.doCompareCallback(from, to);
      }
    } catch (e) {
-      helpers.showAndLogErrorMessage(e.message);
+      showAndLogErrorMessage(e.message);
    }
  }

@@ -403,6 +396,11 @@ export class QueryHistoryManager extends DisposableObject {
    if (!this.assertSingleQuery(finalMultiSelect)) {
      return;
    }
+
+    if (!finalSingleItem) {
+      throw new Error(NO_QUERY_SELECTED);
+    }
+
    this.treeDataProvider.setCurrentItem(finalSingleItem);

    const now = new Date();
@@ -433,7 +431,7 @@ export class QueryHistoryManager extends DisposableObject {
    if (singleItem.logFileLocation) {
      await this.tryOpenExternalFile(singleItem.logFileLocation);
    } else {
-      helpers.showAndLogWarningMessage('No log file available');
+      showAndLogWarningMessage('No log file available');
    }
  }

@@ -445,6 +443,10 @@ export class QueryHistoryManager extends DisposableObject {
      return;
    }

+    if (!singleItem) {
+      throw new Error(NO_QUERY_SELECTED);
+    }
+
    const queryName = singleItem.queryName.endsWith('.ql')
      ? singleItem.queryName
      : singleItem.queryName + '.ql';
@@ -474,7 +476,7 @@ export class QueryHistoryManager extends DisposableObject {
      );
    } else {
      const label = singleItem.getLabel();
-      helpers.showAndLogInformationMessage(
+      showAndLogInformationMessage(
        `Query ${label} has no interpreted results.`
      );
    }
@@ -513,7 +515,7 @@ export class QueryHistoryManager extends DisposableObject {

  addQuery(info: QueryWithResults): CompletedQuery {
    const item = new CompletedQuery(info, this.queryHistoryConfigListener);
-    this.treeDataProvider.push(item);
+    this.treeDataProvider.pushQuery(item);
    this.updateTreeViewSelectionIfVisible();
    return item;
  }
@@ -553,7 +555,7 @@ export class QueryHistoryManager extends DisposableObject {
        ) ||
        e.message.includes('too large to open')
      ) {
-        const res = await helpers.showBinaryChoiceDialog(
+        const res = await showBinaryChoiceDialog(
          `VS Code does not allow extensions to open files >50MB. This file
 exceeds that limit. Do you want to open it outside of VS Code?

@@ -564,11 +566,11 @@ the file in the file explorer and dragging it into the workspace.`
          try {
            await vscode.commands.executeCommand('revealFileInOS', uri);
          } catch (e) {
-            helpers.showAndLogErrorMessage(e.message);
+            showAndLogErrorMessage(e.message);
          }
        }
      } else {
-        helpers.showAndLogErrorMessage(`Could not open file ${fileLocation}`);
+        showAndLogErrorMessage(`Could not open file ${fileLocation}`);
        logger.log(e.message);
        logger.log(e.stack);
      }
@@ -622,7 +624,7 @@ the file in the file explorer and dragging it into the workspace.`

  private assertSingleQuery(multiSelect: CompletedQuery[] = [], message = 'Please select a single query.') {
    if (multiSelect.length > 1) {
-      helpers.showAndLogErrorMessage(
+      showAndLogErrorMessage(
        message
      );
      return false;
@@ -686,7 +688,7 @@ the file in the file explorer and dragging it into the workspace.`
    };
  }

-  async updateTreeItemContextValue(element: CompletedQuery): Promise<void> {
-    this.treeDataProvider.updateTreeItemContextValue(element);
+  async refreshTreeView(completedQuery: CompletedQuery): Promise<void> {
+    this.treeDataProvider.refresh(completedQuery);
  }
 }
--- a/extensions/ql-vscode/src/query-results.ts
+++ b/extensions/ql-vscode/src/query-results.ts
@@ -1,4 +1,4 @@
-import { env, TreeItem } from 'vscode';
+import { env } from 'vscode';

 import { QueryWithResults, tmpDir, QueryInfo } from './run-queries';
 import * as messages from './pure/messages';
@@ -17,7 +17,6 @@ export class CompletedQuery implements QueryWithResults {
  readonly database: DatabaseInfo;
  readonly logFileLocation?: string;
  options: QueryHistoryItemOptions;
-  treeItem?: TreeItem;
  dispose: () => void;

  /**
--- a/extensions/ql-vscode/src/queryserver-client.ts
+++ b/extensions/ql-vscode/src/queryserver-client.ts
@@ -8,6 +8,7 @@ import { QueryServerConfig } from './config';
 import { Logger, ProgressReporter } from './logging';
 import { completeQuery, EvaluationResult, progress, ProgressMessage, WithProgressId } from './pure/messages';
 import * as messages from './pure/messages';
+import { SemVer } from 'semver';

 type ServerOpts = {
  logger: Logger;
@@ -47,6 +48,12 @@ type WithProgressReporting = (task: (progress: ProgressReporter, token: Cancella
 * to restart it (which disposes the existing process and starts a new one).
 */
 export class QueryServerClient extends DisposableObject {
+
+  /**
+   * Query Server version where database registration was introduced
+   */
+  private static VERSION_WITH_DB_REGISTRATION = new SemVer('2.4.1');
+
  serverProcess?: ServerProcess;
  evaluationResultCallbacks: { [key: number]: (res: EvaluationResult) => void };
  progressCallbacks: { [key: number]: ((res: ProgressMessage) => void) | undefined };
@@ -55,7 +62,12 @@ export class QueryServerClient extends DisposableObject {
  withProgressReporting: WithProgressReporting;
  public activeQueryName: string | undefined;

-  constructor(readonly config: QueryServerConfig, readonly cliServer: cli.CodeQLCliServer, readonly opts: ServerOpts, withProgressReporting: WithProgressReporting) {
+  constructor(
+    readonly config: QueryServerConfig,
+    readonly cliServer: cli.CodeQLCliServer,
+    readonly opts: ServerOpts,
+    withProgressReporting: WithProgressReporting
+  ) {
    super();
    // When the query server configuration changes, restart the query server.
    if (config.onDidChangeConfiguration !== undefined) {
@@ -104,6 +116,11 @@ export class QueryServerClient extends DisposableObject {
  private async startQueryServerImpl(progressReporter: ProgressReporter): Promise<void> {
    const ramArgs = await this.cliServer.resolveRam(this.config.queryMemoryMb, progressReporter);
    const args = ['--threads', this.config.numThreads.toString()].concat(ramArgs);
+
+    if (await this.supportsDatabaseRegistration()) {
+      args.push('--require-db-registration');
+    }
+
    if (this.config.debug) {
      args.push('--debug', '--tuple-counting');
    }
@@ -157,6 +174,10 @@ export class QueryServerClient extends DisposableObject {
    this.evaluationResultCallbacks = {};
  }

+  async supportsDatabaseRegistration() {
+    return (await this.cliServer.getVersion()).compare(QueryServerClient.VERSION_WITH_DB_REGISTRATION) >= 0;
+  }
+
  registerCallback(callback: (res: EvaluationResult) => void): number {
    const id = this.nextCallback++;
    this.evaluationResultCallbacks[id] = callback;
--- a/extensions/ql-vscode/src/quick-query.ts
+++ b/extensions/ql-vscode/src/quick-query.ts
@@ -5,8 +5,18 @@ import { CancellationToken, ExtensionContext, window as Window, workspace, Uri }
 import { ErrorCodes, ResponseError } from 'vscode-languageclient';
 import { CodeQLCliServer } from './cli';
 import { DatabaseUI } from './databases-ui';
-import * as helpers from './helpers';
 import { logger } from './logging';
+import {
+  getInitialQueryContents,
+  getPrimaryDbscheme,
+  getQlPackForDbscheme,
+  showAndLogErrorMessage,
+  showBinaryChoiceDialog,
+} from './helpers';
+import {
+  ProgressCallback,
+  UserCancellationException
+} from './commandRunner';

 const QUICK_QUERIES_DIR_NAME = 'quick-queries';
 const QUICK_QUERY_QUERY_NAME = 'quick-query.ql';
@@ -16,21 +26,6 @@ export function isQuickQueryPath(queryPath: string): boolean {
  return path.basename(queryPath) === QUICK_QUERY_QUERY_NAME;
 }

-/**
- * `getBaseText` heuristically returns an appropriate import statement
- * prelude based on the filename of the dbscheme file given. TODO: add
- * a 'default import' field to the qlpack itself, and use that.
- */
-function getBaseText(dbschemeBase: string) {
-  if (dbschemeBase == 'semmlecode.javascript.dbscheme') return 'import javascript\n\nselect ""';
-  if (dbschemeBase == 'semmlecode.cpp.dbscheme') return 'import cpp\n\nselect ""';
-  if (dbschemeBase == 'semmlecode.dbscheme') return 'import java\n\nselect ""';
-  if (dbschemeBase == 'semmlecode.python.dbscheme') return 'import python\n\nselect ""';
-  if (dbschemeBase == 'semmlecode.csharp.dbscheme') return 'import csharp\n\nselect ""';
-  if (dbschemeBase == 'go.dbscheme') return 'import go\n\nselect ""';
-  return 'select ""';
-}
-
 function getQuickQueriesDir(ctx: ExtensionContext): string {
  const storagePath = ctx.storagePath;
  if (storagePath === undefined) {
@@ -51,7 +46,7 @@ export async function displayQuickQuery(
  ctx: ExtensionContext,
  cliServer: CodeQLCliServer,
  databaseUI: DatabaseUI,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: CancellationToken
 ) {

@@ -85,7 +80,7 @@ export async function displayQuickQuery(
    // being undefined) just let the user know that they're in for a
    // restart.
    if (workspace.workspaceFile === undefined) {
-      const makeMultiRoot = await helpers.showBinaryChoiceDialog('Quick query requires multiple folders in the workspace. Reload workspace as multi-folder workspace?');
+      const makeMultiRoot = await showBinaryChoiceDialog('Quick query requires multiple folders in the workspace. Reload workspace as multi-folder workspace?');
      if (makeMultiRoot) {
        updateQuickQueryDir(queriesDir, workspaceFolders.length, 0);
      }
@@ -105,7 +100,9 @@ export async function displayQuickQuery(
    }

    const datasetFolder = await dbItem.getDatasetFolder(cliServer);
-    const { qlpack, dbscheme } = await helpers.resolveDatasetFolder(cliServer, datasetFolder);
+    const dbscheme = await getPrimaryDbscheme(datasetFolder);
+    const qlpack = await getQlPackForDbscheme(cliServer, dbscheme);
+
    const quickQueryQlpackYaml: any = {
      name: 'quick-query',
      version: '1.0.0',
@@ -114,21 +111,21 @@ export async function displayQuickQuery(

    const qlFile = path.join(queriesDir, QUICK_QUERY_QUERY_NAME);
    const qlPackFile = path.join(queriesDir, 'qlpack.yml');
-    await fs.writeFile(qlFile, getBaseText(path.basename(dbscheme)), 'utf8');
+    await fs.writeFile(qlFile, getInitialQueryContents(dbItem.language, dbscheme), 'utf8');
    await fs.writeFile(qlPackFile, yaml.safeDump(quickQueryQlpackYaml), 'utf8');
    Window.showTextDocument(await workspace.openTextDocument(qlFile));
  }

  // TODO: clean up error handling for top-level commands like this
  catch (e) {
-    if (e instanceof helpers.UserCancellationException) {
+    if (e instanceof UserCancellationException) {
      logger.log(e.message);
    }
    else if (e instanceof ResponseError && e.code == ErrorCodes.RequestCancelled) {
      logger.log(e.message);
    }
    else if (e instanceof Error)
-      helpers.showAndLogErrorMessage(e.message);
+      showAndLogErrorMessage(e.message);
    else
      throw e;
  }
--- a/extensions/ql-vscode/src/run-queries.ts
+++ b/extensions/ql-vscode/src/run-queries.ts
@@ -15,7 +15,8 @@ import { ErrorCodes, ResponseError } from 'vscode-languageclient';
 import * as cli from './cli';
 import * as config from './config';
 import { DatabaseItem, getUpgradesDirectories } from './databases';
-import * as helpers from './helpers';
+import { getOnDiskWorkspaceFolders, showAndLogErrorMessage } from './helpers';
+import { ProgressCallback, UserCancellationException } from './commandRunner';
 import { DatabaseInfo, QueryMetadata, ResultsPaths } from './pure/interface-types';
 import { logger } from './logging';
 import * as messages from './pure/messages';
@@ -79,7 +80,7 @@ export class QueryInfo {

  async run(
    qs: qsClient.QueryServerClient,
-    progress: helpers.ProgressCallback,
+    progress: ProgressCallback,
    token: CancellationToken,
  ): Promise<messages.EvaluationResult> {
    let result: messages.EvaluationResult | null = null;
@@ -121,7 +122,7 @@ export class QueryInfo {

  async compile(
    qs: qsClient.QueryServerClient,
-    progress: helpers.ProgressCallback,
+    progress: ProgressCallback,
    token: CancellationToken,
  ): Promise<messages.CompilationMessage[]> {
    let compiled: messages.CheckQueryResult | undefined;
@@ -138,6 +139,7 @@ export class QueryInfo {
          localChecking: false,
          noComputeGetUrl: false,
          noComputeToString: false,
+          computeDefaultStrings: true
        },
        extraOptions: {
          timeoutSecs: qs.config.timeoutSecs
@@ -208,7 +210,7 @@ export interface QueryWithResults {
 export async function clearCacheInDatabase(
  qs: qsClient.QueryServerClient,
  dbItem: DatabaseItem,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: CancellationToken,
 ): Promise<messages.ClearCacheResult> {
  if (dbItem.contents === undefined) {
@@ -284,10 +286,10 @@ async function checkDbschemeCompatibility(
  cliServer: cli.CodeQLCliServer,
  qs: qsClient.QueryServerClient,
  query: QueryInfo,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: CancellationToken,
 ): Promise<void> {
-  const searchPath = helpers.getOnDiskWorkspaceFolders();
+  const searchPath = getOnDiskWorkspaceFolders();

  if (query.dbItem.contents !== undefined && query.dbItem.contents.dbSchemeUri !== undefined) {
    const { scripts, finalDbscheme } = await cliServer.resolveUpgrades(query.dbItem.contents.dbSchemeUri.fsPath, searchPath);
@@ -344,7 +346,7 @@ async function promptUserToSaveChanges(document: TextDocument): Promise<boolean>
    else {
      const yesItem = { title: 'Yes', isCloseAffordance: false };
      const alwaysItem = { title: 'Always Save', isCloseAffordance: false };
-      const noItem = { title: 'No (run anyway)', isCloseAffordance: false };
+      const noItem = { title: 'No (run version on disk)', isCloseAffordance: false };
      const cancelItem = { title: 'Cancel', isCloseAffordance: true };
      const message = 'Query file has unsaved changes. Save now?';
      const chosenItem = await window.showInformationMessage(
@@ -363,7 +365,7 @@ async function promptUserToSaveChanges(document: TextDocument): Promise<boolean>
      }

      if (chosenItem === cancelItem) {
-        throw new helpers.UserCancellationException('Query run cancelled.', true);
+        throw new UserCancellationException('Query run cancelled.', true);
      }
    }
  }
@@ -453,7 +455,7 @@ export async function compileAndRunQueryAgainstDatabase(
  db: DatabaseItem,
  quickEval: boolean,
  selectedQueryUri: Uri | undefined,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: CancellationToken,
  templates?: messages.TemplateDefinitions,
 ): Promise<QueryWithResults> {
@@ -473,10 +475,14 @@ export async function compileAndRunQueryAgainstDatabase(
  }

  // Get the workspace folder paths.
-  const diskWorkspaceFolders = helpers.getOnDiskWorkspaceFolders();
+  const diskWorkspaceFolders = getOnDiskWorkspaceFolders();
  // Figure out the library path for the query.
  const packConfig = await cliServer.resolveLibraryPath(diskWorkspaceFolders, queryPath);

+  if (!packConfig.dbscheme) {
+    throw new Error('Could not find a database scheme for this query. Please check that you have a valid qlpack.yml file for this query, which refers to a database scheme either in the `dbscheme` field or through one of its dependencies.');
+  }
+
  // Check whether the query has an entirely different schema from the
  // database. (Queries that merely need the database to be upgraded
  // won't trigger this check)
@@ -528,7 +534,7 @@ export async function compileAndRunQueryAgainstDatabase(
    if (result.resultType !== messages.QueryResultType.SUCCESS) {
      const message = result.message || 'Failed to run query';
      logger.log(message);
-      helpers.showAndLogErrorMessage(message);
+      showAndLogErrorMessage(message);
    }
    return {
      query,
@@ -559,9 +565,9 @@ export async function compileAndRunQueryAgainstDatabase(
      qs.logger.log(formatted);
    }
    if (quickEval && formattedMessages.length <= 3) {
-      helpers.showAndLogErrorMessage('Quick evaluation compilation failed: \n' + formattedMessages.join('\n'));
+      showAndLogErrorMessage('Quick evaluation compilation failed: \n' + formattedMessages.join('\n'));
    } else {
-      helpers.showAndLogErrorMessage((quickEval ? 'Quick evaluation' : 'Query') +
+      showAndLogErrorMessage((quickEval ? 'Quick evaluation' : 'Query') +
        ' compilation failed. Please make sure there are no errors in the query, the database is up to date,' +
        ' and the query and database use the same target language. For more details on the error, go to View > Output,' +
        ' and choose CodeQL Query Server from the dropdown.');
--- a/extensions/ql-vscode/src/test-adapter.ts
+++ b/extensions/ql-vscode/src/test-adapter.ts
@@ -220,14 +220,13 @@ export class QLTestAdapter extends DisposableObject implements TestAdapter {
      const state = event.pass
        ? 'passed'
        : event.messages?.length
-        ? 'errored'
-        : 'failed';
+          ? 'errored'
+          : 'failed';
      let message: string | undefined;
      if (event.diff?.length) {
        message = ['', `${state}: ${event.test}`, ...event.diff, ''].join('\n');
        testLogger.log(message);
      }
-      (event.diff || []).join('\n');
      this._testStates.fire({
        type: 'test',
        state,
--- a/extensions/ql-vscode/src/upgrades.ts
+++ b/extensions/ql-vscode/src/upgrades.ts
@@ -1,6 +1,7 @@
 import * as vscode from 'vscode';
 import { DatabaseItem } from './databases';
-import * as helpers from './helpers';
+import { showAndLogErrorMessage } from './helpers';
+import { ProgressCallback, UserCancellationException } from './commandRunner';
 import { logger } from './logging';
 import * as messages from './pure/messages';
 import * as qsClient from './queryserver-client';
@@ -24,7 +25,7 @@ async function checkAndConfirmDatabaseUpgrade(
  db: DatabaseItem,
  targetDbScheme: vscode.Uri,
  upgradesDirectories: vscode.Uri[],
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: vscode.CancellationToken,
 ): Promise<messages.UpgradeParams | undefined> {
  if (db.contents === undefined || db.contents.dbSchemeUri === undefined) {
@@ -75,13 +76,20 @@ async function checkAndConfirmDatabaseUpgrade(
  if (curSha != targetSha) {
    // Newlines aren't rendered in notifications: https://github.com/microsoft/vscode/issues/48900
    // A modal dialog would be rendered better, but is more intrusive.
-    await helpers.showAndLogErrorMessage(`Database cannot be upgraded to the target database scheme.
+    await showAndLogErrorMessage(`Database cannot be upgraded to the target database scheme.
    Can upgrade from ${checkedUpgrades.initialSha} (current) to ${curSha}, but cannot reach ${targetSha} (target).`);
    // TODO: give a more informative message if we think the DB is ahead of the target DB scheme
    return;
  }

  logger.log(descriptionMessage);
+
+
+  // If the quiet flag is set, do the upgrade without a popup.
+  if (qs.cliServer.quiet) {
+    return params;
+  }
+
  // Ask the user to confirm the upgrade.

  const showLogItem: vscode.MessageItem = { title: 'No, Show Changes', isCloseAffordance: true };
@@ -107,7 +115,7 @@ async function checkAndConfirmDatabaseUpgrade(
    return params;
  }
  else {
-    throw new helpers.UserCancellationException('User cancelled the database upgrade.');
+    throw new UserCancellationException('User cancelled the database upgrade.');
  }
 }

@@ -121,7 +129,7 @@ export async function upgradeDatabase(
  qs: qsClient.QueryServerClient,
  db: DatabaseItem, targetDbScheme: vscode.Uri,
  upgradesDirectories: vscode.Uri[],
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: vscode.CancellationToken,
 ): Promise<messages.RunUpgradeResult | undefined> {
  const upgradeParams = await checkAndConfirmDatabaseUpgrade(qs, db, targetDbScheme, upgradesDirectories, progress, token);
@@ -135,7 +143,7 @@ export async function upgradeDatabase(
    compileUpgradeResult = await compileDatabaseUpgrade(qs, upgradeParams, progress, token);
  }
  catch (e) {
-    helpers.showAndLogErrorMessage(`Compilation of database upgrades failed: ${e}`);
+    showAndLogErrorMessage(`Compilation of database upgrades failed: ${e}`);
    return;
  }
  finally {
@@ -144,17 +152,24 @@ export async function upgradeDatabase(

  if (compileUpgradeResult.compiledUpgrades === undefined) {
    const error = compileUpgradeResult.error || '[no error message available]';
-    helpers.showAndLogErrorMessage(`Compilation of database upgrades failed: ${error}`);
+    (`Compilation of database upgrades failed: ${error}`);
    return;
  }

  try {
    qs.logger.log('Running the following database upgrade:');
-    qs.logger.log(compileUpgradeResult.compiledUpgrades.scripts.map(s => s.description.description).join('\n'));
+    // We use the presence of compiledUpgradeFile to check
+    // if it is multifile or not. We need to explicitly check undefined
+    // as the types claim the empty string is a valid value
+    if (compileUpgradeResult.compiledUpgrades.compiledUpgradeFile === undefined) {
+      qs.logger.log(compileUpgradeResult.compiledUpgrades.scripts.map(s => s.description.description).join('\n'));
+    } else {
+      qs.logger.log(compileUpgradeResult.compiledUpgrades.descriptions.map(s => s.description).join('\n'));
+    }
    return await runDatabaseUpgrade(qs, db, compileUpgradeResult.compiledUpgrades, progress, token);
  }
  catch (e) {
-    helpers.showAndLogErrorMessage(`Database upgrade failed: ${e}`);
+    showAndLogErrorMessage(`Database upgrade failed: ${e}`);
    return;
  }
  finally {
@@ -165,7 +180,7 @@ export async function upgradeDatabase(
 async function checkDatabaseUpgrade(
  qs: qsClient.QueryServerClient,
  upgradeParams: messages.UpgradeParams,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: vscode.CancellationToken,
 ): Promise<messages.CheckUpgradeResult> {
  progress({
@@ -180,12 +195,13 @@ async function checkDatabaseUpgrade(
 async function compileDatabaseUpgrade(
  qs: qsClient.QueryServerClient,
  upgradeParams: messages.UpgradeParams,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: vscode.CancellationToken,
 ): Promise<messages.CompileUpgradeResult> {
  const params: messages.CompileUpgradeParams = {
    upgrade: upgradeParams,
-    upgradeTempDir: upgradesTmpDir.name
+    upgradeTempDir: upgradesTmpDir.name,
+    singleFileUpgrades: true
  };

  progress({
@@ -201,7 +217,7 @@ async function runDatabaseUpgrade(
  qs: qsClient.QueryServerClient,
  db: DatabaseItem,
  upgrades: messages.CompiledUpgrades,
-  progress: helpers.ProgressCallback,
+  progress: ProgressCallback,
  token: vscode.CancellationToken,
 ): Promise<messages.RunUpgradeResult> {

--- a/extensions/ql-vscode/src/view/result-table-utils.tsx
+++ b/extensions/ql-vscode/src/view/result-table-utils.tsx
@@ -28,7 +28,8 @@ export interface ResultTableProps {
 }

 export const className = 'vscode-codeql__result-table';
-export const tableSelectionHeaderClassName = 'vscode-codeql__table-selection-header';
+export const tableHeaderClassName = 'vscode-codeql__table-selection-header';
+export const tableHeaderItemClassName = 'vscode-codeql__table-selection-header-item';
 export const alertExtrasClassName = `${className}-alert-extras`;
 export const toggleDiagnosticsClassName = `${className}-toggle-diagnostics`;
 export const evenRowClassName = 'vscode-codeql__result-table-row--even';
@@ -45,7 +46,9 @@ export function jumpToLocationHandler(
    jumpToLocation(loc, databaseUri);
    e.preventDefault();
    e.stopPropagation();
-    if (callback) callback();
+    if (callback) {
+      callback();
+    }
  };
 }

@@ -57,6 +60,13 @@ export function jumpToLocation(loc: ResolvableLocationValue, databaseUri: string
  });
 }

+export function openFile(filePath: string): void {
+  vscode.postMessage({
+    t: 'openFile',
+    filePath
+  });
+}
+
 /**
 * Render a location as a link which when clicked displays the original location.
 */
--- a/extensions/ql-vscode/src/view/result-tables.tsx
+++ b/extensions/ql-vscode/src/view/result-tables.tsx
@@ -11,19 +11,23 @@ import {
  SELECT_TABLE_NAME,
  getDefaultResultSetName,
  ParsedResultSets,
-  IntoResultsViewMsg
+  IntoResultsViewMsg,
 } from '../pure/interface-types';
 import { PathTable } from './alert-table';
 import { RawTable } from './raw-results-table';
 import {
  ResultTableProps,
-  tableSelectionHeaderClassName,
+  tableHeaderClassName,
+  tableHeaderItemClassName,
  toggleDiagnosticsClassName,
-  alertExtrasClassName
+  alertExtrasClassName,
+  openFile
 } from './result-table-utils';
 import { vscode } from './vscode-api';


+const FILE_PATH_REGEX = /^(?:.+[\\/])*(.+)$/;
+
 /**
 * Properties for the `ResultTables` component.
 */
@@ -38,6 +42,8 @@ export interface ResultTablesProps {
  sortStates: Map<string, RawResultsSortState>;
  interpretedSortState?: InterpretedResultsSortState;
  isLoadingNewResults: boolean;
+  queryName: string;
+  queryPath: string;
 }

 /**
@@ -62,7 +68,7 @@ function getResultCount(resultSet: ResultSet): number {

 function renderResultCountString(resultSet: ResultSet): JSX.Element {
  const resultCount = getResultCount(resultSet);
-  return <span className="number-of-results">
+  return <span className={tableHeaderItemClassName}>
    {resultCount} {resultCount === 1 ? 'result' : 'results'}
  </span>;
 }
@@ -213,28 +219,44 @@ export class ResultTables
      });
    };

-    return <span className="vscode-codeql__table-selection-header">
-      <button onClick={prevPage} >&#xab;</button>
-      <input
-        type="number"
-        size={3}
-        value={this.state.selectedPage}
-        min="1"
-        max={numPages}
-        onChange={onChange}
-        onBlur={e => choosePage(e.target.value)}
-        onKeyDown={e => {
-          if (e.keyCode === 13) {
-            choosePage((e.target as HTMLInputElement).value);
-          }
-        }
-        }
-      />
-      <span>
-        / {numPages}
+    const openQuery = () => {
+      openFile(this.props.queryPath);
+    };
+    const fileName = FILE_PATH_REGEX.exec(this.props.queryPath)?.[1] || 'query';
+
+    return (
+      <span className="vscode-codeql__table-selection-pagination">
+        <button onClick={prevPage} >&#xab;</button>
+        <input
+          type="number"
+          size={3}
+          value={this.state.selectedPage}
+          min="1"
+          max={numPages}
+          onChange={onChange}
+          onBlur={e => choosePage(e.target.value)}
+          onKeyDown={e => {
+            if (e.keyCode === 13) {
+              choosePage((e.target as HTMLInputElement).value);
+            }
+          }}
+        />
+        <span>
+          /&nbsp;{numPages}
+        </span>
+        <button value=">" onClick={nextPage} >&#xbb;</button>
+        <div className={tableHeaderItemClassName}>
+          {this.props.queryName}
+        </div>
+        <div className={tableHeaderItemClassName}>
+          <a
+            href="#"
+            onClick={openQuery}
+            className="vscode-codeql__result-table-location-link"
+          >Open {fileName}</a>
+        </div>
      </span>
-      <button value=">" onClick={nextPage} >&#xbb;</button>
-    </span>;
+    );
  }

  render(): React.ReactNode {
@@ -248,32 +270,35 @@ export class ResultTables

    const resultSetOptions =
      resultSetNames.map(name => <option key={name} value={name}>{name}</option>);
-
-    return <div>
-      {this.renderPageButtons()}
-      <div className={tableSelectionHeaderClassName}>
-        <select value={selectedTable} onChange={this.onTableSelectionChange}>
-          {resultSetOptions}
-        </select>
-        {numberOfResults}
-        {selectedTable === ALERTS_TABLE_NAME ? this.alertTableExtras() : undefined}
+    return (
+      <div>
+        {this.renderPageButtons()}
+        <div className={tableHeaderClassName}>
+        </div>
+        <div className={tableHeaderClassName}>
+          <select value={selectedTable} onChange={this.onTableSelectionChange}>
+            {resultSetOptions}
+          </select>
+          {numberOfResults}
+          {selectedTable === ALERTS_TABLE_NAME ? this.alertTableExtras() : undefined}
+          {
+            this.props.isLoadingNewResults ?
+              <span className={UPDATING_RESULTS_TEXT_CLASS_NAME}>Updating results…</span>
+              : null
+          }
+        </div>
        {
-          this.props.isLoadingNewResults ?
-            <span className={UPDATING_RESULTS_TEXT_CLASS_NAME}>Updating results…</span>
-            : null
+          resultSet &&
+          <ResultTable key={resultSet.schema.name} resultSet={resultSet}
+            databaseUri={this.props.database.databaseUri}
+            resultsPath={this.props.resultsPath}
+            sortState={this.props.sortStates.get(resultSet.schema.name)}
+            nonemptyRawResults={nonemptyRawResults}
+            showRawResults={() => { this.setState({ selectedTable: SELECT_TABLE_NAME }); }}
+            offset={this.getOffset()} />
        }
      </div>
-      {
-        resultSet &&
-        <ResultTable key={resultSet.schema.name} resultSet={resultSet}
-          databaseUri={this.props.database.databaseUri}
-          resultsPath={this.props.resultsPath}
-          sortState={this.props.sortStates.get(resultSet.schema.name)}
-          nonemptyRawResults={nonemptyRawResults}
-          showRawResults={() => { this.setState({ selectedTable: SELECT_TABLE_NAME }); }}
-          offset={this.getOffset()} />
-      }
-    </div>;
+    );
  }

  handleMessage(msg: IntoResultsViewMsg): void {
--- a/extensions/ql-vscode/src/view/results.tsx
+++ b/extensions/ql-vscode/src/view/results.tsx
@@ -11,7 +11,7 @@ import {
  QueryMetadata,
  ResultsPaths,
  ALERTS_TABLE_NAME,
-  ParsedResultSets
+  ParsedResultSets,
 } from '../pure/interface-types';
 import { EventHandlers as EventHandlerList } from './event-handler-list';
 import { ResultTables } from './result-tables';
@@ -37,6 +37,8 @@ interface ResultsInfo {
   */
  shouldKeepOldResultsWhileRendering: boolean;
  metadata?: QueryMetadata;
+  queryName: string;
+  queryPath: string;
 }

 interface Results {
@@ -96,6 +98,8 @@ class App extends React.Component<{}, ResultsViewState> {
          shouldKeepOldResultsWhileRendering:
            msg.shouldKeepOldResultsWhileRendering,
          metadata: msg.metadata,
+          queryName: msg.queryName,
+          queryPath: msg.queryPath,
        });

        this.loadResults();
@@ -127,6 +131,8 @@ class App extends React.Component<{}, ResultsViewState> {
          interpretation: msg.interpretation,
          shouldKeepOldResultsWhileRendering: true,
          metadata: msg.metadata,
+          queryName: msg.queryName,
+          queryPath: msg.queryPath,
        });
        this.loadResults();
        break;
@@ -280,6 +286,8 @@ class App extends React.Component<{}, ResultsViewState> {
            this.state.isExpectingResultsUpdate ||
            this.state.nextResultsInfo !== null
          }
+          queryName={displayedResults.resultsInfo.queryName}
+          queryPath={displayedResults.resultsInfo.queryPath}
        />
      );
    } else {
--- a/extensions/ql-vscode/src/view/resultsView.css
+++ b/extensions/ql-vscode/src/view/resultsView.css
@@ -8,13 +8,24 @@
  display: flex;
  padding: 0.5em 0;
  align-items: center;
+  justify-content: space-between;
+}
+
+.vscode-codeql__table-selection-pagination {
+  display: flex;
+  padding: 0.5em 0;
+  align-items: center;
+}
+
+.vscode-codeql__table-selection-header-item {
+  padding-left: 2em;
 }

 .vscode-codeql__table-selection-header select {
  border: 0;
 }

-.vscode-codeql__table-selection-header button {
+.vscode-codeql__table-selection-pagination button {
  padding: 0.3rem;
  margin: 0.2rem;
  border: 0;
@@ -25,11 +36,11 @@
  opacity: 0.8;
 }

-.vscode-codeql__table-selection-header button:hover {
+.vscode-codeql__table-selection-pagination button:hover {
  opacity: 1;
 }

-.vscode-codeql__table-selection-header input[type="number"] {
+.vscode-codeql__table-selection-pagination input[type="number"] {
  border-radius: 0;
  padding: 0.3rem;
  margin: 0.2rem;
@@ -185,10 +196,6 @@ td.vscode-codeql__path-index-cell {
  opacity: 0.6;
 }

-.number-of-results {
-  padding-left: 3em;
-}
-
 .vscode-codeql__compare-header {
  display: flex;
 }
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data/qlpack.yml
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data/qlpack.yml
@@ -0,0 +1,3 @@
+name: integration-test-queries-javascript
+version: 0.0.0
+libraryPathDependencies: codeql-javascript
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data/simple-query.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data/simple-query.ql
@@ -0,0 +1,8 @@
+predicate edges(int i, int j) {
+  i = 1 and j = 2 or i = 2 and j = 3
+}
+
+
+from int i, int j
+where edges(i, j)
+select i, j
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/databases.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/databases.test.ts
@@ -0,0 +1,85 @@
+import * as sinon from 'sinon';
+import * as path from 'path';
+import { fail } from 'assert';
+import { expect } from 'chai';
+import { extensions, CancellationToken, Uri, window } from 'vscode';
+
+import { CodeQLExtensionInterface } from '../../extension';
+import { DatabaseManager } from '../../databases';
+import { promptImportLgtmDatabase, importArchiveDatabase, promptImportInternetDatabase } from '../../databaseFetcher';
+import { ProgressCallback } from '../../commandRunner';
+import { dbLoc, DB_URL, storagePath } from './global.helper';
+
+/**
+ * Run various integration tests for databases
+ */
+describe('Databases', function() {
+  this.timeout(60000);
+
+  const LGTM_URL = 'https://lgtm.com/projects/g/aeisenberg/angular-bind-notifier/';
+
+  let databaseManager: DatabaseManager;
+  let sandbox: sinon.SinonSandbox;
+  let inputBoxStub: sinon.SinonStub;
+  let progressCallback: ProgressCallback;
+
+  beforeEach(async () => {
+    try {
+      const extension = await extensions.getExtension<CodeQLExtensionInterface | {}>('GitHub.vscode-codeql')!.activate();
+      if ('databaseManager' in extension) {
+        databaseManager = extension.databaseManager;
+      } else {
+        throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
+      }
+
+      sandbox = sinon.createSandbox();
+      // the uri.fsPath function on windows returns a lowercase drive letter
+      // so, force the storage path string to be lowercase, too.
+      progressCallback = sandbox.spy();
+      inputBoxStub = sandbox.stub(window, 'showInputBox');
+    } catch (e) {
+      fail(e);
+    }
+  });
+
+  afterEach(() => {
+    try {
+      // dispose();
+      sandbox.restore();
+    } catch (e) {
+      fail(e);
+    }
+  });
+
+  it('should add a database from a folder', async () => {
+    const progressCallback = sandbox.spy() as ProgressCallback;
+    const uri = Uri.file(dbLoc);
+    let dbItem = await importArchiveDatabase(uri.toString(true), databaseManager, storagePath, progressCallback, {} as CancellationToken);
+    expect(dbItem).to.be.eq(databaseManager.currentDatabaseItem);
+    expect(dbItem).to.be.eq(databaseManager.databaseItems[0]);
+    expect(dbItem).not.to.be.undefined;
+    dbItem = dbItem!;
+    expect(dbItem.name).to.eq('db');
+    expect(dbItem.databaseUri.fsPath).to.eq(path.join(storagePath, 'db', 'db'));
+  });
+
+  it('should add a database from lgtm with only one language', async () => {
+    inputBoxStub.resolves(LGTM_URL);
+    let dbItem = await promptImportLgtmDatabase(databaseManager, storagePath, progressCallback, {} as CancellationToken);
+    expect(dbItem).not.to.be.undefined;
+    dbItem = dbItem!;
+    expect(dbItem.name).to.eq('aeisenberg_angular-bind-notifier_106179a');
+    expect(dbItem.databaseUri.fsPath).to.eq(path.join(storagePath, 'javascript', 'aeisenberg_angular-bind-notifier_106179a'));
+  });
+
+  it('should add a database from a url', async () => {
+    inputBoxStub.resolves(DB_URL);
+
+    let dbItem = await promptImportInternetDatabase(databaseManager, storagePath, progressCallback, {} as CancellationToken);
+    expect(dbItem).not.to.be.undefined;
+    dbItem = dbItem!;
+    expect(dbItem.name).to.eq('db');
+    expect(dbItem.databaseUri.fsPath).to.eq(path.join(storagePath, 'simple-db', 'db'));
+  });
+
+});
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/global.helper.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/global.helper.ts
@@ -0,0 +1,81 @@
+import * as path from 'path';
+import * as tmp from 'tmp';
+import * as fs from 'fs-extra';
+import fetch from 'node-fetch';
+
+import { fail } from 'assert';
+import { ConfigurationTarget, extensions, workspace } from 'vscode';
+import { CodeQLExtensionInterface } from '../../extension';
+
+// This file contains helpers shared between actual tests.
+
+export const DB_URL = 'https://github.com/github/vscode-codeql/files/5586722/simple-db.zip';
+
+// We need to resolve the path, but the final three segments won't exist until later, so we only resolve the
+// first portion of the path.
+export const dbLoc = path.join(fs.realpathSync(path.join(__dirname, '../../../')), 'build/tests/db.zip');
+export let storagePath: string;
+
+// See https://github.com/DefinitelyTyped/DefinitelyTyped/pull/49860
+// Should be of type Mocha
+export default function(mocha: /*Mocha*/ any) {
+  // create an extension storage location
+  let removeStorage: tmp.DirResult['removeCallback'] | undefined;
+
+  mocha.globalSetup([
+    // ensure the test database is downloaded
+    async () => {
+      fs.mkdirpSync(path.dirname(dbLoc));
+      if (!fs.existsSync(dbLoc)) {
+        try {
+          await new Promise((resolve, reject) => {
+            fetch(DB_URL).then(response => {
+              const dest = fs.createWriteStream(dbLoc);
+              response.body.pipe(dest);
+
+              response.body.on('error', reject);
+              dest.on('error', reject);
+              dest.on('close', () => {
+                resolve(dbLoc);
+              });
+            });
+          });
+        } catch (e) {
+          fail('Failed to download test database: ' + e);
+        }
+      }
+    },
+
+    // Set the CLI version here before activation to ensure we don't accidentally try to download a cli
+    async () => {
+      await workspace.getConfiguration().update('codeQL.cli.executablePath', process.env.CLI_PATH, ConfigurationTarget.Global);
+    },
+
+    // Create the temp directory to be used as extension local storage.
+    () => {
+      const dir = tmp.dirSync();
+      storagePath = fs.realpathSync(dir.name);
+      if (storagePath.substring(0, 2).match(/[A-Z]:/)) {
+        storagePath = storagePath.substring(0, 1).toLocaleLowerCase() + storagePath.substring(1);
+      }
+
+      removeStorage = dir.removeCallback;
+    }
+  ]);
+
+  mocha.globalTeardown([
+    // ensure etension is cleaned up.
+    async () => {
+      const extension = await extensions.getExtension<CodeQLExtensionInterface | {}>('GitHub.vscode-codeql')!.activate();
+      // This shuts down the extension and can only be run after all tests have completed.
+      // If this is not called, then the test process will hang.
+      if ('dispose' in extension) {
+        extension.dispose();
+      }
+    },
+    // ensure temp directory is cleaned up.
+    () => {
+      removeStorage?.();
+    }
+  ]);
+}
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/index.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/index.ts
@@ -0,0 +1,8 @@
+import 'mocha';
+import 'sinon-chai';
+import { runTestsInDirectory } from '../index-template';
+
+// The simple database used throughout the tests
+export function run(): Promise<void> {
+  return runTestsInDirectory(__dirname, true);
+}
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/queries.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/queries.test.ts
@@ -0,0 +1,103 @@
+import { fail } from 'assert';
+import { CancellationToken, extensions, Uri } from 'vscode';
+import * as sinon from 'sinon';
+import * as path from 'path';
+import * as fs from 'fs-extra';
+import 'mocha';
+import { expect } from 'chai';
+
+import { DatabaseItem, DatabaseManager } from '../../databases';
+import { CodeQLExtensionInterface } from '../../extension';
+import { dbLoc, storagePath } from './global.helper';
+import { importArchiveDatabase } from '../../databaseFetcher';
+import { compileAndRunQueryAgainstDatabase } from '../../run-queries';
+import { CodeQLCliServer } from '../../cli';
+import { QueryServerClient } from '../../queryserver-client';
+import { skipIfNoCodeQL } from '../ensureCli';
+
+
+/**
+ * Integration tests for queries
+ */
+describe('Queries', function() {
+  this.timeout(20000);
+
+  before(function() {
+    skipIfNoCodeQL(this);
+  });
+
+  let dbItem: DatabaseItem;
+  let databaseManager: DatabaseManager;
+  let cli: CodeQLCliServer;
+  let qs: QueryServerClient;
+  let sandbox: sinon.SinonSandbox;
+  let progress: sinon.SinonSpy;
+  let token: CancellationToken;
+
+  beforeEach(async () => {
+    sandbox = sinon.createSandbox();
+
+    try {
+      const extension = await extensions.getExtension<CodeQLExtensionInterface | {}>('GitHub.vscode-codeql')!.activate();
+      if ('databaseManager' in extension) {
+        databaseManager = extension.databaseManager;
+        cli = extension.cliServer;
+        qs = extension.qs;
+        cli.quiet = true;
+      } else {
+        throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
+      }
+
+      progress = sandbox.spy();
+      token = {} as CancellationToken;
+
+      // Add a database
+      const uri = Uri.file(dbLoc);
+      const maybeDbItem = await importArchiveDatabase(
+        uri.toString(true),
+        databaseManager,
+        storagePath,
+        progress,
+        token
+      );
+
+      if (!maybeDbItem) {
+        throw new Error('Could not import database');
+      }
+      dbItem = maybeDbItem;
+    } catch (e) {
+      fail(e);
+    }
+  });
+
+  afterEach(() => {
+    try {
+      sandbox.restore();
+    } catch (e) {
+      fail(e);
+    }
+  });
+
+  it('should run a query', async () => {
+    try {
+      const queryPath = path.join(__dirname, 'data', 'simple-query.ql');
+      const result = await compileAndRunQueryAgainstDatabase(
+        cli,
+        qs,
+        dbItem,
+        false,
+        Uri.file(queryPath),
+        progress,
+        token
+      );
+
+      // just check that the query was successful
+      expect(result.database.name).to.eq('db');
+      expect(result.options.queryText).to.eq(fs.readFileSync(queryPath, 'utf8'));
+    } catch (e) {
+      console.error('Test Failed');
+      fail(e);
+    }
+  });
+
+});
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/query.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/query.test.ts
@@ -8,14 +8,14 @@ import { CancellationTokenSource } from 'vscode-jsonrpc';
 import * as messages from '../../pure/messages';
 import * as qsClient from '../../queryserver-client';
 import * as cli from '../../cli';
-import { ProgressReporter, Logger } from '../../logging';
 import { ColumnValue } from '../../pure/bqrs-cli-types';
-import { FindDistributionResultKind } from '../../distribution';
+import { extensions } from 'vscode';
+import { CodeQLExtensionInterface } from '../../extension';
+import { fail } from 'assert';
+import { skipIfNoCodeQL } from '../ensureCli';


-declare module 'url' {
-  export function pathToFileURL(urlStr: string): Url;
-}
+const baseDir = path.join(__dirname, '../../../test/data');

 const tmpDir = tmp.dirSync({ prefix: 'query_test_', keep: false, unsafeCleanup: true });

@@ -61,13 +61,19 @@ type QueryTestCase = {
 // Test cases: queries to run and their expected results.
 const queryTestCases: QueryTestCase[] = [
  {
-    queryPath: path.join(__dirname, '../data/query.ql'),
+    queryPath: path.join(baseDir, 'query.ql'),
    expectedResultSets: {
      '#select': [[42, 3.14159, 'hello world', true]]
    }
  },
  {
-    queryPath: path.join(__dirname, '../data/multiple-result-sets.ql'),
+    queryPath: path.join(baseDir, 'compute-default-strings.ql'),
+    expectedResultSets: {
+      '#select': [[{ label: '(no string representation)' }]]
+    }
+  },
+  {
+    queryPath: path.join(baseDir, 'multiple-result-sets.ql'),
    expectedResultSets: {
      'edges': [[1, 2], [2, 3]],
      '#select': [['s']]
@@ -77,69 +83,33 @@ const queryTestCases: QueryTestCase[] = [

 describe('using the query server', function() {
  before(function() {
-    if (process.env['CODEQL_PATH'] === undefined) {
-      console.log('The environment variable CODEQL_PATH is not set. The query server tests, which require the CodeQL CLI, will be skipped.');
-      this.skip();
-    }
+    skipIfNoCodeQL(this);
  });

  // Note this does not work with arrow functions as the test case bodies:
  // ensure they are all written with standard anonymous functions.
-  this.timeout(10000);
+  this.timeout(20000);

-  const codeQlPath = process.env['CODEQL_PATH']!;
  let qs: qsClient.QueryServerClient;
  let cliServer: cli.CodeQLCliServer;
  const queryServerStarted = new Checkpoint<void>();
-  after(() => {
-    if (qs) {
-      qs.dispose();
-    }
-    if (cliServer) {
-      cliServer.dispose();
+
+  beforeEach(async () => {
+    try {
+      const extension = await extensions.getExtension<CodeQLExtensionInterface | {}>('GitHub.vscode-codeql')!.activate();
+      if ('cliServer' in extension && 'qs' in extension) {
+        cliServer = extension.cliServer;
+        qs = extension.qs;
+        cliServer.quiet = true;
+      } else {
+        throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
+      }
+    } catch (e) {
+      fail(e);
    }
  });

  it('should be able to start the query server', async function() {
-    const consoleProgressReporter: ProgressReporter = {
-      report: (v: { message: string }) => console.log(`progress reporter says ${v.message}`)
-    };
-    const logger: Logger = {
-      log: async (s: string) => console.log('logger says', s),
-      show: () => { /**/ },
-      removeAdditionalLogLocation: async () => { /**/ },
-      getBaseLocation: () => ''
-    };
-    cliServer = new cli.CodeQLCliServer(
-      {
-        async getCodeQlPathWithoutVersionCheck(): Promise<string | undefined> {
-          return codeQlPath;
-        },
-        getDistribution: async () => {
-          return {
-            kind: FindDistributionResultKind.NoDistribution
-          };
-        }
-      },
-      {
-        numberTestThreads: 2
-      },
-      logger
-    );
-    qs = new qsClient.QueryServerClient(
-      {
-        codeQlPath,
-        numThreads: 1,
-        queryMemoryMb: 1024,
-        timeoutSecs: 1000,
-        debug: false
-      },
-      cliServer,
-      {
-        logger
-      },
-      task => task(consoleProgressReporter, token)
-    );
    await qs.startQueryServer();
    queryServerStarted.resolve();
  });
@@ -156,7 +126,7 @@ describe('using the query server', function() {
      try {
        const qlProgram: messages.QlProgram = {
          libraryPath: [],
-          dbschemePath: path.join(__dirname, '../data/test.dbscheme'),
+          dbschemePath: path.join(baseDir, 'test.dbscheme'),
          queryPath: queryTestCase.queryPath
        };
        const params: messages.CompileQueryParams = {
@@ -168,6 +138,7 @@ describe('using the query server', function() {
            localChecking: false,
            noComputeGetUrl: false,
            noComputeToString: false,
+            computeDefaultStrings: true
          },
          queryToCheck: qlProgram,
          resultPath: COMPILED_QUERY_PATH,
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/run-cli.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/run-cli.test.ts
@@ -0,0 +1,53 @@
+import { expect } from 'chai';
+import { extensions } from 'vscode';
+import { SemVer } from 'semver';
+
+import { CodeQLCliServer } from '../../cli';
+import { CodeQLExtensionInterface } from '../../extension';
+import { skipIfNoCodeQL } from '../ensureCli';
+import { getOnDiskWorkspaceFolders } from '../../helpers';
+
+/**
+ * Perform proper integration tests by running the CLI
+ */
+describe('Use cli', function() {
+  this.timeout(60000);
+
+  let cli: CodeQLCliServer;
+
+  beforeEach(async () => {
+    const extension = await extensions.getExtension<CodeQLExtensionInterface | {}>('GitHub.vscode-codeql')!.activate();
+    if ('cliServer' in extension) {
+      cli = extension.cliServer;
+    } else {
+      throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
+    }
+  });
+
+  it('should have the correct version of the cli', async () => {
+    expect(
+      (await cli.getVersion()).toString()
+    ).to.eq(
+      new SemVer(process.env.CLI_VERSION || '').toString()
+    );
+  });
+
+  it('should resolve ram', async () => {
+    const result = await (cli as any).resolveRam(8192);
+    expect(result).to.deep.eq([
+      '-J-Xmx4096M',
+      '--off-heap-ram=4096'
+    ]);
+  });
+
+  it('should resolve query packs', async function() {
+    skipIfNoCodeQL(this);
+    const qlpacks = await cli.resolveQlpacks(getOnDiskWorkspaceFolders());
+    // should have a bunch of qlpacks. just check that a few known ones exist
+    expect(qlpacks['codeql-cpp']).not.to.be.undefined;
+    expect(qlpacks['codeql-csharp']).not.to.be.undefined;
+    expect(qlpacks['codeql-java']).not.to.be.undefined;
+    expect(qlpacks['codeql-javascript']).not.to.be.undefined;
+    expect(qlpacks['codeql-python']).not.to.be.undefined;
+  });
+});
--- a/extensions/ql-vscode/src/vscode-tests/ensureCli.ts
+++ b/extensions/ql-vscode/src/vscode-tests/ensureCli.ts
@@ -0,0 +1,150 @@
+import * as fs from 'fs-extra';
+import * as path from 'path';
+import { DistributionManager, extractZipArchive, codeQlLauncherName } from '../distribution';
+import fetch from 'node-fetch';
+import { workspace } from 'vscode';
+
+/**
+ * This module ensures that the proper CLI is available for tests of the extension.
+ * There are two environment variables to control this module:
+ *
+ *     - CLI_VERSION: The version of the CLI to install. Defaults to the most recent
+ *       version. Note that for now, we must maintain the default version by hand.
+ *
+ *     - CLI_BASE_DIR: The base dir where the CLI will be downloaded and unzipped.
+ *       The download location is `${CLI_BASE_DIR}/assets` and the unzip loction is
+ *       `${CLI_BASE_DIR}/${CLI_VERSION}`
+ *
+ * After downloading and unzipping, a new environment variable is set:
+ *
+ *     - CLI_PATH: Points to the cli executable for the specified CLI_VERSION. This
+ *       is variable is available in the unit tests and will be used as the value
+ *       for `codeQL.cli.executablePath`.
+ *
+ * As an optimization, the cli will not be unzipped again if the executable already
+ * exists. And the cli will not be re-downloaded if the zip already exists.
+ */
+
+process.on('unhandledRejection', e => {
+  console.error('Unhandled rejection.');
+  console.error(e);
+  process.exit(-1);
+});
+
+const _1MB = 1024 * 1024;
+const _10MB = _1MB * 10;
+
+// CLI version to test. Hard code the latest as default. And be sure
+// to update the env if it is not otherwise set.
+const CLI_VERSION = process.env.CLI_VERSION || 'v2.4.0';
+process.env.CLI_VERSION = CLI_VERSION;
+
+// Base dir where CLIs will be downloaded into
+// By default, put it in the `build` directory in the root of the extension.
+const CLI_BASE_DIR = process.env.CLI_DIR || path.normalize(path.join(__dirname, '../../build/cli'));
+
+export async function ensureCli(useCli: boolean) {
+  try {
+    if (!useCli) {
+      console.log('Not downloading CLI. It is not being used.');
+      return;
+    }
+
+    const assetName = DistributionManager.getRequiredAssetName();
+    const url = getCliDownloadUrl(assetName);
+    const unzipDir = getCliUnzipDir();
+    const downloadedFilePath = getDownloadFilePath(assetName);
+    const executablePath = path.join(getCliUnzipDir(), 'codeql', codeQlLauncherName());
+
+    // Use this environment variable to se to the `codeQL.cli.executablePath` in tests
+    process.env.CLI_PATH = executablePath;
+
+    if (fs.existsSync(executablePath)) {
+      console.log(`CLI version ${CLI_VERSION} is found ${executablePath}. Not going to download again.`);
+      return;
+    }
+
+    if (!fs.existsSync(downloadedFilePath)) {
+      console.log(`CLI version ${CLI_VERSION} zip file not found. Downloading from '${url}' into '${downloadedFilePath}'.`);
+
+      const assetStream = await fetch(url);
+      const contentLength = Number(assetStream.headers.get('content-length') || 0);
+      console.log('Total content size', Math.round(contentLength / _1MB), 'MB');
+      const archiveFile = fs.createWriteStream(downloadedFilePath);
+      const body = assetStream.body;
+      await new Promise<void>((resolve, reject) => {
+        let numBytesDownloaded = 0;
+        let lastMessage = 0;
+        body.on('data', (data) => {
+          numBytesDownloaded += data.length;
+          if (numBytesDownloaded - lastMessage > _10MB) {
+            console.log('Downloaded', Math.round(numBytesDownloaded / _1MB), 'MB');
+            lastMessage = numBytesDownloaded;
+          }
+          archiveFile.write(data);
+        });
+        body.on('finish', () => {
+          archiveFile.end(() => {
+            console.log('Finished download into', downloadedFilePath);
+            resolve();
+          });
+        });
+        body.on('error', reject);
+      });
+    } else {
+      console.log(`CLI version ${CLI_VERSION} zip file found at '${downloadedFilePath}'.`);
+    }
+
+    console.log(`Unzipping into '${unzipDir}'`);
+    fs.mkdirpSync(unzipDir);
+    await extractZipArchive(downloadedFilePath, unzipDir);
+    console.log('Done.');
+  } catch (e) {
+    console.error('Failed to download CLI.');
+    console.error(e);
+    process.exit(-1);
+  }
+}
+
+/**
+ * Heuristically determines if the codeql libraries are installed in this
+ * workspace. Looks for the existance of a folder whose path ends in `/codeql`
+ */
+function hasCodeQL() {
+  const folders = workspace.workspaceFolders;
+  return !!folders?.some(folder => folder.uri.path.endsWith('/codeql'));
+}
+
+export function skipIfNoCodeQL(context: Mocha.Context) {
+  if (!hasCodeQL()) {
+    console.log([
+      'The CodeQL libraries are not available as a folder in this workspace.',
+      'To fix in CI: checkout the github/codeql repository and set the \'TEST_CODEQL_PATH\' environment variable to the checked out directory.',
+      'To fix when running from vs code, see the comment in the launch.json file in the \'Launch Integration Tests - With CLI\' section.'
+    ].join('\n\n'));
+    context.skip();
+  }
+}
+
+/**
+ * Url to download from
+ */
+function getCliDownloadUrl(assetName: string) {
+  return `https://github.com/github/codeql-cli-binaries/releases/download/${CLI_VERSION}/${assetName}`;
+}
+
+/**
+ * Directory to place the downloaded cli into
+ */
+function getDownloadFilePath(assetName: string) {
+  const dir = path.join(CLI_BASE_DIR, 'assets', CLI_VERSION);
+  fs.mkdirpSync(dir);
+  return path.join(dir, assetName);
+}
+
+/**
+ * Directory to unzip the downloaded cli into.
+ */
+function getCliUnzipDir() {
+  return path.join(CLI_BASE_DIR, CLI_VERSION);
+}
--- a/extensions/ql-vscode/src/vscode-tests/index-template.ts
+++ b/extensions/ql-vscode/src/vscode-tests/index-template.ts
@@ -1,6 +1,18 @@
 import * as path from 'path';
 import * as Mocha from 'mocha';
 import * as glob from 'glob';
+import { ensureCli } from './ensureCli';
+
+
+// Use this handler to avoid swallowing unhandled rejections.
+process.on('unhandledRejection', e => {
+  console.error('Unhandled rejection.');
+  console.error(e);
+  // Must use a setTimeout in order to ensure the log is fully flushed before exiting
+  setTimeout(() => {
+    process.exit(-1);
+  }, 2000);
+});

 /**
 * Helper function that runs all Mocha tests found in the
@@ -26,35 +38,51 @@ import * as glob from 'glob';
 * After https://github.com/microsoft/TypeScript/issues/420 is implemented,
 * this pattern can be expressed more neatly using a module interface.
 */
-export function runTestsInDirectory(testsRoot: string): Promise<void> {
+export async function runTestsInDirectory(testsRoot: string, useCli = false): Promise<void> {
  // Create the mocha test
  const mocha = new Mocha({
    ui: 'bdd',
    color: true
  });

-  return new Promise((c, e) => {
-    console.log(`Adding test cases from ${testsRoot}`);
-    glob('**/**.test.js', { cwd: testsRoot }, (err, files) => {
+  await ensureCli(useCli);
+
+  return new Promise((resolve, reject) => {
+    console.log(`Adding test cases and helpers from ${testsRoot}`);
+    glob('**/**.js', { cwd: testsRoot }, (err, files) => {
      if (err) {
-        return e(err);
+        return reject(err);
      }

-
-      // Add files to the test suite
-      files.forEach(f => mocha.addFile(path.resolve(testsRoot, f)));
-
      try {
+        // Add test files to the test suite
+        files
+          .filter(f => f.endsWith('.test.js'))
+          .forEach(f => {
+            console.log(`Adding test file ${f}`);
+            mocha.addFile(path.resolve(testsRoot, f));
+          });
+
+        // Add helpers. Helper files add global setup and teardown blocks
+        // for a test run.
+        files
+          .filter(f => f.endsWith('.helper.js'))
+          .forEach(f => {
+            console.log(`Executing helper ${f}`);
+            const helper = require(path.resolve(testsRoot, f)).default;
+            helper(mocha);
+          });
+
        // Run the mocha test
        mocha.run(failures => {
          if (failures > 0) {
-            e(new Error(`${failures} tests failed.`));
+            reject(new Error(`${failures} tests failed.`));
          } else {
-            c();
+            resolve();
          }
        });
      } catch (err) {
-        e(err);
+        reject(err);
      }
    });
  });
--- a/extensions/ql-vscode/src/vscode-tests/minimal-workspace/activation.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/minimal-workspace/activation.test.ts
@@ -9,6 +9,7 @@ import * as determiningSelectedQueryTest from './determining-selected-query-test
 chai.use(chaiAsPromised);

 describe('launching with a minimal workspace', async () => {
+
  const ext = vscode.extensions.getExtension('GitHub.vscode-codeql');
  it('should install the extension', () => {
    assert(ext);
@@ -19,6 +20,9 @@ describe('launching with a minimal workspace', async () => {
  });

  it('should activate the extension when a .ql file is opened', async function() {
+    this.timeout(60000);
+    await delay();
+
    const folders = vscode.workspace.workspaceFolders;
    assert(folders && folders.length === 1);
    const folderPath = folders![0].uri.fsPath;
@@ -26,10 +30,13 @@ describe('launching with a minimal workspace', async () => {
    const document = await vscode.workspace.openTextDocument(documentPath);
    assert(document.languageId === 'ql');
    // Delay slightly so that the extension has time to activate.
-    this.timeout(4000);
-    await new Promise(resolve => setTimeout(resolve, 2000));
+    await delay();
    assert(ext!.isActive);
  });
+
+  async function delay() {
+    await new Promise(resolve => setTimeout(resolve, 4000));
+  }
 });

 determiningSelectedQueryTest.run();
--- a/extensions/ql-vscode/src/vscode-tests/minimal-workspace/databases.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/minimal-workspace/databases.test.ts
@@ -0,0 +1,438 @@
+import 'vscode-test';
+import 'mocha';
+import * as sinon from 'sinon';
+import * as tmp from 'tmp';
+import * as fs from 'fs-extra';
+import * as path from 'path';
+import { expect } from 'chai';
+import { CancellationToken, ExtensionContext, Uri, workspace } from 'vscode';
+
+import {
+  DatabaseEventKind,
+  DatabaseManager,
+  DatabaseItemImpl,
+  DatabaseContents,
+  FullDatabaseOptions
+} from '../../databases';
+import { Logger } from '../../logging';
+import { QueryServerClient } from '../../queryserver-client';
+import { registerDatabases } from '../../pure/messages';
+import { ProgressCallback } from '../../commandRunner';
+import { CodeQLCliServer } from '../../cli';
+import { encodeArchiveBasePath, encodeSourceArchiveUri } from '../../archive-filesystem-provider';
+
+describe('databases', () => {
+
+  const MOCK_DB_OPTIONS: FullDatabaseOptions = {
+    dateAdded: 123,
+    ignoreSourceArchive: false,
+    language: ''
+  };
+
+  let databaseManager: DatabaseManager;
+  let updateSpy: sinon.SinonSpy;
+  let getSpy: sinon.SinonStub;
+  let dbChangedHandler: sinon.SinonSpy;
+  let sendRequestSpy: sinon.SinonSpy;
+  let supportsDatabaseRegistrationSpy: sinon.SinonStub;
+  let supportsLanguageNameSpy: sinon.SinonStub;
+  let resolveDatabaseSpy: sinon.SinonStub;
+
+  let sandbox: sinon.SinonSandbox;
+  let dir: tmp.DirResult;
+
+  beforeEach(() => {
+    dir = tmp.dirSync();
+
+    sandbox = sinon.createSandbox();
+    updateSpy = sandbox.spy();
+    getSpy = sandbox.stub();
+    getSpy.returns([]);
+    sendRequestSpy = sandbox.stub();
+    dbChangedHandler = sandbox.spy();
+    supportsDatabaseRegistrationSpy = sandbox.stub();
+    supportsDatabaseRegistrationSpy.resolves(true);
+    supportsLanguageNameSpy = sandbox.stub();
+    resolveDatabaseSpy = sandbox.stub();
+    databaseManager = new DatabaseManager(
+      {
+        workspaceState: {
+          update: updateSpy,
+          get: getSpy
+        },
+        // pretend like databases added in the temp dir are controlled by the extension
+        // so that they are deleted upon removal
+        storagePath: dir.name
+      } as unknown as ExtensionContext,
+      {
+        sendRequest: sendRequestSpy,
+        supportsDatabaseRegistration: supportsDatabaseRegistrationSpy
+      } as unknown as QueryServerClient,
+      {
+        supportsLanguageName: supportsLanguageNameSpy,
+        resolveDatabase: resolveDatabaseSpy
+      } as unknown as CodeQLCliServer,
+      {} as Logger,
+    );
+
+    // Unfortunately, during a test it is not possible to convert from
+    // a single root workspace to multi-root, so must stub out relevant
+    // functions
+    sandbox.stub(workspace, 'updateWorkspaceFolders');
+    sandbox.spy(workspace, 'onDidChangeWorkspaceFolders');
+  });
+
+  afterEach(async () => {
+    dir.removeCallback();
+    databaseManager.dispose();
+    sandbox.restore();
+  });
+
+  it('should fire events when adding and removing a db item', async () => {
+    const mockDbItem = createMockDB();
+    const spy = sinon.spy();
+    databaseManager.onDidChangeDatabaseItem(spy);
+    await (databaseManager as any).addDatabaseItem(
+      {} as ProgressCallback,
+      {} as CancellationToken,
+      mockDbItem
+    );
+
+    expect((databaseManager as any)._databaseItems).to.deep.eq([mockDbItem]);
+    expect(updateSpy).to.have.been.calledWith('databaseList', [{
+      options: MOCK_DB_OPTIONS,
+      uri: dbLocationUri().toString(true)
+    }]);
+    expect(spy).to.have.been.calledWith({
+      item: undefined,
+      kind: DatabaseEventKind.Add
+    });
+
+    sinon.reset();
+
+    // now remove the item
+    await databaseManager.removeDatabaseItem(
+      {} as ProgressCallback,
+      {} as CancellationToken,
+      mockDbItem,
+    );
+    expect((databaseManager as any)._databaseItems).to.deep.eq([]);
+    expect(updateSpy).to.have.been.calledWith('databaseList', []);
+    expect(spy).to.have.been.calledWith({
+      item: undefined,
+      kind: DatabaseEventKind.Remove
+    });
+  });
+
+  it('should rename a db item and emit an event', async () => {
+    const mockDbItem = createMockDB();
+    const spy = sinon.spy();
+    databaseManager.onDidChangeDatabaseItem(spy);
+    await (databaseManager as any).addDatabaseItem(
+      {} as ProgressCallback,
+      {} as CancellationToken,
+      mockDbItem
+    );
+
+    sinon.restore();
+
+    await databaseManager.renameDatabaseItem(mockDbItem, 'new name');
+
+    expect(mockDbItem.name).to.eq('new name');
+    expect(updateSpy).to.have.been.calledWith('databaseList', [{
+      options: { ...MOCK_DB_OPTIONS, displayName: 'new name' },
+      uri: dbLocationUri().toString(true)
+    }]);
+
+    expect(spy).to.have.been.calledWith({
+      item: undefined,
+      kind: DatabaseEventKind.Rename
+    });
+  });
+
+  describe('add / remove database items', () => {
+    it('should add a database item', async () => {
+      const spy = sandbox.spy();
+      databaseManager.onDidChangeDatabaseItem(spy);
+      const mockDbItem = createMockDB();
+
+      await (databaseManager as any).addDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+
+      expect(databaseManager.databaseItems).to.deep.eq([mockDbItem]);
+      expect(updateSpy).to.have.been.calledWith('databaseList', [{
+        uri: dbLocationUri().toString(true),
+        options: MOCK_DB_OPTIONS
+      }]);
+
+      const mockEvent = {
+        item: undefined,
+        kind: DatabaseEventKind.Add
+      };
+      expect(spy).to.have.been.calledWith(mockEvent);
+    });
+
+    it('should add a database item source archive', async function() {
+      const mockDbItem = createMockDB();
+      mockDbItem.name = 'xxx';
+      await (databaseManager as any).addDatabaseSourceArchiveFolder(mockDbItem);
+
+      // workspace folders should be updated. We can only check the mocks since
+      // when running as a test, we are not allowed to update the workspace folders
+      expect(workspace.updateWorkspaceFolders).to.have.been.calledWith(1, 0, {
+        name: '[xxx source archive]',
+        // must use a matcher here since vscode URIs with the same path
+        // are not always equal due to internal state.
+        uri: sinon.match.has('fsPath', encodeArchiveBasePath(sourceLocationUri().fsPath).fsPath)
+      });
+    });
+
+    it('should remove a database item', async () => {
+      const mockDbItem = createMockDB();
+      sandbox.stub(fs, 'remove').resolves();
+
+      // pretend that this item is the first workspace folder in the list
+      sandbox.stub(mockDbItem, 'belongsToSourceArchiveExplorerUri').returns(true);
+
+      await (databaseManager as any).addDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+
+      updateSpy.resetHistory();
+
+      await databaseManager.removeDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+
+      expect(databaseManager.databaseItems).to.deep.eq([]);
+      expect(updateSpy).to.have.been.calledWith('databaseList', []);
+      // should remove the folder
+      expect(workspace.updateWorkspaceFolders).to.have.been.calledWith(0, 1);
+
+      // should also delete the db contents
+      expect(fs.remove).to.have.been.calledWith(mockDbItem.databaseUri.fsPath);
+    });
+
+    it('should remove a database item outside of the extension controlled area', async () => {
+      const mockDbItem = createMockDB();
+      sandbox.stub(fs, 'remove').resolves();
+
+      // pretend that this item is the first workspace folder in the list
+      sandbox.stub(mockDbItem, 'belongsToSourceArchiveExplorerUri').returns(true);
+
+      await (databaseManager as any).addDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+      updateSpy.resetHistory();
+
+      // pretend that the database location is not controlled by the extension
+      (databaseManager as any).ctx.storagePath = 'hucairz';
+
+      await databaseManager.removeDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+
+      expect(databaseManager.databaseItems).to.deep.eq([]);
+      expect(updateSpy).to.have.been.calledWith('databaseList', []);
+      // should remove the folder
+      expect(workspace.updateWorkspaceFolders).to.have.been.calledWith(0, 1);
+
+      // should NOT delete the db contents
+      expect(fs.remove).not.to.have.been.called;
+    });
+
+    it('should register and deregister a database when adding and removing it', async () => {
+      // similar test as above, but also check the call to sendRequestSpy to make sure they send the
+      // registration messages.
+      const mockDbItem = createMockDB();
+      const registration = {
+        databases: [{
+          dbDir: mockDbItem.contents!.datasetUri.fsPath,
+          workingSet: 'default'
+        }]
+      };
+
+      sandbox.stub(fs, 'remove').resolves();
+
+      await (databaseManager as any).addDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+      // Should have registered this database
+      expect(sendRequestSpy).to.have.been.calledWith(registerDatabases, registration, {}, {});
+
+      await databaseManager.removeDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+
+      // Should have deregistered this database
+      expect(sendRequestSpy).to.have.been.calledWith(registerDatabases, registration, {}, {});
+    });
+
+    it('should avoid registration when query server does not support it', async () => {
+      // similar test as above, but now pretend query server doesn't support registration
+      supportsDatabaseRegistrationSpy.resolves(false);
+      const mockDbItem = createMockDB();
+      sandbox.stub(fs, 'remove').resolves();
+
+      await (databaseManager as any).addDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+      // Should NOT have registered this database
+      expect(sendRequestSpy).not.to.have.been.called;
+
+      await databaseManager.removeDatabaseItem(
+        {} as ProgressCallback,
+        {} as CancellationToken,
+        mockDbItem
+      );
+
+      // Should NOT have deregistered this database
+      expect(sendRequestSpy).not.to.have.been.called;
+    });
+  });
+
+  describe('resolveSourceFile', () => {
+    it('should fail to resolve when not a uri', () => {
+      const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
+      (db as any)._contents.sourceArchiveUri = undefined;
+      expect(() => db.resolveSourceFile('abc')).to.throw('Scheme is missing');
+    });
+
+    it('should fail to resolve when not a file uri', () => {
+      const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
+      (db as any)._contents.sourceArchiveUri = undefined;
+      expect(() => db.resolveSourceFile('http://abc')).to.throw('Invalid uri scheme');
+    });
+
+    describe('no source archive', () => {
+      it('should resolve undefined', () => {
+        const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
+        (db as any)._contents.sourceArchiveUri = undefined;
+        const resolved = db.resolveSourceFile(undefined);
+        expect(resolved.toString(true)).to.eq(dbLocationUri().toString(true));
+      });
+
+      it('should resolve an empty file', () => {
+        const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
+        (db as any)._contents.sourceArchiveUri = undefined;
+        const resolved = db.resolveSourceFile('file:');
+        expect(resolved.toString()).to.eq('file:///');
+      });
+    });
+
+    describe('zipped source archive', () => {
+      it('should encode a source archive url', () => {
+        const db = createMockDB(encodeSourceArchiveUri({
+          sourceArchiveZipPath: 'sourceArchive-uri',
+          pathWithinSourceArchive: 'def'
+        }));
+        const resolved = db.resolveSourceFile(Uri.file('abc').toString());
+
+        // must recreate an encoded archive uri instead of typing out the
+        // text since the uris will be different on windows and ubuntu.
+        expect(resolved.toString()).to.eq(encodeSourceArchiveUri({
+          sourceArchiveZipPath: 'sourceArchive-uri',
+          pathWithinSourceArchive: 'def/abc'
+        }).toString());
+      });
+
+      it('should encode a source archive url with trailing slash', () => {
+        const db = createMockDB(encodeSourceArchiveUri({
+          sourceArchiveZipPath: 'sourceArchive-uri',
+          pathWithinSourceArchive: 'def/'
+        }));
+        const resolved = db.resolveSourceFile(Uri.file('abc').toString());
+
+        // must recreate an encoded archive uri instead of typing out the
+        // text since the uris will be different on windows and ubuntu.
+        expect(resolved.toString()).to.eq(encodeSourceArchiveUri({
+          sourceArchiveZipPath: 'sourceArchive-uri',
+          pathWithinSourceArchive: 'def/abc'
+        }).toString());
+      });
+
+      it('should encode an empty source archive url', () => {
+        const db = createMockDB(encodeSourceArchiveUri({
+          sourceArchiveZipPath: 'sourceArchive-uri',
+          pathWithinSourceArchive: 'def'
+        }));
+        const resolved = db.resolveSourceFile('file:');
+        expect(resolved.toString()).to.eq('codeql-zip-archive://1-18/sourceArchive-uri/def/');
+      });
+    });
+
+    it('should handle an empty file', () => {
+      const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
+      const resolved = db.resolveSourceFile('');
+      expect(resolved.toString()).to.eq('file:///sourceArchive-uri/');
+    });
+  });
+
+  it('should not support the primary language', async () => {
+    supportsLanguageNameSpy.resolves(false);
+
+    const result = (await (databaseManager as any).getPrimaryLanguage('hucairz'));
+    expect(result).to.be.undefined;
+  });
+
+  it('should get the primary language', async () => {
+    supportsLanguageNameSpy.resolves(true);
+    resolveDatabaseSpy.resolves({
+      languages: ['python']
+    });
+    const result = (await (databaseManager as any).getPrimaryLanguage('hucairz'));
+    expect(result).to.eq('python');
+  });
+
+  it('should handle missing the primary language', async () => {
+    supportsLanguageNameSpy.resolves(true);
+    resolveDatabaseSpy.resolves({
+      languages: []
+    });
+    const result = (await (databaseManager as any).getPrimaryLanguage('hucairz'));
+    expect(result).to.eq('');
+  });
+
+  function createMockDB(
+    // source archive location must be a real(-ish) location since
+    // tests will add this to the workspace location
+    sourceArchiveUri = sourceLocationUri(),
+    databaseUri = dbLocationUri()
+  ): DatabaseItemImpl {
+
+    return new DatabaseItemImpl(
+      databaseUri,
+      {
+        sourceArchiveUri,
+        datasetUri: databaseUri
+      } as DatabaseContents,
+      MOCK_DB_OPTIONS,
+      dbChangedHandler,
+    );
+  }
+
+  function sourceLocationUri() {
+    return Uri.file(path.join(dir.name, 'src.zip'));
+  }
+
+  function dbLocationUri() {
+    return Uri.file(path.join(dir.name, 'db'));
+  }
+});
--- a/extensions/ql-vscode/src/vscode-tests/minimal-workspace/index.ts
+++ b/extensions/ql-vscode/src/vscode-tests/minimal-workspace/index.ts
@@ -1,4 +1,12 @@
 import { runTestsInDirectory } from '../index-template';
+
+import * as sinonChai from 'sinon-chai';
+import * as chai from 'chai';
+import * as chaiAsPromised from 'chai-as-promised';
+chai.use(chaiAsPromised);
+chai.use(sinonChai);
+
+
 export function run(): Promise<void> {
  return runTestsInDirectory(__dirname);
 }
--- a/extensions/ql-vscode/src/vscode-tests/minimal-workspace/qltest-discovery.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/minimal-workspace/qltest-discovery.test.ts
@@ -2,20 +2,27 @@ import 'vscode-test';
 import 'mocha';
 import { Uri, WorkspaceFolder } from 'vscode';
 import { expect } from 'chai';
+import * as fs from 'fs-extra';
+import * as sinon from 'sinon';

 import { QLTestDiscovery } from '../../qltest-discovery';

 describe('qltest-discovery', () => {
  describe('discoverTests', () => {
-    it('should run discovery', async () => {
-      const baseUri = Uri.parse('file:/a/b');
-      const baseDir = baseUri.fsPath;
-      const cDir = Uri.parse('file:/a/b/c').fsPath;
-      const dFile = Uri.parse('file:/a/b/c/d.ql').fsPath;
-      const eFile = Uri.parse('file:/a/b/c/e.ql').fsPath;
-      const hDir = Uri.parse('file:/a/b/c/f/g/h').fsPath;
-      const iFile = Uri.parse('file:/a/b/c/f/g/h/i.ql').fsPath;
-      const qlTestDiscover = new QLTestDiscovery(
+    let sandbox: sinon.SinonSandbox;
+
+    const baseUri = Uri.parse('file:/a/b');
+    const baseDir = baseUri.fsPath;
+    const cDir = Uri.parse('file:/a/b/c').fsPath;
+    const dFile = Uri.parse('file:/a/b/c/d.ql').fsPath;
+    const eFile = Uri.parse('file:/a/b/c/e.ql').fsPath;
+    const hDir = Uri.parse('file:/a/b/c/f/g/h').fsPath;
+    const iFile = Uri.parse('file:/a/b/c/f/g/h/i.ql').fsPath;
+    let qlTestDiscover: QLTestDiscovery;
+
+    beforeEach(() => {
+      sandbox = sinon.createSandbox();
+      qlTestDiscover = new QLTestDiscovery(
        {
          uri: baseUri,
          name: 'My tests'
@@ -31,6 +38,15 @@ describe('qltest-discovery', () => {
        } as any
      );

+    });
+
+    afterEach(() => {
+      sandbox.restore();
+    });
+
+    it('should run discovery', async () => {
+      sandbox.stub(fs, 'pathExists').resolves(true);
+
      const result = await (qlTestDiscover as any).discover();
      expect(result.watchPath).to.eq(baseDir);
      expect(result.testDirectory.path).to.eq(baseDir);
@@ -56,5 +72,15 @@ describe('qltest-discovery', () => {
      expect(children[0].path).to.eq(iFile);
      expect(children[0].name).to.eq('i.ql');
    });
+
+    it('should avoid discovery if a folder does not exist', async () => {
+      sandbox.stub(fs, 'pathExists').resolves(false);
+      const result = await (qlTestDiscover as any).discover();
+      expect(result.watchPath).to.eq(baseDir);
+      expect(result.testDirectory.path).to.eq(baseDir);
+      expect(result.testDirectory.name).to.eq('My tests');
+
+      expect(result.testDirectory.children.length).to.eq(0);
+    });
  });
 });
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/astViewer.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/astViewer.test.ts
@@ -15,19 +15,25 @@ const expect = chai.expect;

 describe('AstViewer', () => {
  let astRoots: AstItem[];
-  let viewer: AstViewer;
+  let viewer: AstViewer | undefined;
+  let sandbox: sinon.SinonSandbox;
  beforeEach(async () => {
+    sandbox = sinon.createSandbox();
    // the ast is stored in yaml because there are back pointers
    // making a json representation impossible.
    // The complication here is that yaml files are not copied into the 'out' directory by tsc.
    astRoots = await buildAst();

-    sinon.stub(commands, 'registerCommand');
-    sinon.stub(commands, 'executeCommand');
+    sandbox.stub(commands, 'registerCommand');
+    sandbox.stub(commands, 'executeCommand');
  });

  afterEach(() => {
-    sinon.restore();
+    sandbox.restore();
+    if (viewer) {
+      viewer.dispose();
+      viewer = undefined;
+    }
  });

  it('should update the viewer roots', () => {
@@ -56,7 +62,6 @@ describe('AstViewer', () => {
    doSelectionTest(undefined, new Range(2, 3, 4, 5));
  });

-
  function doSelectionTest(
    expectedSelection: any,
    selectionRange: Range | undefined,
@@ -65,7 +70,7 @@ describe('AstViewer', () => {
    const item = {} as DatabaseItem;
    viewer = new AstViewer();
    viewer.updateRoots(astRoots, item, fsPath);
-    const spy = sinon.spy();
+    const spy = sandbox.spy();
    (viewer as any).treeView.reveal = spy;
    Object.defineProperty((viewer as any).treeView, 'visible', {
      value: true
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/contextual/queryResolver.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/contextual/queryResolver.test.ts
@@ -16,7 +16,8 @@ const expect = chai.expect;
 describe('queryResolver', () => {
  let module: Record<string, Function>;
  let writeFileSpy: sinon.SinonSpy;
-  let resolveDatasetFolderSpy: sinon.SinonStub;
+  let getQlPackForDbschemeSpy: sinon.SinonStub;
+  let getPrimaryDbschemeSpy: sinon.SinonStub;
  let mockCli: Record<string, sinon.SinonStub>;
  beforeEach(() => {
    mockCli = {
@@ -60,7 +61,7 @@ describe('queryResolver', () => {

  describe('qlpackOfDatabase', () => {
    it('should get the qlpack of a database', async () => {
-      resolveDatasetFolderSpy.returns({ qlpack: 'my-qlpack' });
+      getQlPackForDbschemeSpy.resolves('my-qlpack');
      const db = {
        contents: {
          datasetUri: {
@@ -70,20 +71,22 @@ describe('queryResolver', () => {
      };
      const result = await module.qlpackOfDatabase(mockCli, db);
      expect(result).to.eq('my-qlpack');
-      expect(resolveDatasetFolderSpy).to.have.been.calledWith(mockCli, '/path/to/database');
+      expect(getPrimaryDbschemeSpy).to.have.been.calledWith('/path/to/database');
    });
  });

  function createModule() {
    writeFileSpy = sinon.spy();
-    resolveDatasetFolderSpy = sinon.stub();
+    getQlPackForDbschemeSpy = sinon.stub();
+    getPrimaryDbschemeSpy = sinon.stub();
    return proxyquire('../../../contextual/queryResolver', {
      'fs-extra': {
        writeFile: writeFileSpy
      },

      '../helpers': {
-        resolveDatasetFolder: resolveDatasetFolderSpy,
+        getQlPackForDbscheme: getQlPackForDbschemeSpy,
+        getPrimaryDbscheme: getPrimaryDbschemeSpy,
        getOnDiskWorkspaceFolders: () => ({}),
        showAndLogErrorMessage: () => ({})
      }
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/databases.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/databases.test.ts
@@ -1,188 +0,0 @@
-import 'vscode-test';
-import 'mocha';
-import * as sinon from 'sinon';
-import { expect } from 'chai';
-import { ExtensionContext, Uri } from 'vscode';
-
-import {
-  DatabaseEventKind,
-  DatabaseItem,
-  DatabaseManager,
-  DatabaseItemImpl,
-  DatabaseContents,
-  isLikelyDbLanguageFolder
-} from '../../databases';
-import { QueryServerConfig } from '../../config';
-import { Logger } from '../../logging';
-import { encodeSourceArchiveUri } from '../../archive-filesystem-provider';
-
-describe('databases', () => {
-  let databaseManager: DatabaseManager;
-  let updateSpy: sinon.SinonSpy;
-
-  beforeEach(() => {
-    updateSpy = sinon.spy();
-    databaseManager = new DatabaseManager(
-      {
-        workspaceState: {
-          update: updateSpy,
-          get: sinon.stub()
-        },
-      } as unknown as ExtensionContext,
-      {} as QueryServerConfig,
-      {} as Logger,
-    );
-  });
-
-  it('should fire events when adding and removing a db item', () => {
-    const mockDbItem = {
-      databaseUri: { path: 'file:/abc' },
-      name: 'abc',
-      getPersistedState() {
-        return this.name;
-      }
-    };
-    const spy = sinon.spy();
-    databaseManager.onDidChangeDatabaseItem(spy);
-    (databaseManager as any).addDatabaseItem(mockDbItem);
-
-    expect((databaseManager as any)._databaseItems).to.deep.eq([mockDbItem]);
-    expect(updateSpy).to.have.been.calledWith('databaseList', ['abc']);
-    expect(spy).to.have.been.calledWith({
-      item: undefined,
-      kind: DatabaseEventKind.Add
-    });
-
-    sinon.reset();
-
-    // now remove the item
-    databaseManager.removeDatabaseItem(mockDbItem as unknown as DatabaseItem);
-    expect((databaseManager as any)._databaseItems).to.deep.eq([]);
-    expect(updateSpy).to.have.been.calledWith('databaseList', []);
-    expect(spy).to.have.been.calledWith({
-      item: undefined,
-      kind: DatabaseEventKind.Remove
-    });
-  });
-
-  it('should rename a db item and emit an event', () => {
-    const mockDbItem = {
-      databaseUri: 'file:/abc',
-      name: 'abc',
-      getPersistedState() {
-        return this.name;
-      }
-    };
-    const spy = sinon.spy();
-    databaseManager.onDidChangeDatabaseItem(spy);
-    (databaseManager as any).addDatabaseItem(mockDbItem);
-    sinon.restore();
-
-    databaseManager.renameDatabaseItem(mockDbItem as unknown as DatabaseItem, 'new name');
-
-    expect(mockDbItem.name).to.eq('new name');
-    expect(updateSpy).to.have.been.calledWith('databaseList', ['new name']);
-    expect(spy).to.have.been.calledWith({
-      item: undefined,
-      kind: DatabaseEventKind.Rename
-    });
-  });
-
-  describe('resolveSourceFile', () => {
-    it('should fail to resolve when not a uri', () => {
-      const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
-      (db as any)._contents.sourceArchiveUri = undefined;
-      expect(() => db.resolveSourceFile('abc')).to.throw('Scheme is missing');
-    });
-
-    it('should fail to resolve when not a file uri', () => {
-      const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
-      (db as any)._contents.sourceArchiveUri = undefined;
-      expect(() => db.resolveSourceFile('http://abc')).to.throw('Invalid uri scheme');
-    });
-
-    describe('no source archive', () => {
-      it('should resolve undefined', () => {
-        const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
-        (db as any)._contents.sourceArchiveUri = undefined;
-        const resolved = db.resolveSourceFile(undefined);
-        expect(resolved.toString()).to.eq('file:///database-uri');
-      });
-
-      it('should resolve an empty file', () => {
-        const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
-        (db as any)._contents.sourceArchiveUri = undefined;
-        const resolved = db.resolveSourceFile('file:');
-        expect(resolved.toString()).to.eq('file:///');
-      });
-    });
-
-    describe('zipped source archive', () => {
-      it('should encode a source archive url', () => {
-        const db = createMockDB(encodeSourceArchiveUri({
-          sourceArchiveZipPath: 'sourceArchive-uri',
-          pathWithinSourceArchive: 'def'
-        }));
-        const resolved = db.resolveSourceFile(Uri.file('abc').toString());
-
-        // must recreate an encoded archive uri instead of typing out the
-        // text since the uris will be different on windows and ubuntu.
-        expect(resolved.toString()).to.eq(encodeSourceArchiveUri({
-          sourceArchiveZipPath: 'sourceArchive-uri',
-          pathWithinSourceArchive: 'def/abc'
-        }).toString());
-      });
-
-      it('should encode a source archive url with trailing slash', () => {
-        const db = createMockDB(encodeSourceArchiveUri({
-          sourceArchiveZipPath: 'sourceArchive-uri',
-          pathWithinSourceArchive: 'def/'
-        }));
-        const resolved = db.resolveSourceFile(Uri.file('abc').toString());
-
-        // must recreate an encoded archive uri instead of typing out the
-        // text since the uris will be different on windows and ubuntu.
-        expect(resolved.toString()).to.eq(encodeSourceArchiveUri({
-          sourceArchiveZipPath: 'sourceArchive-uri',
-          pathWithinSourceArchive: 'def/abc'
-        }).toString());
-      });
-
-      it('should encode an empty source archive url', () => {
-        const db = createMockDB(encodeSourceArchiveUri({
-          sourceArchiveZipPath: 'sourceArchive-uri',
-          pathWithinSourceArchive: 'def'
-        }));
-        const resolved = db.resolveSourceFile('file:');
-        expect(resolved.toString()).to.eq('codeql-zip-archive://1-18/sourceArchive-uri/def/');
-      });
-    });
-
-    it('should handle an empty file', () => {
-      const db = createMockDB(Uri.parse('file:/sourceArchive-uri/'));
-      const resolved = db.resolveSourceFile('');
-      expect(resolved.toString()).to.eq('file:///sourceArchive-uri/');
-    });
-    function createMockDB(
-      sourceArchiveUri = Uri.parse('file:/sourceArchive-uri'),
-      databaseUri = Uri.parse('file:/database-uri')
-    ) {
-      return new DatabaseItemImpl(
-        databaseUri,
-        {
-          sourceArchiveUri
-        } as DatabaseContents,
-        {
-          dateAdded: 123,
-          ignoreSourceArchive: false
-        },
-        () => { /**/ }
-      );
-    }
-  });
-
-  it('should find likely db language folders', () => {
-    expect(isLikelyDbLanguageFolder('db-javascript')).to.be.true;
-    expect(isLikelyDbLanguageFolder('dbnot-a-db')).to.be.false;
-  });
-});
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/distribution.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/distribution.test.ts
@@ -214,7 +214,6 @@ describe('Launcher path', () => {

  it('should warn when using a hard-coded deprecated launcher name', async () => {
    launcherThatExists = 'codeql.cmd';
-    path.sep;
    const result = await getExecutableFromDirectory('abc');
    expect(fsSpy).to.have.been.calledWith(pathToExe);
    expect(fsSpy).to.have.been.calledWith(pathToCmd);
@@ -252,8 +251,12 @@ describe('Launcher path', () => {
    expect(result).to.equal(undefined);
  });

-  it('should not warn when deprecated launcher is used, but no new launcher is available', async () => {
-    const manager = new (createModule().DistributionManager)(undefined as any, { customCodeQlPath: pathToCmd } as any, undefined as any);
+  it('should not warn when deprecated launcher is used, but no new launcher is available', async function() {
+    const manager = new (createModule().DistributionManager)(
+      { customCodeQlPath: pathToCmd } as any,
+      {} as any,
+      undefined as any
+    );
    launcherThatExists = 'codeql.cmd';

    const result = await manager.getCodeQlPathWithoutVersionCheck();
@@ -265,7 +268,11 @@ describe('Launcher path', () => {
  });

  it('should warn when deprecated launcher is used, and new launcher is available', async () => {
-    const manager = new (createModule().DistributionManager)(undefined as any, { customCodeQlPath: pathToCmd } as any, undefined as any);
+    const manager = new (createModule().DistributionManager)(
+      { customCodeQlPath: pathToCmd } as any,
+      {} as any,
+      undefined as any
+    );
    launcherThatExists = ''; // pretend both launchers exist

    const result = await manager.getCodeQlPathWithoutVersionCheck();
@@ -277,7 +284,11 @@ describe('Launcher path', () => {
  });

  it('should warn when launcher path is incorrect', async () => {
-    const manager = new (createModule().DistributionManager)(undefined as any, { customCodeQlPath: pathToCmd } as any, undefined as any);
+    const manager = new (createModule().DistributionManager)(
+      { customCodeQlPath: pathToCmd } as any,
+      {} as any,
+      undefined as any
+    );
    launcherThatExists = 'xxx'; // pretend neither launcher exists

    const result = await manager.getCodeQlPathWithoutVersionCheck();
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/helpers.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/helpers.test.ts
@@ -1,126 +1,228 @@
 import { expect } from 'chai';
 import 'mocha';
 import { ExtensionContext, Memento } from 'vscode';
-import { InvocationRateLimiter } from '../../helpers';
+import * as yaml from 'js-yaml';
+import * as tmp from 'tmp';
+import * as path from 'path';
+import * as fs from 'fs-extra';
+import * as sinon from 'sinon';

-describe('Invocation rate limiter', () => {
-  // 1 January 2020
-  let currentUnixTime = 1577836800;
+import { getInitialQueryContents, InvocationRateLimiter, isLikelyDbLanguageFolder } from '../../helpers';
+import { reportStreamProgress } from '../../commandRunner';

-  function createDate(dateString?: string): Date {
-    if (dateString) {
-      return new Date(dateString);
+describe('helpers', () => {
+  let sandbox: sinon.SinonSandbox;
+
+  beforeEach(() => {
+    sandbox = sinon.createSandbox();
+  });
+
+  afterEach(() => {
+    sandbox.restore();
+  });
+
+  describe('Invocation rate limiter', () => {
+    // 1 January 2020
+    let currentUnixTime = 1577836800;
+
+    function createDate(dateString?: string): Date {
+      if (dateString) {
+        return new Date(dateString);
+      }
+      const numMillisecondsPerSecond = 1000;
+      return new Date(currentUnixTime * numMillisecondsPerSecond);
    }
-    const numMillisecondsPerSecond = 1000;
-    return new Date(currentUnixTime * numMillisecondsPerSecond);
+
+    function createInvocationRateLimiter<T>(funcIdentifier: string, func: () => Promise<T>): InvocationRateLimiter<T> {
+      return new InvocationRateLimiter(new MockExtensionContext(), funcIdentifier, func, s => createDate(s));
+    }
+
+    it('initially invokes function', async () => {
+      let numTimesFuncCalled = 0;
+      const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncCalled++;
+      });
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
+      expect(numTimesFuncCalled).to.equal(1);
+    });
+
+    it('doesn\'t invoke function again if no time has passed', async () => {
+      let numTimesFuncCalled = 0;
+      const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncCalled++;
+      });
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
+      expect(numTimesFuncCalled).to.equal(1);
+    });
+
+    it('doesn\'t invoke function again if requested time since last invocation hasn\'t passed', async () => {
+      let numTimesFuncCalled = 0;
+      const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncCalled++;
+      });
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
+      currentUnixTime += 1;
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(2);
+      expect(numTimesFuncCalled).to.equal(1);
+    });
+
+    it('invokes function again immediately if requested time since last invocation is 0 seconds', async () => {
+      let numTimesFuncCalled = 0;
+      const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncCalled++;
+      });
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(0);
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(0);
+      expect(numTimesFuncCalled).to.equal(2);
+    });
+
+    it('invokes function again after requested time since last invocation has elapsed', async () => {
+      let numTimesFuncCalled = 0;
+      const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncCalled++;
+      });
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(1);
+      currentUnixTime += 1;
+      await invocationRateLimiter.invokeFunctionIfIntervalElapsed(1);
+      expect(numTimesFuncCalled).to.equal(2);
+    });
+
+    it('invokes functions with different rate limiters', async () => {
+      let numTimesFuncACalled = 0;
+      const invocationRateLimiterA = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncACalled++;
+      });
+      let numTimesFuncBCalled = 0;
+      const invocationRateLimiterB = createInvocationRateLimiter('funcid', async () => {
+        numTimesFuncBCalled++;
+      });
+      await invocationRateLimiterA.invokeFunctionIfIntervalElapsed(100);
+      await invocationRateLimiterB.invokeFunctionIfIntervalElapsed(100);
+      expect(numTimesFuncACalled).to.equal(1);
+      expect(numTimesFuncBCalled).to.equal(1);
+    });
+  });
+
+  describe('codeql-database.yml tests', () => {
+    let dir: tmp.DirResult;
+    beforeEach(() => {
+      dir = tmp.dirSync();
+      const contents = yaml.safeDump({
+        primaryLanguage: 'cpp'
+      });
+      fs.writeFileSync(path.join(dir.name, 'codeql-database.yml'), contents, 'utf8');
+    });
+
+    afterEach(() => {
+      dir.removeCallback();
+    });
+
+    it('should get initial query contents when language is known', () => {
+      expect(getInitialQueryContents('cpp', 'hucairz')).to.eq('import cpp\n\nselect ""');
+    });
+
+    it('should get initial query contents when dbscheme is known', () => {
+      expect(getInitialQueryContents('', 'semmlecode.cpp.dbscheme')).to.eq('import cpp\n\nselect ""');
+    });
+
+    it('should get initial query contents when nothing is known', () => {
+      expect(getInitialQueryContents('', 'hucairz')).to.eq('select ""');
+    });
+  });
+
+  it('should find likely db language folders', () => {
+    expect(isLikelyDbLanguageFolder('db-javascript')).to.be.true;
+    expect(isLikelyDbLanguageFolder('dbnot-a-db')).to.be.false;
+  });
+
+  class MockExtensionContext implements ExtensionContext {
+    subscriptions: { dispose(): unknown }[] = [];
+    workspaceState: Memento = new MockMemento();
+    globalState: Memento = new MockMemento();
+    extensionPath = '';
+    asAbsolutePath(_relativePath: string): string {
+      throw new Error('Method not implemented.');
+    }
+    storagePath = '';
+    globalStoragePath = '';
+    logPath = '';
  }

-  function createInvocationRateLimiter<T>(funcIdentifier: string, func: () => Promise<T>): InvocationRateLimiter<T> {
-    return new InvocationRateLimiter(new MockExtensionContext(), funcIdentifier, func, s => createDate(s));
+  class MockMemento implements Memento {
+    map = new Map<any, any>();
+
+    /**
+     * Return a value.
+     *
+     * @param key A string.
+     * @param defaultValue A value that should be returned when there is no
+     * value (`undefined`) with the given key.
+     * @return The stored value or the defaultValue.
+     */
+    get<T>(key: string, defaultValue?: T): T {
+      return this.map.has(key) ? this.map.get(key) : defaultValue;
+    }
+
+    /**
+     * Store a value. The value must be JSON-stringifyable.
+     *
+     * @param key A string.
+     * @param value A value. MUST not contain cyclic references.
+     */
+    async update(key: string, value: any): Promise<void> {
+      this.map.set(key, value);
+    }
  }

-  it('initially invokes function', async () => {
-    let numTimesFuncCalled = 0;
-    const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncCalled++;
+  it('should report stream progress', () => {
+    const spy = sandbox.spy();
+    const mockReadable = {
+      on: sandbox.spy()
+    };
+    const max = 1024 * 1024 * 4;
+    const firstStep = (1024 * 1024) + (1024 * 600);
+    const secondStep = 1024 * 1024 * 2;
+
+    (reportStreamProgress as any)(mockReadable, 'My prefix', max, spy);
+
+    // now pretend that we have received some messages
+    mockReadable.on.getCall(0).args[1]({ length: firstStep });
+    mockReadable.on.getCall(0).args[1]({ length: secondStep });
+
+    expect(spy).to.have.callCount(3);
+    expect(spy).to.have.been.calledWith({
+      step: 0,
+      maxStep: max,
+      message: 'My prefix [0.0 MB of 4.0 MB]',
+    });
+    expect(spy).to.have.been.calledWith({
+      step: firstStep,
+      maxStep: max,
+      message: 'My prefix [1.6 MB of 4.0 MB]',
+    });
+    expect(spy).to.have.been.calledWith({
+      step: firstStep + secondStep,
+      maxStep: max,
+      message: 'My prefix [3.6 MB of 4.0 MB]',
    });
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
-    expect(numTimesFuncCalled).to.equal(1);
  });

-  it('doesn\'t invoke function again if no time has passed', async () => {
-    let numTimesFuncCalled = 0;
-    const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncCalled++;
-    });
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
-    expect(numTimesFuncCalled).to.equal(1);
-  });
+  it('should report stream progress when total bytes unknown', () => {
+    const spy = sandbox.spy();
+    const mockReadable = {
+      on: sandbox.spy()
+    };
+    (reportStreamProgress as any)(mockReadable, 'My prefix', undefined, spy);

-  it('doesn\'t invoke function again if requested time since last invocation hasn\'t passed', async () => {
-    let numTimesFuncCalled = 0;
-    const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncCalled++;
-    });
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(100);
-    currentUnixTime += 1;
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(2);
-    expect(numTimesFuncCalled).to.equal(1);
-  });
+    // There are no listeners registered to this readable
+    expect(mockReadable.on).not.to.have.been.called;

-  it('invokes function again immediately if requested time since last invocation is 0 seconds', async () => {
-    let numTimesFuncCalled = 0;
-    const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncCalled++;
+    expect(spy).to.have.callCount(1);
+    expect(spy).to.have.been.calledWith({
+      step: 1,
+      maxStep: 2,
+      message: 'My prefix (Size unknown)',
    });
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(0);
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(0);
-    expect(numTimesFuncCalled).to.equal(2);
-  });
-
-  it('invokes function again after requested time since last invocation has elapsed', async () => {
-    let numTimesFuncCalled = 0;
-    const invocationRateLimiter = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncCalled++;
-    });
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(1);
-    currentUnixTime += 1;
-    await invocationRateLimiter.invokeFunctionIfIntervalElapsed(1);
-    expect(numTimesFuncCalled).to.equal(2);
-  });
-
-  it('invokes functions with different rate limiters', async () => {
-    let numTimesFuncACalled = 0;
-    const invocationRateLimiterA = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncACalled++;
-    });
-    let numTimesFuncBCalled = 0;
-    const invocationRateLimiterB = createInvocationRateLimiter('funcid', async () => {
-      numTimesFuncBCalled++;
-    });
-    await invocationRateLimiterA.invokeFunctionIfIntervalElapsed(100);
-    await invocationRateLimiterB.invokeFunctionIfIntervalElapsed(100);
-    expect(numTimesFuncACalled).to.equal(1);
-    expect(numTimesFuncBCalled).to.equal(1);
  });
 });
-
-class MockExtensionContext implements ExtensionContext {
-  subscriptions: { dispose(): unknown }[] = [];
-  workspaceState: Memento = new MockMemento();
-  globalState: Memento = new MockMemento();
-  extensionPath = '';
-  asAbsolutePath(_relativePath: string): string {
-    throw new Error('Method not implemented.');
-  }
-  storagePath = '';
-  globalStoragePath = '';
-  logPath = '';
-}
-
-class MockMemento implements Memento {
-  map = new Map<any, any>();
-
-  /**
-   * Return a value.
-   *
-   * @param key A string.
-   * @param defaultValue A value that should be returned when there is no
-   * value (`undefined`) with the given key.
-   * @return The stored value or the defaultValue.
-   */
-  get<T>(key: string, defaultValue?: T): T {
-    return this.map.has(key) ? this.map.get(key) : defaultValue;
-  }
-
-  /**
-   * Store a value. The value must be JSON-stringifyable.
-   *
-   * @param key A string.
-   * @param value A value. MUST not contain cyclic references.
-   */
-  async update(key: string, value: any): Promise<void> {
-    this.map.set(key, value);
-  }
-}
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/query-history.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/query-history.test.ts
@@ -5,7 +5,9 @@ import * as vscode from 'vscode';
 import * as sinon from 'sinon';
 import * as chaiAsPromised from 'chai-as-promised';
 import { logger } from '../../logging';
-import { QueryHistoryManager } from '../../query-history';
+import { QueryHistoryManager, HistoryTreeDataProvider } from '../../query-history';
+import { CompletedQuery } from '../../query-results';
+import { QueryInfo } from '../../run-queries';

 chai.use(chaiAsPromised);
 const expect = chai.expect;
@@ -19,32 +21,30 @@ describe('query-history', () => {
  let showQuickPickSpy: sinon.SinonStub;

  let tryOpenExternalFile: Function;
+  let sandbox: sinon.SinonSandbox;

  beforeEach(() => {
-    showTextDocumentSpy = sinon.stub(vscode.window, 'showTextDocument');
-    showInformationMessageSpy = sinon.stub(
+    sandbox = sinon.createSandbox();
+
+    showTextDocumentSpy = sandbox.stub(vscode.window, 'showTextDocument');
+    showInformationMessageSpy = sandbox.stub(
      vscode.window,
      'showInformationMessage'
    );
-    showQuickPickSpy = sinon.stub(
+    showQuickPickSpy = sandbox.stub(
      vscode.window,
      'showQuickPick'
    );
-    executeCommandSpy = sinon.stub(vscode.commands, 'executeCommand');
-    sinon.stub(logger, 'log');
+    executeCommandSpy = sandbox.stub(vscode.commands, 'executeCommand');
+    sandbox.stub(logger, 'log');
    tryOpenExternalFile = (QueryHistoryManager.prototype as any).tryOpenExternalFile;
  });

  afterEach(() => {
-    (vscode.window.showTextDocument as sinon.SinonStub).restore();
-    (vscode.commands.executeCommand as sinon.SinonStub).restore();
-    (logger.log as sinon.SinonStub).restore();
-    (vscode.window.showInformationMessage as sinon.SinonStub).restore();
-    (vscode.window.showQuickPick as sinon.SinonStub).restore();
+    sandbox.restore();
  });

  describe('tryOpenExternalFile', () => {
-
    it('should open an external file', async () => {
      await tryOpenExternalFile('xxx');
      expect(showTextDocumentSpy).to.have.been.calledOnceWith(
@@ -205,6 +205,67 @@ describe('query-history', () => {
      expect(queryHistory.compareWithItem).to.be.eq('a');
    });
  });
+
+  describe('HistoryTreeDataProvider', () => {
+    let historyTreeDataProvider: HistoryTreeDataProvider;
+    beforeEach(() => {
+      historyTreeDataProvider = new HistoryTreeDataProvider(vscode.Uri.file('/a/b/c').fsPath);
+    });
+
+    it('should get a tree item with raw results', async () => {
+      const mockQuery = {
+        query: {
+          hasInterpretedResults: () => Promise.resolve(false)
+        } as QueryInfo,
+        didRunSuccessfully: true,
+        toString: () => 'mock label'
+      } as CompletedQuery;
+      const treeItem = await historyTreeDataProvider.getTreeItem(mockQuery);
+      expect(treeItem.command).to.deep.eq({
+        title: 'Query History Item',
+        command: 'codeQLQueryHistory.itemClicked',
+        arguments: [mockQuery],
+      });
+      expect(treeItem.label).to.eq('mock label');
+      expect(treeItem.contextValue).to.eq('rawResultsItem');
+      expect(treeItem.iconPath).to.be.undefined;
+    });
+
+    it('should get a tree item with interpreted results', async () => {
+      const mockQuery = {
+        query: {
+          // as above, except for this line
+          hasInterpretedResults: () => Promise.resolve(true)
+        } as QueryInfo,
+        didRunSuccessfully: true,
+        toString: () => 'mock label'
+      } as CompletedQuery;
+      const treeItem = await historyTreeDataProvider.getTreeItem(mockQuery);
+      expect(treeItem.contextValue).to.eq('interpretedResultsItem');
+    });
+
+    it('should get a tree item that did not complete successfully', async () => {
+      const mockQuery = {
+        query: {
+          hasInterpretedResults: () => Promise.resolve(true)
+        } as QueryInfo,
+        // as above, except for this line
+        didRunSuccessfully: false,
+        toString: () => 'mock label'
+      } as CompletedQuery;
+      const treeItem = await historyTreeDataProvider.getTreeItem(mockQuery);
+      expect(treeItem.iconPath).to.eq(vscode.Uri.file('/a/b/c/media/red-x.svg').fsPath);
+    });
+
+    it('should get children', () => {
+      const mockQuery = {
+        databaseName: 'abc'
+      } as CompletedQuery;
+      historyTreeDataProvider.allHistory.push(mockQuery);
+      expect(historyTreeDataProvider.getChildren()).to.deep.eq([mockQuery]);
+      expect(historyTreeDataProvider.getChildren(mockQuery)).to.deep.eq([]);
+    });
+  });
 });

 function createMockQueryHistory(allHistory: {}[]) {
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/run-queries.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/run-queries.test.ts
@@ -52,6 +52,7 @@ describe('run-queries', () => {
            localChecking: false,
            noComputeGetUrl: false,
            noComputeToString: false,
+            computeDefaultStrings: true
          },
          extraOptions: {
            timeoutSecs: 5
--- a/extensions/ql-vscode/src/vscode-tests/run-integration-tests.ts
+++ b/extensions/ql-vscode/src/vscode-tests/run-integration-tests.ts
@@ -1,16 +1,17 @@
 import * as path from 'path';
 import * as os from 'os';
-import { runTests } from 'vscode-test';
+import * as cp from 'child_process';
+import {
+  runTests,
+  downloadAndUnzipVSCode,
+  resolveCliPathFromVSCodeExecutablePath
+} from 'vscode-test';
+import { assertNever } from '../pure/helpers-pure';
+
+// For some reason, `TestOptions` is not exported directly from `vscode-test`,
+// but we can be tricky and import directly from the out file.
+import { TestOptions } from 'vscode-test/out/runTest';

-// A subset of the fields in TestOptions from vscode-test, which we
-// would simply use instead, but for the fact that it doesn't export
-// it.
-type Suite = {
-  extensionDevelopmentPath: string;
-  extensionTestsPath: string;
-  launchArgs: string[];
-  version?: string;
-};

 // Which version of vscode to test against. Can set to 'stable' or
 // 'insiders' or an explicit version number. See runTest.d.ts in
@@ -22,11 +23,21 @@ type Suite = {
 // testing against old versions if necessary.
 const VSCODE_VERSION = 'stable';

+// List if test dirs
+//   - no-workspace - Tests with no workspace selected upon launch.
+//   - minimal-workspace - Tests with a simple workspace selected upon launch.
+//   - cli-integration - Tests that require a cli to invoke actual commands
+enum TestDir {
+  NoWorksspace = 'no-workspace',
+  MinimalWorksspace = 'minimal-workspace',
+  CliIntegration = 'cli-integration'
+}
+
 /**
 * Run an integration test suite `suite`, retrying if it segfaults, at
 * most `tries` times.
 */
-async function runTestsWithRetryOnSegfault(suite: Suite, tries: number): Promise<void> {
+async function runTestsWithRetryOnSegfault(suite: TestOptions, tries: number): Promise<void> {
  for (let t = 0; t < tries; t++) {
    try {
      // Download and unzip VS Code if necessary, and run the integration test suite.
@@ -58,34 +69,35 @@ async function runTestsWithRetryOnSegfault(suite: Suite, tries: number): Promise
 */
 async function main() {
  try {
-    // The folder containing the Extension Manifest package.json
-    // Passed to `--extensionDevelopmentPath`.
    const extensionDevelopmentPath = path.resolve(__dirname, '../..');
+    const vscodeExecutablePath = await downloadAndUnzipVSCode(VSCODE_VERSION);

-    // List of integration test suites.
-    // The path to the extension test runner script is passed to --extensionTestsPath.
-    const integrationTestSuites: Suite[] = [
-      // Tests with no workspace selected upon launch.
-      {
-        version: VSCODE_VERSION,
-        extensionDevelopmentPath: extensionDevelopmentPath,
-        extensionTestsPath: path.resolve(__dirname, 'no-workspace', 'index'),
-        launchArgs: ['--disable-extensions'],
-      },
-      // Tests with a simple workspace selected upon launch.
-      {
-        version: VSCODE_VERSION,
-        extensionDevelopmentPath: extensionDevelopmentPath,
-        extensionTestsPath: path.resolve(__dirname, 'minimal-workspace', 'index'),
-        launchArgs: [
-          path.resolve(__dirname, '../../test/data'),
-          '--disable-extensions',
-        ]
-      }
-    ];

-    for (const integrationTestSuite of integrationTestSuites) {
-      await runTestsWithRetryOnSegfault(integrationTestSuite, 3);
+    // Which tests to run. Use a comma-separated list of directories.
+    const testDirsString = process.argv[2];
+    const dirs = testDirsString.split(',').map(dir => dir.trim().toLocaleLowerCase());
+
+    if (dirs.includes(TestDir.CliIntegration)) {
+      console.log('Installing required extensions');
+      const cliPath = resolveCliPathFromVSCodeExecutablePath(vscodeExecutablePath);
+      cp.spawnSync(cliPath, ['--install-extension', 'hbenl.vscode-test-explorer'], {
+        encoding: 'utf-8',
+        stdio: 'inherit'
+      });
+    }
+
+    console.log(`Running integration tests in these directories: ${dirs}`);
+    for (const dir of dirs) {
+      const launchArgs = getLaunchArgs(dir as TestDir);
+      console.log(`Next integration test dir: ${dir}`);
+      console.log(`Launch args: ${launchArgs}`);
+      await runTestsWithRetryOnSegfault({
+        version: VSCODE_VERSION,
+        vscodeExecutablePath,
+        extensionDevelopmentPath,
+        extensionTestsPath: path.resolve(__dirname, dir, 'index'),
+        launchArgs
+      }, 3);
    }
  } catch (err) {
    console.error(`Unexpected exception while running tests: ${err}`);
@@ -94,3 +106,30 @@ async function main() {
 }

 main();
+
+
+function getLaunchArgs(dir: TestDir) {
+  switch (dir) {
+    case TestDir.NoWorksspace:
+      return [
+        '--disable-extensions'
+      ];
+
+    case TestDir.MinimalWorksspace:
+      return [
+        '--disable-extensions',
+        path.resolve(__dirname, '../../test/data')
+      ];
+
+    case TestDir.CliIntegration:
+      // CLI integration tests requires a multi-root workspace so that the data and the QL sources are accessible.
+      return [
+        path.resolve(__dirname, '../../test/data'),
+        process.env.TEST_CODEQL_PATH!
+      ];
+
+    default:
+      assertNever(dir);
+  }
+  return undefined;
+}
--- a/extensions/ql-vscode/src/vscode-utils/ui-service.ts
+++ b/extensions/ql-vscode/src/vscode-utils/ui-service.ts
@@ -1,6 +1,6 @@
 import { TreeDataProvider, window } from 'vscode';
 import { DisposableObject } from './disposable-object';
-import { commandRunner } from '../helpers';
+import { commandRunner } from '../commandRunner';

 /**
 * A VS Code service that interacts with the UI, including handling commands.
--- a/extensions/ql-vscode/test/data/.gitignore
+++ b/extensions/ql-vscode/test/data/.gitignore
@@ -0,0 +1 @@
+.vscode
--- a/extensions/ql-vscode/test/data/compute-default-strings.ql
+++ b/extensions/ql-vscode/test/data/compute-default-strings.ql
@@ -0,0 +1,12 @@
+// Test that computeDefaultStrings is set correctly.
+
+newtype TUnit = MkUnit()
+
+class Unit extends TUnit {
+  Unit() { this = MkUnit() }
+
+  string toString() { none() }
+}
+
+from Unit u
+select u
--- a/extensions/ql-vscode/test/pure-tests/files.test.ts
+++ b/extensions/ql-vscode/test/pure-tests/files.test.ts
@@ -47,11 +47,12 @@ describe('files', () => {
  });

  it('should scan a directory', async () => {
-    const singleFile = path.join(dataDir, 'query.ql');
-    const otherFile = path.join(dataDir, 'multiple-result-sets.ql');
+    const file1 = path.join(dataDir, 'compute-default-strings.ql');
+    const file2 = path.join(dataDir, 'multiple-result-sets.ql');
+    const file3 = path.join(dataDir, 'query.ql');

    const result = await gatherQlFiles([dataDir]);
-    expect(result.sort()).to.deep.equal([[otherFile, singleFile], true]);
+    expect(result.sort()).to.deep.equal([[file1, file2, file3], true]);
  });

  it('should scan a directory and some files', async () => {
@@ -64,10 +65,12 @@ describe('files', () => {
  });

  it('should avoid duplicates', async () => {
-    const singleFile = path.join(dataDir, 'query.ql');
-    const otherFile = path.join(dataDir, 'multiple-result-sets.ql');
+    const file1 = path.join(dataDir, 'compute-default-strings.ql');
+    const file2 = path.join(dataDir, 'multiple-result-sets.ql');
+    const file3 = path.join(dataDir, 'query.ql');

-    const result = await gatherQlFiles([singleFile, dataDir, otherFile]);
-    expect(result.sort()).to.deep.equal([[singleFile, otherFile], true]);
+    const result = await gatherQlFiles([file1, dataDir, file3]);
+    result[0].sort();
+    expect(result.sort()).to.deep.equal([[file1, file2, file3], true]);
  });
 });
Author	SHA1	Message	Date
Andrew Eisenberg	a7c73cc421	v1.3.10 Some checks failed Code Scanning - CodeQL / codeql (push) Has been cancelled Details Release / Release (push) Has been cancelled Details Release / Publish to VS Code Marketplace (push) Has been cancelled Details Release / Publish to Open VSX Registry (push) Has been cancelled Details	2021-01-20 14:15:45 -08:00
Andrew Eisenberg	044bc30d96	Clarify how to run CLI tests locally Also, remove an errant `only`, which was preventing some tests from running.	2021-01-20 13:05:53 -08:00
Andrew Eisenberg	9c72e81264	Update changelog	2021-01-20 13:05:53 -08:00
Andrew Eisenberg	3a718ee6e0	Include the full stack in error log messages Ensure we only show the truncated error message in the popup. This will help with debugging.	2021-01-20 13:05:53 -08:00
Andrew Eisenberg	540124478b	Refactor: Move commandRunner to its own module Also, extract related functions and types. There are no behavioral changes in this commit. Only refactorings.	2021-01-19 12:51:12 -08:00
Henry Mercer	6074a1a7c8	Fix minor typo in welcome content Replace "Code QL database" with "CodeQL database" for consistency with [documentation](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/).	2021-01-19 06:51:18 -08:00
Andrew Eisenberg	093a51cee3	Fix Code Scanning warnings	2021-01-11 15:27:43 -08:00
Andrew Eisenberg	cace4acb1e	Update internal docs for publishing And remove unused file.	2021-01-11 13:38:21 -08:00
Andrew Eisenberg	696c16b5b4	Add workflow jobs to deploy extension This adds two new jobs to the `Release` workflow. These jobs are blocked behind an environment. When approved by a committer, the extension will be deployed to Open VSX and VS Code marketplace. Also, update contributing docs for open-vsx publishing.	2021-01-11 13:38:21 -08:00
Andrew Eisenberg	7b439e4511	Make typing more explicit	2021-01-04 08:55:47 -08:00
alexet	402700f56f	SingleFileUpgrades: Address comments	2021-01-04 08:55:47 -08:00
alexet	8eaeefb9ea	Use single file upgrades where possible.	2021-01-04 08:55:47 -08:00
aeisenberg	49ac9796a1	Bump version to v1.3.9	2020-12-17 11:55:58 -08:00
Andrew Eisenberg	89b6b5a945	v1.3.8 Some checks failed Code Scanning - CodeQL / codeql (push) Has been cancelled Details Release / Release (push) Has been cancelled Details	2020-12-17 11:52:33 -08:00
Andrew Eisenberg	53ac1ed70d	Update changelog Clarify a few entries.	2020-12-17 11:07:37 -08:00
Andrew Eisenberg	5824e3607a	Add View AST Command to text editor where appropriate Also, hide the `CodeQL: Run Queries` command inside of zip folders since we do not allow queries to be in archives. I wish we could be more specific about when to show that command, eg- only *.ql files and directories, but I couldn't find a way to restrict a command to only appear on directories.	2020-12-17 11:07:37 -08:00
Andrew Eisenberg	e6eb914783	Add unit tests for query-history	2020-12-16 13:26:09 -08:00
Andrew Eisenberg	b0e032be2c	Fix set label command on history items This removes the cached treeItem that is a property of the completedQuery. We should not be caching them since they are cached by the vscode api itself. Instead, we should recreate whenever requested. Also, this change fixes #598 in a new way. Instead of adding the context to the cached treeItem, we simply refresh only the item that has changed. This is a fast operation.	2020-12-16 13:26:09 -08:00
Andrew Eisenberg	3ea3eda8aa	Add descriptive text and a link at top of query results The descriptive text is the same as the label in the query history view. The link opens the file that ran the query.	2020-12-16 13:18:49 -08:00
Andrew Eisenberg	ca9510c08d	Add unit test for test discovery When directory is not present.	2020-12-14 17:47:49 -08:00
Andrew Eisenberg	303cb3284c	Avoid uninteresting user facing errors This change avoids popping up error messages in two cases: 1. When doing test discovery, do not run discovery on non-existant directories. Also, if there is an error, print to the log, and do not pop up an error window. The reason is that test discovery is a background operation and these should not normally cause pop-ups. 2. When looking for orphaned databases, don't pop up an error if the storagePath can't be found. This is normal when working in a new, single root workspace.	2020-12-14 17:47:49 -08:00
Andrew Eisenberg	5ad433775b	Move query.test.ts to the cli integration tests * Now query.test.ts runs on multiple cli versions * Removed most `dispose` calls in cli tests because each test shares the same instance of the extension and all of its properties. So, we shouldn't be disposing until the last test completes. It's likely that we will need to be more careful about cleaning up state between test runs, but we haven't hit that yet and this can happen in a later commit.	2020-12-14 12:36:46 -08:00
Andrew Eisenberg	69ca0f55ba	Re-enable the queries cli test * Requires that QL_PATH environment variable is set and points to a checkout of github/codeql * Adds the `quiet` flag to the cli. When set, this flag will prevent some modal dialogs from disrupting the flow. Currently, we only ensure that the upgrades dialog is avoided. * Update the main.yml workflow to checkout the codeql repo	2020-12-14 12:36:46 -08:00
Andrew Eisenberg	b5e708796d	Fix failing test Also, small change to ensure `qlpackOfDatabase` never returns undefined. It will either return a value or throw.	2020-12-14 10:20:28 -08:00
Tom Hvitved	2516a62469	Add missing call to `getPrimaryDbscheme` in `qlpackOfDatabase`	2020-12-14 10:20:28 -08:00
Andrew Eisenberg	9ffb3a14c7	Save downloaded DB archives to disk before unzipping (#700 ) This fixes two classes of DBs that can't be installed directly from downloading: 1. DBs whose central directories do not align with their file headers. We need to download and save the entire archive before we can read the central directory and use that to guide the unzipping. 2. Large DBs require too much memory so can't be downloaded and unzipped in a single stream. We also add proper progress notifications to the download progress monitor so users are aware of how many more MBs are left to download. It's not yet possible to do the same for unzipping using the current unzipper library, since unzipping using the central directory does not expose a stream. Co-authored-by: Alexander Eyers-Taylor <alexet@github.com>	2020-12-14 16:26:29 +00:00
dependabot[bot]	51835a2466	Bump ini from 1.3.5 to 1.3.8 in /extensions/ql-vscode Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.8. - [Release notes](https://github.com/isaacs/ini/releases) - [Commits](https://github.com/isaacs/ini/compare/v1.3.5...v1.3.8) Signed-off-by: dependabot[bot] <support@github.com>	2020-12-13 15:52:35 -08:00
Andrew Eisenberg	b470e41431	Make error messages clearer for some common problems 1. Clicking on query history menu items when nothing is selected. Error message is clearer. It would be better to disable when nothing is selected, but waiting on https://github.com/microsoft/vscode/issues/99767 to be released. 2. Trying to run query with a missing or invalid qlpack has better message. 3. Better hover text for "Open query". Co-authored-by: Aditya Sharad <6874315+adityasharad@users.noreply.github.com>	2020-12-09 15:35:59 -08:00
Andrew Eisenberg	370dbcbfae	Update integration test version numbers Also, fix a typo and remove comment.	2020-12-04 10:08:15 -08:00
Andrew Eisenberg	6046cf1472	Add integration test for running a query In order to do this, needed to add a few extra pieces: * extracted the simple database download so that it only happens once and is shared across all tests. * needed to update mocha to latest version since that has the new API * But typings isn't updated yet, so submitted a PR into DefinitelyTyped for that. * Added a concept of helper files for test runs. These helper files will contain all the shared global setup. Unfortunately, at this point, we can't run using a language pack since we would also need to download the the ql repository from somewhere.	2020-12-04 10:08:15 -08:00
Andrew Eisenberg	864041efcb	Add integration tests for database fetching	2020-12-04 10:08:15 -08:00
Andrew Eisenberg	16eac45822	Add integration tests with the CLI This commit adds integration tests that run commands using the CLI. This change introduces a number of enhancements in order to get there. 1. Augments the index-template.ts file so that it downloads an appropriate cli version if requested. 2. Adds the ensureCli.ts that performs the download if a a suitable version is not already installed. See the comments in the file for how this is done. 3. Changes how run-integration-tests is done so that the directories run are specified through a cli argument. 4. Updates the main.yml workflow so that it also runs the cli-integration tests. 5. Takes advantage of the return value of the call to `activate` on the extension. This allows the integration tests to have access to internal variables of the extension like the context, cli, and query server. 6. And of course, adds a handful of simple tests that ensure we have a cli installed of the correct version.	2020-12-04 10:08:15 -08:00
Andrew Eisenberg	06a1fd91e4	Fix typos and augment comments around language labels	2020-12-04 09:36:54 -08:00
Andrew Eisenberg	67e8c86ccc	Use codeql resolve database to get language This commit moves to using codeql resolve database instead of inspecting the `codeql-database.yml` file. When the extension starts and if the cli supports it, the extension will attempt to get the name for any databases that don't yet have a name. Once a name is searched for once by the cli, it will be cached so we don't need to rediscover the name again.	2020-12-04 09:36:54 -08:00
Andrew Eisenberg	43ef44ff12	Add a language label next to databases in the UI This change will only work on databases created by cli >= 2.4.1. In that version, a new `primaryLanguage` field in the `codeql-database.yml` file. We use this property as the language. This change also includes a refactoring of the logic around extracting database information heuristically based on file location. As much as possible, it is extracted to the `helpers` module. Also, the initial quick query text is generated based on the language (if known) otherwise it falls back to the old style of generation.	2020-12-04 09:36:54 -08:00
Andrew Eisenberg	0d04c5d463	Ensure all tests run (#696 ) Accidentally committed `.only`.	2020-12-02 15:57:40 -08:00
Andrew Eisenberg	b6c7837fd7	Pluralize registration message names And do a version check before adding `--require-db-registration` flag.	2020-12-02 11:49:56 -08:00
Andrew Eisenberg	d76f912903	Add version check for db registration Database registration is available in versions >= 2.4.1	2020-12-02 11:49:56 -08:00
Andrew Eisenberg	1b4a992182	Update changelog	2020-12-02 11:49:56 -08:00
Andrew Eisenberg	2795184e70	Add support for database registration	2020-12-02 11:49:56 -08:00
Andrew Eisenberg	3c08baf062	Add query running test for computeDefaultStrings flag	2020-12-01 14:31:39 -08:00
Andrew Eisenberg	6afb946200	Update changelog	2020-12-01 14:31:39 -08:00
Andrew Eisenberg	bfe4aa386c	Pass computeDefaultStrings to query server when compiling queries	2020-12-01 14:31:39 -08:00
Max Schaefer	f4624f3dbf	Fix dubious index check (#692 ) * Fix dubious index check * Add unit tests for add/remove database In order to do this, needed to move `databases.test.ts` to the `minimal-workspace` test folder because these tests require that there be some kind of workspace open in order to check on workspace folders. Unfortunately, during tests vscode does not allow you to convert from a single root workspace to multi-root and so several of the workspace functions needed to be stubbed out. * Update changelog Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>	2020-11-30 11:34:47 -08:00
aeisenberg	1b4d8e303d	Bump version to v1.3.8	2020-11-24 14:11:16 -08:00