v1.5.8

Merge pull request #1009 from github/aeisenberg/remote-nested-queries
Remote queries: Handle nested queries
2021-12-02 11:26:36 +00:00 · 2021-12-01 19:24:10 +00:00 · 2021-12-01 10:53:51 -08:00 · 2021-12-01 09:40:06 +00:00 · 2021-12-01 09:40:06 +00:00 · 2021-12-01 09:40:06 +00:00
54 changed files with 2404 additions and 694 deletions
--- a/.github/ISSUE_TEMPLATE/new-extension-release.md
+++ b/.github/ISSUE_TEMPLATE/new-extension-release.md
@@ -1,18 +0,0 @@
---
-name: New extension release
-about: Create an issue with a checklist for the release steps (write access required
-  for the steps)
-title: Release Checklist for version xx.xx.xx
-labels: ''
-assignees: ''
-
---
-
- [ ] Update this issue title to refer to the version of the release
- [ ] Trigger a release build on Actions by adding a new tag on branch `main` of the format `vxx.xx.xx`
- [ ] Monitor the status of the release build in the `Release` workflow in the Actions tab.
- [ ] Download the VSIX from the draft GitHub release that is created when the release build finishes.
- [ ] Log into the [Visual Studio Marketplace](https://marketplace.visualstudio.com/manage/publishers/github).
- [ ] Click the `...` menu in the CodeQL row and click **Update**.
- [ ] Drag the `.vsix` file you downloaded from the GitHub release into the Marketplace and click **Upload**.
- [ ] Publish the draft GitHub release and confirm the new release is marked as the latest release at https://github.com/github/vscode-codeql/releases.
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -74,7 +74,6 @@ jobs:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
-        version: [stable, nightly]
    steps:
      - name: Checkout
        uses: actions/checkout@v2
@@ -104,31 +103,16 @@ jobs:
        run: |
          npm run lint

-      - name: Install CodeQL
-        run: |
-          mkdir codeql-home
-          if [ ${{ matrix.version }} = "stable" ]
-          then
-            curl -L --silent https://github.com/github/codeql-cli-binaries/releases/latest/download/codeql.zip -o codeql-home/codeql.zip
-          else
-            curl -L --silent ${{ needs.find-nightly.outputs.url }}/codeql.zip -o codeql-home/codeql.zip
-          fi
-          unzip -q -o codeql-home/codeql.zip -d codeql-home
-          unzip -q -o codeql-home/codeql.zip codeql/codeql.exe -d codeql-home
-          rm codeql-home/codeql.zip
-        shell: bash
-
      - name: Run unit tests (Linux)
        working-directory: extensions/ql-vscode
        if: matrix.os == 'ubuntu-latest'
        run: |
-          CODEQL_PATH=$GITHUB_WORKSPACE/codeql-home/codeql/codeql npm run test
+          npm run test

      - name: Run unit tests (Windows)
        if: matrix.os == 'windows-latest'
        working-directory: extensions/ql-vscode
        run: |
-          $env:CODEQL_PATH=$(Join-Path $env:GITHUB_WORKSPACE -ChildPath 'codeql-home/codeql/codeql.exe')
          npm run test

      - name: Run integration tests (Linux)
@@ -151,7 +135,7 @@ jobs:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
-        version: ['v2.3.3', 'v2.4.6', 'v2.5.9', 'v2.6.1', 'nightly']
+        version: ['v2.3.3', 'v2.4.6', 'v2.5.9', 'v2.6.3', 'v2.7.2', 'nightly']
    env:
      CLI_VERSION: ${{ matrix.version }}
      NIGHTLY_URL: ${{ needs.find-nightly.outputs.url }}
--- a/.vscode/extensions.json
+++ b/.vscode/extensions.json
@@ -3,7 +3,7 @@
  // Extension identifier format: ${publisher}.${name}. Example: vscode.csharp
  // List of extensions which should be recommended for users of this workspace.
  "recommendations": [
-    "eamodio.tsl-problem-matcher",
+    "amodio.tsl-problem-matcher",
    "dbaeumer.vscode-eslint",
    "eternalphane.tsfmt-vscode"
  ],
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@ -21,6 +21,9 @@
        // change to 'true' debug the IDE or Query servers
        "IDE_SERVER_JAVA_DEBUG": "false",
        "QUERY_SERVER_JAVA_DEBUG": "false",
+        "CLI_SERVER_JAVA_DEBUG": "false",
+        // Uncomment to set the JAVA_HOME for the codeql instance to use
+        // "CODEQL_JAVA_HOME": "/Library/Java/JavaVirtualMachines/jdk-12.0.1.jdk/Contents/Home"
      }
    },
    {
@@ -93,6 +96,19 @@
        // available in the workspace for the tests.
        // "${workspaceRoot}/../codeql"
      ],
+      "env": {
+        // Optionally, set the version to use for the integration tests.
+        // Use "nightly" to use the latest nightly build.
+        // "CLI_VERSION": "2.7.0",
+
+        // If CLI_VERSION is set to nightly, set this to the url of the nightly build.
+        // "NIGHTLY_URL": "some url to grab the nightly build",
+
+        // Optionally, add a path to the codeql executable to be used during these tests.
+        // If not specified, one will be downloaded automatically.
+        // This option overrides the CLI_VERSION option.
+        // "CLI_PATH": "${workspaceRoot}/../semmle-code/target/intree/codeql/codeql",
+      },
      "stopOnEntry": false,
      "sourceMaps": true,
      "outFiles": [
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -77,9 +77,9 @@ $ vscode/scripts/code-cli.sh --install-extension dist/vscode-codeql-*.vsix # if

 You can use VS Code to debug the extension without explicitly installing it. Just open this directory as a workspace in VS Code, and hit `F5` to start a debugging session.

-### Running the unit/integration tests
+### Running the unit tests and integration tests that do not require a CLI instance

-Ensure the `CODEQL_PATH` environment variable is set to point to the `codeql` cli executable.
+Unit tests and many integration tests do not require a copy of the CodeQL CLI.

 Outside of vscode, run:

@@ -89,6 +89,16 @@ npm run test && npm run integration

 Alternatively, you can run the tests inside of vscode. There are several vscode launch configurations defined that run the unit and integration tests. They can all be found in the debug view.

+Only the _With CLI_ tests require a CLI instance to run. See below on how to do that.
+
+Running from a terminal, you _must_ set the `TEST_CODEQL_PATH` variable to point to a checkout of the `github/codeql` repository. The appropriate CLI version will be downloaded as part of the test.
+
+### Running the integration tests
+
+The _Launch Integration Tests - With CLI_ tests require a CLI instance in order to run. There are several environment variables you can use to configure this.
+
+From inside of VSCode, open the `launch.json` file and in the _Launch Integration Tests - With CLI_ uncomment and change the environment variables appropriate for your purpose.
+
 ## Releasing (write access required)

 1. Double-check the `CHANGELOG.md` contains all desired change comments and has the version to be released with date at the top.
--- a/extensions/ql-vscode/CHANGELOG.md
+++ b/extensions/ql-vscode/CHANGELOG.md
@@ -1,5 +1,31 @@
 # CodeQL for Visual Studio Code: Changelog

+## 1.5.8 - 2 December 2021
+
+- Emit a more explicit error message when a user tries to add a database with an unzipped source folder to the workspace. [#1021](https://github.com/github/vscode-codeql/pull/1021)
+- Ensure `src.zip` archives are used as the canonical source instead of `src` folders when importing databases. [#1025](https://github.com/github/vscode-codeql/pull/1025)
+
+## 1.5.7 - 23 November 2021
+
+- Fix the _CodeQL: Open Referenced File_ command for Windows systems. [#979](https://github.com/github/vscode-codeql/pull/979)
+- Support large SARIF results files (>4GB) without crashing VS Code. [#1004](https://github.com/github/vscode-codeql/pull/1004)
+- Fix a bug that shows 'Set current database' when hovering over the currently selected database in the databases view. [#976](https://github.com/github/vscode-codeql/pull/976)
+- Fix a bug with importing large databases. Databases over 4GB can now be imported directly from LGTM or from a zip file. This functionality is only available when using CodeQL CLI version 2.6.0 or later. [#971](https://github.com/github/vscode-codeql/pull/971)
+- Replace certain control codes (`U+0000` - `U+001F`) with their corresponding control labels (`U+2400` - `U+241F`) in the results view. [#963](https://github.com/github/vscode-codeql/pull/963)
+- Allow case-insensitive project slugs for GitHub repositories when adding a CodeQL database from LGTM. [#978](https://github.com/github/vscode-codeql/pull/961)
+- Add a _CodeQL: Preview Query Help_ command to generate Markdown previews of `.qhelp` query help files. This command should only be run in trusted workspaces. See https://codeql.github.com/docs/codeql-cli/testing-query-help-files for more information about query help. [#988](https://github.com/github/vscode-codeql/pull/988)
+- Make "Open Referenced File" command accessible from the active editor menu. [#989](https://github.com/github/vscode-codeql/pull/989)
+- Fix a bug where result set names in the result set drop-down were disappearing when viewing a sorted table. [#1007](https://github.com/github/vscode-codeql/pull/1007)
+- Allow query result locations with 0 as the end column value. These are treated as the first column in the line. [#1002](https://github.com/github/vscode-codeql/pull/1002)
+
+## 1.5.6 - 07 October 2021
+
+- Add progress messages to LGTM download option. This makes the two-step process (selecting a project, then selecting a language) more clear. [#960](https://github.com/github/vscode-codeql/pull/960)
+- Remove line about selecting a language from the dropdown when downloading database from LGTM. This makes the download progress visible when the popup is not expanded. [#957](https://github.com/github/vscode-codeql/pull/957)
+- Fix a bug where copying the version information fails when a CodeQL CLI cannot be found. [#958](https://github.com/github/vscode-codeql/pull/958)
+- Avoid a race condition when deleting databases that can cause occasional errors. [#959](https://github.com/github/vscode-codeql/pull/959)
+- Update CodeQL logos. [#965](https://github.com/github/vscode-codeql/pull/965)
+
 ## 1.5.5 - 08 September 2021

 - Fix bug where a query is sometimes run before the file is saved. [#947](https://github.com/github/vscode-codeql/pull/947)
--- a/extensions/ql-vscode/media/VS-marketplace-CodeQL-icon.png
+++ b/extensions/ql-vscode/media/VS-marketplace-CodeQL-icon.png
--- a/extensions/ql-vscode/media/logo.svg
+++ b/extensions/ql-vscode/media/logo.svg
@@ -1,14 +1,4 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<svg width="27px" height="16px" viewBox="0 0 27 16" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
-    <!-- Generator: Sketch 59 (86127) - https://sketch.com -->
-    <title>Slice</title>
-    <desc>Created with Sketch.</desc>
-    <g id="light" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
-        <g id="QL" transform="translate(1.000000, 1.000000)">
-            <rect id="Rectangle-41" stroke="#2088FF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" x="0" y="0" width="25" height="14" rx="2"></rect>
-            <line x1="17" y1="5" x2="19" y2="5" id="Stroke-15" stroke="#2088FF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></line>
-            <line x1="17" y1="9" x2="21" y2="9" id="Stroke-15" stroke="#2088FF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"></line>
-            <path d="M8.85227273,7 C8.85227273,7.51894199 8.76988719,7.97537682 8.60511364,8.36931818 C8.44034009,8.76325955 8.21591051,9.08711994 7.93181818,9.34090909 L8.76420455,10.3863636 L7.61647727,10.3863636 L7.14772727,9.80965909 C6.83143781,9.92897787 6.49147909,9.98863636 6.12784091,9.98863636 C5.61079287,9.98863636 5.14678236,9.8712133 4.73579545,9.63636364 C4.32480855,9.40151398 4.00000119,9.06108178 3.76136364,8.61505682 C3.52272608,8.16903186 3.40340909,7.63068497 3.40340909,7 C3.40340909,6.36552713 3.52272608,5.8257598 3.76136364,5.38068182 C4.00000119,4.93560384 4.32480855,4.59611859 4.73579545,4.36221591 C5.14678236,4.12831322 5.61079287,4.01136364 6.12784091,4.01136364 C6.642995,4.01136364 7.10605855,4.12831322 7.51704545,4.36221591 C7.92803236,4.59611859 8.2533132,4.93560384 8.49289773,5.38068182 C8.73248226,5.8257598 8.85227273,6.36552713 8.85227273,7 Z M5.70170455,7.88636364 L6.74715909,7.88636364 L7.17897727,8.44034091 C7.31344764,8.27935526 7.41808675,8.07859969 7.49289773,7.83806818 C7.56770871,7.59753668 7.60511364,7.31818341 7.60511364,7 C7.60511364,6.38257267 7.47064528,5.91145996 7.20170455,5.58664773 C6.93276381,5.2618355 6.57481284,5.09943182 6.12784091,5.09943182 C5.68086898,5.09943182 5.32291801,5.2618355 5.05397727,5.58664773 C4.78503653,5.91145996 4.65056818,6.38257267 4.65056818,7 C4.65056818,7.61553338 4.78503653,8.08617261 5.05397727,8.41193182 C5.32291801,8.73769102 5.68086898,8.90056818 6.12784091,8.90056818 C6.23958389,8.90056818 6.34564344,8.89015162 6.44602273,8.86931818 L5.70170455,7.88636364 Z M10.1813315,10 L10.1813315,4 L11.4114451,4 L11.4114451,8.98579545 L13.9057633,8.98579545 L13.9057633,10 L10.1813315,10 Z" fill="#2088FF" fill-rule="nonzero"></path>
-        </g>
-    </g>
-</svg>
+<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M8.19789 8C8.19789 8.51894 8.1155 8.97538 7.95073 9.36932C7.78595 9.76326 7.56152 10.0871 7.27743 10.3409L8.10982 11.3864H6.96209L6.49334 10.8097C6.17705 10.929 5.83709 10.9886 5.47346 10.9886C4.95641 10.9886 4.4924 10.8712 4.08141 10.6364C3.67042 10.4015 3.34562 10.0611 3.10698 9.61506C2.86834 9.16903 2.74902 8.63068 2.74902 8C2.74902 7.36553 2.86834 6.82576 3.10698 6.38068C3.34562 5.9356 3.67042 5.59612 4.08141 5.36222C4.4924 5.12831 4.95641 5.01136 5.47346 5.01136C5.98861 5.01136 6.45167 5.12831 6.86266 5.36222C7.27365 5.59612 7.59893 5.9356 7.83851 6.38068C8.0781 6.82576 8.19789 7.36553 8.19789 8ZM5.04732 8.88636H6.09277L6.52459 9.44034C6.65906 9.27936 6.7637 9.0786 6.83851 8.83807C6.91332 8.59754 6.95073 8.31818 6.95073 8C6.95073 7.38257 6.81626 6.91146 6.54732 6.58665C6.27838 6.26184 5.92043 6.09943 5.47346 6.09943C5.02648 6.09943 4.66853 6.26184 4.39959 6.58665C4.13065 6.91146 3.99618 7.38257 3.99618 8C3.99618 8.61553 4.13065 9.08617 4.39959 9.41193C4.66853 9.73769 5.02648 9.90057 5.47346 9.90057C5.5852 9.90057 5.69126 9.89015 5.79164 9.86932L5.04732 8.88636ZM9.52695 11V5H10.7571V9.9858H13.2514V11H9.52695Z" fill="#24292F"/>
+<path fill-rule="evenodd" clip-rule="evenodd" d="M13 1.5H3C2.17157 1.5 1.5 2.17157 1.5 3V13C1.5 13.8284 2.17157 14.5 3 14.5H13C13.8284 14.5 14.5 13.8284 14.5 13V3C14.5 2.17157 13.8284 1.5 13 1.5ZM3 0C1.34315 0 0 1.34315 0 3V13C0 14.6569 1.34315 16 3 16H13C14.6569 16 16 14.6569 16 13V3C16 1.34315 14.6569 0 13 0H3Z" fill="#24292F"/>
+</svg>
--- a/extensions/ql-vscode/package-lock.json
+++ b/extensions/ql-vscode/package-lock.json
--- a/extensions/ql-vscode/package.json
+++ b/extensions/ql-vscode/package.json
@@ -4,7 +4,7 @@
  "description": "CodeQL for Visual Studio Code",
  "author": "GitHub",
  "private": true,
-  "version": "1.5.5",
+  "version": "1.5.8",
  "publisher": "GitHub",
  "license": "MIT",
  "icon": "media/VS-marketplace-CodeQL-icon.png",
@@ -46,6 +46,7 @@
    "onCommand:codeQL.setCurrentDatabase",
    "onCommand:codeQL.viewAst",
    "onCommand:codeQL.openReferencedFile",
+    "onCommand:codeQL.previewQueryHelp",
    "onCommand:codeQL.chooseDatabaseFolder",
    "onCommand:codeQL.chooseDatabaseArchive",
    "onCommand:codeQL.chooseDatabaseInternet",
@@ -213,7 +214,7 @@
        "codeQL.queryHistory.format": {
          "type": "string",
          "default": "%q on %d - %s, %r result count [%t]",
-          "markdownDescription": "Default string for how to label query history items.\n* %t is the time of the query\n* %q is the human-readable query name\n*%f is the query file name\n* %d is the database name\n* %r is the number of results\n* %s is a status string"
+          "markdownDescription": "Default string for how to label query history items.\n* %t is the time of the query\n* %q is the human-readable query name\n* %f is the query file name\n* %d is the database name\n* %r is the number of results\n* %s is a status string"
        },
        "codeQL.runningTests.additionalTestArguments": {
          "scope": "window",
@@ -241,7 +242,7 @@
          "scope": "application",
          "description": "Specifies whether or not to write telemetry events to the extension log."
        },
-        "codeQL.remoteRepositoryLists": {
+        "codeQL.remoteQueries.repositoryLists": {
          "type": [
            "object",
            null
@@ -256,6 +257,13 @@
          },
          "default": null,
          "markdownDescription": "[For internal use only] Lists of GitHub repositories that you want to query remotely. This should be a JSON object where each key is a user-specified name for this repository list, and the value is an array of GitHub repositories (of the form `<owner>/<repo>`)."
+        },
+        "codeQL.remoteQueries.controllerRepo": {
+          "type": "string",
+          "default": "",
+          "pattern": "^$|^(?:[a-zA-Z0-9]+-)*[a-zA-Z0-9]+/[a-zA-Z0-9-_]+$",
+          "patternErrorMessage": "Please enter a valid GitHub repository",
+          "markdownDescription": "[For internal use only] The name of the GitHub repository where you can view the progress and results of the \"Run Remote query\" command. The repository should be of the form `<owner>/<repo>`)."
        }
      }
    },
@@ -288,6 +296,10 @@
        "command": "codeQL.openReferencedFile",
        "title": "CodeQL: Open Referenced File"
      },
+      {
+        "command": "codeQL.previewQueryHelp",
+        "title": "CodeQL: Preview Query Help"
+      },
      {
        "command": "codeQL.quickQuery",
        "title": "CodeQL: Quick Query"
@@ -597,7 +609,7 @@
        {
          "command": "codeQLDatabases.setCurrentDatabase",
          "group": "inline",
-          "when": "view == codeQLDatabases"
+          "when": "view == codeQLDatabases && viewItem != currentDatabase"
        },
        {
          "command": "codeQLDatabases.removeDatabase",
@@ -674,6 +686,11 @@
          "group": "9_qlCommands",
          "when": "view == codeQLQueryHistory"
        },
+        {
+          "command": "codeQL.previewQueryHelp",
+          "group": "9_qlCommands",
+          "when": "view == codeQLQueryHistory && resourceScheme == .qhelp && isWorkspaceTrusted"
+        },
        {
          "command": "codeQLTests.showOutputDifferences",
          "group": "qltest@1",
@@ -705,6 +722,11 @@
          "command": "codeQL.openReferencedFile",
          "group": "9_qlCommands",
          "when": "resourceExtname == .qlref"
+        },
+        {
+          "command": "codeQL.previewQueryHelp",
+          "group": "9_qlCommands",
+          "when": "resourceExtname == .qhelp && isWorkspaceTrusted"
        }
      ],
      "commandPalette": [
@@ -736,6 +758,10 @@
          "command": "codeQL.openReferencedFile",
          "when": "resourceExtname == .qlref"
        },
+        {
+          "command": "codeQL.previewQueryHelp",
+          "when": "resourceExtname == .qhelp && isWorkspaceTrusted"
+        },
        {
          "command": "codeQL.setCurrentDatabase",
          "when": "false"
@@ -893,6 +919,10 @@
        {
          "command": "codeQL.openReferencedFile",
          "when": "resourceExtname == .qlref"
+        },
+        {
+          "command": "codeQL.previewQueryHelp",
+          "when": "resourceExtname == .qhelp && isWorkspaceTrusted"
        }
      ]
    },
@@ -962,6 +992,9 @@
    "react": "^16.8.6",
    "react-dom": "^16.8.6",
    "semver": "~7.3.2",
+    "stream": "^0.0.2",
+    "stream-chain": "~2.2.4",
+    "stream-json": "~1.7.3",
    "tmp": "^0.1.0",
    "tmp-promise": "~3.0.2",
    "tree-kill": "~1.2.2",
@@ -986,7 +1019,7 @@
    "@types/gulp-sourcemaps": "0.0.32",
    "@types/js-yaml": "^3.12.5",
    "@types/jszip": "~3.1.6",
-    "@types/mocha": "^8.2.0",
+    "@types/mocha": "^9.0.0",
    "@types/node": "^12.14.1",
    "@types/node-fetch": "~2.5.2",
    "@types/proxyquire": "~1.3.28",
@@ -996,6 +1029,8 @@
    "@types/semver": "~7.2.0",
    "@types/sinon": "~7.5.2",
    "@types/sinon-chai": "~3.2.3",
+    "@types/stream-chain": "~2.0.1",
+    "@types/stream-json": "~1.7.1",
    "@types/through2": "^2.0.36",
    "@types/tmp": "^0.1.0",
    "@types/unzipper": "~0.10.1",
@@ -1019,8 +1054,8 @@
    "husky": "~4.2.5",
    "jsonc-parser": "^2.3.0",
    "lint-staged": "~10.2.2",
-    "mocha": "^8.2.1",
-    "mocha-sinon": "~2.1.0",
+    "mocha": "^9.1.3",
+    "mocha-sinon": "~2.1.2",
    "npm-run-all": "^4.1.5",
    "prettier": "~2.0.5",
    "proxyquire": "~2.1.3",
--- a/extensions/ql-vscode/src/authentication.ts
+++ b/extensions/ql-vscode/src/authentication.ts
@@ -7,16 +7,11 @@ const GITHUB_AUTH_PROVIDER_ID = 'github';
 // https://docs.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps
 const SCOPES = ['repo'];

-interface OctokitAndToken {
-  octokit: Octokit.Octokit;
-  token: string;
-}
-
 /** 
 * Handles authentication to GitHub, using the VS Code [authentication API](https://code.visualstudio.com/api/references/vscode-api#authentication).
 */
 export class Credentials {
-  private octokitAndToken: OctokitAndToken | undefined;
+  private octokit: Octokit.Octokit | undefined;

  // Explicitly make the constructor private, so that we can't accidentally call the constructor from outside the class
  // without also initializing the class.
@@ -26,20 +21,17 @@ export class Credentials {
  static async initialize(context: vscode.ExtensionContext): Promise<Credentials> {
    const c = new Credentials();
    c.registerListeners(context);
-    c.octokitAndToken = await c.createOctokit(false);
+    c.octokit = await c.createOctokit(false);
    return c;
  }

-  private async createOctokit(createIfNone: boolean): Promise<OctokitAndToken | undefined> {
+  private async createOctokit(createIfNone: boolean): Promise<Octokit.Octokit | undefined> {
    const session = await vscode.authentication.getSession(GITHUB_AUTH_PROVIDER_ID, SCOPES, { createIfNone });

    if (session) {
-      return {
-        octokit: new Octokit.Octokit({
-          auth: session.accessToken
-        }),
-        token: session.accessToken
-      };
+      return new Octokit.Octokit({
+        auth: session.accessToken
+      });
    } else {
      return undefined;
    }
@@ -49,33 +41,22 @@ export class Credentials {
    // Sessions are changed when a user logs in or logs out.
    context.subscriptions.push(vscode.authentication.onDidChangeSessions(async e => {
      if (e.provider.id === GITHUB_AUTH_PROVIDER_ID) {
-        this.octokitAndToken = await this.createOctokit(false);
+        this.octokit = await this.createOctokit(false);
      }
    }));
  }

  async getOctokit(): Promise<Octokit.Octokit> {
-    if (this.octokitAndToken) {
-      return this.octokitAndToken.octokit;
+    if (this.octokit) {
+      return this.octokit;
    }

-    this.octokitAndToken = await this.createOctokit(true);
+    this.octokit = await this.createOctokit(true);
    // octokit shouldn't be undefined, since we've set "createIfNone: true".
    // The following block is mainly here to prevent a compiler error.
-    if (!this.octokitAndToken) {
+    if (!this.octokit) {
      throw new Error('Did not initialize Octokit.');
    }
-    return this.octokitAndToken.octokit;
-  }
-
-  async getToken(): Promise<string> {
-    if (this.octokitAndToken) {
-      return this.octokitAndToken.token;
-    }
-    this.octokitAndToken = await this.createOctokit(true);
-    if (!this.octokitAndToken) {
-      throw new Error('Did not initialize Octokit.');
-    }
-    return this.octokitAndToken.token;
+    return this.octokit;
  }
 }
--- a/extensions/ql-vscode/src/cli.ts
+++ b/extensions/ql-vscode/src/cli.ts
@@ -1,6 +1,5 @@
 import * as cpp from 'child-process-promise';
 import * as child_process from 'child_process';
-import * as fs from 'fs-extra';
 import * as path from 'path';
 import * as sarif from 'sarif';
 import { SemVer } from 'semver';
@@ -17,6 +16,8 @@ import { assertNever } from './pure/helpers-pure';
 import { QueryMetadata, SortDirection } from './pure/interface-types';
 import { Logger, ProgressReporter } from './logging';
 import { CompilationMessage } from './pure/messages';
+import { sarifParser } from './sarif-parser';
+import { dbSchemeToLanguage } from './helpers';

 /**
 * The version of the SARIF format that we are using.
@@ -86,6 +87,15 @@ export type QlpacksInfo = { [name: string]: string[] };
 */
 export type LanguagesInfo = { [name: string]: string[] };

+/** Information about an ML model, as resolved by `codeql resolve ml-models`. */
+export type MlModelInfo = {
+  checksum: string;
+  path: string;
+};
+
+/** The expected output of `codeql resolve ml-models`. */
+export type MlModelsInfo = { models: MlModelInfo[] };
+
 /**
 * The expected output of `codeql resolve qlref`.
 */
@@ -159,6 +169,11 @@ export class CodeQLCliServer implements Disposable {
  /** Version of current cli, lazily computed by the `getVersion()` method */
  private _version: SemVer | undefined;

+  /** 
+   * The languages supported by the current version of the CLI, computed by `getSupportedLanguages()`.
+   */
+  private _supportedLanguages: string[] | undefined;
+
  /** Path to current codeQL executable, or undefined if not running yet. */
  codeQlPath: string | undefined;

@@ -181,12 +196,14 @@ export class CodeQLCliServer implements Disposable {
      this.distributionProvider.onDidChangeDistribution(() => {
        this.restartCliServer();
        this._version = undefined;
+        this._supportedLanguages = undefined;
      });
    }
    if (this.cliConfig.onDidChangeConfiguration) {
      this.cliConfig.onDidChangeConfiguration(() => {
        this.restartCliServer();
        this._version = undefined;
+        this._supportedLanguages = undefined;
      });
    }
  }
@@ -257,11 +274,16 @@ export class CodeQLCliServer implements Disposable {
   */
  private async launchProcess(): Promise<child_process.ChildProcessWithoutNullStreams> {
    const codeQlPath = await this.getCodeQlPath();
+    const args = [];
+    if (shouldDebugCliServer()) {
+      args.push('-J=-agentlib:jdwp=transport=dt_socket,address=localhost:9012,server=n,suspend=y,quiet=y');
+    }
+
    return await spawnServer(
      codeQlPath,
      'CodeQL CLI Server',
      ['execute', 'cli-server'],
-      [],
+      args,
      this.logger,
      _data => { /**/ }
    );
@@ -571,6 +593,12 @@ export class CodeQLCliServer implements Disposable {
    return await this.runJsonCodeQlCliCommand<QueryMetadata>(['resolve', 'metadata'], [queryPath], 'Resolving query metadata');
  }

+  /** Resolves the ML models that should be available when evaluating a query. */
+  async resolveMlModels(additionalPacks: string[]): Promise<MlModelsInfo> {
+    return await this.runJsonCodeQlCliCommand<MlModelsInfo>(['resolve', 'ml-models'], ['--additional-packs',
+      additionalPacks.join(path.delimiter)], 'Resolving ML models', false);
+  }
+
  /**
   * Gets the RAM setting for the query server.
   * @param queryMemoryMb The maximum amount of RAM to use, in MB.
@@ -600,6 +628,29 @@ export class CodeQLCliServer implements Disposable {
    return await this.runJsonCodeQlCliCommand<BQRSInfo>(['bqrs', 'info'], subcommandArgs, 'Reading bqrs header');
  }

+  async databaseUnbundle(archivePath: string, target: string, name?: string): Promise<string> {
+    const subcommandArgs = [];
+    if (target) subcommandArgs.push('--target', target);
+    if (name) subcommandArgs.push('--name', name);
+    subcommandArgs.push(archivePath);
+
+    return await this.runCodeQlCliCommand(['database', 'unbundle'], subcommandArgs, `Extracting ${archivePath} to directory ${target}`);
+  }
+
+  /**
+   * Uses a .qhelp file to generate Query Help documentation in a specified format.
+   * @param pathToQhelp The path to the .qhelp file
+   * @param format The format in which the query help should be generated {@link https://codeql.github.com/docs/codeql-cli/manual/generate-query-help/#cmdoption-codeql-generate-query-help-format}
+   * @param outputDirectory The output directory for the generated file
+   */
+  async generateQueryHelp(pathToQhelp: string, outputDirectory?: string): Promise<string> {
+    const subcommandArgs = ['--format=markdown'];
+    if (outputDirectory) subcommandArgs.push('--output', outputDirectory);
+    subcommandArgs.push(pathToQhelp);
+
+    return await this.runCodeQlCliCommand(['generate', 'query-help'], subcommandArgs, `Generating qhelp in markdown format at ${outputDirectory}`);
+  }
+
  /**
  * Gets the results from a bqrs.
  * @param bqrsPath The path to the bqrs.
@@ -660,22 +711,7 @@ export class CodeQLCliServer implements Disposable {

  async interpretBqrs(metadata: QueryMetadata, resultsPath: string, interpretedResultsPath: string, sourceInfo?: SourceInfo): Promise<sarif.Log> {
    await this.runInterpretCommand(SARIF_FORMAT, metadata, resultsPath, interpretedResultsPath, sourceInfo);
-
-    let output: string;
-    try {
-      output = await fs.readFile(interpretedResultsPath, 'utf8');
-    } catch (e) {
-      const rawMessage = e.stderr || e.message;
-      const errorMessage = rawMessage.startsWith('Cannot create a string')
-        ? `SARIF too large. ${rawMessage}`
-        : rawMessage;
-      throw new Error(`Reading output of interpretation failed: ${errorMessage}`);
-    }
-    try {
-      return JSON.parse(output) as sarif.Log;
-    } catch (err) {
-      throw new Error(`Parsing output of interpretation failed: ${err.stderr || err}`);
-    }
+    return await sarifParser(interpretedResultsPath);
  }

  async generateResultsCsv(metadata: QueryMetadata, resultsPath: string, csvPath: string, sourceInfo?: SourceInfo): Promise<void> {
@@ -767,6 +803,23 @@ export class CodeQLCliServer implements Disposable {
    return await this.runJsonCodeQlCliCommand<LanguagesInfo>(['resolve', 'languages'], [], 'Resolving languages');
  }

+  /**
+   * Gets the list of available languages. Refines the result of `resolveLanguages()`, by excluding
+   * extra things like "xml" and "properties".
+   * 
+   * @returns An array of languages that are supported by the current version of the CodeQL CLI.
+   */
+  public async getSupportedLanguages(): Promise<string[]> {
+    if (!this._supportedLanguages) {
+      // Get the intersection of resolveLanguages with the list of hardcoded languages in dbSchemeToLanguage.
+      const resolvedLanguages = Object.keys(await this.resolveLanguages());
+      const hardcodedLanguages = Object.values(dbSchemeToLanguage);
+
+      this._supportedLanguages = resolvedLanguages.filter(lang => hardcodedLanguages.includes(lang));
+    }
+    return this._supportedLanguages;
+  }
+
  /**
   * Gets information about queries in a query suite.
   * @param suite The suite to resolve.
@@ -792,6 +845,39 @@ export class CodeQLCliServer implements Disposable {
    );
  }

+  async packInstall(dir: string) {
+    return this.runJsonCodeQlCliCommand(['pack', 'install'], [dir], 'Installing pack dependencies');
+  }
+
+  async packBundle(dir: string, workspaceFolders: string[], outputPath: string, precompile = true): Promise<void> {
+    const args = [
+      '-o',
+      outputPath,
+      dir,
+      '--additional-packs',
+      workspaceFolders.join(path.delimiter)
+    ];
+    if (!precompile && await this.cliConstraints.supportsNoPrecompile()) {
+      args.push('--no-precompile');
+    }
+
+    return this.runJsonCodeQlCliCommand(['pack', 'bundle'], args, 'Bundling pack');
+  }
+
+  async packPacklist(dir: string, includeQueries: boolean): Promise<string[]> {
+    const args = includeQueries ? [dir] : ['--no-include-queries', dir];
+    // since 2.7.1, packlist returns an object with a "paths" property that is a list of packs.
+    // previous versions return a list of packs.
+    const results: { paths: string[] } | string[] = await this.runJsonCodeQlCliCommand(['pack', 'packlist'], args, 'Generating the pack list');
+
+    // Once we no longer need to support 2.7.0 or earlier, we can remove this and assume all versions return an object.
+    if ('paths' in results) {
+      return results.paths;
+    } else {
+      return results;
+    }
+  }
+
  async generateDil(qloFile: string, outFile: string): Promise<void> {
    const extraArgs = await this.cliConstraints.supportsDecompileDil()
      ? ['--kind', 'dil', '-o', outFile, qloFile]
@@ -1040,6 +1126,12 @@ export function shouldDebugQueryServer() {
    && process.env.QUERY_SERVER_JAVA_DEBUG?.toLocaleLowerCase() !== 'false';
 }

+export function shouldDebugCliServer() {
+  return 'CLI_SERVER_JAVA_DEBUG' in process.env
+    && process.env.CLI_SERVER_JAVA_DEBUG !== '0'
+    && process.env.CLI_SERVER_JAVA_DEBUG?.toLocaleLowerCase() !== 'false';
+}
+
 export class CliVersionConstraint {

  /**
@@ -1065,7 +1157,7 @@ export class CliVersionConstraint {

  /**
   * CLI version where database registration was introduced
-   */
+  */
  public static CLI_VERSION_WITH_DB_REGISTRATION = new SemVer('2.4.1');

  /**
@@ -1074,6 +1166,26 @@ export class CliVersionConstraint {
   */
  public static CLI_VERSION_WITH_ALLOW_LIBRARY_PACKS_IN_RESOLVE_QUERIES = new SemVer('2.6.1');

+  /**
+   * CLI version where the `database unbundle` subcommand was introduced.
+   */
+  public static CLI_VERSION_WITH_DATABASE_UNBUNDLE = new SemVer('2.6.0');
+
+  /**
+   * CLI version where the `--no-precompile` option for pack creation was introduced.
+   */
+  public static CLI_VERSION_WITH_NO_PRECOMPILE = new SemVer('2.7.1');
+
+  /**
+   * CLI version where remote queries are supported.
+   */
+  public static CLI_VERSION_REMOTE_QUERIES = new SemVer('2.6.3');
+
+  /**
+   * CLI version where the `resolve ml-models` subcommand was introduced.
+   */
+  public static CLI_VERSION_WITH_RESOLVE_ML_MODELS = new SemVer('2.7.3');
+
  constructor(private readonly cli: CodeQLCliServer) {
    /**/
  }
@@ -1105,4 +1217,21 @@ export class CliVersionConstraint {
  async supportsDatabaseRegistration() {
    return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_DB_REGISTRATION);
  }
+
+  async supportsDatabaseUnbundle() {
+    return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_DATABASE_UNBUNDLE);
+  }
+
+  async supportsNoPrecompile() {
+    return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_NO_PRECOMPILE);
+  }
+
+  async supportsRemoteQueries() {
+    return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_REMOTE_QUERIES);
+  }
+
+  async supportsResolveMlModels() {
+    return this.isVersionAtLeast(CliVersionConstraint.CLI_VERSION_WITH_RESOLVE_ML_MODELS);
+  }
+
 }
--- a/extensions/ql-vscode/src/config.ts
+++ b/extensions/ql-vscode/src/config.ts
@@ -298,15 +298,50 @@ export function isCanary() {
 */
 export const NO_CACHE_AST_VIEWER = new Setting('disableCache', AST_VIEWER_SETTING);

+// Settings for remote queries
+const REMOTE_QUERIES_SETTING = new Setting('remoteQueries', ROOT_SETTING);
+
 /**
 * Lists of GitHub repositories that you want to query remotely via the "Run Remote query" command.
 * Note: This command is only available for internal users.
- * 
+ *
 * This setting should be a JSON object where each key is a user-specified name (string),
 * and the value is an array of GitHub repositories (of the form `<owner>/<repo>`).
 */
-const REMOTE_REPO_LISTS = new Setting('remoteRepositoryLists', ROOT_SETTING);
+const REMOTE_REPO_LISTS = new Setting('repositoryLists', REMOTE_QUERIES_SETTING);

 export function getRemoteRepositoryLists(): Record<string, string[]> | undefined {
  return REMOTE_REPO_LISTS.getValue<Record<string, string[]>>() || undefined;
 }
+
+export async function setRemoteRepositoryLists(lists: Record<string, string[]> | undefined) {
+  await REMOTE_REPO_LISTS.updateValue(lists, ConfigurationTarget.Global);
+}
+
+/**
+ * The name of the "controller" repository that you want to use with the "Run Remote query" command.
+ * Note: This command is only available for internal users.
+ *
+ * This setting should be a GitHub repository of the form `<owner>/<repo>`.
+ */
+const REMOTE_CONTROLLER_REPO = new Setting('controllerRepo', REMOTE_QUERIES_SETTING);
+
+export function getRemoteControllerRepo(): string | undefined {
+  return REMOTE_CONTROLLER_REPO.getValue<string>() || undefined;
+}
+
+export async function setRemoteControllerRepo(repo: string | undefined) {
+  await REMOTE_CONTROLLER_REPO.updateValue(repo, ConfigurationTarget.Global);
+}
+
+/**
+ * Whether to insecurely load ML models from CodeQL packs.
+ *
+ * This setting is for internal users only.
+ */
+const SHOULD_INSECURELY_LOAD_MODELS_FROM_PACKS =
+  new Setting('shouldInsecurelyLoadModelsFromPacks', RUNNING_QUERIES_SETTING);
+
+export function shouldInsecurelyLoadMlModelsFromPacks(): boolean {
+  return SHOULD_INSECURELY_LOAD_MODELS_FROM_PACKS.getValue<boolean>();
+}
--- a/extensions/ql-vscode/src/databaseFetcher.ts
+++ b/extensions/ql-vscode/src/databaseFetcher.ts
@@ -1,12 +1,13 @@
 import fetch, { Response } from 'node-fetch';
-import * as unzipper from 'unzipper';
 import { zip } from 'zip-a-folder';
+import * as unzipper from 'unzipper';
 import {
  Uri,
  CancellationToken,
  commands,
  window,
 } from 'vscode';
+import { CodeQLCliServer } from './cli';
 import * as fs from 'fs-extra';
 import * as path from 'path';

@@ -32,6 +33,7 @@ export async function promptImportInternetDatabase(
  storagePath: string,
  progress: ProgressCallback,
  token: CancellationToken,
+  cli?: CodeQLCliServer
 ): Promise<DatabaseItem | undefined> {
  const databaseUrl = await window.showInputBox({
    prompt: 'Enter URL of zipfile of database to download',
@@ -47,7 +49,8 @@ export async function promptImportInternetDatabase(
    databaseManager,
    storagePath,
    progress,
-    token
+    token,
+    cli
  );

  if (item) {
@@ -70,8 +73,14 @@ export async function promptImportLgtmDatabase(
  databaseManager: DatabaseManager,
  storagePath: string,
  progress: ProgressCallback,
-  token: CancellationToken
+  token: CancellationToken,
+  cli?: CodeQLCliServer
 ): Promise<DatabaseItem | undefined> {
+  progress({
+    message: 'Choose project',
+    step: 1,
+    maxStep: 2
+  });
  const lgtmUrl = await window.showInputBox({
    prompt:
      'Enter the project slug or URL on LGTM (e.g., g/github/codeql or https://lgtm.com/projects/g/github/codeql)',
@@ -81,14 +90,15 @@ export async function promptImportLgtmDatabase(
  }

  if (looksLikeLgtmUrl(lgtmUrl)) {
-    const databaseUrl = await convertToDatabaseUrl(lgtmUrl);
+    const databaseUrl = await convertToDatabaseUrl(lgtmUrl, progress);
    if (databaseUrl) {
      const item = await databaseArchiveFetcher(
        databaseUrl,
        databaseManager,
        storagePath,
        progress,
-        token
+        token,
+        cli
      );
      if (item) {
        await commands.executeCommand('codeQLDatabases.focus');
@@ -102,6 +112,16 @@ export async function promptImportLgtmDatabase(
  return;
 }

+export async function retrieveCanonicalRepoName(lgtmUrl: string) {
+  const givenRepoName = extractProjectSlug(lgtmUrl);
+  const response = await checkForFailingResponse(await fetch(`https://api.github.com/repos/${givenRepoName}`), 'Failed to locate the repository on github');
+  const repo = await response.json();
+  if (!repo || !repo.full_name) {
+    return;
+  }
+  return repo.full_name;
+}
+
 /**
 * Imports a database from a local archive.
 *
@@ -115,6 +135,7 @@ export async function importArchiveDatabase(
  storagePath: string,
  progress: ProgressCallback,
  token: CancellationToken,
+  cli?: CodeQLCliServer,
 ): Promise<DatabaseItem | undefined> {
  try {
    const item = await databaseArchiveFetcher(
@@ -122,7 +143,8 @@ export async function importArchiveDatabase(
      databaseManager,
      storagePath,
      progress,
-      token
+      token,
+      cli
    );
    if (item) {
      await commands.executeCommand('codeQLDatabases.focus');
@@ -154,7 +176,8 @@ async function databaseArchiveFetcher(
  databaseManager: DatabaseManager,
  storagePath: string,
  progress: ProgressCallback,
-  token: CancellationToken
+  token: CancellationToken,
+  cli?: CodeQLCliServer,
 ): Promise<DatabaseItem> {
  progress({
    message: 'Getting database',
@@ -168,9 +191,9 @@ async function databaseArchiveFetcher(
  const unzipPath = await getStorageFolder(storagePath, databaseUrl);

  if (isFile(databaseUrl)) {
-    await readAndUnzip(databaseUrl, unzipPath, progress);
+    await readAndUnzip(databaseUrl, unzipPath, cli, progress);
  } else {
-    await fetchAndUnzip(databaseUrl, unzipPath, progress);
+    await fetchAndUnzip(databaseUrl, unzipPath, cli, progress);
  }

  progress({
@@ -244,6 +267,7 @@ function validateHttpsUrl(databaseUrl: string) {
 async function readAndUnzip(
  zipUrl: string,
  unzipPath: string,
+  cli?: CodeQLCliServer,
  progress?: ProgressCallback
 ) {
  // TODO: Providing progress as the file is unzipped is currently blocked
@@ -254,16 +278,22 @@ async function readAndUnzip(
    step: 9,
    message: `Unzipping into ${path.basename(unzipPath)}`
  });
-  // Must get the zip central directory since streaming the
-  // zip contents may not have correct local file headers.
-  // Instead, we can only rely on the central directory.
-  const directory = await unzipper.Open.file(zipFile);
-  await directory.extract({ path: unzipPath });
+  if (cli && await cli.cliConstraints.supportsDatabaseUnbundle()) {
+    // Use the `database unbundle` command if the installed cli version supports it
+    await cli.databaseUnbundle(zipFile, unzipPath);
+  } else {
+    // Must get the zip central directory since streaming the
+    // zip contents may not have correct local file headers.
+    // Instead, we can only rely on the central directory.
+    const directory = await unzipper.Open.file(zipFile);
+    await directory.extract({ path: unzipPath });
+  }
 }

 async function fetchAndUnzip(
  databaseUrl: string,
  unzipPath: string,
+  cli?: CodeQLCliServer,
  progress?: ProgressCallback
 ) {
  // Although it is possible to download and stream directly to an unzipped directory,
@@ -280,7 +310,7 @@ async function fetchAndUnzip(
    step: 1,
  });

-  const response = await checkForFailingResponse(await fetch(databaseUrl));
+  const response = await checkForFailingResponse(await fetch(databaseUrl), 'Error downloading database');
  const archiveFileStream = fs.createWriteStream(archivePath);

  const contentLength = response.headers.get('content-length');
@@ -293,13 +323,14 @@ async function fetchAndUnzip(
      .on('error', reject)
  );

-  await readAndUnzip(Uri.file(archivePath).toString(true), unzipPath, progress);
+  await readAndUnzip(Uri.file(archivePath).toString(true), unzipPath, cli, progress);
+

  // remove archivePath eagerly since these archives can be large.
  await fs.remove(archivePath);
 }

-async function checkForFailingResponse(response: Response): Promise<Response | never> {
+async function checkForFailingResponse(response: Response, errorMessage: string): Promise<Response | never> {
  if (response.ok) {
    return response;
  }
@@ -313,7 +344,7 @@ async function checkForFailingResponse(response: Response): Promise<Response | n
  } catch (e) {
    msg = text;
  }
-  throw new Error(`Error downloading database.\n\nReason: ${msg}`);
+  throw new Error(`${errorMessage}.\n\nReason: ${msg}`);
 }

 function isFile(databaseUrl: string) {
@@ -403,25 +434,40 @@ function convertRawLgtmSlug(maybeSlug: string): string | undefined {
  }
  return;
 }
+ 
+function extractProjectSlug(lgtmUrl: string): string | undefined {
+  // Only matches the '/g/' provider (github)
+  const re = new RegExp('https://lgtm.com/projects/g/(.*[^/])');
+  const match = lgtmUrl.match(re);
+  if (!match) {
+    return;
+  }
+  return match[1];
+}

 // exported for testing
-export async function convertToDatabaseUrl(lgtmUrl: string) {
+export async function convertToDatabaseUrl(
+  lgtmUrl: string,
+  progress: ProgressCallback) {
  try {
    lgtmUrl = convertRawLgtmSlug(lgtmUrl) || lgtmUrl;
-
-    const uri = Uri.parse(lgtmUrl, true);
-    const paths = ['api', 'v1.0'].concat(
-      uri.path.split('/').filter((segment) => segment)
-    ).slice(0, 6);
-    const projectUrl = `https://lgtm.com/${paths.join('/')}`;
-    const projectResponse = await fetch(projectUrl);
-    const projectJson = await projectResponse.json();
+    let projectJson = await downloadLgtmProjectMetadata(lgtmUrl);

    if (projectJson.code === 404) {
-      throw new Error();
+      // fallback check for github repositories with same name but different case
+      // will fail for other providers
+      let canonicalName = await retrieveCanonicalRepoName(lgtmUrl);
+      if (!canonicalName) {
+        throw new Error(`Project was not found at ${lgtmUrl}.`);
+      }
+      canonicalName = convertRawLgtmSlug(`g/${canonicalName}`);
+      projectJson = await downloadLgtmProjectMetadata(canonicalName);
+      if (projectJson.code === 404) {
+        throw new Error('Failed to download project from LGTM.');
+      }
    }

-    const language = await promptForLanguage(projectJson);
+    const language = await promptForLanguage(projectJson, progress);
    if (!language) {
      return;
    }
@@ -438,9 +484,25 @@ export async function convertToDatabaseUrl(lgtmUrl: string) {
  }
 }

+async function downloadLgtmProjectMetadata(lgtmUrl: string): Promise<any> {
+  const uri = Uri.parse(lgtmUrl, true);
+  const paths = ['api', 'v1.0'].concat(
+    uri.path.split('/').filter((segment) => segment)
+  ).slice(0, 6);
+  const projectUrl = `https://lgtm.com/${paths.join('/')}`;
+  const projectResponse = await fetch(projectUrl);
+  return projectResponse.json();
+}
+
 async function promptForLanguage(
-  projectJson: any
+  projectJson: any,
+  progress: ProgressCallback
 ): Promise<string | undefined> {
+  progress({
+    message: 'Choose language',
+    step: 2,
+    maxStep: 2
+  });
  if (!projectJson?.languages?.length) {
    return;
  }
--- a/extensions/ql-vscode/src/databases-ui.ts
+++ b/extensions/ql-vscode/src/databases-ui.ts
@@ -135,6 +135,7 @@ class DatabaseTreeDataProvider extends DisposableObject
        this.extensionPath,
        SELECTED_DATABASE_ICON
      );
+      item.contextValue = 'currentDatabase';
    } else if (element.error !== undefined) {
      item.iconPath = joinThemableIconPath(
        this.extensionPath,
@@ -295,7 +296,7 @@ export class DatabaseUI extends DisposableObject {
        'codeQLDatabases.chooseDatabaseLgtm',
        this.handleChooseDatabaseLgtm,
        {
-          title: 'Adding database from LGTM. Choose a language from the dropdown, if requested.',
+          title: 'Adding database from LGTM',
        })
    );
    this.push(
@@ -451,14 +452,13 @@ export class DatabaseUI extends DisposableObject {
  handleChooseDatabaseInternet = async (
    progress: ProgressCallback,
    token: CancellationToken
-  ): Promise<
-    DatabaseItem | undefined
-  > => {
+  ): Promise<DatabaseItem | undefined> => {
    return await promptImportInternetDatabase(
      this.databaseManager,
      this.storagePath,
      progress,
-      token
+      token,
+      this.queryServer?.cliServer
    );
  };

@@ -470,7 +470,8 @@ export class DatabaseUI extends DisposableObject {
      this.databaseManager,
      this.storagePath,
      progress,
-      token
+      token,
+      this.queryServer?.cliServer
    );
  };

@@ -580,7 +581,8 @@ export class DatabaseUI extends DisposableObject {
          this.databaseManager,
          this.storagePath,
          progress,
-          token
+          token,
+          this.queryServer?.cliServer
        );
      } else {
        await this.setCurrentDatabase(progress, token, uri);
@@ -696,7 +698,6 @@ export class DatabaseUI extends DisposableObject {
    token: CancellationToken,
  ): Promise<DatabaseItem | undefined> {
    const uri = await chooseDatabaseDir(byFolder);
-
    if (!uri) {
      return undefined;
    }
@@ -713,7 +714,8 @@ export class DatabaseUI extends DisposableObject {
        this.databaseManager,
        this.storagePath,
        progress,
-        token
+        token,
+        this.queryServer?.cliServer
      );
    }
  }
--- a/extensions/ql-vscode/src/databases.ts
+++ b/extensions/ql-vscode/src/databases.ts
@@ -121,20 +121,21 @@ async function findDataset(parentDirectory: string): Promise<vscode.Uri> {
  return vscode.Uri.file(dbAbsolutePath);
 }

-async function findSourceArchive(
+// exported for testing
+export async function findSourceArchive(
  databasePath: string, silent = false
 ): Promise<vscode.Uri | undefined> {
-
  const relativePaths = ['src', 'output/src_archive'];

  for (const relativePath of relativePaths) {
    const basePath = path.join(databasePath, relativePath);
    const zipPath = basePath + '.zip';

-    if (await fs.pathExists(basePath)) {
-      return vscode.Uri.file(basePath);
-    } else if (await fs.pathExists(zipPath)) {
+    // Prefer using a zip archive over a directory.
+    if (await fs.pathExists(zipPath)) {
      return encodeArchiveBasePath(zipPath);
+    } else if (await fs.pathExists(basePath)) {
+      return vscode.Uri.file(basePath);
    }
  }
  if (!silent) {
@@ -161,7 +162,6 @@ async function resolveDatabase(
    datasetUri,
    sourceArchiveUri
  };
-
 }

 /** Gets the relative paths of all `.dbscheme` files in the given directory. */
@@ -258,7 +258,7 @@ export interface DatabaseItem {
   * Returns the root uri of the virtual filesystem for this database's source archive,
   * as displayed in the filesystem explorer.
   */
-  getSourceArchiveExplorerUri(): vscode.Uri | undefined;
+  getSourceArchiveExplorerUri(): vscode.Uri;

  /**
   * Holds if `uri` belongs to this database's source archive.
@@ -274,6 +274,11 @@ export interface DatabaseItem {
   * Gets the state of this database, to be persisted in the workspace state.
   */
  getPersistedState(): PersistedDatabaseItem;
+
+  /**
+   * Verifies that this database item has a zipped source folder. Returns an error message if it does not.
+   */
+  verifyZippedSources(): string | undefined;
 }

 export enum DatabaseEventKind {
@@ -459,13 +464,26 @@ export class DatabaseItemImpl implements DatabaseItem {
  /**
   * Returns the root uri of the virtual filesystem for this database's source archive.
   */
-  public getSourceArchiveExplorerUri(): vscode.Uri | undefined {
+  public getSourceArchiveExplorerUri(): vscode.Uri {
    const sourceArchive = this.sourceArchive;
-    if (sourceArchive === undefined || !sourceArchive.fsPath.endsWith('.zip'))
-      return undefined;
+    if (sourceArchive === undefined || !sourceArchive.fsPath.endsWith('.zip')) {
+      throw new Error(this.verifyZippedSources());
+    }
    return encodeArchiveBasePath(sourceArchive.fsPath);
  }

+  public verifyZippedSources(): string | undefined {
+    const sourceArchive = this.sourceArchive;
+    if (sourceArchive === undefined) {
+      return `${this.name} has no source archive.`;
+    }
+
+    if (!sourceArchive.fsPath.endsWith('.zip')) {
+      return `${this.name} has a source folder that is unzipped.`;
+    }
+    return;
+  }
+
  /**
   * Holds if `uri` belongs to this database's source archive.
   */
@@ -603,26 +621,28 @@ export class DatabaseManager extends DisposableObject {
      // This is undesirable, as we might be adding and removing many
      // workspace folders as the user adds and removes databases.
      const end = (vscode.workspace.workspaceFolders || []).length;
+
+      const msg = item.verifyZippedSources();
+      if (msg) {
+        void logger.log(`Could not add source folder because ${msg}`);
+        return;
+      }
+
      const uri = item.getSourceArchiveExplorerUri();
-      if (uri === undefined) {
-        void logger.log(`Couldn't obtain file explorer uri for ${item.name}`);
-      }
-      else {
-        void logger.log(`Adding workspace folder for ${item.name} source archive at index ${end}`);
-        if ((vscode.workspace.workspaceFolders || []).length < 2) {
-          // Adding this workspace folder makes the workspace
-          // multi-root, which may surprise the user. Let them know
-          // we're doing this.
-          void vscode.window.showInformationMessage(`Adding workspace folder for source archive of database ${item.name}.`);
-        }
-        vscode.workspace.updateWorkspaceFolders(end, 0, {
-          name: `[${item.name} source archive]`,
-          uri,
-        });
-        // vscode api documentation says we must to wait for this event
-        // between multiple `updateWorkspaceFolders` calls.
-        await eventFired(vscode.workspace.onDidChangeWorkspaceFolders);
+      void logger.log(`Adding workspace folder for ${item.name} source archive at index ${end}`);
+      if ((vscode.workspace.workspaceFolders || []).length < 2) {
+        // Adding this workspace folder makes the workspace
+        // multi-root, which may surprise the user. Let them know
+        // we're doing this.
+        void vscode.window.showInformationMessage(`Adding workspace folder for source archive of database ${item.name}.`);
      }
+      vscode.workspace.updateWorkspaceFolders(end, 0, {
+        name: `[${item.name} source archive]`,
+        uri,
+      });
+      // vscode api documentation says we must to wait for this event
+      // between multiple `updateWorkspaceFolders` calls.
+      await eventFired(vscode.workspace.onDidChangeWorkspaceFolders);
    }
  }

@@ -731,6 +751,8 @@ export class DatabaseManager extends DisposableObject {
      this._currentDatabaseItem = item;
      this.updatePersistedCurrentDatabaseItem();

+      await vscode.commands.executeCommand('setContext', 'codeQL.currentDatabaseItem', item?.name);
+
      this._onDidChangeCurrentDatabaseItem.fire({
        item,
        kind: DatabaseEventKind.Change
@@ -811,6 +833,9 @@ export class DatabaseManager extends DisposableObject {
      vscode.workspace.updateWorkspaceFolders(folderIndex, 1);
    }

+    // Remove this database item from the allow-list
+    await this.deregisterDatabase(progress, token, item);
+
    // Delete folder from file system only if it is controlled by the extension
    if (this.isExtensionControlledLocation(item.databaseUri)) {
      void logger.log('Deleting database from filesystem.');
@@ -819,9 +844,6 @@ export class DatabaseManager extends DisposableObject {
        e => void logger.log(`Failed to delete '${item.databaseUri.fsPath}'. Reason: ${e.message}`));
    }

-    // Remove this database item from the allow-list
-    await this.deregisterDatabase(progress, token, item);
-
    // note that we use undefined as the item in order to reset the entire tree
    this._onDidChangeDatabaseItem.fire({
      item: undefined,
--- a/extensions/ql-vscode/src/extension.ts
+++ b/extensions/ql-vscode/src/extension.ts
@@ -16,6 +16,7 @@ import {
 import { LanguageClient } from 'vscode-languageclient';
 import * as os from 'os';
 import * as path from 'path';
+import * as tmp from 'tmp-promise';
 import { testExplorerExtensionId, TestHub } from 'vscode-test-adapter-api';

 import { AstViewer } from './astViewer';
@@ -493,12 +494,40 @@ async function activateWithInstalledDistribution(
    }
  }

+  const qhelpTmpDir = tmp.dirSync({ prefix: 'qhelp_', keep: false, unsafeCleanup: true });
+  ctx.subscriptions.push({ dispose: qhelpTmpDir.removeCallback });
+
+  async function previewQueryHelp(
+    selectedQuery: Uri
+  ): Promise<void> {
+    // selectedQuery is unpopulated when executing through the command palette
+    const pathToQhelp =  selectedQuery ? selectedQuery.fsPath : window.activeTextEditor?.document.uri.fsPath;
+    if(pathToQhelp) {
+      // Create temporary directory
+      const relativePathToMd = path.basename(pathToQhelp, '.qhelp') + '.md';
+      const absolutePathToMd = path.join(qhelpTmpDir.name, relativePathToMd);
+      const uri = Uri.file(absolutePathToMd);
+      try {
+        await cliServer.generateQueryHelp(pathToQhelp , absolutePathToMd);
+        await commands.executeCommand('markdown.showPreviewToSide', uri);
+      } catch (err) {
+        const errorMessage =  err.message.includes('Generating qhelp in markdown') ? (
+          `Could not generate markdown from ${pathToQhelp}: Bad formatting in .qhelp file.`
+        ) : `Could not open a preview of the generated file (${absolutePathToMd}).`;
+        void helpers.showAndLogErrorMessage(errorMessage, { fullMessage: `${errorMessage}\n${err}` });
+      }
+    }
+    
+  }
+
  async function openReferencedFile(
    selectedQuery: Uri
  ): Promise<void> {
-    if (qs !== undefined) {
+    // If no file is selected, the path of the file in the editor is selected
+    const path = selectedQuery?.fsPath || window.activeTextEditor?.document.uri.fsPath;
+    if (qs !== undefined && path) {
      if (await cliServer.cliConstraints.supportsResolveQlref()) {
-        const resolved = await cliServer.resolveQlref(selectedQuery.path);
+        const resolved = await cliServer.resolveQlref(path);
        const uri = Uri.file(resolved.resolvedPath);
        await window.showTextDocument(uri, { preview: false });
      } else {
@@ -716,13 +745,25 @@ async function activateWithInstalledDistribution(
  );
  // The "runRemoteQuery" command is internal-only.
  ctx.subscriptions.push(
-    commandRunner('codeQL.runRemoteQuery', async (
+    commandRunnerWithProgress('codeQL.runRemoteQuery', async (
+      progress: ProgressCallback,
+      token: CancellationToken,
      uri: Uri | undefined
    ) => {
      if (isCanary()) {
+        progress({
+          maxStep: 5,
+          step: 0,
+          message: 'Getting credentials'
+        });
        const credentials = await Credentials.initialize(ctx);
-        await runRemoteQuery(cliServer, credentials, uri || window.activeTextEditor?.document.uri);
+        await runRemoteQuery(cliServer, credentials, uri || window.activeTextEditor?.document.uri, false, progress, token);
+      } else {
+        throw new Error('Remote queries require the CodeQL Canary version to run.');
      }
+    }, {
+      title: 'Run Remote Query',
+      cancellable: true
    })
  );
  ctx.subscriptions.push(
@@ -732,6 +773,13 @@ async function activateWithInstalledDistribution(
    )
  );

+  ctx.subscriptions.push(
+    commandRunner(
+      'codeQL.previewQueryHelp',
+      previewQueryHelp
+    )
+  );
+
  ctx.subscriptions.push(
    commandRunnerWithProgress('codeQL.restartQueryServer', async (
      progress: ProgressCallback,
@@ -771,7 +819,7 @@ async function activateWithInstalledDistribution(
    ) =>
      databaseUI.handleChooseDatabaseLgtm(progress, token),
      {
-        title: 'Adding database from LGTM. Choose a language from the dropdown, if requested.',
+        title: 'Adding database from LGTM',
      })
  );
  ctx.subscriptions.push(
@@ -792,11 +840,19 @@ async function activateWithInstalledDistribution(

  ctx.subscriptions.push(
    commandRunner('codeQL.copyVersion', async () => {
-      const text = `CodeQL extension version: ${extension?.packageJSON.version} \nCodeQL CLI version: ${await cliServer.getVersion()} \nPlatform: ${os.platform()} ${os.arch()}`;
+      const text = `CodeQL extension version: ${extension?.packageJSON.version} \nCodeQL CLI version: ${await getCliVersion()} \nPlatform: ${os.platform()} ${os.arch()}`;
      await env.clipboard.writeText(text);
      void helpers.showAndLogInformationMessage(text);
    }));

+  const getCliVersion = async () => {
+    try {
+      return await cliServer.getVersion();
+    } catch {
+      return '<missing>';
+    }
+  };
+
  // The "authenticateToGitHub" command is internal-only.
  ctx.subscriptions.push(
    commandRunner('codeQL.authenticateToGitHub', async () => {
--- a/extensions/ql-vscode/src/helpers.ts
+++ b/extensions/ql-vscode/src/helpers.ts
@@ -10,6 +10,7 @@ import {
  env
 } from 'vscode';
 import { CodeQLCliServer, QlpacksInfo } from './cli';
+import { UserCancellationException } from './commandRunner';
 import { logger } from './logging';

 /**
@@ -414,13 +415,14 @@ export class CachedOperation<U> {
 * @see cli.CliVersionConstraint.supportsLanguageName
 * @see cli.CodeQLCliServer.resolveDatabase
 */
-const dbSchemeToLanguage = {
+export const dbSchemeToLanguage = {
  'semmlecode.javascript.dbscheme': 'javascript',
  'semmlecode.cpp.dbscheme': 'cpp',
  'semmlecode.dbscheme': 'java',
  'semmlecode.python.dbscheme': 'python',
  'semmlecode.csharp.dbscheme': 'csharp',
-  'go.dbscheme': 'go'
+  'go.dbscheme': 'go',
+  'ruby.dbscheme': 'ruby'
 };

 export const languageToDbScheme = Object.entries(dbSchemeToLanguage).reduce((acc, [k, v]) => {
@@ -494,14 +496,23 @@ export async function findLanguage(
      void logger.log('Could not autodetect query language. Select language manually.');
    }
  }
-  const availableLanguages = Object.keys(await cliServer.resolveLanguages());
+
+  // will be undefined if user cancels the quick pick.
+  return await askForLanguage(cliServer, false);
+}
+
+export async function askForLanguage(cliServer: CodeQLCliServer, throwOnEmpty = true): Promise<string | undefined> {
  const language = await Window.showQuickPick(
-    availableLanguages,
+    await cliServer.getSupportedLanguages(),
    { placeHolder: 'Select target language for your query', ignoreFocusOut: true }
  );
  if (!language) {
    // This only happens if the user cancels the quick pick.
-    void showAndLogErrorMessage('Language not found. Language must be specified manually.');
+    if (throwOnEmpty) {
+      throw new UserCancellationException('Cancelled.');
+    } else {
+      void showAndLogErrorMessage('Language not found. Language must be specified manually.');
+    }
  }
  return language;
 }
--- a/extensions/ql-vscode/src/interface-utils.ts
+++ b/extensions/ql-vscode/src/interface-utils.ts
@@ -70,7 +70,7 @@ function resolveFivePartLocation(
    Math.max(0, loc.startLine - 1),
    Math.max(0, loc.startColumn - 1),
    Math.max(0, loc.endLine - 1),
-    Math.max(0, loc.endColumn)
+    Math.max(1, loc.endColumn)
  );

  return new Location(databaseItem.resolveSourceFile(loc.uri), range);
--- a/extensions/ql-vscode/src/interface.ts
+++ b/extensions/ql-vscode/src/interface.ts
@@ -365,8 +365,7 @@ export class InterfaceManager extends DisposableObject {
      const showButton = 'View Results';
      const queryName = results.queryName;
      const resultPromise = vscode.window.showInformationMessage(
-        `Finished running query ${
-        queryName.length > 0 ? ` "${queryName}"` : ''
+        `Finished running query ${queryName.length > 0 ? ` "${queryName}"` : ''
        }.`,
        showButton
      );
@@ -502,7 +501,12 @@ export class InterfaceManager extends DisposableObject {
    );

    const resultSetSchemas = await this.getResultSetSchemas(results, sorted ? selectedTable : '');
-    const resultSetNames = resultSetSchemas.map(schema => schema.name);
+
+    // If there is a specific sorted table selected, a different bqrs file is loaded that doesn't have all the result set names.
+    // Make sure that we load all result set names here.
+    // See https://github.com/github/vscode-codeql/issues/1005
+    const allResultSetSchemas = sorted ? await this.getResultSetSchemas(results, '') : resultSetSchemas;
+    const resultSetNames = allResultSetSchemas.map(schema => schema.name);

    const schema = resultSetSchemas.find(
      (resultSet) => resultSet.name == selectedTable
--- a/extensions/ql-vscode/src/pure/bqrs-utils.ts
+++ b/extensions/ql-vscode/src/pure/bqrs-utils.ts
@@ -83,8 +83,7 @@ export function isLineColumnLoc(loc: UrlValue): loc is LineColumnLocation {
    && 'startLine' in loc
    && 'startColumn' in loc
    && 'endLine' in loc
-    && 'endColumn' in loc
-    && loc.endColumn > 0;
+    && 'endColumn' in loc;
 }

 export function isWholeFileLoc(loc: UrlValue): loc is WholeFileLocation {
--- a/extensions/ql-vscode/src/pure/messages.ts
+++ b/extensions/ql-vscode/src/pure/messages.ts
@@ -711,6 +711,11 @@ export interface EvaluateQueriesParams {

 export type TemplateDefinitions = { [key: string]: TemplateSource }

+export interface MlModel {
+  /** A URI pointing to the root directory of the model. */
+  uri: string;
+}
+
 /**
 * A single query that should be run
 */
@@ -744,6 +749,11 @@ export interface QueryToRun {
   * map should be set to the empty set or give an error.
   */
  allowUnknownTemplates: boolean;
+  /**
+   * The list of ML models that should be made available
+   * when evaluating the query.
+   */
+  availableMlModels?: MlModel[];
 }

 /**
--- a/extensions/ql-vscode/src/quick-query.ts
+++ b/extensions/ql-vscode/src/quick-query.ts
@@ -118,7 +118,7 @@ export async function displayQuickQuery(
    // Only rewrite the qlpack file if the database has changed
    if (shouldRewrite) {
      const quickQueryQlpackYaml: any = {
-        name: 'quick-query',
+        name: 'vscode/quick-query',
        version: '1.0.0',
        libraryPathDependencies: [qlpack]
      };
--- a/extensions/ql-vscode/src/run-queries.ts
+++ b/extensions/ql-vscode/src/run-queries.ts
@@ -8,7 +8,8 @@ import {
  TextDocument,
  TextEditor,
  Uri,
-  window
+  window,
+  workspace
 } from 'vscode';
 import { ErrorCodes, ResponseError } from 'vscode-languageclient';

@@ -86,6 +87,7 @@ export class QueryInfo {
  async run(
    qs: qsClient.QueryServerClient,
    upgradeQlo: string | undefined,
+    availableMlModels: cli.MlModelInfo[],
    progress: ProgressCallback,
    token: CancellationToken,
  ): Promise<messages.EvaluationResult> {
@@ -93,12 +95,15 @@ export class QueryInfo {

    const callbackId = qs.registerCallback(res => { result = res; });

+    const availableMlModelUris: messages.MlModel[] = availableMlModels.map(model => ({ uri: Uri.file(model.path).toString(true) }));
+
    const queryToRun: messages.QueryToRun = {
      resultsPath: this.resultsPaths.resultsPath,
      qlo: Uri.file(this.compiledQueryPath).toString(),
      compiledUpgrade: upgradeQlo && Uri.file(upgradeQlo).toString(),
      allowUnknownTemplates: true,
      templateValues: this.templates,
+      availableMlModels: availableMlModelUris,
      id: callbackId,
      timeoutSecs: qs.config.timeoutSecs,
    };
@@ -612,6 +617,24 @@ export async function compileAndRunQueryAgainstDatabase(
    void logger.log(`Couldn't resolve metadata for ${qlProgram.queryPath}: ${e}`);
  }

+  let availableMlModels: cli.MlModelInfo[] = [];
+  // The `capabilities.untrustedWorkspaces.restrictedConfigurations` entry in package.json doesn't
+  // work with hidden settings, so we manually check that the workspace is trusted before looking at
+  // whether the `shouldInsecurelyLoadMlModelsFromPacks` setting is enabled.
+  if (workspace.isTrusted &&
+    config.isCanary() &&
+    config.shouldInsecurelyLoadMlModelsFromPacks() &&
+    await cliServer.cliConstraints.supportsResolveMlModels()) {
+    try {
+      availableMlModels = (await cliServer.resolveMlModels(diskWorkspaceFolders)).models;
+      void logger.log(`Found available ML models at the following paths: ${availableMlModels.map(x => `'${x.path}'`).join(', ')}.`);
+    } catch (e) {
+      const message = `Couldn't resolve available ML models for ${qlProgram.queryPath}. Running the ` +
+        `query without any ML models: ${e}.`;
+      void showAndLogErrorMessage(message);
+    }
+  }
+
  const query = new QueryInfo(qlProgram, db, packConfig.dbscheme, quickEvalPosition, metadata, templates);

  const upgradeDir = await tmp.dir({ dir: upgradesTmpDir.name, unsafeCleanup: true });
@@ -634,7 +657,7 @@ export async function compileAndRunQueryAgainstDatabase(
    }

    if (errors.length === 0) {
-      const result = await query.run(qs, upgradeQlo, progress, token);
+      const result = await query.run(qs, upgradeQlo, availableMlModels, progress, token);
      if (result.resultType !== messages.QueryResultType.SUCCESS) {
        const message = result.message || 'Failed to run query';
        void logger.log(message);
--- a/extensions/ql-vscode/src/run-remote-query.ts
+++ b/extensions/ql-vscode/src/run-remote-query.ts
@@ -1,29 +1,53 @@
-import { QuickPickItem, Uri, window } from 'vscode';
+import { CancellationToken, QuickPickItem, Uri, window } from 'vscode';
+import * as path from 'path';
 import * as yaml from 'js-yaml';
 import * as fs from 'fs-extra';
-import { findLanguage, showAndLogErrorMessage, showAndLogInformationMessage, showInformationMessageWithAction } from './helpers';
+import * as tmp from 'tmp-promise';
+import { askForLanguage, findLanguage, getOnDiskWorkspaceFolders, showAndLogErrorMessage, showAndLogInformationMessage, showInformationMessageWithAction } from './helpers';
 import { Credentials } from './authentication';
 import * as cli from './cli';
 import { logger } from './logging';
-import { getRemoteRepositoryLists } from './config';
+import { getRemoteControllerRepo, getRemoteRepositoryLists, setRemoteControllerRepo } from './config';
+import { tmpDir } from './run-queries';
+import { ProgressCallback, UserCancellationException } from './commandRunner';
+import { OctokitResponse } from '@octokit/types/dist-types';
 interface Config {
  repositories: string[];
  ref?: string;
  language?: string;
 }

-// Test "controller" repository and workflow.
-const OWNER = 'dsp-testing';
-const REPO = 'qc-controller';
-
+export interface QlPack {
+  name: string;
+  version: string;
+  dependencies: { [key: string]: string };
+  defaultSuite?: Record<string, unknown>[];
+  defaultSuiteFile?: string;
+}
 interface RepoListQuickPickItem extends QuickPickItem {
  repoList: string[];
 }

+interface QueriesResponse {
+  workflow_run_id: number
+}
+
+/**
+ * This regex matches strings of the form `owner/repo` where:
+ * - `owner` is made up of alphanumeric characters or single hyphens, starting and ending in an alphanumeric character
+ * - `repo` is made up of alphanumeric characters, hyphens, or underscores
+ */
+const REPO_REGEX = /^(?:[a-zA-Z0-9]+-)*[a-zA-Z0-9]+\/[a-zA-Z0-9-_]+$/;
+
+/**
+ * Well-known names for the query pack used by the server.
+ */
+const QUERY_PACK_NAME = 'codeql-remote/query';
+
 /**
 * Gets the repositories to run the query against.
 */
-async function getRepositories(): Promise<string[] | undefined> {
+export async function getRepositories(): Promise<string[] | undefined> {
  const repoLists = getRemoteRepositoryLists();
  if (repoLists && Object.keys(repoLists).length) {
    const quickPickItems = Object.entries(repoLists).map<RepoListQuickPickItem>(([key, value]) => (
@@ -35,7 +59,7 @@ async function getRepositories(): Promise<string[] | undefined> {
    const quickpick = await window.showQuickPick<RepoListQuickPickItem>(
      quickPickItems,
      {
-        placeHolder: 'Select a repository list. You can define repository lists in the `codeQL.remoteRepositoryLists` setting.',
+        placeHolder: 'Select a repository list. You can define repository lists in the `codeQL.remoteQueries.repositoryLists` setting.',
        ignoreFocusOut: true,
      });
    if (quickpick?.repoList.length) {
@@ -47,22 +71,16 @@ async function getRepositories(): Promise<string[] | undefined> {
    }
  } else {
    void logger.log('No repository lists defined. Displaying text input box.');
-    /**
-     * This regex matches strings of the form `owner/repo` where:
-     * - `owner` is made up of alphanumeric characters or single hyphens, starting and ending in an alphanumeric character
-     * - `repo` is made up of alphanumeric characters, hyphens, or underscores
-     */
-    const repoRegex = /^(?:[a-zA-Z0-9]+-)*[a-zA-Z0-9]+\/[a-zA-Z0-9-_]+$/;
    const remoteRepo = await window.showInputBox({
      title: 'Enter a GitHub repository in the format <owner>/<repo> (e.g. github/codeql)',
      placeHolder: '<owner>/<repo>',
-      prompt: 'Tip: you can save frequently used repositories in the `codeql.remoteRepositoryLists` setting',
+      prompt: 'Tip: you can save frequently used repositories in the `codeQL.remoteQueries.repositoryLists` setting',
      ignoreFocusOut: true,
    });
    if (!remoteRepo) {
      void showAndLogErrorMessage('No repositories entered.');
      return;
-    } else if (!repoRegex.test(remoteRepo)) { // Check if user entered invalid input
+    } else if (!REPO_REGEX.test(remoteRepo)) { // Check if user entered invalid input
      void showAndLogErrorMessage('Invalid repository format. Must be in the format <owner>/<repo> (e.g. github/codeql)');
      return;
    }
@@ -71,94 +89,354 @@ async function getRepositories(): Promise<string[] | undefined> {
  }
 }

-export async function runRemoteQuery(cliServer: cli.CodeQLCliServer, credentials: Credentials, uri?: Uri) {
-  if (!uri?.fsPath.endsWith('.ql')) {
-    return;
-  }
+/**
+ * Two possibilities:
+ * 1. There is no qlpack.yml in this directory. Assume this is a lone query and generate a synthetic qlpack for it.
+ * 2. There is a qlpack.yml in this directory. Assume this is a query pack and use the yml to pack the query before uploading it.
+ *
+ * @returns the entire qlpack as a base64 string.
+ */
+async function generateQueryPack(cliServer: cli.CodeQLCliServer, queryFile: string, queryPackDir: string, fallbackLanguage?: string): Promise<{
+  base64Pack: string,
+  language: string
+}> {
+  const originalPackRoot = await findPackRoot(queryFile);
+  const packRelativePath = path.relative(originalPackRoot, queryFile);
+  const targetQueryFileName = path.join(queryPackDir, packRelativePath);

-  const queryFile = uri.fsPath;
-  const query = await fs.readFile(queryFile, 'utf8');
-
-  const repositoriesFile = queryFile.substring(0, queryFile.length - '.ql'.length) + '.repositories';
-  let ref: string | undefined;
  let language: string | undefined;
-  let repositories: string[] | undefined;
+  if (await fs.pathExists(path.join(originalPackRoot, 'qlpack.yml'))) {
+    // don't include ql files. We only want the queryFile to be copied.
+    const toCopy = await cliServer.packPacklist(originalPackRoot, false);

-  // If the user has an explicit `.repositories` file, use that.
-  // Otherwise, prompt user to select repositories from the `codeQL.remoteRepositoryLists` setting.
-  if (await fs.pathExists(repositoriesFile)) {
-    void logger.log(`Found '${repositoriesFile}'. Using information from that file to run ${queryFile}.`);
+    // also copy the lock file (either new name or old name) and the query file itself. These are not included in the packlist.
+    [path.join(originalPackRoot, 'qlpack.lock.yml'), path.join(originalPackRoot, 'codeql-pack.lock.yml'), queryFile]
+      .forEach(absolutePath => {
+        if (absolutePath) {
+          toCopy.push(absolutePath);
+        }
+      });

-    const config = yaml.safeLoad(await fs.readFile(repositoriesFile, 'utf8')) as Config;
+    let copiedCount = 0;
+    await fs.copy(originalPackRoot, queryPackDir, {
+      filter: (file: string) =>
+        // copy file if it is in the packlist, or it is a parent directory of a file in the packlist
+        !!toCopy.find(f => {
+          // Normalized paths ensure that Windows drive letters are capitalized consistently.
+          const normalizedPath = Uri.file(f).fsPath;
+          const matches = normalizedPath === file || normalizedPath.startsWith(file + path.sep);
+          if (matches) {
+            copiedCount++;
+          }
+          return matches;
+        })
+    });
+
+    void logger.log(`Copied ${copiedCount} files to ${queryPackDir}`);
+
+    language = await findLanguage(cliServer, Uri.file(targetQueryFileName));

-    ref = config.ref || 'main';
-    language = config.language || await findLanguage(cliServer, uri);
-    repositories = config.repositories;
  } else {
-    ref = 'main';
-    [language, repositories] = await Promise.all([findLanguage(cliServer, uri), getRepositories()]);
-  }
+    // open popup to ask for language if not already hardcoded
+    language = fallbackLanguage || await askForLanguage(cliServer);

+    // copy only the query file to the query pack directory
+    // and generate a synthetic query pack
+    void logger.log(`Copying ${queryFile} to ${queryPackDir}`);
+    await fs.copy(queryFile, targetQueryFileName);
+    void logger.log('Generating synthetic query pack');
+    const syntheticQueryPack = {
+      name: QUERY_PACK_NAME,
+      version: '0.0.0',
+      dependencies: {
+        [`codeql/${language}-all`]: '*',
+      }
+    };
+    await fs.writeFile(path.join(queryPackDir, 'qlpack.yml'), yaml.safeDump(syntheticQueryPack));
+  }
  if (!language) {
-    return; // No error message needed, since `findLanguage` already displays one.
+    throw new UserCancellationException('Could not determine language.');
  }

-  if (!repositories || repositories.length === 0) {
-    return; // No error message needed, since `getRepositories` already displays one.
-  }
+  await ensureNameAndSuite(queryPackDir, packRelativePath);

-  await runRemoteQueriesApiRequest(credentials, ref, language, repositories, query);
+  const bundlePath = await getPackedBundlePath(queryPackDir);
+  void logger.log(`Compiling and bundling query pack from ${queryPackDir} to ${bundlePath}. (This may take a while.)`);
+  await cliServer.packInstall(queryPackDir);
+  const workspaceFolders = getOnDiskWorkspaceFolders();
+  await cliServer.packBundle(queryPackDir, workspaceFolders, bundlePath, false);
+  const base64Pack = (await fs.readFile(bundlePath)).toString('base64');
+  return {
+    base64Pack,
+    language
+  };
 }

-async function runRemoteQueriesApiRequest(credentials: Credentials, ref: string, language: string, repositories: string[], query: string) {
-  const octokit = await credentials.getOctokit();
-  const token = await credentials.getToken();
+async function findPackRoot(queryFile: string): Promise<string> {
+  // recursively find the directory containing qlpack.yml
+  let dir = path.dirname(queryFile);
+  while (!(await fs.pathExists(path.join(dir, 'qlpack.yml')))) {
+    dir = path.dirname(dir);
+    if (isFileSystemRoot(dir)) {
+      // there is no qlpack.yml in this direcory or any parent directory.
+      // just use the query file's directory as the pack root.
+      return path.dirname(queryFile);
+    }
+  }

+  return dir;
+}
+
+function isFileSystemRoot(dir: string): boolean {
+  const pathObj = path.parse(dir);
+  return pathObj.root === dir && pathObj.base === '';
+}
+
+async function createRemoteQueriesTempDirectory() {
+  const remoteQueryDir = await tmp.dir({ dir: tmpDir.name, unsafeCleanup: true });
+  const queryPackDir = path.join(remoteQueryDir.path, 'query-pack');
+  await fs.mkdirp(queryPackDir);
+  return { remoteQueryDir, queryPackDir };
+}
+
+async function getPackedBundlePath(queryPackDir: string) {
+  return tmp.tmpName({
+    dir: path.dirname(queryPackDir),
+    postfix: 'generated.tgz',
+    prefix: 'qlpack',
+  });
+}
+
+export async function runRemoteQuery(
+  cliServer: cli.CodeQLCliServer,
+  credentials: Credentials,
+  uri: Uri | undefined,
+  dryRun: boolean,
+  progress: ProgressCallback,
+  token: CancellationToken
+): Promise<void | string> {
+  if (!(await cliServer.cliConstraints.supportsRemoteQueries())) {
+    throw new Error(`Remote queries are not supported by this version of CodeQL. Please upgrade to v${cli.CliVersionConstraint.CLI_VERSION_REMOTE_QUERIES
+      } or later.`);
+  }
+
+  const { remoteQueryDir, queryPackDir } = await createRemoteQueriesTempDirectory();
  try {
-    await octokit.request(
-      'POST /repos/:owner/:repo/code-scanning/codeql/queries',
-      {
-        owner: OWNER,
-        repo: REPO,
-        data: {
-          ref: ref,
-          language: language,
-          repositories: repositories,
-          query: query,
-          token: token,
-        }
-      }
-    );
-    void showAndLogInformationMessage(`Successfully scheduled runs. [Click here to see the progress](https://github.com/${OWNER}/${REPO}/actions).`);
+    if (!uri?.fsPath.endsWith('.ql')) {
+      throw new UserCancellationException('Not a CodeQL query file.');
+    }

-  } catch (error) {
-    if (typeof error.message === 'string' && error.message.includes('Some repositories were invalid')) {
-      const invalidRepos = error?.response?.data?.invalid_repos || [];
-      const reposWithoutDbUploads = error?.response?.data?.repos_without_db_uploads || [];
-      void logger.log('Unable to run query on some of the specified repositories');
-      if (invalidRepos.length > 0) {
-        void logger.log(`Invalid repos: ${invalidRepos.join(', ')}`);
-      }
-      if (reposWithoutDbUploads.length > 0) {
-        void logger.log(`Repos without DB uploads: ${reposWithoutDbUploads.join(', ')}`);
-      }
+    progress({
+      maxStep: 5,
+      step: 1,
+      message: 'Determining project list'
+    });

-      if (invalidRepos.length + reposWithoutDbUploads.length === repositories.length) {
-        // Every repo is invalid in some way
-        void showAndLogErrorMessage('Unable to run query on any of the specified repositories.');
+    const queryFile = uri.fsPath;
+    const repositoriesFile = queryFile.substring(0, queryFile.length - '.ql'.length) + '.repositories';
+    let ref: string | undefined;
+    // For the case of single file remote queries, use the language from the config in order to avoid the user having to select it.
+    let fallbackLanguage: string | undefined;
+    let repositories: string[] | undefined;
+
+    progress({
+      maxStep: 5,
+      step: 2,
+      message: 'Determining query target language'
+    });
+
+    // If the user has an explicit `.repositories` file, use that.
+    // Otherwise, prompt user to select repositories from the `codeQL.remoteQueries.repositoryLists` setting.
+    if (await fs.pathExists(repositoriesFile)) {
+      void logger.log(`Found '${repositoriesFile}'. Using information from that file to run ${queryFile}.`);
+
+      const config = yaml.safeLoad(await fs.readFile(repositoriesFile, 'utf8')) as Config;
+
+      ref = config.ref || 'main';
+      fallbackLanguage = config.language;
+      repositories = config.repositories;
+    } else {
+      ref = 'main';
+      repositories = await getRepositories();
+    }
+
+    if (!repositories || repositories.length === 0) {
+      throw new UserCancellationException('No repositories to query.');
+    }
+
+    progress({
+      maxStep: 5,
+      step: 3,
+      message: 'Determining controller repo'
+    });
+
+    // Get the controller repo from the config, if it exists.
+    // If it doesn't exist, prompt the user to enter it, and save that value to the config.
+    let controllerRepo: string | undefined;
+    controllerRepo = getRemoteControllerRepo();
+    if (!controllerRepo || !REPO_REGEX.test(controllerRepo)) {
+      void logger.log(controllerRepo ? 'Invalid controller repository name.' : 'No controller repository defined.');
+      controllerRepo = await window.showInputBox({
+        title: 'Controller repository in which to display progress and results of remote queries',
+        placeHolder: '<owner>/<repo>',
+        prompt: 'Enter the name of a GitHub repository in the format <owner>/<repo>',
+        ignoreFocusOut: true,
+      });
+      if (!controllerRepo) {
+        void showAndLogErrorMessage('No controller repository entered.');
+        return;
+      } else if (!REPO_REGEX.test(controllerRepo)) { // Check if user entered invalid input
+        void showAndLogErrorMessage('Invalid repository format. Must be a valid GitHub repository in the format <owner>/<repo>.');
        return;
      }
+      void logger.log(`Setting the controller repository as: ${controllerRepo}`);
+      await setRemoteControllerRepo(controllerRepo);
+    }

-      const popupMessage = 'Unable to run query on some of the specified repositories. [See logs for more details](command:codeQL.showLogs).';
-      const rerunQuery = await showInformationMessageWithAction(popupMessage, 'Rerun on the valid repositories only');
-      if (rerunQuery) {
-        const validRepositories = repositories.filter(r => !invalidRepos.includes(r) && !reposWithoutDbUploads.includes(r));
-        void logger.log(`Rerunning query on set of valid repositories: ${JSON.stringify(validRepositories)}`);
-        await runRemoteQueriesApiRequest(credentials, ref, language, validRepositories, query);
-      }
+    void logger.log(`Using controller repository: ${controllerRepo}`);
+    const [owner, repo] = controllerRepo.split('/');

+    progress({
+      maxStep: 5,
+      step: 4,
+      message: 'Bundling the query pack'
+    });
+
+    if (token.isCancellationRequested) {
+      throw new UserCancellationException('Cancelled');
+    }
+
+    const { base64Pack, language } = await generateQueryPack(cliServer, queryFile, queryPackDir, fallbackLanguage);
+
+    if (token.isCancellationRequested) {
+      throw new UserCancellationException('Cancelled');
+    }
+
+    progress({
+      maxStep: 5,
+      step: 5,
+      message: 'Sending request'
+    });
+
+    await runRemoteQueriesApiRequest(credentials, ref, language, repositories, owner, repo, base64Pack, dryRun);
+
+    if (dryRun) {
+      return remoteQueryDir.path;
    } else {
-      void showAndLogErrorMessage(error);
+      // don't return the path because it has been deleted
+      return;
+    }
+
+  } finally {
+    if (dryRun) {
+      // If we are in a dry run keep the data around for debugging purposes.
+      void logger.log(`[DRY RUN] Not deleting ${queryPackDir}.`);
+    } else {
+      await remoteQueryDir.cleanup();
    }
  }
 }
+
+async function runRemoteQueriesApiRequest(
+  credentials: Credentials,
+  ref: string,
+  language: string,
+  repositories: string[],
+  owner: string,
+  repo: string,
+  queryPackBase64: string,
+  dryRun = false
+): Promise<void> {
+
+  if (dryRun) {
+    void showAndLogInformationMessage('[DRY RUN] Would have sent request. See extension log for the payload.');
+    void logger.log(JSON.stringify({ ref, language, repositories, owner, repo, queryPackBase64: queryPackBase64.substring(0, 100) + '... ' + queryPackBase64.length + ' bytes' }));
+    return;
+  }
+
+  try {
+    const octokit = await credentials.getOctokit();
+    const response: OctokitResponse<QueriesResponse, number> = await octokit.request(
+      'POST /repos/:owner/:repo/code-scanning/codeql/queries',
+      {
+        owner,
+        repo,
+        data: {
+          ref,
+          language,
+          repositories,
+          query_pack: queryPackBase64,
+        }
+      }
+    );
+    void showAndLogInformationMessage(`Successfully scheduled runs. [Click here to see the progress](https://github.com/${owner}/${repo}/actions/runs/${response.data.workflow_run_id}).`);
+
+  } catch (error) {
+    await attemptRerun(error, credentials, ref, language, repositories, owner, repo, queryPackBase64, dryRun);
+  }
+}
+
+/** Attempts to rerun the query on only the valid repositories */
+export async function attemptRerun(
+  error: any,
+  credentials: Credentials,
+  ref: string,
+  language: string,
+  repositories: string[],
+  owner: string,
+  repo: string,
+  queryPackBase64: string,
+  dryRun = false
+) {
+  if (typeof error.message === 'string' && error.message.includes('Some repositories were invalid')) {
+    const invalidRepos = error?.response?.data?.invalid_repos || [];
+    const reposWithoutDbUploads = error?.response?.data?.repos_without_db_uploads || [];
+    void logger.log('Unable to run query on some of the specified repositories');
+    if (invalidRepos.length > 0) {
+      void logger.log(`Invalid repos: ${invalidRepos.join(', ')}`);
+    }
+    if (reposWithoutDbUploads.length > 0) {
+      void logger.log(`Repos without DB uploads: ${reposWithoutDbUploads.join(', ')}`);
+    }
+
+    if (invalidRepos.length + reposWithoutDbUploads.length === repositories.length) {
+      // Every repo is invalid in some way
+      void showAndLogErrorMessage('Unable to run query on any of the specified repositories.');
+      return;
+    }
+
+    const popupMessage = 'Unable to run query on some of the specified repositories. [See logs for more details](command:codeQL.showLogs).';
+    const rerunQuery = await showInformationMessageWithAction(popupMessage, 'Rerun on the valid repositories only');
+    if (rerunQuery) {
+      const validRepositories = repositories.filter(r => !invalidRepos.includes(r) && !reposWithoutDbUploads.includes(r));
+      void logger.log(`Rerunning query on set of valid repositories: ${JSON.stringify(validRepositories)}`);
+      await runRemoteQueriesApiRequest(credentials, ref, language, validRepositories, owner, repo, queryPackBase64, dryRun);
+    }
+  } else {
+    void showAndLogErrorMessage(error);
+  }
+}
+
+/**
+ * Updates the default suite of the query pack. This is used to ensure
+ * only the specified query is run.
+ *
+ * Also, ensure the query pack name is set to the name expected by the server.
+ *
+ * @param queryPackDir The directory containing the query pack
+ * @param packRelativePath The relative path to the query pack from the root of the query pack
+ */
+async function ensureNameAndSuite(queryPackDir: string, packRelativePath: string): Promise<void> {
+  const packPath = path.join(queryPackDir, 'qlpack.yml');
+  const qlpack = yaml.safeLoad(await fs.readFile(packPath, 'utf8')) as QlPack;
+  delete qlpack.defaultSuiteFile;
+
+  qlpack.name = QUERY_PACK_NAME;
+
+  qlpack.defaultSuite = [{
+    description: 'Query suite for remote query'
+  }, {
+    query: packRelativePath.replace(/\\/g, '/')
+  }];
+  await fs.writeFile(packPath, yaml.safeDump(qlpack));
+}
--- a/extensions/ql-vscode/src/sarif-parser.ts
+++ b/extensions/ql-vscode/src/sarif-parser.ts
@@ -0,0 +1,48 @@
+import * as Sarif from 'sarif';
+import * as fs from 'fs-extra';
+import { parser } from 'stream-json';
+import { pick } from 'stream-json/filters/Pick';
+import Assembler = require('stream-json/Assembler');
+import { chain } from 'stream-chain';
+
+const DUMMY_TOOL : Sarif.Tool = {driver: {name: ''}};
+
+export async function sarifParser(interpretedResultsPath: string) : Promise<Sarif.Log> {
+  try {
+    // Parse the SARIF file into token streams, filtering out only the results array.
+    const p = parser();
+    const pipeline = chain([
+      fs.createReadStream(interpretedResultsPath),
+      p,
+      pick({filter: 'runs.0.results'})
+    ]);
+
+    // Creates JavaScript objects from the token stream
+    const asm = Assembler.connectTo(pipeline);
+
+    // Returns a constructed Log object with the results or an empty array if no results were found.
+    // If the parser fails for any reason, it will reject the promise.
+    return await new Promise((resolve, reject) => {
+      pipeline.on('error', (error) => {
+        reject(error);
+      });
+      
+      asm.on('done', (asm) => {
+
+        const log : Sarif.Log = {
+          version: '2.1.0', 
+          runs: [
+            { 
+              tool: DUMMY_TOOL, 
+              results: asm.current ?? []
+            }
+          ]
+        };
+        
+        resolve(log);
+      });
+    });
+  } catch (err) {
+    throw new Error(`Parsing output of interpretation failed: ${err.stderr || err}`);
+  }
+}
--- a/extensions/ql-vscode/src/view/RawTableValue.tsx
+++ b/extensions/ql-vscode/src/view/RawTableValue.tsx
@@ -9,14 +9,14 @@ interface Props {
 }

 export default function RawTableValue(props: Props): JSX.Element {
-  const v = props.value;
+  const rawValue = props.value;
  if (
-    typeof v === 'string'
-    || typeof v === 'number'
-    || typeof v === 'boolean'
+    typeof rawValue === 'string'
+    || typeof rawValue === 'number'
+    || typeof rawValue === 'boolean'
  ) {
-    return <span>{v.toString()}</span>;
+    return <span>{renderLocation(undefined, rawValue.toString())}</span>;
  }

-  return renderLocation(v.url, v.label, props.databaseUri);
+  return renderLocation(rawValue.url, rawValue.label, props.databaseUri);
 }
--- a/extensions/ql-vscode/src/view/result-table-utils.tsx
+++ b/extensions/ql-vscode/src/view/result-table-utils.tsx
@@ -37,6 +37,9 @@ export const oddRowClassName = 'vscode-codeql__result-table-row--odd';
 export const pathRowClassName = 'vscode-codeql__result-table-row--path';
 export const selectedRowClassName = 'vscode-codeql__result-table-row--selected';

+const CONTROL_CODE = '\u001F'.codePointAt(0)!;
+const CONTROL_LABEL = '\u2400'.codePointAt(0)!;
+
 export function jumpToLocationHandler(
  loc: ResolvableLocationValue,
  databaseUri: string,
@@ -67,24 +70,42 @@ export function openFile(filePath: string): void {
  });
 }

+function convertedNonprintableChars(label: string) {
+  // If the label was empty, use a placeholder instead, so the link is still clickable.
+  if (!label) {
+    return '[empty string]';
+  } else if (label.match(/^\s+$/)) {
+    return `[whitespace: "${label}"]`;
+  } else {
+    /**
+     * If the label contains certain non-printable characters, loop through each
+     * character and replace it with the cooresponding unicode control label.
+    */
+    const convertedLabelArray: any[] = [];
+    for (let i = 0; i < label.length; i++) {
+      const labelCheck = label.codePointAt(i)!;
+      if (labelCheck <= CONTROL_CODE) {
+        convertedLabelArray[i] = String.fromCodePoint(labelCheck + CONTROL_LABEL);
+      } else {
+        convertedLabelArray[i] = label.charAt(i);
+      }
+    }
+    return convertedLabelArray.join('');
+  }
+}
+
 /**
 * Render a location as a link which when clicked displays the original location.
 */
 export function renderLocation(
-  loc: UrlValue | undefined,
-  label: string | undefined,
-  databaseUri: string,
+  loc?: UrlValue,
+  label?: string,
+  databaseUri?: string,
  title?: string,
  callback?: () => void
 ): JSX.Element {

-  // If the label was empty, use a placeholder instead, so the link is still clickable.
-  let displayLabel = label;
-  if (!label) {
-    displayLabel = '[empty string]';
-  } else if (label.match(/^\s+$/)) {
-    displayLabel = `[whitespace: "${label}"]`;
-  }
+  const displayLabel = convertedNonprintableChars(label!);

  if (loc === undefined) {
    return <span>{displayLabel}</span>;
@@ -93,7 +114,7 @@ export function renderLocation(
  }

  const resolvableLoc = tryGetResolvableLocation(loc);
-  if (resolvableLoc !== undefined) {
+  if (databaseUri !== undefined && resolvableLoc !== undefined) {
    return (
      <a href="#"
        className="vscode-codeql__result-table-location-link"
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-no-qlpack/in-pack.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-no-qlpack/in-pack.ql
@@ -0,0 +1,2 @@
+// This file should not be included the remote query pack.
+select 1
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-no-qlpack/lib.qll
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-no-qlpack/lib.qll
@@ -0,0 +1,3 @@
+int number() {
+  result = 1
+}
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-no-qlpack/not-in-pack.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-no-qlpack/not-in-pack.ql
@@ -0,0 +1,4 @@
+import javascript
+import lib
+
+select number()
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/not-in-pack.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/not-in-pack.ql
@@ -0,0 +1,2 @@
+// This file should not be included the remote query pack.
+select 1
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/otherfolder/lib.qll
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/otherfolder/lib.qll
@@ -0,0 +1,3 @@
+int number() {
+  result = 1
+}
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/qlpack.yml
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/qlpack.yml
@@ -0,0 +1,4 @@
+name: github/remote-query-pack
+version: 0.0.0
+dependencies:
+  codeql/javascript-all: '*'
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/subfolder/in-pack.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack-nested/subfolder/in-pack.ql
@@ -0,0 +1,4 @@
+import javascript
+import otherfolder.lib
+
+select number()
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/in-pack.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/in-pack.ql
@@ -0,0 +1,4 @@
+import javascript
+import lib
+
+select number()
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/lib.qll
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/lib.qll
@@ -0,0 +1,3 @@
+int number() {
+  result = 1
+}
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/not-in-pack.ql
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/not-in-pack.ql
@@ -0,0 +1,2 @@
+// This file should not be included the remote query pack.
+select 1
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/qlpack.yml
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/data-remote-qlpack/qlpack.yml
@@ -0,0 +1,4 @@
+name: github/remote-query-pack
+version: 0.0.0
+dependencies:
+  codeql/javascript-all: '*'
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/databases.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/databases.test.ts
@@ -5,6 +5,7 @@ import { expect } from 'chai';
 import { extensions, CancellationToken, Uri, window } from 'vscode';

 import { CodeQLExtensionInterface } from '../../extension';
+import { CodeQLCliServer } from '../../cli';
 import { DatabaseManager } from '../../databases';
 import { promptImportLgtmDatabase, importArchiveDatabase, promptImportInternetDatabase } from '../../databaseFetcher';
 import { ProgressCallback } from '../../commandRunner';
@@ -17,10 +18,11 @@ describe('Databases', function() {
  this.timeout(60000);

  const LGTM_URL = 'https://lgtm.com/projects/g/aeisenberg/angular-bind-notifier/';
-
+  
  let databaseManager: DatabaseManager;
  let sandbox: sinon.SinonSandbox;
  let inputBoxStub: sinon.SinonStub;
+  let cli: CodeQLCliServer;
  let progressCallback: ProgressCallback;

  beforeEach(async () => {
@@ -53,7 +55,7 @@ describe('Databases', function() {
  it('should add a database from a folder', async () => {
    const progressCallback = sandbox.spy() as ProgressCallback;
    const uri = Uri.file(dbLoc);
-    let dbItem = await importArchiveDatabase(uri.toString(true), databaseManager, storagePath, progressCallback, {} as CancellationToken);
+    let dbItem = await importArchiveDatabase(uri.toString(true), databaseManager, storagePath, progressCallback, {} as CancellationToken, cli);
    expect(dbItem).to.be.eq(databaseManager.currentDatabaseItem);
    expect(dbItem).to.be.eq(databaseManager.databaseItems[0]);
    expect(dbItem).not.to.be.undefined;
@@ -64,7 +66,7 @@ describe('Databases', function() {

  it('should add a database from lgtm with only one language', async () => {
    inputBoxStub.resolves(LGTM_URL);
-    let dbItem = await promptImportLgtmDatabase(databaseManager, storagePath, progressCallback, {} as CancellationToken);
+    let dbItem = await promptImportLgtmDatabase(databaseManager, storagePath, progressCallback, {} as CancellationToken, cli);
    expect(dbItem).not.to.be.undefined;
    dbItem = dbItem!;
    expect(dbItem.name).to.eq('aeisenberg_angular-bind-notifier_106179a');
@@ -74,7 +76,7 @@ describe('Databases', function() {
  it('should add a database from a url', async () => {
    inputBoxStub.resolves(DB_URL);

-    let dbItem = await promptImportInternetDatabase(databaseManager, storagePath, progressCallback, {} as CancellationToken);
+    let dbItem = await promptImportInternetDatabase(databaseManager, storagePath, progressCallback, {} as CancellationToken, cli);
    expect(dbItem).not.to.be.undefined;
    dbItem = dbItem!;
    expect(dbItem.name).to.eq('db');
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/index.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/index.ts
@@ -1,6 +1,10 @@
-import 'mocha';
-import 'sinon-chai';
 import { runTestsInDirectory } from '../index-template';
+import 'mocha';
+import * as sinonChai from 'sinon-chai';
+import * as chai from 'chai';
+import * as chaiAsPromised from 'chai-as-promised';
+chai.use(chaiAsPromised);
+chai.use(sinonChai);

 // The simple database used throughout the tests
 export function run(): Promise<void> {
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/queries.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/queries.test.ts
@@ -68,7 +68,8 @@ describe('Queries', function() {
        databaseManager,
        storagePath,
        progress,
-        token
+        token,
+        cli
      );

      if (!maybeDbItem) {
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/run-cli.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/run-cli.test.ts
@@ -14,22 +14,22 @@ import { KeyType } from '../../contextual/keyType';
 * Perform proper integration tests by running the CLI
 */
 describe('Use cli', function() {
-  const supportedLanguages = ['cpp', 'csharp', 'go', 'java', 'javascript', 'python'];
-
  this.timeout(60000);

  let cli: CodeQLCliServer;
+  let supportedLanguages: string[];

  beforeEach(async () => {
    const extension = await extensions.getExtension<CodeQLExtensionInterface | Record<string, never>>('GitHub.vscode-codeql')!.activate();
    if ('cliServer' in extension) {
      cli = extension.cliServer;
+      supportedLanguages = await cli.getSupportedLanguages();
    } else {
      throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
    }
  });

-  if (process.env.CLI_VERSION !== 'nightly') {
+  if (process.env.CLI_VERSION && process.env.CLI_VERSION !== 'nightly') {
    it('should have the correct version of the cli', async () => {
      expect(
        (await cli.getVersion()).toString()
@@ -58,12 +58,11 @@ describe('Use cli', function() {
    }
  });

-  it('should resolve languages', async function() {
+  it('should support the expected languages', async function() {
    skipIfNoCodeQL(this);
-    const languages = await cli.resolveLanguages();
-    for (const expectedLanguage of supportedLanguages) {
-      expect(languages).to.have.property(expectedLanguage).that.is.not.undefined;
-    }
+    // Just check a few examples that definitely are/aren't supported.
+    expect(supportedLanguages).to.include.members(['go', 'javascript', 'python']);
+    expect(supportedLanguages).to.not.include.members(['xml', 'properties']);
  });

  it('should resolve query by language', async function() {
@@ -73,15 +72,15 @@ describe('Use cli', function() {
    expect((Object.keys(queryInfo.byLanguage))[0]).to.eql('javascript');
  });

+  it('should resolve printAST queries for supported languages', async function() {
+    skipIfNoCodeQL(this);
+    supportedLanguages.forEach(async lang => {
+      if (lang === 'go') {
+        // The codeql-go submodule is not available in the integration tests.
+        return;
+      }

-  supportedLanguages.forEach(lang => {
-    if (lang === 'go') {
-      // The codeql-go submodule is not available in the integration tests.
-      return;
-    }
-    it(`should resolve printAST queries for ${lang}`, async function() {
-      skipIfNoCodeQL(this);
-
+      console.log(`resolving printAST queries for ${lang}`);
      const pack = await getQlPackForDbscheme(cli, languageToDbScheme[lang]);
      expect(pack.dbschemePack).to.contain(lang);
      if (pack.dbschemePackIsLibraryPack) {
--- a/extensions/ql-vscode/src/vscode-tests/cli-integration/run-remote-query.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/cli-integration/run-remote-query.test.ts
@@ -0,0 +1,287 @@
+import { assert, expect } from 'chai';
+import * as path from 'path';
+import * as sinon from 'sinon';
+import { CancellationToken, extensions, QuickPickItem, Uri, window } from 'vscode';
+import 'mocha';
+import * as fs from 'fs-extra';
+import * as os from 'os';
+import * as yaml from 'js-yaml';
+
+import { QlPack, runRemoteQuery } from '../../run-remote-query';
+import { Credentials } from '../../authentication';
+import { CliVersionConstraint, CodeQLCliServer } from '../../cli';
+import { CodeQLExtensionInterface } from '../../extension';
+import { setRemoteControllerRepo, setRemoteRepositoryLists } from '../../config';
+import { UserCancellationException } from '../../commandRunner';
+import { lte } from 'semver';
+
+describe('Remote queries', function() {
+  const baseDir = path.join(__dirname, '../../../src/vscode-tests/cli-integration');
+
+  let sandbox: sinon.SinonSandbox;
+
+  // up to 3 minutes per test
+  this.timeout(3 * 60 * 1000);
+
+  let cli: CodeQLCliServer;
+  let credentials: Credentials = {} as unknown as Credentials;
+  let token: CancellationToken;
+  let progress: sinon.SinonSpy;
+  let showQuickPickSpy: sinon.SinonStub;
+
+  // use `function` so we have access to `this`
+  beforeEach(async function() {
+    sandbox = sinon.createSandbox();
+
+    const extension = await extensions.getExtension<CodeQLExtensionInterface | Record<string, never>>('GitHub.vscode-codeql')!.activate();
+    if ('cliServer' in extension) {
+      cli = extension.cliServer;
+    } else {
+      throw new Error('Extension not initialized. Make sure cli is downloaded and installed properly.');
+    }
+
+    if (!(await cli.cliConstraints.supportsRemoteQueries())) {
+      console.log(`Remote queries are not supported on CodeQL CLI v${CliVersionConstraint.CLI_VERSION_REMOTE_QUERIES
+        }. Skipping this test.`);
+      this.skip();
+    }
+    credentials = {} as unknown as Credentials;
+    token = {
+      isCancellationRequested: false
+    } as unknown as CancellationToken;
+
+    progress = sandbox.spy();
+    // Should not have asked for a language
+    showQuickPickSpy = sandbox.stub(window, 'showQuickPick')
+      .onFirstCall().resolves({ repoList: ['github/vscode-codeql'] } as unknown as QuickPickItem)
+      .onSecondCall().resolves('javascript' as unknown as QuickPickItem);
+
+    // always run in the vscode-codeql repo
+    await setRemoteControllerRepo('github/vscode-codeql');
+    await setRemoteRepositoryLists({ 'vscode-codeql': ['github/vscode-codeql'] });
+  });
+
+  afterEach(() => {
+    sandbox.restore();
+  });
+
+  it('should run a remote query that is part of a qlpack', async () => {
+    const fileUri = getFile('data-remote-qlpack/in-pack.ql');
+
+    const queryPackRootDir = (await runRemoteQuery(cli, credentials, fileUri, true, progress, token))!;
+    printDirectoryContents(queryPackRootDir);
+
+    // to retrieve the list of repositories
+    expect(showQuickPickSpy).to.have.been.calledOnce;
+
+    // check a few files that we know should exist and others that we know should not
+
+    // the tarball to deliver to the server
+    expect(fs.readdirSync(queryPackRootDir).find(f => f.startsWith('qlpack-') && f.endsWith('-generated.tgz'))).not.to.be.undefined;
+
+    const queryPackDir = path.join(queryPackRootDir, 'query-pack');
+    printDirectoryContents(queryPackDir);
+
+    expect(fs.existsSync(path.join(queryPackDir, 'in-pack.ql'))).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'lib.qll'))).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'qlpack.yml'))).to.be.true;
+
+    // depending on the cli version, we should have one of these files
+    expect(
+      fs.existsSync(path.join(queryPackDir, 'qlpack.lock.yml')) ||
+      fs.existsSync(path.join(queryPackDir, 'codeql-pack.lock.yml'))
+    ).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'not-in-pack.ql'))).to.be.false;
+
+    // the compiled pack
+    const compiledPackDir = path.join(queryPackDir, '.codeql/pack/github/remote-query-pack/0.0.0/');
+    printDirectoryContents(compiledPackDir);
+
+    expect(fs.existsSync(path.join(compiledPackDir, 'in-pack.ql'))).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'lib.qll'))).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'qlpack.yml'))).to.be.true;
+    // depending on the cli version, we should have one of these files
+    expect(
+      fs.existsSync(path.join(compiledPackDir, 'qlpack.lock.yml')) ||
+      fs.existsSync(path.join(compiledPackDir, 'codeql-pack.lock.yml'))
+    ).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'not-in-pack.ql'))).to.be.false;
+    verifyQlPack(path.join(compiledPackDir, 'qlpack.yml'), 'in-pack.ql', '0.0.0', await pathSerializationBroken());
+
+    // dependencies
+    const libraryDir = path.join(compiledPackDir, '.codeql/libraries/codeql');
+    const packNames = fs.readdirSync(libraryDir).sort();
+    expect(packNames).to.deep.equal(['javascript-all', 'javascript-upgrades']);
+  });
+
+  it('should run a remote query that is not part of a qlpack', async () => {
+    const fileUri = getFile('data-remote-no-qlpack/in-pack.ql');
+
+    const queryPackRootDir = (await runRemoteQuery(cli, credentials, fileUri, true, progress, token))!;
+
+    // to retrieve the list of repositories
+    // and a second time to ask for the language
+    expect(showQuickPickSpy).to.have.been.calledTwice;
+
+    // check a few files that we know should exist and others that we know should not
+
+    // the tarball to deliver to the server
+    printDirectoryContents(queryPackRootDir);
+    expect(fs.readdirSync(queryPackRootDir).find(f => f.startsWith('qlpack-') && f.endsWith('-generated.tgz'))).not.to.be.undefined;
+
+    const queryPackDir = path.join(queryPackRootDir, 'query-pack');
+    printDirectoryContents(queryPackDir);
+
+    expect(fs.existsSync(path.join(queryPackDir, 'in-pack.ql'))).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'qlpack.yml'))).to.be.true;
+    // depending on the cli version, we should have one of these files
+    expect(
+      fs.existsSync(path.join(queryPackDir, 'qlpack.lock.yml')) ||
+      fs.existsSync(path.join(queryPackDir, 'codeql-pack.lock.yml'))
+    ).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'lib.qll'))).to.be.false;
+    expect(fs.existsSync(path.join(queryPackDir, 'not-in-pack.ql'))).to.be.false;
+
+    // the compiled pack
+    const compiledPackDir = path.join(queryPackDir, '.codeql/pack/codeql-remote/query/0.0.0/');
+    printDirectoryContents(compiledPackDir);
+    expect(fs.existsSync(path.join(compiledPackDir, 'in-pack.ql'))).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'qlpack.yml'))).to.be.true;
+    verifyQlPack(path.join(compiledPackDir, 'qlpack.yml'), 'in-pack.ql', '0.0.0', await pathSerializationBroken());
+
+    // depending on the cli version, we should have one of these files
+    expect(
+      fs.existsSync(path.join(compiledPackDir, 'qlpack.lock.yml')) ||
+      fs.existsSync(path.join(compiledPackDir, 'codeql-pack.lock.yml'))
+    ).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'lib.qll'))).to.be.false;
+    expect(fs.existsSync(path.join(compiledPackDir, 'not-in-pack.ql'))).to.be.false;
+    // should have generated a correct qlpack file
+    const qlpackContents: any = yaml.safeLoad(fs.readFileSync(path.join(compiledPackDir, 'qlpack.yml'), 'utf8'));
+    expect(qlpackContents.name).to.equal('codeql-remote/query');
+    expect(qlpackContents.version).to.equal('0.0.0');
+    expect(qlpackContents.dependencies?.['codeql/javascript-all']).to.equal('*');
+
+    // dependencies
+    const libraryDir = path.join(compiledPackDir, '.codeql/libraries/codeql');
+    printDirectoryContents(libraryDir);
+    const packNames = fs.readdirSync(libraryDir).sort();
+    expect(packNames).to.deep.equal(['javascript-all', 'javascript-upgrades']);
+  });
+
+  it('should run a remote query that is nested inside a qlpack', async () => {
+    const fileUri = getFile('data-remote-qlpack-nested/subfolder/in-pack.ql');
+
+    const queryPackRootDir = (await runRemoteQuery(cli, credentials, fileUri, true, progress, token))!;
+
+    // to retrieve the list of repositories
+    expect(showQuickPickSpy).to.have.been.calledOnce;
+
+    // check a few files that we know should exist and others that we know should not
+
+    // the tarball to deliver to the server
+    printDirectoryContents(queryPackRootDir);
+    expect(fs.readdirSync(queryPackRootDir).find(f => f.startsWith('qlpack-') && f.endsWith('-generated.tgz'))).not.to.be.undefined;
+
+    const queryPackDir = path.join(queryPackRootDir, 'query-pack');
+    printDirectoryContents(queryPackDir);
+
+    expect(fs.existsSync(path.join(queryPackDir, 'subfolder/in-pack.ql'))).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'qlpack.yml'))).to.be.true;
+    // depending on the cli version, we should have one of these files
+    expect(
+      fs.existsSync(path.join(queryPackDir, 'qlpack.lock.yml')) ||
+      fs.existsSync(path.join(queryPackDir, 'codeql-pack.lock.yml'))
+    ).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'otherfolder/lib.qll'))).to.be.true;
+    expect(fs.existsSync(path.join(queryPackDir, 'not-in-pack.ql'))).to.be.false;
+
+    // the compiled pack
+    const compiledPackDir = path.join(queryPackDir, '.codeql/pack/github/remote-query-pack/0.0.0/');
+    printDirectoryContents(compiledPackDir);
+    expect(fs.existsSync(path.join(compiledPackDir, 'otherfolder/lib.qll'))).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'subfolder/in-pack.ql'))).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'qlpack.yml'))).to.be.true;
+    verifyQlPack(path.join(compiledPackDir, 'qlpack.yml'), 'subfolder/in-pack.ql', '0.0.0', await pathSerializationBroken());
+
+    // depending on the cli version, we should have one of these files
+    expect(
+      fs.existsSync(path.join(compiledPackDir, 'qlpack.lock.yml')) ||
+      fs.existsSync(path.join(compiledPackDir, 'codeql-pack.lock.yml'))
+    ).to.be.true;
+    expect(fs.existsSync(path.join(compiledPackDir, 'not-in-pack.ql'))).to.be.false;
+    // should have generated a correct qlpack file
+    const qlpackContents: any = yaml.safeLoad(fs.readFileSync(path.join(compiledPackDir, 'qlpack.yml'), 'utf8'));
+    expect(qlpackContents.name).to.equal('codeql-remote/query');
+    expect(qlpackContents.version).to.equal('0.0.0');
+    expect(qlpackContents.dependencies?.['codeql/javascript-all']).to.equal('*');
+
+    // dependencies
+    const libraryDir = path.join(compiledPackDir, '.codeql/libraries/codeql');
+    printDirectoryContents(libraryDir);
+    const packNames = fs.readdirSync(libraryDir).sort();
+    expect(packNames).to.deep.equal(['javascript-all', 'javascript-upgrades']);
+  });
+
+  it('should cancel a run before uploading', async () => {
+    const fileUri = getFile('data-remote-no-qlpack/in-pack.ql');
+
+    const promise = runRemoteQuery(cli, credentials, fileUri, true, progress, token);
+
+    token.isCancellationRequested = true;
+
+    try {
+      await promise;
+      assert.fail('should have thrown');
+    } catch (e) {
+      expect(e).to.be.instanceof(UserCancellationException);
+    }
+  });
+
+  function verifyQlPack(qlpackPath: string, queryPath: string, packVersion: string, pathSerializationBroken: boolean) {
+    const qlPack = yaml.safeLoad(fs.readFileSync(qlpackPath, 'utf8')) as QlPack;
+
+    if (pathSerializationBroken) {
+      // the path serialization is broken, so we force it to be the path in the pack to be same as the query path
+      qlPack.defaultSuite![1].query = queryPath;
+    }
+
+    // don't check the build metadata since it is variable
+    delete (qlPack as any).buildMetadata;
+
+    expect(qlPack).to.deep.equal({
+      name: 'codeql-remote/query',
+      version: packVersion,
+      dependencies: {
+        'codeql/javascript-all': '*',
+      },
+      library: false,
+      defaultSuite: [{
+        description: 'Query suite for remote query'
+      }, {
+        query: queryPath
+      }]
+    });
+  }
+
+  /**
+   * In version 2.7.2 and earlier, relative paths were not serialized correctly inside the qlpack.yml file.
+   * So, ignore part of the test for these versions.
+   *
+   * @returns true if path serialization is broken in this run
+   */
+  async function pathSerializationBroken() {
+    return lte((await cli.getVersion()), '2.7.2') && os.platform() === 'win32';
+  }
+  function getFile(file: string): Uri {
+    return Uri.file(path.join(baseDir, file));
+  }
+
+  function printDirectoryContents(dir: string) {
+    console.log(`DIR ${dir}`);
+    if (!fs.existsSync(dir)) {
+      console.log(`DIR ${dir} does not exist`);
+    }
+    fs.readdirSync(dir).sort().forEach(f => console.log(`  ${f}`));
+  }
+});
--- a/extensions/ql-vscode/src/vscode-tests/ensureCli.ts
+++ b/extensions/ql-vscode/src/vscode-tests/ensureCli.ts
@@ -44,7 +44,7 @@ const _10MB = _1MB * 10;

 // CLI version to test. Hard code the latest as default. And be sure
 // to update the env if it is not otherwise set.
-const CLI_VERSION = process.env.CLI_VERSION || 'v2.6.1';
+const CLI_VERSION = process.env.CLI_VERSION || 'v2.7.2';
 process.env.CLI_VERSION = CLI_VERSION;

 // Base dir where CLIs will be downloaded into
@@ -58,6 +58,15 @@ export async function ensureCli(useCli: boolean) {
      return;
    }

+    if ('CLI_PATH' in process.env) {
+      const executablePath = process.env.CLI_PATH;
+      console.log(`Using existing CLI at ${executablePath}`);
+
+      // The CLI_VERSION env variable is not used when the CLI_PATH is set.
+      delete process.env.CLI_VERSION;
+      return;
+    }
+
    const assetName = DistributionManager.getRequiredAssetName();
    const url = getCliDownloadUrl(assetName);
    const unzipDir = getCliUnzipDir();
--- a/extensions/ql-vscode/src/vscode-tests/minimal-workspace/databases.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/minimal-workspace/databases.test.ts
@@ -12,7 +12,8 @@ import {
  DatabaseManager,
  DatabaseItemImpl,
  DatabaseContents,
-  FullDatabaseOptions
+  FullDatabaseOptions,
+  findSourceArchive
 } from '../../databases';
 import { Logger } from '../../logging';
 import { QueryServerClient } from '../../queryserver-client';
@@ -179,7 +180,7 @@ describe('databases', () => {
      expect(spy).to.have.been.calledWith(mockEvent);
    });

-    it('should add a database item source archive', async function () {
+    it('should add a database item source archive', async function() {
      const mockDbItem = createMockDB();
      mockDbItem.name = 'xxx';
      await (databaseManager as any).addDatabaseSourceArchiveFolder(mockDbItem);
@@ -478,6 +479,49 @@ describe('databases', () => {
      const db = createMockDB(sourceLocationUri(), Uri.file('/path/to/dir/dir.testproj'));
      expect(await db.isAffectedByTest('/path/to/test.ql')).to.false;
    });
+
+  });
+
+  describe.only('findSourceArchive', function() {
+    // not sure why, but some of these tests take more than two second to run.
+    this.timeout(5000);
+
+    ['src', 'output/src_archive'].forEach(name => {
+      it(`should find source folder in ${name}`, async () => {
+        const uri = Uri.file(path.join(dir.name, name));
+        fs.createFileSync(path.join(uri.fsPath, 'hucairz.txt'));
+        const srcUri = await findSourceArchive(dir.name);
+        expect(srcUri!.fsPath).to.eq(uri.fsPath);
+      });
+
+      it(`should find source archive in ${name}.zip`, async () => {
+        const uri = Uri.file(path.join(dir.name, name + '.zip'));
+        fs.createFileSync(uri.fsPath);
+        const srcUri = await findSourceArchive(dir.name);
+        expect(srcUri!.fsPath).to.eq(uri.fsPath);
+      });
+
+      it(`should prioritize ${name}.zip over ${name}`, async () => {
+        const uri = Uri.file(path.join(dir.name, name + '.zip'));
+        fs.createFileSync(uri.fsPath);
+
+        const uriFolder = Uri.file(path.join(dir.name, name));
+        fs.createFileSync(path.join(uriFolder.fsPath, 'hucairz.txt'));
+
+        const srcUri = await findSourceArchive(dir.name);
+        expect(srcUri!.fsPath).to.eq(uri.fsPath);
+      });
+    });
+
+    it('should prioritize src over output/src_archive', async () => {
+      const uriSrc = Uri.file(path.join(dir.name, 'src.zip'));
+      fs.createFileSync(uriSrc.fsPath);
+      const uriSrcArchive = Uri.file(path.join(dir.name, 'src.zip'));
+      fs.createFileSync(uriSrcArchive.fsPath);
+
+      const resultUri = await findSourceArchive(dir.name);
+      expect(resultUri!.fsPath).to.eq(uriSrc.fsPath);
+    });
  });

  function createMockDB(
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/data/sarif/emptyResultsSarif.sarif
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/data/sarif/emptyResultsSarif.sarif
@@ -0,0 +1,34 @@
+{
+  "version": "2.1.0",
+  "$schema": "http://json.schemastore.org/sarif-2.1.0-rtm.4",
+  "runs": [
+    {
+      "tool": {
+        "driver": {
+          "name": "ESLint",
+          "informationUri": "https://eslint.org",
+          "rules": [
+            {
+              "id": "no-unused-vars",
+              "shortDescription": {
+                "text": "disallow unused variables"
+              },
+              "helpUri": "https://eslint.org/docs/rules/no-unused-vars",
+              "properties": {
+                "category": "Variables"
+              }
+            }
+          ]
+        }
+      },
+      "artifacts": [
+        {
+          "location": {
+            "uri": "file:///C:/dev/sarif/sarif-tutorials/samples/Introduction/simple-example.js"
+          }
+        }
+      ],
+      "results": []
+    }
+  ]
+}
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/data/sarif/invalidSarif.sarif
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/data/sarif/invalidSarif.sarif
@@ -0,0 +1,33 @@
+{
+  "version": "2.1.0",
+  "$schema": "http://json.schemastore.org/sarif-2.1.0-rtm.4",
+  "runs": [
+    {
+      "tool": {
+        "driver": {
+          "name": "ESLint",
+          "informationUri": "https://eslint.org",
+          "rules": [
+            {
+              "id": "no-unused-vars",
+              "shortDescription": {
+                "text": "disallow unused variables"
+              },
+              "helpUri": "https://eslint.org/docs/rules/no-unused-vars",
+              "properties": {
+                "category": "Variables"
+              }
+            }
+          ]
+        }
+      },
+      "artifacts": [
+        {
+          "location": {
+            "uri": "file:///C:/dev/sarif/sarif-tutorials/samples/Introduction/simple-example.js"
+          }
+        }
+      ]
+    }
+  ]
+}
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/data/sarif/validSarif.sarif
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/data/sarif/validSarif.sarif
@@ -0,0 +1,57 @@
+{
+  "version": "2.1.0",
+  "$schema": "http://json.schemastore.org/sarif-2.1.0-rtm.4",
+  "runs": [
+    {
+      "tool": {
+        "driver": {
+          "name": "ESLint",
+          "informationUri": "https://eslint.org",
+          "rules": [
+            {
+              "id": "no-unused-vars",
+              "shortDescription": {
+                "text": "disallow unused variables"
+              },
+              "helpUri": "https://eslint.org/docs/rules/no-unused-vars",
+              "properties": {
+                "category": "Variables"
+              }
+            }
+          ]
+        }
+      },
+      "artifacts": [
+        {
+          "location": {
+            "uri": "file:///C:/dev/sarif/sarif-tutorials/samples/Introduction/simple-example.js"
+          }
+        }
+      ],
+      "results": [
+        {
+          "level": "error",
+          "message": {
+            "text": "'x' is assigned a value but never used."
+          },
+          "locations": [
+            {
+              "physicalLocation": {
+                "artifactLocation": {
+                  "uri": "file:///C:/dev/sarif/sarif-tutorials/samples/Introduction/simple-example.js",
+                  "index": 0
+                },
+                "region": {
+                  "startLine": 1,
+                  "startColumn": 5
+                }
+              }
+            }
+          ],
+          "ruleId": "no-unused-vars",
+          "ruleIndex": 0
+        }
+      ]
+    }
+  ]
+}
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/databaseFetcher.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/databaseFetcher.test.ts
@@ -13,19 +13,23 @@ import {
  looksLikeLgtmUrl,
  findDirWithFile,
 } from '../../databaseFetcher';
+import { ProgressCallback } from '../../commandRunner';
 chai.use(chaiAsPromised);
 const expect = chai.expect;

-describe('databaseFetcher', function() {
+describe('databaseFetcher', function () {
  // These tests make API calls and may need extra time to complete.
  this.timeout(10000);

  describe('convertToDatabaseUrl', () => {
    let sandbox: sinon.SinonSandbox;
    let quickPickSpy: sinon.SinonStub;
+    let progressSpy: ProgressCallback;
+
    beforeEach(() => {
      sandbox = sinon.createSandbox();
      quickPickSpy = sandbox.stub(window, 'showQuickPick');
+      progressSpy = sandbox.spy();
    });

    afterEach(() => {
@@ -35,7 +39,7 @@ describe('databaseFetcher', function() {
    it('should convert a project url to a database url', async () => {
      quickPickSpy.resolves('javascript');
      const lgtmUrl = 'https://lgtm.com/projects/g/github/codeql';
-      const dbUrl = await convertToDatabaseUrl(lgtmUrl);
+      const dbUrl = await convertToDatabaseUrl(lgtmUrl, progressSpy);

      expect(dbUrl).to.equal(
        'https://lgtm.com/api/v1.0/snapshots/1506465042581/javascript'
@@ -48,28 +52,31 @@ describe('databaseFetcher', function() {
      quickPickSpy.resolves('python');
      const lgtmUrl =
        'https://lgtm.com/projects/g/github/codeql/subpage/subpage2?query=xxx';
-      const dbUrl = await convertToDatabaseUrl(lgtmUrl);
+      const dbUrl = await convertToDatabaseUrl(lgtmUrl, progressSpy);

      expect(dbUrl).to.equal(
        'https://lgtm.com/api/v1.0/snapshots/1506465042581/python'
      );
+      expect(progressSpy).to.have.been.calledOnce;
    });

    it('should convert a raw slug to a database url with extra path segments', async () => {
      quickPickSpy.resolves('python');
      const lgtmUrl =
        'g/github/codeql';
-      const dbUrl = await convertToDatabaseUrl(lgtmUrl);
+      const dbUrl = await convertToDatabaseUrl(lgtmUrl, progressSpy);

      expect(dbUrl).to.equal(
        'https://lgtm.com/api/v1.0/snapshots/1506465042581/python'
      );
+      expect(progressSpy).to.have.been.calledOnce;
    });

    it('should fail on a nonexistent project', async () => {
      quickPickSpy.resolves('javascript');
      const lgtmUrl = 'https://lgtm.com/projects/g/github/hucairz';
-      await expect(convertToDatabaseUrl(lgtmUrl)).to.rejectedWith(/Invalid LGTM URL/);
+      await expect(convertToDatabaseUrl(lgtmUrl, progressSpy)).to.rejectedWith(/Invalid LGTM URL/);
+      expect(progressSpy).to.have.callCount(0);
    });
  });

--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/run-remote-query.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/run-remote-query.test.ts
@@ -0,0 +1,205 @@
+import 'vscode-test';
+import 'mocha';
+import * as chaiAsPromised from 'chai-as-promised';
+import * as sinon from 'sinon';
+import * as chai from 'chai';
+import { window } from 'vscode';
+import * as pq from 'proxyquire';
+
+const proxyquire = pq.noPreserveCache();
+chai.use(chaiAsPromised);
+const expect = chai.expect;
+
+describe('run-remote-query', function() {
+
+  describe('getRepositories', () => {
+    let sandbox: sinon.SinonSandbox;
+    let quickPickSpy: sinon.SinonStub;
+    let showInputBoxSpy: sinon.SinonStub;
+    let getRemoteRepositoryListsSpy: sinon.SinonStub;
+    let showAndLogErrorMessageSpy: sinon.SinonStub;
+    let mod: any;
+    beforeEach(() => {
+      sandbox = sinon.createSandbox();
+      quickPickSpy = sandbox.stub(window, 'showQuickPick');
+      showInputBoxSpy = sandbox.stub(window, 'showInputBox');
+      getRemoteRepositoryListsSpy = sandbox.stub();
+      showAndLogErrorMessageSpy = sandbox.stub();
+      mod = proxyquire('../../run-remote-query', {
+        './config': {
+          getRemoteRepositoryLists: getRemoteRepositoryListsSpy
+        },
+        './helpers': {
+          showAndLogErrorMessage: showAndLogErrorMessageSpy
+        },
+      });
+    });
+
+    afterEach(() => {
+      sandbox.restore();
+    });
+
+    it('should run on a repo list that you chose from your pre-defined config', async () => {
+      // fake return values
+      quickPickSpy.resolves(
+        { repoList: ['foo/bar', 'foo/baz'] }
+      );
+      getRemoteRepositoryListsSpy.returns(
+        {
+          'list1': ['foo/bar', 'foo/baz'],
+          'list2': [],
+        }
+      );
+
+      // make the function call
+      const repoList = await mod.getRepositories();
+
+      // Check that the return value is correct
+      expect(repoList).to.deep.eq(
+        ['foo/bar', 'foo/baz']
+      );
+    });
+
+    // Test the regex in various "good" cases
+    const goodRepos = [
+      'owner/repo',
+      'owner-with-hyphens/repo-with-hyphens_and_underscores',
+      'ownerWithNumbers58/repoWithNumbers37'
+    ];
+    goodRepos.forEach(repo => {
+      it(`should run on a valid repo that you enter in the text box: ${repo}`, async () => {
+        // fake return values
+        getRemoteRepositoryListsSpy.returns({}); // no pre-defined repo lists
+        showInputBoxSpy.resolves(repo);
+
+        // make the function call
+        const repoList = await mod.getRepositories();
+
+        // Check that the return value is correct
+        expect(repoList).to.deep.equal(
+          [repo]
+        );
+      });
+    });
+
+    // Test the regex in various "bad" cases
+    const badRepos = [
+      'invalid_owner/repo',
+      'owner/repo+some&invalid&stuff',
+      'owner-with-no-repo/',
+      '/repo-with-no-owner'
+    ];
+    badRepos.forEach(repo => {
+      it(`should show an error message if you enter an invalid repo in the text box: ${repo}`, async () => {
+        // fake return values
+        getRemoteRepositoryListsSpy.returns({}); // no pre-defined repo lists
+        showInputBoxSpy.resolves(repo);
+
+        // make the function call
+        await mod.getRepositories();
+
+        // check that we get the right error message
+        expect(showAndLogErrorMessageSpy.firstCall.args[0]).to.contain('Invalid repository format');
+      });
+    });
+
+  });
+
+  describe('attemptRerun', () => {
+    let sandbox: sinon.SinonSandbox;
+    let showAndLogErrorMessageSpy: sinon.SinonStub;
+    let showInformationMessageWithActionSpy: sinon.SinonStub;
+    let mockRequest: sinon.SinonStub;
+    let logSpy: sinon.SinonStub;
+    let mod: any;
+
+    const error = {
+      message: 'Unable to run query on the specified repositories. Some repositories were invalid or don\'t have database uploads enabled.',
+      response: {
+        data: {
+          invalid_repos: ['abc/def', 'ghi/jkl'],
+          repos_without_db_uploads: ['mno/pqr', 'stu/vwx']
+        }
+      }
+    };
+    const ref = 'main';
+    const language = 'javascript';
+    const credentials = getMockCredentials(0);
+    const query = 'select 1';
+    const owner = 'owner';
+    const repo = 'repo';
+
+    beforeEach(() => {
+      sandbox = sinon.createSandbox();
+      logSpy = sandbox.stub();
+      showAndLogErrorMessageSpy = sandbox.stub();
+      showInformationMessageWithActionSpy = sandbox.stub();
+      mod = proxyquire('../../run-remote-query', {
+        './helpers': {
+          showAndLogErrorMessage: showAndLogErrorMessageSpy,
+          showInformationMessageWithAction: showInformationMessageWithActionSpy
+        },
+        './logging': {
+          'logger': {
+            log: logSpy
+          }
+        },
+      });
+    });
+    afterEach(() => {
+      sandbox.restore();
+    });
+
+    it('should return and log error if it can\'t run on any repos', async () => {
+      const repositories = ['abc/def', 'ghi/jkl', 'mno/pqr', 'stu/vwx'];
+
+      // make the function call
+      await mod.attemptRerun(error, credentials, ref, language, repositories, query, owner, repo);
+
+      // check logging output
+      expect(logSpy.firstCall.args[0]).to.contain('Unable to run query');
+      expect(logSpy.secondCall.args[0]).to.contain('Invalid repos: abc/def, ghi/jkl');
+      expect(logSpy.thirdCall.args[0]).to.contain('Repos without DB uploads: mno/pqr, stu/vwx');
+      expect(showAndLogErrorMessageSpy.firstCall.args[0]).to.contain('Unable to run query on any');
+    });
+
+    it('should list invalid repos and repos without DB uploads, and rerun on valid ones', async () => {
+      const repositories = ['foo/bar', 'abc/def', 'ghi/jkl', 'mno/pqr', 'foo/baz'];
+
+      // fake return values
+      showInformationMessageWithActionSpy.resolves(true);
+
+      // make the function call
+      await mod.attemptRerun(error, credentials, ref, language, repositories, query, owner, repo);
+
+      // check logging output
+      expect(logSpy.firstCall.args[0]).to.contain('Unable to run query');
+      expect(logSpy.secondCall.args[0]).to.contain('Invalid repos: abc/def, ghi/jkl');
+      expect(logSpy.thirdCall.args[0]).to.contain('Repos without DB uploads: mno/pqr');
+
+      // check that the correct information message is displayed
+      expect(showInformationMessageWithActionSpy.firstCall.args[0]).to.contain('Unable to run query on some');
+      expect(showInformationMessageWithActionSpy.firstCall.args[1]).to.contain('Rerun');
+
+      // check that API request is made again, with only valid repos
+      expect(logSpy.lastCall.args[0]).to.contain('valid repositories: ["foo/bar","foo/baz"]');
+      // test a few values in the octokit request
+      expect(mockRequest.firstCall.args[1].data.language).to.eq('javascript');
+      expect(mockRequest.firstCall.args[1].data.repositories).to.deep.eq(['foo/bar', 'foo/baz']);
+
+    });
+
+    function getMockCredentials(response: any) {
+      mockRequest = sinon.stub().resolves(response);
+      return {
+        getOctokit: () => ({
+          request: mockRequest
+        })
+      };
+    }
+  });
+
+  describe('runRemoteQuery', () => {
+    // TODO
+  });
+});
--- a/extensions/ql-vscode/src/vscode-tests/no-workspace/sarifParser.test.ts
+++ b/extensions/ql-vscode/src/vscode-tests/no-workspace/sarifParser.test.ts
@@ -0,0 +1,25 @@
+import * as path from 'path';
+import * as chai from 'chai';
+import * as chaiAsPromised from 'chai-as-promised';
+
+import { sarifParser } from '../../sarif-parser';
+
+chai.use(chaiAsPromised);
+const expect = chai.expect;
+
+describe.only('sarif parser', function() {
+  const sarifDir = path.join(__dirname, 'data/sarif');
+  it('should parse a valid SARIF file', async () => {
+    const result = await sarifParser(path.join(sarifDir, 'validSarif.sarif'));
+    expect(result.version).to.exist;
+    expect(result.runs).to.exist;
+    expect(result.runs[0].tool).to.exist;
+    expect(result.runs[0].tool.driver).to.exist;
+    expect(result.runs.length).to.be.at.least(1);
+  });
+
+  it('should return an empty array if there are no results', async () => {
+    const result = await sarifParser(path.join(sarifDir, 'emptyResultsSarif.sarif'));
+    expect(result.runs[0].results).to.be.empty;
+  });
+});
--- a/extensions/ql-vscode/test/pure-tests/sarif-utils.test.ts
+++ b/extensions/ql-vscode/test/pure-tests/sarif-utils.test.ts
@@ -6,11 +6,12 @@ import {
  getPathRelativeToSourceLocationPrefix,
  parseSarifLocation,
  parseSarifPlainTextMessage,
-  unescapeSarifText
+  unescapeSarifText,
 } from '../../src/pure/sarif-utils';


 describe('parsing sarif', () => {
+
  it('should be able to parse a simple message from the spec', async function() {
    const message = 'Tainted data was used. The data came from [here](3).';
    const results = parseSarifPlainTextMessage(message);
--- a/package-lock.json
+++ b/package-lock.json
@@ -0,0 +1,6 @@
+{
+  "name": "vscode-codeql",
+  "lockfileVersion": 2,
+  "requires": true,
+  "packages": {}
+}
Author	SHA1	Message	Date
shati-patel	ff751cc877	v1.5.8 Some checks failed Release / Release (push) Has been cancelled Details Release / Publish to VS Code Marketplace (push) Has been cancelled Details Release / Publish to Open VSX Registry (push) Has been cancelled Details	2021-12-02 11:26:36 +00:00
Robin Neatherway	d7ba941803	Merge pull request #1009 from github/aeisenberg/remote-nested-queries Remote queries: Handle nested queries	2021-12-01 19:24:10 +00:00
Andrew Eisenberg	e58201e24b	Ensure server uses a well-known query pack name	2021-12-01 10:53:51 -08:00
Henry Mercer	81e60286f2	Require canary for loading models from packs	2021-12-01 09:40:06 +00:00
Henry Mercer	8e156d69d7	Apply suggestions from code review Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>	2021-12-01 09:40:06 +00:00
Henry Mercer	dfcaa27235	Update lockfile	2021-12-01 09:40:06 +00:00
Henry Mercer	ed0553c6b6	Gate loading ML models behind a hidden setting	2021-12-01 09:40:06 +00:00
Henry Mercer	84ecbfc7a1	Resolve ML models and pass them to the queryserver	2021-12-01 09:40:06 +00:00
Andrew Eisenberg	e13349ceb0	Update changelog	2021-11-29 11:16:49 -08:00
Andrew Eisenberg	a1bcb7519f	Ensure src.zip is prioritized over src folder Fixes a bug where legacy databases with both unzipped and zipped sources were incorrectly being loaded with the src folder.	2021-11-29 11:16:49 -08:00
Andrew Eisenberg	b481441052	Emit more relevant error message when failing to add source folder (#1021 ) * Emit more relevant error message when failing to add source folder Fixes #1020 * Update changelog * Clarify changelog and error message Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com> Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>	2021-11-29 16:28:07 +00:00
github-actions[bot]	6a1d1a492e	Bump version to v1.5.8 (#1017 ) Co-authored-by: aeisenberg <aeisenberg@users.noreply.github.com>	2021-11-23 21:13:41 +00:00
Andrew Eisenberg	1dcd9c495c	Prepare for the 1.5.7 release (#1016 ) Some checks failed Release / Release (push) Has been cancelled Details Release / Publish to VS Code Marketplace (push) Has been cancelled Details Release / Publish to Open VSX Registry (push) Has been cancelled Details	2021-11-23 12:52:40 -08:00
Andrew Eisenberg	a9b9502dbd	Fix failing tests on windows 1. Acknowledge that the CLI has a bug for path serialization on <=2.7.2. Avoid testing the query path on that version. 2. Fix calculation of root path on windows.	2021-11-22 12:53:52 -08:00
Andrew Eisenberg	16c0bea799	Merge pull request #1004 from marcnjaramillo/fix-large-sarif-handling Fix large sarif handling	2021-11-22 12:24:17 -08:00
marcnjaramillo	ad81127267	Move test files into data directory	2021-11-22 11:49:40 -08:00
Andrew Eisenberg	30d01cb0e0	Merge pull request #1007 from github/aeisenberg/sorted-result-sets Ensure all result set names are loaded	2021-11-22 09:08:55 -08:00
Shati Patel	2584971a07	Update CLI version in tests	2021-11-22 14:02:09 +00:00
marcnjaramillo	9d9f48bcf8	Fix tests for sarif parser	2021-11-19 20:43:22 -08:00
marcnjaramillo	0bb1501e72	Move sarif parser and tests, build completing	2021-11-19 17:21:42 -08:00
marcnjaramillo	d53abd815d	Make suggested changes, build currently failing	2021-11-19 16:01:18 -08:00
Andrew Eisenberg	d9c5ecf462	Fix failing test and remove changelog note	2021-11-19 14:32:38 -08:00
Shati Patel	51ed2cd480	Delete outdated issue template I don't think we need this issue template any more... The release process is in https://github.com/github/vscode-codeql/blob/main/CONTRIBUTING.md#releasing-write-access-required.	2021-11-19 17:47:23 +00:00
Andrew Eisenberg	4c83805030	Update changelog	2021-11-18 18:13:29 -08:00
marcnjaramillo	c3eca5b1b7	Update test for valid SARIF file	2021-11-18 16:05:31 -08:00
Andrew Eisenberg	742bca1cf5	Remote queries: Handle nested queries This change allows remote queries to run a query from a directory that is not in the root of the qlpack. The change is the following: 1. walk up the directory hierarchy to check for a non-local qlpack.yml 2. Copy over the files as before, but keep track of the relative location of the query compared to the location of the qlpack.yml. 3. Change the defaultSuite of the qlpack.yml so that _only_ this query is run as part of the default query. Also, this adds a new integration test to ensure the nested query is packaged appropriately.	2021-11-18 15:27:29 -08:00
Andrew Eisenberg	5ab55bb5a5	Merge branch 'main' into aeisenberg/sorted-result-sets	2021-11-18 10:49:44 -08:00
Musab Guma'a	3743895b66	Add "Preview Query Help" command	2021-11-18 16:10:26 +00:00
marcnjaramillo	ca5e5e23e6	Finish tests	2021-11-17 16:37:56 -08:00
marcnjaramillo	a666619289	Remove error handling for now	2021-11-17 16:37:56 -08:00
marcnjaramillo	63129236d0	Work on tests for new behavior	2021-11-17 16:37:56 -08:00
marcnjaramillo	4374f409a8	Add changelog entry and add missing dependencies	2021-11-17 16:37:37 -08:00
marcnjaramillo	c49aa8e05e	Fix issue with large SARIF files crashing view Authored by: Marc Jaramillo marcnjaramillo@github.com Authored by: Musab Guma'a mgsium@github.com	2021-11-17 16:33:49 -08:00
Andrew Eisenberg	c590e2f36c	Update package lock (#1003 ) * Add leniency in how positions are handled Previously, positions with end column of 0 were rejected by the extension. CodeQL positions are supposed to be 1-based, but the CLI does handle 0-based and negative positions by using character offsets from the current line start. Instead of rejecting these kinds of positions, the extension should handle them as gracefully as possible. Fixes #999 * Update package lock	2021-11-17 16:24:59 -08:00
Andrew Eisenberg	03d4aca639	Add leniency in how positions are handled (#1002 ) * Add leniency in how positions are handled Previously, positions with end column of 0 were rejected by the extension. CodeQL positions are supposed to be 1-based, but the CLI does handle 0-based and negative positions by using character offsets from the current line start. Instead of rejecting these kinds of positions, the extension should handle them as gracefully as possible. Fixes #999 * Add changelog entry	2021-11-17 16:24:48 -08:00
Andrew Eisenberg	01f24523ac	Update changelog	2021-11-17 09:39:49 -08:00
Andrew Eisenberg	98312a72a7	Ensure all result set names are loaded When the extension loads a sorted result set, it takes a shortcut and avoids loads a file with only the bqrs results for that sorted table. However, it does not load the results for any other table. This causes result set names to go away. This change ensures that if we are loading a sorted table, we also load the result set names for all other tables in that query. Fixes #1005.	2021-11-17 09:37:42 -08:00
Shati Patel	d579cd6541	Update CLI version Version 2.7.1 was released this week	2021-11-17 13:30:34 +00:00
shati-patel	38e5d8babc	Attempt to fix tests	2021-11-12 16:03:17 +00:00
shati-patel	c1fceab8d9	Re-use `dbSchemeToLanguage`	2021-11-12 16:03:17 +00:00
shati-patel	ae555969b5	Tidy up language dropdown	2021-11-12 16:03:17 +00:00
Andrew Eisenberg	3e0ea1ba77	Merge pull request #997 from github/aeisenberg/cli-version-test Avoid testing for the correct CLI_VERSION if CLI_PATH is set	2021-11-08 09:37:18 -08:00
Andrew Eisenberg	ce1ebd2218	Avoid testing for the correct CLI_VERSION if CLI_PATH is set	2021-11-08 09:26:25 -08:00
Andrew Eisenberg	6215c2763e	Merge pull request #996 from rneatherway/rneatherway/use-workflow-run-id Start linking to the exact workflow run	2021-11-08 08:20:58 -08:00
Robin Neatherway	07437000ce	Start linking to the exact workflow run Now that the queries endpoint returns the id we can link straight there. We'll make more changes to the progress tracking, but I think it's worth making this minimal change now.	2021-11-08 15:19:29 +00:00
Andrew Eisenberg	0ef635bc68	Use the correct environment variable in integration tests (#994 ) Also, fix documentation and launch configs to specify the correct and complete set of environment variables we should be using.	2021-11-03 23:04:14 +00:00
Andrew Eisenberg	e9574d33a9	Merge pull request #985 from github/qc-packs Remote Queries: Create packs for remote queries	2021-11-03 08:49:08 -07:00
Musab Guma'a	848869e3f4	Make "Open Referenced File" command functional on open .qlref	2021-11-02 11:19:39 +00:00
Andrew Eisenberg	4a65b6a8b2	Ensure anonymous and scope-less packs can be used as remote queries When we generate the synthetic pack, just ensure that there is a valid name.	2021-11-01 15:18:23 -07:00
Andrew Eisenberg	28c76bece0	Change to 0.0.0 as synthetic version	2021-11-01 14:45:55 -07:00
shati-patel	56faf36edf	Fix Windows path issue	2021-10-29 17:56:23 +01:00
Andrew Eisenberg	440044d2aa	Add more debugging logic Help understand why tests are failing.	2021-10-29 09:18:34 -07:00
Andrew Eisenberg	48468ff354	Allow custom setting of codeql cli for integration tessts	2021-10-29 08:29:45 -07:00
shati-patel	31dc11ed73	Fix recommended extensions	2021-10-29 14:53:45 +01:00
Andrew Eisenberg	903f5db707	Avoid running remote queries on v2.6.3 cli or earlier Also: - Fix the count of copied files - A few typos - Ensure the correct settings are applied for remote queries before running tests.	2021-10-28 16:08:43 -07:00
shati-patel	8317f39459	Update CLI version in `ensureCli.ts`	2021-10-28 15:25:22 +01:00
Andrew Eisenberg	42051f1620	Remote Queries: Create packs for remote queries This is still a bit rough, but handles two cases: 1. There is a qlpack.yml or codeql-pack.yml file in the same directory as the query to run remotely. In this case, run `codeql pack packlist` to determine what files to include (and also always include the lock file and the query itself. Copy to a temp folder and run `pack install`, then `pack bundle`. Finally upload. 2. There is no qlpack in the current directory. Just copy the single file to the temp folder and generate a synthetic qlpack before installing, bundling and uploading. Two cases that are not handled: 1. The query file is part of a workspace. Peer dependencies will not be found. 2. The query file and its qlpack file are not in the same directory. These should be possible to handle later. Also, need to create some unit and integration tests for this.	2021-10-27 21:09:34 -07:00
Andrew Eisenberg	9b90579160	Merge pull request #984 from github/aeisenberg/add270 Add v2.7.0 CLI run tests against	2021-10-27 15:30:23 -07:00
Andrew Eisenberg	541367122e	Add v2.7.0 CLI run tests against	2021-10-27 15:23:06 -07:00
Andrew Eisenberg	0a0500a60d	Merge pull request #979 from mgsium/windows-paths Fix the "CodeQL: Open Referenced File" command for windows paths	2021-10-27 09:00:23 -07:00
Musab Guma'a	746086b761	Fix "Open Referenced File" command for windows paths	2021-10-27 16:54:21 +01:00
Andrew Eisenberg	412d96409e	Merge pull request #982 from github/aeisenberg/debug-cli-server Add support for debugging the CLI server	2021-10-26 09:59:35 -07:00
Andrew Eisenberg	93e15b43a3	Remove hunks Accidentally included from a different PR.	2021-10-26 09:42:32 -07:00
Andrew Eisenberg	dbc8198daa	Add support for debugging the CLI server	2021-10-24 16:39:08 -07:00
Andrew Eisenberg	b3a51d7afd	Merge pull request #978 from mgsium/case-insensitive-slugs Case insensitive fallback check for GitHub repositories	2021-10-20 13:50:48 -07:00
Musab Guma'a	3d24328402	Update CHANGELOG.md	2021-10-20 20:46:22 +01:00
Musab Guma'a	1014c4bdda	Added case-insensitive fallback check for GitHub repos when downloading an LGTM project.	2021-10-20 19:35:41 +01:00
shati-patel	b2a6263431	Send a query pack	2021-10-20 15:06:33 +01:00
Andrew Eisenberg	20cdca77a3	Merge pull request #977 from marcnjaramillo/fix-changelog-entries Update CHANGELOG.md entries for most recent changes	2021-10-19 11:25:46 -07:00
marcnjaramillo	98d48a3709	Update CHANGELOG.md entries for most recent changes	2021-10-19 11:14:19 -07:00
Andrew Eisenberg	6b57993b2a	Merge pull request #976 from marcnjaramillo/selected-database-view Fix issue where 'Set current database' shows on selected database in …	2021-10-19 11:04:22 -07:00
marcnjaramillo	34ac30e403	Fix issue where 'Set current database' shows on selected database in the database view	2021-10-19 10:52:06 -07:00
Andrew Eisenberg	b8618aa87e	Merge pull request #971 from marcnjaramillo/integrate-codeql-database-unbundle Integrate codeql database unbundle	2021-10-19 10:01:40 -07:00
marcnjaramillo	7d8e63c1d1	Make changes requested by @aeisenberg Co-authored by: Marc Jaramillo mnj.webdeveloper@gmail.com Co-authored by: Musab Guma'a mgsium@github.com	2021-10-18 18:38:25 -07:00
marcnjaramillo	b22a8692c8	Integrated CLI database unbundle command for archive download Co-authored by: Marc Jaramillo mnj.webdeveloper@gmail.comm Co-authored by: Musab Guma'a mgsium@github.com	2021-10-18 14:55:17 -07:00
Marc Jaramillo	b5cdd833e2	Merge branch 'github:main' into main	2021-10-14 10:41:05 -07:00
Andrew Eisenberg	81a2f9c428	Merge pull request #963 from marcnjaramillo/handle-nonprint-chars Handle nonprint chars	2021-10-08 14:35:24 -07:00
Marc Jaramillo	b43b824da6	Simplify changelog entry Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>	2021-10-08 13:52:07 -07:00
marcnjaramillo	22616c5582	Add changes to changelog	2021-10-08 13:05:48 -07:00
Marc Jaramillo	2570d179bc	Merge branch 'github:main' into handle-nonprint-chars	2021-10-08 13:00:10 -07:00
Marc Jaramillo	1980f862c6	Merge branch 'github:main' into main	2021-10-08 12:57:36 -07:00
marcnjaramillo	d1eb31e231	Finish creating check for non-printing characters	2021-10-08 12:56:56 -07:00
shati-patel	68863e3b90	Bump version to v1.5.7	2021-10-07 18:47:29 +01:00
shati-patel	b38b884715	Add 1.5.6 header Some checks failed Release / Release (push) Has been cancelled Details Release / Publish to VS Code Marketplace (push) Has been cancelled Details Release / Publish to Open VSX Registry (push) Has been cancelled Details	2021-10-07 18:31:36 +01:00
shati-patel	cc6f2d8886	Prepare release	2021-10-07 18:31:36 +01:00
marcnjaramillo	245db7ca28	Add a check for strings with only new line chars	2021-10-07 09:09:48 -07:00
Andrew Eisenberg	197ab99db8	Merge pull request #959 from alexet/fix-db-remove Delete database after removing it from query server control.	2021-10-07 08:50:22 -07:00
Andrew Eisenberg	6292adf491	Merge branch 'main' into fix-db-remove	2021-10-07 08:17:24 -07:00
Andrew Eisenberg	112d40ff1c	Update CHANGELOG.md	2021-10-07 08:15:35 -07:00
Andrew Eisenberg	b92d6bab7c	Merge pull request #965 from github/aeisenberg/logo-update Update CodeQL logo	2021-10-07 08:08:51 -07:00
Andrew Eisenberg	0a4879c9a8	Merge branch 'main' into aeisenberg/logo-update	2021-10-07 08:01:05 -07:00
shati-patel	7d4d57104a	Update integration test version	2021-10-07 10:13:53 +01:00
Andrew Eisenberg	f06c9abb35	Update integration test versions	2021-10-07 10:13:53 +01:00
marcnjaramillo	2f7d175a76	Make edits per feedback	2021-10-06 14:01:39 -07:00
Andrew Eisenberg	85eaa8b275	Update CodeQL logo Both the marketplace icon and the sideview svg	2021-10-06 11:12:53 -07:00
marcnjaramillo	4783ad6bff	Create rough solution for handling non-printing characters in results	2021-10-05 19:37:14 -07:00
Marc Jaramillo	9f0a975a0c	Merge pull request #1 from github/main Add progress messages to LGTM download option. (#960)	2021-10-05 19:22:49 -07:00
Marc Jaramillo	21dda65871	Add progress messages to LGTM download option. (#960 ) * Add progress messages to LGTM download option. * Add additional argument to get test passing again. * Make edits requested by @aeisenerg * Fix assertion in test case * Update extensions/ql-vscode/CHANGELOG.md	2021-10-04 17:22:11 +01:00
Andrew Eisenberg	39fdd0cad5	Merge pull request #957 from marcnjaramillo/fix-lgtm-download-message Remove line about selecting a language from the dropdown.	2021-10-01 12:07:46 -07:00
Marc Jaramillo	3fb2c71390	Merge branch 'main' into fix-lgtm-download-message	2021-10-01 11:19:57 -07:00
marcnjaramillo	b40f648a87	Remove line about selecting a language from the dropdown.	2021-10-01 11:07:37 -07:00
alexet	57216249c2	Delete database after removing it from query server control.	2021-10-01 18:40:07 +01:00
shati-patel	fbadc15ae9	Also prompt user if the repo in the config is invalid	2021-10-01 18:11:53 +01:00
shati-patel	89b00eaef8	Show input box if there's no controller repo defined in settings	2021-10-01 18:11:53 +01:00
shati-patel	4bc5086cfb	Update test	2021-10-01 18:11:53 +01:00
shati-patel	7a79d39e23	Add new setting to specify controller repo	2021-10-01 18:11:53 +01:00
shati-patel	41ae5a4b5f	Create new "remoteQueries" setting	2021-10-01 18:11:53 +01:00
Musab Guma'a	0493e316c0	Update extensions/ql-vscode/CHANGELOG.md Co-authored-by: Edoardo Pirovano <6748066+edoardopirovano@users.noreply.github.com>	2021-10-01 17:37:54 +01:00
Musab Guma'a	137e17c2e1	Add fix entry to CHANGELOG.md	2021-10-01 17:37:54 +01:00
Musab Guma'a	31db2ffb82	Fix version copy for missing cli	2021-10-01 17:37:54 +01:00
Aditya Sharad	df18ff3052	Update CLI tests to use 2.6.2 (#955 ) * Actions: Update tests to use CLI 2.6.2 * Integration tests: Update to CLI 2.6.2	2021-09-21 22:44:01 +00:00
Benjamin Muskalla	74555510b4	Fix formatting for history format preference There was a space missing for one of the items, making it not rendered as part of the list	2021-09-16 07:53:23 -07:00
shati-patel	a2b8e7d193	Rename function	2021-09-15 10:52:47 +01:00
shati-patel	b59638bd2e	Test the regex for "getRepositories"	2021-09-15 10:52:47 +01:00
shati-patel	b0e19926da	Tests for "validateRepositories"	2021-09-15 10:52:47 +01:00
shati-patel	2e1b83588c	Put error handling into separate "validateRepositories" function	2021-09-15 10:52:47 +01:00
shati-patel	ab441ef75c	Tests for "getRepositories"	2021-09-15 10:52:47 +01:00
shati-patel	b4478e9b54	Remove `token` for running a remote query	2021-09-09 14:06:29 +01:00
aeisenberg	a715ce13c9	Bump version to v1.5.6	2021-09-08 16:15:15 -07:00