Merge pull request #12321 from github/fc-7775-docs-update

CodeQL extension for VS Code docs update

docs/codeql/codeql-for-visual-studio-code/about-codeql-for-visual-studio-code.rst

@@ -14,7 +14,7 @@ CodeQL for Visual Studio Code provides an easy way to run queries from the large
 With these queries, or your own custom queries, you can analyze databases generated from source code to find errors and security vulnerabilities.
 The Results view shows the flow of data through the results of path queries, which is essential for triaging security results.
 
-The CodeQL extension also adds a **CodeQL** sidebar view to VS Code. This contains a list of databases, and an overview of the queries that you have run in the current session.
+The CodeQL extension also adds a **CodeQL** sidebar view to VS Code. This contains a list of local CodeQL databases, an overview of the queries that you have run in the current session, and a variant analysis view for large scale analysis.
 
 The extension provides standard `IntelliSense <https://code.visualstudio.com/docs/editor/intellisense>`__
 features for query files (extension ``.ql``) and library files (extension ``.qll``) that you open in the Visual Studio Code editor.
@@ -36,4 +36,5 @@ Further reading
 -------------------
 
 - ":doc:`Setting up CodeQL in Visual Studio Code <setting-up-codeql-in-visual-studio-code>`"
-- ":doc:`Analyzing your projects <analyzing-your-projects>`"
+- ":doc:`Analyzing your projects <analyzing-your-projects>`"
+- ":doc:`Running CodeQL queries at scale with multi-repository variant analysis <running-codeql-queries-at-scale-with-mrva>`"

docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst

@@ -5,7 +5,7 @@
 Analyzing your projects
 =================================================
 
-You can run queries on CodeQL databases and view the results in Visual Studio Code.
+You can run queries on CodeQL databases and view the results in Visual Studio Code. This article explains how to get a CodeQL database and analyze it on your local machine. For information on running analysis at scale across many CodeQL databases, see ":ref:`Running CodeQL queries at scale with multi-repository variant analysis <running-codeql-queries-at-scale-with-mrva>`."
 
 Choosing a database
 ------------------------
@@ -24,8 +24,8 @@ To analyze a project, you need to add a :ref:`CodeQL database <codeql-database>`
 
 #. Once you've chosen a database, it is displayed in the Databases view. To see the menu options for interacting with a database, right-click an entry in the list. You can select multiple databases using **Ctrl/Cmd+click**.
 
-Obtaining a local database
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Importing a local database
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP file, you can add it to Visual Studio Code. There are several ways to obtain a local CodeQL database.
 
@@ -37,6 +37,9 @@ If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP
 
   For more information about running query tests, see "`Testing custom queries <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/testing-custom-queries>`__" in the CodeQL CLI help.
 
+Downloading a database from GitHub
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 .. include:: ../reusables/download-github-database.rst
 
 Running a query

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

@@ -25,6 +25,8 @@ Editing settings
 
 3. Edit a setting. The new settings are saved automatically.
 
+Alternatively, you can edit the settings in JSON format by opening the command palette and selecting **Preferences: Open User Settings (JSON)**.
+
 Choosing a version of the CodeQL CLI
 --------------------------------------
 
@@ -55,8 +57,8 @@ By default, items in the query history view are retained for 30 days. You can se
 
 .. _configuring-settings-for-running-queries:
 
-Configuring settings for running queries
------------------------------------------
+Configuring settings for running queries locally
+------------------------------------------------
 
 There are a number of settings for **Running Queries**. If your queries run too slowly and time out frequently, you may want to increase the memory. 
 
@@ -64,8 +66,49 @@ There are a number of settings for **Running Queries**. If your queries run too
 
 To save query server logs in a custom location, edit the **Running Queries: Custom Log Directory** setting. If you use a custom log directory, the extension saves the logs permanently, instead of deleting them automatically after each workspace session. This is useful if you want to investigate these logs to improve the performance of your queries.
 
-Configuring settings for testing queries
------------------------------------------
+Configuring settings for variant analysis
+------------------------------------------
+
+You can define or edit lists of GitHub repositories for variant analysis, and change to a different controller repository using the **Variant analysis** settings.
+
+For information on the purpose and requirements for a controller repository, see ":ref:`Setting up a controller repository for variant analysis <controller-repository>`."
+
+You can also edit the items shown in the Variant Analysis Repositories panel by editing a file in your Visual Studio Code workspace called ``databases.json``. This file contains a JSON representation of all the items displayed in the panel. To open your ``databases.json`` file in an editor window, click the **{ }** icon in the top right of the Variant Analysis Repositories panel. You can then see a structured representation of the repos, orgs and lists in your panel. For example:
+
+.. code-block:: json
+
+  {
+    "version": 1,
+    "databases": {
+      "variantAnalysis": {
+        "repositoryLists": [
+          {
+            "name": "My favorite JavaScript repos",
+            "repositories": [
+              "facebook/react",
+              "babel/babel",
+              "angular/angular"
+            ]
+          }
+        ],
+        "owners": [
+          "microsoft"
+        ],
+        "repositories": [
+          "apache/hadoop"
+        ]
+      }
+    },
+    "selected": {
+      "kind": "variantAnalysisSystemDefinedList",
+      "listName": "top_10"
+    }
+  }
+
+You can change the items shown in the panel or add new items by directly editing this file.
+
+Configuring settings for testing queries locally
+------------------------------------------------
 
 To increase the number of threads used for testing queries, you can update the **Running Tests > Number Of Threads** setting.
 

docs/codeql/codeql-for-visual-studio-code/exploring-data-flow-with-path-queries.rst

@@ -3,7 +3,7 @@
 .. _exploring-data-flow-with-path-queries:
 
 Exploring data flow with path queries
-=================================================
+=====================================
 
 You can run CodeQL queries in VS Code to help you track the flow of data through a program, highlighting areas that are potential security vulnerabilities.
 
@@ -20,8 +20,8 @@ You can also modify the existing queries to model data flow more precisely for t
 To ensure that your path query uses the correct format and metadata, follow the instructions in ":ref:`Creating path queries <creating-path-queries>`."
 This topic also contains detailed information about how to define new sources and sinks, as well as templates and examples of how to extend the CodeQL libraries to suit your analysis.
 
-Running path queries in VS Code
------------------------------------
+Running path queries in VS Code locally
+---------------------------------------
 
 #. Open a path query in the editor.
 #. Right-click in the query window and select **CodeQL: Run Query on Selected Database**. (Alternatively, run the command from the Command Palette.)
@@ -30,6 +30,8 @@ Running path queries in VS Code
 #. Click each step to jump to it in the source code and investigate the problem further.
 #. To navigate the results from your keyboard, you can bind shortcuts to the **CodeQL: Navigate Up/Down/Left/Right in Result Viewer** commands.
 
+When you are ready to run a path query at scale, you can use the Variant Analysis Repositories panel to run the query against up to 1,000 repositories on GitHub.com. For information on running analysis at scale across many CodeQL databases, see ":ref:`Running CodeQL queries at scale with multi-repository variant analysis <running-codeql-queries-at-scale-with-mrva>`."
+
 Further reading
 -----------------
 

docs/codeql/codeql-for-visual-studio-code/index.rst

@@ -27,6 +27,11 @@ The CodeQL extension for Visual Studio Code adds rich language support for CodeQ
   VS Code to help you track the flow of data through a program, highlighting
   areas that are potential security vulnerabilities.  
 
+- :doc:`Running CodeQL queries at scale with multi-repository variant analysis
+  <running-codeql-queries-at-scale-with-mrva>`: You can run queries against groups
+  of repositories on GitHub.com and view results in Visual Studio Code as each analysis
+  finishes. 
+
 - :doc:`Testing CodeQL queries in Visual Studio Code
   <testing-codeql-queries-in-visual-studio-code>`: You can run unit tests for
   CodeQL queries using the Visual Studio Code extension.
@@ -40,7 +45,13 @@ The CodeQL extension for Visual Studio Code adds rich language support for CodeQ
 
 - :doc:`Troubleshooting CodeQL for Visual Studio Code
   <troubleshooting-codeql-for-visual-studio-code>`: You can use the detailed 
-  information written to the extension's log files if you need to troubleshoot problems.
+  information written to the extension's log files if you need to troubleshoot problems with
+  analysis of local CodeQL databases.
+
+- :doc:`Troubleshooting variant analysis
+  <troubleshooting-variant-analysis>`: You can use the detailed 
+  information written to workflow log files in your controller repository if you need to
+  troubleshoot problems with analysis of CodeQL databases stored on GitHub.com.
 
 - :doc:`About telemetry in CodeQL for Visual Studio Code <about-telemetry-in-codeql-for-visual-studio-code>`: If you specifically opt in to permit GitHub to do so, GitHub will collect usage data and metrics for the purposes of helping the core developers to improve the CodeQL extension for VS Code.
 
@@ -53,8 +64,10 @@ The CodeQL extension for Visual Studio Code adds rich language support for CodeQ
    analyzing-your-projects
    exploring-the-structure-of-your-source-code
    exploring-data-flow-with-path-queries
+   running-codeql-queries-at-scale-with-mrva
    testing-codeql-queries-in-visual-studio-code
    working-with-codeql-packs-in-visual-studio-code
    customizing-settings
    troubleshooting-codeql-for-visual-studio-code
+   troubleshooting-variant-analysis
    about-telemetry-in-codeql-for-visual-studio-code

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

@@ -0,0 +1,163 @@
+:tocdepth: 1
+
+.. _running-codeql-queries-at-scale-with-mrva:
+
+Running CodeQL queries at scale with multi-repository variant analysis
+======================================================================
+
+.. include:: ../reusables/beta-note-mrva.rst
+
+About multi-repository variant analysis
+---------------------------------------
+
+When you write a query to find variants of a security vulnerability and finish testing it locally, the next step is to run it on a large group of repositories. Multi-repository variant analysis (variant analysis) makes it easy run a query on up to 1000 repositories without leaving Visual Studio Code.
+
+The core functionality of the CodeQL extension helps you write queries and run them locally against a CodeQL database. In contrast, variant analysis allows you to send your CodeQL query to GitHub.com to be tested against a list of repositories.
+
+When you run variant analysis against a list of repositories, your query is run against each repository that has a CodeQL database available to analyze. GitHub creates and stores the latest CodeQL database for the default branch of thousands of public repositories, including every repository that runs code scanning using CodeQL. 
+
+If you want to run variant analysis on your repositories, you need to enable code scanning using CodeQL on GitHub.com before adding your repository to a list for analysis (either default setup, or advanced setup using the CodeQL action). For information about enabling code scanning using CodeQL, see "`Configuring code scanning automatically <https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically>`__."
+
+.. _controller-repository:
+
+Setting a controller repository for variant analysis
+----------------------------------------------------
+
+When you run variant analysis, the analysis is run entirely using GitHub Actions. You don't need to create any workflows, but you must specify which GitHub repository the CodeQL extension should use as the "controller repository." Controller repositories can be empty, but they must have at least one commit. The ``GITHUB_TOKEN`` must also have "Read and write permissions" to run workflows in that repository. For more information, see "`Managing GitHub Actions settings for a repository <https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#setting-the-permissions-of-the-github_token-for-your-repository>`__."
+
+.. pull-quote::
+   
+   Note
+
+   - The controller repository visibility can be "public" if you plan to analyze public repositories. The variant analysis will be free.
+   - The controller repository visibility must be "private" if you need to analyze any private or internal repositories. Any actions minutes used by variant analysis, above the free limit, will be charged to the repository owner. For more information about free minutes and billing, see "`About billing for GitHub Actions <https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions>`__." 
+
+You must define a controller repository before you can run your first variant analysis.
+
+.. image:: ../images/codeql-for-visual-studio-code/controller-repository.png
+    :width: 350
+    :alt: Screenshot of the CodeQL extension in Visual Studio Code. The "Variant Analysis Repositories" section is expanded and the "Set up controller repository" button is highlighted with a dark orange outline.
+
+#. In Visual Studio Code, click **QL** in the left sidebar to display the CodeQL extension.
+
+#. Expand **Variant Analysis Repositories** and click **Set up controller repository** to display a field for the controller repository.
+
+#. Type the owner and name of the repository on GitHub.com that you want to use as your controller repository and press the **Enter** key.
+
+#. If you are prompted to authenticate with GitHub, follow the instructions and sign into your personal or organization account. When you have finished following the process, a prompt from GitHub Authentication may ask for permission to open a URI in Visual Studio Code, click **Open**.
+
+The name of the controller repository is saved in your settings for the CodeQL extension. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."
+
+Running a query at scale using variant analysis
+-----------------------------------------------
+
+#. Expand the **Variant Analysis Repositories** section, to show the default lists which include a selection of 10, 100, and 1,000 public repositories on GitHub.com for the language that you are analyzing.
+
+#. Select which GitHub repository or repositories you want to run your query against. Click a row to highlight it, and then click **Select** to select that repository, organization, or list of repositories. If you want to add a new repository, organization, or list, use the options in the header panel. For information, see ":ref:`Creating custom lists of repositories <custom-lists>`", later in this article.
+
+    .. image:: ../images/codeql-for-visual-studio-code/variant-analysis-repo-lists.png
+        :width: 350
+        :alt: Screenshot of the CodeQL extension in Visual Studio Code. The "Variant Analysis Repositories" section is expanded. The "Top 10 repositories" item has a checkmark to show that it is currently selected for analysis. The user has clicked on the row for a single repository "octo-org/octo-repo" and it is highlighted blue. The "Select" button for that row is highlighted with a dark orange highlight.
+
+#. Open the query you want to run, right-click in the query file, and select **CodeQL: Run Variant Analysis** to start variant analysis.
+
+The CodeQL extension builds a CodeQL pack with your library and any library dependencies. The CodeQL pack and your selected repository list are posted to an API endpoint on GitHub.com which triggers a GitHub Actions dynamic workflow in your controller repository. The workflow spins up multiple parallel jobs to execute the CodeQL query against the repositories in the list, optimizing query execution. As each repository is analyzed, the results are processed and displayed in a Variant Analysis Results view in Visual Studio Code.
+
+.. pull-quote::
+
+    Note
+
+    If you need to cancel the variant analysis run for any reason, click **Stop query** in the Variant Analysis Results view.
+
+Exploring your results
+----------------------
+
+When you run variant analysis, as soon as a workflow to run your analysis on GitHub is running, a Variant Analysis Results view opens to display the results as they are ready. You can use this view to monitor progress, see any errors, and access the workflow logs in your controller repository.
+
+.. image:: ../images/codeql-for-visual-studio-code/variant-analysis-results-view.png
+    :alt: Screenshot of the "Variant Analysis Results" view showing a partially complete run. Analysis of ``angular/angular`` is still running but all other results are displayed. ``facebook/create-react-app`` has three results for this query.
+
+When your variant analysis run is scheduled, the results view automatically opens. Initially the view shows a list of every repository that was scheduled for analysis. As each repository is analyzed, the view is updated to show a summary of the number of results. To view the detailed results for a repository (including results paths), click the repository name.
+
+For each repository, you can see:
+
+- Number of results found by the query
+- Visibility of the repository
+- Whether analysis is still running (black, moving circle) or finished (green checkmark)
+- Number of stars the repository has on GitHub
+- When the repository was last updated
+
+To see the results for a repository:
+
+.. image:: ../images/codeql-for-visual-studio-code/variant-analysis-result.png
+    :alt: Screenshot of an example result in the "Variant Analysis Results" view. The result has blue links to the source files in GitHub so you can go straight to the repository to fix the problem. There is also a "Show paths" link because this is a data flow query.
+
+#. Click the repository name to show a summary of each result.
+
+#. Explore the information available for each result using links to the source files in GitHub.com and, for data flow queries, the **Show paths** link. For more information, see ":ref:`Exploring data flow with path queries <exploring-data-flow-with-path-queries>`."
+
+Exporting your results
+----------------------
+
+You can export your results for further analysis or to discuss them with collaborators. In the results view, click **Export results** to export the results to a secret gist on GitHub.com or to a markdown file in your workspace.
+
+.. _custom-lists:
+
+Creating custom lists of repositories
+-------------------------------------
+
+After you have defined a controller repository, the Variant Analysis Repositories panel shows the lists of repositories that you can select for variant analysis. You can use the options in the panel header to add a specific repository or organization to the panel, and to create and manage custom lists of repositories for variant analysis.
+
+.. pull-quote::
+
+    Note
+
+    CodeQL analysis always requires a CodeQL database to run queries against. When you run variant analysis against a list of repositories, your query will only be executed against the repositories that currently have a CodeQL database available to download.  The best way to make a repository available for variant analysis is to enable code scanning with CodeQL. For information about enabling code scanning using CodeQL, see "`Configuring code scanning automatically <https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically>`__."
+
+Selecting a single GitHub repository or organization for analysis
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. In the Variant Analysis Repositories panel, click the **+**, add new database, icon. 
+
+#. From the dropdown menu, click **From a GitHub repository** or **All repositories of GitHub org or owner**.
+
+#. Type the identifier of the repository or organization that you want to use into the field.
+
+    .. image:: ../images/codeql-for-visual-studio-code/variant-analysis-repo-and-org.png
+        :width: 350
+        :alt: Screenshot of the CodeQL extension in Visual Studio Code. The "Variant Analysis Repositories" section is expanded to show a repository (octo-org/octo-repo) and an organization (octo-org). These items are highlighted with a dark orange outline.
+
+Creating a custom list of repositories
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. In the Variant Analysis Repositories panel, click the |add-list| icon.
+
+#. Type a name for the new list and press **Enter**.
+
+#. Select your list in the panel and then click **+**, to add a repository to your list.
+
+You can manage and edit your custom lists by right-clicking on either the list name, or a repository name within the list, and selecting an option from the context menu.
+
+The custom lists are stored in your workspace in a ``databases.json`` file. If you want to edit this file directly, you can open it by clicking **{ }** in the panel header. 
+
+For example, if you want to continue analyzing a set of repositories that had results for your query, click **Copy repository list** in the Variant Analysis Results view to add a list of only the repositories that have results to the clipboard as JSON. For example:
+
+.. code-block:: json
+
+    {
+        "name": "new-repo-list",
+        "repositories": [
+            "facebook/create-react-app"
+        ]
+    }
+
+You can then insert the ``new-repo-list`` of repositories into your list of custom repository lists for easy access in the Variant Analysis Repositories panel.
+
+Troubleshooting variant analysis
+--------------------------------
+
+For information on troubleshooting variant analysis, see
+":ref:`Troubleshooting variant analysis <troubleshooting-variant-analysis>`."
+
+.. |add-list| image:: ../images/codeql-for-visual-studio-code/variant-analysis-add-list.png
+    :height: 2ex

docs/codeql/codeql-for-visual-studio-code/testing-codeql-queries-in-visual-studio-code.rst

@@ -5,7 +5,7 @@
 Testing CodeQL queries in Visual Studio Code
 ============================================
 
-You can run unit tests for CodeQL queries using the Visual Studio Code extension.
+You can run unit tests for CodeQL queries using the Visual Studio Code extension. When you are sure that your query finds the results you want to identify, you can use variant analysis to run it at scale. For information on running analysis at scale across many CodeQL databases, see ":ref:`Running CodeQL queries at scale with multi-repository variant analysis <running-codeql-queries-at-scale-with-mrva>`."
 
 About testing queries in VS Code
 ---------------------------------

docs/codeql/codeql-for-visual-studio-code/troubleshooting-codeql-for-visual-studio-code.rst

@@ -5,7 +5,12 @@
 Troubleshooting CodeQL for Visual Studio Code
 =============================================
 
-You can use the detailed information written to the extension's log files if you need to troubleshoot problems.
+This article explains how to debug problems with the analysis of CodeQL databases that are stored on your local
+machine. For information on troubleshooting variant analysis, which runs on GitHub.com, see
+":ref:`Troubleshooting variant analysis <troubleshooting-variant-analysis>`."
+
+You can use the detailed information written to the extension's log files if you need to troubleshoot problems
+analyzing CodeQL databases that are stored locally.
 
 About the log files
 --------------------

docs/codeql/codeql-for-visual-studio-code/troubleshooting-variant-analysis.rst

@@ -0,0 +1,30 @@
+:tocdepth: 1
+
+.. _troubleshooting-variant-analysis:
+
+Troubleshooting variant analysis
+================================
+
+.. include:: ../reusables/beta-note-mrva.rst
+
+This article explains how to debug problems with variant analysis, that is, analysis run using GitHub Actions
+and not locally on your machine.
+For information on troubleshooting local analysis, see
+":ref:`Troubleshooting CodeQL for Visual Studio Code <troubleshooting-codeql-for-visual-studio-code>`."
+
+When you run variant analysis, there are two key places where errors and warnings are displayed:
+
+#. **Visual Studio Code errors** - any problems with creating a CodeQL pack and sending the analysis to GitHub.com are reported as Visual Studio Code errors in the bottom right corner of the application. The problem information is also available in the **Problems** view.
+#. **Variant Analysis Results** - any problems with the variant analysis run are reported in this view.
+
+Variant analysis warning: Problem with controller repository
+------------------------------------------------------------
+
+If there are problems with the variant analysis run, you will see a warning banner at the top of the Variant Analysis Results tab. For example:
+
+.. image:: ../images/codeql-for-visual-studio-code/variant-analysis-results-warning.png
+    :width: 600
+    :alt: Screenshot of the "Variant Analysis Results" view showing a warning banner with the text "warning: Problem with controller repository" and "Publicly visible controller repository can't be used to analyze private repositories. 1 private repository was not analyzed." The "Show logs" button is highlighted with a dark orange outline.
+
+In this example, the user ran variant analysis on a custom list of two repositories. One of the repositories was a private repository and could not be analyzed because they had a public controller repository. Only the public repository was analyzed. To analyze both repositories, this user needs to edit their settings and update the controller repository to a private repository. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."
+

docs/codeql/reusables/beta-note-mrva.rst

@@ -0,0 +1,7 @@
+.. pull-quote::
+
+    Note
+
+    Multi-repository variant analysis is currently available as a beta release and is subject to change. To use this feature, you must upgrade the CodeQL extension for Visual Studio Code to a minimum of version 1.8.0.
+    
+    You can report your feedback in the community discussion for the beta release: https://gh.io/mrva-public-beta-discussion.