@@ -1,10 +1,11 @@
|
|||||||
# CodeQL for Visual Studio Code: Changelog
|
# CodeQL for Visual Studio Code: Changelog
|
||||||
|
|
||||||
## [UNRELEASED]
|
## 1.7.11 - 1 March 2023
|
||||||
|
|
||||||
- Enable collection of telemetry concerning interactions with UI elements, including buttons, links, and other inputs. [#2114](https://github.com/github/vscode-codeql/pull/2114)
|
- Enable collection of telemetry concerning interactions with UI elements, including buttons, links, and other inputs. [#2114](https://github.com/github/vscode-codeql/pull/2114)
|
||||||
|
- Prevent the automatic installation of CodeQL CLI version 2.12.3 to avoid a bug in the language server. CodeQL CLI 2.12.2 will be used instead. [#2126](https://github.com/github/vscode-codeql/pull/2126)
|
||||||
|
|
||||||
# 1.7.10 - 23 February 2023
|
## 1.7.10 - 23 February 2023
|
||||||
|
|
||||||
- Fix bug that was causing unwanted error notifications.
|
- Fix bug that was causing unwanted error notifications.
|
||||||
|
|
||||||
|
|||||||
@@ -2,15 +2,17 @@
|
|||||||
|
|
||||||
This document describes the manual test plan for the QL extension for Visual Studio Code.
|
This document describes the manual test plan for the QL extension for Visual Studio Code.
|
||||||
|
|
||||||
The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
|
The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
|
||||||
effort vs value basis).
|
effort vs value basis).
|
||||||
|
|
||||||
#### What this doesn't cover
|
## What this doesn't cover
|
||||||
|
|
||||||
We don't need to test features (and permutations of features) that are covered by automated tests.
|
We don't need to test features (and permutations of features) that are covered by automated tests.
|
||||||
|
|
||||||
### Before releasing the VS Code extension
|
## Before releasing the VS Code extension
|
||||||
|
|
||||||
- Go through the required test cases listed below
|
- Go through the required test cases listed below
|
||||||
- Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
|
- Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
|
||||||
choose to go through some of the Optional Test Cases.
|
choose to go through some of the Optional Test Cases.
|
||||||
- Run a query using the existing version of the extension (to generate an "old" query history item)
|
- Run a query using the existing version of the extension (to generate an "old" query history item)
|
||||||
|
|
||||||
@@ -24,23 +26,25 @@ choose to go through some of the Optional Test Cases.
|
|||||||
|
|
||||||
1. Open the [UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
|
1. Open the [UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
|
||||||
2. Run a MRVA against the following repo list:
|
2. Run a MRVA against the following repo list:
|
||||||
```
|
|
||||||
{
|
```json
|
||||||
"name": "test-repo-list",
|
{
|
||||||
"repositories": [
|
"name": "test-repo-list",
|
||||||
"angular-cn/ng-nice",
|
"repositories": [
|
||||||
"apache/hadoop",
|
"angular-cn/ng-nice",
|
||||||
"apache/hive"
|
"apache/hadoop",
|
||||||
]
|
"apache/hive"
|
||||||
}
|
]
|
||||||
```
|
}
|
||||||
|
```
|
||||||
|
|
||||||
3. Check that a notification message pops up and the results view is opened.
|
3. Check that a notification message pops up and the results view is opened.
|
||||||
4. Check the query history. It should:
|
4. Check the query history. It should:
|
||||||
- Show that an item has been added to the query history
|
- Show that an item has been added to the query history
|
||||||
- The item should be marked as "in progress".
|
- The item should be marked as "in progress".
|
||||||
5. Once the query starts:
|
5. Once the query starts:
|
||||||
- Check the results view
|
- Check the results view
|
||||||
- Check the code paths view, including the code paths drop down menu.
|
- Check the code paths view, including the code paths drop down menu.
|
||||||
- Check that the repository filter box works
|
- Check that the repository filter box works
|
||||||
- Click links to files/locations on GitHub
|
- Click links to files/locations on GitHub
|
||||||
- Check that the query history item is updated to show the number of results
|
- Check that the query history item is updated to show the number of results
|
||||||
@@ -74,7 +78,7 @@ choose to go through some of the Optional Test Cases.
|
|||||||
1. Click a history item (for MRVA):
|
1. Click a history item (for MRVA):
|
||||||
- Check that exporting results works
|
- Check that exporting results works
|
||||||
- Check that sorting results works
|
- Check that sorting results works
|
||||||
- Check that copying repo lists works
|
- Check that copying repo lists works
|
||||||
2. Open the query results directory:
|
2. Open the query results directory:
|
||||||
- Check that the correct directory is opened and there are results in it
|
- Check that the correct directory is opened and there are results in it
|
||||||
3. View logs
|
3. View logs
|
||||||
@@ -84,12 +88,12 @@ choose to go through some of the Optional Test Cases.
|
|||||||
|
|
||||||
Run one of the above MRVAs, but cancel it from within VS Code:
|
Run one of the above MRVAs, but cancel it from within VS Code:
|
||||||
- Check that the query is canceled and the query history item is updated.
|
- Check that the query is canceled and the query history item is updated.
|
||||||
- Check that the workflow run is also canceled.
|
- Check that the workflow run is also canceled.
|
||||||
- Check that any available results are visible in VS Code.
|
- Check that any available results are visible in VS Code.
|
||||||
|
|
||||||
### Test Case 6: MRVA - Change to a different colour theme
|
### Test Case 6: MRVA - Change to a different colour theme
|
||||||
|
|
||||||
Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
|
Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
|
||||||
Are there any components that are not showing up?
|
Are there any components that are not showing up?
|
||||||
|
|
||||||
## Optional Test Cases
|
## Optional Test Cases
|
||||||
@@ -99,9 +103,10 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
|
|||||||
### Selecting repositories to run on
|
### Selecting repositories to run on
|
||||||
|
|
||||||
#### Test case 1: Running a query on a single repository
|
#### Test case 1: Running a query on a single repository
|
||||||
1. When the repository exists and is public
|
|
||||||
1. Has a CodeQL database for the correct language
|
1. When the repository exists and is public
|
||||||
2. Has a CodeQL database for another language
|
1. Has a CodeQL database for the correct language
|
||||||
|
2. Has a CodeQL database for another language
|
||||||
3. Does not have any CodeQL databases
|
3. Does not have any CodeQL databases
|
||||||
2. When the repository exists and is private
|
2. When the repository exists and is private
|
||||||
1. Is accessible and has a CodeQL database
|
1. Is accessible and has a CodeQL database
|
||||||
@@ -109,14 +114,16 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
|
|||||||
3. When the repository does not exist
|
3. When the repository does not exist
|
||||||
|
|
||||||
#### Test case 2: Running a query on a custom repository list
|
#### Test case 2: Running a query on a custom repository list
|
||||||
|
|
||||||
1. The repository list is non-empty
|
1. The repository list is non-empty
|
||||||
1. All repositories in the list have a CodeQL database
|
1. All repositories in the list have a CodeQL database
|
||||||
2. Some but not all repositories in the list have a CodeQL database
|
2. Some but not all repositories in the list have a CodeQL database
|
||||||
3. No repositories in the list have a CodeQL database
|
3. No repositories in the list have a CodeQL database
|
||||||
2. The repository list is empty
|
2. The repository list is empty
|
||||||
|
|
||||||
#### Test case 3: Running a query on all repositories in an organization
|
#### Test case 3: Running a query on all repositories in an organization
|
||||||
1. The org exists
|
|
||||||
|
1. The org exists
|
||||||
1. The org contains repositories that have CodeQL databases
|
1. The org contains repositories that have CodeQL databases
|
||||||
2. The org contains repositories of the right language but without CodeQL databases
|
2. The org contains repositories of the right language but without CodeQL databases
|
||||||
3. The org contains repositories not of the right language
|
3. The org contains repositories not of the right language
|
||||||
@@ -126,20 +133,25 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
|
|||||||
### Using different types of controller repos
|
### Using different types of controller repos
|
||||||
|
|
||||||
#### Test case 1: Running a query when the controller repository is public
|
#### Test case 1: Running a query when the controller repository is public
|
||||||
|
|
||||||
1. Can run queries on public repositories
|
1. Can run queries on public repositories
|
||||||
2. Can not run queries on private repositories
|
2. Can not run queries on private repositories
|
||||||
|
|
||||||
#### Test case 2: Running a query when the controller repository is private
|
#### Test case 2: Running a query when the controller repository is private
|
||||||
|
|
||||||
1. Can run queries on public repositories
|
1. Can run queries on public repositories
|
||||||
2. Can run queries on private repositories
|
2. Can run queries on private repositories
|
||||||
|
|
||||||
#### Test case 3: Running a query when the controller repo exists but you do not have write access
|
#### Test case 3: Running a query when the controller repo exists but you do not have write access
|
||||||
|
|
||||||
1. Cannot run queries
|
1. Cannot run queries
|
||||||
|
|
||||||
#### Test case 4: Running a query when the controller repo doesn’t exist
|
#### Test case 4: Running a query when the controller repo doesn’t exist
|
||||||
|
|
||||||
1. Cannot run queries
|
1. Cannot run queries
|
||||||
|
|
||||||
#### Test case 5: Running a query when the "config field" for the controller repo is not set
|
#### Test case 5: Running a query when the "config field" for the controller repo is not set
|
||||||
|
|
||||||
1. Cannot run queries
|
1. Cannot run queries
|
||||||
|
|
||||||
### Query History
|
### Query History
|
||||||
@@ -150,6 +162,7 @@ The first test case specifies actions that you can do when the query is first ru
|
|||||||
with this since it has quite a limited number of actions you can do.
|
with this since it has quite a limited number of actions you can do.
|
||||||
|
|
||||||
#### Test case 1: When variant analysis state is "pending"
|
#### Test case 1: When variant analysis state is "pending"
|
||||||
|
|
||||||
1. Starts monitoring variant analysis
|
1. Starts monitoring variant analysis
|
||||||
2. Cannot open query history item
|
2. Cannot open query history item
|
||||||
3. Can delete a query history item
|
3. Can delete a query history item
|
||||||
@@ -160,8 +173,8 @@ with this since it has quite a limited number of actions you can do.
|
|||||||
2. By query date
|
2. By query date
|
||||||
3. By result count
|
3. By result count
|
||||||
5. Cannot open query directory
|
5. Cannot open query directory
|
||||||
6. Can open query that produced these results
|
6. Can open query that produced these results
|
||||||
1. When the file still exists and has not moved
|
1. When the file still exists and has not moved
|
||||||
2. When the file does not exist
|
2. When the file does not exist
|
||||||
7. Cannot view logs
|
7. Cannot view logs
|
||||||
8. Cannot copy repository list
|
8. Cannot copy repository list
|
||||||
@@ -171,6 +184,7 @@ with this since it has quite a limited number of actions you can do.
|
|||||||
12. Cannot cancel analysis
|
12. Cannot cancel analysis
|
||||||
|
|
||||||
#### Test case 2: When the variant analysis state is not "pending"
|
#### Test case 2: When the variant analysis state is not "pending"
|
||||||
|
|
||||||
1. Query history is loaded when VSCode starts
|
1. Query history is loaded when VSCode starts
|
||||||
2. Handles when action workflow was canceled while VSCode was closed
|
2. Handles when action workflow was canceled while VSCode was closed
|
||||||
3. Can open query history item
|
3. Can open query history item
|
||||||
@@ -204,12 +218,14 @@ with this since it has quite a limited number of actions you can do.
|
|||||||
4. A popup allows you to open the directory
|
4. A popup allows you to open the directory
|
||||||
|
|
||||||
#### Test case 3: When variant analysis state is "in_progress"
|
#### Test case 3: When variant analysis state is "in_progress"
|
||||||
|
|
||||||
1. Starts monitoring variant analysis
|
1. Starts monitoring variant analysis
|
||||||
1. Ready results are downloaded
|
1. Ready results are downloaded
|
||||||
2. Can cancel analysis
|
2. Can cancel analysis
|
||||||
1. Causes the actions run to be canceled
|
1. Causes the actions run to be canceled
|
||||||
|
|
||||||
#### Test case 4: When variant analysis state is in final state ("succeeded"/"failed"/"canceled")
|
#### Test case 4: When variant analysis state is in final state ("succeeded"/"failed"/"canceled")
|
||||||
|
|
||||||
1. Stops monitoring variant analysis
|
1. Stops monitoring variant analysis
|
||||||
1. All results are downloaded if state is succeeded
|
1. All results are downloaded if state is succeeded
|
||||||
2. Otherwise, ready results are downloaded, if any are available
|
2. Otherwise, ready results are downloaded, if any are available
|
||||||
@@ -220,6 +236,7 @@ with this since it has quite a limited number of actions you can do.
|
|||||||
This requires running a MRVA query and seeing the results view.
|
This requires running a MRVA query and seeing the results view.
|
||||||
|
|
||||||
#### Test case 1: When variant analysis state is "pending"
|
#### Test case 1: When variant analysis state is "pending"
|
||||||
|
|
||||||
1. Can open a results view
|
1. Can open a results view
|
||||||
2. Results view opens automatically
|
2. Results view opens automatically
|
||||||
- When starting variant analysis run
|
- When starting variant analysis run
|
||||||
@@ -227,9 +244,10 @@ This requires running a MRVA query and seeing the results view.
|
|||||||
3. Results view is empty
|
3. Results view is empty
|
||||||
|
|
||||||
#### Test case 2: When variant analysis state is not "pending"
|
#### Test case 2: When variant analysis state is not "pending"
|
||||||
|
|
||||||
1. Can open a results view
|
1. Can open a results view
|
||||||
2. Results view opens automatically
|
2. Results view opens automatically
|
||||||
1. When starting variant analysis run
|
1. When starting variant analysis run
|
||||||
2. When VSCode opens (if view was open when VSCode was closed)
|
2. When VSCode opens (if view was open when VSCode was closed)
|
||||||
3. Can copy repository list
|
3. Can copy repository list
|
||||||
1. Text is copied to clipboard
|
1. Text is copied to clipboard
|
||||||
@@ -240,43 +258,45 @@ This requires running a MRVA query and seeing the results view.
|
|||||||
6. Can open query file
|
6. Can open query file
|
||||||
1. When the file still exists and has not moved
|
1. When the file still exists and has not moved
|
||||||
2. When the file does not exist
|
2. When the file does not exist
|
||||||
7. Can open query text
|
7. Can open query text
|
||||||
8. Can sort repos
|
8. Can sort repos
|
||||||
1. By name
|
1. By name
|
||||||
2. By results
|
2. By results
|
||||||
3. By stars
|
3. By stars
|
||||||
4. By last updated
|
4. By last updated
|
||||||
9. Can filter repos
|
9. Can filter repos
|
||||||
10. Shows correct statistics
|
10. Shows correct statistics
|
||||||
1. Total number of results
|
1. Total number of results
|
||||||
2. Total number of repositories
|
2. Total number of repositories
|
||||||
3. Duration
|
3. Duration
|
||||||
11. Can see live results
|
11. Can see live results
|
||||||
1. Results appear in extension as soon as each query is completed
|
1. Results appear in extension as soon as each query is completed
|
||||||
12. Can view interpreted results (i.e. for a "problem" query)
|
12. Can view interpreted results (i.e. for a "problem" query)
|
||||||
1. Can view non-path results
|
1. Can view non-path results
|
||||||
2. Can view code paths for "path-problem" queries
|
2. Can view code paths for "path-problem" queries
|
||||||
13. Can view raw results (i.e. for a non "problem" query)
|
13. Can view raw results (i.e. for a non "problem" query)
|
||||||
1. Renders a table
|
1. Renders a table
|
||||||
14. Can see skipped repositories
|
14. Can see skipped repositories
|
||||||
1. Can see repos with no db in a tab
|
1. Can see repos with no db in a tab
|
||||||
1. Shown warning that explains the tab
|
1. Shown warning that explains the tab
|
||||||
2. Can see repos with no access in a tab
|
2. Can see repos with no access in a tab
|
||||||
1. Shown warning that explains the tab
|
1. Shown warning that explains the tab
|
||||||
3. Only shows tab when there are skipped repos
|
3. Only shows tab when there are skipped repos
|
||||||
15. Result downloads
|
15. Result downloads
|
||||||
1. All results are downloaded automatically
|
1. All results are downloaded automatically
|
||||||
2. Download status is indicated by a spinner (Not currently any indication of progress beyond "downloading" and "not downloading")
|
2. Download status is indicated by a spinner (Not currently any indication of progress beyond "downloading" and "not downloading")
|
||||||
3. Only 3 items are downloaded at a time
|
3. Only 3 items are downloaded at a time
|
||||||
4. Results for completed queries are still downloaded when
|
4. Results for completed queries are still downloaded when
|
||||||
1. Some but not all queries failed
|
1. Some but not all queries failed
|
||||||
2. The variant analysis was canceled after some queries completed
|
2. The variant analysis was canceled after some queries completed
|
||||||
|
|
||||||
#### Test case 3: When variant analysis state is in "succeeded" state
|
#### Test case 3: When variant analysis state is in "succeeded" state
|
||||||
|
|
||||||
1. Can view logs
|
1. Can view logs
|
||||||
2. All results are downloaded
|
2. All results are downloaded
|
||||||
|
|
||||||
#### Test case 4: When variant analysis is in "failed" or "canceled" state
|
#### Test case 4: When variant analysis is in "failed" or "canceled" state
|
||||||
|
|
||||||
1. Can view logs
|
1. Can view logs
|
||||||
1. Results for finished queries are still downloaded.
|
1. Results for finished queries are still downloaded.
|
||||||
|
|
||||||
@@ -305,14 +325,17 @@ This requires running a MRVA query and seeing the results view.
|
|||||||
1. Collapse/expand tree nodes
|
1. Collapse/expand tree nodes
|
||||||
|
|
||||||
Error cases that trigger an error notification:
|
Error cases that trigger an error notification:
|
||||||
1. Try to add a list with a name that already exists
|
|
||||||
|
1. Try to add a list with a name that already exists
|
||||||
1. Try to add a top-level database that already exists
|
1. Try to add a top-level database that already exists
|
||||||
1. Try to add a database in a list that already exists in the list
|
1. Try to add a database in a list that already exists in the list
|
||||||
|
|
||||||
Error cases that show an error in the panel (and only the edit button should be visible):
|
Error cases that show an error in the panel (and only the edit button should be visible):
|
||||||
|
|
||||||
1. Edit the db config file directly and save invalid JSON
|
1. Edit the db config file directly and save invalid JSON
|
||||||
1. Edit the db config file directly and save valid JSON but invalid config (e.g. add an unknown property)
|
1. Edit the db config file directly and save valid JSON but invalid config (e.g. add an unknown property)
|
||||||
1. Edit the db config file directly and save two lists with the same name
|
1. Edit the db config file directly and save two lists with the same name
|
||||||
|
|
||||||
Cases where there the welcome view is shown:
|
Cases where there the welcome view is shown:
|
||||||
1. No controller repo is set in the user's settings JSON.
|
|
||||||
|
1. No controller repo is set in the user's settings JSON.
|
||||||
|
|||||||
Reference in New Issue
Block a user