mirror of
https://github.com/github/codeql.git
synced 2025-12-16 16:53:25 +01:00
102 lines
4.1 KiB
Markdown
102 lines
4.1 KiB
Markdown
# Query classification and display
|
||
|
||
## Attributable Queries
|
||
|
||
The results of some queries are unsuitable for attribution to individual
|
||
developers. Most of them have a threshold value on which they trigger,
|
||
for example all metric violations and statistics based queries. The
|
||
results of such queries would all be attributed to the person pushing
|
||
the value over (or under) the threshold. Some queries only trigger when
|
||
another one doesn't. An example of this is the MaybeNull query which
|
||
only triggers if the AlwaysNull query doesn't. A small change in the
|
||
data flow could make an alert switch from AlwaysNull to MaybeNull (or
|
||
vice versa). As a result we attribute both a fix and an introduction to
|
||
the developer that changed the data flow. For this particular example
|
||
the funny attribution results are more a nuisance than a real problem;
|
||
the overall alert count remains unchanged. However, for the duplicate
|
||
and similar code queries the effects can be much more severe, as they
|
||
come in versions for "duplicate file" and "duplicate function" among
|
||
many others, where "duplicate function" only triggers if "duplicate
|
||
file" didn't. As a result adding some code to a duplicate file might
|
||
result in a "fix" of a "duplicate file" alert and an introduction of
|
||
many "duplicate function" alerts. This would be highly unfair.
|
||
Currently, only the duplicate and similar code queries exhibit this
|
||
"exchanging one for many" alerts when trying to attribute their results.
|
||
Therefore we currently exclude all duplicate code related alerts from
|
||
attribution.
|
||
|
||
The following queries are excluded from attribution:
|
||
|
||
- Metric violations, i.e. the ones with metadata properties like
|
||
`@(error|warning|recommendation)-(to|from)`
|
||
- Queries with tag `non-attributable`
|
||
|
||
This check is applied when the results of a single attribution are
|
||
loaded into the datastore. This means that any change to this behaviour
|
||
will only take effect on newly attributed revisions but the historical
|
||
data remains unchanged.
|
||
|
||
## Query severity and precision
|
||
|
||
We currently classify queries on two axes, with some additional tags.
|
||
Those axes are severity and precision, and are defined using the
|
||
query-metadata properties `@problem.severity` and `@precision`.
|
||
|
||
For severity, we have the following categories:
|
||
|
||
- Error
|
||
- Warning
|
||
- Recommendation
|
||
|
||
These categories may change in the future.
|
||
|
||
For precision, we have the following categories:
|
||
|
||
- very-high
|
||
- high
|
||
- medium
|
||
- low
|
||
|
||
As [usual](https://en.wikipedia.org/wiki/Precision_and_recall),
|
||
precision is defined as the percentage of query results that are true
|
||
positives, i.e., precision = number of true positives / (number of true
|
||
positives + number of false positives). There is no hard-and-fast rule
|
||
for which precision ranges correspond to which categories.
|
||
|
||
We expect these categories to remain unchanged for the foreseeable
|
||
future.
|
||
|
||
### A note on precision
|
||
|
||
Intuitively, precision measures how well the query performs at finding the
|
||
results it is supposed to find, i.e., how well it implements its
|
||
(informal, unwritten) rule. So how precise a query is depends very much
|
||
on what we consider that rule to be. We generally try to sharpen our
|
||
rules to focus on results that a developer might actually be interested
|
||
in.
|
||
|
||
## Which queries to run and display on LGTM
|
||
|
||
The following queries are run:
|
||
|
||
Precision: | very-high | high | medium | low
|
||
---------------|-----------|---------|---------|----
|
||
Error | **Yes** | **Yes** | **Yes** | No
|
||
Warning | **Yes** | **Yes** | **Yes** | No
|
||
Recommendation | **Yes** | **Yes** | No | No
|
||
|
||
The following queries have their results displayed by default:
|
||
|
||
Precision: | very-high | high | medium | low
|
||
---------------|-----------|---------|--------|----
|
||
Error | **Yes** | **Yes** | No | No
|
||
Warning | **Yes** | **Yes** | No | No
|
||
Recommendation | **Yes** | No | No | No
|
||
|
||
Results for queries that are run but not displayed by default can be
|
||
made visible by editing the project configuration.
|
||
|
||
Queries from custom query packs (in-repo or site-wide) are always run
|
||
and displayed by default. They can be hidden by editing the project
|
||
config, and "disabled" by removing them from the query pack.
|