Previously, the refined info was collected and the CID computed before saving.
This was a major development time sink, so the CID is now computed in the
following step (bin/mc-db-unique).
The columns previously chosen for the CID are not enough. If these columns are
empty for any reason, the CID repeats. Just including the owner/name won't help,
because those are duplicates.
Some possibilities considered and rejected:
1. Could use a random number for missing columns. But this makes
the CID nondeterministic.
2. Switch to the file system ctime? Not unique across owner/repo pairs,
but unique within one. Also, this could be changed externally and cause
*very* subtle bugs.
3. Use the file system path? It has to be unique at ingestion time, but
repo collections can move.
Instead, this patch
4. Drops rows that don't have the
| cliVersion |
| creationTime |
| language |
| sha |
columns. There are very few (16 out of 6000) and their DBs are
quesionable.
The current api (<2024-07-26 Fri>) is set up only for (owner,name). This is
insufficient for distinguishing CodeQL databases.
Other differences must be considered; this patch combines the fields
| cliVersion |
| creationTime |
| language |
| sha |
into one called CID. The CID field is a hash of these others and therefore can be
changed in the future without affecting workflows or the server.
The cid is combined with the owner/name to form one
identifier. This requires no changes to server or client -- the db
selection's interface is separate from VS Code and gh-mrva in any case.
To test this, this version imports multiple versions of the same owner/repo pairs from multiple directories. In this case, from
~/work-gh/mrva/mrva-open-source-download/repos
and
~/work-gh/mrva/mrva-open-source-download/repos-2024-04-29/
The unique database count increases from 3000 to 5360 -- see README.md,
./bin/mc-db-view-info < db-info-3.csv &
Other code modifications:
- Push (owner,repo,cid) names to minio
- Generate databases.json for use in vs code extension
- Generate list-databases.json for use by gh-mrva client
This collection already provides significant meta-information
ctime : str = '2024-05-13T12:04:01.593586'
language : str = 'cpp'
name : str = 'nanobind'
owner : str = 'wjakob'
path : Path = Path('/Users/hohn/work-gh/mrva/mrva-open-source-download/repos/wjakob/nanobind/code-scanning/codeql/databases/cpp/db.zip')
size : int = 63083064
There is some more in the db.zip files, to be added
The .csv.gz format is the simplest and most universal. It's also the smallest
on disk.
The comparison of saved/reloaded dataframe shows no difference.
The ctime_raw column caused serialization problems, so only ctime (in
iso-8601 format) is used.
Previously:
- There is confusion between nameWithOwner and queryLanguage. Both are strings.
Between
runResult, err := codeql.RunQuery(databasePath, job.QueryLanguage, queryPackPath, tempDir)
(agent.go l205)
and
func RunQuery(database string, nwo string, queryPackPath string, tempDir string) (*RunQueryResult, error)
QueryLanguage is suddenly name with owner in the code.
Added some debugging, the value is the query language in the two places it gets used:
server | 2024/07/03 18:30:15 DEBUG Processed request info location="{Data:map[bucket:packs key:1]}" language=cpp
...
agent | 2024/07/03 18:30:15 DEBUG XX: is nwo a name/owner, or the original callers' queryLanguage? nwo=cpp
...
agent | 2024/07/03 18:30:19 DEBUG XX: 2: is nwo a name/owner, or the original callers' queryLanguage? nwo=cpp
Changes:
- Introduce explicit type QueryLanguage = string and update code to clarify
- inline trivial function
