Previously, the refined info was collected and the CID computed before saving.
This was a major development time sink, so the CID is now computed in the
following step (bin/mc-db-unique).
The columns previously chosen for the CID are not enough. If these columns are
empty for any reason, the CID repeats. Just including the owner/name won't help,
because those are duplicates.
Some possibilities considered and rejected:
1. Could use a random number for missing columns. But this makes
the CID nondeterministic.
2. Switch to the file system ctime? Not unique across owner/repo pairs,
but unique within one. Also, this could be changed externally and cause
*very* subtle bugs.
3. Use the file system path? It has to be unique at ingestion time, but
repo collections can move.
Instead, this patch
4. Drops rows that don't have the
| cliVersion |
| creationTime |
| language |
| sha |
columns. There are very few (16 out of 6000) and their DBs are
quesionable.
The current api (<2024-07-26 Fri>) is set up only for (owner,name). This is
insufficient for distinguishing CodeQL databases.
Other differences must be considered; this patch combines the fields
| cliVersion |
| creationTime |
| language |
| sha |
into one called CID. The CID field is a hash of these others and therefore can be
changed in the future without affecting workflows or the server.
The cid is combined with the owner/name to form one
identifier. This requires no changes to server or client -- the db
selection's interface is separate from VS Code and gh-mrva in any case.
To test this, this version imports multiple versions of the same owner/repo pairs from multiple directories. In this case, from
~/work-gh/mrva/mrva-open-source-download/repos
and
~/work-gh/mrva/mrva-open-source-download/repos-2024-04-29/
The unique database count increases from 3000 to 5360 -- see README.md,
./bin/mc-db-view-info < db-info-3.csv &
Other code modifications:
- Push (owner,repo,cid) names to minio
- Generate databases.json for use in vs code extension
- Generate list-databases.json for use by gh-mrva client
