b7b4839fe0
Previously, the refined info was collected and the CID computed before saving. This was a major development time sink, so the CID is now computed in the following step (bin/mc-db-unique). The columns previously chosen for the CID are not enough. If these columns are empty for any reason, the CID repeats. Just including the owner/name won't help, because those are duplicates. Some possibilities considered and rejected: 1. Could use a random number for missing columns. But this makes the CID nondeterministic. 2. Switch to the file system ctime? Not unique across owner/repo pairs, but unique within one. Also, this could be changed externally and cause *very* subtle bugs. 3. Use the file system path? It has to be unique at ingestion time, but repo collections can move. Instead, this patch 4. Drops rows that don't have the | cliVersion | | creationTime | | language | | sha | columns. There are very few (16 out of 6000) and their DBs are quesionable.
54 lines
1.5 KiB
Python
Executable File
54 lines
1.5 KiB
Python
Executable File
#!/usr/bin/env python
|
|
""" Read an initial table of CodeQL DB information, produced by
|
|
mc-db-initial-info, and collect more detailed information from the database
|
|
files. Write out an extended table in CSV format.
|
|
"""
|
|
import qldbtools.utils as utils
|
|
import argparse
|
|
import logging
|
|
import pandas as pd
|
|
import sys
|
|
|
|
#
|
|
#* Configure logger
|
|
#
|
|
logging.basicConfig(format='%(asctime)s %(message)s')
|
|
|
|
#
|
|
#* Process command line
|
|
#
|
|
parser = argparse.ArgumentParser(
|
|
description="""Read an initial table of CodeQL DB information, produced by
|
|
mc-db-initial-info, and collect more detailed information from the database
|
|
files. Write out an extended table in CSV format. """)
|
|
args = parser.parse_args()
|
|
|
|
#
|
|
#* Collect the information
|
|
# This step is time-intensive so we save the results right after.
|
|
d = pd.read_csv(sys.stdin)
|
|
joiners = []
|
|
for left_index in range(0, len(d)-1):
|
|
try:
|
|
cqlc, metac = utils.extract_metadata(d.path[left_index])
|
|
except utils.ExtractNotZipfile:
|
|
continue
|
|
except utils.ExtractNoCQLDB:
|
|
continue
|
|
try:
|
|
detail_df = utils.metadata_details(left_index, cqlc, metac)
|
|
except utils.DetailsMissing:
|
|
continue
|
|
joiners.append(detail_df)
|
|
joiners_df = pd.concat(joiners, axis=0)
|
|
full_df = pd.merge(d, joiners_df, left_index=True, right_on='left_index', how='outer')
|
|
|
|
#
|
|
#* Save results
|
|
#
|
|
full_df.to_csv(sys.stdout, index=False)
|
|
|
|
# Local Variables:
|
|
# python-shell-virtualenv-root: "~/work-gh/mrva/mrvacommander/client/qldbtools/venv/"
|
|
# End:
|