• MRVA download

MRVA download

This repository provides two scripts to download CodeQL databases given a partial url path of the form

/repos/protocolbuffers/protobuf/code-scanning/codeql/databases/cpp

The lists are from the same source as the CodeQL MRVA plugin. They are the top 1000 projects in cpp/java/python respectively.

Following https://docs.github.com/en/rest/code-scanning/code-scanning?apiVersion=2022-11-28#get-a-codeql-database-for-a-repository gives the two scripts ./bin/download and ./bin/download-all

The lists to use are external; their converted forms are in repos-*.

The download is done via

  ~/work-gh/mrva/mrva-open-source-download/
  ./bin/download-all repos-cpp && ./bin/download-all repos-java && ./bin/download-all repos-python

If you run across a rate limit problem, there will be very small DBs. Those can be found and pruned via e.g.,

  export FIND=gfind

  # Count small DBs
  $FIND repos -name db.zip -size -2k -printf "%k\t\t%p\n"  | sort -n | wc -l

  # Remove invalid downloads
  $FIND repos -name db.zip -size -2k -printf "%k\t\t%p\n"  |sort -n |\
      while read s f
      do
          rm "$f"
      done

After that, just re-run ./bin/download-all.