mrva/mrvacommander
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
149 Commits 3 Branches 4 Tags
7de3ee59cecb39be36d84fffec4a5fc9fe1c79e6
Commit Graph

149 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Hohn
26dd69c976 minor doc update 2024-07-23 15:18:32 -07:00
Michael Hohn
731b44b187 Add scripts for automatic codeql db data and metadata collection 
- updated instructions
- cli scripts mirror the interactive session*.py files
 2024-07-23 15:05:03 -07:00
Michael Hohn
aaeafa9e88 Automate metadata collection for all DBs 
Several errors are handled; on extraction
    ExtractNotZipfile:
    ExtractNoCQLDB:

On detail extraction
    DetailsMissing:
 2024-07-22 19:12:12 -07:00
Michael Hohn
129b8cc302 interim: collect metadata from one DB zip file 2024-07-22 12:54:57 -07:00
Michael Hohn
d64522d168 Collect CodeQL database information from the file system and save as CSV 
This collection already provides significant meta-information

    ctime : str = '2024-05-13T12:04:01.593586'
    language : str = 'cpp'
    name : str = 'nanobind'
    owner : str = 'wjakob'
    path : Path = Path('/Users/hohn/work-gh/mrva/mrva-open-source-download/repos/wjakob/nanobind/code-scanning/codeql/databases/cpp/db.zip')
    size : int = 63083064

There is some more in the db.zip files, to be added
 2024-07-22 11:07:00 -07:00
Michael Hohn
6b4e753e69 Experiment with formats for saving/loading the database index 
The .csv.gz format is the simplest and most universal.  It's also the smallest
on disk.
The comparison of saved/reloaded dataframe shows no difference.
The ctime_raw column caused serialization problems, so only ctime (in
iso-8601 format) is used.
 2024-07-12 14:41:05 -07:00
Michael Hohn
3df1cac5ae Clean up package info 2024-07-10 15:38:59 -07:00
Michael Hohn
dcc32ea8ab Add documentation style sheet and Makefile entry 2024-07-10 15:27:09 -07:00
Michael Hohn
3c8db9cbe4 Put the DB code into a package 2024-07-10 15:04:09 -07:00
Michael Hohn
be1304bdd9 Add to .dockerignore to reduce build time 2024-07-10 13:23:08 -07:00
Michael Hohn
8965725e42 Replaced the dynamic table type ArtifactLocation with struct keys 
The original is present in comment form for reference
 2024-07-10 13:08:40 -07:00
Michael Hohn
2df48b9f98 Collect DB information from file system and render it 2024-07-10 09:11:21 -07:00
Michael Hohn
8d80272922 Add client/ setup and plan 2024-07-09 10:37:41 -07:00
Michael Hohn
e3f4d9f012 Use QL_DB_BUCKET_NAME in shell and go 2024-07-08 14:23:25 -07:00
Michael Hohn
3566f5169e Type checking fix: Restrict the keys / values for ArtifactLocation and centralize the common ones 2024-07-08 12:07:46 -07:00
Michael Hohn
b3cf7a4f65 Introduce explicit type QueryLanguage = string and update code to clarify 
Previously:
- There is confusion between nameWithOwner and queryLanguage.  Both are strings.
  Between

        runResult, err := codeql.RunQuery(databasePath, job.QueryLanguage, queryPackPath, tempDir)
    (agent.go l205)

  and

        func RunQuery(database string, nwo string, queryPackPath string, tempDir string) (*RunQueryResult, error)

  QueryLanguage is suddenly name with owner in the code.

  Added some debugging, the value is the query language in the two places it gets used:

        server         | 2024/07/03 18:30:15 DEBUG Processed request info location="{Data:map[bucket:packs key:1]}" language=cpp
        ...
        agent          | 2024/07/03 18:30:15 DEBUG XX: is nwo a name/owner, or the original callers' queryLanguage? nwo=cpp
        ...
        agent          | 2024/07/03 18:30:19 DEBUG XX: 2: is nwo a name/owner, or the original callers' queryLanguage? nwo=cpp

Changes:
- Introduce explicit type QueryLanguage = string and update code to clarify
- inline trivial function
 2024-07-03 13:30:02 -07:00
Michael Hohn
07f93f3d27 Tested zero-length repository list 
By submitting a non-existent target repositories.
 2024-07-03 08:55:27 -07:00
Michael Hohn
7413e23bab Add reverse proxy and configuration for replay capability 2024-07-01 12:03:24 -07:00
Michael Hohn
380e90135a Add the submitEmptyStatusResponse special case 2024-07-01 10:54:46 -07:00
Michael Hohn
1642894ccf Added note about querypackurl 2024-06-27 14:53:52 -07:00
Michael Hohn
c54bda8432 fix regression from 0cffb3c8 2024-06-27 14:22:52 -07:00
Michael Hohn
17bf9049e4 Add note about docker-compose force rebuild 2024-06-27 11:59:34 -07:00
Michael Hohn
62a7b227f0 Add 'revive' linter setup 2024-06-26 10:35:10 -07:00
Michael Hohn
b543cebfac Add golangci linter setup 2024-06-26 10:21:52 -07:00
Michael Hohn
d145731c4b WIP: marked special case of 0 jobs 2024-06-26 09:27:27 -07:00
Michael Hohn
0cffb3c849 Simplify struct SessionInfo and adjoining code 2024-06-25 18:57:27 -07:00
Michael Hohn
9d1a891c72 Add to README, add Makefile with development targets 2024-06-25 12:04:46 -07:00
Nicolas Will
b4d9833da3 Resolve status logic error and refactor server.go 2024-06-24 22:31:19 -04:00
Nicolas Will
e0cbc01d21 Fully implement local and container MRVA 2024-06-24 01:31:28 -04:00
Nicolas Will
ef7552c43f Update .gitignore 2024-06-17 13:16:06 +02:00
Nicolas Will
50da8eefe8 Fix ZipSlip vuln and integer conversion issue 2024-06-17 12:55:28 +02:00
Nicolas Will
fc9fcc7ae6 Add server queue logic and refactor 2024-06-17 11:30:46 +02:00
Michael Hohn
8b310e43ad Fix storage modules types and interfaces to compile server 2024-06-16 20:16:26 -07:00
Michael Hohn
6229c08900 Remove postgres and references to it 2024-06-16 19:43:29 -07:00
Michael Hohn
b756668e70 Fix merge so server compiles 2024-06-16 19:36:31 -07:00
Michael Hohn
2c5ecd3a1e Merge the agent-impl branch into the server branch 2024-06-16 19:21:42 -07:00
Michael Hohn
3be3229ecd Remove duplicate test DB 2024-06-16 19:09:32 -07:00
Michael Hohn
335a61c28e For testing: Add request capture script. Replay via curl 2024-06-16 19:09:32 -07:00
Michael Hohn
cd0647836e Combine New/Setup functions 2024-06-16 19:09:32 -07:00
Michael Hohn
f611f02d1c try git lfs again 2024-06-16 19:09:31 -07:00
Michael Hohn
252859ae18 Add Go access test for the DB store and update README 2024-06-16 19:09:31 -07:00
Michael Hohn
9270230f08 Set up separate minio container for query pack storage 2024-06-16 19:09:31 -07:00
Michael Hohn
7347779072 Change container name to dbstore 2024-06-16 19:09:31 -07:00
Michael Hohn
bbb54caaf3 Set up the minio ql db 
- Set up the minio ql db
- Use bind mounts in docker compose to get persistence
- Populate the ql db with the sample DBs
 2024-06-16 19:09:31 -07:00
Michael Hohn
8df9673897 wip: Mark update slots with XX:, add pkg/server/container.go 2024-06-16 19:09:30 -07:00
Nicolas Will
903ca5673e Add dynamic worker management 2024-06-16 17:07:13 +02:00
Nicolas Will
7ea45cb176 Separate queue and agent logic and refactor 2024-06-16 11:18:22 +02:00
Nicolas Will
e107f6cf80 Fix ENV CODEQL_JAVA_HOME in Dockerfile 2024-06-15 23:10:37 +02:00
Nicolas Will
1a574c2f7f Add RabbitMQ connect retry and healthcheck 2024-06-15 00:39:21 +02:00
Nicolas Will
ea10403f6c Bump server Ubuntu version to 24.10 2024-06-15 00:23:30 +02:00