d956f47db3
The problem was missing fields in the SARIF output. After the debugging
below, the cause were conversions from JSON to Go to JSON; the Go ->
JSON conversion only output the fields defined in the Go struct.
Because SARIF has so many optional fields, no attempt is made to enforce
a statically-defined structure. Instead, the JSON -> Go conversion is
now to a fully dynamic structure; unused fields are simply passed through
Debugging:
Comparing two SARIF files shows
{
"$schema" : "https://json.schemastore.org/sarif-2.1.0.json",
"version" : "2.1.0",
"runs" : [ {...
} ]
}
and
{
"runs": [...
]
}
so there are missing fields.
The Problem
1. Problem origin
// Modify the sarif: start by extracting
var sarif Sarif
if err := json.Unmarshal(sarifData, &sarif); err != nil {
return nil, fmt.Errorf("failed to unmarshal SARIF: %v", err)
}
...
// now inject version control info
...
// and write it back
sarifBytes, err := json.Marshal(sarif)
if err != nil {
return nil, fmt.Errorf("failed to marshal SARIF: %v", err)
}
2. But the struct only has one of the needed fields
type Sarif struct {
Runs []SarifRun `json:"runs"`
}
3. From the docs:
// To unmarshal JSON into a struct, Unmarshal matches incoming object
// keys to the keys used by [Marshal] (either the struct field name or its tag),
// preferring an exact match but also accepting a case-insensitive match. By
// default, object keys which don't have a corresponding struct field are
// ignored (see [Decoder.DisallowUnknownFields] for an alternative).
Overview
TODO diagram
TODO Style notes
- NO package init() functions
- Dynamic behaviour must be explicit
Client CodeQL Database Selector
Separate from the server's downloading of databases, a client-side interface is needed to generate the databases.json file. This
- must be usable from the shell
- must be interactive (Python, Jupyter)
- is session based to allow iterations on selection / narrowing
- must be queryable. There is no need to reinvent sql / dataframes
Python with dataframes is ideal for this; the project is in client/.
Reverse proxy
For testing, replay flows using mitmweb. This is faster and simpler than using gh-mrva or the VS Code plugin.
-
Set up the virtual environment and install tools
python3.11 -m venv venv source venv/bin/activate pip install mitmproxy
For intercepting requests:
-
Start mitmproxy to listen on port 8080 and forward requests to port 8081, with web interface
mitmweb --mode reverse:http://localhost:8081 -p 8080 -
Change
serverports indocker-compose.ymltoports: - "8081:8080" # host:container -
Start the containers.
-
Submit requests.
-
Save the flows for later replay.
One such session is in tools/mitmweb-flows; it can be loaded to replay the
requests:
- start
mitmweb --mode reverse:http://localhost:8081 -p 8080 file>open>tools/mitmweb-flows- replay at least the submit, status, and download requests
Cross-compile server on host, run it in container
These are simple steps using a single container.
-
build server on host
GOOS=linux GOARCH=arm64 go build -
build docker image
cd cmd/server docker build -t server-image . -
Start container with shared directory
docker run -it \ -v /Users/hohn/work-gh/mrva/mrvacommander:/mrva/mrvacommander \ server-image -
Run server in container
cd /mrva/mrvacommander/cmd/server/ && ./server
Using docker-compose
Steps to build and run the server
Steps to build and run the server in a multi-container environment set up by docker-compose.
-
Built the server-image, above
-
Build server on host
cd ~/work-gh/mrva/mrvacommander/cmd/server/ GOOS=linux GOARCH=arm64 go build -
Start the containers
cd ~/work-gh/mrva/mrvacommander/ docker-compose down docker-compose up -d -
Run server in its container
cd ~/work-gh/mrva/mrvacommander/ docker exec -it server bash cd /mrva/mrvacommander/cmd/server/ ./server -loglevel=debug -mode=container -
Test server from the host via
cd ~/work-gh/mrva/mrvacommander/tools sh ./request_16-Jun-2024_11-33-16.curl -
Follow server logging via
cd ~/work-gh/mrva/mrvacommander docker-compose up -d docker-compose logs -f server -
Completely rebuild all containers. Useful when running into docker errors
cd ~/work-gh/mrva/mrvacommander docker-compose up --build -
Test server via remote client by following the steps in gh-mrva
Some general docker-compose commands
-
Get service status
docker-compose ps -
Stop services
docker-compose down -
View all logs
docker-compose logs -
check containers from server container
docker exec -it server bash curl -I http://rabbitmq:15672
Use the minio ql database db
-
Web access via
open http://localhost:9001/loginusername / password are in
docker-compose.ymlfor now. The ql db listing will be athttp://localhost:9001/browser/qldb -
Populate the database by running
./populate-dbstore.shfrom the host.
-
The names in the bucket use the
owner_repoformat for now, e.g.google_flatbuffers_db.zip. TODO This will be enhanced to include other data later -
Test Go's access to the dbstore -- from the host -- via
cd ./test go test -vThis should produce
=== RUN TestDBListing dbstore_test.go:44: Object Key: google_flatbuffers_db.zip dbstore_test.go:44: Object Key: psycopg_psycopg2_db.zip
Use the minio query pack db
-
Web access via
open http://localhost:19001/loginusername / password are in
docker-compose.ymlfor now. The ql db listing will be athttp://localhost:19001/browser/qpstore