187 lines
4.5 KiB
Markdown
187 lines
4.5 KiB
Markdown
# Overview
|
|
|
|
TODO diagram
|
|
|
|
TODO Style notes
|
|
- NO package init() functions
|
|
- Dynamic behaviour must be explicit
|
|
|
|
|
|
## Client CodeQL Database Selector
|
|
Separate from the server's downloading of databases, a client-side interface is needed to generate the `databases.json` file. This
|
|
|
|
1. must be usable from the shell
|
|
2. must be interactive (Python, Jupyter)
|
|
3. is session based to allow iterations on selection / narrowing
|
|
4. must be queryable. There is no need to reinvent sql / dataframes
|
|
|
|
Python with dataframes is ideal for this; the project is in `client/`.
|
|
|
|
## Reverse proxy
|
|
For testing, replay flows using mitmweb. This is faster and simpler than using
|
|
gh-mrva or the VS Code plugin.
|
|
|
|
- Set up the virtual environment and install tools
|
|
|
|
python3.11 -m venv venv
|
|
source venv/bin/activate
|
|
pip install mitmproxy
|
|
|
|
For intercepting requests:
|
|
|
|
1. Start mitmproxy to listen on port 8080 and forward requests to port 8081, with
|
|
web interface
|
|
|
|
mitmweb --mode reverse:http://localhost:8081 -p 8080
|
|
|
|
1. Change `server` ports in `docker-compose.yml` to
|
|
|
|
ports:
|
|
- "8081:8080" # host:container
|
|
|
|
1. Start the containers.
|
|
|
|
1. Submit requests.
|
|
|
|
3. Save the flows for later replay.
|
|
|
|
One such session is in `tools/mitmweb-flows`; it can be loaded to replay the
|
|
requests:
|
|
|
|
1. start `mitmweb --mode reverse:http://localhost:8081 -p 8080`
|
|
2. `file` > `open` > `tools/mitmweb-flows`
|
|
3. replay at least the submit, status, and download requests
|
|
|
|
## Cross-compile server on host, run it in container
|
|
These are simple steps using a single container.
|
|
|
|
1. build server on host
|
|
|
|
GOOS=linux GOARCH=arm64 go build
|
|
|
|
2. build docker image
|
|
|
|
cd cmd/server
|
|
docker build -t server-image .
|
|
|
|
3. Start container with shared directory
|
|
|
|
```sh
|
|
docker run -it \
|
|
-v /Users/hohn/work-gh/mrva/mrvacommander:/mrva/mrvacommander \
|
|
server-image
|
|
```
|
|
|
|
4. Run server in container
|
|
|
|
cd /mrva/mrvacommander/cmd/server/ && ./server
|
|
|
|
## Using docker-compose
|
|
### Steps to build and run the server
|
|
|
|
Steps to build and run the server in a multi-container environment set up by
|
|
docker-compose.
|
|
|
|
1. Built the server-image, above
|
|
|
|
1. Build server on host
|
|
|
|
cd ~/work-gh/mrva/mrvacommander/cmd/server/
|
|
GOOS=linux GOARCH=arm64 go build
|
|
|
|
1. Start the containers
|
|
|
|
cd ~/work-gh/mrva/mrvacommander/
|
|
docker-compose down
|
|
docker-compose up -d
|
|
|
|
4. Run server in its container
|
|
|
|
cd ~/work-gh/mrva/mrvacommander/
|
|
docker exec -it server bash
|
|
cd /mrva/mrvacommander/cmd/server/
|
|
./server -loglevel=debug -mode=container
|
|
|
|
1. Test server from the host via
|
|
|
|
cd ~/work-gh/mrva/mrvacommander/tools
|
|
sh ./request_16-Jun-2024_11-33-16.curl
|
|
|
|
1. Follow server logging via
|
|
|
|
cd ~/work-gh/mrva/mrvacommander
|
|
docker-compose up -d
|
|
docker-compose logs -f server
|
|
|
|
1. Completely rebuild all containers. Useful when running into docker errors
|
|
|
|
cd ~/work-gh/mrva/mrvacommander
|
|
docker-compose up --build
|
|
|
|
1. Test server via remote client by following the steps in [gh-mrva](https://github.com/hohn/gh-mrva/blob/connection-redirect/README.org#compacted-edit-run-debug-cycle)
|
|
|
|
### Some general docker-compose commands
|
|
|
|
2. Get service status
|
|
|
|
docker-compose ps
|
|
|
|
3. Stop services
|
|
|
|
docker-compose down
|
|
|
|
4. View all logs
|
|
|
|
docker-compose logs
|
|
|
|
5. check containers from server container
|
|
|
|
docker exec -it server bash
|
|
curl -I http://rabbitmq:15672
|
|
|
|
### Use the minio ql database db
|
|
|
|
1. Web access via
|
|
|
|
open http://localhost:9001/login
|
|
|
|
username / password are in `docker-compose.yml` for now. The ql db listing
|
|
will be at
|
|
|
|
http://localhost:9001/browser/qldb
|
|
|
|
1. Populate the database by running
|
|
|
|
./populate-dbstore.sh
|
|
|
|
from the host.
|
|
|
|
1. The names in the bucket use the `owner_repo` format for now,
|
|
e.g. `google_flatbuffers_db.zip`.
|
|
TODO This will be enhanced to include other data later
|
|
|
|
1. Test Go's access to the dbstore -- from the host -- via
|
|
|
|
cd ./test
|
|
go test -v
|
|
|
|
This should produce
|
|
|
|
=== RUN TestDBListing
|
|
dbstore_test.go:44: Object Key: google_flatbuffers_db.zip
|
|
dbstore_test.go:44: Object Key: psycopg_psycopg2_db.zip
|
|
|
|
### Use the minio query pack db
|
|
|
|
1. Web access via
|
|
|
|
open http://localhost:19001/login
|
|
|
|
username / password are in `docker-compose.yml` for now. The ql db listing
|
|
will be at
|
|
|
|
http://localhost:19001/browser/qpstore
|
|
|
|
|
|
### To run Use the minio query pack db
|