mrva-docker/README.org at 55c74ae9a3a8591d5f4e10c72bd62fde4a937682

mrva/mrva-docker

Fork 0

Files

Michael Hohn 55c74ae9a3 fix: reconciled status names between server/agent/vscode-codeql

2025-03-14 12:45:36 -07:00

22 KiB

Raw Blame History

Using the Containers
Update Container Images
Project Tools

Using the Containers

Running the containers

Start the containers

  cd ~/work-gh/mrva/mrva-docker/
  docker-compose -f docker-compose-demo.yml down
  docker ps
  docker-compose -f docker-compose-demo.yml up

View all logs
```
docker-compose logs
```
Follow all logs if started with -d
```
  docker-compose logs -f
```

Follow single container, server, logging via

  cd ~/work-gh/mrva/mrvacommander
  docker-compose up -d
  docker-compose logs -f server

Cleanup in case of obscure errors (network or other)

  docker-compose -f docker-compose-demo.yml down --volumes --remove-orphans
  docker network prune
  docker-compose -f docker-compose-demo.yml up --build

Updating binaries in running container

To update the binaries in a running container – mainly during development:

server

  #* Cross-compile locally
  cd ~/work-gh/mrva/mrvaserver
  make msla

  #* check for running containers
  docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"

  #* Copy the new binary
  cd ~/work-gh/mrva/mrvaserver
  docker cp mrvaserver mrva-server:/usr/local/bin/mrvaserver

  #* Restart the binary
  docker exec mrva-server pkill mrvaserver

agent

  #* Cross-compile locally
  cd ~/work-gh/mrva/mrvaagent
  make mala

  #* Look for the agent's name in the process table
  docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"

  #* Copy the new binary
  cd ~/work-gh/mrva/mrvaagent
  docker cp mrvaagent mrva-agent:/usr/local/bin/mrvaagent

  #* Restart the binary
  docker exec mrva-agent pkill mrvaagent

Use gh-mrva container to send request via cli

Start container and check gh-mrva tool

   # Start an interactive bash shell inside the running Docker container
   docker exec -it mrva-docker-client-ghmrva-1 bash

   # Check if the gh-mrva tool is installed and accessible
   gh-mrva -h

Set up gh-mrva configuration

   # Create configuration directory and generate config file for gh-mrva
   mkdir -p ~/.config/gh-mrva
   cat > ~/.config/gh-mrva/config.yml <<eof
   # Configuration file for the gh-mrva tool
   # codeql_path: Path to the CodeQL distribution (not used in this setup)
   # controller: Placeholder for a controller NWO (not relevant in this setup)
   # list_file: Path to the repository selection JSON file

   codeql_path: not-used/codeql-path
   controller: not-used/mirva-controller
   list_file: $HOME/work-gh/mrva/gh-mrva/gh-mrva-selection.json
   eof

Create repository selection list

   # Create a directory and generate the JSON file specifying repositories
   mkdir -p ~/work-gh/mrva/gh-mrva
   cat > ~/work-gh/mrva/gh-mrva/gh-mrva-selection.json <<eof
   {
       "mirva-list": [
           "Serial-Studio/Serial-Studio",
           "UEFITool/UEFITool",
           "aircrack-ng/aircrack-ng",
           "bulk-builder/bulk-builder",
           "tesseract/tesseract"
       ]
   }
   eof

Create and submit the first query (FlatBuffersFunc.ql)

   # Generate a sample CodeQL query for functions of interest
   cat > ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql <<eof
   /**
    ,* @name pickfun
    ,* @description Pick function from FlatBuffers
    ,* @kind problem
    ,* @id cpp-flatbuffer-func
    ,* @problem.severity warning
    ,*/

   import cpp

   from Function f
   where
     f.getName() = "MakeBinaryRegion" or
     f.getName() = "microprotocols_add"
   select f, "definition of MakeBinaryRegion"
   eof

   # Submit the MRVA job with the first query
   cd ~/work-gh/mrva/gh-mrva/
   gh-mrva submit --language cpp --session mirva-session-1172 \
             --list mirva-list                                \
             --query ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql

Check status and download results for the first session

   # Check the status of the submitted session
   gh-mrva status --session mirva-session-1172

   # Download SARIF files and databases if there are results.  For the current
   # query / database combination there are zero result hence no downloads
   cd ~/work-gh/mrva/gh-mrva/
   gh-mrva download --session mirva-session-1172   \
           --download-dbs                          \
           --output-dir mirva-session-1172

Next, run a query with results

   #**  Set up QLPack for the next query
   # Create a qlpack.yml file required for the next query
   cat > ~/work-gh/mrva/gh-mrva/qlpack.yml <<eof
   library: false
   name: codeql-dataflow-ii-cpp
   version: 0.0.1
   dependencies:
     codeql/cpp-all: 0.5.3
   eof

   #**  Create and submit the second query (Fprintf.ql)
   # Generate a CodeQL query to find calls to fprintf
   cat > ~/work-gh/mrva/gh-mrva/Fprintf.ql <<eof
   /**
    ,* @name findPrintf
    ,* @description Find calls to plain fprintf
    ,* @kind problem
    ,* @id cpp-fprintf-call
    ,* @problem.severity warning
    ,*/

   import cpp

   from FunctionCall fc
   where
     fc.getTarget().getName() = "fprintf"
   select fc, "call of fprintf"
   eof

   # Submit a new MRVA job with the second query
   cd ~/work-gh/mrva/gh-mrva/
   gh-mrva submit                                      \
           --language cpp --session mirva-session-1861 \
           --list mirva-list                           \
           --query ~/work-gh/mrva/gh-mrva/Fprintf.ql

Check status and download results for the second session

   # Check the status of the second session
   gh-mrva status --session mirva-session-1861

   # Download SARIF files and databases for the second query
   cd ~/work-gh/mrva/gh-mrva/
   gh-mrva download --session mirva-session-1861   \
           --download-dbs                          \
           --output-dir mirva-session-1861

As shell script

  #*  Start container and check gh-mrva tool
  # Start an interactive bash shell inside the running Docker container
  docker exec -it mrva-docker-client-ghmrva-1 bash

  # Check if the gh-mrva tool is installed and accessible
  gh-mrva -h

  #*  Set up gh-mrva configuration
  # Create configuration directory and generate config file for gh-mrva
  mkdir -p ~/.config/gh-mrva
  cat > ~/.config/gh-mrva/config.yml <<eof
  # Configuration file for the gh-mrva tool
  # codeql_path: Path to the CodeQL distribution (not used in this setup)
  # controller: Placeholder for a controller NWO (not relevant in this setup)
  # list_file: Path to the repository selection JSON file

  codeql_path: not-used/codeql-path
  controller: not-used/mirva-controller
  list_file: $HOME/work-gh/mrva/gh-mrva/gh-mrva-selection.json
  eof

  #*  Create repository selection list
  # Create a directory and generate the JSON file specifying repositories
  mkdir -p ~/work-gh/mrva/gh-mrva
  cat > ~/work-gh/mrva/gh-mrva/gh-mrva-selection.json <<eof
  {
      "mirva-list": [
          "Serial-Studio/Serial-Studio",
          "UEFITool/UEFITool",
          "aircrack-ng/aircrack-ng",
          "bulk-builder/bulk-builder",
          "tesseract/tesseract"
      ]
  }
  eof

  #*  Create and submit the first query (FlatBuffersFunc.ql)
  # Generate a sample CodeQL query for functions of interest
  cat > ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql <<eof
  /**
   ,* @name pickfun
   ,* @description Pick function from FlatBuffers
   ,* @kind problem
   ,* @id cpp-flatbuffer-func
   ,* @problem.severity warning
   ,*/

  import cpp

  from Function f
  where
    f.getName() = "MakeBinaryRegion" or
    f.getName() = "microprotocols_add"
  select f, "definition of MakeBinaryRegion"
  eof

  # Submit the MRVA job with the first query
  cd ~/work-gh/mrva/gh-mrva/
  gh-mrva submit --language cpp --session mirva-session-1172 \
            --list mirva-list                                \
            --query ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql

Send request via gui, using vs code

The following sequence works when run from a local vs code with the custom codeql plugin.

Connect to vscode-codeql container at http://localhost:9080/?folder=/home/coder

Provide settings

The file

/home/coder/.local/share/code-server/User/settings.json

  cat > /home/coder/.local/share/code-server/User/settings.json << EOF
  {
      "codeQL.runningQueries.numberOfThreads": 2,
      "codeQL.cli.executablePath": "/opt/codeql/codeql",

      "codeQL.variantAnalysis.enableGhecDr": true,
      "github-enterprise.uri": "http://server:8080/"
  }
  EOF

Provide list of repositories to analyze

ql tab > variant analysis repositories > {}, put this into databases.json

  {
      "version": 1,
      "databases": {
          "variantAnalysis": {
              "repositoryLists": [
                  {
                      "name": "mrva-list",
                      "repositories": [
                          "Serial-Studio/Serial-Studio",
                          "UEFITool/UEFITool",
                          "aircrack-ng/aircrack-ng",
                          "bulk-builder/bulk-builder",
                          "tesseract/tesseract"
                      ]
                  }
              ],
              "owners": [],
              "repositories": []
          }
      },
      "selected": {
          "kind": "variantAnalysisUserDefinedList",
          "listName": "mirva-list"
      }
  }

Make the list current

ql tab > variant analysis repositories > 'select' mrva-list

Provide a query

Select file qldemo/simple.ql and put Fprintf.ql parallel to it:

  cat > /home/coder/qldemo/Fprintf.ql <<eof
  /**
   ,* @name findPrintf
   ,* @description find calls to plain fprintf
   ,* @kind problem
   ,* @id cpp-fprintf-call
   ,* @problem.severity warning
   ,*/

  import cpp

  from FunctionCall fc
  where
    fc.getTarget().getName() = "fprintf"
  select fc, "call of fprintf"
  eof

  /**
   ,* @name findPrintf
   ,* @description find calls to plain fprintf
   ,* @kind problem
   ,* @id cpp-fprintf-call
   ,* @problem.severity warning
   ,*/

  import cpp

  from FunctionCall fc
  where
    fc.getTarget().getName() = "fprintf"
  select fc, "call of fprintf"

Provide the qlpack specification

Create qlpack.yml for cpp:

  cat > /home/coder/qldemo/qlpack.yml <<eof
  library: false
  name: codeql-dataflow-ii-cpp
  version: 0.0.1
  dependencies:
    codeql/cpp-all: 0.5.3
  eof

Then

Delete qlpack.lock file

In shell,

  cd ~/qldemo
  /opt/codeql/codeql pack install

In GUI, 'install pack dependencies'
In GUI, 'reload windows'

Submit the analysis job

Fprintf.ql > right click > run variant analysis

XX: post-interface reload <2025-03-12 Wed>

  No orphaned databases found.
  Reading query history
  Reading cached query history from '/home/coder/.local/share/code-server/User/workspaceStorage/579340b4/GitHub.vscode-codeql/workspace-query-history.json'.
  Successfully finished extension initialization.
  Variant analysis view loaded

XX: bug: <2025-02-19 Wed>

  +lle1Oo44XUE87h82ShEVmzTAe02Csf9HKt7kLw2xPU0mt3mYNtbaFEsO3HQNh3tWNS7B+lFoaLw/9t2t9r1//h4P/z1P77+fP35+vP15xb//D9DrYDfAKi6AA==","repositories":["Serial-Studio/Serial-Studio","UEFITool/UEFITool","aircrack-ng/aircrack-ng","bulk-builder/bulk-builder","tesseract/tesseract"]}
  Fetch request URL: http://server:8080/repositories/0/code-scanning/codeql/variant-analyses
  Response status: 200
  Response data: [object Object]
  Variant analysis findPrintf submitted for processing
  Variant analysis view loaded
  Unhandled error: Error saving query history to /home/coder/.local/share/code-server/User/workspaceStorage/579340b4/GitHub.vscode-codeql/workspace-query-history.json: Internal error: exhaustivity checking failure
  Error: Error saving query history to /home/coder/.local/share/code-server/User/workspaceStorage/579340b4/GitHub.vscode-codeql/workspace-query-history.json: Internal error: exhaustivity checking failure
      at writeQueryHistoryToFile (/home/coder/.local/share/code-server/extensions/github.vscode-codeql-1.13.2-dev.2024.12.10.23.51.57/out/extension.js:115146:11)
      at QueryHistoryManager.writeQueryHistory (/home/coder/.local/share/code-server/extensions/github.vscode-codeql-1.13.2-dev.2024.12.10.23.51.57/out/extension.js:115818:5)
      at QueryHistoryManager.refreshTreeView (/home/coder/.local/share/code-server/extensions/github.vscode-codeql-1.13.2-dev.2024.12.10.23.51.57/out/extension.js:116309:5)
      at n.value (/home/coder/.local/share/code-server/extensions/github.vscode-codeql-1.13.2-dev.2024.12.10.23.51.57/out/extension.js:115770:11)

After adding console.logs to extension.js, restart container so they take effect:

docker compose -f docker-compose-demo.yml restart code-server

This plugin is literally retarded. It reports this error as fatal, but all the UI functionality is present…

monkey-patch the writeQueryHistoryToFile() function? ~/work-gh/mrva/vscode-codeql/extensions/ql-vscode/src/query-history/store/query-history-store.ts::import { pathExists, remove, mkdir, writeFile, readJson } from "fs-extra";

  . Edit the assertNever Function
  Find the function assertNever in the extension files.
  It's likely in:

  swift
  Copy
  Edit
  /home/coder/.local/share/code-server/extensions/github.vscode-codeql-*/out/extension.js
  Modify:

  js
  Copy
  Edit
  export function assertNever(value: never): never {
    throw new ExhaustivityCheckingError(value);
  }
  to:

  js
  Copy
  Edit
  export function assertNever(value: never): never {
    console.warn("Bypassing assertNever for value:", value);
    return undefined as never;
  }
  This will log the issue instead of crashing.

XX: bug:

  Could not download the results for variant analysis with id: 1. Error: request to
  http://localhost:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoidGVzc2VyYWN0IiwiUmVwbyI6InRlc3NlcmFjdCJ9
  failed, reason: connect ECONNREFUSED 127.0.0.1:8080
  (codeQL.autoDownloadVariantAnalysisResult) Error: Could not download the results
  for variant analysis with id: 1. Error: request to
  http://localhost:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoidGVzc2VyYWN0IiwiUmVwbyI6InRlc3NlcmFjdCJ9
  failed, reason: connect ECONNREFUSED 127.0.0.1:8080

  #* Try download from gh-mrva: works
  docker exec -it mrva-docker-client-ghmrva-1 bash

  curl http://localhost:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0= 
  root@66ca60f7ce3b:/app# curl http://localhost:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0=
  Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your terminal anyway, or
  Warning: consider "--output <FILE>" to save to a file.

  curl http://server:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0= 
  root@66ca60f7ce3b:/app# curl http://server:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0=
  Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your terminal anyway, or
  Warning: consider "--output <FILE>" to save to a file.

  #* Try download from vs-codeql: works for http://server:8080
  docker exec -it mrva-docker-code-server-1 bash

  curl http://localhost:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0= 
  coder@924c44460749:~$ curl http://localhost:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0=
  curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server

  curl http://server:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0= 
  coder@924c44460749:~$ curl http://server:8080/download/eyJTZXNzaW9uSUQiOjEsIk93bmVyIjoiU2VyaWFsLVN0dWRpbyIsIlJlcG8iOiJTZXJpYWwtU3R1ZGlvIn0=
  Warning: Binary output can mess up your terminal. Use "--output -" to tell
  Warning: curl to output it to your terminal anyway, or consider "--output
  Warning: <FILE>" to save to a file.

XX: ?fixed? via

SERVER_HOST=server
SERVER_PORT=8080

But have to go through the sequence again:

restart
patch binaries

Update Container Images

XX:

grep 'docker tag' containers/*/*.org containers/*/Makefile
(grep "grep --color=auto -nH --null -e 'docker tag' containers/*/*")

  # To snapshot a running Docker container and create a new image from it, use the
  # following CLI sequence: 

  #* Get the container IDs

  docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"
  # 0:$ docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"
  # CONTAINER ID   IMAGE                                         NAMES
  # 99de2a875317   ghcr.io/hohn/client-ghmrva-container:0.1.24   mrva-docker-client-ghmrva-1
  # 081900278c0e   ghcr.io/hohn/mrva-server:0.1.24               server
  # a23352c009fb   ghcr.io/hohn/mrva-agent:0.1.24                agent
  # 9e9248a77957   minio/minio:RELEASE.2024-06-11T03-13-30Z      dbstore
  # cd043e5bad77   ghcr.io/hohn/code-server-initialized:0.1.24   mrva-docker-code-server-1
  # 783e30d6f9d0   rabbitmq:3-management                         rabbitmq
  # d05f606b4ea0   ghcr.io/hohn/mrva-hepc-container:0.1.24       hepc
  # 7858ccf18fad   ghcr.io/hohn/dbsdata-container:0.1.24         dbssvc
  # 85d85484849b   minio/minio:RELEASE.2024-06-11T03-13-30Z      artifactstore

  #* Commit the running containers to new images
  # Commit the running container to a new image:
  ( cd ~/work-gh/mrva/mrva-docker/ && rg 'docker (commit)' )

  docker commit 99de2a875317 mrva-client-ghmrva:0.2.0 
  # sha256:2eadb76a6b051200eaa395d2f815bad63f88473a16aa4c0a6cdebb114c556498

  docker commit 081900278c0e   mrva-server-server:0.2.0
  # sha256:0ec38b245021b0aea2c31eab8f75a9141cce8ee789e406cec4dabac484e03aff

  docker commit a23352c009fb   mrva-server-agent:0.2.0
  # sha256:75c6dee1dc57cda571482f7fdb2d3dd292f51e423c1733071927f21f3ab0cec5

  docker commit cd043e5bad77   mrva-client-vscode:0.2.0
  # sha256:b239d13f44637cac3601697dca49325faf123be8cf040c05b6dafe2b11504cc8

  docker commit d05f606b4ea0   mrva-server-hepc:0.2.0
  # sha256:238d39313590837587b7bd235bdfe749e18417b38e046553059295cf2064e0d2

  docker commit 7858ccf18fad   mrva-server-dbsdata:0.2.0
  # sha256:a283d69e6f9ba03856178149de95908dd6fa4b6a8cf407a1464d6cec5fa5fdc0

  #* Verify the newly created images
  docker images

  #* Tag the images for a registry
  ( cd ~/work-gh/mrva/mrva-docker/ && rg 'docker (tag)' )

  tagpushimg () {
      name=$1
      version=$2

      docker tag $name:$version ghcr.io/hohn/$name:$version
      docker push ghcr.io/hohn/$name:$version
  }

  tagpushimg mrva-client-ghmrva 0.2.0

  tagpushimg mrva-server-server 0.2.0

  tagpushimg mrva-server-agent 0.2.0

  tagpushimg mrva-client-vscode 0.2.0

  tagpushimg mrva-server-hepc 0.2.0

  tagpushimg mrva-server-dbsdata 0.2.0

view container image list on ghcr.io: https://github.com/hohn?tab=packages

Project Tools

This project, mrva-docker, is the highest-level part of the project as it packages all others. So it also houses simple project tools.

  # On macos

  # install uv
  curl -LsSf https://astral.sh/uv/install.sh | sh
  uv self update

  # set up mrva-env
  cd ~/work-gh/mrva/mrva-docker
  uv venv mrva-env

  # activate mrva-env
  source mrva-env/bin/activate

  # link scripts (lazy 'install')
  cd  mrva-env/bin/
  ln -s ../../bin/* .

22 KiB Raw Blame History Unescape Escape