Move tree-sitter-extrator project to ruby/extractor/codeql-extractor.

I investigated porting our ruby extractor build to bazel, and unfortunately,
bazels `rules_rust` only support local dependencies inside a workspace,
c.f. https://github.com/bazelbuild/rules_rust/issues/1525.

Therefore, to be able to build the ruby extractor with bazel,
I had to move the `tree-sitter-extractor` project out of `shared/`, where
it morally belongs, into the ruby extractor workspace.
Alternatives considered:
- Provide a symlink from inside the workspace into `shared/`. This is possible,
  but due to the poor symlink support on Windows, it'd make working with
  the entire codeql repo on Windows more difficult.
  Our internal developers all (have to) have developer mode enabled, so that
  we can use symlinks in Bazel, but all external contributers on Windows would
  presumably run into trouble while cloning the repo. Imo, that's not acceptable.
- Fake an environment that looks like it's a workspace using advanced
  bazel magic. That's possible, but would make the build much harder to understand.
  In particular, bazel doesn't (easily) allow writing to the source tree,
  so this is not an easy/quick hack. It's important to me that the build system
  can be understood by non-bazel experts, so this isn't acceptable either.
- Fix the upstream issue. I don't have a good enough understanding of the issue
  or the code to do that.
- Drop the `Cargo.toml` for the ruby extractor, and model all dependencies only
  on the bazel side. This'd break IDE support, and the bazel<->cargo integration
  is actually quite good (besides its limitations), so I don't want to do that.

Out of all these alternatives, moving the shared code into the ruby workspace
seemed the least bad option.
Note that this works, because we want to build the ruby language pack with bazel,
but not the ql-for-ql one - for now, I don't see an advantage of porting that to
bazel anyways, so it's okay that it's blocked from being ported to bazel right now.

I'm happy to discuss this change and alternatives further.
If we agree that this is the way forward, then this needs to be merged together
with a simple corresponding change in the internal repo.

.github/workflows/ql-for-ql-build.yml

@@ -32,7 +32,7 @@ jobs:
           path: |
             ql/extractor-pack/
             ql/target/release/buramu
-          key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-extractor-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('shared/tree-sitter-extractor') }}-${{ hashFiles('ql/**/*.rs') }}
+          key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-extractor-${{ hashFiles('ql/**/Cargo.lock') }}-${{ hashFiles('ruby/extractor/codeql-extractor') }}-${{ hashFiles('ql/**/*.rs') }}
       - name: Cache cargo
         if: steps.cache-extractor.outputs.cache-hit != 'true'
         uses: actions/cache@v3
@@ -44,20 +44,20 @@ jobs:
           key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-rust-cargo-${{ hashFiles('ql/**/Cargo.lock') }}
       - name: Release build
         if: steps.cache-extractor.outputs.cache-hit != 'true'
-        run: cd ql; ./scripts/create-extractor-pack.sh       
+        run: cd ql; ./scripts/create-extractor-pack.sh
         env:
-          GH_TOKEN: ${{ github.token }}   
+          GH_TOKEN: ${{ github.token }}
       - name: Cache compilation cache
         id: query-cache
         uses: ./.github/actions/cache-query-compilation
-        with: 
+        with:
           key: run-ql-for-ql
       - name: Make database and analyze
         run: |
           ./ql/target/release/buramu | tee deprecated.blame # Add a blame file for the extractor to parse.
           ${CODEQL} database create -l=ql --search-path ql/extractor-pack ${DB}
           ${CODEQL} database analyze -j0 --format=sarif-latest --output=ql-for-ql.sarif ${DB} ql/ql/src/codeql-suites/ql-code-scanning.qls  --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}"
-        env: 
+        env:
           CODEQL: ${{ steps.find-codeql.outputs.codeql-path }}
           DB: ${{ runner.temp }}/DB
           LGTM_INDEX_FILTERS: |

.github/workflows/ruby-build.yml

@@ -61,7 +61,7 @@ jobs:
             ruby/extractor/target/release/codeql-extractor-ruby
             ruby/extractor/target/release/codeql-extractor-ruby.exe
             ruby/extractor/ql/lib/codeql/ruby/ast/internal/TreeSitter.qll
-          key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-ruby-extractor-${{ hashFiles('ruby/extractor/rust-toolchain.toml', 'ruby/extractor/Cargo.lock') }}-${{ hashFiles('shared/tree-sitter-extractor') }}-${{ hashFiles('ruby/extractor/**/*.rs') }}
+          key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-ruby-extractor-${{ hashFiles('ruby/extractor/rust-toolchain.toml', 'ruby/extractor/Cargo.lock') }}-${{ hashFiles('ruby/extractor/codeql-extractor') }}-${{ hashFiles('ruby/extractor/**/*.rs') }}
       - uses: actions/cache@v3
         if: steps.cache-extractor.outputs.cache-hit != 'true'
         with:
@@ -119,7 +119,7 @@ jobs:
       - name: Cache compilation cache
         id: query-cache
         uses: ./.github/actions/cache-query-compilation
-        with: 
+        with:
           key: ruby-build
       - name: Build Query Pack
         run: |

.github/workflows/tree-sitter-extractor-test.yml

@@ -3,14 +3,14 @@ name: Test tree-sitter-extractor
 on:
   push:
     paths:
-      - "shared/tree-sitter-extractor/**"
+      - "ruby/extractor/codeql-extractor/**"
       - .github/workflows/tree-sitter-extractor-test.yml
     branches:
       - main
       - "rc/*"
   pull_request:
     paths:
-      - "shared/tree-sitter-extractor/**"
+      - "ruby/extractor/codeql-extractor/**"
       - .github/workflows/tree-sitter-extractor-test.yml
     branches:
       - main
@@ -21,7 +21,7 @@ env:
 
 defaults:
   run:
-    working-directory: shared/tree-sitter-extractor
+    working-directory: ruby/extractor/codeql-extractor
 
 jobs:
   test:
@@ -33,13 +33,13 @@ jobs:
       - name: Run tests
         run: cargo test --verbose
   fmt:
-    runs-on: ubuntu-latest  
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - name: Check formatting
         run: cargo fmt --check
   clippy:
-    runs-on: ubuntu-latest  
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - name: Run clippy

ql/extractor/Cargo.toml

@@ -17,4 +17,4 @@ tracing = "0.1"
 tracing-subscriber = { version = "0.3.18", features = ["env-filter"] }
 rayon = "1.8.1"
 regex = "1.10.3"
-codeql-extractor = { path = "../../shared/tree-sitter-extractor" }
+codeql-extractor = { path = "../../ruby/extractor/codeql-extractor" }

ruby/actions/create-extractor-pack/action.yml

@@ -10,7 +10,7 @@ runs:
       uses: actions/cache@v3
       with:
         path: ruby/extractor-pack
-        key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-extractor-${{ hashFiles('ruby/extractor/rust-toolchain.toml', 'ruby/scripts/create-extractor-pack.sh', 'ruby/extractor/**/Cargo.lock', 'ruby/actions/create-extractor-pack/action.yml') }}-${{ hashFiles('shared/tree-sitter-extractor') }}-${{ hashFiles('ruby/extractor/**/*.rs') }}-${{ hashFiles('ruby/codeql-extractor.yml', 'ruby/downgrades', 'ruby/tools', 'ruby/ql/lib/ruby.dbscheme', 'ruby/ql/lib/ruby.dbscheme.stats') }}
+        key: ${{ runner.os }}-${{ steps.os_version.outputs.version }}-extractor-${{ hashFiles('ruby/extractor/rust-toolchain.toml', 'ruby/scripts/create-extractor-pack.sh', 'ruby/extractor/**/Cargo.lock', 'ruby/actions/create-extractor-pack/action.yml') }}-${{ hashFiles('ruby/extractor/codeql-extractor') }}-${{ hashFiles('ruby/extractor/**/*.rs') }}-${{ hashFiles('ruby/codeql-extractor.yml', 'ruby/downgrades', 'ruby/tools', 'ruby/ql/lib/ruby.dbscheme', 'ruby/ql/lib/ruby.dbscheme.stats') }}
     - name: Cache cargo
       uses: actions/cache@v3
       if: steps.cache-extractor.outputs.cache-hit != 'true'

ruby/extractor/Cargo.toml

@@ -1,3 +1,5 @@
+[workspace]
+
 [package]
 name = "codeql-extractor-ruby"
 description = "CodeQL Ruby extractor"
@@ -18,4 +20,4 @@ rayon = "1.5.0"
 regex = "1.7.1"
 encoding = "0.2"
 lazy_static = "1.4.0"
-codeql-extractor = { path = "../../shared/tree-sitter-extractor" }
+codeql-extractor = { path = "codeql-extractor" }

ruby/extractor/Cross.toml

@@ -1,8 +1,2 @@
 [target.x86_64-unknown-linux-gnu]
 image = "centos/devtoolset-7-toolchain-centos7"
-
-[build.env]
-# Provide the path to the shared extractor
-# Cross mounts this directory as a volume, so builds inside the docker container
-# can see it.
-volumes = ["__CODEQL-EXTRACTOR=../../shared/tree-sitter-extractor"]