mirror of
https://github.com/github/codeql.git
synced 2026-07-30 23:13:01 +02:00
unified: Parse Swift in-process instead of spawning a parser
The Swift front-end shelled out to a separate `swift-syntax-parse` executable
and read a JSON syntax tree back over a pipe. That kept the Swift toolchain off
the extractor's build path, so working on the other (tree-sitter based)
languages needed no Swift -- but it meant the extractor had to *find* that
executable at run time, and everything downstream of that was a workaround:
- the binary is a shell wrapper that sets `LD_LIBRARY_PATH` and execs
`swift-syntax-parse.real`, because the Swift runtime libraries have to sit
beside it;
- that only works in a flattened layout, so packaging went through
`codeql_pkg_runfiles` and the runtime libraries reached the pack only as a
side effect of shipping the parser;
- and the corpus tests skipped themselves when they could not find the binary.
A skip still prints `test result: ok`, so the suite silently tested nothing.
Supporting Swift is the current priority, so the extractor may now depend on
building the Swift half. Link `swift-syntax-rs` in and call `parse_to_json`
directly: `parse.rs` loses ~100 lines of process plumbing, and the tree is
necessarily produced by the swift-syntax build this extractor was built
against, so it cannot silently run a stale parser.
The parser survives as a debugging aid for looking at raw swift-syntax JSON:
echo 'let x = 1' | bazel run //unified/swift-syntax-rs:swift-syntax-parse
It is no longer shipped, so the wrapper, the `.real` split and the runfiles
packaging all go; the Swift runtime libraries are now packaged explicitly.
Linking Swift means the dynamic loader must resolve those libraries before
`main` runs. Bazel links against them through the toolchain's own directory,
which does not exist in an installed pack, so `swift_syntax_rs` now carries an
`$ORIGIN` runpath and the executable finds them beside itself. It arrives
through `CcInfo` rather than `rustc_flags` because `experimental_use_cc_common_link`
makes the link a `cc_common.link` action that `rustc_flags` never reaches.
`cargo test` can no longer build the tests without a local Swift toolchain, so
they run through Bazel, whose toolchain is hermetic on Linux. That also fixes
two things the old arrangement hid: `//unified/extractor:all_tests` covers all
three test files rather than only the corpus, and `test_corpus` now fails
instead of passing vacuously when it finds no cases at all.
`scripts/update-corpus.sh` drops the sandbox so the test can write the
regenerated `.output` files back through the runfiles symlinks. Passing that on
the command line rather than tagging the target keeps the ordinary test run
hermetic.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 297ff885-7c13-4202-a9d0-bc0b17f68d8e
This commit is contained in:
1
Cargo.lock
generated
1
Cargo.lock
generated
@@ -449,6 +449,7 @@ dependencies = [
|
||||
"rayon",
|
||||
"regex",
|
||||
"serde_json",
|
||||
"swift-syntax-rs",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"yeast",
|
||||
|
||||
@@ -4,13 +4,17 @@ This is a CodeQL extractor that maps a language's parse tree onto a shared AST
|
||||
using the `yeast` desugaring engine. Swift, the only language so far, is parsed
|
||||
by Apple's swift-syntax rather than by tree-sitter.
|
||||
|
||||
Build and test with Bazel, whose Swift toolchain is hermetic on Linux, so
|
||||
nothing needs to be installed locally. The extractor links `swift-syntax`, so a
|
||||
`cargo` build additionally needs a local Swift toolchain.
|
||||
|
||||
## Building
|
||||
- To build the extractor, run `scripts/create-extractor-pack.sh`
|
||||
- To build the extractor pack, run `scripts/create-extractor-pack.sh`.
|
||||
|
||||
## Swift Parser
|
||||
- Swift source is parsed by `swift-syntax-parse`, a small Swift/Rust binary in
|
||||
`swift-syntax-rs` that wraps Apple's swift-syntax and emits the parse tree as
|
||||
JSON. There is no grammar in this repository to edit.
|
||||
- Swift source is parsed by the `swift-syntax-rs` crate, which wraps Apple's
|
||||
swift-syntax and emits the parse tree as JSON. The extractor links it and
|
||||
calls it in-process. There is no grammar in this repository to edit.
|
||||
|
||||
- `extractor/src/languages/swift/adapter.rs` converts that JSON into a yeast AST.
|
||||
|
||||
@@ -22,11 +26,7 @@ by Apple's swift-syntax rather than by tree-sitter.
|
||||
|
||||
- The mapping from the parse tree to the target AST is found in `extractor/src/languages/swift/swift.rs`
|
||||
|
||||
- To run tests for the parser and mapping, run `cargo test` in the `extractor`
|
||||
directory. The tests need the `swift-syntax-parse` binary: point
|
||||
`CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` at it, or put it on `PATH`.
|
||||
Corpus tests skip themselves when it cannot be found, so check for skips
|
||||
before concluding a change is clean.
|
||||
- To run tests for the parser and mapping, run `bazel test //unified/extractor:all_tests`.
|
||||
|
||||
- Extractor test cases are located at `extractor/tests/corpus/swift/*/*.swift`.
|
||||
|
||||
|
||||
@@ -47,15 +47,13 @@ codeql_pkg_files(
|
||||
prefix = "tools/{CODEQL_PLATFORM}",
|
||||
)
|
||||
|
||||
# The Swift front-end parser (wrapper + real binary + bundled Swift runtime),
|
||||
# shipped next to the extractor. Only on platforms where swift-syntax builds
|
||||
# (Linux/macOS); elsewhere the group is empty so the pack still builds (Swift
|
||||
# extraction is simply unavailable there).
|
||||
pkg_filegroup(
|
||||
name = "swift-syntax-parse-arch",
|
||||
srcs = select_os(
|
||||
# The Swift runtime, which the extractor loads at startup. Linux only: macOS
|
||||
# provides it with the OS.
|
||||
codeql_pkg_files(
|
||||
name = "swift-runtime-arch",
|
||||
exes = select_os(
|
||||
linux = ["//unified/swift-syntax-rs:swift_runtime_libs"],
|
||||
otherwise = [],
|
||||
posix = ["//unified/swift-syntax-rs:swift-syntax-parse-pkg"],
|
||||
),
|
||||
prefix = "tools/{CODEQL_PLATFORM}",
|
||||
)
|
||||
@@ -66,7 +64,7 @@ codeql_pack(
|
||||
":codeql-extractor-yml",
|
||||
":dbscheme-group",
|
||||
":extractor-arch",
|
||||
":swift-syntax-parse-arch",
|
||||
":swift-runtime-arch",
|
||||
"//unified/tools",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
load("@rules_rust//rust:defs.bzl", "rust_test")
|
||||
load("//misc/bazel:rust.bzl", "codeql_rust_binary")
|
||||
load("//misc/bazel/3rdparty/tree_sitter_extractors_deps:defs.bzl", "aliases", "all_crate_deps")
|
||||
|
||||
exports_files(["Cargo.toml"])
|
||||
|
||||
# swift-syntax builds on Linux and macOS only.
|
||||
_SWIFT_SUPPORTED_PLATFORMS = select({
|
||||
"@platforms//os:linux": [],
|
||||
"@platforms//os:macos": [],
|
||||
"//conditions:default": ["@platforms//:incompatible"],
|
||||
})
|
||||
|
||||
codeql_rust_binary(
|
||||
name = "extractor",
|
||||
srcs = glob(["src/**/*.rs"]),
|
||||
@@ -11,14 +19,75 @@ codeql_rust_binary(
|
||||
"ast_types.yml",
|
||||
"swift_node_types.yml",
|
||||
],
|
||||
# Only for running from the build tree: there the Swift runtime is resolved
|
||||
# through the toolchain-relative part of the runpath, which needs the
|
||||
# libraries in the runfiles tree. An installed pack uses `$ORIGIN` instead.
|
||||
data = select({
|
||||
"@platforms//os:linux": ["//unified/swift-syntax-rs:swift_runtime_libs"],
|
||||
"//conditions:default": [],
|
||||
}),
|
||||
proc_macro_deps = all_crate_deps(
|
||||
proc_macro = True,
|
||||
),
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
visibility = ["//visibility:public"],
|
||||
deps = all_crate_deps(
|
||||
normal = True,
|
||||
) + [
|
||||
"//shared/tree-sitter-extractor",
|
||||
"//shared/yeast",
|
||||
"//unified/swift-syntax-rs:swift_syntax_rs",
|
||||
],
|
||||
)
|
||||
|
||||
# One target per file in `tests/`. Each pulls in `src/**` too, because the tests
|
||||
# reach into the crate's modules with `#[path]`.
|
||||
_TESTS = {
|
||||
"corpus_tests": {
|
||||
"data": glob(["tests/corpus/**"]),
|
||||
"compile_data": [],
|
||||
"size": "medium",
|
||||
},
|
||||
# `include_str!`s a checked-in `parse_to_json` dump.
|
||||
"swift_syntax_pipeline": {
|
||||
"data": [],
|
||||
"compile_data": glob(["tests/fixtures/**"]),
|
||||
"size": "small",
|
||||
},
|
||||
}
|
||||
|
||||
[
|
||||
rust_test(
|
||||
name = test_name,
|
||||
size = spec["size"],
|
||||
srcs = ["tests/%s.rs" % test_name] + glob(["src/**/*.rs"]),
|
||||
aliases = aliases(),
|
||||
compile_data = [
|
||||
"ast_types.yml",
|
||||
"swift_node_types.yml",
|
||||
] + spec["compile_data"],
|
||||
crate_root = "tests/%s.rs" % test_name,
|
||||
data = spec["data"] + select({
|
||||
"@platforms//os:linux": ["//unified/swift-syntax-rs:swift_runtime_libs"],
|
||||
"//conditions:default": [],
|
||||
}),
|
||||
edition = "2024",
|
||||
proc_macro_deps = all_crate_deps(
|
||||
proc_macro = True,
|
||||
),
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
deps = all_crate_deps(
|
||||
normal = True,
|
||||
) + [
|
||||
"//shared/tree-sitter-extractor",
|
||||
"//shared/yeast",
|
||||
"//unified/swift-syntax-rs:swift_syntax_rs",
|
||||
],
|
||||
)
|
||||
for test_name, spec in _TESTS.items()
|
||||
]
|
||||
|
||||
test_suite(
|
||||
name = "all_tests",
|
||||
tests = [":%s" % test_name for test_name in _TESTS],
|
||||
)
|
||||
|
||||
@@ -18,3 +18,8 @@ serde_json = "1.0.145"
|
||||
|
||||
codeql-extractor = { path = "../../shared/tree-sitter-extractor" }
|
||||
yeast = { path = "../../shared/yeast" }
|
||||
# The Swift front-end links swift-syntax through this crate's FFI shim. Its
|
||||
# Swift half is built by Bazel, so `cargo build`/`test` cannot link the
|
||||
# extractor: use `bazel build //unified/extractor` and
|
||||
# `bazel test //unified/extractor:all_tests`.
|
||||
swift-syntax-rs = { path = "../swift-syntax-rs" }
|
||||
|
||||
@@ -1,33 +1,18 @@
|
||||
//! Swift front-end parser: shells out to the separate `swift-syntax-parse`
|
||||
//! binary (which links swift-syntax) to obtain a JSON syntax tree, then adapts
|
||||
//! that JSON into a `yeast::Ast` via the pure-Rust [`swift_adapter`] module.
|
||||
//!
|
||||
//! Running the parser in a separate process keeps the Swift toolchain out of
|
||||
//! the extractor's own build: the extractor never links Swift, so working on
|
||||
//! other (e.g. tree-sitter based) languages needs no Swift toolchain. Each call
|
||||
//! spawns the parser afresh; a longer-lived parser process could be swapped in
|
||||
//! behind this same seam later without touching the extraction pipeline.
|
||||
|
||||
use std::io::Write;
|
||||
use std::process::{Command, Stdio};
|
||||
//! Swift front-end parser: calls into the `swift-syntax-rs` crate (which links
|
||||
//! swift-syntax) to obtain a JSON syntax tree, then adapts that JSON into a
|
||||
//! `yeast::Ast` via the pure-Rust [`swift_adapter`] module.
|
||||
|
||||
use codeql_extractor::extractor::ParsedTree;
|
||||
|
||||
use super::swift_adapter;
|
||||
|
||||
/// Environment variable naming the `swift-syntax-parse` executable. When unset,
|
||||
/// the parser is resolved next to the extractor executable, then on `PATH`.
|
||||
const PARSE_BIN_ENV: &str = "CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE";
|
||||
|
||||
/// Base name of the `swift-syntax-parse` executable as shipped / looked up.
|
||||
const PARSE_BIN_NAME: &str = "swift-syntax-parse";
|
||||
|
||||
/// Parse Swift `source` into a [`ParsedTree`] (a raw `yeast::Ast` plus
|
||||
/// side-channel `extra` tokens), ready to be desugared via `run_from_ast`.
|
||||
pub fn parse(source: &[u8]) -> Result<ParsedTree, String> {
|
||||
let source =
|
||||
std::str::from_utf8(source).map_err(|e| format!("Swift source is not valid UTF-8: {e}"))?;
|
||||
let json = run_parser(source)?;
|
||||
let json =
|
||||
swift_syntax_rs::parse_to_json(source).map_err(|e| format!("Swift parser failed: {e}"))?;
|
||||
let mut adapted = swift_adapter::json_to_ast(&json)?;
|
||||
adapted.ast.set_source(source.as_bytes().to_vec());
|
||||
Ok(ParsedTree {
|
||||
@@ -35,87 +20,3 @@ pub fn parse(source: &[u8]) -> Result<ParsedTree, String> {
|
||||
extras: adapted.extras,
|
||||
})
|
||||
}
|
||||
|
||||
/// The `swift-syntax-parse` executable to invoke, resolved in priority order:
|
||||
///
|
||||
/// 1. the `CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` override, if set;
|
||||
/// 2. a copy shipped next to the extractor executable — this is how the CodeQL
|
||||
/// extractor pack lays it out (`tools/<platform>/{extractor,
|
||||
/// swift-syntax-parse}`), so a packaged extractor is self-contained with no
|
||||
/// environment setup;
|
||||
/// 3. a bare `swift-syntax-parse`, looked up on `PATH`.
|
||||
fn parse_bin() -> String {
|
||||
if let Ok(bin) = std::env::var(PARSE_BIN_ENV) {
|
||||
if !bin.is_empty() {
|
||||
return bin;
|
||||
}
|
||||
}
|
||||
if let Ok(exe) = std::env::current_exe() {
|
||||
if let Some(sibling) = exe.parent().map(|dir| dir.join(PARSE_BIN_NAME)) {
|
||||
if sibling.is_file() {
|
||||
return sibling.to_string_lossy().into_owned();
|
||||
}
|
||||
}
|
||||
}
|
||||
PARSE_BIN_NAME.to_string()
|
||||
}
|
||||
|
||||
/// Whether the `swift-syntax-parse` executable can be launched at all.
|
||||
///
|
||||
/// This reports availability of the *executable*, deliberately not whether
|
||||
/// parsing succeeds: a binary that launches but then crashes or emits invalid
|
||||
/// JSON is still "available", so callers run and surface the failure rather
|
||||
/// than silently skipping. Only a genuinely missing/unlaunchable binary (e.g.
|
||||
/// no Swift toolchain is installed) reports `false`.
|
||||
pub fn binary_available() -> bool {
|
||||
match Command::new(parse_bin())
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.spawn()
|
||||
{
|
||||
Ok(mut child) => {
|
||||
let _ = child.wait();
|
||||
true
|
||||
}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => false,
|
||||
// Any other spawn failure (e.g. a permissions problem) is a genuine
|
||||
// issue worth surfacing, so treat the parser as available and let the
|
||||
// caller fail rather than masking it as "unavailable".
|
||||
Err(_) => true,
|
||||
}
|
||||
}
|
||||
|
||||
/// Run the external parser, feeding `source` on stdin and returning its JSON
|
||||
/// stdout.
|
||||
fn run_parser(source: &str) -> Result<String, String> {
|
||||
let bin = parse_bin();
|
||||
let mut child = Command::new(&bin)
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.map_err(|e| format!("failed to spawn Swift parser `{bin}`: {e}"))?;
|
||||
|
||||
// The parser reads all of stdin before writing any stdout, so writing the
|
||||
// whole source and then closing stdin (by dropping it) cannot deadlock.
|
||||
child
|
||||
.stdin
|
||||
.take()
|
||||
.expect("child stdin was piped")
|
||||
.write_all(source.as_bytes())
|
||||
.map_err(|e| format!("failed to write source to Swift parser `{bin}`: {e}"))?;
|
||||
|
||||
let output = child
|
||||
.wait_with_output()
|
||||
.map_err(|e| format!("failed to run Swift parser `{bin}`: {e}"))?;
|
||||
if !output.status.success() {
|
||||
return Err(format!(
|
||||
"Swift parser `{bin}` failed ({}): {}",
|
||||
output.status,
|
||||
String::from_utf8_lossy(&output.stderr).trim()
|
||||
));
|
||||
}
|
||||
String::from_utf8(output.stdout)
|
||||
.map_err(|e| format!("Swift parser produced non-UTF-8 output: {e}"))
|
||||
}
|
||||
|
||||
@@ -20,20 +20,6 @@ fn update_mode_enabled() -> bool {
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
/// Whether the external swift-syntax parser is available. When the parser
|
||||
/// binary genuinely cannot be found/launched (e.g. no Swift toolchain, and
|
||||
/// neither `CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` nor a `swift-syntax-parse`
|
||||
/// on `PATH`), the corpus test is skipped rather than failed — it cannot run
|
||||
/// without the Swift-backed parser.
|
||||
///
|
||||
/// Crucially this checks only that the executable *launches*: a parser that is
|
||||
/// present but crashes, emits invalid JSON, or otherwise regresses is
|
||||
/// considered available, so the suite runs and fails (rather than silently
|
||||
/// skipping the very failures CI needs to catch).
|
||||
fn parser_available() -> bool {
|
||||
languages::swift_parse::binary_available()
|
||||
}
|
||||
|
||||
/// Parse a corpus `.output` file. The file holds a single test case made of
|
||||
/// three sections separated by `---` delimiter lines:
|
||||
///
|
||||
@@ -110,19 +96,32 @@ fn collect_corpus_stems(dir: &Path, out: &mut Vec<std::path::PathBuf>) {
|
||||
}
|
||||
}
|
||||
|
||||
/// The corpus root, in the runfiles tree.
|
||||
///
|
||||
/// Bazel runs tests from the runfiles root, under a directory named after the
|
||||
/// repository the test came from — `_main` when `github/codeql` is built
|
||||
/// standalone, `ql+` when it is consumed as a dependency — so look for it
|
||||
/// rather than hard-coding either name.
|
||||
fn corpus_dir() -> std::path::PathBuf {
|
||||
let srcdir = std::env::var_os("TEST_SRCDIR")
|
||||
.expect("TEST_SRCDIR is unset; these tests are run with `bazel test`");
|
||||
let entries = fs::read_dir(&srcdir)
|
||||
.unwrap_or_else(|e| panic!("failed to read TEST_SRCDIR {srcdir:?}: {e}"));
|
||||
for entry in entries.flatten() {
|
||||
let candidate = entry.path().join("unified/extractor/tests/corpus");
|
||||
if candidate.is_dir() {
|
||||
return candidate;
|
||||
}
|
||||
}
|
||||
panic!("no `unified/extractor/tests/corpus` under TEST_SRCDIR {srcdir:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_corpus() {
|
||||
if !parser_available() {
|
||||
eprintln!(
|
||||
"skipping test_corpus: the swift-syntax parser is unavailable \
|
||||
(set CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE or put \
|
||||
`swift-syntax-parse` on PATH)"
|
||||
);
|
||||
return;
|
||||
}
|
||||
let update_mode = update_mode_enabled();
|
||||
let all_languages = languages::all_language_specs();
|
||||
let corpus_dir = Path::new("tests/corpus");
|
||||
let corpus_dir = corpus_dir();
|
||||
let mut tested = 0usize;
|
||||
|
||||
for lang in all_languages {
|
||||
let output_schema = yeast::node_types_yaml::schema_from_yaml(languages::OUTPUT_AST_SCHEMA)
|
||||
@@ -139,6 +138,7 @@ fn test_corpus() {
|
||||
stems.dedup();
|
||||
|
||||
for stem in stems {
|
||||
tested += 1;
|
||||
let swift_path = stem.with_extension("swift");
|
||||
let output_path = stem.with_extension("output");
|
||||
let mut failures = Vec::new();
|
||||
@@ -265,4 +265,13 @@ fn test_corpus() {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every language whose corpus directory is missing is skipped silently
|
||||
// above, which is right when a language simply has no corpus — but if that
|
||||
// leaves nothing at all to check, the run is vacuous and must not pass.
|
||||
assert!(
|
||||
tested > 0,
|
||||
"no corpus cases found under {}; the suite would have passed vacuously",
|
||||
corpus_dir.display()
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,25 +1,21 @@
|
||||
#!/bin/bash
|
||||
set -eux
|
||||
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
|
||||
platform="linux64"
|
||||
elif [[ "$OSTYPE" == "darwin"* ]]; then
|
||||
platform="osx64"
|
||||
else
|
||||
echo "Unknown OS"
|
||||
exit 1
|
||||
fi
|
||||
# Build the extractor pack into `extractor-pack/`, ready for
|
||||
# `codeql test run --search-path extractor-pack`.
|
||||
#
|
||||
# `unified.dbscheme` and `Ast.qll` are generated from `extractor/ast_types.yml`,
|
||||
# so they are regenerated here before the pack is assembled.
|
||||
set -euo pipefail
|
||||
IFS=$'\n\t'
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
root=$PWD
|
||||
|
||||
(cd extractor && cargo build --release)
|
||||
|
||||
# we are in a cargo workspace rooted at the git checkout
|
||||
BIN_DIR=../target/release
|
||||
"$BIN_DIR/codeql-extractor-unified" generate --dbscheme ql/lib/unified.dbscheme --library ql/lib/codeql/unified/Ast.qll
|
||||
# `bazel run` executes from the runfiles tree, so pass absolute output paths.
|
||||
bazel run //unified/extractor -- generate \
|
||||
--dbscheme "$root/ql/lib/unified.dbscheme" \
|
||||
--library "$root/ql/lib/codeql/unified/Ast.qll"
|
||||
|
||||
codeql query format -i ql/lib/codeql/unified/Ast.qll
|
||||
|
||||
rm -rf extractor-pack
|
||||
mkdir -p extractor-pack
|
||||
cp -r codeql-extractor.yml tools ql/lib/unified.dbscheme ql/lib/unified.dbscheme.stats extractor-pack/
|
||||
mkdir -p extractor-pack/tools/${platform}
|
||||
cp "$BIN_DIR/codeql-extractor-unified" extractor-pack/tools/${platform}/extractor
|
||||
exec bazel run //unified:install "$@"
|
||||
|
||||
@@ -1,8 +1,22 @@
|
||||
#!/bin/bash
|
||||
# Regenerate the extractor corpus: rerun the corpus tests with update mode on,
|
||||
# so each `.output` file is rewritten from what the extractor currently
|
||||
# produces.
|
||||
set -euo pipefail
|
||||
IFS=$'\n\t'
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
cd extractor
|
||||
UNIFIED_UPDATE_CORPUS=1 cargo test
|
||||
# A sandboxed test cannot touch the source tree. `--strategy=TestRunner=local`
|
||||
# drops the sandbox, and the corpus files in the runfiles tree are symlinks back
|
||||
# to the real ones, so update mode writes through them. Setting this here rather
|
||||
# than tagging the target keeps the ordinary test run sandboxed.
|
||||
#
|
||||
# `--nocache_test_results` because a cached PASS would skip the run, and the
|
||||
# side effect is the point.
|
||||
exec bazel test \
|
||||
--strategy=TestRunner=local \
|
||||
--test_env=UNIFIED_UPDATE_CORPUS=1 \
|
||||
--nocache_test_results \
|
||||
//unified/extractor:corpus_tests \
|
||||
"$@"
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
load("@rules_cc//cc:defs.bzl", "cc_library")
|
||||
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_test")
|
||||
load("@rules_shell//shell:sh_binary.bzl", "sh_binary")
|
||||
load("//misc/bazel:pkg.bzl", "codeql_pkg_runfiles")
|
||||
load(":swift_runtime.bzl", "swift_runtime_libs")
|
||||
load(":xcode_transition.bzl", "xcode_transition_swift_library")
|
||||
|
||||
@@ -35,6 +34,24 @@ xcode_transition_swift_library(
|
||||
],
|
||||
)
|
||||
|
||||
# The Swift runtime has to be resolvable at load time, and Bazel links against
|
||||
# it through the toolchain's own directory, which no installed pack has. The
|
||||
# `$ORIGIN` runpath makes an executable look beside itself instead, where
|
||||
# `//unified:swift-runtime-arch` puts the libraries.
|
||||
#
|
||||
# This must arrive through `CcInfo`, not `rustc_flags`: `.bazelrc` enables
|
||||
# `experimental_use_cc_common_link` on Linux, making the link a
|
||||
# `cc_common.link` action that rustc flags never reach. `:swift_syntax_rs`
|
||||
# depends on it so that every binary linking swift-syntax inherits it.
|
||||
cc_library(
|
||||
name = "swift_runtime_rpath",
|
||||
linkopts = select({
|
||||
"@platforms//os:linux": ["-Wl,-rpath,$$ORIGIN"],
|
||||
"//conditions:default": [],
|
||||
}),
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
)
|
||||
|
||||
# Safe Rust bindings on top of the C ABI. Under Bazel the Swift side comes
|
||||
# from `:swift_syntax_ffi`; `build.rs` is only used by the `cargo` workflow.
|
||||
rust_library(
|
||||
@@ -46,54 +63,27 @@ rust_library(
|
||||
edition = "2024",
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
deps = [
|
||||
":swift_runtime_rpath",
|
||||
":swift_syntax_ffi",
|
||||
],
|
||||
)
|
||||
|
||||
# The Swift front-end parser. We ship it like `//swift/extractor`: a small shell
|
||||
# wrapper (`swift-syntax-parse`) sets `LD_LIBRARY_PATH`/`DYLD_LIBRARY_PATH` to its
|
||||
# own directory and execs the real binary (`swift-syntax-parse.real`); the Swift
|
||||
# runtime shared libraries are packaged alongside them. `parse.rs` resolves the
|
||||
# wrapper as a sibling of the extractor executable.
|
||||
# A debugging aid, for looking at the raw swift-syntax JSON for some input:
|
||||
#
|
||||
# echo 'let x = 1' | bazel run //unified/swift-syntax-rs:swift-syntax-parse
|
||||
rust_binary(
|
||||
name = "swift-syntax-parse.real",
|
||||
name = "swift-syntax-parse",
|
||||
srcs = ["src/main.rs"],
|
||||
# Target name carries `.real` (invalid in a crate name), so set it explicitly.
|
||||
crate_name = "swift_syntax_parse",
|
||||
# On Linux, carry the toolchain's runtime shared libraries as runfiles so
|
||||
# they get packaged next to the binary. On macOS the OS provides the Swift
|
||||
# runtime, so nothing extra is bundled.
|
||||
# See the note on `data` in `//unified/extractor:extractor`.
|
||||
data = select({
|
||||
"@platforms//os:macos": [],
|
||||
"@platforms//os:linux": [":swift_runtime_libs"],
|
||||
"//conditions:default": [],
|
||||
}),
|
||||
edition = "2024",
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
deps = [":swift_syntax_rs"],
|
||||
)
|
||||
|
||||
# `swift-syntax-parse` wrapper (see `swift-syntax-parse.sh`). Its runfiles carry
|
||||
# the real binary and the runtime libraries; packaging flattens them into one
|
||||
# directory.
|
||||
sh_binary(
|
||||
name = "swift-syntax-parse",
|
||||
srcs = ["swift-syntax-parse.sh"],
|
||||
data = [":swift-syntax-parse.real"],
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
)
|
||||
|
||||
# Packaged form for the extractor pack: the wrapper (as `swift-syntax-parse`),
|
||||
# the real binary, and the runtime libraries, flattened into one directory.
|
||||
codeql_pkg_runfiles(
|
||||
name = "swift-syntax-parse-pkg",
|
||||
# The `.sh` source is shipped as `swift-syntax-parse` (the wrapper); drop the
|
||||
# original filename.
|
||||
excludes = ["swift-syntax-parse.sh"],
|
||||
exes = [":swift-syntax-parse"],
|
||||
target_compatible_with = _SWIFT_SUPPORTED_PLATFORMS,
|
||||
visibility = ["//unified:__pkg__"],
|
||||
)
|
||||
|
||||
rust_test(
|
||||
name = "swift_syntax_rs_test",
|
||||
size = "small",
|
||||
|
||||
@@ -1,12 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Wrapper that lets the shipped `swift-syntax-parse` find its Swift runtime
|
||||
# libraries, which are packaged in the same directory as this script (and the
|
||||
# real binary). Mirrors `swift/extractor/extractor.sh`.
|
||||
if [[ "$(uname)" == Darwin ]]; then
|
||||
export DYLD_LIBRARY_PATH=$(dirname "$0")
|
||||
else
|
||||
export LD_LIBRARY_PATH=$(dirname "$0")
|
||||
fi
|
||||
|
||||
exec -a "$0" "$0.real" "$@"
|
||||
Reference in New Issue
Block a user