Testing

Most of ry is tested with fixtures: plain-text files that pair a piece of R with the output ry should produce for it. A fixture reads well in a diff, and a new case takes seconds to add. Reach for an ordinary Rust test only when the behavior is awkward to express as rendered output.

A fixture suite describes the desired semantics, not what the implementation happens to pass. Record a known gap explicitly, in .agents/memory/backlog.md or as a case pinned to the wrong-but-current output with a comment that says so, because an unrecorded gap looks intentional to the next reader.

Fixture format

Every suite in the shipping crates runs through one harness, syntax::testing::run_fixture_suite, which syntax, semantics, format, and ide all share. A .test file holds groups of cases:

#==== group_name
#---- case_name
<input source>
#++++
<expected rendered output>

group__case is the case’s stable identity. It must be unique across the suite, and a duplicate is rejected rather than allowed to shadow the other case silently.
What the expected output shows (schemes, diagnostics, a tree dump, hover text) is up to the suite’s runner. Behavior specific to one suite belongs in its runner, never in the fixture syntax.
A typing or IDE case may hold several files. Each #~~~~ <path> line starts one (see syntax::testing::split_files), and a path under R/ is a package file while anything else is a script. A typing case renders each file’s output under an == <path> line.
The runner loads only .test files, so anything else in a suite directory is ignored.

Running one case

To run a single case, set FIXTURE_FILTER and pick the suite’s test target:

FIXTURE_FILTER=group__case cargo test -p semantics --test test_typing_fixtures -- --nocapture
FIXTURE_FILTER=group__case cargo test -p format --test test_format_fixtures -- --nocapture
FIXTURE_FILTER=group__case cargo test -p ide --test test_ide_fixtures -- --nocapture

One test target can drive several suites, so a suite that does not hold the case skips it quietly. But a filter that matches no case in any suite fails: that is almost certainly a typo, and a run of zero cases would otherwise report a pass.

While iterating, cargo test -p semantics is the default command, since it covers analysis behavior. just gate runs every test plus clippy and a formatting check. CI runs the same recipes, but only when started by hand: the orchestrating agent starts it on main at its hourly check when main has new commits, so a push or a pull request gets no CI run of its own.

Blessing expectations

With RY_BLESS=1, a mismatch does not fail. Instead, the harness rewrites the #++++ blocks in the .test files in place from the runner’s actual output:

RY_BLESS=1 cargo test -p semantics --test test_typing_fixtures

Only the body of each #++++ block is rewritten. Directive lines and the spacing around them are kept, so a blessed file is byte-identical to what a person would have written, and a second run without bless passes. FIXTURE_FILTER still applies, so you can bless a single case, and blessing a suite that is already correct changes no bytes.

Review every blessed change before you commit it. Bless records whatever the runner produces, wrong outcomes included, so accept a changed expectation only when the behavior or the wording improved on purpose.

The fixture suites

crates/syntax/tests/syntax: golden lossless trees and syntax errors, rendered by debug_dump.
crates/syntax/tests/tsr: tree-sitter-r’s parser corpus, converted to the same format.
crates/syntax/tests/errors: golden Elm-style error messages, organized by area (the lexer, delimiters, control flow, functions and calls, the #: type and directive grammar, and recovery locality). Its coverage contract has three parts: every distinct error message the lexer or parser can emit has at least one case here, valid but tricky shapes are pinned as no errors, and recovery cases assert that one broken construct reports once and leaves the rest of the file clean.
crates/semantics/tests/naming: the name-resolution suite. It records which binding every name in an item resolves to, rendered flat in source order. This is the only suite that tests naming directly. Everywhere else naming is tested through its consequences, a type that comes out right or a diagnostic that does or does not fire, and consequences cannot see a read that resolves to the wrong binding while still producing the same type.
crates/semantics/tests/typing: the typing suite. Each case runs the full semantic pipeline on one package file, or on several, with the shipped stubs installed. It renders the exported scheme of every top-level definition (name: <T: numeric> fn(x: T) -> T), the type of every other top-level statement as a bare line, and then the file’s diagnostics (start..end severity[code] message, in byte offsets). Each test thread reuses one database across cases (semantics::testing::with_fixture_project), because parsing the stub corpus costs far more than checking a case. Cases are grouped by feature, with one exception: programs.R.test holds whole small programs written the way real projects are written. A case earns its place there by exercising several features at once, and a clean case with an empty diagnostics block is a promise of zero false positives for that style of code.
crates/semantics/tests/lowering: the HIR each item lowers to, dumped as an indented tree under the item’s name (or under # <statement> for an unnamed item). Argument tags are rendered inline, because matching arguments is what a call’s lowering has to get right. The suite exists because rewrites, such as the native pipe becoming a call, a replacement form, or local, would otherwise be tested only through the type that falls out at the far end, which cannot tell “lowered correctly” from “lowered to the wrong shape but typed the same”. A case whose comment says WRONG-BUT-CURRENT pins a known defect next to the shape it should have, so fixing the defect turns the case red.
crates/semantics/tests/typing-scripts: the same pipeline over script documents, which have one sequential, top-down scope. Besides script semantics, this suite holds the rules for single expressions (literals, operators, indexing, lists, and calls to stubbed functions), rendered one line per statement.
crates/semantics/tests/annotations: one #: block per case, where the case’s lines are the block. It renders what the block lowers to (the kind of coercion, the declared type, the @new target, and each @type or @alias definition), followed by its grammar and lowering errors. This is the compact contract for the type notation, while the syntax suite pins the trees and the errors suite the wording.
crates/semantics/tests/stubs: .Rtypes declarations. Each case renders the declarations it loads, with overload candidates in order, and the problems that make the loader drop a line, in the wording the editor and the override report show.
crates/semantics/tests/typing-inline-off: the ordinary rendering with [check] inline-annotations = false, which pins the ry-syntax warning on records, tuples, tags and match, and shows that literals are still typed while match coverage is not checked.
crates/semantics/tests/typing-strict: the strict stream. After the ordinary rendering, each case appends the file’s typing mode and its strict diagnostics.
crates/semantics/tests/top-level: what a package file’s top level runs under strict mode. A case lists files with #~~~~ separators, so R/ files are library files and tests/ or root files are entries. Each case loads the shipped stubs and renders, per file, every top-level item as definition, constant or effect, with (pure) when the item declares its function pure, followed by the file’s top-level strict diagnostics. Run it with cargo test -p semantics --test test_typing_fixtures top_level_fixtures.
crates/semantics/tests/typing-imports: package metadata. A leading #namespace line supplies the case’s NAMESPACE source and a #description line its DESCRIPTION source, and both are installed as the PackageMetadata input before rendering. The directive lines stay in the analyzed text as ordinary comments, so expected offsets are honest. Attaching a package needs no directive: a library(data.table) statement in the source activates the conditional stub namespace through the same scan the hosts run.
crates/semantics/tests/lints, lints-style and lints-object-systems-off: lints::lint_file, first under the default configuration, then under naming-style = "snake_case", unused-parameter = "warn", shadows-builtin = "warn", and shadows-namespace = "warn", all of which are off by default, and last under [check] object-systems = false. Each case renders every finding as start..end severity[code] message, or clean. Run all three with cargo test -p semantics --test test_lint_fixtures.
crates/semantics/tests/load_index: the package-load index. A case lists R/ files, and optionally NAMESPACE and DESCRIPTION, with #~~~~ separators, and renders the package’s name, the NAMESPACE directives a load honors, and, per file, each item’s position and whether it is bound lazily (as a function or, in a strict file, a constant) or runs eagerly, with what it writes, what it reads while running, and the registrations it makes. Run it with cargo test -p semantics --test test_load_index_fixtures.
crates/loader/tests/reload: reload plans. A file headed #~~~~ before <path> or #~~~~ after <path> exists on that side only, so the same path on both sides is an edit and one side alone is an addition or a deletion, while a plain #~~~~ <path> is unchanged. The case loads the before side, then renders the plan that takes it to the after side: what is bound, run, removed, unregistered, re-registered, and which datasets are read again. R/ files run in the order DESCRIPTION’s collation field gives them, and a data/ file is a dataset whose text stands in for its size and modification time. Run it with cargo test -p loader --test test_reload_fixtures.
crates/format/tests/format: the formatter’s golden suite. Each case’s source must format to the expected block, and the runner formats that output again to assert idempotence on every case. When the expected result is a refusal, the case renders the structured FormatError.
crates/format/tests/format-range: range formatting. The source marks the selection with $0 markers (two for a selection, one for a bare caret, none for the whole file), and the expectation lists the edits by one-based position and shows the document they leave behind, with each replacement between « and ». Run it with cargo test -p format --test test_format_range_fixtures.
crates/ide/tests/ide: the editor features. The source carries one $0 cursor marker, which is stripped before analysis, and the expectation renders each feature’s answer at that position: the hover line with its absolute range, the definition’s target range, and the reference ranges. A target inside the stub corpus renders the token its range covers, as in declared in stub source 0 at `print` , rather than a byte offset, because an offset shifts whenever an unrelated declaration is added, while the token proves the range is right.
crates/ide/tests/structure: whole-file editor features. Each case renders the outline (nested by indentation), the folds as zero-based inclusive line ranges, the semantic tokens for R names, the quick fixes the diagnostics and default lints offer, and, when the source has a $0 marker, every step of selection expansion from it. Run it with cargo test -p ide --test test_ide_fixtures structure.
crates/ide/tests/migrate: the data.table to DataFrame rewrites behind the code actions and ry migrate data-table. Each case is one script whose attached packages are the ones it attaches, and the expectation lists every rewrite as its title, the original lines (- ) and the replacement (+ ), or no rewrite. A $0 marker also lists the code actions offered there. Run it with cargo test -p ide --test test_ide_fixtures migrate.

Fuzzing and property tests

Fixtures check the answers someone wrote down. The harnesses in this section check properties that must hold for every input, over a great many inputs. Every pipeline stage gets fuzz coverage from its first commit.

The batteries of invariants are exported, so that the in-tree harnesses and the coverage-guided targets share exactly the same contracts: syntax::testing::check_parse_invariants for the parser, format::check_format_invariants for the formatter, and semantics::testing::check_semantics_input for the semantic pipeline. The last one folds every lint, under a configuration that turns them all on, into its rendering, so the lint layer inherits the same invariants. The format and semantics harnesses, and those of syntax and ide, also carry a fuzz_regressions_hold_invariants battery that pins every input the coverage-guided targets have ever broken.

The semantic pipeline

crates/semantics/tests/test_fuzz.rs runs generated programs, token soup, and two-file projects through the full semantic pipeline, with the generator biased toward reference cycles, closures, annotations, and typing-mode directives. On every input it checks four properties:

Nothing panics, which among other things means every salsa fixpoint converges.
The output is deterministic across fresh databases.
Diagnostic ranges are well formed.
Incremental equals fresh. Editing a file through the salsa setter gives the same output as a fresh database on the edited text, and editing it back restores the original.

FUZZ_ITERS scales the budgets. A bounded default runs inside cargo test -p semantics, the ignored fuzz_deep test carries the long runs, and on a panic the harness prints the inputs that produced it.

The formatter

crates/format/tests/test_fuzz.rs feeds the formatter valid-program seeds, byte-level mutations of those seeds, token soup, random bytes, every fixture case source in the repository, and real corpus files once they are fetched. It checks four properties: nothing panics (the formatter either succeeds or refuses with a structured error), the output is deterministic, the code is preserved, and formatting is idempotent, meaning that formatting a successful output again succeeds and reproduces it byte for byte.

The refusal path is part of the contract. A file with a syntax error in its R is refused, while an error in a #: annotation, which the parser marks in_annotation, only sends the affected block down the verbatim path.

Preservation is the one property that can catch the formatter changing your code. Determinism, idempotence, and “the output formats again” all hold for a formatter that silently drops a statement or misspells a token. Preservation compares the sequence of non-trivia token kinds and spellings between input and output, with two allowances:

Four kinds are ignored entirely ({, }, ;, and #:), because the formatter may add or move them: it braces single-statement bodies, splits ; chains onto separate lines, and re-lays out annotation blocks.
A string is compared by its content rather than its text, because the formatter chooses the quote character. Raw strings are copied byte for byte, so they keep their full spelling.

Comparing kinds alone would miss a miscompile: a stale or off-by-one source range makes the formatter emit the wrong bytes for an identifier, which renames a variable while preserving every kind. An injected bug of exactly that shape passed a kind-only comparison and fails the spelling comparison.

Every generated input also runs the range battery, check_range_format_invariants, over a bounded spread of selections (the whole file, carets, whole lines, part-line spans, a range past the end). Range edits must be deterministic, refuse exactly when the whole file refuses, cover whole lines in order without overlapping, preserve the code, leave a document that still formats to the same result, be a subset of the edits formatting the whole file in place makes, reproduce that result when the selection is everything, and be stable when re-run on what they produced. On inputs of up to 24 lines it also checks additivity (a span of lines makes exactly the union of its lines’ edits) and composition (formatting line by line until nothing changes reaches the whole-file result, and every step still formats to it). Composition is the strongest of these: it found whole-file idempotence bugs that no single-pass check sees. crates/format/tests/test_format_range.rs runs the battery over every fixture source, the program corpus, and re-indented copies of the fixtures (four spaces, tabs, mixed steps), and checks every byte selection of a few short sources against an independent line oracle.

The editor features

crates/ide/tests/test_fuzz.rs sweeps every feature over a sample of byte offsets in each input: every token boundary plus a coarse stride, so positions off token boundaries and in the middle of multi-byte characters are covered too. At each offset nothing may panic, and every range handed to the editor must lie inside the text it indexes.

Containment alone is a weak check, though: it says a range is somewhere in the file, not that it is on the right thing. Wherever the cursor sits on a name, two stronger properties hold as well:

Name identity: every rename edit, every reference, and the definition target cover text that spells the same name (with backtick quoting normalized, since `x` and x are one binding).
Round trip: if the cursor navigates to a declaration, asking for references from that declaration leads back to the cursor’s own token.

An off-by-one injected where item-relative ranges convert back to absolute offsets passes the containment check and fails name identity on the first seed.

The data.table rewrites are checked once per input: they never overlap, every range lies inside the text, and when the input parses cleanly, so does the text with every rewrite applied.

Completion is sampled once per completion context (the kind of token the cursor sits in or after) rather than at every offset, because it costs far more per call than any other feature and its answer depends on the syntactic context rather than the exact byte.

The program corpus

crates/syntax/tests/programs/*.R.corpus holds 1,967 distinct R programs, read through syntax::testing::program_corpus_sources. The syntax, format, and semantics batteries each run the whole set. The corpus holds inputs only, with no expected output.

The inputs are usable on their own because the invariants need no expected output: nothing panics, syntax checks the lossless reprint and the tree geometry, format checks preservation, and semantics checks that every diagnostic range lies inside its file. The semantics arm shares one database across the whole corpus instead of using the per-input battery, because a fresh database re-parses the stubs every time. It therefore asserts only what a shared database can, and leaves determinism and incremental equivalence to the generated inputs. Every arm also asserts that the corpus is non-empty, so deleting it fails loudly instead of passing on zero cases.

The annotation round trip

crates/semantics/tests/test_roundtrip.rs asserts that every type the checker prints is a type you could write back. A #: TYPE annotation says the annotated value is compatible with TYPE, and the checker has just proved the value has the type it rendered, so pasting an inferred scheme above its own definition must add no finding.

No other test compares the renderer with the annotation grammar: every other suite reads a rendered type as a string, so a type that prints well but parses back as something else is invisible to them.

The test runs over every fixture case source. It skips cases that do not already check clean, because “re-declaring adds no finding” needs a baseline with no findings, and it skips definitions that already carry a #: block, because those render the user’s own text rather than a type the checker chose. It re-points one database with the salsa setter instead of building a new one per probe, because a fresh database re-parses the whole stub corpus.

KNOWN_UNWRITABLE lists schemes that are allowed not to round-trip. It is empty, and worth keeping that way. Its entries name case ids rather than skipping by shape, so a new unwritable rendering of any kind fails the test instead of blending into a known category.

Tree-sitter as a second opinion

crates/syntax/tests/test_corpus.rs compares our verdict on “does this file have errors?” with tree-sitter-r’s, in both directions and over two sets of inputs:

corpus_acceptance runs over the fetched corpus/ and gates the direction where we reject what tree-sitter accepts, which means a gap in our grammar.
in_tree_acceptance runs over inputs that are always present (the program corpus plus every fixture case source) and gates the other direction, where we accept what tree-sitter rejects.

That second direction is the only check that bounds the number of parse errors from below. Every other parser invariant bounds it from above, so without it an over-suppressing parser, the regression most likely to slip in given how much deduplication the parser does, would pass every fuzz battery while broken files went silently clean.

Each direction is gated only where it means something. Rejecting what tree-sitter accepts is noise on in-tree input, which is full of #: annotations (comments, to tree-sitter) and of deliberately broken sources. Accepting what tree-sitter rejects is gated there against crates/syntax/tests/in-tree-acceptance-allowlist.txt. Each allowlist entry is adjudicated by running R itself, and must say which of two things it is: tree-sitter being wrong, or a known gap in this parser.

Inline type annotations are ry syntax that R rejects by design, so tree-sitter judges the plain R program a source erases to (syntax::erase_inline_annotations), not the source itself. That makes the test a check on erasure too: an annotated fixture whose erased text tree-sitter rejects fails it. check_parse_invariants pins erasure from the other side, for every fuzz input that parses cleanly: the erased text parses cleanly, holds no inline annotation, and erases to itself.

The runtime

The runtime is tested against models and against GNU R rather than with fixtures. The oracle’s cases (crates/oracle/cases/, Corpus and oracle) hold the R-level behavior, and crates/interp/tests/test_fuzz.rs fuzzes lowering and checks that the tier-0 bytecode agrees with the evaluator. crates/jit/tests/test_tiers.rs does the same one tier up: it runs random loop, branch and vector programs as closure bodies under tier 0 and under the JIT forced on (jit::Mode::Force, which compiles a body on its second call and a loop on its first backward jump), and every call must give the same value or error. Most generated programs must run to a value, so a generator that drifts into erroring early fails the test instead of passing without comparing anything. The test interpreter has no base R closures (numeric, as.list, withCallingHandlers), so programs use builtins, and each hand-written seed states whether it is meant to fail. Behavior that needs base R, such as a warning handler watching a compiled loop, goes in the oracle case bytecode__hot_loops.R. crates/interp/tests/test_linalg.rs pins base R’s linear algebra to GNU R’s results bit for bit. The matrix products are ry’s own kernels, so their bits hold on every machine; the LAPACK cases hold only on the reference libraries, so on another BLAS (such as CI’s OpenBLAS) the test compares them with a GNU R on the same libraries, and skips them without one. crates/table/tests/test_table.rs checks the table engine against values read from data.table, test_lazy.rs there checks the Polars-style verbs, and crates/libr/tests/test_frame_altrep.rs checks the columns as ALTREP vectors. crates/interp/tests/test_records.rs, test_arrays.rs and test_standard_library.rs cover ry’s own values at run time, the last one also holding the runtime’s function table against the declarations in types/std.Rtypes. crates/libr/tests/test_rmath.rs, test_numerics.rs and test_linpack.rs compare the C API’s numerics with answers recorded from GNU R, so they run without R, and test_fuzz_rmath.rs checks properties of the distribution functions on random and extreme arguments. crates/heap/tests/test_gc.rs builds random object graphs (vectors, lists, interned strings, environments), drops roots, stores young objects into old ones and collects at random points, with and without collection stress, and after every collection checks each object the model still reaches. It also re-interns every string ever made, so an intern-table entry that outlived its string shows up. A disabled write barrier, a frame whose cells are not traced, and a minor collection that keeps dead strings in the table each turn it red.

Tests that load base R take it from the GNU R installation R_HOME names, and the runtime then behaves as that version; without R_HOME they use ry’s own base R (just base-r), and the scenarios skip when it is not built. The oracle’s expectations are GNU R 4.3.3 output, so CI sets R_HOME to 4.3.3. Run them once per supported version, for example R_HOME=/opt/R/4.6.1/lib/R cargo test -p interp, and the oracle the same way with ORACLE_R (Corpus and oracle).

crates/libr/tests/test_fuzz_native.rs fuzzes the native boundary itself. It nests every non-local exit native code can set up (Rf_eval, R_tryEval, R_ToplevelExec, R_tryCatchError, R_UnwindProtect, R_ExecWithCleanup and a cpp11-style C++ unwind) around values and errors, and checks the outcome, which cleanups ran, and that every C context, protection and R_alloc block was released. A second battery drives random API sequences under collection stress against a model.

crates/oracle/tests/test_differential.rs runs whole generated R programs (oracle::fuzz::program) under GNU R, ry’s tier 0 and ry’s JIT, and under collection stress against ry unstressed, and every engine must print the same output, warnings and errors. The generator covers loops shaped for the JIT, closures, S3 dispatch, conditions and on.exit, environments, attributes, NA and overflow, and about one program in eight ends on a deliberate error. cargo run -p oracle -- fuzz --seed N --count K --engines ry-tier0,ry-jit,ry-gctorture runs any range, prints each divergent program and its replay command, and just fuzz-differential and just gc-torture are the long runs. crates/oracle/tests/test_gallery.rs renders a seeded sample of the plot gallery under both engines and compares the images (Corpus and oracle). just gc-torture runs every oracle case on every core and takes about ten minutes on four cores in a debug build. RY_GC_STRESS=1 stresses from the script’s start and RY_GC_STRESS=startup from the first allocation, which takes about seven seconds on an empty script in a debug build.

crates/interp/tests/test_debugger.rs drives the debug adapter (interp::debugger) in-process. Scripted sessions pin breakpoints, stepping, the stack, variables, evaluate, browser() and uncaught errors as transcripts, and a seeded simulator plays a client that sends random, sometimes invalid, requests at every stop. Every request must get exactly one response, the session must end, and the interpreter must be left clean. FUZZ_SEED=<seed> replays a failing session.

The table engine

crates/table/tests/test_table.rs drives the engine from Rust. Every expected value that encodes data.table behavior was read from data.table under GNU R. Its differential fuzz generates random tables and queries and checks the native kernels against the Polars plans (query::execute_with_plans_only), which serve as the reference. FUZZ_ITERS raises the iteration count; run it in release, as FUZZ_ITERS=3000 cargo test-release -p table fuzz. The fuzz tables are small, so the kernels’ parallel paths get large-table tests of their own.

just bench-table <dir> [--rows 5e7 --groupby-only] runs the db-benchmark questions under data.table, as R code under ry run and on the engine from Rust, fails if any checksum differs from data.table’s, and rewrites the matching results table in development/design/r-runtime.md.

just bench-base [--rows 1e5,1e6 --only REGEX --memory] times base R’s own kernels (sorting, string formatting and regex, grouping, merge, match, matrix margins, cumulative sums, xts and zoo rolling operations) under ry run and GNU R, and prints the ratio ry / GNU R per kernel. A kernel whose checksum differs from GNU R’s is marked MISMATCH. With --memory every kernel runs in its own process, and the table adds both engines’ peak RSS. zoo and xts load from RY_BENCH_RLIB (default ~/R/library).

just bench-stdlib [filter] [n] times every List, Str, Dict and Result function at n = 1e6 by default against its fastest GNU R equivalent, and prints one row per function with the ratio ry / GNU R. The cases live in tools/bench/stdlib: ry.R holds ry’s side and gnu.R the R alternatives in base R, purrr, stringr and stringi under the same case names, and the fastest alternative sets the bar. A callback function is compared with R’s callback idioms (vapply, Filter, Reduce, a for loop, purrr), and a function on one string with R’s vectorized string functions. Where ry’s input is a vector, the case also runs it on an Array, ry’s unboxed vector, and the faster of the two counts. Each case runs in its own process, so a case slower than --timeout shows as timeout without stopping the rest, and a check column compares checksums of both results. Packages load from R_LIBS_USER, and an alternative whose package is missing is skipped and named in the row. --sort ratio puts the slowest functions first, and --write saves the table, as in tools/bench/stdlib/results.md.

Coverage-guided fuzzing

fuzz/ is a cargo-fuzz crate, in its own workspace and excluded from the main one. Each target is a thin wrapper around a battery its crate exports, which the crate’s in-tree harness runs too:

Target	Battery
`parse`	`syntax::testing::check_parse_invariants`
`format`	`format::check_format_invariants`
`semantics`	`semantics::testing::check_semantics_input`
`ide`	`ide::testing::check_ide_invariants`
`position`	`ry::position::check_position_invariants`
`config`	`ry::config::check_config_invariants`
`lsp`	the language-server simulator, `run_session`, reading its choices from the input
`load_index`, `reload`	`loader::testing::check_load_index_input`, `check_random_reload`
`sniff`	`sniff::check_sniff_invariants`
`csv`	`table::io::check_csv_input`
`interp_lower`, `interp_unserialize`, `bytecode`	`interp::testing::check_lowering_input`, `check_unserialize_input`, `check_bytecode_agreement`
`unserialize`	`R::serialize::check_unserialize_input`

The semantics target derives its document kind and an incremental edit from the input bytes, so a single byte stream drives the full battery, the splice cache included. The generator-driven targets (lsp, reload, bytecode) read the generator’s choices from the input through syntax::testing::ByteStream, so a byte mutation is a mutation of the program or session. The runtime targets link polars, so cargo fuzz build --no-default-features builds only the analysis targets on a machine with little memory.

It needs nightly, which the repository’s pinned toolchain already is. Run cargo install cargo-fuzz, then just fuzz-run format runs one target, and just fuzz-all 120 runs every target for two minutes each after seeding the corpora. Without just, from fuzz/:

cargo -Zscript ../scripts/seed-fuzz-corpus.rs   # seed from all fixture case sources
cargo fuzz run format -- -max_total_time=600 -print_final_stats=1
cargo fuzz run parse  -- -max_total_time=600

The seeder copies every fixture case’s source in the repository into fuzz/corpus/{parse,format,semantics}/. Files are content-addressed, so running it again only adds. The corpus and the artifacts are gitignored, because a durable regression belongs in the REGRESSIONS battery or in a fixture case, not in the corpus.

When the fuzzer finds something, minimize it with cargo fuzz tmin format <artifact>, fix the bug, replay the artifact, and add the minimized input to REGRESSIONS, plus a golden fixture case when the shape deserves a pinned rendering. To judge corpus quality, you can get a coverage report (the pinned toolchain includes the llvm-tools component it needs) with cargo fuzz coverage format && cargo cov -- show, as the cargo-fuzz coverage guide describes. Deep runs are manual or scheduled work; the bounded in-tree batteries remain the gate in the default suite.

Reloading

crates/loader/tests/test_reload_fuzz.rs generates packages from statements that define, wrap, read and rewrite a small pool of names, edits them at random, and checks that applying the reload plan to the loaded package gives what a fresh load of the edited package gives. A model executes both sides: a lazy binding’s value is its definition’s text, and a statement stores its text plus the values it read, so a statement that should have rerun shows up as a stale value. The same test checks that the incrementally derived index equals one built from scratch, and a second test feeds the index arbitrary bytes and asserts its positions stay inside the text. Removing either of the diff’s ordering rules turns it red within a few hundred seeds.

End-to-end suites

The CLI

crates/ry/tests/test_cli.rs drives the real ry binary end to end. It covers:

diagnostic rendering, with 1-based character positions, snippet windows, and underlines;
JSON Lines output;
the exit-code contract;
finding the configuration, and failing to;
per-file # typing: directives and suppression comments;
NAMESPACE validation, and project stub overrides including their loader-problem reports;
data-masked evaluation;
the formatter’s --check, --diff, and in-place modes;
ry run and ry R without R: help output, usage errors, a missing script, and strict mode refusing a script before any R starts.

crates/ry/tests/test_command_line.rs property-tests and fuzzes argument parsing for ry run, ry R and the ry repl alias against a model: everything after the script reaches it verbatim, options resolve to the documented runtime and mode, and no argument vector panics the parser. test_repl_e2e.rs runs every entry point on both runtimes: script arguments, -i, piped consoles, and the earlier spellings (--runtime, RY_RUNTIME, ry repl).

Docs examples

crates/ry/tests/test_docs.rs runs the examples in the docs through the real ry binary, so a page that shows stale output fails the suite. A block opts in with a marker: the word ry after the fence’s language, then the command and its options. Expressive Code ignores the extra words, so the site shows a plain block, and the pages’ Markdown versions drop the marker. Here an output block follows a tab group:

<Tabs syncKey="annotation-style">
<TabItem label="Inline">

```r
greet <- function(name: character, greeting: character = "hello") -> character {
  paste(greeting, name)
}
greet("Ada", greting = "hi")
```

</TabItem>
<TabItem label="Comments">

```r
#: fn(name: character, [greeting]: character) -> character
greet <- function(name, greeting = "hello") {
  paste(greeting, name)
}
greet("Ada", greting = "hi")
```

</TabItem>
</Tabs>

```text ry check typing
type-mismatch

  x this function has no parameter `greting`. Did you mean `greeting`?
```

A marked text block is the output of the nearest r block above it, or of every tab of the nearest tab group above it, and each tab must produce it on its own. Inside a <TabItem>, after the tab’s r block, it is the output of that tab alone, and a tab of the same group with no output block of its own must report no errors. The code runs in a throwaway project of one ry.toml and one file, main.R unless the output’s snippet names another file, as --[R/both.R:2:31] does. A tab group that no output block follows must give the same findings in every tab, with typing on. An unmarked output block is not tested.

Word	Meaning
`check` / `run`	Run `ry check`, or `ry run main.R`.
`typing`, `strict`	Set that `[check]` key to `true`. Type checking is off unless a block says `typing`.
`KEY=VALUE`	Any other `[check]` key, written as in `ry.toml`, such as `object-systems=false`.
`runtime=ry` / `runtime=gnu-r`	Which runtime `run` uses; GNU R by default, as on the command line.
`exact`	Compare the whole output instead of an excerpt.
`continue`	Run the page’s earlier `r` blocks first, as one session, and compare only this block’s output. A `check` takes every earlier block; a `run` takes only those whose output the page checks, so an illustration that needs a package the machine lacks does not stop it.
`tab=LABEL`	Only that tab produces this output, for the rare example whose spellings legitimately differ.
`dput`	On an `r` block with `#>` output, print each result with `dput()` instead of `print()`.

An r block can also hold its own output, as #> lines after the code that prints it, with a marker such as ```r ry run runtime=ry dput. Each run of code before a #> group shows its last value, and the comparison is always exact. A page’s #> blocks share one R process, each in its own local() environment, because a runtime start costs seconds and the standard library reference has over a hundred of them.

An excerpt matches when every non-blank line of it equals a line of the real output, both trimmed of surrounding whitespace, in the page’s order. Lines the page leaves out, such as the summary, a source snippet, or other findings, are free. A message the page wraps by hand matches one output line when its lines are joined with single spaces. This rule lets a page show the part of the output that matters while still pinning every word it does show. An exact block matches when the whole output agrees, ignoring trailing whitespace and the project’s temporary directory. Both compare standard output and standard error as one stream, interleaved as a terminal shows them, with color off.

just docs-test [PAGE] runs the pages whose path contains PAGE, and just docs-bless [PAGE] rewrites the output of exact and #> blocks from the real tool. An excerpt is curated by hand, so bless leaves it alone: the failure prints the full output to choose the lines from. A run block is skipped with a note when GNU R is missing, or for runtime=ry, when libR.so is not built beside ry. The landing page’s samples in docs/src/pages/index.astro are checked as excerpts with typing on, without markers.

Package loading

crates/ry/tests/test_load.rs runs ry load and plain Rscript against a real GNU R and checks the loaded result: Collate order, imports, S3 and S4 registration, Depends, sysdata.rda, source references, reloads that rebind, rerun and remove, undoing deleted S4 definitions, the error locations, and the ry load --r-entry entry point. test_repl_e2e.rs checks that ry R loads the package it starts in and reloads an edit before the next command. Both skip without R.

Native packages

The libr tests load libR.so the way an R host does, build packages from source with the system C and C++ compilers, load them with dyn.load (which runs R_init_<pkg>), and call their routines through .Call, all through the exported C API. The shared harness is crates/libr/tests/harness.rs. The packages are brio from the benchmark closure, vendored in tests/packages/brio, and tests/packages/rytest, written for these tests.

test_native.rs builds both against ry’s headers and checks the results, collection stress, the handle scope, PROTECT imbalance, errors unwinding through C and C++ frames, R_alloc release, finalizers, coercion, and the random number generator against GNU R’s output.
test_interrupts.rs checks that an interrupt raised while interrupts are suspended, from R or from C, stays pending until the next interrupt check. It loads base R, which changes how a C callable resolves for every later test in the session, so it runs in a process of its own.
test_gnu_r_binaries.rs builds both with GNU R’s R CMD SHLIB, against GNU R’s headers and linked to GNU R’s libR.so, and runs them under ry’s. It runs in a process of its own, so the two builds of one package never share a session.
test_headers.rs compiles every header as C11 and C++17 with warnings as errors, checks with the compiler that every declared name is exported, runs BLAS and LAPACK calls through the headers, and compares the graphics structs’ layout with GNU R’s. It also requires every name in tests/headers/r_api_names.txt (GNU R’s public header names that its R.dll exports, plus the internals packages link) to be declared by ry’s headers and exported by libR. With RY_GNU_R_INCLUDE and RY_GNU_R_EXPORTS pointing at a GNU R include/ and a list of its R.dll exports, api_names_cover_gnu_r checks the list against that release, and RY_UPDATE_API_NAMES=1 regenerates it.
test_altrep.rs and test_altrep_api.rs drive the altreptest package’s classes of every kind and base R’s own classes, against a report recorded from GNU R.
test_serialize.rs and test_serialize_native.rs read and write GNU R’s recorded streams byte for byte in every format and version, read installed lazy-load databases, and fuzz the reader.
test_connections.rs runs one connection scenario list in GNU R and in ry and compares the transcripts; test_connections_gnu_binary.rs repeats it with the ryconn package built by GNU R.
test_install.rs assembles ry’s R home and installs packages with R CMD INSTALL semantics. With RY_SOURCE_PACKAGES set to a directory of source tarballs, it builds each one against ry’s headers and reports the outcome.
test_corpus_binaries.rs loads every compiled package of the installed corpus (RY_CORPUS_LIBRARY, default .corpus/library) under ry, each in a child process with its dependencies, and writes target/tmp/corpus-binaries.tsv.
test_corpus_calls.rs replays native routine calls that tests/data/corpus_calls/record.R recorded under GNU R, reading arguments and results with ry’s unserializer, and requires every routine whose library loads to return what it returned under GNU R.

The two corpus suites pass without a corpus and report that they skipped.

These suites need a C and C++ toolchain, and GNU R for the second suite and the generator check. The parts that need GNU R find it through the R on PATH, taking its headers from R CMD config --cppflags and its Rscript from PATH, so Debian’s split layout and a source build both work. When no R is on PATH, or RY_SKIP_GNU_R=1 is set, those parts print why and skip.

The language server

crates/ry/tests/test_lsp.rs spawns the real ry server process and drives it over stdio with an async-lsp test client. It covers:

capability negotiation: position encodings, pull diagnostics, refresh, label offsets, and snippets;
document sync, both incremental and full-text, including untitled and non-file: buffers and re-reading from disk on close;
the two-wave push contract, its settled-superset invariant, and how a burst of edits settles under latest-edit-wins cancellation;
pull diagnostics: result ids, unchanged reports, retryable cancellation, and push suppression;
diagnostic refresh on save and on a configuration change;
configuration: live reload, a configuration in an ancestor directory, a broken configuration keeping the previous one in force, and the diagnostic on the configuration file itself;
every feature endpoint, including correct UTF-16 and UTF-8 ranges over BMP and non-BMP text, and safety for out-of-bounds positions;
semantic tokens for a #: body, and serving .Rtypes and NAMESPACE buffers.

The cancelled-pull test is made deterministic by a fault-injection seam. When the RY_TEST_DELAY_PULL_MS environment variable is set, the server announces each diagnostics pull by creating the RY_TEST_PULL_MARKER file, then holds the pull (for at most the delay) until the cancellation token flips. The test sends its didChange only after the marker appears, so the cancellation provably lands while the pull is in flight, and the retryable SERVER_CANCELLED response and the successful retry after it can be asserted without sleeping and hoping. Nothing else sets these variables.

crates/ry/tests/test_simulation.rs runs the in-process language-server simulator: random editor sessions under a seeded schedule, checked at random points against a server started fresh on the same disk and buffers. crates/ry/tests/test_scenarios.rs runs the scripted user journeys in crates/ry/tests/scenarios/. Robustness documents both, including the scenario format and the replay variables.

Incrementality

A fixture checks what an edit produces, but not how much work it took to get there. crates/semantics/tests/test_incremental.rs checks the work: semantics::testing::ProbeDatabase logs every query execution, and checked_since names the items whose item_check ran. The tests pin these contracts:

A body edit that keeps the item’s scheme re-checks the edited item alone.
A body edit that changes the scheme also re-checks the items that read it.
A new definition re-checks only the items that read its name.
An inserted statement re-checks only the reads it intercepts.
A body edit re-derives one item’s load facts and its own file’s index layer, a NAMESPACE edit re-derives no item, and a statement that only moved keeps its load facts.
A one-shot check (ProbeDatabase::one_shot) runs no package_name, test_name or interface_group and one project_interface_groups, reports exactly what the incremental check reports, and agrees with the per-name and per-item answers under invariants (checked in a child process, because invariants are process-wide and would perturb the counting tests). The incremental database still goes through the per-name and per-item firewalls.

The same file checks that a chain of re-exports resolves to its base, that a cycle of them pins to Unknown, and that a chain of 4,096 links resolves without overflowing the analysis thread’s stack. In the IDE harness, point_queries_recheck_only_what_an_edit_changed holds the features that run on every keystroke to the same rule.

Performance and memory

crates/semantics/tests/test_perf.rs asserts performance budgets over the real-file corpus. It is ignored by default, because it needs the fetched corpus and a release build, and CI does not fetch the corpus:

cargo test-release -p semantics --test test_perf -- --ignored --nocapture

just perf runs the same command. Its budgets are measured numbers plus headroom, and cover the cold-pass wall time per line, resolve steps per line (the tripwire for a regression in resolve memoization), resident bytes per line, and the p50 and p95 latency from a keystroke to fresh diagnostics on the corpus’s largest package. It prints what it measured, so it doubles as the corpus-wide measurement, while ry debug analysis-stats diagnoses one workspace phase by phase.

Run optimized tests with cargo test-release, an alias for cargo test --profile bench, which is what just perf and cargo bench use. The bench profile keeps opt-level = 3 but uses thin local LTO and 16 codegen units. Plain cargo test --release builds with the release profile’s fat LTO: building libr’s test_linpack that way peaks at 11.1 GiB of RSS and takes 30 minutes on four cores (it was OOM-killed once on a 16 GB machine), against 3.6 GiB and 11 minutes under cargo test-release. Shipped binaries and every benchmark number in the docs come from cargo build --release, so a budget measured under cargo test-release can be slightly slower than a release binary. Debug builds keep line tables only ([profile.dev] debug = "line-tables-only"), which keeps backtraces symbolized; set CARGO_PROFILE_DEV_DEBUG=true to inspect locals in a debugger.

crates/ry/tests/test_perf.rs holds the memory budget of a whole cold ry check --no-cache with type errors on. It generates the three 300,000-line benchmark code bases with tools/bench/generate.py, runs the release binary on each, and fails when one peaks above 256 MiB resident. It reads the child’s own peak with wait4, so it measures the process the user would run, allocator included. It is ignored by default for the same reasons:

cargo test-release -p ry-lang --test test_perf -- --ignored --nocapture

just perf runs both suites.

Guidelines

Add or tighten a fixture before writing a parser-local or crate-local unit test.
A new phase or module gets a fixture suite before it relies on ad hoc unit tests.
Use the lightest fixture change that captures the failing shape.
Change a fixture expectation deliberately, never as a side effect.