“Cuando todo era nada Era nada el principio Él era el Principio Y de la noche hizo luz Y fue el Cielo Y esto que está aquí” — Vox Dei

genesis-vibes

TL;DR: Shared Rust crate for cross-cutting CLI/AIX/self-healing infrastructure in the charly-vibes tool suite. Every tool depends on it instead of reimplementing the same conventions.

What is genesis-vibes?

genesis-vibes is the shared foundation of the charly-vibes suite. It generalizes patterns that appear across multiple tools — structured CLI output, config management, diagnostics, scaffolding, test fixtures, and agent feedback — into a single crate with consistent conventions.

Boundary rule: if only one tool uses it, it does not belong in genesis. Domain logic (metrics, stores, engines, analysis) stays in each tool.

Quick links

• Getting Started — add genesis-vibes to your project
• Modules Overview — full module reference
• Architecture — how the modules fit together
• Design Decisions — rationale behind key trade-offs

How-to guides

• Using the Envelope — structured CLI output
• Building a CLI with Guide — verbosity, format, error handling
• Adding a DoctorCheck — diagnostic checks with auto-fix
• Writing Tests with Fixture — test scratch environments

Modules at a glance

Status: “stable” modules have a settled API. “new” modules are functional but their API may evolve in minor version bumps.

charly-vibes Tool Ecosystem

Getting Started with genesis-vibes

TL;DR: Add genesis-vibes as a dependency, import the modules you need, and use Output::success() for structured CLI output.

What You Will Learn

By the end of this tutorial you will have added genesis-vibes to a Rust tool project, emitted structured CLI output with the envelope, and printed version info in machine-readable JSON. No prior knowledge of genesis-vibes is assumed.

Prerequisites

• A Rust project that uses clap for CLI argument parsing
• Rust 1.75 or later

Step 1: Add the dependency

Add genesis-vibes to your Cargo.toml:

[dependencies]
genesis-vibes = "0.13"

If you want the bleeding-edge version from the repository instead of the crates.io release:

[dependencies]
genesis-vibes = { git = "git@cv:charly-vibes/genesis.git", tag = "v0.13.2" }

Step 2: Emit structured output

Replace raw println!() calls with genesis’s Output type. Every command returns an Envelope<T> — callers check ok first.

use genesis::guide::{Output, Verbosity};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let output = Output::success("Project initialized")
        .with_next_step("Run `my-tool doctor` to verify setup");

    let verbosity = Verbosity::Normal;
    let mut stdout = std::io::stdout();
    let mut stderr = std::io::stderr();

    output.print(verbosity, &mut stdout, &mut stderr)?;
    Ok(())
}

Run your tool — you will see the result on stdout and a next-step hint on stderr:

$ my-tool init
"Project initialized"
→ Run: Run `my-tool doctor` to verify setup

The first line is the data payload (rendered with Debug formatting, hence the quotes); the → Run: line is the next-step hint, written to stderr so it stays out of piped output.

Step 3: Add --json output with format auto-detection

Embed CliFormat into your clap args to get automatic TTY detection:

use clap::Parser;
use genesis::guide::{CliFormat, CliVerbosity, Output, OutputFormat, Verbosity};

#[derive(Parser)]
#[command(name = "my-tool")]
struct Cli {
    #[command(flatten)]
    verbose: CliVerbosity,
    #[command(flatten)]
    format: CliFormat,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    let format: OutputFormat = cli.format.format();
    let verbosity: Verbosity = cli.verbose.verbosity();

    let output = Output::success(vec!["item1", "item2"]);
    output.emit(
        env!("CARGO_PKG_VERSION"), // your tool's version, not genesis's
        format,
        verbosity,
        &mut std::io::stdout(),
        &mut std::io::stderr(),
    )?;
    Ok(())
}

emit() requires the data payload to implement Serialize (needed for the JSON branch). Passing cli_version is your responsibility — see the cli-version ownership contract.

This auto-detects:

• TTY (interactive terminal) → human-readable text
• Piped/redirected (|, >, CI) → JSON envelope

Either can be overridden with --json or --human.

Step 4: Print version as JSON

Pre-parse --version --json before clap processes the rest of the args. The function returns true if it printed the version envelope — and in that case you must exit (it does not call std::process::exit() for you):

use genesis::cli::maybe_print_version_json;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // If `--version --json` (or `--version -j`) is passed, this prints the version
    // envelope and returns `true` — exit so clap doesn't handle --version too.
    // Plain `--version` is left for clap; the function returns `false`.
    if maybe_print_version_json("my-tool", env!("CARGO_PKG_VERSION")) {
        return Ok(());
    }

    // ... rest of your clap setup
    Ok(())
}

Note: The examples in Step 3 require clap with the derive feature in your Cargo.toml:

[dependencies]
clap = { version = "4", features = ["derive"] }

Recap & Next Steps

You added genesis-vibes to your project, emitted structured CLI output, enabled TTY-aware format detection, and wired up --version --json. Next, explore:

• Using the Envelope — deeper dive into warning/hint/error patterns
• Building a CLI with Guide — verbosity levels, error sinks, suggestions
• Adding a DoctorCheck — diagnostic checks with auto-fix

Using the Envelope

TL;DR: Every command returns an Envelope<T>. Check ok first, then inspect data. Use ErrorResult for errors — it enforces a non-empty remediation suggestion.

Context & Prerequisites

This guide explains how to use genesis’s structured output envelope for consistent CLI output. Before starting, ensure you have:

• Added genesis-vibes to your Cargo.toml
• Read Getting Started for basic usage

Constructing a success envelope

Use Envelope::success() (always passing your tool’s own CLI version as the first argument) or the Output helper for common cases:

#![allow(unused)]
fn main() {
use genesis::envelope::Envelope;
use genesis::guide::Output;

// Direct envelope construction — cli_version is YOUR tool's version
let env: Envelope<&str> = Envelope::success(
    env!("CARGO_PKG_VERSION"), // your tool, not genesis-vibes
    genesis::envelope::EnvelopeKind::Ok,
    "operation completed",
    vec![], // warnings
    vec![], // hints
);

// Using the Output helper (recommended for CLI commands)
let output = Output::success(vec!["item1", "item2"])
    .with_warning("config file is deprecated, migrate to config.toml");
}

cli_version is caller-supplied. It identifies the tool that emits the envelope (e.g. env!("CARGO_PKG_VERSION") in your crate). Genesis never injects its own package version — there is no zero-argument constructor. See the CLI version ownership contract.

Adding warnings and hints

Warnings signal non-blocking concerns. Hints suggest next steps.

#![allow(unused)]
fn main() {
let output = Output::success("Project initialized")
    .with_warning("Check your network connection for remote sync")
    .with_next_step("Run `my-tool doctor` to verify setup");
}

Returning errors

ErrorResult enforces Invariant 3.2.5: every error must include a remediation suggestion. The constructor returns Err if remediation is empty.

#![allow(unused)]
fn main() {
use genesis::envelope::{ErrorResult, RemediationEntry};

// Good — remediation is non-empty
let err = ErrorResult::new(
    "E_CONFIG_NOT_FOUND",                      // code
    "config file not found",                   // message
    None,                                      // rule_name
    None,                                      // spec_ref
    None,                                      // entity_id
    vec![],                                    // unmet_clauses
    vec![RemediationEntry {
        command: "my-tool init".to_string(),
        description: "Create a default config".to_string(),
    }],
)?;

// Bad — this returns Err
let err = ErrorResult::new("E_BROKE", "something broke", None, None, None, vec![], vec![]);
// => Err("remediation must be non-empty (Invariant 3.2.5)")
}

Each RemediationEntry pairs a runnable command with a short description. Consumers render them after the error message — see Reading the envelope below.

Choosing the envelope kind

EnvelopeKind is a closed enum. Use it to signal the type of response:

Reading the envelope

Consumers always check ok first:

#![allow(unused)]
fn main() {
use genesis::envelope::ErrorResult;

// A success envelope carries whatever payload the command produced
let envelope: Envelope<Vec<String>> = /* ... */;

if envelope.ok {
    for item in &envelope.data {
        println!("  - {item}");
    }
}
}

An error envelope (Envelope::error(...)) carries an ErrorResult as its data — check ok first, then render the remediation entries:

#![allow(unused)]
fn main() {
let envelope: Envelope<ErrorResult> = /* ... from a failed command ... */;

if !envelope.ok {
    let err = &envelope.data;
    eprintln!("Error [{}]: {}", err.code, err.message);
    for entry in &err.remediation {
        eprintln!("  → {} — {}", entry.command, entry.description);
    }
}
}

Troubleshooting: Common Fail-States

Check vs Doctor kinds

Both are success-path kinds; machine readers dispatch on envelope.kind:

• check — the payload is consumer-defined. Your check pipeline serializes whatever it computes (see dont’s check commands); another tool parsing your payload needs your docs. There is no shared schema.
• doctor — the payload is a DoctorReport with per-check name/status/message entries; fix commands are lifted into envelope hints and CheckStatus::Warn checks are surfaced as warnings. Produced by DoctorRunner / DoctorReport::into_envelope().

The cross-tool parse failure mode: deserializing a doctor payload as a generic check payload (or vice versa) fails or silently misreads — always branch on envelope.kind before choosing a schema.

Further Exploration

• Building a CLI with Guide — progressive-disclosure verbosity levels
• Adding a DoctorCheck — diagnostic framework that uses the envelope

Building a CLI with Guide

TL;DR: Use GuideBuilder to assemble genesis modules into a coherent CLI with progressive-disclosure verbosity, TTY-aware output format, and auto-generated suggestions.

Context & Prerequisites

This guide explains how to build a CLI tool using the guide module. Before starting, ensure you have:

• Added genesis-vibes to your Cargo.toml
• Familiarity with clap argument parsing

Adding verbosity and format to your CLI

Embed CliVerbosity and CliFormat as flattened clap args:

use clap::Parser;
use genesis::guide::{CliVerbosity, CliFormat, Verbosity, OutputFormat};

#[derive(Parser)]
#[command(name = "my-tool")]
struct Cli {
    #[command(flatten)]
    verbose: CliVerbosity,

    #[command(flatten)]
    format: CliFormat,
}

fn main() {
    let cli = Cli::parse();
    let verbosity = cli.verbose.verbosity();
    let format = cli.format.format();
    // ...
}

This gives you:

• -v / -vv / -vvv for progressive verbosity
• -q / --quiet for silencing output
• --json / --human for output format, with auto-detection

Using the Output helper

The Output type wraps an Envelope with a fluent builder API:

#![allow(unused)]
fn main() {
use genesis::guide::{Output, Verbosity};
use std::io::Write;

fn greet(name: &str, verbosity: Verbosity) -> Result<(), Box<dyn std::error::Error>> {
    let output = Output::success(format!("Hello, {name}!"))
        .with_warning("Name contains uppercase characters")
        .with_next_step("Try `my-tool wave` for a wave");

    let mut stdout = std::io::stdout();
    let mut stderr = std::io::stderr();
    output.print(verbosity, &mut stdout, &mut stderr)?;
    Ok(())
}
}

Format-dispatching with emit()

Use Output::emit() to let the format decide — human-readable text or JSON envelope:

#![allow(unused)]
fn main() {
use genesis::guide::{Output, OutputFormat, Verbosity};

fn list_items(
    cli_version: &str,
    format: OutputFormat,
    verbosity: Verbosity,
) -> Result<(), Box<dyn std::error::Error>> {
    let output = Output::success(vec!["item1", "item2", "item3"]);

    output.emit(
        cli_version,
        format,
        verbosity,
        &mut std::io::stdout(),
        &mut std::io::stderr(),
    )?;
    Ok(())
}
}

emit() takes your tool’s cli_version and the current Verbosity, and requires the data payload to implement Serialize (needed for the JSON branch).

Progressive-disclosure verbosity

The verbosity levels control what the user sees. CliVerbosity maps the -v count via Verbosity::from_verbose_count():

Use Verbosity::help_footer() to display a hint in your CLI help text:

fn main() {
    println!("{}", Verbosity::help_footer()); // "Use -v for..."
}

Assembling a CLI with GuideBuilder

GuideBuilder (created via Guide::builder()) collects all genesis modules into a coherent CLI runner. It registers valid commands for typo detection, sets up verbosity, and prepares config integration:

#![allow(unused)]
fn main() {
use genesis::guide::Guide;

let guide = Guide::builder("my-tool", env!("CARGO_PKG_VERSION"))
    .about("Does something useful")
    .commands(&["init", "doctor", "status"])
    .build();

// Run a command with format-aware output
// guide.run(|g| { ... });
// guide.run_formatted(format, |g| { ... });
}

See the source code of wai or dont for complete Guide usage examples.

Exit codes

Guide::run and Guide::run_formatted follow a documented exit-code contract, stable across versions:

Panics are never masked: there is no panic hook, and stack traces are never swallowed. A panicking command unwinds with Rust’s default runtime behavior (typically exit 101). Eval harnesses and shell scripts can therefore distinguish “graceful failure with hint envelope” (1) from “crashed” (any other nonzero).

Error handling with ErrorSink

ErrorSink handles errors with self-healing suggestions — it prints the error, optionally persists it to the error scratch, and optionally suggests the tool’s feedback subcommand:

#![allow(unused)]
fn main() {
use genesis::guide::ErrorSink;
use genesis::suggestions::Suggestion;

let sink = ErrorSink::new("my-tool");
let err = std::io::Error::new(std::io::ErrorKind::NotFound, "config.toml not found");

// Print the error with a suggestion footer to stderr
let mut stderr = std::io::stderr();
sink.handle(&err, &mut stderr);

// Or with an explicit suggestion footer override
let fix = Suggestion::Fix {
    description: "create a default config".to_string(),
    command: Some("my-tool init".to_string()),
};
sink.handle_with_footer(&err, &fix, &mut stderr);
}

ErrorSink is configurable: scratch persists the last error, suggest prints a Suggestion::Fix footer, context includes the full ContextBundle, and feedback_subcommand names the subcommand to suggest (set None to disable). Verbosity is honored via the verbosity field.

Troubleshooting: Common Fail-States

Further Exploration

• Using the Envelope — detailed envelope patterns
• Adding a DoctorCheck — diagnostic framework for your CLI’s doctor subcommand

Adding a DoctorCheck

TL;DR: Implement the DoctorCheck trait and register it with DoctorRunner to add structured diagnostics with optional auto-fix to your tool.

Context & Prerequisites

This guide explains how to add diagnostic checks to your tool’s doctor subcommand. Before starting, ensure you have:

• Added genesis-vibes to your Cargo.toml
• A working CLI (see Building a CLI with Guide)

Implementing a basic check

Create a struct that implements DoctorCheck:

#![allow(unused)]
fn main() {
use genesis::doctor::{DoctorCheck, DoctorRunner};
use genesis::suite_linter::{LintResult, Severity};
use std::path::Path;

struct ConfigFileCheck;

impl DoctorCheck for ConfigFileCheck {
    fn name(&self) -> &'static str {
        "config-file"
    }

    fn description(&self) -> &'static str {
        "Checks that the tool config file exists and is valid"
    }

    fn run(&self, repo: &Path) -> Result<Vec<LintResult>, Box<dyn std::error::Error>> {
        let config_path = repo.join("my-tool.toml");

        if !config_path.exists() {
            return Ok(vec![LintResult::error(
                "config-file",
                "Config file not found",
                "Run `my-tool init` to create a default config",
            )]);
        }

        Ok(vec![]) // pass — no issues
    }
}
}

Adding auto-fix

Implement fix to provide automatic remediation:

#![allow(unused)]
fn main() {
impl DoctorCheck for ConfigFileCheck {
    // ... name, description, run as above ...

    fn can_fix(&self) -> bool {
        true
    }

    fn fix(&self, repo: &Path) -> Result<Vec<LintResult>, Box<dyn std::error::Error>> {
        let config_path = repo.join("my-tool.toml");

        if config_path.exists() {
            return Ok(vec![]); // already fixed
        }

        std::fs::write(&config_path, "# Default config\nkey = \"value\"\n")?;

        Ok(vec![LintResult::info(
            "config-file",
            "Created default config file",
        )])
    }
}
}

Running checks with DoctorRunner

Register checks and run them:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runner = DoctorRunner::new(vec![
        Box::new(ConfigFileCheck),
    ]);

    let repo = std::env::current_dir()?;
    let report = runner.run(&repo, false)?; // false = don't fix

    println!("pass={} warn={} fail={}",
        report.summary.pass,
        report.summary.warn,
        report.summary.fail,
    );

    Ok(())
}

Pass true to enable auto-fix:

#![allow(unused)]
fn main() {
let report = runner.run(&repo, true)?; // true = run fixes
}

Integrating with the envelope

DoctorReport is serializable, so you can emit it as a JSON envelope:

#![allow(unused)]
fn main() {
use genesis::doctor::DoctorReport;
use genesis::envelope::Envelope;

let report: DoctorReport = /* ... */;
let envelope = Envelope::from(report);
envelope.print(&mut std::io::stdout())?;
}

Panic isolation

If a DoctorCheck::run() panics, the runner propagates the panic — it does not catch it. Wrap checks in std::panic::catch_unwind if you need panic isolation for individual checks.

Troubleshooting: Common Fail-States

Further Exploration

• Using the Envelope — structured output for doctor reports
• Module Reference — full DoctorCheck trait API

Writing Tests with Fixture

TL;DR: Use Fixture::new() to create temporary scratch directories with markers, config files, and git initialization for integration tests.

Context & Prerequisites

This guide explains how to use the fixture module for test scratch environments. Before starting, ensure you have:

• Added genesis-vibes to your Cargo.toml (dev-dependencies is fine)
• A tool that operates on a project directory

Creating a basic fixture

Build a temporary directory with markers and files:

#![allow(unused)]
fn main() {
use genesis::fixture::Fixture;

#[test]
fn test_detects_marker() {
    let fixture = Fixture::new()
        .with_marker(".my-tool")
        .with_file("config.toml", "key = \"value\"\n")
        .build()
        .expect("build fixture");

    // The fixture path is a real temp directory
    assert!(fixture.path(".my-tool").exists());
    assert!(fixture.path("config.toml").exists());

    // Contents match
    let contents = std::fs::read_to_string(fixture.path("config.toml")).unwrap();
    assert_eq!(contents, "key = \"value\"\n");
}
}

Adding git initialization

Use with_git_init() to set up a git repo for commands that detect git state:

#![allow(unused)]
fn main() {
let fixture = Fixture::new()
    .with_marker(".wai")
    .with_git_init()
    .build()
    .expect("build fixture");

assert!(fixture.path(".git").exists());
}

Running commands inside the fixture

Use Fixture::run() to execute a command with the fixture as the working directory:

#![allow(unused)]
fn main() {
#[test]
fn test_tool_init() {
    let fixture = Fixture::new()
        .build()
        .expect("build fixture");

    let output = fixture.run("my-tool", &["init"])
        .expect("run my-tool init");

    assert!(output.status.success());
    assert!(fixture.path(".my-tool/config.toml").exists());
}
}

Writing TOML configs

Use with_toml() to write a serializable struct as a TOML file:

#![allow(unused)]
fn main() {
use serde::Serialize;

#[derive(Serialize)]
struct ToolConfig {
    name: String,
    enabled: bool,
}

#[test]
fn test_reads_config() {
    let fixture = Fixture::new()
        .with_toml("my-tool.toml", &ToolConfig {
            name: "test".into(),
            enabled: true,
        })
        .build()
        .expect("build fixture");

    assert!(fixture.path("my-tool.toml").exists());
}
}

Verifying fixture contents

Check that the fixture has the expected structure using standard Rust assertions:

#![allow(unused)]
fn main() {
let fixture = Fixture::new()
    .with_marker(".wai")
    .with_file("data.txt", "hello")
    .build()?;

assert!(fixture.path(".wai").exists());
assert!(fixture.path("data.txt").exists());
}

Note: The fixture directory is automatically cleaned up when the Fixture value is dropped (it uses tempfile::TempDir internally).

Troubleshooting: Common Fail-States

Further Exploration

• Module Reference — full Fixture API

Creating Agent Evals for Your Tool

TL;DR: Build evals that check whether LLM agents use your tool correctly — not whether they can write code. Run scenarios in sandboxed fixtures against real subprocesses, assert on the machine-checkable signals your tool already emits (envelopes, exit codes, state transcripts, managed blocks), and classify every failure into a taxonomy that tells you what to fix.

Context & Prerequisites

This guide explains how to author agent evaluations — scenarios that measure whether an autonomous agent reads, parses, and acts on your tool’s output channels. It is different from unit testing: a unit test checks your binary against inputs you control; an eval checks an agent’s behavior against a task you’ve framed.

Before starting, ensure you have:

• Added genesis-vibes to your Cargo.toml
• Read Writing Tests with Fixture — fixtures are the eval sandbox
• Read Using the Envelope — envelopes are the primary assertion target

Why evals differ from tests

Your tool communicates with agents through structured channels: the JSON envelope (ok, warnings, hints, data), self-healing suggestions (DidYouMean / Fix), managed blocks in AGENTS.md, and discovery artifacts (llms.txt, .genesis/tools.toml). An agent can pass a task while ignoring every one of these channels — brute-forcing until something works — or fail a task while behaving flawlessly. A good eval battery measures protocol adherence separately from task success, because each failure mode points at a different fix: hint-blindness means your output channel needs work; task failure with perfect adherence means the task was misframed.

The core discipline: never score free-form agent text. Score only what crossed the process boundary — subprocess exit codes, exact stdout/stderr envelopes, and filesystem diffs. This eliminates the tool-hallucination confound, where a model narrates plausible tool output without ever running the command.

The eval formula

Every eval scenario is four things:

1. Fixture — the initial sandbox state
2. Agent prompt — what the agent is asked (framed at a difficulty tier)
3. Deterministic checks — assertions over envelopes, exit codes, and file state
4. Error code — what a failure means, in taxonomy terms

Step 1: Provision the sandbox

(Elaborates the Fixture provisioning and Sandbox confinement requirements of the evals-guidelines spec: the tool binary is provisioned before the agent’s first turn; live runs use an isolated HOME, a scrubbed environment, network denied by default, and a per-command timeout.)

Use Fixture for the scratch environment. The tool binary must be pre-built and on PATH before the agent’s turn — compiling from source inside a trial burns budget and measures build skills, not tool comprehension. (Installation is only in-scope if the scenario itself is about installation.)

Two safety rules for the sandbox: no network access unless the scenario explicitly requires it, and run the agent with an isolated HOME/environment (a live agent in a temp directory can still reach the real user home unless the harness blocks it).

Driving the agent depends on the cadence. For CI regression (every commit, zero-cost), use a mock or replay agent: a script that replays a recorded trajectory or a stub agent that follows a simple decision rule — the deterministic checks are the value, not the model. For external capability runs (nightly/scheduled, multi-model), use a harness like Inspect AI with per-trial container sandboxing and multi-provider model support.

#![allow(unused)]
fn main() {
use genesis::fixture::Fixture;

let fixture = Fixture::new()
    .with_marker(".genesis")
    .with_file(".genesis/tools.toml", "unknown_key = \"invalid\"\n")
    .with_file("AGENTS.md", "<!-- my-tool:START -->\nexisting context\n<!-- my-tool:END -->\n")
    .build()
    .expect("build fixture");
}

If the scenario depends on AIX artifacts, provision them explicitly — and see Step 5 for why you also want the ablated variant.

Step 2: Inject contrived failures

(Elaborates the Scenario provenance requirement of the evals-guidelines spec: every scenario traces to an observed failure source — a corpus entry, an error-analysis note, or a ticket. Contrived-failure scenarios are declared as channel stress tests; their results bound detection claims, never prevalence claims.)

Your tool’s self-healing output (hints, suggestions, doctor --fix) is a promise: errors help the agent recover. Evals are where you prove it. Deliberately trigger error paths and check whether the agent’s next action consumes the payload:

• Flag perturbation — invoke with a mistyped subcommand; expect the agent’s next call to use the DidYouMean suggestion.
• Invalid state — place the environment in an illegal state (e.g., a dont-style state machine mid-transition); expect the agent to follow the remediation in the hint envelope rather than editing state files directly.
• Corrupt config — malformed .genesis/tools.toml; expect the agent to run doctor / doctor --fix within a step or two of receiving the hint.

These are the highest-value scenarios in any battery: they test the exact value proposition of the AIX investment.

Step 3: Assert deterministically

(Elaborates the Process-boundary scoring, One fault per check, and Empty-trajectory guard requirements of the evals-guidelines spec: assert only on signals that crossed the process boundary; one fault classification per check; a passive model cannot pass by doing nothing.)

Prefer these checks, in order of reliability:

Step 4: Classify failures with a taxonomy

When a trial fails, assign an error code from a fixed vocabulary so results aggregate into dashboards instead of pass/fail noise. The codes and the agent/tool fault vocabulary they ride on are exactly what the report contract serializes:

• ERR_ENVELOPE_HINT_BLINDNESS — the tool returned ok: false with hints; the agent’s next command ignored the suggested fix
• ERR_STATE_MACHINE_VIOLATION — illegal transition, or bypassed state checks by hand-editing files
• ERR_MANAGED_BLOCK_CORRUPTION — agent overwrote or deleted managed-block markers
• ERR_TOOL_EXECUTION_HALLUCINATION — agent narrated a command it never executed (absent from the subprocess log)
• ERR_CONTEXT_RECOVERY_FAILURE — agent hit an unexpected error but never ran the ecosystem’s orientation commands (wai prime / wai status for context recovery, doctor for diagnostics)
• ERR_TOOL_DISCOVERY_FAILURE — agent defaulted to a generic approach, never discovering the specialized tool
• ERR_ACTION_FORMAT_VIOLATION — live-tier agent failed to produce a parseable structured action after one re-ask (the re-ask consumes a turn). Distinct from ERR_TOOL_EXECUTION_HALLUCINATION, which stays a replay-tier code for claims of runs that never happened — in the live tier the harness executes every action, so that failure mode is impossible there.

Step 5: A/B your AIX artifacts

Run every scenario twice: full (with llms.txt, managed AGENTS.md guidance, .genesis/tools.toml) and ablated (raw binaries, no AIX context). The score delta is the measured value of your documentation and signaling investment. If a hint channel shows no delta, it is either unread, unparsable, useless — or your scenarios are too easy for the agent to need it (or your sample count is too small to detect the delta). All five are actionable findings; the last two mean fix the scenario, not the tool.

Step 6: Run across model tiers

The same scenario should discriminate across capability tiers. Knobs that scale difficulty without changing the scenario:

• Prompt specificity — easy tiers get exact subcommands; hard tiers get intent only (“resolve the failing claim”), forcing discovery via --help / llms.txt
• Distractors — noisy logs, unused config files, decoy tools in the registry
• Perturbation for contamination defense — the suite’s docs (llms.txt, mdBook, crates.io) are public and likely in model training data, so a frontier model may be recalling syntax rather than comprehending it. Rename flags/subcommands in the sandbox (e.g., conclude → finalize) and require discovery via the local llms.txt; keep a set of private held-out fixtures that have never been published
• Step and token caps — budgets enforced by the harness; trajectories exceeding a step limit without progress are terminated early
• Sample counts — agentic runs are stochastic; run n ≥ 3 trials per scenario per model and report pass@k and variance, never single-run pass/fail

Anatomy of a complete scenario

Illustrative sketch. The transcript API below does not exist yet — it is the shape the planned evals module (genesis-zxv) will provide. Write your checks against captured subprocess logs directly today.

#![allow(unused)]
fn main() {
// evals/hint_adherence.rs — sketch
let fixture = Fixture::new()
    .with_file(".genesis/tools.toml", "unknown_key = \"invalid\"\n")
    .build()?;

// 1. Agent turn (via your harness): "Run diagnostics and resolve any errors."
//    Harness captures every subprocess call: argv, exit code, stdout.

// 2. Deterministic checks over the transcript:
let calls = transcript.calls();
assert!(calls.iter().any(|c| c.envelope_failed() && c.hints_mention("doctor --fix")),
    "tool must surface a self-healing hint for invalid config");
assert!(calls.iter().any(|c| c.argv.starts_with("my-tool doctor")),
    "agent should run the hinted fix — else ERR_ENVELOPE_HINT_BLINDNESS");

// 3. Post-fix state:
let final_env = run(&fixture, "my-tool doctor");
assert_eq!(final_env.exit_code, 0);
assert!(final_env.envelope.ok);
}

Step 7: Plant distractors and test doc-drift blindness

(Elaborates the Distractor registration requirement of the evals-guidelines spec: distractors are materialized into the replay environment but are not task material — whether consulting them is a fault is the check’s decision, never their presence.)

A frontier model that recalls your public docs will trust them over your tool’s live output — the most expensive failure mode in agentic use. genesis ships the mechanism for testing exactly this (add-aix-eval-loop §3):

#![allow(unused)]
fn main() {
use genesis::evals::{doc_drift_blindness, AgentStep, DistractorKind, Scenario};

let scenario = Scenario::new("doc-drift", "Initialize my-tool")
    // Bait: materialized in the sandbox, registered as stale docs.
    .distractor_file(
        "AGENTS.md",
        "<!-- my-tool:START -->\nRun `my-tool configure` to initialize.\n<!-- my-tool:END -->\n",
        DistractorKind::StaleDocs,
    )
    .check("doc-drift", doc_drift_blindness("my-tool configure"));

// Agent read --help and followed the envelope → pass.
// Agent ran the stale `configure` instead → agent fault
// ERR_DOC_DRIFT_BLINDNESS with the distractor path in the reason.
}

Distractors never fault a run by themselves — presence is not fault, the check returns the verdict. Compose doc_drift_blindness with ok_envelope / agent_followed_hint for action-level assertions, and attribute replays to models via ScenarioReport::with_model for matrix comparison.

Step 8: Run the tier ladder

(Elaborates the Tier ladder and Battery maintenance requirements of the evals-guidelines spec — openspec/specs/evals-guidelines/spec.md.)

Every consumer battery runs three tiers. Each tier states what it may claim — citing a tier for more than it measures is the failure the ladder exists to prevent:

Live trials never gate a push: they are directional evidence, not CI gates. A permanently green battery is never evidence of quality — see battery maintenance below.

The live action protocol

Your tier-2 harness requires the agent to return one structured action per turn — the exact command line plus a done flag. Per turn the model receives only: the scenario prompt and the fixture root path inside the sandbox. Never the check logic, the expected outcomes, or the distractor registry.

1. Malformed output → re-ask once, with the format error attached; the re-ask consumes a turn of the cap.
2. Malformed again → trial ends invalid_output, agent fault ERR_ACTION_FORMAT_VIOLATION.
3. Parseable action → execute the command exactly once, record the step with executed: true.

Bounds

Every live trial is bounded by a turn cap, a wall-clock timeout, and a token-estimate budget (the aix chars/4 heuristic over the trial’s full prompt and completion traffic — labeled as heuristic). The applied values are recorded in the report. A trial stopped at a bound ends failed and names the reached bound; the stop itself is not attributed as an agent or tool fault.

Free-tier entry and attribution

Use OpenRouter :free model ids as the entry tier — the weakest readers. If a weak model parses the envelope and honors the hint, the channel is robust (why). Record the raw id verbatim (deepseek/deepseek-v4-flash:free, suffix included) — matrix comparisons across time depend on the unnormalized id. Run n ≥ 3 repetitions per scenario × model; each repetition is a distinct report row. On HTTP 429: retry once after ≥ 2 s (record the delay used); a second 429 ends the trial rate_limited — never a tool fault, never a silent substitution of a different model.

Battery maintenance (staleness review)

Checks that never fail are rotting, not succeeding. Review the battery on a cadence: any check that has not failed since its scenario was created is reviewed for continued relevance (kept, sharpened, or retired) before the next tier-2 rotation cycle, and the decision is recorded in the scenario’s source. Declare contrived-failure scenarios as channel stress tests — their results bound detection claims, never prevalence claims (“we detect X when it happens”, never “X happens at rate Y”).

Fault routing

(Elaborates the Fault routing requirement of the evals-guidelines spec.)

A failure is a tool fault or an agent fault, never both:

• Tool faults (the tool misbehaved — envelope missing, exit code wrong) become tickets in your tool’s repo. They are product bugs wearing eval clothes.
• Agent faults (taxonomy-classified, ERR_* code attached) are routed by check code × output channel across ≥ 3 model ids (or all configured ids when fewer). A pattern that reproduces across models points at your output channel (AIX work in genesis), not at the model. Sub-threshold occurrences are recorded without action.

Reporting and aggregation

Every trial emits one report row in the interoperable JSON contract — see the eval report contract and the normative fixtures in tests/golden/eval_report_*.json. Dashboards aggregate rows by report_version with no per-tool mapping.

What genesis provides today (and what it doesn’t)

Available now: Fixture for sandboxing, envelope for structured output your assertions parse, suggestions for DidYouMean / Fix payloads, managed_block markers for boundary audits, doctor with auto-fix as a recovery target, and the evals module for deterministic scenario replay: Scenario::run over recorded AgentStep transcripts, envelope-assertion helpers (parse_envelope, ok_envelope, error_envelope_with_hint, agent_followed_hint, agent_executed_all), the ErrorTaxonomy classification, distractor fixtures with the doc_drift_blindness check, and feedback→Scenario conversion so user-reported failures become regression scenarios (see reference/modules.md).

Gaps worth tracking: live-agent harness capture (running a real model and recording its steps) stays outside the crate — Scenario::run replays recorded transcripts deterministically; orchestrate live runs and per-model matrices in your harness. AIX-ablation provisioning helpers remain a follow-up slice (genesis-zxv).

Related

• Token budgets for LLM consumption: estimate_token_cost and the bounded generators (generate_llms_txt_bounded / generate_llm_txt_bounded) degrade artifacts deterministically instead of overflowing a declared budget — useful when provisioning context for tier-limited models.

Running the Eval Tier Ladder in CI

TL;DR: Tier 0 on every push (no model, fast), tier 1 replay nightly (no model, recorded transcripts), tier 2 live on a schedule with a rotated :free matrix and a hard budget. Live trials never gate a push.

(Elaborates the Tier ladder requirement of the evals-guidelines spec — openspec/specs/evals-guidelines/spec.md.)

Context & Prerequisites

• An eval battery authored per Creating Agent Evals
• The tier definitions and their claims: Step 8: Run the tier ladder
• Reports emitted per the eval report contract

Tier 0 — static checks on push

No model is involved, so this tier belongs in the normal test job and gates the push:

• Envelope shape and schema checks over golden fixtures
• Doc-sync guards (llms.txt / llm.txt staleness)
• Scenario declarations well-formed: every scenario names its provenance (corpus entry, error-analysis note, or ticket), every declared check is present in the report

Runtime: seconds. May claim only “the channels are well-formed”.

Tier 1 — nightly scripted replay

Scenario::run over recorded AgentStep transcripts — deterministic, no model, no network. Schedule it nightly (GitHub Actions schedule), not on push: it exercises detection of every recorded failure mode and takes minutes, not seconds.

# .github/workflows/evals-replay.yml (sketch)
on:
  schedule:
    - cron: "0 3 * * *"   # nightly, off-peak
jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cargo test --test evals_replay -- --nocapture
      # Rows append to the report store in report_version 1 shape.

May claim “recorded failure modes are still detected”. A replay failure is a regression signal — triage it as the fault it reports (tool faults become tickets; agent faults route by check code × output channel).

Tier 2 — live, scheduled, rotated

Live model runs are a separate scheduled workflow that never gates a push:

1. Rotation — iterate the ordered model registry; :free ids run first (entry tier). Order defines scheduling priority, not failover: fallback to a different model is a caller-level decision, recorded per trial.
2. Repetitions — n ≥ 3 per scenario × model cell; each repetition a distinct report row (model verbatim, repetition 0-based).
3. Action protocol — one structured action per turn, one re-ask on malformed output, then ERR_ACTION_FORMAT_VIOLATION.
4. Bounds — turn cap, wall-clock timeout, token-estimate budget (aix chars/4 heuristic); applied values recorded in every row.

429 semantics

On HTTP 429 from the model provider: retry once after a delay of at least two seconds (record the delay used as retry_delay_secs); a second 429 ends the trial rate_limited — no fault attribution, no silent substitution of another model. Never let a rate-limited cell silently disappear: the row is the record that the cell did not run.

Budget

Cap tier-2 spend before the first run:

• A token-estimate budget per trial (the aix chars/4 heuristic, labeled as heuristic) — recorded in bounds.token_budget; exceeding it stops the trial failed with bound_reached: "token_budget" and no fault attribution.
• A per-run wall-clock budget for the workflow itself — if the schedule slips past the window, remaining cells are skipped and recorded as absent, not failed.
• Free-tier ids keep the marginal cost near zero; paid ids enter the registry only when the claim they buy justifies their cost.

What each tier buys you

A permanently green battery — any tier — is reviewed for staleness, not celebrated: checks that never fail are reviewed for continued relevance (kept, sharpened, or retired) before the next tier-2 rotation cycle.

Related

• Creating Agent Evals — authoring scenarios and the live action protocol
• Why weak readers — why :free ids are the entry tier
• The eval report contract — the JSON rows all tiers emit

Modules Overview

TL;DR: Reference table of all genesis-vibes modules with key types, traits, and entry points.

Module Map

envelope

Signature: genesis::envelope

Structured CLI output envelope. Every command returns an Envelope<T>.

Key Types

Functions

Constructors

CLI version ownership contract

cli_version is caller-supplied at construction — the first argument to every constructor. It must be the version of the tool that emits the envelope (typically env!("CARGO_PKG_VERSION") in the downstream tool’s own crate).

Genesis-vibes never injects its own package version: the misleading genesis-derived CLI_VERSION default was removed in the version that introduced this change. There is no zero-argument constructor that silently emits genesis’s version — pass your own version explicitly.

Migration path (for tools adopting the new contract):

1. At each Envelope::success / success_with_tx / error call site, add the tool’s own version as the first argument:

   #![allow(unused)]
   fn main() {
   Envelope::success(env!("CARGO_PKG_VERSION"), EnvelopeKind::Ok, data, vec![], vec![])
   }

2. For envelope-producing helpers (GuideOutput::to_envelope, DoctorReport::to_envelope, StatusReport::to_envelope), the cli_version parameter is now required and must be threaded from the caller.

3. Remove any use genesis::envelope::CLI_VERSION — the constant no longer exists.

4. Verify with cargo test that serialized envelopes carry the tool’s own version, not genesis-vibes’ version.

Receipt metadata (opt-in)

Envelope carries an optional receipt: Option<ReceiptMeta> recording the terminal outcome, retry identity, and user-visible evidence of a command run (add-aix-eval-loop D1). It is additive and opt-in:

• Envelopes constructed without with_receipt() serialize byte-identically to pre-receipt output — the receipt key is omitted entirely (golden-file tested in tests/envelope_golden.rs). No downstream change is required.
• TerminalOutcome classifies how a run ended: Success, Failure, Timeout, or Cancelled. Timeout is explicit, not silent failure, and is independent of the envelope’s ok field (a delivered result with a timed-out follow-up is representable).
• attempt: u32 and optional idempotency_key make retries of mutating commands distinguishable. Which commands qualify is a per-tool policy — genesis ships the mechanism, not the policy.
• evidence: Option<String> is a verifiable statement of the user-visible edge (“file X exists at path Y”), not free-form narrative.

#![allow(unused)]
fn main() {
use genesis::envelope::{Envelope, EnvelopeKind, ReceiptMeta, TerminalOutcome};

let env = Envelope::success(env!("CARGO_PKG_VERSION"), EnvelopeKind::Ok, data, vec![], vec![])
    .with_receipt(ReceiptMeta {
        terminal_outcome: TerminalOutcome::Success,
        attempt: 1,
        idempotency_key: Some("deploy-config".into()),
        evidence: Some("file exists at /tmp/out.txt".into()),
    });
}

guide

Signature: genesis::guide

CLI scaffold: verbosity, output format, error handling, and command dispatch.

Key Types

Functions

suggestions

Signature: genesis::suggestions

Self-healing error suggestions and typo detection.

Key Types

Functions

managed_block

Signature: genesis::managed_block

Managed block injector for <!-- NAME:START --> / <!-- NAME:END --> markers in markdown files.

Key Types

Functions

config

Signature: genesis::config

Shared config management with validation and error reporting.

Key Types

Functions

aix

Signature: genesis::aix

AIX artifact generation helpers for llms.txt, llm.txt, and AGENTS.md blocks, plus token-cost estimation and budget-bounded generation (add-aix-eval-loop §2).

Key Types

Functions

Token-cost heuristic

estimate_token_cost uses the chars/4 heuristic (ceiling division) with a documented ±25% error band against typical English prose in cl100k-class vocabularies — good enough for budget arbitration, not for billing. The heuristic name ships inside every TokenCost so an approximation is never mistaken for a tokenizer count. No tokenizer dependency is pulled in.

Budget degradation ladder

Both bounded generators degrade deterministically instead of overflowing or refusing. Granularity steps, in order:

1. at or under budget → byte-identical to the unbudgeted generator
2. truncate descriptions to their first sentence
3. drop optional (Raw) sections
4. drop table content — headings always survive; module names are the floor

The floor artifact is returned even if it still exceeds the budget: a verbose artifact beats a missing one. Existing generate_llms_txt / generate_llm_txt signatures are untouched (golden-file pinned in tests/aix_bounded.rs).

evals

Signature: genesis::evals

Deterministic evaluation harness: scenarios replayed against fake-agent transcripts, with envelope-assertion helpers, reusable checks, distractor fixtures, and the doc-drift blindness check (add-aix-eval-loop §3). No live LLM, no subprocess runner.

Key Types

Functions

Distractors and doc-drift blindness

Distractor files share the replay environment with real fixtures but are registered separately, so checks can tell bait from task material. Presence of a distractor never faults a run by itself — the check decides:

• doc_drift_blindness(bait_cmd) passes when the agent received a parseable envelope from an executed step and never issued bait_cmd (the action only correct per the stale docs). Action-level correctness composes with ok_envelope / agent_followed_hint.
• Doc-following steps fail as agent fault ERR_DOC_DRIFT_BLINDNESS with the distractor path in the reason.
• A scenario without a StaleDocs distractor makes the check a tool fault (misconfiguration), not an agent fault.

ScenarioReport serializes to JSON (name, passed, failures with taxonomy codes, model when attributed) for per-model matrix comparison; run()’s signature is unchanged — attribution happens post-hoc via with_model.

feedback

Signature: genesis::feedback

Agent issue reporting — wraps scratch (error persistence), context (env bundle), redactor (privacy), and gh (GitHub issue creation) into a single command.

Key Types

Functions

Issue title derivation

The title always starts with [{kind}] . What follows is resolved by precedence:

1. FeedbackArgs::with_title() / the downstream tool’s --title flag — wins over everything.
2. First stdin line, when the piped input is multi-line (the remainder becomes the ## Description body).
3. auto-reported error: {command} on the --from-last-error path.
4. Generic feedback report for single-line stdin.

Stdin contract

handle_feedback reads all of piped stdin (read_to_string, not a single line) — multi-paragraph reports are never truncated. A single-line input (including a trailing newline from echo) keeps the generic title and lands verbatim in the Description. Empty input is an error.

Sub-modules

Converting feedback to regression scenarios

Captured feedback becomes a replayable evals scenario (add-aix-eval-loop §4, design D4) — the aix-gap bundle → regression path:

• Scenario::from_feedback_context(bundle, fixtures) embeds the recorded failure signature (command, exit code, footer hint) as the reproduces-recorded-failure check. Bundle-only conversion (empty fixtures) yields a prompt-only scenario; caller-supplied (path, content) pairs become scenario fixtures via the existing fixture mechanism (the converting tool knows which files were in play — genesis never re-snapshots the working tree).
• feedback::from_last_error(tool_name, fixtures) wraps scratch::read_last_error; a missing record returns the typed ConversionError::NoScratchRecord, never a panic.
• Conversion + replay run in-process: no LLM call, no subprocess runner.

suite_linter

Signature: genesis::suite_linter

Suite-wide config lint checks. Foundation for the doctor module.

Key Types

Functions

fixture

Signature: genesis::fixture

Test scratch environments and runners.

Key Types

Fixture builder methods

doctor

Signature: genesis::doctor

Diagnostic framework with auto-fix.

Key Types

cli

Signature: genesis::cli

CLI helpers.

Functions

status

Signature: genesis::status

Cross-tool status dashboard.

Key Types

scaffold

Signature: genesis::scaffold

Init scaffolding builder.

Key Types

Scaffold builder methods

discovery

Signature: genesis::discovery

Tool discovery via .genesis/tools.toml manifest.

Key Types

Functions

git

Signature: genesis::git

Shared read-only git plumbing: one subprocess-based module for the repo queries every suite tool previously re-implemented with drift. Read-only only — no git init/add/commit (write paths stay with callers), no hook wiring (git_hooks owns that), no git library (the git binary is spawned as a subprocess).

Key Types

Functions

Policy-aware variants

Every named helper that spawns git has a *_with_policy(root, ..., policy) variant taking [EnvPolicy] as its last argument, sharing one internal implementation with the plain variant. The default stays Inherit because hook harnesses legitimately inject GIT_DIR et al.; call the variant with EnvPolicy::StripHookContext when running inside a git hook (e.g. the dont bd hooks) and the repository must be discovered from root alone.

git_hooks

Signature: genesis::git_hooks

Shared git-hook primitives consolidating overlapping mechanics from three donors (pretender, wai, espectacular). Contains only git-hook mechanics — consuming tools pass their own gate commands, markers, and block contents as parameters (guard-tested boundary).

Key Types

Functions

update_check

Signature: genesis::update_check — feature-gated behind update-check

Crates.io update availability checks for dependent binaries. Opt in by enabling the feature on the consumer’s genesis-vibes dependency (update-check = [...]); nothing activates at runtime otherwise.

Design contract ([genesis-2ex])

• Binaries, not libs, notify. Call check() with your own published crate name and installed version — genesis never checks itself from inside a dependent. The crate name must match the published crates.io package exactly: a mismatch 404s and fails silent by design (e.g. a crate published as wai-cli must not pass "wai").
• Cached-passive. At most one fetch per cache TTL (default 7 days, DEFAULT_TTL_SECS); results cached at <cache>/genesis/update-check/<crate>.json, honoring XDG_CACHE_HOME.
• Fail-silent. Any error — network, timeout, corrupt cache, unwritable cache dir — degrades to None. Never panics, never blocks meaningfully (2s connect / 5s total: CONNECT_TIMEOUT, TOTAL_TIMEOUT), never produces user-facing errors.
• CI-aware. CI=true (or 1) or GENESIS_NO_UPDATE_CHECK=<non-empty> skips the check entirely, before any IO.
• Rate-limit polite. crates.io 403/429 responses double the cache TTL and send a descriptive User-Agent.
• Scheme-agnostic versioning. Comparison is current != latest with yanked and semver pre-release versions filtered out — calendar versions need no special-casing.

Key Types

Functions

Validation and debugging

• Crate names are validated (is_valid_crate_name): non-empty, ASCII alphanumeric plus - and _ only — invalid names return None without IO.
• GENESIS_UPDATE_CHECK_DEBUG=1 emits one stderr line per skip/fail reason ([genesis-4mq]) — de-risks silent 404s while wiring a dependent.
• Minimal wiring in a binary’s main:

#![allow(unused)]
fn main() {
if let Some(info) = genesis::update_check::check(
    env!("CARGO_PKG_NAME"),
    env!("CARGO_PKG_VERSION"),
) {
    eprintln!("{}", genesis::update_check::notice(&info));
}
}

env!("CARGO_PKG_NAME") guarantees the crate name matches the published package — hardcoding a name invites exactly the silent mismatch the design contract warns about.

Eval Report Contract

The interoperable JSON contract for scenario eval reports (add-evals-guidelines, requirement Interoperable report contract). A report describes one trial: one scenario, replayed or run live, in one tool repo. Dashboards aggregate rows by report_version with no per-tool mapping.

The contract is normative at the docs/fixture level: this page and the two checked-in fixtures are the source of truth for v1.

• Tier-2 example (live trial, failed): tests/golden/eval_report_tier2.json
• Non-tier-2 example (replay trial, passed): tests/golden/eval_report_replay.json

Contract (report_version 1)

Required fields

Per-check entries

Each element of checks carries:

Tier-2 fields (required for live rows, optional otherwise)

Rows without attribution group as non-tier-2 in aggregation.

Optional fields (omit, never null)

The three vocabularies are distinct

1. status — the trial outcome. invalid_output carries the ERR_ACTION_FORMAT_VIOLATION agent fault; rate_limited carries no fault.
2. fault — the agent-vs-tool attribution of a failed check (or an invalid_output trial).
3. code — the shared ERR_* taxonomy classification of an agent fault. Never present on a tool fault.

Evolution rules

• Additive — a new optional field keeps report_version at 1; older consumers ignore unknown fields.
• Breaking — removing or re-typing a field bumps report_version. Consumers aggregate by version; a tool running an older battery stays legible.

Delta against the current ScenarioReport

ScenarioReport (src/evals.rs) keeps its replay-tier shape until the first consumer harness lands (design D2/D4: no live-runner machinery in genesis). Mapping and delta:

ScenarioReport’s serialization is validated against the non-tier-2 fixture by tests/eval_report_contract.rs. When a consumer harness lands, ScenarioReport evolves to this shape additively (or the version bumps) — per the additive-evolution rule.

Architecture

TL;DR: genesis-vibes is a shared crate that generalizes cross-cutting concerns (CLI output, config, diagnostics, scaffolding) from the charly-vibes tool suite into reusable abstractions.

Why a shared crate?

Before genesis, each tool in the charly-vibes suite reimplemented the same patterns: structured JSON output, verbosity levels, config file management, test fixtures, and agent feedback. Each implementation was identical in intent but diverged in details — a classic case of structural duplication.

genesis-vibes extracts these patterns into a single crate that every tool depends on. This ensures:

• Consistent CLI output — every tool speaks the same envelope protocol
• Single source of truth — a bug fix in the envelope propagates to all tools
• Lower maintenance — new modules (doctor, status, discovery) ship once and are available everywhere

Module boundaries

Modules are organized by cross-cutting concern. The boundary rule: if only one tool uses it, it does not belong in genesis.

┌─────────────────────────────────────────────────────────┐
│                    genesis-vibes                         │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ envelope │  │  guide   │  │  config  │  ...          │
│  │ (output) │  │  (CLI)   │  │  (files) │              │
│  └──────────┘  └──────────┘  └──────────┘              │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │  doctor  │  │  status  │  │ scaffold │              │
│  │ (checks) │  │(dashboard)│  │ (init)   │              │
│  └──────────┘  └──────────┘  └──────────┘              │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ fixture  │  │ feedback │  │ discovery│              │
│  │ (tests)  │  │ (issues) │  │ (manifest)│              │
│  └──────────┘  └──────────┘  └──────────┘              │
└─────────────────────────────────────────────────────────┘
         ▲            ▲            ▲
         │            │            │
    ┌────┴────┐  ┌────┴────┐  ┌────┴────┐
    │  wai    │  │  dont   │  │testaruda│  ...
    └─────────┘  └─────────┘  └─────────┘

Module relationships

Some modules build on others:

• guide uses envelope — Output<T> wraps Envelope<T>
• doctor uses suite_linter — DoctorCheck produces LintResult
• status uses doctor — DoctorStatusBridge wraps DoctorRunner as a StatusContributor
• feedback uses envelope — error reporting follows the envelope protocol
• scaffold uses managed_block — managed_block() delegates to BlockInjector
• discovery uses config — Manifest is a ConfigFile

Extending genesis

New modules should follow the same patterns:

1. Define a trait for the abstraction (e.g., DoctorCheck, StatusContributor)
2. Provide a runner/builder that orchestrates implementations
3. Implement envelope serialization for structured output
4. Document the module’s boundary rule — what belongs in genesis vs. what stays in the tool

Agent workflow integration

genesis modules are designed to be consumed by AI agents as well as humans:

• Envelope — agents parse JSON output; ok field is a binary success check
• Feedback — agents can call handle_feedback() to report issues
• Discovery — agents scan .genesis/tools.toml to discover registered tools
• Scaffold — agents use init commands to set up project structure
• AIX — agents generate llms.txt/llm.txt files for project context

Design Decisions

TL;DR: Key trade-offs and rationale behind genesis-vibes’s architecture — why the envelope is the single output format, why TTY detection matters, and how modules stay focused.

Single envelope format

Decision: Every command returns an Envelope<T>. Callers check ok first.

Rationale: A single output format eliminates the “parse the help text” anti-pattern. Whether the consumer is a human, an agent, or a CI pipeline, the response shape is identical. The ok boolean is a universal success indicator; structured data is in data, errors in error.

Trade-off: Every command must construct an envelope. The overhead is negligible (a few allocations per command), but it imposes a discipline on output that some tools may find constraining.

Mandatory error remediation

Decision: ErrorResult::new() returns Err if the remediation string is empty.

Rationale: An error without a suggested fix is a dead end for the user — and worse, a dead end for an AI agent that cannot ask for clarification. The ErrorResult constructor enforces that every error includes a recovery path.

Trade-off: Some errors genuinely have no remediation (e.g., disk full). In those cases, the remediation should say something like “Free up disk space and retry” — a general suggestion is better than none.

TTY-aware output format

Decision: CliFormat::format() auto-detects stdout: TTY → Human, piped → Json.

Rationale: Agents and CI pipelines pipe stdout — they always get parseable JSON without any flags. Humans at a terminal get readable output. Either can be overridden with --json or --human.

Trade-off: Auto-detection can surprise users who pipe to less (TTY) and expect JSON. The explicit override exists for this case.

Closed EnvelopeKind enum

Decision: EnvelopeKind is a closed enum — adding a variant requires a deliberate decision and a conformance test update.

Rationale: An open set of kinds would let each tool invent its own variant, fragmenting the protocol. A closed enum forces coordination: if you need a new kind, you modify genesis, and all tools benefit.

Trade-off: Tools that need a truly unique kind must either reuse an existing kind (e.g., Info) or open a discussion to extend the enum.

DoctorCheck trait over function pointers

Decision: DoctorCheck is a trait with name(), description(), run(), can_fix(), fix().

Rationale: A trait provides a stable contract that can be extended with new methods (e.g., suggest()) without breaking existing implementations. Function pointers would require a breaking change for any new capability.

Trade-off: More boilerplate for simple checks. A macro (impl_doctor_check!) could reduce this if needed.

Discovery via TOML manifest

Decision: Tools register themselves in .genesis/tools.toml during init.

Rationale: Hardcoded tool lists in orchestrators like wai require a code change every time a tool is added or renamed. A filesystem manifest decouples tool registration from orchestrator code — wai just scans the manifest.

Trade-off: The manifest can drift from reality (a tool is removed but the manifest entry persists). The DetectedTool::detected field mitigates this by checking if the tool’s marker still exists.

Fixture uses real temp directories

Decision: Fixture creates real temporary directories on disk, not in-memory fakes.

Rationale: Integration tests that run real subprocesses need a real filesystem. In-memory fakes (e.g., tempfile::TempDir) would miss filesystem-level issues like permission errors, symlink resolution, and cross-device moves.

Trade-off: Slower than in-memory mocks. For unit tests that don’t touch the filesystem, use standard Rust mocks.

Scaffold returns created/existed paths

Decision: Scaffold::build() returns ScaffoldResult with created and existed path lists.

Rationale: Tools need to know which paths were newly created (for reporting) vs. which already existed (for idempotency). A simple boolean return would lose this information.

Trade-off: The caller must handle the result struct. For most use cases, the idiomatic pattern is let result = scaffold.build().unwrap(); and ignore the result.

Why Weak Readers

Why the eval tier ladder makes OpenRouter :free model ids the entry tier of the model registry, and why variability across weak models is a signal, not noise.

(Elaborates the Free-tier entry and attribution requirement of the evals-guidelines spec — openspec/specs/evals-guidelines/spec.md.)

The weakest reader defines the channel’s strength

A tool communicates with agents through structured channels: the JSON envelope, hints, self-healing suggestions, managed blocks, discovery artifacts. Frontier models will often recover from a muddy signal by brute-forcing — reading your source, guessing flags, retrying. That recovery hides channel defects: the model succeeds despite the channel, and the battery goes green while real users’ weaker models fail.

The :free tier inverts this. A weak model that cannot compensate has one path to success: parse the envelope, read the hint, follow it. If a weak model parses the envelope and honors the hint, the channel is robust — the protocol, not the model, carried the interaction. That is the property worth measuring.

Variability as signal

Weak models are noisy: the same scenario yields different trajectories run to run. In this design the variance is not an annoyance to average away — it is the measurement:

• High pass-rate variance across repetitions on one scenario says the scenario’s signal is marginal relative to the channel’s noise floor — the scenario is not yet sharp enough to discriminate.
• A failure that reproduces across ≥ 3 weak model ids (or all configured ids when fewer) points at the output channel, not the model — route it to AIX work in genesis, not to a model-choice decision.
• A failure isolated to one model id stays directional evidence; it does not route anywhere by itself.

This is why the contract records the raw model id verbatim (including :free) and one row per repetition: matrix comparisons across time depend on unnormalized ids and distinct rows. Aggregated averages would erase exactly the variance that is the signal.

Nothing special-cased

The ordering is the only privilege :free ids get: they are the default first candidates of an ordered registry, which defines trial scheduling priority. No failover semantics, no scoring bonus, no separate code path — model fallback is a caller-level decision, recorded per trial when used. A weak reader that passes is a channel success; a frontier model that fails is a channel finding too, and often a cheaper one to reproduce on the entry tier first.

Trade-offs, stated plainly

• Free tiers are rate-limited (hence the 429 protocol: one spaced retry, then rate_limited — recorded, never silently substituted) and can be slow or de-prioritized at peak; tier-2 runs are scheduled, not gating.
• Weak readers cap scenario complexity: intents a weak model cannot parse at all measure the model, not the channel. Scale prompt specificity by tier (exact subcommands for entry tiers, intent-only for frontier tiers) so each tier measures the channel within its reach.
• Free-tier ids rotate and deprecate; the verbatim id + report rows make the drift visible instead of silently changing what “the battery” measured.

Related

• Creating Agent Evals — the tier ladder and action protocol
• Running the ladder in CI — rotation, 429 semantics, budget
• The eval report contract — raw attribution, repetition rows

Removed & Renamed APIs

TL;DR: This is the maintained source of the doc-drift blocklist in tests/doc_sync.rs. When a public API is removed or renamed, add the old call pattern as a row here — CI then fails if the old usage resurfaces in any markdown doc.

Why this exists

Between v0.4 and v0.6 the API moved (emit signatures, ErrorSink, ErrorResult, output data handling), but the onboarding docs kept showing the v0.4 snippets — nothing compiled them, so nothing failed. A documentation review found 4 of 6 copy-paste snippets broken. This table, consumed by tests/doc_sync.rs::forbidden_api_patterns, is the preventive layer: once a pattern is listed here, any reappearance of the old usage in the book (or README) fails cargo test.

Maintenance contract

• Add a row whenever a public API is removed or renamed, quoting the old call shape as the pattern.
• The pattern cell must be a single backtick-quoted literal matched verbatim against every markdown file under docs/ plus README.md.
• Keep the reason cell short and actionable: name the replacement.
• Exception: this page is excluded from the scan — it necessarily quotes the forbidden patterns.

Forbidden patterns

Related

• CLI version ownership contract
• Drift guards: tests/doc_sync.rs; snippet mirrors: tests/doc_examples.rs

aix Specification

Purpose

TBD - created by archiving change add-aix-eval-loop. Update Purpose after archive.

Requirements

Requirement: Token-cost estimate on generated artifacts

genesis SHALL provide a token-cost estimate for generated llms.txt / llm.txt artifacts, clearly labeled as a heuristic estimate, without changing the existing generator function signatures.

Scenario: estimate is labeled, not mistaken for a count

• WHEN any generated artifact is measured with estimate_token_cost
• THEN the returned TokenCost SHALL include the chars/4 heuristic name alongside the estimate

Scenario: existing generator signatures are unchanged

• WHEN generate_llms_txt or generate_llm_txt is called
• THEN the return type SHALL remain String
• AND existing callers SHALL compile without modification

Scenario: adding modules increases the estimate

• WHEN a ModuleEntry is added to the project metadata
• THEN the reported estimate SHALL be strictly greater than before

Requirement: Budget-bounded generation with graceful degradation

genesis SHALL provide bounded variants of both generators (modules-based llms.txt and section-based llm.txt) that produce artifacts under a declared token budget by degrading content granularity deterministically instead of overflowing or refusing.

Scenario: artifact within budget is generated verbatim

• WHEN the full artifact’s estimate is at or under the declared budget
• THEN the bounded variant SHALL return byte-identical output to the unbudgeted generator

Scenario: over-budget artifact degrades descriptions first

• WHEN the full artifact exceeds the budget
• THEN module descriptions SHALL be truncated to their first sentence before any section is dropped

Scenario: degradation drops optional sections before tables

• WHEN degradation must remove content beyond description truncation
• THEN raw/optional sections SHALL be dropped before module tables
• AND module headings SHALL survive all degradation levels

Scenario: degradation is deterministic

• WHEN the same inputs and budget are used twice
• THEN both outputs SHALL be byte-identical

Requirement: Provenance footer on generated artifacts

Generated llms.txt / llm.txt artifacts SHALL support an opt-in provenance footer line recording the generator version and a content hash. The footer SHALL be appended after all content sections. Deterministic generators SHALL remain deterministic: the timestamped footer variant is a separate function.

Scenario: default generators stay deterministic

• WHEN generate_llms_txt or generate_llms_txt_bounded is called twice with identical inputs
• THEN both outputs SHALL be byte-identical and free of timestamps

Scenario: footer states generator version and content hash

• WHEN an artifact is generated with the provenance footer enabled
• THEN the final lines SHALL identify the generating tool version and a hash of the artifact content
• AND the token-cost estimate of the artifact SHALL include the footer

Scenario: bounded generation degrades content but keeps the footer

• WHEN a bounded artifact overflows its budget and degrades descriptions
• THEN the footer SHALL reflect the hash of the degraded content

envelope Specification

Purpose

TBD - created by archiving change add-aix-eval-loop. Update Purpose after archive.

Requirements

Requirement: Receipt metadata on envelopes

genesis SHALL allow tools to attach optional receipt metadata to an envelope recording the terminal outcome, retry identity, and user-visible evidence of a command run.

Scenario: envelope without receipt serializes unchanged

• WHEN an envelope is constructed without with_receipt()
• THEN the serialized JSON SHALL NOT contain a receipt key
• AND parse_envelope SHALL accept it exactly as before this change

Scenario: receipt round-trips

• WHEN an envelope carries ReceiptMeta with terminal outcome, attempt, and evidence
• THEN serde round-trip SHALL preserve all fields
• AND parse_envelope SHALL accept the enriched envelope without change to its return type

Requirement: Terminal outcome classification

ReceiptMeta SHALL classify the terminal outcome of a run as Success, Failure, Timeout, or Cancelled.

Scenario: every terminal boundary is representable

• WHEN a tool records a run that ended in success, failure, timeout, or cancellation
• THEN each case SHALL map to exactly one TerminalOutcome variant

Scenario: timeout is explicit, not silent failure

• WHEN a tool’s command hits its deadline
• THEN the tool SHALL be able to emit TerminalOutcome::Timeout in the receipt
• AND the envelope’s ok field and the receipt’s outcome SHALL remain independently settable (a delivered result with a timed-out follow-up is representable)

Requirement: Retry identity for mutating commands

ReceiptMeta SHALL support an attempt counter and optional idempotency key so that duplicate or retried runs are distinguishable.

Scenario: retried command carries attempt number

• WHEN a tool emits envelopes for attempt 1 and attempt 3 of the same mutating command
• THEN the receipts SHALL show attempt 1 and 3 respectively
• AND both MAY share the same idempotency key

evals Specification

Purpose

TBD - created by archiving change add-aix-eval-loop. Update Purpose after archive.

Requirements

Requirement: Distractor fixtures in scenarios

genesis SHALL allow scenarios to declare distractor files that exist in the replay environment but are not part of the task’s happy path.

Scenario: distractor is visible to the replay environment

• WHEN a scenario declares a distractor file and runs against agent steps
• THEN the fixture environment SHALL contain the distractor file at its declared path

Scenario: distractors do not fail an otherwise-correct run

• WHEN the agent ignores a distractor and satisfies all scenario checks
• THEN the scenario report SHALL pass

Requirement: Doc-drift blindness check

genesis SHALL provide a reusable check asserting that the agent trusted the tool’s structured output over stale documentation contradicting it.

Scenario: agent trusts envelope over stale docs

• WHEN a scenario includes a stale-docs distractor contradicting the tool’s envelope
• AND the agent’s steps show tool invocation followed by envelope-consistent action
• THEN the doc_drift_blindness check SHALL pass

Scenario: agent follows stale docs over the envelope

• WHEN the agent’s steps show action consistent with the stale doc and inconsistent with the envelope
• THEN the check SHALL fail as agent_fault
• AND the reason SHALL identify the distractor path

Requirement: Model attribution on scenario reports

Scenario reports SHALL optionally record which model replayed the scenario, recording the raw model id verbatim — including :free and any deployment suffix — because matrix comparisons across time depend on the unnormalized id. (Repetition indexes, added by the evals-guidelines capability, ride on the same report.)

Scenario: report carries model when provided

• WHEN a replay is attributed to a model name
• THEN the scenario report SHALL include that name for matrix comparison
• AND the id SHALL be recorded verbatim, including :free or deployment suffixes

Scenario: report without attribution stays valid

• WHEN no model name is provided
• THEN the report SHALL serialize without a model field

Requirement: Action-format-violation fault code

genesis SHALL extend ErrorTaxonomy with ActionFormatViolation (ERR_ACTION_FORMAT_VIOLATION): the live agent failed to produce a parseable structured action after one re-ask. It SHALL be distinct from ToolExecutionHallucination, which remains a replay-tier code for claims of runs that never happened.

Scenario: round-trip through the code

• WHEN from_code("ERR_ACTION_FORMAT_VIOLATION") is called
• THEN it SHALL return the ActionFormatViolation variant
• AND code() on that variant SHALL return the same string

Scenario: existing codes unchanged

• WHEN any pre-existing ERR_* code is round-tripped
• THEN the mapping SHALL be unchanged

Requirement: Receipt terminal-outcome check

A composable ScenarioCheck SHALL exist that evaluates a run’s final envelope: it passes when the envelope carries receipt.terminal_outcome. The check itself does not know which runs are mutating — the caller scopes it (e.g., applies it only to steps it has marked as mutating), mirroring the envelope contract where tools decide which commands qualify as mutating.

Scenario: envelope with receipt passes the check

• WHEN the replayed step’s final envelope contains a receipt with a terminal outcome
• THEN the check SHALL return a passing CheckOutcome

Scenario: missing receipt is a tool fault

• WHEN the caller applies the check to a step it has marked as mutating and the replayed step’s final envelope carries no receipt
• THEN the check SHALL return tool_fault citing the missing receipt

Scenario: success receipt over failing envelope is a lie

• WHEN the final envelope has ok: false but its receipt records terminal_outcome: success
• THEN the check SHALL return tool_fault with a reason naming the contradiction

evals-guidelines Specification

Purpose

TBD - created by archiving change add-evals-guidelines. Update Purpose after archive.

Requirements

Requirement: Tier ladder

Consumer eval batteries SHALL be organized as three tiers with distinct cadences and claims: tier 0 static checks (no model, every push), tier 1 scripted replay of authored trajectories (no model, nightly), tier 2 live model runs (scheduled, rotated across model × scenario cells). A tier’s results SHALL NOT be cited beyond what that tier measures.

Scenario: tier claims are bounded

• WHEN a consumer documents its eval results
• THEN tier-1 results SHALL be presented as check-logic validation over authored trajectories, not as evidence of live agent behavior

Scenario: tier cadences

• WHEN a consumer configures CI
• THEN tier 0 SHALL run on every push, tier 1 on a nightly schedule, and tier 2 on a scheduled rotation that runs every configured cell (scenario × model id) at least once per declared cycle, with a cycle no longer than 14 days
• AND the rotation schedule SHALL be declared in the consumer’s CI configuration

Scenario: contrived failures bound prevalence claims

• WHEN results come from a scenario whose failures are injected or contrived
• THEN they SHALL be presented as detection-capability evidence for the tool’s error channels, not as an estimate of real-world failure frequency

Requirement: Battery maintenance

Consumers SHALL review the battery for staleness: a check that has never failed since its scenario was created SHALL be reviewed for continued relevance (kept, sharpened, or retired) before the next tier-2 rotation cycle, and the decision SHALL be recorded in the scenario’s source. A permanently green battery SHALL NOT be cited as evidence of tool quality.

Scenario: never-failing checks are reviewed

• WHEN a check has never failed across its scenario’s recorded runs
• THEN the consumer SHALL review it for continued relevance before the next tier-2 rotation cycle
• AND the decision SHALL be recorded in the scenario’s source

Requirement: Process-boundary scoring

Checks SHALL assert only on signals that crossed the process boundary: subprocess exit codes, captured stdout/stderr envelopes, and filesystem state. Free-form agent text SHALL NOT be scored.

Scenario: envelope is the assertion target

• WHEN a check validates a tool’s error handling
• THEN it SHALL parse the captured stdout envelope and assert on ok, error codes, and hint commands — not on agent commentary

Scenario: hallucination remains detectable

• WHEN an agent claims a command ran without a corresponding subprocess spawn
• THEN the step’s executed field SHALL be false
• AND the scenario MAY classify this as ERR_TOOL_EXECUTION_HALLUCINATION

Requirement: Fixture provisioning

Each scenario SHALL declare a fixture (tool binary pre-installed on PATH before the agent turn) and an agent prompt.

Scenario: tool binary provisioned before the turn

• WHEN a scenario is provisioned
• THEN the tool binary SHALL be pre-built and on the sandbox PATH
• AND compiling from source inside a trial SHALL NOT occur unless the scenario is explicitly about installation

Requirement: Distractor registration

Scenarios MAY register distractor files, which SHALL be registered separately from task material. Contrived failures, where applicable, SHALL be declared as part of the fixture declaration.

Scenario: distractors are not task material

• WHEN a scenario includes distractor files
• THEN they SHALL live in the distractor registry, distinct from the fixture’s task material

Requirement: Scenario provenance

Every scenario SHALL trace to an observed failure source recorded in its declaration: an entry in the tool’s failure corpus, an error-analysis note, or a ticket. Contrived failures MAY stress-test an error channel, but such scenarios SHALL be declared as channel stress tests — distinct from observed-failure scenarios — and their results SHALL NOT be cited as evidence about failure modes they do not probe.

Scenario: provenance is recorded

• WHEN a scenario is authored
• THEN its declaration SHALL name the observed failure source it derives from, or declare itself a channel stress test

Requirement: One fault per check

Checks SHALL be deterministic, and each check SHALL attribute at most one fault: either a tool fault (no taxonomy code) or exactly one agent-fault taxonomy code.

Scenario: one fault per check

• WHEN a check fails
• THEN it SHALL attribute either a tool fault (no taxonomy code) or exactly one agent-fault taxonomy code

Requirement: Empty-trajectory guard

Every tier-2 scenario SHALL include at least one empty-trajectory guard: a check that fails when the recorded trajectory contains zero executed steps.

Scenario: passive models cannot pass by doing nothing

• WHEN a live model returns an immediate done signal with zero executed steps
• THEN the scenario’s empty-trajectory guard SHALL fail
• AND the trial SHALL NOT be recorded as passed

Requirement: Live action protocol

Consumer harnesses running tier 2 SHALL require the agent to return one structured action per turn (exact command line plus done flag). Each turn the model SHALL receive a standardized minimum context: the scenario prompt and the fixture root path inside the sandbox. The fixture surface — the files the fixture contains and their contents — is observed only through executing actions. The context SHALL never include the check logic, the expected outcomes, or the distractor registry. On malformed output the harness SHALL re-ask once with the format error attached (the re-ask consumes a turn of the cap); persistent malformation SHALL end the trial as invalid_output with agent fault ERR_ACTION_FORMAT_VIOLATION. Every live trial SHALL be bounded by a turn cap, a wall-clock timeout, and a token-estimate budget — measured with the aix chars/4 heuristic over the trial’s full prompt and completion traffic, and labeled as heuristic. The applied cap, timeout, and budget values SHALL be recorded in the report. A trial stopped at a bound SHALL end failed, SHALL name the reached bound in the report, and the bound stop itself SHALL NOT be attributed as an agent or tool fault.

Scenario: one re-ask on malformed output

• WHEN the model’s response is not a parseable action
• THEN the harness SHALL re-ask once, including the format error

Scenario: re-asks consume the turn cap

• WHEN a malformed response triggers the one re-ask
• THEN that re-ask SHALL count as one turn toward the cap

Scenario: persistent malformation is an agent fault

• WHEN the retried response is also not a parseable action
• THEN the trial SHALL end invalid_output
• AND the report SHALL carry ERR_ACTION_FORMAT_VIOLATION as an agent fault, distinct from ERR_TOOL_EXECUTION_HALLUCINATION

Scenario: parseable actions are executed once

• WHEN the model returns a parseable action
• THEN the harness SHALL execute the command exactly once
• AND the resulting step SHALL be recorded with executed: true

Scenario: bounds stop runaway trials

• WHEN a trial exceeds its turn cap, wall-clock timeout, or token budget
• THEN the harness SHALL stop at the first bound reached
• AND the trial SHALL end failed with the reached bound named in the report and no agent- or tool-fault attribution for the stop itself

Scenario: check logic is never exposed

• WHEN the harness builds the model’s per-turn context
• THEN it SHALL include only the scenario prompt and the fixture root path inside the sandbox
• AND it SHALL NOT include check definitions, expected outcomes, or the distractor registry

Requirement: Free-tier entry and attribution

The model registry SHALL be ordered with OpenRouter :free model ids as the default first candidates of the registry — nothing about them is special-cased beyond that ordering. The ordering defines trial scheduling priority (which cells run first); it is not failover semantics — model fallback is a caller-level decision that SHALL be recorded when used. Reports SHALL record the raw model id verbatim (including the :free suffix), one row per repetition, indexed 0-based per scenario × model cell. On HTTP 429 the harness SHALL retry once after a delay of at least two seconds; the delay used SHALL be recorded in the report. A second 429 SHALL end the trial rate_limited — never attributed as a tool fault, never silently retried against a different model.

Scenario: raw attribution

• WHEN a trial runs against deepseek/deepseek-v4-flash:free
• THEN the report SHALL record that id verbatim

Scenario: repetitions are distinct rows

• WHEN a cell (scenario × model) is replayed more than once
• THEN each repetition SHALL be a separate report row

Scenario: 429 is retried once then recorded

• WHEN two consecutive OpenRouter calls return 429
• THEN the trial SHALL end rate_limited with no fault attribution

Scenario: no silent model substitution

• WHEN a harness falls back to a different model for a cell
• THEN the substitution SHALL be recorded
• AND each trial’s report SHALL name the model that actually produced its trajectory

Requirement: Sandbox confinement

Tier-2 execution SHALL run in a sandbox with isolated HOME, scrubbed environment, network egress denied unless the scenario opts in, and a per-command timeout that terminates runaway processes. The harness SHALL execute each action exactly once and record captured streams verbatim.

Scenario: isolated execution

• WHEN a live action is executed
• THEN it SHALL run with the sandbox’s isolated HOME and environment
• AND the resulting step’s executed field SHALL be true

Scenario: network denied by default

• WHEN a live action attempts network access and the scenario declaration does not request network access
• THEN the attempt SHALL fail within the sandbox

Scenario: runaway command terminated

• WHEN an action exceeds the per-command timeout
• THEN the process SHALL be terminated and the timeout recorded
• AND the terminated step SHALL be recorded as a distinct step outcome
• AND checks MAY treat post-termination filesystem state as indeterminate rather than attributing it to the agent

Requirement: Interoperable report contract

Scenario reports SHALL serialize to a versioned JSON shape (report_version, additive evolution only within a version) carrying at minimum: scenario name, tool, outcome status (passed | failed | rate_limited | invalid_output), per-check outcomes — one per declared check, passing and failing — each carrying an outcome status, an ERR_* code when and only when the failure is an agent fault, and a human-readable reason (for humans and tickets, not parsed by aggregators), and the bounds applied with their values. Optional fields SHALL be omitted, never null. Tier-2 rows SHALL additionally carry the raw model id and repetition index (0-based, per scenario × model cell); non-tier-2 rows MAY omit them. Three vocabularies are distinct and defined as follows: the status is the trial outcome (invalid_output carries the ERR_ACTION_FORMAT_VIOLATION agent fault; rate_limited carries no fault); a fault is the agent-vs-tool attribution of a failure; an ERR_* code is the shared taxonomy classification of an agent fault. The contract SHALL be published as a reference page with an example.

Scenario: tool faults carry no code

• WHEN a check fails with a tool fault
• THEN its per-check outcome SHALL carry a reason but no ERR_* code
• AND the code field SHALL be omitted, not null

Scenario: suite-wide aggregation

• WHEN two tools emit reports under the same report_version
• THEN a dashboard SHALL aggregate them without per-tool mapping
• AND rows without model attribution SHALL group as non-tier-2

Scenario: additive evolution

• WHEN a new optional field is added to the report
• THEN report_version SHALL remain unchanged
• AND older consumers SHALL ignore the unknown field

Scenario: breaking change bumps the version

• WHEN a field is removed or re-typed
• THEN report_version SHALL be incremented

Requirement: Fault routing

Tool faults SHALL be filed as tickets in the tool’s repository and SHALL NOT be routed as AIX-channel work. Agent faults SHALL be routed by grouping the ERR_* agent-fault code within the same output channel (the suggestions, envelope, managed-block, or artifact surface the check probes): when the same agent-fault code appears across three or more distinct model ids for that channel — or across all configured ids when fewer than three are configured — the finding SHALL be routed as AIX-channel work (work on the output channel itself, e.g. a hint channel that weak models reliably miss → suggestions-module work in genesis or the tool’s AIX surface), not dismissed as model noise. Agent faults below the routing threshold for their channel — one model id, or fewer than the threshold within the channel — SHALL be recorded without action. The scenario is a reporting dimension, not the routing key.

Scenario: tool fault becomes a ticket

• WHEN a check fails with a tool fault
• THEN a ticket SHALL be filed in the tool’s tracker referencing the report

Scenario: cross-model agent fault points at the channel

• WHEN the same agent-fault code appears across the routing threshold of model ids for the same output channel
• THEN the finding SHALL be routed to AIX-channel work, not to the model
• AND single-model occurrences SHALL be recorded without action

Scenario: sub-threshold occurrences are recorded without action

• WHEN the same agent-fault code appears in fewer model ids than the routing threshold for an output channel (e.g. two of four configured)
• THEN the finding SHALL be recorded without routing action

Scenario: small registries can still route

• WHEN a consumer configures fewer than three model ids
• AND the same agent-fault code appears across all of them for the same output channel
• THEN the finding SHALL be routed to AIX-channel work

feedback Specification

Purpose

TBD - created by archiving change add-aix-eval-loop. Update Purpose after archive.

Requirements

Requirement: Feedback context converts to scenarios

genesis SHALL convert captured feedback context (a ContextBundle, optionally with caller-supplied fixture files) into a replayable Scenario so that agent-reported failures become regression scenarios.

Scenario: bundle-only conversion yields a prompt-only scenario

• WHEN from_feedback_context is called with a ContextBundle and no fixture files
• THEN conversion SHALL succeed with a prompt-only Scenario derived from the recorded command and error expectation
• AND the Scenario SHALL expect an error envelope carrying the bundle’s suggestion footer as the hint

Scenario: caller-supplied fixtures become scenario fixtures

• WHEN from_feedback_context is called with a bundle and a fixture list of (path, content) pairs
• THEN the converted Scenario SHALL contain those fixtures via the existing fixture mechanism

Scenario: last-error path converts from scratch

• WHEN from_last_error(tool_name, fixtures) is called and a scratch ErrorRecord exists
• THEN the converted Scenario SHALL reflect the recorded argv, exit code, and footer

Scenario: missing scratch record is an error, not a panic

• WHEN from_last_error is called with no scratch record for the tool
• THEN conversion SHALL return a typed error

Scenario: converted scenario runs without a live agent

• WHEN the converted scenario is replayed against recorded AgentStep transcripts
• THEN no subprocess runner or LLM SHALL be required

Requirement: aix-gap feedback kind

The feedback flow SHALL accept aix-gap as a valid issue kind, routed through the same validation, redaction, and ContextBundle capture as the existing kinds. An aix-gap report SHALL be convertible to a regression Scenario via the existing from_feedback_context path.

Scenario: aix-gap kind is accepted

• WHEN feedback is filed with --kind aix-gap
• THEN the kind SHALL pass validation and the report SHALL flow through the standard feedback pipeline

Scenario: aix-gap near-miss gets a typo suggestion

• WHEN feedback is filed with kind aix_gap or aixgap
• THEN the suggestion engine SHALL propose aix-gap

Scenario: aix-gap capture converts to a scenario

• WHEN an aix-gap report captures a ContextBundle for a stale-artifact failure
• THEN from_feedback_context SHALL convert the bundle into a replayable Scenario exactly as for other kinds

genesis Specification

Purpose

TBD - created by archiving change add-genesis-foundation. Update Purpose after archive.

Requirements

Requirement: Shared crate for cross-cutting CLI infrastructure

The suite SHALL provide a single shared crate, genesis, that owns cross-cutting CLI, AIX, and self-healing infrastructure used by two or more charly-vibes tools.

Scenario: a tool needs the JSON envelope

• WHEN a tool’s command emits --json output
• THEN it SHALL wrap its payload in genesis::envelope::Envelope
• AND the emitted JSON SHALL share the top-level keys (ok, envelope_version, cli_version, envelope_kind, data, warnings, hints, meta) across every tool.

Scenario: a tool needs self-healing errors

• WHEN a tool encounters an unknown subcommand or a fixable error
• THEN it SHALL use genesis::suggestions::Suggestion to emit a “→ Run: …” footer
• AND SHALL NOT emit a bare error without a fix or context hint.

Scenario: tools need git hook primitives

• WHEN a tool needs to install, detect, or wire a git hook (ownership-marked install/uninstall, hook owner or framework detection, lefthook managed-block wiring)
• THEN it SHALL use genesis::git_hooks
• AND SHALL NOT maintain a private copy of these primitives

Requirement: Boundary rule

A capability SHALL be accepted into genesis only if two or more suite tools need it. Domain logic SHALL NOT be extracted into genesis.

Scenario: a proposed module serves one tool

• WHEN a contributor proposes a new module and only one tool consumes it
• THEN the proposal SHALL be rejected with a request to keep the module in the consuming tool.

Requirement: Distribution

genesis SHALL be distributed as a git dependency pinned by tag from github.com/charly-vibes/genesis. It SHALL NOT be published to crates.io until its interface is tagged stable.

Scenario: a tool adopts genesis

• WHEN a tool adds genesis to its Cargo.toml
• THEN the dependency line SHALL pin a tag (e.g. tag = "v0.1.0")
• AND SHALL NOT track a branch.

Requirement: Versioning

genesis SHALL follow SemVer. A breaking change SHALL bump the minor version, tag a new release, and open a coordinated change updating every dependent’s tag pin. Removed surfaces SHALL be deprecated for one release cycle before removal.

Scenario: a breaking change is made to a genesis module

• WHEN a contributor makes a breaking change to a genesis module
• THEN the minor version SHALL be bumped and a new tag SHALL be cut
• AND a coordinated change SHALL be opened updating every dependent tool’s tag pin
• AND the removed surface SHALL be deprecated for one release cycle before removal.

git-hooks Specification

Purpose

TBD - created by archiving change add-git-hooks. Update Purpose after archive.

Requirements

Requirement: Managed block format compatibility

The git_hooks module’s lefthook injection SHALL use the existing genesis::managed_block injector and block format, not bespoke string surgery.

Scenario: injected block is recognized by managed_block reader

• WHEN ensure_wired() injects a block into lefthook.yml
• AND the file is parsed with managed_block’s reader
• THEN the block SHALL be found with its tool tag and contents intact

Requirement: Tool-specific gates stay in tools

The git_hooks module SHALL contain only git-hook mechanics. No tool gate command (e.g. ah check, pretender check, testaruda select, just check-claims) SHALL appear in the module.

Scenario: module has no tool-specific defaults

• WHEN the git_hooks module source is inspected
• THEN it SHALL contain no hardcoded gate command strings from consuming tools
• AND callers SHALL pass their marker and block contents as parameters

Requirement: Repository root discovery

genesis SHALL provide repo_root() that locates the enclosing git repository by walking parent directories until a .git entry is found.

Scenario: root found from a subdirectory

• WHEN repo_root() is called from a directory nested inside a git repository
• THEN it SHALL return the path of the directory containing .git

Scenario: outside a repository fails

• WHEN repo_root() is called from a directory with no .git entry in any parent
• THEN it SHALL return an error naming the problem

Requirement: Marker-based hook installation

genesis SHALL provide install() that writes an executable hook script to the repository’s hook directory, identified by an ownership marker comment.

Scenario: install writes an executable hook

• WHEN install(root, HookName::PreCommit, marker, script) is called
• THEN <hooks>/pre-commit SHALL contain the marker and script
• AND the file SHALL have executable permissions (0755)

Scenario: hooks directory is created on demand

• WHEN install() is called and <hooks>/ does not exist
• THEN the directory SHALL be created before the hook is written

Scenario: install respects core.hooksPath at any scope

• WHEN the repository has a core.hooksPath config set at the local, global, or system scope
• AND install() is called
• THEN the hook SHALL be written into the resolved hooks directory instead of .git/hooks/

Scenario: install refuses a foreign hook

• WHEN install() is called on a hook file that exists
• AND the file does not contain the ownership marker
• THEN an error SHALL be returned
• AND the existing hook SHALL NOT be modified

Scenario: install is idempotent for own hooks

• WHEN install() is called on a hook file that already contains the ownership marker
• THEN the file SHALL be overwritten with the new contents
• AND no error SHALL be returned

Requirement: Marker-based hook uninstall

genesis SHALL provide uninstall() that removes a hook only when the ownership marker matches.

Scenario: uninstall removes an owned hook

• WHEN uninstall(root, hook_name, marker) is called
• AND the hook file exists and contains the marker
• THEN the hook file SHALL be removed

Scenario: uninstall is a no-op for missing hooks

• WHEN uninstall() is called and the hook file does not exist
• THEN it SHALL return success without error

Scenario: uninstall refuses a foreign hook

• WHEN uninstall() is called on a hook file without the marker
• THEN an error SHALL be returned
• AND the hook file SHALL NOT be removed

Requirement: Hook owner detection

genesis SHALL provide owner() that identifies which known tool owns a hook file, using the same sigil conventions wai’s way/hooks.rs uses.

Scenario: lefthook-owned hook is detected

• WHEN a pre-commit hook contains the string lefthook
• AND owner(root, HookName::PreCommit) is called
• THEN it SHALL return Owner::Lefthook

Scenario: more-specific sigils win over generic ones

• WHEN a hook contains both a generic and a more-specific sigil
• THEN owner() SHALL return the more-specific owner

Scenario: unknown or missing hook has no owner

• WHEN the hook file is missing or contains no known sigil
• THEN owner() SHALL return None

Requirement: Hook framework detection

genesis SHALL provide framework() that detects which hook-management framework a repository uses, superseding espectacular’s detect_hook_framework() with wai’s delegation handling. A repository is assumed to use at most one framework; framework() SHALL report that one framework generically (including husky, which has no root config and is detected via hook-file sigil).

Scenario: lefthook config present

• WHEN the repository root contains lefthook.yml or lefthook.yaml
• AND framework(root) is called
• THEN it SHALL return Framework::Lefthook

Scenario: prek config present

• WHEN no lefthook config exists and the root contains prek.toml
• THEN framework() SHALL return Framework::Prek

Scenario: husky-owned hook without root config

• WHEN no lefthook or prek config exists in the repository root
• AND the hook file contains the husky sigil
• THEN framework() SHALL return Framework::Husky

Scenario: framework signals coexist

• WHEN more than one framework signal exists (a root config and/or hook-file sigils) — expected to be rare since a repository is assumed to use one framework
• THEN framework() SHALL report deterministically by precedence: Lefthook, then Prek, then Husky

Scenario: no known framework

• WHEN no known framework config exists in the repository
• THEN framework() SHALL return Framework::None

Requirement: Lefthook managed-block wiring

genesis SHALL provide lefthook::ensure_wired() that injects a managed_block-formatted entry into the pre-commit: or pre-push: section of a lefthook config, idempotently.

Scenario: block injected under existing pre-commit

• WHEN lefthook.yml contains a pre-commit: section
• AND ensure_wired(root, Stage::PreCommit, block) is called
• THEN the managed block SHALL be inserted directly after pre-commit:
• AND the rest of the file SHALL be unchanged

Scenario: section created when missing

• WHEN lefthook.yml exists but has no pre-push: section
• AND ensure_wired(root, Stage::PrePush, block) is called
• THEN a pre-push: section SHALL be created containing the managed block

Scenario: wiring is idempotent

• WHEN ensure_wired() is called and the config already contains the block’s content
• THEN the file SHALL be unchanged and success returned

Scenario: missing config is reported, not created

• WHEN no lefthook.yml or lefthook.yaml exists in the repository root
• THEN ensure_wired() SHALL return an error telling the caller no supported hook framework config was found

Scenario: unanchorable config errors without modification

• WHEN the lefthook config cannot be anchored (e.g. the stage key is quoted, as in "pre-push":, or the structure is unrecognized)
• AND ensure_wired() is called
• THEN an error SHALL be returned describing the anchor failure
• AND the config file SHALL NOT be modified

Requirement: Wiring verification

genesis SHALL provide lefthook::is_wired() that reports whether a given gate command appears in a stage’s section of the lefthook config, for use by doctor checks (wai’s lefthook-wiring check, espectacular’s doctor).

Scenario: gate command present in stage

• WHEN lefthook.yml’s pre-commit: section contains run: ah check
• AND is_wired(root, Stage::PreCommit, "ah check") is called
• THEN it SHALL return true

Scenario: gate command absent from stage

• WHEN the stage’s section does not contain the command
• OR the stage section does not exist
• OR no lefthook config exists
• THEN is_wired() SHALL return false

Scenario: command in the wrong stage is not wired

• WHEN the command appears only under pre-push:
• AND is_wired(root, Stage::PreCommit, command) is called
• THEN it SHALL return false

Requirement: Effective hooks directory resolution

genesis SHALL provide effective_hooks_dir() that resolves core.hooksPath across all config scopes (local → global → system), matching git’s own precedence, and reports which scope the value came from so consumers can detect scope mismatches (e.g. a hook installed locally while git invokes hooks from a global path).

Scenario: global scope is honored

• WHEN no local core.hooksPath is set
• AND the user’s global git config sets core.hooksPath
• THEN effective_hooks_dir(root) SHALL resolve to the global value
• AND resolve_hooks_dir(root) SHALL return the same path

Scenario: local scope wins over global

• WHEN both local and global core.hooksPath are set
• THEN the resolved directory SHALL be the local value

Scenario: system scope is honored when higher scopes are unset

• WHEN neither local nor global core.hooksPath is set
• AND the system git config sets core.hooksPath
• THEN the resolved directory SHALL be the system value

Scenario: scope is reported to consumers

• WHEN effective_hooks_dir(root) is called
• THEN the result SHALL report the scope the value was found in: Local, Global, System, Default (unset anywhere), or Disabled (empty-string value)

Scenario: empty-string config is surfaced as disabled

• WHEN core.hooksPath is set to the empty string at the effective scope
• THEN effective_hooks_dir(root) SHALL report HooksDirScope::Disabled
• AND resolve_hooks_dir(root) SHALL keep its documented fallback (the default .git/hooks) so existing callers are unaffected

Scenario: unset config falls back to the default

• WHEN no scope sets core.hooksPath
• THEN effective_hooks_dir(root) SHALL report HooksDirScope::Default with the path .git/hooks under the repository root

Scenario: relative values resolve against the repository root

• WHEN the effective core.hooksPath value is a relative path
• THEN the resolved directory SHALL be the value joined onto the repository root