Anvil
fangorn/anvil-cli public
ref:main
Preview Code 
# A Manifesto on Excellence in Software Testing



*Written for anvil-cli, but the principles are not local to it.*



---



## 0. The premise



A test suite is not a safety net. A safety net is passive — it catches you after

you fall, and its only measure of quality is whether it has holes. A test suite is

an **executable specification with an adversary attached**. It states what the

program must do, and it actively tries to prove the program doesn't do it.



Everything below follows from taking that sentence literally.



---



## I. Coverage is a map of ignorance, not a measure of quality



Line coverage tells you exactly one thing: which lines were *executed*. It says

nothing about whether anything was *checked*. A test that runs a function and

asserts nothing raises coverage. A test that asserts `result.is_ok()` on a

function whose entire job is computing the value inside the `Ok` raises coverage

by exactly as much as a test that checks the value.



So:



- **Uncovered lines are a reliable signal.** They are proof of absence. Chase them.

- **Covered lines are not a reliable signal.** They are absence of proof. Audit them.

- **Never write a test to move the number.** The number is a byproduct of testing

  well; the moment it becomes the goal, it stops measuring anything. A suite at

  85% where every assertion is load-bearing is worth more than a suite at 98%

  padded with `assert!(foo().is_ok())`.



The right use of a coverage report is as a **worklist**, read in this order:



1. Code that is uncovered *and* would be catastrophic if wrong (auth, secrets,

   path handling, money, data destruction).

2. Code that is uncovered *and* pure (cheap to test, no excuse).

3. Code that is covered but only *incidentally* — executed as a side effect of

   testing something else, with no assertion pointed at it. This is the most

   dangerous category because the report says it's green.



---



## II. Assert on the consequence, not the mechanism



The question a test must answer is *"what would a user notice if this broke?"* —

and then assert on precisely that.



```rust

// Weak: passes if the function returns Ok having done nothing at all.

assert!(config.save(Some(path)).is_ok());



// Strong: passes only if the observable consequence actually happened.

config.save(Some(path)).unwrap();

let round_tripped = RunnerConfig::load(Some(path)).unwrap();

assert_eq!(round_tripped.runner_token, "secret");

assert_eq!(mode_bits(path), 0o600, "runner token must not be world-readable");

Corollaries:

  • Prefer round-trips to spot-checks. save → load → compare proves both halves and the format between them. Testing save by reading the file with a hand-rolled parser tests your parser.
  • Assert the negative too. "The secret is in the output" is half a test. "…and the raw token is not" is the other half.
  • Test the error path's content, not just its existence. An error that says "error" and an error that says "config not found at /x — run \anvil runner configure` first"are bothErr`. Only one of them is the feature. If a diagnostic message is the product, assert on the message.

III. A test that cannot fail is worse than no test

No test is honest ignorance. A test that cannot fail is a false claim of safety — it occupies the slot where a real test would go, and it makes the report say "checked."

Before a test is finished, it must pass this gate:

Break the code on purpose. Does this test go red?

If you can't articulate a specific one-line mutation that turns the test red, you haven't written a test — you've written an exercise. Common non-tests:

  • assert!(x.is_ok()) where the payload is the point.
  • Asserting a Vec is non-empty when the bug would be wrong contents.
  • assert!(output.contains("error")) — matches the word "error" anywhere, including in an unrelated field or a success message about an error count.
  • Comparing a value to itself through the code under test (assert_eq!(f(x), f(x))).
  • Snapshot tests blessed without ever being read.

IV. Test at the seam where the meaning lives

Not everything deserves the same instrument.

| What it is | Where to test it | Why | |---|---|---| | Pure logic (parsers, path resolution, formatting, ID validation) | Unit test, in-module #[cfg(test)] | Fast, exhaustive, no fixtures. There is no excuse for an untested pure function. | | Filesystem effects (perms, atomicity, rotation, PID files) | Unit test against a TempDir | The effect is the behavior; assert on the real filesystem, not a mock of it. | | HTTP request shape (path, method, headers, body) | Mock server (wiremock) | Protocol contracts break silently and are caught by nothing else. | | End-to-end output contracts (--json, exit codes) | Spawn the real binary | The user runs the binary, not a function. Only this catches "a stray println! corrupted the JSON stream." | | Concurrency, signals, teardown | Deterministic harness with explicit synchronization | Never sleep and hope. |

The rule of thumb: test as low as you can while still testing something a user would care about. Push a test lower and it gets faster and more precise; push it too low and it starts asserting on implementation details that a legitimate refactor will break. The seam you want is the lowest one where the assertion is still phrased in the vocabulary of the requirement.

V. The three questions that generate the good tests

For every function worth testing, ask:

  1. What does it do? — the happy path. One test, and it's the least interesting one you'll write.
  2. What are its edges? — empty, zero, one, max, missing, duplicate, unicode, the boundary and both sides of it. Most bugs live here.
  3. What does it do when it's attacked? — the input the author didn't imagine. .. in a path. An absolute path where a relative one was assumed. A field present but null. A field that is a string where a number was expected. A response that is 200 with a body that says failure.

Question 3 is where the real defects are, and it's the one people skip because it requires imagining a hostile world instead of a cooperative one. The happy-path test documents the feature; the adversarial test is the one that finds the bug.

A useful trigger: whenever you see unwrap_or_default(), unwrap_or_else, a silent continue, a filter_map that drops errors, or Path::join on externally-supplied data — stop. Something is being swallowed there, and the test that proves what it swallows has almost certainly not been written.

VI. Tests are read more than they are run

The failure output of a test is a bug report written in advance, addressed to someone at 3am who did not write the code. Optimize for that reader.

  • The name is the specification. test_config() says nothing. save_writes_runner_token_with_owner_only_permissions() tells the reader what broke without opening the file.
  • Put the diagnosis in the assertion message. Include the actual value, the expected value, and enough context to act. A test asserting on subprocess output should dump stdout and stderr on failure — otherwise the first thing the reader must do is edit the test.
  • One behavior per test. A test with six assertions reports only the first failure; the other five stay hidden until you fix and rerun, five times.
  • No logic in tests. A loop or conditional in a test is a second program that can itself be wrong, and it is not tested. Table-driven cases are fine — the table is data; a branch inside the assertion is not.
  • Comment the why, never the what. // regression: #375 — a 200 with an empty body used to be treated as success is worth more than the test body.

VII. Determinism is non-negotiable

A flaky test is worse than a deleted test, because it teaches the team to ignore red. The response to a flake is to fix the determinism or delete the test — never to rerun until green.

The usual sources, and their fixes:

  • Time → inject a clock; never assert on wall-clock durations.
  • Sleeps used as synchronization → wait on the actual condition with a bounded poll and a timeout, or use a real notification primitive.
  • Shared global state — the real filesystem, $HOME, env vars, fixed ports, a shared DB → give each test its own TempDir, its own ephemeral port, its own scoped state. Rust runs tests in parallel by default; a test that mutates process-global state is a test that will randomly fail its neighbors.
  • Ordering assumptions on hash-map iteration, filesystem read_dir, or concurrent completion → sort before asserting, or assert on sets.
  • The network → mock it. A test that reaches the real internet is a test that fails on a plane.

Every test must pass alone, in a full run, in a random order, and a thousand times in a row.

VIII. A bug that escaped is a missing test, and the test comes first

When a defect is found in the wild, the sequence is fixed:

  1. Write the test that reproduces it. Watch it fail. This is the only moment you can be certain the test is actually pointed at the bug.
  2. Fix the code.
  3. Watch it pass.
  4. Leave the issue number in a comment on the test.

The value of step 1 is not ceremony. A test written after the fix is a test that has never been observed to fail, which by §III is not yet known to be a test.

IX. Do not test the language, the framework, or the mock

Effort spent asserting that serde deserializes, that clap parses --flag, or that the mock returns what you configured it to return is effort not spent on your logic. Test your code's contribution: the defaults you supplied, the validation you added, the shape you chose.

The special case of this that matters most: do not let a mock become the subject. If a test only proves "the mock was called," it will keep passing forever after the real server changes its contract. Assert on request shape — the path, the method, the body the server will actually receive — because that's the part that has to stay true of the real system.

X. When you touch code, you owe it a test

Not a rule about ceremony — a rule about ratchets. A codebase's test quality only ever moves in the direction of the marginal change. If every change leaves the tested surface slightly better than it found it, quality compounds. If changes are allowed to leave it flat, it decays, because the code around it keeps growing.

The concrete obligation, in priority order:

  1. If you fixed a bug, §VIII applies. No exceptions.
  2. If you added a branch, test both sides of it.
  3. If you touched a function with a weak existing test, strengthen the assertion while you're there. This is the cheapest quality improvement that exists — the context is already loaded in your head.
  4. If you found a function that is pure, public, and untested, and it took you thirty seconds to see what it does, it takes ninety seconds to test it. Do it.

The short version

Uncovered code is a worklist; covered code is an unaudited claim. Assert on consequences, not mechanisms. If you can't name the mutation that turns it red, it isn't a test. The happy path documents; the hostile input finds. Write the failure message for a stranger at 3am. Flaky is worse than absent. Reproduce before you fix.