S7 — Testing & Verification, Part 1: Unit Tests

Florian Oswald

13 October, 2026

Why this session exists

Since S2: “where is this wrong, and how do I check?”

• So far: read the code, run it, look at the output — doesn’t scale
• Scales even less once an agent writes most of the code and you review it
• A unit test writes down “here is what I checked” so it’s checked again automatically, every time the code changes — including every agent edit

This session runs remotely (over video call rather than in person) but is still hands-on: work through it at your own computer, running everything as you go. S8 builds directly on this with property-based tests and golden-output checks; S7 is the foundation.

Objectives

• Know what a unit test is, and what it is not worth testing.
• Write tests in testthat (R) and pytest (Python) using arrange-act-assert.
• Understand fixtures: fixed, known inputs with a known correct output.
• Have a first test suite covering real data-wrangling transformations.

A stand-in for the S6 pipeline

• S6 built a DuckDB pipeline on a real dataset — the habits below apply directly to it
• Self-contained here: Brexit referendum poll data (data/brexit.csv), via DuckDB
• check_shares() — flags polls whose remain + leave + undecided doesn’t sum to ~1 (a silent data problem an agent won’t flag unless asked)
• pollster_summary() — aggregates valid polls by pollster: number of polls, average remain/leave share

The full pipeline code is in scripts/s7/pipeline.R and scripts/s7/pipeline.py. If you already have your own S6 pipeline, treat this as a worked example and then repeat the exercise on your own transformations.

The stand-in pipeline, running

• R
• Python

    pollster remain leave undecided share_total valid
1     YouGov   0.52  0.48      0.00        1.00  TRUE
2    Populus   0.55  0.45      0.00        1.00  TRUE
3     YouGov   0.51  0.49      0.00        1.00  TRUE
4 Ipsos MORI   0.49  0.46      0.01        0.96 FALSE
5    Opinium   0.44  0.45      0.09        0.98 FALSE
6     ComRes   0.54  0.46      0.00        1.00  TRUE

# A tibble: 13 × 4
   pollster                         n_polls avg_remain avg_leave
   <chr>                              <int>      <dbl>     <dbl>
 1 BMG Research                           4      0.42      0.442
 2 ComRes                                 9      0.5       0.408
 3 ICM                                   24      0.434     0.431
 4 Ipsos MORI                             3      0.527     0.377
 5 ORB                                   14      0.503     0.469
 6 Opinium                                5      0.412     0.424
 7 Populus                                1      0.55      0.45 
 8 Populus/Number Cruncher Politics       2      0.48      0.41 
 9 Survation                              7      0.436     0.386
10 Survation/IG Group                     1      0.45      0.44 
11 TNS                                    8      0.38      0.396
12 YouGov                                 5      0.434     0.446
13 YouGov/The Times                       1      0.36      0.45 

     pollster  remain  leave  undecided  share_total  valid
0      YouGov    0.52   0.48       0.00         1.00   True
1     Populus    0.55   0.45       0.00         1.00   True
2      YouGov    0.51   0.49       0.00         1.00   True
3  Ipsos MORI    0.49   0.46       0.01         0.96  False
4     Opinium    0.44   0.45       0.09         0.98  False

                            pollster  n_polls  avg_remain  avg_leave
0                       BMG Research        4    0.420000   0.442500
1                             ComRes        9    0.500000   0.407778
2                                ICM       24    0.433750   0.431250
3                         Ipsos MORI        3    0.526667   0.376667
4                                ORB       14    0.502857   0.468571
5                            Opinium        5    0.412000   0.424000
6                            Populus        1    0.550000   0.450000
7   Populus/Number Cruncher Politics        2    0.480000   0.410000
8                          Survation        7    0.435714   0.385714
9                 Survation/IG Group        1    0.450000   0.440000
10                               TNS        8    0.380000   0.396250
11                            YouGov        5    0.434000   0.446000
12                  YouGov/The Times        1    0.360000   0.450000

Notice something already: some real polls in this dataset have shares that don’t quite sum to 1 (rounding in the source data), and check_shares() has to decide on a tolerance for what counts as “close enough.” That tolerance (tol = 0.01 by default) is exactly the kind of decision that deserves a test — it’s a number someone chose, and choices deserve to be pinned down.

What to test (and what not to bother testing)

Rough triage, for code you write and code an agent drafts:

Worth testing:

• Your own transformation logic — filtering rules, aggregation formulas, edge cases (empty input, all-invalid input, a single row)
• Anywhere a silent decision gets made (a tolerance, a default, a “drop rows where X” rule an agent added without telling you)
• Anything embarrassing or costly to get wrong quietly — a mean over the wrong denominator, a join that silently drops rows

Not worth testing:

• Well-tested library behavior you’re calling correctly (test your usage, not dplyr::filter()’s internals)
• Trivial getters/setters, one-line wrappers with no logic
• Anything that varies run-to-run for reasons outside your control (current time, live network) — unless designed around (mocking) — S8 territory

The line isn’t always obvious, and that’s fine — the question to ask is: “if this were subtly wrong, would anything catch it before a reader of my output does?” If the answer is no, write a test.

Fixtures

A fixture: a small, fixed, known input, factored out so multiple tests can reuse it and the input is obvious at a glance.

Both languages below build the same four-row table: two polls from pollster A (one valid, one that doesn’t sum to 1), one valid poll from pollster B, one poll from pollster C with an invalid (negative) sample size.

Fixtures — R

make_polls <- function() {
  data.frame(
    pollster   = c("A", "A", "B", "C"),
    samplesize = c(1000, 1000, 500, -1),
    remain     = c(0.52, 0.48, 0.50, 0.50),
    leave      = c(0.48, 0.50, 0.50, 0.50),
    undecided  = c(0.00, 0.00, 0.00, 0.00),
    stringsAsFactors = FALSE
  )
}

testthat has no special fixture decorator for this kind of case — a plain helper function called at the top of each test_that() block is idiomatic. (For fixtures that need setup/teardown, e.g. a temp file, see withr::local_tempfile() — out of scope today, but worth knowing the name.)

Fixtures — Python

@pytest.fixture
def polls() -> pd.DataFrame:
    return pd.DataFrame(
        {
            "pollster": ["A", "A", "B", "C"],
            "samplesize": [1000, 1000, 500, -1],
            "remain": [0.52, 0.48, 0.50, 0.50],
            "leave": [0.48, 0.50, 0.50, 0.50],
            "undecided": [0.00, 0.00, 0.00, 0.00],
        }
    )

pytest fixtures are functions decorated with @pytest.fixture; any test function that names polls as an argument receives a fresh copy automatically.

Arrange — Act — Assert

Every test below (and every test you should write) has the same three-part shape:

1. Arrange: build the input — the fixture above.
2. Act: call the function under test, exactly once, with that input.
3. Assert: check the output against a value you computed by hand — never a value you got by running the code and copy-pasting whatever it printed.

That third step verifies the code agrees with itself, not that it’s correct — a test whose expected value came from running the code is not a test.

Arrange — Act — Assert, in code

• R (testthat)
• Python (pytest)

test_that("check_shares flags rows that sum to 1 as valid", {
  # Arrange
  polls <- make_polls()

  # Act
  out <- check_shares(polls)

  # Assert
  expect_equal(out$share_total, c(1.00, 0.98, 1.00, 1.00))
  expect_equal(out$valid, c(TRUE, FALSE, TRUE, TRUE))
})

def test_check_shares_flags_rows_that_sum_to_one(polls):
    # Arrange: `polls` fixture
    # Act
    out = check_shares(polls)
    # Assert
    # Note: floats need pytest.approx, not `==` -- exact equality on a
    # computed sum passes only "by luck" of binary representation.
    assert out["share_total"].tolist() == pytest.approx([1.00, 0.98, 1.00, 1.00])
    assert list(out["valid"]) == [True, False, True, True]

Notice the 1.00, 0.98, 1.00, 1.00 values in both — those were computed by hand from the fixture above, not generated by running check_shares() and copying its output. Also notice the asymmetry between the two languages: testthat::expect_equal() compares numerics with a small default tolerance built in, so raw values compare safely; pytest’s == does not, so floating-point assertions in Python need pytest.approx() explicitly — forgetting it is an easy way to write a test that’s flaky for the wrong reasons.

The full test suites

• R: scripts/s7/test-pipeline.R (6 test_that() blocks, 8 expectations)
• Python: scripts/s7/test_pipeline.py (6 tests)

Both cover:

1. Shares summing to 1 → valid; not summing to 1 → invalid
2. A wider tolerance can let a borderline row through
3. Invalid rows and non-positive sample sizes dropped before aggregation
4. Aggregation numerically correct for a single surviving row
5. Edge case: nothing survives filtering → zero-row summary, not an error
6. Missing required columns raise a clear error, not a confusing failure deep inside dplyr/pandas

Case 5 is the one agents most often get wrong on a first draft: ask an agent to write pollster_summary() cold, and there’s a good chance it doesn’t handle the all-filtered-out case at all, or handles it inconsistently (an empty data frame with the wrong columns, or a KeyError/NULL crash). Writing that test before trusting the function is the whole point of this session.

Run them yourself:

Rscript -e ‘testthat::test_file(“scripts/s7/test-pipeline.R”)’

pytest scripts/s7/test_pipeline.py -v

Exercise

1. Open your own S6 pipeline (or use scripts/s7/pipeline.R / .py as-is)
2. Pick one untested transformation. Ask OpenCode to draft a test file for it, using arrange-act-assert and at least one fixture
3. Before running: does each expected value come from your own reasoning, or did the agent just paste back the code’s output? The latter is not a test — it always passes, even after a bug
4. Fix any assertion that fails check 3, run the suite, add one missed edge case (empty input is the classic one)
5. Note what your agent’s first draft got wrong or skipped — bring it to S8

Looking ahead

S8 goes beyond “does this input give this output” into:

• Invariants that must hold for any input (property-based testing)
• Golden-output comparisons for outputs too complex to hand-compute
• Assertions embedded directly in agent-facing scripts

Today’s unit tests are the floor, not the ceiling.