S11 — Reproducibility & Failure Modes

Florian Oswald

13 October, 2026

Why this session exists

• Remote session — no lecture, a debrief, a standard, and a personal document you write and keep
• By now (S4–S10) an agent has almost certainly produced something that looked right and wasn’t
• That’s not hypothetical — it’s the single most reliable thing about working with LLMs

That is the reason this course spends two full sessions (S7, S8) on testing before circling back to it here. This page connects two things usually taught separately: (1) what a professional reproducibility standard actually demands — the lens I use as Data Editor of the Journal of Political Economy when checking whether an accepted paper’s results can be regenerated from its code and data; (2) what you have personally already seen go wrong in this course, and what caught it.

Objectives

• Understand what a journal-grade replication package requires, and why each requirement exists.
• Catalogue every case in S4–S10 where an agent produced a plausible-but-wrong result, and identify what caught it.
• Write a personal checklist you will actually use the next time an agent writes code for you.

Part 1 — The Data Editor’s lens

When a paper is conditionally accepted at a journal like the JPE, authors submit a replication package: code, data, documentation — sufficient for someone uninvolved to regenerate every table and figure.

My job as Data Editor: check whether that’s actually true, not whether the authors say it’s true.

The standard is not exotic. It is a small number of concrete requirements, each of which maps onto a failure mode you have already lived through in this course.

The standard, mapped to failure modes

Requirement	What it checks	Failure mode it prevents
One-command run	No private “the trick” knowledge needed	Code that only runs on the author’s machine
Pinned versions	Results don’t silently drift	“Worked yesterday”
No hardcoded paths	Runs on any machine	/Users/you/Desktop/data.csv (S3’s requirement)
Data provenance	Every input traceable	Silent undocumented filtering upstream
README	Stranger finds code → output mapping	Nobody, incl. you in 6 months, can find how it was made
Seeds set	Exactly reproducible stochastics	Results that quietly drift each run
Auditable sample construction	Every row-drop visible & justified	Silent sample-selection changes the answer

Notice what this standard is really about: it does not ask “is the code good style” or “is the paper’s argument correct.” It asks can this specific, exact result be regenerated by someone who is not you, using only what you gave them. That is a narrower and more mechanical question than “is this correct” — and it is exactly the right question to ask about anything an agent wrote, because an agent will happily hand you code that runs, produces a plausible number, and is only correct by accident, once, on your machine, with your data. This is also why S3 made you restructure your repo (paths, README, config) before you had a “publication” to worry about: reproducibility discipline is infrastructure you build in from session one, not a checklist you apply retroactively to a finished project.

Part 2 — Debrief: the failure log

Across S4–S10 you used agents for: scripts (S2), a DuckDB pipeline (S6), unit tests (S7), invariant/property checks + seeded-bug hunt (S8), spatial joins/CRS (S10), cost/scale reasoning (S9).

Task: go back through your own work and fill in the table with at least four real entries — not hypothetical.

S4 gave you the underlying reason all of this happens: hallucination is a structural property of how an LLM generates text, not an occasional glitch. If you genuinely cannot find four entries, that is itself informative — it usually means you weren’t checking closely enough, and you should look again, harder, before concluding the agent was flawless four sessions running.

The failure log template

Session	What the agent produced	Why plausible	What caught it	Could S7/S8-style test catch it?
e.g. S6	Join silently dropped unmatched rows	Ran without error, row count “looked reasonable”	Manual row-count check before/after join	Yes — row-count invariant

For each row, be specific about the mechanism: was it a hallucinated number (S4: the model predicted a plausible-sounding token, not a looked-up fact)? A silent unit/CRS mismatch (S10: the map “looked fine” at first glance)? A test that passed but tested the wrong thing (S7/S8: arrange-act-assert on the wrong assertion)? A cost/scale estimate that assumed the wrong pricing tier (S9)? Naming the mechanism is what turns this from a diary into a diagnostic tool — it’s the difference between “the agent was wrong” and “the agent was wrong in this particular, recognizable way.” If you cross-reference against your S7/S8 test suite: does a test already exist that would have caught this case? If not, is it worth adding one? You do not have to add it now, but note whether it’s the kind of thing a test could catch (an invariant, a golden-output comparison, a row-count check) versus the kind of thing that fundamentally requires a human to look at output and judge whether it’s sensible (a spatial join that “looks like a map” but uses the wrong projection — S10’s example of a silent bug that survives casual inspection).

Part 3 — Your personal checklist

This is the graded outcome. Using Part 1 (the standard) + Part 2 (your own evidence), write a personal checklist you will genuinely use next time an agent writes code that matters.

• Not a copy of the Part 1 table — shorter, specific to your recurring failure patterns
• Phrased as things you will actually do, ~6–10 items

Some prompts to get you started (do not just answer these one-for-one — synthesize): Which of your Part 2 entries would you have caught immediately if you had just followed one habit? Put that habit at the top. Is there a category of check (row counts, unit tests, CRS/units sanity checks, seed-setting, path hardcoding) that shows up more than once in your table? That is a signal, not a coincidence. What’s the cheapest check you can run every time, versus the expensive check you’ll only run before something ships (a presentation, a submission, handing code to a collaborator)? Given S9 (cost/model economics): is there a point where “run every check every time” becomes wasteful, and how do you decide? If a reviewer (or a Data Editor) picked up your S12 capstone repo cold, what from Part 1’s table would already be satisfied by your habits, and what would still be missing?

Deliverable

A short written checklist (can live at the top of your S12 capstone README, or as a standalone note). Bring it to S12 — you will use it.

Checklist for this session

• Can explain why “can a stranger regenerate this exact number” is narrower and more useful than “is this correct”
• Filled in ≥4 real entries in the Part 2 failure log, each naming the underlying mechanism
• Cross-referenced ≥1 entry against what a S7/S8-style test could or couldn’t have caught
• Written a personal checklist of 6–10 concrete habits, specific to your own recurring failure patterns
• Saved the checklist somewhere you’ll actually see it again before S12