A passing test
suite is a claim,
not a fact

A live matching algorithm had no oracle. Every constant — axis weights, specificity multipliers, boosts, gates, surfacing thresholds — had been chosen by human judgment, and the system was already in front of users. There was no correct score to assert against, which makes the obvious test not merely unavailable but unfalsifiable.

Every number in the matching formula is a versioned hypothesis, not a truth.

So I registered each constant with its value, the rationale it was chosen on, and the observable evidence that would falsify it — including five named falsification criteria, written before any data existed so they could not be tuned afterwards to flatter the result. The golden suite pins exact scores for fixture pairs, each pair isolating one slice of the configuration. Changing a value means a new version row with its evidence basis and a re-pinned golden test in the same commit; an unannounced change goes back to the dev team before anything is re-pinned. Never re-pin silently.

The registry also states in writing that these weights are an expert prior — not statistically optimal, not proven, not validated — and must not be described that way to investors, partners or users until the evidence exists. That sentence was the hardest part to write and is the part I would defend first.

A content platform I built by directing AI coding agents against written specifications, and the verification layer that decides whether they complied. Nothing is implemented until there is a design spec; nothing is built until that spec has an implementation plan. Only then do the agents touch the repository. The test suite is not there to prove the code runs — it is there to prove the agent built the thing the spec asked for.

Its evaluation harness grades generated content against recorded human accept and reject verdicts using embedding similarity plus deterministic checks. I deliberately did not use LLM-as-judge for subjective quality, and wrote down why rather than burying the decision in configuration. The standing rule here: an agent never writes the acceptance test for its own work, because the code and the test come out of the same misunderstanding and you get a green suite that proves nothing.

Red-team and benign control suites against LLM assistants, built on one principle: the attack has to defeat the grading mechanism, not only the model. Pattern bypass where checks are regex; intent-based attacks where grading is rubric-only. Three of the six silent errors above came out of this work — the under-detecting regex, the meaningless carried-over threshold, and the default grader nobody selected.

The output is not a score. It is a suite whose failures are legible: what was attempted, which mechanism let it through, and what the threshold is now calibrated against.

Sole quality owner through pre-launch, building the entire automation framework against an application written by other engineers, and authoring the conventions it is maintained by: locators by role, label and stable test ID with CSS and XPath banned; no fixed waits, only auto-retrying assertions; a single source of environment configuration; and a deployed-versus-local split so authenticated and captcha-dependent suites run where they can actually pass.

The eight suites slice the gate by risk: API, guest UI, authenticated UI, scoring engine, accessibility, asset-weight budgets, published legal documents, and a mobile device profile. Under it runs a known-defect protocol I designed — every open defect carries a test asserting the correct behaviour, flagged so the gate stays green until the fix lands and the test reports an unexpected pass. That is how a suite tracks 35 open defects without going red, and how a red build keeps meaning something.

Sole QA for a platform handling real money. I wrote the test strategy across 16 priority areas — browse, cart, Stripe checkout and refunds, booking workflows, authorization, commission accounting — built the automation for it, and did the exploratory work that automation cannot do.

Reconciliation was owned at the data layer, not the interface: every completed sale had to split into seller net plus platform commission at the documented rate, driven through a live payment end to end. Two further payment-integrity concerns went to code review rather than the defect tracker — a client-supplied amount not cross-checked against the order, and lenient handling of a bogus payment intent. Not everything worth raising is a bug report.