Fix XmR validator verdict math: volume gate + pooled CTR (RFD Option A) (#50)
* Onboard to ops; status.md with full backlog + constraint Constraint (★): the XmR validator's verdict math is unsound. Verified in
- Onboard to ops; status.md with full backlog + constraint
Constraint (★): the XmR validator's verdict math is unsound. Verified in code: min_sample_impressions is pre-registered on every experiment but read by zero lines; CTR is fed to XmR as a daily mean-of-ratios. The RFD flagging this (specs/RFD-gsc-verdict-math-sparse-ctr.md) has been Open since 2026-05-24, and its "pause the cron if you don't fix it" escape clause was never followed — validate-experiments.yml has committed verdicts daily for 51 days.
This has propagated into the biggest decision in the repo: GDA's 2026-07-14 S4 all-in fired on a baseline center of 0.054 clicks/day vs test weeks of 1.43/1.00/1.14 — roughly 8 clicks/week, with no volume floor on the promote gate. Re-check before building the page factory.
Also records a prod audit finding that corrects specs/gda-page-factory.md: its premise that the price-guide generator is "the gap to fill" is wrong — 430 price-guide, 336 compare and 85 best-* pages are already live.
No code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Validator: enforce the pre-registered volume gate (RFD Option A, step 1)
min_sample_impressions is pre-registered on every GSC experiment but was read by zero lines of code. Before emitting any non-pending verdict for an organic metric, fetch the cohort's impressions series and require both the baseline and treatment windows to independently clear min_sample_impressions; otherwise the verdict is pending, not a false signal off sparse data.
This alone kills the RFD's documented false positive: exp_gda_002 (3 clicks over 28 days on ~1840 impressions) now returns pending instead of "validated +18.85%".
Boundary tests added in eval/xmr-validator.test.mjs. validateTrial now takes an injectable fetchSeries so the verdict math is testable without the network.
Evidence: specs/RFD-gsc-verdict-math-sparse-ctr.md defect 1; grep confirms min_sample_impressions had no reader outside specs/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Validator: pooled CTR verdict for organic ctr metrics (RFD Option A, step 2)
Replace daily mean-of-ratios with pooled CTR = sum(clicks)/sum(impressions) per window for organic *.ctr.* metrics, compared as an ABSOLUTE percentage-point delta — the semantics predicted_magnitude_min is actually pre-registered with (exp_gda_001 promote_threshold: "CTR > baseline + 0.001"). The old relative (treatmentMean-baselineMean)/baselineMean was numerically unstable near a zero baseline; pooled absolute delta is not.
For these 14-28d windows the volume-gated pooled comparison IS the verdict (too few weekly points for XmR rules 2/3 to fire soundly, per the RFD). The non-CTR XmR daily path (clicks counts, ai_referrals, crank) is unchanged.
Tests: pooledCtr helper (pooling + zero-impression window), and validated / inconclusive / failed verdicts on cleared volume. Full suite 9/9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- CI: run validator boundary tests before applying verdicts
Gate the daily/dispatch validator run on eval/xmr-validator.test.mjs so broken verdict math can never commit to trials.jsonl. Directly addresses the RFD's escape clause ("if the math isn't right, the cron silently commits garbage") — now it can't: red tests fail the job before --apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>