Comment + Reply exchange
Against the published exchange, not just the Comment
Post-publication critique in a top journal is an adversarial exchange: a Comment is published, the authors Reply, sometimes a Rejoinder follows — and a Comment is only as good as whether it survives that Reply. Correctness scored the engine against the Comment; this adds the Reply. For real Comment+Reply exchanges, Critical AI critiqued the originalpaper blind; we then ask whether it surfaced the published Comment’s flaws, and how the authors’ actual published Reply responded to each.
What it found
Reading only abstracts and blind to the published debate, the engine surfaced 67% (4/6) of the abstract-detectable flaws that drove a real Comment — and every surfaced concern was a point the original authors disputed in a published rejoinder. It surfaced the substantive flaw on the CEO-effect and reproducibility exchanges; it did not on GOTV.
Audit correction (G54, 2026-06-21): the first run reported a clean 6/6 = 100%. An independent re-decomposition refuted it: the per-exchange denominators were misrecorded, and on GOTV the engine’s two credited matches were topical, not substantive— Imai’s load-bearing flaw is randomization imbalance + matching (full-text-only), which the engine never surfaced; its concerns were about contact and compliance. Those two are reclassified not-surfaced, correcting the headline to 67%. A uniform “100%” was the under-firing tell.
Read honestly: “4/4 rebutted, 0conceded” is shaped by selecting exchanges that had a published rejoinder(authors who chose to defend), so “rebutted” is near-structural — it shows the concerns are at the live frontier, not that they are wrong. Adding exchanges where authors largely conceded (e.g. the Reinhart-Rogoff spreadsheet error) is the obvious next step. The steelman dimension (did the engine predict the Reply?) returned 3/3 missed but is exploratory and mis-specified: the engine predicted a defense against its own critique, while the actual Reply rebuts a different specific published Comment it never saw — so that number is not a clean engine limitation.
Exchange by exchange
- Strategic management / organizationrecall 100% (1/1)
The use of variance decomposition in the investigation of <scp>CEO</scp> effects: How large must the <scp>CEO</scp> effect be to rule out chance?
Comment: Quigley & Graffin (Strategic Management Journal, 2017), Comment · Reply: Fitza (Strategic Management Journal, 2017), Rejoinder
- ✓ surfacedauthors rebuttedThe chance 'baseline' in Fitza's simulation is mis-specified: he treats a mechanical property of the estimator (random data still producing nonzero R-squared) as evidence that real CEO performance variance is chance, conflating sampling/estimation noise with a substantive randomness-in-performance claim.
- Political science (experimental methods)recall 0% (0/2)
The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment
Comment: Imai (American Political Science Review, 2005), reanalysis · Reply: Gerber & Green (American Political Science Review, 2005), Rejoinder
- ✗ missedauthors rebuttedThe headline contrast across modes (canvassing large, mail slight, phone null) is partly an artifact of imbalance/method rather than a genuine ranking of mobilization technologies, weakening the inferential basis for the contrast.
- ✗ missedauthors rebuttedThe substantive thesis that turnout decline is attributable to the decline in face-to-face mobilization is overstated given that the empirical results (including the phone null) do not survive corrected analysis.
- Psychology (metascience)recall 100% (3/3)
Estimating the reproducibility of psychological science
Comment: Gilbert, King, Pettigrew & Wilson (Science, 2016), Comment · Reply: Anderson et al. / Open Science Collaboration (Science, 2016), Response
- ✓ surfacedauthors rebuttedThe replication studies were statistically underpowered: many replications had low power to detect the true effect even when the original effect was real, so a substantial fraction were expected to fail by chance alone. The reported low replication rate is therefore inflated as evidence of irreproducibility.
- ✓ surfacedauthors rebuttedThe replicated studies were not a representative or random sample of the literature, and protocols deviated from the originals (different populations, settings, stimuli). With non-random selection and infidelity to original methods, the project gives a biased, non-generalizable estimate of reproducibility, and protocol infidelity depresses the observed rate.
- ✓ surfacedauthors rebuttedThe subjective 'endorsement'/replication-success criterion and the other success metrics are misleading and biased toward declaring non-replication; once expected agreement and the uncertainty in original and replication estimates are properly accounted for, the data are consistent with near-ceiling reproducibility rather than the low rate claimed.
Redesigned steelman: anticipating author rebuttals
The steelman above was mis-specified (it scored the engine’s defence of its owncritique against a Reply to a different Comment). This is the fix: the engine is given the paper + the published Comment’s objections and predicts the authors’ actual Reply (concede / rebut / partial, and the substantive argument), scored against the real reply and audited for leakage. Card & Krueger (1994) is scored against its full-text reply; the other three reuse the stored comment + reply gold. It answers a clean question: can the engine anticipate author rebuttals to real published critiques?
- The use of variance decomposition in the investigation of <scp>CEO</scp> effects: How large must the <scp>CEO</scp> effect be to rule out chance?0/1 anticipated (0 strong · 0 partial)
- The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment0/2 anticipated (0 strong · 0 partial)
- Estimating the reproducibility of psychological science2/3 anticipated (0 strong · 2 partial)
- Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania2/2 anticipated (0 strong · 2 partial)
Read honestly: disposition accuracy (7/8) is inflated by selection— these exchanges have published rejoinders, so the actual disposition is almost always “rebut.” The honest signal is substance: the engine anticipated the authors’ actual load-bearing rebuttal on 4 of 8 objections (0 strongly, 4partially — a 2026-06-22 audit downgraded two over-credited “strong” reproducibility matches to “partial”), and the audit confirmed 0 genuine leakage — it predicted the direction of defence but missed idiosyncratic specifics (e.g. that the entire payroll-data employment decline traced to 23 restaurants from a single franchisee), which is the signature of clean reasoning, not recall of the famous debate. A clean improvement over the superseded 0/3.
Adversarial robustness: do our critiques survive the authors’ rebuttal?
Recall (above) asks whether the engine surfaces a flaw; the steelman asks whether it anticipatesthe reply. Neither asks the load-bearing question for a critique journal’s credibility: when the target authors mount their strongest good-faith rebuttal, does the engine’s core objection still stand? For 8critiques spanning severity and the engine’s stated confidence, single adversarial panels first scored each verdict — but a follow-up audit found that the survives / weakened boundary was judgment-unstable (refute-by-default panels over-kill; defender-primed corrections over-restore). The numbers below are the authoritative re-grounding: a 5-lens convergence panel(refute, defender, strict-neutral, authors’-advocate, reader-utility) votes per objection and we record the majority — so no single framing can flip a verdict. (The earlier single-panel runs are kept in the API as history.)
- Generative AI at Workweakened5/5 agree
- The Impact of AI on Developer Productivityweakened5/5 agree
- The Cybernetic Teammateweakened5/5 agree
- The rise of AI sovereigntyweakened5/5 agree
- Backfiring AI? AI Deployment in Workplaceweakened5/5 agree
- Making GenAI valuableweakened5/5 agree
- Being literate, behaving literate? A mixed-methods approach to adolescents’ algorithm literacy and behavioral strategies on social mediaweakened5/5 agree
- Charismatic machinesweakened5/5 agree
Read honestly: under the 5-lens convergence panel, 0/8of the engine’s critiques land an un-conceded substantive knockout — but 8/8 retain a genuinely valid residual objection and 0 are defeated, at 98% mean agreement with 0unstable verdicts (the defender lens itself voted “weakened” on every item). The engine’s critiques are reliably valid-but-bounded: directionally right, but overclaimed relative to what survives the authors’ rebuttal — which independently quantifies the abstract-access ceiling this journal already discloses. The deeper lesson: single-panel LLM-judge verdicts on the survives/weakened boundary are judgment-unstable(the first runs reported 1–2 “survivors” that this multi-lens vote dissolves), so the convergence majority — not any single adversarial framing — is the reliable scorer.
Self-improvement: can the engine remove its own overclaim?
If overclaim is the dominant failure (above), can the engine fix it? For each of the 7 overclaimed critiques, a revision pass rewrote the objection to its sharpest defensible + substantive core — blind to what survived, so it cannot copy the answer — then a fresh rebuttal + refute-by-default panel re-judged it under a substance guard(a revision that survives only by hedging into a truism, or by restating the paper’s own disclosed caveat, does not count). Because refute-by-default re-judges over-kill, an over-kill audit (defender + neutral arbiter) re-checked the substantive-but-weakened revisions.
- Generative AI at Workweakened (substantive residue)
- The Impact of AI on Developer Productivitycollapsed to non-substantive
- The Cybernetic Teammateweakened (substantive residue)
- The rise of AI sovereigntyweakened (substantive residue)
- Backfiring AI? AI Deployment in Workplaceweakened (substantive residue)convergence reversed over-kill
- Making GenAI valuablecollapsed to non-substantive
- Charismatic machinesweakened (substantive residue)convergence reversed over-kill
Read honestly: the calibration-revision pass does notlift the engine’s critiques to an un-conceded substantive knockout — 0/7 under the convergence panel. It does sharpen substance: the narrowed objections drop the provable overreach and 5/7 stay genuinely substantive, with 0 regressions — but the survives ceiling on abstract-based critique is structural (abstract access, not framing), so removing overclaim there needs full text, not better prompting. A single-panel over-kill audit briefly flipped 2/7to “survives” (raw floor was 0/7), but the 5-lens convergence panel reversed 2/2 of them: the survives/weakened boundary is judgment-unstable, and only a multi-lens majority vote is reliable.
Falsifiability: what would overturn each critique?
The arc above shows the engine’s objections are abstract-access-bounded — directionally right, but the decisive evidence often lives in the full paper. The integrity-first response is to make that bound contestable: for each of the 8 objections the engine declares the specific full-text evidence (a named table, estimator, design detail, or number) that would overturn it — and a 3-lens convergence panel (specificity · decisiveness · skeptic-honesty) votes whether that is a genuine falsifiable test. A critique that can name a concrete, decisive disconfirming test is Popper-falsifiable; one that cannot is unfalsifiable or self-servingly framed.
- Generative AI at Workfalsifiable2/3 agree
Overturned if: In the full paper, the objection is overturned by concrete artifacts of three kinds. (1) Headline (C1/C2): a pre-trend that is not flat — i.e., a statistically significant pre-period (anticipation) coefficient in the Sun-Abraham event study…
- The Impact of AI on Developer Productivitypartial2/3 agree
Overturned if: The objection would be overturned by concrete artifacts in the full paper, each strand checkable: (1) PRECISION — the paper's own regression table reporting the 21–89% interval would CONFIRM rather than refute the precision strand, so that …
- The Cybernetic Teammatefalsifiable3/3 agree
Overturned if: The objection is overturned if the full paper contains, concretely: (a) a task/role heterogeneity table showing the AI-equals-team and emotion effects hold across MULTIPLE distinct knowledge-work task types and worker functions (e.g., R&D v…
- The rise of AI sovereigntyfalsifiable3/3 agree
Overturned if: Three concrete artifacts in the full paper, each defeating one prong: (1) For the single-source / typology gap — a stated multi-case or multi-source corpus, i.e., a methods passage or coding table showing the "authoritarian imaginary" frame…
- Backfiring AI? AI Deployment in Workplacefalsifiable3/3 agree
Overturned if: The objection is overturned if the full paper contains: (1) an explicit proposition or labeled region-of-parameter-space result with stated inequality conditions and a remark (or a phase-diagram/figure partitioning the (heterogeneity, hard-…
- Making GenAI valuablefalsifiable3/3 agree
Overturned if: The objection is overturned if the full paper contains a stated benchmark corpus with explicit boundaries — e.g., "we examined N benchmarks released between year X and year Y, drawn from [source: arXiv/Papers-with-Code/Hugging Face/leaderbo…
- Being literate, behaving literate? A mixed-methods approach to adolescents’ algorithm literacy and behavioral strategies on social mediafalsifiable3/3 agree
Overturned if: The full paper contains all four of the following concrete artifacts: (1) a reported sample size N for the "representative survey" plus the named sampling frame/quota method; (2) a measurement table reporting the awareness and knowledge sca…
- Charismatic machinesfalsifiable3/3 agree
Overturned if: The objection is falsifiable, and each strand has a concrete overturning artifact. Strand 1 (the "not through actual understanding" / performance premise) is overturned if the body contains a stated definitional criterion or section disting…
Read honestly: 7/8 objections name a concrete, decisive full-text test that would overturn them (88% falsifiable, 92% agreement, 0 unstable) — convergence-judged and discriminating, not rubber-stamped: the panel downgraded one critique to partialbecause its precision strand is unfalsifiable by the engine’s own admission, and dissented on borderline cases. The one limit, stated plainly: this proves the engine can name a falsifiable test, not that the test resolves as predicted — confirming that needs the full text, the exact abstract-access ceiling this arc established. A critique that tells you what would prove it wrong is the contestable end-state the journal is built for.
What this proves — and what it doesn’t
It proves the engine, blind and abstract-only, reliably surfaces the substantive flaws that drove real top-journal Comments — and that those flaws are the very points serious enough to provoke a published authorial Reply. It does not resolve who is right in each dispute (these are unresolved debates — the authors rebutted), the cohort is small (3 exchanges), the Reply is read as its published summary (no-reproduce policy), and the steelman dimension needs a corrected design before its number means anything. Machine-readable at /critique/api/exchanges. See also the 3-exchange benchmark corpus these exchanges are drawn from.