Is Critical AI right — not just faithful?

How the number is earned

1. 1. Blind critique. An isolated agent critiques each target from its title + abstract only, with no sight of the expert verdict — the isolation is the blindness guarantee.
2. 2. Ground truth. A separate agent decomposes the authoritative published critique into discrete flaws and labels each abstract-detectable vs full-text/external-only.
3. 3. Agreement. A strict + charitable panel scores, per detectable flaw, whether a blind concern substantively matches (refute-by-default; topical overlap does not count).
4. 4. Adversarial audit. An independent run re-judges every credited match, re-classifies detectability blind to the engine, and scans the blind critiques for leaked outside knowledge and fabricated rigour. A match counts only if it survives all of this.

The audit mattered. It cut the headline from a pre-audit 89% to a confirmed 63% — overturning 6 matches as merely topical and voiding 1 that leaned on knowledge a blind reader could not have. The confirmed figure is the headline precisely because it is the one that survived being attacked.

Cross-domain generality: does the integrity discipline hold in every domain, or just where it started?

Recall (above) is measured on a small benchmark cohort. A different worry is generality: the engine could be disciplined in the domains it started in (economics, management) and sloppy in the ones it reached late. This proof re-derives, for every social-science domain the journal has critiqued, whether its hard integrity floor holds — separating what the publication gate enforces from what is a discovered signal. Every number re-derives in-app from the live registries.

In every domain critiqued so far, the hard floor holds: 0 of 62 critiques distort their source and 0 fabricate a citation, in any domain. That zero-distortion / zero-fabrication result is the load-bearing part — unlike over-reach (a publication gate, so its 100% is true by construction, not discovered), faithfulness is not gated, so zero distortions is a real signal. Honest bound: this rests on thin per-domain counts (2–11 critiques), so it is a held-so-far property — one event from breaking in a thin domain — not yet a robust population guarantee. The 6 disclosed contested (not unfaithful) reviews are a quality gradient, detailed below.

Target by target

Each abstract-detectable flaw the authoritative critique rests on, and whether Critical AI’s blind critique surfaced it. Full-text/external-only flaws are listed muted — they are outside abstract reach and excluded from recall.

• AI-directComputational social science50% (2/4)

  Measuring the predictability of life outcomes with a scientific mass collaboration

  vs Garip (2020), PNAS invited commentary, 'What failure to predict life outcomes can teach us'

  • ✓ surfaced (audit-confirmed)The headline 'practical limits to the predictability of life outcomes' generalizes far beyond what was actually shown — namely that one particular pipeline (a fixed set of FFCWS predictors plus ML applied to six specific outcomes) failed to predict well; the abstract itself hedges with 'in some settings,' exposing that the limit is conditional on this design rather than a general property of life trajectories.

    engine, blind: “The headline framing 'How predictable are life trajectories?' is broad, but the evidence comes from a single cohort (Fragile Families) and exactly six outcomes, so the abstract's own hedge ('in some settings') sits in tension with the sweeping opening question.”

  • ✓ surfaced (audit-confirmed)The benchmark comparison is the load-bearing claim ('only slightly better than a simple benchmark model'), but how weak or strong that benchmark is, and how 'accuracy' was scored (e.g., R^2 / hold-out MSE on rare or noisy outcomes), determines whether the conclusion of poor predictability is warranted; a near-baseline result can reflect a low ceiling set by outcome noise rather than model inadequacy.

    engine, blind: “The central claim rests on a comparison to 'a simple benchmark model,' but the abstract never specifies what that benchmark is, so the reader cannot judge whether 'only slightly better' reflects genuine predictive limits or an unusually strong/weak baseline.”

  • ✗ missedThe constructive thesis Garip foregrounds — that the value lies in the common-task framework and out-of-sample testing rather than any model's accuracy — is itself signaled by the abstract, which already pivots to 'illustrate the value of mass collaborations,' so the commentary's positive reframing builds on what the abstract concedes.
  • ✗ topical only (audit-overturned)Concluding that 160 teams converging on near-benchmark accuracy demonstrates a genuine predictability ceiling assumes the teams collectively exhausted the useful modeling and feature-engineering space; if all teams worked from the same provided feature set under the same task constraints, their convergence reflects a shared input ceiling, not proof that the outcomes are intrinsically unpredictable.

  Outside abstract reach (2): Failure to predict is attributed to limits of ML/data, but the result is equally consistent with limits of the specific measured predictors and the chosen outcomes (data-quality / feature-set ceiling, missingness, attrition in the cohort) rather than an intrinsic ceiling on predictability. · The finding that prediction error is 'strongly associated with the family being predicted and weakly with the technique' is presented as evidence about predictability limits, but it could instead reflect heterogeneous/idiosyncratic measurement error or a small number of hard-to-predict cases driving variance — an interpretation that requires examining the error distribution across families.

• Economics100% (2/2)

  The Colonial Origins of Comparative Development: An Empirical Investigation

  vs Albouy (2012), American Economic Review, Comment

  • ✓ surfaced (audit-confirmed)The identification rests entirely on a single instrument (settler mortality) whose validity and measurement are assumed rather than independently verified, creating a single point of failure for all downstream claims.

    engine, blind: “The exclusion restriction for the instrument is asserted but not defended: European settler mortality is assumed to affect modern income only through institutions, yet historical mortality could plausibly proxy for present-day disease environment, geography, or human-capital channels that bear directly on income.”

  • ✓ surfaced (audit-confirmed)The abstract makes strong, far-reaching causal claims (large effects of institutions; geography/Africa dummies become insignificant once institutions are controlled) from a cross-country observational design, which is vulnerable to fragile identification.

    engine, blind: “The conclusion that Africa and equatorial countries lack lower incomes 'once institutions are controlled for' risks overclaiming, since it treats geography purely as a confounder to be partialled out while geography may operate partly through the same channel as the instrument and the mediator.”

  Outside abstract reach (4): 36 of the 64 settler-mortality observations are not actual mortality rates measured in the country itself, but rates imputed/borrowed from other countries, making the key instrument's data partly fabricated by assignment. · The mortality series combines incomparable populations (laborers, bishops, soldiers on campaign) whose death rates are not on the same scale, and the combinations were made in directions favorable to the institutions hypothesis. · Once the data problems are corrected, the first-stage mortality-expropriation relationship is no longer robust, undermining instrument relevance. · After fixing the data, the IV estimates lose robustness, often producing effectively infinite confidence intervals — i.e. the headline 'large effects of institutions' is not statistically identified.

• Psychology (metascience)80% (4/5)

  Estimating the reproducibility of psychological science

  vs Gilbert, King, Pettigrew & Wilson (2016), Science, 'Comment on Estimating the reproducibility of psychological science'

  • ✓ surfaced (audit-confirmed)Many replications were themselves underpowered, so a non-significant replication is expected by sampling error even when the original effect is real; the headline 'low replication rate' conflates true failure to replicate with the replications' own statistical limitations.

    engine, blind: “Using statistical significance of the replication as a reproducibility criterion conflates effect existence with the dichotomous p<.05 outcome, and the abstract reports significance counts (97% vs 36%) without acknowledging that low replication power or smaller true effects, rather than non-reproducibility, could drive the gap.”

  • ✓ surfaced (audit-confirmed)The 100 studies were not a representative random sample of the psychology literature; studies were selected under feasibility and assignment constraints, so the aggregate 'reproducibility' estimate cannot be generalized to the field as the abstract implies.

    engine, blind: “The abstract does not state how the 100 studies were selected from the candidate pool in the three journals, so whether the estimate is representative or subject to selection effects cannot be assessed from the abstract alone.”

  • ✗ topical only (audit-overturned)The 'subjective replication' endorsement criterion (and the family of dichotomous success metrics) is misleading: subjective ratings and significance-counting are noisy, low-reliability gauges that understate agreement between original and replication; better metrics (e.g., whether the original effect falls in the replication CI) suggest much higher reproducibility.
  • ✓ surfaced (audit-confirmed)Comparing the magnitude of replication effects to original effects to infer 'a substantial decline' ignores regression to the mean and selection/publication bias inflating originals, so the halving of effect sizes does not by itself index irreproducibility.

    engine, blind: “The claim that replication effects represent a 'substantial decline' interprets a halving of effect magnitude as decline, but regression to the mean and publication-driven inflation of originals are plausible alternative explanations the abstract does not address.”

  • ✓ surfaced (audit-confirmed)Replications used 'original materials when available' — i.e., not always — and protocol/infidelity differences (population, setting, procedure) plausibly account for non-replications rather than the originals being false positives.

    engine, blind: “'Original materials when available' implies some replications used non-original materials, a heterogeneity that could systematically depress observed replication and is not quantified in the abstract.”

  Outside abstract reach (2): Gilbert et al.'s reanalysis using a high-fidelity benchmark (independent direct-replication confidence intervals) shows the observed replication rate is statistically consistent with ~100% true reproducibility once expected sampling error and infidelity are accounted for. · Endorsement/CI-overlap metrics were not adjusted for the error rate of the replication estimates themselves, so the count of 'replications whose CI excludes the original effect' overstates failures because some exclusions arise purely from replication noise.

• Social psychology75% (3/4)

  Power Posing

  vs Ranehill et al. (2015), Psychological Science (large replication)

  • ✓ surfaced (audit-confirmed)The original study was underpowered: its sample was too small to reliably estimate the claimed hormonal and behavioral effects, making them vulnerable to false positives.

    engine, blind: “The abstract reports a single study yet draws broad causal physiological and behavioral conclusions, with no sample size disclosed, leaving statistical power and reliability unverifiable.”

  • ✗ missedOnly the self-reported 'feelings of power' effect replicates; the subjective self-report outcome is the weakest, most demand-susceptible measure and cannot license the physiological/behavioral conclusions.
  • ✓ surfaced (audit-confirmed)The abstract over-generalizes to 'real-world, actionable implications' from a single laboratory study, asserting instant, universal effects not warranted by the evidence base.

    engine, blind: “The closing sentence extrapolates from a lab manipulation to sweeping real-world prescriptions, treating a 1-minute pose as something that makes a person 'instantly become more powerful' — a generalization the within-study evidence described cannot support.”

  • ✓ surfaced (audit-confirmed)The strong causal claim ('can these postures actually cause power?... posing... caused... changes') rests on a single original study without independent confirmation, a contested-phenomenon risk.

    engine, blind: “No information is given on replication, data, or code availability, so the strong actionable claims rest on a single unreplicated study as described in the abstract.”

  Outside abstract reach (2): The original study's neuroendocrine and behavioral effects (testosterone increase, cortisol decrease, risk tolerance) fail to replicate in a larger, better-powered sample (N=200), so the physiological and behavioral causal claims are not robust. · Cortisol and testosterone responses are noisy, assay- and timing-sensitive measures whose claimed clean directional effects may reflect uncontrolled measurement/analytic variability rather than a true posture effect.

• Economics67% (2/3)

  The Impact of Legalized Abortion on Crime

  vs Foote & Goetz (2008), Quarterly Journal of Economics, Comment

  • ✗ topical only (audit-overturned)The cross-state test (high-abortion states in the 1970s-80s showing greater 1990s crime declines) is not robust to allowing differential state-specific time trends; once such trends are permitted, the abortion-crime relationship is not reliably identified.
  • ✓ surfaced (audit-confirmed)The attribution of 'as much as 50 percent' of the crime drop to legalized abortion is an overstated magnitude claim that collapses once the coding error is fixed and per-capita / trend-robust specifications are used.

    engine, blind: “The 50 percent figure is presented as an upper bound ('as much as') yet functions as the abstract's takeaway, an overclaiming pattern where the most striking number is foregrounded without the accompanying lower bound, confidence interval, or uncertainty.”

  • ✓ surfaced (audit-confirmed)The overall identification leans on observational timing coincidences (crime falling 'roughly eighteen years after' legalization; five early-legalizing states declining earlier) that are confounded by many other contemporaneous determinants of 1990s crime trends.

    engine, blind: “The five-state early-legalization comparison rests on a very small treated group whose crime trends could diverge for many reasons unrelated to abortion; the abstract reports the directional difference without any control for the states' other characteristics.”

  Outside abstract reach (2): The headline within-state cohort result ('only arrests of those born after abortion legalization fall relative to low abortion states') rests on a regression that was intended to include state-year interaction terms but, due to a coding mistake, omitted them; correcting this guts the effect. · The within-state results depend on using arrest counts rather than a per-capita (arrests-divided-by-population) specification; moving to per-capita crime rates sharply weakens the estimated abortion effect because cohort size itself drives arrest counts.

• Social psychology67% (2/3)

  Inhibiting and facilitating conditions of the human smile: A nonobtrusive test of the facial feedback hypothesis.

  vs Wagenmakers et al. (2016), Registered Replication Report, Perspectives on Psychological Science

  • ✗ topical only (audit-overturned)The original two-study finding rests on a small single-source sample, making the reported effect vulnerable to sampling variability and false-positive inflation typical of pre-power-revolution social psychology.
  • ✓ surfaced (audit-confirmed)The pen-in-mouth manipulation is a subtle, demand-prone, and fragile procedure whose effect on self-reported humor may not survive procedural variation (e.g., presence/absence of cameras, instructions, stimuli) across labs.

    engine, blind: “The sole outcome appears to be a subjective self-report ('reported more intense humor responses'), so the affective effect is measured by demand-sensitive ratings rather than any independent behavioral or physiological index of emotion.”

  • ✓ surfaced (audit-confirmed)The abstract over-interprets the data by drawing fine-grained theoretical conclusions (facial feedback affects the affective but not the cognitive component; both inhibitory and facilitatory mechanisms contribute) that depend on an effect that itself does not reliably exist.

    engine, blind: “The dissociation claim that 'facial feedback operates on the affective but not on the cognitive component' is a claim of a null effect on cognition, which the abstract presents as a positive finding without indicating it was powered to detect such an absence.”

  Outside abstract reach (2): The headline effect (induced smiling increases rated funniness) does not replicate: a pooled estimate across 17 high-powered direct replications is 0.03 rating units (95% CI -0.11 to 0.16), versus the original 0.82, indicating the original effect is essentially absent or massively overstated. · The reported original effect size (0.82) is implausibly large relative to what the construct can plausibly produce, consistent with publication-era effect-size inflation rather than a stable phenomenon.

• Economics50% (1/2)

  Growth in a Time of Debt

  vs Herndon, Ash & Pollin (2014), Cambridge Journal of Economics

  • ✓ surfaced (audit-confirmed)The headline causal/policy reading -- that crossing a 90% debt/GDP threshold sharply depresses growth -- rests on a discontinuity (median growth falls 1%, average falls much more above 90%) that the abstract itself presents as a sharp threshold despite an associational, observational design.

    engine, blind: “The abstract describes only a correlational threshold relationship yet the framing 'Growth in a Time of Debt' and the structure of the claims invite a causal reading (debt depresses growth) that the stated descriptive methods cannot support.”

  • ✗ topical only (audit-overturned)Pooling forty-four heterogeneous countries across ~200 years with wildly unequal numbers of observations per country invites sensitivity to how unequal-length panels are averaged.

  Outside abstract reach (4): A spreadsheet (Excel) coding error caused several countries (e.g., Australia, Austria, Belgium, Canada, Denmark) to be silently omitted from the high-debt average, mechanically lowering the reported growth rate for the 90%+ debt category. · Selective exclusion of available country-year data (e.g., New Zealand 1946-49, early postwar Canada/Australia) dropped low-growth-erasing or high-growth high-debt episodes, biasing the high-debt average downward. · An unconventional weighting scheme (weighting each country equally rather than by the number of country-year observations, so a single year for one country counts as much as decades for another) distorted the high-debt growth estimate. · Correcting all three problems raises average real growth for the high-debt (>90%) group to +2.2% from the published -0.1%, so the headline 90% threshold / cliff in growth does not exist.

• Psychology / neuroscience25% (1/4)

  Methylphenidate Blocks Effort-Induced Depletion of Regulatory Control in Healthy Volunteers

  vs Hagger et al. (2016), Registered Replication Report, Perspectives on Psychological Science

  • ✗ voided (used leaked knowledge)The paper treats regulatory depletion as a settled, well-replicated phenomenon ('more than 100 studies over the last decade') and builds its entire claim on that premise, rather than treating it as a contested effect requiring independent verification.
  • ✗ missedIf the baseline depletion effect is itself absent/near-zero, there is no genuine depletion for methylphenidate to 'block,' so the headline causal claim that the drug reverses a real regulatory-control deficit is unsupported.
  • ✗ topical only (audit-overturned)The study is a single-lab demonstration whose central effect (a drug-by-depletion interaction) is reported with strong, definitive language ('fully blocks', 'demonstrated specificity') without independent replication.
  • ✓ surfaced (audit-confirmed)The secondary mechanistic finding (spectral analysis localizing methylphenidate effects to the slow-4 band, linked to mind-wandering brain networks) is an exploratory, post-hoc neurophysiological interpretation layered on top of the unverified depletion effect.

    engine, blind: “The leap from a behavioral RT spectral signature to claims about resting-state brain networks and mind wandering is a reverse-inference: an association of a band with networks elsewhere does not establish that those networks drove this effect, and no neuroimaging is described.”

  Outside abstract reach (1): The foundational ego-depletion (sequential-task regulatory depletion) effect that the paper presumes is not robust: a 23-lab preregistered replication (~2000 participants) found a meta-analytic effect indistinguishable from zero (d ~ 0.04).

Breaking the ceiling: full text vs abstract

The headline above is bound by abstract-only access. This A/B runs the same blind engineon a target’s full text and on its abstract, scoring each arm against the same full-text-decomposed expert-flaw gold (now including the full-text-only flaws) and adversarially auditing each arm for leakage. A full-text arm needs the target paper’s full text; of the articles downloaded for benchmarking, Card & Krueger (1994) is the one paired target — the others are abstract-arm only, scored against a full-text gold to show the ceiling.

• Estimating Marginal Returns to Medical Care: Evidence from At-Risk Newborns (Almond, Doyle, Kowalski & Williams 2010)

  vs Barreca, Guldi, Lindo & Waddell (2011), QJE Comment (10.1093/qje/qjr042)

  Abstract arm only (target full text not in hand): confirmed 1/7 (14%). With all 7 expert flaws labelled, this documents the abstract-only ceiling: the engine reaches the generic version of a flaw but not the critic-supplied microdata specifics. 0 leakage voids.

• Estimates of the Economic Return to Schooling from a New Sample of Twins (Ashenfelter & Krueger 1994)

  vs Rouse (1999), Economics of Education Review (10.1016/S0272-7757(98)00038-7)

  Abstract arm only (target full text not in hand): confirmed 0/6 (0%). With all 6 expert flaws labelled, this documents the abstract-only ceiling: the engine reaches the generic version of a flaw but not the critic-supplied microdata specifics. 0 leakage voids.

Integrity check across every arm: 0matches voided for leakage of outside knowledge. Famous papers are a leakage trap — the audit confirmed each credited match traces to the paper’s own text, not the published debate. A 2026-06-22 cadence audit re-attacked these numbers and corrected two self-serving detectability labels (ADKW’s heaping flaw is abstract-reachable; the Ashenfelter–Krueger flaws need the critic’s own re-estimation, so it is honestly “0 of 0 abstract-reachable”); it split on the Card–Krueger ceiling-break, which is retained because the match rests on the paper’s own reported levels-vs-proportional numbers (a sensitivity the engine observed, not the published critique’s result-overturning thesis). Aggregator re-derives every figure from the records; JSON ↗ (see fulltext_ab).

Full-text re-critique: do the objections hold against the whole paper?

After ingesting 24 of the 26 target full texts, two passes re-examined the abstract-based critiques against the verbatim text. Part A is the sharpest test: for the 6 robustness-cohort critiques, the engine had already declared the specific full-text artifact that would overturn each objection (its falsifiability condition) — so we check whether that artifact is actually in the paper. Part B re-critiques all 24 afresh, classifying whether the full text confirms, weakens, or overturns the abstract objection, or reveals a new flaw. Every quoted span is deterministically verified to be an exact substring of the verbatim text.

• The Cybernetic Teammatestands vs full text✓ grounded
• Backfiring AI? AI Deployment in Workplacepreprintweakened✓ grounded
• Charismatic machinesobjection overturned✓ grounded
• The rise of AI sovereigntyweakened✓ grounded
• Making GenAI valuablestands vs full text✓ grounded
• Being literate, behaving literate? A mixed-methods approach to adolescents’ algorithm literacy and behavioral strategies on social mediastands vs full text✓ grounded

Read honestly: full text enriches the critiques and refutes almost none. Part B overturned 0/24 abstract objections — it added a new methods/identification flaw in 14 and confirmed 7 — and the sharper Part A falsification test overturned only 1/6, leaving 3standing even against the full text (the refuting artifact simply isn’t in the paper). So the abstract-based objections were directionally sound but incomplete — validated by full text, not refuted by it. Integrity: every span deterministically checked against the verbatim text, catching 1 fabricated span (flagged, excluded). This records re-critique resultsonly — it does not change any critique’s access basis or severity, which still requires a span-verified full-text critique through the build pipeline. JSON ↗ (fulltext_recritique).

Staged full-text upgrades — held for the editorial gate

The re-critique above is turned into action here: for the 13 papers where full text surfaced a new flaw, plus a revision of the one critique the full text overturned, the upgrade content is staged with a severity recalibrated under the full-text access cap. Crucially these are held, not live— promotion to a published critique (which would lift its access basis and severity cap) requires the journal’s named-editor publish gate, the same human boundary as the produce/publish pipeline. Severity was set refute-by-default against inflation and a second verifier re-checked each.

• AI meets politicsseverity moderatestaged
• Beyond disruption and invisibilityseverity moderatestaged
• Crafting computer vision through human eyesseverity moderatestaged
• Into the black boxseverity moderatestaged
• Made With AIseverity moderatestaged
• Making GenAI valuableseverity moderatestaged
• Resilience and disempowerment in algorithmic systemsseverity moderatestaged
• The politics of artificial intelligence alignmentseverity lowstaged
• Working the algorithmseverity moderatestaged
• Backfiring AI? AI Deployment in Workplaceseverity moderatestaged
• More Versus Betterseverity moderatestaged
• Refusal as silenceseverity moderatestaged
• The Cybernetic Teammateseverity moderatestaged
• Charismatic machinesrevision · overturnedseverity lowstaged

Read honestly: full text upgrades the critiques in depth and grounding, not severity — 0/14 of the recalibrated upgrades warrant raising severity above the abstract moderate cap (12 stay moderate, 2 drop to low, including the overturned charismatic prong). The refute-by-default verifier corrected several over-rated severities down. All 14 are span-grounded and staged; nothing changes in the live journal without the named-editor gate. JSON ↗ (fulltext_upgrades_staged).

Coverage scorecard: every aspect of the critique vs the full paper

The most comprehensive test of the critiques. With 24 papers in hand at full text, each Critical AI critique — written from the abstract — is scored against its full published paper across 8 critique-quality dimensions: did it cover the issue the full text warrants? A span-grounded scorer rated each, then a refute-by-default anti-self-flattery verifierre-adjudicated every “addressed” call (the journal grading its own coverage over-credits by construction). Coverage rate = (addressed + ½·partial) / issues found.

• Reproducibility / transparency25%2/2/8 a/p/mblind spot
• Sample / data quality27%2/4/9 a/p/mblind spot
• Measurement / construct validity30%3/7/12 a/p/mblind spot
• Authors' disclosed limitations31%3/5/10 a/p/mblind spot
• Generalizability / scope50%7/6/7 a/p/m
• Statistical inference58%6/2/4 a/p/m
• Identification / causal inference68%8/3/3 a/p/m
• Overclaim / scope of conclusions89%19/3/1 a/p/m

Read honestly: the abstract-based critiques cover only 49% of what the full papers warrant. They are strong exactly where the abstract is informative — overclaim and generalizability/scope — and have systematic blind spotson the full-text-only dimensions: authors’ own disclosed limitations (the worst, rarely visible in an abstract), sample/data, reproducibility, statistical inference, and measurement. This is the abstract-access ceiling made precise, dimension by dimension — and it names exactly where full-text critiques would add the most. The anti-flattery verifier corrected 23 over-credited cells; 97% of evidence spans are verbatim. Benchmark result only — no critique changed. JSON ↗ (critique_coverage).

Audit caveat (G78). A 3-lens convergence re-check of 16 sampled cells found the single-panel scorer + refute-by-default verifier under-credited (6 cells rated higher by the multi-lens majority vs 1 lower; 9/16 exact, 14/16 within one step). So these coverage rates are a conservative lower bound — true coverage is modestly higher and the blind-spot magnitudes are somewhat overstated; the addressed/partial boundary is panel-sensitive. The blind-spot pattern holds; the absolute numbers read as a floor. Deterministic span + provenance probes passed clean.

Structure & grounding audit: is the corpus parsimonious — and well-read?

Two questions about how the 29 published critiques are written. First, can they be consolidated — do the free-form titled sections merely duplicate the structured spine (claims, strongest critique, fair defence, final judgment)? Second, how far do they incorporate existing literature, academic and grey? A parsimony analyst proposed cuts; a refute-by-default skeptic (default keep) plus an independent manual re-check then adjudicated every proposed cut — and the literature figures are counted deterministically from the corpus.

Parsimony — premise refuted. The consolidation hypothesis did not survive verification: of 35 sections flagged as redundant, 33 were overturned and 0 confirmed. The structured spine is evaluative; the free-form sections carry the factual setup— sample sizes, populations, the firm, the design, the mechanism, the paper’s own quoted claims and genre — that the spine presupposes but never states. They are complementary, not duplicative. The corpus is already near-parsimonious; bulk consolidation would orphan load-bearing content. (Two sections went unchecked when a skeptic hit a rate limit; both were re-checked by hand and kept.)

Literature — the real gap. Only 7 of 29 critiques cite any prior work; 20 references in total, all academic, zero grey literature (reports, working papers, standards, regulator and lab publications), mean 0.69 per critique, and 0 of 29 reach “grounded.” Every one of the 29 has relevant grey literature it does not cite. This — not section redundancy — is where the structure is genuinely thin, and it names the next real investment: grounding the critiques in the scholarly and grey conversation, not restructuring them. Audit only — no critique changed. JSON ↗ (critique_structure).

Closing the grounding gap: the literature apparatus (staged)

Acting on that finding. For the 22 un-grounded critiques, candidate references — academic and grey — were proposed and put through two gates before being staged for the editorial publish gate. Gate 1, identity (the fabrication firewall): every reference must resolve against a real registry via doi.org content-negotiation — one path covering Crossref and DataCite (arXiv, NBER, NIST, OECD) — with the resolved title, first author, and year all agreeing; a hallucinated or mis-attributed DOI is dropped. Gate 2, grounding (refute-by-default): a per-reference skeptic panel judged whether the real work supports the specific objection in the same direction and scope — no padding.

The gap is closing — honestly. Of 28 proposed references, 27 resolved to a real registered work (1 had no registry DOI), and the refute-by-default grounding gate then dropped 11 as topical-but-not-grounding — leaving 16 staged across 14 critiques, including the journal’s first grey literature (3: arXiv preprints + the NIST AI Risk Management Framework). That ~40% grounding-gate drop is the same heavy, honest funnel the original reference apparatus showed — the gate caught genuine over-reach (a foundation-models survey that actually supported the paper; an exposure study pointing the other way) rather than padding the bibliography. The 8 still-un-grounded critiques are an honest negative: their candidate citations were topical, not claim-grounding.

Staged, not live. Status staged_pending_editor — these references are held for the named-editor publish gate and change no published critique. Promotion (writing them onto a critique’s references[]) is the editor’s call. JSON ↗ (literature_grounding_staged).

Head-to-head: is Critical AI competitive with the best human and machine critics?

Every other benchmark scores the engine against an absolute gold. This asks the comparativequestion. For 8 papers each with an authoritative published human Comment, three critique arms were compared: Critical AI (its structured blind critique), an alternative engine (a generic strong-reviewer prompt on the same model and access, withoutthe journal’s methodology — so any gap is method, not horsepower), and the human Comment (its abstract-detectable objections, for equal access). All three were neutralised to one voice and the labels shuffled — defeating the register tell — then scored by a blind 3-lens convergence panelon validity, specificity, importance, and calibration (0–3).

• Critical AI (structured)2.77mean rank 2.08
• Alternative engine (generic, same model)2.72mean rank 1.54
• Human Comment (published)2.43mean rank 2.38

Competitive — but it does not dominate, and that’s the honest result. Against the published human Comment, Critical AI wins 5, ties 2, loses 1 on abstract-detectable substance. Against a generic strong-reviewer on the same model it merely ties (3–2–3); the baseline actually ranks first most often (mean rank 1.54 vs 2.08). What the journal’s methodology buys is calibration — it is the best-calibrated, least-overclaiming arm (2.96 vs 2.29) and strong on validity (2.96) — at a cost in specificity (2.42 vs 2.83): it casts a wider, more-hedged net where the baseline lands a sharper punch.

Read the bounds. The human had full-text access; the AI arms are abstract-only, so this compares abstract-detectable substance and the human’s 19full-text-only flaws are excluded (the access ceiling, again — the human’s real edge). The human arm is its abstract-detectable flaw set, not the full Comment prose; the baseline differs in method, not model; and n=8. So this is “at equal abstract access, the engine is competitive with human and machine critics and best-calibrated” — not “Critical AI is a better critic than humans.” Benchmark result only — no critique changed. JSON ↗ (head_to_head).

Audit caveat (G83). Prompted by the discovery that the blind panel over-credits, a source-grounded audit re-checked this live result. The over-credit worry is refuted: neutralisation was unbiased (7/8 targets, 0 favouring Critical AI) and Critical AI’s arm was clean (1/67 leakage, 0 fabricated). The opposite emerged: the baseline’s specificity edge was substantially leakage — 26/102 (25.5%)of its objections imported paper-specific facts absent from the abstract, which the panel rewarded as specificity. So the “ties baseline” result understates Critical AI, whose discipline avoids exactly that leakage. The raw numbers stand; this reframes them. (16/16 staged references still resolve and match title+author+year.)

A compounding improvement loop — that caught its own over-reach, then fixed it

The head-to-head exposed a gap; this is the standing engine that closes it — benchmark → diagnose → propose a change → held-out re-benchmark → two gates→ repeat. Each iteration generates three arms blind on 6 fresh held-out papers(v1 current, improved, baseline generic), neutralised, shuffled, scored by the blind panel (quality gate) — and a source-grounded gate audits whether the improved arm’s new specificity is abstract-licensed or fabricated/leaked. A change activates only if both pass.

Iteration 1 (sharpen): blocked. The first directive raised specificity but a source-grounded audit caught a ~13% over-reach tail (1 fabricated + 8 leakage of 68) the panel’s calibration lens missed— the same “be more specific” failure mode the correctness loop hit before. Not activated.

Iteration 2 (sharpen + grounding guard): activated.Informed by the head-to-head audit — which found leakage, not fabrication, was the dominant tail — iteration 2 added a guard: use only what the abstract licenses; if you can’t ground a specific, give its general form. Both gates now pass: specificity stays +0.39 over v1 (to 3) with calibration up (+0.16), and the source-grounded audit is clean (0 fabricated, 3 leakage of 72 = 4%, down from 13%). So improvementActive() is true— the first validated, fabrication-clean, baseline-competitive methodology improvement. The engine produced an improvement, caught its own over-reach, and fixed it. Validation result only; folding it into live generation stays an editor decision. JSON ↗ (improvement_engine).

Self-improvement, gated on held-out evidence

The benchmark doesn’t just measure — it diagnoses where the engine falls short, into 6reusable failure modes that feed the next critique’s self-check. But an improvement you assume works is not an improvement. So a candidate lesson-set activates only if it beats a no-lessons baseline on a held-out A/B — fresh papers it was never derived from, scored within one run (the only valid comparison, since the flaw decomposition is re-run each time).

• v1 — "be more specific" — sharpen every concern to a specific mechanism: FAILED → not activatedlessons 60% vs baseline 80%

  REGRESSED (delta -0.20): pruned correct-simple concerns and over-reached into specific-but-wrong ones. Failed the gate; not activated.

• v2 — additive — sharpen only where the abstract licenses it; never drop a valid concern or invent a false specific one: PASSED → activatedlessons 73% vs baseline 64%

  PASSED (delta +0.09): zero per-target regression, zero pruning; +1 real substantive catch (Brady length confound). Activated, with the margin (one flaw) disclosed.

v1 (“be more specific”) regressed by pruning correct-but-simple concerns; the v2 additive meta-rule (sharpen only where the abstract licenses it, never drop a valid concern) removed that harm and passed by a one-flaw marginwith zero pruning — a safe, modest, honestly-bounded improvement, not a large effect. The loop is falsifiable, and it caught v1’s regression before it could degrade a published critique.