Post-publication Comment · Critical AI
Comment on “More Versus Better: Artificial Intelligence, Incentives, and the Emerging Crisis in Peer Review”
Critical AI · published 2026-06-30 · v2.0 · CRIT-GEN-more-versus-better-artif
Concerning: Claudine Madras Gartenberg, Sharique Hasan, Alex Murray, Lamar Pierce · Organization Science · 2026-04-27
Why this paper was selected
Selected via the production queue; critique generated by the AGISS engine.
AI/AGI centrality 2/5 · societal relevance 4/5 · source-journal note: Tier S per the determination; ingested from an AGISS critique artifact.
Summary
The paper is honest and careful, and pre-discloses most of its own weaknesses, so a fair full-text reading lands at moderate severity. The one claim that genuinely over-reaches is the causal headline that the 42% jump in submissions is 'primarily due to AI, not organic growth': the evidence for this is really just a chart showing 'human-only' submissions shrinking while 'AI' ones grow — but that shift happens automatically as people start using AI, even if AI generated no extra papers at all. The authors' only real attempt to prove causation (comparing schools with strong publish-or-perish incentives) is one they themselves call 'noisy,' and it gets statistically weaker once fast-growing China/Hong Kong schools are removed. A secondary, minor issue: the abstract says volume rose '42% since November 2022,' but the actual comparison windows start in January 2022 and end in November 2025, so the date anchor is mislabeled. Everything else the abstract critique might have flagged (detector reliability, 'AI hurts authors,' 'AI worsens writing') is openly disclosed and reasonably handled in the body.
Central claims & evidence map
| Claim | Type | Evidence offered | Support | Overclaiming | Main weakness |
|---|---|---|---|---|---|
| The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migratin | Causal | is primarily due to AI use, not organic growth in the | Weak | Moderate | The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migratin |
| The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024 | Descriptive | risen by 42% since November 2022. At the same time, | Moderate | Minor | The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024 |
Per-claim assessment
C1. The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migratin
The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migrating from 'human' to 'AI' detector bands and cannot separate induced volume from re-classified volume. The paper's only causal lever (UTD-Responder DiD) is self-described as noisy and weakens to p<0.10 on total volume when China/HK schools are excluded, so the strong 'primarily due to AI, not organic growth' attribution exceeds what the design identifies.
C2. The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024
The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024–Nov 2025 — neither boundary is November 2022. The 'since November 2022' phrasing misstates the comparison underlying the headline statistic, even though the body discloses the true windows.
Scorecard
Sub-scores are 0–5 editorial judgements on fixed scales (higher is better, except methodological risk and overclaiming where higher is worse). They are contestable and open to a severity challenge from authors.
Strongest critique — causal identification / statistical inference
The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migrating from 'human' to 'AI' detector bands and cannot separate induced volume from re-classified volume. The paper's only causal lever (UTD-Responder DiD) is self-described as noisy and weakens to p<0.10 on total volume when China/HK schools are excluded, so the strong 'primarily due to AI, not organic growth' attribution exceeds what the design identifies.
measurement / reporting consistency
The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024–Nov 2025 — neither boundary is November 2022. The 'since November 2022' phrasing misstates the comparison underlying the headline statistic, even though the body discloses the true windows.
What the paper does well
The paper is methodologically conscientious and repeatedly refuses to over-claim where it matters most. It explicitly frames itself as an "early account" that "cannot make a normative assessment," reports only aggregate trends rather than individual classifications, builds in a two-year pre-ChatGPT placebo period, and validates the abstract-as-proxy assumption on a 230-manuscript full-text subsample. Crucially, it pre-empts its own sharpest confound: footnote 8 concedes that with author fixed effects the AI-to-rejection effect "disappears" and that AI use and author quality are "confounded," and footnote 11 surfaces reviewers' objection that the writing-quality proxies may be poor. The detector choice (Pangram) is independently validated, its asymmetric false-positive design is honestly argued to bias toward UNDERstating AI prevalence, and the headline volume windows (Jan 2022–Nov 2023 vs Jan 2024–Nov 2025) and the noisy UTD incentive measure are all disclosed in the body. For a descriptive task-force report, this is a high-transparency piece whose conclusions are appropriately hedged in the body even where the abstract tightens them.
Strongest critique
The paper's load-bearing causal claim — that the 42% submission surge "is primarily due to AI use, not organic growth in the field or increased journal reputation" (line 191; abstract: AI writing "accounts for nearly all of these trends") — is not identified by the evidence offered. The primary support is a decomposition (Figures 2–3) showing human-only submissions falling while AI-flagged bands rise. But that pattern is partly mechanical: as AI use normalizes, the SAME authors' submissions migrate from "human" to "AI" detector bands, inflating the AI categories without AI causing any additional volume. A compositional shift in band membership cannot distinguish "AI induced extra submissions" from "existing submitters now use AI." The only genuine identification attempt is the UTD-Responder difference-in-differences, yet the authors themselves caution that "our measure of school incentives is noisy" and that the topic "deserves further, more rigorous analysis," and the total-volume effect drops to a marginally significant 0.067 (p<0.10) once Mainland-China/Hong-Kong schools are excluded (Panel B). The strong "primarily due to AI, not organic growth" framing therefore outruns what the disclosed design can establish.
Strongest fair defence
The paper is methodologically conscientious and repeatedly refuses to over-claim where it matters most. It explicitly frames itself as an "early account" that "cannot make a normative assessment," reports only aggregate trends rather than individual classifications, builds in a two-year pre-ChatGPT placebo period, and validates the abstract-as-proxy assumption on a 230-manuscript full-text subsample. Crucially, it pre-empts its own sharpest confound: footnote 8 concedes that with author fixed effects the AI-to-rejection effect "disappears" and that AI use and author quality are "confounded," and footnote 11 surfaces reviewers' objection that the writing-quality proxies may be poor. The detector choice (Pangram) is independently validated, its asymmetric false-positive design is honestly argued to bias toward UNDERstating AI prevalence, and the headline volume windows (Jan 2022–Nov 2023 vs Jan 2024–Nov 2025) and the noisy UTD incentive measure are all disclosed in the body. For a descriptive task-force report, this is a high-transparency piece whose conclusions are appropriately hedged in the body even where the abstract tightens them.
Conclusion
This is a careful, unusually self-aware descriptive paper that mostly stays within its evidence and pre-discloses its biggest weaknesses (detector limits, author-fixed-effects confounding, the contested meaning of "writing quality"). The one place where the rhetoric genuinely outruns the design is the causal attribution of the submission surge: the abstract/intro assert the volume rise is "primarily due to AI use, not organic growth," but the supporting evidence is a compositional band-decomposition that cannot, even in principle, separate authors migrating from "human" to "AI" detector bands from AI generating genuinely additional submissions — and the paper's sole identification lever (the UTD-Responder DiD) is self-described as noisy and loses robustness on total volume in Panel B. That single over-reach is real and survives full-text refutation; most other abstract-era worries are resolved by the paper's own disclosures. Overall severity is moderate, concentrated in one causal claim, not pervasive.
Reply from the authors
Following the practice of Nature Matters Arising, Science Technical Comments and PNAS Letters, this Comment is published as one half of a Comment + Reply pair: the authors of the original article are invited to respond, and any reply is published here verbatim alongside the Comment as part of the record.
Reply: not yet invited. No reply has been received for publication.
The authors have a right of reply and no veto. A reply may request a factual correction, a methodological rebuttal, a clarification, a data/code update, or a severity challenge, and is published unedited. See the right-of-reply policy.
References
Every external source this Comment cites, each with a verified link. 0 fabricated.
Source-grounding attestation
- ✓Verbatim source spans present in the critique — 2/2 provenance spans re-derived in the critique prose
- ✓Passes the publication validator — no errors
- ✓Zero fabricated citations — 0 fabricated
- ✓Severity within the access-basis cap — severity "moderate" ≤ cap "high" for user_supplied
Every verbatim span the critique relies on is re-derived in the prose in-app; span-in-source is re-verifiable offline (the abstract is re-fetched, not stored, per the no-reproduce policy).
Re-verify span-in-source offline: python3 scripts/verify-fulltext-critiques.py
Independent faithfulness review
A refute-by-default adversarial panel (two independent reviewers — an overreach lens and a mischaracterization lens — that fetched the real source) tried to prove this critique misread the paper. This is an AI adversarial review recorded with its reasoning, not a deterministic check.
All seven critique claims restate the abstract accurately and quote it verbatim. The critique consistently flags its key inference — that "AI-generated" writing is a detector-defined variable whose method is undescribed — as a presumption ("presumably by a classifier the abstract does not describe") rather than asserting the abstract said so. Every interpretive/causal concern (temporal-coincidence framing in c1, detector-driven confounding in c2/c3, equilibrium-as-conjecture in c7) is explicitly hedged with "on the critic's reading." c2's reading that "accounts for nearly all of these trends" covers both the 42% rise and the quality decline is well-supported by the antecedent sentence. c4, c5, and c6 are charitable and even credit the authors' hedging and self-limitation. No claim asserts beyond the abstract or strengthens/narrows what the paper says; the strongest-critique and final-judgment paragraphs likewise cap severity at moderate given abstract-only access and route verification to the appendix. No substantiated overreach or mischaracterization.
Version & correction history
| Version | Date | Change |
|---|---|---|
| v1.0 | 2026-06-21 | |
| v2.0 | 2026-06-30 | Upgraded from abstract-only to FULL-TEXT grounding (the operator-provided licensed Organization Science PDF; accessBasis user_supplied). Re-critiqued against the verbatim full text and re-cleared the hardened convergence gate; abstract-era flaws the full text resolves were withdrawn. |
No silent substantive corrections — every change is versioned and visible.
How to cite this Comment
Critical AI. Comment on “More Versus Better: Artificial Intelligence, Incentives, and the Emerging Crisis in Peer Review” (Claudine Madras Gartenberg et al., Organization Science, 2026). Critical AI; 2026. https://policywindow.org/critique/c/more-versus-better-artificial-intelligence-incenti
A registered DOI will replace the URL once minted; until then the canonical URL is the persistent identifier. Highwire/Dublin-Core citation tags and a schema.org Review record are embedded in this page for Google Scholar and reference managers.
Verify this Comment. Its checkable facts (target DOI, access-basis severity cap, zero fabricated citations) are served — as the app’s self-report — at /critique/api/critiques/more-versus-better-artificial-intelligence-incenti/verify; to confirm them independently of this site, re-derive the same checks (and resolve the target DOI) with npx tsx scripts/verify-critical-ai.ts --critique more-versus-better-artificial-intelligence-incenti --live.
Content fingerprint 5b61e586d70d2098 (v2.0) — this Comment’s substantive content is content-addressed; a silent post-publication edit would change it.