{"$schema":"https://policywindow.org/critique/api/schema","critique_id":"CRIT-GEN-more-versus-better-artif","slug":"more-versus-better-artificial-intelligence-incenti","url":"https://policywindow.org/critique/c/more-versus-better-artificial-intelligence-incenti","doi":null,"status":"published","critique_type":"editorially_approved_ai_native_critique","publication_date":"2026-06-30","current_version":"2.0","target_paper":{"title":"More Versus Better: Artificial Intelligence, Incentives, and the Emerging Crisis in Peer Review","authors":["Claudine Madras Gartenberg","Sharique Hasan","Alex Murray","Lamar Pierce"],"journal":"Organization Science","doi":"10.1287/orsc.2026.ed.v37.n3","url":"https://doi.org/10.1287/orsc.2026.ed.v37.n3","publicationDate":"2026-04-27","paperType":"conceptual","accessBasis":"user_supplied","fullTextUsed":true,"fictional":false,"doi_url":"https://doi.org/10.1287/orsc.2026.ed.v37.n3"},"source_journal":{"tier":"S","rankingSources":["resolved from the monitored-venue determination"],"rankingNote":"Tier S per the determination; ingested from an AGISS critique artifact."},"selection_provenance":{"id":"more-versus-better-artificial-intelligence-incenti","venue":"Organization Science","inMonitoredSet":true,"determinedTier":"S","recordedTier":"S","effectiveTier":"S","kind":"monitored","disclosed":true,"offListPeerReviewed":false},"selection":{"aiAgiCentralityScore":2,"societalRelevanceScore":4,"aiAgiCategories":[],"selectionReason":"Selected via the production queue; critique generated by the AGISS engine."},"scores":{"aiAgiContribution":2,"evidentiarySupport":3,"methodologicalRisk":3,"overclaiming":3,"reproducibilityOrAuditability":3,"societalImpactRelevance":4,"severity":"moderate","confidence":"high"},"severity_cap_for_access_basis":"high","plain_language_summary":"The paper is honest and careful, and pre-discloses most of its own weaknesses, so a fair full-text reading lands at moderate severity. The one claim that genuinely over-reaches is the causal headline that the 42% jump in submissions is 'primarily due to AI, not organic growth': the evidence for this is really just a chart showing 'human-only' submissions shrinking while 'AI' ones grow — but that shift happens automatically as people start using AI, even if AI generated no extra papers at all. The authors' only real attempt to prove causation (comparing schools with strong publish-or-perish incentives) is one they themselves call 'noisy,' and it gets statistically weaker once fast-growing China/Hong Kong schools are removed. A secondary, minor issue: the abstract says volume rose '42% since November 2022,' but the actual comparison windows start in January 2022 and end in November 2025, so the date anchor is mislabeled. Everything else the abstract critique might have flagged (detector reliability, 'AI hurts authors,' 'AI worsens writing') is openly disclosed and reasonably handled in the body.","claims":[{"id":"C1","text":"The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migratin","type":"causal","evidenceOffered":"is primarily due to AI use, not organic growth in the","support":"weak","overclaiming":"moderate","assessment":"The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migrating from 'human' to 'AI' detector bands and cannot separate induced volume from re-classified volume. The paper's only causal lever (UTD-Responder DiD) is self-described as noisy and weakens to p<0.10 on total volume when China/HK schools are excluded, so the strong 'primarily due to AI, not organic growth' attribution exceeds what the design identifies.","mainWeakness":"The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migratin","confidence":"high"},{"id":"C2","text":"The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024","type":"descriptive","evidenceOffered":"risen by 42% since November 2022. At the same time,","support":"moderate","overclaiming":"minor","assessment":"The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024–Nov 2025 — neither boundary is November 2022. The 'since November 2022' phrasing misstates the comparison underlying the headline statistic, even though the body discloses the true windows.","mainWeakness":"The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024","confidence":"high"}],"sections":[{"id":"flaw1","title":"Strongest critique — causal identification / statistical inference","body":"The headline claim that the submission surge is caused by AI rather than organic growth is supported only by a band-composition decomposition, which is partly a mechanical artifact of authors migrating from 'human' to 'AI' detector bands and cannot separate induced volume from re-classified volume. The paper's only causal lever (UTD-Responder DiD) is self-described as noisy and weakens to p<0.10 on total volume when China/HK schools are excluded, so the strong 'primarily due to AI, not organic growth' attribution exceeds what the design identifies."},{"id":"flaw2","title":"measurement / reporting consistency","body":"The abstract and intro headline the surge as 'risen by 42% since November 2022', but the 42% is actually computed from a window that begins BEFORE ChatGPT (Jan 2022–Nov 2023) compared against Jan 2024–Nov 2025 — neither boundary is November 2022. The 'since November 2022' phrasing misstates the comparison underlying the headline statistic, even though the body discloses the true windows."},{"id":"strengths","title":"What the paper does well","body":"The paper is methodologically conscientious and repeatedly refuses to over-claim where it matters most. It explicitly frames itself as an \"early account\" that \"cannot make a normative assessment,\" reports only aggregate trends rather than individual classifications, builds in a two-year pre-ChatGPT placebo period, and validates the abstract-as-proxy assumption on a 230-manuscript full-text subsample. Crucially, it pre-empts its own sharpest confound: footnote 8 concedes that with author fixed effects the AI-to-rejection effect \"disappears\" and that AI use and author quality are \"confounded,\" and footnote 11 surfaces reviewers' objection that the writing-quality proxies may be poor. The detector choice (Pangram) is independently validated, its asymmetric false-positive design is honestly argued to bias toward UNDERstating AI prevalence, and the headline volume windows (Jan 2022–Nov 2023 vs Jan 2024–Nov 2025) and the noisy UTD incentive measure are all disclosed in the body. For a descriptive task-force report, this is a high-transparency piece whose conclusions are appropriately hedged in the body even where the abstract tightens them."}],"strongest_critique":"The paper's load-bearing causal claim — that the 42% submission surge \"is primarily due to AI use, not organic growth in the field or increased journal reputation\" (line 191; abstract: AI writing \"accounts for nearly all of these trends\") — is not identified by the evidence offered. The primary support is a decomposition (Figures 2–3) showing human-only submissions falling while AI-flagged bands rise. But that pattern is partly mechanical: as AI use normalizes, the SAME authors' submissions migrate from \"human\" to \"AI\" detector bands, inflating the AI categories without AI causing any additional volume. A compositional shift in band membership cannot distinguish \"AI induced extra submissions\" from \"existing submitters now use AI.\" The only genuine identification attempt is the UTD-Responder difference-in-differences, yet the authors themselves caution that \"our measure of school incentives is noisy\" and that the topic \"deserves further, more rigorous analysis,\" and the total-volume effect drops to a marginally significant 0.067 (p<0.10) once Mainland-China/Hong-Kong schools are excluded (Panel B). The strong \"primarily due to AI, not organic growth\" framing therefore outruns what the disclosed design can establish.","strongest_fair_defence":"The paper is methodologically conscientious and repeatedly refuses to over-claim where it matters most. It explicitly frames itself as an \"early account\" that \"cannot make a normative assessment,\" reports only aggregate trends rather than individual classifications, builds in a two-year pre-ChatGPT placebo period, and validates the abstract-as-proxy assumption on a 230-manuscript full-text subsample. Crucially, it pre-empts its own sharpest confound: footnote 8 concedes that with author fixed effects the AI-to-rejection effect \"disappears\" and that AI use and author quality are \"confounded,\" and footnote 11 surfaces reviewers' objection that the writing-quality proxies may be poor. The detector choice (Pangram) is independently validated, its asymmetric false-positive design is honestly argued to bias toward UNDERstating AI prevalence, and the headline volume windows (Jan 2022–Nov 2023 vs Jan 2024–Nov 2025) and the noisy UTD incentive measure are all disclosed in the body. For a descriptive task-force report, this is a high-transparency piece whose conclusions are appropriately hedged in the body even where the abstract tightens them.","final_judgment":"This is a careful, unusually self-aware descriptive paper that mostly stays within its evidence and pre-discloses its biggest weaknesses (detector limits, author-fixed-effects confounding, the contested meaning of \"writing quality\"). The one place where the rhetoric genuinely outruns the design is the causal attribution of the submission surge: the abstract/intro assert the volume rise is \"primarily due to AI use, not organic growth,\" but the supporting evidence is a compositional band-decomposition that cannot, even in principle, separate authors migrating from \"human\" to \"AI\" detector bands from AI generating genuinely additional submissions — and the paper's sole identification lever (the UTD-Responder DiD) is self-described as noisy and loses robustness on total volume in Panel B. That single over-reach is real and survives full-text refutation; most other abstract-era worries are resolved by the paper's own disclosures. Overall severity is moderate, concentrated in one causal claim, not pervasive.","review_process":{"aiAgentsUsed":["claim_extraction","ai_agi_relevance","adversarial","author_defence","citation_integrity","legal_risk","meta_review"],"reviewRounds":1,"humanEditor":{"name":"","role":"","approvalDate":"","declaredConflict":"none"},"expertCertification":{"used":false}},"author_response":{"notified":false,"status":"not_yet_invited"},"versions":[{"version":"1.0","date":"2026-06-21","note":"","changeType":"initial"},{"version":"2.0","date":"2026-06-30","note":"Upgraded from abstract-only to FULL-TEXT grounding (the operator-provided licensed Organization Science PDF; accessBasis user_supplied). Re-critiqued against the verbatim full text and re-cleared the hardened convergence gate; abstract-era flaws the full text resolves were withdrawn.","changeType":"revision"}],"transparency":{"modelCardUrl":"/critique/model-card","publicAuditSummary":"Full-text critique grounded in the operator-provided licensed Organization Science PDF (accessBasis user_supplied — re-verification requires source access). Every span is an exact substring of the stored full text; cleared the hardened 3-lens convergence gate. Upgraded from abstract-only v1.0; abstract-era flaws withdrawn where the full text resolves them. Targets claims/methods/inference only, never the authors.","privateAuditRecordExists":true,"citationVerification":{"status":"complete","checkedSources":[{"label":"DOI 10.1287/orsc.2026.ed.v37.n3 — Crossref-verified","url":"https://doi.org/10.1287/orsc.2026.ed.v37.n3","verified":true},{"label":"Full text used for span verification (licensed publisher PDF, provided to the editor; not redistributable)","url":"https://doi.org/10.1287/orsc.2026.ed.v37.n3","verified":true}],"fabricatedCitations":0},"riskReview":{"copyright":"completed","defamation":"completed","note":"Licensed full text quoted sparingly under criticism/review; not redistributed (the PDF and extracted text are never committed). Targets claims/methods/inference only."}}}