Comment on “The Cybernetic Teammate: A Field Experiment on Generative AI and Teamwork”

A widely-discussed preregistered field experiment on generative AI and teamwork in a major firm; its results are already cited in debates about AI replacing collaboration, making the generalisation step worth scrutinising.

This is a strong, preregistered 2×2 field experiment (791 P&G professionals) with a credible design, transparent robustness checks, and unusually candid disclosure of limitations. The full text resolves most concerns the abstract alone raised. The one inferential over-reach that survives adversarial reading is the flagship "AI matched teams" claim: it is an equivalence statement supported by separate significance and visual similarity, but the authors never report the one pairwise test (Individual+AI vs Team No AI) or equivalence test that the claim logically requires — and their point estimates actually favor AI (0.37 vs 0.24). Secondary, lower-severity issues: the abstract elevates an admittedly exploratory emotion result to a co-equal "pillar"; self-reported emotion gains are attributed to "the actual experience of working with AI" in a non-blind design vulnerable to novelty/demand effects; and the human-selection-advantage framing rests on an untested 50% vs 37% gap. Calibrated severity: moderate.

A preregistered (AEARCTR-0013603) 2×2 field experiment at Procter & Gamble (791 professionals) randomising AI access and team vs solo work on real product-development tasks, with blinded expert evaluation. This critique is grounded in the full published text (Organization Science), provided to the editor as a licensed PDF.

The paper's flagship finding is an EQUIVALENCE claim ('AI matched / was comparable to teams without AI'), but it is supported only by each arm being separately significant versus the control and by visual distribution overlap (Figure 3). No equivalence test (e.g., TOST) is reported, and — tellingly — Table 2 reports the pairwise p-value for Team+AI vs Team No AI (p=0.242) but NEVER reports the Individual+AI vs Team No AI difference test that the 'matched' claim actually requires. This is 'absence of a significant difference' being read as 'evidence of equivalence.' Worse, the point estimates do not support equivalence: Individual+AI = 0.373 SD vs Team No AI = 0.245 SD, i.e., AI is numerically HIGHER, so the data are at least as consistent with 'AI beats teams' as with 'AI equals teams.' The central organizational implication (AI can substitute for a human teammate) is built on an untested null.

The abstract presents emotional response as finding (3), a co-equal 'pillar' alongside performance and expertise. The body, however, discloses that sociality was preregistered only 'with unclear effects' and that the emotional analysis 'emerged as a significant factor during the study' — i.e., it is an exploratory/emergent result, not a confirmatory preregistered hypothesis. Presenting an exploratory finding with the same evidentiary weight as the two confirmatory preregistered outcomes inflates its standing.

The positive-emotion result is self-reported and the AI condition is inherently non-blind (participants knew they had AI, received AI-specific training, and used a novel tool). The paper attributes the observed positive shift to 'the actual experience of working with AI,' arguing that post-assignment baselines net out initial reactions to assignment. But a post-assignment baseline does not control for demand effects or the novelty/excitement of using a new AI tool DURING the task — exactly the period the emotion change captures. Self-reported affect toward a salient new technology in a non-blind upskilling program is a textbook setting for demand and novelty confounds, which the paper does not address for this outcome.

The 'humans retain a selection advantage' narrative rests on a 50% (human teams) vs ~37% (AI) probability-of-picking-best-idea gap from Figure 8 panel (b), but no significance test or confidence interval for this specific difference is reported, and the relevant pairwise comparisons across these conditions are small-sample. The paper itself notes elsewhere (endnote 24) that incremental cross-condition differences 'are not statistically significant,' so presenting selection superiority as a robust human-vs-AI contrast outruns the reported evidence.

This is a genuinely high-quality study and most abstract-era worries dissolve on full reading. It is preregistered (AEARCTR-0013603) with the performance and expertise hypotheses clearly designated primary and the emotion pillar honestly flagged as exploratory. The 2×2 design is embedded in real P&G product-development work with real stakes, blinded expert evaluators, multiple evaluations per solution, stratified randomization, and pre-treatment balance (Table 1). Robustness is unusually thorough: wild-cluster-bootstrap SEs at the eight randomization units (Table A3), AI-generated re-evaluations, alternative team-aggregation specifications, typo controls, and a composite-index replication. The authors repeatedly and explicitly hedge the very claims most open to attack — they call the 40/60 augment-vs-substitute split 'indicative rather than causal,' concede selection differences are not statistically significant (endnote 24), warn emotion results are 'immediate reactions' not long-term dynamics, and lay out scope conditions (single firm, one-day virtual flash teams, single AI model). The equivalence-claim critique is real but narrow: the distributions in Figure 3 are visibly similar and both arms clearly beat control, so the substantive story is directionally sound even if the 'matched' phrasing overstates the statistical warrant.