Post-publication Comment · Critical AI
Comment on “AI meets politics: Examining the effects of different targeting strategies across 15 countries”
Critical AI · published 2026-06-30 · v2.0 · CRIT-000008
Concerning: Sanne Kruikemeier, Svenja Schäfer, Alice Hamilton, Puck Guldemond, Jade Vrielink, Carmen Dymanus, Annelien Van Remoortere, Sanne Tamboer · New Media & Society · 2026-06-04
Why this paper was selected
A large cross-national experiment on AI-generated political microtargeting bears directly on election integrity debates, making the scope of its persuasion claims worth scrutinising.
AI/AGI centrality 4/5 · societal relevance 5/5 · source-journal note: New Media & Society is a top-tier communication and media-studies journal. Tier A.
Summary
This is a well-powered, honestly-hedged cross-national experiment (N=7118, 15 countries) whose headline finding — small persuasive effects of party-congruent AI messages, null effects for age and combination targeting — survives full-text scrutiny. The full text actually *defuses* most abstract-era worries: the authors repeatedly stress effects are "small," call several results "only marginally significant," and disclose limitations (small per-country samples, single AI model, two-party design, manual stimulus editing). The defensible criticisms are localized inference/wording over-reaches, not design failures. The single hardest-to-refute problem is a flat self-contradiction in the Results: after reporting that the multi-factor interaction terms are all null (p>.05), the paper writes that it therefore "does find evidence of a reinforcing effect" — the exact opposite of what its own data and abstract say. Secondary issues: the Discussion claims older-EU-membership voters are more affected by *political* targeting when no political×length interaction reaches p<.05; H1c is declared "consistent with predictions" on a p=.097 (non-significant) coefficient; and the political-orientation manipulation confounds party label with differing argument content. Overall severity is moderate: the design and main null/small-effect conclusions are sound; the flaws are over-statements at the seams.
Central claims & evidence map
| Claim | Type | Evidence offered | Support | Overclaiming | Main weakness |
|---|---|---|---|---|---|
| Internal self-contradiction in the Results | Causal | and, thus, we do find evidence of a reinforcing effect. | Weak | Moderate | Internal self-contradiction in the Results |
| Discussion over-generalizes the moderation result | Causal | more likely to be affected by political and personal- | Moderate | Minor | Discussion over-generalizes the moderation result |
| A hypothesis is declared confirmed on a non-significant coefficient | Causal | ings are consistent with the predictions of H1c. | Moderate | Minor | A hypothesis is declared confirmed on a non-significant coefficient |
| The political-orientation manipulation confounds party identity with argument content | Methodological | The left-wing versions of the stimuli highlighted societal benefits, | Moderate | Minor | The political-orientation manipulation confounds party identity with argument content |
Per-claim assessment
C1. Internal self-contradiction in the Results
Internal self-contradiction in the Results. The sentence reports that the two-way interaction terms among the manipulated factors are all non-significant ("we do not find proof that the effects of the manipulated factors are conditional on each other") and then concludes the opposite in the same breath: that the study therefore DOES find a reinforcing effect. Null interactions are evidence AGAINST a reinforcing/cumulative effect, and this directly contradicts the paper's own abstract ("a combination of multiple categories does not affect persuasive outcomes"). Whether read as a dropped "not" or as a substantive error, the published inferential statement in the load-bearing Results section asserts a finding the data do not support.
C2. Discussion over-generalizes the moderation result
Discussion over-generalizes the moderation result. It states older-EU-membership voters are more affected by BOTH political and personality targeting, but in Table 2 the Political×length interaction is p=.267 (evaluation), p=.100 (importance), and p=.098 (agreement) — none below the conventional .05 threshold. Only the Personality×length interaction reaches significance (and only for evaluation/importance). Attributing a membership-length moderation to *political* targeting is not supported by the reported interaction coefficients.
C3. A hypothesis is declared confirmed on a non-significant coefficient
A hypothesis is declared confirmed on a non-significant coefficient. Issue agreement under correct political targeting is b=.083 with p=.097 — above the .05 threshold — yet the paper states the findings are consistent with the predictions of H1c, treating a null-at-conventional-alpha result as supportive. This inflates the count of confirmed sub-hypotheses for the headline finding.
C4. The political-orientation manipulation confounds party identity with argument content
The political-orientation manipulation confounds party identity with argument content. Left-wing and right-wing stimulus versions did not differ only in the party name: the left-wing versions emphasized societal benefits while right-wing versions emphasized economic advantages. Because "correct" political targeting is congruence between participant ideology and message, the H1 effect cannot cleanly separate party-label matching from argument-content matching — undercutting the interpretation that party congruence per se drives the headline effect. This content asymmetry is not flagged in the Limitations.
Scorecard
Sub-scores are 0–5 editorial judgements on fixed scales (higher is better, except methodological risk and overclaiming where higher is worse). They are contestable and open to a severity challenge from authors.
Strongest critique — statistical inference / reproducibility
Internal self-contradiction in the Results. The sentence reports that the two-way interaction terms among the manipulated factors are all non-significant ("we do not find proof that the effects of the manipulated factors are conditional on each other") and then concludes the opposite in the same breath: that the study therefore DOES find a reinforcing effect. Null interactions are evidence AGAINST a reinforcing/cumulative effect, and this directly contradicts the paper's own abstract ("a combination of multiple categories does not affect persuasive outcomes"). Whether read as a dropped "not" or as a substantive error, the published inferential statement in the load-bearing Results section asserts a finding the data do not support.
statistical inference
Discussion over-generalizes the moderation result. It states older-EU-membership voters are more affected by BOTH political and personality targeting, but in Table 2 the Political×length interaction is p=.267 (evaluation), p=.100 (importance), and p=.098 (agreement) — none below the conventional .05 threshold. Only the Personality×length interaction reaches significance (and only for evaluation/importance). Attributing a membership-length moderation to *political* targeting is not supported by the reported interaction coefficients.
statistical inference
A hypothesis is declared confirmed on a non-significant coefficient. Issue agreement under correct political targeting is b=.083 with p=.097 — above the .05 threshold — yet the paper states the findings are consistent with the predictions of H1c, treating a null-at-conventional-alpha result as supportive. This inflates the count of confirmed sub-hypotheses for the headline finding.
measurement / internal validity
The political-orientation manipulation confounds party identity with argument content. Left-wing and right-wing stimulus versions did not differ only in the party name: the left-wing versions emphasized societal benefits while right-wing versions emphasized economic advantages. Because "correct" political targeting is congruence between participant ideology and message, the H1 effect cannot cleanly separate party-label matching from argument-content matching — undercutting the interpretation that party congruence per se drives the headline effect. This content asymmetry is not flagged in the Limitations.
What the paper does well
The paper is methodologically strong and unusually candid, so most abstract-era suspicions do not survive the full text. It is well-powered (N=7118 across 15 countries), pre-registers clear hypotheses, uses appropriate random-intercept/random-slope multilevel models nesting respondents in countries, and reports robustness checks (continuous moderators, alternative cut-points, excluding ambiguous cases) in Appendix 4 that reproduce the main pattern. Crucially, the authors do not over-sell: they repeatedly state the effects are "small," label the issue-agreement and EU-membership results "only marginally significant," and devote a substantial Limitations section to the very weaknesses an external critic would raise — small per-country samples that cannot detect small country-level effects, a two-party operationalization in nominally multi-party systems, single-model/single-prompt AI dependence, manual editing of stimuli, and limited ecological validity. The central conclusions — a modest party-congruence effect and genuine nulls for age and combination targeting — are well-supported and appropriately hedged. The surviving criticisms are localized over-statements at the wording/inference seams, not failures of design or data.
Strongest critique
In the Results, the paper reports that all multi-factor interaction terms are non-significant (p>.05) and then writes "and, thus, we do find evidence of a reinforcing effect." — a flat self-contradiction. Null interactions are evidence against a cumulative/reinforcing effect, and the conclusion drawn here is the exact opposite of both the immediately preceding clause and the paper's own abstract ("a combination of multiple categories does not affect persuasive outcomes"). Whether this is a dropped "not" or a substantive slip, as published it is a defective inferential statement sitting in the load-bearing Results section; a refuter cannot rescue it on the merits, only excuse it as a typo — which itself concedes the printed claim is wrong.
Strongest fair defence
The paper is methodologically strong and unusually candid, so most abstract-era suspicions do not survive the full text. It is well-powered (N=7118 across 15 countries), pre-registers clear hypotheses, uses appropriate random-intercept/random-slope multilevel models nesting respondents in countries, and reports robustness checks (continuous moderators, alternative cut-points, excluding ambiguous cases) in Appendix 4 that reproduce the main pattern. Crucially, the authors do not over-sell: they repeatedly state the effects are "small," label the issue-agreement and EU-membership results "only marginally significant," and devote a substantial Limitations section to the very weaknesses an external critic would raise — small per-country samples that cannot detect small country-level effects, a two-party operationalization in nominally multi-party systems, single-model/single-prompt AI dependence, manual editing of stimuli, and limited ecological validity. The central conclusions — a modest party-congruence effect and genuine nulls for age and combination targeting — are well-supported and appropriately hedged. The surviving criticisms are localized over-statements at the wording/inference seams, not failures of design or data.
Conclusion
A solid, honestly-reported study whose main conclusions hold up; the defensible critique is confined to a handful of inference/wording over-reaches rather than design flaws. The single most damaging item is the self-contradictory "reinforcing effect" sentence in the Results, which as printed asserts a finding the study's own null interactions and abstract refute. The EU-membership-on-political-targeting claim, the treatment of a p=.097 coefficient as confirming H1c, and the party-vs-content confound are real but each is partly disclosed or partly inherent to the design, so they are secondary. None rises to the level of overturning the paper's headline (small, mostly-null) findings. Overall severity: moderate.
Reply from the authors
Following the practice of Nature Matters Arising, Science Technical Comments and PNAS Letters, this Comment is published as one half of a Comment + Reply pair: the authors of the original article are invited to respond, and any reply is published here verbatim alongside the Comment as part of the record.
Reply: not yet invited. No reply has been received for publication.
The authors have a right of reply and no veto. A reply may request a factual correction, a methodological rebuttal, a clarification, a data/code update, or a severity challenge, and is published unedited. See the right-of-reply policy.
Automated re-evaluation after reply: Authors may reply at any time; replies are published alongside, and a reply flagging a factual error triggers automated re-evaluation and a versioned correction; this critique addresses claims, framing and generalisation only, never the authors.
References
Every external source this Comment cites, each with a verified link. 0 fabricated.
Source-grounding attestation
- ✓Verbatim source spans present in the critique — 4/4 provenance spans re-derived in the critique prose
- ✓Passes the publication validator — no errors
- ✓Zero fabricated citations — 0 fabricated
- ✓Severity within the access-basis cap — severity "moderate" ≤ cap "high" for user_supplied
Every verbatim span the critique relies on is re-derived in the prose in-app; span-in-source is re-verifiable offline (the abstract is re-fetched, not stored, per the no-reproduce policy).
Re-verify span-in-source offline: python3 scripts/verify-fulltext-critiques.py
Independent faithfulness review
A refute-by-default adversarial panel (two independent reviewers — an overreach lens and a mischaracterization lens — that fetched the real source) tried to prove this critique misread the paper. This is an AI adversarial review recorded with its reasoning, not a deterministic check.
Both refuters retrieved the genuine abstract (via the OpenAlex API, reconstructed from the abstract_inverted_index and cross-checked: New Media & Society, 2026, N=7,118, three targeting strategies, the verbatim "Contrary to popular belief, targeting based on age or a combination of multiple categories does not affect persuasive outcomes," and the closing claim of "a robust analysis of how multiple targeting strategies influence the electorate in an EU election context"). Against that source, the critique's two load-bearing claims hold up. C1 quotes the orientation-congruence effect and the age/combined null accurately, correctly avoids inventing a personality-trait result, and frames its power-analysis point as a methodological caution rather than a claim the paper contradicts. C2 accurately attributes the paper's strong "influence the electorate" language while flagging the legitimate gap between attitudinal outcomes (ad likability, issue importance) and real electoral influence, and it preserves the paper's "EU election context" qualifier. The only blemish either refuter surfaced is rhetorical compression in the strongestCritique field ("under-powered null claims" overstating an absence of demonstrated power), but it is hedged at the claim level, low-severity, and substantively defensible — not a sustained misrepresentation. Neither refuter sustained a misreading; the critique is faithful to the retrieved source.
Version & correction history
| Version | Date | Change |
|---|---|---|
| v1.0 | 2026-06-15 | Initial publication. |
| v2.0 | 2026-06-30 | Upgraded from abstract-only to FULL-TEXT grounding (the operator-provided licensed New Media & Society PDF; accessBasis user_supplied). Re-critiqued against the verbatim full text and re-cleared the hardened convergence gate; abstract-era flaws the full text resolves were withdrawn. |
No silent substantive corrections — every change is versioned and visible.
How to cite this Comment
Critical AI. Comment on “AI meets politics: Examining the effects of different targeting strategies across 15 countries” (Sanne Kruikemeier et al., New Media & Society, 2026). Critical AI; 2026. https://policywindow.org/critique/c/ai-meets-politics-examining-the-effects-of-differe
A registered DOI will replace the URL once minted; until then the canonical URL is the persistent identifier. Highwire/Dublin-Core citation tags and a schema.org Review record are embedded in this page for Google Scholar and reference managers.
Verify this Comment. Its checkable facts (target DOI, access-basis severity cap, zero fabricated citations) are served — as the app’s self-report — at /critique/api/critiques/ai-meets-politics-examining-the-effects-of-differe/verify; to confirm them independently of this site, re-derive the same checks (and resolve the target DOI) with npx tsx scripts/verify-critical-ai.ts --critique ai-meets-politics-examining-the-effects-of-differe --live.
Content fingerprint 8662dbebb512aa56 (v2.0) — this Comment’s substantive content is content-addressed; a silent post-publication edit would change it.