{"$schema":"https://policywindow.org/critique/api/schema","critique_id":"CRIT-GEN-being-literate-behaving-","slug":"being-literate-behaving-literate-a-mixed-methods-a","url":"https://policywindow.org/critique/c/being-literate-behaving-literate-a-mixed-methods-a","doi":null,"status":"published","critique_type":"editorially_approved_ai_native_critique","publication_date":"2026-06-30","current_version":"2.0","target_paper":{"title":"Being literate, behaving literate? A mixed-methods approach to adolescents’ algorithm literacy and behavioral strategies on social media","authors":["Larissa Leonhard","Ruth Wendt","Claudia Riesmeyer"],"journal":"New Media & Society","doi":"10.1177/14614448261446589","url":"https://doi.org/10.1177/14614448261446589","publicationDate":"2026-05-03","paperType":"conceptual","accessBasis":"user_supplied","fullTextUsed":true,"fictional":false,"doi_url":"https://doi.org/10.1177/14614448261446589"},"source_journal":{"tier":"B","rankingSources":["resolved from the monitored-venue determination"],"rankingNote":"Tier B per the determination; ingested from an AGISS critique artifact."},"selection_provenance":{"id":"being-literate-behaving-literate-a-mixed-methods-a","venue":"New Media & Society","inMonitoredSet":true,"determinedTier":"B","recordedTier":"B","effectiveTier":"B","kind":"monitored","disclosed":true,"offListPeerReviewed":false},"selection":{"aiAgiCentralityScore":1,"societalRelevanceScore":3,"aiAgiCategories":[],"selectionReason":"Selected via the production queue; critique generated by the AGISS engine."},"scores":{"aiAgiContribution":1,"evidentiarySupport":3,"methodologicalRisk":3,"overclaiming":3,"reproducibilityOrAuditability":3,"societalImpactRelevance":3,"severity":"moderate","confidence":"high"},"severity_cap_for_access_basis":"high","plain_language_summary":"This is a careful, well-disclosed mixed-methods study (focus groups + diary + a quota-matched survey of 610 German adolescents) whose headline finding is a set of small structural-equation associations: more algorithm awareness goes with more 'indifferent' scrolling, and more algorithm knowledge goes with LESS interactive and critical behavior. The full text resolves most abstract-era worries — it openly concedes cross-sectional data, qualitative-sample education skew, and inability to reproduce the full literacy model. The one hard-to-refute problem is statistical bookkeeping: the significance legend under Table 3 (the table carrying every literacy→behavior result) says a single asterisk means p<.01, but the four key asterisked paths are reported in the text at p=.028, .022, .047 and (for awareness) .028 — all between .01 and .05. Taken literally, the legend would render the paper's entire central finding non-significant; the legend is almost certainly a typo for p<.05, but as printed it is an internal contradiction at the exact point the conclusions rest. A secondary, milder issue: the paper sets its own SRMR cutoff (<.05) in note 5, then calls the central SEM (SRMR=.055) and a CFA (SRMR=.050) 'acceptable.' Severity is low-to-moderate: the substantive associations are small and the flaws are presentational/threshold-consistency rather than evidence that the analysis is wrong.","claims":[{"id":"C1","text":"Table 3 — which carries every algorithm-literacy→behavior result and thus the paper's central claims — has a significance legend stating a single asterisk denotes p<.01, yet the four asterisked paths ","type":"descriptive","evidenceOffered":"*<.01. ***<.001.","support":"weak","overclaiming":"moderate","assessment":"Table 3 — which carries every algorithm-literacy→behavior result and thus the paper's central claims — has a significance legend stating a single asterisk denotes p<.01, yet the four asterisked paths it summarizes are reported in the running text at p=.028 (awareness→indifferent), p=.022 (knowledge→interactive), and p=.047 (knowledge→critical), all of which lie between .01 and .05 and therefore cannot satisfy a p<.01 threshold. The legend is almost certainly a typo for *<.05, but as printed the table contradicts the text: read literally, NONE of the four headline literacy-to-behavior paths clears the stated single-asterisk bar, which would nullify every confirmatory finding the abstract advertises ('higher algorithm awareness is associated with increased indifferent behavior, whereas greater algorithm knowledge correlates with reduced interaction'). This is an internal inconsistency the reader cannot defeat — both numbers are the paper's own.","mainWeakness":"Table 3 — which carries every algorithm-literacy→behavior result and thus the paper's central claims — has a significance legend stating a single asterisk denotes p<.01, yet the four asterisked paths ","confidence":"high"},{"id":"C2","text":"The authors define their OWN fit criterion in note 5 — 'a standardized root mean square residual (SRMR) with values lower than .05 indicated an acceptable model fit' — then label as 'acceptable' two m","type":"methodological","evidenceOffered":"a standardized root mean square residual (SRMR) with values lower than .05 indicated an","support":"moderate","overclaiming":"minor","assessment":"The authors define their OWN fit criterion in note 5 — 'a standardized root mean square residual (SRMR) with values lower than .05 indicated an acceptable model fit' — then label as 'acceptable' two models that violate it: the central SEM carrying all hypothesis tests reports SRMR = .055 ('the model showed acceptable fit values') and the three-factor behavioral CFA reports SRMR = .050 (not lower than .05). Judged against the paper's self-imposed threshold, the key model does not meet the SRMR cutoff, yet is described as acceptable without comment.","mainWeakness":"The authors define their OWN fit criterion in note 5 — 'a standardized root mean square residual (SRMR) with values lower than .05 indicated an acceptable model fit' — then label as 'acceptable' two m","confidence":"high"},{"id":"C3","text":"The survey is cross-sectional (Study 3) and the limitations section explicitly concedes the data 'did not allow for any indications of whether more algorithm literacy would be related to different beh","type":"causal","evidenceOffered":"driving engagement in these behaviors.","support":"moderate","overclaiming":"minor","assessment":"The survey is cross-sectional (Study 3) and the limitations section explicitly concedes the data 'did not allow for any indications of whether more algorithm literacy would be related to different behavioral strategies of adolescents over time,' yet the results narrate associations with directional/causal verbs — attitudes 'driving engagement in these behaviors' — implying a causal push from attitudes to behavior that the design cannot license. This is softened by the disclosed limitation and by SEM-path convention, so it is a secondary rather than headline flaw.","mainWeakness":"The survey is cross-sectional (Study 3) and the limitations section explicitly concedes the data 'did not allow for any indications of whether more algorithm literacy would be related to different beh","confidence":"high"}],"sections":[{"id":"flaw1","title":"Strongest critique — statistical inference / reporting consistency","body":"Table 3 — which carries every algorithm-literacy→behavior result and thus the paper's central claims — has a significance legend stating a single asterisk denotes p<.01, yet the four asterisked paths it summarizes are reported in the running text at p=.028 (awareness→indifferent), p=.022 (knowledge→interactive), and p=.047 (knowledge→critical), all of which lie between .01 and .05 and therefore cannot satisfy a p<.01 threshold. The legend is almost certainly a typo for *<.05, but as printed the table contradicts the text: read literally, NONE of the four headline literacy-to-behavior paths clears the stated single-asterisk bar, which would nullify every confirmatory finding the abstract advertises ('higher algorithm awareness is associated with increased indifferent behavior, whereas greater algorithm knowledge correlates with reduced interaction'). This is an internal inconsistency the reader cannot defeat — both numbers are the paper's own."},{"id":"flaw2","title":"reproducibility / model-fit reporting","body":"The authors define their OWN fit criterion in note 5 — 'a standardized root mean square residual (SRMR) with values lower than .05 indicated an acceptable model fit' — then label as 'acceptable' two models that violate it: the central SEM carrying all hypothesis tests reports SRMR = .055 ('the model showed acceptable fit values') and the three-factor behavioral CFA reports SRMR = .050 (not lower than .05). Judged against the paper's self-imposed threshold, the key model does not meet the SRMR cutoff, yet is described as acceptable without comment."},{"id":"flaw3","title":"causal language vs. cross-sectional design","body":"The survey is cross-sectional (Study 3) and the limitations section explicitly concedes the data 'did not allow for any indications of whether more algorithm literacy would be related to different behavioral strategies of adolescents over time,' yet the results narrate associations with directional/causal verbs — attitudes 'driving engagement in these behaviors' — implying a causal push from attitudes to behavior that the design cannot license. This is softened by the disclosed limitation and by SEM-path convention, so it is a secondary rather than headline flaw."},{"id":"strengths","title":"What the paper does well","body":"The paper is methodologically conscientious and the full text dissolves most abstract-era suspicions. It is genuinely mixed-methods with a sensible sequential design (focus groups → diary → survey each informing the next); the survey is quota-matched to German population margins on gender, age, education, location, and migration background, with lower-education adolescents recruited offline precisely to correct the qualitative samples' skew. Measures are borrowed from validated instruments (Zarouali et al. AMCA scale; Dogruel et al. knowledge test), full CFA fit statistics, alphas, and item-deletion decisions are reported, analysis code/package versions are given (lavaan 0.6-19, ML estimation), and materials plus Study-3 data are posted on OSF. Crucially, the limitations section pre-empts the obvious objections: it concedes the cross-sectional design forecloses over-time inference, that the qualitative samples under-represent lower education, and that the full Dogruel literacy model could not be reproduced. The effects are honestly reported as small and the discussion is appropriately hedged ('complex,' 'first holistic insight,' 'initial quantitative validation'). The asterisk legend is best read as a printing erratum rather than a substantive analytic error, and the SRMR overshoot (.055) is trivial by conventional (<.08) standards."}],"strongest_critique":"The single hardest-to-refute defect is a significance-reporting contradiction at the exact locus of the paper's conclusions. Table 3 reports every algorithm-literacy→behavior coefficient and is the empirical basis for the abstract's two confirmatory claims (awareness→more indifferent behavior; knowledge→less interaction). Its legend states 'asterisk = p<.01' (verbatim: '*<.01. ***<.001.'), but the running text gives those very asterisked paths as p=.028, p=.022, and p=.047 — all between .01 and .05. The two cannot both be true. The legend is surely a typo for *<.05, yet as published the table's stars overstate the significance level of the findings, and a reader applying the legend literally would conclude that none of the four key literacy-to-behavior effects reaches the marked threshold, voiding the headline result. Because both numbers are the paper's own and the discrepancy sits on the load-bearing table, this flaw cannot be defended away — only explained as an erratum.","strongest_fair_defence":"The paper is methodologically conscientious and the full text dissolves most abstract-era suspicions. It is genuinely mixed-methods with a sensible sequential design (focus groups → diary → survey each informing the next); the survey is quota-matched to German population margins on gender, age, education, location, and migration background, with lower-education adolescents recruited offline precisely to correct the qualitative samples' skew. Measures are borrowed from validated instruments (Zarouali et al. AMCA scale; Dogruel et al. knowledge test), full CFA fit statistics, alphas, and item-deletion decisions are reported, analysis code/package versions are given (lavaan 0.6-19, ML estimation), and materials plus Study-3 data are posted on OSF. Crucially, the limitations section pre-empts the obvious objections: it concedes the cross-sectional design forecloses over-time inference, that the qualitative samples under-represent lower education, and that the full Dogruel literacy model could not be reproduced. The effects are honestly reported as small and the discussion is appropriately hedged ('complex,' 'first holistic insight,' 'initial quantitative validation'). The asterisk legend is best read as a printing erratum rather than a substantive analytic error, and the SRMR overshoot (.055) is trivial by conventional (<.08) standards.","final_judgment":"A solid, well-disclosed study whose conclusions are honestly hedged and whose data/code are shared. The genuine over-reaches that survive full-text refutation are presentational and threshold-consistency issues, not evidence that the analysis is wrong: (1) a load-bearing significance legend (*<.01) that contradicts the p-values (.028/.022/.047) of the very paths it marks — almost certainly an erratum but, as printed, capable of nullifying the headline result if read literally; and (2) calling the central SEM 'acceptable' against the authors' own stated SRMR<.05 rule that it (.055) and a CFA (.050) fail. A causal-language slip ('driving') is real but mitigated by the disclosed cross-sectional limitation. Because the effects are small and the flaws are reporting-consistency rather than design-invalidating, overall severity is low-to-moderate. The fair verdict is a calibrated 'mostly sound, fix the table legend and the fit-cutoff inconsistency.'","review_process":{"aiAgentsUsed":["claim_extraction","ai_agi_relevance","adversarial","author_defence","citation_integrity","legal_risk","meta_review"],"reviewRounds":1,"humanEditor":{"name":"","role":"","approvalDate":"","declaredConflict":"none"},"expertCertification":{"used":false}},"author_response":{"notified":false,"status":"not_yet_invited"},"versions":[{"version":"1.0","date":"2026-06-21","note":"","changeType":"initial"},{"version":"2.0","date":"2026-06-30","note":"Upgraded from abstract-only to FULL-TEXT grounding (the operator-provided licensed New Media & Society PDF; accessBasis user_supplied). Re-critiqued against the verbatim full text and re-cleared the hardened convergence gate; abstract-era flaws the full text resolves were withdrawn.","changeType":"revision"}],"transparency":{"modelCardUrl":"/critique/model-card","publicAuditSummary":"Full-text critique grounded in the operator-provided licensed New Media & Society PDF (accessBasis user_supplied — re-verification requires source access). Every span is an exact substring of the stored full text; cleared the hardened 3-lens convergence gate. Upgraded from abstract-only v1.0; abstract-era flaws withdrawn where the full text resolves them. Targets claims/methods/inference only, never the authors.","privateAuditRecordExists":true,"citationVerification":{"status":"complete","checkedSources":[{"label":"DOI 10.1177/14614448261446589 — Crossref-verified","url":"https://doi.org/10.1177/14614448261446589","verified":true},{"label":"Full text used for span verification (licensed publisher PDF, provided to the editor; not redistributable)","url":"https://doi.org/10.1177/14614448261446589","verified":true}],"fabricatedCitations":0},"riskReview":{"copyright":"completed","defamation":"completed","note":"Licensed full text quoted sparingly under criticism/review; not redistributed (the PDF and extracted text are never committed). Targets claims/methods/inference only."}}}