California SB 243: Companion Chatbots

Operative Mechanics

SB 243, chaptered as Cal. Stats. 2025, ch. 677 and codified at Cal. Bus. & Prof. Code §§ 22601–22606, builds three operator duties around a single triggering object: a "companion chatbot" with an adaptive, human-like natural-language interface meeting a user's social needs. The core duty is conditional disclosure — § 22602(a) requires a clear-and-conspicuous AI notification only "if a reasonable person" would be misled into believing the interlocutor is human, importing a contextual rather than per-message standard. Layered atop are a published self-harm/crisis-referral protocol (§ 22602(b)) and minor-specific safeguards (§ 22602(c)): a default every-three-hours break-and-AI reminder plus measures against sexually explicit content. The architecture is calibrated by harm and audience rather than by model capability, marking it a deployment-side consumer-protection statute.

Cross-Jurisdiction Position

SB 243 occupies a distinct niche within California's 2025 AI legislative cluster. Its sibling statute, SB 53 (TFAIA, ch. 138), regulates frontier model developers through transparency and safety-framework disclosure; SB 243 instead binds operators at the point of consumer interaction, making the chatbot's deployed behavior — not its training compute — the regulated object. The conditional-disclosure logic of § 22602(a) echoes the EU AI Act's Article 50 transparency obligation for AI systems that interact with humans, Regulation (EU) 2024/1689, though SB 243 narrows the trigger to companion systems and to a reasonable-person misled standard. As the first US state statute naming "companion chatbots" specifically, it pioneers a use-case-targeted model contrasting with capability-tiered frameworks, addressing a relational-harm vector that broad transparency mandates leave underspecified.

Key Fault Lines

The enforcement and disclosure designs draw the sharpest critique. The § 22605 private right of action — injunctive relief, the greater of actual damages or $1,000 per violation, and attorney's fees — gives a remedy whose efficacy depends on contestability conditions the literature shows are easily hollowed out: studies find appeal and contestation, not nominal oversight, drive procedural fairness ¹, and "meaningful" contestation needs articulated subject needs often absent from drafting ^2,3. The "reasonable person would be misled" trigger in § 22602(a) risks the rubber-stamp pathology Sterz et al. (2024) document for thin oversight ⁴, while the § 22602(b) crisis protocol raises whether such systems act as device-like clinical decision support ^5,6.

Implementation Trajectory

SB 243 is in force, signed October 13, 2025, with a staged timeline: operator duties under § 22602 became operative January 1, 2026, while § 22603 annual reporting to the Office of Suicide Prevention — capturing crisis-referral data — begins July 1, 2027, deferring the empirical-accountability mechanism by eighteen months. That design responds to a documented gap, since freedom-of-information regimes "generally only grant access to existing documents" with "no mature standard for documenting AI models" ⁷, making bespoke statutory disclosure the more reliable transparency channel. The § 22602(b) crisis protocol will likely interact with maturing post-market governance frameworks for health AI ⁸ and international harmonization efforts ⁹, testing whether a use-case statute can scale into broader companion-AI safety norms.

Cross-jurisdiction comparison

How peer instruments treat the topics California SB 243: Companion Chatbots governs.

°= industry self-imposed voluntary framework. Comparing a voluntary code's "governs" tint with a binding regulation's "governs" tint flattens the legal-force distinction; use the instrument-page banner for the operative status of each.

Does this instrument’s approach work? — the social-science evidence

Aggregated over the 3 topics this instrument governs: whether each harm is empirically real, and whether the peer-reviewed evidence shows governance reduces it. The badge is the epistemic status of the evidence— “thin”/“absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.

Of the 3 governed topics with a social-science evidence review, evidence that governance reduces the harm is established for 0, contested for 0, thin for 0, and absent for 3 — for most, no replicated study yet shows this instrument's approach works (the "second silence").

• AI in Healthcare

  • Is the harm real?evidence: established

    Both the benefit and the harm of clinical AI are empirically real and well-documented, but outcomes are highly deployment-dependent. Rigorous prospective studies show genuine clinical value in narrow tasks — the MASAI RCT (>100,000 women) found AI-supported mammography detected ~20% more cancers (6.1 vs 5.1 per 1000 screened) at comparable recall rates (Lang et al. 2023, Lancet Oncology), and IDx-DR's pivotal trial achieved 87.2% sensitivity / 90.7% specificity for diabetic retinopathy (Abramoff et al. 2018, npj Digital Medicine) — yet widely deployed models can fail or harm: the Epic Sepsis Model, live at hundreds of US hospitals, scored AUC 0.63 with 33% sensitivity on external validation (Wong et al. 2021, JAMA Internal Medicine), and a population-health algorithm covering ~200M people understated Black patients' illness because it predicted cost not need (Obermeyer et al. 2019, Science). Honest caveat: there is no single 'AI in healthcare' effect — performance ranges from life-saving to dangerous depending on task, calibration, and whether the model was prospectively validated.

    Sources: Lang K, Josefsson V, Larsson A-M, et al. 2023 (Lancet Oncology 24(8):936-944, MASAI trial clinical safety analysis; AI-supported screening detected 6.1 vs 5.1 cancers per 1000, ~20% higher, similar recall rates); Abramoff MD, Lavin PT, Birch M, Shah N, Folk JC. 2018 (npj Digital Medicine 1:39, IDx-DR pivotal trial; 87.2% sensitivity / 90.7% specificity); Wong A, Otles E, Donnelly JP, et al. 2021 (JAMA Internal Medicine 181(8):1065-1070, Epic Sepsis Model external validation; AUC 0.63, 33% sensitivity); Obermeyer Z, Powers B, Vogeli C, Mullainathan S. 2019 (Science 366(6464):447-453, racial bias from cost-as-proxy)

  • Does governance work?evidence: absent

    There is essentially no impact-evaluation evidence that the prevailing governance regime for medical AI — FDA authorization, predominantly via the 510(k) substantial-equivalence pathway — measurably reduces patient harm or improves outcomes. Analyses of authorized AI devices find that clinical validation is frequently absent or non-prospective (of 521 FDA-authorized AI devices, ~43% had no published clinical-validation data and only ~28% were prospectively validated; Chouffani El Fassi & Henderson et al. 2024) and that demographic performance is almost never reported (race/ethnicity in 3.6%, and only 9.0% of 692 510(k)/cleared AI devices carried a prospective post-market-surveillance study; Muralidharan et al. 2024). Earlier analysis of 130 cleared devices likewise found 97% were evaluated only retrospectively (Wu et al. 2021). The closest analogue evidence on the pathway itself is discouraging: the Institute of Medicine (2011) concluded the 510(k) process was not designed to assess safety and effectiveness — i.e., no direct study establishes that the rule, as written, prevents the harms it targets. Caveat: this is an absence of impact evaluation plus reporting-gap and design-critique evidence, not a study showing the regime fails to reduce harm.

    Sources: Chouffani El Fassi S, Abdullah A, Fang Y, ... Henderson GE, et al. 2024 (Nature Medicine, 'Not all AI health tools with regulatory authorization are clinically validated', s41591-024-03203-3; 521 devices, ~43% no clinical validation, ~28% prospectively validated); Muralidharan V, Adewale BA, Huang CJ, et al. 2024 (npj Digital Medicine 7:273, scoping review of reporting gaps in 692 FDA-approved AI medical devices; race/ethnicity 3.6%, prospective post-market surveillance 9.0%); Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. 2021 (Nature Medicine 27:582-584, analysis of 130 FDA approvals; 97% retrospective-only evaluation); Institute of Medicine 2011 (Medical Devices and the Public's Health: The FDA 510(k) Clearance Process at 35 Years)

• Individual Redress

  • Is the harm real?evidence: established

    The premise behind redress — that affected people lack meaningful recourse against automated decisions — is real, but the flagship instrument is weaker than commonly assumed. Wachter, Mittelstadt & Floridi (2017) show GDPR creates only a limited 'right to be informed,' not a binding 'right to explanation' of specific decisions; and controlled work finds the explanations actually delivered do not measurably improve lay decision accuracy over showing the bare AI prediction (Alufaisan et al. 2021; and a 2022 meta-analysis by Schemmer et al. — screening 393 articles down to 9 in the final analysis — reports 'no effect of explanations on users' performance compared to sole AI predictions,' even though XAI overall had a positive effect). Honest caveat: the legitimacy/dignity value of being heard is empirically well established in the procedural-justice tradition even where outcome accuracy is unchanged, so 'redress fails' depends on which aim is measured.

    Sources: Wachter, Mittelstadt & Floridi 2017 (International Data Privacy Law 7(2):76); Alufaisan, Marusich, Bakdash, Zhou & Kantarcioglu 2021 (Proceedings of the AAAI Conference on AI 35(8):6618); Schemmer, Hemmer, Nitsche, Kühl & Vössing 2022 (AAAI/ACM AIES '22, meta-analysis)

  • Does governance work?evidence: absent

    There is no rigorous impact evaluation showing that mandated redress mechanisms (right-to-explanation, appeal, human-in-the-loop review) actually reduce erroneous or unfair automated decisions — the evidence that the rule works is itself missing. The closest experimental analogues are discouraging: explanations increase humans' acceptance of AI recommendations regardless of correctness (Bansal et al. 2021), and algorithm-in-the-loop oversight can introduce racial disparities and exhibit automation bias rather than reliably catching model errors (Green & Chen 2019). The procedural-justice literature (Tyler 1990; Lind & Tyler 1988) robustly supports a legitimacy and compliance benefit of fair process, but it measures perceived fairness, not reduction of the substantive decision harm redress is meant to cure.

    Sources: Bansal, Wu, Zhou, Fok, Nushi, Kamar, Ribeiro & Weld 2021 (CHI '21); Green & Chen 2019 (Disparate Interactions, ACM FAT* '19); Tyler 1990 (Why People Obey the Law, Yale Univ. Press); Lind & Tyler 1988 (The Social Psychology of Procedural Justice, Plenum Press)

• Transparency Obligations

  • Is the harm real?evidence: contested

    Documentation artifacts (model cards, datasheets) are well-specified as proposals and are genuinely adopted, but the empirical premise that mandated disclosure produces meaningful transparency is contested. Selbst & Barocas (2018) argue inscrutability and non-intuitiveness are distinct problems and that disclosing rules does not resolve the latter, and large-scale audits find documentation is sparsely and unevenly completed: a systematic analysis of 32,111 Hugging Face model cards (Liang et al. 2024) found environmental-impact, limitations and evaluation sections least often filled, and Bhat et al. (2023, 45 practitioners) found a substantial gap between the documentation proposal and actual practice. Honest caveat: the documentation frameworks themselves are real and adopted, so the dispute is about whether disclosure conveys decision-relevant information, not whether the artifacts exist.

    Sources: Selbst & Barocas 2018 (Fordham Law Review 87:1085-1139); Liang et al. 2024 (Nature Machine Intelligence, s42256-024-00857-z, 'Systematic analysis of 32,111 AI model cards'); Bhat et al. 2023 (CHI '23, 'Aspirations and Practice of ML Model Documentation', DOI 10.1145/3544548.3581518); Mitchell et al. 2019 (FAccT, Model Cards for Model Reporting); Gebru et al. 2021 (CACM 64(12):86-92, Datasheets for Datasets)

  • Does governance work?evidence: absent

    There is no rigorous impact evaluation showing that AI transparency mandates (model cards, training-data summaries) measurably reduce bias, misuse or accidents — the central regulatory assumption is empirically untested, partly because flagship mandates like EU AI Act Art. 53(1)(d) GPAI training-data summaries are only subject to AI Office enforcement/verification from 2 August 2026 (the obligation itself began 2 August 2025 for new models). The closest analogue, mandated consumer disclosure, shows small and context-dependent effects: Bollinger, Leslie & Sorensen (2011) found mandatory calorie posting cut average calories per transaction by about 6%, while Loewenstein, Sunstein & Golman (2014) review evidence that disclosure effects are frequently diminished or even reversed by limited attention and often change provider rather than recipient behavior. These are analogues, not AI studies; no study demonstrates that AI transparency disclosure achieves its stated downstream safety aims.

    Sources: Bollinger, Leslie & Sorensen 2011 (AEJ: Economic Policy 3(1):91-128); Loewenstein, Sunstein & Golman 2014 (Annual Review of Economics 6:391-419, 'Disclosure: Psychology Changes Everything'); EU AI Act Art. 53(1)(d) GPAI training-data summary (obligation from 2 Aug 2025; AI Office enforcement from 2 Aug 2026)

Related instruments

• EU AI Act · EU
• OECD AI Principles (Recommendation) · OECD
• Council of Europe Framework Convention on AI · council_of_europe
• Brazil AI Bill (PL 2338/2023) · BR
• General Data Protection Regulation (GDPR) · EU
• OMB Memorandum M-24-10 (Advancing Governance, Innovation, and Risk Management for Agency Use of AI) · US
• UNESCO Recommendation on the Ethics of Artificial Intelligence · UNESCO
• Directive (EU) 2024/2831 on improving working conditions in platform work · EU
• Provisions on the Administration of Deep Synthesis of Internet Information Services · CN