Why Generic LLMs Hallucinate Clinical Trial Content

A large language model does not know what it does not know. It produces text that is grammatically coherent and structurally plausible, and in a clinical trial context, that is precisely what makes it dangerous.

Regulatory professionals have begun using general-purpose AI tools to draft informed consent forms, summarize protocols, and generate sections of investigator brochures. The efficiency gains are real. But so is a problem that is now being documented with increasing precision across the medical and regulatory literature: generic LLMs, when applied to clinical trial content, fabricate information at rates that should alarm any team using them without rigorous human review.

This article examines why this happens, what the data show, and what distinguishes AI systems that are engineered for the specific demands of clinical research from those that are not.

The Measurement Problem: LLM Hallucination in Clinical Trial Documents Is Quantifiable, Not Just Theoretical

The instinct when discussing LLM hallucinations is to treat them as edge cases, the model occasionally saying something wrong. The published evidence does not support that framing.

A 2024 comparative study published in the Journal of Medical Internet Research evaluated GPT-3.5, GPT-4, and Bard (an earlier Gemini version) on their ability to retrieve real references for systematic medical reviews ^[1]. The hallucination rates, meaning citations that did not correspond to any real published paper, were 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard ^[1]. These are not occasional slips. Nearly three in ten references generated by the best-performing model in that study did not exist.

Separate work by Walters and Wilder (2023) ^[2], analyzing 636 citations across 84 AI-generated literature reviews, found that 55% of GPT-3.5 citations and 18% of GPT-4 citations were entirely fabricated. Among the references that did correspond to real papers, citation errors (wrong publication dates, incorrect page numbers, misidentified journals) were found in 43% of GPT-3.5 outputs and 24% of GPT-4 outputs ^[2].

A 2025 study in JMIR Mental Health ^[3], which prompted GPT-4o to generate six literature reviews across mental health topics of varying research depth, found that 19.9% of all 176 citations were entirely fabricated. Among the citations that referred to real publications, 45.4% contained bibliographic errors. Notably, 64% of the fabricated citations that included digital object identifiers pointed to real but completely unrelated papers ^[3]. These are not obviously wrong outputs. They are wrong in ways that require careful expert verification to detect.

For clinical trial sponsors, CROs, and medical writers, this is not an abstract quality concern. Regulatory submissions containing fabricated citations or incorrect statistical references create significant verification burdens, and in consent documents, inaccurate AI-generated content can expose participants to misinformed decision-making.

Why Clinical Trial Content Is Structurally Different from General Text

To understand why generic LLMs struggle specifically with clinical trial material, consider what that content actually requires.

A clinical trial protocol is not a narrative. It is an operational and regulatory control document. Every eligibility criterion must be precise enough to apply consistently at a screening visit across all participating sites. Every endpoint definition must map to a statistical analysis plan. Every dosing instruction must agree with the investigational medicinal product dossier. Cross-document consistency is not optional; it is a foundational expectation under ICH E6(R3) ^[4], adopted by ICH in January 2025 and issued as final FDA guidance in September 2025. E6(R3) formalizes risk-based quality management and quality-by-design principles under which the protocol serves as the primary operational reference for trial conduct, informing the consistency of downstream documentation across sites and functions.

An informed consent form adds another layer. ICFs must satisfy FDA 21 CFR Part 50 ^[5] requirements for the elements of informed consent, follow institution-specific IRB templates, reflect the exact risks and procedures described in the protocol, and be written in plain language accessible to lay participants. These requirements are not advisory. They are the legal basis on which human subjects research is permitted to proceed.

General-purpose LLMs are pre-trained predominantly on web text and publicly available corpora. While some regulatory and clinical documents appear in those corpora, they are not curated, annotated, or weighted to represent the structured regulatory logic governing clinical trial documentation. The model learns statistical associations between words, not the hierarchical document logic that governs what a clinical trial record must contain and why.

Documented Failures in Regulatory Document Generation

The performance data on LLMs applied to specific clinical trial documents make the scale of this problem concrete.

A study published in the Journal of the American Medical Informatics Association ^[6] introduced InformBench, a benchmark comprising 900 paired clinical trial protocol and ICF documents, and evaluated how well baseline LLMs performed on the task of generating an ICF from a protocol. GPT-4o, one of the most capable general-purpose models available, achieved only 70% to 80% compliance with the 18 core regulatory rules derived from FDA guidance, and exhibited factual errors in 18% to 43% of generated ICF sections ^[6]. A domain-optimized pipeline incorporating retrieval-augmented generation and mandatory human review outperformed the baseline model by up to 30 percentage points on compliance in those tasks ^[6].

A separate study in JMIR Medical Informatics (2025) ^[7] found that in one protocol, an LLM-generated ICF omitted common discomforts associated with a COVID-19 diagnostic test entirely, because that information was present in the clinical context but not in the specific protocol document the model was processing. The human-authored ICF included it. The LLM did not infer it, and it did not ask for clarification. It simply produced an incomplete consent document with no indication that anything was missing ^[7].

A preprint evaluating ChatGPT-4o, Gemini 2.5 Pro, and DeepSeek R1 on FDA regulatory review documents and clinical trial data ^[8] found that FDA review files, which contain nuanced methodological detail, cross-referenced analyses, and interpretive regulatory commentary, are structurally more complex than the drug labels that prior research had used as a proxy. No LLM tested in that preprint could reliably extract, synthesize, and summarize that content without errors, even when given direct access to the relevant documents.

A study in the Journal of Medical Internet Research examining LLMs in RCT design ^[9] found that while models performed reasonably on certain high-level design tasks, their accuracy on eligibility criteria design was only 55% and on outcomes measurement design only 53%. In both domains, getting it wrong carries direct consequences for patient safety and data integrity.

The Mechanism: How Hallucinations Actually Occur in Clinical Contexts

Understanding why these failures happen, not just that they happen, is necessary for evaluating AI tools in a regulatory environment.

General-purpose LLMs generate output by predicting the most statistically probable continuation of a given input, given what the model learned during training. When the model encounters a query that falls outside its training distribution (clinical trial eligibility criteria, SAP language, DSUR content requirements), it does not flag uncertainty. It generates text that resembles the expected structure of such content, drawing on whatever statistical associations exist in the training corpus ^[10]. The result is prose that looks like a protocol amendment eligibility criterion, sounds like an ICH-compliant statement, and cites a plausible-seeming journal, but may be partially or entirely wrong.

Research published in Frontiers in Artificial Intelligence in 2025 ^[10] identified two distinct sources of LLM hallucination. The first is prompting-induced, arising when vague or under-specified inputs push the model toward generative interpolation. The second is model-internal, arising from the architecture's pretraining data distribution: when the model lacks sufficient exposure to a specific domain, it fills gaps with probabilistic guesses rather than grounded facts. Clinical trial content suffers from both. Regulatory and operational workflows require structurally precise answers; models lacking deep regulatory training fill in the gaps.

A study published in Communications Medicine in August 2025 ^[11] tested how LLMs respond to prompts containing a single fabricated clinical detail, such as a false laboratory value or a non-existent condition. Models hallucinated by elaborating on or confirming the false detail in 50% to 82.7% of cases across multiple tested models ^[11]. For GPT-4o specifically, hallucination rates under adversarial conditions declined from 53% to 23% with a specialized mitigation prompt, but never reached zero ^[11]. In a clinical documentation workflow, any fabricated clinical detail that enters a protocol, ICF, or regulatory filing has the potential to propagate downstream to every dependent document.

The inadequate training data coverage problem is particularly acute for rare therapeutic areas and specialized regulatory frameworks. The JMIR Mental Health citation study ^[3] found that LLM fabrication rates rose significantly for topics with lower public and literature visibility, which may imply thinner representation in training data. ICH E3 guidelines for clinical study reports, DSUR content requirements under ICH E2F, or Phase I dose-escalation schema for novel cell and gene therapies are not heavily represented in general web text. Models asked to generate content in these areas are operating at the edge of their knowledge, where hallucination risk is highest.

What "Confident but Wrong" Costs in Clinical Research

The specific danger in a clinical research context is not just that errors occur. It is that LLM errors are presented with the same fluency and confidence as accurate content.

A human expert who does not know a specific ICH E6(R3) requirement will say so, or will hedge. A generic LLM asked the same question will produce a well-formatted, appropriately hedged paragraph that may cite an ICH document, use the right terminology, and still be factually wrong on a specific requirement. The output reads as authoritative. Detecting the error requires the kind of deep regulatory expertise that the model was supposed to be assisting.

This produces a verification paradox. The people most likely to rely on AI assistance for regulatory documents are those for whom expert review capacity is limited: early-phase sponsors without large regulatory teams, CROs managing high document volumes, or sites producing locally adapted ICFs for multinational studies. These are also the contexts where comprehensive expert review of AI output is hardest to guarantee.

A medRxiv preprint on medical hallucination in foundation models (2025) ^[12] found that in non-clinical contexts, LLM errors are more likely to be caught because users typically possess the background knowledge needed to identify inaccuracies. In clinical and regulatory contexts, that assumption breaks down. Subtle, plausible misinformation in a consent form, an eligibility criterion, or a safety reporting timeline can influence decisions and conduct without being immediately detectable.

Regulatory and Documentation Considerations

FDA guidance does not yet provide a comprehensive framework specifically governing LLM use in clinical trial document generation. However, existing requirements establish the baseline against which any AI-generated output must be measured.

Under 21 CFR Part 50 ^[5], informed consent must contain specific required elements and must accurately reflect the study as it will be conducted. Under 21 CFR Part 56 ^[5b], that consent document must be reviewed and approved by an IRB before use. The regulatory obligation falls on the sponsor and investigator, not on the tool. An LLM generating an ICF that omits a required risk disclosure or mischaracterizes a procedure does not create regulatory liability for the model; it creates it for the team that used the output without adequate verification.

FDA's 2021 action plan for AI and machine learning in medical devices ^[13] outlined the agency's intended regulatory approach for AI/ML-based software as a medical device, including principles of transparency, good machine learning practice, real-world performance monitoring, and future work on predetermined change control plans. That plan was developed specifically for regulated medical devices and does not directly govern AI-assisted clinical document drafting. It does, however, signal FDA's broader regulatory posture: AI tools used in clinical contexts are expected to demonstrate characterized, validated performance, not just plausible output.

ICH E6(R3) ^[4], finalized in January 2025 and adopted as FDA guidance in September 2025, establishes quality-by-design and risk-proportionate oversight expectations. Sponsors are required to identify and mitigate risks to trial data integrity from the earliest stages of trial planning. Deploying an unvalidated, general-purpose LLM for regulatory document generation without adequate human oversight is difficult to square with the quality management framework E6(R3) establishes, even where a specific LLM prohibition does not yet exist.

The EMA's September 2024 reflection paper on the use of AI in the medicines lifecycle ^[14] acknowledges AI's potential while emphasizing that regulatory submissions must maintain traceability of information and human accountability for conclusions. Any AI tool supporting document generation must be positioned as a drafting aid subject to expert review, not as an autonomous generator of regulatory content.

Traceability is worth emphasizing on its own terms. A clinical trial document that an AI system generates without source-linked citations provides reviewers and auditors with no path back to the originating data. When a discrepancy surfaces, whether at an IRB review, a site inspection, or a regulatory query, the inability to trace a specific claim back to a protocol section, a published study, or a guidance document creates a documentation gap that well-controlled clinical document workflows are expected to preserve. Well-designed AI systems built for clinical research document generation can address this by providing inline source references linked to specific protocol sections or regulatory guidance clauses, enabling the human reviewer to verify, not just read, the output.

AI and Automation in Clinical Research: The Architecture Gap

The performance gap between general-purpose LLMs and purpose-built clinical research AI systems is not primarily about model size or raw capability. It is about architecture.

General-purpose models generate output based on their pretraining knowledge, which is static, domain-general, and not grounded in the specific protocol, study design, or regulatory context at hand. When asked to draft an ICF from a protocol, they do not retrieve the protocol's actual content through a controlled, traceable process. They generate text that resembles an ICF, drawing on whatever ICF-like patterns exist in their training data.

Domain-specific systems engineered for clinical trial document generation use fundamentally different architectures. Retrieval-augmented generation (RAG) pipelines ground every output in the actual source documents provided, enabling traceability of claims back to specific protocol sections, ICH guidance clauses, or FDA-approved labels. The InformGen system ^[6], for example, achieved near-100% regulatory compliance on 18 core FDA ICF rules using a RAG-based architecture with human-in-the-loop review, compared to 70% to 80% compliance from a standard GPT-4o baseline on those same tasks.

A 2026 scoping review of fine-tuning and RAG integration in healthcare AI systems ^[15] found that combining domain-specific adaptation with retrieval-augmented pipelines improved factual consistency, answer accuracy, and robustness to unseen clinical content compared to general-purpose LLMs alone, particularly for knowledge-intensive tasks such as medical question answering and clinical summarization. The same architectural logic applies in regulatory document generation: pipelines adapted to the structure, vocabulary, and compliance requirements of ICH guidelines and clinical trial protocols perform differently from those that have not been built with that domain in mind. Sponsors and CROs evaluating AI tools for document generation should ask not just whether a model can draft a convincing-looking paragraph, but whether its output is traceable, auditable, and grounded in the specific study it is supposed to describe. (See how KScribe approaches this for regulatory document generation in clinical trials.)

How Kitsa Fits Into This Problem

KScribe, Kitsa's AI-powered regulatory document generation product, is built around the principle that every clinical trial document must be grounded in the specific study it describes. Every output traces back to a source: the protocol, the investigational brochure, or the applicable regulatory guidance. That traceability is what makes it possible for a medical writer or regulatory affairs professional to review, audit, and approve AI-generated content with confidence rather than assumption. KScribe is not designed to replace expert review; it is designed to make expert review faster, more complete, and less reliant on manual extraction from documents that can span hundreds of pages. More at kitsa.ai/regulatory-document-generation.

Key Takeaways

• •Fabricated citations in general-purpose LLM outputs have been measured at 18% to 91% across multiple published studies, depending on the model and task; even GPT-4-level models produce fabricated medical references at rates that are not acceptable for clinical or regulatory use without verification.
• •GPT-4o achieved only 70% to 80% regulatory compliance and 18% to 43% factual error rates when generating informed consent forms from clinical trial protocols in the InformBench benchmark.
• •Clinical trial documents have a structural precision requirement that general web-trained LLMs are not built to satisfy: every eligibility criterion, endpoint definition, and consent disclosure must trace back to a specific verified source.
• •LLMs hallucinate more frequently in domains with thin training coverage; rare therapeutic areas, specialized regulatory frameworks, and newer guidance documents represent elevated-risk territory.
• •Adversarial prompting research published in Communications Medicine shows that LLMs elaborate on fabricated clinical details in 50% to 82.7% of cases, meaning plausible but incorrect inputs to a clinical workflow can propagate into documented outputs.
• •In the InformBench ICF benchmark, RAG-based architecture with human-in-the-loop review outperformed vanilla GPT-4o by up to 30 percentage points on regulatory compliance; broader healthcare AI evidence supports the pattern, though clinical-trial-specific benchmarks remain limited.
• •Regulatory responsibility for AI-generated clinical content rests with the sponsor and investigator, not with the tool; source traceability, audit trails, and expert sign-off are non-optional, not optional add-ons.

FAQ

References

Related Articles