Why Generic LLMs Fail at Regulatory Document Generation

A 57% Accuracy Floor Is Not a Regulatory Document Strategy

When researchers evaluated vanilla GPT-4o on informed consent form drafting across 900 clinical trials, the model achieved factual accuracy ranging from 57% to 82%, depending on study complexity ^[1]. In a document governed by FDA 21 CFR Part 50.25, where the sponsor carries legal accountability for ensuring that a participant genuinely understood the procedures, risks, and their rights before enrolling ^[14], a potential 43% error rate is a clinical and regulatory liability, not an acceptable margin.

The same class of risk extends to other regulated document types. Research evaluating LLM performance on FDA regulatory review documents found meaningful limitations across all three models tested ^[8], and hallucination rates in eligibility criteria conversion reached 32.7% ^[7], suggesting the failure modes documented in ICF generation are not format-specific. Protocols, Investigator's Brochures, Development Safety Update Reports, and Clinical Study Reports can carry similar structural vulnerabilities: outputs that read authoritatively, flow fluently, and contain errors nearly invisible without deep protocol knowledge.

This article draws on peer-reviewed evidence, regulatory guidances, and benchmark data to explain why the failure mode is structural, and why prompt engineering alone cannot address it.

Why Regulatory Documents Are Different From Any Other Writing Task

A protocol is not a long technical brief. An ICF is not a patient information leaflet. These documents exist within an interlocking system of regulatory obligations: what content must appear, what language is permissible, how terms must remain consistent across an entire submission dossier, and what evidence trail must exist when an inspector asks how the document was produced.

That compliance perimeter expanded substantially when ICH E6(R3) was finalized on January 6, 2025. The guidance introduced a comprehensive data governance framework explicitly addressing data integrity, traceability, and computer system accountability across the trial lifecycle ^[5]. ICH E6(R3) does not create a separate carve-out for AI-generated content. The guidance's principles are explicitly intended to remain relevant as technology evolves, and tools used to create, modify, or maintain GCP-regulated records are subject to the same data integrity and computer system oversight requirements as any other electronic system in the trial environment. Sponsors deploying general-purpose models for regulated document generation without corresponding computer system validation documentation may carry a compliance gap into submission workflows.

Protocol complexity has been escalating in parallel. A Tufts Center for the Study of Drug Development analysis published in 2024, drawing on data from 950 protocols and 2,188 amendments across 16 pharmaceutical companies and CROs, found that the prevalence of protocol amendments across Phase I through Phase IV trials had risen from 57% in 2015 to 76%, with the mean number of amendments per protocol climbing 60% to 3.3 ^[3]. The financial consequences are well-documented: a separate Tufts CSDD analysis found the median direct cost of implementing a substantial amendment ranges from $141,000 for a Phase II protocol to $535,000 for a Phase III protocol, covering rewrites, IRB resubmissions, patient re-consent, and site retraining ^[4]. In that environment, a document generation tool that introduces errors or internal inconsistencies is not merely inconvenient. It adds directly to that bill.

What the Research Says About Generic LLMs in Regulatory Contexts

The evidence against deploying unmodified general-purpose models for regulated clinical documents has accumulated steadily since 2022.

A 2025 study published in npj Digital Medicine evaluated 18 experimental configurations of LLMs for clinical note generation across 12,999 clinician-annotated sentences and found a 1.47% hallucination rate and a 3.45% omission rate, with 44% of those hallucinations rated as clinically "major" in significance ^[2]. That study examined clinical notes rather than regulatory submissions, but it provides a relevant baseline for understanding the error distribution in AI-generated clinical text at scale. Published research examining a sample of industry-sponsored clinical study reports found a median document length of 644 pages ^[18]; at that scale, even a low per-sentence error rate translates to multiple substantive inaccuracies within a single submission.

For the specific task of handling clinical eligibility criteria, which form the backbone of both the protocol and the safety population described in the CSR, the performance gap is wider. A 2025 study evaluating LLMs on the conversion of trial eligibility criteria to structured queries found an overall hallucination rate of 32.7%, with wrong domain assignments in 34.2% of outputs. For certain diagnostic categories, including pregnancy status, the model failed completely, producing a Jaccard similarity of 0.00 against the reference ^[7].

The InformGen benchmark, evaluated across ICFs from 900 clinical trial protocols, quantified the gap directly. A vanilla GPT-4o model produced 57% to 82% factual accuracy, while a purpose-built system with structured compliance logic, combined with human review, reached above 90%. The domain-specific system also achieved near-100% compliance with 18 core FDA regulatory rules derived from 21 CFR Part 50 guidelines, outperforming the general model by up to 30 percentage points ^[1]. A 2025 preprint evaluating ChatGPT-4o, Gemini 2.5 Pro, and DeepSeek R1 on FDA regulatory review documents found each demonstrated meaningful limitations in extracting nuanced methodological details from submissions containing cross-referenced analyses ^[8].

The Cross-Document Consistency Problem

The failure mode that receives the least discussion in AI-and-medical-writing conversations is also the one with the most direct regulatory consequence: cross-document consistency.

Under ICH M4E(R2) and the Common Technical Document format adopted by FDA, EMA, and PMDA, regulatory submissions require alignment across Module 2.5 (Clinical Overview), Module 2.7 (Clinical Summaries), and the Clinical Study Reports in Module 5 ^[13]. The patient population defined in the protocol must appear without deviation in the SAP, the ICF, and the CSR. The primary endpoint specified in the protocol must map to the estimand framework defined under ICH E9(R1) ^[12], which must in turn underpin the statistical methodology documented in the SAP. Cross-document discrepancies in these elements are among the items regulatory reviewers examine when assessing submission quality, and ICH M4E(R2)'s emphasis on alignment across modules reflects the expectation that sponsors will maintain internal consistency throughout the dossier ^[13].

General-purpose LLMs, when deployed through standard commercial interfaces, generate each document session without persistent state. There is no mechanism within such a system to cross-check the endpoint terminology used when drafting the protocol six weeks ago against the SAP being drafted today, or to detect that the exclusion criterion in Module 2.5 has diverged from the text in Module 5. A sponsor generating documents piecemeal through a commercial chat interface can produce a submission dossier with internally inconsistent definitions, without any automated signal that the inconsistency exists.

Regulatory Compliance Logic Is Not General Knowledge

FDA 21 CFR Part 50 requires that informed consent language be understandable to the participant or their legally authorized representative ^[14]. Many IRBs, institutions, and plain-language standards use a sixth to eighth grade reading level as a practical benchmark for meeting that requirement. A 2024 peer-reviewed analysis of 5,239 ICFs drawn from ClinicalTrials.gov found that 91% were written above the 8th-grade level those IRB standards target, with an average Flesch-Kincaid Grade Level of 10.99 ^[15]. The gap between the readability target and typical ICF output is not accidental. It reflects the technical density of source protocols. A general-purpose LLM prompted to draft an ICF from a complex oncology protocol defaults to the register of its training data, which skews toward academic and technical text. Prompt instructions alone cannot reliably enforce a reading level constraint across a 15-page section covering complex pharmacokinetic monitoring procedures.

The same gap appears in the statistical domain. ICH E9(R1), adopted November 20, 2019, introduced an estimands framework requiring sponsors to define not just the primary outcome, but how intercurrent events will be handled, which of five estimand strategies applies, and how the sensitivity analysis will test whether the main estimand's conclusions hold under different analytical assumptions ^[12]. A general-purpose LLM asked to draft an SAP has no reliable internal representation of estimand construction logic. It produces a document that looks structurally like an SAP without the protocol-specific estimand alignment that FDA and EMA reviewers expect to find.

The Audit Trail Gap Under 21 CFR Part 11

For electronic records that fall under predicate rules and are created, modified, or transmitted in a GxP context, FDA 21 CFR Part 11 requires secure, computer-generated, time-stamped audit trails documenting who performed each action, when, and what changed ^[6]. It is worth clarifying that not every use of an AI drafting tool triggers Part 11; the compliance risk escalates when outputs are incorporated into controlled GxP workflows or submitted as part of a regulated record. At that point, the absence of a creation history becomes a gap that computer system validation documentation would ordinarily fill. Computer systems used to create such records are expected to be validated to demonstrate they perform as intended before use in regulated activities. Data integrity has been a consistent FDA enforcement priority: an industry analysis of FDA drug GMP warning letters found approximately 80% in 2015 and 2016 included a data integrity component, reflecting the agency's long-standing focus on electronic records controls ^[17].

General-purpose LLMs deployed through commercial chat interfaces do not generate regulated audit trails suitable for GxP or Part 11 workflows. They do not record who submitted a prompt, which protocol version was active at the time of generation, what earlier draft was modified, or why a given section was revised. A regulated document produced through an unvalidated commercial interface may lack a verifiable origin within the sponsor's validated documentation environment.

Long-Document Performance Breaks Down

Clinical trial protocols have grown substantially longer as complexity has increased; a Tufts CSDD analysis documented a 139% increase in procedure quantity and 214% rise in endpoints since 2005 ^[3]. CSRs reach 644 pages or more at the median ^[18]. Research evaluating multiple models, including Claude 3, GPT-3.5 Turbo, and Gemini Pro, has found that model accuracy on analysis tasks degrades as input length grows, with information located in the middle sections of long contexts receiving less reliable attention than content near the beginning or end ^[16]. This pattern is sometimes called "Lost in the Middle."

For regulatory documents, the practical consequence is specific. A general-purpose LLM generating a CSR synopsis from a 200-page protocol and study results package may correctly reflect the primary endpoint stated in the opening pages, and miss the subgroup amendment defined on page 147, or the safety assessment added in the second protocol amendment. Prompting alone does not reliably address this risk, because the limitation lies in the model's capacity to maintain coherent attention across very long inputs, not in how the prompt is phrased.

Regulatory and Documentation Considerations

Three regulatory frameworks define the environment in which these failure modes become concrete compliance risks.

ICH E6(R3), finalized January 6, 2025, introduced a standalone data governance section covering data integrity, traceability, security, and computer system accountability for all aspects of trial conduct ^[5]. The guidance makes explicit that its GCP principles are intended to remain relevant as technology evolves, and that sponsors bear oversight responsibility for any electronic systems involved in GCP-regulated activities. A clinical document generated by an AI tool is subject to the same authenticity and traceability standards as one drafted manually, and the tool generating it carries the same computer system oversight obligations as any other system in the sponsor's GxP environment.

ICH E3, which governs the structure and content of Clinical Study Reports, requires that CSRs provide a clear, complete, transparent, and reproducible account of the study ^[11]. "Reproducible" in this context means that a reviewer must be able to trace every analytical claim in the report to the underlying protocol and data. A CSR section generated by a general-purpose model that interpolates across source material, rather than grounding claims in the protocol, SAP, and study data, may fail this standard without the failure being visible on a surface read.

ICH E9(R1), adopted November 2019, introduced the estimands framework to align trial objectives, design, analysis, and reporting in a single coherent structure ^[12]. Every element of an SAP, from the primary estimand through sensitivity analysis strategy, must trace back to the corresponding protocol section. That alignment requires a document generation process that treats the SAP and the protocol as parts of the same evidentiary chain, not two independent writing tasks.

AI and Automation in Regulatory Writing: What Evidence Supports

The research on domain-specific models points in a consistent direction. A 2025 peer-reviewed analysis in Frontiers in Artificial Intelligence found that adapting a general-purpose LLM with medical domain knowledge increases performance "dramatically" compared to the unadapted base model, and that inadequate domain-specific constraints can lead to erroneous outputs in clinical settings ^[9]. Published commentary in the biomedical AI space has further noted that high-quality domain training data tends to matter more than raw model parameter count for clinical reliability, with smaller fine-tuned models outperforming larger general-purpose alternatives on specialized benchmarks ^[10].

For regulatory documents, domain accuracy is only part of what the task requires. Beyond getting clinical facts right, a purpose-built system must also maintain cross-document consistency by tracking terminology, population definitions, endpoints, and estimand alignment across the full document set; enforce regulatory compliance logic against specific guidances including ICH E6(R3), ICH E3, ICH E9(R1), and FDA 21 CFR Part 50; operate within an environment with audit trail functionality; and surface discrepancies rather than generating plausible-sounding text that obscures them.

The table below summarizes where the structural gaps between general-purpose and purpose-built systems appear in practice.

These differences are architecture requirements, not prompt quality issues. Prompt engineering improves output fluency; it does not by itself add persistent state, create regulated audit trails, or encode compliance logic against specific regulatory guidances. The structural gap between general-purpose and purpose-built systems is not addressable through better prompting alone.

How Kitsa Fits Into This Problem

KScribe, Kitsa's regulatory document generation platform (kitsa.ai/regulatory-document-generation), addresses the document types where these failure modes matter most: protocols, ICFs, Investigator's Brochures, Development Safety Update Reports, and Clinical Study Reports. The platform is designed to track cross-document consistency across a related document set and maintain document lineage for audit purposes.

Human review remains necessary in any AI-assisted regulatory writing workflow, including purpose-built systems, and medical writers retain professional accountability for the accuracy and completeness of every regulated document they submit. The difference a purpose-built regulatory platform makes is in reducing the structural risks that generic tools introduce before human review even begins. For sponsors and CROs evaluating AI tools for regulatory medical writing, the distinction between a general-purpose commercial model and a system built to the regulatory context is not a product preference. It is the difference between a drafting tool and an auditable clinical infrastructure.

A Practical Illustration: Where Inconsistency Enters a CTD Dossier

The following scenario is hypothetical but reflects the structural risks that arise from piecemeal document generation across separate LLM sessions.

Consider a Phase III oncology trial where a sponsor uses a general-purpose LLM to draft, in separate sessions over several months, the protocol, ICF, SAP, and CSR synopsis.

The protocol defines the primary analysis population as patients who received at least one dose of study drug and had a baseline tumor assessment. Two months later, when drafting the SAP, the same general-purpose LLM is prompted with a summary of the study design. It renders a slightly different population definition, adding a qualification about measurable disease at baseline, because that language appeared frequently in its training data for oncology SAPs. The CSR synopsis, drafted six months after the protocol, reflects the SAP's wording, not the protocol's.

The ICF, drafted separately, describes eligibility requirements in plain language drawn from the eligibility criteria section of the protocol, but omits one of the post-amendment exclusion criteria added in Protocol Amendment 2, which was incorporated into the SAP but not explicitly provided to the LLM when drafting the ICF.

By submission, four documents have been produced. Each reads well in isolation. Together, they contain at least two cross-document discrepancies. A system that tracked source documents across the full document set would be designed to flag both.

What to Ask Any AI Vendor Before Deploying for Regulatory Documents

Sponsors and CROs evaluating tools for regulated document generation, including purpose-built platforms like KScribe, should request clear answers to the following before deployment.

Does the system generate a time-stamped audit trail identifying who initiated each document generation, which source version was active, and what edits were made post-generation? This is the type of electronic record traceability that FDA 21 CFR Part 11 and ICH E6(R3) expect from computer systems operating in GxP environments ^[5],[6].

Is the system validated as a GxP computer system, and is validation documentation available for review? This includes IQ/OQ/PQ reports and SOPs for document workflows.

Does the platform track cross-document consistency? Specifically, if a protocol section is amended, does the system identify downstream documents (ICF, SAP, CSR) that may require updating?

Is the system grounded in specific regulatory guidances, including ICH E6(R3), ICH E3, ICH E9(R1), and FDA 21 CFR Part 50, with compliance logic built into generation rather than relying on prompts?

What is the human review workflow? Purpose-built systems should not position AI output as submission-ready; human expert review of every regulated document remains a requirement, not an option.

Key Takeaways

• •A vanilla GPT-4o model achieved 57% to 82% factual accuracy on ICF drafting across 900 trials; a purpose-built system with human review reached above 90%, outperforming the general model by up to 30 percentage points on FDA regulatory compliance ^[1].
• •LLMs evaluated on clinical eligibility criteria conversion produced a 32.7% overall hallucination rate, including complete failure on certain diagnostic categories ^[7].
• •76% of Phase I through Phase IV protocols now require at least one amendment, with Phase III amendment costs reaching a median of $535,000 per substantial amendment, and mean amendments per protocol up 60% since 2015 ^[3],[4].
• •ICH E6(R3), finalized January 2025, extends data governance, traceability, and computer system oversight requirements to electronic tools used in trial conduct, including AI document generation ^[5].
• •General-purpose LLMs deployed through commercial interfaces generate each document session without persistent state, with no mechanism to enforce the cross-document consistency required across a CTD submission ^[13].
• •For electronic records within scope of predicate rules, FDA 21 CFR Part 11 requires validated systems with time-stamped audit trails; commercial LLM interfaces carry none of the validation documentation those requirements imply ^[6].
• •Domain-specific training and high-quality clinical data can substantially improve LLM performance on medical and regulatory tasks relative to unadapted general-purpose models ^[9],[10].
• •Human review remains necessary in any AI-assisted regulatory writing workflow, including purpose-built systems; the value of a purpose-built regulatory platform is in reducing the structural risks that generic tools introduce before human review begins.

FAQ

References

Related Articles