FDA, EMA, and ICH Expectations for AI Regulatory Systems

Introduction

In September 2024, the EMA finalized its Reflection Paper on AI in the medicinal product lifecycle, establishing expectations for transparency, reproducibility, and human oversight across the full drug development process ^[2]. Four months later, in January 2025, the FDA published its first draft guidance specifically addressing artificial intelligence in drug and biological product regulation: a 7-step credibility assessment framework that applies to any AI model generating data intended to inform regulatory decisions about safety, effectiveness, or quality ^[1]. Then, on January 14, 2026, the two agencies jointly published ten Guiding Principles of Good AI Practice in Drug Development ^[3], the first time FDA and EMA have formally converged on a shared regulatory vision for AI across the full medicines lifecycle.

These three documents, read alongside ICH E6(R3) (finalized January 2025 and adopted by the FDA in September 2025) ^[4] and the finalized ICH M11 CeSHarP protocol standard (published in the Federal Register May 2026) ^[5], represent one of the most significant recent convergences of AI regulatory expectations produced by major health authorities. They span a range of legal instruments: FDA's drug-development guidance is a non-binding draft; EMA's Reflection Paper is a non-binding position; the joint principles are explicitly voluntary; ICH E6(R3) is a formal guidance standard adopted by regulators; and the EU AI Act is binding law. The practical force of the non-binding frameworks is substantial nonetheless, since they help define what regulatory reviewers and inspectors are likely to evaluate or reference when AI is used in a submission or regulated trial. For sponsors, CROs, and medical writing teams deploying AI to generate or support regulatory documentation, these frameworks now define the baseline that reviewers and inspectors are likely to reference when AI is disclosed in submissions or identified during inspections.

This article breaks down the operative expectations from each framework, explains where they converge and where they differ, and draws out what the combined regulatory environment means for any organization running AI in clinical or regulatory operations today. Status of all cited instruments is current as of June 2026.

Why the Regulatory Community Is Paying Attention Now

The regulatory frameworks arrived because the underlying activity reached a scale that demanded them. FDA has received more than 500 drug and biological product submissions containing AI components since 2016, with a measurable increase year-over-year as AI integration accelerated across oncology, neurology, and pharmacovigilance applications ^[1],[19]. As of December 2024, FDA had authorized more than 1,000 AI-based medical devices, a figure that reflects years of incremental AI governance work that CDRH began well before CDER's January 2025 drug-development guidance arrived ^[6].

On the European side, the EU AI Act (Regulation 2024/1689) entered into force in August 2024 ^[7]. Its requirements apply on a staggered timeline: prohibitions and AI literacy duties began applying in February 2025, governance obligations for general-purpose AI models took effect in August 2025. Under the original Act, Annex III standalone high-risk systems were due to comply from August 2, 2026, and Article 6(1) systems embedded in Annex I-regulated products from August 2, 2027. Under the Digital Omnibus provisional agreement of May 7, 2026, which is expected to receive formal adoption before August 2, 2026, those deadlines would be extended by different amounts: the Annex III standalone high-risk deadline would move by approximately 16 months to December 2, 2027, and the Article 6(1) product-integrated deadline would move by 12 months to August 2, 2028 ^[16].

Not all pharmaceutical or clinical AI is automatically high-risk under the Act. Classification follows two distinct routes in Article 6 ^[7]. The first covers AI that is a safety component of, or itself a product under, Union harmonization legislation listed in Annex I, most relevantly the EU Medical Device Regulation (MDR 2017/745); AI clinical decision-support tools that meet the MDR definition of a medical device fall here. The second covers AI whose specific intended use falls within one of the Annex III categories; the health-related Annex III entries are narrow, covering biometric categorization using sensitive attributes, AI used by public authorities to determine eligibility for public healthcare benefits, AI for risk assessment and pricing in life and health insurance, and patient triage systems ^[7]. Clinical trial management AI, pharmacovigilance analytics, and regulatory document generation do not appear in Annex III and are not automatically classified as high-risk: that classification requires analysis of intended use and whether the system materially influences decisions affecting individuals' health, safety, or fundamental rights ^[7].

The EU AI Act is sector-agnostic; it does not address the pharmaceutical-specific quality standards that govern how clinical evidence is generated, reviewed, or submitted. The EMA Reflection Paper and the joint FDA-EMA principles fill that gap, translating the Act's broad requirements into the vocabulary of GxP, GCP, and marketing authorization ^[2],[3].

The combined effect is this: AI systems that operate anywhere in the regulatory evidence chain are now subject to overlapping, mutually reinforcing expectations from four sources simultaneously: FDA guidance, EMA guidance, ICH standards, and EU law. Organizations that treat these as separate compliance checklists will find themselves managing contradictions. Organizations that understand how the frameworks interlock will find that meeting one set of expectations goes a long way toward meeting the others.

The FDA Credibility Assessment Framework

FDA's January 2025 draft guidance, "Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products" (Docket FDA-2024-D-4689), is the agency's first comprehensive guidance for AI in drug and biologics regulation ^[1]. It applies to any AI model whose output is intended to inform regulatory decisions about a product's safety, effectiveness, or quality, covering the full drug lifecycle from nonclinical through post-marketing and manufacturing phases, but explicitly excluding AI used solely in drug discovery or internal operational efficiency ^[1].

Context of Use

The guidance centers on a concept called the context of use (COU): the specific role and scope of an AI model for addressing a defined question of interest. Before any other assessment can proceed, sponsors must define the COU precisely: what question the model addresses, what inputs it processes, and what outputs it produces for regulatory purposes ^[1]. This requirement is deceptively demanding. Generic system descriptions will not satisfy it; COU documentation must be granular enough that a reviewer can assess whether the model's design is actually fit for the claimed purpose.

Seven-Step Credibility Assessment

FDA proposes a structured seven-step process for establishing and documenting AI model credibility ^[1]:

1. Define the question of interest
2. Define the context of use
3. Assess the AI model risk
4. Develop a credibility assessment plan proportionate to that risk
5. Execute the plan
6. Document results and deviations
7. Determine the adequacy of the model for the COU

Risk, under this framework, is not simply clinical severity; it is a function of two dimensions: the influence the model has on the regulatory decision, and the consequence of a decision error ^[1]. A model that categorizes patients for outpatient versus inpatient monitoring occupies a different risk tier than a model that flags data anomalies for human review. The guidance provides a worked example of the former to illustrate why the stakes of miscategorization demand proportionally rigorous credibility assessment ^[1].

Lifecycle and ALCOA+

FDA's guidance describes lifecycle evaluation as important in certain contexts of use, particularly where AI models are applied to data distributions that may change over time: not a single validation event, but ongoing monitoring for model drift, distribution shift, and performance degradation relative to the established COU ^[1]. Data pipelines feeding AI must meet the data integrity standards applicable to the regulatory context. In GMP environments, FDA CGMP guidance articulates these through ALCOA+ principles: Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, and Available ^[17]. FDA's AI guidance similarly requires that data used for model development and evaluation be accurate, complete, and of sufficient quality to support the intended COU ^[1]. Under 21 CFR Part 11, AI systems that create, modify, maintain, archive, or transmit electronic records required under an FDA predicate rule are subject to Part 11 obligations, which FDA interprets narrowly and applies with enforcement discretion in certain areas, focusing compliance effort on systems that directly affect product quality, safety, or the integrity of regulatory submissions ^[8]. For systems within Part 11's scope, validation, tamper-resistant audit trails, and access controls are required in proportion to the system's role in the regulated record-keeping process ^[8].

FDA explicitly encourages sponsors to engage early through formal meeting requests or other available channels to set expectations for credibility assessment activities before proceeding, since the appropriate approach depends on the specific COU and risk profile ^[1].

EMA's Reflection Paper: Fit-for-Purpose Evidence, Not Just Compliance

EMA's Reflection Paper on the use of AI in the medicinal product lifecycle (EMA/CHMP/CVMP/83833/2023, final September 2024) takes a different but complementary angle ^[2]. Where FDA's guidance is structured around a formal framework with defined steps, EMA's paper articulates governing principles and places responsibility squarely on the sponsor or marketing authorization applicant to demonstrate that AI systems are fit for purpose across the range of legal, ethical, technical, scientific, and regulatory standards that apply.

The paper's core accountability statement is unequivocal: it is the responsibility of the sponsor, MAH, or manufacturer to ensure that all algorithms, models, datasets, and data processing pipelines used are fit for purpose and comply with EU legislation, GxP standards, and current EMA guidelines ^[2]. The paper does not provide safe harbor for systems that merely meet minimum technical specifications; it demands that each AI application be evaluated against the specific scientific and regulatory context in which it operates.

Transparency and Explainability

EMA acknowledges directly that full explainability is not always achievable: some model architectures are inherently opaque, and black-box outputs cannot always be traced to interpretable causal paths ^[2]. The paper does not disqualify such models outright. Instead, it introduces a practical threshold: where full explainability is not possible, sponsors must demonstrate interpretability, meaning substantive evidence that human oversight is in place and that the model's performance can be monitored, interrogated, and acted upon when it fails or underperforms ^[2],[9].

This is a meaningful distinction. Explainability is about understanding why a model makes a specific prediction. Interpretability, as EMA uses it, is about whether the system is governed well enough that a human can identify problems and intervene. A model does not need to be a glass box to meet EMA's expectations, but it must be embedded in a governance structure that prevents unchecked reliance on its outputs.

What EMA Requires in Scientific Advice

For any request for scientific advice or opinion involving an AI-supported application, EMA expects sponsors to provide the full model architecture, logs from model development, documentation of training and validation datasets, and evidence of the model's generalizability to the specific target population and regulatory context ^[2]. This is substantively more than a description of what the model does; it is a technical dossier covering how the model was built and why it performs as claimed.

EMA's emphasis on dataset integrity is particularly salient for regulatory document generation. A model trained on data that does not reflect the population, endpoints, or regulatory standards relevant to a specific application cannot demonstrate generalizability; and EMA reviewers will ask about this ^[2].

The Joint FDA-EMA Principles: Ten Points of Convergence

On January 14, 2026, FDA's CDER and CBER, together with EMA, published ten Guiding Principles of Good AI Practice in Drug Development ^[3]. The principles are explicitly non-prescriptive; they do not themselves create binding obligations; but they represent the first formally harmonized regulatory statement on AI across the two largest pharmaceutical markets in the world. As Jones Day's regulatory analysis noted, they are likely to shape future mandatory guidance on both sides of the Atlantic ^[3],[10].

The ten principles are ^[3]:

1. Human-centric by design: AI systems must reflect ethical and human values, with patient interests and public health as the primary goal
2. Risk-based approach: validation, oversight, and safeguards must be proportionate to the COU and associated risk level
3. Adherence to standards: AI applications must comply with applicable legal, ethical, scientific, and regulatory standards
4. Clear context of use: the intended purpose and scope of every AI application must be precisely defined
5. Multidisciplinary expertise: AI development and deployment must draw on diverse expertise, including clinical, statistical, computational, and regulatory knowledge
6. Data governance and documentation: data pipelines must be documented, controlled, and auditable throughout the AI lifecycle
7. Model design and development practices: model architecture, training, and validation must be appropriate to the COU and documented
8. Risk-based performance assessment: performance testing must be proportionate to the AI system's influence on regulated decisions
9. Lifecycle management: AI systems require ongoing monitoring, maintenance, and periodic reassessment
10. Clear, essential information: users of AI systems must receive accurate and sufficient information about model outputs, limitations, and uncertainty

Applied Clinical Trials' analysis of the principles noted that the joint publication builds on prior FDA-EMA collaboration following bilateral discussions in 2024, and signals intent for deeper convergence in future jurisdiction-specific guidance ^[11]. For sponsors, the practical effect is that a credibility assessment plan designed to satisfy FDA's seven-step framework will, if adequately detailed, also address the governance and documentation expectations embedded in EMA's principles and vice versa.

What the Frameworks Mean for Specific AI Use Cases

How the combined framework applies depends substantially on what the AI system does. The table below maps common pharmaceutical AI uses to their primary compliance obligations.

This mapping is not exhaustive, does not constitute legal advice, and does not substitute for formal legal or regulatory analysis of specific systems and their intended use. EU AI Act classification in particular requires a documented Article 6 assessment for each system; the entries above reflect the most likely classification but not a definitive determination. Regulatory expectations in this area are actively evolving: FDA's January 2025 drug AI guidance remains in draft, EMA GMP Annex 22 is under development, and the EU AI Act Digital Omnibus has not yet been formally enacted as of this article's publication in June 2026.

ICH E6(R3): GCP Meets the Digital Trial

ICH E6(R3), finalized January 6, 2025 by the ICH Assembly, adopted by EMA with effect from July 23, 2025 ^[18], and adopted by FDA in September 2025 ^[4], is not an AI-specific guideline. Its scope is Good Clinical Practice broadly; trial design, conduct, monitoring, documentation, and data governance. But it carries material implications for any organization deploying AI within a GCP-regulated trial.

The guidance introduces a dedicated chapter on computerized systems with requirements for validation, access controls, user accountability, and audit trails ^[12]. Systems used in data capture, remote monitoring, or source record management must be validated to a standard appropriate to their clinical risk; digital platforms performing these functions are explicitly subject to the expanded computerized systems requirements that R3 adds to the R2 baseline ^[12]. For AI systems operating in these roles; including systems that flag data anomalies, generate monitoring signals, or automate documentation workflows; the validation obligation applies in full.

E6(R3)'s data governance framework also extends to CROs and technology vendors. Sponsors must ensure adequate oversight even where trial tasks are outsourced; the responsibility for AI system performance does not transfer to a vendor ^[12]. This has specific implications for organizations using commercially developed AI tools for protocol generation, clinical data review, or regulatory writing, where the underlying model training data and validation documentation typically reside with the vendor, not the sponsor.

ICH M11 CeSHarP, finalized by the ICH Assembly in November 2025 and adopted by FDA in May 2026 ^[5], adds a structured protocol format layer. The standard provides harmonized content structure, terminologies, and data fields for clinical trial protocols to enable interoperable electronic exchange across regulatory regions ^[5]. For AI-assisted protocol generation, this creates both a compliance target and an opportunity: a system that outputs CeSHarP-structured protocol content aligned with the standard's data elements is, by design, producing content formatted for direct regulatory review. Sponsors planning global trials in 2026 and beyond should author protocols using the CeSHarP template; the structured format facilitates electronic extraction by regulators rather than manual assessment ^[5].

EMA GMP Annex 22: AI in Manufacturing Takes Shape

EMA's efforts to govern AI extend beyond clinical and regulatory operations. The EMA GMP/GDP Inspectors Working Group is actively developing Annex 22 of the EU GMP Guide, which will establish binding requirements for AI in pharmaceutical manufacturing ^[13]. A July 2025 draft introduced a structured, risk-based framework for AI adoption in manufacturing, with expectations for intended use documentation, validation, lifecycle management, explainability, and human-in-the-loop oversight ^[14].

The draft initially stated that dynamic, adaptive, and probabilistic AI models; including generative AI and large language models; should not be used in critical GMP applications. Following stakeholder consultation in 2025 that revealed significant industry support for enabling these technologies, EMA convened a multistakeholder expert workshop in June-July 2026 to reassess this position and develop appropriate control measures ^[13]. The outcome will determine whether generative AI can be used in GMP-critical contexts under defined safeguards, or whether the categorical restriction will hold.

For organizations using LLM-based tools in regulated manufacturing or documentation contexts, this is an active regulatory boundary. The question of whether generative AI's probabilistic outputs can meet the deterministic, auditable standards that GMP requires is not yet settled in European regulation, and the answer will have direct implications for how regulatory document generation systems are validated and deployed.

EU AI Act: The Legal Floor Beneath All Guidance

All of the guidance frameworks discussed above operate above a legal baseline established by the EU AI Act (Regulation 2024/1689) ^[7]. Unlike the EMA Reflection Paper, FDA's draft guidance, or the joint principles, the Act is law once its provisions take effect: obligations are binding and enforceable by national market surveillance authorities.

Whether a pharmaceutical or clinical AI system qualifies as high-risk requires analysis under Article 6 ^[7]. The first classification pathway (Article 6(1)) covers AI embedded as a safety component in a regulated product that requires third-party conformity assessment under Annex I-listed Union harmonization legislation, most relevantly the EU Medical Device Regulation (MDR 2017/745). An AI clinical decision-support tool that meets the MDR definition of a device, and whose safety function requires third-party assessment, is high-risk on this pathway. The second pathway (Article 6(2)) covers AI whose specific intended use falls within one of the eight Annex III categories. Health-related Annex III entries are narrow: they cover biometric categorization using sensitive attributes, AI used by public authorities to determine eligibility for public healthcare benefits, AI for risk assessment and pricing in life and health insurance, and patient triage systems ^[7]. General pharmaceutical R&D applications, including regulatory document generation, protocol development tools, and pharmacovigilance analytics, are not listed in Annex III and do not automatically qualify as high-risk ^[7]. Even where a system does fall within an Annex III category, Article 6(3) provides that it need not be treated as high-risk if it does not pose a significant risk of harm to health, safety, or fundamental rights and does not materially influence the outcome of decision-making ^[7].

When high-risk classification does apply, obligations include ^[7]:

• Mandatory risk management systems covering the entire lifecycle
• Data and data governance requirements for training and test datasets
• Technical documentation of system design and intended use
• Record-keeping sufficient to support post-deployment auditing
• Transparency and information provision to users
• Human oversight mechanisms built into system design
• Performance standards for accuracy, robustness, and cybersecurity

Under the Digital Omnibus provisional agreement of May 7, 2026, expected to be formally adopted before August 2, 2026, the compliance timeline would be extended: standalone Annex III systems would comply by December 2, 2027, and AI embedded in Annex I regulated products by August 2, 2028 ^[16]. Compliance with EU AI Act requirements does not discharge obligations under EMA guidelines or GxP standards; the Act is additive, not substitutive ^[7].

The financial scale of compliance is not trivial. Research published in a peer-reviewed analysis of the Act's healthcare implications estimated annual compliance costs of approximately EUR 29,277 per AI unit, with certification costs ranging from EUR 16,800 to EUR 23,000 per unit ^[15]. These figures reinforce why organizations that integrate regulatory compliance into system design from the start, rather than retrofitting it, will carry a material cost advantage.

Regulatory and Documentation Considerations for AI Systems

Across FDA, EMA, ICH, and EU AI Act requirements, several documentation obligations apply to any AI system operating in the regulatory evidence chain:

None of these documentation requirements are novel in principle; they reflect the same quality management expectations that apply to validated computerized systems in GxP environments. What changes with AI is the complexity of producing this documentation for systems whose internal logic is probabilistic, whose training data shapes outputs in ways that are difficult to trace, and whose performance may shift over time without obvious warning.

How Kitsa Fits Into This Problem

Kitsa positions KScribe as a regulatory document generation platform built for this compliance environment. Kitsa describes KScribe as producing regulatory documents from structured clinical intelligence rather than from general probabilistic text generation: protocol content, regulatory specifications, and cross-document dependencies are represented explicitly and intended to be auditable. For sponsors and medical writing teams whose documentation will face regulatory inspection, the compliance question is not just whether the output is accurate; it is whether the system generating it can demonstrate how.

Key Takeaways

• •FDA's January 2025 draft guidance (FDA-2024-D-4689) introduced a seven-step credibility assessment framework that applies to any AI model generating data for regulatory decision-making on drug safety, efficacy, or quality; the first such framework from the agency.
• •EMA's September 2024 Reflection Paper (EMA/CHMP/CVMP/83833/2023) places full accountability on sponsors and MAHs for ensuring AI systems are fit for purpose, and accepts that full explainability is not always achievable; but requires interpretability and human oversight in its place.
• •The ten FDA-EMA Guiding Principles published January 14, 2026 represent the first formally harmonized regulatory statement on AI across both agencies, emphasizing risk-based approaches, defined context of use, lifecycle management, and human-centric governance.
• •ICH E6(R3), finalized January 2025 and adopted by FDA in September 2025, adds dedicated computerized systems requirements; including validation, audit trails, and access controls; that apply directly to AI systems operating in GCP-regulated trials.
• •ICH M11 CeSHarP, finalized November 2025 and adopted by FDA in May 2026, provides a structured protocol standard that AI-assisted protocol generation tools should target for output alignment.
• •The EU AI Act (Regulation 2024/1689) creates binding obligations for AI systems that qualify as high-risk under Article 6; pharmaceutical and clinical AI is not automatically classified as high-risk and requires a specific Article 6 classification assessment. Under the May 2026 Digital Omnibus provisional agreement, pending formal adoption, the compliance deadline for standalone Annex III systems would move to December 2, 2027 ^[16].
• •Across all frameworks, four documentation requirements are consistent: credibility or validation plans, training and test data provenance, model architecture records, and lifecycle monitoring evidence.

Frequently Asked Questions

References

This article reflects regulatory status as of June 2026. FDA's January 2025 AI guidance remains in draft. EMA GMP Annex 22 is under development. The EU AI Act Digital Omnibus amendments are provisionally agreed but pending formal enactment. Readers should verify current status of all cited instruments before implementation decisions.

Related Articles