Multi-Agent AI For GMP Audits In Pharma | Alyra

Specialist agents over an 11,268-page GMP corpus, answering supplier audit questions with a citation on every line and a CI gate that blocks regressions.

Every Answer Cited: A Multi-Agent Audit System For Pharma Quality

Custom Software & Integrations. Japan's largest pharmaceutical company. Pharmaceutical manufacturing. Research and development quality, and Good Manufacturing Practice (GMP) audit.

At a glance

The brief came from a quality lead who wanted audit preparation to stop starting from a blank page. Instead of reading across years of audit history, standard operating procedure (SOP) revisions, supplier change notices and overlapping regulatory text by hand, they wanted a system that answers a question in an auditor's own words and shows where every part of the answer came from.

Audit preparation that took days of reading across disconnected systems now comes back in under two minutes. A set of specialist agents covers the twelve kinds of question an auditor actually asks, across ten regulatory regimes and an 11,268-page body of evidence. Every line of every answer names the document it came from, so a finding can go straight into an audit report, and the system is retested on every change: if the answers get worse, the change cannot ship.

The evidence existed. Nobody could read it in time.

Eleven thousand pages, and an audit date that does not move.

Big pharma does not manufacture most of its own drugs. It outsources to CDMOs, the contract development and manufacturing organisations that run the actual production. Outsourcing the work does not outsource the liability. The sponsor stays legally responsible for product quality, which means it has to audit every CDMO it uses, repeatedly, for as long as it uses them. Qualification audits before anything starts, routine audits after, for-cause audits when something goes wrong.

That audit burden is the problem this system was built for. It is expensive, it is manual, it never ends, and the consequence of doing it badly is an FDA finding against the sponsor, not against the supplier.

An auditor therefore prepares for two kinds of audit: internal audits of their own sites, and supplier audits of the CDMOs. Before either one they have to assemble a single picture out of systems that do not talk to each other. SOP revision history sits in one place. Deviations, corrective actions and change controls sit in the quality management system. Supplier change notifications arrive as their own stream. The regulations themselves are spread across a dozen public agency sites that keep moving.

So the auditor pulls each thread by hand and cross-references them to build an agenda, a risk briefing and a checklist. The reading load is genuinely beyond a person. The reference corpus supplied for this build came to 11,268 pages across 93 documents, plus 354 structured quality records. Nobody reads that before a Tuesday audit.

The failure mode that matters is narrow and expensive: an SOP that has fallen out of step with guidance that changed since the last audit, because nobody reread both documents side by side. Miss it and it becomes an FDA 483 observation. A repeat 483 becomes a Warning Letter, and that means remediation cost, delayed approvals, and at the far end a consent decree.

There is a second constraint that rules out most of what a general-purpose assistant does. In a regulated context an answer without a source is worthless. A finding that cannot be traced back to a named document cannot go in an audit report, so a system that is usually right and occasionally unattributable is not a smaller version of the right tool. It is the wrong tool.

The brief was a system that answers questions the way an auditor asks them, and cites every line it returns.

Map the questions before building anything

The first deliverable was not code. It was the list of questions the system had to answer, written in the words a quality lead would actually use, from a pre-audit briefing through to a supplier health profile. Twelve were agreed, and each was mapped to the agents and sources it would need. That list became the acceptance criteria, which meant scope was a document both sides had signed off rather than an argument held later.

Written in his own words they read like this. What changed since our last audit of this CDMO in 2023. Give me a customised checklist with priority sections. Tell me the health of this CDMO before I qualify them. Here are my raw notes, photos and audio, generate my audit report. What the system hands back is audit agendas, risk-ranked checklists, observation logs and full reports in the client's own template.

Build the corpus, then the agents that read it

Ninety-three source documents were ingested, chunked and embedded, coming to 11,268 pages, alongside 354 structured records covering supplier change notices, SOP revisions and quality events. The largest single document ran to 1,388 pages. More agents were built than the brief asked for, each owning one compliance domain, because splitting the work across specialists rather than one large prompt is a traceability decision: each domain can be tuned, tested and trusted on its own, and every step leaves a log. In a GxP environment, meaning any process a medicines regulator can inspect, that log is not a debugging convenience. It is a regulatory requirement.

The orchestrator prompt is where the regulatory discipline lives. It does not behave like a chatbot. It behaves like a qualified GMP auditor, with deficiency classification, regulatory justification, audit-ready formatting, a hard requirement that every piece of information carries a citation naming its source document, and a second that a question about a named company is answered only once every source the system holds on that company has been correlated. The system is built from the sponsor's side, looking outward at its manufacturers, which is why the watchlist tracks ten named CDMOs for 483 observations, Warning Letters, recalls, data integrity findings and leadership changes. The sponsor sits on that list too, because it manufactures for other companies as well. It is both the auditor and the audited.

Make quality a gate, not a promise

GAMP 5 is the industry framework for validating computerised systems, and it assumes software that behaves the same way every time. Generative AI does not. That gap is the real engineering problem here, and a change-control gate is the only honest answer to it. If you cannot promise the system behaves identically on every run, you have to prove continuously that its quality has not moved.

The part nobody asked for is the part that matters most. A golden evaluation dataset, under version control with named authors and approvers, runs on every pull request that touches an agent, a prompt or the knowledge base. Four retrieval metrics have agreed thresholds, and a pull request that drops below any of them cannot merge. The runner also refuses to pass vacuously: if fewer than eighty percent of the evaluation samples return valid results, the run fails rather than reporting a clean sheet off a broken pipeline. That gate is the answer to the question every quality organisation eventually asks, which is what happens when the model changes and your validation is six months old.

The part that was not asked for is the part worth keeping

The system went to handover on day fourteen with more agents than the brief scoped, a written SOP and a recorded walkthrough.

The more durable outcome is the change-control gate. In a regulated environment, a retrieval system that cannot prove it still works after a change is a liability rather than an asset, and the gate turns that proof into something the build pipeline enforces rather than something a person remembers to check.

Two acceptance criteria were reported open rather than estimated. Both depended on a test set that was never supplied, and reporting them as pending was the correct answer. In this domain, a number nobody can trace is worth less than an honest gap.

The wider question the build answers is whether an agentic system can go through a real GxP validation lifecycle at all: user requirements, design qualification, installation, operational and performance qualification, risk management and continuous monitoring. Treating each agent as a configurable module with its own validation evidence is what makes that tractable, and the guardrails become the evidence. Mandatory citations, confidence thresholds and change-control triggers are all things you can put in front of an inspector. That question is the subject of a paper now in submission to peer-reviewed journals, with the validation mapping and the evaluation evidence from this build behind it.

The clearest signal came eight months after handover, when the client came back with three technical observations about how the system retrieved rather than a list of complaints. All three were built. A system nobody uses does not generate informed requests about its own ranking behaviour, and the work that followed those three observations is the part of this build we would keep.

"I can now walk into a CDMO audit prep with a cross-GxP picture pulled from SOPs, CAPAs, supplier history, and recent regulatory signals in minutes, where the same reading used to take me days across siloed systems. Every answer lands with its own citations and a verifier-agent cross-check, so I spend my time questioning the finding instead of chasing the source. The system also flags when a document I previously relied on has quietly changed." Gourav Pandey, R&D Quality Professional, a top-20 global pharmaceutical company

Specialist agents over an 11,268-page regulated corpus, with a citation on every line and an automated gate that blocks any change which drops retrieval quality.

Start a conversation