Auditable medical AI in surgery: an architectural response to the ethics imperative

INTRODUCTION: A FORENSIC QUESTION, ASKED FROM A CLINICAL CHAIR

In Mata v. Avianca (S.D.N.Y., 2023), counsel was sanctioned for filing generative-model citations that could not be verified when challenged^[1]. The same forensic question - on what basis did the system say this, from which source, and when - will arise when a surgical artificial intelligence (AI) output is reviewed after an adverse event. The system either produces a reconstructible evidentiary artefact or it does not.

The journal’s founding editorial placed surgeons at the centre of defining Artificial Intelligence Surgery^[2]. Capelli et al. articulated trustworthy surgical AI as explainable, fair, accountable, robust, and safe^[3]; World Health Organization (WHO) framed comparable commitments for health AI^[4]; the surgical-training consensus extended them to pedagogy^[5]; and the Artificial Intelligence Organization for the Next Generation of Surgeons (AIONS) supplied working definitions for AI Surgery, surgomics, and robotics^[6].

These documents articulate what trustworthy systems should be; they do not specify the architectural artefacts by which those claims can be externally verified. This Perspective addresses that implementation gap.

The proposition is that high-trust surgical AI requires an assurance architecture spanning pre-operative support, training, and intra- and post-operative ambient assist, built around four externally inspectable properties. These are proposed as engineering determinants of defensibility, not regulatory mandates.

WHERE THE ETHICS FRAMEWORK REACHES ITS OPERATIONAL LIMIT

Ethical principles remain necessary but do not by themselves make system behaviour reconstructible. Transparency, for example, cannot be established from fluent outputs alone; it depends on what the system is structurally required to retrieve, refuse, and record. Architecture therefore supplies artefacts against which ethical claims can be tested.

Evidence and adjacent professional failures illustrate the problem: a widely implemented sepsis model performed substantially worse on external validation than originally reported^[7]; generative systems can fabricate citations^[1]; and clinical language-model outputs may require verification against the health record^[8]. Aggregate performance or fluent output alone therefore cannot establish defensibility at the point of use.

The task is to identify architectural properties that make ethical claims auditable rather than merely asserted. The four properties below are candidate choices for high-trust clinical AI environments and are treated as load-bearing commitments rather than features.

THE FIRST ARCHITECTURAL COMMITMENT: SOURCE-GROUNDED RETRIEVAL

A clinical AI system that generates medical content from model parameters without runtime retrieval against a curated, version-controlled source corpus is structurally weaker under audit: it cannot reconstruct the source pathway supporting a contested answer.

Source-grounded retrieval is not a novel form of retrieval-augmented generation (RAG). Conventional RAG combines parametric generation with retrieved non-parametric memory^[9], and recent conceptual clinical work has proposed curated medical knowledge bases, provenance-aware RAG, and tamper-evident logging^[10]. The contribution advanced here is their coupling to permission and authorisation gates: for evidence-dependent assertions, retrieved material must support the claim within identifiable provenance, version, and authorised scope; otherwise, the system enters a typed defer/refusal state and records it.

Support is assessed at claim level, not by exact-text matching. Multiple sources may be synthesised if each material claim remains traceable and disagreement is preserved. When no guideline directly addresses atypical anatomy or a rare intra-operative presentation, the system may provide a bounded synthesis of relevant primary literature, state the evidence gap, request context, or defer a prescriptive recommendation. Refusal applies to the unsupported or unauthorised claim, not to all potentially useful evidence.

Independent clinical work supports separating information supply from claim authorisation. In a controlled breast-cancer decision-snapshot evaluation, RAG without an authorisation gate did not reduce unjustified inference relative to an unconstrained large language model (LLM), whereas an evidence-graded authorisation framework reduced unsupported claims and increased appropriate refusal^[11]. This preprint does not establish a standard, but supports the distinction: retrieval supplies evidence; governance determines what may be asserted.

The audit objective is reconstructibility: which source passage, corpus version, authorisation rule, and system state supported or prevented an assertion. Benchmarks such as MedHELM remain useful for task-level performance^[12], but do not replace output-level provenance.

In surgery, the pre-operative guideline query is one text-based example, not the architecture’s modality limit. Before laparoscopic cholecystectomy or liver resection, a query about updated society guidance should return an answer bound to an identifiable document section and date, or an evidence-insufficiency/defer state rather than an unsupported assertion. Intra-operative observations may instead originate from video or sensors; their governance is addressed below.

THE SECOND ARCHITECTURAL COMMITMENT: PERMISSION-FIRST GOVERNANCE

Medical AI often operates at the boundary between patient care and governed clinical content, including society guidelines, copyrighted literature, and institutional protocols. A permission-first architecture inverts ingestion: content is ingested, indexed, retrieved for synthesis, or used to generate an output only when an explicit upstream permission or licence covers that use. Content scopes are defined before deployment and enforced at retrieval time rather than moderated after the fact.

Machine-readable frameworks establish precedents for ex ante control. The World Wide Web Consortium (W3C) Open Digital Rights Language (ODRL) represents permissions, prohibitions, duties, and constraints over digital assets^[13], while the Global Alliance for Genomics and Health (GA4GH) Data Use Ontology (DUO) encodes allowable uses of biomedical datasets to support authorisation decisions^[14]. They govern different objects but show that permitted uses can be represented with content and evaluated computationally.

For guideline authorities, retrospective takedown cannot fully preserve source authority once content has been ingested and paraphrased. Under this proposal, issuing bodies define permitted uses ex ante and those constraints are enforced during retrieval and synthesis, preserving source authority, version, scope, and jurisdiction.

For surgical societies and guideline authorities, permission-first governance therefore offers a mechanism for AI-mediated distribution while preserving enforceable control over content, version, scope, and jurisdiction^[13,14]. Figure 1 illustrates the evidence-governance path when an evidence-dependent question lacks an eligible, current source in the authorised corpus.

Figure 1. Permission-first evidence governance: supported output, defer state, human fallback, and auditable record. A clinician’s evidence-dependent question is evaluated only against an authorised evidence corpus. When an eligible and current source is available within the permitted corpus, the system may produce a supported output linked to the retrieved source, its version, scope, and permission status. When no eligible source is available, the system enters an evidence-insufficiency/defer state and does not generate an evidence-dependent clinical assertion from material that has not been retrieved or is not authorised for synthesis. Official publisher links may optionally be presented outside the synthesis boundary for direct consultation, but are not retrieved, ingested, indexed, or synthesised. The clinician retains the standard clinical pathway. The request identifier, timestamp, corpus version, permission policy, retrieval event, output state, and hash-chain reference are retained in a tamper-evident audit record. The “less than five years” condition shown in the figure reflects the corpus-currency policy applied to guideline sources in the illustrated architecture; it is not proposed as a universal criterion for clinical validity. This figure is conceptual and does not constitute clinical validation.

THE THIRD ARCHITECTURAL COMMITMENT: REFUSAL AS A FIRST-CLASS BEHAVIOUR

Abstention is established in selective prediction and learning-to-defer, where models withhold or defer predictions when uncertainty or expected error is unacceptable^[15,16]; medical-LLM work also shows failures to abstain appropriately under clinical uncertainty^[17]. Here, refusal is extended beyond statistical uncertainty as a typed governance event. The working taxonomy separates capability states (out-of-scope, insufficient retrieval evidence, internal source conflict) from authorisation states (out-of-licence, out-of-mandate)^[18]. Each event is recorded with its trigger in the audit trail.

Refusal is an AI output state, not an instruction to halt care. Intra-operatively, the system withholds an unsupported or unauthorised assertion while the surgeon retains control and the standard clinical pathway. Fallbacks may request missing context - an interactive strategy evaluated in MediQ^[19] - provide a traceable bounded synthesis, expose unresolved conflicts, defer to the clinician, or withhold the unsupported recommendation. Selective-prediction studies frame abstention against the costs of error and non-assistance^[20], while ClinDet-Bench shows that incomplete information can produce both premature conclusions and excessive abstention^[21]. Refusal frequency should therefore depend on task, case mix, evidentiary availability, and institutional mandate; the auditable requirement is the recorded trigger, resulting state, and human override.

THE FOURTH ARCHITECTURAL COMMITMENT: A CRYPTOGRAPHICALLY CHAINED AUDIT TRAIL

The three commitments converge on a fourth: retrievals, generations, refusals, and overrides are recorded in an integrity-protected audit trail. Secure logging mechanisms are well established: Schneier and Kelsey described constructions designed to make earlier entries resistant to undetectable modification or destruction after compromise^[22]. In the proposed architecture, records are hash-linked so each payload binds to the digest of the preceding record. Independent recomputation additionally requires deterministic serialisation or canonicalisation and an initial trust anchor; these are implementation specifications of this proposal, not requirements established by reference^[22].

National Institute of Standards and Technology (NIST) Special Publication (SP) 800-92 addresses operational log management^[23]; International Organization for Standardization (ISO) 27789:2021 defines electronic health record (EHR) audit trigger events and audit data^[24]; and healthcare implementations have demonstrated immutable access logs on permissioned distributed ledgers^[25]. These precedents ground durable, reviewable, tamper-evident records without prescribing this architecture.

The audit trail is the artefact available to regulators, review boards, or expert witnesses. Without it, post-hoc reconstruction of how unsupported content was generated or surfaced becomes substantially more difficult.

Three properties are central: integrity protection and tamper-evidence, content selectivity, and retention. Integrity protection makes unauthorised modification detectable; selectivity enables reconstruction of retrieved sources, refusal class, model and corpus versions, and applied thresholds without exposing clear-text patient data; retention keeps the artefact available on a medically meaningful time horizon.

Stronger implementations may add external timestamping, independent verification, or distributed-ledger anchoring; these are implementation options rather than requirements of the proposed architecture^[22-25].

THE SURGICAL TRIAD IS ALREADY THERE: PREPARATION, TRAINING, DAILY COMPANION

Surgical practice already spans three operational phases: preparation, training, and daily clinical work. Source checking, institutional protocols, curricula, case records, and handoffs already embody expectations of justification and traceability; AI should preserve rather than erode them.

The question is therefore whether AI preserves the auditability expected of these established practices, not how the four commitments map onto a product line.

In preparation, a source-grounded retrieval system with a tamper-evident audit trail instruments the surgeon’s existing source-checking reflex^[9,10,23,24]. Assertions should remain traceable to retrieved passages, refusals classified, and interactions recorded for later reconstruction.

In training, the 2025 consensus supports simulation, AI-enabled objective feedback, automated skill assessment, error identification, and validation before widespread adoption^[5]. The proposed architecture adds inspectability of evidentiary basis, system version, trainee interaction, and feedback, allowing training or accreditation bodies to review an auditable artefact.

In the daily companion layer, Article 14 of the EU AI Act makes human oversight salient for high-risk AI, while medical-device AI remains subject to the Medical Devices Regulation (MDR) and In Vitro Diagnostic Medical Devices Regulation (IVDR) and their interplay with the AI Act^[26,27]. Mascagni et al. demonstrated a deep-learning computer-vision system that segments hepatocystic anatomy and assesses critical-view-of-safety criteria from laparoscopic images^[28]. That perception layer is distinct from the proposed governance layer: provenance can identify video or frame context, model and version, and observation; permission and mandate constrain use; abstention or refusal governs outputs outside validated or authorised conditions; and audit records the resulting events. Source-grounded retrieval becomes relevant when an observation is converted into an evidence-dependent assertion or recommendation. The same governance pattern can surround future multimodal or vision-language components without requiring them.

Latency controls need not share one path. Pre- or post-operative consultation can perform retrieval, authorisation checks, and durable logging synchronously. In high-frequency intra-operative perception, the validated perception model remains on the real-time path while audit records are committed at event level asynchronously or in short batches; retrieval is invoked when an observation becomes an evidence-dependent assertion rather than on every frame. Event granularity, batching, and acceptable latency must be validated for intended use and risk; no universal latency budget is proposed. Auditable clinical RAG architectures likewise identify latency and usability as feasibility constraints^[10].

Thus, the four commitments provide a coherent governance and assurance posture across phases, with implementation adapted to modality and workflow.

REGULATORY CONVERGENCE: OVERLAPPING OBLIGATIONS AND ARCHITECTURAL IMPLICATIONS

The EU AI Act, MDR/IVDR, and General Data Protection Regulation (GDPR) impose overlapping obligations rather than a single system architecture. The AI Act classifies systems as high risk in specified circumstances and Articles 9-15 address risk management, data governance, technical documentation, record-keeping, transparency, human oversight, accuracy, robustness, and cybersecurity^[26]. Where personal data are processed, GDPR adds accountability, data protection by design and default, records of processing where applicable, and security^[29]. Medical Device Coordination Group (MDCG) 2025-6 addresses the interplay between MDR/IVDR and the AI Act for medical-device AI^[27].

These regimes do not mandate source-grounded retrieval, typed refusal, or cryptographic chaining^[26,27,29]. The four commitments are candidate engineering patterns that may help operationalise and evidence requirements for documentation, record-keeping, transparency, oversight, governance, accountability, and security. Their records can contribute to broader conformity and accountability artefacts but do not establish regulatory compliance by themselves.

EXTENDING THE SAME COMMITMENTS TO ADJACENT LAYERS

The same governance principles extend beyond the immediate clinical decision to institutional governance and scholarly knowledge, with implementation adapted to modality, content type, and workflow.

At the institutional layer, hospitals, academic centres, and guideline authorities define which AI behaviour is permitted, against which protocols, and under which oversight. Applying the same provenance, authorisation, refusal, and audit principles can connect a clinical event to the rule governing it.

At the scholarly knowledge layer, the relevant objects are the corpus, its version discipline, and its relationship to source authorities. Permission-first, machine-readable policies can support AI-mediated distribution while preserving explicit control over scope, version, and jurisdiction^[13,14].

These extensions locate the argument at the structural layer where clinical assertions, institutional rules, and published guidance must remain reconstructible and attributable.

CONCLUSION: THE NEXT COLLECTIVE TASK

Surgery has advanced by embedding ethical duties into practices that alter the conditions under which harm occurs. Auditable medical AI requires the same move from principle to inspectable architecture.

Source-grounding, permission-first governance, typed refusal, and cryptographically chained audit trails are proposed as architectural conditions supporting defensibility in surgical AI.

The author-developed Refusal Stack specification underlying the taxonomy is publicly deposited^[18] as a citable starting point, not a final standard. The surgical community should now subject the architectural layer to the same collective scrutiny applied to definitions, training standards, and clinical practice.