An Evidence-Grounded Framework for Medical Devices

Summary & Methodology Analysis

Medical-device safety analysis requires maintaining costly and expert-intensive risk-management evidence across the lifecycle. Current LLM approaches address isolated tasks using generic prompts without providing necessary source links, traceability, uncertainty handling, lifecycle updates, or recorded expert review. To address this, the framework uses an evidence intake component to collect product descriptions, requirements, design and software documents, user information, risk-management files, verification evidence, complaints, post-market reports, and relevant standards. It stores and retrieves device and safety knowledge using vector databases, knowledge graphs, structured tables, or hybrid forms while preserving source links. A task setup component defines the device or device function under analysis, system boundary, intended use, use environment, users, patient context, and selected safety method. The system then produces method-specific candidate safety items, such as hazards, hazardous situations, failure modes, causes, effects, risk controls, safety requirements, verification ideas, safety-case claims, or review questions, using task definitions and retrieved evidence.

Before expert review, the system performs critique and uncertainty checks on generated items, including duplicate detection, format checks, consistency checks, source-evidence checks, multi-model comparisons, and generation of defeaters or review questions. Qualified experts then inspect source evidence and accept, reject, edit, merge, or split generated items, recording short justifications for important decisions. When requirements, software, design elements, risk controls, verification results, complaints, or post-market data change, the system identifies affected source-linked safety items and proposes updates. The underlying models or datasets mentioned include Scopus, IEEE Xplore, Google Scholar, ChatGPT, and Baidu Apollo. A structured literature review returned an initial search of more than 200 papers, resulting in 90 papers published between 2023 and 2026 after screening.

Despite these capabilities, prompt-only use can support early ideation, but it is not sufficient as the sole support method for regulated safety analysis. Public model training data is unlikely to contain device and company-specific evidence, making LLM outputs plausible but weakly supported by the actual device. Public case studies create a risk of evaluation data contamination where related material may have been present in model training data. Common natural language metrics such as BLEU are insufficient because they do not assess safety correctness, traceability, risk control quality, or review value. Expert reference analyses may be incomplete or reflect expert specific judgment. Non-public or newly built case studies reduce training data overlap, but cannot prove that no related knowledge was present in model training data. Furthermore, review effort may depend on reviewer experience, device familiarity, and the quality of the existing risk management file, and an MDR survey reports certification costs as the most frequently named reason for discontinuing products in the EU, cited by 91 percent of respondents.