VeriLLMed: Visual Debugging of Medical Large Language Models with Knowledge Graphs
Jump to Discussion
Presentation
- Session
- Can we have a (real) doctor, please?
- Time
- Tuesday, Nov 10, 13:00 – 13:12 (US/Eastern) · session 13:00 – 14:30
- Room
- Hall America north
- Presenting from
- Boston
Links
Sign in to access the Paper PDF.
- Preprint PDF(opens in new tab)
Abstract
Large language models (LLMs) show promise in medical diagnosis, but real-world deployment remains challenging due to high-stakes clinical decisions and imperfect reasoning reliability. As a result, careful inspection of model behavior is essential for assessing whether diagnostic reasoning is reliable and clinically grounded. However, debugging medical LLMs remains difficult. First, developers often lack sufficient medical domain expertise to interpret model errors in clinically meaningful terms. Second, models can fail across a large and diverse set of instances involving different input types, tasks, and reasoning steps, making it challenging for developers to prioritize which errors deserve focused inspection. Third, developers struggle to identify recurring error patterns across cases, as existing debugging practices are largely instance-centric and rely on manual inspection of isolated failures. To address these challenges, we present VeriLLMed, a visual analytics system that integrates external biomedical knowledge to audit and debug medical LLM diagnostic reasoning. VeriLLMed transforms model outputs into comparable reasoning paths, constructs knowledge graph-grounded reference paths, and identifies three recurring classes of diagnosis errors: missing errors, relation errors, and branch errors. Case studies and expert evaluation demonstrate that VeriLLMed helps developers identify clinically implausible reasoning and generate actionable insights that can inform the improvement of medical LLMs.
For Practitioners
Medical AI developers, healthcare data scientists, and teams evaluating diagnosis-oriented LLMs would be interested in this work. They can use the proposed knowledge-grounded visual analytics workflow to identify recurrent reasoning errors, compare model reasoning with biomedical reference paths, prioritize problematic behaviors, and guide model refinement.