University of East LondonMultimodal Chest X-Ray RAG
My MSc dissertation. A fully local RAG system that finds similar chest X-rays among 29,101 MIMIC-CXR images and drafts a radiology report with Mistral 7B. Adding CheXbert label filtering raised retrieval precision from 10.33% to 50.33%, and an ensemble router took it to 60.67%.
[ Overview ]
Retrieval-augmented generation can improve LLM-written radiology reports by giving the model similar past cases to work from. The problem is that X-rays which look alike can show different conditions, so retrieving by visual similarity alone often returns the wrong reference cases. Cloud-hosted LLMs also raise patient privacy and compliance concerns.
For my dissertation I designed, built and evaluated a fully local, privacy-preserving multimodal RAG framework that adds structured clinical knowledge to retrieval. It was awarded 94%.
[ My role ]
- Designed the research question, the retrieval pipeline and the evaluation.
- Built the Python and FastAPI backend that runs every stage of the pipeline locally.
- Built the React workstation used to run cases, compare pipelines and inspect results.
- Ran the experiments and statistical tests, and wrote up the findings.
[ Screens ]
[ What I built ]
Retrieval pipeline
- BiomedCLIP image embeddings stored in Qdrant for similarity search over 29,101 chest X-rays from MIMIC-CXR.
- Qwen2.5-VL generates a visual caption for each input image.
Diagnostic filtering
- CheXbert labels filter retrieved cases so they match the diagnosis, not just the appearance.
- A parallel ensemble router runs several filters and picks the best one for each case.
Local report generation
- Mistral 7B runs locally through Ollama and writes the report from the top 3 retrieved cases.
- No patient data leaves the machine.
Explainability
- Zero-shot patch-similarity heatmaps show which regions the model attended to.
- Measured with normalised cross-correlation, which showed the heatmaps follow anatomy rather than the specific pathology.
Clinical workstation UI
- React, Vite and Tailwind CSS interface with a live pipeline view: input, encode, retrieve, filter, top 3, report.
- Tabs for report generation, an evidence matrix, visual grounding heatmaps and pathology meters.
- Side-by-side comparison of retrieval pipelines (no filter, keyword, CheXbert, LLM, text RAG and hybrid) for each case.
- Asynchronous image upload, streamed reports and a model picker for switching local LLMs.
[ Impact ]
- Retrieval precision rose from 10.33% with visual similarity alone (26.00% with keyword matching) to 50.33% with CheXbert filtering, an improvement of 40 percentage points.
- The ensemble router reached 60.67%, 10.33 points above the best single filter (p < 0.001).
- Better retrieval led to better reports: BLEU-4 rose from 0.0395 to 0.0699 and ROUGE-L from 0.1870 to 0.2549.
- Evaluated on a stratified set of 100 X-rays for retrieval and 30 for report generation.
[ Stack ]
- Python
- FastAPI
- Qdrant
- BiomedCLIP
- CheXbert
- Qwen2.5-VL
- Mistral 7B
- Ollama
- React
- Vite
- Tailwind CSS
[ What I learned ]
- Visual similarity alone isn't enough for medical retrieval; structured labels make the difference
- Measuring retrieval and generation quality with proper metrics and significance tests
- Keeping a responsive web UI separate from slow, GPU-heavy model inference