Medical Q&A ChatbotQLoRA On Mistral-7B
A QLoRA fine-tune of Mistral-7B-Instruct into a doctor-style medical Q&A assistant, served behind FastAPI and Streamlit with safety disclaimers wired through every layer.
Headline results
- Base
- Mistral-7B, 4-bit NF4
- Adapter
- LoRA r=64, merged
- Data
- 329 pairs, demo scale
- Serving
- FastAPI + Streamlit
System architecture
Problem
Patients ask free-text health questions and a base instruction model answers in the register of a chatbot rather than a clinician. Adapting a 7B model to a specific voice and domain used to need multiple GPUs, which puts the whole parameter-efficient fine-tuning loop out of reach for most portfolios.
Approach
QLoRA end to end. Mistral-7B-Instruct loaded in 4-bit NF4 with double quantization and bfloat16 compute, LoRA adapters at rank 64 on the attention and MLP projections, trained with TRL's SFTTrainer over doctor-patient exchanges wrapped in Mistral's instruction format. The adapter is merged back into the base weights, then served behind a FastAPI /chat endpoint with a Streamlit chat UI. An educational-only disclaimer is wired through every layer: system prompt, API response, answer text and UI.
Impact
The full parameter-efficient fine-tuning to serving loop, working: 4-bit quantized base, LoRA adapter, supervised fine-tune, adapter merge, API, and UI. It is demo-scale by design at 329 training pairs and the training run is GPU-bound rather than executed in-repo, so this is the pipeline rather than a quality claim. The safety layering is the part worth reading, because a medical demo that only disclaims in the README has not thought about how it will be used.
Decisions & tradeoffs
Layer the safety disclaimer, do not place it
A medical demo gets screenshotted, embedded and quoted out of context. The disclaimer lives in the system prompt, the API payload, the generated answer and the interface, so it survives every one of those.
Merge the adapter for serving
Keeping LoRA weights separate is right during training and wrong at inference, where it adds a load step and a version to get wrong. Merging trades the ability to hot-swap adapters for a simpler serving path.
Call the scale what it is
329 pairs teaches a register, not medicine. Presenting it as a demo of the fine-tuning pipeline rather than as a capable medical assistant is the only honest framing.
Build spec
- Base
- Mistral-7B-Instruct, 4-bit NF4, double quant, bf16
- Adapter
- LoRA r=64, alpha 16, dropout 0.1, q/k/v/o + MLP
- Trainer
- TRL SFTTrainer, Mistral instruction format
- Data
- 329 doctor/patient pairs, demo scale
- Serving
- FastAPI /chat + Streamlit UI
System notes
- 4-bit NF4 with double quantization and bfloat16 compute, which is what puts a 7B fine-tune within reach
- LoRA rank 64 on both attention and MLP projections, merged back into base weights for serving
- The educational-only disclaimer appears in the system prompt, the API response, the answer and the UI
- TRL SFTTrainer with a formatting function that wraps each record in Mistral's own instruction template
What this does not show
- Educational demo only, not medical advice. 329 pairs teaches a register, not medicine, and the training run is GPU-bound rather than executed in-repo.
Stack
Mistral-7B · QLoRA / PEFT · TRL SFTTrainer · bitsandbytes · FastAPI · Streamlit