MEDIQ.

Clinicians lose time to charting and to hunting through records, and in some settings they can’t touch a keyboard at all. An assistant that helps there has to be easy to drive by voice or gesture — and it must never be the one that decides what goes into a patient’s record.

A team hackathon project. I built the clinical dashboard, the data layer — IndexedDB stores behind a validated repository, with the room and admission rules — and testing. Teammates built the voice, gesture and AI features.

1. 01InputVoice, hands or typing
2. 02VerifyOn-device speaker check
3. 03RouteRule-based commands
4. 04DraftSOAP note + safety audit
5. 05ConfirmClinician reviews exact text

Everything runs in the browser except AI calls, which go through a small Node server that holds the keys. Voice, hands and AI can move around the app and prepare a draft, but every write — a note, an admission, a transfer — passes one confirmation dialog, and room rules are checked again inside the database transaction.

1. 01
   Doctor-only voice

   An on-device speaker-verification model (WeSpeaker ResNet34, ONNX in the browser) checks who is speaking before any command runs, and fails closed when nobody is enrolled.

2. 02
   Deterministic commands

   A rule-based router handles navigation and reports — “show me the patient report”, “close the report” — so no language model decides where you go.

3. 03
   Touchless reports

   MediaPipe hand tracking runs locally: point to aim, pinch to drag a report, pull two hands apart to resize, hold a fist to close.

4. 04
   Drafts with a safety audit

   A structured SOAP draft and an independent audit sit side by side. Local rules always run, so the whole workflow works with no API keys.

• Nothing saves itself. Voice, gestures and AI can only navigate or prepare a draft. Saving goes through one dialog that shows the target, the source and the exact content, and needs an “I have reviewed this exact change” tick.
• Every result says where it came from. Outputs are tagged by source — model, local rule, record or simulation — so a warning from a local rule is never presented as an AI finding.
• Secrets stay on the server. Provider keys live only in a local Node process; a build step scans the browser bundle to confirm no key ships to the client.

A hackathon prototype on synthetic data — not a medical device. Speaker verification has no anti-spoofing, browser storage isn’t tamper-proof, and real-room microphone and gesture behaviour still needs testing on real hardware.

• TypeScript
• React
• Vite
• Tailwind CSS
• Node.js / Express
• Zod
• IndexedDB
• onnxruntime-web
• MediaPipe
• Web Speech API
• Vitest
• Playwright