Why Clinical ASR Needs a New Evaluation Approach
Clinical voice AI is rapidly becoming part of dictation, ambient documentation, call-center workflows, patient intake, and post-visit follow-up. These systems must understand terms that are rare in general speech but central to clinical tasks: medication names, procedure names, anatomy, diagnoses, devices, symptoms, and specialty abbreviations.
Collecting real clinical audio is difficult and expensive. It requires annotation pipelines, IRB approvals, and strict HIPAA compliance. Patient recordings are protected health information (PHI), so they cannot be freely shared across teams or used in automated test pipelines. Synthetic audio, by design, contains no PHI, making it the only form of clinical speech data that teams can version, share, and test.
But generating synthetic audio isn't enough. If the synthesized speech mispronounces a medication or procedure name, it teaches the ASR model the wrong pronunciation, making failures harder to detect. This is where a pronunciation-aware flywheel becomes essential.
The Flywheel: A Repeatable Loop for ASR Improvement
The flywheel is a full improvement loop: build a benchmark, evaluate ASR behavior, use the results to decide what to change, and reevaluate after the change. It starts as a conversation with an AI agent, not a static config file. You can run the build skill in any agent harness (Claude Code, Codex, etc.) and describe the clinical workflow you want to stress test.
The agent asks a series of questions to define the clinical profile:
- What specialty or workflow?
- Which ASR failure modes have been observed?
- Which terms are encountered daily, and which are difficult?
Common terms become the baseline, and difficult terms become the signal that drives benchmark design. For example, in an orthopedic practice, the agent might focus on post-op instructions and PACU handoff, highlighting terms like Cefazolin, Ketorolac, and Total knee arthroplasty.
Generating Pronunciation-Aware Synthetic Audio
The pipeline uses NVIDIA NeMo Data Designer to expand seed terms into a richer dataset. Five output columns transform a clinical term into a phoneme-annotated, TTS-ready sentence:
| Column | Purpose |
|---|---|
| sample_id | Unique ID for the generated sample |
| sentence | Clinical sentence containing the exact target term |
| ipa_pronunciation | Reviewed or dictionary-derived pronunciation candidate |
| ssml_sentence | Sentence wrapped in SSML with phoneme markup |
| audio_filepath | Target path for the synthesized audio file |
The generated sentence must preserve the exact target term. If the model substitutes a brand name or abbreviation, the benchmark no longer tests the intended entity. The agent skill checks for this and regenerates or rejects rows that don't contain the exact term.
SSML Phoneme Tag Injection
SSML (Speech Synthesis Markup Language) is an XML-based markup language that provides TTS engines with instructions on how to synthesize speech. It's critical for controlling pronunciation, pacing, volume, and emphasis. The SSML step wraps the generated sentence in a <speak> element and injects a <phoneme> tag around every occurrence of the target term.
<speak>The nurse administered <phoneme alphabet="ipa" ph="əˌsiːtəˈmɪnəfɛn">Acetaminophen</phoneme> to the patient after surgery to manage mild pain.</speak>
Manual Pronunciation Review for IPA Gaps
Dictionary lookup covers many clinical terms, but not all. Newer drug names, trade names, and rare procedure terms may be missing. The flywheel handles these gaps with an explicit manual review path:
- Flag rows with missing or low-confidence IPA
- Use the agent harness to propose candidate IPA strings
- Validate the candidate against the TTS phoneme inventory
- Synthesize a short QA clip for the term in context
- Review to accept, edit, or reject the candidate
- Write accepted pronunciations to a reviewed override file
- Regenerate the affected SSML and audio
The skill enforces a pause: the agent cannot move on until the user has listened to the clips. This turns pronunciation gaps into a small review queue instead of a hidden benchmark-quality problem.
Synthesizing Audio and Producing the Manifest
Once each row has an SSML sentence and target audio path, the workflow synthesizes one audio file per sample. NVIDIA Magpie TTS Multilingual supports SSML phoneme tags with IPA and ARPAbet, allowing the synthesizer to render the clinical term using the reviewed phoneme sequence.
The final output is a NeMo-compatible JSONL manifest:
{
"audio_filepath": "data/audio/audio_Acetaminophen_3c7a1f02.wav",
"text": "The nurse administered Acetaminophen to the patient after surgery to manage mild pain.",
"duration": 3.914,
"term": "Acetaminophen",
"entity_category": "drug",
"ipa_source": "reviewed"
}
This manifest is the handoff point between SDG, ASR evaluation, and model adaptation. It keeps the metadata needed for slicing results by entity category, pronunciation source, context type, voice, or acoustic condition.
Benchmarking ASR Performance
The flywheel reports familiar ASR metrics, but presents them as decision signals:
| Metric | What it measures | Skill use |
|---|---|---|
| WER | Word error rate across the full sentence | General ASR quality signal |
| CER | Character error rate | Near-miss signal for long clinical terms |
| KER | Keyword error rate on the target clinical entity | Primary signal for workflow-critical terms |
| SER | Sentence error rate | Shows whether any error occurred in the sentence |
In the orthopedic practice simulation, entity-level metrics made the next step clear: medication names were the weakest category, so the next cycle focused on pronunciation review and additional drug-name coverage.
Limitations and Next Steps
Synthetic audio is not a substitute for real clinical audio. It's a controllable way to create targeted stress tests, especially for rare terms, but production validation still requires real-world audio. Pronunciation control still needs human review. The current benchmark is small; stronger claims require held-out terms, more contexts, more speakers, acoustic perturbations, and repeated runs.
Clinical environments include alarms, overlapping speakers, masks, telehealth microphones, room reverberation, ambulance noise, and dictation artifacts. The next version of the benchmark should include acoustic stress profiles.
If you're looking to understand how AI inference cold starts can impact real-time applications, check out this article on NVIDIA Dynamo Snapshot slashing AI inference cold start from minutes to seconds on Kubernetes.
For more on the infrastructure powering these AI workloads, see Azure's AI datacenters built for NVIDIA's Rubin platform.
Conclusion
Clinical ASR improvement requires more than a one-time dataset or aggregate score. You need a workflow that helps you define the clinical profile, generate pronunciation-aware synthetic audio, measure ASR quality on the terms that matter, adapt the model when appropriate, and reevaluate the result. The flywheel described here starts with a simple conversation and ends with a repeatable ASR improvement loop.
Ready to get started? Explore NVIDIA agent skills to use this workflow as a guide for building profile-driven benchmarks, reviewing pronunciations, generating synthetic clinical audio, and evaluating ASR output with entity-level metrics. Read the original article on NVIDIA Developer Blog for more details.
