AURAVIO RESEARCH
A physician evaluation of multilingual clinical translation
We evaluated Auravio against a fixed OpenAI Whisper-1 speech-to-English translation endpoint using fictional clinical scenarios in five languages. Ten physicians independently reviewed anonymized output pairs.
Relative reduction in physician-rated serious clinical errors
Compared with the tested Whisper-1 endpoint on this five-language benchmark. Serious errors were defined as major or critical errors in the final English output.
26 of 500 Auravio ratings identified a serious error, compared with 158 of 500 baseline ratings. The absolute difference was 26.4 percentage points; the relative reduction was 83.5% (95% confidence interval: 73.3–91.2%).
This developer-led study used scripted, fictional recordings with one speaker per language. The comparison includes whether systems produced the required English output, as well as the clinical meaning of that output. These results do not establish safety or patient benefit in clinical use.
How the study worked
Two pilot physicians reviewed 50 fictional clinical scenarios. Native translator-speakers translated and recorded them in Latin American Spanish, Italian, Brazilian Portuguese, France French, and Standard German, producing 250 recordings.
Both systems processed the same recordings. Ten additional physicians reviewed anonymized output pairs, with two physicians independently assessing each pair. This produced 500 paired assessments and 1,000 individual severity ratings.
Twelve physicians participated overall: two in the pilot review and ten in the final blinded evaluation.
Lower serious-error rates in all five tested languages
The direction of the results favored Auravio in every tested language. Language-level results and confidence intervals are reported in the preprint.
Auravio: A Blinded Physician Evaluation of Multilingual Clinical Translation
Michael Cacioli