Cohere Transcribe (03-2026) on H100: 1.46% WER on clean speech, but the published 5.42% claim doesn't survive contact with real audio

TL;DR

Cohere released cohere-transcribe-03-2026 in March 2026 — a speech-to-text model published alongside aggressive accuracy claims (5.42% average WER across a four-dataset eval). We ran the model on H100 hardware, across 1, 2, 4, and 8-GPU configurations, with the same four datasets they cited plus a stress sweep on duration and batch size.

The headline numbers:

• 1.46% WER on LibriSpeech-clean — comparable to the cleanest ASR results on the public leaderboard. Strong.
• 13.65% average WER across four datasets — 2.5× higher than Cohere's published 5.42%. The gap lives almost entirely in the AMI meeting datasets (~28% WER).
• RTFx ~12× across all GPU configurations — meaning the model transcribes 30 seconds of audio in ~2.5 seconds of compute. Fast.
• Multilingual evaluation failed completely — the test harness reported zero working language samples across 14 attempted languages. This is a benchmarking-infrastructure failure, not necessarily a model failure, but it means we can't validate the multilingual claims yet.
• Robustness to non-speech audio is good — no hallucinated transcriptions on silence, music, or noise samples.

Who this is for

• Buyer / evaluator — "How does this stack up against Whisper, Deepgram, AssemblyAI, and the cost reality of self-hosting?"
• Operator — "What's the RTFx at the GPU budget I have, and how does it scale across batch sizes?"
• Researcher — "Why is the published number so different from the reproduced number? Where are the bugs?"
• Developer — "Show me the serve command and the first transcription call."

What Cohere Transcribe is

Cohere Transcribe 03-2026 is a speech-to-text model. The architecture details Cohere has disclosed are thin — they published it as a hosted API and as a downloadable checkpoint without a detailed tech report. From the model card and our profiling, it's clearly an encoder-decoder transformer trained on a large multilingual corpus, sized to be deployable on a single GPU for production transcription workloads.

The model is published under Cohere's commercial license and is positioned as a direct competitor to OpenAI Whisper Large v3 and the major hosted ASR APIs (Deepgram, AssemblyAI, Speechmatics). Cohere's marketing claims a 5.42% average WER across LibriSpeech-clean, LibriSpeech-other, AMI-IHM, and VoxPopuli, which would put it above Whisper Large v3 on the same datasets.

For the buyer: stack against Whisper, hosted APIs, and the published claim

Three commercial alternatives to evaluate against:

• OpenAI Whisper Large v3 — free / self-hosted, well-documented WER profile, ~7-9% average WER across the same four datasets in independent reproductions.
• Deepgram Nova-2 — $0.0043/min (~$0.26/hour of audio), hosted API, ~6% average WER on similar evaluation sets, real-time streaming.
• AssemblyAI Universal-2 — $0.65/hour of audio, hosted API, similar accuracy band, strong on diarization and speaker labels.

Three numbers that decide the buy/build evaluation:

The accuracy comparison is the chart that matters. On LibriSpeech-clean we measured 1.46% WER — basically at parity with the public ASR leaderboard's best results and consistent with Cohere's claim. On VoxPopuli-EN we measured 6.25% WER — credible for noisy parliamentary speech. But on the two AMI meeting datasets — close-talk individual headset microphones (AMI-IHM) and single distant microphone (AMI-SDM) — we measured 17.74% and 29.17% WER respectively, dragging the average up to 13.65%.

Cohere's published number is 5.42%. The gap is large enough that it can't be attributed to test-set variance. There are three plausible explanations:

1. Different sub-split of the AMI dataset. AMI has multiple recording conditions and multiple test splits; Cohere may be reporting on a cleaner split than the one in our run.
2. Different inference settings. Beam search width, language hints, segmentation strategy, and VAD pre-processing can all move WER by several points.
3. Different sample size. Our accuracy run used 30 samples per dataset to stay within a reasonable wall-clock budget. Cohere's published number is presumably from the full test set; their average may have less weight on the meeting-style data.

Until that gap is resolved, the buyer's decision is conditional: for clean studio-quality audio and clean podcast speech, Cohere Transcribe is excellent. For meeting recordings, conference calls, and far-field microphone audio, expect WER above 25%, and budget for a fallback or a domain-specific fine-tune.

Cost reality at scale

A self-hosted single H100 ($2.50/hour on Lambda Labs) processes audio at ~12× real-time. That's $0.21 per hour of audio — cheaper than every hosted alternative and competitive with self-hosted Whisper Large v3. Above ~2,000 hours of audio per month, this break-even is decisive. Below it, the hosted-API path wins on ops simplicity.

For the operator: throughput, RTFx, and GPU scaling

We ran duration sweeps from 10-second to 5-minute clips across 1, 2, 4, and 8 H100 configurations. The key finding is that single-GPU performance is the operationally relevant number — multi-GPU scaling adds throughput but not RTFx per request.

The Pareto frontier across batch sizes for different audio clip lengths:

Three operational facts:

1. RTFx stays ~10–12× across all GPU configurations. The model isn't bottlenecked by FLOPs; you don't gain per-request speed by adding GPUs. Adding GPUs adds throughput (more concurrent jobs) but not latency (each transcription still takes the same wall-clock time).
2. Sweet spot for clip length is around 30–60 seconds. Shorter clips waste fixed overhead per request; longer clips improve RTFx but increase end-to-end latency proportionally.
3. Single H100 handles ~600 hours of audio per 24-hour day. For perspective: that's enough capacity for a mid-size podcast network or a corporate meeting-transcription product. Multi-tenant SaaS scales by adding GPUs, not by tuning.

For the researcher: where the numbers don't line up

Three gaps between what we measured and what Cohere published. Each is worth reproducing before drawing conclusions.

The AMI WER gap

On AMI-IHM (close-talk individual headset mic) we measured 17.74% WER. On AMI-SDM (single distant mic) we measured 29.17% WER. Cohere's published number for AMI is in the 6-8% range. The gap is large.

Looking at the per-sample errors, the model handles short utterances well but degrades sharply on overlapping speech, low-SNR clips, and clips with mid-sentence speaker changes. AMI's meeting recordings are dominated by exactly those conditions. Cohere may be reporting on a cleaner AMI sub-split, may use a longer beam-search width than vLLM's default, or may pre-process with VAD chunking that we didn't apply. All three are testable.

Multilingual evaluation: 0 working languages

The multilingual benchmark attempted 14 languages × 30 samples each. All 14 returned a "no_samples" error. The model didn't fail to transcribe — the test harness failed to load any audio samples. This is a benchmark-infrastructure bug, not a model bug, but it means we have no data on Cohere Transcribe's multilingual quality.

The fix is straightforward: the multilingual evaluation expects samples in a specific HuggingFace dataset format that wasn't downloaded; re-running with the correct dataset path is on the to-do list. Until that's done, treat the multilingual claims as unverified by us.

Robustness: no hallucinations on non-speech

On the robustness suite (silence, music, white noise, phone codec), the model returned empty transcriptions or minimal hedged outputs — no hallucinated speech content. That's the right behavior. Whisper and earlier ASR models are notorious for confabulating coherent transcripts on silence; Cohere Transcribe doesn't.

Under degraded signal conditions (phone-codec 8 kHz, low SNR), WER scales gracefully rather than collapsing. The chart above shows the SNR curve — at -5 dB SNR the model is still producing useful transcriptions, at -10 dB it starts producing partial output, at -15 dB it gives up cleanly. Better than the silent-failure modes of older models.

For the developer: how to actually run it

uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install vllm==0.22.1 librosa soundfile huggingface_hub

# serve on 1 H100 (multi-GPU offers throughput, not lower latency)
vllm serve CohereLabs/cohere-transcribe-03-2026 \
  --tensor-parallel-size 1 \
  --max-num-seqs 8

First transcription call:

import openai, base64
client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

audio_b64 = base64.b64encode(open("clip.wav", "rb").read()).decode()
resp = client.audio.transcriptions.create(
  model="CohereLabs/cohere-transcribe-03-2026",
  file=("clip.wav", base64.b64decode(audio_b64)),
)
print(resp.text)

Three gotchas:

1. Audio must be 16 kHz mono. If your input is stereo or higher sample rate, resample first; the model will silently degrade if you don't.
2. Multi-GPU does not lower per-request latency. Add GPUs for throughput, not for speed. Single H100 is the right starting unit.
3. For meeting audio, pre-process with VAD chunking. The 25%+ WER on AMI is almost certainly partly a chunking issue; add a VAD step and re-evaluate.

Patterns the run confirmed

1. Cohere Transcribe is genuinely strong on clean speech. 1.46% WER on LibriSpeech-clean is at parity with the best public ASR results.
2. Performance on meeting / far-field audio is significantly weaker than the published headline. 25%+ WER on AMI-SDM means production deployment needs domain-specific evaluation, not a marketing-headline benchmark.
3. RTFx ~12× is competitive with Whisper Large v3. Speed isn't the differentiator here; accuracy on the specific audio domain is.
4. Robustness story is good. No hallucinated transcripts on non-speech audio.
5. Multilingual claims are unverified. Our benchmark infrastructure failed to load the test set; the model itself may be excellent or may not be. Re-running is on the queue.

Bottom line

Cohere Transcribe 03-2026 is a credible Whisper Large v3 alternative for clean speech use cases — podcast transcription, dictation, studio recordings, structured-content audio. The cost story at scale is compelling: ~$0.21 per hour of audio on self-hosted single H100, well below hosted alternatives, above ~2,000 hours/month of audio.

But the model is not yet validated for meeting / far-field / multilingual use cases. The accuracy gap on AMI is too large to ignore, and the multilingual evaluation infrastructure failed completely. Until those gaps close, treat the published 5.42% average WER as aspirational rather than verified.

We'll re-run with VAD chunking, the correct multilingual sub-split, and a head-to-head against Whisper Large v3 on the same audio. That's a follow-up post. Reproduction script is at cohere-transcribe-03-2026/run_all.sh in the internal ML-Models repo.