Accent and Dialect Robustness in ASR Systems

Accent and dialect robustness in speech recognition is not a side issue that shows up occasionally with a heavily accented speaker. It's a structural gap built into how these systems get trained and tested, and it decides whether the transcript your software hands back is trustworthy for the person actually speaking into the microphone. Understanding where that gap comes from is the only way to judge whether a given tool can be trusted with a real, mixed-accent conversation rather than a lab recording of one kind of speaker.

What the evidence shows about how large the accuracy gap is

The clearest data point remains the Koenecke et al. study published in PNAS in 2020. Researchers ran five commercial systems (Amazon, Apple, Google, IBM, and Microsoft) on 19.8 hours of structured audio from 42 white speakers and 73 Black speakers, matched for age and gender so the comparison wasn't confounded by anything else. Average word error rate came out to 0.19 for white speakers and 0.35 for Black speakers, nearly double, and every single one of the five systems showed the same direction of gap. Error rates were worst for Black male speakers. Because the pattern held across five separate vendors built on different engineering teams and different data, the study ruled out the easy excuse that one company just built a bad model.

The non-native speaker gap tells a similar story from a different angle. One baseline ASR system showed a substantial word error rate gap between native and non-native speakers, on both read speech and human-machine interaction speech. That's a gap between a usable transcript and one a person has to rewrite by hand. That's the difference between a usable transcript and one a person has to rewrite by hand.

A 2025 comparison published by NHSJS ran 3,038 recordings drawn from 293 linguistic groups in the Speech Accent Archive against three widely used models. Whisper large-v3 posted a mean word error rate of 9.3%. Wav2Vec2 came in at 26.1%. Vosk en-0.22 hit 32.9%, more than three times Whisper's error rate on the exact same audio. Even inside Whisper's own results, groups with less representation in training data, Sylheti and Haitian Creole speakers among them, showed error rates 15 to 20 percentage points worse than well-represented groups like English and French speakers. Within the UK alone, regional accent variation follows the same pattern: varieties with less representation in training data tend to see higher error rates, a gap that tracks exposure in the training corpus, not how clearly anyone is speaking.

Spontaneous speech makes all of this worse. For some systems, word error rate for non-native speakers nearly doubles moving from a scripted sentence to an unscripted conversation. Two systems in the comparison, Google and another model, degraded to roughly the same error rate for native and non-native speakers once the speech became spontaneous, which suggests a chunk of the "native speaker advantage" seen in other tests is really an artifact of clean, read-aloud test conditions rather than a true measure of how the model handles real talk. Every study, every model, every linguistic comparison points the same direction. This isn't one vendor's flaw or one language's quirk.

Why the training data pipeline produces these gaps

Accents differ from each other along a few concrete acoustic dimensions: stress, tone, and duration. Those are exactly the features an acoustic model has to learn to match sound to text, so any imbalance in how much of each pattern the model has heard becomes an imbalance in how well it recognizes that pattern later.

Whisper, a widely used open-weight ASR model, trained on 680,000 hours of multilingual speech, a genuinely large number. But the hours aren't evenly split across accents and dialects, and performance on lower-resource language varieties drops off substantially compared to standard English. More hours of one pronunciation style means the model's internal probability weighting treats that style as the default guess, and everything that departs from it starts at a disadvantage before a single word gets decoded.

Research presented at FAccT '24 identified a second, less obvious problem: most ASR research treats accent as something only some speakers have, rather than a property that every speaker carries, including those who set the evaluation standards. That framing quietly turns disparate performance into a niche problem instead of a systemic one, and it makes the evaluation circular: systems get scored against a "standard" transcription written by listeners who already share the training data's accent assumptions. The baseline against which "accuracy" gets measured was never accent-neutral to begin with.

These failures compound once you leave a single-speaker lab setup. Diarization, the process of figuring out who said what, depends on clean acoustic separation between voices, and speakers whose accents share overlapping features, or who are speaking against background noise, make that separation harder to achieve reliably. Language model rescoring, the second stage that cleans up acoustic decoding using probability weights from text corpora, draws on text data that similarly skews toward standard varieties, meaning an error introduced acoustically may not get corrected at the rescoring stage. Add technical jargon into a sentence spoken with a non-standard phoneme inventory, and now two separate error sources are stacking on top of each other in the same clause. Child speech, elderly speech, and clinical speech introduce their own physiological wrinkles, but they trace back to the same root cause: a pipeline built around one prototype speaker who represents a small slice of the people actually talking into microphones.

How researchers are approaching accent robustness technically

Four distinct lines of research have gained real traction, and each one attacks a different stage of the pipeline.

Saliency-driven spectrogram masking, described in a 2025 paper by Sameti and colleagues submitted to ICASSP 2026, trains a classifier to spot accent-specific cues inside the spectrogram, masks those regions out, and uses the masked audio as augmented training data. Tested on Whisper for both English and Persian, it produced meaningful word error rate reductions and introduced the first systematic Persian ASR accent benchmark, filling a gap that simply didn't exist before.

Domain adversarial training paired with accent pre-training takes a different route: pre-train a discriminator on unlabeled accented speech, then apply adversarial training during fine-tuning. In low-resource settings where only one accent's annotated data was available, this method reported a 33% average improvement across multiple accents, a substantial gain given how little labeled data it needed to get there.

Data augmentation through cross-lingual voice conversion combined with speed perturbation synthesizes new acoustic variants of existing training speech, widening the range of accent-adjacent examples the model sees. Combining multiple augmentation techniques addresses the problem from more than one angle simultaneously.

A November 2025 paper in the EURASIP Journal on Audio, Speech, and Music Processing introduced Manifold Mixup for cross-accent generalization, a two-stage setup: framewise character recognition first, then a CTC-based end-to-end ASR model on top. Tested against Indic-TIMIT (six Indic accents), Common Voice accent groups, and LibriSpeech, it let a model trained on one accent generalize to accents it had never seen, with roughly a 2 percentage point absolute gain over a non-mixup baseline.

Plain fine-tuning on accent-specific data still works and still matters. Fine-tuning on accent-specific data for varieties like Scottish English has been shown to improve recognition for that variety, and approaches that layer general accent-diverse training with domain-specific fine-tuning show that robustness and narrow adaptation can be pursued together rather than treated as competing goals.

Benchmark infrastructure is catching up too. The ML-SUPERB 2.0 Challenge at Interspeech 2025 built a test suite spanning more than 200 languages, accents, and dialects. The winning submission achieved a 23 percentage point absolute improvement in language identification accuracy and an 18 percentage point reduction in character error rate on the general multilingual set. On accented and dialectal data specifically, the top submission hit 30.2% lower character error rate and 15.7% higher language identification accuracy, numbers large enough to suggest just how much room for improvement existed in the prior baselines. Still, Whisper's own robustness has limits: despite handling natural speech variation reasonably well, performance on lower-resource language varieties drops off substantially compared to standard English, and strong benchmark scores don't guarantee the same robustness under all real-world conditions. Strong scores on a curated benchmark don't guarantee the same model holds up once it meets the real world.

Where the accent gap lands in real-world meeting transcription

Meeting transcription stacks nearly every hard case ASR faces into one recording: spontaneous, unscripted speech, several people talking over each other, inconsistent microphone quality, crosstalk, specialized jargon, and speakers from different linguistic backgrounds all on the same call. That's precisely the environment where accent-driven error rates climb the fastest.

Published accuracy figures for AI meeting transcription tools in 2025 ranged from 82% to 98% depending on conditions. Top providers advertise word error rates as low as 4 to 7% under clean audio, but that same figure can climb past 15% under harder conditions, a gap wide enough to change whether a meeting summary is useful or misleading. Recordings drawn from Common Voice, a crowdsourced dataset with a wide range of accents, generally show accuracy 5 to 10 percentage points lower than clean LibriSpeech audio, a decent stand-in for what happens once the speaker population widens past whoever the model was mostly trained on.

A 2025 study by McGuire tested five current ASR systems against the L2-ARCTIC corpus, covering Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese first-language speakers reading scripted sentences. Whisper and AssemblyAI both landed close to human-level transcription, with mean match error rates of 0.054 and 0.056. But that result is for read speech specifically. Clinical speech research published in npj Digital Medicine in 2026 found meaningfully higher error rates for non-native speakers in clinical conversation settings, with the same read-versus-spontaneous gap appearing again even in current-generation systems.

A separate 2024 study by Graham and Roll in JASA Express Letters evaluated Whisper across native and non-native English accents and found accuracy higher for American and Canadian speakers than for British and Australian speakers, with native accents outperforming non-native ones overall. The errors tracked back to first-language prosody, vowel inventory differences, and whether the speech was scripted or spontaneous.

None of that stays contained inside the transcript. A transcription error caused by an accent mismatch turns into a wrong action item, a commitment attributed to the wrong person, or a technical term dropped entirely from the summary, and the mistake travels into every document generated downstream of that transcript. Diarization, meanwhile, now works reliably for up to roughly eight distinct speakers in most products under clean audio conditions, but similarity between two speakers' accents raises the odds of a diarization mistake, which then compounds with whatever transcription errors are already there. Testing in 2026 found that leading tools have largely converged above 90 to 95% accuracy for clean, native English audio. That convergence says nothing about how those same tools handle a globally distributed team on an ordinary weekday call with three accents and a poor wireless connection.

Evaluating an ASR-based Meeting Tool for Accent Robustness

The headline accuracy number on a vendor's site is almost never the number that matters. Figures like Zoom's reported 99.05% or Webex's reported 98.71% describe performance under clean audio and native-speaker conditions, a far cry from a meeting with six accents and a shared, spotty conference-room mic.

A few direct questions cut through the marketing faster than any spec sheet. What accent distribution trained or fine-tuned the underlying ASR model, and is that documented anywhere, or just implied? Does the vendor publish word error rate broken down by accent, language, or speaker demographic, or only a single blended figure that hides the worst cases? How does performance change between scripted and spontaneous speech, since that gap is one of the more reliable predictors of real-world degradation? And when the transcript does contain errors, does the system flag the low-confidence stretches, or does it just hand back a confidently wrong sentence with no warning attached?

Language coverage is a related but separate signal that should be checked on its own. A tool supporting a wide range of languages has likely put real investment into varied training data, more so than a tool built and tuned mostly around English. But coverage and accuracy are two different claims, and one doesn't guarantee the other, so confirm both rather than assuming breadth implies quality.

Audio quality still matters on its own terms, independent of anyone's accent. Background noise, a weak laptop mic, echo, and a shaky internet connection all raise error rates for every speaker on the call, so any evaluation should run on realistic recording conditions rather than a quiet room with a good headset.

The step from raw transcript to structured summary adds a second point where accent-related mistakes either get caught or get worse. Tools that run a large language model over the transcript to produce a summary can sometimes correct a phoneme-level slip using surrounding context, but that same correction step can also confidently rewrite something into a different meaning than what was actually said. The most useful test is an end-to-end one: run the tool against a recording that includes at least one non-native English speaker or a regional accent outside standard American English, then check not just the raw transcript's error rate but whether the resulting summary, action items, and any fields pushed into a CRM or project tool actually reflect what was said. Tools built around a full pipeline, transcription feeding into summarization, action item extraction, and routing into systems like a CRM or task tracker, carry more places for an accent-driven error to spread, but also more chances to catch and correct it before it reaches a teammate's inbox. Both layers need to hold up.

For any team with participants from different regions or first languages on a regular basis, accent robustness deserves consideration alongside price and integrations from the start of the tool selection process. It's the difference between a transcript that actually reflects the meeting and one that quietly rewrites it.