ASR Model Architectures for Meeting Audio

Word error rate counts how often a model's transcript diverges from a human-verified reference: substitutions, deletions, insertions, all of it tallied as error. It's the standard metric in speech recognition, and it's also the most commonly misread one. Deepgram's published analysis shows a model can post a 2% WER on clean audiobook narration, then climb past 16% on a meeting recording pulled from the same benchmark infrastructure. That gap is the product of architectural choices most leaderboards never test for, because most leaderboards don't use meeting audio. They use short clips of one person reading clearly into a good microphone.

The Open ASR Leaderboard analysis says as much directly: the dominant benchmark format is short-form, clean English audio under 30 seconds. That format skips over multilingual performance and model throughput almost entirely, and both start to matter enormously once audio runs long and speakers talk over each other. A single WER number also buries three failure modes that show up constantly in meeting transcription and never in a headline score: hallucinated words inserted with false confidence, accent bias that quietly inflates error rates for non-standard speakers, and diarization mistakes that hand the right words to the wrong person. Vendors quote one clean-audio number because it's the number that makes the product look finished, not because it tells you anything about a Tuesday standup with three people talking over a bad speakerphone.

Meeting audio breaks from benchmark audio in four specific ways. Speakers talk over each other: Research on state-of-the-art diarization systems consistently finds meaningful error rates, driven largely by crosstalk. Acoustic conditions swing wildly inside a single call, speakerphone pickups, laptop mics, someone dialing in on a compressed codec from a hotel wifi network, all stitched into one recording. Meetings carry domain vocabulary, product names, internal shorthand, none of which showed up in training data. And meetings run long, 30 to 90 minutes typically, while benchmark clips cap out under 30 seconds, which means long-form handling is an architectural consideration most leaderboards never stress-test.

Four dominant architectures answer these four conditions differently, and they don't answer them equally well. Streaming Conformers lose the meeting fight before it starts. Speech LLMs and LLM hybrids are where the real ground gets contested. What follows takes each one on its own terms, then states which bets pay off for meetings and which ones only look good on a leaderboard.

How ASR architectures work at a mechanical level

Automatic speech recognition turns spoken audio into text, and how well that holds up under real conditions gets decided by design choices made long before anyone hits record. When branding is stripped off any system, the same three-step skeleton is underneath: audio processing produces the same core stages regardless of what a vendor calls its system. Audio first gets converted into log-mel spectral features, a compact numeric picture of frequency and energy over time. An encoder maps that picture into a high-dimensional space the model can reason over. Then a decoder, or a CTC head, or in newer systems an LLM, produces the output text and handles the alignment between acoustic frames and word sequences.

This isn't a research curiosity. MarketsandMarkets figures cited in Deepgram's analysis put the speech and voice recognition market at $9.66 billion in 2025, on track for $23.11 billion by 2030 at a 19.1% compound annual growth rate. Money like that means production deployment, not lab demos, decides where engineering effort goes.

The old split between "hybrid" and "end-to-end" systems doesn't describe the field anymore. Deepgram reports that as of 2026, four distinct architectures are in active production or active research: streaming Conformer models, encoder-decoder transformers, Speech LLMs, and LLM post-processing hybrids. The Open ASR Leaderboard, comparing more than 60 models from 18 organizations across 11 datasets as of November 21, 2025, is a decent signal of what the field bothers to measure. It says nothing on its own about how any of these models handle a 90-minute product review call with six people on speakerphone, because nobody built the leaderboard around that recording.

Each architecture makes a different core bet: on latency, on raw accuracy, on generalization, or on how cleanly it slots into an existing pipeline. None of the four wins on every axis. Anyone telling you otherwise is selling something.

Streaming Conformer models: the real-time tradeoff in meeting conditions

Conformer architectures pair convolutional layers, good at local acoustic detail, with self-attention, good at longer-range dependencies across an utterance. Streaming variants process audio as it arrives instead of waiting for a speaker to finish, which is the entire point for live captions, real-time translation, or a voice command that has to fire mid-meeting rather than after the fact.

The throughput numbers aren't subtle. NVIDIA's Parakeet CTC 1.1B posts a real-time factor of 2793.75: it processes audio roughly 2,800 times faster than it takes to play back. Whisper Large v3, on the same measure, hits 68.56. Yet the Open ASR Leaderboard shows accuracy between them barely moves, 6.68% WER for Parakeet against 6.43% for Whisper Large v3. Parakeet gives up almost nothing in accuracy for a throughput gain measured in orders of magnitude, on clean audio.

Meetings aren't clean audio, and that's exactly where the streaming bet stops paying off. A streaming Conformer working in 960-millisecond chunks hits 6.47% WER on LibriSpeech test-other, against 3.9% for a non-streaming Conformer paired with a language model on the same data, a figure from an arXiv paper on low-latency voice agents cited by Deepgram. A streaming model can't attend to audio that hasn't happened yet. It has to commit to a transcript before it knows what comes next, and that structural limit is exactly where crosstalk and mid-sentence reversals do the most damage: the model locks in a segment before it can hear the interruption, and there's no walking it back once the audio has committed.

Language coverage narrows the case further. Parakeet has historically focused on English. Current releases such as Parakeet-TDT-0.6B-v3 and Parakeet RNNT 1.1B cover 25 European languages, useful for a multinational team but still leaving most of the world outside that set. The Open ASR Leaderboard flags this as a pattern across the whole field, not just Parakeet: strong performance in a narrow language band, weak coverage everywhere else.

Decoder choice compounds it. CTC and TDT decoders run several times faster than decoders built on large language models, in exchange for a modest bump in error rate, a fine trade when someone's waiting on live captions and a bad one when nobody is. For after-the-fact meeting transcription, the speed advantage buys nothing. Streaming Conformers should be ruled out for that job outright, not weighed as a maybe. They're built to solve a different problem, and they solve it well. Meeting transcription just isn't that problem.

Encoder-decoder transformers: high accuracy at rest, and the on-device shift for sensitive meetings

Encoder-decoder transformers split the job cleanly, with an encoder building a high-dimensional representation of the audio and a decoder handling temporal alignment while generating text, and the whole thing trains end to end as one sequence-to-sequence system. Whisper set the template. Whisper Large v2 trained on 680,000 hours of weakly supervised and labeled audio, and that scale is a big part of why encoder-decoder transformers became the default for high-accuracy transcription.

Whisper Large v3 covers 99 languages, the strongest multilingual open baseline according to the Open ASR Leaderboard, though fine-tuned or distilled versions like Distil-Whisper and CrisperWhisper often beat the original on English-only benchmarks. The same analysis puts Whisper Large v3 ahead of other open models specifically on long-form audio, which matters because a meeting is close to the platonic ideal of a long-form transcription problem.

The real shift in this category over 2025 and 2026 has nothing to do with accuracy scores and everything to do with where the model runs. WhisperKit, described in a July 2025 paper, runs Whisper Large v3 Turbo (roughly 809 million parameters) entirely on Apple hardware, matching or beating GPT-4o-transcribe on accuracy without ever sending audio to a server. That changes what's possible for healthcare, legal, and financial services meetings, where compliance rules often mean audio can't leave the device. Frontier-quality transcription no longer needs a network call, and that's a genuinely different product constraint than it was a year earlier. Any vendor still pitching cloud-only transcription for a regulated meeting is pitching last year's constraint.

Alibaba has pushed a related evolution with Fun-ASR (September 2025) and Qwen3-ASR (January 2026), folding LLM components and reinforcement learning into what started as a pure encoder-decoder lineage. Both target a well-documented weakness: models that look excellent on open benchmarks often fall apart once tested against real industry evaluation sets, where vocabulary and acoustic conditions stop being clean.

None of this touches the meeting-specific problems, though. Encoder-decoder transformers, even the best of them, were never built to handle overlapping speakers as part of the core architecture. Diarization runs bolted on as a separate stage, carrying its own error budget on top of transcription error. Chunking strategy differs from vendor to vendor and is rarely published in detail, which makes it hard to know how any given model handles a 90-minute recording once it blows past whatever window it was trained on.

Speech LLMs: what happens when the language model processes audio directly

Traditional voice AI pipelines chain stages together: ASR produces a transcript, diarization assigns speakers, then NLP does whatever semantic work the product needs. Speech LLMs collapse that chain. Instead of taking a transcript as text input, models like GPT-4o and Gemini 1.5 attend directly to the acoustic signal alongside text context, reasoning over sound and language in one pass instead of three sequential ones.

The pattern currently topping accuracy leaderboards pairs a Conformer encoder with an LLM decoder. NVIDIA's Canary-Qwen 2.5B, released June 2025, combines a FastConformer encoder with a Qwen3-1.7B decoder and tops the Open ASR Leaderboard at 5.63% average WER. IBM's Granite-Speech-3.3-8B and Microsoft's Phi-4-Multimodal-Instruct sit near the top of the same board. Qwen3-ASR, from Alibaba's Qwen team, is a multilingual open-source model covering 52 languages and dialects.

GLM-ASR-Nano-2512, with a modest parameter count, posts the lowest average error rate (4.10) among comparable open-source models, with a particular edge on Wenet Meeting, a benchmark the source describes as reflecting real-world meeting conditions with background noise and overlapping speech. It was trained specifically for robustness on low-volume audio. OpusLM, released June 2025, joins the same wave: an open unified speech language model family treating speech and language as one modeling problem instead of two.

A model hearing raw audio directly, instead of a transcript stripped of everything but words, could pick up prosody, speaker affect, and other paralinguistic signal that vanishes the moment audio gets flattened into text. Whether current Speech LLMs actually cash that in on meeting audio specifically is a claim the available benchmarks don't settle either way, and anyone selling certainty here is reading further into the numbers than the numbers support.

There's a cost, and it's the mirror image of the streaming Conformer's advantage. LLM decoders run slower than CTC or TDT decoders, the same throughput gap as before, just felt from the other side. Speech LLMs trade speed for reasoning depth. As of November 21, 2025 there were 150 Audio-Text-to-Text models and 27,000 ASR models on Hugging Face Hub, and the overwhelming majority get evaluated on short-form English audio, not meetings. Which of these architectures actually holds up on a full-length meeting recording stays an open question the leaderboards don't answer, and won't until someone builds a benchmark out of real meeting audio instead of audiobook chapters.

LLM post-processing hybrids: where the industry has landed for production meeting transcription

Most meeting transcription products running in production today are pipelines, and the shape is consistent across vendors. Audio gets captured, through a meeting bot or at the device. An ASR model, usually an encoder-decoder or Conformer variant, turns it into text. A separate diarization model assigns speaker labels, carrying its own independent error budget. An LLM then handles the post-processing: summarization, action item extraction, topic detection, formatting. The output lands wherever the team actually works.

This structure wins for a plain reason: each stage gets swapped, tuned, or replaced on its own schedule. A proven LLM handles the semantic layer, a proven ASR model handles the acoustic layer, and no single unified architecture yet reliably does both at production quality across a full-length meeting. A proven LLM handles the semantic layer, a proven ASR model handles the acoustic layer, and no single unified architecture yet reliably does both at production quality across a full-length meeting. It's the only approach that's actually shipped at scale, and the reason is boring but decisive: modularity beats elegance when the acoustic layer and the language layer are still both unsolved.

Microsoft Teams runs this pattern, with MAI-Transcribe through Azure Speech in Foundry Tools handling ASR, and Copilot layered on top for summarization, task extraction, and sentiment analysis (sentiment analysis specifically ships through the Copilot for Sales add-on, not standard Teams Copilot). Zoom AI Companion runs a comparable hybrid: Zoom's own ASR engine paired with federated LLMs, including proprietary SLMs and Nemotron-based models, for the summarization layer. Mistral's Voxtral Realtime, announced for February 2026, targets the same tension between offline transcription quality and the constraints of real-time streaming, another entrant in the same production-hybrid category.

The pipeline inherits diarization's weakest link no matter how good the ASR or LLM stage is. research on state-of-the-art diarization consistently finds substantial error rates, with overlapping speech driving much of it. A pipeline can't outrun its worst stage, and diarization is very often that stage.

LLM post-processing doesn't fix ASR mistakes. It amplifies them. Mishear a client's name or drop a negation in the transcript, and the LLM summarizing it carries that error straight into the action item, the decision log, or the CRM field, with nothing downstream to catch it. The ASR stage sets a ceiling nothing after it can raise. Even the strongest LLMs are not immune to errors on meeting-specific summarization tasks. A meeting turns out to be a genuinely harder summarization target than a document, and not simply because it runs longer.

Where this architecture has actually improved is structure. Leading tools now output decisions, action items with named owners and deadlines, fields formatted for a CRM, instead of a wall of undifferentiated text. Competition has shifted from who transcribes most accurately to who organizes the output best, and that shift by itself tells you the transcription layer has matured about as far as the current architectures allow, gaps and all.

The four meeting-audio conditions that separate architectures in practice

Overlapping speech is unsolved, industry-wide, and no vendor pitch should get to imply otherwise. Diarization error at the state of the art is driven substantially by simultaneous speech, not any other factor. Streaming Conformers are particularly exposed here, since they have to commit to a transcript chunk before an overlap even has a chance to resolve. Encoder-decoder transformers and Speech LLMs, working from the full recording, have more room to use later context to untangle who said what, but neither treats overlapping speech as solved, and nothing in the current research describes an architecture that does.

Variable acoustics punish every architecture that wasn't trained on audio resembling those conditions: speakerphone pickups, laptop mics, someone dialing in over a compressed codec. Training data composition, more than architecture family, decides how a model handles this. A model trained heavily on studio-quality audiobook narration struggles with a call center headset regardless of whether it's a Conformer or a Speech LLM underneath, simply because it never saw that acoustic profile during training.

Domain vocabulary, the product names and internal jargon absent from general training data, stays a persistent weak spot across every architecture. It's the specific gap Fun-ASR and Qwen3-ASR were built to close, between open benchmark performance and real industry evaluation sets, and that gap exists precisely because benchmark vocabulary and meeting-room vocabulary barely overlap.

Long-form duration separates the architectures most cleanly of the four. Most benchmarks still test short clips. Actual meetings run considerably longer, and that mismatch means chunking strategy, an engineering decision most vendors don't disclose in detail, ends up mattering as much as the model's raw accuracy. Whisper Large v3's edge on long-form audio stands out precisely because so few benchmarks bother testing for it. Until meeting-length audio becomes standard in the evaluation suite, headline WER numbers will keep measuring something other than what a 90-minute meeting actually demands.