Speaker Diarization in Overlapping Speech

Speaker diarization answers one question: who spoke when. That sounds simple until two people talk at once, which happens constantly in real meetings, and that single moment is where most diarization systems fall apart. Everything downstream of that failure, the transcript, the summary, the action items, the CRM entry, inherits the mistake and never fixes it on the way down.

This isn't a solved problem with a few rough edges left to sand off. Overlapping speech remains one of the central unsolved challenges in the field, and it explains most of the gap between AI meeting tools that get things right and tools that quietly don't. Anyone shopping for one of these tools should treat overlap handling as the real test, more telling than the transcription accuracy number sitting on the landing page.

Diarization splits an audio stream into segments and tags each one with a speaker identity. Get that tagging wrong, and the transcript layer built on top goes wrong with it, along with whatever summary or task list gets pulled from that transcript. If a system can't tell Sarah's voice from James's, no downstream model can reliably say Sarah agreed to own the deliverable. The error doesn't stay put. It moves through the pipeline the way a rumor moves through an office, picking up distortion at every retelling.

Why overlapping speech breaks the standard diarization pipeline

Overlap means any stretch of audio where two or more speakers talk at the same time. Standard diarization pipelines aren't built for that. They segment audio, pull a speaker embedding (a numeric fingerprint of vocal characteristics) from each segment, then cluster those embeddings into speaker groups. The whole approach rests on one quiet assumption: one dominant voice per segment.

Overlap breaks that assumption. Feed a segment with two voices talking at once into an embedding extractor, and it won't produce two clean fingerprints. It produces one blurred, mixed vector that resembles neither speaker's actual voice profile. Cluster on that, and the system guesses, often wrong.

Most commercial pipelines still rely on simplified heuristics rather than principled overlap handling, and those shortcuts only hold up when the overlap is short and the speakers were already clearly established beforehand. Bring in a third voice mid-overlap, let the crosstalk run long, or catch it early in a meeting before speaker profiles have settled, and the heuristic falls apart.

None of this is some rare edge case. Overlapping speech occurs constantly in ordinary conversation, and any meeting with real cross-talk and backchanneling pushes it further still. Even a hypothetically flawless diarization system, one that never confuses two non-overlapping speakers, still misses every additional speaker in every overlap region if it has no way to handle overlap at all. The error scales directly with how much overlap the audio contains. Tools that ignore overlap fail on schedule, with errors scaling directly with how much overlap the audio contains. They fail on schedule.

What the diarization error rate actually measures, and why a single headline number misleads

Diarization error rate, or DER, adds up three kinds of mistakes: missed speech, false alarm speech, and speaker confusion, then divides that total by the overall duration of speech. It's the industry's go-to scorecard, and it's also routinely gamed. Plenty of published DER figures quietly exclude overlapping speech and the messy zones right at speaker turn boundaries, which happen to be exactly where diarization struggles most. Plenty of published DER figures quietly exclude overlapping speech and the messy zones right at speaker turn boundaries, an exclusion of exactly where diarization struggles most. That's the hard part of the problem getting scored around.

Strict and permissive scoring produce very different numbers from the same system. The most rigorous benchmark conditions score under the least forgiving rules: all speech scored, including overlaps, no speaker count fed to the system in advance, no oracle voice activity detection, no tuning to the test set, and no forgiveness collar around turn boundaries. A vendor who cites DER without naming these conditions is letting the number flatter the system, whether or not that's the intent.

The instability across datasets tells the real story. Pyannote's benchmark table runs the same open-source diarization pipeline across twelve different datasets and returns DERs ranging from 8.9% up to 46.8%. Nothing about the pipeline changed between runs, only the audio did, and that's roughly a five-fold spread on identical code. A single headline DER tells a buyer almost nothing about how a system will behave on their own meeting recordings. Anyone treating it as a stand-alone spec is trusting a number that was never built to travel outside its own test set.

Every other speaker-scoped metric sits on top of this same shaky foundation. Talk ratios, interruption counts, sentiment tied to a specific speaker, compliance flags on a regulated call: all of it degrades the moment diarization mislabels a speaker. Ask a vendor what their DER looks like on audio that resembles the meetings a team actually runs, not a clean benchmark set, and most vendors, pressed on it, simply don't have an answer ready.

Overlap detection in modern systems before assignment

Before a system can correctly assign overlapping speech to the right speakers, it has to know the overlap is there in the first place. That's a distinct subtask called overlap speech detection, or OSD, and it's treated separately from assignment for good reason: a system that fails to flag a region as overlapping will force it onto a single speaker and get it wrong by design, every single time.

Earlier OSD approaches leaned on energy-based and spectral audio features, later paired with CNNs and LSTMs. Two things held that generation back. Real-world audio with labeled overlap regions is scarce, so a lot of training relied on simulated overlap, mixtures built by artificially layering clips from a widely used speech corpus. Models trained that way tend to stumble once they hit messier, real conversational audio: a textbook case of domain mismatch between the training data and the world the model actually has to work in.

A more recent approach, described at Interspeech 2025 by researchers from Northwestern Polytechnical University and Li Auto, folds a pretrained WavLM self-supervised speech model in as the front end, adds a speaker attention module, and trains the whole thing with what the researchers call a progressive strategy. Voice activity detection logits mask the encoder's hidden states before the overlap detector ever sees them, so the system uses voice activity information to inform overlap detection instead of running two blind, independent classifiers side by side. The speaker attention module pulls frame-level speaker representations and merges them with the WavLM acoustic features, patching a real weakness in earlier systems: single-speaker segments and overlapping segments often look statistically similar without added speaker-specific context to tell them apart.

The payoff is an F1 score of 82.76% on the AMI test set, reported as state of the art for the task. That's a strong number, and it also means something close to one in six overlap regions still gets missed entirely or falsely flagged, and every one of those errors carries straight into the assignment stage that follows. Self-supervised front ends like WavLM and wav2vec 2.0 now consistently beat handcrafted acoustic features here, and that's become the standard starting point for anyone building a competitive OSD system.

Two architectures for assigning overlapping speech: modular pipelines vs. end-to-end models

Once overlap gets flagged, something has to decide who's actually talking. Two broad architectures compete here, and the direction of travel favors one of them, even if neither has won outright yet.

Modular, or cascaded, pipelines break the job into stages: voice activity detection, segmentation, speaker embedding extraction, clustering, then a separate overlap-handling step bolted on afterward. That last step usually runs target-speaker voice activity detection, or TS-VAD, which takes an overlap region and a set of candidate speaker embeddings and estimates per-speaker activity within that region. The advantage of modular design is flexibility: each stage can be swapped or retuned on its own, and clustering methods like agglomerative hierarchical clustering, VBx, or spectral clustering are well studied and well understood. The downside is that errors compound. The overlap-handling step is only ever as good as everything that fed into it.

End-to-end models take a different route. A single neural network handles segmentation and speaker assignment together, trained to output activity probabilities for every speaker at once rather than picking one label per segment. That design handles overlap natively, since nothing forces the model to choose just one speaker for a given moment. One recent entry, EEND-TA, is a unified, non-autoregressive model that scales its pretraining through multi-speaker simulation and has been benchmarked across AliMeeting-far, AliMeeting-near, AMI-Mix, AMI-SDM, DIHARD III, and MagicData RAMC. It posts a DER of 14.49% on DIHARD III, reported as ahead of every other end-to-end diarization model as of early 2025. End-to-end systems also tend to run with a lower real-time factor than modular pipelines, which matters once a system has to process meetings live instead of after the fact.

For meetings with real, sustained overlap, end-to-end is the better bet, and treating modular clustering as the default for that kind of audio is caution dressed up as rigor when it's actually just outdated. A third path has emerged that concedes as much while hedging on the low-overlap case: hybrid systems that pick their method per meeting based on how much overlap is actually present. The winning system in MISP 2025's Track 2 and Track 3, built by Beijing Fosafer, measured overlap proportion for each meeting and routed accordingly, sending meetings with less than 1% overlapping speech to VBx clustering and everything else to an end-to-end WavLM-based segmenter. The logic holds: VBx performs best when speakers are cleanly separated, and end-to-end models handle the messy assignment problem in overlap regions better than clustering ever will. A separate fourth-place system, also from Fosafer, added multi-channel fusion using DOVER-lap to combine results across eight microphone channels. Hybrid routing is the most defensible answer available right now, not because it's elegant, but because neither pure modular nor pure end-to-end design wins across every condition on its own.

Current benchmark results and the gaps that remain

The MISP 2025 Challenge, built around multi-channel, real meeting audio, gives a useful snapshot of where things stand. System S5, using overlap decision fusion, posted a DER of 7.99% on the dev set and 9.09% on eval. System S6, which fused those same results across eight microphone channels, brought that down to 7.75% on dev and 8.88% on eval, a 42.78% relative reduction compared to the baseline. On the combined transcription-and-speaker-labeling track, the same team's system posted a cpCER of 11.56% and took first place. On transcription alone, a CER of 9.48%, also first.

Other challenges tell a similar story, and it becomes most visible in the gap between DER and JER. The BUCEA system at VoxSRC 2022, tested on VoxConverse, a dataset built from YouTube-sourced multi-speaker audio, scored a DER of 5.48%, but its Jaccard error rate came in at 32.1%. That gap matters, since JER weights overlap regions more heavily than DER does. A low DER sitting next to a much higher JER is a fairly direct signal of how much harder assignment gets once real overlap enters the picture. DKU-DukeECE, also at VoxSRC 2022, used TS-VAD for overlap detection, fused four separate systems together with DOVER-lap, and landed a DER of 4.75%, good enough for first place in that track.

There's a ceiling on the multimodal side, too. MISP 2025 tested whether video, specifically lip tracking, could give diarization and speech recognition an extra edge. At least one top team, Fosafer, found the opposite: blurred lip footage and poor lighting caused the extracted visual features to actively interfere with the audio-based system rather than help it. Adding a modality can just as easily degrade performance as improve it. It's a bet that the new signal is cleaner than the one already in hand, and that bet doesn't always pay off.

None of these numbers came from a laptop mic in a cramped hybrid meeting room. They came from controlled challenge conditions: known speaker counts, clean recording setups, carefully engineered multi-channel rigs. The gap between benchmark DER and whatever a production system delivers on ordinary business audio persists across deployments, and most vendors don't quantify it. Pyannote's five-fold spread across twelve datasets, from 8.9% to 46.8%, is still the most honest public illustration of how wide that gap gets. A separate challenge, DISPLACE 2023, adds another dimension: multilingual, code-switched conversation recorded far-field. Diarization systems tuned for monolingual audio tend to fail substantially under those conditions, and that describes a large share of enterprise meetings happening globally right now.

Overlap errors and their degrading effect on downstream output in AI meeting tools

The failure doesn't stop at the transcript. It moves forward: a diarization error produces a transcript speaker-label error, which produces a summarization error, which produces a misattributed action item, which produces a wrong entry in a CRM record. Each stage trusts the one before it completely, with no mechanism to check its work.

Action items sit at the sharpest edge of this. Roughly 47% of the action items actually discussed in a meeting never get captured by anyone or anything at all, and among the ones that do get captured, wrong diarization means they're frequently pinned to the wrong person. Human memory doesn't reliably catch that error afterward, either: 63% of workers can't recall every task assigned to them from earlier the same day. If the system gets attribution wrong and no one notices, that commitment just disappears.

Talk ratio, interruption counts, sentiment tagged to a specific speaker, compliance flags on regulated calls: all of these run on the same underlying speaker labels, and all of them fail together the moment those labels are wrong. In voice-agent research from AssemblyAI, 52.5% of builders named accuracy as their single biggest challenge, and diarization accuracy doesn't separate cleanly from transcription accuracy in practice. They degrade together, or they hold together.

CRM integration raises the cost of getting this wrong. A note pushed into the wrong contact's record, or a commitment dropped entirely because the wrong speaker got credited, is worse than no note at all. It creates false confidence that the record is complete when it isn't, which is a harder failure to catch than an obvious gap someone would notice and go check by hand.

Backchannels distort turn counts on their own, quietly. A quick "yeah," "right," or "uh-huh" dropped into someone else's sentence is technically an overlap, and it gets miscounted as its own speaker turn often enough to inflate one person's apparent talk time while shrinking another's, even when the primary transcript reads perfectly clean. Meeting intelligence, as a category, has to clear a higher accuracy bar than general-purpose transcription for exactly this reason. A 5% word error rate sounds fine on paper, until that 5% happens to land on the one sentence where someone commits to a contract term.

What to look for in an AI meeting tool's overlap handling

Ask a vendor two things: how the system handles overlapping speech specifically, and what kind of audio that number was measured on. Most vendors will answer the first question and quietly dodge the second, and that dodge is itself the answer.

Plain transcription accuracy has largely become a commodity: leading tools in 2025 clear 90 to 95%-plus accuracy on clean English audio. The real differentiation now lives in diarization, specifically in how a system behaves once speakers start talking over each other. Published word error rate figures, typically 4 to 7% for top providers on high-quality audio and climbing past 15% in harder conditions, don't separate recognition error from diarization error. A system can post an excellent WER and still hand over a transcript where half the quotes are attributed to the wrong person, and a buyer reading only the WER would never know it.

A more honest number is diarization cpWER on a real benchmark like CHiME-6 or DIHARD III. AssemblyAI's Universal-3.5 Pro reports a diarization cpWER of 30.17, ahead of Deepgram Nova-3 EN at 37.92 and ElevenLabs Scribe v2 at 35.26. Those figures actually show whether a speaker label can be trusted, which a headline transcription-accuracy number on a landing page never will.

How the audio gets captured matters just as much. A bot that visibly joins a call can often pull audio straight from the meeting platform's own stream, which tends to run cleaner. Silent desktop recording captures system audio directly but stays more exposed to local mic bleed and room noise. That choice affects more than consent and how participants perceive the tool: it affects raw audio quality, and audio quality feeds diarization performance directly. Multi-channel setups, like a dedicated conference room mic array, produce meaningfully better results than a single laptop mic picking up a whole room. MISP 2025's eight-channel fusion, which delivered a 42.78% relative DER reduction over a single-channel baseline, shows how large that gap gets. In-person meetings are harder still, since there's no platform-level speaker metadata or per-participant audio stream to lean on, only raw room audio and whatever the diarization model can pull out of it on its own.

Anyone evaluating an AI meeting tool should push past the landing-page number and ask a short set of pointed questions before signing anything. Does the system run dedicated overlap detection ahead of speaker assignment, or is it quietly relying on a before-and-after proximity guess dressed up as a model? Is the underlying diarization approach end-to-end, modular, or hybrid, and has it actually been tested on audio that resembles the meetings it'll be used on, not just a clean benchmark set? And does it support diarization across the number of languages a global team actually needs, not just transcription? Code-switched, multilingual conversation is a condition where diarization systems tuned for monolingual audio tend to fail substantially.