An evaluation set for diarization that does not flatter the model

Diarization error rate computed over a whole call is dominated by the easy minutes. A system can hold a steady speaker map through long single-speaker stretches, lose every overlap in the call, and still report a respectable number.

The failures that matter in a multi-party call are concentrated in a small share of the audio: two people starting at once, cross-talk carrying one speaker into another's microphone, and backchannels — a word or two of agreement — that are shorter than the window most systems need to attribute a speaker at all. An evaluation set that samples audio uniformly barely contains them.

The set was built for those cases specifically, across both in-person recordings and online calls with up to ten speakers, so the two conditions could be scored separately rather than averaged into one figure.

Segmentation and speaker-boundary generation were defined as a process rather than a judgement call, so that two annotators presented with the same overlapping passage would place the boundary the same way. The annotation protocol and the tooling annotators worked in were written alongside it, and every batch went through quality control on the labels before it entered the set.

The finished set is used to evaluate diarization models on the conditions it was built around — overlapping speech, cross-talk and short backchannels — which makes the differences between models visible where they were previously averaged away.

Because the protocol and tooling are in-house, the set extends: new conditions can be added without renegotiating what a speaker boundary means.