From diarization segments to a readable transcript

Speech-to-text tools with speaker separation usually return a list of short segments. Each one has a speaker label, a start time, an end time and a few words. That shape suits a player and reads badly as a document:

SPEAKER_00 00:00:01.2 Thanks for
SPEAKER_00 00:00:02.1 joining us today.
SPEAKER_01 00:00:03.4 Happy to
SPEAKER_00 00:00:04.0 So first
SPEAKER_01 00:00:04.3 be here.

The speaker changes four times in three seconds, and a human reader sees noise. A small post-processing step fixes most of it. Here is one that runs on plain arrays, with no dependencies.

A two-person recording setup. Photo: Austin Distel, Unsplash

Sort first

Diarization output is not always in order, especially after merging results from several chunks of a long file. Sort by start time before doing anything else. Everything below assumes that order.

Merge neighbours from one speaker

Two segments from the same speaker belong together when the pause between them is short. A gap threshold of about 1.5 seconds works as a starting point. Longer pauses often mark a new thought, so they stay separate.

Clips on an editing timeline, each with its own start and end. Photo: Peter Stumpf, Unsplash

Absorb short interruptions

"Yeah", "right" and "mm-hm" break one person's sentence into pieces. They are rarely worth a line of their own. The rule used here: a segment shorter than 0.7 seconds, with the same speaker on both sides, is an interruption. Its text is dropped from the flow and the two sides are joined.

This is a judgement call. In a legal or research transcript, the backchannel can be the evidence, so make the rule optional and leave it off for those jobs.

The code

function tidy(segments, { maxGap = 1.5, minTurn = 0.7, dropBackchannel = true } = {}) {
  const sorted = [...segments].sort((a, b) => a.start - b.start);

  const kept = dropBackchannel
    ? sorted.filter((s, i) => {
        const prev = sorted[i - 1];
        const next = sorted[i + 1];
        const short = s.end - s.start < minTurn;
        const between = prev && next && prev.speaker === next.speaker && prev.speaker !== s.speaker;
        return !(short && between);
      })
    : sorted;

  const turns = [];
  for (const s of kept) {
    const last = turns[turns.length - 1];
    if (last && last.speaker === s.speaker && s.start - last.end <= maxGap) {
      last.end = Math.max(last.end, s.end);
      last.text += " " + s.text.trim();
    } else {
      turns.push({ speaker: s.speaker, start: s.start, end: s.end, text: s.text.trim() });
    }
  }
  return turns;
}

function stamp(seconds) {
  const t = Math.floor(seconds);
  const pad = (n) => String(n).padStart(2, "0");
  return `${pad(Math.floor(t / 3600))}:${pad(Math.floor(t / 60) % 60)}:${pad(t % 60)}`;
}

const render = (turns, names = {}) =>
  turns.map((t) => `[${stamp(t.start)}] ${names[t.speaker] || t.speaker}: ${t.text}`).join("\n\n");

Trying it

const segments = [
  { speaker: "S0", start: 1.2, end: 2.0, text: "Thanks for" },
  { speaker: "S0", start: 2.1, end: 3.3, text: "joining us today." },
  { speaker: "S1", start: 3.4, end: 3.9, text: "Happy to" },
  { speaker: "S1", start: 4.3, end: 5.0, text: "be here." },
  { speaker: "S0", start: 5.2, end: 7.0, text: "Let's start with the budget," },
  { speaker: "S1", start: 7.0, end: 7.3, text: "Right." },
  { speaker: "S0", start: 7.4, end: 9.0, text: "because it changed in March." },
];

console.log(render(tidy(segments), { S0: "Host", S1: "Guest" }));

This prints three turns:

[00:00:01] Host: Thanks for joining us today.

[00:00:03] Guest: Happy to be here.

[00:00:05] Host: Let's start with the budget, because it changed in March.

Limits worth knowing

• Overlapping speech stays out of order, because sorting by start time cannot untangle it. Marking such turns for a human is better than guessing.
• The thresholds depend on the speakers. A slow lecture needs a larger gap than a fast sales call.
• Speaker labels are local to one run. SPEAKER_00 in one file can be a different person in the next, so names have to be mapped per file.

Keep the raw segments next to the tidy version. When someone disputes a quote, the original timing is what you check against the audio.

Hosted services do this step for you. AudioVText, for example, returns transcripts with timecodes and separated speakers, with Russian as its primary language and 99 languages supported. A script like this one is still useful when you need your own rules for a batch of files.

Speaker turns with timecodes in a hosted transcription tool (Russian interface)