Modeling Podcast Transcripts for Search: Segments, Speakers and Timestamps

If you have ever tried to build search over a pile of podcast transcripts, you probably started the obvious way: dump each episode into one big text field, index it, and call it done. It works for about a day. Then someone searches for a phrase, gets a match somewhere in a ninety-minute episode, and asks the reasonable question: where? Who said it, and at what minute?

This post walks through a data model that answers those questions without much ceremony. It is the shape I keep coming back to when I work with long-form audio turned into text.

The unit is the segment, not the episode

A transcript is not a document in the usual sense. It is a sequence of short spoken segments, each with a start time, an end time and a speaker. Treat that sequence as the primary data and the full text as a derived view.

A minimal segment record looks like this:

{
  "episode_id": "ep_2026_10_12",
  "seq": 418,
  "start_ms": 1834200,
  "end_ms": 1841950,
  "speaker": "Guest",
  "text": "We tried that in the first version and it was too slow."
}

Keep seq even though you have timestamps. Speech recognition output sometimes produces overlapping or zero-length segments, and a plain integer order is the only thing you can always trust when you rebuild the text.

Speakers are labels, then names

Diarization, the step that separates voices, produces anonymous labels such as SPEAKER_00 and SPEAKER_01. Mapping those to real names is a separate, editable step. Store the mapping per episode in its own table:

{ "episode_id": "ep_2026_10_12", "label": "SPEAKER_01", "name": "Guest", "confirmed": false }

Two practical reasons. First, the mapping is often wrong for the first minute, when the host introduces the guest and voices overlap. Second, a correction should not require rewriting thousands of segments. Join at query time, or denormalise into the search index only when you rebuild it.

Index windows, not single segments

Individual segments are often too short to search well. "Yes, exactly" is a perfectly valid segment and a useless search hit. On the other hand, indexing the whole episode loses the location.

A good compromise is a sliding window: group consecutive segments into chunks of roughly 30 to 60 seconds, overlapping by one segment, and index each chunk with the start time of its first segment and the list of speakers inside it. A hit then points to a window, and the UI can scroll to the exact segment that contains the matched words.

The same windows work for embedding-based retrieval. If you later add question answering over episodes, retrieve windows, then quote the underlying segments with their timestamps so a reader can check the original audio.

Export formats are views too

You will be asked for subtitles. SRT and WebVTT are both simple cue lists with start and end times, and both can be generated from segments. WebVTT also supports a voice tag, so you can carry speaker names into the cue text. Generate these on demand rather than storing them; the segment table stays the single source of truth.

For reading, render paragraphs by merging consecutive segments from the same speaker, and show a timestamp at the start of each paragraph. That one rule turns a choppy machine transcript into something that reads like an interview.

Measure quality where users feel it

Word error rate is the standard metric for speech recognition, but users rarely complain about it directly. They complain when a name is wrong, when a quote is assigned to the wrong person, or when a timestamp lands thirty seconds early. Add three cheap checks to your pipeline:

• Spot-check the first two minutes of each new show for speaker mapping.
• Keep a small list of proper nouns per show and flag segments where a near-miss spelling appears.
• Sample a few search hits and verify that the timestamp plays the matched words.

Where the transcripts come from

Building your own speech pipeline is a project of its own. For prototypes and research, I often start from an existing service and focus on the search side. Readpodcast AI is one option: you paste a public RSS feed or a Spotify or Apple Podcasts link and get transcripts with speaker turns and timestamps, which maps neatly onto the segment model above. Whatever source you use, keep the raw segments, keep the speaker mapping separate and keep the timestamps attached to every quote. Your future search features will thank you.

Andreas Keller works on Readpodcast AI.