0

Retiming generated video to a cloned voice, and other pipeline notes from a 1:46 AI film

A frame from our founder film: the founder in a loft office holding a coffee. AI-generated, nothing was filmed.

A film with no footage still needs a pipeline. Ours produced a 1:46 launch film from three kinds of input, none of them filmed: shots from generative models, a voice clone for the narration, and an HTML page rendered frame by frame for the screen animations. In it, our founder walks a street, works late and speaks to the camera. The result is on YouTube (1:46).

Iteration count: five versions and roughly 800 model runs. Most failures did not show inside a single generated clip; they surfaced in the edit. Below is the pipeline as it stands, then each stage and what broke there.

The pipeline as it stands

Stage Input Rule we kept
Talking shots 10+ generated takes per shot pick the take whose pace fits the line; warp the video to the narration via forced-alignment anchors
Finishing raw and upscaled frames skin-colored areas from the raw frame; skin lift toward face brightness capped at +24%; eyes and brows protected
Narration clone sample with the missing band above 11 kHz restored generated voice used raw, never stretched
Music a single track from a single prompt no stretching; pauses on bar lines; trimmed in whole 4-bar loops
Screens an HTML page rendered frame by frame
Review a multimodal model plus human eyes audio A/B must hold in both orders; lip sync checked by eye

Talking shots: retime the picture, not the audio

No generated talking shot matched our narration's timing. Time-stretching the audio was one of the things that degraded the cloned voice (see the narration section), so the audio stays fixed and the video is retimed.

Forced alignment runs on two signals, the clip's own audio and the narration line, and returns word and character timestamps for each. Matching timestamps become anchor points, and the video is warped through them.

Measuring 34 mouth closures over 11 clips exposed an offset: the generated lips led their audio by about 90 ms. We apply only about 60 ms of shift, deliberately leaving the lips slightly early, because that looks natural on playback.

Take selection is the other half. Each talking shot was generated 10 or more times, and we kept the take whose pace already matched the line. Shortened pauses can pass in standing shots; in walking shots they show.

Finishing: raw-frame skin over an upscaled frame

Raw generations came out soft, so an upscaler ran on every shot. Sharpening is not selective, so skin picked up lines, stubble shadow and a darker tone. Version 4 drew a blunt note from the founder: "Hands and skin look old, wrinkled and too dark."

Each frame is now a composite:

  1. Take skin-colored areas from the raw frame.
  2. Take everything else from the sharpened frame.
  3. Raise skin brightness toward the face's measured value, up to a +24% cap.
  4. Keep the eyes and brows protected.

The founder chose the lift strength from a sheet of four side-by-side columns. One shot had a side effect, a warm brick wall that softened slightly. He reviewed it and accepted it.

Narration: restore the sample, then stop processing

v1 used the clone output untouched, and it sounded good. v2 time-stretched lines to match lip movement. v3 added sample cleanup, EQ and a de-esser. Perceptually it degraded in stages, through muffled and echoey to phone-call quality.

Post-processing is now off entirely. One fix remains on the input side: the source recording had nothing above 11 kHz, so the clone sample gets that missing band restored before cloning. Whatever the model generates is used as is.

Music: one generation, edited on the grid

Building the score as chunks from a composition plan failed: stitched together, the chunks played back as five distinct songs. The current score is a single generation from a single prompt, never time-stretched. The edit follows the music's grid instead: film pauses were shifted onto bar lines, and where the music was too long, it lost whole 4-bar loops.

Review: a model judge needs guardrails

The review stage used a multimodal model as a comparator. Its still-image judgments held up. On audio A/B comparisons it showed position bias, preferring whichever clip came first, so an audio verdict only counted if it survived swapping the order. As a sketch:

def stable_pick(judge, a, b):
    # judge(x, y) returns the clip it prefers: x or y
    first = judge(a, b)
    second = judge(b, a)
    return first if first == second else None  # None: the order decided, discard

Its lip-sync judgment failed outright: a 400 ms offset came back as "in sync". Lip sync stays with humans: we step through frame strips at every m, p and b, where the lips close.

Practices worth carrying over

  • Playbook first. From day one, each note from the founder became a rule the next project starts from.
  • Every artifact stored. Clips, timings, music and the render page are kept, so one scene can be redone without redoing the film around it.
  • Frames, not summaries. A summary of the output is not the output.

The models themselves were all called through Wiro, our API for image, video, audio and language models.

The first version of this article ran on dev.to.


All rights reserved

Viblo
Hãy đăng ký một tài khoản Viblo để nhận được nhiều bài viết thú vị hơn.
Đăng kí