Six Video Model Providers, One Polling Loop: Notes on Normalizing Async Generation APIs

Draft for Viblo — English, original, not published elsewhere

Text generation converged on a shape. Whatever you think of OpenAI's API design, POST /v1/chat/completions won, and every serious provider either speaks it natively or ships a compatibility layer. You can swap GPT for Claude for Qwen by changing a string.

Video generation has no such convergence, and I don't think it's going to get one soon. I spent a while building a normalization layer over six video model families — Kling, Seedance, Wan, Vidu, PixVerse, and HappyHorse — and the interesting part was not what I expected going in.

The transport layer is the easy part

Every video model works roughly the same way at the transport level, because physics forces it to. Generating five seconds of 720p video takes tens of seconds to minutes. No provider is going to hold an HTTP connection open that long, so they all do the same thing: accept a job, return an ID, make you poll.

So the envelope is obvious:

POST /v1/videos/generations
{
  "model": "wan2.7-t2v",
  "prompt": "A cinematic drone shot over a forest at sunrise",
  "duration": 5,
  "quality": "720p",
  "aspect_ratio": "16:9"
}

-> {"id": "task_xxxxx", "status": "pending"}

GET /v1/tasks/task_xxxxx

-> {"status": "completed", "results": ["https://.../output.mp4"]}

Two calls. Poll every 3–5 seconds. If you've written a job queue client before, you've written this.

The temptation early on is to force this into the chat-completions shape, because then you get "OpenAI compatible" on the marketing page and existing SDKs Just Work. We tried. It's a bad idea, and I want to explain why, because the reasoning generalizes.

Chat completions is a request-response primitive with an optional streaming mode. Both modes assume the result arrives on the connection you opened. To make video fit, you have to either (a) block for two minutes and hope nothing in the path has a 60-second idle timeout, or (b) return a fake completion whose content is a task ID, which means the caller has to parse a message body to find out that the thing they asked for hasn't happened yet.

Option (b) is the one that looks clever in a design doc. In practice you've now got an API where choices[0].message.content sometimes contains prose and sometimes contains a job handle, and every consumer needs a branch. You haven't removed the polling code. You've hidden it behind a type that lies.

We kept video on its own endpoint pair. Text stays genuinely OpenAI-compatible; media does not pretend to be. Being honest about the seam turned out to be cheaper than papering over it — and it's the same call I'd make again.

The hard part is capability fragmentation

Here's the matrix that actually caused the work. These six families do not support the same operations:

Text-to-video and image-to-video are universal. Everything past that is a patchwork. Reference-to-video exists on four of six. Video editing on three. Keyframe-to-video only on PixVerse. Start-end-frame only on Vidu.

And look at the Kling row: reference-to-video and editing exist on O3 but not V3. Capability varies within a family, by version. That single fact kills the cleanest design.

The obvious approach is a capability enum on the family — KLING supports [t2v, i2v, ref2v, edit] — and you validate against it. It's wrong the moment a provider ships a version that drops or adds a mode. You end up with either a lie (the enum says a thing the model can't do) or a combinatorial mess of family-version pairs hardcoded in your validator.

What worked better: treat capability as a property of the model ID, not the family, and derive the family grouping for display purposes only. kling-v3-video-generation and a hypothetical O3 model ID are separate entries with separate capability sets that happen to share a vendor label in the UI. The family is a presentation concept. The model ID is the unit of truth.

This sounds obvious written down. It was not obvious while staring at six vendor doc sites that all organize themselves by family.

Three normalization decisions worth arguing about

1. Reject unsupported modes at submit, not at poll

If a caller asks Vidu for reference-to-video, you can find out at submit time that this is impossible. Do that. Return a 4xx immediately with the reason and, ideally, the list of models that do support the mode.

The alternative — forward it upstream and let the provider fail the job — is worse in a specific way: the failure arrives asynchronously, thirty seconds later, in a poll response, formatted however that vendor formats errors. You've converted a synchronous validation error into an async one for no reason. Now the caller's error handling has to cover "job failed because the model can't do this" as a runtime path, when it was knowable before the job existed.

2. Normalize terminal states hard, intermediate states loosely

Providers have wildly different intermediate vocabularies: queued, pending, submitted, processing, running, rendering, uploading, finalizing. Chasing a canonical intermediate taxonomy is a losing game — every new provider adds a state you didn't model.

But there are only three terminal outcomes anyone actually branches on: it worked, it failed, or it's still going. Collapse to completed / failed / everything-else-is-pending, and pass the provider's raw status through in a secondary field for anyone who wants to build a nicer progress bar.

The asymmetry is deliberate. Terminal states are a contract — clients write if (status === "completed") and that must never break. Intermediate states are telemetry. Locking down the contract while leaving telemetry open is what lets you add a seventh provider without a breaking change.

3. Poll interval belongs to the server's knowledge, not the client's guess

3–5 seconds is a reasonable default, but it's a default, not a law. A 5-second 720p clip and a 10-second 1080p clip have very different expected latencies, and the provider's queue depth matters more than either.

If you're building a client: don't hammer at a fixed 1-second interval because it feels responsive. You'll get rate-limited, and on a job that takes 90 seconds you've made 90 useless requests to save at most 4 seconds of latency.

If you're building the gateway: you know things the client doesn't — which provider, what queue depth, typical completion time for that model and duration. Expose that. A retry_after hint on the pending response costs nothing and moves the decision to where the information lives.

What still isn't solved

Some honest gaps, because posts like this usually skip them.

Parameter semantics don't fully normalize. quality: "720p" means something slightly different across vendors — different bitrates, different encoder settings, sometimes different actual pixel dimensions for the same label. You can normalize the name of the knob. You cannot make the outputs identical. Anyone benchmarking across providers needs to know the label is approximate.

Aspect ratio support is ragged. Not every model takes every ratio, and the failure mode varies: some reject, some silently letterbox, some crop. Normalizing this properly means per-model validation tables that have to be maintained by hand against vendor docs that change without notice.

Output retention is a policy question wearing an engineering costume. Result URLs point at hosted files. How long they live, whether they're signed, whether the upstream provider also retains a copy — these are decisions with legal weight, and they don't have a technically "correct" answer. Worth deciding explicitly and documenting, rather than inheriting whatever each upstream does by default.

Prompt portability is a myth. The same prompt string produces meaningfully different results across families, because they were trained differently and respond to different phrasings. A unified API makes it cheap to try another model. It does not make your prompt work there. Anyone selling model-switching as a zero-cost operation is eliding this.

The part I'd emphasize to anyone building something similar

The submit-and-poll envelope took a couple of days. The capability matrix and its version-level exceptions took weeks, and it's the part that still requires maintenance every time a vendor ships.

If you're evaluating whether to build this yourself: for one provider, don't. Use their SDK. The abstraction costs more than it saves. The break-even is somewhere around the second or third provider, and it arrives not because of the HTTP plumbing but because that's when you start needing a place to encode "which of these models can actually do the thing I'm asking for" — and that logic has to live somewhere whether or not you call it a gateway.

The full endpoint reference for the task API described here is at docs.velokey.ai/api/video-models/introduction if you want the concrete parameter shapes.

Disclaimer: This post is intended as a technical write-up on API design for asynchronous media generation. It is not commercial advertising for any service.