I Tried a Different Way to Handle Multimodal Video Inputs in Node.js

I was working on a small video generation prototype recently and ran into a problem I hadn't really thought about when I started.

The first version was extremely simple:

prompt → API → video model → result

The request body wasn't much more complicated than this:

{
  prompt: "A person walking through a rainy street"
}

Then I started adding references.

First an image. Then another image for the environment. Later I wanted to attach audio and occasionally an existing video.

My request object slowly turned into this:

{
  prompt,
  characterImage,
  backgroundImage,
  styleImage,
  audioReference,
  videoReference
}

It still worked, but I didn't like where it was going.

Every new experiment meant another field, another validation rule, and usually another condition somewhere in the backend.

So I changed the way I represented the input.

I stopped creating fields for every reference

Instead of giving every possible input its own property, I treated references as resources:

const request = {
  prompt: "A person walking through a rainy street",

  resources: [
    {
      type: "image",
      role: "character",
      url: "/assets/person.png"
    },
    {
      type: "image",
      role: "environment",
      url: "/assets/street.jpg"
    },
    {
      type: "audio",
      role: "atmosphere",
      url: "/assets/rain.wav"
    }
  ],

  output: {
    duration: 5,
    aspectRatio: "16:9"
  }
};

The small thing that helped most was keeping type and role separate.

An image is just an image as far as storage is concerned. But once it reaches the generation layer, a character reference and a style reference aren't really the same thing.

So I can have:

{ type: "image", role: "character" }

and:

{ type: "image", role: "style" }

without changing the request format.

This also made validation less annoying.

const allowedTypes = new Set([
  "image",
  "audio",
  "video"
]);

function validateResources(resources = []) {
  for (const resource of resources) {
    if (!allowedTypes.has(resource.type)) {
      throw new Error(
        `Unsupported resource: ${resource.type}`
      );
    }

    if (!resource.url) {
      throw new Error("Missing resource URL");
    }
  }
}

The real validation is obviously more complicated. File size and MIME type need checking, and video duration matters too.

But I learned pretty quickly that these checks should happen early.

In my first version I let the generation worker deal with invalid inputs. That meant a bad request could get accepted by the API, sit in the queue for a while, reach a worker and only then fail.

Not ideal.

Uploading the references caused another problem

Originally I sent the actual files together with the generation request.

It was convenient:

browser
   ↓
upload + generate
   ↓
model

Then one generation failed and I tried to retry it.

The browser had to upload all the same files again.

That was fine with a couple of small images. It became noticeably worse when I started testing video references.

I ended up separating uploads from generation completely.

                   ┌─────────────┐
                   │   Browser   │
                   └──────┬──────┘
                          │
                    upload assets
                          │
                   ┌──────▼──────┐
                   │   Storage   │
                   └──────┬──────┘
                          │
                     asset IDs
                          │
                   ┌──────▼──────┐
                   │     API     │
                   └──────┬──────┘
                          │
                        queue
                          │
                   ┌──────▼──────┐
                   │   Worker    │
                   └──────┬──────┘
                          │
                   ┌──────▼──────┐
                   │ Video Model │
                   └─────────────┘

Now the upload happens first.

The API only receives references to files that already exist in storage.

{
  "prompt": "A short cinematic street scene",
  "resources": [
    {
      "type": "image",
      "role": "character",
      "assetId": "asset_1024"
    }
  ]
}

Apart from making retries easier, this had a useful side effect: the same asset could be reused.

If I wanted to generate five variations using the same character image, I no longer had to upload that image five times.

I hadn't planned that feature. It just fell out of separating storage from generation.

The model itself became the next awkward part

At first I called the video provider directly inside my Express route.

Something roughly like:

app.post("/generate", async (req, res) => {
  // validation...

  const result = await provider.generate(req.body);

  res.json(result);
});

This was okay until I wanted to try a different model.

The second model accepted references differently, so suddenly the route contained conditions for two providers.

I moved that translation into a separate adapter instead.

class VideoAdapter {
  async generate(input) {
    const images = input.resources.filter(
      item => item.type === "image"
    );

    const audio = input.resources.filter(
      item => item.type === "audio"
    );

    const videos = input.resources.filter(
      item => item.type === "video"
    );

    return provider.generate({
      prompt: input.prompt,
      images,
      audio,
      videos,
      output: input.output
    });
  }
}

The adapter isn't particularly clever. That's kind of the point.

The rest of the application works with its own resource format. Only this layer needs to understand what the current video model expects.

I ran into this again while looking at newer multimodal workflows with MiniMax H3. Once text, images, audio and video can all be part of the generation context, designing the whole application around generate(prompt) starts to feel a little restrictive.

I now prefer thinking about it as:

generate({
  instructions,
  resources,
  output
});

The provider adapter can figure out what to do with those resources.

The application doesn't need to care quite as much.

One thing I wouldn't do again

I also made the generation endpoint synchronous in an early prototype.

That was a bad idea.

Video generation can take long enough that keeping an HTTP request open just creates unnecessary problems. So the current endpoint only creates a job:

app.post("/api/video/jobs", async (req, res) => {
  validateResources(req.body.resources);

  const job = await queue.add(
    "generate-video",
    req.body
  );

  res.status(202).json({
    id: job.id,
    status: "queued"
  });
});

Then:

POST /api/video/jobs

returns immediately.

The frontend checks:

GET /api/video/jobs/:id

and gets something simple:

{
  "id": "job_823",
  "status": "processing"
}

I actually had a progress field in the first implementation:

{
  "progress": 63
}

Then I realized the number didn't mean much.

The provider wasn't giving me real frame-level progress, so I was basically inventing precision based on which step the worker had reached.

I removed it.

queued, processing, completed and failed turned out to be enough.

There's still a problem I haven't solved properly

Different models accept different numbers and types of references.

One might support several images plus audio. Another might only accept a prompt and one image.

Right now some of those limits still live in configuration.

If I keep working on this, I'd like to make model capabilities explicit:

{
  image: true,
  audio: true,
  video: true,
  maxImages: 9,
  maxAudio: 3,
  maxVideos: 3
}

Then the frontend could ask:

GET /api/models/:id/capabilities

before displaying its upload controls.

That would also give the backend one place to validate provider-specific limits instead of spreading them across routes and UI code.

I haven't implemented this part yet, so I don't know whether the capability object will stay this simple. There are probably edge cases around duration, resolution and combinations of references that will make it messier.

But it seems like a better direction than hardcoding model names in the frontend.

For now, the useful change was much smaller: stop treating the prompt as the entire generation request.

Once I did that, the backend became easier to change.

The current mental model is basically:

instructions + resources
          ↓
     application
          ↓
    model adapter
          ↓
       provider

It's not a particularly sophisticated architecture, but it has held up better than my original collection of characterImage, styleImage, audioReference and videoReference fields.

If I were rebuilding the prototype today, that's probably the one decision I'd make earlier.