Designing an API That Is Allowed to Be Unsure

Most APIs are built on an assumption that is never written down: that the operation either succeeds and returns the answer, or fails and returns an error. Two states, and the answer is a fact.

That model fits a database lookup. It does not fit anything that infers. Inference produces a distribution, not a fact, and if your response shape cannot express a distribution, your service will lie — not because anyone intended it to, but because the schema left it no other way to speak.

I work on a system that estimates where a photograph was taken from what is visible in it. It is a good forcing function for this problem, because the honest answer is almost never a coordinate. It is "probably this region, possibly that one, and here is what made me think so." Below is what I have learned about designing for that, generalised past my own domain.

The failure mode has a shape

Watch what happens when a confident schema meets an unconfident model.

{ "lat": 41.9028, "lng": 12.4964 }

The model that produced this may have been 90% sure, or it may have been picking between five candidates that scored within noise of each other. The response cannot tell you which, so every consumer downstream treats both cases identically. The mobile client drops a pin. The map zooms to street level, because that is what you do with a coordinate. The user sees a specific street corner in Rome and reasonably concludes the system knows.

Nobody lied. The schema had exactly one way to say anything, and it was used.

Now add a confidence field, which is the usual first fix:

{ "lat": 41.9028, "lng": 12.4964, "confidence": 0.62 }

Better, and still bad. What does 0.62 mean? Sixty-two percent chance that this exact point is correct? That the city is correct? That the country is correct? Is it calibrated — do things scored 0.6 actually come true 60% of the time — or is it a softmax output that happens to live between zero and one? Consumers will not ask. They will threshold it at some round number a product manager picked and move on.

A number without a stated referent and a stated calibration is decoration.

Make the shape carry the uncertainty

The response format that has worked for us has four properties. None are exotic; the discipline is in refusing to skip any of them.

1. Return candidates, not an answer. The top level is a list, ordered, even when there is exactly one plausible option. A single-element list and a scalar are the same information, but they produce very different client code. The list forces the caller to handle "there are others" from day one instead of at the first bug report.

2. Attach an explicit granularity to every candidate. Not a coordinate — a claim about a level: continent, country, region, city, neighbourhood, exact site. The system should return the finest level it can actually defend, and no finer. "Country: Portugal" as a confident answer is more useful than a street address the model invented to satisfy a schema that demanded lat/lng.

3. Attach the evidence. Every candidate carries the observations that support it, in terms the caller can check independently: the script on visible signage, the driving side and road-marking convention, utility-pole style, vegetation zone, roof material, the sun angle if a shadow is legible. This is the part teams skip because it is work, and it is the part that makes the whole response auditable. A caller who disagrees can go and look.

4. Make "I don't know" a first-class success. An empty candidate list with a stated reason is a valid 200 response, not an error. It is also the correct answer far more often than product owners like. If your only way to express ignorance is a 4xx, the model will be pushed to guess, because guessing looks like it works.

Roughly:

{
  "candidates": [
    {
      "granularity": "city",
      "place": "Porto, Portugal",
      "confidence": 0.58,
      "evidence": [
        { "type": "signage_script", "detail": "Portuguese-language street plate" },
        { "type": "road_marking", "detail": "right-hand traffic, EU-standard markings" },
        { "type": "architecture", "detail": "azulejo tile facade, common in northern Portugal" }
      ]
    },
    { "granularity": "country", "place": "Portugal", "confidence": 0.81, "evidence": [] }
  ],
  "calibration": {
    "note": "confidence is the empirical hit rate at this granularity on our evaluation set",
    "set": "mixed real-world uploads, n=4200"
  },
  "insufficient_evidence": false
}

Note the second candidate. It is not a competitor to the first — it is the same conclusion at a coarser granularity with a higher confidence. Letting the caller choose how much precision to buy at what confidence is more honest than collapsing that trade-off yourself.

Calibration is a contract, not a metric

If you ship a confidence field, you have made a promise, and the promise is testable: among all responses scored near 0.6, about 60% should be right.

Two things follow. First, you have to measure it, on the input distribution you actually receive rather than the benchmark you trained against — and for anything involving user uploads, those differ enormously. Public benchmarks in my field are full of geotagged landmark photography; real uploads are ordinary streets, interiors, and screenshots with every scrap of metadata stripped by whatever platform they passed through. A model that scores well on the former and reports those numbers is quoting a fiction.

Second, you have to state the referent in the response, next to the number, not in a documentation page nobody opens. The calibration block above is not garnish. It is the difference between a number a caller can use and a number a caller will misuse.

What this costs, and what it buys

The costs are real. The response is bigger. Client code branches more. Someone will ask why the API cannot "just return the location," and you will have that conversation repeatedly. Product will point out that a competitor returns one clean coordinate and looks smarter in a demo.

The counter-argument is what happens after the demo. A single-answer API fails silently: the wrong answer looks exactly like the right one, so nobody catches it until a decision has already been made on top of it. An evidence-bearing shortlist fails loudly and locally — a caller looking at "Portuguese signage" on a photo they know is Brazilian has caught the error immediately, and knows which part to distrust.

For anything that feeds a human decision — verification, moderation, research, journalism — that property is the product. The pin is not.

The transferable rule

This is not really about geolocation. The same shape applies to any service whose output is an inference: document classification, entity resolution, fraud scoring, medical triage, content moderation, most things currently described as AI features.

Your response schema is a claim about how certain your system is. If the schema can only express certainty, your system will express certainty, whatever the model actually computed. Design the shape first, and design it so the honest answer is always sayable.

If you want to see this in practice, the tool I work on is at whereisthisplace.online — it returns a ranked shortlist with the visual clues attached and a confidence level, rather than a single pin, and it says so when the evidence is too thin. Free analysis runs without an account, capped at two photos a day. On our calibration set it gets the country right about 87% of the time and the city about 62%; those are the numbers behind the design above, and they are the reason the response shape has to be able to say "not sure."

Disclosure: I build that product. The API design argument stands on its own — apply it to whatever you are building, including against us.

— Ray Lin, Singapore