JSON mode isn't a contract: making structured output reliable in production

Every model provider now offers some flavor of structured output — OpenAI’s response_format with strict JSON schema, Anthropic’s tool-use forcing, Gemini’s responseSchema. Teams treat “the API guarantees valid JSON” as the end of the problem. It’s the start of a different one.

Constrained decoding (grammar-based token masking, in most implementations) guarantees the output parses. It says nothing about whether the values are right. A model can emit a perfectly well-formed JSON object with a hallucinated order ID, a status field that doesn’t match the actual outcome, or a confidence score it invented because your schema requires one. Syntactic validity and semantic correctness are different failure surfaces, and only one of them is solved for you.

Where it still breaks

• Enum drift. You define status: "approved" | "denied" | "pending". Under strict schema enforcement the model can’t emit a fourth value — so instead it picks the closest of the three, even when none fit, because the grammar forces a choice. You lose the signal that the case was ambiguous.
• Required-field hallucination. Mark a field required and the model will fill it even with no grounding for the value, because omitting it isn’t a legal token sequence. A summary_source_url field marked required on a task with no browsing tool will get a plausible-looking fake URL.
• Silent truncation. Long structured outputs (large arrays, deeply nested objects) can hit max_tokens mid-object. Some SDKs return a parse error; others return a truncated-but-technically-valid prefix if the cutoff lands on a boundary, and you ship a partial record.
• Schema-prompt mismatch. The schema constrains shape; it doesn’t constrain meaning. If your prompt says “list the top 3 risks” and your schema says risks: array with no maxItems, you’ll get anywhere from one to nine, all schema-valid.

The reliability layer that’s actually required

flowchart LR
  P["Prompt + schema"] --> M["Model call
(constrained decoding)"]
  M --> S["Syntactic check
(parses? matches schema?)"]
  S -->|fail| RT["Retry with error
fed back into prompt"]
  RT --> M
  S -->|pass| V["Semantic checks
(business rules, refs exist)"]
  V -->|fail| RT
  V -->|pass| OUT["Accepted output"]

Three layers, not one:

1. Syntactic validation — you get this mostly for free from strict mode, but don’t skip it. Providers’ “guarantee” still has edge cases (nested oneOf, recursive schemas) where compliance degrades. Validate with a real schema library (ajv, pydantic) on every response regardless of provider promises.
2. Semantic validation — checks the model provider cannot run because they require your domain data. Does the referenced order ID exist? Is the date in the future when it shouldn’t be? Does the enum value match what your downstream state machine actually expects? This is plain code, and it’s where most real bugs live.
3. Retry with the error in context — on failure, don’t just resample. Feed the validator’s specific complaint back to the model (“order_id 8823-A does not exist in this account”) so the retry has new information instead of the same odds of repeating the mistake. Cap at 2–3 retries and fall back to a cheaper deterministic extraction or a human queue — don’t loop indefinitely on a model that’s confidently wrong.

Schema design choices that prevent failures upstream

• Make required fields truly always-derivable. If a field can’t be grounded from the input in every legal case, make it nullable and handle null downstream. Required-but-sometimes-unknowable fields are the single biggest source of hallucinated values.
• Add explicit array bounds. minItems/maxItems where you can state them. “Return between 1 and 5 risks” removes an entire class of scope ambiguity.
• Prefer closed enums over free text for anything routed downstream, but add an "other" or "uncertain" member so the model has a truthful escape hatch instead of being forced into a wrong bucket.
• Keep nesting shallow. Deep nested objects increase both truncation risk and the chance a constrained-decoding implementation falls back to a looser mode. Flatten where the domain allows it.
• Version your schemas. Structured-output schemas are an API contract with your own downstream code — treat schema changes like you’d treat breaking changes to an endpoint: versioned, with a migration path for records already in flight.

What to log

Log the raw model output, the validation result (syntactic and semantic separately), and which retry attempt succeeded — not just the final accepted value. When a semantic check starts failing at a higher rate after a prompt or model change, that rate is your earliest signal of drift, and you won’t have it without the breakdown by layer. Treat structured-output failure rate as a first-class metric on any agent or extraction pipeline, the same way you’d track latency or cost.

Structured output turns “the model said something” into “the model produced a value your system will act on automatically.” That’s a much higher bar than valid JSON, and it’s on you to enforce it.