No. 372026LLM reliability · Real Bedrock run

Structured Output EvalValid JSON Was Never the Failure Mode

A structured-output reliability benchmark across plain, schema-in-prompt, native constrained decoding, and strict tool use, where the real Bedrock run showed schema adherence rather than JSON validity is the failure that matters.

Valid JSON: 1.000 even on plain prompting
Schema adherence, plain: 25%
Adversarial: 33%
Schema in prompt: 100%, ties constrained decoding

System architecture

Problem

Agents break when the model returns malformed JSON or the wrong tool arguments, and the standard defense is to wrap everything in constrained decoding so the output parses. That defense assumes a failure mode nobody measured. The real question is how often the output parses, how often it matches the schema, and how much each strategy actually buys.

Approach

Four strategies run across the same 8 labeled extraction and classification tasks, ordered from weakest to strongest constraint: a plain return-JSON prompt, the JSON Schema pasted into the prompt, native output_config.format constrained decoding, and a strict tool whose input_schema is the target schema. Each output is scored on four axes: valid-JSON rate with lenient recovery of fences and preamble, schema adherence under jsonschema Draft 2020-12, field-level accuracy against gold, and schema adherence on the adversarial subset alone. The task set deliberately includes adversarial inputs, missing required fields, prompt injection inside the input, and preamble bait. Headline numbers come from a real Claude Haiku 4.5 run on AWS Bedrock.

Impact

The real run overturned the mock. Claude Haiku returned valid JSON 100% of the time on every strategy including plain, so malformed JSON simply was not the failure mode. The severe failure was schema adherence: plain prompting produced valid JSON matching the target schema only 25% of the time, falling to 33% on adversarial inputs, at 0.562 field accuracy. Pasting the schema into the prompt fixed it completely to 100% schema adherence and 100% field accuracy, tying native constrained decoding and the strict tool. Valid does not mean correct, and on this model the cheap intervention was enough.

Decisions & tradeoffs

Score valid JSON and schema adherence separately

Collapsing them into one reliability number would have hidden the entire finding. Splitting them showed the model scores 1.000 on parseability and 0.250 on shape, which relocates the whole problem.

Keep the mock as a contrasting fixture, not the headline

The offline mock is calibrated with designed malformed-JSON failures and reproduces a different table where plain scores 0.500 valid_json. It stays in the repo to run the pipeline key-free, but the real model's behavior is the reported result.

Report the honest ceiling on constrained decoding

Native and strict_tool tied schema-in-prompt at 1.000, so no measured lift is claimed for them on this set. Their value is the guarantee at scale and on harder schemas, which is stated rather than dressed up as a number.

Build spec

Model: Claude Haiku 4.5 (Bedrock)
Tasks: 8 (clean + adversarial)
Plain schema rate: 0.250 (0.333 adversarial)
Schema-in-prompt: 1.000 schema, 1.000 field acc
Stack: Python, jsonschema, anthropic[bedrock]

System notes

Real Claude Haiku on Bedrock: valid_json 1.000 on every strategy, plain included
Plain prompting: 25% schema adherence, 33% on adversarial inputs
Schema-in-prompt reaches 100% schema and field accuracy, tying constrained decoding
Adversarial subset scored separately: missing fields, injection, preamble bait