Getting Reliable JSON out of an LLM

Prompting for JSON is not a parsing strategy. Schema validation, one smart retry, and tool-based output turn a probabilistic model into something your TypeScript code can trust.

The moment an LLM output feeds another function instead of a human, "usually valid JSON" stops being good enough. A model that returns perfect JSON 98 percent of the time still fails 20 times per thousand calls, and each failure is a runtime exception somewhere downstream.

The fix is not a better prompt. It is treating model output the way you treat user input: parse, validate, and recover.

Validate at the boundary, always

Define the shape once with a schema library like Zod and parse every response through it. This catches truncated JSON, missing fields, wrong types, and hallucinated extra structure in one place, and it gives the rest of your codebase a real TypeScript type instead of "any".

const InvoiceSchema = z.object({
    vendor: z.string(),
    total: z.number(),
    currency: z.enum(['USD', 'EUR', 'PKR']),
    lineItems: z.array(z.object({
        description: z.string(),
        amount: z.number(),
    })),
});

const result = InvoiceSchema.safeParse(JSON.parse(raw));
if (!result.success) {
    // do not throw yet; this is where the retry lives
}

Retry once, with the error in the prompt

When validation fails, send one follow-up request containing the invalid output and the validation errors, asking the model to fix it. In practice this single corrective round trip resolves the vast majority of failures. Cap it at one retry: if the second attempt also fails, log the raw output and fall back, because a model that failed twice will usually fail a third time.

Better: make JSON the only possible output

Modern APIs let you define a tool with a JSON schema and force the model to call it. The model then cannot reply with prose, markdown fences, or apologies; the API layer constrains the output to your schema. If your provider supports forced tool use or a native structured output mode, prefer it over prompt instructions, and keep the Zod parse anyway as a cheap second line of defense.

Takeaways

• Never JSON.parse a model response without schema validation behind it.
• One retry with the validation errors included fixes most failures cheaply.
• Forced tool use beats "please respond with only JSON" every time it is available.
• Log every raw failure; the patterns tell you when the prompt or schema needs work.