Structured Output with Local LLMs Why Valid JSON Isnt Enough (Ollama + Pydantic)

Why Structured Output Matters for Local LLMs

Local LLMs are a natural fit when your workflow touches sensitive data — household notes, medical records, internal logs. But running the model is only step one. In real applications, the LLM's output feeds another component: a scheduler, a database writer, a downstream cloud model. Free-form text breaks that pipeline. You need predictable structure.

Structured Output solves this: define the expected shape (a schema) up front, constrain the model's generation to that schema, and receive a regular Python object you can parse and validate. In this guide, we'll use Gemma 4 (4B) as the local model, Ollama as the serving runtime, and Pydantic for schema definition and validation.

The pattern works, but as we'll see, there's a catch that trips up most first-time implementations. (For a deeper look at the underlying approach, see the original Towards Data Science write-up.)

Implementation: A Smart-Home Case Study

The Scenario

A user asks: "Should the dishwasher run now or later?" Before answering, the app must extract device info, timing constraints, and electricity tariffs from private household notes. A local LLM sanitizes those notes into a structured object, which is then passed to a cloud LLM for reasoning.

Step 1 — Define the Schema with Pydantic

from pydantic import BaseModel
from typing import List

class DeviceToSchedule(BaseModel):
    name: str
    runtime_minutes: int
    earliest_start: str
    latest_end: str

class SchedulingContext(BaseModel):
    current_time: str
    device_in_question: str
    controller_capacity_kw: float
    electricity_prices: dict[str, float]
    devices_to_schedule: List[DeviceToSchedule]

Step 2 — Install Ollama and Pull the Model

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull the compact 4B variant
ollama pull gemma4:4b

# Python client + Pydantic
pip install ollama pydantic

Step 3 — Connect Pydantic to Ollama

import ollama
from pydantic import BaseModel

response = ollama.chat(
    model="gemma4:4b",
    messages=[{"role": "user", "content": prompt}],
    format=SchedulingContext.model_json_schema(),  # enforce schema
)

# Parse the JSON back into a validated Python object
context = SchedulingContext.model_validate_json(response["message"]["content"])

Two key calls: model_json_schema() converts the Pydantic model into a JSON Schema that Ollama uses to constrain generation. model_validate_json() parses the response back into a typed object — ready for downstream consumption.

The Catch: Valid Structure ≠ Correct Content

Run the one-shot call and inspect the result. On the surface, everything looks fine: valid JSON, correct schema, Pydantic parses it cleanly. But look closer — the robot vacuum appears in devices_to_schedule, even though the household notes explicitly say it finished at 16:10 and needs no further scheduling today.

This is the critical distinction:

Structured output enforces the shape of the response. It does not, by itself, guarantee the content is correct.

Why the Model Gets It Wrong

In a single call, Gemma 4B must simultaneously:

1. Determine which devices still need scheduling
2. Extract relevant facts for those devices
3. Map facts to the correct schema fields
4. Assemble the final nested object

That's a lot of coordinated decisions for a small local model. As the schema grows more complex, the pressure to coordinate more decisions in a single generation increases — and so does the error rate.

The Fix: Decompose the Task

Split the work into two focused steps.

Step 1 — Determine scope (small schema, narrow task):

class SchedulingScope(BaseModel):
    devices_to_schedule: List[str]

scope = SchedulingScope.model_validate_json(
    ollama.chat(
        model="gemma4:4b",
        messages=[{"role": "user", "content": scope_prompt}],
        format=SchedulingScope.model_json_schema(),
    )["message"]["content"]
)

Result: ["dishwasher", "ev_charger", "washing_machine"] — robot vacuum correctly excluded.

Step 2 — Extract facts for those selected devices into the full SchedulingContext schema, feeding step 1's output into the prompt.

Comparison

Pitfalls to Watch

• Don't skip validation. Always call model_validate_json() — a schema-constrained model can still emit malformed output on edge cases.
• Watch schema depth. Nested models amplify the coordination problem. Flatten where possible.
• Model size matters. 4B models tolerate shallow schemas; deep nesting needs bigger models or more decomposition.

Final Thoughts

Local LLMs + structured output is a clean pattern for sensitive-data workflows: define the schema with Pydantic, constrain generation via Ollama's format argument, parse the response back into a typed object. The implementation is genuinely straightforward.

But the lesson that separates working prototypes from production systems is this: valid structure does not guarantee correct content. When a small model struggles with a complex schema, decomposition — separating scope determination from fact extraction — is the practical fix.

Next steps to explore:

• Try outlines or llama.cpp grammars for stricter generation-time constraints
• Benchmark schema-following across model sizes (4B vs. 12B vs. 27B)
• Add a validation layer that cross-checks extracted facts against source text

For teams running reliable cloud workflows at scale, the same reliability mindset applies — see how Temporal powers reliable cloud operations at Netflix and how Meta migrated petabyte-scale ingestion without data loss.