What are Structured Outputs? And why they are just a gate on reliability and not privacy?

On August 6, 2024, OpenAI introduced Structured Outputs, giving developers a native way to make model responses conform to a supplied JSON Schema.

Developers had already been chasing this behavior with careful prompting, third-party libraries, validation code, and repeated requests. Structured Outputs moved much of that work into the model provider’s infrastructure, and it quickly became useful for generating interfaces, extracting fields from financial documents, and producing reports that downstream systems could reliably parse.

OpenAI combined model training with something called "constrained decoding". The supplied schema is converted into a grammar (a set of rules that that defines a language) determines which tokens are valid at each point in the response. Invalid tokens are removed as options during generation.

If the model has already produced the beginning of a specific JSON object, for example, it can't suddenly insert a curly brace somewhere the schema doesn't allow.

I'm heavily summarizing the technical pieces, but the basic idea is straightforward: limit the model to the subset of tokens that can still produce a valid result.

That solves syntax and how strictly the schema is followed, but it doesn't guarantee that the values inside the response are accurate. 

Perfectly valid JSON can still contain the wrong number, a fabricated fact, or a bad conclusion.

Structured Outputs makes the response predictable, but it doesn't make the input safe. A model can return perfectly fine JSON while the original prompt still contains client names, account information, case details, participant quotes, or private business plans.

And as AI workflows get faster and more reliable, sensitive information moves faster too. It passes through model infrastructure, application databases, logs, exports, email, Slack, and whatever system comes next in your workflow. Each copy creates another place that needs access controls, and retention rules. 

Structured Outputs can control the shape of the data, but they can not decide whether that data should have been sent in the first place, and that is the part I think many teams are overlooking and will continue to overlook until it's too late.

At Nonymize, our goal is to help people remove identifying details from transcripts, documents, and other text before sending that material into downstream AI tools.

Nonymize finds potentially sensitive details, lets the user review each proposed replacement, and produces a less-identifying copy for the next step in the workflow.

Our privacy policy is specific about what happens during that process. Customer transcripts are not used to train, fine-tune, evaluate, or improve models. The original text is temporarily retained so the user can review the result, then deleted when the anonymization is finalized, and temporary replacement keys you'd use to reference the anonymized version expire after seven days. The anonymized result remains in the user’s account until they delete it.

Highlighting that is important because privacy claims should describe what a product actually does, not what we hope readers assume it does.

We built Nonymize to be a deliberate privacy layer between sensitive information and the AI tools people use every day.

Structured Outputs controls the shape of what comes out.

Nonymize helps control what goes in, check us out.