Goodbye JSON Parsing: nobodywho's Decision Models Output Probabilities Directly

When building LLM application chains, we often need the model to make a "choice": which retriever to call, which tool to use, whether content is safe. The conventional approach is to have the LLM output JSON, parse it with Pydantic, and add retry mechanisms—once the format breaks, the whole pipeline can get stuck. The nobodywho project recently introduced Decision Models, offering a new approach: instead of generating any tokens, it directly scores candidate answers, returning the probability and confidence for each option.

Pain Points of Traditional Approaches: Parsing and Retries

In most chained calls, the "choice" step relies on the LLM's text output, which is then extracted into structured results via a parser. Developers need to handle cases where the model's output doesn't conform to the schema, typically by adding retry logic. This not only increases latency but also complicates the code. What's worse, when there are many options, the model may generate ambiguous or contradictory answers, further reducing parsing success rates.

A nobodywho contributor points out that this pattern is essentially using a generative model for a classification task, and classification tasks don't require token-by-token text generation. (Source: Reddit post, where the author discloses their contributor status)

What Are Structured Output and Retry Mechanisms

Having the LLM output JSON and then parsing it is a naive implementation of "structured output": the model still generates text token by token, developers rely on prompts to constrain the format, and then use libraries like Pydantic to validate field types and required fields. Since generation is stochastic, format errors, missing fields, and type mismatches occur, necessitating a "parse failure → re-invoke → re-parse" retry loop. Retries multiply token consumption and end-to-end latency, and converge slowly at higher temperatures or with many options. More robust approaches include constrained decoding and function calling, which can restrict the selectable tokens during the decoding phase—but they still fall under the generation paradigm, outputting text rather than a directly usable probability distribution.

How Decision Models Work

nobodywho's new Decision Models are designed specifically for this kind of "choice" step. You simply list all possible answers, and the model computes a score for each option, ultimately returning a result object containing the probabilities for all options and an overall confidence. Since no tokens are generated, JSON parsing and format validation are entirely avoided.

The feature is accessed via Python's Decision or DecisionAsync interfaces, with the model running locally and no network requests required. According to the post, it uses the laya model under the hood (449 MB, Apache-2.0 license), which can run on the user's own machine.

Scoring-Based vs. Generative Classification

Traditional approaches disguise classification as generation: the model first generates text like "Option A," and the system then maps it back to a label. Scoring-based classification has the model compute a logit or similarity score for each candidate answer individually, normalized via softmax to produce a probability distribution. The benefits: the output space is constrained to the candidate set, eliminating the failure mode of "generating gibberish outside the candidate set"; the probabilities themselves carry uncertainty information that can be directly used for threshold decisions; inference typically requires only a single forward pass, without autoregressive token-by-token decoding, resulting in lower and more stable latency. The trade-off is that candidate options must be enumerated in advance, and open-ended answers cannot be handled.

The laya Model and Local Inference

laya is the small model underlying nobodywho's Decision Models, roughly 449 MB in size, Apache-2.0 licensed, and fully runnable on local CPUs or consumer-grade hardware. Compared to generative large models that are often tens of GB in size, these small models sacrifice general generative capability in exchange for low latency, zero network dependency, and the privacy advantage of data never leaving the device. It is essentially a sentence embedding/scoring model that computes match degree by comparing the representation of the input with each candidate option. Note that the small model size means limited semantic coverage; its discrimination quality in specialized domains or multilingual scenarios needs to be confirmed through actual testing.

Use Cases and Community Discussion

The post author asks the community: "Where in the chain would you put a step like this?" This sparks thinking about the role of decision steps in complex workflows. Common candidate positions include:

• Routing: Deciding which retriever or tool to use based on user input.
• Safety filtering: Determining whether content is suitable for further propagation.
• Branching in multi-step reasoning: Replacing LLM calls in chains that require explicit branching.

Compared to generative approaches, the probability distribution from Decision Models allows developers to set dynamic thresholds—for example, triggering human review or a fallback path when confidence falls below a certain value.

Limitations and Open Questions

Currently, the solution only supports local execution, and the model size (449 MB) may still be a burden for resource-constrained environments. Additionally, the specific capability boundaries of the laya model and its performance in multilingual or complex semantic scenarios are not detailed in the post. Since this is a single source (a Reddit post by a project contributor), actual effectiveness requires more independent testing.

Nevertheless, this "no generation, only scoring" approach offers a lightweight alternative for decision steps in chained calls, especially suitable for scenarios requiring explicit probability outputs. The project is open-sourced on GitHub, and the documentation provides Python interface examples.

Repository: https://github.com/nobodywho-ooo/nobodywho

Documentation: https://docs.nobodywho.ooo/python/decision-models