DPP Protocol Released: Unifying Document Parsing Standards with 140 Rules

Why Does Document Parsing Need a "Protocol"?

There are hundreds, if not thousands, of OCR models and tools, yet the same document can yield vastly different parsing results. Some errors are obvious at a glance: fabricated text, completely misaligned tables. But many more errors fall into a gray area—when a table with a shared header spans multiple pages, should it be split into two separate tables or kept as one logical whole? Both approaches are semantically valid, yet they can lead to completely different evaluation outcomes.

According to the publisher's introduction on Twitter, existing parsing benchmarks share a structural flaw: they inherently reward one particular style, even when other styles are equally semantically correct. This causes model optimization to be driven by benchmark preferences rather than genuine improvements in parsing quality.

What Is DPP: A 140-Rule "Constitution" for Parsing

Document Parsing Protocol (DPP) offers an answer: first, standardize the very question of "what counts as correct parsing." The team has compiled 140 rules covering virtually every aspect of document parsing, serving as a rubric for evaluating LLM parsing quality.

The rules are designed starting from concrete scenarios, clarifying ambiguous areas one by one:

• Tables: borderless whitespace-aligned tables, header-only tables, horizontally sliced tables, side-by-side mini tables
• Forms: key-value regions, checkboxes, lists, IDs and labels, fields outside the form
• Charts: standardized chart output, multiple charts, Gantt charts, line charts

Each rule corresponds to a type of structure that easily produces disagreement in real-world parsing, effectively transforming problems that previously relied on evaluator subjective judgment into comparable, reproducible entries.

Rubric in evaluation methodology refers to a predefined set of item-by-item scoring criteria, rather than a single overall score. Each rubric item typically describes an observable feature or behavior and specifies its weight or attribution in determining correctness. Compared to end-to-end black-box scoring, rubrics make evaluation results interpretable and reproducible: when different evaluators score the same output against the same criteria, disagreements converge onto specific items. DPP's 140 rules essentially decompose "whether document parsing is correct" from an overall impression into atomic judgments that can be verified item by item—this is also the premise that allows it to serve both human evaluation and model-assisted evaluation.

Positioning: An Evaluation Guide, Not a Complete Benchmark

The publisher explicitly states that DPP is not yet a complete evaluation benchmark, but an initial evaluation guide. Its goal is not to produce an ultimate leaderboard, but to advance community discussion: what should the "best" way to parse documents actually look like?

This positioning is worth noting. Standardizing parsing quality is difficult for a single team to accomplish in one go; it is more pragmatic to first put forward a rule set and let more models and tools expose disagreements under a shared language. For users, DPP can immediately serve as a checklist for human or model-assisted evaluation; for developers, it provides a list of improvement directions to compare against.

The distinction between benchmark and evaluation guide deserves clarification. A complete benchmark typically includes a fixed dataset, ground truth, automated scoring scripts, and a leaderboard, with the goal of producing comparable quantitative scores; an evaluation guide only specifies "how to look at it" without binding to specific samples or scoring implementations. The advantage of a benchmark is scalable comparison, but its disadvantage is that it easily solidifies one annotation style as the only correct answer—precisely the structural flaw in existing parsing benchmarks noted in the article. An evaluation guide trades lower upfront cost for greater flexibility, allowing the community to apply the same judgment criteria across different datasets, at the cost that result comparability depends on the consistency of those executing it.

Potential Impact on the Document Parsing Ecosystem

If DPP is accepted by the community, the most direct change will be improved comparability of evaluations. Different models scored under the same rule set reduce ranking distortions caused by style preferences. Second, the rules themselves can inversely guide the construction of training data—incorporating the boundary cases covered by the 140 rules into the annotation system helps models learn to handle ambiguous structures that were previously overlooked.

Of course, the maintenance cost of the rule set, the degree of adaptation to different document types, and possible conflicts between rules are all issues that need further discussion. The publisher positions the protocol as a starting point for discussion, not an endpoint, which also means its value will evolve with community feedback.

More details can be found at the project link and official blog.

Boundary cases in training data construction refer to samples that lie at category boundaries and are prone to annotator disagreement, such as cross-page tables, borderless alignment, and fields outside forms. Such samples are often discarded or handled according to a single convention in standard annotation workflows, resulting in models never seeing multiple reasonable interpretations of ambiguous structures during training. Explicitly incorporating the boundary cases covered by the rule set into the annotation system is equivalent to providing the model with an ambiguity checklist, making it inclined to produce semantically coherent results rather than mechanically applying a certain format when encountering similar structures. This is also why the rule set can permeate from evaluation tools back into the data production process.

One-line Summary

DPP standardizes the correct format of document parsing with 140 rules, attempting to solve the style bias problem in evaluation.

Paragraph Summary

The article introduces Document Parsing Protocol (DPP), a document parsing evaluation standard consisting of 140 rules. The article points out that existing parsing benchmarks have a structural flaw: they inherently reward one particular style, even when other styles are equally semantically correct, causing model optimization to be driven by benchmark preferences. DPP's approach is to first standardize "what counts as correct parsing," with rules covering tables, forms, charts, and other ambiguity-prone structures, transforming problems that previously relied on subjective judgment into comparable, reproducible entries. The publisher explicitly positions DPP as an initial evaluation guide rather than a complete benchmark, aiming to advance community discussion on the best parsing approach. If accepted, DPP is expected to improve evaluation comparability and inversely guide training data construction.