SR-SWA
An attention mechanism variant that combines sliding-window attention so that the per-step computational cost during long-sequence decoding remains constant and does not grow with context length.
Timeline (last 90 days)
Unlimited OCR 2 uses the SR-SWA attention mechanism, keeping the decoding cost per step almost constant.
SR-SWA's specific implementation details, VRAM usage curves or latency data at different page counts were not disclosed in the original material.
Unlimited OCR 2's SR-SWA introduces a reference set mechanism on top of the sliding window, enabling text and visual information to interact across windows and avoiding the problem of pure sliding windows forgetting distant information.
The core innovation of Unlimited OCR 2 is sparse reference sliding window attention (SR-SWA), which compresses text-vision attention interactions into a fixed-size reference set (for example, 512 tokens).
The 512-token reference set mentioned officially is an example value, and the configuration space and tuning strategies in actual deployment have not yet been disclosed.
The robustness of SR-SWA on real, complex documents, its adaptability to layout diversity, and whether a fixed reference set will lose key details in extremely long documents all require further verification.
SR-SWA makes per-step decoding latency and memory usage barely grow with document length.
All Facts (7)
Unlimited OCR 2 uses the SR-SWA attention mechanism, keeping the decoding cost per step almost constant.
50%UnverifiedSR-SWA's specific implementation details, VRAM usage curves or latency data at different page counts were not disclosed in the original material.
50%UnverifiedUnlimited OCR 2's SR-SWA introduces a reference set mechanism on top of the sliding window, enabling text and visual information to interact across windows and avoiding the problem of pure sliding windows forgetting distant information.
50%UnverifiedThe core innovation of Unlimited OCR 2 is sparse reference sliding window attention (SR-SWA), which compresses text-vision attention interactions into a fixed-size reference set (for example, 512 tokens).
50%UnverifiedThe 512-token reference set mentioned officially is an example value, and the configuration space and tuning strategies in actual deployment have not yet been disclosed.
50%UnverifiedThe robustness of SR-SWA on real, complex documents, its adaptability to layout diversity, and whether a fixed reference set will lose key details in extremely long documents all require further verification.
50%UnverifiedSR-SWA makes per-step decoding latency and memory usage barely grow with document length.
50%