Metrecept
An open-source LangChain caching gateway project released under the MIT license on GitHub. It provides caching acceleration for LLM API requests and supports normalized sorting of tool calls as well as caching of SSE streaming responses.
Timeline (last 90 days)
Normalizing and sorting the tool call list before generating the cache key can resolve the problem of low cache hit rates.
The caching gateway must fully receive and reassemble all SSE chunks on the server side before storing the complete response in the cache, and then replay it in the form of synthesized SSE.
Some streaming responses will never become cache entries; incomplete responses are not saved if the client disconnects midway or if generation is abnormally interrupted.
Waiting for the complete streaming response before deciding whether to cache it increases memory pressure and latency on the caching layer.
The Metrecept project chose an exact matching strategy rather than semantic similarity matching.
Introducing fuzzy semantic matching at the caching layer is essentially trading accuracy for performance, and may return incorrect answers that the user did not request.
All Facts (6)
Normalizing and sorting the tool call list before generating the cache key can resolve the problem of low cache hit rates.
50%UnverifiedThe caching gateway must fully receive and reassemble all SSE chunks on the server side before storing the complete response in the cache, and then replay it in the form of synthesized SSE.
50%UnverifiedSome streaming responses will never become cache entries; incomplete responses are not saved if the client disconnects midway or if generation is abnormally interrupted.
50%UnverifiedWaiting for the complete streaming response before deciding whether to cache it increases memory pressure and latency on the caching layer.
50%UnverifiedThe Metrecept project chose an exact matching strategy rather than semantic similarity matching.
50%UnverifiedIntroducing fuzzy semantic matching at the caching layer is essentially trading accuracy for performance, and may return incorrect answers that the user did not request.
50%