Redis LangCache: Semantic Caching That Cuts LLM Costs Up to 90%

Redis has launched LangCache, a managed semantic caching service now in public preview on Redis Cloud. It sits between an application and an LLM, comparing incoming prompts by meaning rather than exact wording. When a close enough match exists in the cache, it returns the stored answer and skips the model call entirely. Redis claims savings of up to 90% on API costs and cache-hit response times up to 15x faster than re-querying the model. The service is available today via REST API with Python and JavaScript SDKs.

What happened

Detail	Fact
Product	Redis LangCache, managed semantic caching service
Status	Public preview on Redis Cloud
Access	REST API, Python and JavaScript SDKs
Claimed cost savings	Up to 90% on LLM API spend
Claimed speed on cache hit	Up to 15x faster than re-querying the model
Demo: direct inference	2.232 seconds, 514 input + 250 output tokens
Demo: cache hit	0.37 seconds, 0 tokens consumed (roughly 6x faster)
Customer result (Mangoes.ai)	70% hit rate, 70% lower LLM spend, 4x faster responses

LangCache targets a specific inefficiency in production AI systems. Support assistants and RAG pipelines (retrieval-augmented generation, where a model is fed retrieved documents before answering) field the same questions thousands of times a day, each phrased a little differently. Without a semantic cache, every phrasing is treated as a fresh request and billed in full.

Prefix caching, the approach many LLM providers already offer, only goes so far. It reuses the computed states for a shared system prompt or context prefix, making the generation cheaper. But the request still hits the model, new tokens still get processed, and the full answer still gets decoded. A prefix-cache hit is a discounted call, not an avoided one. LangCache avoids the call entirely.

How LangCache actually works

The integration is a two-step loop around every model call:

1. Before calling the LLM, send the prompt to POST /v1/caches/{cacheId}/entries/search. LangCache generates a vector embedding for the prompt and runs a similarity search over stored entries.
2. If the similarity score clears the configured threshold, the cached response comes back immediately and the model is never called.
3. On a miss, call the LLM as normal, then store the prompt and response via POST /v1/caches/{cacheId}/entries so future paraphrases can hit it.

Embedding generation is handled by the service. You can use the default models or bring your own. Cache behavior is tuned through similarity thresholds, TTLs (time-to-live expiry), and eviction policies. The service also offers adaptive controls that balance precision and recall. Everything is built on Redis’s vector database and monitored through the Redis Cloud console.

On data privacy, Redis states that customer data stays on the customer’s own Redis servers and is not accessed by Redis or used for model training.

How to estimate your actual savings

The savings formula from the Redis documentation is straightforward:

Estimated monthly savings = (Monthly output token costs) x (Cache hit rate)

Redis gives a worked example: $200 per month in LLM spend, with 60% of that on output tokens and a 50% hit rate, equals $60 saved per month. Output tokens matter more than input tokens here because cached responses eliminate decoding entirely, while input token costs are largely offset by embedding and storage costs.

Redis also publishes a savings calculator for annual estimates on its product page.

What can go wrong with semantic caching?

Threshold tuning is the main operational risk. Set the similarity threshold too low and the cache may return a refund policy in response to an upgrade question. Set it too high and nearly every paraphrase misses, making the cache pointless. Production deployments need:

• Carefully tuned similarity thresholds per use case
• TTL and eviction policies so stale answers age out
• Tenant data isolation for multi-tenant apps
• Active monitoring for incorrect matches

LangCache addresses these with access scopes, custom filtering, TTL controls, and Redis Cloud monitoring. But the configuration work is real and falls on the team building the app. This is not a setting you enable once and forget.

If you are already working on AI integration for your business, semantic caching is worth evaluating early, before your token bill scales with traffic. The earlier you design for it, the less retrofitting is needed later.

Our take

The core idea is sound. Most production AI apps have more repeated intent than their builders realise, and skipping redundant LLM calls is a cleaner solution than optimising prompt length. The demo numbers (0.37 seconds vs 2.232 seconds, 6x faster) are more honest than the headline claim of 15x, which likely reflects a best-case scenario. The savings formula is also refreshingly transparent rather than a marketing black box.

The real question is how much safe repetition your traffic contains. A general-purpose chatbot probably has less than a customer support assistant with a narrow topic domain. Mangoes.ai’s 70% hit rate on a patient-care voice app is plausible precisely because that kind of app fields a constrained set of questions repeatedly. Your mileage will depend entirely on your use case.

One caution: public preview means the API and behavior can change. We would not architect a critical production system around it yet, but it is worth prototyping now to measure your actual hit rate before committing. We have covered how AI infrastructure costs are a central concern as these workloads scale, and LangCache is one practical tool for managing that.

What to do about it

1. Log a sample of recent prompts from your app and identify repeated intents, even with different phrasing. This tells you whether your hit rate potential is high enough to justify the integration effort.
2. Sign up for Redis Cloud and test LangCache with the Python or JavaScript SDK against that prompt sample.
3. Set a conservative similarity threshold to start. Review false matches before loosening it.
4. Apply the savings formula to your actual output token spend to project real ROI before committing to production.
5. Build in TTL policies from day one so cached answers don’t persist after your product or pricing changes.

If you want help evaluating where semantic caching fits in your AI stack, the Lumien team is available to talk through the architecture.

Source: Marktechpost