Building an Open-Source Semantic Cache for LLM Calls

Every time your application makes an LLM API call, there's a roughly 30% chance that call is semantically identical to one you've already made. Not string-identical — semantically identical. Different words, same meaning, same expected output.

At scale, this is a hemorrhage. A mid-size SaaS running GPT-4o at moderate volume is burning through $2,000–$8,000 per month in redundant calls. The math is straightforward: if you cache 30% of calls and your average response costs $0.03, the savings compound fast.

GPTCache launched with promise in early 2023. By mid-2024, it was effectively unmaintained. The architecture was monolithic, the embedding model was hardcoded, and the eviction strategy was naive LRU with no semantic awareness.

The core insight that GPTCache missed: cache invalidation for semantic data isn't the same problem as cache invalidation for deterministic data. You can't just hash the input. You need to embed it, compare it against a vector space of prior queries, and make a confidence-weighted decision about whether the cached response is still valid.

The semantic cache sits between your application and the LLM provider as a transparent proxy:

1. Ingress — Incoming request is intercepted
2. Embed — Query is embedded using a lightweight model (we use gte-small)
3. Search — Vector similarity search against the cache index
4. Decide — If similarity > threshold (configurable, default 0.92), return cached response
5. Passthrough — If no match, forward to provider, cache the response on return

The embedding step adds ~12ms of latency. The vector search adds ~3ms with HNSW indexing. Total overhead on a cache miss: 15ms. On a cache hit, you save 800–2400ms of LLM latency plus the API cost.

We're building this in Python with a Rust core for the vector operations. The index uses HNSW via hnswlib for now, with plans to move to a custom implementation once the API surface stabilizes.

The eviction strategy is semantic-aware: entries that are frequently "near-missed" (similarity between 0.85–0.92) get promoted in the index, while entries that haven't been matched in N hours decay.

class SemanticCache:
    def __init__(self, threshold=0.92, model="gte-small"):
        self.threshold = threshold
        self.embedder = load_model(model)
        self.index = HNSWIndex(dim=384)

    async def get(self, query: str):
        embedding = self.embedder.encode(query)
        results = self.index.search(embedding, k=1)
        if results and results[0].score >= self.threshold:
            return results[0].payload
        return None

The v0.1 release targets a single-node deployment with Redis-backed metadata and an in-memory HNSW index. The roadmap includes distributed caching, streaming response support, and provider-specific optimizations.

The goal isn't to build a product — it's to build infrastructure that makes LLM applications cheaper to run for everyone. Open source. No vendor lock-in. No managed service tax.

Follow the build on GitHub or subscribe to field notes for weekly updates.