Blogs

Counting requests can't control LLM spend: one request may be a one-line question or an agent run making a dozen model calls. Limit by tokens instead — estimate, reserve atomically in a shared Redis counter, then reconcile with actual usage. Also covers token buckets vs. window quotas and per-run agent budgets.

Read on Medium·Read on DEV·Read on LinkedIn

Cache-aside breaks in front of an LLM agent: a question has no fixed key, rebuilding an answer means paying the model again, and answers depend on document version and permissions. Three Redis caches — exact match, semantic, and tool results — and why a semantic hit for timer T3410 must never answer T3411.

Read on Medium·Read on DEV·Read on LinkedIn

TL;DRA token limit is a reservation: reserve an estimate before the model call, correct it after, and give every agent run its own budget.

1. 01Decide what each counter protects
2. 02Estimate before the call
3. 03Reserve in one atomic script
4. 04Call the model, then reconcile
5. 05Give every agent run its own budget
6. 06Tell the caller which limit it hit

Read on Hashnode

TL;DRPut three caches in Redis and check them in order of cost — exact match, semantic, then tool results inside the agent loop — and know the cases where a hit must be treated as a miss.

1. 01Check in order of cost
2. 02Add the exact-match cache
3. 03Add the semantic cache, with an identifier check
4. 04Add the tool-result cache inside the agent loop
5. 05Re-check authorisation on every hit
6. 06Invalidate by version
7. 07Fail open when Redis is down
8. 08Leave prefix caching to the model provider

Read on Hashnode

In a shared RAG pipeline the login is the easy part; leaks happen in the search. The rule: a client may select a workspace but never assert one. Four checkpoints — entry, retrieval, prompt, and proof — keep the workspace inside every SQL and vector query, backed by a 30+ attack suite that passes only when nothing leaks.

Read on Medium·Read on DEV·Read on LinkedIn

TL;DROne pipeline, many users: the client may select a workspace but never assert one. Four checkpoints keep every answer inside the asker’s own documents.

1. 01The server decides the workspace on every request
2. 02Every path to the data carries the limit
3. 03Document text never makes an access decision
4. 04Prove it with refusal tests and an audit trail

Read on Hashnode

If an attacker stole everything inside your API, who could they become? With an identity provider issuing tokens and the API only verifying them, the answer is no one. Four questions for RAG auth — including two attackers most designs forget: the model and the documents — and why every failure should close the door.

Read on Medium·Read on DEV·Read on LinkedIn

TL;DRLet an identity provider (Keycloak, Okta, Cognito) issue tokens; the RAG API only verifies them, locally, against cached public keys.

1. 01The config holds nothing that signs
2. 02Verify locally, then check membership
3. 03The route and every tool take identity from the principal

Read on Hashnode