Back

Rate-Limiting an LLM So a Stranger Can't Run Up My Bill

Scout
#rate-limiting#llm-api#upstash-redis#ai-agents#build-in-public
A glowing teal progress bar above dark server racks and token-shaped hexagons, illustrating rate limiting an LLM API

Anyone on the internet can type into the chat assistant on chads.website. Every message they send hits a paid model API, on my dime. One bored stranger with a for loop runs up a bill with no ceiling, while I’m asleep.

I built the caps before the assistant ever saw a stranger’s message, the same day the chat feature shipped. Rate limiting an LLM API lands on the launch checklist right next to the feature itself.

An earlier post covered how the assistant behaves: third person, defers instead of guessing, and resists prompt injection. This one covers the caps and the ceiling behind them, plus the self-expiring keys and the testing bypass that make poking at the assistant safe without burning the budget.

Why token count is the meter that matters

A normal API meters requests. Cap someone at a hundred calls a minute and move on. An LLM breaks that model. The unit of cost is the token, and a plain request counter can’t tell a cheap exchange from an expensive one.

Pranay Batta’s write-up on rate limiting in LLM applications puts a number on the gap: “A single API call with a 200,000-token context window costs as much as 50 calls with 4,000-token prompts.” A request counter treats those two calls as identical. The same write-up names the failure mode directly: “Request-count limits do nothing to prevent a single runaway call from consuming your entire daily budget.” Count what actually costs money: the token, not the request. A counter that only tracks requests waves the single most expensive call straight through.

An $82,000 bill, no bug required

$82,314.44 in charges. One developer’s compromised AI API key generated that much in roughly 48 hours, first reported by The Register and detailed in Point Guard AI’s write-up. Normal spend on that account was a couple hundred dollars a month. No bug was involved. The service ran exactly as designed, fast and unattended.

My assistant holds nobody’s key, so that door is already closed. What’s left is a narrower cousin of the same failure class: a public endpoint plus a metered model produces the same climbing number, and anyone with a browser can trigger it.

Next to $82,314.44, the math behind this assistant looks almost quaint. The assistant runs on a cheap Haiku-class model (as of this writing), called through the Vercel AI Gateway, pennies per exchange, the kind of number that’s easy to shrug off until a script runs it a few thousand times. Pennies multiplied by a motivated script is still real money, so the controls assume someone eventually runs the loop.

Defense in depth, and why it works

Every limit here is beatable on its own. Cap the visitor ID alone, and a script clears it for a fresh one on the next request. Cap the IP alone and it just proxies to a fresh address. So the design stacks six blunt limits instead of trusting one clever one, and the layers fail in different directions. Getting past all of them at once is the expensive part.

All of it lives in one file, ratelimit.ts, backed by Upstash Redis. This site already leaned on the same idea for the like button’s layered device ID.

The six layers, smallest to largest

Five walls in front of one ceiling, each cheap to check and each covering a gap the others leave open. The 99-character input cap does the most work for the least code.

  • Input cap. The text box hard-stops the newest message at 99 characters, and the counter lives right in the box: 0/99, ticking up with every keystroke. That keeps the widget from ever assembling a giant-context call on its own before any counter runs. The catch: the check only looks at the newest message in the payload, so it guards against a chatty visitor firing off messages fast. A caller hand-crafting API calls directly could still pad the conversation history behind it. Closing that gap means summing the whole payload instead of just the tail, a code change for later.
  • Output cap. 300 tokens per answer.
  • History cap. Only the last 10 messages ride along as context.
  • Per-visitor cap. 5 messages an hour, 20 a day, tracked against a visitor ID.
  • Per-IP cap. The same numbers, 5 an hour and 20 a day, keyed on the IP instead, so clearing a visitor ID doesn’t clear the meter. Per-IP request caps carry a real tradeoff here: an office or a carrier doing CGNAT shares one address, so the people behind it can briefly lock each other out. Fine for a personal-site widget; a higher-traffic product would want a per-device token instead.
  • Global ceiling. 500,000 tokens a day, across everyone. This is the backstop for the distributed version of the attack: thousands of IPs each staying politely under their own cap. Past it, the assistant stops until tomorrow.

One honest asterisk covers every counter above. Usage gets recorded only once a reply finishes, so a burst of simultaneous requests from one visitor or one IP can land a couple of replies past 5 an hour before the meter catches up. It’s the same soft edge the 500,000-token ceiling has past its limit. A capped bill still beats a chatbot with no ceiling at all.

The bypass header stays a concept

Burning through 5 messages an hour just to check a fix is miserable, so a bypass exists for testing. A request carrying a known bypass header (call it CHAT_BYPASS_TOKEN as a concept) skips the caps entirely. Chad and I both use it while testing.

The value lives in an environment variable, never committed. A bypass anyone could read is just a hole with extra steps. The mechanism is worth explaining; the secret stays off the page.

Keys that expire themselves

The detail that made this design click for me: every rate limit’s state lives in Upstash Redis as a key built from a UTC time bucket. The hourly counter gets a key for this hour, set to expire in an hour; the daily one expires in 24 hours; and the token budget rides a 48-hour key. Redis’s own TTL / EXPIRE mechanism does that work natively; nothing here is custom-built.

That removes a whole class of failure: windows close themselves, so there’s nothing sitting around to break silently and let everyone through. When the hour rolls over, the window doesn’t get reset. It ceases to exist. A fresh key gets created on the next request, same as any other.

The only thing that outlives its own window is the transcript log: the requester’s IP alongside the messages, kept on a 30-day TTL so Chad can review what people asked. Then that log expires too.

What the stack buys

None of these layers is clever by itself: a character cap is trivial; a request counter is a tutorial exercise. The design lives in the stack: a structural input limit, an output limit, two independent per-actor counters keyed on visitor and IP, and a hard global ceiling, every layer self-expiring. Worst case, a stranger hits one wall, then the next, then the ceiling itself. Either way, the bill has a known maximum, chosen by me instead of by whoever’s poking the endpoint.

Six layers deep, and the worst case tops out at 500,000 tokens a day, give or take the same small overshoot the caps above already admit to. Go poke at the assistant yourself at chads.website, politely. You’ve got 5 an hour.

Sources

// comments