Rate limits and queueing

Two separate mechanisms shape throughput, and they behave very differently. Rate limiting caps how many requests you may start. Queueing decides how long a started request waits for the model. Under load you will meet the queue far more often than the rate limit.

Rate limits

Limits are applied per workspace, per minute — not per API key, not per model. Every key in a workspace draws on the same allowance.

Every response, not just rejections, carries the current state:

Header	Meaning
X-RateLimit-Limit	Requests permitted per minute.
X-RateLimit-Remaining	Requests left in the current window.
X-RateLimit-Reset	When the window resets, as a Unix timestamp.

X-RateLimit-Reset is an absolute timestamp

It is seconds since the epoch, not seconds from now. Treating it as a delay produces a wait of decades. Compute the delay as X-RateLimit-Reset - now, or just use Retry-After on a 429, which is already in seconds.

Exceeding the limit returns 429 with Retry-After. Read the headers as you go and slow down before you hit it — reacting to X-RateLimit-Remaining approaching zero is cheaper than absorbing rejections.

If your workload needs a higher limit, talk to us; it is adjustable per account.

Queueing

Serverless models are shared, and GPU capacity is finite. Rather than rejecting requests the moment a model is busy, we hold them briefly and admit them as capacity frees up.

Admission is based on a token budget, not a request count. Each model has:

• a total token budget across all in-flight requests,
• a maximum number of concurrent requests,
• a maximum queue depth, and
• a maximum time any request may wait.

A request is admitted when the model is below both its concurrency limit and its token budget. The budget is charged against your request’s estimated size — input tokens plus the output you allow — so a large max_tokens makes a request harder to admit. Setting max_tokens close to what you actually need measurably improves admission under load.

If the model is busy, your request joins a first-in-first-out queue. If the queue is already at its depth limit, it is rejected immediately with queue_full rather than being made to wait.

What the client sees while queued

Nothing. No headers, no partial body, no keep-alive — the connection is simply open and silent until the request is admitted. This is the single most important operational fact on this page:

A queued request is indistinguishable from a hung one

Your HTTP client cannot tell a queued request from a stalled connection. If your client timeout is shorter than the model’s maximum wait, you will abandon requests that were about to succeed — and, because streaming responses only begin emitting once the model starts generating, a short timeout can cut a healthy request too.

Waits can reach several minutes on large models at peak. Set client and proxy timeouts accordingly — well above your worst expected wait, not just above the typical one. Successful responses include X-Queue-Wait-Ms, so you can measure what your workload actually experiences and tune from data rather than guesswork.

If the wait elapses before admission, you get 503 model_overloaded with Retry-After: 5.

Checking current capacity

curl https://api.resetdata.ai/api/v1/queue/status \
  -H "Authorization: Bearer $RESETDATA_API_KEY"

Returns, per model, the in-flight request count, committed tokens, available budget and current queue depth — useful for deciding whether to dispatch a batch now or wait.

Practical guidance

• Set generous timeouts. Minutes, not seconds, for large models. This is the most common integration mistake.
• Right-size max_tokens. It is charged against the admission budget up front, so an unnecessarily large value makes you wait longer.
• Honour Retry-After on 429 and 503.
• Back off with jitter. Synchronised retries from a fleet turn a busy model into a queue-full one.
• Prefer streaming for long generations. Once tokens flow you get continuous evidence the request is alive, and intermediaries are far less likely to cut an active connection than a silent one.
• Watch X-Queue-Wait-Ms. Rising values are your early warning to spread load or move to dedicated capacity.

Concurrency is shared

A model’s limits apply across everyone using it, so waits reflect total demand, not just yours. If your workload needs guaranteed throughput and predictable latency, dedicated capacity is the answer — see Models.