# Unstable AI inference > The six ways an LLM call fails that have nothing to do with the model, what each one costs, and which of them a durable proxy removes. Source: https://docs.stg.perdurance.dev/unstable-ai-inference/ --- **Most AI inference failures are not model failures.** The model produced an answer; something between your process and that answer did not survive. A dropped socket, a serverless timeout, a backgrounded tab, a pod eviction, a rate limit at the wrong moment — none of them are the model's fault, and all of them cost you a generation you already paid for. This page is a catalogue of those failures, what each one costs, and which of them go away when the execution outlives the connection that started it. ## Why AI inference is less stable than the rest of your stack An ordinary API call answers in tens of milliseconds. An inference call answers in seconds to minutes, and streams for most of that. That single difference breaks assumptions the whole transport stack is built on: - **A long-held connection is a fragile connection.** Every load balancer, proxy, NAT and mobile radio between you and the provider has an idle timeout and a lifetime, and a two-minute generation gives all of them time to act. - **There is no cheap retry.** A failed `GET` costs nothing to repeat. A failed generation costs the full price of the generation, twice, and answers differently the second time. - **The work is not resumable by the protocol.** HTTP has no way to say "carry on from token 4,000". A dropped stream is a stream restarted from nothing. - **The failure is invisible from the outside.** A connection that dies at 40 seconds looks identical whether the provider never saw the request or is 40 seconds into answering it. ## The six failures | # | Failure | What it looks like | What it costs | Removed by Perdurance | |---|---|---|---|---| | 1 | Client disconnect mid-stream | Socket closes, partial output | The whole generation | Yes — execution continues, chunks are stored | | 2 | Client process dies | Nothing at all | The whole generation | Yes — collect by id later | | 3 | Timeout at an intermediary | `502`, `504`, or a reset at 30/60/100s | The whole generation | Yes — the hold is ours, not the socket's | | 4 | Serverless execution limit | Function killed at its hard limit | The whole generation | Yes — submit and collect in two invocations | | 5 | Retry after an ambiguous failure | A second identical call | A second generation, billed | Yes — identical bytes attach | | 6 | Worker or provider transient error | `429`, `5xx`, a dropped upstream | Latency, and a retry | Partly — re-dispatched, and the retry is free | ### 1. The client disconnects mid-stream A phone enters a tunnel. A laptop lid closes. A tab is backgrounded and the browser throttles its timers. The socket goes and the tokens produced after it are gone with it — on a normal provider connection there is nothing listening, so there is nothing kept. Perdurance persists each chunk as it arrives, whether or not anything is reading. When the client comes back it replays from the beginning, or resumes at an exact sequence number: ```http GET /requests/01J8F2ZK9QX3M4NBVWT7 Accept: text/event-stream Last-Event-ID: 25 ``` The reply starts at sequence 26 and live-tails to completion. That is a chunk address, not a timestamp, so a reader that died knows exactly where it was. ### 2. The client process dies A deploy rolls, a container is OOM-killed, a worker is evicted. The generation in flight has no owner and no record. Nothing about the request survives except what you happened to log before you sent it. With the request written down before the upstream call is made, the execution has an identity that does not depend on the caller existing. `GET /requests/{id}` works afterwards — from another process, another machine, another day. ### 3. Something in the middle times out Idle timeouts of 30, 60 or 100 seconds are the default in most proxies, load balancers and API gateways. A generation that thinks for 90 seconds before its first token meets them all. The synchronous routes hold a connection for up to the deployment's **synchronous hold** — five minutes by default — and when a generation outlasts even that, the connection is released with `504 held_too_long`. **That is not a failure.** The execution continues and the answer is stored; the message carries the request id. This is the single most misread status in the API, because the reflex it invites — submit again with a different body — is the one action that starts a second provider call. See [Errors](/errors). ### 4. Serverless hits its execution limit A Lambda, a Cloud Run request, an edge function: each has a hard ceiling it cannot argue with, and a long generation walks straight into it. Holding a socket open for two minutes inside a function billed by the millisecond is also expensive on its own terms. Split the work across two invocations: ```bash # Invocation 1 — submit, return immediately curl -X POST "$PERDURANCE_URL/requests?dialect=openai" \ -H "Authorization: Bearer $PERDURANCE_KEY" \ -H 'Content-Type: application/json' -d "$BODY" # → 202 { "request_id": "01J8F2ZK9QX3M4NBVWT7", "status": "pending" } # Invocation 2 — later, from anywhere curl "$PERDURANCE_URL/requests/01J8F2ZK9QX3M4NBVWT7" \ -H "Authorization: Bearer $PERDURANCE_KEY" ``` Neither invocation holds a connection for the length of the generation, and the generation does not care that neither is running. ### 5. The retry after an ambiguous failure This is the expensive one, and it is the one your code performs whether or not you wrote it: the OpenAI and Anthropic SDKs retry automatically by default. A retry after a connection error is a second provider call for work that is very likely still in progress. A body hash makes the second call attach to the first execution instead. [LLM inference retries](/llm-inference-retries) is the long form. ### 6. Transient errors from the provider or a worker `429` from a provider at its capacity, a `5xx`, an upstream connection dropped part-way. These are the failures a retry genuinely fixes — but only if the retry is free. A worker that dies holding a request does not lose it: the lease expires, a sweep re-dispatches it, and the record's `recoveries` counter goes up by one. A stream being replayed while that happens carries an `event: failed` with `status: "superseded"` — the dead attempt's partial output is discarded rather than spliced onto the new one, and re-opening the stream gives you the new attempt from its beginning. `attempts`, `recoveries` and `next_retry_at` on a record are execution history, not an error budget. `recoveries` going up means a worker died and the request was picked back up — which is the behaviour you adopted this for. ## What is still unstable afterwards Being honest about the boundary matters more than the list above. - **The provider being down is still the provider being down.** A stored `502 upstream_failed` is a durable record of a failure, not a way around it. - **The model's output is still non-deterministic.** Durability makes a generation retrievable, not reproducible in the sense of producing the same answer if genuinely re-run. - **Your tools are still not idempotent.** See [Agent SDK retries](/agent-sdk-retries). - **Ten minutes is the window for body-identity.** After it, identical bytes are a new request. The id works for as long as your retention policy keeps the record. ## Sizing it up for your own traffic The failures worth counting are the ones you are already paying for and probably not measuring: generations abandoned mid-stream, automatic SDK retries, and `504`s treated as failures. All three are visible per key and per model in `GET /usage`, alongside the difference between requests accepted and provider calls actually made — which is the number that tells you what deduplication is saving. See [Usage and storage](/usage). - [LLM inference retries](https://docs.stg.perdurance.dev/llm-inference-retries/): Why a retry bills twice, and what makes one safe. - [Agent SDK retries](https://docs.stg.perdurance.dev/agent-sdk-retries/): The same problem inside a loop, where it compounds. - [Frequently asked questions](https://docs.stg.perdurance.dev/faq/): Short answers to what people ask before adopting this. - [Deployment](https://docs.stg.perdurance.dev/deployment/): What a deployment is made of: one image, one Postgres.