ai agents

A classification endpoint has been running at temperature 0 for a month. A team replays last Tuesday’s inputs to debug a complaint, and a handful of the labels come back different. Nobody changed the prompt, the model ID or the code. (This is a hypothetical scenario, but the pattern is common enough that Anthropic’s own glossary warns about it.) The glossary says that “even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”

The usual fix, turning temperature down, only controls one step: how the next token is picked from the model’s probabilities. It does not control how those probabilities were computed, which hardware computed them, which other requests shared the batch, which model version sat behind the name, or what the tools returned. LLM determinism is a property of the whole inference pipeline, not of one parameter. Where in that pipeline the variation enters decides what an application can honestly promise about it.

Key Takeaways

• Temperature 0 controls token selection, not computation. Anthropic documents that it is “not fully deterministic”, and OpenAI described its seed as “best effort”.
• The main documented cause on modern stacks is a lack of batch invariance, not concurrency. A request’s arithmetic can depend on which other requests share its batch. Batch-invariant kernels removed the variation in Thinking Machines’ test, at a speed cost.
• A pinned model ID freezes the weights, not the serving stack. Anthropic says infrastructure updates are the most likely cause of behaviour changes on a stable ID.
• In agents, one different token can redirect a whole run. Safety must come from validation and approval gates, not from the model repeating itself.
• Evaluate distributions, not single runs. Report standard errors, resample where variance is high, and track failure modes.
• Pick a reproducibility level: exact, best-effort, statistical, or application-level correctness. Then control the layer that matters for that level.

Quick Navigation

What LLM determinism actually means

A system is deterministic if the same inputs always produce the same output. The difficulty with LLMs is that “the same inputs” hides most of the inputs. The prompt is visible. The model weights, numerical kernels, batch composition, hardware, decoding settings, tokenizer and every tool response along the way are not, and all of them feed into the result.

It helps to separate four questions that often get merged:

• Repeat within one fixed environment. Same weights, same kernels, same hardware, same batch shape. This is the narrowest case, and the one most likely to be reproducible bit for bit.
• Repeat across serving conditions. Same model and prompt, but different load, batch sizes or machines. This is what an API user actually experiences.
• Reproduce after a change. The provider updates the model, the serving stack or the infrastructure. The question is now whether the behaviour is preserved, not the bits.
• Reproduce a workflow. An agent’s final state depends on many calls, plus tools and external data. Matching one response is not the same as matching the run.

Reproducibility is also not correctness. A model can return the same wrong answer every time. It can also return different wording that is equally correct. Exact text equality, equivalent meaning, task success and safe execution are four different success criteria, and an application should choose one deliberately.

Temperature is only one LLM determinism control

At each step, the model produces a score (a logit) for every token in its vocabulary. Decoding turns those scores into a choice. Greedy decoding takes the highest-scoring token. Sampling draws from the probability distribution. Temperature reshapes that distribution: lower values concentrate probability on the top token, and temperature 0 is generally treated as greedy. Top-p and top-k limits trim the candidate set before the draw.

A seed fixes the random number generator used for sampling, so that two runs with the same probabilities make the same draws. OpenAI introduced a seed parameter in 2023. Its cookbook guide promised “(mostly) consistent outputs” on a “best effort” basis and stated that “determinism is not guaranteed”. It paired the seed with a system_fingerprint field identifying the backend configuration, warning that when the fingerprint differs “you may see different outputs”. The current Chat Completions reference now marks system_fingerprint as “Deprecated”. So the visible signal for backend change is going away even on the API that introduced it. We found no seed parameter documented for Anthropic’s Messages API, and Anthropic’s glossary says outright that temperature 0 is not fully deterministic, on its own service and through third-party clouds.

Both controls act on the last step of the computation. Greedy decoding is deterministic given its input, but its input is a vector of floating-point logits computed by everything upstream. If two top candidates are close and the upstream arithmetic shifts one of them slightly, greedy decoding faithfully picks a different token. There may also be implementation details in tie-breaking. Temperature and seeds narrow the variation that sampling adds. They do nothing about variation in the numbers being sampled from.

How the serving system breaks LLM determinism

The root mechanism is that floating-point addition is not associative. Real-number arithmetic says (a + b) + c equals a + (b + c). Floating-point arithmetic rounds after every operation, so the order matters. In 32-bit floats:

• (100,000,000 + (−100,000,000)) + 1 = 1
• 100,000,000 + ((−100,000,000) + 1) = 0

In the second order, adding 1 to −100,000,000 is lost to rounding, because 32-bit floats near that magnitude are spaced 8 apart. The same effect, at smaller scale, occurs whenever a GPU kernel sums thousands of products in a matrix multiplication or attention reduction. Change the order of the sum and the last few bits of the result can change.

That alone does not make a system nondeterministic. A fixed kernel on fixed hardware sums in a fixed order. NVIDIA’s cuBLAS documentation says its routines “generate the same bit-wise results at every run” on GPUs “with the same architecture and the same number of SMs”. It lists the exceptions: multi-stream execution, atomics mode and certain workspace conditions. It also notes that results are not guaranteed to match across toolkit versions.

The more interesting finding came from Thinking Machines Lab. In Defeating Nondeterminism in LLM Inference (September 2025), Horace He and colleagues argued that the common explanation, “concurrency plus floating point”, misses the main cause. A typical forward pass, they wrote, uses essentially no atomic operations and is run-to-run deterministic. What it is not is batch-invariant. Many kernels choose a different reduction strategy depending on how many tokens are in the batch, so one request’s numbers depend on how many other requests were processed alongside it. Server load sets the batch size, and load is outside any user’s control.

Their experiment makes the scale concrete. They ran 1,000 completions of the same prompt at temperature 0 on Qwen3-235B-A22B-Instruct-2507 and got 80 unique outputs. All 1,000 matched for the first 102 tokens, then split at token 103: 992 continued “Queens, New York” and 8 “New York City”. With batch-invariant kernels, all 1,000 completions were identical. The cost was speed. In their unoptimised test on a smaller model, a run that took 26 seconds with default vLLM took 55 seconds deterministically, or 42 seconds with an improved attention kernel. That is one implementation on one stack, but it establishes the mechanism. It also shows why providers optimising for throughput may not choose batch-invariant kernels by default.

Two qualifications matter. First, a numerical difference only changes text when it lands on a near-tie between candidate tokens. Most batch changes alter nothing visible, which is why the divergence above appeared at token 103 and not token 1. Second, once a single token differs, every later token is conditioned on a different prefix, so a one-bit difference can become a different paragraph.

Mixture-of-experts (MoE) models add a possible amplifier. A router scores each token and sends it to a few experts. If two experts score almost equally, a small numerical shift could change the assignment, and with it the computation path. Some MoE designs also cap how many tokens each expert accepts. The Switch Transformer paper sets expert capacity from the tokens in a batch and a “capacity factor”. Tokens beyond that capacity are “dropped” and passed through the residual connection without the expert’s computation. In such a design, whether a token is processed can depend on what else is in the batch. Whether any particular commercial model uses capacity limits at inference, and how often near-ties occur, is not publicly documented. This is a plausible mechanism, not a measured cause for any named API.

LLM determinism and the model behind the name

Even with perfectly deterministic kernels, the model itself can change. Four things are worth distinguishing:

• A model family: “Claude Sonnet”, “GPT-6”. This is a marketing label, not something you can reproduce against.
• A mutable alias: a name that resolves to the latest snapshot. Anthropic’s model ID documentation describes an alias such as claude-sonnet-4-5 as “a convenience pointer that resolves to the most recent dated snapshot”.
• A pinned snapshot: a fixed version. Anthropic says its IDs from the 4.6 generation onward map “to a single, fixed model snapshot” and that it “does not update the weights or configuration of an existing model ID”. OpenAI’s text generation guide recommends pinning production applications to snapshots for consistent behaviour.
• The serving implementation: kernels, hardware, quantisation and routing infrastructure.

The fourth layer can change while the third stays fixed. Anthropic’s documentation says so directly: “If you notice unexpected behavioral differences on a previously stable model ID, an infrastructure update is the most likely cause.” That is a candid statement, and it applies in principle to any hosted model. A pinned snapshot freezes the weights. It does not freeze the arithmetic.

Unchanged application code therefore proves very little. Tokenizer versions, default parameters, safety filters, system-level prompt handling and the hardware a request lands on can all differ while the request body stays byte-identical. And snapshots do not last forever. When one is retired, the replacement is a different model, and UniverseBlend’s account of what breaks when a model version retires covers the migration that follows.

When LLM nondeterminism becomes an agent security and reliability problem

In a single chat answer, a changed word is usually harmless. In an agent, outputs become inputs. A one-token difference can change:

• which tool the model selects, or the arguments it passes;
• the text of a retrieval query, and therefore which documents come back;
• an intermediate conclusion that later steps rely on;
• whether the agent decides it is finished, or retries;
• the exact downstream API call, and the state it leaves behind.

Because each step conditions the next, small divergences compound. Hypothetically: two runs of a support agent read the same ticket. One run’s first query returns a refund-policy document. The other’s, worded slightly differently, returns a shipping FAQ. From then on, the runs are reasoning about different evidence. Neither run is broken. The system simply has more than one plausible path, and a single test exercised only one of them.

This is where reliability shades into security, and the two should be kept apart. Ordinary variation is not an attack. But a safety property that holds on 99 runs and fails on the 100th is not a property the system has. Any defence that relies on the model “usually” refusing a dangerous action inherits that variance. The safer design assumes the model will sometimes choose differently and makes high-impact actions safe regardless:

• idempotent operations, so a repeated call does no extra harm;
• deterministic validation of tool arguments in code;
• bounded retries;
• approval gates for irreversible steps;
• logs detailed enough to reconstruct which path a run took.

It also explains why an agent can pass an evaluation one day and fail a near-identical run the next without having become less capable. The capability did not change. The sample did.

Why one evaluation run cannot establish LLM determinism

If outputs vary, every evaluation score is an estimate with error. Anthropic’s 2024 research note, A statistical approach to model evaluations (with the companion paper by Evan Miller), recommends:

• reporting the standard error of the mean with every score;
• clustering standard errors when questions are grouped;
• “resampling answers from the same model several times” to reduce within-question variance;
• comparing models with paired differences on the same questions;
• using power analysis to decide how many questions an evaluation needs.

None of those recommendations prescribes a universal sample size, and neither does this article. The right number of repeats depends on how variable the task is, how costly each run is, how risky a wrong conclusion would be, and how important the decision is. A model-selection decision for a regulated workflow justifies more trials than a prompt tweak for an internal tool. The practical rule is to keep running until the confidence interval is narrow enough that a plausible value at either end would not change your decision.

Variance is also only one of the reasons a public number misleads. Benchmarks can be contaminated, harnesses differ, and few report error bars. UniverseBlend’s case for building your own evals instead of trusting benchmark scores works through those gaps. For agents, measure the distribution of outcomes, not just the mean: how often the task succeeds, how it fails when it fails, and whether any failure involved an unsafe action. Two configurations with the same success rate can have very different worst cases.

A practical LLM determinism protocol for production teams

The aim is to make every surprising output diagnosable, then decide which layer to control.

1. Record the model identifier exactly as sent and as returned, preferring pinned snapshots to aliases. Store any version or fingerprint field the provider still returns.
2. Capture every request parameter: temperature, top-p, top-k, max tokens, stop sequences, seed, reasoning or thinking settings, and response format.
3. Version the context, not just the prompt: system instructions, tool definitions, JSON schemas, retrieval configuration and any preprocessing.
4. Log identifiers and timing: provider request IDs, timestamps, region, retries and their causes. A retried call is a second sample, not the same call.
5. Snapshot tool and retrieval results for each step, so a replay can separate model variation from changes in the data.
6. Test repetition deliberately. Run the same request many times in sequence and in parallel. Parallel requests are more likely to share batches with different neighbours, which exposes batch-dependent variation.
7. Diagnose before controlling. If the variation disappears at temperature 0, it was sampling. If it persists with fixed inputs, suspect serving and ask the provider, or test on a controlled stack. If it appears after a date, suspect a model or infrastructure change. If it appears only in multi-step runs, look at tools and external state.
8. Re-run a representative workload after any model change, alias move or announced retirement, and compare distributions, not single outputs.
9. Track failure modes by category (wrong tool, wrong argument, premature stop, unsafe action), not only pass rates.
10. Write down the reproducibility contract: the success criterion, the acceptable variation, and who signs off when it changes.

Logging a seed does not make a result reproducible across providers, model versions or serving stacks. It is one field in a record whose purpose is diagnosis.

What teams should promise about LLM determinism

The useful framework here is a contract with four levels. It is a synthesis, not an industry standard. Choose the level from the task’s risk and from what a changed answer would cost.

Most production teams need the last two and only claim the second. Hosted APIs generally cannot offer the first, because the batch a request lands in is not the customer’s to choose. Self-hosting can achieve it at a throughput cost, but self-hosting is not a cure-all. It moves the responsibility for kernels, hardware and versioning onto the team. Who controls each of those layers is also a vendor question. The model vendor risk checklist covers what to ask about notice periods and snapshot pinning.

Conclusion

A repeatable prompt is not a reproducible system. Temperature and seeds control the last step of a long computation. Batching, kernels, hardware, model versions and tools sit upstream and downstream of it. The practical principle is to decide which level of reproducibility the task needs, find out which layer is producing the variation, and control that layer. For any action that cannot be undone, rely on validation in code rather than on the model making the same choice twice.

FAQ

Does temperature zero guarantee deterministic LLM outputs?

No. Temperature 0 makes decoding choose the most likely token, but the probabilities themselves come from floating-point computation that can vary with batch size, kernels and hardware. Anthropic’s documentation states that results at temperature 0 “will not be fully deterministic”. Other hosted APIs behave similarly, unless the provider guarantees otherwise.

Why can the same prompt produce different answers?

Sampling is the obvious reason, but not the only one. The same prompt can be processed in a different batch, on different hardware or with a different kernel. It can also reach an updated model behind an alias, or receive different tool and retrieval results. Any of these can change one token, and later tokens follow from it.

Can a random seed make LLM responses reproducible?

A seed fixes the randomness used in sampling, so it removes one source of variation. It does not fix the numbers being sampled from. OpenAI’s own guidance called seed-based determinism “best effort” and said that “determinism is not guaranteed”. Seeds also don’t carry across model versions or providers.

Do model updates affect reproducibility?

Yes. A new snapshot behind an alias is a different model. Pinning a dated or fixed snapshot avoids that, but the serving infrastructure can still change. Anthropic names infrastructure updates as the most likely cause of behaviour differences on a stable model ID. When a snapshot retires, the replacement must be re-evaluated.

How should teams evaluate nondeterministic models?

Treat each evaluation as a statistical estimate. Run tasks repeatedly, report the mean with a standard error or confidence interval, and compare models on paired questions. Scale the number of trials to the task’s variance and the decision’s importance. For agents, also categorise failures, because averages hide unsafe outliers.

Can LLM agents be made fully deterministic?

Rarely, in practice. You would need deterministic inference (fixed weights, batch-invariant kernels, fixed hardware) and frozen tools and external data. That is achievable in a controlled test harness, not across live systems. Production agents should instead be made safe under variation, using validated actions, idempotency and approval gates.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Read more

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Read more

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

Checked against Hirundo’s and Lazarus AI’s published results, CrowdStrike’s DeepSeek-R1 study, Lawfare and the Federal Register on 4 November 2026. Ask Alibaba’s Qwen3.6-35B-A3B about Tiananmen …

Read more

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

Read more