Welcome to my digital garden. 🌱

Tim Le

Temperature, Top-p and Max Tokens in a real SaaS app

What temperature, top-p and max tokens actually do, and the values I pick for real SaaS features: extraction, support replies, summaries, copywriting and agents. Includes a settings table, cost and latency effects, and the mistakes I see most.

Temperature, Top-p and Max Tokens in a real SaaS app

In a demo, you leave every setting on default and the model looks great. In a real SaaS product, those defaults quietly decide three things your customers notice: how consistent the answers are, how long they take, and what each request costs.

Three parameters do most of that work: temperature, top-p and max tokens. This post explains what each one does in plain words, and the values I actually start with for common SaaS features.

How a model picks the next word

A model writes one token at a time. At each step it produces a probability for every possible next token, for example "the" 40%, "a" 25%, "this" 10%, and a long tail of unlikely ones. Temperature and top-p change how it chooses from that list. Max tokens decides when it must stop.

Temperature: how adventurous the choice is

Temperature reshapes the probabilities before a token is picked.

  • Low (0 to 0.3): the top choices dominate. Answers are focused and repeat well. Good for anything a program will read or a customer will rely on.
  • Medium (0.4 to 0.7): some variety, still on topic. Good for support replies and summaries.
  • High (0.8 and up): unlikely tokens get a real chance. Answers vary a lot and can drift. Good for brainstorming and creative copy, risky for facts.

Two things worth knowing:

  • Temperature 0 is more consistent, not perfectly identical. Small differences can still appear between calls.
  • Higher temperature does not make a model smarter or more "creative" in a useful way. It makes it less predictable.

Top-p: how many choices are allowed

Top-p (nucleus sampling) keeps only the smallest group of tokens whose probabilities add up to p, and ignores the rest. With top-p 0.9, the model picks only from the most likely tokens that together cover 90% of the probability.

  • Low top-p (0.1 to 0.5): very few candidates, very safe output.
  • High top-p (0.9 to 1.0): nearly everything is allowed.

My rule: change one of temperature or top-p, not both. They overlap, and moving both makes results hard to reason about. I tune temperature and leave top-p at its default (usually 1.0). Some newer models also restrict or fix these values, so check the provider's docs before relying on them.

Max tokens: the ceiling, not the target

Max tokens caps the length of the response. It does not make the model write that much, and it does not shorten the answer politely. If the limit is hit, the output stops mid-sentence.

Why it matters in a SaaS app:

  • Cost. You pay for output tokens, and they usually cost more than input tokens. A sensible cap limits the worst case.
  • Latency. Output is generated token by token, so a long answer is a slow answer. A short cap keeps the UI snappy.
  • Safety net. A runaway response, or a loop in an agent, stops at the cap instead of burning budget.
  • Broken output. If you ask for JSON and the cap cuts it off, parsing fails. Set the cap above the largest valid answer you expect.

Always check the stop reason the API returns. If it says the length limit was reached, treat the answer as incomplete: retry with a higher cap, or ask for a shorter answer.

Starting values by feature

These are starting points, not laws. I always test on real examples before shipping.

FeatureTemperatureTop-pMax tokensWhy
Data extraction to JSON0defaultJust above the largest expected objectA parser reads it, so it must be stable and complete
Classification, tagging, routing0defaultVery small (10 to 50)One label is enough, and it is cheap and fast
Support reply drafts0.3 to 0.5default300 to 600Natural wording, but no invented policies
Summaries of documents0.2 to 0.4defaultMatch the summary length you promiseFaithful to the source, predictable size
Marketing copy, titles, ideas0.7 to 1.0defaultDepends on the formatVariety is the point
Code generation or fixes0 to 0.2defaultGenerousCorrectness beats variety
Agents and tool calling0 to 0.3defaultModerate per stepPredictable steps; a hard cap per step prevents loops

Three things that bite in production

1. Retries behave differently. At temperature 0 a retry gives almost the same answer, so retrying a bad answer rarely helps. At a higher temperature a retry can fix it. For flaky formats I prefer to fix the prompt or validate the output, and use a retry with a slightly higher temperature only as a fallback.

2. Per-customer limits. Max tokens is also a product decision. A free plan can get a lower cap than a paid plan, and the cap belongs in your plan configuration, not hard-coded in a prompt.

3. Streaming hides length, not cost. Streaming makes a long answer feel faster, but the user still waits for it and you still pay for every token. Keep the cap anyway.

A simple way to tune them

  1. Collect 20 to 50 real inputs from your own product, including awkward ones.
  2. Start from the table above and run each input several times.
  3. Look at three things: is the output correct, is it consistent between runs, and did it ever get cut off?
  4. Change one setting at a time, and log the settings with every request so you can explain a bad answer later.
  5. Track average output tokens and latency per feature. Lower the cap if answers are consistently shorter than it.

Conclusion

Treat these three settings as part of the feature, not as an afterthought. Use low temperature when a program or a customer depends on the answer, a bit more when natural wording matters, and high only when variety is the goal. Leave top-p alone, and set max tokens just above the longest answer you expect, so cost and latency stay under control and nothing gets cut off silently.

Get these right and the model feels reliable, and your bill stays predictable.