Temperature, Top-p and Max Tokens in a real SaaS app
What temperature, top-p and max tokens actually do, and the values I pick for real SaaS features: extraction, support replies, summaries, copywriting and agents. Includes a settings table, cost and latency effects, and the mistakes I see most.
In a demo, you leave every setting on default and the model looks great. In a real SaaS product, those defaults quietly decide three things your customers notice: how consistent the answers are, how long they take, and what each request costs.
Three parameters do most of that work: temperature, top-p and max tokens. This post explains what each one does in plain words, and the values I actually start with for common SaaS features.
How a model picks the next word
A model writes one token at a time. At each step it produces a probability for every possible next token, for example "the" 40%, "a" 25%, "this" 10%, and a long tail of unlikely ones. Temperature and top-p change how it chooses from that list. Max tokens decides when it must stop.
Temperature: how adventurous the choice is
Temperature reshapes the probabilities before a token is picked.
- Low (0 to 0.3): the top choices dominate. Answers are focused and repeat well. Good for anything a program will read or a customer will rely on.
- Medium (0.4 to 0.7): some variety, still on topic. Good for support replies and summaries.
- High (0.8 and up): unlikely tokens get a real chance. Answers vary a lot and can drift. Good for brainstorming and creative copy, risky for facts.
Two things worth knowing:
- Temperature 0 is more consistent, not perfectly identical. Small differences can still appear between calls.
- Higher temperature does not make a model smarter or more "creative" in a useful way. It makes it less predictable.
Top-p: how many choices are allowed
Top-p (nucleus sampling) keeps only the smallest group of tokens whose probabilities add up to p, and ignores the rest. With top-p 0.9, the model picks only from the most likely tokens that together cover 90% of the probability.
- Low top-p (0.1 to 0.5): very few candidates, very safe output.
- High top-p (0.9 to 1.0): nearly everything is allowed.
My rule: change one of temperature or top-p, not both. They overlap, and moving both makes results hard to reason about. I tune temperature and leave top-p at its default (usually 1.0). Some newer models also restrict or fix these values, so check the provider's docs before relying on them.
Max tokens: the ceiling, not the target
Max tokens caps the length of the response. It does not make the model write that much, and it does not shorten the answer politely. If the limit is hit, the output stops mid-sentence.
Why it matters in a SaaS app:
- Cost. You pay for output tokens, and they usually cost more than input tokens. A sensible cap limits the worst case.
- Latency. Output is generated token by token, so a long answer is a slow answer. A short cap keeps the UI snappy.
- Safety net. A runaway response, or a loop in an agent, stops at the cap instead of burning budget.
- Broken output. If you ask for JSON and the cap cuts it off, parsing fails. Set the cap above the largest valid answer you expect.
Always check the stop reason the API returns. If it says the length limit was reached, treat the answer as incomplete: retry with a higher cap, or ask for a shorter answer.
Starting values by feature
These are starting points, not laws. I always test on real examples before shipping.
| Feature | Temperature | Top-p | Max tokens | Why |
|---|---|---|---|---|
| Data extraction to JSON | 0 | default | Just above the largest expected object | A parser reads it, so it must be stable and complete |
| Classification, tagging, routing | 0 | default | Very small (10 to 50) | One label is enough, and it is cheap and fast |
| Support reply drafts | 0.3 to 0.5 | default | 300 to 600 | Natural wording, but no invented policies |
| Summaries of documents | 0.2 to 0.4 | default | Match the summary length you promise | Faithful to the source, predictable size |
| Marketing copy, titles, ideas | 0.7 to 1.0 | default | Depends on the format | Variety is the point |
| Code generation or fixes | 0 to 0.2 | default | Generous | Correctness beats variety |
| Agents and tool calling | 0 to 0.3 | default | Moderate per step | Predictable steps; a hard cap per step prevents loops |
Three things that bite in production
1. Retries behave differently. At temperature 0 a retry gives almost the same answer, so retrying a bad answer rarely helps. At a higher temperature a retry can fix it. For flaky formats I prefer to fix the prompt or validate the output, and use a retry with a slightly higher temperature only as a fallback.
2. Per-customer limits. Max tokens is also a product decision. A free plan can get a lower cap than a paid plan, and the cap belongs in your plan configuration, not hard-coded in a prompt.
3. Streaming hides length, not cost. Streaming makes a long answer feel faster, but the user still waits for it and you still pay for every token. Keep the cap anyway.
A simple way to tune them
- Collect 20 to 50 real inputs from your own product, including awkward ones.
- Start from the table above and run each input several times.
- Look at three things: is the output correct, is it consistent between runs, and did it ever get cut off?
- Change one setting at a time, and log the settings with every request so you can explain a bad answer later.
- Track average output tokens and latency per feature. Lower the cap if answers are consistently shorter than it.
Conclusion
Treat these three settings as part of the feature, not as an afterthought. Use low temperature when a program or a customer depends on the answer, a bit more when natural wording matters, and high only when variety is the goal. Leave top-p alone, and set max tokens just above the longest answer you expect, so cost and latency stay under control and nothing gets cut off silently.
Get these right and the model feels reliable, and your bill stays predictable.
