Every LLM feature is governed by three forces that refuse to sit still at once: how much it costs, how fast it responds, and how good the output is. Pull one and the other two move. Reach for the biggest model to lift quality and your latency and bill both jump. Cache aggressively to cut cost and latency and you risk serving stale or subtly wrong answers. Like the three-body problem in physics, there is no clean closed-form solution — only stable configurations you engineer toward and then defend.

The failure mode we are called in to fix is almost always the same: a team optimised one axis by accident. Someone picked a model in week one, wired it to a synchronous endpoint, and shipped. Six months later the feature is slow, expensive and nobody can change it without fear. This is a field guide to trading the three forces deliberately instead.

Measure before you tune

You cannot optimise what you have not instrumented. Before touching a single knob, put numbers on all three axes for every LLM call in the system:

  • Cost: input and output tokens per request, and cost per successful task — not per call, because retries and multi-step agents hide the real number.
  • Latency: time-to-first-token and total time, at p50, p95 and p99. Averages lie; your angriest users live in the tail.
  • Quality: a task-specific score from an eval suite, not a gut feeling. If you cannot express quality as a number, every optimisation below is flying blind.
Optimising cost or latency without a quality gate is not optimisation — it is quietly degrading your product and calling it a win. The eval suite is the seatbelt that lets you drive fast.

Route, don't standardise

The most expensive mistake is using one large model for every request. Most traffic is easy; a minority is hard. A router that sends simple requests to a small, cheap, fast model and escalates only the hard ones to a frontier model routinely cuts spend by half while improving tail latency, because the easy cases no longer queue behind heavyweight inference.

async def answer(query, context):
    tier = classify_difficulty(query, context)   # cheap, fast, cached

    if tier == "simple":
        return await small_model(query, context)  # ~10x cheaper

    draft = await small_model(query, context)
    if judge_confidence(draft) >= 0.8:
        return draft                               # good enough, done

    return await frontier_model(query, context)    # escalate the hard 15%

The classifier itself must be cheap — a small model, a heuristic, or a distilled scorer — or it eats the savings. And every routing decision belongs in your logs, so you can see the escalation rate drift and re-tune the thresholds as traffic evolves.

Caching is the highest-leverage lever

Nothing beats the latency and cost of a request you never make. Three layers compound:

  1. Exact-match cache. Identical prompt in, cached response out. Trivial to build, and in many products a surprising share of traffic is genuinely repeated.
  2. Semantic cache. Embed the query, and if a past query is close enough in vector space, reuse its answer. Powerful, but it needs a similarity threshold you have validated — too loose and you serve confidently wrong answers.
  3. Prompt (prefix) caching. Most providers now cache a stable prompt prefix so a long system prompt or retrieved context is not re-processed on every call. Put your static instructions and large fixed context first; keep the volatile user input last. This alone can cut both latency and input cost dramatically on long-context features.

Latency is mostly perception

Total generation time matters, but for interactive features time-to-first-token matters more. Stream tokens as they arrive and a two-second response feels instant because the user sees motion immediately. Beyond streaming, the biggest latency wins are structural:

  • Parallelise independent steps. If an agent needs three retrievals and a classification, fire them concurrently instead of in a chain. Latency collapses to the slowest branch, not the sum.
  • Shorten outputs. Output tokens dominate both latency and cost. Ask for structured, concise results; do not let the model narrate. Capping max_tokens honestly is free speed.
  • Cut round-trips. Every agent hop is a full network-plus-inference round-trip. Fewer, better-scoped steps usually beat many small ones.

Buy quality where it is cheapest

When quality is the axis you need to move, reach for the cheapest lever first. In our experience the ranking is remarkably consistent:

  1. Better context. Improving retrieval — cleaner chunks, tighter ranking, the right documents — lifts answer quality more than any model upgrade, and it is cheap and fast.
  2. Better prompt. A sharper instruction and one or two well-chosen examples often close the gap without changing the model at all.
  3. A verification pass. A second, cheaper call that checks the first against the source catches a large fraction of errors for a fraction of the cost of a bigger model everywhere.
  4. A bigger model. Real, but the most expensive lever on both cost and latency. Use it for the residue the cheaper levers cannot reach — which is exactly what the router is for.

Set budgets, not just targets

Give each feature an explicit budget: a cost ceiling per task and a p95 latency SLO, with a minimum quality floor that must never be breached. Wire those into CI. A pull request that saves 20% on cost but drops the quality score below the floor should fail the build, loudly, the same way a failing test does. Budgets turn the three-body problem from an endless argument into a set of numbers the whole team can reason about.

The stable configuration

Put together, a well-balanced LLM feature tends to look like this: a cheap classifier routing traffic by difficulty, a layered cache absorbing repeated and near-repeated work, streamed responses for perceived speed, parallelised sub-steps, disciplined output lengths, and a frontier model reserved for the genuinely hard minority — all sitting behind an eval suite and explicit budgets enforced in CI.

None of these forces is ever solved for good. Traffic shifts, models change, prices move, and a configuration that was optimal last quarter drifts. The teams that win are not the ones who found the perfect settings once; they are the ones who instrumented all three axes, made every trade-off visible, and can re-balance in an afternoon instead of a re-write. Treat cost, latency and quality as one system, measure all three, and never move one without watching the other two.

N
Netatum AI Team
AI Integration & Consulting