Two requests can hit the same model at the same GPU utilization and still feel completely different. Input length, expected output, queue depth, batching and cache state all matter. Adding accelerators may quiet the symptom, but it does not explain which delay was removed or what the next bottleneck will be.
Measure the delay users actually notice
Track queue time, time to first token, inter-token latency and total generation time separately. A chat product is sensitive to the first token, while an offline extraction job mostly cares about completion time.
Slice P50, P95 and P99 by model, feature, input length and priority. A single service-wide average conceals the workload that repeatedly waits.
A request count is not a capacity unit
Short classification and long document synthesis are both one API call but consume very different amounts of memory and compute. Record input tokens, output tokens, cancellation and routing decisions.
Prompt changes can alter infrastructure demand even when traffic is flat. Treat major prompt and context changes like deploys and compare the distributions before and after.
Batching trades latency for throughput
Continuous batching can improve accelerator use, but waiting for a batch and mixing long with short requests may hurt interactive traffic. Separate online, internal automation and bulk evaluation with queues or concurrency limits.
Autoscaling must also include model load and warm-up time. Scaling after a queue is already deep is often too late for the users currently waiting.
Price completed work, not GPU hours
Include retries, failed generations, cache, network and idle warm capacity in cost per successful request. Compare that number by product feature because quality, latency and model requirements differ.
A useful load test mixes short and long conversations, disconnects and multi-turn context. Record the point where the queue stops recovering and agree on a degradation policy before production traffic reaches it.