Asked which model to use, most teams debate benchmark scores. In practice the decision is made by four other factors, and capability is usually the least binding of them once the task is well specified.
Volume changes the arithmetic
At low volume, a hosted frontier model is almost always cheaper once you account for the engineering time to run anything yourself. At high, sustained volume on a narrow task, self-hosting an open-weight model can invert that — but only if utilisation is high. Idle GPUs are more expensive than API calls, and bursty traffic favours hosted endpoints strongly.
Self-hosting pays when your GPUs are busy. Most workloads are burstier than teams assume.
Data residency is often the real constraint
For regulated sectors the question is not which model is best but which deployment is permissible. Where data cannot leave a jurisdiction or a VPC, open-weight models in your own infrastructure may be the only compliant option — and that constraint, not benchmark performance, decides the architecture.
Latency and the tail
Average latency is a poor guide; the 95th percentile is what users feel. Hosted frontier endpoints are fast in the median and occasionally slow under load. A self-hosted small model can offer far more predictable latency, which matters more than raw quality for interactive features where a two-second wait breaks the interaction.
The operational cost nobody prices
Running your own inference means owning GPU capacity planning, model upgrades, batching, quantisation choices and being on call when it is down. That is a real engineering commitment. For a team without platform engineers, a hosted endpoint is not a compromise, it is a correct allocation of scarce attention.
Tier rather than choose
The framing as a binary is the mistake. Production systems typically use both: a small open-weight model for high-volume classification and extraction, a frontier model for the reasoning steps that justify the price. This gets most of the cost saving without putting hard tasks on a model that cannot do them.
Keep the door open
Whatever you choose, put the model behind an interface of your own so switching is a configuration change rather than a rewrite. Pin versions explicitly, keep the evaluation suite provider-agnostic, and re-run it when a new model appears. The field moves fast enough that today's right answer expires.
