For the past two years, the conversation about AI has mostly been about access. Which API? Which model tier? Which provider? The underlying assumption of most of that discussion is that the computing power is located elsewhere, accessible by calling it, and paid for by the token.
This assumption is worth revisiting.
What changed
The open-weight models are good. They’re not just “good enough for demos.” They’re actually competitive with Frontier Cloud APIs for most real tasks. These models include Qwen 2.5, Mistral, Llama 3, and Gemma 2. For tasks such as coding assistance, document analysis, writing, internal Q&A, classification, and data extraction, the difference in performance between these models running locally and a paid cloud API is negligible for most business applications.
This occurred at the same time that hardware advanced. For example, Apple’s M4 Ultra ships with 192 GB of unified memory, meaning it can run models that would have required a rack of servers two years ago. NVIDIA’s RTX 50-series cards have dedicated inference cores. AMD’s Ryzen AI processors integrate neural processing units. A mid-range desktop build with an RTX 4070 can handle 13-billion-parameter models in real time.
The software has caught up, too. Tools like Ollama and LM Studio have almost eliminated friction. Pulling and running a capable model locally is now an afternoon task, not a week-long infrastructure project.
What it means for developers
Developers are the group that feels this shift the most directly. In practice, a local model running in the background is now considered a legitimate development tool rather than a research curiosity.
This includes code completion, commit message drafting, test generation, documentation, and quick explanations of unfamiliar codebases. All of these features can run on the same machine you’re working on with no API latency, rate limits, or cost per call. For developers who spend a lot of time with AI tooling, the monthly API bill is becoming harder and harder to ignore.
The deeper change is that local inference makes certain workflows viable that weren’t before. These include tight loops where you call a model dozens of times to process or transform data; pipelines that run on sensitive internal code that you’d rather not send anywhere; and local agents that need low latency to be usable. These workflows are genuinely different when the model is on your machine.
What it means for everyone else
For non-developers, the question is simpler: Do you want an AI assistant that processes your data without leaving your building?
For most individual users, cloud APIs are still the easier option. Examples include ChatGPT, Claude, and Copilot. They’re well-supported and constantly improving. The privacy tradeoff is acceptable for most personal and general professional use.
However, this changes when the data is sensitive. For example, a legal professional running contracts through an AI. An accountant analyzing client financials. A healthcare administrator could be summarizing patient documents. For these use cases, local inference isn’t just a technical preference; it’s also a compliance consideration. The model doesn’t call home. Nothing is logged on a third-party server. The data stays exactly where it started.
The tools are now accessible enough that a technically confident person outside of development can set something up. However, it’s still not a five-minute setup if you’re starting from scratch. However, it’s no longer a six-month project.
What it means for companies
For larger organizations, the use of local AI has evolved from a niche infrastructure discussion to a legitimate strategic consideration.
The cost is real. For a ten-person team with moderate AI usage, a local inference server in the €2,500 to €7,500 range usually pays for itself in three to five months compared to cloud API costs. Afterward, it runs for free. At scale, the economics shift decisively toward local deployment once usage crosses a threshold that most active teams reach sooner than expected.
However, the cost argument is almost secondary to the control argument. Running your own models means no vendor dependency, no surprise pricing changes, and no terms-of-service updates affecting how your data is handled. It means having AI that works inside your network perimeter and is accessible via an internal API. You can audit and govern its usage. For companies in regulated industries or those with significant intellectual property concerns, this level of control is valuable and cannot be captured in a simple cost comparison.
The risk lies in the operational overhead. Running a local inference stack requires infrastructure maintenance. Hardware fails, models need updating, and configurations drift. Companies that embark on this path without designating a clear owner for the stack often end up with a capability that technically exists but practically doesn’t, much like that Kubernetes cluster nobody understood.
The “token chase” problem
Cloud providers are in a race that benefits them more than their customers.
Each new model release is presented as a significant advancement, and some genuinely are. However, the practical result for most business users is higher prices and more complex pricing tiers chasing capabilities that far exceed what everyday tasks require. The difference in capability between GPT-4-class cloud APIs and a capable open model running locally used to justify the premium. For most use cases today, however, it doesn’t – at least not by as much as the pricing difference suggests.
The providers know this, which is why the frontier race keeps moving. Longer context windows, multimodal inputs, and reasoning models are a few examples. Some of these matter for specific applications. Most of them solve problems that a three-person accounting firm doesn’t have.
Where things actually stand
Local AI is not right for everyone, nor is it a replacement for cloud APIs in every situation. If you need a state-of-the-art reasoning model, access to live data, or a system that a nontechnical team can use without setup, then cloud APIs are still the better option.
However, “local is too complicated” and “local models aren’t good enough” are outdated objections. The hardware is there. The software is there. The models are there. The only thing left is a genuine decision about whether the trade-offs make sense for a specific situation. There are no technical barriers that rule it out.
It’s worth making that decision now, before the cloud API bills make it for you.
get in touch. I reply within 24 hours.