You have probably noticed the pattern by now.
First, you experiment with GPT-4 for internal tools. Then, a few customer-facing features appear. Finally, someone forwards you the monthly OpenAI bill, and your eyes widen.
At what point does “just using the API” become a problem worth solving?
Not every company should self-host LLMs. For many companies, the API is the right solution: no infrastructure to manage and you only pay for what you use. Plus, the models keep improving.
However, there is a clear threshold. Here are five signs that you have crossed it.
Sign 1: Your Monthly API Bill Exceeds $500 (and Keeps Growing)
OpenAI’s pricing structure is straightforward. It costs $15 for every million output tokens for GPT-4o and $5 for GPT-4o-mini, plus input token charges.
Even a modest production workload, such as a support ticket summarizer, content classifier, or internal search assistant, can easily reach 1–2 million output tokens per day.
Do the math:
One million output tokens per day equals $450 per month for GPT-4o output alone.
Add input tokens, and you’re over $500.
Scale up to three to five features, and you’re in the thousands!
At $500–$1,000 per month, self-hosting becomes financially attractive. A dedicated AWS GPU instance (g4dn.xlarge) costs ~$0.53 per hour, or about $380 per month if running 24/7. However, you will not run it 24/7. Most workloads are bursty. With auto-shutdown on idle, your actual monthly cost will often be between $50 and $150.
The sign: You can point to a line item on your AWS or OpenAI bill and say, “This is growing faster than our usage.”
Sign 2: You Are Sending Sensitive Data to a Third-Party API
This is a non-negotiable requirement for some industries.
Every prompt sent to OpenAI, Anthropic, or Google passes through their servers. Despite their data privacy assurances, the legal risk may be too high.
Consider what your prompts contain:
- Customer PII (names, emails, addresses)
- Proprietary code or algorithms
- Internal strategy documents
- Financial data or trade secrets
- Medical or legal information
If any of these apply to your business, your legal or compliance team will eventually tell you to “stop.” And they will be right.
The sign: Your legal or security team has asked, “Where is this data going?” – and you did not have a good answer.
Self-hosted inference on your own AWS account ensures that your data never leaves your infrastructure. For companies under GDPR, HIPAA, SOC2, or similar frameworks, this is not an optimization – it’s a requirement.
Sign 3: You Have Predictable, Recurring Usage
OpenAI’s pay-per-token model is ideal for experimentation and unpredictable workloads. You only pay when you use it.
However, once your usage becomes predictable – such as a daily batch job, a steady stream of API calls from 9 to 5, or a consistent workload — you are paying a premium for elasticity that you don’t need.
Self-hosting flips the cost model. You pay a fixed hourly rate for the GPU instance, regardless of how many tokens you generate. The more you use it, the lower your per-token cost.
The break-even point is: Roughly 300,000–500,000 output tokens per day, depending on your instance choice and workload. Below that threshold, the API may be cheaper. Above that, self-hosting is more cost-effective.
The sign: Your daily token usage does not vary wildly. You can look at a graph and see a stable, predictable pattern.
Sign 4: You Are Frustrated by Rate Limits or Latency
OpenAI imposes rate limits based on your usage tier. These limits are expressed in terms of tokens or requests per minute. If you experience a sudden spike due to a marketing campaign, product launch, or batch processing, you may exceed these limits and start seeing HTTP 429 errors.
Your only options are to wait or request an increase in your limits, which takes time and negotiation.
With your own infrastructure, there are no rate limits beyond what your GPU can handle. For example, a g4dn.xlarge running Llama 3.1 8B can generate approximately two million tokens per hour (34 tokens per second × 3,600 seconds). Most workloads never come close.
Latency is another hidden advantage. API calls traverse the public internet, adding 100–300 ms of network delay before the model starts generating. A self-hosted endpoint in your AWS region (or peered VPC) can reduce that delay to a few milliseconds.
The sign: You have encountered a “Service Unavailable” or “Rate Limit Exceeded” error from an LLM provider, which cost you time or money.
Sign 5: You Want Model Independence (Vendor Lock-In is Real)
OpenAI has changed its pricing, deprecated models, and altered its terms of service. They have already done all three.
If your application is tightly coupled to the GPT-4 API, you are at their mercy. They could:
- Raise prices (they have);
- Discontinue a model you rely on (they have);
- Introduce new usage restrictions (they have); or
They could experience an outage.
Open-source models provide a hedge. Llama 3, Mistral, Qwen, and others are competitive with GPT-3.5 and are approaching the level of GPT-4 for many tasks. More importantly, they run on your infrastructure using an OpenAI-compatible API.
The key is that if you build your application using the OpenAI API specification (messages, chat completions, and streaming), you can point it at any compatible endpoint. Switching from OpenAI to a self-hosted Llama instance requires only a single URL change.
The sign: You should consider this if you’re concerned about what would happen if OpenAI changed something overnight, or if you simply want the option to switch providers without rewriting your application.
So, Should You Self-Host?
Add up the signs that apply to you.
| Signs You Checked | Verdict |
| 0–1 | Stick with managed APIs. The overhead of self-hosting is not worth it for you yet. |
| 2–3 | You are in the gray zone. A conversation makes sense – some workloads may be ready, others not. |
| 4–5 | You are likely overpaying, taking on unnecessary risk, or both. Talk to someone who has done this. |
If you checked three or more boxes, let’s have a 30-minute, no-obligation conversation. I will ask you about your workload, data, and constraints, and I will give you an honest answer about whether self-hosted AI is a good fit for your company.
I have already done this for my own infrastructure. Here are the numbers from my last deployment: 34 tokens per second generated and 1,093 tokens per second processed with prompt processing, all for $0.53 per hour on a g4dn.xlarge instance.
Your situation will be different. However, the above framework will help you determine if it is worth exploring.
About the Author
I am a freelance infrastructure engineer based in Romania, working with European, US and Israeli companies. I deploy private, cost-controlled AI endpoints on AWS. My last project: Llama 3.1 8B running at 34 tokens/second for $0.53/hour.