Self-Hosted AI vs Cloud: Which Is Right for Your Business?

The question is not which is better. It is which fits your specific situation: your workload, your compliance exposure, and whether you have engineers who can actually run this thing. Self-hosted AI and cloud AI are not competing philosophies. They are different tools with different tradeoffs, and the right answer depends almost entirely on what you are building and for whom.

This guide covers the real tradeoffs. Cost structure, compliance requirements, operational overhead, and the criteria that matter for production deployments. No cheerleading for either side.

the short answer: it depends on three things#

Most self-hosted vs. cloud discussions bury the decision criteria deep in the post. Here it is upfront.

your compliance requirements#

If you operate in healthcare, legal, or financial services and your workloads touch regulated data -- patient records, client communications, financial transactions -- your compliance framework may make the decision for you before the cost math is even relevant.

Cloud APIs can satisfy some requirements via Business Associate Agreements (BAAs) and data processing addenda, but they cannot satisfy all of them. HIPAA requires that electronic Protected Health Information (ePHI) never leaves a controlled environment. A 2026 ruling in the Southern District of New York found that documents created using commercial generative AI tools and shared with an attorney are not protected by attorney-client privilege, because communications with public AI platforms lack the required confidentiality elements (Debevoise Data Blog, February 2026). For legal professionals, this turns an architecture question into a legal risk question -- the kind you probably want answered before you build anything.

Self-hosted is not optional in these contexts. It is what makes compliance possible.

your token volume and usage pattern#

Cloud AI is cheaper at low and irregular volume. Self-hosted is cheaper at high and predictable volume. The break-even varies by model and hardware, but a consistent pattern holds: organizations processing fewer than 5-10 million tokens per month are almost always better served by cloud APIs. Organizations processing 100M or more tokens per month -- particularly those with predictable workloads -- can see annual savings of $5M to $50M by owning their inference layer (IDC, 2025).

your operational capacity#

Self-hosting is not plug-and-play. It requires real engineering to deploy, harden, and maintain. A production deployment involves model selection, hardware sizing, inference engine configuration, security hardening, monitoring, and ongoing update management -- and that is before anything breaks. If that capacity does not exist in-house, the operational overhead of self-hosting often exceeds the cost savings, at least in the first year.

what "self-hosted AI" actually means#

Self-hosted AI means running a language model entirely inside your own infrastructure: on your hardware, on servers you control, inside your network perimeter. No data leaves your environment. No third party processes your prompts or completions.

on-premise vs. private cloud vs. hybrid#

There are three common deployment models:

• On-premise: Models run on physical hardware inside your facility. Maximum control, no dependency on external cloud infrastructure. Requires upfront hardware investment and physical maintenance.
• Private cloud: Models run on cloud infrastructure (AWS, Azure, GCP) provisioned exclusively for your organization. You get elasticity without shared tenancy. The data lives on cloud infrastructure, but is not commingled with other customers.
• Hybrid: Sensitive workloads run on-premise or in a private cloud. General-purpose workloads run on managed cloud APIs. Most mature production deployments land here after a few iterations.

what self-hosting solves that a BAA or VPN does not#

This distinction matters for compliance work: a Business Associate Agreement with a cloud AI provider is not the same as self-hosting. A BAA defines who is responsible when something goes wrong. It does not prevent the data from transiting or being processed by the provider's infrastructure.

If your requirement is that data never leaves your environment -- not just that there is a contract in place if it does -- a BAA is not sufficient. Self-hosted is the architectural requirement.

the tools that make self-hosting viable: Ollama, vLLM, Open WebUI#

The open-source tooling for self-hosted AI has matured considerably over the past two years. Three tools are in wide production use:

• Ollama: Best for single-user and small-team deployments. Simple setup, runs capable models on a laptop or small workstation.
• vLLM: High-throughput inference engine for production. The right choice for multi-user or high-volume deployments.
• Open WebUI: User-facing interface layer with access controls, conversation history, and model management.

Deploying a private LLM is now an engineering project, not a research project. That is a meaningful shift from where things were in 2023.

what "cloud AI" actually means#

Cloud AI typically refers to accessing language models via managed APIs: you send a request, receive a response, and pay per token. Three tiers are worth distinguishing.

fully managed cloud APIs (OpenAI, Anthropic, Google Vertex)#

The fastest path to production. No infrastructure to manage, immediate access to frontier models, simple per-token billing. Data is processed on shared infrastructure. Most providers have enterprise agreements and data processing addenda, but the data transits and is processed by their systems.

private cloud endpoints (AWS Bedrock, Azure OpenAI): closer but not the same#

A middle tier. You access models through a major cloud provider's infrastructure within your existing cloud environment, with stronger isolation than public APIs. But the data still lives on the provider's infrastructure. You do not own the hardware, and the provider's terms govern.

what the terms of service actually say about your data#

Most enterprise cloud AI agreements explicitly prohibit training on customer data. But "not training" is different from "not retaining" or "not processing." Sensitive data now makes up 34.8% of employee inputs to AI tools as of 2025, up from 11% in 2023 (LeanLaw / industry research, 2025). Read the data processing addenda carefully. Do not assume cloud AI is compliant for your use case.

when self-hosted wins#

compliance requirements that cloud APIs cannot satisfy#

Healthcare providers whose workloads touch ePHI, law firms processing confidential client matter data, and financial services firms under SEC Regulation S-P and FINRA Rule 3110 face compliance requirements that cannot be satisfied by routing data through a third party's infrastructure, regardless of the contract terms.

44% of enterprises cite data privacy and security as the top barrier to LLM adoption (Kong Enterprise AI Report, 2025). For these organizations, self-hosting is not a preference. It is the prerequisite.

high-volume, predictable workloads where the cost math flips#

Self-hosted infrastructure amortizes quickly when inference load is high and consistent. Organizations processing 100M+ tokens monthly can save $5M-$50M annually compared to cloud API pricing at scale (IDC, 2025). The break-even for most configurations falls between 5M and 60M tokens per month depending on model size and hardware.

GPU prices have dropped 40-60% since 2024 (Northflank AI Hosting Report, 2026). The hardware case for self-hosting is stronger now than it was 18 months ago, and it keeps improving.

proprietary data and IP protection#

If your AI system is trained on or retrieves from proprietary internal data -- customer records, internal documentation, trade-sensitive workflows -- you may have business reasons beyond regulatory compliance to keep that data from transiting external infrastructure. Self-hosting keeps it inside your environment by architecture.

when cloud AI wins#

early-stage and variable-volume workloads#

If you are building your first AI feature, running a pilot, or have unpredictable usage that spikes and drops, cloud APIs are almost always the right starting point. No upfront hardware cost, no infrastructure to maintain, instant access to capable models. This is not a concession. It is the correct technical decision at that stage.

frontier model access without infrastructure build#

The best-performing models are only available via cloud APIs. Open-source models have closed the gap significantly for many tasks -- Llama 4, Mistral, and Qwen perform well on broad benchmarks -- but for tasks where frontier model capability matters, cloud is your only path today.

speed to production#

A cloud API integration can ship in days. A self-hosted deployment takes weeks at minimum: hardware procurement or cloud provisioning, model evaluation, infrastructure setup, security hardening, access control, and documentation. If time to production is the constraint, cloud wins.

the real cost comparison#

The cost comparison between self-hosted and cloud AI has three components that most analyses get wrong.

cloud API cost structure: per-token pricing at scale#

Pay-per-token pricing is predictable at low volume and expensive at high volume. At 10 million tokens per month, a typical cloud API cost is manageable. At 1 billion tokens per month, the math shifts significantly. Organizations running large-scale AI workloads regularly encounter five-figure monthly API bills that were not anticipated when the project started small.

self-hosted cost structure: upfront hardware + ongoing ops#

The true costs:

• Hardware or cloud GPU rental
• Initial deployment and configuration engineering (typically 2-4 weeks of senior engineer time for a production deployment)
• Ongoing operations (10-20 hours of DevOps time per month)

A proper production deployment ranges from $15,000-$25,000 for a single-model on-premise setup to $40,000-$80,000 for a multi-model, multi-user enterprise deployment with compliance documentation. The single most underestimated cost is model update management. Each major model update requires 1-2 weeks of engineering time, adding approximately $17,000-$46,000 annually in labor at senior engineer rates (AI Pricing Master, 2026). Most teams only learn this after the first major update cycle.

the break-even: what volume makes self-hosting cheaper#

Most configurations break even between 5M and 60M tokens per month. At 60M+ tokens per month with a 70B model, self-hosting is typically cheaper than cloud API pricing -- often significantly so. Below 5M tokens per month, cloud APIs are almost always cheaper once you account for all self-hosting costs.

compliance: where the decision gets made for you#

HIPAA: ePHI and the business associate agreement gap#

The HIPAA Security Rule requires covered entities and business associates to implement technical safeguards that ensure the confidentiality, integrity, and availability of ePHI. A Business Associate Agreement with a cloud AI provider creates contractual accountability. It does not prevent the ePHI from transiting or being processed by the provider's infrastructure.

Some cloud providers offer HIPAA-eligible configurations, but the implementation details require careful due diligence. Self-hosting eliminates this category of risk by keeping ePHI inside your network perimeter.

For a detailed breakdown of HIPAA compliance for AI systems, see Building HIPAA-Compliant AI Systems.

attorney-client privilege: why cloud AI creates legal exposure#

The February 2026 SDNY ruling determined that confidential client matter data processed through public commercial AI platforms loses its privilege protection because the communications lack the required confidentiality elements. The ruling is narrow but the implication is clear: law firms that use public cloud AI tools to process client matter data are creating legal exposure.

Bar association guidance in multiple states has trended toward requiring firms to conduct due diligence on how AI tools handle client data. Self-hosted infrastructure -- where client data never reaches a third-party system -- is the architecturally sound response.

GDPR and data sovereignty: where your data physically lives matters#

The EU's General Data Protection Regulation requires that personal data of EU residents be processed in accordance with data subject rights, including the right to erasure. If your AI system processes personal data of EU residents, data residency matters. GDPR fines reached 1.2 billion euros in 2024 -- enforcement is active and increasing (Secure Privacy, 2026).

the hybrid architecture most production systems land on#

Few mature production deployments are purely one or the other. The pattern that most engineering teams arrive at after 12-18 months looks like this:

• Sensitive or regulated workloads run on self-hosted or private-cloud infrastructure: patient records, legal documents, financial transactions, proprietary internal data.
• General-purpose or public-facing workloads run on managed cloud APIs: customer-facing interfaces, content generation, search and summarization where input data is not sensitive.
• Frontier model capability when needed is accessed via cloud APIs for specific high-stakes tasks where open-source models are not yet competitive.

You get private infrastructure where the compliance and cost math requires it, and cloud APIs where you need speed or frontier capability. It is not a compromise -- it is just how the math works out.

what ongoing operations actually look like#

self-hosted: maintenance, updates, and the engineering overhead nobody advertises#

A production self-hosted deployment is a live system. Model updates are not automatic -- you evaluate new versions, test them against your workloads, and deploy deliberately. Each major model update typically requires 1-2 weeks of engineering time. Security patches for the inference stack, operating system, and supporting tools need to be applied on their own schedule. Monitoring and alerting need to be configured, and then someone has to watch them.

The realistic ongoing ops cost for a single-model deployment is 10-20 hours of DevOps time per month, plus engineering time for model updates. If you do not have that capacity in-house, it needs to be contracted.

cloud: dependency risk, vendor pricing changes, and rate limits#

Managed APIs require almost no operational overhead on your end. In exchange: you do not control when models change (providers update on their own schedule, which can affect outputs), you hit rate limits during peak demand, and you are exposed to vendor pricing changes. API prices have trended downward, but that trend is not guaranteed. Dependency on a single provider's pricing and terms is a real business risk for any system with significant AI spend.

decision framework: which path is right for your workload#

Choose self-hosted if:

• Your workload touches ePHI, confidential legal matter data, or financial data subject to SEC Regulation S-P or FINRA Rule 3110
• You process more than 30-60 million tokens per month consistently
• You have proprietary training data or retrieval data that cannot leave your network
• You need full control over model selection, updates, and fine-tuning
• Your compliance documentation requires architectural evidence that data never left your environment

Choose cloud AI if:

• You are in early development or running a pilot
• Your usage volume is low, irregular, or growing unpredictably
• You need frontier model capability for tasks where open-source models are not competitive
• You have no data sensitivity requirements for the specific workload
• You need to ship in days, not weeks

Consider hybrid if:

• Your organization has both regulated and non-regulated workloads
• You want to start with cloud and migrate high-volume workloads to self-hosted once usage stabilizes
• Different departments have different compliance requirements

For a technical review of your specific workload and compliance environment, Silverthread Labs offers a free automation audit that covers architecture decisions alongside operational assessment.

FAQ#

What is the difference between self-hosted AI and cloud AI? Self-hosted AI runs language models entirely inside your own infrastructure: on hardware you control, inside your network. Your data never leaves your environment. Cloud AI routes your requests through a third-party provider's infrastructure via API. Self-hosted gives you full data sovereignty; cloud gives you faster deployment and access to frontier models.

When does self-hosting AI become cheaper than cloud APIs? The break-even depends on model size, hardware configuration, and usage patterns. The general range is 5M-60M tokens per month. Below that, cloud APIs are typically cheaper once you account for hardware, deployment engineering, and ongoing operations. Above 60M tokens per month, self-hosting almost always wins on cost. Organizations processing 100M+ tokens monthly can save $5M-$50M annually by owning their inference layer (IDC, 2025).

Is self-hosted AI HIPAA compliant? It can be -- ePHI never leaves your network, which eliminates the third-party data exposure that cloud APIs create. But that requires proper implementation: network segmentation, access controls, audit logging, encryption at rest and in transit, and documentation. Self-hosting is the prerequisite for HIPAA compliance in AI workloads, not a guarantee of it.

Does using cloud AI violate attorney-client privilege? A 2026 ruling by a U.S. District Court (SDNY) found that documents created using commercial generative AI tools and shared with an attorney are not protected by attorney-client privilege, because the communications lack required confidentiality elements when processed by a public AI platform. Self-hosted infrastructure -- where client matter data never reaches a third-party system -- removes that exposure.

What are the hidden costs of self-hosting an LLM? The costs most often underestimated: deployment engineering (2-4 weeks of senior engineer time for a production setup), ongoing model update management (1-2 weeks per major update, roughly $17,000-$46,000 in annual labor), and ongoing DevOps (10-20 hours per month). Hardware or cloud GPU costs are usually well-estimated. Engineering time is not.

Can a small business run a self-hosted AI model? Technically yes -- Ollama can run capable models on a laptop or a single GPU workstation. Practically, a reliable, secure, and maintained production deployment requires engineering capacity most small businesses do not have in-house. Without compliance requirements, cloud APIs are almost always the right choice. In regulated industries, the compliance case for self-hosting is real, but the operational overhead usually means contracting the deployment and maintenance out.