Reference architecture for enterprise LLM deployment covering multi-tenant isolation, GPU sizing, compliance controls, cost allocation, and the gateway layer that ties production AI together.
The gap between a working prototype and a production enterprise LLM is not about the model. It is about everything around the model: who can access it, what data touches it, how much it costs per request, and whether you can prove all of that to an auditor.
Worldwide spending on generative AI is forecast to reach $644 billion in 2025, a 76.4% jump from 2024. More than 80% of enterprises will have used generative AI APIs or deployed generative AI-enabled applications by 2026. Most of that spending will not go toward model weights. It will go toward the infrastructure that makes those models safe, observable, and economically viable at scale.
This guide covers the reference architecture for getting there.
A true enterprise LLM is not just ChatGPT with a business wrapper. It is a system grounded in proprietary data, optimized for internal workflows, and deployed with compliance, cost, and control requirements that consumer products never face.
Consumer LLMs ship with basic encryption and user authentication. Enterprise LLMs require multi-layered security, zero-trust architecture, and role-based access control. Consumer LLMs are standalone apps. Enterprise LLMs integrate deeply with ERP, CRM, and data warehouses. Consumer LLMs have minimal compliance safeguards. Enterprise LLMs must align with sector-specific regulations like GDPR, HIPAA, and financial reporting standards.
The paradigm is shifting. We are moving from treating LLMs as magic black boxes to viewing them as modular AI systems, where orchestration, grounding, observability, and feedback loops matter just as much as the base model itself.
A production-ready enterprise LLM setup includes four distinct layers:
Layer 1: Data processing and governance. This is where ingestion pipelines clean, classify, and route data before it reaches a model. PII detection, access tagging, and retention policies live here. Without this layer, every subsequent layer inherits uncontrolled data.
Layer 2: Model training and fine-tuning. Base model selection, domain adaptation through fine-tuning, and evaluation benchmarks. For most enterprises, this means parameter-efficient methods like LoRA rather than full fine-tuning, which for a 70B parameter model requires 8x H100 80GB GPUs.
Layer 3: Deployment and serving infrastructure. Containerization, GPU allocation, API endpoints, autoscaling, and the runtime environment. This layer determines latency, throughput, and cost per request.
Layer 4: Security with role-based access control. Authentication, authorization, audit logging, and policy enforcement across every interaction with the model.
The order matters. Each layer depends on the one below it. Skip data governance and your fine-tuning ingests garbage. Skip serving infrastructure planning and your security layer protects an application nobody can use at scale.
The first architectural fork every enterprise LLM project hits is where inference runs.
Cloud API works when volumes are low and data sensitivity is manageable. The tradeoff is straightforward: zero infrastructure overhead in exchange for per-token costs that scale linearly with usage and can exceed dedicated infrastructure costs for high-volume applications. Customer service platforms processing thousands of requests per hour, internal knowledge systems serving entire organizations, RAG pipelines processing document corpora: these are the workloads where API pricing breaks down.
Private deployment is the right answer when regulations require zero data egress, or when compliance auditors evaluate architecture rather than outcome. Classified government data, healthcare PHI under strict interpretation, banking data in jurisdictions that forbid even encrypted egress, defense contractor environments with ITAR requirements.
Three patterns have emerged for how enterprises structure private deployments: air-gap, BYOC (bring your own cloud), and hybrid-with-boundary. Air-gapped deployments, where the inference server has no outbound network connectivity, represent the strongest compliance guarantee. BYOC keeps compute in the enterprise's own cloud tenancy. Hybrid-with-boundary routes sensitive workloads to private infrastructure while allowing non-sensitive traffic to use cloud APIs.
Most enterprises land on hybrid. The question is where to draw the boundary, and that decision should be driven by data classification, not convenience.
Model size dictates hardware. A 7B parameter model in FP16 requires approximately 14GB of GPU memory. A 70B model requires approximately 140GB, typically requiring multi-GPU configurations with tensor parallelism.
| Model size | GPU memory (FP16) | Typical configuration |
|---|---|---|
| 7B parameters | ~14 GB | Single GPU |
| 70B parameters | ~140 GB | Multi-GPU with tensor parallelism |
| 70B fine-tuning | 640 GB+ | 8x H100 80GB |
Beyond raw memory, containerization introduces challenges unique to LLMs. Model files often exceed 10GB, and containers must handle GPU driver compatibility and memory optimization that standard applications never deal with. The five main technical areas of deployment, namely containerizing the model, allocating GPU resources, creating API endpoints, setting up autoscaling, and securing the environment, each carry LLM-specific complexity that generic container orchestration does not address out of the box.
Latency numbers tell the deployment story clearly. Local inference on datacenter GPUs delivers P50 TTFT of 15 to 30ms for appropriately sized models on H100-class hardware. Cloud APIs deliver P50 TTFT of 100 to 300ms depending on provider and model.
That 3 to 10x gap matters for interactive applications. But latency is only half the problem. API services operate on shared infrastructure where response times vary with overall demand. Production applications that require consistent sub-second response times, or that need to process large batches with predictable throughput, may find API latency unacceptable. You can sign an SLA with a cloud provider, but you cannot eliminate the variance inherent in shared infrastructure.
Autoscaling for LLM workloads is harder than for standard web services. GPU instances take minutes to provision, not seconds. Cold starts with 10GB+ model loading add further delay. Traffic patterns for LLM applications are less predictable than traditional web traffic, making reactive autoscaling too slow and proactive autoscaling expensive if overprovisioned.
Hallucinations are not just an accuracy problem. In enterprise settings, they are a liability. LLMs generate confident but incorrect answers that can lead to operational mistakes, legal risks, and loss of trust. A customer-facing system that fabricates a contract term or a compliance bot that invents a regulation creates exposure that no disclaimer can cover.
RAG is becoming the standard grounding pattern in enterprise LLMs, connecting models to vector databases and existing business systems (CRM, ERP, CMS, intranet) so answers stay accurate, current, and grounded in company knowledge. The alternative, fine-tuning on proprietary data, bakes knowledge into model weights where it cannot be updated without retraining.
RAG does not eliminate hallucinations. It reduces them. Production RAG pipelines need faithfulness metrics as ongoing benchmarks, measuring whether the model's output is actually supported by the retrieved context or whether the model is ignoring the context and generating from its parametric memory. For more on building these pipelines, see the RAG pipeline architecture guide.
Every production challenge described above converges on a single architectural need: a control plane that sits between applications and models.
Continuous monitoring of latency, throughput, token usage, error rates, and GPU load is essential to keep enterprise LLM solutions reliable and cost-effective. But monitoring alone is not enough. The control plane must also enforce policies: budget caps per team, PII redaction before prompts reach models, prompt injection defense, and routing logic that sends requests to the right model based on cost or latency requirements.
Enterprise LLMs face four critical challenges: data privacy and security, hallucinations, prompt injection, and high inference cost. Each of these is addressed differently, but all of them require a centralized enforcement point. Bolting security onto individual model deployments does not scale. Tracking costs per model without a unified view does not give finance teams what they need. Detecting prompt injection at the application layer means every application team reinvents the same defense.
A gateway can centralize selected controls, but its boundary must be explicit. shim provides PII placeholder processing, policy, and accounting for native OpenAI, Anthropic, and Google routes; provider selection, prompt-injection defenses, host security, and deep observability remain application and security-tooling responsibilities.
Compliance is not a checkbox. It is an architectural constraint that shapes every other decision.
Every API call to a third-party LLM service sends organizational data to external servers. Customer queries, internal documents, financial records, clinical notes. For healthcare organizations processing PHI, financial institutions handling transaction data, or any enterprise subject to data residency regulations, this creates compliance risk.
Sending prompts to third-party APIs creates a data processing relationship that triggers obligations under GDPR and HIPAA, and may require controls under SOC 2 compliance programs. For some regulated data, that relationship may be prohibited outright.
The compliance architecture question is not "are we compliant?" but "can we prove it?" That proof requires audit logs showing every prompt and response, data flow diagrams showing where information travels, access control records showing who used which model and when, and retention policies governing how long outputs persist.
For teams navigating specific regulatory frameworks, we have covered the EU AI Act compliance requirements, broader AI compliance patterns, the AI governance framework, and the NIST AI framework in detail.
Teams that choose private deployment often budget it as a project. It is not. On-prem LLM deployment is a recurring operational commitment requiring model refreshes as new versions release, runtime upgrades as serving frameworks evolve, hardware migrations as GPU generations turn over, and eval-loop maintenance as domain requirements shift.
Who should self-host: organizations where regulations explicitly prohibit data egress for the workloads in question, where compliance auditors evaluate architecture rather than outcome, and where the volume of inference justifies dedicated GPU infrastructure over API costs.
Who should not: teams without dedicated ML infrastructure engineers, organizations running low-volume or exploratory workloads, and companies where the compliance requirement can be met with a properly governed cloud API and a data processing agreement.
For everyone in between, the hybrid-with-boundary pattern offers a middle path. Route sensitive workloads to private infrastructure, let everything else use cloud APIs through a unified control plane, and draw the boundary based on data classification rather than organizational politics.
The architecture is not the hard part. Keeping it running is.