More companies are bringing AI in-house. An Enterprise Technology Research survey reported in September 2026 found 42% of organisations run open-weight models in production, with cost savings cited by 69% of them — and that 46% host them in their own cloud accounts while 42% run them on-premises (Techstrong.ai). The same survey found the biggest obstacles were security and compliance reviews and the cost of hosting. This guide covers how to do it well.
Why self-host?
- Data privacy and sovereignty. Prompts, documents and outputs never leave your infrastructure — important for healthcare, finance, legal and government work.
- Cost at scale. At high, steady volumes, owning or renting GPUs can be cheaper than paying per token.
- Control. No surprise model retirements or behaviour changes; you choose when to upgrade.
- Customisation. Fine-tune models on your data and run them however you like.
- Offline and edge use. Factories, ships, clinics and devices without reliable connectivity.
When an API is still the better choice
- You need frontier capability for the hardest reasoning or agentic tasks.
- Your volume is low or unpredictable — idle GPUs are expensive.
- You lack people to operate, patch and monitor GPU infrastructure.
Many companies end up hybrid: sensitive or high-volume workloads on a self-hosted model, everything else on a commercial API.
Choosing a model
Start with the smallest model that passes your evaluation set. As of October 2026, strong options include Gemma 4 (2B to 31B, Apache 2.0), Qwen3.8-27B (Apache 2.0), Muse Glimmer (30B, Apache 2.0), Granite 4.1 (3B–30B, Apache 2.0) and, for clusters, DeepSeek V4, Kimi K3 or GLM-5.3. Our guide to the best open-source LLMs compares them, including licences.
The serving tools
| Tool | Best for | Notes |
|---|---|---|
| vLLM | Production serving with many users | High throughput, OpenAI-compatible API, broad model support |
| SGLang | High-performance production serving | Strong for structured output and complex workloads |
| llama.cpp | CPUs, laptops, edge devices, GGUF models | Runs almost anywhere; joined Hugging Face in 2026 |
| Ollama | Developers and small teams | Simplest way to run models locally |
| LM Studio | Desktop use with a GUI | OpenAI- and Anthropic-compatible local endpoints |
| TensorRT-LLM | Maximum performance on NVIDIA GPUs | More engineering effort |
A few 2026 developments worth knowing:
- vLLM shipped version 0.31 in October with support for the newest models — and a fix for a remote-code-execution vulnerability affecting versions 0.7.3 to 0.30.x. If you run vLLM, upgrade (vLLM releases).
- llama.cpp and ggml joined Hugging Face in February, while remaining fully open-source (Hugging Face).
- Hugging Face's Text Generation Inference is now in maintenance mode; Hugging Face recommends vLLM, SGLang, llama.cpp or MLX instead (Hugging Face docs).
- Ollama now builds on upstream llama.cpp for GGUF models and MLX on Apple silicon, and remains the easiest way to get started (Ollama releases).
Rule of thumb: prototype with Ollama or LM Studio; deploy multi-user production with vLLM or SGLang; use llama.cpp for CPU and edge.
How much GPU memory do you need?
Memory has two parts: the model weights and the KV cache, which grows with context length and the number of simultaneous users.
| Model size | Weights at 4-bit (approx.) | Typical hardware |
|---|---|---|
| 2–4B | 2–3 GB | Laptop, phone, small GPU |
| 7–9B | 5–6 GB | Consumer GPU (8–12 GB) |
| 27–31B | 17–22 GB | One 24 GB GPU (more for many users) |
| 70B | 35–40 GB | 48–80 GB GPU; more for high concurrency |
| Frontier MoE (500B–2.8T) | Hundreds of GB to ~1.4 TB | Multi-GPU servers (e.g. 8× H200) |
These are approximations: serving around 50 concurrent users at 8K context on a 70B model can need more than 80 GB once the KV cache is included (PremAI). Large mixture-of-experts models are firmly data-centre scale — NVIDIA's Nemotron 3 Ultra, for instance, lists eight B200 or H200 GPUs, or sixteen H100s, as a minimum (model card).
Quantisation (running models at 8-, 4- or even lower-bit precision) cuts memory dramatically with modest quality loss, and many 2026 models ship natively in low-precision formats such as FP8 or FP4.
Estimating cost
Compare three numbers:
- API cost = your monthly tokens × the provider's price per token.
- Self-hosted cost = GPU rental or amortised purchase + power + engineering and operations time.
- Utilisation — self-hosting wins only if GPUs are busy. A GPU that sits idle most of the day is usually more expensive than an API.
Run a pilot, measure real tokens per day and peak concurrency, then decide.
Security checklist
- Keep inference servers private — behind your network, never directly on the internet.
- Patch quickly — inference servers have had serious vulnerabilities; subscribe to release notes.
- Download weights only from official sources and verify checksums.
- Authenticate every request and apply per-user rate limits.
- Log prompts and outputs according to your data-retention policy.
- Apply the same AI-application defences you would with an API — permissions, output validation and prompt-injection protection.
A sensible rollout plan
- Pick one use case and build an evaluation set.
- Test two or three candidate models with Ollama or vLLM on a rented GPU.
- Choose the smallest model that passes, then load-test for your peak concurrency.
- Deploy with vLLM or SGLang behind an API gateway with authentication, logging and monitoring.
- Route requests that need frontier capability to a commercial API if required.
- Re-evaluate quarterly — better open models arrive every few months.
How Zinetics can help
We design and deploy private AI — from a single-GPU assistant over your documents to multi-GPU clusters — and integrate it with your applications through a secure API layer. See what we build with AI, read about open-source vision models and OCR, or talk to us about your requirements.