How to Set Up a Private AI Server for Your Team (2026)

If you are still deciding whether to build one at all, start with our local AI server for business guide — it covers the build itself, the ROI case against cloud API spend, and security hardening. This guide assumes you have decided and now need to size it.

The gap between those two questions is where most private AI server projects go wrong. A team lead runs Ollama on a laptop, it works beautifully, and they extrapolate: buy a bigger GPU, put it in the closet, done. Then the fifth person hits it at 10am on a Tuesday and the whole thing feels broken. Nobody warned them, because almost nothing written about self-hosted AI distinguishes between one user and twelve.

Start here: what "team-ready" actually means

A personal local LLM setup and a team AI server run the same model on the same hardware with the same software. They differ in exactly one dimension: concurrency. That single difference changes which inference engine you pick, how much VRAM you buy, and what "enough" means.

If you want the single-user contrast case, our home AI server build guide optimizes for one person running the biggest model they can afford. That is a fundamentally different optimization than what follows here — a team server usually wants a smaller model with more cache headroom, not the largest model that technically loads.

By the end of this guide you will have two things: a seat number for any GPU you are considering, and a shopping list matched to your team size. Start at the local LLM guide hub if you want the broader map of the category first.

The number everyone gets wrong: how many users one GPU really serves

Here is the mental model almost everyone brings to this: "the model is 20GB, my card has 24GB, so it fits, so it works." That is true for one user. It is catastrophically wrong for twelve.

Model weights are a fixed cost. You pay them once when the model loads. The KV cache is a per-sequence cost, and it scales with the number of concurrent sequences multiplied by how long each one's context is. Every simultaneous conversation on your server holds its own cache of key and value tensors for every token it has seen. That is the resource you actually run out of.

What one seat actually costs

KV cache per token is deterministic from the model architecture: two tensors (key and value) × layers × KV heads × head dimension × bytes per element. Modern models use grouped-query attention, which cuts the KV head count dramatically and is the reason a 70B model in 2026 is far cheaper per seat than a 70B model was in 2023.

The 32K column is the one that ambushes people. A team that uploads contracts, codebases, or research papers into a RAG pipeline is not running 8K conversations — it is running 32K ones, and each seat costs four times as much. Sizing on chat-length contexts and then deploying a document Q&A tool is the single most common way these servers get undersized.

Seats per card

Applying the rule to real hardware, at Q4 weights, 8K context window, FP16 KV cache, and 10% overhead:

Concurrent users are not total users

This is the number that saves you money. A 30-person team does not generate 30 simultaneous requests. People type, read, think, go to meetings, and use the tool in bursts. In practice, observed peak simultaneous request counts for internal AI tools land somewhere between a quarter and a sixth of headcount.

Planning heuristic: divide headcount by 4 for a heavy-usage team (engineering, support), by 6 for a light-usage team (general office). Label this a heuristic, because it is one — it is a planning starting point derived from typical usage patterns, not a measured constant. Instrument your actual peak concurrency within the first month and resize from real data.

Run through it: 30 engineers ÷ 4 = 8 peak concurrent. From the table, 8 concurrent seats on a 14B model fits comfortably on a 24GB card and with real headroom on 32GB. That is a $1,599–$2,199 GPU serving a 30-person engineering org, and it is why the tier recommendations below are less expensive than most people expect.

Ollama stops at four users — here's the wall

The most useful and least-known fact in this entire guide: Ollama's OLLAMA_NUM_PARALLEL defaults to 4. Per Ollama's own documentation, the default auto-selects either 4 or 1 parallel requests depending on available memory. Request number five does not get a slower response — it gets no response until a slot frees.

Worse, Ollama does not implement continuous batching. When a slot is occupied by a 900-token generation, the queued request waits for that entire generation to finish. This is why the failure mode is a cliff rather than a slope: p99 latency does not degrade gracefully as load rises, it falls off a ledge the moment you cross the parallel limit.

vLLM's continuous batching injects newly arrived requests into the running batch at every decoding iteration rather than waiting for the current batch to complete, and PagedAttention allocates KV cache in non-contiguous blocks so memory is not reserved for a sequence's maximum possible length.

That architectural difference is the whole ballgame for a team server. Under vLLM, a long generation cannot head-of-line-block a short one: the short request joins the batch on the next iteration and starts streaming tokens immediately. Under Ollama, it waits.

Independent testing by Rost Glukhov (glukhov.org, "How Ollama Handles Parallel Requests") documents the parallel-slot behavior hands-on and is worth reading before you commit to a stack. A widely-circulated 2026 teardown, "vLLM Teardown: How Many Users Can One GPU Really Hold?", reports that at 20 concurrent users Ollama queues 19 of them while vLLM batches all 20, with p99 latency of roughly 9 seconds versus roughly 1.5 seconds. Treat those figures as third-party reported, not verified by us — the direction is well-established by the architecture, but the exact numbers depend on model, hardware, and prompt length.

If your team lands under the line — a three-person practice, a small design studio — Ollama is genuinely the better choice: simpler, faster to stand up, easier to maintain. Our Ollama setup guide walks the full install. Other engines worth knowing: TGI (Hugging Face's server, similar batching model to vLLM) and NVIDIA TensorRT-LLM, which squeezes more throughput out of NVIDIA silicon at the cost of a per-model compilation step. FlashAttention is enabled by default in vLLM on supported cards and is part of why its memory profile is as tight as it is.

Three build tiers by team size

Prices are GPU street ranges from our catalog; total build figures assume a mid-range CPU, motherboard, PSU, and case around the card. Cross-check card choices against the AI GPU buying guide hub.

Small team: 5–15 people (~$1,500–$2,500)

Peak concurrency of 1–4. A used RTX 3090 ($699–$999) is the value champion here and has been for three years running — 24GB of VRAM at a third of the price of a new 24GB-class card. It runs 7B–32B models at Q4 and gives you the full 24GB row of the seat table above. If you would rather buy new with a warranty, the RTX 5060 Ti 16GB ($429–$479) covers 8B–14B models for a team this size. Our used RTX 3090 vs RTX 5060 Ti breakdown and the RTX 4090 vs RTX 3090 comparison settle that call.

Pair it with 64GB of system RAM and a fast NVMe drive. Models that suit this tier: Mistral 7B and Phi-4 14B for speed, Gemma 3 27B if you want more capability and can live with 1–3 concurrent seats.

Growing team: 15–30 people (~$3,500–$6,000)

This is the post's hero recommendation, and for most readers it is the right answer. One RTX 5090 ($1,999–$2,199) gives you 32GB of GDDR7 at 1,792 GB/s — enough to serve 11–13 concurrent seats on a 14B model or 4–6 on a 32B, in a single-card build that needs no NVLink, no tensor parallelism, and no multi-GPU debugging. Peak concurrency for a 15–30 person team is 3–8. The math clears with room to spare.

The alternative is two RTX 4090 cards ($1,599–$1,999 each) for 48GB combined, which unlocks a 70B model at Q4 — barely, at 1–2 concurrent seats. That trade is only worth making if a 70B model is genuinely required for your work rather than aspirationally nice. See RTX 5090 vs RTX 4090 and our multi-GPU setup guide before committing to two cards. Budget-constrained? The RTX 5080 ($999–$1,099) is half the price but only 16GB — check RTX 5090 vs RTX 5080 for whether that halving costs you the seats you need.

Model fits at this tier: Qwen 3 72B and Llama 4 Maverick 70B on the dual-card path, DeepSeek R1 70B if reasoning-heavy work dominates, CodeLlama 34B for an internal coding assistant.

Company-wide: 30–50+ people ($12,000 and up)

At 8–15 peak concurrent users on a 70B model you need 80GB in one memory space. The A100 80GB PCIe ($12,000–$15,000) is the sane entry point; the H100 PCIe 80GB ($25,000–$33,000) buys throughput rather than capacity and is only worth it if you are compute-bound, not cache-bound. Either belongs in a real chassis with real airflow — the Supermicro SYS-421GE-TNRT ($8,000–$15,000 barebones) handles multiple double-width cards with redundant power. Be honest with your budget: card plus chassis at this tier is a $20,000–$30,000 conversation, not a $12,000 one.

A workstation-class middle path exists: the RTX PRO 6000 96GB gives you more VRAM than an A100 in a card you can put in a tower. Our RTX PRO 6000 96GB review covers whether it beats the datacenter option for a team server.

The quiet-office alternative (and its honest limit)

The NVIDIA DGX Spark ($3,999), Mac Studio M4 Max ($1,999–$5,999), and GMKtec EVO-X2 ($1,999–$3,649) all put 128GB or more of unified memory in a silent box that draws a fraction of a 575W GPU's power. Models that will not load on a 32GB card load here without complaint.

And they are still the wrong choice for a 20-person team. Unified memory solves capacity; concurrency is a bandwidth problem. Batching many sequences means streaming the entire model's weights through the memory bus on every decoding step for the whole batch, and LPDDR5X or on-package memory moves far less data per second than GDDR7. These machines are genuinely excellent for 2–5 heavy users working with very large models — and genuinely poor at serving 20 light ones. Compare the trade in our DGX Spark vs Strix Halo piece, or browse the mini PC for AI hub.

What you need besides the GPU

The supporting spend is where team servers quietly fail, because it gets budgeted as an afterthought.

System RAM

Floor: 2× your total VRAM. A 32GB card wants 64GB minimum, 128GB if you intend to keep multiple models resident or swap between them. This is not about running the model on CPU — it is about model loading, page cache, and the embedding/reranking stack that sits alongside a RAG deployment. Our how much RAM for local AI guide has the full breakdown.

Storage

Models run 4GB to 140GB each and teams hoard them — five to ten is normal within six months. Use NVMe, not SATA: load time is felt on every cold start and every model switch. A Samsung 990 Pro 4TB ($289–$339) is the standard answer; see our best NVMe SSD for local AI roundup for alternatives.

Networking

This is where self-hosted team servers actually fall over, and nobody writes about it. Streaming chat tokens over 1GbE is fine — a token is bytes. Uploading a 200MB document set into a RAG index from twelve desks simultaneously is not. If your use case is document-heavy, budget for 10GbE between the server and your switch. A UniFi Dream Machine Pro ($379–$449) handles the VLAN isolation and firewall rules you want around this box, and a MikroTik CRS326 ($149–$199) adds SFP+ uplinks cheaply.

Shared document storage for RAG

If the server is going to answer questions about company documents, those documents need to live somewhere the ingestion pipeline can reach on a schedule. A Synology DS1821+ ($949–$1,099) gives you that plus snapshots — useful when someone re-indexes the wrong folder.

Power and thermals

An RTX 5090 pulls 575W under sustained inference load. In a closet, that is an HVAC question, not a footnote — sustained GPU load is not the same duty cycle as gaming, and a server that thermally throttles at 2pm every day will look like a software problem. Size the PSU with our AI workstation PSU guide (1000W+ for a single 5090) and model the running cost with our local AI electricity cost calculator before you sign off on the budget.

The software stack: accounts, access, and not exposing this to the internet

Keep this layer boring. The reference stack for a team server is three components:

Give every person their own account. Shared credentials defeat the compliance reason you built this. If you cannot answer "who asked the model about that client file," you have rebuilt the audit problem you were escaping.

Do not port-forward this server. Keep it on the LAN or behind a VPN, full stop. An inference endpoint reachable from the open internet is both a data exfiltration path and free compute for whoever finds it. There is no configuration of an open port that is worth the convenience.

One genuinely useful pattern once you are running: model routing. Serve a small fast model for autocomplete, classification, and title generation, and a larger one for reasoning, from the same GPU with separate KV budgets. Most requests hitting your server do not need the big model, and routing them away from it is the cheapest concurrency you will ever buy. Llama 4 Scout 8B is a good small-tier pick alongside a 32B reasoning model.

When to scale up (and the signals that tell you)

Sizing on day one is a guess. Sizing on month three is data. Watch four signals:

• Queue depth consistently above zero at peak. vLLM exposes this on its metrics endpoint. Occasional spikes are fine; a queue that never drains during business hours is not.
• p95 time-to-first-token above 2 seconds. Research on LLM streaming quality of experience (Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services, arXiv:2404.16283) grounds why time-to-first-token dominates perceived responsiveness far more than total generation time. Two seconds is the practical threshold where a tool starts feeling slow rather than instant.
• Per-user tokens per second below reading speed. Roughly 10–15 tok/s is where streaming stops outpacing a reader. Below that, batching has gone too far.
• Users routing around the server back to ChatGPT. This is the real failure signal and the only one that shows up in your subscription bill instead of your dashboard. Ask about it directly in your first monthly check-in.

When those fire, escalate in this order — cheapest first:

1. Raise --max-num-seqs. vLLM's default is conservative. If VRAM headroom exists, this is a free concurrency increase.
2. Quantize the KV cache to FP8. Roughly halves per-seat cost with minimal quality impact — usually the highest-leverage single change available.
3. Quantize weights harder (Q4 → Q3). Frees weight VRAM for cache, but quality degrades noticeably below Q4 on smaller models. See our quantization primer and the MoE on small GPUs guide — a sparse MoE model can give you 70B-class quality at a fraction of the active-parameter cost.
4. Add a second GPU with tensor parallelism. Doubles both weights capacity and cache. Requires a board with the right lane layout — check our dual-GPU motherboard guide and the notes on NVLink and pipeline parallelism.
5. Split into two single-GPU servers by workload. Often better than one bigger server: put coding assistance on one box and document Q&A on the other. Different context lengths, different KV profiles, no contention between them. This beats model sharding for most teams under 50 people.

One more lever worth knowing: speculative decoding raises per-user token throughput without more VRAM, at the cost of some extra compute. It helps when you are compute-rich and cache-poor — the opposite of most team servers, but worth testing.

The bottom line

Divide your headcount by 4 to 6 to get peak concurrent users. Look up that number against your candidate GPU in the seats table. If the answer is four or more, run vLLM, not Ollama. For most companies between 15 and 30 people, one RTX 5090 in a $3,500–$6,000 build running a 14B–32B model at Q4 is the correct purchase — and then you spend the first month measuring real concurrency and adjusting --max-num-seqs rather than buying a second card.

The failure mode to avoid is not buying too little GPU. It is buying the largest model your card can technically hold, leaving no cache headroom, and discovering at the fifth simultaneous user that capacity and concurrency were never the same thing.