How to Choose a GPU Cloud Server: Training vs Inference Sizing

How do you choose a GPU cloud server? It depends on whether you are training or serving, and more importantly on which wall you will hit first. The most common sizing mistake is buying by FLOPS leaderboard, then discovering halfway through model load that the card is short on memory. In most deep learning workloads VRAM binds before compute does. This guide separates the training and inference decision paths, then covers precision, CPU and network pairing, CUDA compatibility, spot instances and multi-GPU scaling, with sizing math you can run yourself.

Training and inference are not the same purchase

The same model, two almost opposite bottlenecks. Getting this straight removes half the trial and error.

• Training is bound by compute and memory bandwidth: dense matrix multiplies, GPU utilisation routinely above 90%, and parallelism sets throughput
• Training must also hold optimiser state: full fine-tuning stores gradients and moments on top of weights, typically 5–8× the memory of pure inference
• Training is sensitive to interconnect: multi-node jobs spend a large share of step time in all-reduce
• Training tolerates interruption: checkpoints make it resumable, which makes it a good fit for spot capacity
• Inference is bound by memory capacity and bandwidth: single-stream decoding is a memory-read problem, not a compute problem
• Inference is judged on concurrency and per-card throughput: at the same total QPS, one large-memory card is usually cheaper and easier to operate

Short version: training buys compute, inference buys memory and bandwidth. The classic wrong turn is buying eight training cards for a serving workload whose real constraint is KV cache, or renting eight H100s for one 7B fine-tune while six of them wait on collective communication.

VRAM is the wall: you run out before you run out of compute

Look at the order in which OOM hits, and the sizing problem becomes obvious:

1. Inference, step one: weights do not fit. You change GPU, or you are forced to a smaller model and a shorter context
2. Inference, step two: weights fit, KV cache does not. Concurrency stalls, or long requests get truncated
3. Training, step one: optimiser state does not fit. ZeRO / FSDP sharding is forced, and step time typically rises 30–60%
4. Training, step two: activations fill the card. Gradient checkpointing trades compute for memory, adding another 20–40% per step

The other underweighted number is effective memory bandwidth, which sets the ceiling on decode speed: bandwidth ÷ weight size is roughly the per-token upper bound. A 7B model in FP16 is about 14GB of weights, so a 2TB/s-class card tops out near 140 tokens/s per stream in theory and 80–120 in practice, while a 1TB/s consumer card tops out near 70 and lands at 40–60. Capacity is the pass/fail test; bandwidth is what you actually feel.

GPU types compared: datacentre, consumer, inference

Consumer cards are cheap, and there are four hard limits to design around:

• No ECC memory: long training runs can accumulate silent data errors that show up as mysterious loss divergence
• Most models have no NVLink: multi-GPU falls back to PCIe, where bandwidth can differ by almost 20×
• No MIG: one physical card cannot be partitioned into instances for separate tenants
• Virtualisation and enterprise support need extra licensing for commercial multi-tenant use

How precision rewrites the bill: FP32 / FP16 / BF16 / INT8 / INT4

Precision is your most reliable memory discount and your easiest place to get burned. The memory effect first:

On speed, FP16 and BF16 are effectively equal on Ampere-class Tensor Cores, which is why BF16 costs you old-card compatibility rather than performance. INT8 is typically 1.5–2× FP16, and INT4 reaches 2–4× on Ampere. But end-to-end gains are not linear: memory traffic, kernel coverage and dequantisation overhead eat into it, and real-world wins of 1.3–1.8× are more common than the theoretical number.

• Trading precision for memory is linear and dependable, because weight size scales directly with bytes per element
• Trading precision for speed is not linear: it depends on model shape, batch size and kernel coverage
• Do not train in INT8 or INT4: optimiser state, gradient scaling and small updates behave badly at low dynamic range

Quality impact depends on the task. INT8 (symmetric or grouped) usually moves metrics by under 0.5% and is safe in production. INT4 (GPTQ, AWQ) must be measured: code generation, long chains of reasoning and maths degrade noticeably, while chat, classification and retrieval are largely unaffected. Always evaluate on your own benchmark before shipping.

Size your own workload: the memory math

Three formulas cover almost every case. Take the lower bound, then add 20–30% headroom — that is your target VRAM.

• 7B inference, FP16: 7e9 × 2 bytes = 14GB of weights → about 17GB with headroom → a 24GB card works, a 16GB card forces INT8 or a shorter context
• 7B inference, INT4: about 3.5GB of weights → 4.5GB to start → a single 16GB card handles dozens of concurrent requests
• 70B inference, FP16: 140GB of weights → no single card → 2×80GB with tensor parallelism (TP=2); a 141GB card technically fits the weights but leaves no room for KV cache
• 7B LoRA, BF16: 14GB weights + small optimiser state + activations → a 24GB card works with gradient checkpointing, 40GB is comfortable
• 7B full fine-tune, Adam: 7e9 × 16 bytes = 112GB → at least 2×80GB; ZeRO-3 across 4×40GB also fits but visibly lengthens step time

On the serving side you must also size KV cache separately — it is the hidden cost of long context:

• Llama-2-7B (32 layers, hidden 4096, no GQA): about 512KB per token, roughly 2GB for a 4K context, so 32 concurrent requests need 64GB
• Llama-3-8B (32 layers, GQA with 8 KV heads): about 128KB per token, roughly 0.5GB for the same 4K context
• Takeaway: at identical weight size, GQA multiplies your concurrency at a given memory budget. Capacity planning without this term under-counts by several times

One more trap: batch size does not scale memory linearly. Linear layers and activations grow linearly, but PyTorch reuses memory aggressively and the intermediate tensors inside attention grow with the square of sequence length. Moving from 2K to 8K context can multiply activation memory by more than 4×. Size for the 8K context you intend to serve, not the 512 default.

Look past the GPU: CPU, RAM, storage, network

The GPU sets the ceiling; the rest of the machine decides how much of it you reach. Plenty of cheap GPU instances underperform because of exactly this. VRAM, CPU-to-RAM ratios and the card families on offer: GPU cloud servers.

• CPU: loading, decoding and augmentation are CPU-bound. Four vCPU per GPU is the floor; 2 vCPU machines starve the GPU and utilisation commonly drops to 50–70%
• System RAM: must hold the shard you are reading. A useful rule is system memory ≥ total GPU memory, plus pinned memory to speed up host-to-device copies
• Storage: local NVMe reads at roughly 1–2GB/s sequentially, while reading straight from object storage only reaches a few hundred MB/s per stream. Sync the dataset to local NVMe first, then train
• Network: start at 10Gbps inside a host. For multi-node training, look for 100Gbps RDMA (InfiniBand or RoCE) — gradient sync over a 25Gbps shared NIC is often slower than the GPU compute itself

Inference weights the trade-offs differently: under concurrency, the CPU schedules, preprocesses and merges batches. Running vLLM or SGLang on a CPU-rich instance with one or two large-memory inference cards is often cheaper than piling on GPU count — fewer GPUs with more CPU wins for high-concurrency small-model serving.

NVLink vs PCIe: what multi-GPU training really costs

NVLink 3 is roughly 18–19× the bandwidth of PCIe 4.0, and tensor parallelism amplifies that gap: every layer forward and backward performs an all-reduce whose volume grows with depth. Over PCIe, communication can consume 30–60% of step time — you paid twice as much per card and gained little effective compute.

• Single-GPU jobs: interconnect is irrelevant, so buy on memory and compute
• 2–8 GPU training: prefer same-node NVLink / NVSwitch; this is most of the premium in a training card price
• Across nodes: keep tensor parallelism inside the node and use data parallelism, or pipeline parallelism, across nodes
• Consumer multi-GPU: run nvidia-smi topo -m first. PIX (over PCIe) versus NV# (over NVLink) can be a difference of tens of percent of throughput

CUDA and driver compatibility: where projects break

Right spec, wrong software stack is the usual reason a well-sized machine will not start. Settle these before signing, not while debugging.

• Drivers are backward compatible, frameworks are not: CUDA 12.x runtimes generally require driver r525 or newer
• PyTorch wheels bundle their own CUDA runtime: the system CUDA in your container must match the wheel — upgrading the driver alone is not a fix
• cuDNN and NCCL must line up: if torch.cuda.nccl.version() will not print, NCCL is mismatched, and multi-node jobs hang at initialisation
• Architecture support is a hard gate: no BF16 on Turing, FP8 from Hopper, FP4 only on Blackwell. New cards with old frameworks can require compiling extensions from source
• The classic error CUDA driver version is insufficient for CUDA runtime version is almost always a driver or image mismatch, not a code bug

Verification is one command: nvidia-smi for the driver and detected cards, then python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_capability())" for the framework side, then an NCCL all-reduce microbenchmark from nccl-tests to confirm communication. One slow node in ten is the one you will be paged about, so verify every machine rather than sampling.

Spot GPUs: train at a fraction of on-demand

Spot and preemptible GPU capacity commonly lands at one third to one tenth of the on-demand price; check the live official quote for your region. For resumable training this is the most direct saving available. Billing mode drives the long-run bill just as hard as spot does — price bands across five workload tiers: cloud server price guide.

• Good fit: hyperparameter search, bulk labelling, offline evaluation, and training with a checkpoint interval of 15 minutes or less
• Bad fit: online serving, non-interruptible long runs, and any task whose state lives only on the local disk of that instance
• Keep all GPUs of a job on the same instance: reclamation then releases the whole group. Otherwise the remaining quarter of the job keeps billing, and this straggler residue is the most common hidden cost of spot
• Make the training loop re-entrant: checkpoint regularly and use a framework retry mode (PyTorch elastic is one example)

Design storage the same way: keep datasets and checkpoints in object storage, never on the reclaimable local disk of the instance, and sync checkpoints back as you go. Then a vanished instance costs you at most 15 minutes of compute instead of a whole night.

Choosing a multi-GPU strategy

The order is largely fixed — walk down it based on what is actually running out of memory:

1. Weights and optimiser state do not fit → ZeRO-3 / FSDP; memory falls roughly linearly with GPU count and the overhead is acceptable
2. Weights of a single layer do not fit (usually 70B and up) → tensor parallelism to split each layer across cards, inside one node over NVLink
3. You need more GPUs than one node holds → pipeline parallelism by layer, orchestrated with Slurm or Kubernetes
4. You still want the parameter count to grow → MoE sparsifies parameters and relieves memory, but load balancing and all-to-all traffic become the new problem

Inference inverts the logic. You usually want tensor parallelism to fit one model in a node, then horizontal replicas, not data parallelism, because there are no gradient synchronisation steps to amortise. In vLLM or SGLang that is the combination of --tensor-parallel-size and --data-parallel-size.

Eight selection mistakes to avoid

1. Comparing FLOPS without comparing memory — the most common sizing error, and it ends in OOM or forced sharding
2. Assuming a consumer card is production-grade — no ECC, no MIG, licensing limits; long runs carry silent-error risk and commercial use needs checking
3. Forgetting KV cache — tests pass at short context, long prompts fill the card in production, and scaling up comes too late
4. Specifying a GPU count without checking topology — PIX versus NV# in nvidia-smi topo -m is worth tens of percent of throughput
5. Mixing GPU generations — the slowest card sets the speed of every collective operation
6. Running online serving on spot — reclamation takes the service down with no graceful drain
7. Committing to a long term before measuring — pay hourly, capture real step times and tokens/s, then convert
8. Skipping checkpoint and dataset backups — recovery from a GPU failure or an accidental delete costs far more than a month of instances

The first four are technical judgement; the last four are operational habits. The two that actually blow a budget are number 6 and number 7: using the cheap tier for the wrong job, and signing a long term before you have a measurement.

Next step: validate in 24 hours

Sizing should not be guesswork. Five steps get you a decision backed by real measurements inside one working day.

1. Measure peak memory: run 50 training steps or 100 inference requests and record peak usage from nvidia-smi and max_memory_allocated
2. Compute the lower bound: use the formulas above for weights, optimiser state and KV cache, then add 20–30% headroom to get a target VRAM figure
3. Shortlist two or three cards: match them against the GPU table, and verify driver, PyTorch version and NCCL support
4. Open the smallest hourly instance: run 30 minutes of the real workload, record step time (tokens/s) and total wall time, and convert to cost per training hour or per million tokens
5. Write the comparison table before signing: spot for the bulk of training with on-demand for the final stretch, on-demand plus a committed plan for serving, and account for the two separately

If you want the math run against your own model, send over parameter count, precision, context length, concurrency, and whether you are doing a full fine-tune, and we can return a candidate spec range with a memory estimate. GPU pricing moves with chip supply and region, so treat any figure as indicative and confirm the live official quote.