GPU inference (vLLM)

Inference is the act of running a trained model to produce an output. For LLMs this is dominated by matrix multiplication and memory bandwidth, which is why GPUs, not CPUs, are the practical hardware. A single request on a GPU is fast; the engineering challenge is serving hundreds of concurrent requests without wasting the card.

vLLM is an open-source inference server built for exactly that. Its key idea, paged attention, treats the model's working memory (the KV cache) like virtual memory pages, so many requests share the GPU without fragmenting it. Combined with continuous batching, which slots new requests into a running batch instead of waiting for the batch to finish, it typically delivers several times the throughput of a naive serving loop. It exposes an OpenAI-compatible API, so applications rarely need to change.

vLLM is not a model and not a fine-tuning tool. It is the layer between your open-weight model files and your application. Alternatives include TGI, TensorRT-LLM and SGLang; the choice depends on hardware, model format and the latency profile you need.

• CTO / Head of Engineering: the serving engine determines how many concurrent users one GPU supports, which sets the real cost of self-hosting.
• CFO: throughput per GPU is the number that turns a hardware quote into a cost per request; a badly configured server can double the bill.
• Operations head: inference servers need monitoring for queue depth, memory pressure and latency, the same as any production service.
• Data lead: quantisation and batching settings change output quality slightly; evals must be re-run when serving configuration changes.

Serving engines like vLLM exist because raw GPU capacity is expensive and a naive implementation leaves most of it idle. Without batching, a card waits on one user's slow generation while others queue. Without memory paging, long conversations exhaust GPU memory long before compute is saturated. vLLM addresses both, which is what makes a single mid-range GPU viable for a department-sized workload. The trade-off is added operational complexity: version compatibility with new model architectures, tuning of batch sizes and memory limits, and the need to benchmark latency under realistic concurrency rather than trusting a single-request demo.

• A bank serving a document-extraction model to its operations team from two GPUs in its own data centre.
• A SaaS product hosting a support-agent model for enterprise tenants who require in-region processing.
• A hospital running a clinical-protocol assistant on a single L4 card with predictable sub-second first-token latency.
• A retailer generating product descriptions in bulk overnight, where batching throughput matters more than per-request latency.
• A logistics platform parsing thousands of proof-of-delivery images daily through a vision-language model on shared GPU capacity.

Tool / technologyvLLM is software you deploy and operate, and GPU serving is a skill your platform team must hold or borrow. Eazyware sizes hardware, configures serving and hands over runbooks as part of Private Agentic AI.

Eazyware service that covers it: Agentic AI Solutions (self-hosted). Starting prices are on the pricing page.