Running Models Locally: Quantization, vLLM, Ollama and LoRA

Open-weight models can be downloaded and run on hardware you control, from a laptop to a rack of GPUs. Doing it well comes down to four questions: will the model fit in memory, how much quality does compression cost, which runtime suits the job, and is fine-tuning worth it? This article answers each with worked numbers and real commands.

Why run models locally, and why not

Hosted APIs are the easiest way to use an LLM. Running models yourself makes sense when one of these matters more than convenience:

• Privacy and data residency. Prompts and documents never leave your network, which can settle compliance questions outright.
• Cost at steady volume. A GPU costs the same whether it is busy or idle. With high, predictable traffic, self-hosted inference can cost less per token than an API; with low or spiky traffic it usually costs more.
• Latency and offline use. There is no network round trip, and the system keeps working on a plane, a factory floor or an air-gapped network.
• Control. You pin the exact model version, choose sampling settings, read raw logits and fine-tune the weights. Nobody retires your model.

The costs are real too. As of 2026 the strongest closed models are generally still ahead of open-weight ones on the hardest tasks, although the gap varies by task. You also take on drivers, upgrades, monitoring, scaling and security patches. A common pattern is hybrid: a local model for high-volume or sensitive work, and a hosted frontier model for the hard cases.

Memory maths: will it fit?

Memory, not compute, is usually the first limit. Three things share it.

Weights take parameters × bytes per parameter. At 16-bit precision (FP16 or BF16) each parameter takes 2 bytes, so an 8-billion-parameter model needs about 16 GB for weights alone. At 8-bit it needs about 8 GB, and at 4-bit roughly 4–5 GB, because 4-bit formats also store scale factors and keep a few sensitive tensors at higher precision.

The KV cache holds the attention keys and values of every token in the context (the mechanism is explained in What Is a Large Language Model?). Its size per token is 2 × layers × KV heads × head dimension × bytes per value. For Llama 3.1 8B, with 32 layers and 8 KV heads of dimension 128 stored at 16-bit, that is 128 KiB per token: 1 GiB for an 8,000-token conversation, 4 GiB at 32,000 tokens and 16 GiB at the full 128,000-token context window. It is also per sequence, so a server handling twenty conversations at once needs twenty caches.

Overhead covers activations, runtime buffers and the GPU driver. Budget a gigabyte or two.

Two consequences follow. Quantizing weights alone does not make long contexts cheap; most runtimes can also store the KV cache at 8-bit, which halves it. And a mixture-of-experts model must hold all of its experts in memory even though only a few run for each token. Its memory needs follow the total parameter count while its speed follows the much smaller active count, which makes MoE models a good match for machines with plenty of memory but modest compute, such as high-memory Macs.

Quantization

Quantization stores weights with fewer bits. Weights are split into small groups, and each group is stored as low-bit integers plus a scale factor that maps them back to approximate real values. Matrix multiplications dequantize on the fly or run directly on low-precision hardware. Because generating tokens for a single user is limited mainly by memory bandwidth, smaller weights also mean faster output, not just a smaller footprint.

Quality degrades gracefully, up to a point. 8-bit is close to lossless for most models. 4-bit typically costs a small and often hard-to-notice amount of quality, which is why it is the usual sweet spot for local use. Below about 3 bits errors grow quickly, and small models suffer more than large ones. With a fixed memory budget, a larger model at 4-bit often beats a smaller one at 16-bit. Test on your own tasks, though: losses are uneven and can show up first in maths, code or less common languages.

Quantized versions of popular models usually appear on Hugging Face soon after release, so you rarely need to quantize anything yourself. GGUF is the format of the desktop world; GPTQ, AWQ and FP8 are the formats of GPU servers.

Desktop runtimes: llama.cpp, Ollama, LM Studio and MLX

For one person on a laptop or workstation, four tools cover most needs:

• llama.cpp is a C/C++ inference engine for GGUF models. It runs on CPUs and on GPUs through Metal, CUDA, Vulkan and other back ends, can split a model between GPU memory and system RAM when it does not fit on the GPU, and its llama-server exposes an OpenAI-compatible HTTP API.
• Ollama wraps the same ecosystem in a friendly command-line tool: it pulls models from its library by name, manages them and runs a local server.
• LM Studio is a desktop application with a model browser, chat interface and local server, for people who prefer a GUI.
• MLX is Apple's array framework for Apple Silicon. Its mlx-lm package runs and fine-tunes models using the unified memory that the CPU and GPU share.

# Download on first use, then chat in the terminal
# (the default tag is usually a 4-bit build)
ollama run llama3.1:8b

ollama list    # models on disk
ollama ps      # models in memory, and how much of each is on the GPU

# The same models are served over HTTP on port 11434,
# including an OpenAI-compatible API under /v1

Serving many users: vLLM and SGLang

Desktop runtimes are tuned for one conversation at a time. Serving many concurrent users from GPUs is a different problem, and the main open-source answers are vLLM and SGLang.

vLLM's central idea is PagedAttention. Instead of reserving one contiguous block of memory for each request's maximum possible length, it stores the KV cache in small fixed-size blocks, like pages of virtual memory, and allocates them as sequences grow. Less memory is wasted, so more sequences fit on the GPU at once, and requests with identical prefixes can share blocks. vLLM pairs this with continuous batching: new requests join the running batch at every generation step instead of waiting for the current batch to finish, which keeps the GPU busy. SGLang uses similar techniques and adds RadixAttention, which reuses cached prefixes across requests automatically; that helps agents and chat apps that resend the same long system prompt. Both support tensor parallelism across several GPUs and the server quantization formats above.

# Linux with a supported GPU; install into a virtual environment
pip install vllm

# Llama weights on Hugging Face are gated: accept the licence and log in first
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90

--gpu-memory-utilization sets the share of GPU memory vLLM may claim; whatever the weights leave free becomes KV-cache blocks. --max-model-len caps the context of a single request.

OpenAI-compatible endpoints

vLLM, SGLang, llama.cpp's server, Ollama and LM Studio all expose an API modelled on OpenAI's /v1/chat/completions. Existing clients, SDKs and frameworks can switch to a local model by changing the base URL:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [
          {"role": "system", "content": "You are a concise assistant."},
          {"role": "user", "content": "Explain PagedAttention in two sentences."}
        ],
        "temperature": 0.2,
        "max_tokens": 200
      }'

With the official openai Python package, set base_url="http://localhost:8000/v1" and any placeholder API key. Compatibility is good for plain chat but uneven for extras. Tool calling, for example, often needs additional server flags and a model-specific parser, so test the features you depend on before connecting a local model to an agent.

When fine-tuning is the right tool

Fine-tuning continues training a model on your own examples. It is powerful, and often reached for too early. Work through the cheaper options first:

• Prompting. Clearer instructions and a few examples fix many problems in minutes.
• Retrieval. If the model lacks knowledge, especially knowledge that changes, fetch it at query time with RAG (see embeddings, vector databases and RAG). Fine-tuning is unreliable at teaching facts.
• Fine-tuning. Use it for consistent behaviour that prompts cannot achieve: a strict output format, a house style, domain jargon, or a narrow high-volume task such as classification or extraction, where a fine-tuned small model can replace a much larger one.

The data matters more than the method. A few hundred to a few thousand clean, consistent, representative examples usually beat a large noisy set. Format them with the model's own chat template, remove duplicates, include edge cases and examples of correct refusals, and hold out a test set before training starts. Without that test set you cannot tell whether the fine-tune helped.

LoRA and QLoRA

Full fine-tuning updates every weight. With the Adam optimiser in mixed precision that takes roughly 16 bytes per parameter for weights, gradients and optimiser state: well over 100 GB for an 8B model before activations. LoRA avoids most of this. It freezes the original weights and learns a low-rank update for selected matrices: for a weight matrix W, it trains two thin matrices B and A whose product BA has the same shape as W but a small rank r, typically between 8 and 64. Only A and B receive gradients.

The savings are large. With rank 16 on all seven attention and MLP projection matrices of an 8B Llama-style model, the adapter has about 42 million parameters, roughly half a percent of the model, and saves to well under 100 MB. After training you can merge BA into W, so inference costs nothing extra, or keep adapters separate and swap them; vLLM can serve many LoRA adapters on one base model.

QLoRA goes further: it loads the frozen base model in 4-bit NF4 and trains 16-bit LoRA adapters on top. The QLoRA paper fine-tuned a 65-billion-parameter model on a single 48 GB GPU; an 8B model fits easily on a 24 GB consumer card. With Hugging Face libraries the setup is short:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# QLoRA: load the frozen base model in 4-bit NF4
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model)

lora = LoraConfig(
    r=16,                 # rank of the update
    lora_alpha=32,        # the update is scaled by lora_alpha / r
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # a tiny fraction of the total

# Train with TRL's SFTTrainer or your own loop, then save only the adapter:
# model.save_pretrained("support-style-adapter")

Around this core, Hugging Face PEFT implements LoRA and its variants, TRL provides trainers for supervised fine-tuning and preference methods such as DPO, Unsloth focuses on fast, memory-efficient training on a single GPU, and Axolotl drives runs, including multi-GPU ones, from a YAML config. A finished adapter can be merged and converted to GGUF to run in llama.cpp or Ollama.

Hardware guidance

For a single user, generation speed depends mostly on memory bandwidth, because every new token reads all the active weights. Compute matters more for long prompts and many concurrent users. The table gives rules of thumb, assuming moderate context lengths; check the actual file size and your context needs before buying anything.