llama.cpp is a high-performance, open-source C/C++ library and set of tools for running Large Language Models (LLMs) locally on your own computer without relying on cloud services. It was originally created to run Meta's LLaMA models, but today it supports well over 100 different model families, including Gemma, Qwen, DeepSeek, Mistral, Phi, GLM, GPT-OSS, and many others.
How it works
Instead of training AI models, llama.cpp performs inference—loading a pretrained model and generating responses.
The model itself is stored in a GGUF file, while llama.cpp provides the runtime that executes it.
Major components
llama-cli
Interactive command-line chat.
or
llama-server
Runs an OpenAI-compatible REST API.
Applications can then use:
just like the OpenAI API.
llama-bench
Measures performance.
Example output includes:
-
Prompt processing speed
-
Token generation speed
-
GPU utilization
-
Memory bandwidth
Useful for comparing quantizations such as Q4, Q5, and IQ4.
Quantization tools
Reduce model size with minimal quality loss.
Examples:
| Quant | Size | Speed | Quality |
|---|
| IQ2 | Very small | Very fast | Lower |
| IQ4_XS | Small | Fast | Good |
| Q4_K_M | Medium | Fast | Very good |
| Q5_K_M | Larger | Moderate | Excellent |
| Q8_0 | Large | Slower | Near FP16 |
Hardware support
One of the biggest strengths of llama.cpp is its broad hardware compatibility.
| Hardware | Backend |
|---|
| CPU | AVX/AVX2/AVX512 |
| Intel Arc | SYCL |
| Intel CPU | SYCL/oneAPI |
| NVIDIA | CUDA |
| AMD | HIP or Vulkan |
| Apple Silicon | Metal |
| Any Vulkan GPU | Vulkan |
It also supports hybrid CPU+GPU inference, allowing models larger than your GPU memory to run by splitting work between CPU RAM and GPU VRAM.
Why people like it
-
Completely local
-
No internet required
-
No subscriptions
-
Open source (MIT license)
-
Excellent performance
-
Supports many model families
-
Runs on Linux, Windows, and macOS
-
Can expose an OpenAI-compatible API
Typical workflow
GGUF
llama.cpp primarily uses the GGUF model format.
A GGUF file contains:
-
Model weights
-
Tokenizer
-
Metadata
-
Chat template
-
Quantization information
GGUF is designed for fast local loading and efficient inference.
Typical commands
Chat:
Use Hugging Face directly:
Run an API server:
Recent capabilities
The project has expanded well beyond a simple CLI. Recent releases include:
-
OpenAI-compatible HTTP server
-
Built-in WebUI
-
Tool calling
-
Structured JSON output
-
Vision support for compatible multimodal models
-
MCP (Model Context Protocol) support
-
Function calling
-
Grammar-constrained generation
-
Embeddings
-
RAG-friendly APIs
How it compares
| Feature | llama.cpp | Ollama | LM Studio |
|---|
| Open source | ✅ | ✅ | ❌ |
| GUI | No (WebUI optional) | Minimal | Full desktop app |
| REST API | ✅ | ✅ | ✅ |
| Maximum flexibility | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| Best for developers | ✅ | Good | Fair |
| GGUF support | Native | Native | Native |
Since you've been running Gentoo, Intel Arc, and SYCL, you're using one of llama.cpp's fastest configurations for Intel hardware. Your setup can take advantage of oneAPI/SYCL acceleration to offload inference to the Arc GPU while still falling back to the CPU when needed