llama.cpp is a fast, open-source inference engine for running large language models (LLMs) locally on your own computer. It was originally created by Georgi Gerganov to run Meta's Llama models efficiently on CPUs, but it now supports many model families including Llama, Mistral, Gemma, Qwen, Phi, DeepSeek, and others.
Its main advantages are:
-
Runs locally without requiring cloud APIs or subscriptions.
-
Highly optimized for CPUs while also supporting GPUs through CUDA (NVIDIA), Metal (Apple Silicon), ROCm (AMD), Vulkan, and other backends.
-
Uses GGUF model files, a format designed for fast loading and efficient quantization.
-
Supports quantized models (e.g. 4-bit, 5-bit, 8-bit), allowing surprisingly capable models to run on consumer hardware with much less RAM.
-
Provides multiple interfaces, including:
-
a command-line tool (
llama-cli)
-
an OpenAI-compatible HTTP server (
llama-server)
-
libraries that can be embedded into applications.
For example, after downloading a GGUF model, you can run it with:
Or start a local API server:
Applications can then communicate with it using an API similar to OpenAI's Chat Completions API.
How it compares to other tools
-
llama.cpp: The underlying inference engine focused on performance and flexibility.
-
Ollama: A user-friendly application built on top of llama.cpp that simplifies downloading and managing models.
-
vLLM: Designed primarily for high-throughput model serving in production, especially on GPUs.
If you're interested in running AI models entirely on your laptop or desktop without relying on cloud services, llama.cpp is one of the most widely used and mature options.