// 7 picks · reviewed August 2026
The Best Tools for Running LLMs Locally in 2026
Running an LLM on your own machine means picking a server that turns a GGUF file (or a full-precision checkpoint) into something you can talk to over HTTP. Every option here does that. What splits them apart is what happens once you go past a single chat window: how many GPUs it can use, how many requests it can handle at once, and how much you have to know about quantization before anything runs at all.
The honest split is between ease and ceiling. Ollama and Jan get you a working model in minutes and ask nothing of you. vLLM and llama.cpp ask a lot more and pay it back in throughput and hardware control. LocalAI, textgen, and Shimmy sit in between, each solving one specific problem — no GPU, no backend lock-in, no dependency chain — well enough to be worth the extra reading.
How we picked these
Every repo here was read and assessed on its own before it was considered for this page: we pull the README, the directory layout and a key source file, and write the assessment from those rather than from the project's own marketing. Candidates for this list came from a full-text and topic search over the 56 reviewed repositories that matched this category. Anything with no commit in the last 12 months was cut, as were link collections, tutorials and boilerplates; the floor for inclusion was 1.000 stars. The ordering is a judgement call about who should pick what, not a ranking by stars. finds.dev is independent of every project listed here: nobody paid for a place on this page, there are no affiliate or referral links on it, and we have no commercial relationship with any of them. That is the point of writing down what each one is bad at as well as what it is good at.
// 1 of 7
★ 174,049 · Go · MIT · updated Jun 2026
the default first choice — easiest setup and by far the biggest ready-to-pull model library
Ollama is the one to install first. `curl localhost:11434/api/chat` works within minutes, the model library is the biggest of any tool here, and the Modelfile system means you can version a system prompt the same way you'd version code.
The cost shows up once you push it. Requests queue serially by default, so throughput collapses the moment more than one thing hits it at once — a problem vLLM doesn't have. There's no real multi-GPU splitting either, so a model too big for one card is a model Ollama can't run well, full stop. And the quantization you get is whatever the community happened to upload to the registry, with no first-party quality bar on which quant level actually suits your use case.
The app itself has also outgrown 'lean daemon' — there's a React frontend, browser tooling, and cloud config bolted on now, which is more surface area than a thing running with network access probably needs.
View on GitHub → Our full take →
// 2 of 7
★ 125,954 · C++ · MIT · updated Aug 2026
squeezing the most out of your exact hardware with every quantization format and platform backend
llama.cpp is the engine everything else on this page either sits on top of or was clearly inspired by. If you want to hand-tune inference for one exact GPU or CPU — every quantization level from 1.5-bit up, every backend from CUDA to RISC-V vector extensions — this is the actual reference implementation, not someone's wrapper around it.
That control comes with none of Ollama's guardrails. There's a pinned issue literally asking what the public API is supposed to be, which tells you how stable it is to build against. Chat templates, tool-calling, and the various grammar and parser layers grew organically and are genuinely hard to trace through the codebase. New model architectures show up here days to weeks after they land in transformers, and figuring out which quant level keeps quality intact for a given model is mostly trial and error, not documented guidance.
View on GitHub → Our full take →
// 3 of 7
★ 46,821 · Go · MIT · updated Jun 2026
an OpenAI-compatible server on a machine with no GPU at all
LocalAI earns its place for one reason: it's the pick that assumes no GPU exists. CPU-only, Raspberry Pi, multi-GPU cluster — same OpenAI-compatible API either way, and the backend-per-container design means the core binary stays small instead of bundling every engine it supports.
The tradeoff for that breadth is consistency. Thirty-plus backends means the ones nobody uses much — ROCm, Intel oneAPI — get less attention, and issues linger there. The macOS build isn't Apple-signed, so you're running `xattr` to clear a quarantine flag before first launch, the kind of friction Ollama's installer avoids entirely. And if you want the distributed, VRAM-aware routing that's genuinely unique to this project, you're now also running PostgreSQL and NATS — a real operational step up from 'run one container,' which is the pitch everywhere else on the README.
View on GitHub → Our full take →
// 4 of 7
★ 47,588 · Python · AGPL-3.0 · updated Aug 2026
a full chat UI plus API without committing to one backend
textgen is the one to reach for if you want an actual chat UI, not just an API, and you don't want to commit to one inference backend. It runs GGUF through llama.cpp, EXL3, Transformers, or TensorRT-LLM interchangeably, and the portable builds with bundled CUDA/ROCm/Vulkan binaries solve the setup pain that trips people up on llama.cpp directly.
The price is a genuinely confusing install: conda, portable, or Docker, each with its own requirements file, and picking wrong for your GPU is easy. The extensions are a grab-bag of scripts of wildly different quality — some bundle stale dependencies and look unmaintained. And the project has renamed itself (text-generation-webui to textgen) on top of a huge accumulated flag surface, so a saved config from a year ago isn't a safe bet to still work.
View on GitHub → Our full take →
// 5 of 7
★ 82,723 · Python · Apache-2.0 · updated Jun 2026
anyone with a real GPU who needs concurrent throughput, not just a single chat window
vLLM is the pick once 'local' means a real GPU serving more than one request at a time. PagedAttention and continuous batching are the reason its throughput numbers hold up in practice, not just in benchmarks, and the quantization support — FP8, INT4, GPTQ, AWQ, GGUF — is more complete in one place than anywhere else on this list.
None of that is free. Loading a 70B model and spinning up CUDA graphs takes minutes, which makes vLLM a bad fit for anything bursty — Ollama or Jan will have you talking to a model before vLLM finishes initializing. Support is Linux-and-CUDA only, so if you're on Windows or Apple Silicon this isn't even an option the way it is for the rest of the list. And with 2000+ contributors, the codebase has genuinely inconsistent corners — the ongoing V1 engine migration means you'll still trip over flags like `--use-v2-block-manager` and have to guess which generation of code you're actually running.
View on GitHub → Our full take →
// 6 of 7
★ 42,980 · TypeScript · NOASSERTION · updated Jun 2026
a polished, fully offline desktop app for non-technical users
Jan is the pick for someone who wants a ChatGPT-style app, not a server to configure — it ships to the Microsoft Store and Flathub, and the localhost:1337 OpenAI-compatible endpoint means tools like Cursor or Continue can point at it exactly like they'd point at Ollama.
It still doesn't remove the one decision every tool here eventually forces on you: the UI downloads models but leaves you to work out whether Q4_K_M or Q8_0 is the right call for your machine's memory, same as everywhere else. Windows users on AMD cards get no documented GPU acceleration path at all, and there's no first-class ARM64 Linux build — just a linked GitHub issue with a workaround. And under the polished app is a monorepo with four separate build systems, which is a lot of surface for what's supposed to be the simple option.
View on GitHub → Our full take →
// 7 of 7
★ 5,715 · Rust · Apache-2.0 · updated Jul 2026
a single dependency-free binary — no Python, no separate llama.cpp install
Shimmy's pitch is the most different one here: a single Rust binary with no Python and no llama.cpp dependency, running GGUF models through its own WebGPU engine so the same code path gets you CUDA-class acceleration on Vulkan, D3D12, or Metal without picking a backend. The INT4 KV cache compression is a real, measurable win — about 7x less VRAM for the cache, which is what lets a 3B model actually fit on a 4GB card.
It's also the newest and smallest project on this list by a wide margin, and it shows. GPU-verified model support tops out at 3B parameters — 7B is explicitly listed as pending validation, which undercuts the main selling point for anyone who wants a serious model. It only runs one model per process, so serving more than one means juggling multiple ports the way you'd never have to with Ollama. And the install path is a trap: `cargo install shimmy` quietly gives you the CPU engine, not the GPU one — you need the prebuilt GitHub release binary for that, and it's buried in a footnote rather than the main instructions.
View on GitHub → Our full take →
Questions people ask
Do I need a GPU to run any of these?
No — LocalAI and llama.cpp both run well on CPU-only machines, and Ollama's CPU path works too, just slower. If you genuinely have no GPU, LocalAI is built around that case specifically rather than treating it as a fallback.
What's actually different between Ollama and llama.cpp?
Ollama is built on llama.cpp — it wraps it in a friendlier CLI, a model registry, and a REST API so you don't touch build flags or compile anything yourself. llama.cpp gives you the same inference engine with none of the wrapper, so you get every quantization format and backend option but have to wire it up by hand.
Which of these should I use behind a coding tool like Cursor or Continue?
Ollama, Jan, LocalAI, and textgen all expose OpenAI-compatible endpoints that these tools already speak, so any of them works as a drop-in backend. Ollama has the widest first-party integration support if you want the least friction; Jan is the better pick if you also want a usable chat window on the same backend.
When does it make sense to use vLLM instead of Ollama?
When you have a real GPU and more than one person or process hitting the model at once. Ollama queues requests by default and throughput drops fast under load; vLLM's continuous batching is built for exactly that concurrency, at the cost of a much slower cold start and Linux/CUDA-only support.
None of these is wrong, but they're wrong for different people, which is exactly the kind of decision worth a second opinion. If you'd rather have picks like this land in your inbox instead of hunting for them, that's what the weekly finds.dev email is for.
Get finds like these weekly →