Last reviewed: September 2026. Tool licenses change; I re-check this page twice a year and note changes in the log at the end.
The stack under a private AI deployment has seven layers: a runtime that executes the model, an interface people talk to it through, an orchestration layer for retrieval and agents, a vector store, tooling for fine-tuning, tooling for evaluation, and observability. This page lists what I use and recommend at each layer, with the license of every tool, and it separates the tools that are genuinely open source from the ones that are free to use but proprietary.
This is chapter 5 of the Open Source AI Hub. A tool is listed as open source here only if its license is approved by the Open Source Initiative; open-core and source-available projects are labeled as such, because for a hosted product the difference can matter as much as the model license in chapter 3.
Inference runtimes
llama.cpp (MIT) is the foundation of local inference: a C++ implementation that runs quantized GGUF models on CPUs and every common GPU, and the engine underneath several of the tools below. If you want to understand what your machine is doing, start here.
Ollama (MIT for the server and command-line tool) wraps llama.cpp in a model library, a one-line install and an HTTP API with OpenAI-compatible endpoints. It is what I put on every developer machine and the first server I stand up for a team. Its native desktop application is a newer, separate component; check the repository for the license of the piece you deploy. My Ollama guide and Ollama for Teams cover it end to end.
vLLM (Apache-2.0) is the high-throughput server for GPUs: continuous batching, paged attention and tensor parallelism across several cards. When a single Ollama box stops keeping up with a team, vLLM is the next step, and running LLMs inside your company network explains when to make it.
SGLang (Apache-2.0) competes with vLLM on serving performance, with strengths in structured output and multi-turn caching. Text Generation Inference (Apache-2.0) is Hugging Face’s server; it spent a period under a restrictive license from July 2023 and returned to Apache-2.0 in April 2024, a small reminder to check even the tools you trust. LocalAI (MIT) gives an OpenAI-compatible API over many model types. llamafile (Apache-2.0) packs a model and llama.cpp into one executable, handy for demos. MLX and mlx-lm (MIT) are Apple’s framework for running models on Apple silicon, and the fastest option on a Mac.
How I choose: llama.cpp or Ollama for a laptop or a single machine, Ollama for a small team on one server, vLLM or SGLang for a multi-user service on real GPUs.
Chat interfaces
Open WebUI is the most popular self-hosted chat interface, and its license changed in 2025: from version 0.6.6 it uses its own license, based on BSD-3-Clause, that adds a requirement to keep the Open WebUI branding visible unless you have a small deployment or a separate agreement. That makes it source-available rather than OSI open source. Read the clause before you put it in front of customers with your own logo on it.
LibreChat (MIT) is a full-featured alternative with multi-provider support and no branding clause; it is what I now default to for client deployments. AnythingLLM (MIT) bundles chat with document retrieval for teams that want one install. text-generation-webui (AGPL-3.0) is the power-user interface, and the AGPL matters: if you host a modified version as a service, you must offer your changes to its users.
Free, but not open source
LM Studio is a polished desktop application for running local models, free to use including at work, and proprietary. Its command-line tool and SDKs are MIT; the application is not. It is a fine tool. It simply belongs in a different column in a procurement review, and it should not be described as open source in one.
Orchestration and RAG frameworks
LangChain and LangGraph (MIT) are the most widely used libraries for chaining models, tools and retrieval; note that LangSmith, the company’s observability product, is a proprietary service. LlamaIndex (MIT) is the stronger choice when retrieval over documents is the core of the product. Haystack (Apache-2.0) is the most production-minded of the three, with explicit pipelines. DSPy (MIT) treats prompts as programs to be optimized rather than written, and is worth learning once you have an evaluation set.
My honest position: most private deployments need far less framework than they think. A retrieval pipeline is a few hundred lines of plain code, and my private RAG guide builds one without any of the above. Reach for a framework when you need its integrations, not because a tutorial did.
Vector stores
pgvector (PostgreSQL License) is my default: a PostgreSQL extension that gives you vector search inside the database you already run, with backups, permissions and transactions you already understand. Qdrant (Apache-2.0) is the purpose-built store I choose when filtering and scale outgrow Postgres. Chroma (Apache-2.0) is the easiest to start with for prototypes. Weaviate (BSD-3-Clause), Milvus (Apache-2.0, a Linux Foundation project) and LanceDB (Apache-2.0) are all sound. FAISS (MIT) is a library rather than a server, ideal when the index lives inside your application. For contrast, Pinecone is a proprietary managed service; useful, but it is not something you can run inside your own network.
Fine-tuning
Fine-tuning continues training an existing model on your own examples so that it adopts a style, a format or domain knowledge that prompting alone cannot reach. The Hugging Face libraries are the foundation: transformers, PEFT, TRL, datasets and accelerate are all Apache-2.0. On top of them, Axolotl (Apache-2.0) and LLaMA-Factory (Apache-2.0) turn a YAML file into a training run, and Unsloth (Apache-2.0 for the core library; check the licenses of its companion packages and its enterprise features) makes single-GPU fine-tuning two to five times faster.
The methods in one paragraph each. LoRA trains small adapter matrices alongside frozen weights, so a fine-tune is a few hundred megabytes and can be swapped at runtime. QLoRA does the same over a 4-bit quantized base, which is what lets a 70B model be tuned on a single large GPU. DPO and its relatives train on pairs of preferred and rejected answers to shape behavior without a separate reward model, and GRPO, popularized by DeepSeek R1, extends that idea to reasoning.
When fine-tuning is the wrong tool: whenever the problem is that the model does not know your facts. That is a retrieval problem, and retrieval is cheaper, updatable and auditable. Fine-tune for style, format, tone and narrow skills; retrieve for knowledge. My RAG guide makes the same point at length.
Evaluation
An evaluation suite is a fixed set of real examples from your workload, scored the same way every time, that tells you whether a model, a prompt or an upgrade made things better or worse. lm-evaluation-harness (MIT, from EleutherAI) runs the standard academic benchmarks. Inspect (MIT, from the UK AI Security Institute) is a clean framework for writing your own evaluations. promptfoo (MIT) is the pragmatic choice for teams: a YAML file of prompts, expected behaviors and assertions, run against any model or endpoint. RAGAS (Apache-2.0) and DeepEval (Apache-2.0) score retrieval pipelines on faithfulness and relevance.
The one rule: fifty cases from your own work beat every public leaderboard, and you should build them before you choose a model, not after. This is the same discipline I ask of AI-generated code in You Are the Quality Control: the tool proposes, you verify, and you verify with a test you wrote.
Observability
Langfuse is open-core: the core is MIT, with some enterprise features under a commercial license, and it is what I deploy for tracing prompts, latency and cost inside a client’s network. Arize Phoenix is source-available under the Elastic License 2.0, which lets you use and modify it but not offer it as a managed service to others; that is fine for internal use and worth knowing before you build a product on it. OpenLLMetry (Apache-2.0) adds LLM spans to standard OpenTelemetry, so your existing tracing stack can carry them.
The Hugging Face Hub
The Hub is where nearly every open model and dataset is published. The website is a service; the libraries you use to talk to it, including huggingface_hub, are Apache-2.0. Three things to know. Gated models require you to accept the maker’s terms before downloading, and those terms are the license, so read them at that moment. The license field on a model page is filled in by whoever uploaded the files and is sometimes wrong, which is why chapter 3 says to open the license file itself. And for air-gapped deployments you can mirror what you need to an internal store and point the libraries at it; the company-network guide shows how.
A reference stack for a private deployment
For a team of a few dozen people who want private AI without vendor lock-in, the stack I most often build is: Ollama for developer machines and vLLM for the shared service; LibreChat as the interface; pgvector in the existing PostgreSQL for retrieval, with nomic-embed-text or a Qwen3 embedding model; promptfoo for evaluation; Langfuse for observability; and a Qwen3 or Mistral Small model at the center, with a larger model behind it for hard requests. Every component is under MIT, Apache-2.0, the PostgreSQL License or BSD except where I have said otherwise, and the whole thing runs on hardware the client owns. What to audit before it goes live is in the Private AI Architecture Checklist.
Change log
- September 2026: first published.
Next steps
Continue to chapter 6, Open Source AI Coding Assistants, which covers the tools developers use these models through. For the retrieval layer in practice, read how to build a private RAG system, and if you would rather have the whole stack designed and run for you, that is private AI deployment consulting.