When I review a private AI deployment, planned or already running, I walk the same list every time. It exists because every item on it has bitten a real team somewhere: the endpoint that was “temporarily” internet-reachable, the model nobody could roll back, the GPU server that became a single point of failure the day the whole company started depending on it. My guide to running LLMs inside a company network covers how to build the deployment in order; this checklist is how you audit one. Print it, walk it, and be honest about the gaps.
Scope and workloads
- The workloads are written down. Coding assistance, internal knowledge search, batch processing: each named, each with an owner. “We will run AI internally” is an ambition, not a scope.
- Each workload has a success measure agreed before launch: adoption, time saved, tickets deflected. Without one, the pilot never ends and never graduates.
- Someone has said no to something. A deployment that promises every team everything is sized for nothing in particular. The workloads you deferred are as important as the ones you accepted.
Data
- Data classes are mapped to paths. Which data may reach the internal models, which may still use cloud APIs, and which may touch no AI at all: written down, not tribal knowledge. This is the core decision from on-premise vs cloud, made explicit.
- RAG sources are inventoried and permissioned. A retrieval system that indexes everything quietly becomes a way to read documents you were never granted. Retrieval must respect the same access control as the source systems.
- Prompts are not retained by default. If you log requests, you log who and how much, with prompt bodies excluded or redacted, and a stated retention period.
Models
- Every model in production is pinned to a specific version and quantization, recorded somewhere a new engineer can find.
- There is an evaluation suite: representative prompts with known-good answers, run on every model or quantization change. If quality regressed last month, you should be able to name the day it happened.
- Promotion and rollback are defined. Who evaluates a new model, who approves it, and how the previous one comes back in minutes rather than a weekend.
- The embedding model is versioned too. Change it and your vector index silently mismatches; re-embedding is part of the upgrade plan, not a surprise.
Hardware and serving
- Capacity matches written workloads, with measured headroom for concurrency, not laptop-benchmark optimism. (My tool ailane answers what existing machines can run.)
- The failure mode is known. One GPU server is fine for a pilot; production needs an answer to “what happens when it is down”, even if the answer is a documented degraded mode.
- The serving layer fits the concurrency: single-user tools are not serving forty engineers, and the API in front is OpenAI-compatible so clients survive a stack swap.
- Utilization is monitored: GPU load, queue depth, time-to-first-token, with someone actually looking at the trend.
Network and access
- No inbound route from the internet to any inference endpoint. Verified by scan, not by assumption.
- TLS and internal DNS, so clients are configured once and traffic is encrypted even inside the perimeter.
- Authentication in front of every endpoint: per-team keys or SSO, with rate limits that stop one runaway script from starving everyone.
- The model-update path is deliberate: weights arrive through an egress proxy, artifact repository, or, for air-gapped sites, a defined media process. “The server has internet access so we can pull models” is a finding, not a feature.
Tooling and use
- Each approved tool has a published two-line setup pointing it at the internal endpoint. Adoption dies in configuration friction.
- Agent permissions follow least privilege and the security discipline for AI coding tools applies unchanged: self-hosting removes the data-exposure class, not the insecure-output class or the review obligation.
- Generated work is reviewed by humans under the same standards as human work. The private deployment does not relax quality control; it makes it affordable at volume.
People and operations
- The service has an owner: a named person or team, not “whoever set it up”.
- Runbooks exist for the five routine events: model update, rollback, capacity increase, endpoint outage, and credential rotation.
- The team has been trained, and there is a channel where usage questions get answered, because an unused deployment fails just as surely as a broken one.
- The decision is revisited on a schedule. Open-model capability and hardware economics move quickly; a twice-yearly review of the cloud/on-premise split keeps the architecture honest.
How to use the result
Score it plainly: items you can prove, items you believe but cannot prove, and items missing. The believed-but-unproven column is where incidents live; “verified by assumption” is the phrase to hunt down. A pilot can carry a few gaps knowingly. A production deployment carrying them unknowingly is the thing this checklist exists to prevent.
Next steps
If the checklist surfaced more gaps than your team can close alone, that is a normal outcome, not a failure: most of these items are one focused week each. The build order lives in How to Run LLMs Inside Your Company Network, the bigger picture on the private AI deployment consulting page, and if you want an outside review of your architecture against this list, tell me about your situation.
