VPS Tutorials
Build an AnythingLLM Private Knowledge Base on a VPS (2026): Docker, OpenAI/Ollama, RAG Document Q&A and HTTPS Deployment Guide
This tutorial explains how to self-host AnythingLLM on a VPS and build a private AI knowledge base that can upload PDFs, Markdown and web content and answer questions over documents with RAG. It covers VPS sizing, Docker deployment, persistent storage, connecting both OpenAI and Ollama models, Caddy HTTPS, workspace permissions, backup and upgrade, security hardening, and common connection failures.
I've recently seen many people wanting to build "internal company knowledge base AI Q&A," but they pick the wrong tool at the very first step.
Some people just throw PDFs at ChatGPT. It's genuinely convenient, but contracts, customer data and internal SOPs all have to be uploaded to a third-party platform. Others deploy Dify, which is feature-complete, but if they just want a few people to look up documents, the workflow feels heavy. Still others install Open WebUI, which has a nice chat experience, but the document knowledge base side isn't necessarily the smoothest entry point.
If your need is very specific:
Upload a batch of documents, then let the AI answer questions based on those materials.
AnythingLLM fits well.
It's not just a chat shell, and it's not a complex AI application development platform. It's more like a "private document Q&A backend": you create Workspaces, upload PDFs, Markdown, TXT and web content, configure models and vector retrieval, and then you can chat around those materials.
This article covers the VPS self-hosting approach: how to deploy, how big the VPS should be, how to choose between OpenAI and Ollama, which data you must back up, and which pitfalls to avoid.
Don't just install it because an AI tool is trending.
I'd judge it like this:
| Need | Suitable for AnythingLLM | Better tool |
|---|---|---|
| Private document Q&A | Suitable | AnythingLLM |
| Multi-user shared knowledge base | Suitable | AnythingLLM Docker |
| Complex AI workflows | So-so | Dify / n8n |
| Just chatting with local models | So-so | Open WebUI |
| Unifying multiple model APIs | Not suitable | One-API / New-API / LiteLLM |
| Customer service embedded in a website | Possible | AnythingLLM Chat Widget |
In one sentence: AnythingLLM's strengths are documents, Workspaces, RAG and team access.
If you just want to ask a model "help me write some copy," Open WebUI is more direct. If you want to orchestrate a whole AI application, Dify is more of a platform. If you want to put company materials, product docs, tutorials and contract templates in and query them, AnythingLLM is closer to the need.
There are two cases here.
This saves the most VPS resources. Model inference doesn't run on your VPS; the VPS only handles:
- AnythingLLM web backend
- Document upload and splitting
- Vector indexing
- User login and workspaces
- Communication with external LLM APIs
Recommended specs:
| Usage | VPS specs |
|---|---|
| Personal testing | 1 core / 2GB / 20GB SSD |
| Small team of 2-5 | 2 cores / 4GB / 40GB SSD |
| Many documents, multiple users | 4 cores / 8GB / 80GB SSD |
If you're only connecting to the OpenAI API, you don't need a GPU.
Many people hear "AI tool" and go looking for a GPU VPS, but that's unnecessary. When you use cloud models, what really costs money is API calls, not VPS compute.
This is completely different.
Running a 7B small model with Ollama already strains memory noticeably. It can run on CPU too, but it won't be fast; without a GPU, generation speed may be so slow you suspect the service is broken.
My recommendation:
| Model approach | Recommended specs |
|---|---|
| Only AnythingLLM + cloud models | 2 cores 4GB is comfortable enough |
| AnythingLLM + Ollama small model | Start at 4 cores 16GB |
| Want to run larger local models | GPU server or a local machine with a graphics card |
So don't force a 1C1G tiny box to run large local models. Being able to start doesn't mean it's usable; being able to answer doesn't mean the experience is good.
This article uses an Ubuntu / Debian VPS as an example.
First, log in via SSH:
ssh root@your_server_ip
Update the system:
apt update && apt upgrade -y
Install basic tools:
apt install -y ca-certificates curl gnupg ufw
Check system resources:
free -h
df -h
AnythingLLM listens on port 3001 by default. First confirm it's not occupied:
ss -tlnp | grep 3001
If there's no output, the port is free.
If Docker isn't installed on the VPS yet, you can install it from the official repository.
Add the Docker GPG key:
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
Add the Docker repository:
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" > /etc/apt/sources.list.d/docker.list
Install Docker:
apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Confirm the version:
docker --version
docker compose version
The official Docker deployment approach is straightforward: prepare a persistent directory and mount it to the container's /app/server/storage.
I recommend putting data in /opt/anythingllm:
mkdir -p /opt/anythingllm
cd /opt/anythingllm
touch .env
Run the container:
docker run -d \
--name anythingllm \
--restart unless-stopped \
-p 3001:3001 \
--cap-add SYS_ADMIN \
-v /opt/anythingllm:/app/server/storage \
-v /opt/anythingllm/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm:latest
Check container status:
docker ps --filter "name=anythingllm"
Check logs:
docker logs anythingllm --tail=100
Access in the browser:
http://your_server_ip:3001
The first visit goes through the setup wizard, creates the admin account, then selects the LLM Provider, Embedding Provider and vector database.
If you're more used to Compose, you can write it like this.
Create the directory:
mkdir -p /opt/anythingllm
cd /opt/anythingllm
touch .env
Create docker-compose.yml:
services:
anythingllm:
image: mintplexlabs/anythingllm:latest
container_name: anythingllm
restart: unless-stopped
ports:
- "127.0.0.1:3001:3001"
cap_add:
- SYS_ADMIN
environment:
STORAGE_DIR: /app/server/storage
JWT_SECRET: "replace-with-a-long-random-string-at-least-32-chars"
PASSWORDMINCHAR: "10"
DISABLE_TELEMETRY: "true"
volumes:
- /opt/anythingllm:/app/server/storage
- /opt/anythingllm/.env:/app/server/.env
Start:
docker compose up -d
Here I wrote the port as:
127.0.0.1:3001:3001
This means only the local machine can access 3001, not directly exposed to the public internet. Later, use Caddy to reverse proxy to an HTTPS domain.
This is safer than opening 3001 to the whole internet.
Assume your domain is:
ai.example.com
Add an A record in DNS:
ai.example.com -> your_server_ip
Install Caddy:
apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' > /etc/apt/sources.list.d/caddy-stable.list
apt update
apt install -y caddy
Edit the Caddyfile:
nano /etc/caddy/Caddyfile
Write:
ai.example.com {
reverse_proxy 127.0.0.1:3001
}
Check the config:
caddy validate --config /etc/caddy/Caddyfile
Reload:
systemctl reload caddy
Now access:
https://ai.example.com
Caddy automatically requests and renews HTTPS certificates.
If you use Caddy as a reverse proxy, the public internet only needs SSH, 80 and 443 open.
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable
ufw status
Don't expose 3001 directly to the public internet:
ufw deny 3001/tcp
If your Docker Compose earlier already bound to 127.0.0.1:3001:3001, even if the cloud provider's security group mistakenly opens 3001, external access to the container port won't work.
AnythingLLM is just the frontend and knowledge base; it still needs to connect to a model.
This suits most VPS users.
Pros:
- Low VPS spec requirements
- Stable answer speed
- No need to manage local models
- Most worry-free for small team deployments
Cons:
- Document content is sent to the corresponding API provider for processing
- There are API costs
- Depends on external service availability
If the materials aren't extremely sensitive, such as public product docs, customer service FAQs, blog content, ordinary internal process docs, this option is more realistic.
Suits users who want to keep data on their own machine as much as possible.
First install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
Pull a small model to test:
ollama pull llama3.1:8b
Test the endpoint:
curl http://127.0.0.1:11434/api/tags
If AnythingLLM is in Docker, localhost inside the container refers to the container itself, not the host.
On Linux you can add this in Compose:
extra_hosts:
- "host.docker.internal:host-gateway"
Then in AnythingLLM set the Ollama address to:
http://host.docker.internal:11434
If you use Docker Run, you need to add:
--add-host=host.docker.internal:host-gateway
Many people trip up here: they fill in http://localhost:11434 inside the container and keep getting connection failures.
The easiest thing to confuse in AnythingLLM is this.
Simply put:
| Method | Suitable scenario | Scope of effect |
|---|---|---|
| Attach file | Temporarily ask about one file | Better for the current conversation |
| Embed into Workspace | Long-term knowledge base | Retrievable in the workspace afterward |
If you just want the AI to look at one PDF and ask a few questions, Attach is enough.
If you want "company policy Q&A," "product manual Q&A," "customer support knowledge base," you should embed documents into the Workspace so they become a retrievable knowledge base.
My advice:
- One Workspace per department
- One Workspace per product line
- Don't mix sensitive materials with public materials in the same Workspace
- For files you're unsure about using long-term, Attach first, don't rush to Embed
RAG isn't about stuffing all files in at once. The clearer the material boundaries, the more reliable the answers.
AnythingLLM Docker supports multi-user and permission management.
But multi-user isn't "enable it first and figure it out later." You need to think through first:
- Who can see which Workspace
- Which documents can be shared
- Who keeps the admin account
- How to remove departed users
- Whether to allow a public Chat Widget
If it's just you testing, run the flow as a single user first.
Once you confirm workspaces, document classification and model settings are stable, then consider multi-user. Opening team access before the permission design is clear easily leads to chaos later.
The scariest thing for a self-hosted knowledge base isn't a brief service outage, but losing document indexes and configuration.
AnythingLLM's key data is in the storage directory you mounted, for example:
/opt/anythingllm
At minimum back up:
.env- The database and configuration in the storage directory
- Uploaded documents
- Vector index related data
You can first make a local archive:
cd /opt
tar -czf anythingllm-backup-$(date +%F).tar.gz anythingllm
Then sync to object storage with rclone:
rclone copy /opt/anythingllm-backup-$(date +%F).tar.gz r2:backup-bucket/anythingllm/
Backing up without restoring is the same as not finishing.
Find an off-peak time to rehearse once:
docker compose down
mv /opt/anythingllm /opt/anythingllm-old
mkdir -p /opt/anythingllm
tar -xzf /opt/anythingllm-backup-2026-07-12.tar.gz -C /opt
docker compose up -d
Confirm accounts, Workspaces, documents and chat all still work.
Upgrading a Docker deployment is simple, but don't upgrade blindly in production.
Enter the directory:
cd /opt/anythingllm
Back up first:
tar -czf /opt/anythingllm-before-upgrade-$(date +%F).tar.gz /opt/anythingllm
Pull the new image:
docker compose pull
Restart:
docker compose up -d
Check logs:
docker logs anythingllm --tail=100
After upgrading, focus on checking:
- Can you log in
- Are Workspaces still there
- Can documents be retrieved
- Is the LLM Provider working
- Is the Embedding Provider working
- Is Caddy HTTPS working
If you find anomalies, don't delete data first. Keep the old directory and backup, then decide on rollback.
Check whether the container is running:
docker ps --filter "name=anythingllm"
Check logs:
docker logs anythingllm --tail=100
Check the port:
ss -tlnp | grep 3001
If you use Caddy, keep checking:
systemctl status caddy --no-pager -l
journalctl -u caddy -n 100 --no-pager
Usually AnythingLLM isn't up, or the Caddyfile points to the wrong place.
Confirm the local machine can access it:
curl -I http://127.0.0.1:3001
If the local machine can't access it either, fix Docker first; if the local machine can access it, then fix Caddy.
Test on the host first:
curl http://127.0.0.1:11434/api/tags
If the host is fine but the container isn't, it's most likely a wrong address.
When AnythingLLM in Docker accesses host Ollama, usually don't write:
http://localhost:11434
Change to:
http://host.docker.internal:11434
And make sure Compose has:
extra_hosts:
- "host.docker.internal:host-gateway"
This usually isn't a server problem, but a knowledge base design problem.
Check first:
- Are the documents too old
- Are unrelated materials mixed in the same Workspace
- Is the question too broad
- Is the Embedder configured correctly
- Can relevant chunks be retrieved after document splitting
Don't expect RAG to understand everything automatically. If you give it a pile of messy files, it will find answers from the messy files.
Check memory:
free -h
Check OOM:
journalctl -k | grep -i "out of memory"
If you run AnythingLLM and Ollama at the same time, running out of memory is normal.
Short term you can add swap:
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
But swap is only for emergencies; it can't turn a 2GB tiny box into an AI server.
Once AnythingLLM is on the public internet, at minimum do these things:
- Access only via HTTPS domain
- Don't directly expose port 3001
- Use a strong admin password
- Don't write API keys into public repositories
- Separate different teams' materials into different Workspaces
- Regularly back up
/opt/anythingllm - Disable departed accounts promptly
- Confirm compliance requirements before uploading sensitive materials
If it's just for your own use, you can also add a layer of Cloudflare Access or Tailscale to allow only your own devices to access.
These three tools are often asked about together.
| Tool | Better for |
|---|---|
| AnythingLLM | Document Q&A, private knowledge base, small team RAG |
| Dify | AI applications, workflows, Agents, external service orchestration |
| Open WebUI | ChatGPT-like chat interface, local model chat, multi-model entry point |
My choice is simple:
- Want to "upload documents then ask questions": AnythingLLM
- Want to "build an AI application for others to use": Dify
- Want to "give Ollama a good chat frontend": Open WebUI
These three don't fully replace each other; they're just different entry points.
If building internal document Q&A for a small team of 3-5 people, I'd configure it like this:
- VPS: start at 2 cores 4GB
- Model: connect to OpenAI / OpenRouter first, don't rush to run locally
- Deployment: Docker Compose + Caddy HTTPS
- Ports: only open 80/443, don't open 3001
- Documents: create Workspaces by department or product line
- Backup: back up
/opt/anythingllmto object storage daily - Permissions: test as a single user first, enable multi-user after the flow is stable
Once you truly confirm the materials can't leave the network, then consider heavier options like Ollama, local embedding and GPU servers.
The value of AnythingLLM isn't "installed an AI tool," but turning scattered documents into a knowledge base you can ask, query and share. The VPS is just the carrier; what really needs long-term maintenance is material boundaries, permissions and backups.