VPS Tutorials
Airflow 3 on a VPS: Celery, private services and tested recovery
Deploy Airflow 3.4.0 with Compose, FAB, PostgreSQL, Redis and Caddy. Configure worker API access and authenticated health checks, size concurrency, protect keys, and rehearse backup recovery.

Airflow becomes useful when a workflow needs dependencies, retries, backfills and visible execution history. Running it reliably involves much more than installing a Python package: the scheduler, DAG processor, API server, workers, broker, metadata database and encryption keys must work together.
This guide builds a recoverable single-VPS deployment on Ubuntu 24.04 with Docker Compose, CeleryExecutor, PostgreSQL 17, Redis 7.4, FAB authentication and Caddy HTTPS. The original example pins Airflow 3.3.1. The official release checked on November 16, 2026 is 3.4.0, released November 6. For a new installation, use the matching 3.4.0 Python 3.12 image and constraints, with the explicit changes below. Avoid latest; check core, provider and security release notes before subsequent updates.
- Extract data from APIs, databases or object storage, then transform and load it.
- Generate scheduled reports, synchronize search indexes or archive logs.
- Coordinate machine-learning and other batch pipelines.
- Express dependencies, branches, retries, timeouts and manual intervention.
- Keep run status, parameters, logs and failure history.
Airflow does not serve millisecond request paths or replace a queue or continuously running service. A few independent scripts may be better handled by cron or systemd timers; real-time event streams may need Kafka or Flink.
Airflow 3 uses api-server for the UI/API and requires a separate DAG processor. The old airflow webserver command is unsuitable: an accessible UI alone does not prove that DAGs parse or tasks execute.
LocalExecutor can simplify a small host. Celery separates scheduling from execution and supports additional workers:
| Component | Responsibility | Example count |
|---|---|---|
| API server | UI, authentication, public API and health | 1 |
| Scheduler | Queue eligible tasks | 1 |
| DAG processor | Parse and serialize DAG definitions | 1 |
| Celery workers | Receive and run queued tasks | 2, adjusted to capacity |
| Triggerer | Handle deferrable-operator triggers | 1 |
| PostgreSQL | Airflow metadata and Celery results | 1 |
| Redis | Celery message broker | 1 |
Celery needs its provider, broker and result backend. Redis is not the authoritative copy of workflow history. Broker loss can interrupt dispatch; password protection, persistence, memory policy and monitoring still matter.
Two workers on one VPS share the same host and disk failure domain. Critical workloads need independently placed database, broker, control plane and workers, potentially using the official Helm chart or managed Airflow.
The official Docker quick start's 4GB minimum is a startup floor, not a production sizing guarantee.
| Workload | Suggested starting VPS | Worker concurrency | Conditions |
|---|---|---|---|
| Learning | 4 vCPU, 8GB RAM, 80GB SSD | 2 | One worker, reduced component limits, no heavy tasks |
| Small production | 8 vCPU, 16GB RAM, 160GB NVMe | 4–8 after testing | Begin conservatively and measure peaks |
| Larger pipelines | 16 vCPU, 32GB+ RAM | Load-tested | Split workers, database and log storage |
A DAG can spawn memory-intensive Python, Pandas, browser or ML processes. Measure peak memory per task before raising concurrency. Exit 137 often means OOM; inspect the container restart checklist.
The full sample's container limits total about 18.3GiB. They are caps, not reservations, but simultaneous demand can exceed a 16GB host. For an initial 16GB deployment, reduce each worker to mem_limit: 2g and concurrency 2, then tune with measurements. An 8GB laboratory also needs one worker and lower control-plane/database limits.
| Port | Function | Exposure |
|---|---|---|
80/TCP | ACME validation and HTTPS redirects | Caddy only |
443/TCP | Public UI/API over TLS | Caddy only |
8080/TCP | API server | Docker network, no host mapping |
5432/TCP | PostgreSQL | Docker network |
6379/TCP | Redis | Docker network |
22/TCP | Administration | Restricted sources, SSH keys |
Do not publish 0.0.0.0:5432 or 0.0.0.0:6379. See database exposure and ports, firewalls and security groups. The sample also maps UDP 443 for Caddy HTTP/3; allow it only if needed.
Install Docker Engine and Compose v2 using the official Ubuntu instructions, then check the tools:
sudo apt update
sudo apt full-upgrade -y
sudo apt install -y ca-certificates curl jq openssl
docker --version
docker compose version
Review system upgrades and reboot requirements first. Point airflow.example.com to the host and open only restricted SSH and Caddy's public ports. Publish AAAA only with working IPv6. For Cloudflare proxy issues, check DNS, TLS mode and origin connectivity rather than removing firewall rules.
Run deployment maintenance as a dedicated non-root user. The following initialization creates new secrets and overwrites the environment file; use it only for a fresh instance:
sudo install -d -m 0750 -o "$USER" -g "$USER" \
/opt/airflow/{dags,logs,plugins,config,backups,caddy-data,caddy-config}
cd /opt/airflow
AIRFLOW_FERNET_KEY="$(docker run --rm apache/airflow:3.3.1-python3.12 \
python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"
umask 077
cat > .env <<EOF
AIRFLOW_IMAGE_NAME=airflow-local:3.3.1
AIRFLOW_UID=$(id -u)
AIRFLOW_DOMAIN=airflow.example.com
AIRFLOW_ADMIN_USER=airflowadmin
AIRFLOW_ADMIN_PASSWORD=$(openssl rand -hex 24)
[email protected]
POSTGRES_DB=airflow
POSTGRES_USER=airflow
POSTGRES_PASSWORD=$(openssl rand -hex 32)
REDIS_PASSWORD=$(openssl rand -hex 32)
AIRFLOW_FERNET_KEY=${AIRFLOW_FERNET_KEY}
AIRFLOW_API_SECRET_KEY=$(openssl rand -hex 32)
AIRFLOW_JWT_SECRET=$(openssl rand -hex 32)
EOF
chmod 600 .env
Replace the domain and administrator email. .env must stay private. Fernet encrypts connection and variable passwords; JWT signing and API secrets must remain consistent across the components that use them. Back them up securely and keep Airflow bind-mounted directories writable by the selected UID. If the host user is root, select a non-root AIRFLOW_UID and adjust ownership before starting.
Simple Auth Manager is intended for development/testing. This deployment uses the official FAB provider, with an eventual path to OAuth, LDAP or enterprise SSO. Create /opt/airflow/Dockerfile:
FROM apache/airflow:3.3.1-python3.12
ARG AIRFLOW_VERSION=3.3.1
ARG PYTHON_VERSION=3.12
ARG CONSTRAINT_URL=https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt
RUN pip install --no-cache-dir \
"apache-airflow==${AIRFLOW_VERSION}" \
apache-airflow-providers-celery \
apache-airflow-providers-fab \
--constraint "${CONSTRAINT_URL}"
For the reviewed new installation, update both image and constraint-version values before building:
cd /opt/airflow
sed -i 's/^FROM apache\/airflow:.*/FROM apache\/airflow:3.4.0-python3.12/;s/^ARG AIRFLOW_VERSION=.*/ARG AIRFLOW_VERSION=3.4.0/' Dockerfile
sed -i 's/^AIRFLOW_IMAGE_NAME=.*/AIRFLOW_IMAGE_NAME=airflow-local:3.4.0/' .env
chmod 600 .env
The 3.4.0 Python 3.12 constraints include Celery 3.26.0 and FAB 3.10.2 providers. Keep core and constraints aligned; do not run an unrestricted pip install -U. Add application dependencies through requirements.txt, rebuild and record the image digest. The key-generation image above only generates a Fernet key; these edits select the actual running core image.
Create /opt/airflow/compose.yml:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME}
build:
context: .
env_file:
- .env
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS: airflow.providers.fab.auth_manager.models.db.FABDBManager
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}
AIRFLOW__CELERY__BROKER_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW_FERNET_KEY}
AIRFLOW__API__SECRET_KEY: ${AIRFLOW_API_SECRET_KEY}
AIRFLOW__API_AUTH__JWT_SECRET: ${AIRFLOW_JWT_SECRET}
AIRFLOW__API__BASE_URL: https://${AIRFLOW_DOMAIN}
AIRFLOW__CORE__LOAD_EXAMPLES: "false"
AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: "true"
AIRFLOW__LOGGING__LOGGING_LEVEL: INFO
user: "${AIRFLOW_UID:-50000}:0"
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- ./config:/opt/airflow/config
depends_on: &airflow-common-depends-on
postgres:
condition: service_healthy
redis:
condition: service_healthy
airflow-init:
condition: service_completed_successfully
restart: unless-stopped
logging:
driver: json-file
options:
max-size: 20m
max-file: "5"
services:
postgres:
image: postgres:17-alpine
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 10s
timeout: 5s
retries: 10
restart: unless-stopped
mem_limit: 2g
redis:
image: redis:7.4-alpine
command:
- redis-server
- --appendonly
- "yes"
- --requirepass
- ${REDIS_PASSWORD}
- --maxmemory
- 768mb
- --maxmemory-policy
- noeviction
volumes:
- redis-data:/data
healthcheck:
test: ["CMD-SHELL", "redis-cli -a '$${REDIS_PASSWORD}' ping | grep PONG"]
interval: 10s
timeout: 5s
retries: 10
restart: unless-stopped
mem_limit: 1g
airflow-init:
<<: *airflow-common
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
restart: "no"
command:
- bash
- -c
- |
set -e
airflow db migrate
if ! airflow users create \
--username "$${AIRFLOW_ADMIN_USER}" \
--password "$${AIRFLOW_ADMIN_PASSWORD}" \
--firstname Airflow \
--lastname Admin \
--role Admin \
--email "$${AIRFLOW_ADMIN_EMAIL}"; then
airflow users list | grep -Fq "$${AIRFLOW_ADMIN_USER}"
fi
airflow-apiserver:
<<: *airflow-common
command: api-server
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/monitor/health"]
interval: 20s
timeout: 10s
retries: 10
start_period: 60s
mem_limit: 2g
airflow-scheduler:
<<: *airflow-common
command: scheduler
mem_limit: 2g
airflow-dag-processor:
<<: *airflow-common
command: dag-processor
mem_limit: 2g
airflow-triggerer:
<<: *airflow-common
command: triggerer
mem_limit: 1g
airflow-worker-1:
<<: *airflow-common
command: celery worker
environment:
<<: *airflow-common-env
AIRFLOW__CELERY__WORKER_CONCURRENCY: "4"
mem_limit: 4g
airflow-worker-2:
<<: *airflow-common
command: celery worker
environment:
<<: *airflow-common-env
AIRFLOW__CELERY__WORKER_CONCURRENCY: "4"
mem_limit: 4g
caddy:
image: caddy:2-alpine
environment:
AIRFLOW_DOMAIN: ${AIRFLOW_DOMAIN}
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- ./caddy-data:/data
- ./caddy-config:/config
depends_on:
airflow-apiserver:
condition: service_healthy
restart: unless-stopped
mem_limit: 256m
volumes:
postgres-data:
redis-data:
Apply these two configuration additions before the first startup. Add the internal execution endpoint to x-airflow-common.environment, keeping the public base URL unchanged:
AIRFLOW__CORE__EXECUTION_API_SERVER_URL: http://airflow-apiserver:8080/execution/
Workers need this routable Docker address; localhost inside a worker refers to that worker, and the public proxy should not be required for internal task execution. Keep the shared JWT signing secret consistent.
Add an environment mapping to the existing redis service. Its health check reads a container variable, whereas Compose interpolation alone does not inject one:
environment:
REDIS_PASSWORD: ${REDIS_PASSWORD}
Do not duplicate the redis service or replace its command. mem_limit caps each container. For small hosts, remove airflow-worker-2 and reduce concurrency and component limits as described above. Follow the Compose production checklist for logs and resource boundaries.
Redis noeviction prevents deliberate message eviction but makes writes fail when memory is exhausted. Monitor broker capacity, persistence and errors. Consider RabbitMQ when its acknowledgements, routing or operations model fits better.
Create /opt/airflow/Caddyfile:
{$AIRFLOW_DOMAIN} {
encode zstd gzip
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains"
X-Content-Type-Options "nosniff"
Referrer-Policy "strict-origin-when-cross-origin"
}
reverse_proxy airflow-apiserver:8080
}
Caddy obtains and renews certificates. Keep 8080 unpublished so the proxy's TLS cannot be bypassed. Set AIRFLOW__API__BASE_URL to the real public address and define trusted proxy/TLS boundaries when other intermediaries exist. Apply HSTS includeSubDomains only if all affected subdomains support HTTPS.
The quiet configuration check avoids printing expanded secrets. Never publish full Compose configuration:
cd /opt/airflow
docker compose config --quiet
docker compose build --pull
docker compose up airflow-init
docker compose up -d
docker compose ps
Initialization runs airflow db migrate, including the configured FAB database manager, then creates the administrator. Review each component's logs:
docker compose logs --tail=100 airflow-init
docker compose logs --tail=100 airflow-apiserver
docker compose logs --tail=100 airflow-scheduler
docker compose logs --tail=100 airflow-dag-processor
docker compose logs --tail=100 airflow-worker-1
docker compose logs --tail=100 caddy
Load only your trusted local environment before using shell-side domain/database variables:
cd /opt/airflow
set -a
. ./.env
set +a
Check component health:
curl -fsS "https://${AIRFLOW_DOMAIN}/api/v2/monitor/health" | jq
docker compose exec airflow-scheduler \
airflow jobs check --job-type SchedulerJob --local
docker compose exec airflow-dag-processor \
airflow jobs check --job-type DagProcessorJob --local
docker compose exec airflow-worker-1 \
celery --app airflow.providers.celery.executors.celery_executor.app \
inspect ping
/api/v2/monitor/health is an unauthenticated monitoring endpoint. HTTP 200 proves the request succeeded; inspect the returned database and job statuses too. Verify DAG parsing, actual worker execution, complete UI logs and sustained Redis/PostgreSQL health. Use Prometheus and Grafana for host/container metrics.
Create /opt/airflow/dags/vps_smoke_test.py:
from datetime import datetime
from airflow.sdk import DAG, task
with DAG(
dag_id="vps_smoke_test",
schedule=None,
start_date=datetime(2026, 1, 1),
catchup=False,
tags=["smoke-test"],
) as dag:
@task
def show_worker() -> str:
import socket
return socket.gethostname()
@task
def confirm(hostname: str) -> None:
print(f"Celery task completed on {hostname}")
confirm(show_worker())
Trigger it through the UI after parsing. Both tasks should succeed, workers should receive them and the UI should display complete logs. If the DAG is absent:
docker compose exec airflow-dag-processor airflow dags list-import-errors
docker compose exec airflow-dag-processor python /opt/airflow/dags/vps_smoke_test.py
Fix imports, mounted-file permissions or missing dependencies rather than repeatedly restarting the API server. The historical start date with catchup disabled does not request a backlog.
Control external API/database access separately from worker process counts:
docker compose exec airflow-apiserver \
airflow pools set external_api 4 "第三方 API 并发限制"
docker compose exec airflow-apiserver airflow pools list
Set pool="external_api" on relevant tasks. The pool description is human-readable and may be replaced with your team's language. Use smaller pools for memory-heavy tasks, plus per-DAG limits. Raising parallelism, worker concurrency and database pools together can overload PostgreSQL, disk or external APIs.
Backing up dags/ alone is insufficient. Preserve:
- PostgreSQL metadata, users, connections, variables and execution history.
dags/,plugins/,config/and deployment files..env, including Fernet/JWT/API keys and database credentials.- Task logs, or a verified remote-log repository.
- Dockerfile, additional requirements, image tags and digests.
Create /opt/airflow/backup.sh:
#!/usr/bin/env bash
set -Eeuo pipefail
cd /opt/airflow
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
dest="backups/${stamp}"
mkdir -p "${dest}"
docker compose exec -T postgres \
pg_dump -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" \
--format=custom --no-owner --no-privileges \
> "${dest}/airflow.dump"
tar --exclude='./backups' \
-czf "${dest}/airflow-files.tar.gz" \
.env Dockerfile compose.yml Caddyfile dags plugins config
sha256sum "${dest}"/* > "${dest}/SHA256SUMS"
find backups -mindepth 1 -maxdepth 1 -type d -mtime +14 -print
Before running it, insert the following immediately after its cd /opt/airflow line. Compose reads .env for interpolation, but that does not populate the script's POSTGRES_USER/POSTGRES_DB variables:
set -a
. ./.env
set +a
umask 077
Add logs to the tar input list when using local task logs; include requirements.txt if used. Restrict the script and backup directories to the maintenance account, make backup.sh executable, and keep archives encrypted off-host. Check pg_dump status and archive/checksum readability. The find command only lists directories older than 14 days; deletion belongs to an explicit retention policy after confirming independent copies.
Fernet loss prevents decryption even with a complete database dump. Rotate using new-key,old-key order (新密钥,旧密钥), run airflow rotate-fernet-key against the shared configuration, confirm successful re-encryption, then remove the old key. Preserve recovery access during the transition.
Use a separate VPS, or a distinct Compose project with new volumes, sandbox DNS and nonconflicting ports. Restore the saved deployment/environment first and load its trusted .env. The commands below replace the destination database:
cd /opt/airflow
docker compose down
docker compose up -d postgres redis
docker compose exec -T postgres \
dropdb -U "${POSTGRES_USER}" --if-exists "${POSTGRES_DB}"
docker compose exec -T postgres \
createdb -U "${POSTGRES_USER}" "${POSTGRES_DB}"
docker compose exec -T postgres \
pg_restore -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" \
--clean --if-exists --no-owner --no-privileges \
< backups/替换为时间戳/airflow.dump
docker compose run --rm airflow-init
docker compose up -d
Replace the timestamp placeholder with the actual archived path. Use the original compatible image initially; inspect PostgreSQL readiness before dropdb/createdb. Check administrator login, DAGs, historical runs, connection decryption, a test task and logs. Broker recovery requires reconciling queued/running tasks against external side effects, especially for non-idempotent operations.
Read core/provider release notes and take an off-host backup. Stop every writing component before metadata migration:
cd /opt/airflow
./backup.sh
docker compose stop \
airflow-apiserver airflow-scheduler airflow-dag-processor \
airflow-triggerer airflow-worker-1 airflow-worker-2
# 修改 Dockerfile 和 .env 中的固定版本后
docker compose build --pull
docker compose run --rm airflow-init
docker compose up -d
docker compose ps
Update pinned Dockerfile/environment versions together. If you removed the second worker, omit it from service commands. Use airflow db migrate, not removed airflow db init or historical db upgrade. Rehearse major changes against a restored database. An incompatible migration needs the old image and the pre-upgrade database/configuration backup for rollback.
Check AIRFLOW__CORE__AUTH_MANAGER, FAB installation, airflow-init and AIRFLOW__API__BASE_URL. API instances must share the API secret; check proxy headers and public URL too.
Check airflow-dag-processor, mount ownership and airflow dags list-import-errors. Airflow 3 requires the processor independently of the UI.
Check Redis authentication/health, broker URL, worker availability, queue names, provider installation, pools and concurrency. Then verify the internal execution API address and JWT configuration.
Verify shared API/JWT secrets and synchronized clocks. Expired or incorrectly timed tokens can reject API and log requests.
Count usage across API, scheduler, processor, triggerer and workers before tuning SQLAlchemy pools. Increasing max_connections alone consumes more RAM and can hide leaks.
Use docker compose ps, docker inspect and journalctl -k. Reduce worker concurrency, constrain heavy pools or add capacity; see the restart guide.
The 4GB quick-start floor is for experimentation. Start small production near 8 vCPU/16GB with reduced worker limits, then size from parsing and task-memory measurements.
Use airflow api-server instead of webserver, and run the mandatory DAG processor. Update health checks and proxy upstreams together.
PostgreSQL, DAGs, configuration and keys are essential. Redis AOF helps with short outages but is not workflow history; reconcile incomplete tasks before replaying them.
FAB supports managed users/roles and enterprise identity integration. Add appropriate SSO/MFA, restricted access or VPN for administrative use; installing FAB alone does not establish those controls.
No. Preserve keys independently and use the multi-key transition with airflow rotate-fernet-key for rotation.
They increase host-local parallelism, but independent failure domains, remote logs and off-host backups are required for host-failure tolerance.
- Pin core, providers, PostgreSQL and Redis; avoid
latest. - Publish only Caddy and restricted SSH; leave 8080/5432/6379 private.
- Run API server, scheduler, DAG processor, triggerer and selected workers.
- Inspect
/api/v2/monitor/health, job checks, the smoke DAG and complete logs. - Use a random FAB administrator password and plan enterprise access controls.
- Keep offline copies of Fernet/JWT/API secrets and database credentials.
- Maintain independent metadata, deployment and log backups; rehearse restoration.
- Size worker concurrency, pools, database connections and RAM from measurements.
- Back up, stop components and run
airflow db migratebefore upgrading. - Monitor host resources, containers, queue depth and task failures.