VPS Tutorials
Kafka KRaft on a VPS: explicit TLS, ACLs and recoverable storage
Deploy single-node Kafka with the supported 4.3 patch, explicit TLS stores and controller authentication. Configure client routing, topic and group ACLs, test application access, and plan retention, monitoring and cold recovery.

When several services produce and consume payments, logs or order events, direct database and HTTP calls can tightly couple them. Kafka stores events in a partitioned, replayable append-only log, allowing producers and independent consumer groups to work at their own pace.
This Ubuntu 24.04 and Docker Compose guide uses KRaft, official images, SASL/PLAIN over TLS, ACLs, persistent storage and a pinned version. The September 15, 2026 example used Kafka 4.3.1. The official download page checked on November 16 lists 4.4.0, 4.3.2 and 4.2.2 as supported releases and 4.3.1 as archived. The configuration below retains the original example and adds a required override for explicit TLS settings; choose the supported 4.3.2 patch for a fresh deployment after reading its release notes.
- Separate order, payment and inventory services with events.
- Feed logs or analytics into several independent downstream systems.
- Save consumer offsets and resume after failure within retained history.
- Let multiple consumer groups read the same topic independently.
- Use partitions for parallelism and retain events for a defined period.
For individual tasks, flexible routing and explicit message acknowledgments, RabbitMQ is often simpler. Kafka is useful for replay, stream processing and sharing an event history.
| Architecture | Benefit | Limitation | Use |
|---|---|---|---|
| Single-node KRaft | Lower cost and simple operation | Host, disk or process failure interrupts service; replication factor 1 | Development or recoverable low-risk workloads |
| Three containers on one VPS | Tests cluster configuration | Shares host, disk and network failures | Laboratory only |
| Independent production nodes | Three topic replicas can tolerate one unavailable replica | Higher cost and operational complexity | Critical services |
Combined mode runs broker and controller in one process. Official guidance does not recommend it for critical deployments. Design independent broker failure domains and a controller quorum of 3 or 5, rather than treating three copied containers as HA.
For three topic replicas, replication.factor=3, min.insync.replicas=2 and producer acks=all require enough in-sync replicas before confirming a write. The acknowledgment applies to the current ISR, subject to its minimum; it is not a backup or a promise that every configured replica always acknowledged.
Kafka 4.x uses KRaft instead of ZooKeeper. Dynamic controller quorums are supported, but this single-node example uses static controller.quorum.voters. Plan a supported role/quorum migration before expanding; never clone a populated data directory to create a different node.
Plan peak write rate, message size, partitions, retention, replica count, consumer replay and disk/network performance.
| Workload | Suggested VPS | Memory/storage assumption |
|---|---|---|
| Development | 2 vCPU, 4GB RAM, 50GB SSD | About 1GB heap and disposable data |
| Small single node | 4 vCPU, 8GB RAM, at least 100GB NVMe | 2GB heap with room for page cache |
| Critical cluster | At least 3 broker failure domains, 4–8 vCPU/8–16GB each, independent NVMe | Separate controller planning and realistic load tests |
These are estimates, not measured guarantees. Do not consume all RAM with JVM heap: page cache is essential. The default example limits the container to 4GB and heap to 2GB; for a 4GB development host reduce heap to around 1GB and the container limit appropriately. Avoid sharing the data disk with another high-I/O database.
| Port | Purpose | Exposure |
|---|---|---|
9094/TCP | Client SASL_SSL | 127.0.0.1; a WireGuard IP for remote clients |
19094/TCP | Internal broker listener | Not published |
29093/TCP | KRaft controller | Not published |
Restrict the client listener to application sources even with TLS/ACLs. Do not expose the controller network or plaintext 9092 publicly. Check Docker publication and cloud rules using the middleware exposure checklist.
Install Docker Engine and Compose v2 using the official Ubuntu instructions before this block:
sudo apt update
sudo apt full-upgrade -y
sudo apt install -y ca-certificates curl openssl openjdk-21-jre-headless
docker --version
docker compose version
keytool -help >/dev/null
The host Java installation supplies keytool; the image supplies Kafka's runtime. Install netcat-openbsd if using the later nc diagnostic. Review reboots, log rotation, resource limits and recovery with the Compose production checklist.
sudo install -d -m 0750 -o "$USER" -g "$USER" /opt/kafka
cd /opt/kafka
install -d -m 0750 data backups
install -d -m 0700 secrets pki
umask 077
docker run --rm --entrypoint /opt/kafka/bin/kafka-storage.sh \
apache/kafka:4.3.1 random-uuid
Save the generated Cluster ID. Create .env with your actual ID, hostname and private bind address:
CLUSTER_ID=请替换为上一步输出
KAFKA_HOST=mq.example.com
KAFKA_BIND_IP=127.0.0.1
Replace the entire example placeholder after CLUSTER_ID with that generated value. Never generate a new ID when reusing populated storage.
For a WireGuard interface at 10.66.0.1, set KAFKA_BIND_IP to that address and resolve mq.example.com to it from the application hosts. Loopback mode requires hostname resolution to loopback on the VPS for external-listener tests.
Generate hex credentials in the same shell session:
ADMIN_PASSWORD="$(openssl rand -hex 32)"
APP_PASSWORD="$(openssl rand -hex 32)"
STORE_PASSWORD="$(openssl rand -hex 24)"
printf '%s\n' "$STORE_PASSWORD" > secrets/kafka_keystore_creds
printf '%s\n' "$STORE_PASSWORD" > secrets/kafka_ssl_key_creds
printf '%s\n' "$STORE_PASSWORD" > secrets/kafka_truststore_creds
printf 'KafkaServer {\n org.apache.kafka.common.security.plain.PlainLoginModule required\n username="admin"\n password="%s"\n user_admin="%s"\n user_app="%s";\n};\n' \
"$ADMIN_PASSWORD" "$ADMIN_PASSWORD" "$APP_PASSWORD" \
> secrets/broker_jaas.conf
Reserve admin for operations and use app for applications. PLAIN itself does not encrypt passwords; TLS is mandatory. Keep JAAS, client properties and .env private. Preserve the generated application password securely before ending this shell session.
Issue certificates covering the container hostname kafka and client hostname mq.example.com:
cd /opt/kafka/pki
openssl genrsa -out ca.key 4096
openssl req -x509 -new -sha256 -days 3650 \
-key ca.key \
-subj '/CN=Kafka Internal CA' \
-out ca.crt
openssl genrsa -out broker.key 4096
openssl req -new -sha256 \
-key broker.key \
-subj '/CN=mq.example.com' \
-out broker.csr
printf '%s\n' \
'subjectAltName=DNS:kafka,DNS:mq.example.com' \
'extendedKeyUsage=serverAuth' > broker.ext
openssl x509 -req -sha256 -days 825 \
-in broker.csr \
-CA ca.crt \
-CAkey ca.key \
-CAcreateserial \
-extfile broker.ext \
-out broker.crt
openssl pkcs12 -export \
-name kafka-broker \
-in broker.crt \
-inkey broker.key \
-certfile ca.crt \
-out /opt/kafka/secrets/kafka.keystore.p12 \
-passout "pass:$STORE_PASSWORD"
keytool -importcert -noprompt \
-alias kafka-ca \
-file ca.crt \
-keystore /opt/kafka/secrets/kafka.truststore.p12 \
-storetype PKCS12 \
-storepass "$STORE_PASSWORD"
Use the real DNS name both in the SAN and KAFKA_HOST. Do not mount or distribute ca.key; clients receive only the CA certificate/truststore. Track expiration.
Create the administrator properties used by the container's health check:
printf 'security.protocol=SASL_SSL\nsasl.mechanism=PLAIN\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="%s";\nssl.truststore.location=/etc/kafka/secrets/kafka.truststore.p12\nssl.truststore.type=PKCS12\nssl.truststore.password=%s\n' \
"$ADMIN_PASSWORD" "$STORE_PASSWORD" \
> /opt/kafka/secrets/client-admin.properties
chmod 600 /opt/kafka/.env /opt/kafka/secrets/*
sudo chown -R 1000:1000 /opt/kafka/data /opt/kafka/secrets
Check the selected image's actual UID/GID before applying the example ownership 1000:1000. Its non-root process needs access through the secrets directory and to each 600-mode file; do not fix permissions by making secrets world-readable.
Create /opt/kafka/compose.yml:
services:
kafka:
image: apache/kafka:4.3.1
container_name: kafka
hostname: kafka
restart: unless-stopped
ports:
- "${KAFKA_BIND_IP:-127.0.0.1}:9094:9094"
volumes:
- ./data:/var/lib/kafka/data
- ./secrets:/etc/kafka/secrets:ro
environment:
CLUSTER_ID: ${CLUSTER_ID}
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL
KAFKA_LISTENERS: CONTROLLER://:29093,INTERNAL://:19094,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19094,EXTERNAL://${KAFKA_HOST}:9094
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf
KAFKA_SSL_KEYSTORE_FILENAME: kafka.keystore.p12
KAFKA_SSL_KEYSTORE_TYPE: PKCS12
KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka_keystore_creds
KAFKA_SSL_KEY_CREDENTIALS: kafka_ssl_key_creds
KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.truststore.p12
KAFKA_SSL_TRUSTSTORE_TYPE: PKCS12
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kafka_truststore_creds
KAFKA_SSL_CLIENT_AUTH: none
KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
KAFKA_SUPER_USERS: User:admin
KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
KAFKA_LOG_DIRS: /var/lib/kafka/data
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_HEAP_OPTS: -Xms2g -Xmx2g
KAFKA_LOG_RETENTION_HOURS: 168
healthcheck:
test:
- CMD-SHELL
- /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server kafka:19094 --command-config /etc/kafka/secrets/client-admin.properties >/dev/null 2>&1
interval: 30s
timeout: 10s
retries: 10
start_period: 60s
mem_limit: 4g
cpus: 2.0
ulimits:
nofile:
soft: 100000
hard: 100000
logging:
driver: json-file
options:
max-size: 10m
max-file: "5"
The controller listener is internal. KAFKA_ADVERTISED_LISTENERS tells clients where to reconnect after bootstrap: incorrect DNS, hostname or port can cause failure after the first connection succeeds.
Do not start this definition alone. The official image's helper detects SSL from advertised listener names containing SSL; the custom INTERNAL/EXTERNAL names above do not trigger that shortcut. File-name variables alone therefore do not supply the broker's actual store paths/passwords. Also, a plaintext controller connection would use an anonymous principal while ACL enforcement is enabled.
Preserve the already generated store password in the restricted environment file, once:
cd /opt/kafka
printf 'KAFKA_TLS_STORE_PASSWORD=%s\n' "$(sudo cat secrets/kafka_keystore_creds)" >> .env
chmod 600 .env
Add compose.override.yml for a fresh supported-patch installation:
services:
kafka:
image: apache/kafka:4.3.2
environment:
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:SASL_SSL,INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL
KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN
KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka.keystore.p12
KAFKA_SSL_KEYSTORE_PASSWORD: ${KAFKA_TLS_STORE_PASSWORD:?store password required}
KAFKA_SSL_KEY_PASSWORD: ${KAFKA_TLS_STORE_PASSWORD:?store password required}
KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.truststore.p12
KAFKA_SSL_TRUSTSTORE_TYPE: PKCS12
KAFKA_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_TLS_STORE_PASSWORD:?store password required}
The controller uses the same admin identity defined in KafkaServer JAAS and trusts the certificate for kafka. Existing clusters must review version and listener migration compatibility before applying changes. For the 4GB development case, also override heap to -Xms1g -Xmx1g and lower the container limit; the 2GB heap example is intended for the larger host.
Validate quietly and then start:
cd /opt/kafka
docker compose config --quiet
docker compose pull
docker compose up -d
docker compose ps
docker compose logs --tail=100 kafka
Create the three-partition, one-replica orders topic:
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server kafka:19094 \
--command-config /etc/kafka/secrets/client-admin.properties \
--create \
--topic orders \
--partitions 3 \
--replication-factor 1
Grant the application topic permissions and its consumer-group access:
docker exec kafka /opt/kafka/bin/kafka-acls.sh \
--bootstrap-server kafka:19094 \
--command-config /etc/kafka/secrets/client-admin.properties \
--add --allow-principal User:app \
--operation Read --operation Write --operation Describe \
--topic orders
docker exec kafka /opt/kafka/bin/kafka-acls.sh \
--bootstrap-server kafka:19094 \
--command-config /etc/kafka/secrets/client-admin.properties \
--add --allow-principal User:app \
--operation Read \
--group orders-workers
Create an application properties file from client-admin.properties, replacing the user with app and password with APP_PASSWORD. Keep security.protocol=SASL_SSL, sasl.mechanism=PLAIN and CA trust. Host-side clients need a truststore path on their own host, not the container's /etc path. Never set ssl.endpoint.identification.algorithm= to disable hostname validation.
The following administrator test verifies basic broker delivery:
printf 'order.created id=1001\n' | docker exec -i kafka \
/opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server kafka:19094 \
--producer.config /etc/kafka/secrets/client-admin.properties \
--topic orders
docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server kafka:19094 \
--consumer.config /etc/kafka/secrets/client-admin.properties \
--topic orders \
--from-beginning \
--max-messages 1
Repeat with the application properties, and explicitly pass consumer group orders-workers; a randomly generated group does not match the granted ACL. Also test that the app is denied unrelated topics and administrative operations. The administrator's successful test alone does not prove app permissions.
Use acks=all in business producers. Understand idempotence, transaction permissions and message-key ordering, and grant only documented required privileges. A successful send does not mean the downstream business transaction has completed.
The example retains logs for 168 hours. Retention deletes history; it is not permanent archiving. Estimate raw storage as peak MB/s × 86400 × retained days × replicas:
峰值写入 MB/s × 86400 × 保留天数 × 副本数
Allow extra space for indexes, segments, replay peaks and delayed cleanup. Sustained 70%–80% disk use leaves little margin for compaction or slow consumers.
Match partition count to consumers and throughput rather than maximizing it. Partitions can be added, not directly removed; adding partitions can change key placement and affect assumptions about ordering.
Remote JMX is not exposed by default. Use a JMX Exporter Java agent and Prometheus on restricted private endpoints:
UnderReplicatedPartitionsandUnderMinIsrPartitionCount.- Active-controller state, controller queues and election failures.
- Request latency/errors and BytesIn/BytesOut.
- Maximum consumer-group lag and processing rate.
- GC, heap, process health and file descriptors.
- Disk capacity, inodes, I/O latency and network loss.
On one broker, UnderReplicatedPartitions=0 does not imply host availability. Combine Kafka and host alerts using Prometheus and Grafana.
Replication improves availability but propagates deletions and logical mistakes. For a single node, pause producers/consumers and take a stopped-node data snapshot:
cd /opt/kafka
docker compose stop kafka
sudo tar -C /opt/kafka -czf \
"backups/kafka-data-$(date +%F-%H%M%S).tar.gz" \
data
docker compose start kafka
Check tar succeeded; apply restricted ownership/permissions, encryption and off-host retention. The snapshot includes logs and KRaft metadata only when all configured log/metadata directories reside in the backed-up path; verify mounts if customized.
Back up the environment, Compose and override, JAAS, CA, keystores and client properties separately with encryption. Restore to an isolated stopped instance with the original Cluster ID, node configuration, metadata and ownership. Do not reformat populated storage. Start with compatible versions, check topics, retained events and consumer offsets, and test clients before rerouting production.
For low RPO/RTO, plan a separate cluster with MirrorMaker 2 topic replication and consumer-group checkpoints. Check filtering, checkpoint/offset synchronization and failover behavior; it is neither automatic zero-loss failover nor a replacement for independent archives.
Read release notes and check client, plugin, Kafka Connect and protocol compatibility. A single node requires maintenance and a cold backup:
cd /opt/kafka
docker compose config --quiet
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server kafka:19094 \
--command-config /etc/kafka/secrets/client-admin.properties \
--list
docker compose pull
docker compose up -d
docker compose ps
docker compose logs --tail=100 kafka
Change the pinned image/override before pulling; pulling the same tag is not a version transition. Avoid latest. Follow the supported metadata and KRaft feature-version sequence. After irreversible metadata changes, image downgrade may be unsupported; restore a compatible pre-upgrade snapshot instead.
Check advertised.listeners:
getent hosts mq.example.com
nc -vz mq.example.com 9094
Run DNS/routing checks from the client host. Verify VPN/public routes, cloud rules, Docker bindings and the port-access troubleshooting guide.
Inspect SAN and the chain:
openssl s_client -connect mq.example.com:9094 \
-servername mq.example.com \
-showcerts </dev/null
This command displays certificates; to validate them, supply the CA, -verify_hostname with the actual DNS name and -verify_return_error. Correct the certificate/DNS rather than disabling verification.
Use PLAIN and SASL_SSL. The JAAS account key illustrated by user_用户名 must become the actual user_app or user_admin entry. Reload broker configuration after editing broker_jaas.conf. Prefer staged new credentials and tested clients before removing old credentials.
Do not change CLUSTER_ID in .env when reusing data. Stop the node and compare meta.properties, configuration and backups. Reformatting a business-data directory destroys recovery information.
cd /opt/kafka
docker compose logs --tail=200 kafka
namei -l /opt/kafka/data /opt/kafka/secrets
sudo chown -R 1000:1000 /opt/kafka/data /opt/kafka/secrets
Confirm the image UID/GID before chown, and check explicit TLS store settings, parent permissions, free space and broker logs.
No. It uses KRaft; do not mix this setup with an old ZooKeeper Compose definition.
For development and light testing, with roughly 1GB heap and strict retention/disk monitoring. Small production is better planned around 8GB/NVMe and measured workload.
Only when downtime and recovery are acceptable. Critical deployments need independent broker failure domains and a properly designed controller quorum; combined mode is not recommended there.
Prefer VPN/private access, SASL_SSL, ACLs and source restrictions. Keep controller traffic outside client networks.
Use it only over TLS. Consider SCRAM, OAuthBearer or external authentication where appropriate.
No. Add independent backups, archiving or cross-cluster replication and rehearse restoration.