In the last post I described a simple setup to manually run two instances of the same server to keep the website available even if one of them fails. Now I have tried the same with Kubernetes. For Kubernetes many of these things are provided out of the box, which makes the setup partially simpler but it seems to require more RAM on the nodes.
What we want is a webserver with a storage to save user uploaded files and a database:
But it should be highly available, i.e. distributed over multiple servers so that it continues to run even if one of them fails. With Kubernetes it will look like this:
Now let’s install this step by step:
Linux
We need three Linux nodes with Debian or Ubuntu. They should have 4GB RAM or more (it might work with 2GB RAM but then there won’t be much RAM left for your applications). And we need to install wireguard and open-iscsi:
apt-get update apt install -y wireguard open-iscsi systemctl enable iscsid systemctl start iscsid modprobe iscsi_tcp echo "iscsi_tcp" | sudo tee /etc/modules-load.d/iscsi_tcp.conf
K3s – a small Kubernetes version
Kubernetes can be installed with a single command. It can also automatically integrate Wireguard encryption between the nodes:
curl -sfL https://get.k3s.io | sh -s - server --cluster-init --flannel-backend=wireguard-native
After it has started, we need the IP and the token of this first node. The token can be retrieved using
cat /var/lib/rancher/k3s/server/node-token
Now we can start it on the other two nodes by running this command on node 2 and 3:
curl -sfL https://get.k3s.io | sh -s - server --server https://<your node1 IP>:6443 --flannel-backend=wireguard-native --token <your node1 token>
This way we have three nodes that can communicate with each other using an encrypted Wireguard connection.
Longhorn (Synchronized File System)
For different purposes we want to have the same files on all nodes. E.g. when a user uploads a file, it should be available on all three nodes. For this purpose we install Longhorn. Just run this on node 1, it will be installed on the others automatically:
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash helm repo add longhorn https://charts.longhorn.io helm repo update kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/deploy/longhorn.yaml
Traefik
K3s already contains Traefik, which can create Let’s encrypt certificates and route traffic to our webserver. When a certificate was created, it should be saved in a space where Longhorn can sync it with the other nodes, a persistent volume. First we need to create this persistent volume. Create a file traefik-pvc.yaml on node 1 with this content:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: traefik
namespace: kube-system
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
storageClassName: longhorn
Then run “kubectl apply -f traefik-pvc.yaml” to create this persistent volume.
Afterward we have to configure Traefik. Because it is already included in K3s, we have to edit the existing configuration using the command
kubectl edit deployment traefik -n kube-system
We have to make three changes:
args
In the “args” section we have to add some arguments to specify that we want to create Let’s encrypt certificates for our domains:
- --certificatesresolvers.default.acme.httpchallenge=true
- --certificatesresolvers.default.acme.httpchallenge.entrypoint=web
- --certificatesresolvers.default.acme.email=<your email address>
- --certificatesresolvers.default.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
- --certificatesresolvers.default.acme.storage=/data/acme.json
Then, above the “priorityClassName” section we insert a block that will set the file permissions of the acme.json in the persistent volume correctly so that Traefik can access it:
initContainers:
- command:
- sh
- -c
- touch /data/acme.json; chmod -v 600 /data/acme.json
image: busybox:latest
imagePullPolicy: Always
name: volume-permissions
resources: {}
securityContext:
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data
priorityClassName: system-cluster-critical
Then, in the existing “securityContext” section we add a “fsGroup” row to mount the persistent volume with a certain group id:
securityContext:
fsGroup: 65532
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
And finally, in the “volumes” section we change the volume “data” to use our new persistent volume:
volumes:
- name: data
persistentVolumeClaim:
claimName: traefik
- emptyDir: {}
name: tmp
After saving these changes, Kubernetes should automatically restart Traefik. You can see all existing “pods” using
kubectl get pods -A
If you want to restart Traefik, you can do this by just deleting it:
kubectl delete pod <traefik pod name> -n kube-system
Container Registry
To be able to install own containers, e.g. our PHP webserver, we need a container registry, i.e. a service where our nodes can load container images from when they need to install a service. Installing it is quite simple. First we create a separate namespace:
kubectl create namespace registry
Then we create a file registry-all.yaml with its configuration. At the end you have to replace “registry.yourdomain.com” with your real domain for the registry. I.e. you have to create entries in your DNS with the name “registry.yourdomain.com” and the IPs of all three nodes (i.e. three A entries).
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
namespace: registry
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: registry
spec:
replicas: 1
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
containers:
- name: registry
image: registry:2
ports:
- containerPort: 5000
volumeMounts:
- name: registry-storage
mountPath: /var/lib/registry
volumes:
- name: registry-storage
persistentVolumeClaim:
claimName: registry-pvc
---
apiVersion: v1
kind: Service
metadata:
name: registry-service
namespace: registry
spec:
type: NodePort
ports:
- port: 5000
targetPort: 5000
nodePort: 30500 # Choose a port between 30000-32767
selector:
app: registry
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: registry-ingress
namespace: registry
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.tls.certresolver: default
traefik.ingress.kubernetes.io/service.serverPort: "5000"
spec:
tls:
- hosts:
- registry.yourdomain.com
rules:
- host: registry.yourdomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: registry-service
port:
number: 5000
Then install it using
kubectl apply -f registry-all.yaml
That’s all. Now Traefik will create a Let’s encrypt certificate for it and you should be able to access it on https://registry.yourdomain.com . In the future you should add authentication to the registry. I will try to add documentation for it here.
Galera Cluster (Distributed MariaDB database)
Galera Cluster is a MariaDB (MySQL compatible) database where each node is a master, i.e. the nodes will sync with each other and you can do read and write operations on all of them. Thanks to an existing helm chart from Bitnami, it is easy to install. Just run this on node 1:
helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update kubectl create namespace galera
Then create a file galera-values.yaml:
## galera-values.yaml
## Global Docker image parameters
##
image:
registry: docker.io
repository: bitnami/mariadb-galera
tag: 10.11.4-debian-11-r0
pullPolicy: IfNotPresent
debug: false
## Kubernetes resource requests and limits
##
resources:
requests:
memory: 512Mi
cpu: 250m
limits:
memory: 1024Mi
cpu: 500m
## Persistence configuration
##
persistence:
enabled: true
storageClass: longhorn
accessModes:
- ReadWriteOnce
size: 8Gi
## MariaDB configuration
##
auth:
rootPassword: my-root-password
replicationUser: repl_user
replicationPassword: my-repl-password
## Service configuration
##
service:
type: ClusterIP
## Number of replicas
##
replicaCount: 3
## Galera configuration
##
galera:
clusterBootstrap: true
And run this afterward:
mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $(id -u):$(id -g) ~/.kube/config helm install my-galera bitnami/mariadb-galera \ --namespace galera \ --values galera-values.yaml
Now Galera should run on all three nodes. You can check it using
kubectl get pods -A -o wide
It will also use a Longhorn persistent volume to make re-syncing a new node faster, because then this node will already have the files of the previous failed node.
To get the automatically created root password for the database, you can use this command:
kubectl get secret --namespace galera my-galera-mariadb-galera -o jsonpath="{.data.mariadb-root-password}" | base64 --decode
To connect to the database and e.g. create databases and tables, you can use this:
kubectl exec -it my-galera-mariadb-galera-1 -n galera -- bash mysql -u root -p
Now you can create an example database that we can use for our PHP example program later:
CREATE DATABASE myappdb; CREATE USER 'myappuser'@'%' IDENTIFIED BY 'myapppassword'; GRANT ALL PRIVILEGES ON myappdb.* TO 'myappuser'@'%'; FLUSH PRIVILEGES; EXIT;
PHP Webserver
Now we have everything to run our PHP webserver. Here is a simple example PHP project that we can deploy. Create a directory “php-app” and save a file index.php into it with this content:
<?php
$servername = getenv('DB_HOST');
$username = getenv('DB_USER');
$password = getenv('DB_PASSWORD');
$dbname = getenv('DB_NAME');
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
echo "Connected successfully to Galera Cluster";
// Close connection
$conn->close();
?>
And create a Dockerfile in the same directory:
FROM php:8.1-apache RUN docker-php-ext-install mysqli COPY . /var/www/html/ RUN chown -R www-data:www-data /var/www/html
Now we have to create a container image from it and upload it to our container registry:
apt-get install docker docker.io docker build -t registry.yourdomain.com/php-app:latest . docker push registry.yourdomain.com/php-app:latest
Next we have to give the database credentials to our PHP script by creating a Kubernetes secret:
kubectl create secret generic galera-credentials \ --from-literal=DB_HOST=my-galera-mariadb-galera.galera.svc.cluster.local \ --from-literal=DB_USER=root \ --from-literal=DB_PASSWORD=<your password> \ --from-literal=DB_NAME=myappdb
Now it is time to set www.yourserver.com to the IPs of all three nodes (i.e. three A entries). Add it in your DNS server.
Then create a file php-all.yaml with this content and using that secret:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: php-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-deployment
spec:
replicas: 1
selector:
matchLabels:
app: php-app
template:
metadata:
labels:
app: php-app
spec:
containers:
- name: php-container
image: registry.yourdomain.com/php-app:latest
ports:
- containerPort: 80
envFrom:
- secretRef:
name: galera-credentials
volumeMounts:
- name: php-storage
mountPath: /var/www/html/uploads
volumes:
- name: php-storage
persistentVolumeClaim:
claimName: php-pvc
---
apiVersion: v1
kind: Service
metadata:
name: php-service
spec:
selector:
app: php-app
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: php-ingress
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.tls.certresolver: default
spec:
tls:
- hosts:
- www.yourdomain.com
rules:
- host: www.yourdomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: php-service
port:
number: 80
and install it using
kubectl apply -f php-all.yaml
That’s all. Now you should have a working 3 node Kubernetes cluster that will continue to work even if one node fails. And when the node comes back it should automatically be re-integrated. You can access it using
$ curl https://www.yourdomain.com Connected successfully to Galera Cluster