Trouble To Run a Replicated Stateful Application

hamza01 · September 23, 2024, 6:26pm

Hi everyone,

I’m following the official Kubernetes documentation for running a replicated stateful application: StatefulSets | Kubernetes. However, I’m encountering issues with my pods. They sometimes run successfully, but also crash occasionally.

I found a potential issue report on the documentation itself: website/content/en/examples/application/mysql/mysql-statefulset.yaml at main · kubernetes/website · GitHub

Would anyone be able to help me deploy a stateful application with a master-slave concept? Alternatively, could you point me towards any accurate documentation or alternative approaches?

Thanks in advance!

rob_kodekloud · September 23, 2024, 7:13pm

To help you here, we’ll need to see at least some of your YAML, and perhaps the errors from the logs of the crashed pods. There will be an error there somewhere; let’s find it.

Alistair_KodeKloud · September 24, 2024, 6:09pm

To be sure, the easier way to deploy something like that would be to use a helm chart. More configurable too.

hamza01 · September 26, 2024, 3:36pm

Thank you, Mr. Rob and Mr. Alistair, for your response. I apologize for providing the incorrect documentation link earlier. Below are the correct links:

Official Documentation: Run a Replicated Stateful Application | Kubernetes
GitHub Discussion: Problem following tutorial Run a Replicated Stateful Application · Issue #33223 · kubernetes/website · GitHub

hamza01 · September 26, 2024, 3:36pm

1- StatefulSet Script:

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: mariadb

spec:

serviceName: “mariadb-service”

replicas: 3

selector:

matchLabels:

  app: mariadb

template:

metadata:

  labels:

    app: mariadb

spec:

  securityContext:

    runAsUser: 0

  initContainers:

  - name: init-mariadb

    image: bitnami/mariadb:10.1

    command:

    - bash

    - "-c"

    - |

      set -ex

      # Generate mariadb server-id from pod ordinal index.

      [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1

      ordinal=${BASH_REMATCH[1]}

      echo [mysqld] > /mnt/conf.d/server-id.cnf

      # Add an offset to avoid reserved server-id=0 value.

      echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf

      # Copy appropriate conf.d files from config-map to emptyDir.

      if [[ $ordinal -eq 0 ]]; then

        cp /mnt/config-map/primary.cnf /mnt/conf.d/

      else

        cp /mnt/config-map/replica.cnf /mnt/conf.d/

      fi

    volumeMounts:

    - name: conf

      mountPath: /mnt/conf.d

    - name: config-map

      mountPath: /mnt/config-map

  - name: clone-mariadb

    image: gcr.io/google-samples/xtrabackup:1.0

    command:

    - bash

    - "-c"

    - |

      set -ex

      # Skip the clone if data already exists.

      [[ -d /bitnami/mariadb/data/mysql ]] && exit 0

      # Skip the clone on primary (ordinal index 0).

      [[ `hostname` =~ -([0-9]+)$ ]] || exit 1

      ordinal=${BASH_REMATCH[1]}

      [[ $ordinal -eq 0 ]] && exit 0

      echo "Starting to clone data from mariadb-$(($ordinal-1))..."

      ncat --recv-only mariadb-$(($ordinal-1)).mariadb-service 3307 | xbstream -x -C /bitnami/mariadb/data

      echo "Preparing the backup with xtrabackup..."

      xtrabackup --prepare --target-dir=/bitnami/mariadb/data;   

    volumeMounts:

    - name: mariadb-pvc-rec-canada

      mountPath: /bitnami/mariadb/data

    - name: conf

      mountPath: /etc/mysql/conf.d

  containers:

  - name: mariadb

    image: bitnami/mariadb:10.1

    env:

    - name: MARIADB_USER

      value: "myuser"

    - name: MARIADB_PASSWORD

      value: "newpassword"

    - name: MARIADB_DATABASE

      value: "mydatabase"

    - name: ALLOW_EMPTY_PASSWORD

      value: "yes"

    ports:

    - name: mariadb

      containerPort: 3306

    volumeMounts:

    - name: mariadb-pvc-rec-canada

      mountPath: /bitnami/mariadb/data

    - name: conf

      mountPath: /etc/mysql/conf.d

    resources:

      requests:

        cpu: 500m

        memory: 1Gi

    livenessProbe:

      exec:

        command: ["mysqladmin", "ping"]

      initialDelaySeconds: 30

      periodSeconds: 10

      timeoutSeconds: 5

    readinessProbe:

      exec:

        # Check we can execute queries over TCP (skip-networking is off).

        command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]

      initialDelaySeconds: 5

      periodSeconds: 2

      timeoutSeconds: 1   

  - name: xtrabackup

    image: gcr.io/google-samples/xtrabackup:1.0

    ports:

    - name: xtrabackup

      containerPort: 3307

    command:

    - bash

    - "-c"

    - |

      set -ex

      cd /bitnami/mariadb/data



      # Determine binlog position of cloned data, if any.

      if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then

        cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in

        rm -f xtrabackup_slave_info xtrabackup_binlog_info

      elif [[ -f xtrabackup_binlog_info ]]; then

        [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1

        rm -f xtrabackup_binlog_info xtrabackup_slave_info

        echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\

              MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in

      fi



      if [[ -f change_master_to.sql.in ]]; then

        echo "Waiting for mysqld to be ready (accepting connections)"

        until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 2; done



        echo "Initializing replication from clone position"

        mysql -h 127.0.0.1 \

              -e "$(<change_master_to.sql.in), \

                  MASTER_HOST='mariadb-0.mariadb-service', \

                  MASTER_USER='root', \

                  MASTER_PASSWORD='', \

                  MASTER_CONNECT_RETRY=10; \

                  START SLAVE;" || exit 1

        mv change_master_to.sql.in change_master_to.sql.orig

      fi



      exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \

        "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"

    volumeMounts:

    - name: mariadb-pvc-rec-canada

      mountPath: /bitnami/mariadb/data

    - name: conf

      mountPath: /etc/mysql/conf.d

    resources:

      requests:

        cpu: 300m

        memory: 500Mi

volumes:

name: conf

emptyDir: {}
name: config-map

configMap:

name: mariadb-config

volumeClaimTemplates:

metadata:

name: mariadb-pvc-rec-canada

spec:

accessModes: [ “ReadWriteOnce” ]

resources:
```
requests:

  storage: 15Gi
```

hamza01 · September 26, 2024, 3:38pm

2- ConfigMap Script:

apiVersion: v1

kind: ConfigMap

metadata:

name: mariadb-config

labels:

app: mariadb-config

data:

primary.cnf: |

# Apply this config only on the primary.

[mysqld]

log-bin

datadir=/bitnami/mariadb/data

replica.cnf: |

# Apply this config only on replicas.

[mysqld]

super-read-only

datadir=/bitnami/mariadb/data

3- I am using headless service by setting ClusterIP to None.

4- My PersistentVolumes (PVs) are working correctly, and I have configured them using a pv.yaml file. The PVs are correctly bound since I have 3 replicas for MariaDB.