kunmidevOpstories

How to Achieve Real Zero-Downtime in Kubernetes Rolling Deployments: Avoiding broken client connections

Tokede Akinkunmi — Fri, 16 Feb 2024 15:10:55 GMT

The popular idiom nothing is constant except change was from a Greek philosopher named Heraclitus. Although Heraclitus lived around 500 BCE, this quote remains valid. Thanks to super-efficient orchestration tools like Kubernetes, making changes to our applications has become more seamless.

In software engineering, we make changes almost every day, but how do we avoid these changes from impacting users negatively? One of the negative impacts on users is through broken connections. I would have loved to discuss the impact of broken client connections, but not in this article.

By default, the Kubernetes deployment strategy involves rolling deployments. Yes! Rolling deployment sounds very interesting, but there is more to that. We need to ask ourselves some questions. What happens during rolling deployments?

Rolling deployments means incrementally replacing the current pods with new ones. During this process, there is always downtime spanning from microseconds to seconds. It might be insignificant for applications with a low user base. But it is significant for big applications, especially payment gateways, where every second counts.

Note: There are other ways to achieve zero downtime while deploying to production in Kubernetes, such as utilising a service mesh like Istio or implementing a blue-green deployment. These options consume more resources than rolling deployments, leading to an increase in infrastructural costs.

The question, What happens during rolling deployments? can be broken into two.

First, what happens when a pod starts up, and what happens when a pod shuts down?

Before we continue, here are the pre-requisites for this tutorial:

Kubernetes knowledge
Experience using Docker

The startup stage of a pod

When a pod starts in a rolling deployment without the readiness probe configured, the endpoint controller updates the corresponding service objects with the pods endpoints, which means the pod starts receiving traffic even though the pod is not ready. The absence of a readiness probe makes the application unstable.

Having a readiness probe set is recommended for applications. The implication is that it only receives traffic when it is ready; the endpoint controller continues to monitor the pods based on the pods readiness probe result. When the probe is successful, the endpoints are updated on the service objects to receive traffic.

Below is an example of a Kubernetes deployment file with a readiness probe configured:

apiVersion: apps/v1kind: Deploymentmetadata:  name: my-highly-available-appspec:  replicas: 3 # Define desired number of replicas  selector:    matchLabels:      app: highly-available  template:    metadata:      labels:        app: highly-available    spec:      containers:      - name: highly-available        image: highly-available-image:latest        ports:        - containerPort: 80        readinessProbe: # Readiness probe configuration          httpGet:            path: /health-check            port: 80            initialDelaySeconds: 5 # Wait 5 seconds before starting probes            periodSeconds: 10 # Check every 10 seconds            failureThreshold: 3 # Mark unhealthy after 3 consecutive failures

We have established what happens during the startup stage of a pod; it is time to analyse what happens in the shutdown stage.

The shutdown stage of a pod

It is crucial to understand that components in a Kubernetes cluster are more like micro-services and not monolithic. The way micro-services work is different from how monolithic processes run. In microservices, it takes more time for all the components to sync.

When the API server receives a notification of a pod deletion from a client or during rolling deployment, it first modifies the state of the pod in etcd, then notifies the endpoint controller and the Kubelet. Upon receiving the pod deletion notification from the API server, the endpoint controller removes the pod endpoint from every Service with which the pod is associated.

The endpoint controller on the control plane does this by sending a REST API to the API server. The API server then notifies its watchers, of which Kube-proxies are one; the Kube-proxies update the iptables rules to reflect the changes in the set of endpoints associated with that Service. Updating the iptables rules will prevent new traffic from being directed to the terminating pods.

The scenario described above is where the downtime occurs because it takes more time to update the iptables rules than for the containers to be terminated by the Kubelet. These phases happen concurrently. When a request to delete a pod is received from a client or during a rolling deployment, this request reaches the API server on the control plane. The Kubelet and the endpoint controller watch the API server for changes once the Kubelet and the endpoint controller receive a deletion notification. The Kubelet immediately sends a SIGTERM signal to the container, and the endpoints controller sends a request back to the API server for the pod endpoints to be removed from all service objects, a task performed by Kube-proxies on the worker nodes.

The reason for this downtime is that the containers would have been terminated by Kubelet (which is a shorter process and therefore takes less time) before the pod endpoints are updated on the corresponding Service (which involves more processes and thus takes more time). Due to the difference in task completion time, Services still route traffic to the endpoints of the terminating pods, leading to messages like connection error or connection refused.

The diagram below provides a pictorial view of what happens internally within the Kubernetes architecture.

We have been able to establish reasons for broken connections during rolling deployments; how then do we solve this issue?

The Solution

Kubernetes was never designed as a plug and play orchestration tool; it requires proper configuration to suit every use case accordingly. Since we have discovered that the difference in task completion time is the main issue, the simple solution is to define a wait time for the proxies to update the iptables.

We can achieve this by adding a preStop hook to the deployment configuration. Before the container shuts down completely, we will configure the container to wait for 20 seconds. It is a synchronous action, which means the container will only shut down when this wait time is complete. By then, the Kube-proxies would have updated the iptables, and new connections would be routed to running pods instead of terminating pods.

Note: The preStop hook is a mechanism used in pod lifecycle management to execute a specific command or action before a pod terminates

It is crucial to understand that when the iptables are updated, connections to the old pods (pods that are terminating) are still maintained, and client connections are not broken until all processes are complete and the pod shuts down gracefully, but new connections are directed to stable pods.

Below is an example of a deployment file with the preStop hook configured:

apiVersion: apps/v1kind: Deploymentmetadata:  name: my-highly-available-appspec:  replicas: 3 # Define desired number of replicas  selector:    matchLabels:      app: highly-available  template:    metadata:      labels:        app: highly-available    spec:      containers:      - name: highly-available        image: highly-available-image:latest        ports:        - containerPort: 80        readinessProbe: # Readiness probe configuration          httpGet:            path: /health-check            port: 80            initialDelaySeconds: 5 # Wait 5 seconds before starting probes            periodSeconds: 10 # Check every 10 seconds            failureThreshold: 3 # Mark unhealthy after 3 consecutive failures        lifecycle:          preStop:            exec:              command: ["/bin/bash", "-c", "sleep 20"]

With the configuration above, rolling deployments will no longer cause downtime on our infrastructure.

Finally, we should always ensure that the sleep time given is less than terminationGracePeriodSeconds, which is 30 seconds by default. A higher value will only cause the container to shut down forcefully.

Conclusion

To sum up, we have made significant progress in ensuring stable user connections during rolling deployments, regardless of the number of deployment versions released daily. We have modified our deployment file to include a readiness probe and a pre-stop hook. These changes enable us to manage traffic during pod startup and shutdown more effectively.

It has indeed been a productive time with you. Happy automating!

]]>

HOW TO RUN DOCKER IN DOCKER [Leveraging on Unix Sockets]

Tokede Akinkunmi — Wed, 02 Aug 2023 23:00:00 GMT

In a world where everyone seeks convenience, complications can arise when attempting to simplify processes. However, its interesting to discover that providing solutions to these complications can make the whole process a lot easier and more exciting.

Containerization represents a significant improvement over the traditional way of running applications. The advantages of containerization cannot be overstated, from its ability to run applications in any environment to efficient versioning through multiple available registries and more.

In this tutorial, we will focus on a specific use case that commonly appears in production environments.

Heres the scenario: we have pulled a Jenkins image from the official Docker registry, and its running in a container on port 8080. Jenkins will be utilized for continuous integration and deployment of our applications. As mentioned earlier, we have decided to containerize our applications, so in the Jenkins pipeline, we will need to push each version of our application to AWS ECR (Elastic Container Registry).

In the given scenario, the Jenkins agent running in Docker needs to execute Docker commands to build the Docker image and push it to AWS ECR. This requires having Docker present in that container. So, how can we achieve this?

Prerequisites

Ubuntu 20.04 AMI on AWS
Root SSH access or sudo privileges
Knowledgeable about Jenkins and Docker

Step 1: Install Docker on the Host Machine

We shouldnt forget that the host machine is where the Jenkins container will be running. The following commands will help you with the installation of Docker on Ubuntu.

Lets update the package index and install all necessary dependencies. Run:

sudo apt-get updatesudo apt install apt-transport-https ca-certificates curl software-properties-common

Next, add Dockers GPG key and repository to your host machine

echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/docker.list

Note: The command above is specifically for Ubuntu 20.04

Finally, install the Docker engine. Run:

sudo apt-get updatesudo apt-get install docker-ce

We will need to enable and start the Docker service. Run:

sudo systemctl start dockersudo systemctl enable docker

Yes! We have Docker running on our host machine. Lets verify it by running:

docker --version

A result similar to this should display:

ubuntu@ip-45-67-23:$ docker --versionDocker version 24.0.4, build 3713ee1

The next step is quite optional, but its recommended. If you wish to run Docker commands on your host machine without preceding those commands with sudo every time, run:

sudo usermod -aG docker $USER

Step 2: Customize the official Jenkins Image

One of the greatest advantages of containerization is the ability to pull an image from a Docker repository to be used directly or as a base image for further customization to suit a specific use case. For the use case stated above, this is how our Dockerfile will look like:

FROM jenkins/jenkins:latestUSER root# Install sudoRUN apt-get update && \    apt-get install -y sudo# Install dependenciesRUN apt-get update && \    apt-get install -y unzip && \    apt-get install -y python3-pip && \    pip3 install --upgrade pip# Install AWS CLIRUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \Unzip awscliv2.zip and \    ./aws/installUSER jenkins

The above Dockerfile indicates that we pulled the official Jenkins image and then switched the user to root just to install all required dependencies, after which we switched back to the Jenkins user to ensure Jenkins ran with non-root privileges for security purposes.

To build the Jenkins image from the above Dockerfile, run:

sudo docker build -t test_jenkins:latest <path to dockerfile>

The above Docker command builds a customized Jenkins image with test_jenkins as its name and tags it as latest. It is important for us to indicate the path to our Dockerfile.

To verify if we have successfully built our Jenkins image, lets run:

sudo docker images

After running the command above, you should see a result similar to this in your shell:

ubuntu@ip-174-42-12-23:$ sudo docker imagesREPOSITORY        TAG          IMAGE ID       CREATED        SIZEtest_jenkins     latest       63f37aa302   2 minutes ago   1.2GB

The output above shows the name of the image, the tag, the image ID, which is also very useful, the creation time, and the size. The output gives a summary of the image built.

Note: Docker images take up a lot of space on the host machine over time after building several images. In a production environment, it is advisable to set up a cron job to help clean up the unused Docker images at certain intervals.

Step 3: Perform the Magic

Here is the most critical part. We are about to perform magic! It is not magic per se, but just a smart move. Let us take a little dive into how Docker works.
When we install Docker on our host machine, in our case, Ubuntu AMI is our host machine. It creates a Unix socket in this location: /var/run/docker.sock . This Unix socket is used for communication between the Docker client and the Docker daemon.

Note: The Unix socket is a special file-based communication mechanism in Unix-like operating systems.

Okay, lets break it down: the Docker client allows us to interact with Docker using docker commands, while the Docker daemon listens to requests from the Docker client and manages Docker objects such as images, containers, volumes, and so on. What facilitates communication between the Docker client and the Docker daemon is the Unix socket.

Below is a diagram from Docker's official documentation on Docker architecture:

From the explanation given above, we now know that /var/run/docker.sock is the most important location for docker communication. Therefore, we will be mapping it as a volume to the container. This implies that when we run a docker command in the container, the container has access to the socket /var/run/docker.sock, which will help pass the request to the Docker daemon. Remember that the Docker daemon manages Docker objects such as images, volumes, and so on.

We will be using the docker run command to start our container and map the necessary volumes. Lets run:

sudo docker run -d --name jenkins -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock -v jenkins_data:/var/jenkins_home test_jenkins:latest

It is important to note that in the command above, I created an extra Docker volume called jenkins_data mapped to the Jenkins_home directory for data persistence in case the container dies and we need to run a new one. External volume mapping is a good example of best practices.

To confirm if our Docker container is running or not, lets run:

sudo docker ps

This should return an output similar to this:

ubuntu@ip-176-45-67-23:$ sudo docker psCONTAINER ID   IMAGE                  COMMAND                 CREATED       STATUS       PORTS                                                                                      NAMES9ed41r31067w test_jenkins:latest "/usr/bin/tini -- /u..." 1 minute ago   1 minute ago 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp,   jenkins

Well done if you can see an output similar to the one shown above. Finally, to ensure this works perfectly, let us make this location executable: /var/run/docker.sock. The purpose of this tutorial is not only to provide a solution but also to educate you on how it works, as most articles out there wont explain the underlying principles. To achieve this, on the host machine, run:

chmod +x /var/run/docker.sock

The command above allows us to run docker commands from the container without any errors. We can therefore proceed with writing our pipeline scripts to push to AWS ECR (Elastic Container Registry).

Congratulations! You now have a clear understanding of how to achieve Docker in Docker (DinD) functionality. By mapping the Docker daemons Unix socket into a Docker container using the -v /var/run/docker.sock:/var/run/docker.sock option, you enable the container's Docker client to interact with the Docker daemon running on the host machine. This allows you to manage sibling containers and run Docker commands within the container itself. Happy containerizing!

]]>