When your application is in production mode, you want to configure health checks to know if something goes wrong. Distributed systems can be hard to manage because there are many parts that all need to work for the system to function, and if any part breaks the system has to detect it, wrap around it and fix it. To detect and resolve problems with your system and application, you can run health checks. Health checks are a simple way to let your system know if an instance of your application is working or not. If an instance of your application is not working, other services shouldn’t access it or send requests to it, instead, the requests should be sent to another instance of the App that is ready or rescheduled at a later time. By default, Kubernetes will start to send traffic to a pod when all the containers inside the pod start and will restart containers when they crash. To know when there is a malfunction or an issue, you can make your deployments more robust by creating custom health checks.

Types of Health Checks:

  1. Readiness Probes: Readiness Probes are designed to let Kubernetes know when your application is ready to serve traffic.  Kubernetes will make sure that the readiness probe passes before allowing the service to send traffic to the pod. If a readiness probe starts to fail, Kubernetes will stop sending traffic to the pod until it passes again.

    Readiness Probes indicate whether your container is ready to serve requests. If the check fails, the container will not be restarted, but your Pod’s IP address will be removed from the service, so it will not serve any further requests. The readiness test will make sure that at startup, the pod will only receive traffic when the test succeeds.

    Using a Readiness Probe, Kubernetes will wait until your application is fully started before it allows the service to send traffic to the new copy.  

  2. Liveness Probes: Liveness Probes indicate whether your container is running. If the check fails your container will be restarted.  Liveness Probes lets Kubernetes know if your application is alive or dead. If your application is running or alive, Kubernetes will leave it alone. If your application is not working or is dead, Kubernetes will start a new one to replace it.

  • Running a command in your container periodically. You can execute a command in your container to check if the container is still working. For commands probes, Kubernetes will run a command inside your container. If the command returns with an exit code zero, your container will be marked healthy. Otherwise it will be marked unhealthy. The command probe is useful when you can’t or don’t want to run an HTTP server, but you can run a command to check if your application is healthy or not.

  • Periodic checks on a URL (HTTP): HTTP Probes are the most common type of custom probe. If your Application is not an HTTP Server, you can usually create a lightweight HTTP server inside your app to respond to the liveness probe. Kubernetes will ping a path if it gets an HTTP response in the 200 or 300 range, it will be marked as healthy otherwise it will be marked as unhealthy.

  • TCP Probe: Kubernetes will try to establish a TCP connection on your specified port. If it can establish a connection, the container is considered healthy, and if it can’t it’s, considered a failure.

DEMO

  1. This livenessProbe is going to check for our path `/` on the expressjs-port `5000` It has an initial delay of 15 seconds and timeout of 30 seconds.
apiVersion: apps/v1
kind: Deployment
metadata:
 name: healthcheck-deployment
spec:
 replicas: 3
 selector:
   matchLabels:
     app: express-app
 template:
   metadata:
     labels:
       app: express-app
   spec:
     containers:
     - name: express-app
       image: tolatemitope/express-app
       ports:
       - name: expressjs-port
         containerPort: 5000
       livenessProbe:
         httpGet:
           path: /
           port: expressjs-port
         initialDelaySeconds: 15
         timeoutSeconds: 30

2. Create your healthcheck deployment using kubectl create -f <path_to_your_healtcheck_file>

3. Get your pods using kubectl get pods

  • You can also get the status of any pod using kubectl describe pod <name_of_pod>

In the image above, you can see that there is a Liveness http-get  with a timeout of 30s and a period of 10s. The pod will consider it healthy when there’s 1 success and unhealthy when there are 3 failures.

4. If your application crashes for some reason, Kubernetes will terminate the pod and launch a new pod.  Do not route traffic to pods that are not healthy.

5. The readinessProbe is different from the livenessProbe because it will wait until the readinessProbe succeeds before it starts serving requests.

Create your liveness and readinessProbe using the kubectl create -f readiness-liveness.yaml command on your terminal.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: expressapp-readiness
spec:
 replicas: 3
 selector:
   matchLabels:
     app: express-app
 template:
   metadata:
     labels:
       app: express-app
   spec:
     containers:
     - name: express-app
       image: tolatemitope/express-app
       ports:
       - name: expressapp-port
         containerPort: 5000
       livenessProbe:
         httpGet:
           path: /
           port: expressapp-port
         initialDelaySeconds: 15
         timeoutSeconds: 30
       readinessProbe:
         httpGet:
           path: /
           port: expressapp-port
         initialDelaySeconds: 15
         timeoutSeconds: 30

  • You will see that your container is being created, but it’s not ready yet.

The initial delay of 15 seconds needs to pass and the check will be executed immediately. Only when the check is passed or executed, and the response is successful then your pod will be ready to serve the request.

  • You can see now that all our pods are ready and running. This is the difference between the liveness and readiness probe.

Pod State

Kubernetes doesn’t run containers directly, it wraps one or more containers into a higher-level structure called a pod. Any containers in the same pod will share the same resources and local network. Containers can easily communicate with other containers in the same pod. Pods are used as a unit of replication in Kubernetes. If you need your application to be scaled up you can increase the number of pods. Kubernetes can be automatically configured to scale up and down based on the load.  As a container that can have multiple processes, a pod can have multiple containers inside.  When provisioning your pods they should remain as small as possible to avoid unnecessary bills.

Pods have status fields which you can see when you input kubectl get pods in your terminal. In this scenario, all pods are in the running status. They are in the running state because they are bound to a Node.

There are other valid statuses like:

  • Succeeded: All your containers within your pod have been terminated successfully and will not be restarted.
  • Failed:  All your containers within your pod have been terminated, and at least one container returned a failure code. The failure code is the exit code of the process when a container terminates.  Exit code zero means okay, and other exit codes are a failure.

  • Pending: The pod has been accepted but is not running. This happens mostly when your container image is still downloading and if you have a slow network connection this can actually take some time. If your pod cannot be scheduled because of resource constraints, it’ll also be in this status.  If you don’t have enough CPU or memory anymore your pod can still remain in pending status.

  • Unknown: The state of your pod couldn’t be determined. A network error might have been occurred (like the node where your pod is running can be down or have a malfunction)
  • You can get the condition of your pod using kubectl describe pod <name_of_your_pod>

The pod conditions are conditions which the pod has passed.  In the image above the pod has passed the conditions of Initialized,  Ready, ContainersReady and PodScheduled.

Different types of PodConditions:

  1. PodScheduled: The pod has been scheduled to a Node.
  2. Ready: Pod can serve requests and be added to a service.
  3. Initialized: The initialization containers have been started successfully.
  4. Unschedulable: The pod can’t be scheduled due to constraints from your application. ContainersReady: All the containers in the pod are ready.

  • You can see your container statuses using kubectl get pod <name_of_pod> -n <namespace> -o yaml  You will be able to see your containerID, image, imageID and exitcode.

  • You can also get when exactly the container started running and its current state.

The container state can be running, terminated or waiting.

Enabling a Developer Control Plane on your Kubernetes Workloads

Check out how you can deliver your resources and engineering teams with an intuitive PaaS-like CI/CD workflow that will enable even the most junior developer on your team to deliver on DevOps premises.

If you have enjoyed reading this, be sure to check out our other blog posts.