Optimizing startup pod & readiness in K8s

Ensuring the reliability, efficiency, and readiness of applications is paramount. In this article I will explore key concepts and techniques in Kubernetes deployment, shedding light on essential functionalities such as liveness probes, readiness probes, startup probes, and more. From TCP sockets to gRPC integration, each aspect is crucial in optimizing the deployment process and enhancing the overall performance of distributed systems.

Image from Google Cloud

Liveness probe

A Liveness probe is used to determine if a container in a pod is alive and healthy. Kubernetes periodically checks the container’s health based on the probe configuration, and if the probe fails, Kubernetes restarts the container. Below is a code snippet of how it’s configured.

# Example of a liveness probe in Kubernetes Pod configuration
spec:
  containers:
  - name: my-container
    # Other container configurations
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10

livenessProbe: indicates that we are configuring a liveness probe for the container.

httpGet: Specifies that we are using an HTTP GET request to perform the probe. Kubernetes will send an HTTP GET request to the specified path and port.

path: Specifies the endpoint/path Kubernetes will access to check the container’s health. In this example, it’s set to /healthz.

port: Specifies the port on which the container listens for the HTTP requests. Here, it’s set to 8080.

initialDelaySeconds: Specifies the number of seconds Kubernetes should wait after the container starts before performing the first probe. In this case, it’s set to 15 seconds.

periodSeconds: Specifies the frequency at which Kubernetes should perform subsequent probes, in seconds. Here, it’s set to 10 seconds, meaning Kubernetes will perform the probe every 10 seconds after the initial delay.

Readiness probe

A readiness probe is used to determine if a container in a pod is ready to serve traffic. Kubernetes uses the readiness probe to decide whether to include the pod in the load-balancing pool. Below is a code snippet of how it’s configured.

readinessProbe:
  exec:
    command:
    - /usr/local/bin/checker
    - --full-check
    - --data-service
  initialDelaySeconds: 60
  timeoutSeconds: 5

You will notice that it's a similar configuration to our liveness probe except for two additional configs; command and timeoutSeconds. I will explain both sections below.

command: Specifies the command to be executed. In this case, /usr/local/bin/checker --full-check --data-service is the command that will be executed. This command likely checks various aspects of the container's readiness, possibly including connectivity to external services or the completion of initialization tasks.

timeoutSeconds: Specifies the number of seconds Kubernetes should wait for the command to complete. If the command is not complete within this timeout period, the probe will be considered failed. In this case, it’s set to 5 seconds.

This configuration sets up a readiness probe that executes a specific command inside the container after an initial delay of 60 seconds. If the command completes within 5 seconds and returns successfully, the container is considered ready to serve traffic. Readiness probe can be used to check TCP socket, exec, and gPRC service. I will explain them below.

TCP Socket probe

TCP socket probe is used to check if a container can establish a TCP connection to a specified port. It’s useful for checking if a service inside the container is accepting connections.

# Example of a TCP socket probe in Kubernetes Pod configuration
spec:
  containers:
  - name: my-container
    # Other container configurations
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 10

Exec Probe

An exec probe is used to execute a command inside a container and check the exit status of the command. It’s useful for performing custom health checks or readiness checks.

# Example of an exec probe in Kubernetes Pod configuration
spec:
  containers:
  - name: my-container
    # Other container configurations
    readinessProbe:
      exec:
        command:
          - "sh"
          - "-c"
          - "echo 'ready'"
      initialDelaySeconds: 10

gRPC Probe

A gRPC probe is used to check the health of a gRPC service running inside a container. It’s similar to an HTTP probe but tailored for gRPC endpoints.

# Example of a gRPC probe in Kubernetes Pod configuration
spec:
  containers:
  - name: my-container
    # Other container configurations
    readinessProbe:
      exec:
        command:
          - "grpc_health_probe"
          - "-addr=:8080"
      initialDelaySeconds: 10

Startup Probe

A startup probe is used to determine when a container in a pod has started successfully. Unlike liveness and readiness probes, a startup probe only runs during the initial startup of the container.

# Example of a startup probe in Kubernetes Pod configuration
spec:
  containers:
  - name: my-container
    # Other container configurations
    startupProbe:
      httpGet:
        path: /startup
        port: 8080
      initialDelaySeconds: 20

It is used to adopt liveness checks on slow-starting containers, avoiding them getting killed by the kubelet before they are up and running.

Init Containers

Init containers are used to perform initialization tasks before the main application container starts. They are often used for tasks such as downloading dependencies or setting up configurations.

# Example of init containers in Kubernetes Pod configuration
spec:
  initContainers:
  - name: init-container
    image: busybox
    command: ['sh', '-c', 'echo Initializing...']
  containers:
  - name: my-container
    # Other container configurations

Init containers are used for pod-level initialization purposes, like waiting for volumes to be ready.

Pod Readiness Gates

Pod readiness gates allow you to delay pod readiness until certain conditions are met, such as the readiness of a dependent service or external system.

# Example of pod readiness gates in Kubernetes Pod configuration
spec:
  readinessGates:
  - conditionType: MyCustomCondition

readinessGates: This field specifies a list of conditions that must be true for a pod to be considered ready. The conditions defined here are typically custom conditions that are specific to your application or environment.

conditionType: This field specifies the type of condition that must be satisfied. In this example, MyCustomCondition is the custom condition type that Kubernetes will check before considering the pod as ready.

The idea is to extend the concept of pod readiness to check additional conditions in addition to making sure all the containers are ready.

Should I use all these configurations?

This depends on various factors including the specific requirements of your application, the complexity of your deployment environment, and the resources available for managing and monitoring your application. Here are some considerations to help you decide:

1. Application Requirements: Consider what checks are necessary to ensure the reliability, availability, and performance of your application. For example, if your application relies heavily on HTTP endpoints, including readiness and liveness probes that perform HTTP checks may be essential.
2. Complexity and Maintainability: Including multiple types of probes and checks can increase the complexity of your deployment configuration. This can make the configuration harder to understand, maintain, and troubleshoot. Evaluate whether the benefits of including additional checks outweigh the added complexity.
3. Resource Utilization: Each probe or check consumes additional resources, such as CPU and network bandwidth, both within the containers themselves and on the Kubernetes cluster. Assess whether your infrastructure can handle the additional resource overhead introduced by including multiple checks.
4. Monitoring and Alerting: Ensure that you have appropriate monitoring and alerting systems in place to detect and respond to failures detected by these probes. Having comprehensive monitoring can help you quickly identify and resolve issues with your application.

Summary

While including multiple types of probes and checks can provide additional layers of assurance for your application’s health and readiness, it’s essential to consider the trade-offs in terms of complexity, resource utilization, and maintenance overhead. Start with the probes that are most critical for your application’s requirements and gradually add additional checks as needed, while ensuring that you have the necessary monitoring and alerting mechanisms in place to support them.