In the dynamic and often complex environment of Kubernetes, observing and understanding the state of your cluster is essential. If you can't examine the condition of your running applications, effectively managing them becomes a challenge, if not impossible. This is where the concept of observability comes into play, and Kubernetes Events play a pivotal role in this context.

In this blog post, you'll learn what Kubernetes Events are and how to collect them. Let’s get started!

What are Kubernetes Events?

Kubernetes Events are resources that provide insights and visibility into the operational aspects of the cluster, such as why certain actions were taken or why they failed, serving as a useful tool for understanding the state and behavior of Kubernetes resources. 

Kubernetes Events are generated whenever there's a change in the state of cluster resources such as Nodes, Pods, Containers, and other Kubernetes objects like Deployments, Services, etc. 

It’s important to note that Kubernetes Events are temporary in nature and, by default, are not stored permanently. Instead, they are kept in "etcd", the Kubernetes key-value data store, and are retained for a short period (typically one hour). This helps in managing the storage and performance impact on the "etcd" database.

Types of Kubernetes Events

Kubernetes Events can broadly be categorized into two types:

• Normal Events: These events signify expected and routine operations in the cluster, like a Pod being scheduled or an image being successfully pulled.
• Warning Events: Warning events indicate issues that users need to address. These might include failed Pod scheduling, errors pulling an image, or problems with resource limits.

How to Collect Kubernetes Events

In Kubernetes, there are two primary commands for collecting event data: kubectl describe and kubectl get events.

kubectl describe

The kubectl describe <RESOURCE> command provides detailed information about a specified resource, including a section titled Events at the end. 

Below is an example of the Events section from the output of a kubectl describe <POD_NAME> command for a running nginx Pod:

While the kubectl describe command provides a detailed overview of a specific resource, including recent events, its event data can be somewhat limited. It does not offer options for deeper analysis. This is where the kubectl get events command, designed specifically for handling event data, comes into the picture.

kubectl get events

This command displays all events related to all objects in the current namespace. 

Below is a screenshot showing the output of the kubectl get events command for a running nginx Pod:

As you can see, the event feed provides several key pieces of information:

• LAST SEEN: Timestamp of when the event was last observed.
• TYPE: Indicates if the event is Normal or a Warning.
• REASON: A short string that describes the event.
• OBJECT: The Kubernetes object (like a Pod or Deployment) associated with the event.
• MESSAGE: A description of the event.

With kubectl get events, you can use additional flags to focus on exactly what you need. Next, we'll look at how to use these flags to better organize and understand your Kubernetes Events.

Filtering and Sorting Kubernetes Events

When working with Kubernetes Events, the volume of data can be overwhelming, especially in large clusters. Efficiently filtering and sorting these events is key to extracting meaningful insights. Here are some practical tips to help you manage this:

Get a detailed view of events

kubectl get events -o wide 

Add the -o wide flag to get a comprehensive view of each event, including additional details not visible in the standard output.

List events in a specific namespace

kubectl get events --namespace=<NAMESPACE_NAME>

Replace <NAMESPACE_NAME> with the actual namespace. This command filters events to show only those occurring in a specified namespace.

List events across all namespaces

kubectl get events -A

Add the -A flag to see events from all namespaces.

Stream live events

kubectl get events -w

Add the -w command to stream events in real-time. This is particularly useful for monitoring ongoing activities or troubleshooting live issues, as it updates continuously as new events occur. Use Ctrl+C to terminate the stream. 

Use field selectors for precise filtering

kubectl get events --field-selector type=<EVENT_TYPE>

Add the --field-selector flag to filter events based on specific field values. Replace <EVENT_TYPE> with the event type you want to filter by. For example, kubectl get events --field-selector type=Warning will only show events of type Warning. This is particularly useful for isolating events related to errors or critical issues.

Sort events by timestamp

kubectl get events --sort-by='.metadata.creationTimestamp'

Add the –sort-by flag to sort events chronologically. This is useful for tracking the sequence of events and understanding their progression over time.

Use JSON or YAML output for complex queries

kubectl get events -o json

kubectl get events -o yaml

For complex filtering that can't be achieved with kubectl flags, you can output the events in a structured format like JSON or YAML by adding the -o json and -o yaml flags, respectively. You can then use tools like jq (for JSON) to perform advanced queries and analyses.

Tools For Exporting Kubernetes Events

As we've previously discussed, Kubernetes Events are inherently temporary, stored for only a short duration in the "etcd" database. This poses a challenge when it comes to long-term monitoring and analysis. To address this, several external tools have been developed, enabling the export of Kubernetes Events to more persistent destinations for extended analysis and integration with monitoring systems.

Below, we explore some common tools that are widely used for exporting and monitoring Kubernetes Events:

• Eventrouter: Eventrouter is a tool that watches Kubernetes Events and forwards them to a specified destination, such as a logging platform. This helps in persisting Events beyond their default short lifespan, allowing for extended analysis and historical monitoring of the cluster's activities.
• kubernetes-event-exporter: This tool allows for the export of Kubernetes Events to multiple destinations with customizable routing and filtering. It's especially useful for transforming events into a more consumable format for various monitoring and alerting systems. 
• kspan: It’s a tool that converts Kubernetes Events into elements called "spans", which helps in tracing the sequence and connection between different events in your Kubernetes cluster. This makes it much simpler to see how events are related and to understand the overall flow and impact of what's happening in the cluster. 
• kubewatch: It monitors Kubernetes Events and notifies users of changes. It's well-suited for real-time alerting and integrates with various messaging platforms, keeping users informed about the operational state of the cluster. 

Each of these tools comes with its own unique set of features and capabilities, making them suitable for different needs and scenarios. Choose the tool that aligns best with your specific requirements and goals. 

Conclusion

In this blog post, we learnt what Kubernetes Events are, including their various types. We explored how to collect event data using two key commands: kubectl describe and kubectl get events, and discussed the use of different flags with kubectl get events for enhanced filtering and sorting. Lastly, we touched on several popular tools used to export Kubernetes events, allowing for persistent storage and long-term analysis and reporting.

Unlock the Power of Kubernetes Today! 

Dive into the world of hands-on learning with a FREE KodeKloud account. 

Interested in learning more about Kubernetes? Check out the following courses from KodeKloud:

• Kubernetes for the Absolute Beginner.
• Certified Kubernetes Administrator (CKA)
• Certified Kubernetes Application Developer (CKAD)