Unifying Logging Domains With Korrel8r Simplifying Log Access For Enhanced Cluster Observability

Hey guys! Today, we're diving deep into an exciting proposal to simplify how we access logs within our clusters using Korrel8r. As it stands, we're juggling three separate domains for cluster logs, which can be a bit of a headache. Let's break down the current situation, the problems it presents, and how we can make things smoother and more efficient.

The Current Log Landscape: A Triple Threat

Currently, we have three distinct domains for handling cluster logs:

• log: This domain uses a Loki store and assumes the Viaq log format. Think of it as our original setup, designed to work with a specific log structure.
• otellog: This domain also uses a Loki store but is built to handle logs in the OTEL format. OTEL, or OpenTelemetry, is becoming a standard for observability, so this domain is crucial for modern log management.
• podlog: This domain takes a different approach, directly accessing current Pod logs via the API server. It's a direct line to real-time logs from our pods.

The Problem with Multiple Log Domains

Having these three domains might seem like a good way to organize things, but it leads to several issues that we need to address. Let's get into details about these issues.

First off, the log and otellog domains often end up reading the same data. This is a major efficiency killer. Even if one domain can't properly interpret the data, it still tries, leading to wasted effort. Imagine trying to read a book in a language you don't understand – frustrating, right? This wastes resources and slows down the entire process. By unifying the domains, we ensure that only the appropriate processing is applied, saving valuable compute time and reducing unnecessary load on our systems. This streamlined approach not only conserves resources but also enhances the overall responsiveness of our logging infrastructure, leading to faster insights and quicker resolution of issues.

Secondly, podlog only sees a subset of the logs stored in Loki, which can lead to duplicate results. This means we might get the same log entries from both podlog and either log or otellog*. It's like hearing the same story from two different people – redundant and confusing. Moreover, this inconsistency in log visibility can hinder troubleshooting efforts, as engineers might miss crucial information or waste time sifting through duplicate entries. A unified domain, by contrast, provides a single, comprehensive view of all log data, ensuring that no log entry is overlooked. This holistic approach enhances the accuracy of diagnostics and accelerates the process of identifying and resolving issues, ultimately improving the reliability and performance of our systems.

Thirdly, the current setup results in duplicate and unnecessary calls for log data. This inefficiency slows things down and wastes resources. It's like ordering the same meal twice – unnecessary and wasteful. By consolidating our log domains, we can significantly reduce the overhead associated with duplicate requests, allowing us to make the most efficient use of our resources. This optimization not only saves computational power and storage but also reduces network congestion and latency, resulting in a more responsive and efficient logging infrastructure. This streamlined approach ensures that our systems operate at peak performance, providing timely insights without unnecessary strain.

Finally, having multiple log domains is just plain confusing. It makes it harder to understand where to look for logs and how to query them. It’s like having three different dictionaries – you're never quite sure which one has the definition you need. This confusion can lead to delays in troubleshooting, increased training time for new team members, and an overall less efficient workflow. By unifying our log domains, we create a single, intuitive interface for accessing all log data, reducing the cognitive load on engineers and simplifying the overall log management process. This enhanced clarity and ease of use empower teams to quickly find and analyze the information they need, improving their productivity and responsiveness.

The Proposal: One Log Domain to Rule Them All

To tackle these issues, the proposal suggests combining all three domains into a single, unified "log" domain. This new domain would be a one-stop-shop for all our log needs, making things simpler and more efficient.

Key Components of the Unified Log Domain

This new domain would have a few key components:

• Multiple Store Types: The domain would support different types of log stores, including API server access and Loki. This allows us to pull logs from various sources in a unified way. The ability to integrate diverse log stores into a single domain enhances the flexibility and scalability of our logging infrastructure, allowing us to adapt to evolving needs and incorporate new technologies seamlessly. This versatile approach ensures that our systems remain agile and responsive, providing comprehensive log management capabilities across different environments and platforms.

• (Question: Multiple Korrel8r Stores or a Single "Compound" Store?): This is a key design question we need to answer. Should we use multiple Korrel8r stores, or a single "compound" store that can handle different types? Let’s explore the implications and benefits of each approach to make an informed decision. By carefully considering the trade-offs and aligning our architecture with our specific requirements, we can build a robust and efficient logging solution that meets our needs today and scales for the future. This proactive and thoughtful approach ensures that our logging infrastructure is well-designed, maintainable, and capable of handling the demands of our dynamic environment.

• Multiple Log Classes: The domain would support different classes representing distinct log schemas. This includes the existing Viaq formats (infrastructure, application, audit), OTEL format (with potential subtypes), and API server logs presented as OTEL. By supporting multiple log classes, the unified domain can seamlessly handle logs from various sources and formats, providing a comprehensive view of system activity. This versatility ensures that we capture all relevant data, regardless of its origin or structure, enabling thorough analysis and effective troubleshooting. The ability to differentiate log schemas also enhances the accuracy and efficiency of log processing, allowing us to apply specific rules and transformations based on the log type. This tailored approach optimizes resource utilization and ensures that our logging infrastructure remains scalable and responsive.

  • Viaq Log Classes: We'd keep the existing viaq.infrastructure, viaq.application, and viaq.audit classes for logs in the Viaq format. These classes are already well-defined and understood, so maintaining them ensures backward compatibility and minimal disruption. This stability is crucial for maintaining the integrity of our existing workflows and ensuring that our team can continue to rely on familiar tools and processes. By leveraging our existing knowledge and infrastructure, we can focus on enhancing our logging capabilities without introducing unnecessary complexity or risk.
  • OTEL Log Class: We'd introduce an otel class for logs in the OTEL format. But should we have subtypes for infra/app/audit? This is something we need to consider. OTEL, or OpenTelemetry, is designed to provide a unified standard for telemetry data, but it's still important to map OTEL logs to specific categories for effective analysis and management. Subtypes within the otel class could help us further categorize logs based on their source and content, enhancing the granularity of our analysis. This structured approach allows us to apply specific rules and transformations to different types of OTEL logs, optimizing resource utilization and ensuring that our logging infrastructure remains scalable and responsive. By carefully considering the need for subtypes, we can strike a balance between flexibility and simplicity, creating a logging solution that meets our current needs and adapts to future requirements.
  • API Server Logs as OTEL: API server logs would be presented in the OTEL format. This consistency makes it easier to process and analyze them alongside other logs. Standardizing log formats across different sources simplifies our log management processes and enhances our ability to correlate events and identify issues. By adopting OTEL as a common format, we can leverage a wide range of tools and techniques for log analysis, ensuring that our logging infrastructure remains state-of-the-art and capable of handling the demands of our dynamic environment.

Prioritizing Log Lookups and Avoiding Duplicates

With a single domain controlling all the log stores and formats, we can prioritize lookups and avoid duplicate results. The domain should be smart enough to auto-detect the Loki log format, preferring OTEL if possible. This intelligent approach ensures that we always use the most efficient method for accessing log data, optimizing resource utilization and minimizing latency. By prioritizing log lookups, we can quickly retrieve the information we need, even in high-load environments. This responsiveness is critical for effective troubleshooting and proactive management of our systems.

Benefits of the Unified Log Domain

The unified log domain will offer several key benefits:

• Simplified Access: A single entry point for all logs makes it easier to find what you need. This streamlined access reduces the cognitive load on engineers and accelerates the process of identifying and resolving issues. By consolidating log access into a single domain, we eliminate the confusion and complexity of navigating multiple systems, empowering teams to quickly find and analyze the information they need.
• Reduced Duplication: Avoiding duplicate lookups and results saves resources and improves efficiency. This optimization not only saves computational power and storage but also reduces network congestion and latency, resulting in a more responsive and efficient logging infrastructure. By eliminating duplicate calls for log data, we ensure that our systems operate at peak performance, providing timely insights without unnecessary strain.
• Improved Consistency: Standardizing log formats and access methods leads to more consistent and reliable results. This consistency is crucial for effective analysis and accurate diagnostics. By adopting a unified approach to log management, we can ensure that our data is reliable and our insights are trustworthy, empowering us to make informed decisions and proactively address potential issues.
• Better Performance: Prioritizing OTEL format and avoiding unnecessary calls improves overall performance. This enhanced performance translates to faster troubleshooting, quicker response times, and a more efficient use of resources. By optimizing our log management processes, we can minimize delays and ensure that our systems remain responsive and reliable.

Conclusion: A Brighter, More Log-Friendly Future

Combining our logging domains into a single, unified "log" domain is a smart move that will simplify our log access, reduce duplication, improve consistency, and boost performance. It's a significant step towards a more efficient and user-friendly logging experience with Korrel8r. Let's work together to make this proposal a reality!

This proposal depends on https://github.com/korrel8r/korrel8r/issues/76, so be sure to check that out for more context.