From Gartner IOCS 2024 Conference: AI, Observability Data, and Telemetry Pipelines

Last week, I attended one of the last conferences of the year with team Mezmo: the Gartner IT Infrastructure, Operations & Cloud Strategies Conference in Las Vegas.

Not surprisingly, there were over 20 sessions covering observability and how it is getting increasingly critical in the new complex distributed computing environment. Of course, there were many sessions, including all keynotes that addressed the advent and impact of AI on IT operations and observability. Yes, there is tremendous interest, investigation, and optimism about AI and how it can be leveraged to streamline or automate IT operations, however, the teams are feeling overwhelmed.

The way forward is to keep learning and continuing conversations about what AI is and what it’s not and keep exploring how it can streamline and optimize observability operations.

With the increasing complexity of IT applications and infrastructure, the need for observability is growing and compound that with the adoption of AI, the volume of data collected is becoming overwhelming. The findings discussed at Gartner were in line with what we are seeing in the market. The data is growing exponentially and so is the spend to manage and store the data. For certain organizations, the data is growing 40% year over year.

And, most of this telemetry data is logs. Logs by nature are more voluminous and harder to manage. Organizations must take a systematic approach to manage the logs and that starts with understanding what you are collecting and then separating the signal from the noise. Mezmo’s data profiling helps you understand your log patterns so you can make an informed decision on what to send for analysis, what to store, and what to discard. Doing such an audit on your log data and applying optimization techniques using telemetry pipelines, Gartner suggests that you can reduce the volume of data by over 30%. At Mezmo we have seen similar or even better results where our customers are able to reduce the data volumes by over 40%.

‍

Telemetry pipelines are indispensable tools for tackling the data deluge. However, they are more than just filtering and moving the data. The Mezmo telemetry pipeline provides data understanding, as mentioned above, helps with data optimization (dedupe, sample, filter, throttle), and also helps detect data surges and anomalies while the data is in motion. This is a critical aspect of a modern data pipeline. 

‍

The pipeline must be responsive to the business and inform the users when a certain anomaly happens. Mezmo’s Chelsea Wright covered this aspect of a pipeline in her session “ Mezmo: Manage Anomalous Application Behavior Proactively with Telemetry Pipelines.” Beyond detecting, a responsive pipeline can also work in an incident mode where a trigger can automatically update the pipeline configuration or data routing to meet the data needs at that particular time.

To help SREs and developers tame telemetry data and create pipelines, Mezmo recently launched Mezmo Flow, which helps you create telemetry pipelines in just 15 minutes. If your teams are also facing the challenge of taming telemetry data, the first step is to get started with a Free version of the Mezmo Telemetry pipeline and start controlling your data and getting more value from it.