I can’t remember the last time I drove down highway 101 between San Francisco and the South Bay and didn’t see a billboard claiming to be the single tool to solve all of my data problems. I know what most of these products do because I work in the industry and have spent hours defining the word, “data.” But, for the other hundreds of thousands of people who see these billboards every day, I always wonder if they know what the difference is and why someone would need a dozen tools to solve a single problem. The truth is that the term, “data,” is ambiguous. Today, I’m going to talk specifically about observability data and the concept of an observability data pipeline.
Observability represents the ability to see and understand the internal state of a system from its external outputs. Logs, metrics, and traces are known as the three pillars of observability, aka the external outputs. When we use the term, “observability data” these are the types of data that we’re referring to.
In order for companies to survive today, they must have access to this data. It allows them to maintain development velocity, understand and protect against security threats, and provide a positive customer experience. However, the majority of this data isn’t being leveraged. We see two main reasons for this:
I didn’t just break the internet with this information. Vendors realized this was an issue years ago and went on to create solutions, like the infamous single pane of glass, to solve it. Although these solutions bring all three types of observability data into a single tool, they aren’t optimized for use by all of the data consumers. In organizations that operate with a DevOps culture folks from development, operations, and security all need access to their observability data. In order to get to it, I’ve seen teams create manual workarounds that negatively impact operational efficiency. These inefficiencies simply don’t fly anymore because real-time insights can mean the difference between resolving an issue quickly or incurring millions of dollars in damages.
An observability data pipeline is a tool or process that centralizes observability data from multiple sources, enriches it, and sends it to a variety of destinations. This solves multiple problems, including:
This level of flexibility ensures that everyone can use their tools of choice and avoid costly vendor lock-in. The right tool can also put controls in place to manage spikes so that everyone in an organization has access to the data they need in real time, without impacting the budget.
For more than five years Mezmo, formerly LogDNA, has focused on building a modern log management tool for teams that embrace DevOps. Now, we’re working on a new pipeline product that allows organizations to centralize all of their log data from multiple sources; parse, normalize, and enhance it in Mezmo; and then stream it wherever they need—for example, to Mezmo Log Analysis for troubleshooting and debugging, to a SIEM for security, or to a data lake for compliance.
By shifting the control point left to the pipeline, existing Mezmo users can leverage many of their favorite features to unlock more value from their log data at scale. This is how we do it:
We started with log data because it’s the cornerstone of the DevOps culture. To date, we’ve helped thousands of developers and DevOps teams leverage their logs to build and maintain some of the world's most innovative products. Now, we’re helping them get even more value from their log data by streaming it to other destinations for a more unified development, security, and compliance practice.