Our team’s learnings from Kubecon: Use Exemplars, Configuring OTel, and OTTL cookbook

A few weeks ago, members of Mezmo were at Kubecon and attended several sessions. You can see a post with my recap and session highlights. Today, though, I’m going to discuss three sessions that my colleagues found interesting for our peers in Observability. 

Unifying Observability: Correlating Metrics, Traces, and Logs with Exemplars and OpenTelemetry - Kruthika Prasanna Simha & Charlie Le, Apple

This talk with Kruthika Prasanna Simha and Charlie Le from Apple opens up with a case study - when you open an e-commerce website and a picture doesn’t load, what’s the root cause? At a high level, you can start troubleshooting by reviewing metric data, but that’s just telling you something is wrong, not specifically what, so the need to correlate with other data, like traces, becomes important. What if you could easily see your metric data map to a specific trace span? 

This is where exemplars come into play. If you’re not familiar with exemplars, it’s mapping specific traces or log data points together with metrics. Using exemplars makes it easier to correlate metrics with API calls, identify anomalies, or augment trace or log data with metrics. Just note one prerequisite of exemplars is you need to collect other types of telemetry data in order to make this work. The good news, OpenTelemetry collects these out of the box. 

So why would you want to map all these pieces of data together? First, it makes it much easier to do root cause analysis of an issue. Second, it gives more granular insights into anomalies. Finally, it reduces the mean time to respond/resolve. 

To conclude, despite the fact you can store these different data types separately, you would then have to manually manage the overhead of linking metrics with traces and logs. This makes it a one-stop shop for diagnosis instead of a three-stop shop. 

Mastering OpenTelemetry Collector Configuration - Steve Flanders, Cisco

If you’re not familiar with OpenTelemetry (OTel), this was the session to attend as it gave you a general overview of what OTel is, why it matters, and the flexibility it provides to collect metrics, traces, and logs. At a high level, OTel is a collector that exists to import and export data. The collector is not the final destination, but rather an intermediary to send between sources and destinations such as a container to an observability tool, etc. 

As we’ve discussed in our blogs and highlighted in this session, OTel is vendor-agnostic. This means you’re not tied to one vendor for instrumenting this data; you can send this data to other vendors now or in the future. 

Steve Flanders then walked through the setup, at a high level you need to:

1. Properly configure exporters
2. Connect to a pipeline 
3. OTLP with a YAML file
4. Once configured, can have YAML files send data via OpenTelemetry Transformation Language (OTTL)

The OTTL Cookbook: A Collection of Solutions to Common Problems - Tyler Helmuth, Honeycomb & Evan Bradley, Dynatrace

OpenTelmetry Transformation Language (OTTL) is a mouthful, so let’s start off with what it is and why they’re talking about it as a cookbook in this session. OTTL is a way to manipulate telemetry data within the OTel collector, it is a custom-built language specifically for the OpenTelemetry collector. This session took the time to cover some of the recipes available to help you solve common problems with telemetry data - transform and filter. 

The advantage of OTTL over other languages is that it can access any OpenTelmetry Protocol (OTLP) field and enables the expression of complex transformations using a simple syntax. 

Tyler and Evan covered multiple use cases and the recipes for processing such data, such as parsing unstructured log data, manipulating JSON, converting Prometheus metrics, etc. 

All in all - OTel is the winner in these sessions

OTel has gained market traction and, as we’ve seen with its various use cases and intricate details, has a lot to offer. Mezmo supports ingesting data from OTel collectors into our telemetry pipeline. At our Kubecon booth, many people loved the new Mezmo Flow experience that helps you get started using Mezmo Telemetry Pipeline. Give it a try today and see how we can help you manage your telemetry data. 

‍