Managing Variable Log Retention

As systems become more complex and distributed, the total amount of machine data put off by those systems continues to skyrocket. While teams may need access to an ever-increasing scope of machine data to gain insights into their increasingly complicated systems, that same need to access an increasingly large amount of data also creates cost concerns. Those concerns can grow into cost emergencies quickly. Additionally, choosing a limit on the types of logs ingested into observability systems for the sake of cost comes with a risk of not being able to identify your issue fast enough, or even at all, attached. These factors create a bit of a problem when analyzing an increasingly large amount of log data meets cost concerns. 

Variable Retention and Log Data Restoration from Mezmo provides an interesting fix to that issue with two features that are a bit like peanut butter and jelly for your logging issues.

Not All Data is Needed All the Time

Depending on your situation, you may agree or disagree with that statement. That said, for the vast majority of machine data users, that statement will be true - you don’t need all your data all the time. Before we go any further with this blog, let’s talk about two types of storage solutions for your data and why you would want them. The first kind of storage is known as “hot storage.” Hot storage is extremely accessible storage with low capacity where you can retrieve data near instantly. The drawback to using hot storage is that it also has high costs (you’re getting what you’re paying for!). You should use hot storage for data crucial to your business - and in some cases, that includes logs. On the other side of the spectrum is cold storage. Cold storage is storage that is “cheap and deep.” It is typically kept far away from the warmth of a CPU (thus, the name cold storage), has massive capacity, and is used for a longer-term solution. Cold storage is not as accessible, but it’s also more cost-effective - kind of like that pile of papers you have sitting somewhere in your house that contains your mortgage, car loan paperwork, tax returns, and other things that you probably shouldn’t throw out. You can also use this method of storage for logs. 

The easiest way to think about hot vs. cold storage is like a checking account and an investment account. Checking accounts are for money that you might need right now. Your returns might be lower, but you can get your money whenever you need it. If you put your money in an investment account, your returns might be better, but it’s usually a longer-term solution where funds are less liquid. The same is generally valid for hot vs. cold storage options, and all the major cloud providers offer both hot and cold storage solutions.

‍

‍

Variable Retention

Now that we've had a primer on hot vs cold storage, let’s talk about Variable Retention. Variable Retention is a Mezmo feature that allows users to select the time window of the log data ingested into Mezmo’s UI instead of limiting the type of log data ingested into Mezmo’s UI. Mezmo allows users to select a retention window anywhere from 7 to 30 days, which we believe will cover most Mezmo customers’ needs. 

A key advantage to Mezmo’s Variable Retention Rules is that they allow users to keep their log data in hot storage for as long as they need it - up to 30 days! There’s an incredibly long list of the reasons someone would want to keep their logging data inside Mezmo’s UI for a long time. Maybe you’re dealing with compliance issues (Mezmo is HIPPA, ISO 2700:2013, SOC 2, GDP, and PCI compliant!), or you work in an environment where you’d like the ability to review log data for a longer time. 

Conversely, someone may want their retention window to be smaller, most notably to reduce spending. If a user does not need a larger retention window, why pay to keep their logs in hot storage, right? With Mezmo’s Variable Retention rules, you have a choice.

Log Data Restoration

Log Data Restoration creates a yin-yang relationship with Variable Retention to keep logging cost overruns in control. If you’re a skeptical person, you’ve probably noticed that the biggest problem with Mezmo’s Variable Retention is that all of your log data is stored in hot storage, creating cost overruns. On top of that, where does your log data go when the information is outside your retention window?  Log Data Restoration allows users to store their older log data in cold storage - like in an AWS S3 bucket or other means of Object storage - away from Mezmo and then recall the data later if or when they need it. You can read more about Log Data Restoration in this blog post on the topic here. Log Data Restoration from Mezmo allows users to control their log data, even if it is data from outside their variable retention window. Need log data outside of your window? Recall the data back into the Mezmo UI using Log Data Restoration. Log Data Restoration brings greater control over your log data while allowing you to save money by keeping that old log data in cold storage. 

Better Together

To make the most out of Variable Retention Rules from Mezmo, you will want to pair the feature with Log Data Restoration for a yin-yang effect on cost overruns caused by machine data. If you don’t already have access to these features in your Mezmo account or are new to Mezmo altogether, we encourage you to reach out to Mezmo at outreach@mezmo.com.

‍