How to Prevent Log Data Loss When Using Elasticsearch In Production

One of the biggest challenges in infrastructure logging is ensuring that resources for storing and processing events are sufficient for scaling. Most businesses grow throughout the years, and as they grow, they accumulate more infrastructure and logged events. It’s critical that the logging stack captures, processes, and stores every event, or businesses could miss a significant event. 

Elasticsearch is common for searching and analyzing log data, but it needs an intermediate to send processed logs. The common intermediate is Logstash, which will temporarily cache and queue events when Elasticsearch is too busy. Logstash has a buffer that will hold events until Elasticsearch is ready again, leaving a single point of failure. If Logstash fails or its buffer overflows, logged events are lost. If it crashes or is taken offline for maintenance, all queued events are lost. Administrators deploy brokers with the ELK stack to help buffer and queue logged events during a spike in messages.

‍

How does the ELK stack work?

Organizations rely heavily on their infrastructure. It not only powers their productivity, but it’s the foundation for their revenue. Whether the organization houses its infrastructure or provisions it in the cloud, it’s a necessary part of business productivity. Small businesses may not have log management in place. Still, spikes in growth and an increase in provisioned IT resources will require logging so that administrators can identify issues before they cause downtime.

Traditionally, organizations might store only a few logs on the network, but enterprise businesses now could experience terabytes of data from hundreds of components every day. Employees can’t read and analyze these events effectively, prompting the need to implement indexing and storage tools such as Elasticsearch. This tool centralized logged events, so administrators no longer needed to SSH into machines and grep logs to find specific events. 

Elasticsearch remedied indexing and storage issues, but administrators came across new ones. They needed a way to queue messages and avoid disruption from dropped events that could not be indexed quickly enough in Elasticsearch. Logstash will aggregate events, but it does not have the scaling capabilities to process large quantities of data during message spikes. 

The ELK stack (Elasticsearch, Logstash, and Kibana) helps manage this issue, providing the enterprise a complete log management solution. Logstash drops logs when overloaded, resulting in the eventual addition of a buffering agent (or broker) to the stack to help manage spiked volumes of events. The three common broker agents are Redis, Kafka, and RabbitMQ. A broker will add better buffering to the log management solution to hold events until Logstash can process them and send them off to Elasticsearch.

What do brokers do?

In a smaller environment, a broker may not seem necessary, but issues arise when there are spikes in logged events. This spike could result from additional resources pouring into  aggregation, a system failure, or a cybersecurity event. When a spike occurs, your Logstash service could get overloaded and begin to drop events. It can also happen when Elasticsearch cannot handle more events, and Logstash needs to pause the pipeline flow. Should any resource in your pipeline slow from a spike in logged events, the entire stack could be affected, and you could lose data.

A broker is an added resource that helps manage this issue. It adds a better buffer to the stack so that you no longer rely on Logstash to buffer and queue messages. Because Logstash doesn’t scale for unforeseen spikes in data, you eliminate the risk of losing events with brokers who collect data from producers. Any spikes in data will be queued much more efficiently and sent to Logstash once it’s free, meaning that Logstash could fail or go down for maintenance and the organization still preserves its event data. 

‍
We’ll examine the three brokers popular in the enterprise.

RabbitMQ

For smaller organizations that want a simple broker solution, RabbitMQ is a solution that understands basic messaging protocols such as AMQP (Advanced Message Queuing Protocol). It’s a general messaging service that will ensure that Logstash and Elasticsearch have processed the latest events. Only then will it move additional data across the pipeline to prevent message deletion.

To handle spikes in data, RabbitMQ sets up a queue where messages are stored. It will hold these messages until confirmation from the target client (e.g., Logstash or Elasticsearch) that it received the message. If any target client fails, it cannot send a confirmation message, at which point RabbitMQ will hold it until the target client recovers.

When spikes in data happen, RabbitMQ will slow down, so one challenge is latency with high-volume traffic. RabbitMQ can be clustered, which will help with latency, but large enterprises don’t usually prefer how it handles overflow. The latency issues could create a backlog and delay real-time analysis tools, which could be an issue for environments requiring fast response times to ongoing events.

Redis

For larger enterprise environments, you will probably find either Redis or Apache Kafka. Redis is an in-memory data broker, which makes it optimized for performance. You might also see Redis referred to as a database (although there is no query language) because it can handle many different data structures in big data queries. This service is beneficial in log management. It can take several other data structures sent from your publishers, including strings, hashes, lists, ordered sets, bimaps, hyperloglogs, geospatial data, and streams.

The most significant disadvantage of in-memory queues is not having enough memory to support them. Redis will persist data to disk when memory is full, but caching on disk slows down your pipeline. Organizations will often use Redis in the Amazon Web Services (AWS) cloud to provide resources quickly but at a reduced cost. If Redis runs out of memory, you still risk event loss, so Redis is mainly implemented for its speed and not scalability.

Apache Kafka

For a solution with a push and pull model, Kafka offers a way to better store (called partitioning in Kafka) and send messages within a data pipeline. Publishers are the infrastructure resources generating logs, and subscribers are the aggregators and processes that pull logs from Kafka’s storage. It’s much more fault-tolerant than Redis and RabbitMQ since it uses disk caching and storage, which means it’s also a slower solution.

Kafka decouples publishers and subscribers so that either one can fail and Kafka will still function. It sends verification messages to publishers to ensure that messages are received and cached and holds them until subscribers can retrieve them. If subscribers slow down or fail, Kafka holds messages until they recover, making Kafka best for failover and reliability, but at the sacrifice of speed from disk-based storage.

Aside from slight performance loss, Kafka’s other disadvantage is that it requires Apache ZooKeeper in a clustered environment to track cluster state. Adding ZooKeeper creates more complexity which might be unwanted but necessary. Because it’s decoupled, Kafka also relies on consumers to track their messages in their queue.

Conclusion

If your organization is experiencing a loss in logged events, a broker could remediate the issue. These brokers all have their advantages and disadvantages. Another option is working with a SaaS solution like LogDNA. With our solution, you can avoid infrastructure complexities and all the burdens of rolling out your tools.