Introducing LogDNA Agent 3.4

LogDNA is now Mezmo but the product you know and love is here to stay.

In our ongoing efforts to provide our customers with a best-in-class log management solution, we’re excited to launch improvements to our Agent. This new, robust version (3.4) provides significant performance improvements for our customers. Specifically, we focused on two main areas of the Agent: error handling and endurance.

Error Handling

One area of focus for Agent 3.4 is our ability to better handle error states and ensure you are optimizing performance. Specifically, we wanted to eliminate noisy error logs as well as give customers more control over their retry files. With Agent 3.4 we’ve implemented:

• Longer backoff for HTTP 5XX errors
• Immediate exiting on HTTP 4XX errors
• Recovery on notify errors
• Configurable retry file locations
• Maximum retry file sizes
• Additional retry metrics
• Compressed retry files

Endurance

We also focused our efforts on improving the Agent’s endurance. Specifically, we implemented:

• HTTP 2.0 support
• Concurrent uploads

The agent was enhanced to support tracking non-contiguous offsets, which allowed for concurrent uploads to be re-enabled without sacrificing stateful lookback.

Performance Data

As excited as we are for customers to try out this new Agent version and see these improvements for themselves, we know that we are only as good as the data we measure. Because of the focus on performance, we decided to share some of the data and metrics from our own internal testing.  To help you understand the nature of this data, I have included a helpful glossary in the Appendix below.

‍

Benchmark Testing

The goal of benchmark testing is to simulate the Agent under expected conditions. We ran benchmark tests on an EC2 instance in the us-west-1 region with the following specifications:

‍

Instance Type: t2.large

Architecture: x86_64

Processor: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GH

Number of Cores: 2

Memory: 8 GB

Storage: EBS

‍

And here were the results of the test:

‍

File System Metrics

‍

Total Time (sec): 23

Total Lines: 4166932

Line Rate (lines/sec): 181170.95652173914

Total Size (bytes): 52245566.0

File Rate (bytes/sec): 2271546.347826087

‍

Ingestion Metrics

‍

Total Time (sec): 22

Total Size (bytes): 310600357.0

Total Number of Samples: 149

Average Request Duration (sec): 1.0604503484161074

Average Request Size (bytes): 2084566.1543624161

Endurance Testing

The goal here is to test the LogDNA Agent for an extended period under a pseudo-realistic workload. The following endurance test was run on an EC2 instance for 72 hours with the following specifications in the us-east-2 region:

‍

Instance Type: m5d.large

Architecture: x86_64

Processor: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GH

Number of Cores: 2

Memory: 8 GB

Storage: EBS

‍

Memory Consumption:

In these conditions, the logdna-agent consumed around 46 MB VmRSS, which measures how much RAM the process is actually using. 

CPU Consumption:

Outside of the initial program start (which is reflected in the spike in the graph below, and is expected), the Agent consumed less than 2% of CPU throughout the entire endurance test. 

‍

In summary, Agent 3.4 performs well under heavy load for a long period of time and in error states that require some form of recovery. Please check out our documentation for further details. And as always, feel free to reach out to me with any questions: albert.feng@logdna.com

Appendix

‍

Definitions:

File System Metrics

• Total Time: the time of the benchmark run in seconds.

• Total Lines: the total number of lines read off the file system by the agent, while the benchmark process was run.

• Line Rate: the number file lines per second read by the agent. Calculated as (total lines)/(total time)

• Total Size: the total amount of log data, in bytes, consumed by the agent.

• File Rate: the rate, in bytes per second, of log data being consumed by the agent. Calculated as (total size)/(total time)

‍

Ingestion Metrics

• Total Time: same as defined above

• Total Size: The total amount of data sent via HTTP request to the Ingester.

• Total Number of Samples: The amount of samples taken by the metrics recorder during the benchmark run.

• Average Request Duration: the average duration of an HTTP call to the Ingester over the duration of the benchmark run. It is calculated as (total number of seconds spent sending HTTP requests to the ingester)/(total number of ingestion samples taken)

• Average Request Size: the average size of an HTTP request sent to the Ingester over the duration of the benchmark run. Calculated as (sum of all bytes sent to the ingester)/(total number of ingestion samples taken)

‍

Memory Metrics

Max Process Virtual Memory: the maximum process virtual memory observed throughout the benchmark process run.