Introducing LogDNA Agent 3.4

    4 MIN READ

    LogDNA is now Mezmo but the product you know and love is here to stay.

    In our ongoing efforts to provide our customers with a best-in-class log management solution, we’re excited to launch improvements to our Agent. This new, robust version (3.4) provides significant performance improvements for our customers. Specifically, we focused on two main areas of the Agent: error handling and endurance.

    Error Handling

    One area of focus for Agent 3.4 is our ability to better handle error states and ensure you are optimizing performance. Specifically, we wanted to eliminate noisy error logs as well as give customers more control over their retry files. With Agent 3.4 we’ve implemented:

    • Longer backoff for HTTP 5XX errors
    • Immediate exiting on HTTP 4XX errors
    • Recovery on notify errors
    • Configurable retry file locations
    • Maximum retry file sizes
    • Additional retry metrics
    • Compressed retry files


    We also focused our efforts on improving the Agent’s endurance. Specifically, we implemented:

    • HTTP 2.0 support
    • Concurrent uploads

    The agent was enhanced to support tracking non-contiguous offsets, which allowed for concurrent uploads to be re-enabled without sacrificing stateful lookback.

    Performance Data

    As excited as we are for customers to try out this new Agent version and see these improvements for themselves, we know that we are only as good as the data we measure. Because of the focus on performance, we decided to share some of the data and metrics from our own internal testing.  To help you understand the nature of this data, I have included a helpful glossary in the Appendix below.

    Benchmark Testing

    The goal of benchmark testing is to simulate the Agent under expected conditions. We ran benchmark tests on an EC2 instance in the us-west-1 region with the following specifications:

    Instance Type: t2.large

    Architecture: x86_64

    Processor: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GH

    Number of Cores: 2

    Memory: 8 GB

    Storage: EBS

    And here were the results of the test:

    File System Metrics

    Total Time (sec): 23

    Total Lines: 4166932

    Line Rate (lines/sec): 181170.95652173914

    Total Size (bytes): 52245566.0

    File Rate (bytes/sec): 2271546.347826087

    Ingestion Metrics

    Total Time (sec): 22

    Total Size (bytes): 310600357.0

    Total Number of Samples: 149

    Average Request Duration (sec): 1.0604503484161074

    Average Request Size (bytes): 2084566.1543624161

    Endurance Testing

    The goal here is to test the LogDNA Agent for an extended period under a pseudo-realistic workload. The following endurance test was run on an EC2 instance for 72 hours with the following specifications in the us-east-2 region:

    Instance Type: m5d.large

    Architecture: x86_64

    Processor: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GH

    Number of Cores: 2

    Memory: 8 GB

    Storage: EBS

    Memory Consumption:

    In these conditions, the logdna-agent consumed around 46 MB VmRSS, which measures how much RAM the process is actually using. 

    CPU Consumption:

    Outside of the initial program start (which is reflected in the spike in the graph below, and is expected), the Agent consumed less than 2% of CPU throughout the entire endurance test. 

    In summary, Agent 3.4 performs well under heavy load for a long period of time and in error states that require some form of recovery. Please check out our documentation for further details. And as always, feel free to reach out to me with any questions:



    File System Metrics

    • Total Time: the time of the benchmark run in seconds.
    • Total Lines: the total number of lines read off the file system by the agent, while the benchmark process was run.
    • Line Rate: the number file lines per second read by the agent. Calculated as (total lines)/(total time)
    • Total Size: the total amount of log data, in bytes, consumed by the agent.
    • File Rate: the rate, in bytes per second, of log data being consumed by the agent. Calculated as (total size)/(total time)

    Ingestion Metrics

    • Total Time: same as defined above
    • Total Size: The total amount of data sent via HTTP request to the Ingester.
    • Total Number of Samples: The amount of samples taken by the metrics recorder during the benchmark run.
    • Average Request Duration: the average duration of an HTTP call to the Ingester over the duration of the benchmark run. It is calculated as (total number of seconds spent sending HTTP requests to the ingester)/(total number of ingestion samples taken)
    • Average Request Size: the average size of an HTTP request sent to the Ingester over the duration of the benchmark run. Calculated as (sum of all bytes sent to the ingester)/(total number of ingestion samples taken)

    Memory Metrics

    Max Process Virtual Memory: the maximum process virtual memory observed throughout the benchmark process run.