What is Logspout?

Logspout is an open source log router designed specifically for Docker container logs. If you’ve ever looked into log management for Docker, chances are you’ve heard of it. Logspout is a container that collects logs from all other containers running on the same host, then forwards them to a destination of your choice. This lets you send logs to an HTTP/S server, syslog server, or other endpoint without having to monitor files or modify your host systems.

Mezmo, formerly known as LogDNA, provides a version of Logspout that routes logs directly to Mezmo's ingestion servers, making it even easier to deploy. We’ll explain how the Mezmo Logspout container works, how it differs from the Mezmo agent, and when you might want to use one over the other.

How is Logspout Different From the Mezmo Agent?

Mezmo Logspout works by connecting directly to the host’s Docker socket and reading logs that container print to stdout or stderr. It then routes these messages to a destination of your choice through an HTTP endpoint. It is streaming based so any hiccups or spikes will likely result in data loss.

The Mezmo agent works by monitoring files for new events and sending those events to Mezmo. This offers a high level of compatibility to not just Docker logs but across platforms. Since it is file based, it includes recovery and retry mechanisms to minimize the loss of data.

Benefits of Logspout

Let’s look at some of the benefits that Logspout offers for shipping logs to Mezmo. If you're used to using the agent, this will give you a better idea of the differences between the Mezmo agent and Mezmo Logspout.

Better Performance

Mezmo Logspout is a lightweight Go application that runs entirely in memory. The Mezmo Logspout image is roughly 50 MB and uses just over 9 MB of RAM on a fresh start. Since it doesn’t read from log files, it can ingest logs from any number of sources without being constrained by disk speed or memory.

Easier Management

Logspout does not need to access any files or directories and only requires access to the Docker socket, which is located at /var/run/docker.sock. Any newly started containers are automatically detected by Logspout without requiring you to restart the container or add a new volume mount. Like the agent, you can change settings through environment variables.

Logspout can also be deployed using most orchestration tools including Kubernetes, Rancher, Docker Swarm, AWS Elastic Container Service, and Docker Compose. Deploying it as a DaemonSet ensures that each node in your cluster runs a single instance of the container. You can find examples for orchestrating Logspout in the project’s readme.

Customizability

Logspout allows you to ship the same logs to multiple destinations in addition to Mezmo. You can change the default Mezmo endpoint to a separate endpoint (such as an on-premise installation), or send the same logs to multiple destinations. We’ve also added support for tags and custom hostnames, so you can search and filter your logs more easily.

Since Mezmo Logspout image is derived from the original Logspout image, you can leverage other features such as multiline logs, custom routes, and the ability to inspect log streams in real-time.

Other Considerations

The key difference between Logspout and agent is that Mezmo Logspout can only read logs from other Docker containers due to its lack of filesystem access. In other words, Logspout cannot log your hosts or non-Docker containers unless you can route them through a separate container. In addition, Logspout only works when the Docker logging driver is set to either journald or json-file. Drivers that don’t work with the docker logs command are not supported.

Like the Mezmo agent, the Logspout container must run as the root user. This is because the container accesses the Docker API directly. This creates a security risk, since any process with direct access to the Docker API can perform any action including creating containers, effectively gaining root access to the host. We explained the security implications of this in detail on our blog post on why the Mezmo agent runs as root.

Lastly, there is an increased risk of data loss due to Logspout only storing logs in memory. If the container stops unexpectedly, any logs that were buffered but not sent will be lost (they could still be on the host if the Docker Logging Driver processed them, but Logspout won’t attempt to re-send them after restarting). You can reduce this risk by:

• Increasing the memory limit of the container
• Increasing the buffer size
• Increasing the flush interval (how frequently it sends buffered logs to Mezmo)
• Increasing the number of retries in case of a network failure.

When Should You Use Logspout?

To recap, for your log management solution, use Logspout when you

• Are using Docker or a Docker orchestration tool
• Don’t need to log host systems or non-Docker applications
• Want fast performance and a small resource footprint
• Want to ship logs to multiple destinations
• Are okay with streaming logs knowing that there is the possibility of losing logs in the event of a container failure

The best way to determine whether the Logspout container is a good fit is to try it. The Mezmo Logspout container is completely open source and can be deployed in seconds. Let us know if you have any questions, comments, or suggestions!