Choosing The Right Ingestion Client
LogDNA has a range of options by which you can supply your account with log data. If you're not entirely familiar, these are via the Collector Agent, syslog, code libraries, PaaS integration, and REST API.Given all these options, which one is right for you? Unfortunately, like many things in tech (as in life), the question is difficult to answer definitively. I don't mean this in any non-committal way. I only mean that it's hard to say which ingestion client is the right one without some form of well thought through qualifications.Given that, let's work through the five ingestion clients which LogDNA provides, and consider when they would, and when they wouldn't, be the right option to choose. Let's start off with, arguably, the simplest of the five, the logging agent.
The Collector Agent
If you want the easiest option, one supported across all of the major operating systems, (Linux, macOS, and Windows), then the Collector Agent is the choice to make. Written in Node.js, it logs information at the OS level. To quote the official documentation, it:
Reads log files from the computer it is installed on and uploads the log data to your LogDNA account…It opens a secure web socket to LogDNA's ingestion servers and 'tails' for new log files added to your specific logging directories. When those files change, the changes are then sent to LogDNA, via the secure web socket.
In addition to reading log files on your server(s), the Collector Agent can also read from the Windows Application Log.As you may expect from a system daemon or service, after installation, all that's required is to provide a configuration. On Linux/UNIX systems, this is stored in /etc/logdna.conf and on Windows, it is stored in C:\ProgramData\logdna\logdna.conf.As an example, let's assume that my servers are running Ubuntu 16.04 and that I have installed the agent. With that done, the minimum configuration that I'd need to add to /etc/logdna.conf, so that it started sending my log data is:
logdir = /var/log/nginx,/var/log/auth
key = <your logdna="" ingestion="" key=""></your>
I've only needed to provide the directories to read the log files from that are of interest to me, as well as my LogDNA ingestion key. In the example provided, it would send log information from my NGINX server, as well as authentication requests to my LogDNA account.If there are files that shouldn't be included, then I can use the exclude option to have the agent skip over them. And there's another handy option: tags. It allows you to group hosts automatically into dynamic host groups without having to explicitly assign a host to a group within the LogDNA web app.The agent can be installed using package managers, such as Homebrew, APT, and Yum, as well as from source. Additionally, it also integrates with Kubernetes, if you're using Docker containers.Given this level of simplicity, if you're just getting started with LogDNA, want an option that requires a minimum investment, both initially and over time, and want something that's available on all the major platforms, then the Collector Agent is the right one to choose.However, it is considered a bit of a shotgun approach. That is, while it can send your log data straight to LogDNA with little effort, you have only negligible control over how and when it does so. Specifically, you can't choose a level of log granularity, such as debug, info, warning, and critical.
syslog
The next option is syslog. It is the veteran ingestion client, available on all Linux/UNIX installations, since its development by Eric Allman back in the 1980s. If you're not familiar, syslog (also available as syslog-ng and rsyslog) is a logging system which allows for logging on a server locally, as well as to remote syslog servers.It is an excellent choice for logging information across a wide range of devices, such as printers and routers, and a wide range of system-services, such as authentication, servers such as NGINX, CUPS, and Apache, system-level events, and kernel-level events.To send log data to LogDNA using syslog requires a little bit more effort than the logging agent. Gladly, they're well documented under Settings -> "View your logs" -> "Via syslog".Setting up syslog requires adding an entry to /etc/syslog.conf, which can be generated in the "Log Source" modal dialog. For example: *.* @syslog-a.logdna.com:16312. syslog-ng and syslog require a little more configuration effort. But like syslog, the instructions are well documented.syslog is a particularly appealing option if you're looking for simplicity and minimal cost. One of the three daemons is likely universally available on any Linux/UNIX system, as they've been around for so long. What's more, it's a very well understood application, with a wealth of documentation available on how to configure it across a wide variety of use cases.This gives it an advantage over the Collector Agent. Where the agent passes log data as-is, syslog allows you to choose the format in which the log messages are written. This may not be necessary. But it's handy nonetheless, as you can write log data that better suits the needs of your applications. Doing so, you’re better able to parse the log data when the time comes, as it will make more sense.Given this advantage, the fact that syslog is free, and that LogDNA provides detailed configuration instructions and supports host tags/dynamic groups, it's well worth considering.You won't have to look far to find support for it, and you won't have to invest much time in maintaining it. Another handy aspect of sending log data using syslog is that you don't need to explicitly integrate against LogDNA, such as when developing applications, to send the log data from your applications. By logging to the system logs, that information is automatically, and transparently sent by syslog.However, one thing that works against using syslog is a lack of native Windows support. Having said that, there are syslog daemons for Windows, such as Kiwi syslog Server. And you'll find others if you do further research. But they're not natively available like syslog is for Linux/UNIX systems.
PaaS Integration
Let's say that neither the agent nor a syslog daemon or ingestion client suits your purposes, as you're using a PaaS (Platform as a Service), such as Heroku. In that case, you still need to send your log data to LogDNA, but you may not have as much control as you otherwise might.In this case, while the options aren't extensive, you're not without options. LogDNA supports Heroku, Kubernetes, Docker (including Docker Cloud, Convox, and Rancher), Fluentd, Flynn, CloudWatch, Elastic Beanstalk, and Cloud Foundry.Assuming that you're using one in the above list, let's first consider what you get, which depends on the platform. Some provide more configuration and flexibility; some provide less. In many cases, there isn't much choice in how logs can be forwarded.Then there's the setup complexity and time. While I won't say that any of the setups are easy, some will require more time and effort than others. However, regardless of what you're using, LogDNA provides thorough documentation for each one.
Code Libraries
Now to my favorite option, code libraries. As a software developer, you'd naturally expect that it would be. LogDNA has libraries officially developed and supported for four languages -- Node.JS, Python, Rails, and Ruby. There are also libraries for Go, iOS, Java, and PHP.Given that these are some of the most popular languages and that they have runtimes available on the major operating systems, it's likely that at least one of them will suit your needs for ingestion client.Now let’s consider the positives of using code libraries. Unlike the previous two options, code libraries afford you complete control over both how and when your log data is sent. You can integrate logging support into your existing application(s).You can log as much or as little as your application or regulatory environment demands. You can add as much optional extra information as you want. And you can specify, exactly, the information that is sent, such as custom metadata.Here's a minimalist example using PHP, which is often my language of choice.
php
use Monolog\Logger;
use Zwijn\Monolog\Handler\LogdnaHandler;
require_once ('vendor/autoload.php');
$logger = new Logger('general');
$logdnaHandler = new LogdnaHandler(
'YOUR LOGDNA INGESTION KEY',
'myappname',
Logger::DEBUG
);
$logger->pushHandler($logdnaHandler);
# Sends debug level message "mylog" with some related meta-data
$logger->debug(
"mylog", [
'logdna-meta-data-field1' => [
'value1' => 'value',
'value2' => 5
],
'logdna-meta-data-field2' => [
'value1' => 'value'
]
]
);
This example initializes a logger object, which will send log messages, at or above the level of DEBUG to LogDNA. It will also send two sets of custom metadata — metadata which could be anything that makes sense to your application.To be fair, this is rather a "Hello World" example. And integrating logging into your code base would take more effort than it took to create this short example — especially if it's an older and quite complex legacy codebase.That said, I hope you'll appreciate the level of power and customizability that using a code library provides. On the flip side, code libraries also incur a high(er) investment, both in their initial investment, as well as in their maintenance over the longer term.They take special planning, design, testing, and debugging. They also need to be developed using security best practices, both in the code itself, as well as in the deployment process. Given that, while you have much more power, the investment will be higher. Something to keep in mind with ingestion clients.
The REST API
And now to the last option. If none of the previous four options suit — or your particular setup is entirely custom — there's one last option that just may work, the LogDNA REST API.Similar to using a code library, the REST API allows you to send log data, along with custom metadata, to LogDNA.
curl "https://logs.logdna.com/logs/ingest?hostname=EXAMPLE_HOST&mac=C0:FF:EE:C0:FF:EE&ip=10.0.1.101&now=$(date +%s)" \
-u INSERT_INGESTION_KEY: \
-H "Content-Type: application/json; charset=UTF-8" \
-d \
'{
"lines": [
{
"line":"This is an awesome log statement",
"app":"myapp",
"level": "INFO",
"env": "production",
"meta": {
"customfield": {
"nestedfield": "nestedvalue"
}
}
}
]
}'
In the example log request, from the docs, above, you can see that requests to the API are GET requests, and require that a hostname and LogDNA ingestion key are provided. A MAC address, IP address, and a timestamp are able to be included as well.The request body is in JSON format and can contain one or more message lines. JSON is a light-weight and almost universally accepted format, one able to be created by most every popular software language, along with a variety of tools, today.As with the code libraries, and as you can see in the example above, the message lines contain the message’s timestamp, body, application name, and log level, as well as optional metadata.Given that the request is being made using curl, it could be wrapped in a shell script on Linux/UNIX or batch script on Windows. Then, if applicable, the ingestion client could be tied into either CRON or the Windows Scheduler. If written well, it could also take input from any number of services, or be called from an existing process via a system call.However, you don't need to use curl. You could use any software language that can compose and send a GET request, whether natively or via a third-party package, such as GuzzleHTTP for PHP. Similar to how they work, you could integrate logging directly into an existing application, even if it didn't have dedicated support via one of the eight code libraries.The REST API offers quite a significant level of flexibility; a flexibility that the Collector Agent, syslog, and PaaS options simply don't — or can't — provide.Unfortunately though, as with the code library option, there won't be a clear set of steps required to integrate it. Every application is different and has different needs and requirements. What's more, it will require a higher investment, both at the start and over time. However, if it's the one that best suits your needs, then it's worth considering.
In Conclusion
These are the five ways in which you can send log data from your servers and applications to LogDNA. They all have pros and cons, they're not all equal, and they can't be compared in the proverbial "apples to apples" way.The needs of your particular ingestion client, whether server or application, will supply most of the criteria; as will your budget and the experience of your technical staff. But regardless of the choice that you make, you can get started quite quickly, likely using the collector agency or syslog, and then progressing to the more involved options as time goes by, according to your particular use case, and as your budget allows.
Table of Contents
Share Article
RSS Feed
.png)
.png)
