Logging Best Practices, Part 2: General Best Practices

Isn't all logging pretty much the same? Logs appear by default, like magic, without any further intervention by teams other than simply starting a system… right?

While logging may seem like simple magic, there's a lot to consider. Logs don't just automatically appear for all levels of your architecture, and any logs that do automatically appear probably don't have all of the details that you need to successfully understand what a system is doing. We've talked about actionable logs in part 1 of our Logging Best Practices Series. Now let's take a look at some other general best practices for logging: using log levels and ensuring you have logs for all levels of your architecture.

‍Before we can get into different ways of constructing loglines, we need to talk about overall logging practices. All loglines have levels and types.

Log Levels

Log levels refer to how severe a message impacts a system, helping operations understand how to react. On a high level, there are typically seven or eight different levels to choose from, depending on which standard you're following. Let's take a look at the log4j standard [1]:

• TRACE – An action happened—any action at all. This level significantly decreases performance because every tiny thing is dumped into the logs.
• DEBUG – An action happened that transferred data inside of a component (a microservice, an API, a database, etc.). Debug messages are probably very familiar to most developers; we use them to understand when functions or methods were called and how data may have transformed across a single plane.
• INFO – An action happened that transferred data from component to component. Generally, this level is the most used as a general message.
• WARN – There's something wrong, but the problem isn't blocking the component from continuing to function. An example of this level would be a deprecation notice.
• ERROR – Something went wrong with a specific component of your system causing a failure of part of the component's functions, but the rest of the component is still functioning. In this case, a function call may have failed, but the component is still processing data.
• FATAL – The entire component has crashed and has no hope of recovering without manual intervention.

As a general best practice, use log levels consistently. Have a set plan for your entire team. You need to come to a team-wide agreement at the very least on which log levels to use for each situation as language standards aren't always strict. A company-wide agreement is even better for enabling the effective use of logging systems by your operations-focused teams as they often are handling multiple environments owned by various development-focused teams.

Log Only What You Need

On the flip side, don't just use INFO for everything! Log levels enable you to fine-tune your logs to show only the information you need when you need it. Are you on a production system and need to know only when a system is failing? Report only on ERROR or FATAL, and ensure that your code flags each error with the appropriate log level. On a dev box and want to watch the data flowing throughout your system? Report down to DEBUG or even TRACE, and update your code accordingly. If you don't use log levels, your systems will be less efficient as you'll likely be logging too much, and you probably won't have the information you need in a hurry to fix problems when they arise.

You shouldn't need to log everything that happens in your system all the time. Logging everything all the time leads to information overload that takes over your system and fills up your log files, and you don't end up with anything useful—or the useful bits are buried at the bottom of a haystack of noise. If you use logging levels to identify data needs, you can turn up or down the noise with a fine-tuned knob, avoiding overly noisy logging systems.

Log Generation

While you do get general logs from your platforms and operating systems, you won't get logs for your application unless you add them explicitly in your codebase, and you won't get logs for additional needs such as auditing without explicitly adding them. You need to generate logs for all of your different needs throughout the depth of your architecture. There are many types of logs: system logs, application logs, audit logs, etc. If one level of your architecture is not logging to the same log management system that you're using everywhere else, remedy that immediately.

Also, don't just print to stdout. Simply printing to standard output (or standard error) just means the messages you're sending get lost in the sands of time. Ensure you use a logger to send your data to a file, which you can then archive, parse, or otherwise use to match the needs that you have for that data.

On that same thread, don't assume auto-generated logs, especially ones created by an interpreter, are good enough. You're missing data or generating useless, non-actionable data if you're relying simply on auto-generated logs.

By the way, just like in grammar school, spelling counts! When you're trying to parse messages and search logs, a single misspelled word will get amplified tenfold, and you will miss context that you could very well have used in your troubleshooting efforts. Take the time to check your log messages in your codebase before committing them. After all, details do matter.

Finally, don't dump everything into your logs. It may be obvious, but you don't need to add "attachments" to your logs. Any kind of artifact generated that isn't a log message or a structured log object doesn't belong in your log stream. For example, if your function generates a spreadsheet based on a query on a database, don't include the entire spreadsheet in your log object. While that may seem innocuous in your local system, that artifact will fill up your available space extremely quickly on production, and you'll be dealing with noisy logs to boot. Your operations team will not thank you for it.

If you can stick to these recommendations for using log levels and generating logs, you should be well on your way to writing actionable logs, the number one priority of any logging plan.