Log Management Best Practices: Elevated Importance in the DevOps Toolkit

A constantly evolving development environment has completely changed the way we approach the app process. Our command-line forebears would tuck tail and run if they saw the ever-increasing complexities modern developers contend with every day. Scores of data streaming in daily, new frameworks, and technical stacks that are starting to make the biological cell look simple. But for all of this increased complexity – an often overlooked powerful source of making sense of it all has been neglected; we’re now finally elevating this source to its proper place. We’re talking about the limitless potential that logging can provide app developers. By practicing intelligent server logging from the start, we can utilize server logs for enhanced security, data optimization and overall increased performance. For a moment think about how important logged information is in a few select non-technical fields. Banks must have records of money transfers, airplanes have to track their flights throughout the process, and so on. If an issue were to occur right now or in the future, this data would be there to overview and help us come to a quick solution. The log management best practices should no longer be an afterthought, but part of the development process. This line of thinking needs to be at the forefront of any future cloud-based log management strategy.

Develop Intelligent Logging Practices From the Start

There is a crucial importance to how your own log messages are constructed in the first place. Proper log construction is integral to making sense of your own data and allowing third-party applications – like LogDNA – the ability to parse logs efficiently and help you gain increased insights on your data. JSON logging is one of the most efficient ways to format your logs into a database that can be easily searched through and analyzed. This gives you power on your own and helps your other tools get the job done even faster. JSON’s logging format allows for a simple standard for reading and coding, without sacrificing the ability to comfortably store swathes of data your app may be producing. It’s best to begin JSON logging from the get-go when developing a new application. But with enough elbow-grease you can reasonably go back and change an app for future JSON support. JSON logging standards should be widespread and mandatory for the rest of your project team. This way everyone is able to comprehend the same data and avoid any communication mishaps. A majority of libraries will assist in creating metadata tags for your logs and these can then be simply configured into a JSON log. Here is a brief pseudo example of a default logging level output and it’s JSON input.

Logger Output logger.info (“User clicked buy”, {“userId”: “xyz”, “transactionId”: “buy”, “pageId”: “check-out” }};JSON Input{“alert”: “User clicked buy”,“userId”: “xyz”,“transactionId”: “buy”,“pageId”: “check-out”}

A standard like this is important namely for two major reasons. It facilitates a shared understanding that can be read between different departments, including: devops, project-leads and business oriented team members, which creates a central interchangeable environment to utilize shared data from one another. This might include varied business functions across a company, from marketing initiatives on the business front to a developer streamlining a new (UI) on the checkout cart function. Secondly, when the log output is in this format it allows for machines to read the data thoroughly and with greater ease. What could take hours or even days while manually searching has been reduced to a few seconds with the machine’s all-seeing eye.

Making Sense of Levels

The identity of your data is the next step to take once you’ve constructed it in an adequate manner. It helps you monitor how users are experiencing your interface; it can look out for debilitating performance issues, potential security problems, as well as give you the tools to use user trends to create a better experience. Some log levels will not be on a production-level app; an example of this would be debug. Others will be streaming in constantly. Our previous example showed the information level, typically used for simple inputs of information from the user. There is a breadth of great information here – you can count on that. Furthermore, levels such as warning, error, and worst of all fatal or critical, are logs that need immediate attention before besmirching your good name. This leads us to another important practice.

Structuring Log Data

These levels directly translate into a few important JSON fields. The fields of either json.level and json.severity help determine where warnings and errors are coming from. You can catch some of these earlier warnings before they snowball into a major fatal problem. Valuable fields of data like this help catch major events in the process. Some other important fields to look out for include json.appId, json.platform and json.microservices. If you’re running your logs through Kubernetes then you know you’ll be running on a wide variety of different platforms. The json.platform comes in handy for this. In summary of these few listed fields: you can filter logs pertaining to only events that cause errors or warnings, isolate messages from designated add-on apps and microservices, or self select parsed logs from multiple staging platforms. There are several data types that JSON supports and these include numbers, strings and nested objects. The trick is in knowing how to not get these mixed up. Quotes around a number can be misread as a string. It’s important to properly label events and fields so that these mix ups don’t happen. Many developers are apprehensive about saving and storing all of this data, feeling as if they’ll never be able to look through any of it anyhow. There’s a part of that which is true. But the best thing about proper logging practices is that they don’t have to anyways. That’s the machine’s job. More importantly, it’s the machine-based solution that LogDNA has got down to a science. You never know which part of the data you’ll need down the line.

Always Collect Log Data for Further Use

Logs are the nutrient rich pieces of data that can be stored and collected for future use. They don’t take up much space and can be looked over instantly. With log management best practices, you can make sure that your log monitoring is the best way to get a clear picture of your overall system. Logging is a fundamental way to understand your environment and understand the myriad of events going on. Many future trends can be predicted from past data as well. Proper logging monitoring helps detect any potential problems and you can rid them from your app before they become a major issue. With log management best practices, you can make sure you are

Collect as much data as you can.

You may think that not all data is created equally and that is sometimes true. But you’d be surprised what certain innocuous metrics could point towards in the future. With that being said, you should know your system better than anyone else. This extends to what type of data you think is most crucial in logging. As a general rule of thumb the following are two important pieces of information that should always be logged.

Log Poor Performance

If a certain part of your application should be performing at a general high speed, i.e. a database ping – then you should log this event if it is not performing adequately. If an event went above a certain ms, then you’d be rightfully worried about the overall performance. This leads you to look into the problem at hand.

Log Errors

Even with log management best practices, you’re bound to come across a few errors after adding any type of new features to a system. Logging these errors can be a useful piece of information for debugging and future analytics. If this error somehow gets into production, you now have the tools to rid your app of the problem and ensure a happy user experience.

Watch for Performance Deviations & Look at Larger Trends

The macro picture of your logging data will be the deciding factor in telling you how your system is functioning and performing on an overall basis. This is more important than looking at a single data point entry. Graphing this type of data in a log management system will give a visual representation to a process that used to be nearly inaccessible.

Keep the Bottom-Line in Mind

At the end of the day what matters most is that your development processes are always evolving, growing and keeping the end-user in mind. Helping your internal teams take advantage of their own data is liberating. We help find the interwoven connections between data and apply new and innovative features to contend with a rapidly evolving logging sphere. Follow these parameters for the best logging practices through proper log message construction, collaborative standards, structuring and storing data and you will succeed. Log management is a real-time understanding of your metrics, events and more that can mean a lot of different things to different parts of your team. With log management best practices, you can make the very best use of your data.