Logging Best Practices, Part 6: Security and Compliance

Isn't all logging pretty much the same? Logs appear by default, like magic, without any further intervention by teams other than simply starting a system… right?

While logging may seem like simple magic, there's a lot to consider. Logs don't just automatically appear for all levels of your architecture, and any logs that do automatically appear probably don't have all of the details that you need to successfully understand what a system is doing. We've talked about actionable logs, log levels, logs for all components and needs, different methodologies for generating logs, and best practices specifically for text-based logging and structured logging. In this final article in the series, let's touch on security and compliance considerations for logging.

‍Ah, compliance. The bane of every engineer and the boon of any enterprise. If you've ever worked at a company that has a product that is compliant in some form with any system like PCI or HIPAA, you know the pain of working through the annual compliance audits. Logs in various forms are central to generating evidence, and there are some things to be aware of when generating logs in the first place to ensure that your audit experience is less painful [1]. There are a lot of additional things to consider and new practices coming out all the time, but this post should give you a good overview of where to start.

Logging, Security, and Compliance

First, let's set some context. Why should you care about logs in the context of security and compliance, especially if you're not at the point where you need to be compliant? Aside from being a central component to gathering evidence for an audit, logs are critical to identifying when and where unauthorized access occurred in any system. Let's imagine someone has gotten access to your systems. They immediately start rooting around, trying to find something of interest. However, they want to cover their tracks, and one of the first places they go to cover their tracks is to attempt to tamper with the logs [2]. There are ways to ensure that, even with attempts at log tampering, you can secure your logs enough to detect intrusion in this fashion. For that reason alone, logging is as critical to active security measures as it is to forensic ones like compliance.

From a compliance perspective, most if not all security standards require compliance with some sort of logging mechanism. For example, as of version 3.2.1 (the version from May 2018, which is current as of this writing) [3], the PCI DSS requirement 10 explicitly requires audit trails and other logging mechanisms to be present on all systems that can access or provide access to cardholder data. As a result, auditing processes involve gathering logs as compliance evidence and examining them for any potential violations. Good application event logs will speed up your audit process by virtue of being easy to read and clear about which applications were accessed by which users. If your logs are neatly kept, a security researcher or auditor will be able to get what they need quickly and efficiently.

Some Initial Best Practices to Consider

First, ensure you have an append-only log and/or ensure you have read-only archives of all of your logs. Append-only logs means each log line is appended to the end of a log file with no way to erase prior lines save deleting the file. Couple that with a read-only archive of your log files stored in a separately secured location (preferably more than one archive in more than one separate location!), and you can ensure that wiping audit logs takes a significant amount of time, buying your alerting mechanisms time to reach someone who can do something about it. Even if an attacker were to turn off logging, they can't turn it off retroactively other than deleting the log files themselves, and if an archive is stored in a separately secured location, they can't delete the archive without additional work [4].

Speaking of archives, you should take an extra step to mask or sanitize any personally identifiable information (PII) in your logs. While good development practices should dictate that PII only is stored when absolutely necessary and only in secured and hashed databases, a lot of people forget to ensure that the same PII never appears in logs other than a unique identifier (UID) that correlates with the same secured database. Logs are often overlooked when thinking through security, and considering they're often transported across networks, logs are one vector for gathering data that can be a bit more accessible than others.

Ensure that you have process alerts set up to know when a logging process has stopped working, been turned off, or been accessed from an unauthorized user. This kind of tamper detection alert ensures that your logs are always writing, especially to that read-only archive. One of the first things an intruder would try to do would be to turn off logging, and a process alert on a low level would help you notice this problem quickly.

There are no known users in a secure system, and that includes in the logs. Every single access should be logged, even from automated systems and others, as you never know what may have caused a problem or who may have gotten access to a secured system. Most compliance standards require this type of access tracking. As an example scenario, a momentary slip in a commit to source control or someone lurking in your chat system undetected while people are discussing a deployment system could provide an attacker with credentials for your CI/CD system, which then could have admin credentials to a separate system. That kind of scenario is why there are no known users. A happy side effect of following this best practice for security and compliance is that it will get easier to understand where and when a system or an application fails because you can trace back through identification of credentials that were used to the system or user that triggered a deployment or other event.

Separate out your concerns. Typically, most folks consider logging a violation of the classic software engineering separation of concerns principle and file the need under a cross-cutting concern. However, you still can and should abstract the logging need out and separate out the resulting logs from each concern. In Python, for example, abstract your logging library of choice to its own class, and set a separate logging location and any context using individual instances coupled with handlers [5]. In doing something like this, you ensure that someone needing access to logs about one concern doesn't get access to logs from another.

Finally, use the same best practices as before, especially these two: Use standard logging libraries, and use all logging levels wisely. I'm calling these two best practices out explicitly due to a couple reasons. First, a standard logging library that has a history of quality development probably has a lot more bandwidth than you do to think about security, like memory leaks or input data sanitization. A standard library can also ensure your format is standardized and recognized by multiple archive systems or other data storage and retrieval systems that make enacting compliance easier. Using logging levels, as we noted before, decreases the chances that someone turns off logging to reduce noise as they're provided with finer controls. You can then use those logging levels to tune alerting for security purposes or to share specific alerts with different teams.

Overall, logging is critical to security and compliance, and it's often a component that people miss when thinking through their security and compliance needs. The best practices I outlined here are by no means exhaustive, especially as the needs of any system can and will shift when you move from platform to platform. There are many considerations and constant new ways to implement better logging security. Instead, this list is intended to get you thinking about how logging should fit in an overall security and compliance strategy, and hopefully make your audit cycle a bit easier.