Containers brought everything with them, from reduced dependencies to architectural complexities. So it’s not surprising that the difficulty of implementing and managing a centralized log in a microservice can range from tedious to impossible.
When we mostly had low-traffic monolithic applications that ran on a few servers, managing decentralized logs wasn’t awful. With the increasing adoption of cloud and microservices, effective logging means implementing a data infrastructure that speedily aggregates data from several individual services and systems. It also means that we must store logs in a readable format that’s visible, scalable, resilient, and searchable. As a result, centralized logging is becoming increasingly important, even in distributed systems like microservices.
Centralized Versus Decentralized Security Logs
Let’s consider a security breach as an example. A Security Operations Center (SOC) team’s ability to successfully mitigate an attack depends on their mean time to detect (MTTD) and mean time to respond (MTTR) to potential security incidents.
While proactively analyzing log data helps reduce the MTTD and MTTR, you still risk being reactive to attacks with decentralized logs. You’re more vulnerable to attacks that exploit insufficient log monitoring. Even after identifying suspected malicious activity, how fast can the SOC team detect and contain an attack? It takes a long time if the team has to crawl through thousands of data entries on multiple event logs, analyze them to understand the suspected attacker’s intent and move to contain the problem.
Well-configured, centralized logging should give your SOC team complete visibility of your entire system. Your log data enables in-depth forensics on the attack and other potential loopholes in case of a breach.
Managing Logs in a Decentralized Platform
Getting Kubernetes Logs Using Kubectl
The Kubernetes command-line tool, kubectl, acts as a Kubernetes’ cluster manager that helps us manage cluster resources, view logs, and deploy applications.
Kubernetes might have won the containerization spotlight by enabling us to orchestrate and manage containers more efficiently, but it didn’t come without logging challenges. Kubernetes is self-managed, so you lose your container logs when you delete or evict your pod or if it suffers a crash.
Worse still, in the event of a crash, the subsequent loss of the logs also means that there’s too little information left to help you diagnose the problem. You would have to run kubectl commands in your clusters individually to view their logs for other errors making centralized logging in Kubernetes even more critical.
Imagine running a web API microservice application with more than 100 instances, and an error occurred in one of them. Depending on the severity of the error, you either lose your logs completely, or you must dig through clusters using kubectl.
Locked-Down Servers
Some smaller companies, like startups, might have no issues granting their developers direct access to their servers. But in large enterprises, it’s usually the case that only select members of IT have access to the servers.
Since restricted team members also need access to logs, using a decentralized logging solution like Mezmo, formerly LogDNA, is an easy decision. Mezmo lets you define which parts of a log are available to individual team members.
Centralized Logging Gives You Greater Visibility
With a log centralization solution like Mezmo, you can tag every log entry with an identifier, group it, and store it in one place with its accompanying metadata.
This way, your logs are searchable in all your various groups, and you have an eagle’s eye view of your entire application and systems. You will see when seemingly unrelated issues are related, like when a spike in error messages corresponds to periods of memory shortage.
Tips For Creating an Efficient Log Management Strategy
Structure Your Log Data
Not separating log messages from raw text output in log files might seem like the easier non-harmful way to go, but the trade-off is that you don’t have the correct data available in your logs when you have to troubleshoot issues. You should get a log encoder that produces JSON log messages and configure JSON Appenders.
Centralize Your Logs with a Reliable Log Management Tool
Centralized logs may require extra steps to implement, but they make it up to you through more straightforward and efficient log management — especially when choosing the right logging tool. With all your logs stored in one safe location, you can better organize your log data and group logs from related sources.
An ideal log management tool should also be resilient, have fast live-tailing, and make your logs searchable.
Implement Active Log Security
Your log data is your window into the inner workings of your system, so you want to make sure that only authorized persons have access to it. Logs often contain information that should be confidential — and this information could be the target of malicious attackers.
In addition to restricting access to sensitive logs, ensure that you have encrypted your logs before transmitting them and have secured your communication protocols. The Mezmo agent and official code libraries enable encryption by default. Mezmo also encrypts your logs when storing them and only allows access to the web application over secure HTTPS.
Your Logging Approach Should be Scalable
The dilemma is between extensive logging and storage cost. Since logs only get bigger, especially when issues arise in your system, you can’t log everything. Also, it’s not good practice to delete old logs, especially when your system is still relatively the same. Still, the coverage of your logs has to be holistic enough.
You need a logging solution with the added benefit of helping you save costs as your logs grow. Mezmo does this by providing a unique log retention solution that can absorb unprecedented growth in the volume of your log data (as is the case when a problem arises) and manage old log data.
With Mezmo, you can archive old logs that you do not immediately need to save costs. The logs that you archive naturally take up less space while retaining their integrity and remaining accessible. You can choose to unarchive them at any time should the need arise.
Monitor Your Logs Proactively
Data is constantly changing, and so you should update your understanding of events and app performance. Don’t make log monitoring something you only do when you must troubleshoot a problem.
Logs should give you an expansive view of your system so that you can evaluate the efficiency of your services and learn ways to improve your users’ experience.
Conclusion
We gladly let serverless and decoupled services into our lives only to discover during the welcoming party that log aggregation nightmares had snuck in with them. It’s not hard to centralize your logs in a monolith since your log data often shares a standard format and has servers mapped to roles. On the other hand, centralizing logs in a microservice can be challenging — individual containers log data separately and often have formats that vary widely, have storage constraints, and are primarily unmapped.
Extensive centralized logging is the way to go, but it can generate large volumes of data that can be financially and technically overwhelming! Mezmo's solution to this is a logging tool that lets you keep all the data you need while simplifying your access to relevant information when you need it.
Start with Mezmo today and see how our end-to-end solution meets your logging needs.
