The microservices-oriented architectural style is the realization of previous efforts to promote a heterogeneous yet orchestrated way of modeling complex software systems. As business use cases become more advanced, this style has grown in adoption. Logging microservices is a powerful way for developers to extract valuable information from the massive amount of data that modern applications process and produce.
Modern Systems = Complex Systems
In his Introduction to the Modeling and Analysis of Complex Systems, Professor Hirorki Sayama defines complex systems as “networks made of a number of components that interact with each other, typically in a nonlinear fashion.” If we acknowledge that a microservices architecture can be viewed as a network of semi-independent systems that communicate with each other, we can say that it is a complex system rather than a monolith. But what is the reason for this?
The first reason is the environment: microservices architectures respond to the requirements of interconnected systems, which are much more common than isolated systems these days. Whereas, monolithic architectures respond to simpler business requirements. Monoliths are useful for ideation, proof of concepts, and focused work like user stories and one-off projects. While complex applications can be built with a monolithic architecture, as the business case expands it’s often better to be prepared with a distributed architecture. Other benefits of using a distributed architecture are that critical components can be isolated and fault tolerance pathways can be defined, teams can be laser focused on delivering business value, and the entire system can more fluidly scale to meet the needs of a dynamic customer base.
Designing a distributed system is not as simple as creating a federation of monoliths. Microservices architectures create new challenges in areas that were not as difficult to design for a single system (such as consistency, communication, persistence, security, and deployment). In addition, microservices instances are usually considered disposable in order to allow scale-in and scale-out depending on the system load. Under the traditional logging scheme, this leads to numerous, ephemeral log files that must be accessed one at a time. Centralizing your logs into one, single source of truth is essential to understand how these connected systems interact and diagnose errors with the context you need.
The Power of Centralized Logs
In his book, I Heart Logs, Jay Kreps presents the concept of logs as a data structure that solves problems like consensus in order to determine what happened if there was an issue in a distributed system. This means that logging can be used as a tool for data integration by making all of a system’s data available even if it comes from heterogeneous sources. Logging is also a useful way to collect data that is produced in real time, such as data that is gathered by sensors that are connected to the internet and deployed in hundreds or thousands of instances at the same time.
By leaning on this view of logs as power tools for processing and storing data, developers move towards what Kreps calls “log-centric architectures.” In this type of architecture, small and specialized microservices can run in large numbers simultaneously in order to focus on their internal logic and delegate tasks such as consistency, data flow, and recovery. With so many systems interacting at once, centralized log management is the only way to easily find and make sense of the information that you need.
Microservices also provide data about their own operational status, meaning that a centralized logging platform can help developers find and address other operational problems using the power of logging. According to the principles of chaos engineering, organizations need to ask themselves how much confidence they can have in the complex systems they put into production. This will depend entirely on their techniques for gathering data and their ability to understand and take action on it.
The heterogeneous nature of microservices contributes to the chaos that can result from this type of architecture. Microservices are usually not implemented by the same team, so there can be several different ways to log data. Even if that’s not the case, business requirements are different for each microservice, so logs may not represent data in a consistent way. All of this means that the platform used to centralize the logging for microservices must be flexible enough to transform and aggregate the data in order to provide useful information, no matter if the format is standard or custom.
Centralizing log data is not a simple task, given that microservices architectures must comply with higher levels of scalability, traceability, flexibility, and security. A carefully-chosen platform that will connect the logs from your microservices will allow you to harness the power of the information that you can derive from this kind of architecture.
Scaling Microservices With Logging
The business-specific use cases for modern applications add new levels of complexity that must be addressed in new ways. Consequently, the development and deployment environments must become more complex in order to enable this flexibility and scalability. Fortunately, though, concepts like logging can be extended to provide powerful tools for managing the amount and heterogeneity of the data that today's organizations create.
