What is Log Rotation? How Does it Work?

As a system writes information to log files, the log file grows in size. We use log rotation to avoid large files that could create issues when opening them, which transfers log events to a new log file without interrupting the logging process. In many cases, the process involves renaming the current file to something new and then naming the new file the first file’s name. This process might seem simple, but it’s much more complicated than it sounds. The system responsible for logging must make the file change quickly without interruption.

For teams engaged in full stack monitoring, properly configured log rotation ensures critical logs don’t consume disk space or degrade performance across your observability pipeline.

How do Rotated Log Files Look?

“Log rotation” as a term is usually for Linux-based systems, but log rotation can happen in any environment. We’ll use the Linux log rotation to illustrate how it works. Linux has a directory named /var/log/ where we store logged events about the system. To see a list of log files in this directory, we can execute the following command:

ls /var/log

This command will show a list of logs in the log directory. You’ll notice several names of logs, some of them with the same names but with appended data values, and some with names and appended numerical values. The following is an example of what you might see in a Red Hat or CentOS environment:

cron

cron-20210807

cron-20210806

cron-20210805

Notice that the files have the same general name, but the name doesn’t have dates appended to it. . The logs with appended dates are older, already rotated log files. The file with the name “cron” and no date is the currently active file. In this example, the cron job log file is rotated every day. To can identify that it’s rotated every day because the appended date is a daily value. The old log file is given the name “cron” with the current date appended to it. After the rotated old file is stored, the new and existing cron file is created, which will contain log events for the current day.

Some logs compress to save disk space, so you might see a .gz file extension. As an example, you might see the following set of log files for a MySQL server running on the server:

mysql.log

mysql.log.1.gz

mysql.log.2.gz

mysql.log.3.gz

mysql.log.4.gz

Notice that the files have the .gz extension. This file extension indicates a compressed file. You can still access the log data within these files, but you need to extract the file content to view it. Compressed files reduce their size and save disk space, and they will take up less space when you archive or back up your logs to another location.

If you want to view the most recent logs in real-time, you can use the tail -f log command, which allows you to follow active log output even as rotation occurs.

What is The Purpose of Log Rotation?

We should log every event on a critical system. Logged events should include authentication and authorization errors, application errors, hardware failures, potential security incidents, and any other event that can prove helpful to administrators. This includes security logs, which are especially important for compliance and forensic analysis.

On a busy system, these logs can grow by terabytes per day. If a log file is already a terabyte at the end of the day, allowing it to grow another terabyte can create several issues. First, it can take up incredible amounts of disk space within only a few days. Second, opening the file can be impossible if the local device does not have the memory resources. It can also cause trouble for applications that must ingest the log data to process it and export it to a readable format.

Log rotation eliminates the main issues with large files. Administrators can rotate files at various times to keep them smaller in size and move them to another location. In many cases, administrators use log files in conjunction with analytic tools to visualize the data. The visualized data displays valuable information to analysts and administrators to take the necessary steps to remediate a specific problem. It also gives them status updates so that they know how efficiently the environment infrastructure is running.

What Happens to Old Log Files?

Although administrators rotate logs, the old files stay on the local system or a network device. When older files are no longer needed, they often still need to be stored for auditing purposes. To save on storage space, administrators take older files and archive them.

Archives are distinct from backups. When an administrator no longer needs to keep files on the server but must store them for potential audits, an archive will move the file from its current location to a new storage location. Because the cloud has virtually endless storage capacity, usually, this new location is the cloud. 

Administrators should also do backups to log files, making copies and storing them in another location. Backups recover the system should data corrupt, the system crashes, and in the aftermath of a cybersecurity event.“Log rotation” as a term is usually for Linux-based systems, but log rotation can happen in any environment. We’ll use the Linux log rotation to illustrate how it works. Linux has a directory named /var/log/ where we store logged events about the system. To see a list of log files in this directory, we can execute the following command: