Managing Huge Log Files with Logrotate Instead of Truncation

In an earlier article, we’d considered the problem of files that grow too large over time and need to be deleted or archived, and then replaced with new ones so that they can continue being filled. Log files are a prime example of this kind of thing, as are local mail files. The simplest way we saw earlier was to simply empty the file by truncating it if you don’t want to preserve any of the old logs. However, this approach has some huge problems.

First, you can’t keep track of the dozens of log files that your system generates all the time. You might truncate a log file now, but who will do it next week? What if you forget? Are you going to manually keep track of all files and go and truncate them one after the other?

Second, you might want to archive old files and optionally compress them instead of deleting them. This adds a new layer of complexity for manually executing the commands.

Different files have different schedules and limits. Your mail file might grow slower than your Apache log files. So you have to go to each of them, check if they’re growing large, archive and compress the ones that are huge, create new empty log files etc.

It goes without saying, that this is a problem that’s crying out for a better solution. And that solution is called “logrotate”.

Checking logrotate Default Configurations

The “logrotate” package is probably already installed on your system. It’s software that can automatically handle log files (or any files) that periodically need archival or removal. So see the default settings, open up the global configuration file using a text editor like this:

vi /etc/logrotate.conf

This is a wonderfully commented configuration file that tells you what happens by default to a log file if you use logrotate. For example, here’s a screenshot of my logrotate.conf file:

It says here that by default, logrotate will run once a week, keep 4 weeks worth of backlogs, will not compress the files etc. You can go lower down to see further settings. Each of these settings can be overridden as shown below.

How to Rotate a Specific File

For this example, I’m going to use logrotate on my mail folder. The file “/var/spool/mail/root” grows pretty large at regular intervals. To add this file to logrotate, first we go to the directory that contains all of logrotate’s configuration files:

cd /etc/logrotate.d

Each file in this directory instructs logrotate to run on specific files. I create a new file called “mail” and open it up for editing:

>mail
vi mail

Once in there, I paste the following into it:

/var/spool/mail/root {
    size 100M
    postrotate
        /bin/systemctl reload sendmail > /dev/null 2>/dev/null
    endscript
    su root root
}

The first line specifies the exact name and location of the file I want logrotate to work on, followed by code enclosed in curly brackets. Each line is a separate directive. Here is what each one means:

1. size – rotate the file only if it’s larger than 100M;
2. postrotate/endscript – run this script after rotating the file. In this case, I’m just reloading the application so that it resets its state, the inode pointer at which it currently writes etc.;
3.  su root root – run the logrotate process as root. Without this, you might get a permissions error on some systems.

There are other options you can use as well. The logrotate man page has everything you need.

Save this file and exit.

Testing logrotate

After saving your configuration file, you can test logrotate in verbose mode to see what action it will take without actually doing anything! Here’s the command for that:

logrotate -d mail

Here, “mail” is the configuration file I just created. The “-d” stands for debug. Here’s the output I get:

You can see that logrotate correctly detects my /var/spool/mail/root file and determines that it will be rotated based on the size. It also specifies what it will be renamed to and a host of other information. This will let you know exactly what will happen when logrotate runs so there are no surprises later down the line.

Logrotate is ideal for system administrators, frustrated with growing log files that keep eating up space on their system. Text files can be compressed very efficiently, so this is a great way to automatically solve the problem. Or even delete the old archived files entirely. It all depends on what you want to do.