How To Change Log Rate Limiting In Linux

Posted 2021-01-20 数据库技术和故事

By default in Linux there are a few different mechanisms in place that may rate limit logging. These are primarily the systemd journal and rsyslog rate limits that are in place by default.

Here we cover modifying or removing rate limiting for logging.

 

Why Rate Limiting?

Rate limitations on logging are in place to prevent logging from using excessive levels of system resources. To log an event, it needs to be written to disk which uses system resources. If there are too many of these events coming in that need to be recorded to disk they can overwhelm a system and cause more important services to respond slowly or fail.

For this reason it is generally not recommended to completely disable rate limiting, but to tweak it as required. At the same time we do not want to drop important messages that may be required to generate a critical alert, so a balance needs to be found.

Systemd Journal Rate Limiting

How do we know if the journal limits are actually causing us to drop log messages? Generally you will see similar messages in the log files as below.

Jan  9 09:18:07 server1 journal: Suppressed 7124 messages from /system.slice/named.service

In this particular case we have a DNS server running Bind which is logging all DNS queries. 7124 messages were suppressed and dropped (not logged) because they were coming in too fast in this example.

By default systemd allows 1,000 messages within a 30 second period.

The limits are controlled in the /etc/systemd/journald.conf file.

RateLimitInterval=0
RateLimitBurst=0

If more messages than the amount specified in RateLimitBurst are received within the time defined by RateLimitInterval, all further messages within the interval are dropped until the interval is over.

You can modify these values as you see fit, you can completely disable systemd journal logging rate limiting by setting both to 0.

If you make any changes to /etc/systemd/journald.conf you will need to restart the systemd-journald service to apply the changes.

systemctl restart systemd-journald

Rsyslog Rate Limiting

The systemd journal limit is hit before any default rsyslog limits as its default limits are smaller. By default rsyslog will accept 20,000 messages within a 10 minute period.

Therefore if you increase the rate limiting of the systemd journal logging as shown above you may then start to receive similar messages in your syslog logs as shown below.

....
Jan  9 22:42:35 server1 rsyslogd-2177: imjournal: begin to drop messages due to rate-limiting
Jan  9 22:51:26 server1 rsyslogd-2177: imjournal: 143847 messages lost due to rate-limiting
...

The first message states that messages will be dropped as the limit has been reached, and once the interval is over (after 10 minutes by default) the amount of messages that were lost due to rate limiting will then be logged.

The limits are controlled in the /etc/rsyslog.conf file.

$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 3000

For further information see the imjournal rsyslog documentation.

Again you can modify these values as you like, and they can be completely disabled by setting both to 0.

If you make any changes to the /etc/rsyslog.conf file you will need to restart the rsyslog service to apply the changes.

systemctl restart rsyslog

Summary

As shown we can check our log files to find out if logs are being dropped due to either systemd journal or syslog rate limits. The systemd journal default rate limit is much lower than the syslog default rate limit so it will be triggered first. Once you increase the rate limiting on the systemd journal logging you may then start to experience additional rate limiting by syslog, which can then also be increased if required.

 

实例配置：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

vim /etc/rsyslog.conf       #### MODULES ####   # The imjournal module bellow is now used as a message source instead of imuxsock. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal $imjournalRatelimitInterval 0 $SystemLogRateLimitInterval 0 #$ModLoad imklog # reads kernel messages (the same are read from journald) #$ModLoad immark  # provides --MARK-- message capability         systemctl restart rsyslog