内核诊断interrupt took too long

Posted 2021-02-09 linengier

The linux kernel gathers samples using ‘perf’ performance monitor without affecting the latencies.
These include getting interrupt times. If interrupts take too long, a similar message to this prints:

kernel: [ 6491.061361] perf: interrupt took too long (6650 > 6452), lowering kernel.perf_event_max_sample_rate to 30000

Meaning
This essentially means that the machine was stuck on an interrupt for a long amount of time. This can be caused by a number of reasons, including:

DISK IO interrupt taking long would be caused by a faulty, slow or overloaded disk. Alternatively this can be caused by an issue with a disk or raid controller.
Network IO interrupt taking too long would be caused most often by network driver issues being suboptimal. Alternatively, this can be caused by network issues, although the protocol switching should theoretically be preventing it.
Troubleshooting
The disk IO can be easily checked with disk IO stats (sysstat-sar and/or iostat) and confirmed. If the disk IO is not the reason for slow interrupts, the network IO will be. For this, the problem needs to be checked on the network and/or kernel side.

First point of troubleshooting should include checking kernel messages in /var/log/(messages|syslog) as well as dmesg. Should these show tracebacks from vmxnet driver, the interrupt taking too long will be caused by a faulty network driver. Please contact the network card provider for this, or try upgrading to the latest available stable kernel.

If there is no issue with the kernel drivers, the network would be most liekly at fault, most likely first hop. This then needs to be checked on the network side.

Keywords
network interrupt took too long latency

Timestamp
参考网址