Overriding the default Linux kernel 20-second TCP socket connect timeout

Posted 2020-08-26 Time Goes By

Whatever language or client library you‘re using, you should be able to set the timeout on network socket operations, typically split into a connect timeout, read timeout, and write timeout.

However, although you should be able to make these timeouts as small as you want, the connect timeout in particular has an effective maximum value for any given kernel. Beyond this point, higher timeout values you might request will have no effect - connecting will still time out after a shorter time.

The reason TCP connects are special is that the establishment of a TCP connection has a special sequence of packets starting with a SYN packet. If no response is received to this initial SYN packet, the kernel needs to retry, which it may have to do a couple of times. All kernels I know of wait an increasing amount of time between sending SYN retries, to avoid flooding slow hosts.

All kernels put an upper limit on the number of times they will retry SYNs. On BSD-derived kernels, including Mac OS X, the standard pattern is that the second SYN will be second 6 seconds after the first, then a third SYN 18 seconds after that, then the connect times out after a total of around 75 seconds.

On Linux however, the default retry cycle ends after just 20 seconds. Linux does send SYN retries somewhat faster than BSD-derived kernels - Linux supposedly sends 5 SYNs in this 20 seconds, but this includes the original packet (the retries are after 3s, 6s, 12s, 24s).

The end result though is that if your application wants a connect timeout shorter than 20s, no problem, but if your application wants a connect timeout longer than 20s, you‘ll find that the default kernel configuration will effectively chop it back to 20s.

Changing this upper timeout limit is easy, though it requires you to change a system configuration parameter and so you will need to have root access to the box (or get the system administrators to agree to change it for you).

The relevant sysctl is tcp_syn_retries, which for IP v4 is net.ipv4.tcp_syn_retries.

Be conservative in choosing the value you change it to. Like BSD, the SYN retry delays increase in time (albeit doubling rather than tripling), so a relatively small increase in the number of retries leads to a large increase in the maximum connect timeout. In a perfect world, there would be no problem with having a very high timeout because applications‘ connect timeouts will come into play.

However, many applications do not set an explicit connect timeout, and so if you set the kernel to 10 minutes, you‘re probably going to find something hanging for ages sooner or later when a remote host goes down!

I recommend that you set it to a value of 6, 7, or at most 8. 6 gives an effective connect timeout ceiling of around 45 seconds, 7 gives around 90 seconds, and 8 gives around 190 seconds.

To change this in a running kernel, you can use the /proc interface:

# cat /proc/sys/net/ipv4/tcp_syn_retries 
5
# echo 6 > /proc/sys/net/ipv4/tcp_syn_retries 

Or use the sysctl command:

# sysctl net.ipv4.tcp_syn_retries
net.ipv4.tcp_syn_retries = 5
# sysctl -w net.ipv4.tcp_syn_retries=6
net.ipv4.tcp_syn_retries = 6

To make this value stick across reboots however you need to add it to /etc/sysctl.conf:

net.ipv4.tcp_syn_retries = 6

Most Linux installations support reading sysctls from files in /etc/sysctl.d, which is usually better practice as it makes it easier to administer upgrades, so I suggest you put it in a file there instead.

(I see no reason you‘d want to reduce this sysctl, but note that values of 4 or less all seem to be treated as 4 - total timeout 9s.)

[ref]http://www.sekuda.com/overriding_the_default_linux_kernel_20_second_tcp_socket_connect_timeout