TCP Window Scaling

Problem:

When sending mail to a particular target, we were unable to send messages above a certain size (around 500KB). The diagnostic on the relay was:

MDeferred: Connection timed out with <site>

Messages would go back into the queue, and after the predictable retry intervals, time out.

The problem server had recently replaced an otherwise-working Fedora Core 3 server.

Tests from a system running Windows XP configured as a DMZ peer of the problem server succeeded even as parallel tests from the CentOS 5 system failed. This to our mind ruled out all switches, firewalls, and network paths in between the problem server and the remote target.

Detailed tcpdump traces showed bulk transmissions that would progress for a certain amount of data, and then stall with requests for retransmits from the remote side; however even though we could watch the retransmitted packets leave our system, traces on their side showed the retransmits never arrived there. Sendmail would go into a application- defined wait and eventually the application would kill the connection with the above diagnostic in the logs.

Solution:

http://serverfault.com/questions/444515/cant-send-large-messages-using-sendmail-centos-5

These changes have been made to /etc/sysctl.conf:

net.ipv4.tcp_rmem = 4096 87380 174760
net.ipv4.tcp_wmem = 4096 16384 131072
net.ipv4.tcp_window_scaling = 0

...then reboot or execute:

# sysctl -p

It is our view that the load on the system is small enough that a loss of performance due to disabling TCP Window Scaling is an acceptable side-effect.

We are monitoring for undesired side-effects.

Discussion

Over the past few years I have had intermittent issues with Sendmail on CentOS 5 (several revisions from 5.0 through to today's 5.7) like this, where large connections (either single large messages, or a transaction with many small messages) would hang up and time out. Most of the time I was too happy to blame the device on the other end of the connection, something all to easy to do when the other device is a Windows system running home-grown software or some off-the-shelf bargain basement black box.

This solution makes me wonder if the problem has been TCP window scaling the whole time.

I also wonder if this is maybe a wider-spread problem than anyone knows, since in this case the customer was large enough to throw a lot of resources at the issue -- one conference call had something like 12 people on it from network providers, firewall support people, applications specialists, and management from both ends (oh yes and me). We were able to trace out the behavior pretty accurately but were unable to identify TCP window scaling as the problem. The problem was probably aggravated by a firewall device in the chain throwing away the re-transmitted packets for security reasons.

A smaller client would have hacked together a work-around (several were suggested, including relaying mail through a third party who could talk to both ends) and then forgotten about the issue.

It would be interesting if someone with a RedHat support contract could reproduce the issue and then engage RedHat to see if there really is an underlying kernel issue.