I hit an issue at work where SSH connections to some servers would stall for around 80 seconds. This seems like it only started happening recently, and it wasn’t happening for any of the systems I use on a day-to-day basis, but some of our developers were seeing the problem and determined that disabling GSSAPIAuthentication “fixed” the problem.
We predominantly use Ubuntu on the desktop, and CentOS derivatives on the servers. However, this stalled connection didn’t occur for all servers. It also didn’t happen when we made an ssh connection between CentOS servers, just when going from Ubuntu to CentOS. Doing some googling showed a bunch of blog posts and bug reports about this very issue, and the first ones I looked through all said the same thing – the fix was to disable GSSAPIAuthentication.
I wasn’t happy with that as a fix, simply because it didn’t sound relevant. We don’t use GSSAPIAuthentication at all; it’s on by default in the CentOS sshd configuration files, and it’s on by default in Ubuntu and Debian ssh configuration files, but it had never been a noticeable problem until recently. We hadn’t changed anything in our server setups. It was possible we’d changed something in our desktops (we upgrade Ubuntu every now and then, for example), but some fairly old installs were exhibiting the same problems to the same servers.
It felt like a DNS issue, but all the servers had a DNS record. Then I found a bug report that suggested it was related to Avahi, the mDNS daemon. Sure enough, disabling avahi on my client meant all ssh connections were fast. Then, after far too much incidental messing round, the penny dropped. The GSSAPIAuthentication was triggering a reverse DNS lookup on the host being connected to. If this existed in DNS, the connection was fine. If it didn’t, it was doing a reverse DNS lookup via mDNS, which ended up stalling for a long time.
Of course, now that I know this is the case, I can find any number of blog posts and bug reports that spell it out. I’m writing this one just to add some signal to the noise:
If your SSH connection is slow and looks like it’s stalling inside GSSAPI, check that you have valid forward (A) and reverse (PTR) DNS records for the host you are connecting to!. Disabling GSSAPIAuthentication will work, but it hides the real problem, which (in this case), is that for whatever reason your system is falling back to an mDNS lookup and is failing. Disabling GSSAPIAuthentication is simply hiding the symptom – you either need to fix your DNS setup, or tell your host to stop using mDNS.
And now I’m reminded of one of the best-titled blogs I read: Everything is a Freaking DNS Problem.
(This is an old post that I wrote in 2011, but never posted for some reason…)