A few weeks ago, I ran into a weird issue where WinSCP would load only the /home/ directory on a Linux-based terminal, and the ls command in PuTTY would return a list of only the first few files in any directory. I’d never encountered anything like it, but since I finally figured out the problem, I thought it was well worth documenting.
The problem
It all began when one of our techs at work was installing a new terminal and needed me to connect to it to load the customer’s application. I fired up WinSCP, entered the terminal’s IP address, clicked Login, and waited. WinSCP reported that it was connecting and that it was loading the directory contents, but after a while it timed out. I tried again; same result.
Odd, I thought, running a continuous ping to check the terminal’s network connectivity. I could see a packet was getting dropped here and there, but it was nothing excessive, and definitely not something that should be preventing WinSCP from connecting. I was able to connect to the terminal via PuTTY and changed to my target directory, but when I ran the ls command, I got a list of only seven or eight files, and that was it.
I could connect to other terminals on the customer’s system without any trouble but updated WinSCP just to be sure we had the latest and greatest, but that didn’t make any difference. We chalked up the problem as a possible bad flash card and shipped out a replacement card which we tested in a terminal in our office for the tech to install. Even after he plugged in the new card, however, we continued to see the same behavior, which led me to believe that the problem was not with the terminal itself, but with the customer’s network. But what could it be?
The research
You would think that doing a search for “PuTTY and WinSCP won’t list contents of remote directory” would turn up some helpful results, but my initial efforts to figure out what was happening were fruitless. It was only after a couple of hours of searching that I finally ran across an Apple Community thread from 2011 that finally got me pointed in the right direction.
In that thread, the original poster complained about a Linux-based box “hanging” when they issued basic commands when connecting to the machine via SSH. Interesting…that sounded kind of like what I was seeing in PuTTY.
One of the users who replied, Camelot, stated that the problem was “almost certainly a MTU issue,” explaining that the original poster’s VPN was probably not resetting the MTU properly, resulting in “large packets…getting truncated and/or dropped—that’s why small transactions…work, but large ones fail.”
The solution
I’m not going to get into the weeds with a detailed explanation of MTU (maximum transmission unit), but even though no VPN was in use between the customer’s server and terminal, a MTU mismatch turned out to be exactly the problem. Somewhere on the customer’s network between the server and the terminal, there is a router or switch whether the MTU is set to less than the default packet size of 1500 used by both the customer’s Windows server and our Linux-based terminal. This was wreaking havoc with my attempts to transfer data, even basic directory listings, using WinSCP and PuTTY. This happens because the 1500-byte packet gets broken into two smaller fragments; the first makes it through, and the rest gets dumped.
Serenity Networks has a very good explanation of how to determine the MTU between two devices. It boils down to this:
- At a command prompt, enter ping ip -f -l 1500. If you see the message Packet needs to be fragmented but DF set, the MTU is less than 1500; proceed to step 2.
- Repeat the command from step 1, but reduce the last number by 10 (from 1500 to 1490), and keep doing this until you find a number that gives you normal ping results.
- Now, repeat the same command, but increase the last number by 1. Keep doing this until you find a number that once again gives you the Packet needs to be fragmented but DF set message; the last number that gives you normal ping results is your MTU.
In the particular environment that I was troubleshooting, MTU needed to be set to 1348. Since our terminal is Linux-based, I didn’t have to bother getting the customer’s IT team involved to try to track down the weird piece of networking equipment, instead, I was able to set the MTU on the terminal’s NIC by doing the following based on a nixCraft post:
- On terminal, enter sudo nano /etc/network/interfaces.
- Under the gateway line for eth0, enter mtu 1348, then press CTRL + X and Y to save the file.
- Enter sudo reboot to restart the terminal.
Once this was done, I entered ip 1 to confirm that the terminal was using the correct MTU, then fired up WinSCP. WinSCP connected immediately and displayed the remote directory on the terminal with no complaints, and I’ve had no further problems with it.