ssh speed tests

This page documents rough and ready speed tests I performed to investigate how best to copy files between two Linux computers across a fast network. The conclusions are that scp and rsync leave a lot to be desired, that sshfs is the slowest of the bunch by a factor of 6 for lots of small files, and that 3 times faster transfers than with scp can be obtained using tar over ssh while still retaining a secure connection.

Details of the tests

Tests were performed transferring data between two computers running Ubuntu 10.04.4 LTS (Lucid Lynx). The two computers contained dual core Intel Pentium D 3.2 GHz processors and 4 GB of memory, and were connected using Intel pro/1000 ethernet adapters over a gigabit ethernet connection through an old but high performance Cisco switch. The hard drive on each machine had a write speed of approximately 50 MB/s; this is rather slow, and so efficient nonblocking and buffering will be an important feature of these speed tests.

Two data sets were tested. An incompressible data set consisted of 200 files each of 10 MB of random data from /dev/urandom. A compressible data set consisted of the uncompressed GCC 4.6.3 source tree (610 MB).

All measurements are given as average ± standard deviation, where standard deviation is not provided if it is zero to the stated precision. These measurements are taken from a number (usually 5) of independent transfers.

Several methods of transfer were considered. These are listed below for transferring from src to dest on computer remote:

scp
scp -r src remote:dest
rsync
rsync -r src remote:dest, using rsync version 3.0.7 protocol version 30.
tar over ssh
tar -cf- src | ssh -q remote tar -xf- -Cdest
tar over netcat
ssh -f remote "nc -l 7111 | tar -xf- -Cdest" ; tar -cf- src | nc remote 7111. This in fact suffered some problems with ssh not backgrounding properly and netcat not correctly listening or correctly parsing end of file codes.
sshfs
sshfs remote:dest mnt ; cp -r src mnt ; fusermount -u mnt, using sshfs version 2.2, fuse library version 2.8.1, and fuse kernel interface version 7.12.

In all cases, the version of ssh used was OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009, and the version of (GNU) tar used was 1.22. All tests were run using the default options unless specified. For the version of ssh used, the default cipher is aes128-ctr and the default MAC is hmac-md5.

Initial tests and ssh ciphers

The first tests aimed to find the fastest ssh ciphers and compare them with the other methods. All tests in here are on incompressible data.

programcipher
defaultaes128-ctraes128-cbcblowfish-cbccast128-cbcarcfour128arcfour
scp 33.7±0.7 MB/s 33.6±0.1 MB/s 33.7±0.1 MB/s 33.8±0.1 MB/s 31.1±0.1 MB/s 34.4±0.7 MB/s 33.8±0.1 MB/s
tar over ssh 53.3 MB/s 53.4 MB/s 64.9 MB/s 52.9 MB/s 33.9 MB/s 85.3±0.1 MB/s 85.8±0.3 MB/s
rsync 32.4±0.2 MB/s 33.5±0.5 MB/s 33.7±0.6 MB/s 33.7±0.7 MB/s 33.3±0.3 MB/s 34.5±0.3 MB/s 33.8±0.5 MB/s
tar over netcat 58.5±2.0 MB/s

I conclude that:

  1. tar over either ssh or netcat are far faster than the other methods, possibly because tar and the transfer run in parallel, getting around the hard disk speed bottleneck.
  2. arcfour is the fastest cipher, and aes128-cbc is faster than the default aes128-ctr. Note that blowfish-cbc is not particularly fast.
  3. ssh is faster than netcat using either the aes128-cbc or arcfour ciphers.

ssh Message Authentication Codes (MACs)

The second tests aimed to find the fastest ssh Message Authentication Codes (MACs) and compare them with the other methods. All tests here are on incompressible data using the arcfour cipher found above to be the fastest in this case.

programMAC
hmac-md5hmac-md5-96hmac-sha1hmac-sha1-96hmac-ripemd160umac-64@openssh.com
scp 34.0±0.1 MB/s 33.9±0.4 MB/s 34.0±0.4 MB/s 34.0±0.1 MB/s 33.9±0.3 MB/s 34.0±0.1 MB/s
tar over ssh 85.7±0.2 MB/s 83.9±0.3 MB/s 73.2±0.2 MB/s 73.8±0.1 MB/s 62.1±0.1 MB/s 96.7±0.2 MB/s
rsync 33.6±0.7 MB/s 34.0±1.3 MB/s 34.1±0.4 MB/s 33.4±0.7 MB/s 33.8±0.6 MB/s 34.0±0.8 MB/s

I conclude that:

  1. For the scp and rsync methods, the bottleneck is elsewhere and the MAC makes little difference.
  2. For the tar over ssh method, the MAC makes a significant difference, with the new umac-64@openssh.com outperforming everything, and the default hmac-md5 coming a not-too-distant second.

Managing compressibility

The third tests aimed to find the best way to deal with compressibility. In addition to ssh's -C flag which enables compression within ssh, rsync offer the -z flag and tar offers both the -z and -j flags. The -j flag uses bzip2 for compression, while all the others use gzip.

All tests were performed using the arcfour cipher and umac-64@openssh.com MAC.

programdatacompression
none-C-z-j
scpcompressible 9.5±0.3 MB/s 6.5±0.1 MB/s
tar over sshcompressible 100.5±0.9 MB/s 16.7±0.1 MB/s 17.3±0.2 MB/s 5.0 MB/s
rsynccompressible 12.5±0.2 MB/s 11.7±0.2 MB/s 11.4±0.1 MB/s
sshfscompressible 4.4 MB/s 3.3 MB/s
scpincompressible 34.7±0.3 MB/s 13.8 MB/s
tar over sshincompressible 102.7±1.8 MB/s 13.7±0.1 MB/s 13.3±0.2 MB/s 2.5 MB/s
rsyncincompressible 34.5±0.5 MB/s 33.7±0.5 MB/s 33.7±0.5 MB/s
sshfsincompressible 33.2±0.5 MB/s 10.9 MB/s

I conclude that:

  1. On these computers where the connection speed is so fast as to not be limiting, not using compression beats using compression every time even when what is being transferred is compressible.
  2. At least on these computers, tar over ssh (without compression) is significantly faster than every other method. Again, this may be because the two separate processes get around the limitations of the slow hard drive.
  3. sshfs is terrible at transferring a large number of small files (as seen from the compressible case).
  4. bzip2 is too slow to be used for on-the-fly compression.
  5. The speed of rsync appears to be independent of whether and what type of compression is used.
Valid XHTML 1.0 Strict