Using mbuffer to speed up slow zfs send | zfs receive
July 18th, 2010
So, you find yourself doing a zfs send | receive, perhaps a large incremental send. You find the transfer process going really slowly - trickling along at less than 1MB/sec. Yet, you know the sender and the receiver is capable of far far more than this. What’s the deal?
Well, basically, zfs receive is bursty - it can spend ages computing something, doing no receiving, then blat the data out to disk. The issue with this is that it stalls the sender, resulting in a bursty and slow transfer process.
The solution is to deploy mbuffer into the mix. MBuffer will buffer the stream, which you can do at both ends. While ZFS receive can’t receive, mbuffer buffers, when zfs receive can receive, mbuffer sends it data as fast as it can. Let’s see an example:
# Start the receiver first. This listens on port 9090, has a 1GB buffer,
and uses 128kb chunks (same as zfs):
mbuffer -s 128k -m 1G -I 9090 | zfs receive data/filesystem
# Now we send the data, also sending it through mbuffer:
zfs send -i data/filesystem@1 data/filesystem@2 | mbuffer -s
128k -m 1G -O 10.0.0.1:9090
You’ll get a lovely output such as:
in @ 15.8 MB/s, out @ 8923 kB/s, 1.0 GB total, buffer 6% full
And hopefully, you’ll find your zfs send|receive suddenly go a lot lot quicker.
Here is the link to download the useful mbuffer program.
I personally found this approach decreased my send time for 40GB from over 4 hours down to around 30 minutes. Hurrah! :)
Entry Filed under: General
4 Comments Add your own
1. Eric Sproul | July 21st, 2010 at 8:34 pm
This looks interesting, but you don’t mention what transport you were using before mbuffer. Assuming ssh, how much of the speedup can be attributed to mbuffer versus not having to do the ssh crypto?
Folks might also want to remember that the mbuffer traffic is going over the network in the clear, which might be a concern in some environments.
2. Alasdair | July 28th, 2010 at 7:26 am
Hi Eric,
Obviously people can still use mbuffer in conjunction with SSH so you continue to get an encrypted transport stream and you’ll benefit from the buffer as well.
This is actually the case for what I was using mbuffer for; I just dropped mbuffer into the backup script on the receiving side, and SSH is still the transport stream. This gave a large speed up.
SSH Encryption gives you a fairly constant transfer speed. The issue we had specifically was that ZFS Receive would hang about not receiving any data whilst performing computations, then do an enormous burst of activity. Adding mbuffer helps tremendously for this case as it buffered data during the stall, then could feed ZFS receive as fast as it could go during the burst of activity.
Alasdair
3. Andrew Gabriel | August 2nd, 2010 at 10:34 pm
See the bug I raised on this: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6729347
The receiver does a giant read per TXG. Between these reads, the standard TCP network buffers fill and flow control off. When the disks and the network are similar bandwidths, you will see alternate network, then disk activity. By having a buffer on the receive side which can hold 5 seconds worth of data at the rate the disk and network can stream, you will then see the dataflow stream continuously. I didn’t see any significant benefit of a buffer at the sender, but a small one can’t do any harm.
4. sh0x | August 30th, 2010 at 3:23 am
Sweet! Thanks for the tip. My zfs send/recv went 45Mb/s using SSH for transport. I know there is network and cpu overhead with SSH, but the same transfer using mbuffer went 112MB/s (w/out SSH).
For my home whitebox setup I can’t ask for more throughput than that!
Ok I can, time to bundle NICs. :)
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed