Comments on: Using mbuffer to speed up slow zfs send | zfs receive

By: zfs send receive performance issues - The UNIX and Linux Forums

zfs send receive performance issues - The UNIX and Linux Forums — Mon, 29 Oct 2012 22:46:59 +0000

[...] [...]

By: Mozaik. » Blog Archive » Backing up large zpools

Mozaik. » Blog Archive » Backing up large zpools — Wed, 13 Jun 2012 18:03:15 +0000

[...] syncing the changes to USB took long (days), this took weeks. That’s when I learned about mbuffer to speed up zfs sync, but dedup and compression still took their toll. Plain uncompressed and [...]

By: John Ryan

John Ryan — Mon, 12 Mar 2012 15:13:31 +0000

I’ve automated this in a script like:
zfs send pool/tank@now | mbuffer -s 128k -m 1G 2>/dev/null | ssh -c arcfour128 remote.machine “mbuffer -q -s 128k -m 1G 2>/dev/null | zfs recv -F -v pool/tank”

also works with zfs send -i
tests show me arcfour128 speeds it up as well.

By: Jim Sloey

Jim Sloey — Fri, 11 Nov 2011 17:31:18 +0000

Is anyone using this in an automated script?

By: John Laur

John Laur — Sat, 29 Jan 2011 04:34:47 +0000

If you are using mbuffer on a local replication it’s both tedious, complicated and unnecessary to use two copies running on the same machine. Try this syntax:

zfs send … | mbuffer -s 128k -m 2G -o – | zfs receive …

the “-o -” puts mbuffer into file output mode, but then sends to stdout instead. Fancy!

By: Alasdair

Alasdair — Sun, 23 Jan 2011 00:28:26 +0000

Shawn – MBuffer probably won’t help in your case. If you’re getting a really slow zfs send/receive speed locally, it’ll be due to an issue that you’ll need to track down.

If you’re using dedupe, that can be really slow. You might have a dodgy disk/controller and/or drivers, or you could be on an old version of ZFS.

Whenever I see really slow zfs send/receive speeds in the past it has been due to IO errors, either due to dying disks or a disk/controller Solaris didn’t like. “iostat -En” error counts can be helpful here, and “iostat -x” can help identify the slow party.

By: Shawn

Shawn — Mon, 27 Dec 2010 05:10:05 +0000

so mbuffer is working great for network transfers however…How can it be used to speed up local zfs send and receives locally. I have not had any luck speeding up a local sync between to local pools. When I try it the buffers just fill and then I am back to trickle mode again. Any thoughts?

By: Anthony

Anthony — Tue, 12 Oct 2010 19:49:27 +0000

I’m glad I stumbled across this post.

I have an X4600 M2 that’s being replaced by an X4270 M2 with new disks – I have a dedicated interface on both for data migration. Lots of cycles and physmem on both sides. zfs send/receive over ssh defaults got me on the order of 24 MB/s. Using the blowfish cipher maxed at ~52 MB/s, and the arcfour cipher at ~67 MB/s. I’ve tested today using mbuffer as above, and consistently get 109-116 MB/s throughput with the sending buffer pegged 100% full and the receiving buffer mostly empty, occasionally rising up to 25% or so briefly before falling back to 0. I interpret this as indicating that the GigE interfaces are the bottleneck, and suspect that with 10GE interfaces I could easily double and likely triple throughput. I may try adding compression into the pipes to see how that affects time to completion.

By: sh0x

sh0x — Mon, 30 Aug 2010 03:23:35 +0000

Sweet! Thanks for the tip. My zfs send/recv went 45Mb/s using SSH for transport. I know there is network and cpu overhead with SSH, but the same transfer using mbuffer went 112MB/s (w/out SSH).

For my home whitebox setup I can’t ask for more throughput than that!

Ok I can, time to bundle NICs. :)

By: Andrew Gabriel

Andrew Gabriel — Mon, 02 Aug 2010 22:34:50 +0000

See the bug I raised on this: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6729347

The receiver does a giant read per TXG. Between these reads, the standard TCP network buffers fill and flow control off. When the disks and the network are similar bandwidths, you will see alternate network, then disk activity. By having a buffer on the receive side which can hold 5 seconds worth of data at the rate the disk and network can stream, you will then see the dataflow stream continuously. I didn’t see any significant benefit of a buffer at the sender, but a small one can’t do any harm.