Sorry for taking a while to respond!

We are using Crucial SSDs, http://www.crucial.com. We’ve found them to be reliable, reasonable performance, cost effective, and they supply a 3 year warranty. A good all-rounder, even for enterprise workloads.

It’s not too hard if you’re using NFS since NFS is stateless. You also have to use SAS drives and SAS arrays as you can plug a dual-head SAS array into two physical servers and both nodes can see the disks at the same time.

You then just need to write some scripts to:

a. Detect failure (we use NRPE to check writing to an nfs mount mounted via a loopback cable between the primary and the secondary node)
b. Have the secondary pull the plug on the primary (we do this via IPMI)
c. Forcibly import the zpool and bring up the IPs

To improve the c failover time we use VNICs with the same MAC on both machines, so hosts don’t have to learn new MAC addresses.

It’s not a huge amount of code.

With a big array (20TB+) its not fast enough to handle storing virtual machine disk images on it, as the failover can take > 180 seconds which is long enough for Windows and Linux to declare their disks dead and get very unhappy. But it’s fine for NFS clients which recover quite happily.

Basically the setting didn’t help the situation I was experiencing. The sd timeouts don’t seem to be used by the mpt_sas driver. So adjusting them is completely pointless.

The way we reduced the possibility of bad disks impacting our storage is to preemptively zpool offline disks that show any signs of misbehaviour, such as taking drives out of service that exhibit a single hard error, or show any SMART errors. So far this has worked quite well.

But ultimately the storage subsystem and/or drivers in the OS need improving. There are bugs open about this:

https://www.illumos.org/issues/1553
https://www.illumos.org/issues/1069

Yes I read your comment in the oi_a51a release: “if you need clustering, then you should be able to justify the budget for it. Clustering on the cheap is a recipe for disaster”

My justification same reason both illumos and openindiana exist – need I say more?

Thanks in Advance

Can you please comment further on the failure of this setting using mpt_sas? It sounds like what you’re saying is, this setting should in theory work, but the driver is causing issues and it never gets to the ZFS level to use this setting. Is that right? Did you test using the known bad drive from the beginning of the article? Any other details would be appreciated if you have them.

Thanks

I solved the problem by yanking that drive out and throwing it in the bin… Didn’t know about this then! :)

Thanks for the comment about the timeout, it’s a good suggestion.

However in testing with failing harddrives (on mpt_sas anyway), we see that the sd timeouts are completely ignored so my entire post above is moot!

— richard

As far as I know, when Oracle closed down OpenSolaris, they stopped contributing to OpenHA, which was only working with OpenSolaris 2009.06. The source code of OpenHA is now moved to Illumos gate, but activities in that project is low. You can’t even compile it nowadays. Considering Oracle Solaris Cluster 3.3u1 doesn’t run on Illumos based distro, or even Solaris 11 Express, I wonder which software package you used to setup HA ZFS and HA NFS for VM storage. (The only thing I can think of is to license RSF-1 just like NexentaStor HA)

Thanks