Archive for November, 2008
Running 32bit ASP.NET 1.1 Apps on 64bit Windows Server 2008
We don’t treat Microsoft Windows as a second class citizen here at EveryCity – after all it’s a legitimate and very popular hosting platform. Yes, I’m a Solaris convert; it’s my preferred weapon of choice, but Windows Server ain’t that bad.
Indeed, it’s father, Mr David Cutler gets my total respect – the NT kernel is a masterful stroke of genius, even if the Win32 userland deposited on top is a steaming pile of cack. If you haven’t read Showstopper, which is a vivid account of the creation of NT, go out and locate a copy. It’s a cracking good read and I thoroughly enjoyed it. One of my favourite blog authors, Jeff Atwood of Coding Horror, gave it a pretty good writeup.
Anyway, I digress. A whole bunch of months ago we started putting Windows Server 2008 into production. IIS 7 is rather funky – native rewrite rules, FastCGI PHP support, some sane defaults (New application pool for each website) make it a compelling upgrade from Server 2003. Also, so far most ASP.NET applications dumped on it "just work".
However occasionally you have to move a web application from a 32bit Server 2003 box to a 64bit Server 2008 box. If that application has some 32bit DLLs, then you’re going to need to enable a 32bit support, otherwise the app will break at some point.
Thankfully it’s easy – head on over to the Application Pools area, right click the appropriate application pool, click "Advanced Settings", and enable 32bit support:
If your App is an ASP .NET 1.1 application, it should in theory run under .NET 2.0 without too much hassle, but you may just want to select "Classic" as your "Managed Pipeline Mode". I am not a .NET developer, and do not profess to know what this does, but I had to enable it for an ASP .NET 1.1 app I moved recently. According to one of my Microsoft loving friends, don’t install .NET 1.1 on Server 2008 – it won’t work and shoehorning it on may break your box. Apparently.
Add comment November 28th, 2008
Transport Tycoon Deluxe, OpenTTD
During my teenage years, I used to spend vast amounts of time playing Transport Tycoon, followed by Transport Tycoon Deluxe. It was an absolutely fantastic game written by Chris Sawyer. Actually, going a number of years back to my very early teens, I used to play Railroad Tycoon on the Amiga, which was a similar sort of thing, but only for trains, made by Sid Meier.
Recently, I’ve been playing OpenTTD, an open source reimplementation of Transport Tycoon Deluxe, which runs perfectly on modern day operating systems, including Win2k/XP/Vista, Linux, MacOS and others. It features many enhancements, and removes many of the limits that were present in the original game (Such as the maximum number of trains, stations, etc).
It’s definitely worth checking out, although you need files from the original Transport Tycoon Deluxe. You can pick up copies of it on Amazon/eBay for next to nothing. So if you’ve been wondering why I’ve not posted over the past few days, now you know :P
1 comment November 24th, 2008
SNMP Monitoring of LSI MegaRaid Cards
We use LSI 3041E raid cards (which use the SAS1064ET chipset) in a bunch of our Sun x2100 and x2200 Servers, and naturally you want a simple and straight forward method of monitoring the raid status.
Checking the Raid Status on Linux
On Linux, we opted for the simple and easy to use mpt-status utility, which you can script easily. You can install it straight from Debian apt-get, although it doesn’t seem to be in the normal CentOS Yum repositories. It’s pretty easy to use, as this demonstrates:
# mpt-status
open /dev/mptctl: No such file or directory
Try: mknod /dev/mptctl c 10 220
Make sure mptctl is loaded into the kernel
# modprobe mptctl
# mpt-status
You seem to have no SCSI disks attached to your HBA or you have
them on a different scsi_id. To get your SCSI id, run:
mpt-status -p
# mpt-status -p
Checking for SCSI ID:0
Checking for SCSI ID:1
Checking for SCSI ID:2
Found SCSI id=2, use ''mpt-status -i 2`` to get more information.
# mpt-status -i 2
ioc0 vol_id 2 type IM, 2 phy, 135 GB, state OPTIMAL, flags ENABLED
ioc0 phy 1 scsi_id 4 SEAGATE ST314654SSUN146G 022D, 136 GB, state ONLINE, flags NONE
ioc0 phy 0 scsi_id 3 SEAGATE ST3146855SS 0002, 136 GB, state ONLINE, flags NONE
You can then write a simple bash script to check that the status is “OPTIMAL”, and set up some kind of remote monitoring to access it via SNMP or Nagios NRPE.
Checking the Raid Status on Windows
On Windows Server 2003/2008, for remote monitoring your best (only?) option is to install Windows SNMP, and install LSI MegaRaid Storage Manager with the SNMP plugin. You can download the LSI MegaRaid Storage Manager from LSI’s website. Once SNMP and the MegaRaid SNMP plugin are installed, you should be able to snmpwalk your Windows server:
root mibs (mon01): snmpwalk -v1 -c public w01.someserver.everycity.co.uk | head SNMPv2-MIB::sysDescr.0 = STRING: Hardware: x86 Family 15 Model 67 Stepping 3 AT/AT COMPATIBLE - Software: Windows Version 5.2 (Build 3790 Multiprocessor Free) SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.311.1.1.3.1.2 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (819029) 2:16:30.29 SNMPv2-MIB::sysContact.0 = STRING: SNMPv2-MIB::sysName.0 = STRING: W01-SOMESERVER ...
Great! Now, you need the LSI Mib Files. Technically you don’t "need" them to check the relevant SNMP OIDs, but it’s helpful to know what you’re querying. I obtained them by downloading and digging through the Linux version of LSI MegaRaid Storage Manager. At the time of writing this was MSM_Linux_28800.zip Inside this is a tar.gz file called MSM_linux_installer-2.88-00.tar.gz. Inside this are 4 RPM files. This is starting to remind me of Russian dolls. Inside sas_ir_snmp-3.16-1002.i386.rpm and sas_snmp-3.16-1002.i386.rpm (Which you can extract with "rpm2cpio *.rpm | cpio -idmv"). Finally you can get your two MIB files:
./etc/lsi_mrdsnmp/sas/LSI-AdapterSAS.mib ./etc/lsi_mrdsnmp/sas-ir/LSI-AdapterSASIR.mib
If you don’t want to arse around, and lets face it who enjoys arsing around, please enjoy LSI-AdapterSAS.mib and LSI-AdapterSASIR.mib.
On a typical ucd/net SNMP install, you’d place these in /usr/share/snmp/mibs. There’s a good guide on ensuring the mibs get loaded when you call tools such as snmpwalk, which means instead of getting:
# snmpwalk -v1 -c public w01.someserver.everycity.co.uk .1.3.6.1.4.1.3582 | head -n 50 SNMPv2-SMI::enterprises.3582.4.1.1.1 = STRING: "W01-SOMESERVER" SNMPv2-SMI::enterprises.3582.4.1.2.1 = STRING: "Microsoft Windows 2003 Service Pack 2.0" SNMPv2-SMI::enterprises.3582.4.1.3.1.1 = STRING: "1.23-02" SNMPv2-SMI::enterprises.3582.4.1.3.2.1 = STRING: "lsi_mrdsnmpagent.dll" SNMPv2-SMI::enterprises.3582.4.1.3.3.1 = STRING: "3.16.0.1" SNMPv2-SMI::enterprises.3582.4.1.3.4.1 = STRING: "28th May 2008" SNMPv2-SMI::enterprises.3582.4.1.9.1.1 = STRING: "LSI Corporation" SNMPv2-SMI::enterprises.3582.5.1.1.1 = STRING: "W01-SOMESERVER" SNMPv2-SMI::enterprises.3582.5.1.2.1 = STRING: "Microsoft Windows 2003 Service Pack 2.0" SNMPv2-SMI::enterprises.3582.5.1.3.1.1 = STRING: "1.14-01" SNMPv2-SMI::enterprises.3582.5.1.3.2.1 = STRING: "lsi_mrdsnmpagent.dll" SNMPv2-SMI::enterprises.3582.5.1.3.3.1 = STRING: "3.16.0.1" SNMPv2-SMI::enterprises.3582.5.1.3.4.1 = STRING: "28th May 2008"
You get:
LSI-MegaRAID-SAS-MIB::hostName.1 = STRING: "W01-SOMESERVER" LSI-MegaRAID-SAS-MIB::hostOSInfo.1 = STRING: "Microsoft Windows 2003 Service Pack 2.0" LSI-MegaRAID-SAS-MIB::mibVersion.1 = STRING: "1.23-02" LSI-MegaRAID-SAS-MIB::agentModuleName.1 = STRING: "lsi_mrdsnmpagent.dll" LSI-MegaRAID-SAS-MIB::agentModuleVersion.1 = STRING: "3.16.0.1" LSI-MegaRAID-SAS-MIB::releaseDate.1 = STRING: "28th May 2008" LSI-MegaRAID-SAS-MIB::copyright.1 = STRING: "LSI Corporation" LSI-megaRAID-SAS-IR-MIB::hostName.1 = STRING: "W01-SOMESERVER" LSI-megaRAID-SAS-IR-MIB::hostOSInfo.1 = STRING: "Microsoft Windows 2003 Service Pack 2.0" LSI-megaRAID-SAS-IR-MIB::mibVersion.1 = STRING: "1.14-01" LSI-megaRAID-SAS-IR-MIB::agentModuleName.1 = STRING: "lsi_mrdsnmpagent.dll" LSI-megaRAID-SAS-IR-MIB::agentModuleVersion.1 = STRING: "3.16.0.1" LSI-megaRAID-SAS-IR-MIB::releaseDate.1 = STRING: "28th May 2008"
This is obviously much more readable and understandable. You can also view the comments in the MIB file, for example:
pdDiskPredFailureCount OBJECT-TYPE
SYNTAX INTEGER
ACCESS read-only
STATUS optional
DESCRIPTION "Number of disk devices in this adapter those are critical"
alarmStatus OBJECT-TYPE
SYNTAX INTEGER{
status-ok(1),
status-critical(2),
status-nonCritical(3),
status-unrecoverable(4),
status-not-installed(5),
status-unknown(6),
status-not-available(7)
}
Depending on the model of your RAID card, the most useful OIDs to monitor are:
# snmptranslate -IR -On vdDegradedCount .1.3.6.1.4.1.3582.4.1.4.1.2.1.19 # snmptranslate -IR -On vdOfflineCount .1.3.6.1.4.1.3582.4.1.4.1.2.1.20 # snmptranslate -IR -On pdDiskFailedCount .1.3.6.1.4.1.3582.4.1.4.1.2.1.24 # snmptranslate -IR -On pdDiskPredFailureCount .1.3.6.1.4.1.3582.4.1.4.1.2.1.23
Or:
# snmptranslate -IR -On vdDegradedCount .1.3.6.1.4.1.3582.5.1.4.1.1.3.1.20 # snmptranslate -IR -On vdOfflineCount .1.3.6.1.4.1.3582.5.1.4.1.1.3.1.21 # snmptranslate -IR -On pdDiskFailedCount .1.3.6.1.4.1.3582.5.1.4.1.1.3.1.25 # snmptranslate -IR -On pdDiskPredFailureCount .1.3.6.1.4.1.3582.5.1.4.1.1.3.1.24
All of which should be zero. You can script snmpget or use nagios’s snmp plugin directly to monitor these values.
Last bot not least, checking on Solaris
Solaris is the easiest of all:
# raidctl -l
Controller: 1
Volume:c1t0d0
Disk: 0.1.0
Disk: 0.2.0
# raidctl -l c1t0d0
Volume Size Stripe Status Cache RAID
Sub Size Level
Disk
----------------------------------------------------------------
c1t0d0 135.9G N/A OPTIMAL OFF RAID1
0.1.0 135.9G GOOD
0.2.0 135.9G GOOD
Enjoy!
5 comments November 18th, 2008
Solaris as an iSCSI Server with ZFS
iSCSI is a rather funky protocol, that allows you to export a block device (eg, a harddrive partition, zfs zvol or a regular file) as a scsi device, over TCP/IP. Typically it’s used in environments where you have storage on a device (Such as on a SAN) and want to share it with a client/server, for performance reasons, or simply to provide more storage.
In this particular instance, we have a Sun Fire x4500 (Thumper) SAN with ample storage, running Solaris 10 update 6, and I wanted to export a chunk of it to a Windows box.
iSCSI terminology
iSCSI Target: iSCSI Server
iSCSI Initiator: iSCSI Client
I can understand why they chose “target” and “initiator”, but when you’re starting out it’s easier to think in terms of clients and servers.
Installing requisite Solaris Packages
If you’ve performed a Core install, you might not have the required packages installed (You can check with pkginfo). If they’re missing, fetch your Solaris CD. Install and enable as follows:
pkgadd -d . SUNWiscsir SUNWiscsitgtr SUNWiscsitgtu SUNWiscsiu svcadm enable iscsitgt
Sharing a ZFS zvol block storage unit
We need a “block device” to export over iSCSI. ZFS lets you create “sparse” zvols which start off small, and expand to full size as you use them. Let’s add a 10GB sized one:
zfs create -s -V 10G zpool01/myiscsivol
We can now share it as simply as:
zfs set shareiscsi=on zpool01/myiscsivol
Easy huh?
Configuring your iSCSI share with iscsitadm
Your weapon of choice when configuring iSCSI shares is iscsitadm. Unfortunately it’s a slightly obtuse tool, which doesn’t give you friendly feedback when you type the wrong thing in. But there are worse tools out there. You can view details of your iSCSI share with:
iscsitadm list target -v
Restricting iSCSI to a particular IP Address
If you want to restrict your iSCSI share to a particular network interface/ip address, you can do it by creating a Target Portal Group (iSCSI terminology is… odd.) For example:
iscsitadm create tpgt 1 iscsitadm modify tpgt -i 10.0.0.1 1 iscsitadm list tpgt -v 1
This creates a target portal group with a number of 1, and specifies that it should use 10.0.0.1 as it’s interface. Next we associate this tpgt with the iSCSI share:
iscsitadm modify target -p 1 zpool01/myiscsivol
At this point, you can actually go ahead and mount the iSCSI target from your client. I won’t cover this off, but on Windows Server 2003 you download the iSCSI Software Initiator. It’s very easy to use and requires virtually no explanation. All I had to do was fire up the interface, add a discovery target, scan for volumes, connect one, and my iSCSI volume popped up in Disk Management. I formatted it with NTFS, and it delivered a nice 110MB/sec write speed.
Configuring CHAP Authentication
It’s definitely worth configuring CHAP Authentication to protect your iSCSI share. I won’t cover anything funky, such as Radius auth, ipsec or bidirectional target authentication, but they are available if you want them. First, we need to tell iscsitadm about our client (the "initiator"), and give it a friendly name. You’ll need the iqn (iSCSI Qualified Name), which you can get from the client (In Windows, it’s on the iSCSI "General" tab).
iscsitadm create initiator --iqn iqn.1234-01.com.microsoft:win01 myclient
Now set the CHAP authentication parameters, the username and password. The password if you’re not using ipsec must be exactly between 12 and 16 characters:
iscsitadm modify initiator --chap-name myusername myclient iscsitadm modify initiator --chap-secret myclient
You now need to associate the initiator entry you just created, with the ACL (Access Control List) of the iSCSI share:
iscsitadm modify target --acl myclient zpool01/myiscsivol
Now you’re all done! Congratulations :)
A Note on CHAP Passwords
I banged my head against a wall, unable to get the Windows initiator to authenticate against my iSCSI chap details. I kept getting “Authorization denied”. My password began with a “0″ and contained a “!” in the middle. Changing the password to a 15 character letter only password seemed to fix this. I don’t have enough time to investigate this further, but if you run into a similar issue, using a simpler password may fix it for you.
4 comments November 16th, 2008
Sun x4500 Thumper: Mapping logical drives to physical
The Sun x4500 has 48 disk slots, numbered 0 to 47. However on Solaris, drives are named according to their controller/target location. I was wondering how you work out how to go from the logical naming, to the physical one.
Well the answer lays on the x4500 Tools & Drivers CD. On it is a nifty package named "SUNWhd-1.07.pkg", which plonks a utility called "hd" at "/opt/SUNWhd/hd/bin/hd". Running spits out the serial numbers of the disks, their temperature, and at the end, it finally spits out some ASCII art depicting the layout:
---------------------SunFireX4500------Rear---------------------------- 36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: c4t3 c4t7 c3t3 c3t7 c6t3 c6t7 c5t3 c5t7 c1t3 c1t7 c0t3 c0t7 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: c4t2 c4t6 c3t2 c3t6 c6t2 c6t6 c5t2 c5t6 c1t2 c1t6 c0t2 c0t6 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: c4t1 c4t5 c3t1 c3t5 c6t1 c6t5 c5t1 c5t5 c1t1 c1t5 c0t1 c0t5 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: c4t0 c4t4 c3t0 c3t4 c6t0 c6t4 c5t0 c5t4 c1t0 c1t4 c0t0 c0t4 ^b+ ^b+ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ -------*-----------*-SunFireX4500--*---Front-----*-----------*----------
Rather funky, and useful!
Add comment November 16th, 2008
Sun Fire x4500 Thumper: Recommended ZFS Zpool Layout
The x4500 comes with 48 disks, two of which you typically use as a mirrored ZFS pair for the host OS, leaving 46 drives for data. One of the questions you’re faced with, is how to efficiently lay out your zpool configuration to balance performance, reliability and capacity.
For the particular workload we’ll be using the x4500 for, we want a balance across all 3. No particular factor wins out over the others – they’re all equally important. To further complicate matters, the box has six 8-channel SATA controllers, so you want to spread your workload across the controllers in an intelligent fashion.
There are many differing opinions on this. I sparked a debate on #solaris on Freenode posing the question, with some suggesting a single zpool with collection of mirrors if databases are involved, 1 drive per controller. Others suggested lots of small raidz2 sets in a single zpool.
After expirementing, musing, and researching on the web, we finally settled on the following configuration, which provides a fair balance:
pool: zpool01
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zpool01 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c3t1d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c3t5d0 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0
c6t4d0 ONLINE 0 0 0
c5t4d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c3t7d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
c3t6d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
spares
c4t2d0 AVAIL
c4t6d0 AVAIL
This gives 2 spares, and 11 drives across 4 raidz2 groups. The chances of 3 drives failing in a 11 disk raidz2 pool before the spares finish rebuilding are (hopefully!) fairly low. In the unlikely event that 3 drives did fail, they’d more than likely be spread across the 4 raidz2 groups. It’s all about managing risk.
The command to create this would be:
zpool create zpool01 raidz2 c{4,3,6,5,1,0}t1d0 c{3,6,5,1,0}t0d0
zpool add zpool01 raidz2 c{4,3,6,5,1,0}t3d0 c{3,6,5,1,0}t2d0
zpool add zpool01 raidz2 c{4,3,6,5,1,0}t5d0 c{3,6,5,1,0}t4d0
zpool add zpool01 raidz2 c{4,3,6,5,1,0}t7d0 c{3,6,5,1,0}t6d0
zpool add zpool01 spare c4t2d0 c4t2d6
Finally, this inspiration for this configuration came from the Joyent web blog. Those guys know their stuff and have been using ZFS in production for longer than most.
Add comment November 16th, 2008
Sun release new Apache Module mod_privileges for Solaris
Nick Kew, who works for Sun on their Web Stack project, has just posted on the Webstack Discuss Mailing List about an exciting new Solaris-specific Apache HTTPD module they have been working on. Rather than paraphrase, he posted:
I’ve just introduced mod_privileges to Apache HTTPD trunk.
This is a platform-specific module for Solaris 10 and OpenSolaris, that makes the webserver privileges(5)-aware. This enables the server to be run with enhanced security, and with different settings per virtual host.
The feature likely to be of most interest is that it enables different virtual hosts to run under different Unix user and group IDs, using the VHostUser and VHostGroup directives. This is the capability once promised by the “perchild” MPM.
It has one major drawback: it is not suitable for a threaded MPM. However, it is ideally suited for use with PHP, which of course also precludes threads. It should also be of interest to anyone hosting other in-process scripting environments such as mod_perl, mod_python or mod_ruby, or application modules.
This is a really exciting module. Being able to give each VirtualHost it’s own user and group is a killer feature for shared hosting companies, who traditionally have had the nightmare of all PHP scripts running under the Apache user. Although solutions exist, such as php-suexec, they are cumbersome and CGI based, and thus typically slow or memory/process intensive. This kind of Apache module sounds like it has the potential to offer a really slick way of solving this particular problem.
Unfortunately it’s in the Apache HTTPD 2.3 trunk, so yet to be released into the wild. But I’m looking forward to this becoming production-ready in a future release.
Add comment November 14th, 2008