Posts filed under 'Solaris'
Making Solaris SMF ignore core dumps in child processes
I can never ever remember how to do this and googling for it always takes ages, so I thought I’d jot it down here.
When Solaris SMF starts a process, it tracks that process and all its children. If any of those children coredump, SMF treats it as a failure and puts the state into maintenance mode. Not terribly useful if you’re launching buggy software like FFMpeg.
The solution? Simple! Slap this in your SMF Manifest under the exec stop method:
Add comment January 19th, 2010
Enabling 64bit MySQL on Solaris Sun Web Stack 1.4
Sun Web Stack 1.4 includes both a 32bit and 64bit MySQL, with the standard bin/mysqld and bin/amd64/mysqld binaries.
By default, the SMF service sun-mysql50 runs in 32bit mode. To enable 64bit mode, simply:
# svccfg -s sun-mysql50:default svc:/application/database/sun-mysql50:default> listprop sun-mysql50 application sun-mysql50/action_authorization astring solaris.smf.manage.sun-mysql/default sun-mysql50/bin astring /opt/webstack/mysql/5.0/bin sun-mysql50/data astring /var/opt/webstack/mysql/5.0/data sun-mysql50/value_authorization astring solaris.smf.value.sun-mysql/default sun-mysql50/enable_64bit boolean true method_context framework method_context/group astring mysql method_context/limit_privileges astring :default method_context/privileges astring :default method_context/project astring :default method_context/resource_pool astring :default method_context/supp_groups astring :default method_context/use_profile boolean false method_context/user astring mysql method_context/working_directory astring /var/opt/webstack/mysql general framework general/enabled boolean true restarter framework NONPERSISTENT restarter/logfile astring /var/svc/log/application-database-sun-mysql50:default.log restarter/contract count 105 restarter/start_pid count 606 restarter/start_method_timestamp time 1233237617.117424000 restarter/start_method_waitstatus integer 0 restarter/auxiliary_state astring none restarter/next_state astring none restarter/state astring online restarter/state_timestamp time 1233237617.119195000 svc:/application/database/sun-mysql50:default> setprop sun-mysql50/enable_64bit=true svc:/application/database/sun-mysql50:default> exit # svcadm refresh sun-mysql50 # svcadm disable sun-mysql50 # svcadm enable sun-mysql50 # ps -ef | grep mysql mysql 649 490 0 14:00:06 ? 0:00 /bin/sh /opt/webstack/mysql/5.0/bin/64/mysqld_safe --user=mysql --datadir=/var/ mysql 747 649 0 14:00:06 ? 0:22 /opt/webstack/mysql/5.0/bin/64/mysqld --basedir=/opt/webstack/mysql/5.0 --datad
And as we can see from the process list, the 64 bit binary has been launched instead of the 32 bit one.
Add comment January 29th, 2009
Compiling Python 2.6 on Solaris 10
Sorry for not posting so much lately. Work has been busier than ever - it’s quite incredible. Just a quick post on compiling Python 2.6, which was giving me a few problems.
Dependencies
I’d recommend throwing on ncurses and readline from the Solaris 10 companion CD, the packages are SFWncur and SFWrline. The full dependency list is:
P SFWncur P SFWrline P SUNWbzip P SUNWcry P SUNWcsl P SUNWcslr P SUNWcsr P SUNWgccruntime P SUNWlibms P SUNWlibmsr P SUNWopenssl-libraries P SUNWzlib
Compiling
The _ctype module fails to compile with Sun Studio 12. Rather than fix this, I simply used gcc instead. Also Python seemed to be missing _ssl, so I popped in the appropriate library paths. Thus:
export "LDFLAGS=-L/opt/sfw/lib -R/opt/sfw/lib -L/usr/sfw/lib -R/usr/sfw/lib" export "CPPFLAGS=-I/usr/sfw/include -I/opt/sfw/include -I/opt/sfw/include/ncurses" export "CFLAGS=-I/opt/sfw/include" export "LIBS=-lncurses" export CC=gcc CXX=g++ ./configure --prefix=/opt/python26 --enable-shared --disable-ipv6 --with-threads --with-libs="-lncurses" --with-wctype-functions gmake gmake install
Not all the modules will compile, but the ones that were missing were not of importance (sqlite, bsdbd, etc).
1 comment January 27th, 2009
Solaris 10: Swap Space, /tmp and SMF
fork: Not enough space
Solaris 10 by default places /tmp on swap. This is good for speed, but not so good on a general purpose box where some applications may fill up /tmp. If you fill /tmp, you essentially reduce the amount of available swap to 0. This can lead to trouble, run out of physical ram, and new processes may not start. You get lovely fork() errors on the shell, and interesting messages in dmesg:
# ps -ef -bash: fork: Not enough space # free -bash: fork: Not enough space # prstat -bash: fork: Not enough space ... # dmesg ... Dec 7 02:56:27 w01.someserver.everycity.co.uk genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 8193 (munin-node) Dec 7 02:56:51 w01. someserver.everycity.co.uk tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded Dec 7 02:56:57 w01. someserver.everycity.co.uk genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 8223 (exim) Dec 7 02:57:26 w01. someserver.everycity.co.uk genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 563 (httpd) ...
The easiest way to fix this is to immediately disable any services that eat ram using svcadm disable, and clear out /tmp. You can then either move /tmp to a physical partition by editing /etc/vfstab, increase the amount of swap, or my favourite, limit the amount of swap /tmp can use by adding a mount option to /etc/vfstab:
# grep /tmp /etc/vfstab swap - /tmp tmpfs - yes SIZE=2048M
Unfortunately with this you have to reboot the box, which wasn’t an option with the machine I was running on. So I added a bunch more swap for the time being.
SMF Unhappy after running out of swap space
However I encountered a rather bizarre issue, which can only be described as a bug. Services I had stopped using svcadm disable, wouldn’t re-enable with svcadm enable:
# svcs http STATE STIME FMRI disabled 23:26:00 svc:/network/http:apache22-csk # svcadm -v enable http svc:/network/http:apache22-csk enabled. # svcs http STATE STIME FMRI disabled 23:26:00 svc:/network/http:apache22-csk
What’s going on here? The log in /var/svc/log didn’t report the enable command either. After investigating, I came to the conclusion that SMF must have broken when the box ran out of memory. SMF is managed by two processes, svc.startd and svc.configd, and thankfully you can restart them. Simply kill them both:
# ps -ef | grep svc
root 7 1 0 Dec 01 ? 0:01 /lib/svc/bin/svc.startd
root 9 1 0 Dec 01 ? 0:00 /lib/svc/bin/svc.configd
# pkill -9 svc.configd
# pkill -9 svc.startd
# ps -ef | grep svc
root 12803 1 0 23:47:07 ? 0:01 /lib/svc/bin/svc.configd
root 12841 1 0 23:47:09 ? 0:00 /lib/svc/bin/svc.startd
Then enabling the process actually does it this time:
# svcs http STATE STIME FMRI disabled 23:26:00 svc:/network/http:apache22-csk # svcadm -v enable http svc:/network/http:apache22-csk enabled. # svcs http STATE STIME FMRI enabled 23:49:00 svc:/network/http:apache22-csk
Problem solved! However I dislike it when things silently break in this way. You have to wonder, if SMF broke, what else may be having issues?
Add comment December 8th, 2008
Sun x4500 Thumper: Mapping logical drives to physical
The Sun x4500 has 48 disk slots, numbered 0 to 47. However on Solaris, drives are named according to their controller/target location. I was wondering how you work out how to go from the logical naming, to the physical one.
Well the answer lays on the x4500 Tools & Drivers CD. On it is a nifty package named "SUNWhd-1.07.pkg", which plonks a utility called "hd" at "/opt/SUNWhd/hd/bin/hd". Running spits out the serial numbers of the disks, their temperature, and at the end, it finally spits out some ASCII art depicting the layout:
---------------------SunFireX4500------Rear---------------------------- 36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: c4t3 c4t7 c3t3 c3t7 c6t3 c6t7 c5t3 c5t7 c1t3 c1t7 c0t3 c0t7 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: c4t2 c4t6 c3t2 c3t6 c6t2 c6t6 c5t2 c5t6 c1t2 c1t6 c0t2 c0t6 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: c4t1 c4t5 c3t1 c3t5 c6t1 c6t5 c5t1 c5t5 c1t1 c1t5 c0t1 c0t5 ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: c4t0 c4t4 c3t0 c3t4 c6t0 c6t4 c5t0 c5t4 c1t0 c1t4 c0t0 c0t4 ^b+ ^b+ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ -------*-----------*-SunFireX4500--*---Front-----*-----------*----------
Rather funky, and useful!
Add comment November 16th, 2008
Sun Fire x4500 Thumper: Recommended ZFS Zpool Layout
The x4500 comes with 48 disks, two of which you typically use as a mirrored ZFS pair for the host OS, leaving 46 drives for data. One of the questions you’re faced with, is how to efficiently lay out your zpool configuration to balance performance, reliability and capacity.
For the particular workload we’ll be using the x4500 for, we want a balance across all 3. No particular factor wins out over the others - they’re all equally important. To further complicate matters, the box has six 8-channel SATA controllers, so you want to spread your workload across the controllers in an intelligent fashion.
There are many differing opinions on this. I sparked a debate on #solaris on Freenode posing the question, with some suggesting a single zpool with collection of mirrors if databases are involved, 1 drive per controller. Others suggested lots of small raidz2 sets in a single zpool.
After expirementing, musing, and researching on the web, we finally settled on the following configuration, which provides a fair balance:
pool: zpool01
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zpool01 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c3t1d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c3t5d0 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0
c6t4d0 ONLINE 0 0 0
c5t4d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c3t7d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
c3t6d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
spares
c4t2d0 AVAIL
c4t6d0 AVAIL
This gives 2 spares, and 11 drives across 4 raidz2 groups. The chances of 3 drives failing in a 11 disk raidz2 pool before the spares finish rebuilding are (hopefully!) fairly low. In the unlikely event that 3 drives did fail, they’d more than likely be spread across the 4 raidz2 groups. It’s all about managing risk.
The command to create this would be:
zpool create zpool01 raidz2 c{4,3,6,5,1,0}t1d0 c{3,6,5,1,0}t0d0
zpool add zpool01 raidz2 c{4,3,6,5,1,0}t3d0 c{3,6,5,1,0}t2d0
zpool add zpool01 raidz2 c{4,3,6,5,1,0}t5d0 c{3,6,5,1,0}t4d0
zpool add zpool01 raidz2 c{4,3,6,5,1,0}t7d0 c{3,6,5,1,0}t6d0
zpool add zpool01 spare c4t2d0 c4t2d6
Finally, this inspiration for this configuration came from the Joyent web blog. Those guys know their stuff and have been using ZFS in production for longer than most.
Add comment November 16th, 2008
Sun release new Apache Module mod_privileges for Solaris
Nick Kew, who works for Sun on their Web Stack project, has just posted on the Webstack Discuss Mailing List about an exciting new Solaris-specific Apache HTTPD module they have been working on. Rather than paraphrase, he posted:
I’ve just introduced mod_privileges to Apache HTTPD trunk.
This is a platform-specific module for Solaris 10 and OpenSolaris, that makes the webserver privileges(5)-aware. This enables the server to be run with enhanced security, and with different settings per virtual host.
The feature likely to be of most interest is that it enables different virtual hosts to run under different Unix user and group IDs, using the VHostUser and VHostGroup directives. This is the capability once promised by the “perchild” MPM.
It has one major drawback: it is not suitable for a threaded MPM. However, it is ideally suited for use with PHP, which of course also precludes threads. It should also be of interest to anyone hosting other in-process scripting environments such as mod_perl, mod_python or mod_ruby, or application modules.
This is a really exciting module. Being able to give each VirtualHost it’s own user and group is a killer feature for shared hosting companies, who traditionally have had the nightmare of all PHP scripts running under the Apache user. Although solutions exist, such as php-suexec, they are cumbersome and CGI based, and thus typically slow or memory/process intensive. This kind of Apache module sounds like it has the potential to offer a really slick way of solving this particular problem.
Unfortunately it’s in the Apache HTTPD 2.3 trunk, so yet to be released into the wild. But I’m looking forward to this becoming production-ready in a future release.
Add comment November 14th, 2008
Adding mcrypt Support to Sun’s CoolStack
A very boring first post - but I thought I’d blog about this since it’s fresh in my mind!
Sun’s Web Stack - CoolStack
We’ve been using Solaris in an ever increasing way here at EveryCity for a while now. Solaris is an excellent operating system for many reasons, and I’ll no doubt be blogging about it plenty in the weeks and months to come. But one of the key areas Solaris currently lags behind Linux with is the availability of well made, precompiled packages of popular open source software. A very large number of servers these days are deployed to run web services, including Apache, PHP, MySQL, Ruby on Rails, etc, and installing this on Linux can be as simple as “yum install apache php mysql”.
So Sun set out to provide a solution to this on Solaris - enter CoolStack. CoolStack has a good selection of software, including:
- Apache HTTP Server
- PHP Hypertext Processor
- MySQL Database
- Perl Scripting Language
- Ruby (Including rubygems and rails)
- Tomcat J2EE/Java Web Server
- Squid Caching Proxy Server
The PHP package includes a good selection of libraries, so you can get started straight out of the box.
Adding mcrypt Support to PHP on CoolStack 1.3.1
One of the things that CoolStack is missing however is the mcrypt extension, which is used in a number of PHP applications, for example PHPMyAdmin. I was surprised this was missed out, as so many others are included. However it’s not too hard to add this ourself. This example is for CoolStack 1.3.1, but assuming the stack doesn’t change too much, the principles should be the same.
Compiling libmcrypt
Head on over to http://sourceforge.net/projects/mcrypt and download the latest libmcrypt source. You’ll need a healthy development environment, which I’m hoping to blog about in a future post (Hint: Grab Sun Studio 11 and put /opt/SUNWspro/bin at the front of your path).
You’ll want to choose a sensible place to install libmcrypt to - since I was producing a package for deployment to other servers, I chose “/opt/libs/libmcrypt”, but where you place this is up to you - “/opt/coolstack” is another suitable PREFIX path.
Let’s now compile and install libmcrypt (Using cc rather than gcc):
unset CC LDFLAGS CFLAGS export CC=cc ./configure --prefix=/opt/libs/libmcrypt --enable-dynamic-loading \ --with-included-algos=cast-128,gost,rijndael-128,twofish,arcfour,cast-256,loki97,rijndael-192,saferplus,wake,blowfish-compat,des,rijndael-256,serpent,xtea,blowfish,enigma,rc2,tripledes make make install
Et Voilà! You now have libmcrypt installed. If your compiler barfs, post a printout of what you get in the comments and I’ll assist if I can.
Compiling the php mcrypt extension
Assuming you have already downloaded, installed and configured CoolStack, all we need to do is fetch two CoolStack source packages. Nip on over to http://cooltools.sunsource.net/coolstack/, click the link to download, and choose “Source” as the Platform.
Download and install CSKruntime_1.3.1_Src.pkg.bz2 and CSKamp_1.3.1_Src.pkg.bz2 - this should place files under /opt/coolstack/src. Now lets compile the mcrypt extension:
cd /opt/coolstack/src/php5/ ./prepare_src cd php-5.2.6/ext/mcrypt /opt/coolstack/php5/bin/phpize ./configure --with-php-config=/opt/coolstack/php5/bin/php-config \ --with-mcrypt=/opt/libs/libmcrypt/ make make install
You can check to make sure the resulting mcrypt.so file is safe to use by doing:
ldd /opt/coolstack/php5/lib/php/extensions/no-debug-non-zts-20060613/mcrypt.so
This should not show any “(file not found)” errors.
Finally, add extension=mcrypt.so to /opt/coolstack/php5/lib/php.ini. Restart Apache (svcadm restart http) - phpinfo(); in a php file should now show a happy mcrypt installation :)
11 comments October 26th, 2008