Archive for March, 2009
What is Solaris? Why should I be using it?
Solaris is Sun Microsystem’s flagship Unix based operating system. It is free to obtain and use, and Sun opened the Solaris source code under an Open Source license in 2005.
It is robust, highly scalable and incredibly powerful; actively maintained by Sun, new features are being introduced on a regular basis. Paid commercial support is available. It fully supports Intel & AMD CPUs, ensuring it runs on the vast majority of commodity hardware such Dell & HP.
Solaris contains many killer features. If you’re currently using Linux, FreeBSD, Mac OS X, or another Unix varient, you should consider checking it out.
We are a big Solaris user here at EveryCity, providing Managed Solaris Hosting to our customers, via our dedicated and cloud based hosting platform, along side our Windows and Linux hosting offerings.
Killer Features
Solaris has a huge number of features, too many to mention. We will cover off some of the key features that we make heavy use of, many of which we severely miss when we use other operating systems.
ZFS Filesystem
ZFS is a revolutionary filesystem that throws history to the wind, doing away with the traditional link between files, filesystems and partitions. ZFS utilises a notion of “pooled storage”, where you allocate disks to a pool. You can then dynamically create filesystems on the fly, which all share the pool of storage. You can, for example, give each user their own ZFS filesystem.
ZFS filesystems support compression, encryption and quotas. The ZFS filesystem is atomic – transactions are either fully comitted or not comitted at all. There is no fsck/chkdsk tool, data on disk is always consistent, meaning that after an unexpected power loss, the system boots without needing to perform a lengthy disk check.
Storage Pools can be created with RAID levels, with ZFS supporting RAID0, RAID1, RAID10, RAID-Z (Raid 5) and RAID-Z2 (Raid 6). ZFS stores data blocks on disk with a CRC error checking hash, and ZFS will detect silent data corruption and report on it. If you are using RAID with parity (1, 10, Z or Z2), ZFS will recover from the corruption by utilising the parity data. This ensures ZFS can for example recover from the scenario where a disk in a RAID array is silently corrupting data. No commercial RAID card on the market can currently recover from this particular failure mode.
ZFS supports snapshots, which are virtually free and incredibly easy to do. Filesystem snapshots are immediately available, via a hidden “.zfs” directory, that let you view the filesystem at the time of the snapshot. Snapshots are read only, and can be "cloned" to produce full read/write filesystems. They are great for producing backups.
ZFS snapshots are incredibly powerful, and very useful. A great example is the Solaris liveupgrade utility, which is used to upgrade Solaris to a new release. liveupgrade will snapshot the root ZFS filesystem before performing the upgrade. If the upgrade fails, you can rollback to the snapshot, saving you having to restore from backups. We use ZFS snapshots internally on our backup server to store incremental daily backups.
ZFS contains many more features that we haven’t even begun to touch on, such as send/receive, separate log devices, the ARC and L2ARC caching systems, and many many more. We love ZFS, and now simply can’t imagine life without it.
Zones / Containers
Solaris Zones (Or "Containers" as you may hear them being referred to) are similar to FreeBSD Jails. They are virtualised Solaris installations that you can SSH into, install applications inside, and use just like a real physical server. Solaris Zones differ from other virtualisation technologies in that they all run on top of a single Kernel, with little or no overhead. Zones cannot for example run Linux or Windows inside, because they are not a hardware virtualisation solution – they are a logical virtualisation technology that groups system processes/users/resources into discrete units.
Zones are very powerful, and Solaris provides a full management framework for creating, starting, stopping, cloning and modifying them. Since there is little or no overhead, Zones are incredibly fast, and it is entirely feasible to run enterprise applications such as Oracle inside a zone. Zones are cheap to create, using very little memory. As an example, one might create a database Zone for MySQL, a production Zone with Apache for a live website, and a staging Zone also with Apache for development.
Zones support resource controls, such as CPU, memory, number of processes, etc, providing a great method of partitioning up a system. Resources are pooled, in that you set a memory cap, rather than allocating a specific quantity of RAM, allowing you to overcommit.
We use Solaris Zones for our cloud computing environment, and cannot sing their praises enough. We can deploy a brand new fully working Zone, including installing all necessary key applications such as Apache, PHP, MySQL, etc, within 1 to 2 minutes.
Service Management Framework
Sun has done away with the old System V init scripts that users of Linux may be familiar with. Instead, in Solaris, we have the Service Management Framework, part of Sun’s "Predictive self healing" strategy. SMF manages services via a set of command line tools, "svcs", "svcadm" and "svccfg". SMF can detect and restart failed services, tracks dependencies, starts services in parallel, and stores it’s configuration in XML based manifests (which you don’t need to touch, unless you’re creating a new service).
SMF is incredibly powerful, and makes managing services incredibly easy.
Versions of Solaris
Solaris comes in several versions. Solaris 10 is Sun’s commercial, stable operating system, available free of charge. New release come out on a roughly 6 month basis, which often introduce new major features, whilst maintaining backwards compatibility providing maximum stability. We utilise Solaris 10.
OpenSolaris is the open source edition of Solaris. It contains radical new features and bleeding edge technologies. It is relatively new, and evolving quickly with releases every 6 to 12 months. It provides a great way to try out new features, and is very stable, with many companies using it in a production environment. Commercial support is also available from Sun.
Solaris also has development editions, such as Solaris Express Community Edition, and you can also obtain the OpenSolaris codebase and build this yourself.
Conclusion
I hope you found the above interesting. Solaris is not without it’s flaws, and Sun are working hard to address them. For example, Solaris 10 currently lacks an integrated centralised package management system. But this feature has been developed, is present in OpenSolaris, and should hopefully arrive in Solaris 11.
We wouldn’t have made such a heavy investment in time and energy to use Solaris if we didn’t strongly believe in it’s technological benefits, and we urge others to play with it and give it a go. Perhaps you’ll fall in love with it as much as we have.
4 comments March 6th, 2009
Compiling MySQL-python on Solaris 10
This one can be a bit of a nightmare on Solaris, due to typical Solaris complications. There are two key things to watch out for.
First thing to watch out for, is that MySQL-python includes both pyconfig.h and my_config.h, both of which on Solaris may include SIZEOF_ definitions. If you’re using Sun Web Stack 1.4, the my_config.h file is for 64bit, but we’ll no doubt be compiling as 32bit. Our recommendation is to compile up your own mysql client library and link against this (see my previous post about compiling things against the Sun Web Stack 1.4 MySQL).
The second thing to watch out for is that Python and MySQL both record the compiler options they were compiled with. For example, -fPIC, -Wall, etc. MySQL-python blindly passes these to whatever compiler you’re using. These may be the wrong arguments, generating all sorts of warnings and/or errors. Ones like these:
# MySQL compiled with Sun Studio, Python compiled with gcc, we're compiling with Sun Studio: building '_mysql' extension creating build/temp.solaris-2.10-i86pc-2.5 creating build/temp.solaris-2.10-i86pc-2.5/src /opt/SUNWspro/bin/cc -OPT:Olimit=0 -DNDEBUG -O -Kpic -Dversion_info=(1, 3, 0, 'f inal', 0) -D__version__=1.3.0 -I/opt/webstack/mysql/include/mysql -I/opt/python2 .5/include/python2.5 -c src/mysqlmod.c -o build/temp.solaris-2.10-i86pc-2.5/src/ mysqlmod.o -xarch=386 -xchip=pentium -xspace -xildoff -xc99=all -xnorunpath -m32 -DBIG_TABLES -DHAVE_RWLOCK_T cc: Warning: illegal option -OPT:Olimit=0 "/usr/include/sys/feature_tests.h", line 332: #error: "Compiler or options inval id for pre-UNIX 03 X/Open applications and pre-2001 POSIX applications" cc: acomp failed for src/mysqlmod.c error: command '/opt/SUNWspro/bin/cc' failed with exit status 2 # MySQL compiled with gcc, Python compiled with Sun Studio, we're compiling with gcc: building '_mysql' extension creating build/temp.solaris-2.10-i86pc-2.5 creating build/temp.solaris-2.10-i86pc-2.5/src gcc -OPT:Olimit=0 -DNDEBUG -O -Kpic -Dversion_info=(1, 3, 0, 'final', 0) -D__version__=1.3.0 -I/opt/ec/mysql_client/include/mysql -I/opt/python2.5/include/python2.5 -c src/mysqlmod.c -o build/temp.solaris-2.10-i86pc-2.5/src/mysqlmod.o -DHAVE_RWLOCK_T gcc: unrecognized option `-Kpic' cc1: error: invalid option argument `-OPT:Olimit=0' # etc, there are quite a few combinations here
Our solution to this was to find out what your python is compiled with (Sun Studio or gcc) by looking in the “lib/python*/config/Makefile” file (relative to where python is installed), compile up your own MySQL client library using the same compiler, then compile MySQL-python with that same compiler.
Don’t forget to set the LDFLAGS environment variable with -L and -R paths to your MySQL client library path and Python library path, so that MySQL-python can find the libraries when it compiles.
Add comment March 6th, 2009
Compiling things against the Sun Web Stack MySQL
Sun have done an interesting thing with Web Stack 1.4, and have churned out both 32bit and 64bit libraries and binaries for Apache, PHP, MySQL, etc (On Solaris 10 anyway). The 64 bit versions can be found in their amd64 folders, and things like Apache & MySQL can be told to start in 64bit mode by tweaking their manifests. So for example, you have:
In: /opt/webstack/apache2/2.2/bin: ./httpd: ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically linked, stripped ./amd64/httpd: ELF 64-bit LSB executable AMD64 Version 1 [SSE FXSR CMOV FPU], dynamically linked, stripped
One downside of this is that Sun have forgotten a vital subtlety. There is a header file that ships with MySQL called "my_config.h", which contains SIZEOF_ entries for the data types, eg SIZEOF_LONG. It’s autogenerated by autoconf, and the my_config.h that comes with Sun Web Stack 1.4 contains SIZEOF_ entries that are 64bit sized. There is no 32bit version of the file. This is no good if you want to link against MySQL in 32bit mode. We became aware of this error when compiling MySQL-python 1.3, which includes pyconfig.h as well, which also defines SIZEOF_LONG, so we were getting:
warning: "SIZEOF_LONG" redefined
There are perhaps ways of fixing or getting around this, such as stealing a my_config.h from a 32bit sized MySQL installation and altering the #include statement of whatever you’re compiling. But the safest solution we went with was to compile our own MySQL client from source, doing something along the lines of:
export CC=/opt/SUNWspro/bin/cc export CXX=/opt/SUNWspro/bin/CC ./configure --without-server --enable-thread-safe-client --prefix=/opt/mysql_client gmake gmake install
You can then compile your application against this client library. A lot of things that build against mysql use the mysql_config program to get their compiler options. So if you do compile up your own mysql client, you’ll want to make sure it’s mysql_config is in your path.
2 comments March 6th, 2009