“Could not stat /dev/sda1″ when installing Citrix XenServer
If you get this error when installing Citrix XenServer…
Could not stat /dev/sda1 --- No such file or directory
… then I have a solution for you! Basically it’s caused by a race condition - the installer creates the partition table, but then immediately attempts to create a filesystem on /dev/sda1 before the Kernel has caught up with the partition table change.
You can fix the issue by:
1. Hit "Alt-F2" to get a console
2. Use "ps -ef" to get the pid of the installer, and "kill -9" it
3. Type "vi /etc/inittab"
4. Change the line under "# Start the installer on the console" to read:
tty1::respawn:/opt/xensource/installer/preinit
(This causes the installer to be respawned upon death - useful for debugging if things go wrong without requiring a reboot)
5. Type "vi /opt/xensource/installer/backend.py"
6. Near the top add "from time import sleep" under one of the import statements.
7. Near line 565, under the "def createDom0DiskFilesystems(…" bit, add a new line with "sleep(10)" in it:
def createDom0DiskFilesystems(disk, primary_partnum):
sleep(10)
rc, err = util.runCmd2....
Be sure to match the indentation of the lines below, as Python uses indentation as a part of its syntax.
8. Type "kill -HUP 1" to reload the inittab. The installer should respawn on tty1 - simply press “Alt-F1″ to get to it. Perform the install.
This simple fix works by adding a sleep statement before the bit that creates a filesystem. Yay!
2 comments October 20th, 2010
bash: fork: Not enough space
Quite a lot of our clients email asking if they have run out of disk space when they get this error:
bash: fork: Not enough space
Not so - the "space" referred to here means memory (RAM). So in basic terms, it means your server has run out of free memory and can’t start new programs.
Resolving this requires stopping running services or restarting your server. Unfortunately, given you have no free memory to launch new programs, doing this might be quite hard. But thankfully if you get this error in a Solaris Zone with capped memory usage, you can fix the issue from the Global Zone. (If you’re an EveryCity managed hosting customer, we can do this for you).
How do you stop it from happening? Well you need to identify what is gobbling all your memory. If it’s an Apache web server, chances are you just received an influx of visitors, as Apache (by default) has to spawn a new process for each connection, and each process uses RAM. The fix is to buy more memory, or optimise Apache (to use less RAM per process), or optimise your site so that requests take less time (so fewer Apache processes are needed to handle the same throughput).
1 comment October 12th, 2010
Moving Zones between Servers on OpenSolaris
A very very quick post.
If you get:
# zoneadm -z blah attach
ERROR: no active dataset.
Result: Attach Failed.
The issue is that the zone boot environment isn’t mounted. The zbe is set with the legacy option so you have to use the old mount command, eg:
# mount -F zfs rpool/zones/blah/ROOT/zbe /zones/blah/root
# zoneadm -z blah attach
Attaching...
preferred global publisher: openindiana.org
Cache: Using /var/pkg/download.
Updating non-global zone: Output follows
Creating Plan
...
Yay!
Add comment September 15th, 2010
OpenIndiana - a new OpenSolaris Distribution :-)
Well, I can finally talk about it! We’ve done it! OpenIndiana is official :-)
Don’t forget to follow on twitter!
On Tuesday 14th September, we will be unveiling OpenIndiana – an exciting new distribution of OpenSolaris! OpenIndiana is a continuation of the OpenSolaris operating system. It was conceived during the period of uncertainty following the Oracle takeover of Sun Microsystems, after several months passed with no binary updates made available to the public. The formation proved timely, as Oracle has since discontinued OpenSolaris in favour of Solaris 11 Express, a binary distribution with a more closed development model to debut later this year. OpenIndiana is part of the Illumos Foundation, and provides a true open source community alternative to Solaris 11 and Solaris 11 Express, with an open development model and full community participation. -------------------- Announcement Details -------------------- We will be holding a press conference in London (UK) at the JISC offices at 6:30pm UK Time (BST, GMT+1) to formally announce the project and provide full details, including information on gaining access to download our first development release. In addition, we will be broadcasting live on the internet – please see http://openindiana.org/announcement for details of attending in person or on the web. We believe this announcement will deliver the distribution the community has long sought after. Please see our website at: http://openindiana.org/ Please follow us on twitter at: http://twitter.com/openindiana Tickets for attending the press conference: http://openindiana.eventbrite.com/ We look forward to seeing you at the announcement :-) The OpenIndiana Team.
2 comments September 9th, 2010
Upgrading modern Dell Server BIOS/Firmware
I’m from a rather old school way of doing things, and for years now I’ve updated the BIOS on our Dell servers by making a DOS boot disk image and mounting it via the iDRAC enterprise.
However recently Dell have stopped providing the images (or I’ve been unable to find them) and I started tearing my hair out trying to do the BIOS upgrade from a CentOS or Knoppix boot CD using their Linux installer.
However somehow one day I stumbled upon the "F10" option during the system boot screen. This magic option starts up a crazy embedded OS which can update all your firmware in one go - BIOS, RAID Card, NIC Firmware, iDRAC firmware - the lot. You just tell it which IP address settings to use, let it use the default Dell FTP server, and off it goes! Mad!
I have to say this is an incredibly cool feature. It’s very slow but mostly automated and a very welcome change.
Add comment August 27th, 2010
Boot environments with OpenSolaris domUs on xVM
I was having a nightmare with an OpenSolaris domU xen guest I had broken. It had multiple boot environments, all I needed was to boot one of them - the default rpool/ROOT/opensolaris one would do. However when booting the guest, no grub menu was presented by pygrub, it just goes straight to booting the kernel:
root ~ (xvm03.everycity): xm start guest -c v3.4.2-xvm chgset 'Wed Feb 17 02:28:56 2010 -0800 19674:ba65a89eded4' SunOS Release 5.11 Version snv_147 64-bit Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved. /platform/i86xpv/kernel/drv/amd64/rootnex: undefined symbol 'iommulib_nexdma_unmapobject' WARNING: mod_load: cannot load module 'rootnex' panic[cpu0]/thread=fffffffffbc609e0: BAD TRAP: type=e (#pf Page fault) rp=fffffffffbcb4ec addr=20 occurred in module "genunix" due to a NULL pointer dereference ... skipping system dump - no dump device configured rebooting...
I spent ages trying to get the grub menu to display, but couldn’t work out how to, so decided to take a different approach.
First, I needed a list of boot environments out of the guest. If you already know which boot environment you want to boot (such as the original rpool/ROOT/opensolaris boot environment) then you can skip this bit. I include it as it’s quite a nifty method - thanks to a friend from IRC, he provided me with this clever script:
#!/usr/bin/python2.4 import sys import pygrub import fsimage disk = sys.argv[1] file = sys.argv[2] fs = fsimage.open(disk, pygrub.get_fs_offset(disk)) f = fs.open_file(file) print f.read();
Pop this into a file called readfile.py. This script steals some libraries from pygrub, so you’ll need to copy this to the same working directory. The usage is as follows:
# cp /usr/lib/xen/bin/pygrub ./pygrub.py # ./readfile.py /dev/zvol/dsk/rpool/xvm/guest /boot/grub/menu.lst
This will then spit back the grub menu list at you:
# python2.4 readfile.py /dev/zvol/dsk/data/xvm/guest/disk0 /boot/grub/menu.lst timeout 5 default 3 #---------- ADDED BY BOOTADM - DO NOT EDIT ---------- title OpenSolaris Development snv_134 findroot (pool_rpool,0,a) bootfs rpool/ROOT/opensolaris kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS module$ /platform/i86pc/$ISADIR/boot_archive ...
From this list, the information we want is the “bootfs” argument from the boot environment you want to boot. So for example rpool/ROOT/opensolaris. We can then shove this into the xvm xml config file for the guest:
<domain type='xen'>
<name>guest</name
<memory>4194304</memory>
<currentMemory>4194304</currentMemory>
<vcpu>8</vcpu>
<bootloader>/usr/lib/xen/bin/pygrub</bootloader>
<bootloader_args>--args="-B zfs-bootfs=rpool/ROOT/opensolaris"</bootloader_args>
<os>
<type>linux</type>
</os>
...
The key line above there is the bootloader_args argument - this passes the correct bootfs to the kernel telling it which ZFS filesystem to use for the boot. Hey presto, your guest should now boot :)
Add comment August 23rd, 2010
OpenSolaris is now officially dead. RIP.
What follows is a leaked memo sent to Oracle Engineering yesterday. It basically spells the end of OpenSolaris, which is to be replaced with Solaris 11 Express, a binary distribution. The key part is:
"We will not release any other binary distributions, such as nightly or bi-weekly builds of Solaris
binaries, or an OpenSolaris 2010.05 or later distribution. We will determine a simple, cost-effective means of getting enterprise users of prior OpenSolaris binary releases to migrate to S11 Express."
Solaris Engineering,
Today we are announcing a set of decisions regarding the path to Solaris 11, and answering key pending questions on open source, open development, software and binary licenses, and how developers and early adopters will be able to use Solaris 11 technology before its release in 2011.
As you all know, the term “OpenSolaris” has been used colloquially to refer to any or all of a collection of source code, a development model, a web site, a logo, a binary release, a source license, a community, and many other related things. So it’s taken a while to go over each issue from an organizational and business perspective, and align on the correct next step. Therefore, please take the time to read all of the detail here carefully. We’ll discuss our strategy first, and then the decisions and changes to our policies and processes that implement that strategy.
Solaris Strategy
———————-
Solaris is the #1 Enterprise Operating System. We have the leading share of business applications on Solaris today, including both SPARC and x64. We have more than twice the application base of AIX and HP-UX combined. We have a brand that stands for innovation, quality, security, and trust, built on our 20-year investment in Solaris operating system engineering.
From a business perspective, the purpose of our investment in Solaris engineering is to drive our overall server business, including both SPARC and x64, and to drive business advantages resulting from integration of multiple components in the Oracle portfolio. This includes combining our servers with our storage, our servers with our switches, Oracle applications with Solaris, and the effectiveness of the service experience resulting from these combinations. All together, Solaris drives aggregate business measured in many billions of dollars, with significant growth potential.
We are increasing investment in Solaris, including hiring operating system expertise from throughout the industry, as a sign of our commitment to these goals. Solaris is not something we outsource to others, it is not the assembly of someone else’s technology, and it is not a sustaining-only product. We expect the top operating systems engineers in the industry, i.e. all of you, to be creating and delivering innovations that continue to make Solaris unique, differentiated, and valuable to our customers, and a unique asset of our business.
Solaris must stand alone as a best-of-breed technology for Oracle’s enterprise customers. We want all of them to think “If this has to work, then it runs on Solaris.” That’s the Solaris brand. That is where our scalability to more than a few sockets of CPU and gigabytes of DRAM matters. That is why we reliably deliver millions of IOPS of storage, networking, and Infiniband. That is why we have unique properties around file and data management, security and namespace isolation, fault management, and observability. And we also want our customers to know that Solaris is and continues to be a source of new ideas and new technologies– ones that simplify their business and optimize their applications. That’s what made Solaris 10 the most innovative operating system release ever. And that is the same focus that will drive a new set of innovations in Solaris 11.
For Solaris to stand alone as the best-of-breed operating system in Oracle’s complete and open portfolio, it must run well on other server hardware and execute everyone’s applications, while delivering unique optimizations for our hardware and our applications. That is the central value proposition of Oracle’s complete, open, and integrated strategy. And these are complementary and not contradictory goals that we will achieve through proper design and engineering.
The growth opportunity for Solaris has never been greater. As one example, Solaris is used by about 40% of Oracle’s enterprise customers, which means we have a 60% growth opportunity in our top customers alone. In absolute numbers, there are 130,000 Oracle customers in North America alone who don’t use our servers and storage yet, and a global customer base of 350,000 (the prior Sun base was ~35,000). That’s a huge opportunity we can go attack as a combined company that will increase Solaris adoption and the overall Hardware server revenue. Our success will also increase the amount of effort ISVs exert optimizing their applications for Solaris.
We will continue to grow a vibrant developer and system administrator community for Solaris. Delivery of binary releases, delivery of APIs in source or binary form, delivery of open source code, delivery of technical documentation, and engineering of upstream contributions to common industry technologies (such as Apache, Perl, OFED, and many, many others) will be part of that activity. But we will also make specific decisions about why and when we do those things, following two core principles: (1) We can’t do everything. The limiting factor is our engineering bandwidth measured in people and time. So we have to ensure our top priority is driving delivery of the #1 Enterprise Operating System, Solaris 11, to grow our systems business; and (2) We want the adoption of our technology and intellectual property to accelerate our overall goals, yet not permit competitors to derive business advantage (or FUD) from our innovations before we do.
We are using our investment in core Solaris innovation and engineering to drive multiple businesses, through multiple product lines. This already includes our Solaris operating system for Enterprise, and our ZFS Storage product line, and will soon include other Oracle products. This strategy is all about creating more value from a set of common software investments: it makes everything you do more
valuable and used by more people worldwide. It also means you as an individual engineer or manager have an even greater responsibility to understand the broader business and technical contexts in which your engineering is deployed.
Solaris Decisions
————————
We will continue to use the CDDL license statement in nearly all Solaris source code files. We will not remove the CDDL from any files in Solaris to which it already applies, and new source code files that are created will follow the current policy regarding applying the CDDL (simply, that usr/src files will have the CDDL, and the very small minority of files in usr/closed might not have it). Use of other open licenses in non-ON consolidations (e.g. GPL in the Desktop area) will also continue. As before, requests to change the license associated with source code are case-by-case decisions.
We will distribute updates to approved CDDL or other open source- licensed code following full releases of our enterprise Solaris operating system. In this manner, new technology innovations will
show up in our releases before anywhere else. We will no longer distribute source code for the entirety of the Solaris operating system in real-time while it is developed, on a nightly basis.
Anyone who is consuming Solaris code using the CDDL, whether in pieces or as a part of the OpenSolaris source distribution or a derivative thereof, would therefore be able to consume any updates we release at that time, under the terms of the CDDL, LGPL, or whatever license applies.
We will have a technology partner program to permit our industry partners full access to the in-development Solaris source code through the Oracle Technology Network (OTN). This will include both early access to code and binaries, as well as contributions to us where that is appropriate. All such partnerships will be evaluated on a case-by-case basis, but certainly our core, existing technology partnerships, such as the one with Intel, are examples of valued participation.
We will encourage and listen to any and all license requests for Solaris technology, either in part or in whole. All such requests will be evaluated on a case-by-case basis, but we believe there are
many complementary areas where new partnership opportunities exist to expand use of our IP.
We will continue active open development, including upstream contributions, in specific areas that accelerate our overall Solaris goals. Examples include our activities around Gnome and X11, IPS
packaging, and our work to optimize ecosystems like Apache, OpenSSL, and Perl on Solaris.
We will deliver technical design information, in the form of documentation, design documents, and source code descriptions, through our OTN presence for Solaris. We will no longer post advance
technical descriptions of every single ARC case by default, indicating what technical innovations might be present in future Solaris releases. We can at any time make a specific decision to post advance technical information for any project, when it serves a particular useful need to do so.
We will have a Solaris 11 binary distribution, called Solaris 11 Express, that will have a free developer RTU license, and an optional support plan. Solaris 11 Express will debut by the end of this
calendar year, and we will issue updates to it, leading to the full release of Solaris 11 in 2011.
All of Oracle’s efforts on binary distributions of Solaris technology will be focused on Solaris 11. We will not release any other binary distributions, such as nightly or bi-weekly builds of Solaris
binaries, or an OpenSolaris 2010.05 or later distribution. We will determine a simple, cost-effective means of getting enterprise users of prior OpenSolaris binary releases to migrate to S11 Express.
We will have a Solaris 11 Platinum Customer Program, including direct engineering involvement and feedback, for customers using our Solaris 11 technology. We will be asking all of you to participate in this endeavor, bringing with us the benefit of previous Sun Platinum programs, while utilizing the much larger megaphone that is available to us now as a combined company.
We look forward to everyone’s continued work on Solaris 11. Our goal is simply to make it the best and most important release of Solaris ever.
-Mike Shapiro, Bill Nesheim, Chris Armes
22 comments August 13th, 2010
So your keys don’t work properly on your Solaris console
Try: loadkeys us
Add comment August 13th, 2010
Solaris 11 announced for 2011
Well, it was a long time in coming, but Oracle have finally announced Solaris 11 will be arriving in 2011. Which is jolly good news!
The unfortunate thing is that Oracle barely mentioned OpenSolaris, and the whole OpenSolaris community is still left hanging.
But fear not! The community are working on plugging the gap left by Oracle. The Illumos project is a spork of OS/Net (The kernel and core userland tools of OpenSolaris) that will track the official Oracle OpenSolaris repo, whilst merging in community contributions, including replacements for closed source components (such as the closed libc_i18n), and closed userland binaries (such as tr/pax/awk).
The Illumos project provides a community rallypoint for all downstream OpenSolaris distributions, such as Nexenta, Schillix, Belenix, etc. Instead of using the Oracle source tree directly, they build from Illumos. Community changes can be merged in far more easily than was the case with Sun/Oracle, and should Oracle ever stop providing updates to the open source codebase, the community forks gracefully in a very much “Business as usual” type of way, just at a slower pace.
What’s still missing is a fully blown distribution, something to replace the void left by the missing OpenSolaris 2010.03 release. However I can confirm that the community is working on this too, behind closed doors; an announcement should be forthcoming by the end of this month, which is really really exciting news. I happen to be in contact with the people doing this, and if anyone reading this would like to get involved, please get in touch by mailing me on alasdairrr at gmail dot com.
The past year has been a turbulent time for Solaris/OpenSolaris users. The Oracle takeover resulted in a kind of communications blackout, leading to widespread fear, uncertainty and doubt, and a general community meltdown. Finally, news is trickling out of Oracle, at least as far as Solaris 10/11 is concerned, especially now that Oracle confirmed the support options available for Solaris on non-Sun Hardware. Corporates that like to pay for their operating system are no doubt going to be very happy by these recent moves.
However, the future of the OpenSolaris distribution is still very much unclear. But it really doesn’t matter. Oracle’s failure to act regarding OpenSolaris has resulted in the community taking direct action to reclaim what should have been theirs to begin with. OpenSolaris, under Sun/Oracle, was open source, but not led by the community. That, ladies and gentlemen, is about to change.
Thankfully, Oracle’s failure to act has resulted in direct community action to resolve this, and I’m very much of the opinion that this will actually be better for OpenSolaris in the long run.
2 comments August 11th, 2010
pfiles on Solaris 10 shows garbage IP_NEXTHOP value
Discovered an interesting little bug today. On Solaris 10, if you run pfiles against a running process with TCP/IP sockets open, you might find IP_NEXTHOP present against every socket (even ones that didn’t set this flag):
# pfiles 1234
...
10: S_IFSOCK mode:0666 dev:293,0 ino:64096 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(1048576),SO_RCVBUF(1048576),IP_NEXTHOP(0.0.16.0)
sockname: AF_INET 10.0.0.202 port: 1000
11: S_IFSOCK mode:0666 dev:293,0 ino:34793 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(1048576),SO_RCVBUF(1049800),IP_NEXTHOP(200.4.16.0)
sockname: AF_INET 10.0.0.202 port: 1000
peername: AF_INET 10.0.0.114 port: 49724
12: S_IFSOCK mode:0666 dev:293,0 ino:49846 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(1048576),SO_RCVBUF(1055220),IP_NEXTHOP(244.25.16.0)
sockname: AF_INET 10.0.0.114 port: 48161
13: S_IFSOCK mode:0666 dev:293,0 ino:53776 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(1048576),SO_RCVBUF(1049800),IP_NEXTHOP(200.4.16.0)
sockname: AF_INET 10.0.0.202 port: 48123
peername: AF_INET 10.0.0.114 port: 1000
14: S_IFSOCK mode:0666 dev:293,0 ino:17380 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(1048576),SO_RCVBUF(1049800),IP_NEXTHOP(200.4.16.0)
sockname: AF_INET 10.0.0.202 port: 48148
peername: AF_INET 10.0.0.116 port: 1000
This is related to bug 6386915 pfiles reports IP_NEXTHOP set when it isn’t. If you look closely at the list above, you’ll see that where the SO_RECVBUF values match, so do the IP_NEXTHOP values. Someone has gotten their C code wrong.
The bug was fixed in OpenSolaris/onnv but never backported to Solaris 10. If you need this fixed, just grab pfiles from an OpenSolaris box - it seems to work okay (But be sure to grab the platform binaries, eg /usr/bin/i86/pfiles and /usr/bin/amd64/pfiles). Also keep in mind the ABI interface between Solaris 10 and OpenSolaris may have changed so the pfiles binary from OpenSolaris might explode and eat your children. You have been warned. Worked for me though.
Add comment August 4th, 2010