Killing a Solaris 10 Zone stuck in the shutting_down state.
June 11th, 2009
So, you have a Solaris 10 Zone. You’ve run “zoneadm -z zonename shutdown”. It hasn’t quite shut down, and is stuck in the shutting_down state. What can you do to fix it?
Well, sometimes some processes don’t die in a timely fashion. Check what processes are running with the following command:
# ps -fz zonename
If any processes other than zsched are running, kill -9 them. The zone should hopefully shut down.
If it doesn’t, and you’re left with zsched as the only remaining process, then potentially you’ve hit a bug, such as bug 6272846 – "User orders zone death; NFS client thumbs nose". This bug has been outstanding since May 2005, so don’t expect a fix any time soon.
Thankfully there are a few more things you can try to kill the damn zone off. Give some of the following a go:
# zoneadm -z zonename unmount -f # zoneadm -z zonename reboot -- -s # pkill -9 -z zonename
The above combo should hopefully deliver a fatal blow to your Zone. If not, bitch at Sun. Hopefully they’ll sort their lives out.
Entry Filed under: General
9 Comments Add your own
1. Brian | July 10th, 2012 at 10:09 pm
2012 – and same issue, with little on Oracle’s web-site for proper procedures to clean-up mount & zone
Created a case with Oracle for the fun of it.
tried everything with exception to the pkill
waiting for Oracle
Thanks for the info
-Brian
2. Alasdair | July 16th, 2012 at 4:37 pm
Hi Brian,
There are existing bugs in the Solaris codebase related to shutting down zones. If you’re not wedded to Oracle, I’d recommend checking out SmartOS. They have fixed most (all?) of them and we haven’t had any issues shutting down Zones since moving to it.
Good luck with Oracle support!
3. Bruce K | August 16th, 2012 at 3:25 pm
in many cases, when a zone is stuck in “down” state when shuting down, it is because you have a session in the global zone in which you are in a path of the zone you are shutting down. e.g. /zones//root/…..
e.g.in this example zone “laker” is stuck in the “down” state. so check all your windows or try fuser to see who was in any of lakers filesystems via the global zone and kill the process or cd out of it.
caramel# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
5 bobcat running /zones/bobcat native shared
6 laker down /zones/laker native shared
check all your opened windows in global zone and you will find one that is no longer in a valid path. just cd + enter or cd / and zone that was stuck in the “down” state will come all the way down.
4. rajeesh | October 10th, 2012 at 2:32 pm
OMG!!! this works…thanks a ton .. i was about to reboot the server for this bloody prob!!! thanks mate thanks a ton .
5. sharath | February 4th, 2013 at 6:18 pm
The umount command worked. Awesome!!!
6. W Sanders | April 4th, 2013 at 5:32 pm
Still not fixed as of 2013
7. Nakarti | May 16th, 2013 at 4:15 pm
I did this on a global thats scheduled for reboot. The zones are stuck with a sync process that I can’t kill. Good thing a reboot was planned!
8. Alasdair | May 16th, 2013 at 4:25 pm
Nakarti,
Did it work? It might not, “sync” processes usually indicate stuck IO, which is a more serious condition. Which Solaris version was this on?
I can report that we haven’t had any stuck zones since moving to SmartOS, as Joyent have hunted down and fixed all the bugs thanks to their massive cloud scale.
9. Alasdair | May 16th, 2013 at 4:31 pm
W Sanders,
Is that on the official Solaris 10/11?
As I mentioned above, it’s fully fixed in SmartOS, thankfully. We switched to SmartOS and haven’t looked back – recently migrated over 250 Solaris 10 Zones to SmartOS and are very happy with the results :-)
If you’re in a corporate environment, SmartDatacenter may be worth looking at, which is Joyent’s commercial product based on SmartOS.
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed