Appliance fails to boot after power outage
Contents
Overview
This article describes how to recover from an unclean shut down of a Swivel hardware or VM appliance. An unclean shut down can occur as a result of a power outage.
Symptoms
The symptom of failing to boot can include prevention of startup, due to an fsck disk check request. It can also include a very rapid (abnormal) boot sequence which leads to a prompt where you are asked for a root password.
In both instances you need to follow the instructions below to perform an fsck of the partitions.
Remote access to the Swivel hardware appliances can be made using the DRAC, see DRAC Card How To Guide
Solution
Boot from the Sysrescue CD
Our recommended method for performing a fsck is by downloading the free sysrescuecd.
This can be obtained from http://www.sysresccd.org, select downloads and then follow the link to download, this will usually be an iso image which for Swivel hardware appliances can be burnt to a CD from the image. VM's should be able to mount the iso image as CD and allow the image to be booted from CD, the boot from CD option may need to be selected for the Swivel server.
You do not need a root password using this method. Its also a requirement that the filesystem is not mounted by the operating systems when you scan it.
Once you've downloaded and burnt to disc (or mounted the ISO file as a CD if using vmware) you can boot from the disc, select 1). default boot options
Mounting a VM CD/ISO image
For VMware it is possible to specify the CDROM to use an ISO image, then when the system boots up, hitting Esc will bring up the Virtual BIOS menu and allow the CD-ROM Drive to be selected as a boot device.
Run the file system checks
The commands to run once you get to the command line of sysrescuecd are:
fsck /dev/sdX1 -y fsck /dev/sdX2 -y fsck /dev/sdX3 -y fsck /dev/sdX5 -y fsck /dev/sdX7 -y
Please Note: Constitute X in the above commands with the actual device e.g /dev/sda1 or /dev/sdb1.
For more recent appliances sda4 and sda6 do not exist and can be safely ignored. Older appliances with sdX4 and sdX6 should have these checked.
fsck /dev/sdX4 -y fsck /dev/sdX6 -y
Running the fsck from the command line
Fsck running
Fsck with sda4 absent
Fsck repairing file system
Restart the system
Restart the Swivel appliance and ensure that it boots correctly.
Retrieving backups
Should the worst happen and the appliance is not recoverable, you can mount the backups directory to scp the contents from within it to a safe location:
mount /dev/sda2 /root/temp
An example scp command to retrieve the files would be:
scp /root/temp root@anothermachine:/home/admin
Where anothermachine is another linux machine with a username of root and a destination directory of /home/admin
If this doesn't work
If it's not possible to fix the problem with the sysrescuecd solution above, you should consider retrieving Swivel backups (as mentioned in the solution) from your broken appliance and deploying them into a freshly deployed VM. If you have not retained the VM you downloaded from us you may need to contact your reseller for a reissue of the download.
Further Troubleshooting
Check the /var/spool/clientmqueue or /var/log/tomcat folders for large numbers of files, this has been seen to prevent booting.
inode too big
Make a note of any inode numbers given as errors and these can be used to find problems if any partitions can be mounted. To search using an inode use:
find / -inum <inode_number>
Example:
find / -inum 722421
Sample Boot error log
Your system appears to have shut down uncleanly.
Press N within 1 second to not force file system integrity check...
Checking root filesystem
Inode 295605 is too big.
/:UNEXPECTED INCONSISTENCY; RUN fschk MANUALLY
[FAILED]
*** An error occurred during the file system check
*** Droping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type Control-D to continue):
Hardware Fault
If the cause is a hardware fault then it may be necessary to contact Dell Hardware support, see Dell Maintenance
Did you take a backup?
It's important to take backups for these occasions. Some other articles related to backup methods for the Swivel appliance can be found here:
