Today we had an issue with one of our database servers hosted in a VMware cluster. Database (mysql) is running on Centos 5 operating system. We were asked to check the VM due to application failure. I was able to see the below error in the VM console.
Seems it had been hanged because of the out of memory. I performed a reboot to see if it came to normal state. Unfortunately, It didn’t boot into the OS saying file system inconsistency and needed to enter the maintenance mode to run the manual file system check.
Big problem was no one has the root password because this is a bit older legacy environment which is not updated. Finally, the option was breaking the root password.
I tried multiple ways to boot the kernel into single user mode, but nothing helped. After change the kernel into boot as single user mode it’s stop in same place as figure 1.1
Finally, I tried the option below.
1. Set the VMware boot delay by editing the VM’s setting [set it to 10000 (10sec)]
2. Interrupt the boot pressing arrow keys
Once you get to the above page select the Kernel entry (2nd row) hit the “e” to edit the boot loader.
Normally you will see nothing after the rhgb quiet. In my case I had to type single fastboot to get me in to the single user mode.
After editing press Enter to accept the editing part and hit “b” to boot using that single user mode (runlevel 1). Then you can see as below. There is the shell to type the passwd to reset the root password.
Once confirmed the password change type reboot to boot the VM. Sometime when we used fastboot kernel might get panic during the reboot. If something goes wrong, please reset the VM using VMware power operation.
Then I was able to enter the maintained mode and run below command to check the file system.
FSCK -y <enter this command 2, 3 times >
Then VM was successfully booted into login screen.
Thanks for reading. Stay safe!!!
