This is not any knowledge sharing post but one amazing experience that I should never forget and would like to share with you so that in future if you stuck like me this can help you :)
Sunday Afternoon (16 Feb 2014 ). Yes I work on weekends too :( I was working on some database maintenance activity and wanted to install one of the package on XYZ linux server. I typed rpm -qa command to check something and suddenly got an error “Connection refused” ! Nagios started shouting that server is down !! Then I logged into our vm stack and found that kernel crashed because of corrupt/missing C libraries..now no option, tried maintenance mode but no luck :( Only one option was left – boot that vm using live centos image, find existing file system and see if you can correct it.. from where to get live centos DVD ? I searched in entire storage volume and got one :) Booted VM using DVD in rescue mode and found corrupt file. Now how to correct it? I knew that server ABC is centos 5.5 tried copied file from it, matched md5sum and voila!! Server got up! J
Sunday Afternoon (16 Feb 2014 ). Yes I work on weekends too :( I was working on some database maintenance activity and wanted to install one of the package on XYZ linux server. I typed rpm -qa command to check something and suddenly got an error “Connection refused” ! Nagios started shouting that server is down !! Then I logged into our vm stack and found that kernel crashed because of corrupt/missing C libraries..now no option, tried maintenance mode but no luck :( Only one option was left – boot that vm using live centos image, find existing file system and see if you can correct it.. from where to get live centos DVD ? I searched in entire storage volume and got one :) Booted VM using DVD in rescue mode and found corrupt file. Now how to correct it? I knew that server ABC is centos 5.5 tried copied file from it, matched md5sum and voila!! Server got up! J
No comments:
Post a Comment