Everything stoped working after powerutage

[Eriic91]

Had a power outage today, I started the server back up and it was very strange, half of my dockers wouldn't start and gave the message "server error",

My vm's are no where to be seen, and after some troubleshooting "read - probably made it worse" I get the message "Libvirt Service failed to start."

I can't see my shares from the web UI, I SSH into the server and saw that all my shares now is under "user0" and "user" is empty.

Im starting to lose my mind, can someone take a look at my diagnostics and see what caused this?

 

I was consedering using the option at spaceinvaders patreon where he helps you with remote support, but that one was sold out.

Anyone else that offer that service? 

Don't wanna lose my stuff.

tower-diagnostics-20190924-1915.zip

[Hoopster]

6 minutes ago, Eriic91 said:

Had a power outage today, I started the server back up and it was very strange, half of my dockers wouldn't start and gave the message "server error",

My vm's are no where to be seen, and after some troubleshooting "read - probably made it worse" I get the message "Libvirt Service failed to start."

I can't see my shares from the web UI, I SSH into the server and saw that all my shares now is under "user0" and "user" is empty.

Im starting to lose my mind, can someone take a look at my diagnostics and see what caused this?

 

I was consedering using the option at spaceinvaders patreon where he helps you with remote support, but that one was sold out.

Anyone else that offer that service? 

Don't wanna lose my stuff.

tower-diagnostics-20190924-1915.zip 100.88 kB · 1 download

Your data is likely safe and intact.  

 

First lesson: get your server on a UPS.  From what you are describing, I assume you have no UPS for the server?

 

It happens quite frequently that a power outage either corrupts the data on the unRAID flash boot drive or kills the device altogether.  It is likely from what you are describing that data on the flash drive has been corrupted.

 

Shut down the server, remove the flash drive, and insert it in a Windows PC and let it run a chkdsk on it.  That may solve the problem.  If it does not, do you have a recent backup of your flash drive contents (you really should be backing that up somewhat frequently)?  If so, you can wipe the flash drive and restore your backup to it.

 

Even with no backup, all is not lost if there is a flash drive problem.  Your data is probably still OK and you can create a new config.  Don't worry about that until you have determined if there is corruption on the flash drive that can be corrected.

[Hoopster]

FYI - /mnt/user and /mnt/user0 both reference the unRAID user shares.  /mnt/user0 excludes the cache drive from the view but should be otherwise identical to /mnt/user

[Eriic91]

Found an old backup from july, giving that a shoot.

 

I can't see anything in my "user" folder when I SSH into it, it used to be the otherway around.

 

[itimpi]

The syslog in the diagnostics shows that you have a corrupt BTRFS file system, so that probably explains your symptoms.

[Eriic91]

What does that mean?

[Eriic91]

Some succes with the back-up, Got my shares back, but the docker is empty and VM tab still says "Libvirt Service failed to start."

[Hoopster]

Yeah, your diagnostics also showed no share.cfg file.  That's why your shares are not showing up in /mnt/user.

 

Something definitely got corrupted.

 

EDIT:  it looks like you got that resolved

 

You'll need to fix the file system corruption to resolve other issues.  

 

See this wiki and read up on methods to correct BTRFS errors.

Edited September 24, 2019 by Hoopster