Data Loss Problems

[stingray060]

Ok, mad panic here.

 

I restarted my server today, when it loaded up it said disk 1 was unreadable, and therefore unmountable,

 

I ran xfs_repair -v and then -L it changed a load of stuff, once i restarted the server again it seems to have wiped the whole drive.

 

Is there a way i can backup of the parity or am i stuffed?

 

Cheers. 

[Squid]

You should post your diagnostics.

 

[trurl]

4 hours ago, stingray060 said:

xfs_repair -v

Did you do this from the webUI or from the command line?

[stingray060]

9 hours ago, trurl said:

Did you do this from the webUI or from the command line?

From the web interface terminal 

[trurl]

6 hours ago, stingray060 said:

From the web interface terminal 

So from the command line then.

 

What exactly was the command you used? It is better to do it from the webUI so you don't make a mistake specifying the device.

 

Might also be helpful to know exactly what the output was from the command.

 

Wish you had asked for help before doing anything.

 

Also

17 hours ago, Squid said:

You should post your diagnostics.

If possible before rebooting and preferably with the array started
Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

[stingray060]

Yeah, so do I now!

 

I did xfs_repair -v /dev/md1 

didnt fix my problem so did

xfs_repair -L /dev/md1

 

I dont know what the results are, but i will attatch the diagnostics.

 

 

vault-diagnostics-20210128-2242.zip

[trurl]

Disk1 is mounted and though it doesn't have a lot of data, it does have some shares on it including the lost+found share that would have been created by the repair if it had to do that. Unfortunately, lost+found typically means that it couldn't figure out what folders the files belong in or the filenames. You will have to sort that out yourself if you can.

 

Did you have a lot of data on the disk?

 

Did you have backups of everything important and irreplaceable?

 

Parity can't do anything for you in this situation, it agrees with the current contents of disk1.

 

 

[stingray060]

I had around 3.5/4Tb worth of data on there. I am not an experienced user when it comes down to it.

 

I have backups of most of it on google cloud, which totals 2TB. But I'm paying around £35-50 a month for that backup. I cant be more glad, that I had it in this instance, but the money I'm forking out, I cant continue to do it, and really need to find a alternative solution. If you have any suggestions, fire away.

 

What is the best way of deleting all info on that drive, and then I can start downloading from google cloud?

 

I have no professional knowledge on this stuff whatsoever, so my practices will not be efficient, or proper. Which could well be the reason for the corruption in the first place.

 

Which leads me on to my final question, how can i prevent this in future!?

 

Thanks a lot.

 

 

[trurl]

All of my important and irreplaceable files will fit easily on a single external drive. I keep a few in rotation for backups. I have a lot more data than that but I don't consider it important and irreplaceable. Since I happen to have an old build I use it as a backup Unraid server for some of those less important files.

 

As for how to prevent corruption, any idea how it happened? Did you have to do a hard powerdown? Disk1 capacity is about what you say you had already used. Did you try to overfill it?

[Hoopster]

9 minutes ago, stingray060 said:

I have backups of most of it on google cloud, which totals 2TB. But I'm paying around £35-50 a month for that backup. I cant be more glad, that I had it in this instance, but the money I'm forking out, I cant continue to do it, and really need to find a alternative solution. If you have any suggestions, fire away.

Some of us are using CrashPlan Pro (there is a docker container for it) for unlimited backup.  The price is $10 a month (not sure what UK pricing is).

 

A recent new suggestion for those that don't like CrashPlan is Jottacloud  personal (7.50 Euros a month) with a Duplicati frontend.  There is also a Duplicati docker container. 

 

Cloud backup for me is in addition to an on-site backup server and external USB drives for the most important stuff.

Edited January 29, 2021 by Hoopster

[stingray060]

Im fairly confident i didnt overfill it, but i cant rule it out.

 

I was not at home when it had its issue, but i was having issues, so i rebooted, which it then said it was unmountable.

 

The main issue i had, was that the google cloud rclone was playing up, i didnt pay much attention to how, but it was showing errors in the log.

 

That's the only process running when i rebooted.

 

 

On the note of google cloud rclone, what is the correct command for syncing?

I currently use: rclone sync -v /mnt/user/Oli Drive:oli_vault/backup/Oli

 

Thanks.

[trurl]

1 minute ago, stingray060 said:

google cloud rclone

I have no experience with that. You might ask on its support thread.

[stingray060]

6 minutes ago, Hoopster said:

Some of us are using CrashPlan Pro (there is a docker container for it) for unlimited backup.  The price is $10 a month (not sure what UK pricing is).

 

A recent new suggestion for those that don't like CrashPlan is Jottacloud  personal (7.50 Euros a month) with a Duplicati frontend.  There is also a Duplicati docker container. 

 

Cloud backup for me is in addition to an on-site backup server and external USB drives for the most important stuff.

Interesting, best option once i am re-established, is to bin off google cloud, and switch to CrashPlan Pro i think, looking at the price difference. 

 

Thank you.

[stingray060]

@trurl

On second thoughts, do you have your backup server mirroring, or backing up your main server? 

 

What is your method of redundancy?

Thanks.

[trurl]

I just do rsync for a few shares occasionally. Like I said, not important stuff.

[trurl]

And the external drives are also just rsync of a couple of shares, but on a regular schedule.

[Hoopster]

1 minute ago, stingray060 said:

do you have your backup server mirroring, or backing up your main server?

 I know you directed to Trurl, but I will answer from my perspective as well.

 

My backup server is in the basement and is powered off unless a backup is needed.  Since the motherboard has IPMI, I have a script that runs once a week automatically and powers on the backup server, backs up share by share to the backup server (initial backup is everything; subsequent backups are new/modified files only), emails me a summary and then powers down the server.

 

For the USB drive external backup, I use the unassigned devices backup script which is set to copy photos, movies and personal videos to the backup drive whenever I plug it into the server.

[stingray060]

1 minute ago, trurl said:

I just do rsync for a few shares occasionally. Like I said, not important stuff.

What command do you use for that? do you use the mirror one or what?

 

I've just remembered, I only just recently upgraded to 2x 4TB drives. I still have (i think) all my old drives, which is a bit of a mess:

 

2x 2TB Nas Drives

1x 1TB Drive

1x 2TB Standard Drive

 

So to recover a little bit of outdated info, i could put these into a new built and fetch out what i can.

 

I intend to turn that "new" build into a on-site backup for future, I cant be bothered with the risk that I've had this time. If any of you know a good method of synicing between two unraid servers, let me know!!

 

You've been very helpful and I am forever grateful! 

 

 

 

 

[stingray060]

2 minutes ago, Hoopster said:

 I know you directed to Trurl, but I will answer from my perspective as well.

 

My backup server is in the basement and is powered off unless a backup is needed.  Since the motherboard has IPMI, I have a script that runs once a week automatically and powers on the backup server, backs up share by share to the backup server (initial backup is everything; subsequent backups are new/modified files only), emails me a summary and then powers down the server.

 

For the USB drive external backup, I use the unassigned devices backup script which is set to copy photos, movies and personal videos to the backup drive whenever I plug it into the server.

OK, I like this idea! 

 

What software do you use for this? 

[trurl]

2 minutes ago, stingray060 said:

all my old drives

You may be able to mount these directly on your server with the Unassigned Devices plugin.

[stingray060]

Just now, trurl said:

You may be able to mount these directly on your server with the Unassigned Devices plugin.

I shall try, makes me scared to loose things!!

[Hoopster]

1 minute ago, stingray060 said:

What software do you use for this? 

It is just an rsync script.  No particular software needed.  I do use IPMI tools from the Nerd Tools plugin to make the power on/off of server via IPMI very easy within the script.

 

The external USB backup script is also just an rsync script (the same one in the first post of the Unassigned Devices forum) modified to include the shares I want to backup.

[trurl]

Just now, Hoopster said:

The external USB backup script is also just an rsync script (the same one in the first post of the Unassigned Devices forum) modified to include the shares I want to backup.

That's basically what I do.

 

And Unassigned Devices can mount shares on other computers so you can do a similar script to rsync shares to another server.

[stingray060]

1 minute ago, Hoopster said:

It is just an rsync script.  No particular software needed.  I do use IPMI tools from the Nerd Tools plugin to make the power on/off of server via IPMI very easy within the script.

 

The external USB backup script is also just an rsync script (the same one in the first post of the Unassigned Devices forum) modified to include the shares I want to backup.

How do you have the email portion done?

[trurl]

3 minutes ago, stingray060 said:

How do you have the email portion done?

The "notify" line in the scripts does the builtin notifications. You have to set up the email first in Settings - Notifications. And be sure to have Notifications alert you immediately by email or other agent as soon as a problem is detected.