Array disk died & replaced, new disk empty after rebuild

[-C-]

 

 

I had some troubles during the rebuild and had to restart the server, but the rebuild appeared to complete successfully:

 

Any idea what could have caused this?
This is the first time a disk died without warning and I've needed to use parity.

 

I'm assuming the data from the failed drive's gone. Fortunately, I have most of it backed up.

 

I am using syslog to save the logs, so should have logs of everything if they're of use.

[JorgeB]

10 hours ago, -C- said:

I am using syslog to save the logs

Post that and the current diagnostics.

[-C-]

Thanks Jorge

Here they are:

 

syslog-192.168.34.43.log.2.7z

The disk died around 7:20 on 1/9

 

tower-diagnostics-20230918-1104.zip

 

 

[JorgeB]

The disk was formatted zfs during the rebuild:

 

Sep  1 21:11:53 Tower emhttpd: creating volume: disk3 (zfs)

 

So any data there would be gone.

[-C-]

Thanks Jorge,

 

I followed the official guide here: https://docs.unraid.net/unraid-os/manual/storage-management/#replacing-faileddisabled-disks

 

I had missed the point in the notes there "The rebuild process can never be used to change the format of a disk - it can only rebuild to the existing format."

I wasn't using the process to change the format, it was to replace a failed drive, so I skipped over this point. Then when it came to adding the new disk, I figured that as I was having to format the disk anyway, i might as well switch to ZFS so I can take advantage of some of its benefits over XFS like replication from cache etc.

 

Now that I think about how parity works, I realise that there was no way it could have worked with a different format type. It was my mistake. This was the first time I rebuilt a disk from parity, so it was all new to me. Fortunately, none of the data on that disk was irreplaceable and I have a backup of most of it.

 

I do think that there should be a bigger warning about this in the manual to make it clearer that a change of format will stop the rebuild from being able to work. Especially now that I'd imagine there are others like me who'd like to take advantage of ZFS since 6.12 and may be tempted to do what I did.
Even better would be if there's a way for the system to check whether you're trying to rebuild onto a disk with a different format than the one being replaced when attempting to start a rebuild and give a nice clear data loss warning.

[JorgeB]

There already is a warning not to format disks during a rebuild, and the txt changed a couple of times to make as clear as possible, but it seems it doesn't help in some cases.

 

[da02uk]

So if someone had done the same as this and is now in to a 30 hr rebuild of what was a 1tb drive could it be expect that that someone has done it wrong?

 

Its done loads of writes but i cant actually see any data in there at the moment.

I was replacing an older disc and also wanted to update the file system, so have done the same.

I still have the working older disk, so i could just stop the current rebuild, use unbalance to move the data to elsewhere, then just format the new disk and install?

If i move the data around do i need to check parity afterwards?

 

Dont mean to hijack just spotted this thread as being recent.

 

[-C-]

2 hours ago, JorgeB said:

There already is a warning not to format disks during a rebuild, and the txt changed a couple of times to make as clear as possible, but it seems it doesn't help in some cases.

 

 

I took that as a standard "formatting will remove all data on this disk" type warning. I didn't think any of that was relevant to what I was doing- I had an empty disk with no data on it to worry about. To me "...will normally lead to loss of all data on the disk being formatted" is about losing any data on the disk being added, it doesn't mention anything about a different format breaking the rebuild process.

Edited September 18, 2023 by -C-
I wrote that it was an error message. It's a warning.

[-C-]

1 hour ago, da02uk said:

I was replacing an older disc and also wanted to update the file system, so have done the same.

 

Makes me feel slightly less dumb if someone else has done the same thing!

Good luck, I hope you can put that disk in, but I'll leave that to someone more knowledgeable.

[da02uk]

1 minute ago, -C- said:

 

Makes me feel slightly less dumb if someone else has done the same thing!

Good luck, I hope you can put that disk in, but I'll leave that to someone more knowledgeable.

 

I wouldn't judge your dumbness by anything I do, I set a really low bar

 

[-C-]

Having read about people having array drives come up as unavailable due to power issues, or other temporary issue, and turn out to be fine, I figured I'd stick the failed drive in a USB caddy, just to see if there were any signs of life. It's come up as fine, passed quick SMART test. So it may well have been a power glitch or something and a reboot would have fixed it.

 

Not sure whether I should try to use this drive any more, in case there is something up with it.

 

Here's its attributes:

 

I can browse it and access files no problem.

 

Is there any way to put this back into the array with its data intact, or am I better off copying the data onto the disk that's now in the array?

 

So a few lessons learned here- triple read and have a good think about anything array related when there's any possibility of data being lost, and always try turning it off and on again!

Edited September 18, 2023 by -C-
missed some

[-C-]

55 minutes ago, da02uk said:

 

I wouldn't judge your dumbness by anything I do, I set a really low bar

 

Did someone say bar? Great idea!

[itimpi]

9 hours ago, -C- said:

Not sure whether I should try to use this drive any more, in case there is something up with it.

 

I would suggest running an extended SMART test and only consider continuing to use the drive if that completes error free.   If it is not unusual for the short test to work but the extended one to fail.

[JorgeB]

12 hours ago, -C- said:

it doesn't mention anything about a different format breaking the rebuild process.

Not specifically, but it does say format in NEVER part of a rebuild and doing it will result in that loss.

 

9 hours ago, -C- said:

Not sure whether I should try to use this drive any more

Disk looks fine, you can mount it with UD and copy the data to the new disk in the array.