Disk Failed right after Party Build

[a12vman]

Long-time Unraid user here first time I've ever had an issue with a drive.

 

My server spent the last 5 days preclearing and building a brand new 18TB WD Drive from parity to replace a 10TB Drive.

 

Preclear and Parity build completed yesterday with no errors.  

 

I got up this morning and see that the same 18TB Drive is now unmountable.

 

   Where do I go from here?  I have the original 10TB Drive, it's still in the server but unmounted.

 

In looking thru SysLog I see this:

Feb 13 11:21:56 MediaTower kernel: REISERFS warning (device md1): journal-1207 journal_read_transaction: REPLAY FAILURE fsck required! Block to replay is outside of filesystem

Feb 13 11:21:56 MediaTower kernel: REISERFS warning (device md1): reiserfs-2006 journal_init: Replay Failure, unable to mount

Feb 13 11:21:56 MediaTower root: mount: /mnt/disk1: wrong fs type, bad option, bad superblock on /dev/md1, missing codepage or helper program, or other error.

Feb 13 11:21:56 MediaTower kernel: REISERFS warning (device md1): sh-2022 reiserfs_fill_super: unable to initialize journal space

Feb 13 11:21:56 MediaTower emhttpd: shcmd (30): exit status: 32

Feb 13 11:21:56 MediaTower emhttpd: /mnt/disk1 mount error: not mounted

 

 

mediatower-diagnostics-20230213-1132.zip

Edited February 13, 2023 by a12vman

[trurl]

6 minutes ago, a12vman said:

It shows up down in unassigned devices

What shows up as Unassigned? Your screenshot is showing Array Operation, not Unassigned Devices.

 

Disk1 isn't disabled, it is just unmountable.

 

Diagnostics are after reboot so can't see what happened before. Might be a good idea to check connections.

 

Then check filesystem on disk1.

[a12vman]

Here is log info for Disk 1:

 

Feb 13 11:21:25 MediaTower kernel: sd 7:0:2:0: [sdd] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
Feb 13 11:21:25 MediaTower kernel: sd 7:0:2:0: [sdd] 4096-byte physical blocks
Feb 13 11:21:25 MediaTower kernel: sd 7:0:2:0: [sdd] Write Protect is off
Feb 13 11:21:25 MediaTower kernel: sd 7:0:2:0: [sdd] Mode Sense: 7f 00 10 08
Feb 13 11:21:25 MediaTower kernel: sd 7:0:2:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Feb 13 11:21:25 MediaTower kernel: sdd: sdd1
Feb 13 11:21:25 MediaTower kernel: sd 7:0:2:0: [sdd] Attached SCSI disk
Feb 13 11:21:49 MediaTower emhttpd: WDC_WD180EDGZ-11BLDS0_8LG1Z0DF (sdd) 512 35156656128
Feb 13 11:21:49 MediaTower kernel: mdcmd (2): import 1 sdd 64 17578328012 0 WDC_WD180EDGZ-11BLDS0_8LG1Z0DF
Feb 13 11:21:49 MediaTower kernel: md: import disk1: (sdd) WDC_WD180EDGZ-11BLDS0_8LG1Z0DF size: 17578328012
Feb 13 11:21:49 MediaTower emhttpd: read SMART /dev/sdd
Feb 13 11:21:53 MediaTower root: /usr/sbin/wsdd
Feb 13 11:22:03 MediaTower root: /usr/sbin/wsdd
Feb 13 11:22:12 MediaTower wsdd[9419]: udp_send: Failed to send udp packet with Invalid argument
Feb 13 11:22:12 MediaTower wsdd[9419]: udp_send: Failed to send udp packet with Invalid argument
Feb 13 11:22:12 MediaTower wsdd[9419]: Failed to send hello with Invalid argument
Feb 13 12:00:25 MediaTower emhttpd: read SMART /dev/sdd
Feb 13 12:00:51 MediaTower root: /usr/sbin/wsdd
Feb 13 12:03:37 MediaTower kernel: sd 7:0:2:0: [sdd] Synchronizing SCSI cache
Feb 13 12:03:37 MediaTower kernel: sd 7:0:2:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00
Feb 13 12:04:55 MediaTower kernel: sd 7:0:6:0: [sdd] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
Feb 13 12:04:55 MediaTower kernel: sd 7:0:6:0: [sdd] 4096-byte physical blocks
Feb 13 12:04:55 MediaTower kernel: sd 7:0:6:0: [sdd] Write Protect is off
Feb 13 12:04:55 MediaTower kernel: sd 7:0:6:0: [sdd] Mode Sense: 7f 00 10 08
Feb 13 12:04:55 MediaTower kernel: sd 7:0:6:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Feb 13 12:04:55 MediaTower kernel: sdd: sdd1
Feb 13 12:04:55 MediaTower kernel: sd 7:0:6:0: [sdd] Attached SCSI disk
Feb 13 12:04:56 MediaTower unassigned.devices: Disk with ID 'WDC_WD180EDGZ-11BLDS0_8LG1Z0DF (sdd)' is not set to auto mount.
Feb 13 12:04:56 MediaTower emhttpd: WDC_WD180EDGZ-11BLDS0_8LG1Z0DF (sdd) 512 35156656128
Feb 13 12:04:56 MediaTower kernel: mdcmd (2): import 1 sdd 64 17578328012 0 WDC_WD180EDGZ-11BLDS0_8LG1Z0DF
Feb 13 12:04:56 MediaTower kernel: md: import disk1: (sdd) WDC_WD180EDGZ-11BLDS0_8LG1Z0DF size: 17578328012
Feb 13 12:04:56 MediaTower emhttpd: read SMART /dev/sdd
Feb 13 12:06:08 MediaTower root: /usr/sbin/wsdd
Feb 13 12:06:21 MediaTower wsdd[4178]: udp_send: Failed to send udp packet with Invalid argument
Feb 13 12:06:21 MediaTower wsdd[4178]: Failed to send hello with Invalid argument

[itimpi]

Reiserfs cannot handle drives larger than 16 TB (one of the reasons it is now deprecated) so that may be the cause of the problem.   For drives of the larger sizes now available you need to use XFS or BTRFS (or ZFS when Unraid 6.12 is released) as the file system type.

[trurl]

2 hours ago, a12vman said:

Here is log info for Disk 1:

Those timestamps were in the syslog in your diagnostics.

3 hours ago, trurl said:

after reboot so can't see what happened before

  

37 minutes ago, itimpi said:

Reiserfs

You need to get all of your disks off reiserfs by 2025 when it will no longer be supported by linux. 

 

You can't rebuild to another filesystem, the only way is to reformat, so you will need to put their files somewhere else.

3 hours ago, a12vman said:

have the original 10TB Drive

You will probably have to New Config that one back into the array and rebuild parity, then add that large disk to a new slot, format it as xfs, and use it as a destination for copying your reiser disks so you can reformat them. Once you get some of them copied you can reformat them to xfs to use as destinations for files from the other reiser disks.

 

So, a lot of work ahead.

[a12vman]

Oh wow that is a big undertaking.  I am so screwed.  I picked up a new 18TB drive and rebuild from parity. See the screenshot below where Disk 1 is showing invalid size.  

I take it "EB" is short for Exobytes?

 

My default FS in Disk settings is XFS but this drive was formatted as Reiser.  

 

On top of all of this my parity build just completed.  My Movies share is empty even though it still shows as 4.7TB computed space.

 

I have never done the New Config Option before, can someone walk me through it?  

 

 

Edited February 15, 2023 by a12vman

[trurl]

2 hours ago, a12vman said:

My default FS in Disk settings is XFS but this drive was formatted as Reiser

 

On 2/13/2023 at 2:58 PM, trurl said:

You can't rebuild to another filesystem

 

Post new diagnostics

[a12vman]

I stopped the Array, went to tools --> New Config. Choose "All" for Preserve Current Assignments.

Clicked the "Yes i want to do this" checkbox and then clicked "Apply".

Went back to the Main Tab and changed Disk 1 to the original 10TB Drive.

I Started the array. The array came up but started a Parity Sync. I cancelled it immediately.

I browsed my shares and it looks like I have everything except a couple movies/shows that I added in the last couple days.

Now my array is online and everything is running. My Parity Drive says that Parity is Invalid.

There is no "check parity" button.  The only option I see is a "Sync" button which says "Sync will start Parity-Sync and/or Data-Rebuild. "

I just want to be sure that clicking the "Sync" button will re-build the Parity Drive but not write anything to my data drives.

I don't want to lose any data.

[trurl]

2 minutes ago, a12vman said:

clicking the "Sync" button will re-build the Parity Drive but not write anything to my data drives.

Correct.

 

1 hour ago, trurl said:

Post new diagnostics

 

[a12vman]

  

mediatower-diagnostics-20230215-1019.zip

[trurl]

Looks OK so far. You must rebuild parity.

[a12vman]

Thank you  I will re-build.

 

Long-term I need to get these drives away from Reiser.

 

I was reading the LimeTech document about how to migrate away from Reiser.  

 

Why can't I just replace my drives 1 at a time with a new XFS formatted & Pre-Cleared Drive, then Re-Build from Parity?

 

Does the data migration have to involve Rsync or the Unbalance Plugin?

 

Edited February 15, 2023 by a12vman

[trurl]

On 2/13/2023 at 2:58 PM, trurl said:

You can't rebuild to another filesystem

Doesn't matter at all what is on the disk you are going to rebuild to. It can be a clear drive, it can be a drive formatted as XFS or as anything else. It can even be an NTFS drive from Windows full of pron. Rebuild is going to overwrite the entire disk.

 

Parity doesn't know anything about filesystems and files. It is all just bits.

https://wiki.unraid.net/Manual/Overview#Parity-Protected_Array

 

The bits on a disk represent a filesystem with its files, but parity is just going to rebuild the bits. The rebuilt bits will represent the same filesystem with its files as were on the original disk.

 

On 2/13/2023 at 2:58 PM, trurl said:

the only way is to reformat

That will get the disk to the correct filesystem, but it will be an empty filesystem. So first

On 2/13/2023 at 2:58 PM, trurl said:

you will need to put their files somewhere else.

Then after reformat you can copy to the empty filesystem.

[trurl]

Here is the wiki:

https://wiki.unraid.net/Manual/Storage_Management#Converting_to_a_new_File_System_type

Basically, you have to format to a new filesystem, so you have to put the files elsewhere.

There is already another ongoing thread with another user dealing with this. At one point they made the mistake of formatting a drive they hadn't copied yet. We may have even saved most of those files after a lot of effort, but obviously better if the mistake hadn't been made. I link it here in case you might find it instructive.

 

[trurl]

22 minutes ago, a12vman said:

Long-term I need to get these drives away from Reiser.

Before 2025

 

[a12vman]

My parity re-build completed this morning. 

I formatted the 18TB Drive as XFS and started the Pre-Clear with Pre and Post Read.

 

This process is going to take a long time.  I shrunk my array from 4 to 3 drives last year.

 

It took forever to empty a 6TB Drive using UnBalance.  Cant wait to see how long it will take this time.

[trurl]

5 hours ago, a12vman said:

I formatted the 18TB Drive as XFS and started the Pre-Clear with Pre and Post Read.

Don't understand what preclear has to do with anything, and it certainly doesn't make any sense to format a disk and then preclear it.

Are you preclearing a different disk for some purpose? Maybe it was mentioned earlier but I haven't completely reviewed the thread.

[a12vman]

After days of clearing data from Reiser Drives to XFS Drives i am finally finished.

 

All of my drives are formatted as XFS.  Sigh of relief that I didn't lose any data.

 

There were a few things that confused me. 

 

In summary

1. Pre-cleared an 18TB drive.

2. I stopped the array and replaced the 10TB drive with the 18TB drive.   

3.  Started the array to re-build the 18TB from parity.

i started getting errors shortly after the re-build completed.  Soon after that the drive was marked as umountable.  

4.  Stopped the array, pulled the failed 18TB Drive. 

5.  New config--> Preserve All. 

6.  Put the 10TB drive back where the failed 18TB was located.

7.  Mark parity as valid and start the array.

At this point I am back to original state.  

 

I took the failed(by failed I mean drive was making all kinds of clicking sounds that were not normal) 18TB drive back to Best Buy for an Exchange.

New 18TB drive into the server for Pre-Clear.

Formatted Drive XFS.

Followed the process in the "Upgrade Fliesystem" document - emptied each Reiser drive, format, then fill with data. 

 

i am a little suprised that i never got a warning when i attempted to re-build in steps 3 above. 

i don't remember the format procedure the first time, i don't remember being asked for a File Syste Type.