BUG? Failed emulated disk and unmountable after rebuild (12tb->18tb replacement)

[spamalam]

I had a disk that red balled with read/write errors that seem to be related to power/SAS cabling.  I decided to replace this with a pre-cleared drive i keep off the array.  I had recently successfully completed a parity check prior to the disk read/write erroring.

 

I used what I had always done to replace 

1. Stopped array
2. Switched the disk (12tb->18tb, these were in different physical sas slots)
3. Started the array
4. Waited for the rebuild to finish

 

However I'm now met with:

 

Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565106): xfs_growfs /mnt/disk3
Dec  9 18:01:08 beyonder-nas root: meta-data=/dev/md3               isize=512    agcount=13, agsize=268435455 blks
Dec  9 18:01:08 beyonder-nas root:          =                       sectsz=512   attr=2, projid32bit=1
Dec  9 18:01:08 beyonder-nas root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Dec  9 18:01:08 beyonder-nas root:          =                       reflink=1    bigtime=0 inobtcount=0
Dec  9 18:01:08 beyonder-nas root: data     =                       bsize=4096   blocks=3418095603, imaxpct=5
Dec  9 18:01:08 beyonder-nas root:          =                       sunit=0      swidth=0 blks
Dec  9 18:01:08 beyonder-nas root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Dec  9 18:01:08 beyonder-nas root: log      =internal log           bsize=4096   blocks=521728, version=2
Dec  9 18:01:08 beyonder-nas root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Dec  9 18:01:08 beyonder-nas root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565107): mkdir -p /mnt/disk4
Dec  9 18:01:08 beyonder-nas  emhttpd: /mnt/disk4 mount error: Unsupported partition layout
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565108): umount /mnt/disk4
Dec  9 18:01:08 beyonder-nas root: umount: /mnt/disk4: not mounted.
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565108): exit status: 32
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565109): rmdir /mnt/disk4
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565110): mkdir -p /mnt/disk5
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565111): mount -t xfs -o noatime,nouuid /dev/md5 /mnt/disk5
Dec  9 18:01:08 beyonder-nas kernel: XFS (md5): Mounting V5 Filesystem
Dec  9 18:01:08 beyonder-nas kernel: XFS (md5): Ending clean mount
Dec  9 18:01:08 beyonder-nas kernel: xfs filesystem being mounted at /mnt/disk5 supports timestamps until 2038 (0x7fffffff)
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565112): xfs_growfs /mnt/disk5
Dec  9 18:01:08 beyonder-nas root: meta-data=/dev/md5               isize=512    agcount=12, agsize=244188659 blks
Dec  9 18:01:08 beyonder-nas root:          =                       sectsz=512   attr=2, projid32bit=1
Dec  9 18:01:08 beyonder-nas root:          =                       crc=1        finobt=1, sparse=0, rmapbt=0
Dec  9 18:01:08 beyonder-nas root:          =                       reflink=0    bigtime=0 inobtcount=0
Dec  9 18:01:08 beyonder-nas root: data     =                       bsize=4096   blocks=2929721331, imaxpct=5
Dec  9 18:01:08 beyonder-nas root:          =                       sunit=0      swidth=0 blks
Dec  9 18:01:08 beyonder-nas root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Dec  9 18:01:08 beyonder-nas root: log      =internal log           bsize=4096   blocks=476930, version=2
Dec  9 18:01:08 beyonder-nas root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Dec  9 18:01:08 beyonder-nas root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565113): mkdir -p /mnt/disk6
Dec  9 18:01:08 beyonder-nas  emhttpd: shcmd (1565114): mount -t xfs -o noatime,nouuid /dev/md6 /mnt/disk6
Dec  9 18:01:08 beyonder-nas kernel: XFS (md6): Mounting V5 Filesystem
Dec  9 18:01:09 beyonder-nas kernel: XFS (md6): Ending clean mount

 

It looks like unraid failed to emulate the content of disk4 when the disk red-balled which is kind of scary, isn't this the selling point of unraid?  

 

I've got two questions:

1. Did I do something wrong?  I think unraid failed to emulate disk 4 when it red balled the disk, but that feels like a bug in unraid as I'd expect a read/write red balled disk to flip over to parity/emulated.  When i stopped the array and started it, I'd have expect it to copy data from parity with the emulated disk, however it seems unraid never emulated the disk!
2. How do I get my data back/recover?  Format and copy back from the old disk?

 

beyonder-nas-diagnostics-20221209-1712.zip

Edited December 9, 2022 by spamalam
Title clear

[spamalam]

Confirming that parity shouldn't have got corrupted, the parity check was non-write correcting.

 

[itimpi]

There is a bug in the 6.11.2 release that you are running that means disks larger than 2.2 TB are not correctly partitioned.

 

Upgrading to a later release (current one is 6.11.5) and repeating the rebuild should fix your problem.

[spamalam]

1 minute ago, itimpi said:

There is a bug in the 6.11.2 release that you are running that means disks larger than 2.2 TB are not correctly partitioned.

 

Upgrading to a later release (current one is 6.11.5) and repeating the rebuild should fix your problem.

Wow, and i was thinking of doing that upgrade too.  Damn.

 

How do i redo the rebuild?  Unassign and reassign?

[itimpi]

Just now, spamalam said:

Unassign and reassign?

@JorgeB as the expert on these things will probably confirm but I think the steps are:

• stop array
• unassign disk
• start the array.   At this point Unraid should be emulating the drive and it’s contents should be visible
• stop array
• assign disk
• Start array to rebuild.

[spamalam]

14 minutes ago, itimpi said:

@JorgeB as the expert on these things will probably confirm but I think the steps are:

• stop array
• unassign disk
• start the array.   At this point Unraid should be emulating the drive and it’s contents should be visible
• stop array
• assign disk
• Start array to rebuild.

Great, removed the disk and now see a mounted disk 4.

 

I'm pretty sure that there was no emulated disk 4 before building the array though, should i be concerned about that?   I get if there was a bug with formatting at the end it would happen, but surely the mounted disk should be there during the rebuild and the content available?

 

I understood:

1. Red ball event -> disk unmounted from /mnt/disk4 and contents emulated from parity under /mnt/disk4
2. Stopping and restarting array -> contents emulated from parity under /mnt/disk4
3. During rebuild -> contents emulated from parity under /mnt/disk4
4. Completed rebuild -> disk mounted as /mnt/disk4 and contents no longer emulated from parity under /mnt/disk4

?

Edited December 9, 2022 by spamalam

[itimpi]

1 minute ago, spamalam said:

I'm pretty sure that there was no disk 4 before building the array though, should i be concerned about that.   I get if there was a bug with formatting at the end it would happen, but surely the mounted disk should be there during the rebuild and the content available?

No idea what the exact effect of the bug was in terms of what you see during the rebuild.  
 

I THINK what happens is that Unraid uses the sectors from the physical drive for the parts it has so far rebuilt and if the partitioning was incorrect this could have caused problems.

[JorgeB]

1 minute ago, spamalam said:

but surely the mounted disk should be there during the rebuild and the content available?

No because of the bug the new disk wasn't formatted, you could only see the emulated disk before a replacement disk was assigned, just rebuild again and this time it will work (assuming you already updated).

[spamalam]

5 minutes ago, JorgeB said:

No because of the bug the new disk wasn't formatted, you could only see the emulated disk before a replacement disk was assigned, just rebuild again and this time it will work (assuming you already updated).

Understood, thanks for making it clear.

 

I've upgraded and now it emulates during rebuild as expected.