Disk is Unmountable: Unsupported partition layout after upgrading unraid version

[romanodog]

I have a server set up with 5 drives: 2 parity and 3 data, all 8tb.  I recently upgraded from version 6.7 to to 6.8.3. When the server came back up one of my drives, disk 3, said Unmountable: Unsupported partition layout.  To be honest, I can't say for sure what exact steps I took after this:

I believe I tried just rebooting the server.  When the array was started, it was not emulating that drive, it just acted as if it was not there (the files on it were unaccessable and the available space was reduced by 8tb.)  I stopped the array, clicked on disk 3 to remove it from the array (probably a mistake?).  I added it back to the array and it started to rebuild parity.  Now I was able to start the array, it was rebuilding, and the contents of disk 3 were being emulated, so it was useable for now.  After a day or so, I noticed that the parity rebuild hadn't budged past a certain point.  I also began to have some weird issues like the webgui terminal and the logging pop up window not displaying anything.  I checked top and my CPU usage was very low, which was odd because I figured it should be at 100% from the rebuild.  I rebooted my the server and now the drive is still unmountable but I cannot start the array (emulated) if that drive is not selected, there is a message next to "start" that says a disk is missing.  I've tried moving the drive from the raid card that it's been attached to to the motherboard in case it was a hardware problem.  I ran the xfs repair utility and it didn't appear to show any kind of errors:

 xfs_repair -v /dev/md3
Phase 1 - find and verify superblock...
        - block cache size set to 318696 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 0 tail block 0
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
Phase 5 - rebuild AG headers and trees...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...

        XFS_REPAIR Summary    Mon Apr 27 20:41:56 2020

Phase           Start           End             Duration
Phase 1:        04/27 20:41:50  04/27 20:41:51  1 second
Phase 2:        04/27 20:41:51  04/27 20:41:51
Phase 3:        04/27 20:41:51  04/27 20:41:53  2 seconds
Phase 4:        04/27 20:41:53  04/27 20:41:53
Phase 5:        04/27 20:41:53  04/27 20:41:53
Phase 6:        04/27 20:41:53  04/27 20:41:55  2 seconds
Phase 7:        04/27 20:41:55  04/27 20:41:55

Total run time: 5 seconds
done

When I view the disk log, I get this error in there: emhttpd: error: ckmbr, 2115: No such file or directory (2): open: /dev/sdb1

Indeed I have no sdb1, just sdb.

 

I'm not sure what to do at this point.  I am going to grab another drive to use to replace it or to keep as a backup if this drive is usable, but is the data from that drive just gone now or do you think it is still recoverable from the parity?  Not a huge deal because most of it is backed up elsewhere.

Diagnostics attached.

Any help is appreciated, thanks.

tower-diagnostics-20200427-2103.zip

[JorgeB]

I suspect the problem is controller related, and I would recommend replacing that rockeraid with a recommended controller.

 

If you unassing disk3 and start the array the emulated disk should mount correctly, then you'll need to rebuild, ideally on a new controller.

[romanodog]

I did suspect that, but I moved disk 3 to the motherboard and it still is unmountable. 

I cannot start the array after I remove disk 3 from it, it just says a disk is missing and the start button is disabled. 

Edited April 28, 2020 by romanodog

[JorgeB]

but I moved disk 3 to the motherboard and it still is unmountable. 

That's expected, it will need to be rebuilt.

 

I cannot start the array after I remove disk 3 from it, it just says a disk is missing and the start button is disabled. 

There's should be a check mark to allow the array to start with a missing disk.

 

 

[romanodog]

I am running  SMART check on it now, but I will look for that checkbox after.  I didn't notice it before but maybe I missed it.

I see that RocketRaid 2300 is on the list of working controllers.  I've had it hooked up for over a year with no issues, so I'm not sure that that's where the problem is.  https://wiki.unraid.net/Hardware_Compatibility#PCI_SATA_Controllers

[JorgeB]

21 minutes ago, romanodog said:

I see that RocketRaid 2300 is on the list of working controllers.

It is, but it uses the same driver as the SASLP/SAS2LP, and those have several issues with latest Unraid releases, and it's usually similar, they were working great before until they don't anymore.

[romanodog]

Ah, thanks for the info.  Looks like it's time to do some shopping.

[romanodog]

8 hours ago, johnnie.black said:

There's should be a check mark to allow the array to start with a missing disk.

 

Well I'm a dummy and completely missed that check box   I have the array back up now. 

 

How can I rebuild the disk that has the error?  What's the proper procedure for that?

[JorgeB]

https://wiki.unraid.net/Troubleshooting#Re-enable_the_drive

 

Recommend connecting it to one of the onboard SATA ports first to see if it doesn't happen again.

[romanodog]

Thank you so much, i'm rebuilding again.  Back up drive and a new controller are on the way to my house now.  Much appreciated!