Hello all,

 

I booted up by unraid server last week to find Disk 3 disabled.

 

Upon stopping the array, removing the disk, rebooting the server, mounting again the missing disk and rebooting, the sever reports Disk 3 as "Unmountable: Wrong or missing file system", but now Disk 7 and 9, although mounted and running show the same info.

 

After rebooting the server again, Disk 3 is again disabled and Disk 7 & 9 are again mounted and running without the previous error.

 

At the time I can't re-mount Disk 3 and try to rebuild it.

 

I 've uploaded the diagnostics when the server starts, as well as the diagnostics after trying to re-mount Disk 3 and getting some FS errors for Disks 7 & 9 too.

 

Any help would be much appreciated.

tower-diagnostics-20250319-0923.zip tower-diagnostics-20250319-0942.zip

Handling of unmountable drives is covered here in the online documentation accessible via the Manual link at the bottom of the Unraid GUI.  In addition every forum page has a DOCS link at the top and a Documentation link at the bottom.   The Unraid OS->Manual section covers most aspects of the current Unraid release.

 

In your specific case you will be doing disk3 against the emulated drive, and in the meanwhile keep the physical disk intact.

 

The syslog in the diagnostics is the RAM version that starts afresh every time the system is booted and it appears you rebooted before capturing diagnostics.  You can enable the syslog server (probably with the option to Mirror to Flash set) to get a syslog that survives a reboot so we can see what leads up to the problem if it keeps occurring.  The mirror to flash option is the easiest to set up (and if used the file is then automatically included in any diagnostics), but if you are worried about excessive wear on the flash drive you can put your server's address into the remote server field and log to a share instead.

 

Your diagnostics show that the docker.img file is corrupted as well (which is configured to be on disk8) so that will need fixing as well.  Have you run a memtest recently as this is often caused by bad RAM.   It would also do no harm to run a check filesystem on disk8 to check it has no corruption at that level.   You will probably need to recreate the docker.img file and restore its contents via Apps->Previous Apps to get docker running properly again.

 

If in doubt at any point please ask for clarification.

Hello @itimpi,

 

I did a "Check Filesystem Status" without the "-n", got the following as a result:

 

 

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 7
        - agno = 3
        - agno = 5
        - agno = 2
        - agno = 6
        - agno = 4
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

 

 

 

After the results I started the array in Normal mode, Disk 3 is still disabled (with an "X").

 

Docker has some issues from a few years ago when I had this and some Windows VMs running on an SSD that I eventually replaced with an HDD. I will need to properly set it up again at a later day.

tower-diagnostics-20250319-1208.zip

1 hour ago, cyberstyx said:

Disk 3 is still disabled (with an "X").

That's expected, but the emulated disk is now mounting, assuming contents look correct, you can rebuild on top, but would recommend replacing the cables first to rule that out, since there are a few UDMA CRC errors.

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

The array started with Disk 3 enabled and is currently rebuilding.

 

I will let you know as soon as it finishes (it will take many hours).

I checked your diagnostics to see if repair created any lost+found on disk3, but it looks like there may be lost+found on disk5 instead, presumably from a previous repair. Have you examined that user share?

Yes, Disk 5 was an older issue (3 years ago?) that was tackled.

 

I returned home, checked the progress, it looks like there are other issues now.

 

Disk 3 stopped writing, Disks 7 & 9 (which also reported issues when I was trying to mount Disk 3 with the problematic FS but were ok after each array restart), are now showing errors while participating in the rebuild.

 

Should I stop the rebuild and check Disks 7 & 9 FS status too?

 

tower-diagnostics-20250319-2108.zip

You cannot rebuild multiple disks with single parity, could be a power/connection issue with disks 7 and 9, or the controller, since it's Marvell, if you don't have another controller, try replacing the cables and then try to rebuild again.

I replaced one cable that was old, the others had new cables (installed ~3 years ago). The PSU was also replaced ~3 years ago.

 

Disks 3, 7 and 9 where on the same PCI SATA controller, I moved them all on the second PCI SATA controller that had Disk 8 connected on it and started over the rebuild.

 

The controllers are 2x Digitus PCI SATA with 4 SATA connectors each, which I have them installed for many years now. There are no more free SATA connectors available except the non-used controller that had 3, 7 & 9 on it.

Looks like 3, 7, 9 have all disconnected. Nothing to indicate any need filesystem repair, but you should stop rebuild and fix connections.

I checked its status today, the operation paused after 12 hours and at 24%, Disk 3 went to Disabled status. I will need to cancel the paused operation and try something new.

 

I am attaching the latest info.

 

I found out this thread Recommended controllers for Unraid suggesting to a void the MARVEL chipset and giving other viable options. Do you think the Digitus cards (chipset MARVELL 88SE9230) are giving all the issues?

 

The two SATA expansion cards were bought in 2016 and worked fine, but then it was an older version of unraid too.

tower-diagnostics-20250320-1026.zip

1 hour ago, cyberstyx said:

Do you think the Digitus cards (chipset MARVELL 88SE9230) are giving all the issues?

It's a strong possibility, they can sometimes work fine and then start having issues with a different kernel.

I will then look for a suitable replacement card before retrying with the already given walkthrough.

 

I will close this post, thank you for your help @itimpi, @JorgeB, @trurl

Coming back to this issue, I ordered a "10Gtek ‎LSI-3008-8I HBA Card" from Amazon for 94€ plus 2x "Internal HD Mini SAS (SFF-8643 Host) to 4 x SATA (Target) Hard Drive Cable" for 14€ each, which arrived last week.

 

I flashed the card to IT mode on Saturday and took the expected ~15hours to rebuild a 6TB 5200rpm disk, it finished rebuilding yesterday night.

 

The problem is now resolved, the array is up and running again. Looks like the MARVEL based cards are not suitable for unraid 6x and onwards I guess. Thanks again for your help @itimpi, @JorgeB and @trurl.

 

PS: Flashing a card from IR mode (raid functionality) to IT mode (JBOD functionality) is mostly a pain as there is generic information around that may not suite the specific HBA card you may have and you may need to adapt. There 's quite some investigation you will have to do from generic guides and the manufacturer's info, and then you will have to self-improvise. Broadcom's info on this (who bought Avago who bought LSI) is for the lols, with very good generic info, guides and tools that don't work anymore (building a USB bootable with their suite of tools on it) and scattered to no existing links to the actual individual files you need.