December 12, 20205 yr I had a 4 Disk array setup with a cached drive. My Cached drive started going into read only after a few minutes of array being online due to hardware errors it seemed from log. I decided to just replace the SSD i used for the cached drive with a larger drive. First I backed up the appdata from cached drive using plugin CA Backup / Restore Appdata. I then shutdown the system, swapped the SSD for the new one and started it up. All was good, I mounted the new SSD and then started the restore process. This failed due to write errors and then the SSD showed up as unreadable due to bad file system. It showed up as needing a format as it was not the right file system, it was the only disk showing here and i checked multiple times. So I performed the requested format it seemed to be very quick and the page just refreshed and it was not in format list anymore. However the system then seemed to hang and lag really bad, when it recovered from hanging 3 of my HDD then showed "Unmountable: No file system" I have no idea what happened to cause this? I decided to reboot the system. After reboot every single drive showed as "Unmountable: No file system" I then rebooted again and then 1 or 2 of the drives recovered. Now after multiple reboots and re-seating every cable and drive and putting the old cached drive back in, I now seem to have 1 drive that is genuinely failed and 1 drive showing "Unmountable: No file system", the unmountable drive has 0 errors and passed smart test. Also the old cache drive now also shows "Unmountable: No file system". All of these drives have been in the array for years, and have data on them. Is there anyway I can fix this and find out how replacing a completely unrelated SSD Cached drive seems to have wrecked my HDD array? Attaching diagnostic logs and a screenshot of my current drive status tower-diagnostics-20201212-1642.zip Edited December 12, 20205 yr by Pear
December 12, 20205 yr Community Expert Connection issues on multiple disks. Check all connections, power and SATA, all disks, both ends, including splitters. Then post new diagnostics
December 12, 20205 yr Author 4 minutes ago, trurl said: Connection issues on multiple disks. Check all connections, power and SATA, all disks, both ends, including splitters. Then post new diagnostics This is a HP Microserver Gen 8 so all the drives are connected via a single cable attached to a backplane. I have re-seated this cable a few times now and the individual drives. But when I replaced the cache drive the only cable i touched was the sata cable for the SSD and the power splitter for the ssd. Edited December 12, 20205 yr by Pear
December 12, 20205 yr Community Expert 8 minutes ago, trurl said: Connection issues on multiple disks. Dec 12 16:36:19 Tower kernel: ata3.00: exception Emask 0x10 SAct 0x1 SErr 0x4890000 action 0xe frozen Dec 12 16:36:19 Tower kernel: ata3.00: irq_stat 0x08400040, interface fatal error, connection status changed Dec 12 16:36:19 Tower kernel: ata3: SError: { PHYRdyChg 10B8B LinkSeq DevExch } Dec 12 16:36:19 Tower kernel: ata3.00: failed command: READ FPDMA QUEUED Dec 12 16:36:19 Tower kernel: ata3.00: cmd 60/20:00:40:29:97/00:00:02:00:00/40 tag 0 ncq dma 16384 in Dec 12 16:36:19 Tower kernel: res 40/00:04:40:29:97/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Dec 12 16:36:19 Tower kernel: ata3.00: status: { DRDY } Dec 12 16:36:19 Tower kernel: ata3: hard resetting link Dec 12 16:36:24 Tower kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Dec 12 16:36:24 Tower kernel: ata3.00: configured for UDMA/133 Dec 12 16:36:24 Tower kernel: ata3: EH complete Dec 12 16:36:25 Tower kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Dec 12 16:36:25 Tower kernel: ata2.00: configured for UDMA/133 Dec 12 16:36:25 Tower kernel: ata2: limiting SATA link speed to 3.0 Gbps Dec 12 16:36:25 Tower kernel: ata2.00: exception Emask 0x50 SAct 0x2 SErr 0x4090800 action 0xe frozen Dec 12 16:36:25 Tower kernel: ata2.00: irq_stat 0x00400040, connection status changed Dec 12 16:36:25 Tower kernel: ata2: SError: { HostInt PHYRdyChg 10B8B DevExch } Dec 12 16:36:25 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED Dec 12 16:36:25 Tower kernel: ata2.00: cmd 60/20:08:00:fe:4f/00:00:03:00:00/40 tag 1 ncq dma 16384 in Dec 12 16:36:25 Tower kernel: res 40/00:0c:00:fe:4f/00:00:03:00:00/40 Emask 0x50 (ATA bus error) Dec 12 16:36:25 Tower kernel: ata2.00: status: { DRDY } Dec 12 16:36:25 Tower kernel: ata2: hard resetting link Dec 12 16:36:29 Tower kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 320) Dec 12 16:36:29 Tower kernel: ata2.00: configured for UDMA/133 Dec 12 16:36:29 Tower kernel: ata2: EH complete ...
December 12, 20205 yr Author 2 minutes ago, trurl said: Dec 12 16:36:19 Tower kernel: ata3.00: exception Emask 0x10 SAct 0x1 SErr 0x4890000 action 0xe frozen Dec 12 16:36:19 Tower kernel: ata3.00: irq_stat 0x08400040, interface fatal error, connection status changed Dec 12 16:36:19 Tower kernel: ata3: SError: { PHYRdyChg 10B8B LinkSeq DevExch } Dec 12 16:36:19 Tower kernel: ata3.00: failed command: READ FPDMA QUEUED Dec 12 16:36:19 Tower kernel: ata3.00: cmd 60/20:00:40:29:97/00:00:02:00:00/40 tag 0 ncq dma 16384 in Dec 12 16:36:19 Tower kernel: res 40/00:04:40:29:97/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Dec 12 16:36:19 Tower kernel: ata3.00: status: { DRDY } Dec 12 16:36:19 Tower kernel: ata3: hard resetting link Dec 12 16:36:24 Tower kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Dec 12 16:36:24 Tower kernel: ata3.00: configured for UDMA/133 Dec 12 16:36:24 Tower kernel: ata3: EH complete Dec 12 16:36:25 Tower kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Dec 12 16:36:25 Tower kernel: ata2.00: configured for UDMA/133 Dec 12 16:36:25 Tower kernel: ata2: limiting SATA link speed to 3.0 Gbps Dec 12 16:36:25 Tower kernel: ata2.00: exception Emask 0x50 SAct 0x2 SErr 0x4090800 action 0xe frozen Dec 12 16:36:25 Tower kernel: ata2.00: irq_stat 0x00400040, connection status changed Dec 12 16:36:25 Tower kernel: ata2: SError: { HostInt PHYRdyChg 10B8B DevExch } Dec 12 16:36:25 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED Dec 12 16:36:25 Tower kernel: ata2.00: cmd 60/20:08:00:fe:4f/00:00:03:00:00/40 tag 1 ncq dma 16384 in Dec 12 16:36:25 Tower kernel: res 40/00:0c:00:fe:4f/00:00:03:00:00/40 Emask 0x50 (ATA bus error) Dec 12 16:36:25 Tower kernel: ata2.00: status: { DRDY } Dec 12 16:36:25 Tower kernel: ata2: hard resetting link Dec 12 16:36:29 Tower kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 320) Dec 12 16:36:29 Tower kernel: ata2.00: configured for UDMA/133 Dec 12 16:36:29 Tower kernel: ata2: EH complete ... I am thinking the cable for the backplane has just randomly failed when I replaced the cached drive. I am starting to think micro servers are not great due to this cable as it is a single point of failure for all 4 drives and is incredibly difficult to replace. I have a second gen 8 with a failed bay that always shows the disk failed in this bay.
December 12, 20205 yr Community Expert Power or controller or backplane ata2=disk1 Dec 12 16:33:57 Tower kernel: ata2.00: ATA-9: WDC WD40EFRX-68WT0N0, WD-WCC4E3LXY9LF, 82.00A82, max UDMA/133 ata3=disk2 Dec 12 16:33:57 Tower kernel: ata3.00: ATA-9: WDC WD40EFRX-68WT0N0, WD-WCC4E6XXFLTJ, 82.00A82, max UDMA/133
December 12, 20205 yr Community Expert 2 minutes ago, Pear said: the cable for the backplane could be
December 12, 20205 yr Author 11 minutes ago, trurl said: could be It is only actually showing 1 failed drive in the array however, just one is showing with no file system. Is there no way for me to get the one with bad filesystem to do a check and show up correctly? Parity disk shows as working correctly Disk 1 shows as Unmountable: No file system Disk 2 is showing as working correctly Disk 3 is showing as a disabled drive and a smart scan wont complete. I have re-seated everything again and just removed the cached drive completely and same issue with same drives. Edited December 12, 20205 yr by Pear
December 12, 20205 yr Community Expert That looks better so far. Disk3 will have to be rebuilt of course since it is out-of-sync but the emulated disk can be read if necessary so we can leave that for now. Since disk1 isn't emulated you have to assume it is actually corrupt so you can try to repair the filesystem: https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui Disk1 SMART isn't looking good for WD RED though, attributes 1 and 200 need to be monitored, you can set that up by clicking on a disk to get to its settings. Serial Number: WD-WCC4E3LXY9LF ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 1653 200 Multi_Zone_Error_Rate ---R-- 199 199 000 - 541 Parity also has some for attribute 1 Do you have backups of everything important and irreplaceable?
December 12, 20205 yr Community Expert Looks like all these disks have a few years on them. Do you have Notifications setup to alert you immediately by email or other agent as soon as a problem is detected?
December 12, 20205 yr Author 3 minutes ago, trurl said: That looks better so far. Disk3 will have to be rebuilt of course since it is out-of-sync but the emulated disk can be read if necessary so we can leave that for now. Since disk1 isn't emulated you have to assume it is actually corrupt so you can try to repair the filesystem: https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui Disk1 SMART isn't looking good for WD RED though, attributes 1 and 200 need to be monitored, you can set that up by clicking on a disk to get to its settings. Serial Number: WD-WCC4E3LXY9LF ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 1653 200 Multi_Zone_Error_Rate ---R-- 199 199 000 - 541 Parity also has some for attribute 1 Do you have backups of everything important and irreplaceable? To be honest, I do not have backups of this data but this is a media server so it is not incredibly important data on here that cannot be replaced. But I would prefer to not have to go through the time to replace it.
December 12, 20205 yr Author 1 minute ago, trurl said: Looks like all these disks have a few years on them. Do you have Notifications setup to alert you immediately by email or other agent as soon as a problem is detected? Not at moment no, but I do look into the error section of the main page regularly to see if there is errors seen. Also I have Fix Common Problems plugin that alerts me of issues when I check the main UnRaid page.
December 12, 20205 yr Community Expert Start with trying to repair filesystem on disk1 then. Be sure to capture any output so you can post it.
December 12, 20205 yr Community Expert Just now, Pear said: look into the error section of the main page regularly Notifications by email or other agent will let you know without having to open the webUI. In any case, if you add 1 and 200 to the SMART attributes to monitor on all WD RED disks you will already have warnings on parity and disk1.
December 12, 20205 yr Author 5 minutes ago, trurl said: Notifications by email or other agent will let you know without having to open the webUI. In any case, if you add 1 and 200 to the SMART attributes to monitor on all WD RED disks you will already have warnings on parity and disk1. Ok I will do this once up and running again. I did a filesystem check on the drive and I got the below : Phase 1 - find and verify superblock... - block cache size set to 729040 entries Phase 2 - using internal log - zero log... zero_log: head block 1779374 tail block 1779370 ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_fdblocks 347588, counted 1328648 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... Maximum metadata LSN (88:1779391) is ahead of log (88:1779374). Would format log to cycle 91. No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Sat Dec 12 18:43:11 2020 Phase Start End Duration Phase 1: 12/12 18:42:57 12/12 18:42:57 Phase 2: 12/12 18:42:57 12/12 18:42:59 2 seconds Phase 3: 12/12 18:42:59 12/12 18:43:08 9 seconds Phase 4: 12/12 18:43:08 12/12 18:43:08 Phase 5: Skipped Phase 6: 12/12 18:43:08 12/12 18:43:11 3 seconds Phase 7: 12/12 18:43:11 12/12 18:43:11 Total run time: 14 seconds Edited December 12, 20205 yr by Pear
December 12, 20205 yr Author With the above log output should I go ahead and run "xfs_repair -v /dev/md1" from an ssh terminal? Edited December 12, 20205 yr by Pear
December 12, 20205 yr Community Expert That doesn't look too bad. Do it again without -n. You will also probably have to do -L. Since Unraid has already determined the disk is unmountable it won't be able to replay the transaction log on the disk.
December 12, 20205 yr Community Expert Just now, Pear said: With the above log output should I go ahead and run "xfs_repair -v /dev/md1" from an ssh terminal? Or you can do it from the webUI. I usually recommend not doing it from the command line since there are some ways you can specify the wrong device, but if you aren't using encryption then that is the correct device (/dev/md1) to repair disk1 in the parity array.
December 12, 20205 yr Community Expert 2 minutes ago, Pear said: Ok running with -nL now 4 minutes ago, trurl said: without -n -n means nomodify so it only checks doesn't repair. OK, I see you edited your post.
December 12, 20205 yr Author Ok it completed as below : Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... sb_fdblocks 347588, counted 1328648 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (88:1779406) is ahead of log (1:2). Format log to cycle 91. done Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... sb_fdblocks 347588, counted 1328648 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (88:1779406) is ahead of log (1:2). Format log to cycle 91. done However at moment still shows unmountable, do I need to come out of maintenance mode before it will try to mount it again? Or is this still failed?
December 12, 20205 yr Author Ok, thanks a lot that fixed disk 1. Screenshot attached. Disk 3 still disabled, is this disk completely failed or can I now just rebuild this?
Archived
This topic is now archived and is closed to further replies.