DaveW42

Thanks, @JorgeB! The rebuild process concluded and I've been checking things regularly for a day or two. Everything seems to be working just fine, with no data loss. Thanks so much for the help! Dave

Thanks, @JorgeB ! I somehow missed seeing that box/arrow next to the missing drive. The emulated drive looks good. I have continued the process. It looks like the rebuild may take a few days, since the parity drives are large (14TB). I will report back when it is complete. For posterity's sake, below is the box/arrow part I missed. Thanks! Dave

Hi, @JorgeB. I am currently at Step 4, which was as follows: 4. Important: If the drive to be rebuilt is a data drive then check that the emulated drive is showing the content you expect to be there as the rebuild process simply makes the physical drive match the emulated one. If this is not the case then you may want to ask in forums for advice on the best way to proceed. Before starting this process I had checked the emulated version of Disk 1 and things looked good. Now that I am in the midst of this process, however, I don't see a way to actually check that emulated disk, unless I were to mount it again perhaps (see screenshots below, showing the removal of disk 1 and its appearance now as Dev 1, which is not mounted). I suspect this is probably all normal, and that the review of the emulated disk really needs to happen at the beginning (before step 1). However, I thought I should check because I really don't want to lose any data. Could you confirm that I should simply proceed through the next steps (i.e., stopping the array, reassigning the two disabled disks, etc.)? Thanks again! Dave

Thank you! I will start the process. Dave

Thanks, @JorgeB . Just to make sure I understand this correctly, could you confirm that I should take the steps I have copied below? These are copied directly from here: https://docs.unraid.net/unraid-os/manual/storage-management/#replacing-faileddisabled-disks. I have added some green text in brackets, to denote how I believe that text applies to my specific situation. Thanks so much for your help and insight ! Dave Rebuilding a drive onto itself There can be cases where it is determined that the reason a disk was disabled is due to an external factor and the disk drive appears to be fine. In such a case you need to take a slightly modified process to cause Unraid to rebuild a 'disabled' drive back onto the same drive. Stop array Unassign disabled disk [This means I should unassign Parity 2 and Disk 1, which are both currently disabled. Disk 1 is being emulated. Disk 3 is currently displaying a status of 'green' and 'normal operation,' so nothing needs to occur for Disk 3] Start array so the missing disk is registered Important: If the drive to be rebuilt is a data drive then check that the emulated drive is showing the content you expect to be there as the rebuild process simply makes the physical drive match the emulated one. If this is not the case then you may want to ask in forums for advice on the best way to proceed. [I will make sure to do this. When I look at the emulated drive right now (Disk 1), things look ok.] Stop array Reassign disabled disk [This means I should reassign Parity 2 and Disk 1 to the array] (optional) Tick the box to start in Maintenance mode. If you start the array in Maintenance mode you will need to press the Sync button to trigger the rebuild. The advantage of doing this in Maintenance mode is that nothing else can write to the array while the rebuild is running which maximises speed. The disadvantage is that you cannot use the array in the meantime and until you return to normal mode cannot see what the contents of the disk being rebuilt will look like. Click Start to initiate the rebuild process and the system will reconstruct the contents of the emulated disk. This process can be used for both data and parity drives that have been disabled.

Thanks, @JorgeB ! I had one VM running, disabled auto-restart, shut it down, and then rebooted the server. Attached are the new diags. Thanks! Dave nas24-diagnostics-20250110-1214.zip

Thanks for replying and providing the guidance, JorgeB! I checked the settings for each VM and found that one did have a box checked next to that controller (03 - Windows 10 - 02102021-> 01302023 - Windows 10 Steam Oculus). I removed the check (see screenshot below) and updated the settings. I also looked at "Tools – System Devices" and noted that there was no check in the box by the IOMMU grouping for that controller (see screenshots below). Three of the four drives with read errors are associated with that controller, as I am sure you are aware given the instruction you provided! What should I do next? As noted, I have not rebooted the machine or anything since this issue occurred. If it helps, there is only one VM that I am really running/using at present, and it wasn't the one referenced above. My primary concern right now is just making sure those 3 drives work again and that I haven't lost any files. In short, we can talk about the problems with those remaining VMs another day, I am sure you are swamped with the release of Unraid 7! Dave ...

Hi, everyone. I am running Unraid 6.12.13 with two parity disks, 16 disks in an array, an NVME cache drive, and a few unassigned disk devices. Most of the drives are connected to one of two LSI Logic SAS9211-8I 8PORT Int 6GB Sata+SAS Pcie 2.0 cards, with the others connected to SATA ports on the motherboard. I also have an IO CREST Internal 5 Port Non-RAID SATA III 6GB/s M.2 B+M Key Adapter Card connected to one of two NVME slots on my motherboard (Asus ROG STRIX X570-E GAMING) to provide the opportunity to add a few more SATA drives. I can’t recall if any drives are currently connected to that IO Crest device. Yesterday I completed a full parity check, which showed no errors. I was accessing Unraid tonight to try to troubleshoot 4 VMs that I had been experimenting with a year or so ago, to try to also use Unraid as a gaming system. Each VM essentially represented a trial and error attempt. Tonight I was just trying to start up each VM to try and see where I had left off. I tried to start each one, but none were in a working state (i.e., no Windows startup screen even). This didn’t surprise me (this was how I remember leaving it). I checked the VM logs and a couple indicated that the VMs had started up and crashed. After looking at this I came back to the Main Unraid screen and received error messages indicating that three drives were in an error state, including one of my two parity drives. A warning message indicated that I have 3 disks with read errors. When I look at the Main Unraid screen I see a red ‘x’ next to Parity 2 and Disk 1, both with 3,059 errors indicated. Disk 3 still has a green circle next to it, but it shows 1,014 errors. These three disks also now appear in Unassigned drives, as follows: Disk 1 appears as Dev 1 with an ‘x’ and a “Mount” button next to it that it looks like I could press. Disk 3 appears as Dev 2 with an ‘x’ and a “Mount” button next to it that it looks like I could press. Parity 2 appears as Dev 5 with an ‘x’ next to it and a greyed out ‘Mount’ button. I wasn’t quite sure what to do so I immediately created a diagnostic file (see attached). I haven’t changed or touched anything since creating that file (I have not rebooted, etc.) Unraid is still currently running in this state. Help would be greatly appreciated in recovering these three drives and getting this system back in shape! Thanks, Dave nas24-diagnostics-20250109-2035.zip

The rebuild process and parity check has completed, and everything looks great (no data loss!) Thanks so much for all the help, JorgeB, itimpi, and trurl!!!! It is greatly appreciated. Dave

Thanks, JorgeB! Data rebuild is commencing as indicated. As an additional data point in case anyone is curious, despite having so many drives the parity rebuild process is only requiring an additional 30 watts with respect to power consumption (80 plus platinum PSU). Dave

Got it, thank you. I will unassign both Disk 9 and Parity 2. Thank you! Dave

Contents of Disk 1 and Disk 9 look great, thank you!!! When rebuilding the drive back onto itself, should I UNassign both Disk 9 and Parity 2, or would I just unassign Disk 9? Thanks again!!!! Dave

Attached is the new diagnostics file. Thanks! Dave nas24-diagnostics-20230918-1036.zip

Thanks!

Thanks, JorgeB! So I should restart the array (i.e., not in maintenance mode)? Thanks, Dave

Thanks! Below are the results for Disk 1. Dave Phase 1 - find and verify superblock... - block cache size set to 6071976 entries Phase 2 - using internal log - zero log... zero_log: head block 116970 tail block 116966 ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 7 - agno = 12 - agno = 4 - agno = 5 - agno = 6 - agno = 9 - agno = 1 - agno = 10 - agno = 8 - agno = 11 - agno = 3 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (11:116995) is ahead of log (1:2). Format log to cycle 14. XFS_REPAIR Summary Mon Sep 18 10:05:02 2023 Phase Start End Duration Phase 1: 09/18 10:03:08 09/18 10:03:08 Phase 2: 09/18 10:03:08 09/18 10:03:37 29 seconds Phase 3: 09/18 10:03:37 09/18 10:03:49 12 seconds Phase 4: 09/18 10:03:49 09/18 10:03:49 Phase 5: 09/18 10:03:49 09/18 10:03:50 1 second Phase 6: 09/18 10:03:50 09/18 10:03:59 9 seconds Phase 7: 09/18 10:03:59 09/18 10:03:59 Total run time: 51 seconds done

Thanks! Below are the results for Disk 9. Dave Phase 1 - find and verify superblock... - block cache size set to 6101544 entries Phase 2 - using internal log - zero log... zero_log: head block 3794193 tail block 3794187 ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_fdblocks 125178818, counted 127618391 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 8 - agno = 4 - agno = 5 - agno = 6 - agno = 1 - agno = 7 - agno = 9 - agno = 3 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:3794198) is ahead of log (1:2). Format log to cycle 4. XFS_REPAIR Summary Mon Sep 18 09:49:15 2023 Phase Start End Duration Phase 1: 09/18 09:44:43 09/18 09:44:43 Phase 2: 09/18 09:44:43 09/18 09:45:14 31 seconds Phase 3: 09/18 09:45:14 09/18 09:46:41 1 minute, 27 seconds Phase 4: 09/18 09:46:41 09/18 09:46:41 Phase 5: 09/18 09:46:41 09/18 09:46:42 1 second Phase 6: 09/18 09:46:42 09/18 09:48:03 1 minute, 21 seconds Phase 7: 09/18 09:48:03 09/18 09:48:03 Total run time: 3 minutes, 20 seconds done

Thanks, trurl! Just to confirm, so I should go back to the GUI and this time use the following options for both Disk 9 and Disk 1? -vL Thanks, Dave

Thanks, JorgeB. Didn't realize I was supposed to use the GUI, and am happy to use it. Here is the output for Disk 9 with the -v option specified. Phase 1 - find and verify superblock... - block cache size set to 6101544 entries Phase 2 - using internal log - zero log... zero_log: head block 3794193 tail block 3794187 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. Here is the output for Disk 1 with the -v option specified. Phase 1 - find and verify superblock... - block cache size set to 6071976 entries Phase 2 - using internal log - zero log... zero_log: head block 116970 tail block 116966 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. Thanks! Dave

I ran the following in a terminal in maintenance mode: xfs_repair -v /dev/md9 xfs_repair -v /dev/md1 In both cases I receive error messages saying "No such file or directory" and "fatal error -- couldn't initialize XFS library" Dave

Ok, I purchased a new 1500W PSU (Corsair HX1500i). This should be more than enough power for the system. My previous PSU was 850W. I also checked the cables and the seating of the LSI cards and these looked ok. Attached is the new diagnostic file. Thanks! Dave nas24-diagnostics-20230915-2332.zip

Thanks, JorgeB. Will do. You've got me thinking that my current PSU might not be enough given the number of devices I am running (I think current PSU is 800w). I will buy a new PSU, install it, and then post when I have the new diags (might be two or three days). Dave

Hi, I am running Unraid 6.12.4 with two parity disks, 16 disks in the array, an NVME cache drive, and a few unassigned devices drives. Most of the drives are connected to one of two LSI Logic SAS9211-8I 8PORT Int 6GB Sata+SAS Pcie 2.0 cards, with the others connected to SATA ports on the motherboard. I also have an IO CREST Internal 5 Port Non-RAID SATA III 6GB/s M.2 B+M Key Adapter Card connected to one of two NVME slots on my motherboard (Asus ROG STRIX X570-E GAMING) to provide the opportunity to add a few more SATA drives. Before the problems emerged, no drives were connected to that IO Crest device. I had the case open and had installed an SSD drive (SK Hynix 1TB) to (I believe) a SATA port on the motherboard and (separately) an NVME drive (WD 2TB) to one of the open USB ports on the motherboard using an Inateck USB NVME enclosure, with the intention of using either of these for a new gaming VM. After sealing things up, putting the system back in place, and turning it on, the system lost power briefly. When it came back up Disk 1 and Parity 2 came up as disabled, with the contents of Disk 1 being emulated. I ran an extended Smart Test on the parity disk and there were no errors. I tried adding another HDD to the system (i.e., connecting to the IO Crest via the backplane on my computer case) with the intention of using it to replace Disk 1, but the new disk did not show in unassigned devices. I ran a regular smart test on Disk 1, and there were no errors. Given this I rebooted the array in safe mode, unassigned the disabled drives (Disk 1 and Parity 2), and then started up the array to make sure those drives were removed. I then rebooted in safe mode again, shutdown the array again, added Disk 1 and Parity 2 back in their original position, and started parity sync and the data rebuild process. Things went badly very quickly at this point, with disk 9 and disk 16 almost immediately coming up as disabled. I briefly saw a message flash about CRC errors involving an unassigned device. At around 26.6% of the Disk 1 data rebuild, I was no longer able to interact with the unraid system. However, I could see that a separate Unraid Win 10 virtual machine was still running without issue. I didn’t touch anything and about 15 minutes later the system became responsive again and I could click on menus etc. The system currently shows the following: · Parity 2: red x (parity device is disabled) · Disk 1: Green but lists as “unmountable: unsupported or no file system.” · Disk 9: red x, lists as “unmountable: unsupported or no file system.” · Disk 16: Green but lists as “unmountable: unsupported or no file system.” I don’t believe that Parity 2 has seen much/any real activity as a result of the rebuild. In its disabled state it shows 1 read, 4 writes, and 2 errors. Attached is the diagnostic file. In terms of next steps, should I power down the system, check all cables and make sure that the LSI cards are properly seated in the motherboard? Help would be greatly, greatly appreciated. Dave nas24-diagnostics-20230912-0015.zip

I can see files on both drives!!!! Woohooo!!!!! Thanks go to trurl, JorgeB, and itimpi for all of your help with this !!! 👏 🙌 🍾 It is so much appreciated. I can see family photos on these drives, music, etc. There is no way I could have done this myself, thank you!!!!! Dave

Rebuild took approximately 31 hours to complete. Notification messages indicated rebuild was successful, and all drives appear as green. Attached is the diagnostic file. I haven't touched anything else yet (haven't told dockers and vm's to run). Fingers crossed this diagnostic shows a clean bill of health! Dave nas24-diagnostics-20230224-0945.zip