DaveW42

Just a quick update to let everyone know that so far so good (the rebuild process has started). Given the size of Disk 1, this task will probably take 24 hours or so to complete. Thanks, Dave

Great! As indicated, I will: 1) stop the array 2) assign the two hard drives (14TB and 10TB) to their appropriate drive slots (Drive 1 and Drive 12, respectively) within the array 3) start the array in normal mode Dave

Hi, trurl. I see all my directories ... and on both drives! Does this mean we are ready to rebuild the drives and leave emulation behind? I will also send you a PM with the actual output of running these statements. Thanks!!! Dave

Thanks so much, trurl. I removed all of the deprecated and "Unknown" plugins you indicated, unmounted the remote shares, and also set them not to automount. I then rebooted and ran a new diagnostic file (see below) I suspect that that nginx error relates to a SageTV media server I have running separately, that likely keeps trying to record OTA tv shows to a corresponding share on unRAID. That share has largely been unavailable, since my array has been down. Hence the errors. Also, at one point about a week ago I issued an "nginx stop" command via the terminal, because I saw the log filling up and had read a thread indicating it might help to stop and restart that service. That just made things worse, so I rebooted (which I would think would have restored the nginx service properly). I just share that last bit in case it might be germane. Thanks! Dave nas24-diagnostics-20230222-1542.zip

Hi, trurl. Please see the attached file. Dave nas24-diagnostics-20230222-1434.zip

Thanks, JorgeB and trurl !! I am holding my breath at this point. I first used the unRAID GUI to browse emulated Disk 1, and everything looked great (woohoo!). No lost+found directory there, as trurl noted. Then I used the unRAID GUI to browse emulated Disk 12. This actually took quite some time to load (unRAID was showing those wavy "loading" lines), but after about 5 minutes I did see file directories (again, no lost+found), and for the first time in a couple of weeks I saw the file directories associated with the files that had been missing (YAHOOOO!!!!!). I then looked back at Disk 1 and tried to navigate within the folders, to look for specific files. At this point I was greeted by the wavy loading lines again, and that is where I have been since that time, with regard to Disk 1. I also tried to navigate to individual files within Disk 12 and experienced the same story (wavy loading lines). Does the above still sound good, and indicate that I should move to rebuild those drives? And with respect to rebuilding, could you confirm that the task of rebuilding would simply involve the following as immediate next steps: 1) stop the array 2) assign the two hard drives (14TB and 10TB) to their appropriate drive slots (Drive 1 and Drive 12, respectively) within the array 3) start the array in normal mode I will also parenthetically note that I am currently seeing new buttons on the Main screen that say "Check" and "History." Next to Check it says "Check will start Read-Check of all array disks." Thanks! Dave

Thanks, JorgeB and itimpi! Disk 1 and Disk 12 still appear with red x's as "Not installed." I made no changes there, and started the array in normal mode as instructed. Attached is the new diagnostic file. Thanks! Dave nas24-diagnostics-20230222-1300.zip

Thanks, itimpi ! Output is as follows. Dave xfs_repair -Lv /dev/md1 Phase 1 - find and verify superblock... - block cache size set to 2975488 entries Phase 2 - using internal log - zero log... zero_log: head block 2227444 tail block 2227440 ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 132352, counted 133504 sb_ifree 1881, counted 1636 sb_fdblocks 1557016821, counted 1485460369 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 9 - agno = 7 - agno = 6 - agno = 8 - agno = 4 - agno = 10 - agno = 11 - agno = 12 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (8:2227435) is ahead of log (1:2). Format log to cycle 11. XFS_REPAIR Summary Wed Feb 22 12:25:07 2023 Phase Start End Duration Phase 1: 02/22 12:22:42 02/22 12:22:42 Phase 2: 02/22 12:22:42 02/22 12:23:12 30 seconds Phase 3: 02/22 12:23:12 02/22 12:23:32 20 seconds Phase 4: 02/22 12:23:32 02/22 12:23:32 Phase 5: 02/22 12:23:32 02/22 12:23:34 2 seconds Phase 6: 02/22 12:23:34 02/22 12:23:50 16 seconds Phase 7: 02/22 12:23:50 02/22 12:23:50 Total run time: 1 minute, 8 seconds done

Thanks! And thanks also for the additional comment regarding our focus on emulated disks (it helps me better understand this process). Below is the requested output. Dave xfs_repair -v /dev/md1 Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock - block cache size set to 2975488 entries Phase 2 - using internal log - zero log... zero_log: head block 2227444 tail block 2227440 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Thanks, JorgeB! I have stopped the array and started it again in maintenance mode. Should I run that command via the terminal? Note that unRAID does not currently give me the option to run xfs_repair via the GUI on an unassigned device. Dave

Sorry for any missteps on my part! I am trying the best I can to proceed step-by-step through the guidance provided. Since I am a newbie to the inner workings of unRAID, I hadn't realized the significance of unassigning a drive and what the implications were. Thanks for clarifying the next step, trurl: Current state of machine: Disk 1 and Disk 12 still appear with red x's as "Not installed." Array is running in normal mode. Attached is the new diagnostic file. Thanks, Dave nas24-diagnostics-20230222-0939.zip

Hi, trurl. I'm not quite sure what you are asking me to do as a next step. Could you clarify? The array is currently running in normal mode, with no disks assigned as Disk 1 and Disk 12. The 14TB disk that was disabled (i.e., as Disk 1) currently appears as Dev 2 in Unassigned Devices. It is also labelled as sdd and is not mounted. The 10TB disk that we ran -L on (i.e., as Disk 12) is in Unassigned Devices as Dev 1 (sdh), and is not mounted. When I click on Dev 2 in Unassigned Devices, there is no area for me to run xfs_repair via the GUI. If I am understanding the manual correctly, this is the part where I need to be very careful and make sure to fix the drive by going through unRAID parity management. Since the Disk 1 14TB drive is currently placed in Unassigned Devices, I believe this means that that drive is not on the array and is not currently regarded as a participant in unRAID parity management. Do we need to do our next steps via the terminal, since the GUI does not show xfs_repair for Dev 2? Also, as a quick reminder from my first post, this particular 14TB drive was disabled by UNRAID, and I have taken no steps with it since that time (i.e., I have NOT yet tried to replace this drive with a new drive, to let it sync/rebuild based on the emulated version of the disabled drive). Would it be safer to just sync/rebuild this drive onto a new precleared drive, and then run xfs on the replacement drive? Please let me know. Thanks again for the help and insight! Dave

Thanks, trurl. Disk 1 and Disk 12 still appear with red x's as "Not installed." I started the array in normal mode. Attached is the new diagnostic file. Thanks! Dave nas24-diagnostics-20230221-2244.zip

Thanks, trurl! Disk 1 and Disk 12 now appear with red x's as "Not installed." I started the array in maintenance mode. Attached is the diagnostic file. Thanks! Dave nas24-diagnostics-20230221-2204.zip

I believe I assigned disk 1 in order to see the option to run xfs_repair on it. If I don't assign it, it goes to Unassigned Disks and I don't see the option to run xfs_repair on that drive ...

Also, I should note that in the GUI Disk 1 now appears in blue, with the mouseover text showing "New device."

Thanks, trulrl. I stopped the array to run xfs_repair via the GUI, and I have checked the "maintenance mode" box. There is a message next to that box that says "Maintenance mode - if checked, Start array but do not mount disks." However, above that message is another message indicating "Start will start Parity-Sync and/or Data-Rebuild." If I click "Start" to start the array, will UNRAID actually start a parity-sync and/or data-rebuild as the message is indicating, or will it just let me access Disk 1 so I can run xfs_repair in maintenance mode? I just want to make sure it doesn't try to rebuild Disk 1 onto itself, which I think would be bad. Thanks! Dave

Thanks, trurl! I made sure both disks were removed (via GUI), and I started the array in normal mode. Both disks now have a red 'x' next to them. The mouseover says "Device is missing (disabled). Contents emulated" for both disks. Attached is an updated diagnostic file. Thanks! Dave nas24-diagnostics-20230221-1452.zip

Thanks, JorgeB. I proceeded as you indicated. Below are the results (the first block was run with just -v, and the second block was run with -Lv . I then tried to start the array in normal mode (not maintenance), which triggered a rebuild. But then it occurred to me that Disk 1 was set as "Not Installed" (recall that I have two drives with an issue). I had removed Disk 1 using the GUI, to avoid the chance that at some point it might try to rebuild that drive onto itself, as opposed to a replacement drive. I wasn't sure if not having anything assigned to Disk 1 might create a problem for the rebuild of Disk 12, so I immediately cancelled the rebuild of Disk 12 using the GUI, and stopped the array. The GUI currently reads "Start will bring the array on-line and start Data-Rebuild." Which of the following should I do now, or should I do something else? Option A) restart the array in normal mode, with Disk 1 left as "Not Installed" and Disk 12 assigned to the 10TB drive that was just repaired with -L Option B) reassign Disk 1 to the original 14TB that UNRAID disabled (and which I believe UNRAID regards as unmountable), then restart the array in normal mode with Disk 12 assigned to the 10TB drive that was just repaired with -L Option C) replace Disk 1 with one of my new precleared 14 TB drives, then restart the array in normal mode with Disk 12 assigned to the 10TB drive that was just repaired with -L I guess I'm just not quite sure if any of this might trigger a parity check as part of the data rebuild, and I want to make sure the system is properly configured should that occur. I'm also not quite clear how emulation is fitting into this picture as well. For example, is emulation occurring if no drive is assigned to Disk 1? And would UNRAID at some point *forget* the need to emulate Disk 1? I still don't quite understand why certain files appear lost despite messages indicating that both Disk 1 and Disk 12 were being emulated. Is there a link or forum post that you could point me to, so I could understand that part better? Thanks again so much! Dave BLOCK 1 Phase 1 - find and verify superblock... - block cache size set to 3005320 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 1075093 tail block 1075089 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. BLOCK 2 Phase 1 - find and verify superblock... - block cache size set to 3005320 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 1075093 tail block 1075089 ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 0, counted 262400 sb_ifree 0, counted 1948 sb_fdblocks 2441087425, counted 140977217 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 3 - agno = 6 - agno = 2 - agno = 5 - agno = 8 - agno = 7 - agno = 4 - agno = 9 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:1075085) is ahead of log (1:2). Format log to cycle 4. XFS_REPAIR Summary Tue Feb 21 10:56:10 2023 Phase Start End Duration Phase 1: 02/21 10:53:21 02/21 10:53:21 Phase 2: 02/21 10:53:21 02/21 10:53:49 28 seconds Phase 3: 02/21 10:53:49 02/21 10:54:23 34 seconds Phase 4: 02/21 10:54:23 02/21 10:54:23 Phase 5: 02/21 10:54:23 02/21 10:54:23 Phase 6: 02/21 10:54:23 02/21 10:54:54 31 seconds Phase 7: 02/21 10:54:54 02/21 10:54:54 Total run time: 1 minute, 33 seconds done

Also, please let me know if any of the above output suggests I should spin the array up in non-maintenance mode before doing this (i.e., to try to mount drive 12 and play the log back again), and then spin the array down again, click maintenance mode, and then spin the array up again before running -L on disk 12. Thanks! Dave

Thanks so much for those explanations, JorgeB and trurl !!! Very interesting stuff. I think the concept associated with parity drives is brilliant, and I have mentioned the whole idea to non-techy friends as an example of something that appears impossible (being able to protect numerous large drives using just one large drive) but is possible through logic and basic math. Very cool. So, to make a long story short, I decided that trying to take an image of drive 12 might actually cause me more problems in the long run (e.g., if I inserted that drive into another computer, and somehow that other computer decided to write something to that drive). After the drives precleared, I therefore just decided I would just follow the initial direction both of you had provided, which was to move forward with xfs_repair -L as you both were suggesting. Before I did that, just for the heck of it I ran xfs_repair with the -nv options, just to see what would happen. Lo and behold I got a totally different set of output, which I have shared below. I think ultimately this still means I should just run the "-L" option, but I thought I should check first. Please let me know what you think. Say the word, and "-L" will be my next step. Thanks! Dave Phase 1 - find and verify superblock... - block cache size set to 3005320 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 would reset superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 would reset superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 would reset superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 1075093 tail block 1075089 ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_icount 0, counted 262400 sb_ifree 0, counted 1948 sb_fdblocks 2441087425, counted 140977217 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 2 - agno = 3 - agno = 4 - agno = 8 - agno = 6 - agno = 7 - agno = 9 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Tue Feb 21 01:55:39 2023 Phase Start End Duration Phase 1: 02/21 01:54:34 02/21 01:54:34 Phase 2: 02/21 01:54:34 02/21 01:54:34 Phase 3: 02/21 01:54:34 02/21 01:55:08 34 seconds Phase 4: 02/21 01:55:08 02/21 01:55:08 Phase 5: Skipped Phase 6: 02/21 01:55:08 02/21 01:55:39 31 seconds Phase 7: 02/21 01:55:39 02/21 01:55:39 Total run time: 1 minute, 5 seconds

Thanks, JorgeB. I'm not quite sure I followed that. Are you saying that whatever was written to disk 1 and disk 12 immediately before they got disabled would be missing, or are you saying that UNRAID might have tried to continue to write to those drives even after they were disabled (i.e., UNRAID would be writing to and updating an emulated disk)? Or is it something else? I'm also not quite sure I understand how the emulation works (I'll have to look that one up). I believe UNRAID was indicating that both drives were emulated, after the drives were disabled. Having said this, what didn't quite make sense to me was that -- if both drives were truly emulated--then why were those files missing? I would have thought I would have seen those files, since they were being emulated. Also, preclearing is still going as we speak (pre-read, zero, post-read) for all 4 new drives. I will shortly be investigating what programs to use to do the imaging. Not sure whether I should stay in the Linux family for this, or bring things over to Windows, which I am more familiar with. I wish I had learned more about Linux a long time ago, as I do like this framework much better than Windows. Thanks! Dave

Hi, JorgeB. Thank you! That information is very helpful. Time is definitely a factor, but I would also definitely be willing to go the more conservative route and image the drives (both the original drive that UNRAID disabled and the rebuilt, synced drive) if it improved my chances of recovering the data even slightly. With this in mind, I have four 14TB WD drives coming via next day service. These will also come in handy when I get past this issue and one day figure out a local backup solution for my array. It may take me a bit before I am able to update this thread with any new information ... I will need to preclear those drives, running two at a time using the device I mentioned in my first post. I just did this a week ago, and I think it took 3 or 4 days per pair of drives, given all the preclear steps. And then I will need to 1) figure out what software to use to image those two drives and 2) run that imaging process. I really hope "-L" works, it would be nice for a quick solve once I am through all of this setup! I will keep monitoring this thread in case anyone has any new thoughts on all of this, but I think I wouldn't really have an update until I am through all these steps (maybe 7 to 10 days??) Thanks again! Dave

Thanks for your thoughts on this, trurl! Would there be any advantage to imaging the drive before I run xfs repair with -L ? Or would it just be a waste of time to do so, in hopes of additional attempts to salvage files using File Scavenger and TestDisk? Thanks also, JorgeB, for your additional comment. Dave

Also, I suspect we are in very different time zones (almost 4am here). Unfortunately I have to work in the morning and need to catch a bit of sleep. I'll take a look again at your guidance in the morning. Thanks again so much for all your help!!! Dave