March 12, 20251 yr Hi all, I just noticed that I'm missing a bunch of files (episodes and movies, but I'm sure there are other things missing that I haven't noticed yet). However, UnRAID hasn't sent me any notifications of issues so I'm not sure exactly what's going on. I've attached my diagnostics. Additional info, I have two parity drives and 8 data drives. One of those parity drives is currently being synced and another drive on my array is currently being rebuilt. I had a cable come loose while building my new desk which caused some issues. I've made sure to reseat the cables. The drives came loose about a week ago but it wasn't until yesterday that the files went missing. Please let me know what else I can provide to help with this! tower-diagnostics-20250312-0949.zip
March 12, 20251 yr Community Expert You should probably carry out a check filesystem on each of your array drives.
March 12, 20251 yr Community Expert Corruption on disk5. You should see in MAIN - Array Devices that it is unmountable. Check filesystem on disk5 from the webUI and post the output. Unrelated, your appdata and system shares have files on the array.
March 12, 20251 yr Community Expert I see disk5 is also the disk that is rebuilding. Are you rebuilding it onto the same disk? Usually we try to repair the filesystem of the emulated disk before rebuilding on top.
March 12, 20251 yr Community Expert Just now, trurl said: Usually we try to repair the filesystem of the emulated disk before rebuilding on top. That gives us more options in case the filesystem repair doesn't go well. Maybe the physical disk was OK and not corrupt until you began rebuilding an unmountable filesystem to it. Maybe we could have repaired the emulated disk, and also repaired the physical disk, and maybe we would get different results. 46 minutes ago, dapiedude said: UnRAID hasn't sent me any notifications How do you have Notification setup?
March 12, 20251 yr Author Thank you both for responding! Quote carry out a check filesystem on each of your array drives @itimpi Thanks for the link and the suggestion. Since the drives are XFS, I'll need to put the array into maintenance mode. Is it worth letting the the parity sync / drive rebuild finish (it's at 6%) before checking the filesystem? Or should I cancel the sync / rebuild and check the filesystem now? Quote Corruption on disk5, disk5 is also the disk that is rebuilding. Quote Maybe we could have repaired the emulated disk, and also repaired the physical disk, and maybe we would get different results. @trurl it's the same disk. This is definitely highlighting a huge gap in my knowledge and I appreciate you bringing it up. Whenever a disk gets disabled (for parity sync errors or unmountable drives or whatever), my checklist for 'repair' is: stop the array remove the offending drive from the array using the unraid GUI (the Main page) start the array in maintenance mode stop the array add the drive back into the array start the array In the past, that's all I've needed to do. Repairing the disk is a step I'll need to add, correct? If the repair goes well then I won't need to remove / rebuild? Quote How do you have Notification setup? Through Discord! I see: Quote Warning [TOWER] - array has errors Description Array has 1 disk with read errors Disk 5 - ST14000NM001G-2KJ103_ZTM0N72H (sdc) (errors 2) Priority warning 3/8/2025 3:03 PM Quote Unraid Disk 5 error Alert [TOWER] - Disk 5 in error state (disk dsbl) Description ST14000NM001G-2KJ103_ZTM0N72H (sdc) Priority alert 3/8/2025 3:03 PM Then I went through the steps I outlined above, and got these messages: Quote UnRAID APP — 3/9/2025 10:51 PM Tower Unraid Disk 5 message Notice [TOWER] - Disk 5 returned to normal operation Description ST14000NM001G-2KJ103_ZTM0N72H (sdc) Priority normal 3/9/2025 10:51 PM Quote Tower Unraid Disk 5 message Notice [TOWER] - Disk 5, is being reconstructed and is available for normal operation Description ST14000NM001G-2KJ103_ZTM0N72H (sdc) Priority normal 3/9/2025 10:53 PM Then, I noticed that I was missing files at 6:42pm and went through the steps outlined above again. To which I got these notifications: Quote UnRAID APP — Yesterday at 7:44 PM Tower Unraid Disk 5 error Alert [TOWER] - Disk 5 in error state (disk dsbl) Description No device identification () Priority alert Yesterday at 7:44 PM Quote Tower Unraid Disk 5 message Notice [TOWER] - Disk 5, is being reconstructed and is available for normal operation Description ST14000NM001G-2KJ103_ZTM0N72H (sdc) Priority normal Yesterday at 7:56 PM I'm not sure how helpful the above notifications are in our shared understand of what's going on, but that's all the info I was sent! I believe that my expectation is that the second parity drive should be providing a bridge for the media that was in disk 5 (I'm assuming that these are the files I lost), and it was doing so up until yesterday. Could you tell me or point me to some documentation on generally what's wrong with that expectation in this current scenario? Again, thank you both so much for taking the time to respond and for such timely responses. It's hugely appreciated.
March 12, 20251 yr Community Expert Solution Disabled and Unmountable are different, and often independent conditions. And require different solutions. Unraid disables a disk when a write to it fails for any reason. After a disk is disabled, the disk is no longer used by Unraid because it is out-of-sync. Instead, it is emulated by parity by reading all other disks. Reads from the emulated disk read all other disks and get the data from the parity calculation. The initial failed write, and any subsequent writes to the emulated disk, are emulated by updating parity so those writes can be recovered. https://docs.unraid.net/unraid-os/overview/nas/#parity-protected-array A disabled disk is enabled by rebuilding. A disk is unmountable when its filesystem is corrupt, and so the OS can't load it for access. The filesystem needs to be repaired A disk can be disabled, but the emulated disk is mountable. OK to rebuild assuming everything is working well (why did it get disabled? Bad connections are often the cause) A disk can be enabled, but the disk is unmountable. No rebuild required, repair needed. A disk can be disabled, and the emulated disk is unmountable. This is what you have. Repair and rebuild needed. The parity calculation is supplying the data for the emulated disk, but that emulated disk is unmountable. If rebuilding to a new disk, the original disk can still be used to recover data if repair results of the emulated disk aren't good enough. If rebuilding to the same disk, it is better to repair the emulated disk before rebuild so we have more options. You are currently rebuilding an unmountable filesystem. Too late now, we will have to let it finish and deal with the consequences.
March 12, 20251 yr Community Expert Also note, even dual parity is not a substitute for backups. You must always have another copy of anything important and irreplaceable.
March 12, 20251 yr Author Amazing explanation, thank you so much. Next time, I will repair before rebuilding in this situation where my disk is disabled and the emulated disk is unmountable. Luckily, I'm fastidious with my off-site backups for irreplaceable documents. Everything lost here is easily (if annoyingly) replaceable. You're a boon to this community!!
March 12, 20251 yr Community Expert When rebuild completes, let us know. Still more work to be done with repair.
March 16, 20251 yr Author Hi @trurl, the rebuild is finished! I see that Disk 5's utilization is blank which is what you were talking about. I've attached my diagnostics, if you need them, otherwise I'm all ears for next steps! tower-diagnostics-20250315-2310.zip
March 16, 20251 yr Community Expert Some of your diagnostics didn't work. What do you get from command line with this? df -h
March 16, 20251 yr Author It hung when just running `df -h` so I had to run it on the local file system only, `df -h -l` Image attached!
March 16, 20251 yr Community Expert 3 minutes ago, dapiedude said: It hung when just running `df -h` so I had to run it on the local file system only, `df -h -l` What's that about? rclone? Remote mounts? Any idea why all your plugins are showing in Diagnostics as unknown to CA?
March 16, 20251 yr Author Quote What's that about? rclone? Remote mounts? I ran the command from my phone, so my SMB share to my windows PC wasn't active. That's my only guess. My PC is on now and running the command worked flawlessly. Quote Any idea why all your plugins are showing in Diagnostics as unknown to CA? Not really.. they seem to be fine in the GUI!
March 16, 20251 yr Author Quote Check filesystem on disk5 from the webUI and post the results. Disk 5, I haven't clicked "fix" although I would guess that's the next step! I also went ahead and scanned all of the other disks. I found that my cache drive is also corrupt: None of the others had issues and all looked like:
March 16, 20251 yr Author Disk 5: When pressing "Fix" on the cache, it appears that nothing happens. The syslogs show:
March 16, 20251 yr Community Expert Regarding cache, it looks like a bad NVMe device, and xfs_reapair appears to be unable to repair that filesystem, start the array in maintenance mode and type: xfs_repair -v /dev/md5p1 and post the output just to see if it's any different, most likely it won't
March 16, 20251 yr Author I assume that for Disk 5, I should zero the log? I'm not sure exactly what to do with the NVMe cache though.
March 16, 20251 yr Author Disk 5 wouldn't mount, so I zeroed the log and was presented with "file system corruption fixed". I then checked the drive again and was given this: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 7 - agno = 9 - agno = 11 - agno = 2 - agno = 6 - agno = 8 - agno = 3 - agno = 10 - agno = 12 - agno = 4 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... would have reset inode 2147483826 nlinks from 1 to 2 would have reset inode 6442451773 nlinks from 1 to 2 would have reset inode 2147485973 nlinks from 1 to 2 would have reset inode 10737418377 nlinks from 1 to 2 would have reset inode 17179869328 nlinks from 1 to 2 would have reset inode 2147596279 nlinks from 1 to 2 No modify flag set, skipping filesystem flush and exiting. I then fixed and checked again and was "No file system corruption detected."
March 16, 20251 yr Author I believe that my NVMe cache drive is end-of-life. Here is the smart test: root@Tower:~# smartctl -a /dev/nvme0n1 smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.78-Unraid] (local build) Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: PCIe SSD Serial Number: 19060410243337 Firmware Version: ECFM12.2 PCI Vendor/Subsystem ID: 0x1987 IEEE OUI Identifier: 0x6479a7 Total NVM Capacity: 1,024,209,543,168 [1.02 TB] Unallocated NVM Capacity: 0 Controller ID: 1 NVMe Version: 1.3 Number of Namespaces: 1 Namespace 1 Size/Capacity: 1,024,209,543,168 [1.02 TB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 6479a7 2071512020 Local Time is: Sun Mar 16 16:47:29 2025 EDT Firmware Updates (0x12): 1 Slot, no Reset required Optional Admin Commands (0x0007): Security Format Frmw_DL Optional NVM Commands (0x0054): DS_Mngmt Sav/Sel_Feat Timestmp Log Page Attributes (0x0c): Ext_Get_Lg Telmtry_Lg Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 70 Celsius Critical Comp. Temp. Threshold: 90 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.73W - - 0 0 0 0 0 0 1 + 7.69W - - 1 1 1 1 0 0 2 + 6.18W - - 2 2 2 2 0 0 3 - 0.0490W - - 3 3 3 3 2000 2000 4 - 0.0018W - - 4 4 4 4 25000 25000 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 2 1 - 4096 0 1 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 30 Celsius Available Spare: 100% Available Spare Threshold: 5% Percentage Used: 100% Data Units Read: 3,777,511,029 [1.93 PB] Data Units Written: 2,955,125,403 [1.51 PB] Host Read Commands: 23,963,949,223 Host Write Commands: 7,789,239,035 Controller Busy Time: 127,035 Power Cycles: 283 Power On Hours: 48,824 Unsafe Shutdowns: 194 Media and Data Integrity Errors: 36 Error Information Log Entries: 432 Warning Comp. Temperature Time: 836 Critical Comp. Temperature Time: 0 Error Information (NVMe Log 0x01, 16 of 63 entries) Num ErrCount SQId CmdId Status PELoc LBA NSID VS Message 0 432 1 0x8000 0x4502 - 1000204672 1 - Unrecovered Read Error 1 431 1 0x7000 0x4502 - 1000204672 1 - Unrecovered Read Error 2 430 1 0x6000 0x4502 - 1000204672 1 - Unrecovered Read Error 3 429 1 0x5000 0x4502 - 1000204672 1 - Unrecovered Read Error 4 428 1 0x4000 0x4502 - 1000204672 1 - Unrecovered Read Error 5 427 1 0x3000 0x4502 - 1000204672 1 - Unrecovered Read Error 6 426 6 0x3043 0x4502 - 1000204672 1 - Unrecovered Read Error 7 425 6 0x2043 0x4502 - 1000204672 1 - Unrecovered Read Error 8 424 6 0x1043 0x4502 - 1000204672 1 - Unrecovered Read Error 9 423 6 0x0043 0x4502 - 1000204672 1 - Unrecovered Read Error 10 422 6 0xf043 0x4502 - 1000204672 1 - Unrecovered Read Error 11 421 6 0xe043 0x4502 - 1000204672 1 - Unrecovered Read Error 12 420 8 0x3081 0x4502 - 1000204672 1 - Unrecovered Read Error 13 419 8 0x2081 0x4502 - 1000204672 1 - Unrecovered Read Error 14 418 8 0x1081 0x4502 - 1000204672 1 - Unrecovered Read Error 15 417 8 0x0081 0x4502 - 1000204672 1 - Unrecovered Read Error ... (47 entries not read) Self-tests not supported
March 16, 20251 yr Community Expert 1 hour ago, dapiedude said: "No file system corruption detected." Start the array in normal mode and post new diagnostics.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.