dapiedude

Got it! Just checked it thoroughly, nothing important on there but surprisingly some very old files I thought I had deleted

This is what I got

Attached! tower-diagnostics-20250316-1745.zip

I believe that my NVMe cache drive is end-of-life. Here is the smart test: root@Tower:~# smartctl -a /dev/nvme0n1 smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.78-Unraid] (local build) Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: PCIe SSD Serial Number: 19060410243337 Firmware Version: ECFM12.2 PCI Vendor/Subsystem ID: 0x1987 IEEE OUI Identifier: 0x6479a7 Total NVM Capacity: 1,024,209,543,168 [1.02 TB] Unallocated NVM Capacity: 0 Controller ID: 1 NVMe Version: 1.3 Number of Namespaces: 1 Namespace 1 Size/Capacity: 1,024,209,543,168 [1.02 TB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 6479a7 2071512020 Local Time is: Sun Mar 16 16:47:29 2025 EDT Firmware Updates (0x12): 1 Slot, no Reset required Optional Admin Commands (0x0007): Security Format Frmw_DL Optional NVM Commands (0x0054): DS_Mngmt Sav/Sel_Feat Timestmp Log Page Attributes (0x0c): Ext_Get_Lg Telmtry_Lg Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 70 Celsius Critical Comp. Temp. Threshold: 90 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.73W - - 0 0 0 0 0 0 1 + 7.69W - - 1 1 1 1 0 0 2 + 6.18W - - 2 2 2 2 0 0 3 - 0.0490W - - 3 3 3 3 2000 2000 4 - 0.0018W - - 4 4 4 4 25000 25000 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 2 1 - 4096 0 1 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 30 Celsius Available Spare: 100% Available Spare Threshold: 5% Percentage Used: 100% Data Units Read: 3,777,511,029 [1.93 PB] Data Units Written: 2,955,125,403 [1.51 PB] Host Read Commands: 23,963,949,223 Host Write Commands: 7,789,239,035 Controller Busy Time: 127,035 Power Cycles: 283 Power On Hours: 48,824 Unsafe Shutdowns: 194 Media and Data Integrity Errors: 36 Error Information Log Entries: 432 Warning Comp. Temperature Time: 836 Critical Comp. Temperature Time: 0 Error Information (NVMe Log 0x01, 16 of 63 entries) Num ErrCount SQId CmdId Status PELoc LBA NSID VS Message 0 432 1 0x8000 0x4502 - 1000204672 1 - Unrecovered Read Error 1 431 1 0x7000 0x4502 - 1000204672 1 - Unrecovered Read Error 2 430 1 0x6000 0x4502 - 1000204672 1 - Unrecovered Read Error 3 429 1 0x5000 0x4502 - 1000204672 1 - Unrecovered Read Error 4 428 1 0x4000 0x4502 - 1000204672 1 - Unrecovered Read Error 5 427 1 0x3000 0x4502 - 1000204672 1 - Unrecovered Read Error 6 426 6 0x3043 0x4502 - 1000204672 1 - Unrecovered Read Error 7 425 6 0x2043 0x4502 - 1000204672 1 - Unrecovered Read Error 8 424 6 0x1043 0x4502 - 1000204672 1 - Unrecovered Read Error 9 423 6 0x0043 0x4502 - 1000204672 1 - Unrecovered Read Error 10 422 6 0xf043 0x4502 - 1000204672 1 - Unrecovered Read Error 11 421 6 0xe043 0x4502 - 1000204672 1 - Unrecovered Read Error 12 420 8 0x3081 0x4502 - 1000204672 1 - Unrecovered Read Error 13 419 8 0x2081 0x4502 - 1000204672 1 - Unrecovered Read Error 14 418 8 0x1081 0x4502 - 1000204672 1 - Unrecovered Read Error 15 417 8 0x0081 0x4502 - 1000204672 1 - Unrecovered Read Error ... (47 entries not read) Self-tests not supported

Disk 5 wouldn't mount, so I zeroed the log and was presented with "file system corruption fixed". I then checked the drive again and was given this: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 7 - agno = 9 - agno = 11 - agno = 2 - agno = 6 - agno = 8 - agno = 3 - agno = 10 - agno = 12 - agno = 4 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... would have reset inode 2147483826 nlinks from 1 to 2 would have reset inode 6442451773 nlinks from 1 to 2 would have reset inode 2147485973 nlinks from 1 to 2 would have reset inode 10737418377 nlinks from 1 to 2 would have reset inode 17179869328 nlinks from 1 to 2 would have reset inode 2147596279 nlinks from 1 to 2 No modify flag set, skipping filesystem flush and exiting. I then fixed and checked again and was "No file system corruption detected."

I assume that for Disk 5, I should zero the log? I'm not sure exactly what to do with the NVMe cache though.

Thanks @JorgeB! Here is the output:

Disk 5: When pressing "Fix" on the cache, it appears that nothing happens. The syslogs show:

Disk 5, I haven't clicked "fix" although I would guess that's the next step! I also went ahead and scanned all of the other disks. I found that my cache drive is also corrupt: None of the others had issues and all looked like:

I ran the command from my phone, so my SMB share to my windows PC wasn't active. That's my only guess. My PC is on now and running the command worked flawlessly. Not really.. they seem to be fine in the GUI!

It hung when just running `df -h` so I had to run it on the local file system only, `df -h -l` Image attached!

Hi @trurl, the rebuild is finished! I see that Disk 5's utilization is blank which is what you were talking about. I've attached my diagnostics, if you need them, otherwise I'm all ears for next steps! tower-diagnostics-20250315-2310.zip

Amazing explanation, thank you so much. Next time, I will repair before rebuilding in this situation where my disk is disabled and the emulated disk is unmountable. Luckily, I'm fastidious with my off-site backups for irreplaceable documents. Everything lost here is easily (if annoyingly) replaceable. You're a boon to this community!!

Thank you both for responding! @itimpi Thanks for the link and the suggestion. Since the drives are XFS, I'll need to put the array into maintenance mode. Is it worth letting the the parity sync / drive rebuild finish (it's at 6%) before checking the filesystem? Or should I cancel the sync / rebuild and check the filesystem now? @trurl it's the same disk. This is definitely highlighting a huge gap in my knowledge and I appreciate you bringing it up. Whenever a disk gets disabled (for parity sync errors or unmountable drives or whatever), my checklist for 'repair' is: stop the array remove the offending drive from the array using the unraid GUI (the Main page) start the array in maintenance mode stop the array add the drive back into the array start the array In the past, that's all I've needed to do. Repairing the disk is a step I'll need to add, correct? If the repair goes well then I won't need to remove / rebuild? Through Discord! I see: Then I went through the steps I outlined above, and got these messages: Then, I noticed that I was missing files at 6:42pm and went through the steps outlined above again. To which I got these notifications: I'm not sure how helpful the above notifications are in our shared understand of what's going on, but that's all the info I was sent! I believe that my expectation is that the second parity drive should be providing a bridge for the media that was in disk 5 (I'm assuming that these are the files I lost), and it was doing so up until yesterday. Could you tell me or point me to some documentation on generally what's wrong with that expectation in this current scenario? Again, thank you both so much for taking the time to respond and for such timely responses. It's hugely appreciated.

Hi all, I just noticed that I'm missing a bunch of files (episodes and movies, but I'm sure there are other things missing that I haven't noticed yet). However, UnRAID hasn't sent me any notifications of issues so I'm not sure exactly what's going on. I've attached my diagnostics. Additional info, I have two parity drives and 8 data drives. One of those parity drives is currently being synced and another drive on my array is currently being rebuilt. I had a cable come loose while building my new desk which caused some issues. I've made sure to reseat the cables. The drives came loose about a week ago but it wasn't until yesterday that the files went missing. Please let me know what else I can provide to help with this! tower-diagnostics-20250312-0949.zip