December 18, 2025Dec 18 This is a complicated issue, and I'm not sure where to begin, but I'll try my best to describe as briefly, but effectively, as possible. About 2 weeks ago I had 2 drives fail on my dual parity array. I first attempted to fix the perceived root cause of power delivery issues, then proceeded with a parity rebuild. After the rebuild completed (seemingly successfully) about a week ago I've been seeing bizarre behavior that got worse today (described below). Background:Because I have historically had a lot of drives becoming disabled, I decided before proceeding with the parity rebuild to move the files off one of the emulated drives that had comparatively little data, then swap out the drive and rebuild from parity. While moving this data manually from the emulated disk to a good disk via Krusader, I ran into an issue where Krusader appdata wouldn't properly move to the good disk. In retrospect this is probably because I was using Krusader and I couldn't move the app data? Some of that appdata moved, as did all the appdata from the other dockers. Anyway, I don't know if that lead to the issues I figured I should mention it.The other scary thing that happened is that I had another disk issue during the rebuild. I don't remember exactly what happened (there was a lot going on) but I believe I had a separate 3rd drive suddenly show as not present during the parity rebuild. Againn, I'm not sure but something that had me very nervous because there were now 3 drives with issues at the same time. I do recall I paused the rebuild, rebooted the server and that drive returned to normal operation, so I proceeded with the rebuild which eventually completed. Again, seemingly without issue.The current issues:So once the rebuild finished, everything seemed to be operating normally and without issue at first. At some point shortly after though I noticed operations failing because they seemed to not have permissions. First I noticed I would be unable to download something through SABNZB which was giving an error that it was unable to create the necessary folder. I also noticed that the download share in Krusader had different files than what had been in there the day before. At this point I did a server reboot and everything returned (correct files in download share via Krusader & SABNZB able to download again). It was almost like the server was recognizing a separate set of shares (or at least one for downloads). I assumed it was a glitch, but the next day I woke up to alerts that appdata backup had failed. I believe the log for that showed a similar permission issue for the backup as SABNZB had been showing. I restarted the server again and was able to manually run a successful appdata backup though. These issues persisted where I would need to restart my server at least daily. Everything would be running like it historically had, then suddenly it seemed like there was a reconfiguration (or something) and many operations were unable to function. Reboots always fixed it until today.Today I was continuing to download some files while researching potential fixes and the cache drive filled up. I had noticed this has been an issue in the last few days though I previously had been able to manually clear SOME space with Mover and keep operating. Today though, most of the space stayed full so I cancelled the downloads and I rebooted the server, but this time I couldn't get Krusader the Krusader GUI to load. All my other Dockers seem to be working still. I deleted and reinstalled the docker but it still wont load. In fact the Docker logs for Krusader wont come up either. The log window shows up for a split second and vanishes.I assume all these issues are related to some kind of corruption, but I have no idea how to proceed with troubleshooting. There is additional detail I could add but this is already way too long and I think I hit the broad strokes.Logs attached. calculon-diagnostics-20251218-1851.zip
December 19, 2025Dec 19 Author Ok, Something about writing everything out must have spurred some thought because I decided to compare the cache and array contents. What I discovered is that it seems everything currently residing on the cache drive is only new since the parity rebuild. The difference across between the cache drive download share and what is contained in the download share on the array disks represent are a perfect match to what I was seeing in Krusader during reboots (2 separate shares). More concerning is that I've now noticed that while the "main" page in the UNRAID GUI shows everything is good with Disk 5, when I click the "browse" button at the far left to view the contents, it loads a blank page that simply says "Invalid path" instead of showing the disk contents (177GB worth). I'm thinking my parity rebuild did fail. Disk 5 is the original disk I manually moved the data off via Krusader before rebuilding on a new disk. I'm not sure why I believe I was able to make space on the cache drive using Mover now though, as it seems something is preventing anything from actually moving from cache to array. Perhaps items that I had been deleting were now actually represented as free space as a result of mover? Maybe it was in my head. I'm sure that's irreverent now. I'm guessing despite my array being "Healthy" that it's actually not and I will need to rebuild with a new configuration, and whatever is on Disk 5 is lost? For now I am going to run an Extended SMART test on Disk 5 and wait your direction. A quick test came back without error.
December 19, 2025Dec 19 Community Expert Dec 18 15:23:26 Calculon kernel: XFS (md5p1): Corruption detected. Unmount and run xfs_repairCheck filesystem on disk5
December 19, 2025Dec 19 Author Thank you, I finished the extended SMART Test and will proceed with the filesystem checkSMART test results are attached in case they provide any value calculon-smart-20251219-1115.zip
December 19, 2025Dec 19 Author OK, so attempts to start the array in maintenance mode are failing:TL;DR: I'm unable to start the array in Maintenance Mode I have checked the Maintenance Mode box and clicked Start 4x.I believe this is reflected at the bottom this shot from the syslog:Each time I select Start the UNRAID working symbol (I just learned that's apparently called a "throbber"?) pops up for 2 seconds and the Main screen returns to how it was showing the array is not started. Unfortunately Disk 5 seems to agree the array isn't started:System logs are attached if needed calculon-diagnostics-20251219-1222.zip
December 19, 2025Dec 19 Community Expert That suggests a plugin issue, reboot in safe mode and try again.
December 19, 2025Dec 19 Author Running the server in Safe Mode I ran the disk Check and received a "File system corruption detected" response. I ran the fix and the result was a dirty log, so I chose to zero the log which resulted in a response of "File system corruption fixed" - cool.I decided to run a follow up Check to verify the fix and I got a "File system corruption detected" response again. Thoughts? Edited December 19, 2025Dec 19 by beatmurph grammer
December 19, 2025Dec 19 Author OK, so I decided to give the server a standard reboot and everything has returned to normal function. I can view the contents of disk 5 again without issue, and apparently there are 1.04TB of data on that disk instead of 177GB it had been showing previously. Should I assume the disk is good, or should I still attempt to correct the alleged file system corruption on that disk? Edited December 19, 2025Dec 19 by beatmurph
December 20, 2025Dec 20 Community Expert 9 hours ago, beatmurph said:Should I assume the disk is good, or should I still attempt to correct the alleged file system corruption on that disk?If it's still detecting corruption, start the array in maintenance mode and post the output fromxfs_repair -v /dev/md5p1
December 21, 2025Dec 21 Author On 12/20/2025 at 2:46 AM, JorgeB said:start the array in maintenance mode and post the output fromas requested . . . Phase 1 - find and verify superblock... - block cache size set to 3026936 entriesPhase 2 - using internal log - zero log...zero_log: head block 316672 tail block 316672 - scan filesystem freespace and inode maps... - found root inode chunkPhase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1data fork in inode 2147495232 claims metadata block 268635728correcting nextents for inode 2147495232bad data fork in inode 2147495232rebuilding inode 2147495232 data forkMetadata corruption detected at 0x489748, inode 0x80002d40 dinodeinode 2147495232 data fork rebuild failed, error 117, clearingcleared inode 2147495232 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes...Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 6 - agno = 3 - agno = 4 - agno = 1 - agno = 2 - agno = 5 - agno = 7entry "Bugonia.2025.1080p.AMZN.WEB-DL.DDP5.1.Atmos.H.264-FLUX.mkv" in shortform directory 2147493503 references free inode 2147495232junking entry "Bugonia.2025.1080p.AMZN.WEB-DL.DDP5.1.Atmos.H.264-FLUX.mkv" in directory inode 2147493503Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - reset superblock...Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - traversal finished ... - moving disconnected inodes to lost+found ...Phase 7 - verify and correct link counts...resetting inode 66987112 nlinks from 4 to 3resetting inode 6576475519 nlinks from 4 to 3resetting inode 2185476466 nlinks from 8 to 7resetting inode 8661071999 nlinks from 5 to 4resetting inode 2187127208 nlinks from 4 to 3resetting inode 10737964385 nlinks from 7 to 6resetting inode 12947217367 nlinks from 5 to 4resetting inode 15033114733 nlinks from 8 to 7resetting inode 15033550189 nlinks from 6 to 5resetting inode 15034299570 nlinks from 3 to 2resetting inode 15035620842 nlinks from 5 to 4 XFS_REPAIR Summary Sun Dec 21 10:40:57 2025Phase Start End DurationPhase 1: 12/21 10:40:49 12/21 10:40:50 1 secondPhase 2: 12/21 10:40:50 12/21 10:40:50Phase 3: 12/21 10:40:50 12/21 10:40:53 3 secondsPhase 4: 12/21 10:40:53 12/21 10:40:53Phase 5: 12/21 10:40:53 12/21 10:40:53Phase 6: 12/21 10:40:53 12/21 10:40:56 3 secondsPhase 7: 12/21 10:40:56 12/21 10:40:56Total run time: 7 secondsdone
December 21, 2025Dec 21 Community Expert If you restart the array in normal mode you can look at what is now showing on disk5. If there is a lost+found folder that will be where the repair process puts any files/folders for which it could not find the correct name.
December 21, 2025Dec 21 Author There is a lost+found folder. Is that considered the corruption? Do I need to do something with those files?
December 21, 2025Dec 21 Community Expert Linux 'file' command can help you figure out what type of data is in each lost+found file, so you can try to open it in an appropriate application and maybe figure out what its name and folder should be.Maybe simpler to try to recover those from backup.16 minutes ago, beatmurph said:Do I need to do something with those files?They are just taking up space unless you decide what to do with them yourself.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.