HenkaN

I've got some DeLOCK 10x sata expansion card.

It's not always on the same SSD, and this is the first time i saw it on the WD drive. Moved all 6 drives to the motherboard connectors now and only use the expansion card for the two i'm preclearing, and it seems fine right now actually. Might be a problem with that card then, assuming the errors won't come after a while.

So i've seen these errors before (not frequent at all) on my server but they've "fixed themselves" so i didn't bother investigating further. However now it's becoming an actual problem since it's never ending and ended up corrupting stuff once (guess i should've not ignored it in the first place huh). Jul 1 22:19:51 Henkraid kernel: ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Jul 1 22:19:51 Henkraid kernel: ata9.00: failed command: WRITE DMA EXT Jul 1 22:19:51 Henkraid kernel: ata9.00: cmd 35/00:08:58:e1:ea/00:00:e8:00:00/e0 tag 17 dma 4096 out Jul 1 22:19:51 Henkraid kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 1 22:19:51 Henkraid kernel: ata9.00: status: { DRDY } Jul 1 22:19:51 Henkraid kernel: ata9: hard resetting link Jul 1 22:19:51 Henkraid kernel: ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Jul 1 22:19:51 Henkraid kernel: ata9.00: configured for UDMA/133 Jul 1 22:19:51 Henkraid kernel: ata9: EH complete Worth to mention is it doesn't happen frequently at all if i just let the system run normally, but a few days ago i removed 3x1tb drives and added 2x10tb drives and started preclearing them. The preclear combined with docker containers running will trigger this to happen, and it won't stop unless stuff start getting corrupted and unRaid decides the SSD(s) are faulty or i reboot the server when the log won't stop filling with it. I've tried what i can do hardware wise so far, different sata cables, different ports both on motherboard and expansion card, pulled the GPU to make sure there's enough power for the drives, i even precleared the ssd's to see if it reported anything weird but it finished successfully (no idea it that was a good idea). It can handle a parity check + two preclears at the same time but with no docker containers running, so it has something to do with the ssd's i think. I don't believe they are broken though and it's starting to feel like a hardware error, but what do i know... Will try with a different PSU as soon as i can find the right cables i need for my spare one. If anyone have any idea about this please do tell, after i've tried another PSU i don't know what to do. Thanks. henkraid-diagnostics-20190701-2109.zip

From what i've noticed so far i lost very little. Your help is very much appreciated. Thanks alot!

Alrighty that seems to have fixed it now, it's detected as an xfs filesystem and it mounts without any issues from what i can tell so far. Will provide the xfs_repair output and my diagnostics just in case there's anything more to it. I guess now i just have to slowly go through the data and see what's lost due to all of my problems lol. Thanks alot man! root@Henkraid:~# xfs_repair /dev/md6 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair. --------------------------- root@Henkraid:~# xfs_repair -L /dev/md6 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) - scan filesystem freespace and inode maps... ir_freecount/free mismatch, inode chunk 3/35062848, freecount 0 nfree 5 inode rec for ino 6521313728 (3/78862784) overlaps existing rec (start 3/78862784) agi_freecount 57, counted 14 in ag 3 sb_icount 2496, counted 2560 sb_ifree 397, counted 308 sb_fdblocks 496361854, counted 491894032 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 imap claims a free inode 6477511085 is in use, correcting imap and clearing inode cleared inode 6477511085 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 entry "South Park - S01E13 - Cartman's Mom is a Dirty Slut Bluray-720p.mp4" at block 0 offset 2824 in directory inode 6477511060 references free inode 6477511085 clearing inode number in entry at offset 2824... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 6477511060 (no data entry): rebuilding rebuilding directory inode 6477511060 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (4:49915) is ahead of log (1:2). Format log to cycle 7. done --------------------------- root@Henkraid:~# xfs_repair /dev/md6 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Phase 3 - for each AG...de chunk Phase 3 - for each AG...de chunk - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done henkraid-diagnostics-20190520-1541.zip

It doesn't look like disk6 is detected as an xfs filesystem. I restarted the system and it don't look like that changed anything. Not sure if that's a problem? Did this, didn't run the actual repair (the option to check it isn't in the gui). It says "FS auto", "Unmountable: No file system". And on the bottom "Unmountable disk present:Disk 6 • WDC_WD40EFRX-68N32N0_WD-WCC7K2RK8VU8 (sdb)". And i have the option to format it. root@Henkraid:~# xfs_repair -n /dev/md6 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) - scan filesystem freespace and inode maps... ir_freecount/free mismatch, inode chunk 3/35062848, freecount 0 nfree 5 inode rec for ino 6521313728 (3/78862784) overlaps existing rec (start 3/78862784) agi_freecount 57, counted 14 in ag 3 sb_icount 2496, counted 2560 sb_ifree 397, counted 308 sb_fdblocks 496361854, counted 491894032 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 imap claims a free inode 6477511085 is in use, would correct imap and clear inode - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 entry "South Park - S01E13 - Cartman's Mom is a Dirty Slut Bluray-720p.mp4" at block 0 offset 2824 in directory inode 6477511060 references free inode 6477511085 would clear inode number in entry at offset 2824... No modify flag set, skipping phase 5 Inode allocation btrees are too corrupted, skipping phases 6 and 7 Maximum metadata LSN (1:26702) is ahead of log (0:0). Would format log to cycle 4. No modify flag set, skipping filesystem flush and exiting. henkraid-diagnostics-20190519-2056.zip

The disk i was testing in another pc hasn't been put back in the array. That's the disk i'm rebuilding right now. And what i mean by the metadata things not written to parity was since they happened during a bad reboot, it might not have affected the parity disk.

Does that log to a file? Last time i ran xfs_repair it barely returned any information at all.

I have notifications, are there anything you see in the diags that i should be aware of?

Okay here's the full picture. My server kinda froze, i could navigate the UI but i couldn't change anything, reboot, shutdown, nothing. So i had to pull the power to reboot it. When i powered it back up i got this metadata error, tried to correct it with xfs_repair, it needed -L as you said here. That didn't do anything. And every time i tried to start up the array it would get stuck on mounting disk6 and throw this. (And require me to cut the power because nothing worked when that happened). This pissed me off for a while, had people tell me it's probably cable issues, the drive is failing and i don't know what. So i decided to pull the drive and plug it into another pc to see if it was fine there or not. Turns out it was fine, i was still pissed though so i formated it and wrote a ton of data to it to see if anything weird happened. I honestly don't even care anymore if all or some of the data is lost/corrupted. However it feels like that metadata issue will be the next to deal with and i'm back at square one, unless xfs_repair decides to actually fix it now. My hopes was that the corruption wasn't written to parity (like that would actually be the case :p), seems like it is now though? However we'll see in about 5 hours if that's still the case or not.

Alright. This is as of right now with the rebuild ongoing. henkraid-diagnostics-20190519-1103.zip

Alright, we'll see in about 6 hours if it turns out a success. At least it looks better than i hoped for so far, big thanks man!

Seems to be working and it does indeed say "Unmountable: No file system". You want me to start array in maintenance mode after it's done and "Check Filesystem Status"?

Yeah we can't use the original disk6. That has to be rebuilt. I have no idea about these kind of things, how do i enable disk5 again? All i've done with disk5 after this happened is mount it with unassigned devices and pulled the data just in case. It should be perfectly intact as it was left before.

Here we go. henkraid-diagnostics-20190518-1854.zip