Disk16 is missing when viewing /media it also has tons of system files I wouldnt expect to see


Go to solution Solved by JorgeB,

Recommended Posts

So I may have screwed up... while it would appear my files are all there now my Disk16 does not look how I would expect.

SabNZDB is unable to create new folders and I believe this is why.

 

Normally I would see some of the shares and files as they're added to the array but Disk16 looks like this!

 

I did run a parity check and it corrected over a billion sync errors, I assumed that was correcting an issue that occurred when transferring several thousand files and getting an IO error in Windows.

Any ideas why this disk looks like this?

 

Is there anyway to fix it?


This is what I did.

 

disk.PNG

Edited by techlovin
Link to comment

There is no SMART information for disk16 which suggests it dropped offline, and also I/O errors in the syslog reporting possible xfs level corruption.

 

I suggest you power cycle the server, and then as long as disk16 shows up run a file system check on it.   Might also want to consider an extended SMART test although that would take around a day on a disk of that size.

 

if anything looks slightly wrong post new diagnostics after the check.

Link to comment
1 hour ago, JorgeB said:

Check filesystem on disk16.

Phase 1 - find and verify superblock...
        - block cache size set to 1427000 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 456065 tail block 456065
        - scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x3ffffffc8/0x1000
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x5ffffffa8/0x1000
btree block 12/1 is suspect, error -74
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x47fffffc0/0x1000
btree block 9/1 is suspect, error -74
bad magic # 0 in btbno block 9/1
bad magic # 0 in btbno block 12/1
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x57fffffb0/0x1000
btree block 11/1 is suspect, error -74
bad magic # 0 in btbno block 11/1
btree block 8/1 is suspect, error -74
bad magic # 0 in btbno block 8/1
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x37fffffd0/0x1000
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x47fffffc8/0x1000
btree block 9/2 is suspect, error -74
bad magic # 0 in btcnt block 9/2
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x5ffffffb0/0x1000
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x57fffffb8/0x1000btree block 7/1 is suspect, error -74
btree block 12/2 is suspect, error -74
bad magic # 0 in btbno block 7/1
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x3ffffffd0/0x1000
btree block 8/2 is suspect, error -74
bad magic # 0 in btcnt block 8/2
bad magic # 0 in btcnt block 12/2

btree block 11/2 is suspect, error -74
agf_freeblks 1871567, counted 0 in ag 9
agf_longest 1755685, counted 0 in ag 9
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x37fffffd8/0x1000
btree block 7/2 is suspect, error -74
bad magic # 0 in btcnt block 7/2
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x4ffffffb8/0x1000agf_freeblks 103963324, counted 0 in ag 8
agf_longest 101598315, counted 0 in ag 8

btree block 10/1 is suspect, error -74
agf_freeblks 1372606, counted 0 in ag 12
Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x47fffffd0/0x1000bad magic # 0 in btcnt block 11/2
agf_longest 755030, counted 0 in ag 12

bad magic # 0 in btbno block 10/1
agf_freeblks 115350166, counted 0 in ag 7
btree block 9/3 is suspect, error -74
Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x3ffffffd8/0x1000bad magic # 0 in inobt block 9/3
agf_longest 108329326, counted 0 in ag 7
agf_freeblks 1871568, counted 0 in ag 11

Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x5ffffffb8/0x1000agf_longest 1190778, counted 0 in ag 11
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x4ffffffc0/0x1000
btree block 8/3 is suspect, error -74
btree block 12/3 is suspect, error -74

bad magic # 0 in inobt block 12/3
bad magic # 0 in inobt block 8/3
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x47fffffd8/0x1000
Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x37fffffe0/0x1000Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x57fffffc0/0x1000btree block 10/2 is suspect, error -74
bad magic # 0 in btcnt block 10/2

btree block 7/3 is suspect, error -74
btree block 9/4 is suspect, error -74
bad magic # 0 in inobt block 7/3
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x3ffffffe0/0x1000
btree block 8/4 is suspect, error -74
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x5ffffffc0/0x1000
btree block 12/4 is suspect, error -74
bad magic # 0 in finobt block 9/4
bad magic # 0 in finobt block 8/4
agi_count 4928, counted 0 in ag 9
agi_count 13056, counted 0 in ag 8
agf_freeblks 1871567, counted 0 in ag 10
agi_freecount 2, counted 0 in ag 8
agf_longest 1609848, counted 0 in ag 10
bad magic # 0 in finobt block 12/4

agi_count 2304, counted 0 in ag 12
btree block 11/3 is suspect, error -74
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x37fffffe8/0x1000
btree block 7/4 is suspect, error -74
bad magic # 0 in inobt block 11/3
bad magic # 0 in finobt block 7/4
agi_count 11648, counted 0 in ag 7
agi_freecount 2, counted 0 in ag 8 finobt
Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x4ffffffc8/0x1000
agi_freecount 6, counted 0 in ag 7
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x57fffffc8/0x1000btree block 10/3 is suspect, error -74
bad magic # 0 in inobt block 10/3
agi_freecount 6, counted 0 in ag 7 finobt

btree block 11/4 is suspect, error -74
bad magic # 0 in finobt block 11/4
agi_count 3008, counted 0 in ag 11
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x4ffffffd0/0x1000
btree block 10/4 is suspect, error -74
bad magic # 0 in finobt block 10/4
agi_count 1600, counted 0 in ag 10
sb_icount 181504, counted 144960
sb_ifree 130, counted 122
sb_fdblocks 570449352, counted 344148554
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
Metadata corruption detected at 0x4379a3, xfs_inode block 0x330604e50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x330604e70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x330604e90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x330604eb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334a12210/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334a12230/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334ab32d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334ab32f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334a61190/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334a611b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b05f50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b05f70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b05f90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b05fb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b05fd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b05ff0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b06010/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b06030/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3349e49d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3349e49f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3349e4a10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3349e4a30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bc3c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bc3cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b74690/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b746b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c03c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c03cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b12a90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b12ab0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b12ad0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b12af0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c050d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c050f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b290d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334b290f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf07d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf07f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf0810/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf0830/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf0850/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf0870/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf0890/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf08b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf08d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334bf08f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cc3d50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cc3d70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c47110/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c47130/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c41090/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c410b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c65210/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334c65230/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfac10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfac30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfac50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfac70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfac90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfacb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfacd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334cfacf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334d21590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334d215b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334d56490/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334d564b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e03590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e035b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334d05c10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334d05c30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e362d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e362f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e27150/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e27170/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e61a90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e61ab0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334edd050/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334edd070/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334db1790/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334db17b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e75790/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e757b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f23ad0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f23af0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f45e90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f45eb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b250/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b270/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b290/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b2b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b2d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b2f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b310/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b330/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b350/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334e0b370/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f19c50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f19c70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f19c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f19cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f19cd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f19cf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3350144d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3350144f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f58090/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334f580b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335040c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335040cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334fb4550/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x334fb4570/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3350b1c50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3350b1c70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335107490/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351074b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351074d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351074f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335107510/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335107530/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335107550/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335107570/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335107590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351075b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034a50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034a70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034a90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034ab0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034ad0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034af0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034b10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034b30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034b50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335034b70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33512e490/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33512e4b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33519eb10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33519eb30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351e6050/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351e6070/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351e6090/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351e60b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351e60d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351e60f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33511e2d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33511e2f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3350539d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3350539f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335211c10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335211c30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351f3f10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3351f3f30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352f3d50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352f3d70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352f3d90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352f3db0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33531a390/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33531a3b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335278ed0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335278ef0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335360c10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335360c30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335360c50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335360c70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335336c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335336cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335322450/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335322470/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33540afd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33540aff0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33538cd10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33538cd30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33538cd50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33538cd70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335419010/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335419030/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335456950/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335456970/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335456990/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354569b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3353cf710/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3353cf730/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3353cf750/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3353cf770/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335495350/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335495370/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354301d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354301f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354dc890/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354dc8b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354dc8d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354dc8f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33548ea90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33548eab0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354bd750/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3354bd770/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335526ad0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335526af0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33554e7d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33554e7f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335507e50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335507e70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335563850/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335563870/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33556a210/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33556a230/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335528e10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335528e30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335593310/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335593330/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3355c4cd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3355c4cf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc3d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc3f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc410/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc430/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc450/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc470/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc490/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc4b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc4d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc4f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc510/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3352cc530/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335619cd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335619cf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3355ccf10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3355ccf30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356a8e50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356a8e70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356e2d10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356e2d30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335775590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3357755b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356d2810/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356d2830/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356d2850/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356d2870/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356d2890/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356d28b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33578b990/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33578b9b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3357b7c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3357b7cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335780cd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335780cf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335780d10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335780d30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335780d50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335780d70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356ea050/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3356ea070/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33581aa10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335807510/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335807530/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335807550/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335807570/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3355f5510/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3355f5530/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x335810690/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3358106b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33988ac50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33988ac70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33988acb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33988aed0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33988aef0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33988af30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33978ae90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33978aeb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33978aef0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x339d287f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33998ae90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33998aeb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x339c28770/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x339c28850/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x339c28870/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c84ecd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c84ecf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c84ed10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c8e4510/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c8e4530/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c8e4550/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c753bd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c753bf0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c753c30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c9d7750/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c9d7770/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c9d7790/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33c7c4890/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cb4b090/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cb4b0b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cb4b0f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ca8f1d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ca8f1f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ca8f210/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cba5a10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cba5a30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cba5a70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ccf1ad0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ccf1af0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ccf1b30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cf25dd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cf25df0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cf25e30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cc31650/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cc31670/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cc316b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cdb4210/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cdb4230/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33cdb4270/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ce85590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ce855b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33ce855f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d77b9d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d77b9f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d77ba30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d00d5d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d00d5f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d00d610/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d0b45d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d0b45f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d0b4650/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd284f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd28510/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd28590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd285b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd285f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd28670/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dd286d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e842dd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e842df0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e842e10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e842e30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e842e50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78ac70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78acd0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78ad50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78ad70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78adb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78ae10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78ae90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78af10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78af30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33d78af70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bb010/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bb030/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bb1d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bbc70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bbd50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bbd70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bc650/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5bc670/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e9349d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33e9349f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5c32b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5c3710/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5c3730/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5c4d10/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5c4d30/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5c4d50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dc28610/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dc28690/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dc286b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33dc286f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5cb650/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5d13b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x33f5d8870/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340641a50/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340641a70/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340641a90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340716150/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340716170/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3407161b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3407c4110/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3407c4590/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3407c45b0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3407c45f0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x3405d88d0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340695c90/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340695cb0/0x4000
Metadata corruption detected at 0x4379a3, xfs_inode block 0x340695cf0/0x4000
bad CRC for inode 13696519808
bad magic number 0x0 on inode 13696519808
bad version number 0x0 on inode 13696519808
bad next_unlinked 0x0 on inode 13696519808
inode identifier 0 mismatch on inode 13696519808
bad CRC for inode 13696519809
bad magic number 0x0 on inode 13696519809
bad version number 0x0 on inode 13696519809
bad next_unlinked 0x0 on inode 13696519809
inode identifier 0 mismatch on inode 13696519809
bad CRC for inode 13696519810
bad magic number 0x0 on inode 13696519810
bad version number 0x0 on inode 13696519810
bad next_unlinked 0x0 on inode 13696519810
inode identifier 0 mismatch on inode 13696519810
bad CRC for inode 13696519811
bad magic number 0x0 on inode 13696519811
bad version number 0x0 on inode 13696519811
bad next_unlinked 0x0 on inode 13696519811
inode identifier 0 mismatch on inode 13696519811
bad CRC for inode 13696519812
bad magic number 0x0 on inode 13696519812
bad version number 0x0 on inode 13696519812
bad next_unlinked 0x0 on inode 13696519812
inode identifier 0 mismatch on inode 13696519812
bad CRC for inode 13696519813
bad magic number 0x0 on inode 13696519813
bad version number 0x0 on inode 13696519813
bad next_unlinked 0x0 on inode 13696519813
inode identifier 0 mismatch on inode 13696519813
bad CRC for inode 13696519814
bad magic number 0x0 on inode 13696519814
bad version number 0x0 on inode 13696519814
bad next_unlinked 0x0 on inode 13696519814
inode identifier 0 mismatch on inode 13696519814
bad CRC for inode 13696519815
bad magic number 0x0 on inode 13696519815

 

 

This goes on for days.... so definitely a bad filesystem check.
Not seeing a command to repair, if that's even possible.

 

 

Edit:

Here are the results at the end.
 

No modify flag set, skipping phase 5
Inode allocation btrees are too corrupted, skipping phases 6 and 7
No modify flag set, skipping filesystem flush and exiting.

        XFS_REPAIR Summary    Thu Apr 13 11:30:25 2023

Phase		Start		End		Duration
Phase 1:	04/13 11:30:16	04/13 11:30:16
Phase 2:	04/13 11:30:16	04/13 11:30:16
Phase 3:	04/13 11:30:16	04/13 11:30:25	9 seconds
Phase 4:	04/13 11:30:25	04/13 11:30:25
Phase 5:	Skipped
Phase 6:	Skipped
Phase 7:	Skipped

Total run time: 9 seconds

 

Edited by techlovin
Link to comment

So I have NO idea how I let this happen but I think I know what went wrong.

When I was resolving my issue in the previous thread, I started a new config, confirmed parity was fine, but then ran a parity check afterwards.

It turns out I must have got the drives confused because checking the power-on hours of this drive it's clear this is a brand new drive, well 5 days of powered on time now.

 

I've got no idea how I let that happen.

The old drive is currently at 30% being Zeroed....

Edit, to add it's strange because when I was confirming the drive I mounted it, and took a look through it, it appeared fine... I'm just very confused at the moment.

 

I'll do whatever I have to, to just get back to a safe working spot with my array.

Edited by techlovin
Link to comment

WELL RIP.

Don't know what happened because I have not done ANYTHING else to the array and Disk16 has disappeared.

Previously it was visible and I could see the contents of it emulated perfectly fine, now when I looked again it's displaying exactly like the first screenshot in this thread. :(

 

In Krusader Disk16 is no longer a folder but an unknown file under /mnt/

Link to comment

Now all my drives are unmountable!!!!!

 

Edit:
Rebooted the server and my drives are back.... wtf is going on!!?

 

Edit2:

 

My apologies for my panicked tone, I'm terrified to lose 200TBs of data... I'm frantically backing up anything I cant replace but I really really need some help figuring out what the hell is going on!

Edited by techlovin
Link to comment
2 hours ago, Squid said:

 

Best guess is that you've disturbed the cabling and now its hit and miss on whether it makes a proper connection and keeps it.

 

If anything happens again, post your diagnostics before you reboot.

 

The cabling has not been disturbed.

I can reproduce this issue repeatedly. 

 

I reboot the server without ever touching it, drives, or any cables.

 

The server is fine and disk16 is emulated exactly as I expect it should be. After 3 hours approximately disk16 is still emulated but the contents of it have disappeared.

 

When checking via the GUI or Krusader and going to /mnt Disk16 is not readable.

 

I can simply reboot and it will continue working again for 3 hrs.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.