Disk1 will not rebuild due to xfs filesystem error- How can I recover?

glave

By glave
in General Support

Recommended Posts

glave Apprentice

glave

Posted
Posted

During the monthly parity check, disk1 dropped out due to what appeared to be an error with hardware, but not drive failure. I cold booted the server, readded drive1 to the array, and started a rebuild. Not very long after the rebuild began, I noticed that all of my user shares had disappeared. Looking into the log, I see lots of these errors as listed below.

 

I have ran an extended smart check on that drive, and it passes with flying colors. How can I rebuild on this drive? If I cannot, I currently cannot even remove it from the array and start as missing, as the array says too many disks missing and will not even start unprotected.

 

ug  6 16:54:41 arcade kernel: XFS (md1): Metadata corruption detected at xfs_da3_node_read_verify+0xeb/0xf3, xfs_da3_node block 0x8404ec80
Aug  6 16:54:41 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:54:41 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:54:41 arcade kernel: ffff8800927c1000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:54:41 arcade kernel: ffff8800927c1010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:54:41 arcade kernel: ffff8800927c1020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:54:41 arcade kernel: ffff8800927c1030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:54:41 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug  6 16:54:41 arcade shfs/user: shfs_readdir: fstatat: 890 (117) Structure needs cleaning
Aug  6 16:54:41 arcade shfs/user: shfs_readdir: readdir_r: /mnt/disk1/downloads/queue (117) Structure needs cleaning
Aug  6 16:54:41 arcade kernel: XFS (md1): Metadata corruption detected at xfs_da3_node_read_verify+0xeb/0xf3, xfs_da3_node block 0x8404ec80
Aug  6 16:54:41 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:54:41 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:54:41 arcade kernel: ffff8801e7f35000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:54:41 arcade kernel: ffff8801e7f35010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:54:41 arcade kernel: ffff8801e7f35020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:54:41 arcade kernel: ffff8801e7f35030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:54:41 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug  6 16:54:41 arcade shfs/user: shfs_readdir: fstatat: 890 (117) Structure needs cleaning
Aug  6 16:54:41 arcade shfs/user: shfs_readdir: readdir_r: /mnt/disk1/downloads/queue (117) Structure needs cleaning
Aug  6 16:54:50 arcade kernel: XFS (md1): Metadata corruption detected at xfs_da3_node_read_verify+0xeb/0xf3, xfs_da3_node block 0x8404ec80
Aug  6 16:54:50 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:54:50 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:54:50 arcade kernel: ffff88009088a000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:54:50 arcade kernel: ffff88009088a010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:54:50 arcade kernel: ffff88009088a020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:54:50 arcade kernel: ffff88009088a030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:54:50 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug  6 16:54:50 arcade kernel: XFS (md1): Metadata corruption detected at xfs_da3_node_read_verify+0xeb/0xf3, xfs_da3_node block 0x8404ec80
Aug  6 16:54:50 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:54:50 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:54:50 arcade kernel: ffff8801e61ee000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:54:50 arcade kernel: ffff8801e61ee010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:54:50 arcade kernel: ffff8801e61ee020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:54:50 arcade kernel: ffff8801e61ee030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:54:50 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug  6 16:54:50 arcade kernel: XFS (md1): Metadata corruption detected at xfs_da3_node_read_verify+0xeb/0xf3, xfs_da3_node block 0x8404ec80
Aug  6 16:54:50 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:54:50 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:54:50 arcade kernel: ffff8801e6201000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:54:50 arcade kernel: ffff8801e6201010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:54:50 arcade kernel: ffff8801e6201020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:54:50 arcade kernel: ffff8801e6201030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:54:50 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug  6 16:54:50 arcade kernel: XFS (md1): Metadata corruption detected at xfs_da3_node_read_verify+0xeb/0xf3, xfs_da3_node block 0x8404ec80
Aug  6 16:54:50 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:54:50 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:54:50 arcade kernel: ffff880091f23000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:54:50 arcade kernel: ffff880091f23010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:54:50 arcade kernel: ffff880091f23020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:54:50 arcade kernel: ffff880091f23030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:54:50 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug  6 16:55:09 arcade kernel: XFS (md1): Metadata CRC error detected at __read_verify+0xaa/0xb4, xfs_dir3_leafn block 0x8404ec80
Aug  6 16:55:09 arcade kernel: XFS (md1): Unmount and run xfs_repair
Aug  6 16:55:09 arcade kernel: XFS (md1): First 64 bytes of corrupted metadata buffer:
Aug  6 16:55:09 arcade kernel: ffff8801e5ed4000: 76 da fd f3 a8 44 d3 b9 f8 00 e5 8f 49 35 5b b3  v....D......I5[.
Aug  6 16:55:09 arcade kernel: ffff8801e5ed4010: 9c 71 cf b4 2a 39 cd 38 70 cc 58 13 ce d3 38 52  .q..*9.8p.X...8R
Aug  6 16:55:09 arcade kernel: ffff8801e5ed4020: fd 4b f3 f4 30 5a ca f1 ef 14 de 30 b3 74 69 ce  .K..0Z.....0.ti.
Aug  6 16:55:09 arcade kernel: ffff8801e5ed4030: 6c 8b b0 1e 2d 0a 12 cf dc f1 88 51 ed 52 82 38  l...-......Q.R.8
Aug  6 16:55:09 arcade kernel: XFS (md1): metadata I/O error: block 0x8404ec80 ("xfs_trans_read_buf_map") error 74 numblks 8
Aug  6 16:55:09 arcade kernel: XFS (md1): xfs_do_force_shutdown(0x1) called from line 315 of file fs/xfs/xfs_trans_buf.c.  Return address = 0xffffffff81294958
Aug  6 16:55:09 arcade shfs/user: shfs_unlink: unlink: /mnt/disk1/downloads/queue/3571 (117) Structure needs cleaning
Aug  6 16:55:09 arcade shfs/user: shfs_unlink: lookup: downloads/queue/3571s (5) Input/output error
Aug  6 16:55:09 arcade shfs/user: shfs_open: lookup: downloads/queue/n199.log (5) Input/output error
Aug  6 16:55:09 arcade kernel: XFS (md1): I/O Error Detected. Shutting down filesystem
Aug  6 16:55:09 arcade kernel: XFS (md1): Please umount the filesystem and rectify the problem(s)
Aug  6 16:55:09 arcade shfs/user: shfs_open: lookup: downloads/queue/n203.log (5) Input/output error
Aug  6 16:55:09 arcade shfs/user: shfs_open: lookup: downloads/queue/n199.log (5) Input/output error
Aug  6 16:55:09 arcade shfs/user: shfs_open: lookup: downloads/queue/n199.log (5) Input/output error
Aug  6 16:55:09 arcade shfs/user: shfs_open: lookup: downloads/queue/n199.log (5) Input/output error
Aug  6 16:55:09 arcade shfs/user: shfs_open: lookup: downloads/queue/n199.log (5) Input/output error

 

Link to comment
John_M Experienced

John_M

Posted
Posted

The clue is in the error message:

 

Aug  6 16:54:41 arcade kernel: XFS (md1): Unmount and run xfs_repair

 

Start the array in maintenance mode (so that the file systems are not mounted) then run xfs_repair in the disk's settings page, which is accessed by clicking on the affected disk (in your case the actual word "Disk 1").

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing