Disk errors : "Buffer I/O error ... lost async page write"

[aleary]

My system has just generated a number of disk errors, and I'm wondering what the cause would be and what I should do to fix it.

 

System:

• Unraid 6.3.2
• 16GB ECC RAM
• Array : 3 x  4TB array drives, (1 parity and 2 data drives (xfs))
• Cache Pool: 2 x 256GB SSD drives (btrfs).

 

I was logged into Unraid via SSH converting a VMWare vmdk disk image to use with Unraid using "qemu-img", writing about 30GB to a share that does not have caching enabled, so writing directly to the array when this error was returned:

 

qemu-img: error while writing sector 9302014: Input/output error

 

Checking the syslog I found the following:

 

Feb 28 15:49:49 unraid kernel: XFS (md1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3504 of file fs/xfs/libxfs/xfs_btree.c.  Caller xfs_alloc_fixup_trees+0x184/0x2c2
...

Feb 28 15:49:49 unraid kernel: XFS (md1): Internal error xfs_trans_cancel at line 983 of file fs/xfs/xfs_trans.c.  Caller xfs_iomap_write_allocate+0x29c/0x2ca

...
Feb 28 15:49:49 unraid kernel: XFS (md1): xfs_do_force_shutdown(0x8) called from line 984 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff812b1cef
Feb 28 15:49:50 unraid kernel: XFS (md1): Corruption of in-memory data detected.  Shutting down filesystem
Feb 28 15:49:50 unraid kernel: XFS (md1): Please umount the filesystem and rectify the problem(s)
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748705, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748706, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748707, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748708, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748709, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748710, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748711, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748712, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748713, lost async page write
Feb 28 15:49:50 unraid kernel: Buffer I/O error on dev md1, logical block 547748714, lost async page write
Feb 28 15:49:50 unraid shfs/user: err: shfs_write: write: (5) Input/output error
 

The Unraid UI doesn't show any issues other than the syslog, and there don't appear to be any SMART disk errors.

However, any access on the md1 drive from the CLI fail with "Input/output error".

 

Does the "Corruption of in-memory data detected" indicate a potential RAM issue, rather than a disk issue?

Perhaps I should run memtest on the system to check that.

 

Diagnostics are attached.

 

Any suggestions would be gratefully received.

 

Kind regards,

Alan Leary.

 

unraid-diagnostics-20170228-1609.zip

Edited February 28, 2017 by aleary

[JorgeB]

Check filesystem on disk1

https://lime-technology.com/wiki/index.php/Check_Disk_Filesystems#Redoing_a_drive_formatted_with_XFS

[aleary]

Thanks for the pointer, johnnie.black.

 

I ran xfs_repair, as guided, and following some repairs made, all appears to be fine.

 

I'm still not sure why this occurred to start with, but will be keeping a close eye on things.

 

Thanks again,

Alan.