Files Disappear Until Reboot. Kernel Errors

[matthewmalek]

unraid: 6.3.5 

 

For quite a while now my system has been not listing all my files and directories after having been on for a short while. When I reboot the system everything is available, but after a while certain user shares start having many of their files go missing. This is true even when I ssh into the system and list the files directly on its filesystem. I believe the issue is between the discs and the kernel, as is made clear by the logs I've attached. But I don't know enough about what's going on to make a real diagnosis. I would like to make sure my data is safe in case one of these times I turn on my system and my files never show up again.

 

I included a log that shows just the errors. There is a constant "Emask" kernel exception throughout, but I believe the bottom shows the real error as I try to access an affected user share.

 

If anyone can suggest a fix for this issue, I would greatly appreciate the help!

syslog_ERRORS.txt

[CHBMB]

I'd post full diagnostics if I were you....

[matthewmalek]

Oh wow, didn't realize there was such a thing. Very handy. Attached.

multivac-diagnostics-20180204-1217.zip

 

I've also noticed that the issues appear to by with my two latest discs 6 and 7, both of which are xfs whereas the rest are reiserfs. I have tested and discovered they do not spin up anymore. I have not restarted yet, but if history holds, they will spin up on reboot and be accessible until they spin down, for good.

 

Failed attempt to spin up up disc 6:

Feb 4 12:41:23 Multivac kernel: mdcmd (52): spinup 6
Feb 4 12:41:23 Multivac kernel: md: do_drive_cmd: lock_bdev error: -2

[JorgeB]

You're having ATA errors on these 2 disks for hours:

 

Jan 16 19:43:31 Multivac kernel: ata11: SError: { 10B8B }
Jan 16 19:43:31 Multivac kernel: ata11: hard resetting link
Jan 16 19:43:36 Multivac kernel: ata11: controller in dubious state, performing PORT_RST
Jan 16 19:43:41 Multivac kernel: ata11: softreset failed (timeout)
Jan 16 19:43:41 Multivac kernel: ata11: hard resetting link
Jan 16 19:43:44 Multivac kernel: ata10: limiting SATA link speed to 1.5 Gbps
Jan 16 19:43:44 Multivac kernel: ata10: hard resetting link
Jan 16 19:43:49 Multivac kernel: ata10: controller in dubious state, performing PORT_RST
Jan 16 19:43:49 Multivac kernel: ata10: hardreset failed (PHY debouncing failed)
Jan 16 19:43:49 Multivac kernel: ata10: reset failed, giving up
Jan 16 19:43:49 Multivac kernel: ata10: EH complete
Jan 16 19:43:50 Multivac kernel: ata10: exception Emask 0x10 SAct 0x0 SErr 0x80000 action 0xe frozen
Jan 16 19:43:50 Multivac kernel: ata10: irq_stat 0x00100010, PHY RDY changed
Jan 16 19:43:50 Multivac kernel: ata10: SError: { 10B8B }
Jan 16 19:43:50 Multivac kernel: ata10: hard resetting link
Jan 16 19:43:51 Multivac kernel: ata11: softreset failed (timeout)
Jan 16 19:43:51 Multivac kernel: ata11: hard resetting link

Those errors end up with the disks being redetected with new identifiers resulting in this:

 

Jan 20 16:07:50 Multivac kernel: md: disk6 read error, sector=385863320
Jan 20 16:07:50 Multivac kernel: md: disk6 read error, sector=2389788928
Jan 20 16:07:50 Multivac kernel: md: disk6 read error, sector=2389788936
Jan 20 16:07:50 Multivac kernel: md: disk6 read error, sector=2389788944
Jan 20 16:07:50 Multivac kernel: md: disk6 read error, sector=2389788952
Jan 20 16:07:50 Multivac kernel: md: disk7 read error, sector=385863320
Jan 20 16:07:50 Multivac kernel: XFS (md6): metadata I/O error: block 0x16ffce98 ("xfs_trans_read_buf_map") error 5 numblks 8
Jan 20 16:07:50 Multivac kernel: XFS (md6): metadata I/O error: block 0x16ffce98 ("xfs_trans_read_buf_map") error 5 numblks 8
Jan 20 16:07:50 Multivac kernel: md: disk7 read error, sector=2389788928
Jan 20 16:07:50 Multivac kernel: md: disk7 read error, sector=2389788936
Jan 20 16:07:50 Multivac kernel: md: disk7 read error, sector=2389788944
Jan 20 16:07:50 Multivac kernel: md: disk7 read error, sector=2389788952
Jan 20 16:07:50 Multivac kernel: XFS (md6): metadata I/O error: block 0x8e714900 ("xfs_trans_read_buf_map") error 5 numblks 32
Jan 20 16:07:50 Multivac kernel: XFS (md6): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Jan 20 16:07:50 Multivac shfs/user: err: shfs_readdir: fstatat: Wii (5) Input/output error
Jan 20 16:07:50 Multivac shfs/user: err: shfs_readdir: fstatat: Wii (5) Input/output error
Jan 20 16:07:50 Multivac kernel: XFS (md6): metadata I/O error: block 0x8e714900 ("xfs_trans_read_buf_map") error 5 numblks 32
Jan 20 16:07:50 Multivac kernel: XFS (md6): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Jan 20 16:07:50 Multivac kernel: XFS (md6): metadata I/O error: block 0x16ffce98 ("xfs_trans_read_buf_map") error 5 numblks 8

Both disks are on the same Silicon Image controller, try another or different ports if available.

 

There are also a lot of errors on disk4 on a different SIL controller, these possibly caused by a bad cable:

 

Jan 19 15:09:18 Multivac kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x80000 action 0xe frozen
Jan 19 15:09:18 Multivac kernel: ata2: irq_stat 0x00100010, PHY RDY changed
Jan 19 15:09:18 Multivac kernel: ata2: SError: { 10B8B }
Jan 19 15:09:18 Multivac kernel: ata2: hard resetting link
Jan 19 15:09:25 Multivac kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10)
Jan 19 15:09:25 Multivac kernel: ata2.00: configured for UDMA/33
Jan 19 15:09:25 Multivac kernel: ata2: EH complete
Jan 19 15:09:27 Multivac kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x80000 action 0xe frozen
Jan 19 15:09:27 Multivac kernel: ata2: irq_stat 0x00100010, PHY RDY changed
Jan 19 15:09:27 Multivac kernel: ata2: SError: { 10B8B }
Jan 19 15:09:27 Multivac kernel: ata2: hard resetting link
Jan 19 15:09:33 Multivac kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10)
Jan 19 15:09:33 Multivac kernel: ata2.00: configured for UDMA/33
Jan 19 15:09:33 Multivac kernel: ata2: EH complete

Also, change your onboard controller from IDE to SATA/AHCI if that's an option.

[matthewmalek]

Thanks Johnnie. I haven't had time to look into it until now, but changing disks 6 and 7 to different ports fixed the issue. I will also look into changing the controller type.