(Solved) Issues again - Device is disabled, contents emulated - advice on next step

[the1poet]

Hi,

So I was having issues with unRaid and was at work, so I figured a reboot and remote back in to check what was up.

Long story short, I've come home and remoted into my server via web GUI and getting a red X next to one of my drives, and when it's highlighted I see the message "Device is disabled, contents emulated" error.

I've also noticed files are missing, which is odd as I thought the parity disk was supposed to keep the files there until I had time to replace a disk should one drop from the array? Am I missing something?

I'm currently running a parity check (which is running super slow). Normally it would take less than 24 hours, but the ETA on this check is 3 days.

Can someone offer some advice on the next step on where to go from here?

This is the first I'm I've had this type of issue, so I want to make sure I go about it the right way to avoid losing data and wasting time (as I have to work half a day tomorrow... being Saturday, and I'm away for part of Easter next week.).

Thank in advance.

homenas-diagnostics-20180323-2220.zip

Edited April 7, 2018 by the1poet
Marking solved

[JorgeB]

5 minutes ago, the1poet said:

I'm currently running a parity check (which is running super slow).

You can cancel that for now, please post your diagnostics: Tools -> Diagnostics, also and if you got them post the ones from before rebooting also.

[the1poet]

23 minutes ago, johnnie.black said:

You can cancel that for now, please post your diagnostics: Tools -> Diagnostics, also and if you got them post the ones from before rebooting also.

Thanks for the speedy response! I'll do that and get back ASAP

 

edit: please see my original post. I've attached the diag zip to that.

Edited March 23, 2018 by the1poet
Update

[JorgeB]

You were having issues with two disks, ATA9 and ATA10:

 

Quote

Mar 23 20:48:51 HomeNAS kernel: ata10: softreset failed (1st FIS failed)
Mar 23 20:48:51 HomeNAS kernel: ata10: hard resetting link
Mar 23 20:48:51 HomeNAS kernel: ata9: softreset failed (1st FIS failed)
Mar 23 20:48:51 HomeNAS kernel: ata9: hard resetting link
Mar 23 20:49:01 HomeNAS kernel: ata10: softreset failed (1st FIS failed)
Mar 23 20:49:01 HomeNAS kernel: ata10: hard resetting link
Mar 23 20:49:01 HomeNAS kernel: ata9: softreset failed (1st FIS failed)
Mar 23 20:49:01 HomeNAS kernel: ata9: hard resetting link
Mar 23 20:49:36 HomeNAS kernel: ata10: softreset failed (1st FIS failed)
Mar 23 20:49:36 HomeNAS kernel: ata10: limiting SATA link speed to 3.0 Gbps
Mar 23 20:49:36 HomeNAS kernel: ata10: hard resetting link
Mar 23 20:49:36 HomeNAS kernel: ata9: softreset failed (1st FIS failed)
Mar 23 20:49:36 HomeNAS kernel: ata9: limiting SATA link speed to 3.0 Gbps
Mar 23 20:49:36 HomeNAS kernel: ata9: hard resetting link
Mar 23 20:49:41 HomeNAS kernel: ata9: softreset failed (1st FIS failed)
Mar 23 20:49:41 HomeNAS kernel: ata9: reset failed, giving up
Mar 23 20:49:41 HomeNAS kernel: ata9.00: disabled
Mar 23 20:49:41 HomeNAS kernel: ata9: EH complete

 

ATA9 (disk7) ended up being disable, ATA10 (disk8) is working for now, since both share the same controller there could be a problem there, but start by checking/replacing cables on both disks, especially if they share a molex splitter or power connector, then power back on and grab new diags just to check SMART for disk7, since it's not on the previous diags on account of it being dropped offline.

[the1poet]

On 3/23/2018 at 10:40 PM, johnnie.black said:

You were having issues with two disks, ATA9 and ATA10:

 

 

ATA9 (disk7) ended up being disable, ATA10 (disk8) is working for now, since both share the same controller there could be a problem there, but start by checking/replacing cables on both disks, especially if they share a molex splitter or power connector, then power back on and grab new diags just to check SMART for disk7, since it's not on the previous diags on account of it being dropped offline.

Finally have some time today, so I'll do some testing and get back with my results. Thanks again for your feedback.

 

edit: so I pulled the two drives and connected them via an external dock to my Win 10 machine with Hard Disk Sentinel running and both SMART data is coming up fine. Based on this, your suggestion seems like the most likely cause. I'm going to try that now and see how I go.

Edited March 25, 2018 by the1poet
Update

[the1poet]

On 3/23/2018 at 10:40 PM, johnnie.black said:

You were having issues with two disks, ATA9 and ATA10:

 

 

ATA9 (disk7) ended up being disable, ATA10 (disk8) is working for now, since both share the same controller there could be a problem there, but start by checking/replacing cables on both disks, especially if they share a molex splitter or power connector, then power back on and grab new diags just to check SMART for disk7, since it's not on the previous diags on account of it being dropped offline.

So I've replaced the 2 cables and booted unRaid. The disk that was missing before has come back up, but there's still a red X next to it. Do I just added it back to the array? What do I need to do to achieve that since it's currently in a disabled state?

[the1poet]

I followed the instructions here:

 and the array is currently rebuilding the drive.

Think everything should be good now.

[the1poet]

So, my unRaid just started doing a parity check which it's scheduled to do at the start of each month.

I just tried to copy some data over to the array at the same time and was wondering why it stopped responding.

Had a quick look at the current log and noticed this:

Apr 1 13:40:41 HomeNAS kernel: print_req_error: I/O error, dev sdj, sector 6446415432
Apr 1 13:40:41 HomeNAS kernel: md: disk8 read error, sector=6446415368
Apr 1 13:40:41 HomeNAS kernel: md: disk8 read error, sector=6446415376
Apr 1 13:40:41 HomeNAS kernel: md: disk8 read error, sector=6446415384
Apr 1 13:40:41 HomeNAS kernel: md: disk8 read error, sector=6446415392
Apr 1 13:40:41 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x1803c7e08 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:41 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x200bce560 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x200bce560 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x3800008a8 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x900 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x200bce560 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x3800008c8 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x1015eb130 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): metadata I/O error: block 0x1803c7e08 ("xfs_trans_read_buf_map") error 5 numblks 32
Apr 1 13:40:50 HomeNAS kernel: XFS (md8): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.

I had replaced the two SATA cables on those drives, now the same two seem to be misbheaving again.

homenas-diagnostics-20180401-1541.zip

Edited April 1, 2018 by the1poet
Added latest diagnostics

[JorgeB]

Problems again on ATA9 and 10, they share the same Asmedia controller, if you already replaced cables (including power cables) it could be a controller problem.

[the1poet]

On 4/1/2018 at 6:13 PM, johnnie.black said:

Problems again on ATA9 and 10, they share the same Asmedia controller, if you already replaced cables (including power cables) it could be a controller problem.

I didn't replace the power the first time because I wanted to know if that fixed it. I've replaced the PCB board that connects to the drives (it's a Silverstone CS380 case) and power connectors this time. If that fails, I think I'll look at a dedicated SATA controller card.

[the1poet]

So I've made it further than the 4 days when it failed last time. Silly me only had 1 power connector for the 8 drives when it would seem it needed 2.

Since then I haven't had any issues.

My assumption for why it has failed only recently was that I didn't have enough data on the array for more than whatever the threshold for power was to be over utilised (meaning not enough drives were accessed at the same time for the power to be insufficient until now).

Anyway, no issues since, I shall reluctantly mark this solved (again). Thanks again @johnnie.black for the help. Very much appreciated!

Edited April 7, 2018 by the1poet
Grammar