Drive Removed Itself From Array

[QPlus7]

Hi, all.

 

I have an array that is missing a drive. I haven't done anything to my array as it has been working fine for a while. I was alerted to this when my wife told me she couldn't save files to her network folder.

 

Upon further investigation, I noticed the share is not showing all the data it contains. If I browse each individual drive from the console I see data that I don't see when browsing the share from a Windows/OS X machine.

 

Stopping and starting the array takes a really long time. Rebooting the server also takes a really long time.

 

I saw the following in the console and thought I'd share:

 

ErrorWarningSystemArrayLogin


Aug 20 10:18:32 VIA-NAS01 kernel: ata1: link is slow to respond, please be patient (ready=0)
Aug 20 10:18:37 VIA-NAS01 kernel: ata1: device not ready (errno=-16), forcing hardreset
Aug 20 10:18:37 VIA-NAS01 kernel: ata1: soft resetting link
Aug 20 10:18:53 VIA-NAS01 kernel: ata1.00: configured for PIO0
Aug 20 10:18:53 VIA-NAS01 kernel: ata1.01: configured for UDMA/33
Aug 20 10:18:53 VIA-NAS01 kernel: ata1: EH complete
Aug 20 10:19:23 VIA-NAS01 kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 20 10:19:23 VIA-NAS01 kernel: ata1.00: failed command: READ MULTIPLE EXT
Aug 20 10:19:23 VIA-NAS01 kernel: ata1.00: cmd 29/00:08:88:f6:03/00:00:1e:00:00/e0 tag 0 pio 4096 in
Aug 20 10:19:23 VIA-NAS01 kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Aug 20 10:19:23 VIA-NAS01 kernel: ata1.00: status: { DRDY }
Aug 20 10:19:28 VIA-NAS01 kernel: ata1: link is slow to respond, please be patient (ready=0)
Aug 20 10:19:33 VIA-NAS01 kernel: ata1: device not ready (errno=-16), forcing hardreset
Aug 20 10:19:33 VIA-NAS01 kernel: ata1: soft resetting link
Aug 20 10:19:49 VIA-NAS01 kernel: ata1.00: configured for PIO0
Aug 20 10:19:49 VIA-NAS01 kernel: ata1.01: configured for UDMA/33
Aug 20 10:19:49 VIA-NAS01 kernel: ata1: EH complete
Aug 20 10:20:19 VIA-NAS01 kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 20 10:20:19 VIA-NAS01 kernel: ata1.00: failed command: READ MULTIPLE EXT
Aug 20 10:20:19 VIA-NAS01 kernel: ata1.00: cmd 29/00:08:88:f6:03/00:00:1e:00:00/e0 tag 0 pio 4096 in
Aug 20 10:20:19 VIA-NAS01 kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Aug 20 10:20:19 VIA-NAS01 kernel: ata1.00: status: { DRDY }
Aug 20 10:20:24 VIA-NAS01 kernel: ata1: link is slow to respond, please be patient (ready=0)
Aug 20 10:20:29 VIA-NAS01 kernel: ata1: device not ready (errno=-16), forcing hardreset
Aug 20 10:20:29 VIA-NAS01 kernel: ata1: soft resetting link
Aug 20 10:20:45 VIA-NAS01 kernel: ata1.00: configured for PIO0
Aug 20 10:20:45 VIA-NAS01 kernel: ata1.01: configured for UDMA/33
Aug 20 10:20:45 VIA-NAS01 kernel: ata1: EH complete
Aug 20 10:21:16 VIA-NAS01 kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 20 10:21:16 VIA-NAS01 kernel: ata1.00: failed command: READ MULTIPLE EXT
Aug 20 10:21:16 VIA-NAS01 kernel: ata1.00: cmd 29/00:08:88:f6:03/00:00:1e:00:00/e0 tag 0 pio 4096 in
Aug 20 10:21:16 VIA-NAS01 kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Aug 20 10:21:16 VIA-NAS01 kernel: ata1.00: status: { DRDY }
Aug 20 10:21:21 VIA-NAS01 kernel: ata1: link is slow to respond, please be patient (ready=0)
Aug 20 10:21:26 VIA-NAS01 kernel: ata1: device not ready (errno=-16), forcing hardreset
Aug 20 10:21:26 VIA-NAS01 kernel: ata1: soft resetting link
Aug 20 10:21:42 VIA-NAS01 kernel: ata1.00: configured for PIO0
Aug 20 10:21:42 VIA-NAS01 kernel: ata1.01: configured for UDMA/33
Aug 20 10:21:42 VIA-NAS01 kernel: ata1: EH complete

 

I assume one of my drives has failed and therefore been removed from the array but why is my share not showing all the data? I have a parity drive which, unless I have misunderstood, should preserve all my data so that I can replace the failed disk.

 

Any help is greatly appreciated - thanks in advance!

[JorgeB]

Please post the diagnostics: Tools -> Diagnostics

[QPlus7]

43 minutes ago, johnnie.black said:

Please post the diagnostics: Tools -> Diagnostics

Thanks for the quick response. See attached zip.

via-nas01-diagnostics-20190820-1112.zip

Edited August 20, 2019 by QPlus7

[JorgeB]

Multiple issues:

Disk2 is disabled and missing, is that expected? Do you have notifications enable?

Some SATA ports are set to IDE, change to AHCI/SATA

Disk1 has a connection problem, likely a bad cable.

Disk4 also looks like there's a connection problem, though it might be related to IDE mode, it's not uncommon with those AMD chipsets, but might as well change the cable also to rule it out.

 

When all these are fixed check filesystem on disks 2, 3 and 4, then you'll need to rebuild disk2

 

[QPlus7]

8 hours ago, johnnie.black said:

Multiple issues:

Disk2 is disabled and missing, is that expected? Do you have notifications enable?

Some SATA ports are set to IDE, change to AHCI/SATA

Disk1 has a connection problem, likely a bad cable.

Disk4 also looks like there's a connection problem, though it might be related to IDE mode, it's not uncommon with those AMD chipsets, but might as well change the cable also to rule it out.

 

When all these are fixed check filesystem on disks 2, 3 and 4, then you'll need to rebuild disk2

 

Disk2 being disabled is not expected. Everything was working fine then this disk was removed from the array without me doing anything.

 

Where do you see this? I assume its a BIOS setting so I checked that. I didn't see a way to set individual SATA ports but I did see a BIOS setting named "SATA IDE Combined Mode" and this was set to ENABLED. I have no set it to DISABLED.

 

I have replaced ALL SATA cables.

 

How do I check the filesystem on disks 3 and 4? 

 

Thanks in advance!

[QPlus7]

After replacing all SATA cables, I can now see, what I believe is, all the data on the share like I used to. I say what I believe because its over 8TB of data and I cannot confirm everything is there because I simply don't remember. I can confirm that I folder that I was previously unable to see is now available again.

 

I also stopped the array and was able to add Disk 2 back to the array. 

After starting the array again, Disk 2 now says "Device is disabled, contents is emulated". I assume this means its using the parity drive instead of Disk 2? Disk 2 has 264 errors on it but I am able to browse the data from the Unraid console. I purchased an extra drive today - I'm thinking about just replacing the drive and calling it a day.

[itimpi]

1 hour ago, QPlus7 said:

also stopped the array and was able to add Disk 2 back to the array

What exactly do you mean by this?    Disk2 should have come back by itself (albeit disabled) and I am a bit worried that you have taken an action that might mean its contents will be lost.

 

you should provide new diagnostics so that we can see what the current state of the system is.

[QPlus7]

10 hours ago, itimpi said:

What exactly do you mean by this?    Disk2 should have come back by itself (albeit disabled) and I am a bit worried that you have taken an action that might mean its contents will be lost.

 

you should provide new diagnostics so that we can see what the current state of the system is.

The original Disk2 did come back, disabled, as you mentioned. It showed errors and given the situation I decided to replace the drive.

 

I replaced it with a brand new drive last night. I started to do a preclear on the new drive last night and all was good the last time I checked on it at ~75%. This morning I came back to check on it and now the drive is gone. It doesn't show up under unassigned devices at all.

 

That being said, If I click on Dashboard I see an unassigned device.

 

If I click on Main, I see Disk 2 missing but it shows how much data is used and when I click the folder I can actually browse data. I also see that my unassigned drive is mising.

 

I've attached the latest diagnostics as well.

 

via-nas01-diagnostics-20190821-0758.zip

[itimpi]

31 minutes ago, QPlus7 said:

If I click on Main, I see Disk 2 missing but it shows how much data is used and when I click the folder I can actually browse data.

This is perfectly normal   Unraid will be emulating disk2 using the combination of all the good data drives plus parity.   The data showing will be what will end up on disk2 after you have rebuilt it.

[QPlus7]

8 minutes ago, itimpi said:

This is perfectly normal   Unraid will be emulating disk2 using the combination of all the good data drives plus parity.   The data showing will be what will end up on disk2 after you have rebuilt it.

That makes sense - thanks for the confirmation. Any thoughts on my new drive (the one that was in the process of pre-clearing) disappearing?

Edited August 21, 2019 by QPlus7

[QPlus7]

I just restarted the server and now it shows my new drive as unassigned again. I'm going to select is as my Disk 2 and cross my fingers.

 

Well, I tried to add it to Disk 2 and it originally gave me the option but then it disappeared immedately after I selected it from the drop down.

 

I also noticed the pre-clear didn't finish so I am going to resume that and try to add the disk once it completes.

 

I'm sure this is all user error but the disappearing disks are making me nervous.

[JorgeB]

New disk dropped offline, looks more like a connection problem but since there's no SMART report it's harder to say, replace cables and post new diags.

[QPlus7]

15 minutes ago, johnnie.black said:

New disk dropped offline, looks more like a connection problem but since there's no SMART report it's harder to say, replace cables and post new diags.

 

I replaced all SATA cables yesterday. It could be the motherboard but I'm going to replace the drive because I noticed it said something about unable to format the partition. Just to rule that out, I'm going to exchange the drive for a new one later today.