Device Disabled, contents emulated

[Tharnax]

I saw a few threads on this topic but didn't see anything covering my specific issue so started a new thread to request input / support.  I'm setting up a new Unraid server and purchased three new 16TB WD Red Pro drives for the initial parity drive and two disks.  I used one of the 16TB to backup some data from a previous WHS 2011 onto it via external USB connector.  I then proceeded to setup a new Unraid setup with the other 2 16TB drives, one as parity and the other as "Disk 1".  I started the Array and I let the initial parity complete.  Following the completed initial parity I proceeded to copy about 10TB worth of data, including all of the data from what I backed up on the 16TB drive.  I then shut down the Unraid server to install the 16TB drive and add it into the array.  Upon boot everything appeared to be fine.  I selected the 16TB, as "Disk 2" and started the array.  "Disk 1" went into the "Device Disabled, contents emulated" state.  I'm not sure what to do next.  The Disk log information is below and I've attached a server diagnostics zip and server smart zip.  Hopefully some one has some suggestions as I prefer not to loose the 10TB of data I have on the 16TB drive as I don't have a local backup, although worst case I could recover the data from a remote location backup if it comes to it.

 

Disk Log Information:

Jun 21 17:30:31 Server kernel: ata7: SATA max UDMA/133 abar m2048@0xf0710000 port 0xf0710100 irq 31
Jun 21 17:30:31 Server kernel: ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jun 21 17:30:31 Server kernel: ata7.00: ATA-11: WDC WD161KFGX-68AFPN0, 83.00A83, max UDMA/133
Jun 21 17:30:31 Server kernel: ata7.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jun 21 17:30:31 Server kernel: ata7.00: Features: NCQ-sndrcv NCQ-prio
Jun 21 17:30:31 Server kernel: ata7.00: configured for UDMA/133
Jun 21 17:30:31 Server kernel: sd 7:0:0:0: [sdg] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jun 21 17:30:31 Server kernel: sd 7:0:0:0: [sdg] 4096-byte physical blocks
Jun 21 17:30:31 Server kernel: sd 7:0:0:0: [sdg] Write Protect is off
Jun 21 17:30:31 Server kernel: sd 7:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Jun 21 17:30:31 Server kernel: sd 7:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 21 17:30:31 Server kernel: sdg: sdg1
Jun 21 17:30:31 Server kernel: sd 7:0:0:0: [sdg] Attached SCSI removable disk
Jun 21 17:30:50 Server emhttpd: WDC_WD161KFGX-68AFPN0_2BKGGU5R (sdg) 512 31251759104
Jun 21 17:30:51 Server kernel: mdcmd (2): import 1 sdg 64 15625879500 0 WDC_WD161KFGX-68AFPN0_2BKGGU5R
Jun 21 17:30:51 Server kernel: md: import disk1: (sdg) WDC_WD161KFGX-68AFPN0_2BKGGU5R size: 15625879500 
Jun 21 17:30:51 Server emhttpd: read SMART /dev/sdg
Jun 21 18:12:49 Server kernel: ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jun 21 18:13:05 Server kernel: ata7.00: qc timeout (cmd 0x2f)
Jun 21 18:13:05 Server kernel: ata7.00: Read log 0x13 page 0x00 failed, Emask 0x4
Jun 21 18:13:05 Server kernel: ata7.00: Read log 0x00 page 0x00 failed, Emask 0x40
Jun 21 18:13:05 Server kernel: ata7.00: NCQ Send/Recv Log not supported
Jun 21 18:13:05 Server kernel: ata7.00: Read log 0x00 page 0x00 failed, Emask 0x40
Jun 21 18:13:05 Server kernel: ata7.00: ATA Identify Device Log not supported
Jun 21 18:13:05 Server kernel: ata7.00: failed to set xfermode (err_mask=0x40)
Jun 21 18:13:05 Server kernel: ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jun 21 18:13:21 Server kernel: ata7.00: qc timeout (cmd 0x2f)
Jun 21 18:13:21 Server kernel: ata7.00: Read log 0x00 page 0x00 failed, Emask 0x4
Jun 21 18:13:21 Server kernel: ata7.00: NCQ Send/Recv Log not supported
Jun 21 18:13:21 Server kernel: ata7.00: Read log 0x00 page 0x00 failed, Emask 0x40
Jun 21 18:13:21 Server kernel: ata7.00: ATA Identify Device Log not supported
Jun 21 18:13:21 Server kernel: ata7.00: failed to set xfermode (err_mask=0x40)
Jun 21 18:13:21 Server kernel: ata7: limiting SATA link speed to 3.0 Gbps
Jun 21 18:13:21 Server kernel: ata7.00: limiting speed to UDMA/133:PIO3
Jun 21 18:13:21 Server kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Jun 21 18:13:26 Server kernel: ata7.00: qc timeout (cmd 0xec)
Jun 21 18:13:26 Server kernel: ata7.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Jun 21 18:13:26 Server kernel: ata7.00: revalidation failed (errno=-5)
Jun 21 18:13:26 Server kernel: ata7.00: disabled
Jun 21 18:13:27 Server kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Jun 21 18:13:27 Server kernel: sd 7:0:0:0: [sdg] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Jun 21 18:13:27 Server kernel: sd 7:0:0:0: [sdg] Sense not available.
Jun 21 18:13:27 Server kernel: sd 7:0:0:0: [sdg] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Jun 21 18:13:27 Server kernel: sd 7:0:0:0: [sdg] Sense not available.
Jun 21 18:13:27 Server kernel: sd 7:0:0:0: [sdg] 0 512-byte logical blocks: (0 B/0 B)
Jun 21 18:13:27 Server kernel: sd 7:0:0:0: [sdg] 4096-byte physical blocks
Jun 21 18:13:27 Server kernel: sdg: detected capacity change from 31251759104 to 0

** Press ANY KEY to close this window ** 

server-diagnostics-20220621-2238.zip server-smart-20220621-1935.zip

[itimpi]

It looks like disk1 dropped offline for some reason.  The syslog also shows errors occurring on disk2.   In both cases it looks like it could be cabling related.

 

I would suggest powering down the server, carefully check the power and SATA cabling to the drives, reboot the server and then get and post new diagnostics.

 

BTW:  The diagnostics also include SMART reports as standard so normally no reason to provide them separately.

[JorgeB]

Note that Marvell controllers are not recommended because they can sometimes drop disks without a reason.

[Tharnax]

Thank you both for your input.

 

Disk 2 is currently doing the following:  

 

Is it okay to power down the server while the above activity is occurring or should I wait until it completes?  Upon power down I'll check all the cables and move the drives to the onboard Intel SATA controller so it gets them off them Marvel SATA controller to hopefully increase stability.  I'll remove the Marvel SATA controller at the same time if it isn't recommended for Unraid.

 

@JorgeB - I see on another thread you recommended an IO CREST Internal 5-Port Non-Raid SATA III controller card.  This is a Canadian Amazon site but it appears to be the same card: https://www.amazon.ca/Internal-Non-Raid-Controller-Desktop-Bracket/dp/B07ST9CPND/ref=sr_1_4?crid=2FEEM476N86H9&keywords=IO+CREST+Internal+5+Port+Non-Raid+SATA+III&qid=1655902223&sprefix=io+crest+internal+5+port+non-raid+sata+iii%2Caps%2C256&sr=8-4.  Do you still recommend this card?  Seems like I can order it and get it same day to replace the Marvel controller card. 

 

 

[JorgeB]

40 minutes ago, Tharnax said:

Is it okay to power down the server while the above activity is occurring or should I wait until it completes?

It's OK to power down but it will start over from the beginning, might as well see if it finishes.

 

41 minutes ago, Tharnax said:

it appears to be the same card

It is, those are a good option.

[Tharnax]

Awesome, thank you @JorgeB, very much appreciate the replies.  Since it is approximately 6 hours from completion, I'll let it run.  Besides it'll allow time for the new SATA controller to arrive so I can install it at the same time.

[trurl]

7 hours ago, itimpi said:

BTW:  The diagnostics also include SMART reports as standard so normally no reason to provide them separately.

But in this case, diagnostics didn't include SMART for disk1 since it wasn't attached. And the attached SMART report for disk1 didn't work.

 

So I wonder if it is seeing disk1 even now.

 

49 minutes ago, Tharnax said:

Disk 2 is currently doing the following

Not entirely clear about this. It should be rebuilding disabled disk1. If it is doing anything with disk2 it would be clearing it.

 

Post a screenshot of Main - Array Devices and new diagnostics.

 

 

[JorgeB]

4 minutes ago, trurl said:

If it is doing anything with disk2 it would be clearing it.

It is, it shows that in the screenshot, good catch.

[trurl]

8 minutes ago, JorgeB said:

it shows that in the screenshot

So is disk1 not seen and so not being rebuilt? Or will it just not do rebuild while a clear is happening?

 

[JorgeB]

Will be able to rebuild disk1 once the clearing is done, I assume disk2 is really a new disk, because it's getting wiped, and if disk2 was supposed to be used as disk1 he should cancel the clear now and after that will be able to use that disk to rebuild disk1, if the clear finishes can't so easily remove the disk from the array, especially with aa disabled disk.

[trurl]

Different serial numbers on disk1 smart and disk2 diagnostics

[Tharnax]

Hi All, thank you for your continued analysis.  Here is an image of the Array:  

 

 

 

I've also attached the latest diagnostics. 

 

Parity / Disk 1 & Disk 2 are all brand new in the last few weeks.  Disk 2 I purchased first and utilized it to copy some data from the old WHS 2011 server on to another Unraid server I am building, so Disk 2 wasn't completely empty but the data that was on Disk 2 is backed up and Disk 2 is fine to be added to the array.  I assumed it would be cleaned / formatted then added to the existing array of Parity + Disk 1.  Both Parity and Disk 1, were fresh right out of the packaging and added to this brand new Unraid install.  All the other Unassigned disks you see below are from the old WHS 2011 server, and once I get the array working with Parity + Disk 1 + Disk 2, I will mount and rsync copy the data over from each of the unassigned disks.  As each disk's data is copied onto the array, I will clear / format it and add it to the array.  Note: I will only be adding back in the 3x 8TB drives, not the 3x 3TB drives and I'll be adding two 500GB SSD drives (Raid 1) for cache once all is said and done.

 

 

Please let me know if I should stop the current array process or not?  Worst case if the parity drive will rebuild the array on Disk 2, then I can re-add Disk 1 as a new disk.  If Disk 2 is being added to the array as a new "empty" disk, then I should power down server, fix the connection settings and restart the server confirm Disk 1 is "active" and then re-build the array?

 

 

Thank you!

server-diagnostics-20220622-1341.zip

[JorgeB]

Let the clear finish then rebuild disk1.

[Tharnax]

Clear finished, powered down the server.  Replaced the SATA controller, the 16TB was connected to the Marvel SATA controller so for the moment it is now connected to the new SATA controller.  Powered up the server and followed the steps as per the manual:

• Stop array
• Unassign disabled disk
• Start array so the missing disk is registered
• Important: If the drive to be rebuilt is a data drive then check that the emulated drive is showing the content you expect to be there as the rebuild process simply makes the physical drive match the emulated one. Is this is not the case then you may want to ask in forums for advice on the best way to proceed.
• Stop array
• Reassign disabled disk
• Start array to begin rebuild. If you start the array in Maintenance mode you will need to press the Sync button to start the rebuild.

Started the area in maintenance mode and it did present the Sync button.  It appears a sync "Data-Rebuild" is now in progress.  

 

So far it all appears to be okay but I've attached a latest diagnostics log incase you have an opportunity to take a look and see if there are any concerns.  I hope, so far so good.  Thanks for your help!

server-diagnostics-20220622-1932.zip

[trurl]

2 hours ago, Tharnax said:

Started the area in maintenance mode

Don't know why maintenance mode is even mentioned at the end of those. I guess it is an option to do it that way, but not usually done like that.

 

2 hours ago, Tharnax said:

Important: If the drive to be rebuilt is a data drive then check that the emulated drive is showing the content you expect to be there as the rebuild process simply makes the physical drive match the emulated one. Is this is not the case then you may want to ask in forums for advice on the best way to proceed.

And since you are in maintenance mode, no drives are mounted, so no way to know whether you are rebuilding a mountable filesystem or not.

 

[trurl]

Maybe you started in normal mode and checked the contents of the emulated disk, but if so, you didn't mention it (or I missed it).

[Tharnax]

2 minutes ago, trurl said:

Nothing in those instructions said anything about maintenance mode.

 

And since you are in maintenance mode, no drives are mounted, so no way to know whether you are rebuilding a mountable filesystem or not.

 

Nothing in the instructions "https://wiki.unraid.net/Manual/Storage_Management#Rebuilding_a_drive_onto_itself" suggested that I can't use maintenance mode, it just stated if you star the array in maintenance mode you will need to press the Sync button to start the rebuild.  Sounds like a made the wrong assumption that maintenance mode is a "safer" approach.  Hopefully I haven't made things worse, I'm very much learning as I go.

 

What do you mean no drives are mounted?  This is what I see in the Array Devices:

[trurl]

I edited my post

10 minutes ago, trurl said:

Don't know why maintenance mode is even mentioned at the end of those. I guess it is an option to do it that way, but not usually done like that.

 

2 minutes ago, Tharnax said:

What do you mean no drives are mounted?

When a drive is mounted, all of those columns at the end (FS, Size, Used, Free, View) have something in them.

 

And your diagnostics shows no disks mounted, that is why I said

12 minutes ago, trurl said:

no way to know whether you are rebuilding a mountable filesystem or not.

 

7 minutes ago, trurl said:

Maybe you started in normal mode and checked the contents of the emulated disk, but if so, you didn't mention it (or I missed it).

Did you?

[trurl]

I guess it doesn't matter now. Finish rebuild, and if the disk is unmountable, we can try to repair it (check filesystem)

[trurl]

The earlier diagnostics did show disabled disk1 mounted, so it should be fine.

[Tharnax]

1 minute ago, trurl said:

I guess it doesn't matter now. Finish rebuild, and if the disk is unmountable, we can try to repair it (check filesystem)

Crap!  And no, I didn't start it in normal mode.  I made a best guess based on those instructions and since it based on the note it didn't seem to matter if it was normal mode or maintenance mode I made the wrong choice.  I'll let the rebuild complete and then as you stated, see if it is mountable.

[trurl]

My bad, I should have reviewed the thread.

[Tharnax]

1 minute ago, trurl said:

The earlier diagnostics did show disabled disk1 mounted, so it should be fine.

Fingers crossed but after the disk clean of Disk 2, I did shut down the server and replace the SATA controller and restart.  That's when I started to follow the steps above.  Sounds like I made the mistake of not just starting the array in normal mode.

[itimpi]

1 hour ago, Tharnax said:

Crap!  And no, I didn't start it in normal mode.  I made a best guess based on those instructions and since it based on the note it didn't seem to matter if it was normal mode or maintenance mode I made the wrong choice.  I'll let the rebuild complete and then as you stated, see if it is mountable.

If you carried out the earlier listed step to check that the contents of the emulated disk looked OK then I would expect the final result will be fine.

 

The advantage of doing this in Maintenance mode is that nothing else can write to the array while the rebuild is running which maximises speed.  The disadvantage is that you cannot use the array in the meantime and until you return to normal mode cannot see what the contents of the disk being rebuilt will look like.

 

I will edit the instructions to make this clearer.

[Tharnax]

@itimpi - Thank you for clarifying, that helps me breath a sigh of relief.  As this is a new setup, I'm not concerned with not being able to write to the array while the rebuild is running.  I still have another 7 hours in the data-rebuild process but if you could advise on the anticipated next step I should do following the data-Rebuild that would be appreciated.  Then I know what to expect or do next so I don't "guess" myself into another potential concern. 

 

Yes, I did check the emulated disk before starting and it all looked okay from my perspective, as the folder structured appeared accurate not that I necessarily know what I'm looking for.