emhttpd: error: hotplug_devices

[Nitesh]

Hi Guys,

I had this error come up with one of my data drives (disk 1) and I am not sure if its drive failure or something to do with cables.

Once the error came up for the drive in question. that drive showed up in unassigned devices for some reason. Never have I had a drive fail and it show up in unassigned devices.

Attached are some of the screenshots and diagnostics attached.

I always have spare drives on hand so can swap it immediately if its a drive issue.

Any help would be awesome!

Cheers

megaatlantis-diagnostics-20230507-1411.zip

[itimpi]

Looking at the diagnostics disk1 dropped offline (which resulted in write errors and thus it got disabled) and then reconnected with a different device Id which is why it then appeared under Unassigned Devices.  It is highly likely the drive is fine and it is an external factor that caused this (e.g sata or power cabling) as the SMART information for the drive looks fine.

 

You also have an aggressive spindown timeout (15 minutes).   Thought it was worth mentioning this in case there are issues with spinning up the drive as the syslog contains regular occurrences of sequences of the form

Apr 18 05:52:33 MegaAtlantis kernel: mpt2sas_cm0: log_info(0x31120500): originator(PL), code(0x12), sub_code(0x0500)
Apr 18 05:52:33 MegaAtlantis kernel: sd 7:0:3:0: [sde] tag#2836 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=8s
Apr 18 05:52:33 MegaAtlantis kernel: sd 7:0:3:0: [sde] tag#2836 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e5 00
Apr 18 05:52:33 MegaAtlantis  emhttpd: read SMART /dev/sde

where the read SMART indicates that Unraid thinks the drive has just been spun up.

[Nitesh]

4 hours ago, itimpi said:

Looking at the diagnostics disk1 dropped offline (which resulted in write errors and thus it got disabled) and then reconnected with a different device Id which is why it then appeared under Unassigned Devices.  It is highly likely the drive is fine and it is an external factor that caused this (e.g sata or power cabling) as the SMART information for the drive looks fine.

 

You also have an aggressive spindown timeout (15 minutes).   Thought it was worth mentioning this in case there are issues with spinning up the drive as the syslog contains regular occurrences of sequences of the form

Apr 18 05:52:33 MegaAtlantis kernel: mpt2sas_cm0: log_info(0x31120500): originator(PL), code(0x12), sub_code(0x0500)
Apr 18 05:52:33 MegaAtlantis kernel: sd 7:0:3:0: [sde] tag#2836 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=8s
Apr 18 05:52:33 MegaAtlantis kernel: sd 7:0:3:0: [sde] tag#2836 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e5 00
Apr 18 05:52:33 MegaAtlantis  emhttpd: read SMART /dev/sde

where the read SMART indicates that Unraid thinks the drive has just been spun up.

Thanks for checking this for me.

I shut down the server and checked each disk cable by unplugging and plugging back in and when I restarted. Got a message 0 Errors.
But still there is a X next to the disk. Whats the best way to solve this one ? Screen shot attached.


Also thanks for pointing out the aggressive spin down. I did that for a test and never reverted back.
I have reverted it back to 1 hour. Hoping that 1 hour is not considered aggressive.

[itimpi]

It takes a rebuild (or the New Config tool) to clear the red 'x'.  As long as the contents of the emulated disk1 look OK (as a rebuild just makes a physical disk match an emulated one) then you can rebuild the drive onto itself as described here in the online documentation.

[Nitesh]

Will start the rebuild tonight and report back how I go

[Nitesh]

followed the instruction and started rebuild but it failed. Screenshot attached.

Also attached are the latest diagnostics

[itimpi]

15 minutes ago, Nitesh said:

Also attached are the latest diagnostics

These are missing!

[Nitesh]

oops. here it is

megaatlantis-diagnostics-20230508-2116.zip

[itimpi]

Looking at the syslog as soone as you start the rebuild you get 

May  8 21:08:23 MegaAtlantis kernel: sd 7:0:0:0: [sdb] tag#1201 Sense Key : 0x2 [current] 
May  8 21:08:23 MegaAtlantis kernel: sd 7:0:0:0: [sdb] tag#1201 ASC=0x4 ASCQ=0x0 
May  8 21:08:23 MegaAtlantis kernel: sd 7:0:0:0: [sdb] tag#1201 CDB: opcode=0x8a 8a 00 00 00 00 00 01 9b 04 58 00 00 04 00 00 00
May  8 21:08:23 MegaAtlantis kernel: I/O error, dev sdb, sector 26936408 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0
May  8 21:08:23 MegaAtlantis kernel: md: disk1 write error, sector=26936344
May  8 21:08:23 MegaAtlantis kernel: md: disk1 write error, sector=26936352

Followed by a lot more write errors.

 

It may really be a failing disk, so I suggest you run the click on the drive on the Main tab; disable spindown; and run the extended SMART test.   If that fails then you really have a failing drive. 
 

BTW:  it was not relevant in this case, but often we will want the diagnostics with the array started in normal mode so we can see if emulated disks are mounting.

[Nitesh]

I did the extended test as suggested and for sure had read errors and failed at 10%.

Just added a new drive to the array and started the rebuild. Will keep you posted on how it goes.

Thanks for your help with this

[Nitesh]

OMG what a nightmare this has been. While building Drive 1. Drive 7 Failed.

Luckily I was able to build both the drives without issues

Thanks again for your help