Good Afternoon (and happy xmas!)

 

I'm having issues with my Unraid setup after changing to new hardware.  My previous setup ran without issue but I'm re-scaling the hardware to reduce power usage.
 

Everything was tested pre migration as my backup server (5 HDD and 1 SSD) without issue for over a month including full data copy between the servers and multiple parity checks.

I moved the main server USB and disks to the hardware and started having issues within 2 days, 1 of the drives (10TB) kept dropping from the array without warning,  I replaced the sata cable, redone the power to remove all splitters and ended up moving the drive to another controller.  Nothing resolved the issue on the drive (which rebuilt every time and then dropped within 2 days).  I replaced the drive in the end and done a parity swap.

 

Fast forward to a week later and the old parity drive which is now in the same disk slot 2 (from an Unraid perspective) has just done the same.  It's using the same sata/power cables/controller it's had from the initial move and has showed no issues until now.

 

As an FYI the hardware is a laptop using M.2 sata adaptors, the PSU running the drives is an M2-ATX 160W the total power usage at the time of the failure was under 60w with a 130W power brick running both the laptop and PSU. 

 

Can anyone shed some light and point me in the right direction?
Log below and diagnostic's attached.

 

Spoiler

Dec 24 18:21:04 Tower  emhttpd: spinning down /dev/sde
Dec 24 18:21:35 Tower  emhttpd: spinning down /dev/sdc
Dec 24 18:31:23 Tower  emhttpd: spinning down /dev/sdf
Dec 24 18:50:28 Tower  emhttpd: read SMART /dev/sdf
Dec 24 18:50:52 Tower  emhttpd: read SMART /dev/sdc
Dec 24 19:07:25 Tower  emhttpd: spinning down /dev/sdf
Dec 24 19:07:25 Tower  emhttpd: spinning down /dev/sdc
Dec 24 19:11:27 Tower  emhttpd: read SMART /dev/sdf
Dec 24 19:12:01 Tower  emhttpd: read SMART /dev/sdc
Dec 24 19:44:09 Tower  emhttpd: spinning down /dev/sdc
Dec 24 19:45:45 Tower  emhttpd: read SMART /dev/sdc
Dec 24 20:10:59 Tower  emhttpd: read SMART /dev/sde
Dec 24 20:40:00 Tower  emhttpd: spinning down /dev/sde
Dec 24 20:42:37 Tower  emhttpd: read SMART /dev/sde
Dec 24 20:59:08 Tower  emhttpd: spinning down /dev/sde
Dec 24 20:59:08 Tower  emhttpd: spinning down /dev/sdc
Dec 24 23:46:57 Tower  emhttpd: spinning down /dev/sdf
Dec 25 00:00:01 Tower Plugin Auto Update: Checking for available plugin updates
Dec 25 00:00:08 Tower Plugin Auto Update: Update available for tips.and.tweaks.plg (Not set to Auto Update)
Dec 25 00:00:08 Tower Plugin Auto Update: Update available for unassigned.devices.preclear.plg (Not set to Auto Update)
Dec 25 00:00:09 Tower Plugin Auto Update: Community Applications Plugin Auto Update finished
Dec 25 03:09:32 Tower  emhttpd: read SMART /dev/sdf
Dec 25 03:09:49 Tower  emhttpd: read SMART /dev/sdc
Dec 25 03:22:01 Tower  emhttpd: read SMART /dev/sde
Dec 25 03:28:24 Tower kernel: ata8.00: exception Emask 0x10 SAct 0x0 SErr 0x49e0000 action 0xe frozen
Dec 25 03:28:24 Tower kernel: ata8.00: irq_stat 0x00000040, connection status changed
Dec 25 03:28:24 Tower kernel: ata8: SError: { PHYInt CommWake 10B8B Dispar LinkSeq DevExch }
Dec 25 03:28:24 Tower kernel: ata8.00: failed command: FLUSH CACHE EXT
Dec 25 03:28:24 Tower kernel: ata8.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 9
Dec 25 03:28:24 Tower kernel:         res 40/00:00:f8:e5:09/00:00:22:04:00/40 Emask 0x10 (ATA bus error)
Dec 25 03:28:24 Tower kernel: ata8.00: status: { DRDY }
Dec 25 03:28:24 Tower kernel: ata8: hard resetting link
Dec 25 03:28:30 Tower kernel: ata8: found unknown device (class 0)
Dec 25 03:28:34 Tower kernel: ata8: softreset failed (device not ready)
Dec 25 03:28:34 Tower kernel: ata8: hard resetting link
Dec 25 03:28:38 Tower kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 25 03:28:38 Tower kernel: ata8.00: revalidation failed (errno=-2)
Dec 25 03:28:44 Tower kernel: ata8: hard resetting link
Dec 25 03:28:44 Tower kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 25 03:28:44 Tower kernel: ata8.00: revalidation failed (errno=-2)
Dec 25 03:28:49 Tower kernel: ata8: hard resetting link
Dec 25 03:28:50 Tower kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 25 03:28:50 Tower kernel: ata8.00: revalidation failed (errno=-2)
Dec 25 03:28:50 Tower kernel: ata8.00: disable device
Dec 25 03:28:55 Tower kernel: ata8: hard resetting link
Dec 25 03:28:55 Tower kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=31s
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] tag#9 Sense Key : 0x5 [current] 
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] tag#9 ASC=0x21 ASCQ=0x4 
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] tag#9 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
Dec 25 03:28:55 Tower kernel: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Dec 25 03:28:55 Tower kernel: ata8: EH complete
Dec 25 03:28:55 Tower kernel: ata8.00: detaching (SCSI 8:0:0:0)
Dec 25 03:28:55 Tower kernel: device offline error, dev sde, sector 3909943296 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Dec 25 03:28:55 Tower kernel: device offline error, dev sde, sector 3909943296 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Dec 25 03:28:55 Tower kernel: md: disk2 write error, sector=3909943232
Dec 25 03:28:55 Tower kernel: md: disk2 write error, sector=0
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] Synchronizing SCSI cache
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] Stopping disk
Dec 25 03:28:55 Tower kernel: sd 8:0:0:0: [sde] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17581583448
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17581583456
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17581583464
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17581583472
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17581583480
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17750943160
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17750943168
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17750943176
Dec 25 03:38:29 Tower kernel: md: disk2 write error, sector=17750943184
Dec 25 03:53:31 Tower  emhttpd: spinning down /dev/sde
Dec 25 03:53:31 Tower  emhttpd: sdspin /dev/sde down: 2
Dec 25 04:10:40 Tower  emhttpd: spinning down /dev/sdf
Dec 25 04:10:40 Tower  emhttpd: spinning down /dev/sdc
Dec 25 10:59:17 Tower  emhttpd: read SMART /dev/sdf
Dec 25 11:18:46 Tower  emhttpd: spinning down /dev/sdf
Dec 25 11:44:17 Tower  emhttpd: read SMART /dev/sdd
Dec 25 12:00:35 Tower  emhttpd: spinning down /dev/sdd
Dec 25 12:11:44 Tower  emhttpd: read SMART /dev/sdf
Dec 25 12:12:04 Tower  emhttpd: read SMART /dev/sdc
Dec 25 13:31:30 Tower  emhttpd: spinning down /dev/sdf
Dec 25 13:32:00 Tower  emhttpd: spinning down /dev/sdc

 

 

Many thanks in advance to anyone who can assist.

tower-diagnostics-20221225-1407.zip

Edited December 25, 20223 yr by res2cou

Both JMB controllers are showing ATA errors, but worse for one since the disk dropped offline, are they the same model? Try connecting the disk that dropped to the other controller.

Thanks for your reply.

 

I moved the drive that dropped to the other controller with a fresh sata cable late last night and started the rebuild process, it's about 50% so far. I'll post new diag's when I'm back home.

 

Where/what in the diag's point you to the errors? I'd love to know what I'm looking for.

 

Would you know if there's any known compatibility issues with the JMB controllers and drives? Are the errors showing for all drives or just the WD?

The controllers are JMB582 and JMB585. 

24 minutes ago, res2cou said:

The controllers are JMB582 and JMB585.

I believe that the JMB 585 adds a port multiplier to get extra ports, and port multipliers are NOT recommended for use with Unraid as they seem to cause both speed and reliability issues.

Hi itimpi.

 

I believe you might be confusing it with the JMB575 which is the multiplier.

 

I've attached the datasheet for the 585.

 

I did check the forums ahead of selecting the controller.

JMB585.pdf

On 12/25/2022 at 10:50 PM, res2cou said:

the PSU running the drives is an M2-ATX 160W

Only powering hard disk exclude laptop ? I notice disk will spindown, if only powering hard disk ( from 5 to 2 now ) then the loading may be too low. So, you could try not spindown and check any different.

 

And what is the input voltage ?

20 hours ago, res2cou said:

Would you know if there's any known compatibility issues with the JMB controllers and drives?

They are recommend and usually work fine, if it's a problem with the controller it could be that specific controller.

The disk2 rebuild without issue and doesn't look to be have reported any further issues (diag attached).

 

I did notice after running through the old diags that the ata error on the 582 controller was reported 5 seconds after running the powertop script (log section below).

Spoiler

Dec 24 14:58:24 Tower  emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/startScript.sh /tmp/user.scripts/tmpScripts/Powertop autotune/script 
Dec 24 14:58:26 Tower  emhttpd: read SMART /dev/sdc
Dec 24 14:58:29 Tower kernel: ata2.00: exception Emask 0x0 SAct 0x30 SErr 0x0 action 0x6 frozen
Dec 24 14:58:29 Tower kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 24 14:58:29 Tower kernel: ata2.00: cmd 61/08:20:88:33:35/00:00:00:02:00/40 tag 4 ncq dma 4096 out
Dec 24 14:58:29 Tower kernel:         res 40/00:28:90:33:35/00:00:00:02:00/40 Emask 0x4 (timeout)
Dec 24 14:58:29 Tower kernel: ata2.00: status: { DRDY }
Dec 24 14:58:29 Tower kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 24 14:58:29 Tower kernel: ata2.00: cmd 61/08:28:90:33:35/00:00:00:02:00/40 tag 5 ncq dma 4096 out
Dec 24 14:58:29 Tower kernel:         res 40/00:28:90:33:35/00:00:00:02:00/40 Emask 0x4 (timeout)
Dec 24 14:58:29 Tower kernel: ata2.00: status: { DRDY }
Dec 24 14:58:29 Tower kernel: ata2: hard resetting link
Dec 24 14:58:34 Tower kernel: ata2: found unknown device (class 0)
Dec 24 14:58:37 Tower kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 24 14:58:37 Tower kernel: ata2.00: configured for UDMA/133
Dec 24 14:58:37 Tower kernel: ata2: EH complete

I've stopped the powertop autotune not run on this boot.
Could this be an issue with the controllers/drives not working well with powertop?

 

Vr2lo: The input voltage to both the laptop and the M2-ATX is 19V from the same PSU.  Power usage measured at the mains as per the graph below,

(Disk 2 rebuild finishing at the start of the graph, Mostly idle the 2nd half)

The issues also seem to have happened when the disks were already spinning which I think counters the too little power and also too much power draw.  I'll add a meter to monitor the output of the M2-ATX 12v line to see if it dips.

 

My current thinking is to leave it to settle for 3-5 days and see if any errors show up,  if all clear I might run powertop and see if anything changes.  I know powertop has options to exclude devices from it's tuning.

tower-diagnostics-20221227-1443.zip

10 minutes ago, res2cou said:

Could this be an issue with the controllers/drives not working well with powertop?

Definitely, it has been reported before with various controllers.

Evening all,

 

Everything has been working without any issues for over a week and a half without powertop with auto tune being run.

 

Thanks for your assistance.