Hi,

 

Yesterday I added an additional 22TB drive to my existing parity drive and began the parity-sync. Everything was running well and the parity sync was running at decent speeds.

 

This morning I checked on it and it was about 18% complete, however, the pairty-sync speeds were now 2-3MBs. I thought that this may have been due to the mover running and I noticed the timezone was set incorrectly on my server so I fixed that and resumed the pairty sync with the understanding that perhaps it would speed back up once the mover completed its work.

 

However, after pausing and resuming the parity-sync twice maybe three times it now shows it's only writing at 500-800KBs and seems to decrease each time I pause and resume the pairty sync. Any idea what could be going on?

 

Attached below are my diagnostics. (Side note, I've been having a lot of problems with other things since I replaced the entirety of my servers hardware the other day. You may see things that could allude to that as well in the diagnostics.)

 

Thanks!

 

deepwater-diagnostics-20240725-1121.zip

Edited July 25, 20241 yr by mswdev

Jul 25 07:36:38 Deepwater kernel: sd 9:0:0:0: Power-on or device reset occurred
Jul 25 07:36:38 Deepwater kernel: sd 9:0:1:0: Power-on or device reset occurred
Jul 25 07:36:38 Deepwater kernel: sd 9:0:2:0: Power-on or device reset occurred

 

Issues with multiple devices, this usually means a power/connection issue, see if these disks share something, like a power splitter, could also be the miniSAS cable

13 minutes ago, JorgeB said:

Jul 25 07:36:38 Deepwater kernel: sd 9:0:0:0: Power-on or device reset occurred
Jul 25 07:36:38 Deepwater kernel: sd 9:0:1:0: Power-on or device reset occurred
Jul 25 07:36:38 Deepwater kernel: sd 9:0:2:0: Power-on or device reset occurred

 

Issues with multiple devices, this usually means a power/connection issue, see if these disks share something, like a power splitter, could also be the miniSAS cable

 

To add more context: I'm running a 45Drives Storinator S45 as the server case with 18 of the 45 bays filled. It's one of the older styles that doesn't use a backplane. I recently upgraded the cable harness to use 1 meter SFF 8643 -> 4 8482 SFF SAS cables.

 

I'm then running triple Broadcom SAS 9305-16i HBAs. The power for all of the drives is using the cable harness that comes with the Storinator using single molex to power two sata connectors per sas cable. You can see what that looks like in this video from 45drives here at timestamp 1:33: 

 

 

The disks do share a power splitter, but it should be providing more than enough power? In the logs, is there a way to tell which specific drive is being rebooted by the HBA continuously?

 

The server power supply is a ZIPPY M3W-6950P. Here are the specs of the power supply for reference: https://knowledgebase.45drives.com/kb/power-supply-specifications Also attached below as a pdf incase the link ever becomes dead in the future for others who may come across this.

Zippy 950W M3W-6950P 2+1 Redundant.pdf

Edited July 25, 20241 yr by mswdev

They are:

 

Jul 25 03:20:17 Deepwater kernel: md: import disk5: (sdd) WDC_WD142KFGX-68AFPN0_6AG6SPZT size: 13672382412 
Jul 25 03:20:17 Deepwater kernel: md: import disk6: (sdc) WDC_WD142KFGX-68AFPN0_5MG0JX2J size: 13672382412 

Jul 25 03:20:18 Deepwater emhttpd: import 31 cache device: (sdb) Samsung_SSD_860_QVO_4TB_S4PJNF0M405049B

 

And they are the only 3 devices connected to one of the HBAs

26 minutes ago, JorgeB said:

They are:

 

Jul 25 03:20:17 Deepwater kernel: md: import disk5: (sdd) WDC_WD142KFGX-68AFPN0_6AG6SPZT size: 13672382412 
Jul 25 03:20:17 Deepwater kernel: md: import disk6: (sdc) WDC_WD142KFGX-68AFPN0_5MG0JX2J size: 13672382412 

Jul 25 03:20:18 Deepwater emhttpd: import 31 cache device: (sdb) Samsung_SSD_860_QVO_4TB_S4PJNF0M405049B

 

And they are the only 3 devices connected to one of the HBAs

 

I'm thinking this could have something to do specifically with the Samsung 4TB SSD? My reasoning being that I noticed this same issue last night with the HBA continuously attempting to restart that specific drive after it began a TRIM. In those logs you quoted the timestamp is 07:36 and my TRIM settings are set to run a TRIM at 07:30.

 

Perhaps this would explain why I'm only seeing this issue during a TRIM? Is it possible that SSD specifically is damaged? (It was recently gifted to me from a friend and I have not yet tested it. Funny enough, he gave it to me and it was very dirty; he said his dogs took it outside and it could have water damage.) This issue doesn't seem to be occurring anymore and the parity-sync has returned to 170MBs as of right now.

 

Let me know if you think it's possible that the culprit could be that specific SSD or if it makes more sense that it's still an issue with the SAS cables/power?

Edited July 25, 20241 yr by mswdev

I don't think TRIM is the issue, since there are also errors before and after, but you can test.