Bad performance from one drive location

[Mike90]

I have been having an issue with my server that I have had for a while now.

The issue stared a while back but recently has gotten worse.

It seems that ONLY one drive LOCATION has been having this issue.

When transferring data to of from that disk location it will seem alright for a 30 seconds to a minute, but then will drop to 5mb/s and even 0mb/s and sit there till it feels like going back to 5mb/s for a bit then back to 0mb/s.

I say one drive location and not the drive itself because I have replaced the drive in the location several times. each has had this issue.
The just removed a drive from the location last week because it started to get UDMA CRC error counts. 

Over 4k in a week.

I've tested the drive and it passes smart tests from WD.

The drive I replaced it with luckily has not had any UDMA counts.. yet.
The drive is my most used drive as it is where I have my P2P files located. 

 

I have tried swapping the sas cables and cards.. 

I even shut down my P2P Docker to test if it was cause by the dockers read/write to the drive.

Still the same speed issues...

 

if anyone has any suggestions that would be greatly appreciated. 

 

 

 

Chassis: SuperMicro 4U CSE-846 24 Bay  

Backplane: BPN-SAS2-846EL1

Mobo: X9DRi-F/2x 

CPUs: E5-2620 x2

RAM: 32GB 

Controller: SAS2308 x2 (one controller both lanes from BP)

[Mike90]

I tried to transfer a 1.6GB file. 

Below are snap shots I took while it was 'working' on it. 

Its been almost 20 minutes now and still not finished.

Only about 3/4 done.

 

[JorgeB]

Please post the diagnostics: Tools -> Diagnostics

[Mike90]

here is the diagnostics zip. 

tron-diagnostics-20210505-1038.zip

[JorgeB]

10 hours ago, Mike90 said:

It seems that ONLY one drive LOCATION has been having this issue

What drive is that? Disk22 is SMR so degraded write performance is expected.

[Mike90]

1 hour ago, JorgeB said:

What drive is that? Disk22 is SMR so degraded write performance is expected.

yes. however the drive is brand new.

I have the 8TB version of the drive in other spots without issues. 

Previous drives that have been in 22s location were not SMR drives, and had the same issues.

 

[JorgeB]

10 minutes ago, Mike90 said:

I have the 8TB version of the drive in other spots without issues. 

The 8TB drive are not SMR, despite the model being the same.

[Mike90]

3 minutes ago, JorgeB said:

The 8TB drive are not SMR, despite the model being the same.

either way. the previous drives were not SMR and had the same issue. 

I am trying to read from them.. the data is being written to a cache drive. 

I have moved a drive from one location that worked without issue to the physical location where 22 is now and started having the same issues. 
It is not the drive.

[Mike90]

I had this drive WD30EURS-73TLHY0 in that location before, and it experienced the same issues.

So I removed it thinking it was bad. then replaced it with a 4TB version (that 4TB green drive is in the array still just in another spot) same issue.

Then replaced that with a 4TB Blue drive.. Same issue... and had UDCA errors.. 

I've tested all the removed drives and everything seems fine. 

[JorgeB]

19 minutes ago, Mike90 said:

I am trying to read from them..

You didn't mention that before, assuming the diags were taken after a transfer there's nothing logged, so can't guess what the problem is, but it should be easy to rule out by swapping that disk with another one and see if the problems stays with the slot/cable or follows the disk.

[Mike90]

1 minute ago, JorgeB said:

You didn't mention that before, assuming the diags were taken after a transfer there's nothing logged, so can't guess what the problem is, but it should be easy to rule out by swapping that disk with another one and see if the problems stays with the slot/cable or follows the disk.

I have done this. it does not follow the disk. it seems to be the physical location. But I dont know why... the current drive is the 5th i think.

[Mike90]

could it be the controller on the Backplane?

[JorgeB]

6 minutes ago, Mike90 said:

controller on the Backplane?

What kind of backplane? If there's no expander swap cables with another slot/miniSAS cable, if it's an expander backplane it can only be the slot.

[Mike90]

5 minutes ago, JorgeB said:

What kind of backplane? If there's no expander swap cables with another slot/miniSAS cable, if it's an expander backplane it can only be the slot.

Backplane: BPN-SAS2-846EL1 yes its an expander. 

[JorgeB]

Yeah, then AFAIK not much you can do, other than trying with a different backplane, or getting one from ebay, but they can be expensive even used.

[Mike90]

Just now, JorgeB said:

Yeah, then AFAIK not much you can do, other than trying with a different backplane, or getting one from ebay, but they can be expensive even used.

yea.... some people was $500... i paid 500 for the whole system minus drives....

[Mike90]

also its weird that P2P works without issue.... yet i cant move data from drive easily..