Added Drives - Array Read/Write Issues

[Gavin10]

I've previously been using unRAID for a few months using four drives, two 120 GB SSD cache drives(1 data 1 parity) and two 5TB array drives (1 data 1 parity). I have been very happy with the +/- 100Mbs over 1Gbe and +/- 250Mbs local transfers on the machine. 

 

This week, I got a good deal on an additional 3 5TB drives. I quickly upgraded to unRAID Server Plus, pre-cleared them and added them into the array to move from 5TB to a total of 20TB. The next day I almost immediately noticed the crippling read/write speeds that I was getting. Locally on the server using Krusader transferring from cache to the array, it starts out great at 250Mbs and then slows down to 50-10Mbs. Same idea over GBe, maxes out at 112Mbs and slows to 50-10Mbs. I downloaded Dynamix System Stats and noticed that the once the RAM cache gets full during the transfer is when the speed slows down. I've also checked all the drives and the SMART checks have passed and have SATA III 6.0Gbps connectivity. 

 

So far, I haven't 'written' any data at all to the new 3 drives. I only have data on 1 drive and the parity drive of course.

 

Any advice? Is this typical speeds? Just not sure why all was well for months before I upgraded unRAID and added the new drives that aren't even being written/read.

 

FX 8320

16GB RAM

MSI 970A SLI Krait motherboard

Edited September 7, 2018 by Gavin10

[trurl]

Possibly some connection issues from having disturbed things while adding disks. If you post your diagnostics we could take a look. Tools - Diagnostics, post complete zip.

 

But writes to the parity array are typically slower due to the fact parity is updated at the same time. There are 2 ways to configure how parity is updated. The standard method only requires the disk to be written and the parity disk to spin, but also requires more reading before writing in order to calculate parity. The other (turbo) method is faster but requires all disks to spin. Here is a link to more details of how these work and how to set them:

 

https://forums.unraid.net/topic/50397-turbo-write/

 

[Gavin10]

Thanks for the quick response trurl. Attached my diagnostics! FYI if you see my parity-sync check is in progress in the diagnostics, I know. I literally just started running my parity-sync check as it hasn't ran in a while, and a small part of me is hoping that will help probably not...

homeserver-diagnostics-20180906-2203.zip

Edited September 7, 2018 by Gavin10

[trurl]

Looks like you are running a correcting parity check. Parity sync is when you (re)build parity.

 

Something odd about your flash? It seems to be showing up as both sr0 and sda, and also getting a lot of errors on the sr0 device. Anything special about how this is setup? You might try another port next boot to see if it makes a difference.

 

Probably not related to your other complaints though. Nothing about any disk or connection problems I could see. And not sure there is anything unusual about your write speeds. Have you looked at that link yet?

 

 

[Gavin10]

Thanks for the quick lesson! 

 

Nope, my flash drive is just plugged into a USB 2.0 port in the motherboard. I do see in the logs where Sandisk is showing up twice though. Maybe I switched it to a different port when I was putting in the new drives.

 

Yep, I've enabled 'reconstruct write' and still no changes...

 

Could be due to the parity check being run? But to the same results... When the parity check is done (about 8 hrs left @ 164Mbs), i'll shut down and try a different port.

[trurl]

1 minute ago, Gavin10 said:

Could be due to the parity check being run?

Yes. In fact I would say you shouldn't even try any sort of test while parity is being checked. Turbo write is basically doing the same thing parity sync or check does. But the turbo writing is happening at different places on the disk than the parity checking is taking place. If you did read that other link, then just consider doubling everything it said about reading, writing, seeking if you are trying to write during a parity check.

[Gavin10]

Yeah, I understand. Simply just not enough bandwith. When the parity check is finished tomorrow, I'll try change the flashdrive port and re-enabling turbo write and report back. 

 

Thank you sir!

[Gavin10]

I have moved the flashdrive to a different port and it is still showing up as sr0 and sda. Also, I have enabled turbo write and the speeds are all over the place to start 100-300Mbs, but still slows down to 10Mbs after a few seconds.

homeserver-diagnostics-20180907-1337.zip

[JorgeB]

8 minutes ago, Gavin10 said:

and it is still showing up as sr0

Some flash drives have a partition that works as an emulated cd-rom, that's sr0, it may or not be removable, but besides spamming the log with some errors it shouldn't cause any issues.

 

9 minutes ago, Gavin10 said:

I have enabled turbo write and the speeds are all over the place to start 100-300Mbs, but still slows down to 10Mbs after a few seconds.

Is the parity check finished?

[Gavin10]

Yep, parity check completed this morning

[JorgeB]

Transfer one or more large files and grab the diags during the transfer after it slows down.

 

 

[Gavin10]

So, some strange things are happening here.

 

I tried to boot into Safe Mode to see if that would rule out any software plugin issues. But I could not log in for the life of me. So I reboot regularly. My server has gone back to it's default 'Tower' name and for some reason it kicked out one of my Cache drives and picked it up as a new drive? All of my data and apps are still okay as I can tell, so I added it back into my cache pool. Now my cache pool has increased from 120gb to 180gb?!

 

Same speed issue though...  

 

Diagnostics - tower-diagnostics-20180907-1119.zip

 

Thanks for all you guys help.. Kind of worrying right now.

[JorgeB]

Transfer is going to cache, so nothing to do with turbo write, check that your cache devices are fast enough and are being trimmed.

 

The other issues suggest a flash drive problem, make sure you're using an USB 2.0 port.

[Gavin10]

You called it. Was plugged into a USB 3.0 when I switched it this evening. Cache is back to normal now and was able to change the name back. 

 

As far as I know, my SSD drives have been read/write of at least 200Mbs constantly. But I guess that could change? And I have the TRIM run at 4AM which is a few hours after the mover. 

 

Still no changes to speed though. But if you look at my screenshot and the red little circle, when the RAM cache gets filled, the drive speeds get hammered. It's like clockwork every time. Is that expected? Or is it possible my RAM has gone bad?

 

EDIT: This is the same behavior if I am copying from cache --> array or array --> cache.

EDIT 2: Here is the diagnostic for cache --> array after it slows down. homeserver-diagnostics-20180907-1201.zip

 

Edited September 7, 2018 by Gavin10

[JorgeB]

11 minutes ago, Gavin10 said:

Is that expected

Yes, and they can't handle it, hence the problem, enable turbo write and disable cache for that share, then transfer directly to the array, you should get 100MB/s +

[Gavin10]

To rule out the SSDs, I disabled cache and downloaded the 16gb file to my PC over GBe. I got an extremely constant 113Mbs. However, writing to the array started at 100Mbs, and fell down to 50Mbs.

 

[JorgeB]

With turbo write enable?

[Gavin10]

Hey, sorry for the late response. Was out of town for the weekend.

 

I have downloaded and enabled the CA Turbo Write plugin and successfully have been getting 100Mbs writing to my HDD array. But, when copying to the Cache things slow down.

 

Seems like one or both of my SSDs have gone bad of some sort.

[Gavin10]

Bought two new Samsung 860 EVO 500 GB and speeds have been better. Problem solved. Thanks @trurl and @johnnie.black for your help resolving my issue!