[SOLVED] Need help with Slow Transfer Speeds over network

[guyonphone]

I am having an issue with slow transfer speeds from windows to my NAS.

 

Essentially what happens is that transfers will start out at 90MB/s and after the first minute will drop to the 20MB/s range. Many times I will see a copy drop to 0MB/s and sit with no transfer, then it will spike to 200MB/s only to fall back to 10MB/s after a few seconds. Sometimes a copy won't work, and Ill get an error and need to retry the entire copy.

 

My System:

Unraid: 6.17

CPU: AMD FX-8750 (Eight Cores)

Motherboard: Gigabyte GA-990FXA-UD5

Memory: 20 Gigs

Data Drives:

18 Drives formatted with Reiserfs

4 Drives formatted with xfs

 

Cache Drive:

Samsung SSD 850 EVO 500GB

 

Host Bus Adapter:

LSI00244 (SAS 9201-16i) host bus adapter.

(I just recently replaced an AOC-SAS2LP-MV8 and an AOC-SASLP-MV8 with this LSI card because i thought those cards might have been causing my slowness issues however the problem persists)

 

Case: Norco 4224

 

Both My desktop and Server have gigabit nics and my switch is a Dell PowerConnect 5224 fully gigabit.

 

Any help would be greatly appreciated.

 

Thanks!

 

 

unraid-syslog-20160207-1517.zip

[JorgeB]

Do you see the same seesaw speed transferring directly to a xfs disk?

[dvd.collector]

Are you transferring to a cached share?  i.e. writing to your cache drive?

 

I see the same effect as you when I'm writing directly to the array, as it seems I can transfer over the network faster than my array can write.  I think the initial spike you see is it writing to RAM, which then once full causes the "pause" and eventually a failure if you don't limit your connection speed.

 

I don't see this effect writing to my cache SSD though as that can sustain the write speed higher than my network.

 

To work around the issue when I was writing directly to the array I found a copy program that allowed me to throttle the speed I was transferring at.  Limiting to ~40MB/s seemed to allow the transfer to complete successfully.

[guyonphone]

Thanks for your replies guys, let me answer your questions:

 

dvd.collector: Yes, the share I am transferring to, says that cache is enabled.

 

johnnie.black: I will test this and get back to you.

 

Thank You

 

 

[guyonphone]

Hello Johnnie.black,

 

So normally I keep all my disk shares private. I made one public which was using ReiserFS, and then copied to it and got similar results.

I then did the same test with one of my XFS drives, and essentially the same thing happened but it took a lot longer. I got sustained 50MB/s for over half the copy, then it fell to 30MB/s and ultimately 1-2MB/s at which point i paused the copy. Waited a minute, and resumed the copy. That made it spike again, and finish.

 

Thanks

[dvd.collector]

Try installing the dynamix stats plugin and watch the system stats page.  That will show you what speed the disk is writing at.

 

This is how I identified that it seems to write to RAM first, as you see RAM fill up before the disk writing seems to start.

[JorgeB]

You can run diskspeed to look for slow sectors on your disks:

 

diskspeed.sh -s 101

 

This will take some time, about 15 min. per disk, don’t use the array during the test or it will skew results, this is not a complete disk test, but in my experience it’s sufficient to detect slow sectors, like the example below.

[guyonphone]

Hello Johnnie.black,

 

Would you recommend that I run diskspeed on my SSD Cache drive? I assume checking the speed of my array disks in this scenario shouldn't matter because the Cache drive should be intercepting the copy on behalf of the array, is that correct?

 

Thank You

[JorgeB]

Sorry, missed that you are using a cache drive, if you get the same speed transferring to cache drive or directly to the array the disk test is irrelevant and I’m out of ideas.

[gubbgnutten]

Hello Johnnie.black,

 

Would you recommend that I run diskspeed on my SSD Cache drive? I assume checking the speed of my array disks in this scenario shouldn't matter because the Cache drive should be intercepting the copy on behalf of the array, is that correct?

 

Thank You

 

Have you verified that the files actually end up on the cache drive after the transfer?

 

I vaguely remember some thread with a user experiencing performance problems with a BTRFS SSD cache drive after a while. Probably an overly drastic long shot, but if there are no other ideas - what about reformatting the cache drive?

[guyonphone]

Hello Gubbgnutten,

 

I have verified that the files are being saved to the Cache drive. I am going to troubleshoot more this weekend. Things I want to try are:

 

1. Swapping out the switch and all Ethernet Cables.

2. I don't currently know if my Cache drive is connected to a port on my motherboard or my Host Bus Adapter, I will find that out and change from one to other to see if that helps.

3. For some reason I get an error when i try to activate TRIM on my ssd, an IOCTL error. So I want to figure out how to get TRIM running.

4. I want to try copying from another computer on my network to see if I have the same problem transferring, as all copies have been happening from my main desktop.

5. I have several apps installed on my cache drive, (Plex, Ubooquity, Sabnzbd, Sonarr, Sickbeard, Deluge VPN) will the database scans that these apps do interfere with file copies? Is there a way to have two cache drives, one for file processing, and another to host apps?

 

Thanks

[JorgeB]

1. Swapping out the switch and all Ethernet Cables.

 

Always a good idea to try from another pc but the initial 100MB/s suggests that network is working fine.

 

3. For some reason I get an error when i try to activate TRIM on my ssd, an IOCTL error. So I want to figure out how to get TRIM running.

 

FSTRIM doesn’t work on some HBAs, e.g., LSI 9211, should work on onboard ports, not trimming the SSD can explain your poor write performance.

 

[guyonphone]

Reporting back on some of my troubleshooting:

 

1. I connected by cache drive to one of my motherboards sata ports.

2. I was then able to run FSTRIM on my SSD.

3. Trim completed and logged the event.

4. Unfortunately the issue persists.

5. I found a sweet 4 port Intel Nic on Amazon for 54$ so I will be installing that when it comes.

 

 

I have yet to swap out ethernet cables or try a different switch. I will do that tomorrow. 

If that doesn't work ill swap out the Cache drive. Speaking of which I have attached the smart report for my SSD cache drive.

unraid-smart-20160213-0330.zip

[gubbgnutten]

If that doesn't work ill swap out the Cache drive. Speaking of which I have attached the smart report for my SSD cache drive.

 

If if comes to that and you're prepared to swap out the cache drive, try just reformatting it first.

[guyonphone]

Thanks guys for all your help!

 

I have resolved the issue. It was a bad SSD. I still need to perform some further testing on the SSD itself, but it looks like it had some sort of weird internal problem where it is slow when data is copied to the drive (Writing).

 

Thanks again!