Jump to content
Niklas

Transferring big files from Windows to Unraid, everything grinds to a halt

48 posts in this topic Last Reply

Recommended Posts

Have encrypted XFS array, on 6.6.6 I get 160MB/s write speed to array from cache, after update to 6.7.0-rc2 about 10-16MB/s, switch back to 6.6.6 and again ~160MB/s with TurboWrite.

 

There's something wrong with encryption on 6.7.0-rc - CPU utilization is very low (2% on 6.7.0-rc and 12% on 6.6.6)

Share this post


Link to post
Just now, Raptor said:

Have encrypted XFS array, on 6.6.6 I get 160MB/s write speed to array from cache, after update to 6.7.0-rc2 about 10-16MB/s, switch back to 6.6.6 and again ~160MB/s with TurboWrite.

 

There's something wrong with encryption on 6.7.0-rc - CPU utilization is very low (2% on 6.7.0-rc and 12% on 6.6.6)

Yep. Made a separate bug report about it:

 

Share this post


Link to post
2 hours ago, Niklas said:

I thought I had similar problems on 6.6.6 but I could remember wrong. 

May be, I got some performance issue on 6.7RC ( test on 2 and 3 ), but no problem on 6.6.6

Edited by Benson

Share this post


Link to post

Well. It is a problem right now. Lets hope for a fast fix. :)

Share this post


Link to post

Could you simple try `cat file > /dev/null' in array disk, does read speed normal ?

My one top at ~83MB/s max, but I am sure not that slow.

Edited by Benson

Share this post


Link to post

My read speed was ok. Don't know if it was normal but not 10-15MB/s. I can't check now. My array is not encrypted until this problem is solved. :)

Edited by Niklas

Share this post


Link to post

In general SMB file transfer, I won't feel abnormal, but i.e. rsync, file hash generate check ... all slow down.

Edited by Benson

Share this post


Link to post
Just now, Benson said:

In generak SMB file transfe, I wount feel abnormal, but i.e. rsync, file hash generate check ... all slow down.

Yes. Probably true.

Share this post


Link to post

can u access ur shares in unraid? I mean in the UI? I always get

Feb 10 21:08:20 Unraid-Server nginx: 2019/02/10 21:08:20 [error] 6635#6635: *64273 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.86.42, server: , request: "GET /Shares/Share?name=nextcloud HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444", referrer: "https://7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444/Shares"

Share this post


Link to post

I guess time out can happen if Unraid is busy with writing files to the array. Transferring files using SMB made the Unraid UI and some dockers very slow. Browsing network shares was not possible until the content in ram was written to disk.

Edited by Niklas

Share this post


Link to post
6 minutes ago, nuhll said:

can u access ur shares in unraid? I mean in the UI? I always get

Feb 10 21:08:20 Unraid-Server nginx: 2019/02/10 21:08:20 [error] 6635#6635: *64273 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.86.42, server: , request: "GET /Shares/Share?name=nextcloud HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444", referrer: "https://7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444/Shares"

No such error. May be I could try get some file under UI. But not now.

Edited by Benson

Share this post


Link to post

Im currently doin parity and importing files via radarr, no smb transfer. And transfering files with 10mb/s via unbalanced.

 

But even if i stop all, i cant or bad reach unraid ui. or lets say specific categorys of the ui.

Share this post


Link to post

Unraid caches the transfer in ram (20%). When you stop the transfer, unraid is busy writing the content in ram to array. You can see the write going on for some time after stopping the file transfer. Until ram is empty i guess.

 

Well, to my understanding. ;)

Edited by Niklas

Share this post


Link to post

Like i said, even if i dont do anything with unraid, some parts of the ui are very unresponsive. I alos get nginx to low childs on restart.

 

Also i dont really think its writing anything to ram while transfering.

Edited by nuhll

Share this post


Link to post

Do you use cache? I do so my dockers and data is on ssd cache. Not much writing to array

Share this post


Link to post

yeah, ofc i have a 2 ssds cache. for vm dockers and incoming downloads.

Share this post


Link to post
On 2/7/2019 at 5:19 PM, johnnie.black said:

Yes, 20% free RAM by default, you can always use turbo write.

20% free ram used for cache. That's why I got full speed until ram cache was full.

Share this post


Link to post

Turbo write doesnt change anything (for me) i get it enabled automatc while doin mover, but mover also running for 4 hours or so by now... xD nzbget download is 8kbs atm :D

 

For me it shows array 35 - 50 mb/s (normal over 100)

 

and for cache around 300kbs - 8mb/s (dont know what they run normal, but i guess 200-300mbs?)

 

ram is under 50%.


What i also wonder is, unraid shows CPU 100% nearly all time. But if i ssh into it and run htop, cpu is chilling mostly below 30% oO?

1.png

Edited by nuhll

Share this post


Link to post

Hmm, do you have Unassigned Devices installed? And do you share SMB connections with Unassigned Devices?

 

Share this post


Link to post

I have it installed, but not using it. I used it for encrypting my array.

 

I remove it as we speak.

Share this post


Link to post

I did ask, because I see similar results here. Since some days I'm rebooting my main server several times a day. It started when I first applied 6.7.0-rc2 to a test server. I had lots of problems when copying large files on this test system. So I went back to 6.6.6 and now I do see this behaviour on all machines.

 

The other machines did not change so I think - it's just a guess - that a recently updated plugin is the reason.

 

* I do experience these nginx errors (too few childs).

* I do experience this "slow as f*ck when copying large files".

* I do experience memory eating up to 100% on a 64GB machine when copying large files.

* I do experience UI no longer usable when copying large files.

* I can read this "Please wait. Retrieving information..." from Unassigned Devices for hours.

* A clean shutdown is no longer possible.

* Several minutes ago I tried to copy a 47GB file with MC on my main server to a second server via SMB (/mnt/disks/UD01 to /mnt/disks/192.168.178.101_disk17). It stalled. Sending and receiving machine ended up with 100% memory usage. I had to reboot again. 

 

All this since some days weeks. And all this can be read in various threads here. Not in one thread. But if you browse thru the threads you will see some similarities. Something happened to long running 6.6.6 machines.

 

Edited by hawihoney

Share this post


Link to post

U can just restart nginx and then the ui is responsive again, but in web termianl is no longer working.. 

Was coming back from work and parity build is back to normal 150mb/s while yesterday i left with 30mb/s.

 

Now i startet importing files via radarr, and again upstream loss, parity down to 40 mb/s and lags everywhere.

 

The interesting part is that the OVERALL performance degraded from 150 mb/s down to 40mb/s. Jus tbecause im doin 2 thigns at the same time.

 

CPU and RAM is  still fine tho.

 

 

Edited by nuhll

Share this post


Link to post

If you are doing anything in parallel with a parity check/build then you will get very significant decrease in the parity check speed due to the disks continually having to move the read/write heads back and forth.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now