Transferring big files from Windows to Unraid, everything grinds to a halt


Niklas

Recommended Posts

Have encrypted XFS array, on 6.6.6 I get 160MB/s write speed to array from cache, after update to 6.7.0-rc2 about 10-16MB/s, switch back to 6.6.6 and again ~160MB/s with TurboWrite.

 

There's something wrong with encryption on 6.7.0-rc - CPU utilization is very low (2% on 6.7.0-rc and 12% on 6.6.6)

Link to comment
Just now, Raptor said:

Have encrypted XFS array, on 6.6.6 I get 160MB/s write speed to array from cache, after update to 6.7.0-rc2 about 10-16MB/s, switch back to 6.6.6 and again ~160MB/s with TurboWrite.

 

There's something wrong with encryption on 6.7.0-rc - CPU utilization is very low (2% on 6.7.0-rc and 12% on 6.6.6)

Yep. Made a separate bug report about it:

 

Link to comment

can u access ur shares in unraid? I mean in the UI? I always get

Feb 10 21:08:20 Unraid-Server nginx: 2019/02/10 21:08:20 [error] 6635#6635: *64273 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.86.42, server: , request: "GET /Shares/Share?name=nextcloud HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444", referrer: "https://7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444/Shares"

Link to comment
6 minutes ago, nuhll said:

can u access ur shares in unraid? I mean in the UI? I always get

Feb 10 21:08:20 Unraid-Server nginx: 2019/02/10 21:08:20 [error] 6635#6635: *64273 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.86.42, server: , request: "GET /Shares/Share?name=nextcloud HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444", referrer: "https://7e8791b3aa0bbd8fe33ae9faa95413a1af5ece0a.unraid.net:444/Shares"

No such error. May be I could try get some file under UI. But not now.

Edited by Benson
Link to comment

Unraid caches the transfer in ram (20%). When you stop the transfer, unraid is busy writing the content in ram to array. You can see the write going on for some time after stopping the file transfer. Until ram is empty i guess.

 

Well, to my understanding. ;)

Edited by Niklas
Link to comment

Turbo write doesnt change anything (for me) i get it enabled automatc while doin mover, but mover also running for 4 hours or so by now... xD nzbget download is 8kbs atm :D

 

For me it shows array 35 - 50 mb/s (normal over 100)

 

and for cache around 300kbs - 8mb/s (dont know what they run normal, but i guess 200-300mbs?)

 

ram is under 50%.


What i also wonder is, unraid shows CPU 100% nearly all time. But if i ssh into it and run htop, cpu is chilling mostly below 30% oO?

1.png

Edited by nuhll
Link to comment

I did ask, because I see similar results here. Since some days I'm rebooting my main server several times a day. It started when I first applied 6.7.0-rc2 to a test server. I had lots of problems when copying large files on this test system. So I went back to 6.6.6 and now I do see this behaviour on all machines.

 

The other machines did not change so I think - it's just a guess - that a recently updated plugin is the reason.

 

* I do experience these nginx errors (too few childs).

* I do experience this "slow as f*ck when copying large files".

* I do experience memory eating up to 100% on a 64GB machine when copying large files.

* I do experience UI no longer usable when copying large files.

* I can read this "Please wait. Retrieving information..." from Unassigned Devices for hours.

* A clean shutdown is no longer possible.

* Several minutes ago I tried to copy a 47GB file with MC on my main server to a second server via SMB (/mnt/disks/UD01 to /mnt/disks/192.168.178.101_disk17). It stalled. Sending and receiving machine ended up with 100% memory usage. I had to reboot again. 

 

All this since some days weeks. And all this can be read in various threads here. Not in one thread. But if you browse thru the threads you will see some similarities. Something happened to long running 6.6.6 machines.

 

Edited by hawihoney
Link to comment

U can just restart nginx and then the ui is responsive again, but in web termianl is no longer working.. 

Was coming back from work and parity build is back to normal 150mb/s while yesterday i left with 30mb/s.

 

Now i startet importing files via radarr, and again upstream loss, parity down to 40 mb/s and lags everywhere.

 

The interesting part is that the OVERALL performance degraded from 150 mb/s down to 40mb/s. Jus tbecause im doin 2 thigns at the same time.

 

CPU and RAM is  still fine tho.

 

 

Edited by nuhll
Link to comment
  • 10 months later...
On 2/7/2019 at 4:11 PM, Niklas said:

 

 

Hello,

 

I'm pulling my hair here. Using Unraid 6.7 RC2 but had this problem on 6.6.6 too (well, I think so.. bad memory). 

 

When transferring big files to Unraid (array) from Windows via SMB, the transfer grinds to a halt after what seems to be a set size. The speed is 110MB/s (how is this possible? Does it cache in ram or something before writing to the array?) until it dips down to like 2 MB/s or often 0MB/s. It stays like that for a couple of minutes and the network share goes unresponsive (Explorer not responding), same thing with the Unraid web ui (it gets very slow or time out). Suddenly it starts transferring again but grinds to a halt after a while again. When this happens I can see writes going on to one of the data disks and to the parity drive but the speed is like 10-20 MB/s. When it is done, the transfer picks up again. Not using cache for that share. If I cancel the transfer, I can see Unraid still writing to parity and data a while after and until it is done, the share will be unresponsive until done.

 

Sorry, much information but I don't know what to do next. This only happens during write operations. Reading will transfer at a steady 110MB/s.
Specs in signature.
This is recorded while Unraid in safe mode, writing to parity and data seems really slow?

 

 

 

Hello, 

 

Did you manage to solve this issue? I am currently seeing the same problem while trying to move around 9TB onto a newly installed unraid server. I am considering moving from freenas to unraid so installed unraid 6.8 onto my test HP N40L server containing 2x6TB WD reds ( 1 used for parity) and 2x3TB WD greens with no cache drive. The data has been copying for over two days but still far from done. I also see the burst of fast copy (~100MB/s) copying roughly 2GB - 2.5GB before stalling for roughly a minute. I also see the same slow read/write from the drives as shown in your video during this minute of network inactivity which makes me think that it is copying a small portion to RAM before calculating parity and writing to disk. Is this the case?

 

This same machine and drives did not show this behaviour in freenas which also contained the same discs in an array with one used for parity. I have enabled turbo write mode as some suggest but this did not solve this issue. Any suggestions?

 

Thanks for any help or advice.

 

Edited by Jeff123
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.