Pixxelcrasher Posted June 6, 2019 Share Posted June 6, 2019 (edited) Hello, I have upgraded my unraid server with a better network connection. Now I have 90 MB/s if I write something to the array... but only if I upload one large file! If I would upload a lot of little files like 100 4KB files the server goes down to 200 KB/s. That's not normal !!! what can I do? Here are some pieces of information... Please help Thanks tower-diagnostics-20190606-1933.zip Edited June 6, 2019 by Pixxelcrasher Quote Link to comment
Squid Posted June 6, 2019 Share Posted June 6, 2019 Filesystem Overhead. All systems have the same flaw where copying a ton of small files is significantly slower than large filesSent via telekinesis 1 1 Quote Link to comment
Pixxelcrasher Posted June 7, 2019 Author Share Posted June 7, 2019 So It doesnt give anything that I can do to fix that ? Quote Link to comment
glennv Posted June 7, 2019 Share Posted June 7, 2019 Unraid is even worse then the standard effect on any filesystem as in addition to normal filesystem overhead it has to do parity calculations/writes for each small file.I gave up struggling with trying regular incremental backups of huge audio sample libraries (milions) because it would take ages. Different bakcup solution react differently but all suffer. Regardless if you have fast ssd cache aparently. I do them now only once a month to unraid where the bakcup tool has a local database so does not have to recheck every file at every backup run. But still insanely slow always. And daily outside unraid. Normal large files , which is most of my data are rocketing over my 10g net at over 500-700MB/s without issuesIts the only downside of an otherwise stellar Unraid experience. 1 Quote Link to comment
testdasi Posted June 7, 2019 Share Posted June 7, 2019 In your 3rd screenshot, try changing Tuneable (md_write_method) to reconstruct write (used to be called turbo write). Writing a lot of small files will always be slower but the impact of parity will be more severe because it requires repeated seeking and switching between reading and writing. Reconstruct write helps at the cost of needing the rest of your array spun up. As long as the rest of your array is decent with random read, it generally will help. Also, remember your network adds latency and latency is a lot more important with small files. Having to wait 0.1s/file due to network latency is nothing for a 1GB file but for 1000 1MB files, it's 100s! 1 Quote Link to comment
Pixxelcrasher Posted June 7, 2019 Author Share Posted June 7, 2019 So exist any program which make it a little bit faster ? Or settings.. Please anythingThanks for the helpful answers!Gesendet von meinem SM-G950F mit Tapatalk Quote Link to comment
JonathanM Posted June 7, 2019 Share Posted June 7, 2019 1 hour ago, Pixxelcrasher said: So exist any program which make it a little bit faster ? Or settings.. Please anything Thanks for the helpful answers! Gesendet von meinem SM-G950F mit Tapatalk Create an archive file on the source end so you only need to transfer 1 large file. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.