Uploadspeed for small files is very slow !

[Pixxelcrasher]

Hello,

I have upgraded my unraid server with a better network connection. Now I have 90 MB/s if I write something to the array... but only if I upload one large file! If I would upload a lot of little files like 100 4KB files the server goes down to 200 KB/s. That's not normal !!!

what can I do?

 

Here are some pieces of information...

Please help

Thanks

 

tower-diagnostics-20190606-1933.zip

Edited June 6, 2019 by Pixxelcrasher

[Squid]

Filesystem Overhead. All systems have the same flaw where copying a ton of small files is significantly slower than large files

Sent via telekinesis

[Pixxelcrasher]

So It doesnt give anything that I can do to fix that ?

[glennv]

Unraid is even worse then the standard effect on any filesystem as in addition to normal filesystem overhead it has to do parity calculations/writes for each small file.
I gave up struggling with trying regular incremental backups of huge audio sample libraries (milions) because it would take ages. Different bakcup solution react differently but all suffer. Regardless if you have fast ssd cache aparently.
I do them now only once a month to unraid where the bakcup tool has a local database so does not have to recheck every file at every backup run. But still insanely slow always. And daily outside unraid. Normal large files , which is most of my data are rocketing over my 10g net at over 500-700MB/s without issues
Its the only downside of an otherwise stellar Unraid experience.

[testdasi]

In your 3rd screenshot, try changing Tuneable (md_write_method) to reconstruct write (used to be called turbo write). 

 

Writing a lot of small files will always be slower but the impact of parity will be more severe because it requires repeated seeking and switching between reading and writing. Reconstruct write helps at the cost of needing the rest of your array spun up. As long as the rest of your array is decent with random read, it generally will help.

 

Also, remember your network adds latency and latency is a lot more important with small files. Having to wait 0.1s/file due to network latency is nothing for a 1GB file but for 1000 1MB files, it's 100s!

[Pixxelcrasher]

So exist any program which make it a little bit faster ? Or settings.. Please anything


Thanks for the helpful answers!

Gesendet von meinem SM-G950F mit Tapatalk

[JonathanM]

1 hour ago, Pixxelcrasher said:

So exist any program which make it a little bit faster ? Or settings.. Please anything


Thanks for the helpful answers!

Gesendet von meinem SM-G950F mit Tapatalk
 

Create an archive file on the source end so you only need to transfer 1 large file.