Very Slow When Copying Multiple Files to SMB

[ncoolidg]

I do not understand why I cannot copy multiple files via SMB.

 

I tested one large file: 150MB/s

I tested 100 small files: 10 files/sec (about 1-3MB/s)

 

Can someone tell me why this is happening? I did see other posts on here about this issue but no clear answer yet... This is unusable like this.

 

I tried multiple systems too both running Windows and macOS so its not an OS problem.

Edited May 15, 2022 by ncoolidg

[MAM59]

Small files have large overheads. For each file it needs to update directory entries and recalc parity for them and so on. Usually the directories are far away from the data area, so the disks need stepping.

The large file only needs this once and the large data can be written continuously.

But 100 small files need to step back an forth a hundred times, wait until the correct block flys by, rewrite it and wait once more to do the same for parity (whilst the data disk writes the new block, the parity block has already passed and you need one more rotation to get it back into position).

This sums up until it becomes really annoying.

But there is nothing wrong and nothing you can do against it (but to replace all real hds with ssds, which is usually no option because of the size)

 

[ncoolidg]

I ended up figuring out the solution. I disabled user shares and enabled only disk shares. This increased the throughput to 20-50 items/sec. I was also using an SSD so it's just a problem with user shares in Unraid in general. Less overhead is needed with disk shares.

Edited May 15, 2022 by ncoolidg

[Frank1940]

7 hours ago, ncoolidg said:

so it's just a problem with user shares in Unraid in general. Less overhead is needed with disk shares.

 

This is true but many folks (including myself) are willing to tolerate the extra time for the convenience that User Shares provide.  In my case, most of my usage is 'write once, read many' so the extra time required is a one time situation.  I notice it mostly when I make the full backups of our Windows computers once a month.  But I generally just start the backups and walk away until they finish.

 

But everybody's use case is different.

 

One word of caution (of which you may be already aware), don't do Disk_Share-to-User_Share file operations.  This can result in data loss-- big time!

[Confused]

I'm experiencing this problem as well.  I'm not sure it's a user share problem, more of a problem with SMB itself.

 

I got frustrated with the low transfer speed and decided to just scp ssh the files from terminal directly into the user share and it went at max 120MB/s speed.  

 

1800 files, 20mb/file for a total of 23GB in size.  Avg speed 114.55MB/sec

Edited April 5 by Confused

[ncoolidg]

On 4/5/2024 at 10:36 AM, Confused said:

I'm experiencing this problem as well.  I'm not sure it's a user share problem, more of a problem with SMB itself.

 

I got frustrated with the low transfer speed and decided to just scp ssh the files from terminal directly into the user share and it went at max 120MB/s speed.  

 

1800 files, 20mb/file for a total of 23GB in size.  Avg speed 114.55MB/sec

I can't say for sure, but I will say once I switched to disk shares I get an average of 120MB/s with normal file transfers.