Copying within the array still weird


opentoe

Recommended Posts

Can someone try to do a copy with a large file from one user share to another and post the results up here? I did previously post about this maybe couple months ago but wanted a fresh look and maybe new eyes that can possibly help resolve this PIA issue. Copying a single 40 GB file takes about 15 minutes. Anyway I made another small vid of a Windows copy showing what happens and also a small vid of the basic resources page while the copy is happening. I just want to say when I copied on previous version of unraid it would keep a consistent speed. Version 5 was solid too. I remember doing A LOT of copying when going from BFRS to XFS. If I had the space to move all the content I would reformat the entire system. If I had the big bucks I would remove all the drives in the system and install all new drives for testing. Have to add I have replaced everything in the system except the processor. Mainboard, JBOD cards, memory, cables, formatted my USB boot drive. Also I always leave all drives spun up 24/7. Any suggestions are welcome. 

 

 

Link to comment

Looks like its caching and the pauses are due to it writing it out.

 

Few questions though.

 

Is unRaid and Windows both using SMB3?  (SMB3 is tons faster because the data doesn't have to traverse the network if source and destination are the same mount point.)  Similarily, are you copying from U: to U: ?

Link to comment
On 10/27/2019 at 11:13 PM, Squid said:

Looks like its caching and the pauses are due to it writing it out.

 

Few questions though.

 

Is unRaid and Windows both using SMB3?  (SMB3 is tons faster because the data doesn't have to traverse the network if source and destination are the same mount point.)  Similarily, are you copying from U: to U: ?

Hi, all my Windows machines are using SMBV2/V3. Used the powershell command to find out what version it is.

Get-SmbServerConfiguration | Select EnableSMB2Protocol

 

To eliminate all that stuff I consoled into the Unraid box and used MC to perform the copy. That also would come to a complete stop. Here is a video of that using the built in MC, which I do use a lot. I would usually see that blue bar move steadily at 55-60 MB/sec. Something made a change a while back in V6 that somehow slowed my copies big time. Here is a short video that shows how slow and bad it really is. Not being able to move files around easily hinders using Unraid big time. Very weird, right?

 

 

 

 

Link to comment

Pls post diagnostic file.

 

During SMB network transfer, it shoot over 110MB/s, does network in 10G or bonding ?

 

No matter in 6.7 or 6.8, disk tunable parameter need change, otherwise performance will reduce a lot. Direct I/O and turbo-write setting also affect too.

 

Pls try evaluates those setting.

Link to comment
17 hours ago, johnnie.black said:

There's a known issue with v6.7.2 where any array writes starve reads, this also affects moving data from one array device to another, it was fixed on v6.8-rc, you can try that.

It was happening before 6.7.2 but I will def try the beta for sure.

 

Link to comment
18 hours ago, Benson said:

Pls post diagnostic file.

 

During SMB network transfer, it shoot over 110MB/s, does network in 10G or bonding ?

 

No matter in 6.7 or 6.8, disk tunable parameter need change, otherwise performance will reduce a lot. Direct I/O and turbo-write setting also affect too.

 

Pls try evaluates those setting.

 

Here is the diag data. Don't even worry about the Windows video copy. Once we are into the console itself and the copy stays at 0% for 2 minutes we can eliminate the internal network. Not using SMB while doing a console copy, so the network topology should not matter but the network is 1Gb through out, all segments tested while the network was being installed.

Here are my tunable parameters:

Tunable NR REQUESTS: 1024

Tunable MD_NUM STRIPES: 4096

Tunable MD_SYNC WINDOW: 2048

Tunable MD_SYNC THRESH: 2000

Tunable MD_WRITE_METHOD: RECONSTRUCT WRITE

 

I did run a utility that ran for quite a bit that gave me the tunable recommended numbers. I'll see if I can find that utility again and run the test again. I'll see if that utility works with the latest version.

 

I also wanted to add. I have a PRO LICENSE key that I won during our get together one night for Unraid. I have never used this key 'yet' since I already have another small box running with all SSD's just for a couple small things. Whoever finds out what the issue is the key is yours! I'll get it transferred to you. Maybe will motivate. I'm just burned out trying to figure out what it could be and really been slammed with year end projects at work and rarely have the time to tinker.

 

BTW, happy Holllla Ween!

 

 

 

sun-diagnostics-20191101-0208.zip

Edited by opentoe
Change
Link to comment

Ok, this is odd. The only thing I have done was upgrade to the beta 6.8.0rc4. So for the hell of it I tried a Windows copy after the server settled down and idled for a good 10-15 minutes. This is the first time I've seen a normal copy from within the array itself in about a year. There was no camels back with all those troughs and spikes. This was a consistent copy that I am used to seeing. So no one thinks I'm crazy I cut a short video of the end of the copy. You'll see a night and day difference between this copy video and the one in the original post. The difference is very apparent. Could this have been the issue jonnie.black was indicating in his post? I sure as hell hope so. Don't want to get all excited here so I'll do some more testing when time permits maybe over this weekend. I also did a console ( MC ) copy and it didn't stall out at all. Just a 'little' happy here. Haven't seen that type of copy in a very long time. Anyway, check out the first video in the original post and then this one. Wow.

 

 

Link to comment

6.8 have major change in FUSE layer ( assume test on array, not cache pool or UD ), the behavior also different. BTW, tuning on the new set of tunable could improve an make things more smooth.

77.4MB/s should be normal performance.

 

I quick check on the diagnostic, in general no abnormal found.

 

41 minutes ago, opentoe said:

but the network is 1Gb

Strange, I never foud an overshoot > 113 MB/s when doing SMB transfer on 1Gbps ethernet.

 

Below are some practical transfer in Unraid view, most time in saturate but some deeps not avoidable.

 

1.png.a9cf8e81fb1ab0fd137fdbb11d3b03c9.png

 

Edited by Benson
Link to comment

Well can't always rely on what speeds Windows shows. I also did a GCP copy via console and it worked great. I also checked the tunable settings and it looks like they were reset to the defaults which I'm just leaving alone and never touching again. I did see that FUSE layer setting.

 

Experimental: If set to Yes then mount User Share file system with FUSE direct_io mount option. This will increase write performance but might possibly decrease read performance.

Auto selects No

 

Mine is set to AUTO

 

stripes = 4096

queue limit = 80

sync limit = 5

write method = auto

 

Link to comment
15 minutes ago, Squid said:

It will if the source and destination are on the server.  In that case, the data never actually traverses the network with SMB3

Yes, if it is the case. From previous video, I notice a rather long slow/pause period during network transfer, so I assume actual file in transfer there.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.