Jump to content
Niklas

Transferring big files from Windows to Unraid, everything grinds to a halt

48 posts in this topic Last Reply

Recommended Posts

 

 

Hello,

 

I'm pulling my hair here. Using Unraid 6.7 RC2 but had this problem on 6.6.6 too (well, I think so.. bad memory). 

 

When transferring big files to Unraid (array) from Windows via SMB, the transfer grinds to a halt after what seems to be a set size. The speed is 110MB/s (how is this possible? Does it cache in ram or something before writing to the array?) until it dips down to like 2 MB/s or often 0MB/s. It stays like that for a couple of minutes and the network share goes unresponsive (Explorer not responding), same thing with the Unraid web ui (it gets very slow or time out). Suddenly it starts transferring again but grinds to a halt after a while again. When this happens I can see writes going on to one of the data disks and to the parity drive but the speed is like 10-20 MB/s. When it is done, the transfer picks up again. Not using cache for that share. If I cancel the transfer, I can see Unraid still writing to parity and data a while after and until it is done, the share will be unresponsive until done.

 

Sorry, much information but I don't know what to do next. This only happens during write operations. Reading will transfer at a steady 110MB/s.
Specs in signature.
This is recorded while Unraid in safe mode, writing to parity and data seems really slow?

 

 

Edited by Niklas

Share this post


Link to post
7 minutes ago, Niklas said:

Does it cache in ram or something before writing to the array?

Yes, 20% free RAM by default, you can always use turbo write.

Share this post


Link to post
9 minutes ago, johnnie.black said:

Yes, 20% free RAM by default, you can always use turbo write.

Ok, I see. Tried the plugin for turbo write, it always thinks 1 drive is spun down, when not. Set 1 drive allowed to be spun down. Trying now..

One big change I have done is going from the motherboards sata to crossflashed IBM M1015 (IT).

This is with turbo write enabled :/ You can see Unraid going almost unresponsive (see Unassigned Devices trying to load)

Edited by Niklas

Share this post


Link to post

Those ST4000DM004 disks are SMR, and form same family as the ST8000DM004, there are various users with those disks and very poor writes, that's likely your problem, if you have a different disk add it to the array and test writing to it with turbo write enable, parity is not SMR so it won't be a problem.

Share this post


Link to post
1 minute ago, johnnie.black said:

Those ST4000DM004 disks are SMR, and form same family as the ST8000DM004, there are various users with those disks and very poor writes, that's likely your problem, if you have a different disk add it to the array and test writing to it with turbo write enable, parity is not SMR so it won't be a problem.

Oh, that's shingled, right? Didn't know that. Guess I will replace them soon or use the cache drive more.

Share this post


Link to post
Just now, Niklas said:

Oh, that's shingled, right?

Yep, and while the older Seagate Archive shingled drives work well with Unraid, the newer Barracuda models appear to be crap, at least some of them.

Share this post


Link to post
10 minutes ago, johnnie.black said:

Yep, and while the older Seagate Archive shingled drives work well with Unraid, the newer Barracuda models appear to be crap, at least some of them.

Ok, thanks! Not good. Going to replace them all when my budget allows it. The ST4000VN008 (IronWolf) costs almost the same here in Sweden. 😤 I only use 2,67 TB in total so I just need one for now. ;)

Edited by Niklas

Share this post


Link to post

Something else seems to be wrong here.

Removed disk3 (it was empty), removed parity and set that up as the new disk3. Moving everything from disk1 and 2 using unBALANCE makes it go down on it's knees again. Starts off with 110 MB/s and then climbs down.

 

Now unBALANCE show 13 MB/s going down steady.

Writing to the IronWolf show about 12,8 MB/s in Main and very slow Unraid web gui. That's without parity, moving from the two Barracuda's to the IronWolf.
What's wrong here? About 13MB/s seem to be max.
I have used the DiskSpeed docker and did not notice any strange things with transfer speed or controller throughput.

Edit:
notgood.png.7c672d5cc8b3919264721032c816ce66.png
Ehm..

Edit: now over 200 avarage but not much cpu usage? Something related to encryption?

Edited by Niklas
added htop

Share this post


Link to post

Well. Reformatted the IronWolf as unencrypted xfs and writing to it resulted in speeds like 133MB/s.

So, it can't cope well with encrypted drives?

Seeing write speed up to 128,4 MB/s on unencrypted ST4000DM004 without parity.

Edited by Niklas

Share this post


Link to post

Your CPU should handle encryption just fine, only CPUs without AES could have some trouble.

Share this post


Link to post
Just now, johnnie.black said:

Your CPU should handle encryption just fine, only CPUs without AES could have some trouble.

So, that's strange then. No problems moving to unencrypted drives..

Share this post


Link to post
2 minutes ago, johnnie.black said:

Yes, Ironwolf is the same?

IronWolf is removed from the array for now.

This is a test share on disk3 (shingled one) only. Unencrypted. No parity. No problem.

Edited by Niklas

Share this post


Link to post
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       36 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               58
Model name:          Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
Stepping:            9
CPU MHz:             2172.815
CPU max MHz:         3800.0000
CPU min MHz:         1600.0000
BogoMIPS:            6835.49
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d
sort -u /proc/crypto | grep module
module       : aes_x86_64
module       : aesni_intel
module       : crc32_pclmul
module       : crc32c_intel
module       : crct10dif_pclmul
module       : cryptd
module       : ghash_clmulni_intel
module       : kernel
module       : pcbc

 

aes is there..

Edited by Niklas

Share this post


Link to post

Writing to SSD cache (encrypted btrfs) gives me full speed...

Edited by Niklas

Share this post


Link to post

How about non-encrypted btrfs with parity ?

Share this post


Link to post
14 minutes ago, Benson said:

How about non-encrypted btrfs with parity ?

Running non encrypted XFS without parity for now. Gives me full speed over my network but I really want to use encryption (peace of mind if server is stolen). Cache is still encrypted btrfs and that also gives full speed.

I removed parity (IronWolf) and added that one as unencrypted xfs data disk. Moved all files from the other data disks to that one. 


I am wiping one of the data disks (the "slow" ones) ST4000DM004 in Windows. 150-160 MB/s write speed. Enough to max out my 1Gbps LAN. With encrypted xfs in Unraid, speeds was about 12MB/s write speed.

Edited by Niklas

Share this post


Link to post

Maybe your diagnostics would reveal something. Post the complete zip file.

Share this post


Link to post
21 hours ago, John_M said:

Maybe your diagnostics would reveal something. Post the complete zip file.

I had it up before. Just a bit paranoid. ;)


I don't think that it would say mush. The two drives I use now is not encrypted anymore.

You could say that the problem is very slow write speed when using encrypted xfs (10-12MB/s). Writing to cache that is encrypted btrfs is fast and nice.

 

Edited by Niklas
Removed diag, ask me and I will provide it if needed

Share this post


Link to post

I wanted to see if there is anything with multiple errors that keeps resetting. The only thing I notice that might be a little suspicious is this, at the very end of your syslog, but I don't think it's related to your problem:

Feb  8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
Feb  8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 CDB: opcode=0x28 28 00 1d 1c 1c 18 00 00 08 00
Feb  8 15:29:15 Server kernel: print_req_error: I/O error, dev sdd, sector 488381464

 

Share this post


Link to post
20 hours ago, John_M said:

I wanted to see if there is anything with multiple errors that keeps resetting. The only thing I notice that might be a little suspicious is this, at the very end of your syslog, but I don't think it's related to your problem:


Feb  8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
Feb  8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 CDB: opcode=0x28 28 00 1d 1c 1c 18 00 00 08 00
Feb  8 15:29:15 Server kernel: print_req_error: I/O error, dev sdd, sector 488381464

 

 

Yes. Seen that. It's sdd and that one is unassigned, used as temporary storage for some dockers only. 

Edited by Niklas

Share this post


Link to post

I see people over at the forum for prerelease bugs seem to have the same or similar problems like I have.. Maybe 6.7 RC related? 

 

I thought I had similar problems on 6.6.6 but I could remember wrong. 

 

 

Edited by Niklas

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now