Niklas Posted February 7, 2019 Share Posted February 7, 2019 (edited) Hello, I'm pulling my hair here. Using Unraid 6.7 RC2 but had this problem on 6.6.6 too (well, I think so.. bad memory). When transferring big files to Unraid (array) from Windows via SMB, the transfer grinds to a halt after what seems to be a set size. The speed is 110MB/s (how is this possible? Does it cache in ram or something before writing to the array?) until it dips down to like 2 MB/s or often 0MB/s. It stays like that for a couple of minutes and the network share goes unresponsive (Explorer not responding), same thing with the Unraid web ui (it gets very slow or time out). Suddenly it starts transferring again but grinds to a halt after a while again. When this happens I can see writes going on to one of the data disks and to the parity drive but the speed is like 10-20 MB/s. When it is done, the transfer picks up again. Not using cache for that share. If I cancel the transfer, I can see Unraid still writing to parity and data a while after and until it is done, the share will be unresponsive until done. Sorry, much information but I don't know what to do next. This only happens during write operations. Reading will transfer at a steady 110MB/s. Specs in signature. This is recorded while Unraid in safe mode, writing to parity and data seems really slow? capture_2019-02-07_16-50-54_b88724eb.mp4 Edited February 10, 2019 by Niklas Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 7 minutes ago, Niklas said: Does it cache in ram or something before writing to the array? Yes, 20% free RAM by default, you can always use turbo write. Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) 9 minutes ago, johnnie.black said: Yes, 20% free RAM by default, you can always use turbo write. Ok, I see. Tried the plugin for turbo write, it always thinks 1 drive is spun down, when not. Set 1 drive allowed to be spun down. Trying now.. One big change I have done is going from the motherboards sata to crossflashed IBM M1015 (IT). This is with turbo write enabled You can see Unraid going almost unresponsive (see Unassigned Devices trying to load) capture_2019-02-07_17-25-27_e65e2339.mp4 Edited February 7, 2019 by Niklas Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 Those ST4000DM004 disks are SMR, and form same family as the ST8000DM004, there are various users with those disks and very poor writes, that's likely your problem, if you have a different disk add it to the array and test writing to it with turbo write enable, parity is not SMR so it won't be a problem. Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 1 minute ago, johnnie.black said: Those ST4000DM004 disks are SMR, and form same family as the ST8000DM004, there are various users with those disks and very poor writes, that's likely your problem, if you have a different disk add it to the array and test writing to it with turbo write enable, parity is not SMR so it won't be a problem. Oh, that's shingled, right? Didn't know that. Guess I will replace them soon or use the cache drive more. Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 Just now, Niklas said: Oh, that's shingled, right? Yep, and while the older Seagate Archive shingled drives work well with Unraid, the newer Barracuda models appear to be crap, at least some of them. Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) 10 minutes ago, johnnie.black said: Yep, and while the older Seagate Archive shingled drives work well with Unraid, the newer Barracuda models appear to be crap, at least some of them. Ok, thanks! Not good. Going to replace them all when my budget allows it. The ST4000VN008 (IronWolf) costs almost the same here in Sweden. 😤 I only use 2,67 TB in total so I just need one for now. Edited February 7, 2019 by Niklas Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) Something else seems to be wrong here. Removed disk3 (it was empty), removed parity and set that up as the new disk3. Moving everything from disk1 and 2 using unBALANCE makes it go down on it's knees again. Starts off with 110 MB/s and then climbs down. Now unBALANCE show 13 MB/s going down steady. Writing to the IronWolf show about 12,8 MB/s in Main and very slow Unraid web gui. That's without parity, moving from the two Barracuda's to the IronWolf. What's wrong here? About 13MB/s seem to be max. I have used the DiskSpeed docker and did not notice any strange things with transfer speed or controller throughput. Edit: Ehm.. Edit: now over 200 avarage but not much cpu usage? Something related to encryption? Edited February 7, 2019 by Niklas added htop Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) Well. Reformatted the IronWolf as unencrypted xfs and writing to it resulted in speeds like 133MB/s. So, it can't cope well with encrypted drives? Seeing write speed up to 128,4 MB/s on unencrypted ST4000DM004 without parity. Edited February 7, 2019 by Niklas Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 Your CPU should handle encryption just fine, only CPUs without AES could have some trouble. Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 Just now, johnnie.black said: Your CPU should handle encryption just fine, only CPUs without AES could have some trouble. So, that's strange then. No problems moving to unencrypted drives.. Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 2 minutes ago, Niklas said: So, that's strange then. Yes, Ironwolf is the same? Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 Maybe AES is disable in the bios. Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) 2 minutes ago, johnnie.black said: Yes, Ironwolf is the same? IronWolf is removed from the array for now. This is a test share on disk3 (shingled one) only. Unencrypted. No parity. No problem. capture_2019-02-07_20-45-48_4d74a17c.mp4 Edited February 7, 2019 by Niklas Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 36 bits physical, 48 bits virtual CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 58 Model name: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz Stepping: 9 CPU MHz: 2172.815 CPU max MHz: 3800.0000 CPU min MHz: 1600.0000 BogoMIPS: 6835.49 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 6144K NUMA node0 CPU(s): 0-3 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d sort -u /proc/crypto | grep module module : aes_x86_64 module : aesni_intel module : crc32_pclmul module : crc32c_intel module : crct10dif_pclmul module : cryptd module : ghash_clmulni_intel module : kernel module : pcbc aes is there.. Edited February 7, 2019 by Niklas Quote Link to comment
JorgeB Posted February 7, 2019 Share Posted February 7, 2019 Strange then, no more ideas. Quote Link to comment
Niklas Posted February 7, 2019 Author Share Posted February 7, 2019 (edited) Writing to SSD cache (encrypted btrfs) gives me full speed... Edited February 7, 2019 by Niklas Quote Link to comment
Niklas Posted February 8, 2019 Author Share Posted February 8, 2019 Does @limetech have any idea? Quote Link to comment
Vr2Io Posted February 8, 2019 Share Posted February 8, 2019 How about non-encrypted btrfs with parity ? Quote Link to comment
Niklas Posted February 8, 2019 Author Share Posted February 8, 2019 (edited) 14 minutes ago, Benson said: How about non-encrypted btrfs with parity ? Running non encrypted XFS without parity for now. Gives me full speed over my network but I really want to use encryption (peace of mind if server is stolen). Cache is still encrypted btrfs and that also gives full speed. I removed parity (IronWolf) and added that one as unencrypted xfs data disk. Moved all files from the other data disks to that one. I am wiping one of the data disks (the "slow" ones) ST4000DM004 in Windows. 150-160 MB/s write speed. Enough to max out my 1Gbps LAN. With encrypted xfs in Unraid, speeds was about 12MB/s write speed. Edited February 8, 2019 by Niklas Quote Link to comment
John_M Posted February 8, 2019 Share Posted February 8, 2019 Maybe your diagnostics would reveal something. Post the complete zip file. Quote Link to comment
Niklas Posted February 8, 2019 Author Share Posted February 8, 2019 (edited) On 2/8/2019 at 5:28 PM, John_M said: Maybe your diagnostics would reveal something. Post the complete zip file. I had it up before. Just a bit paranoid. I don't think that it would say much. The two drives I use now is not encrypted anymore. You could say that the problem is very slow write speed when using encrypted xfs (10-12MB/s). Writing to cache that is encrypted btrfs is fast and nice. Edited January 1, 2020 by Niklas Removed diag, ask me and I will provide it if needed Quote Link to comment
John_M Posted February 8, 2019 Share Posted February 8, 2019 I wanted to see if there is anything with multiple errors that keeps resetting. The only thing I notice that might be a little suspicious is this, at the very end of your syslog, but I don't think it's related to your problem: Feb 8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00 Feb 8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 CDB: opcode=0x28 28 00 1d 1c 1c 18 00 00 08 00 Feb 8 15:29:15 Server kernel: print_req_error: I/O error, dev sdd, sector 488381464 Quote Link to comment
Niklas Posted February 8, 2019 Author Share Posted February 8, 2019 (edited) 20 hours ago, John_M said: I wanted to see if there is anything with multiple errors that keeps resetting. The only thing I notice that might be a little suspicious is this, at the very end of your syslog, but I don't think it's related to your problem: Feb 8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00 Feb 8 15:29:15 Server kernel: sd 7:0:0:0: [sdd] tag#0 CDB: opcode=0x28 28 00 1d 1c 1c 18 00 00 08 00 Feb 8 15:29:15 Server kernel: print_req_error: I/O error, dev sdd, sector 488381464 Yes. Seen that. It's sdd and that one is unassigned, used as temporary storage for some dockers only. Edited February 9, 2019 by Niklas Quote Link to comment
Niklas Posted February 10, 2019 Author Share Posted February 10, 2019 (edited) I see people over at the forum for prerelease bugs seem to have the same or similar problems like I have.. Maybe 6.7 RC related? I thought I had similar problems on 6.6.6 but I could remember wrong. Edited February 10, 2019 by Niklas Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.