icedragonslair Posted May 5, 2014 Share Posted May 5, 2014 I have been having transfer speed issues, and I am trying to get to bottom of it so any help is greatly appreciated. 1. I have checked all cables and cards 2. I tested all transfer speeds via pulling the drive and popping them into my test server and using (HDTune Pro), all readings (File and drive benchmarks, Seq, 4kb random single/multi, access times, burst rates, etc) are all within acceptable parameters. However I found that my drives are aligned differently (see below), qlso it seems my parity is being written to in real time, how do I change this to manual parity checks and back ups (if this is affecting it as well). Also, wouldan SSD cache drive help these speeds by a lot and any recommendations for size would be great Any help is greatly appreciated as this has been driving me nuts for a while now. Ice Disc alignments parity - WD30EURS Partition 1 size: 2930266532 KB (K=1024) Partition format: GPT: 4K-aligned disk1 - WD20EURX Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned disc2 - WD20EURS Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned disc3 - WD20EURS Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: 4K-aligned disc4 - WD20EVDS Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned disc5 - WD20EURS Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned disc6 - WD20EURS Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned disc7 - WD20EURX Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned disc8 - WD20EURS Partition 1 size: 1953514552 KB (K=1024) Partition format: MBR: unaligned Link to comment
itimpi Posted May 5, 2014 Share Posted May 5, 2014 However I found that my drives are aligned differently (see below), qlso it seems my parity is being written to in real time, how do I change this to manual parity checks and back ups (if this is affecting it as well). Not sure I understand this comment about parity? If you do not have a cache disk then all writes involve writing to the parity disk as well. This is completely different to parity checks which you can schedule when they should be run. Link to comment
superloopy Posted May 5, 2014 Share Posted May 5, 2014 I'm not quite sure but i believe that drives > 2TB (parity in your case) are aligned differently to those < 2TB so your drive alignments look ok to me. There is a ton of info in the FAQ regarding this but no doubt someone will confirm in more detail ... Link to comment
icedragonslair Posted May 6, 2014 Author Share Posted May 6, 2014 Like I said all hardware has checked out but this is really getting to me since before February I had to replace on drive so I upgraded the server and all was well, then all of a sudden I started getting these outrageously slow file copy speeds. Here is the log syslog-2014-05-05.txt Link to comment
doorunrun Posted May 6, 2014 Share Posted May 6, 2014 Have you considered the client side of the equation? Are you experiencing transfer speed problems between unRAID and a Windows client, or are the transfers being done internally to the unRAID server? Here's some ideas from the unRAID wiki...... http://lime-technology.com/wiki/index.php/Improving_unRAID_Performance Link to comment
icedragonslair Posted May 6, 2014 Author Share Posted May 6, 2014 Well, I have tried or at least looked into most of what's in the wiki to no avail. As you can probably assume from my hardware in my signature that most have been adjusted as the wiki suggests. At this point no drives use the onboard sata ports and the parity is on its own pci-e 6gb/s x2 card and the otrher eight drives are on the supermicro card. The data transfer speeds are indeed from a windows system (win7 x64 pro) but I have swapped out three different systems and one was a mac and the speeds are the same. Once again this has only started since I replaced one drive that had an issue...but now that drive is replaced and it is actually a faster drive (3gb/s vs 6gb/s) I have not run a parity check since this stated since it will take 20k mins to complete it and that is ridiculous, I will rebuild my server and trash/recover my data. These speeds also coincide with my install of the new drive and the pro key upgrade (though I also cannot see that making a difference at all). Would it be worth it to reformat the drives so the alignment is the same, but I think that may be a stretch for what is going on here. As I had mentioned I am at a loss and at this point I am thinking of going back to motherboard controller and dumping the extra drives just to see if that makes a difference at all. At this time it is at least still, though without a parity check (3.1-5.3 mb/s vs 89-121 mb/s prior), I have no idea whether I am actually safe or not I am thinking that at this point in time a support ticket might be in order. Thanks Ice My net consists of cat 6 throughout with gigabit nics (system I am using to transfer has an intel I217V onboard) with a Netgear R6300 Link to comment
doorunrun Posted May 6, 2014 Share Posted May 6, 2014 It looks like one drive is configured for UDMA/66 or 1.5Gbps. It's ID'ed as ATA16 and I think it the WDC WD30EURS-63S which might be parity on the SYBA card. Quoted from your syslog: May 5 10:35:57 Tower kernel: ata16: SATA link up 1.5 Gbps (SStatus 113 SControl 300) May 5 10:35:57 Tower kernel: ata15: SATA link down (SStatus 0 SControl 300) May 5 10:35:57 Tower kernel: ata10: SATA link down (SStatus 0 SControl 300) May 5 10:35:57 Tower kernel: ata16.00: ATAPI: MARVELL VIRTUALL, , 1.09, max UDMA/66 May 5 10:35:57 Tower kernel: ata14: SATA link down (SStatus 0 SControl 300) May 5 10:35:57 Tower kernel: ata16.00: configured for UDMA/66 May 5 10:35:57 Tower kernel: ata5: SATA link down (SStatus 0 SControl 300) May 5 10:35:57 Tower kernel: ata6: SATA link down (SStatus 0 SControl 300) May 5 10:35:57 Tower kernel: scsi 10:0:0:0: Direct-Access ATA WDC WD30EURS-63S 80.0 PQ: 0 ANSI: 5 May 5 10:35:57 Tower kernel: sd 10:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB) I wouldn't think this would make your speeds abyssmal but it's all I can see at the minute. The point made about a cache is a good one, parity has to be written when the file is transferred into the array. Link to comment
trurl Posted May 6, 2014 Share Posted May 6, 2014 When you say you put the drives in a test server, do you mean another unRAID server, or something else? unRAID writes to parity-protected array involve reading the data drive, reading the parity drive, calculating the change in parity, writing the data drive, and writing the parity drive. In other words 2 reads and 2 writes, so it will normally be quite a bit slower than writing to just a single drive in another computer. What speeds are you getting exactly? Have you tried eliminating the network completely by copying from one drive to another directly on unRAID from command line or using mc? Link to comment
icedragonslair Posted May 6, 2014 Author Share Posted May 6, 2014 No I have a windows box I set up just to test the drives when this started happening, so I can just do physical tests one drive at a time (using HDTune Pro) I was aware that it would be slower but having transfer rates drop from 45-60mb/s to 3-7mb/s is ridiculous and you saw what the parity check rates were and are now. There seems to be just no way that something isn't wrong here, and nothing has changed with anything but the server, everything else is the same (even though I have tested it all any ways) trurl - When you say you put the drives in a test server, do you mean another unRAID server, or something else? unRAID writes to parity-protected array involve reading the data drive, reading the parity drive, calculating the change in parity, writing the data drive, and writing the parity drive. In other words 2 reads and 2 writes, so it will normally be quite a bit slower than writing to just a single drive in another computer. What speeds are you getting exactly? Have you tried eliminating the network completely by copying from one drive to another directly on unRAID from command line or using mc? I was not aware of this I will look into it or replace the card, however this started prior to that card being installed. doorunrun - It looks like one drive is configured for UDMA/66 or 1.5Gbps. It's ID'ed as ATA16 and I think it the WDC WD30EURS-63S which might be parity on the SYBA card. Quoted from your syslog: Link to comment
doorunrun Posted May 6, 2014 Share Posted May 6, 2014 What type of read speeds are you getting? It looks like you are running a stock unRAID system, no plugins, right? You may also want to consider testing just pure network performance. iPerf can be set up on unRAID. It's a little involved but you need to somehow isolate the various parts of the chain to nail it down. Also, EthTool is there already in unRAID and can give you stats on your connection. WireShark can be installed on your Windows system and you can analyze what goes on network-wise when you transfer. You can save the capture and share it here for others to analyze. One idea would be to set up a "new" unRAID server on your existing mobo/disk-controllers. Don't use any of your existing drives! It would take a single spare PreCleared drive (probably the smaller the better). Disconnect ALL your existing DATA/PARITY drives, use a "demo" version of unRAID, set up the system as a single drive, no parity and test your transfers. If this sounds too weird or doesn't make sense, don't try this! Take it slow and think it through! All of it involves some work. I haven't heard of anyone setting up a ticket (paid consulting) to get it fixed. But, it's certainly within the realm of possibilities. Link to comment
icedragonslair Posted May 7, 2014 Author Share Posted May 7, 2014 Nope, no plugins, I was getting read speeds in the neighborhood of 65-80mb/s but now its down to around 35. I will install one of the other plugins, test the network and post back Link to comment
icedragonslair Posted May 8, 2014 Author Share Posted May 8, 2014 Wow, I guess this is more important than I thought...It just took me 6 hours to transfer a large folder containing 123GB of HD mkv video files averaging 4-8gb each, last month that would have taken 20 minutes Well, this weekend I'll get to and see if I can figure out how to test the network Link to comment
icedragonslair Posted May 9, 2014 Author Share Posted May 9, 2014 First thing I did was wireshark, The cap can be downloaded http://icedragonslair.com/images/unraid/wiresharkcap.rar Will do the others this weekend Any help is greatly appreciated, Thanks, Ice Link to comment
dgaschk Posted May 9, 2014 Share Posted May 9, 2014 May 5 10:35:57 Tower logger: Installing system plugins May 5 10:35:57 Tower logger: installing plugin: webGui-latest May 5 10:35:57 Tower logger: May 5 10:35:57 Tower logger: Warning: simplexml_load_file(): /boot/plugins/webGui-latest.plg:1: parser error : Document is empty in /usr/local/sbin/installplg on line 13 May 5 10:35:57 Tower logger: May 5 10:35:57 Tower logger: Warning: simplexml_load_file(): in /usr/local/sbin/installplg on line 13 May 5 10:35:57 Tower logger: May 5 10:35:57 Tower logger: Warning: simplexml_load_file(): ^ in /usr/local/sbin/installplg on line 13 May 5 10:35:57 Tower logger: May 5 10:35:57 Tower logger: Warning: simplexml_load_file(): /boot/plugins/webGui-latest.plg:1: parser error : Start tag expected, '<' not found in /usr/local/sbin/installplg on line 13 May 5 10:35:57 Tower logger: May 5 10:35:57 Tower logger: Warning: simplexml_load_file(): in /usr/local/sbin/installplg on line 13 May 5 10:35:57 Tower logger: May 5 10:35:57 Tower logger: Warning: simplexml_load_file(): ^ in /usr/local/sbin/installplg on line 13 May 5 10:35:57 Tower logger: xml parse error May 5 10:35:57 Tower logger: Installing user plugins May 5 10:35:57 Tower logger: Starting go script May 5 10:35:57 Tower emhttp: unRAID System Management Utility version 5.0.5 webGui-latest is incompatible with version 5. Link to comment
icedragonslair Posted May 9, 2014 Author Share Posted May 9, 2014 webGui-latest is incompatible with version 5. Dgaschk, I am aware of this, but I have been asking how to uninstall it with no response. I never installed it and have never run anything but 5.x versions. So any help on this is greatly appreciated for at this point I am beginning to dump day and am considering a complete reboot and reload. We are talking a lot of data and losing a lot, so it is definitely a last resort...but I am at the end of my proverbial rope. Link to comment
doorunrun Posted May 9, 2014 Share Posted May 9, 2014 First thing I did was wireshark, The cap can be downloaded http://icedragonslair.com/images/unraid/wiresharkcap.rar I looked at the capture and I'm a novice when it comes to reading them. I will have to play around with it further. Thank for taking the time to capture the data! I don't think you have to dump all your data just to start over; and there should be a way to remove the enhanced webGUI. Dgascchk sig is usually a great help! Link to comment
DaleWilliams Posted May 9, 2014 Share Posted May 9, 2014 You can remove the advanced gui by installing Dynamix (which removes it automatically) Dynamix is the 'successor' to SimpleFeatures and includes much of the 'advanced GUI' functionality. Link to comment
doorunrun Posted May 9, 2014 Share Posted May 9, 2014 I've been looking over your wire capture and compared it against some local local testing between a Win8 system and unRAID. I'm wondering if you have enabled "jumbo frames?" What catches my attention is the write request for large length (65536 bytes?) followed by the tcp segment then another write request of similar size. Then we see some chatter about segments not captures. But also there's a lot of chatter back and forth as if there are errors, but I can't say that for sure. (See attached pix below) I wish I knew more, it just varies greatly from what I saw in my tests. Link to comment
icedragonslair Posted May 10, 2014 Author Share Posted May 10, 2014 I will check on the jumbo frames, i am moving all my data at this point into single layer shares, so when I fire up the backup server (thumb drive, motherboard/cpu/memory and two 4tb externals...lol) I can easily just transfer things off this one if need be 1. Jumbo frames are not enabled on the windows system (though capable it is disabled) 2. How do I check if they are on the unraid server (no settings via bios and I don't have any settings for the supermicro card) I also don't think my router (netgear 6300) and my un-managed switches are capable 3. At this point am I just better off upgrading to 6 beta? Link to comment
doorunrun Posted May 10, 2014 Share Posted May 10, 2014 Which model of SuperMicro controller card are you using? Why not try the on-board SATA connections? It looks like there's 7 of them and they all support SATA3. It seems like this would give you some flexibility in testing whether the SM or SYBA controller is giving you troubles. Remember, unRAID doesn't span a file across all the drives; a file is stored completely on one drive (that's one of the great selling points). So, when you transfer files it's just like copying disk to disk, computer to computer. Computing and writing parity on the unRAID side is really the only difference. The unRAID performance wiki says this about jumbo frames...... add the MTU property to /boot/config/network.cfg with the size, in bytes, of the payload. I don't think the alignment issue is giving you troubles. Adding a cache drive will help the apparent write speeds. I hope you can nail this down! Link to comment
icedragonslair Posted May 10, 2014 Author Share Posted May 10, 2014 1. SUPERMICRO AOC-SAS2LP-MV8 *** I have now switched both cards out (removed Syba and replaced SM with my other one) this does not appear to be the issue *** 2. There are indeed 7 that support sata3, but I was getting IRQ 16 errors, so I moved to the card. I am also moving all drives to sata6 I have 3 more in the wings to replace the others right now and the rest within 2 months. However, none of these items have changed in the last month (since the speeds sent to hell), except one backplane was bad and that has since been replaced, but I am willing to try anything at this point. 3. There is no entry for frame size (MTU) in the network.cfg file at all, I will look into this I am spending so much time on this it may have been easier just to scrap it and start over Link to comment
doorunrun Posted May 10, 2014 Share Posted May 10, 2014 There's a couple of posts on the SAS2LP-MV8 that might be worth considering, maybe you've already seen them. One said slow write speeds went away after going to V6; another suggested reflashing to current firmware even if already installed. Otherwise the SM card is a solid performer and works out of the box. One other post suggested limiting unRAID memory to 4095MB...but it's not been a universal cure. Do you think you're trying to copy to a disk that's pretty full? That can slow things down according to what I've read here. FWIW, there seems to be a consensus here that HDD's on 6Gb/s interfaces doesn't really help transfer speeds. Only SSD are capable of fully utilizing SATA3. I hope this is not too frustrating! Link to comment
icedragonslair Posted May 10, 2014 Author Share Posted May 10, 2014 Yeah, about that frustrating thing...lol... Nah, they are all less than half full. Thanks for the info on the mv8, I will give the new firmware a shot after this move. The only reason I am changing is due to the aging nature of the existing drives, nip things in the bud and so forth, as two of the originals have died and three of the five remaining 'WD20EURS' are all of the same batch, and the sata6's are all I can get in the WD-AV series now (sold on these drives). Thanks, when all is done I will re-post hopefully ewith a TFG or at least numbers and logs Ice Link to comment
dgaschk Posted May 10, 2014 Share Posted May 10, 2014 The TCP trace i not enough to be useful but looks normal; 65536 is the TCP window size and 1466 is the maximum segment size. Jumbo frames is not the issue. After the speed is no longer abysmal you can investigate jumbo frames for further optimization. However, unless the networking hardware is sub-par the maximum gain is about 3% in a LAN. 3% is generally not worth the additional configuration management. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.