pkn Posted April 4, 2015 Share Posted April 4, 2015 Noticed something I don't understand. On the same system, same conditions, parity sync (writing to parity, reading others) and data disk rebuild (writing to data disk, reading others) both go with average speed ~140 MB/sec (start at ~195 MB/s, end at ~90 MB/s). Parity check (reading only) goes with average speed of ~55 MB/s (varies during check randomly between ~45 and ~70 MB/s). Is it the way it should be in unRAID Trial 6.0-beta14b? Setup: Supermicro H8DME-2, Opteron 2346 HE, 4GB RAM, unRAID Trial 6.0-beta14b. Parity: 4+4TB=8TB RAID-0 pool via Areca ARC-1110 in PCI-X slot Disk1: 8TB Seagate Archive in motherboard SATA port Disk2: 8TB Seagate Archive in motherboard SATA port Quote Link to comment
garycase Posted April 4, 2015 Share Posted April 4, 2015 Definitely not ==> I'd certainly expect parity checks to run with a similar speed ... with the same downward ramp as the check moves from the outer cylinders to the inner cylinders. This likely has something to do with the use of the RAID array for parity, but I can't think of a reason why it would be so much slower, since you're using the same configuration for syncs and rebuilds. Do you have a 3rd Seagate 8TB unit? If so ... and if you're inclined to experiment with it, it'd be interesting to see if you get similar behavior if you replace your parity array with a 3rd Archive drive. One other thought => Are you starting this check immediately after doing one of the other activities (sync or rebuild) ?? It's possible that your Seagate drives are "catching up" with band rewrites to empty their persistent cache areas ... this would slow them down a good bit. Quote Link to comment
pkn Posted April 5, 2015 Author Share Posted April 5, 2015 This likely has something to do with the use of the RAID array for parity, but I can't think of a reason why it would be so much slower, since you're using the same configuration for syncs and rebuilds. Yeah... I can't imagine same RAID pool reads fast for data rebuild and reads slow for parity check. Do you have a 3rd Seagate 8TB unit? If so ... and if you're inclined to experiment with it, it'd be interesting to see if you get similar behavior if you replace your parity array with a 3rd Archive drive. Will do. One other thought => Are you starting this check immediately after doing one of the other activities (sync or rebuild) ?? It's possible that your Seagate drives are "catching up" with band rewrites to empty their persistent cache areas ... this would slow them down a good bit. No, at one occasion there was 24+ hours since last activity. Quote Link to comment
pkn Posted April 5, 2015 Author Share Posted April 5, 2015 I'm looking at "top"... mdrecoveryd which, I suppose, is parity check thread, has priority 20 and often waits for disk I/O (status=D). Bu there are three processes called migration/1, migration/2 and migration/3, all with priority RT, and migration/3 constantly taking from 50% to 300% of CPU and having status=R. What are those? Googled it out. "The migration kernel process distributes workload across CPU cores." Hmmm.... kernel issue? But then why it does not affect parity sync and data rebuild? Quote Link to comment
pkn Posted April 6, 2015 Author Share Posted April 6, 2015 Okay... I mean, not OK. Next tests: server "Testt", AMD-based. Downgrading unRAID from 6.0-beta14b to 6.0-beta2 did not change anything. Configuration 8TB Archive parity and two 8TB Archives for data disks showed even worse performance for parity check - around 30 MB/s. Process mdrecoveryd again waits for migration/3 a lot. Parity sync performance is as with all previous tests - ~140 MB/s. My other server (Intel-based, not AMD) "Tower" is overpopulated, underpowered, runs unRAID-5.0, and generally is far form optimum. However, data disk rebuild it does with average speed ~66 MB/s, and parity check (check, not sync) with ~64 MB/sec. During parity check processes unraidd and mdrecoveryd take ~35% CPU each, migration/0 and migration/1 both take 0%. I'm starting to suspect unRAID kernel problem with this particular AMD-based motherboard/chipset/CPU configuration. Thinking about other tests - unRAID 5.0 on AMD-based platform, and how to test on Intel-based platform without risking my real data on it... Quote Link to comment
pkn Posted April 6, 2015 Author Share Posted April 6, 2015 Next test: server "Testt", AMD-based. Downgraded unRAID to 5.0 - no changes. Process migration/3 eats all CPU when parity check, parity sync flies no problem. After consideration I decided that I'm not idiot enough desperate enough brave enough to do this testing on my real data Intel-based server, so I'll wait until another X7SBE (Intel-based) motherboard arrives, should be few days. Quote Link to comment
pkn Posted April 7, 2015 Author Share Posted April 7, 2015 (after googling) Looks like a kernel bug. https://bugzilla.openvz.org/show_bug.cgi?id=1954 https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/1223055 Which was long fixed... or not. I can not do # echo 0 > /proc/sys/kernel/sched_cpulimit_nr_balance - no such file or directory. Quote Link to comment
pkn Posted April 10, 2015 Author Share Posted April 10, 2015 Next: tested it on X7SBE (Intel-based) motherboard - no "migration process eating CPU" problem. Parity check speed is ~120 MB/s at start, versus ~50 MB/s at start on AMD-based H8DME-2 motherboard - all else being equal. This is still considerably slower than parity sync (~195 MB/s at start) but that I understand - in parity check there are two processes, unraidd and mdrecoveryd competing for CPU cycles. In parity sync there is only mdrecoveryd. Well, looks like my long living unlove to AMD just got another reason to thrive. On the other hand, if this kernel bug is known since 2011 why is it still here... Quote Link to comment
garycase Posted April 10, 2015 Share Posted April 10, 2015 Very interesting ... but at least it's trivial to avoid => just stay with Intel ... since all of my systems are (and always have been) Intel based ever since the original Altair 8080 I built in 1975 I've never had this problem Quote Link to comment
pkn Posted April 10, 2015 Author Share Posted April 10, 2015 Very interesting ... but at least it's trivial to avoid => just stay with Intel Yeah... but... I could probably live with ridiculously slow parity checks, but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question. So I'll need a new Intel-based motherboard - can't use X7SBE since it only supports 8GB of RAM. Which also means new expansion cards for PCI-e since almost all of mine are PCI-X. And I was hoping I'm done with hardware purchases for at least a year... another upgrade ...upgrade? ...upgrade! Quote Link to comment
garycase Posted April 10, 2015 Share Posted April 10, 2015 ... can't use X7SBE since it only supports 8GB of RAM ... Have you actually confirmed that you need more than 8GB of RAM for what you want to do? I'd try it before upgrading. One other thought: SuperMicro boards often work fine with larger RAM modules than they were originally specified for. I do NOT know if that's true for an X7SBE, but I know for a fact, for example, that an X7SPA-HF-D525 board works perfectly with a pair of 4GB modules (8GB total) even though the specifications say the board only supports 4GB of RAM [One of my servers has been happily running on this board with 8GB of memory for over 2 years]. Only way to know for sure is to try it -- if you happen to have access to a 4GB DDR2 ECC modules you could try it (or you could buy a pair as long as they're returnable ... worst case is it'd cost you a restocking fee). ... If you happen to try that, be sure and post the results -- I'm sure a lot of folks who are using this board would be interested in knowing that. Note for X7SBE Users: If you're already tested this and know the answer, it'd be nice if you'd post that here Quote Link to comment
tr0910 Posted April 10, 2015 Share Posted April 10, 2015 but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question. What is about Docker and AMD that you don't like?? Quote Link to comment
pkn Posted April 11, 2015 Author Share Posted April 11, 2015 ... can't use X7SBE since it only supports 8GB of RAM ... Have you actually confirmed that you need more than 8GB of RAM for what you want to do? I'd try it before upgrading. No. I'm just reading the latest Dockers threads and it's more like "Wow, I want this Docker, and that one too, no way they both (all) will fit in 8GB memory". Thank you for this question, really. It triggered my serious reconsideration of what I really need - versus what I'd like to play with. And you know... first result of this reconsideration is that I realized that I've just recently bought 'bout 32 TB of storage volume which I enjoy playing with, but not really needed... not sure which smilie put here... this or that or maybe this One other thought: SuperMicro boards often work fine with larger RAM modules than they were originally specified for. ... Oh my... you have just sent me to another round of hardware purchases and testing! Quote Link to comment
pkn Posted April 11, 2015 Author Share Posted April 11, 2015 but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question. What is about Docker and AMD that you don't like?? I've just found that on my H8DME-2 (AMD-based) motherboard parity check (not sync) is going ridiculously slow. All thing equal motherboard X7SBE (Intel-based) provides 2-2.5 faster parity check. I blame this on kernel-level inability to properly (re)distribute load between the CPUs/CPU cores. This is my personal opinion and it does not reflect superiority or inferiority or unrelatediority or whateverriority of the aforementioned fully respected manufacturers. It's just that if the current kernel can't, on H8DME-2 (AMD-based) motherboard, balance load between the CPUs/CPU cores during simple task of parity check, than I don't even want tor try if it can do it with multiple Dockers running. Quote Link to comment
tr0910 Posted April 11, 2015 Share Posted April 11, 2015 but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question. What is about Docker and AMD that you don't like?? I've just found that on my H8DME-2 (AMD-based) motherboard parity check (not sync) is going ridiculously slow. All thing equal motherboard X7SBE (Intel-based) provides 2-2.5 faster parity check. I blame this on kernel-level inability to properly (re)distribute load between the CPUs/CPU cores. I've just started a non-correcting parity sync on my AMD server running a few dockers and it is running at between 110 and 120mb/sec. This is a bit slower than the initial parity generation, but not the kind of numbers you are referring to. Shutting off all the dockers boosts it about 10 mb/sec., but depending on how hard the dockers are working, the parity will slow down. Intel will too. The only thing I notice as difference between my Sandy Bridge Intel Xeon server and this one is that the AMD sucks about 30 watts more power. Dockers are Transmission, Plex, Owncloud, Tonido, MariaDB, It has an Areca 1280 with 2x2 tb parity and other 9 other 3 and 4 tb drives passed through. Nothing is connected to the motherboard sata ports at present. Other specs: Gigabyte Technology Co., Ltd. - GA-880GA-UD3H CPU: AMD Phenom II X6 1055T @ 2800 Memory: 16384 MB (max. installable capacity 16 GB) Network: eth0: 1000Mb/s - Full Duplex Quote Link to comment
pkn Posted April 12, 2015 Author Share Posted April 12, 2015 but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question. What is about Docker and AMD that you don't like?? I've just found that on my H8DME-2 (AMD-based) motherboard parity check (not sync) is going ridiculously slow. All thing equal motherboard X7SBE (Intel-based) provides 2-2.5 faster parity check. I blame this on kernel-level inability to properly (re)distribute load between the CPUs/CPU cores. I've just started a non-correcting parity sync on my AMD server running a few dockers and it is running at between 110 and 120mb/sec. This is a bit slower than the initial parity generation, but not the kind of numbers you are referring to. Shutting off all the dockers boosts it about 10 mb/sec., but depending on how hard the dockers are working, the parity will slow down. Intel will too. The only thing I notice as difference between my Sandy Bridge Intel Xeon server and this one is that the AMD sucks about 30 watts more power. Dockers are Transmission, Plex, Owncloud, Tonido, MariaDB, It has an Areca 1280 with 2x2 tb parity and other 9 other 3 and 4 tb drives passed through. Nothing is connected to the motherboard sata ports at present. Other specs: Gigabyte Technology Co., Ltd. - GA-880GA-UD3H CPU: AMD Phenom II X6 1055T @ 2800 Memory: 16384 MB (max. installable capacity 16 GB) Network: eth0: 1000Mb/s - Full Duplex Thanks for sharing. Well, that tells me that the problem is with Supermicro H8DME-2 motherboard. At least, I don't know where else it would be. I've tested two different H8DME-2 motherboards, with single CPU and with two CPUs, with different versions of unRAID, with different amounts of RAM from 4GB to 16GB... no changes. And, to recap, the Supermicro X7SBE motherboard, in exactly same setup, does not have this problem. Too bad 'cause I was hoping to use the H8DME-2. Quote Link to comment
tr0910 Posted April 12, 2015 Share Posted April 12, 2015 Are both of those motherboards "Tams" data center pulls? I have both the Intel and the Amd versions. Only running v5 though. And I don't see any difference in performance doing parity checks compared with this Amd server. Quote Link to comment
pkn Posted April 12, 2015 Author Share Posted April 12, 2015 Are both of those motherboards "Tams" data center pulls? ... Don't know for sure, but most probably yes. Quote Link to comment
luvmich Posted April 13, 2015 Share Posted April 13, 2015 Hate to burst the bubble on Intel and AMD. It is a linux driver issue of sorts. My two systems are similar except for the controller cards. Server 1 uses the onboard SATA and a AOC-SAS2LP-MV8. System 2 uses only the 2 AOC-SASLP-MV8. Anyway System 1 rebuilds at 130MB/s and parity checks at 40MB/s. System 2 rebuilds at 70MB/s and parity checks at 70MB/s. System 2 controller cards are max out hence the constant 70MB/s. I have not upgraded system 1 to v6 yet so I don't know if this situation has been fixed. But every v5 had the same results on the two systems. I tried bear bone installs of v5 on system 1 and I would always get the same 40MB/s parity checks. I borrowed a AOC-SASLP-MV8 and put it back in system 1. The parity checks went back to 70MB/s again. Made no sense. I'm just waiting for v6 final:) before I put it on System 1. Quote Link to comment
Squid Posted April 13, 2015 Share Posted April 13, 2015 Hate to burst the bubble on Intel and AMD. It is a linux driver issue of sorts. My two systems are similar except for the controller cards. Server 1 uses the onboard SATA and a AOC-SAS2LP-MV8. System 2 uses only the 2 AOC-SASLP-MV8. Anyway System 1 rebuilds at 130MB/s and parity checks at 40MB/s. System 2 rebuilds at 70MB/s and parity checks at 70MB/s. System 2 controller cards are max out hence the constant 70MB/s. I have not upgraded system 1 to v6 yet so I don't know if this situation has been fixed. But every v5 had the same results on the two systems. I tried bear bone installs of v5 on system 1 and I would always get the same 40MB/s parity checks. I borrowed a AOC-SASLP-MV8 and put it back in system 1. The parity checks went back to 70MB/s again. Made no sense. I'm just waiting for v6 final:) before I put it on System 1. Haven't really paid much attention to the conversation, but have you tried playing with the disk tunables? http://lime-technology.com/forum/index.php?topic=29009.0 Quote Link to comment
pkn Posted April 13, 2015 Author Share Posted April 13, 2015 ... Anyway System 1 rebuilds at 130MB/s and parity checks at 40MB/s. ... Sounds suspiciously similar to mine. Could you please run "top" during parity check for a minute or two? You don't need to run full parity check, it can be canceled at any time. In "good" system, during parity check, there are two processes, unraidd and mdrecoveryd, consuming 20-30% of CPU each, all others consume 0-2%. If, in addition to unraidd and mdrecoveryd, your often see process migration or migration/x in the topmost line, consuming a lot of CPU, then it's probably problem similar to mine. Quote Link to comment
JorgeB Posted June 13, 2015 Share Posted June 13, 2015 Anyone found a solution for this? I'm having the exact same problem. Two of my servers have almost identical hardware: Asrock B75 Pro3-M Celeron G550 and G540 4GB DDR3 1333 1 x SAS2LP-MV8 15 x 2TB HDDs, 6 on intel onboard, 1 on asmedia onboard, 8 on SAS2LP When I first built parity got around 105Mb/s, limited by some of the older disks which are only 500Gb/platter. Running a parity check or a disk rebuild I only get 40Mb/s. Either server get the same results, so it can’t be a bad board or controller, board has two pci-e slots, tried the SAS2LP on both. I also did the following tests: Added one SASLP-MV8, connected 4 HDDs from the SAS2LP, still same 40Mb/s parity check. Removed all disks from SAS2LP and connected all 8 to the SASLP, got about 75MB/s. Tested on different Unraid 6 beta versions, including beta15 and RC6. So, even though SASLP is limited by the slower PCI-e connection it's almost twice as fast as the SAS2LP that should not be bus limited, but this only happens on parity check, parity sync works as expected. Quote Link to comment
JorgeB Posted June 15, 2015 Share Posted June 15, 2015 Just to add that tried running disk tunables and there was no significative difference, best result was about 45Mb/s. I suspect a compatibility issue with the SAS2LP but it seams to be a card used by many here, is anyone getting a good parity check speed (>80Mb/s) with this card? Thanks Quote Link to comment
RobJ Posted June 15, 2015 Share Posted June 15, 2015 I've wanted to help, but so far drawing a complete blank. Quote Link to comment
pkn Posted June 15, 2015 Author Share Posted June 15, 2015 Anyone found a solution for this? I'm having the exact same problem. Two of my servers have almost identical hardware: Asrock B75 Pro3-M Celeron G550 and G540 ... So it's an Intel-based motherboard... could you please run "top" (via telnet, Putty, ssh, console, etc.) during parity check for a few minutes to verify? In "good" system, during parity check, there are two processes, unraidd and mdrecoveryd, consuming 20-30% of CPU each, all others consume 0-2%. If, in addition to unraidd and mdrecoveryd, your often see process migration or migration/x in the topmost line, consuming a lot of CPU, then it's probably problem similar to mine. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.