So right now I'm setup like this:

dom0: dom0_max_vcpus=1 dom0_vcpus_pin

archVM: vcpu="1-7"

 

And in most cases this seems to have worked well.  dom0 hovers around 60% when doing xfers, parity goes at full speed, and it hardly moves when archVM-Plex is at 650% transcoding 1080p->480p.

 

However where it did get maxed out, and seemed to actually hamper performance, was during my VM backup.  rsync was just hammering dom0 cpu to 95%.  I observed rsync speed drop to 15MB/s when copying the front of my data.img file (15GB used out of 40GB) and then jumped up to 60MB/s when it was copying the rest and cpu usage fell (I can't remember but less than 95%).

 

So to my eye this is a case of cpu bottleneck.  Ionic suggested it was the md5 check and sent me a new script which turns off hashing.  I'm not home so I can't test it, but I can see how that might work. 

 

BUT ... I'm not thrilled about turning off hashing and I want to see if I can solve the problem with good provisioning first.

 

What if I did this:

dom0: dom0_max_vcpus=2 dom0_vcpus_pin

archVM: vcpu = "1-7"

 

So I'm giving dom0 2 vcpus which will be pinned to cores 0,1 but I'm still allowing archVM access to core 1 but not core 0

 

Does that break some rule or best practice? 

 

Generally how will Xen deal with dividing up core1's cycles?

 

If there is contention for core1 will that cause a lot of context switching between cores 0 and 1?

 

I know I might have to ask this in the Xen mailing list, but I didn't want to jump into the list and was hoping a Xen guru here might have some insight.

 

PS I also know I could just give archVM 6 cores instead of 7, but I really want it to have access to as many cores as possible for Plex but I also know I need to dedicate one to dom0 so this is my compromise solution.  If I were really crazy I'd give it access to all 8 cores and start playing about with vcpu priority blah blah blah that I remember seeing once in the wiki / mailing list acrhives

Short answer...

 

Xen is far better at allocating resources than you are. I just allocate all my VMs all CPUs always.

haha yeah I hear ya.  I guess I was just going from the Xen Wiki's advice that dom0 really should be given dedicated resources (usually a single cpu) so that it isn't staved if the other VM's are heavy on I/O.  My concern is that while Plex isn't I/O heavy, it will indeed consume all the CPU you give it (and I Want it to) which would leave unraid very little left to do anything else ... like serve up the video that plex is transcoding and then manage sending that back out over the network.  Add a little bit of array demand (mover, parity, file management) and I'm worried about context switching and starving unraid.

 

Now of course I'm probably worrying about nothing / the last 5%, but I go back to, "That is what the Xen Wiki says to do" http://wiki.xen.org/wiki/Xen_Project_Best_Practices#Dedicating_a_CPU_core.28s.29_only_for_dom0

 

All else being equal, I'll probably just try what I suggested tonight and see what happens.  Fortunately I have a specific use case that should be repeatable as a test.

haha yeah I hear ya.  I guess I was just going from the Xen Wiki's advice that dom0 really should be given dedicated resources (usually a single cpu) so that it isn't staved if the other VM's are heavy on I/O.

 

That much is true. But by doing 'dom0: dom0_max_vcpus=2 dom0_vcpus_pin' you are allocating dom0 the CPU. The CPU time allocation gives by default priority to dom0 even if you 'over subscribe' the VMs / no of threads available.

 

I would urge you to allocate 2 CPUs to dom0 and then just enjoy your life! 

Well right, I want dom0 to have priority on pcpu1 (and own pcpu0).  Since dom0 won't usually have a need for it my thought is domU will be allowed to use it on a non-interference basis at best, or a 50/50 split at worst.

 

It won't work that way?

 

Yes Yes ... there is the simple way, the right way, and the way I obsess over ;-)  Thanks for indulging me.  As you might have guessed, when I jump into something, I REALLY immerse myself.  it's been an intense effort on my part with a long steep learning curve.

 

I'll play about with it tonight.  It should be pretty easy to set it and if it doesn't error out, I can throw a seriously heavy load at archVM and see if it ever lets it use pcpu1 or not.  I also did manage to sign up for the Xen mailing list and posted my question.

Well I tried it.  I can't tell for sure if it really worked. Something about the cpu times and the way it was jumping around in xl vcpu-list just didn't look right.  htop inside the VM claimed high useage in all 7 cpus and dom0 htop shows two with minimal useage but I'm just not convined.

 

But .... I went with a much more elegant solution until the next edge case: I solved the rsync problem for vm backup.  It was using -z for compression.  Totally not needed for local rsync.  CPU went down to a manageable 70% and speeds were fairly even over the entire .img files just below HDD speed (SSD appdisk to HDD cache) averaging 75MB/s

Well despite my initial doubtfulness I reexamined my screen shots and now I think maybe it was working.  Then I woke up to this http://xen.1045712.n5.nabble.com/Give-dom0-2-pinned-vcpus-but-share-one-with-domU-td5722792.html

 

 

Hi JustinChase, im watching this with interest as i have pinned 2 cores to dom0, the reason i have done this is not for performance issues but because i have/had an ongoing issue with xen crashing and bringing down dom0 as well, see posts below from PeterB and myself, there are other users also reporting the same crash:-

 

http://lime-technology.com/forum/index.php?topic=32618.msg304299#msg304299

http://lime-technology.com/forum/index.php?topic=32618.msg303019#msg303019

 

i found an old post from a user who had the same error as myself and a post from schoolbusdriver who then recommended pinning cores for dom0  which worked for that user, so i tried the same, pinning cpu0 and 1 to dom0 and so far no crashes!, so i think potentially having all cores available to dom0 and also to domU's is not a great idea, im interested to see how you get on.

 

the above is in no way 100% proof yet as its only been up a week but previously i had 3 crashes all within a week period, all with the same error.

 

edit - link to old post http://lime-technology.com/forum/index.php?topic=31653.msg291028#msg291028

You know I could have sworn there was another reason in the back of my mind why I even thought about pinning before I red the Xen wiki.  That must have been it.  Problem is, I was reading so much well before I made the leap that my head was just filled with amorphous facts floating all around with no place to put them.  I do remember that thread.

 

Well in anycase, score another reason for pinning, regardless of my specific desire to allow one of two pinned pcpus to be shred with a guest.  I'm going to change my config again tonight and leave it to see how it fairs over more time.

 

Thanks.

OH Crap! I just made the wiki-todo list http://wiki.xen.org/wiki/Xen_Document_Days/TODO#A_Xen_Performance_Tuning_Guide

Ijc 09:22, 21 May 2014 (UTC): The stuff about VCPU pinning seems to be a bit misleading, it's not always the best thing to do. See e.g. http://lists.xen.org/archives/html/xen-users/2014-05/msg00163.html and in particular http://lists.xen.org/archives/html/xen-users/2014-05/msg00170.html.  In the referenced thread I said: "It does say "might" and "can" perhaps even those are a bit strong. Pinning is one tool in the performance tuning arsenal but it is very workload dependent on whether it will help or hurt (and it can be a lot in either direction).", which I think is accurate (and in any case the docs need to guide people better about when/whether to pin and what the up/downsides are etc).

Hi JustinChase, im watching this with interest as i have pinned 2 cores to dom0, the reason i have done this is not for performance issues but because i have/had an ongoing issue with xen crashing and bringing down dom0 as well, see posts below from PeterB and myself, there are other users also reporting the same crash:-

 

http://lime-technology.com/forum/index.php?topic=32618.msg304299#msg304299

http://lime-technology.com/forum/index.php?topic=32618.msg303019#msg303019

 

i found an old post from a user who had the same error as myself and a post from schoolbusdriver who then recommended pinning cores for dom0  which worked for that user, so i tried the same, pinning cpu0 and 1 to dom0 and so far no crashes!, so i think potentially having all cores available to dom0 and also to domU's is not a great idea, im interested to see how you get on.

 

the above is in no way 100% proof yet as its only been up a week but previously i had 3 crashes all within a week period, all with the same error.

 

edit - link to old post http://lime-technology.com/forum/index.php?topic=31653.msg291028#msg291028

 

I think I've had the same issue. It seems to happen whenever I stress the arch VM e.g. multiple Plex synchs to my tablet and at the same time use the win 8 VM and then say try to access the Xen Domains page in the unraid GUI.

 

I still have console access, but can't shut the server down cleanly even if I first issue the xl destroy cmd to kill the vm's hard reboot is about all that works, which of course means no log written to USB.  How do I manually save a copy of the log before rebooting so that I can try to see if there's anything in the log? Where is the log stored before being saved to USB?

 

Sent from my Nexus 4 using Tapatalk

 

 

Hi JustinChase, im watching this with interest as i have pinned 2 cores to dom0, the reason i have done this is not for performance issues but because i have/had an ongoing issue with xen crashing and bringing down dom0 as well, see posts below from PeterB and myself, there are other users also reporting the same crash:-

 

http://lime-technology.com/forum/index.php?topic=32618.msg304299#msg304299

http://lime-technology.com/forum/index.php?topic=32618.msg303019#msg303019

 

i found an old post from a user who had the same error as myself and a post from schoolbusdriver who then recommended pinning cores for dom0  which worked for that user, so i tried the same, pinning cpu0 and 1 to dom0 and so far no crashes!, so i think potentially having all cores available to dom0 and also to domU's is not a great idea, im interested to see how you get on.

 

the above is in no way 100% proof yet as its only been up a week but previously i had 3 crashes all within a week period, all with the same error.

 

edit - link to old post http://lime-technology.com/forum/index.php?topic=31653.msg291028#msg291028

 

I think I've had the same issue. It seems to happen whenever I stress the arch VM e.g. multiple Plex synchs to my tablet and at the same time use the win 8 VM and then say try to access the Xen Domains page in the unraid GUI.

 

I still have console access, but can't shut the server down cleanly even if I first issue the xl destroy cmd to kill the vm's hard reboot is about all that works, which of course means no log written to USB.  How do I manually save a copy of the log before rebooting so that I can try to see if there's anything in the log? Where is the log stored before being saved to USB?

 

Sent from my Nexus 4 using Tapatalk

 

hi apgood, when you say your seeing the same issue, are you actually seeing the crash on the console? (see screenshots), when im seeing the issue i reported it bring down everything, the only thing i can do is ping dom0 ip, thats all, hard reset required, i have no access from the console to run any commands.

 

if you are seeing the same message on the console then i would highly recommend you try pinning cores to dom0, this seems to of cured it for me, i have a theory as to why it works but i will leave that for now incase im wrong :-). if you need a walk through on how to pin cores to dom0 then please let me know and i will post up the code.

No sure if the messages on my console are identical.  I've pinned a CPU to dom0 now so hopefully it won't happen again but if it does I'll take a photo

 

Sent from my Nexus 4 using Tapatalk

 

 

No sure if the messages on my console are identical.  I've pinned a CPU to dom0 now so hopefully it won't happen again but if it does I'll take a photo

 

Sent from my Nexus 4 using Tapatalk

 

sounds good :-), the new kernel that will be shipped with beta6 MAY actually fix the issue, or at the very least will give more detail on exactly what is crashing the system, see this post http://lists.xen.org/archives/html/xen-devel/2013-10/msg00585.html but for now pinning works for me 

So if anyone dug into those links the xen list guys sent ... I wonder if the problem is unraid being starved or the context switching.  And in either case, maybe changing dom0's credit schedule setting might make the diff.  I suppose I'll try unpinning completely to see if I have an crashes and then if I do I'll muck about with the credit.

So if anyone dug into those links the xen list guys sent ... I wonder if the problem is unraid being starved or the context switching.  And in either case, maybe changing dom0's credit schedule setting might make the diff.  I suppose I'll try unpinning completely to see if I have an crashes and then if I do I'll muck about with the credit.

 

yeah i think starvation of cpu cycles for dom0 is the issue, caused by high cpu usage domU's and the whole thing folds in on itself causing dom0 to crash, havent heard of the credit scheduler, off to do some googling :-).

Which is of course very weird.  No matter how much we think us unraid users are special flowers, people have been banging against Xen for a long time now.  Makes me wonder if it is something in the 3.10 kernel / something not turned on.  Hopefully 6.0b6 fixes it with 3.14 and I think Xen 4.4 as well.

Yep managed to cause it even with one CPU pinned to dom0.... Was trying to expand my archvm .IMG file. Needless to say my archvm is very much stuffed (well actually it was stuffed before that because I was mucking around with automount points and broke it to point that VM wouldn't boot).  Anyway console screen says CPU stalled but stall ended before state dump start... Whatever that means it wasn't good because nothing responded until hard reboot.

 

Sent from my Nexus 4 using Tapatalk

 

 

You know what ... well before I was using Xen (read it wasn't even turned on) I started noticing cpu stall messages.  I can't remember if it was only with 6.0 or if I had it with 5.0.  But I never had crashes.  If my memory is correct (I'll have to search my post history), I wonder if virtualization is less tolerant of stalls.

Which is of course very weird.  No matter how much we think us unraid users are special flowers, people have been banging against Xen for a long time now.  Makes me wonder if it is something in the 3.10 kernel / something not turned on.  Hopefully 6.0b6 fixes it with 3.14 and I think Xen 4.4 as well.

 

i think it could also be kernel related, excerpt from the xen post i found that matched the issue:-

 

I've had a report of a host starting to give this output:

Oct 7 19:36:37 kernel: INFO: rcu_sched self-detected stall on CPU { 2}

(t=273561 jiffies g=17919 c=17918 q=9688)

Oct 7 19:36:37 kernel: sending NMI to all CPUs:

Oct 7 19:36:37 kernel: xen: vector 0x2 is not implemented

This bit is just a symptom triggered by the initial rcu stall which is

your real problem.

 

That said I thought the NMI thing was fixed recently, which might have

gotten you better debugging on the rcu problem.

 

This went into v3.12-rc1 (commit 6efa20e).

 

Steven, can you try your test with newer kernels that have this fix? With

it we should be able to see where the stall is happening.

 

-boris

 

which kinda makes sense as we are on kernel 3.10 right now, looks like we are getting bumped up to 3.14 in b6, and as you stated xen 4.4 too so maybe this will get rid of this issue completely.

So who has control of the cattle prod to use on Tom? ;-)

So who has control of the cattle prod to use on Tom? ;-)

 

ive already used it on him (pm sent) thing is i think he has thick skin hehe

maybe jonp is feeling up to pushing, now that he's back from his Dominican Republic vacation.

 

With that said, what is the verdict on pinning CPU's?  Should we, or should we not currently do this, and if so, how many?  I have 4 CPU's and no hyperthreading.  I've not had any issues that I'm aware of, and am not pinning anything currently, but am about to finish my Windows 7 VM, and intend to start using it heavily.

Turns out my issues were related to bad ram that was slowly getting worse.

 

Anyway got an old HP dl180 G6 from work, so once I get all the bits together I'll migrate unraid over to proper server hardware and use the current box as a backup server.

 

Sent from my Nexus 4 using Tapatalk

 

 

Well it works.  but I'm not sure it was really getting me anything.  Based on the advice from the Xen list I have now set both doms to run with full access to all 8 cpus.  I've had one cpu stall hard crash.  It happened while I was doing some silliness with my network.  Not that it should have caused a crash, but i was not being nice to my network.

 

I've also hammered my archVM with some serious transcoding while also tweaking around with dom0 and didn't ahve any trouble at that time.

 

I'm not sure what to say about memory being an issue.  I'm running ECC ram and it tested good.  But I'm always open to ideas. 

 

I think between xen4.4, 3.12 kernel, and pending todo in the xen docs, I'm just going to sit tight for the time being.