I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

 

It's not possible with dual parity.

 

http://lime-technology.com/forum/index.php?topic=47408.msg454560#msg454560

I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

 

It's not possible with dual parity.

 

http://lime-technology.com/forum/index.php?topic=47408.msg454560#msg454560

Thanks Johnnie, that's what I was afraid of.

 

I didn't realize the answer was already out there.  I'll make it a FAQ item, as it's a small change to the hardware freedom we've enjoyed for quite awhile, a reasonable trade-off for dual parity though.

Has anyone tested performance of the NVMe implementation in 6.2.0?

 

I have the Intel 750 and wondering if I should make it cache or make the Samsung SM951 as cache.

I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

i'm not knowledgeable, but I know that the 6.2 beta will not let you do so.  I am reasonably certain that Tom posted a message saying that order does now matter in response to a comment I made about this.

I'm going to hop into the "moving data breaks things in 6.2B21" camp.

I didn't expect to be, but so be it.

 

However I move things from a vdisk to array from Windows regularly and have not had an issue until this point.

 

I have a Win8VM (headless) that I use for torrents, it has a 2nd vdisk with a raw image I use as a HDD in the VM to store stuff.

I wanted to download a game I just purchased using the GOG downloader, this was ~40GB and a bunch of files.

Keep in mind I have had zero issues with 100+ GB of torrents down/up as this is a normal occurrence for this VM.

Anyhow this game downloading leads to the VM acting all kinds of weird, slow, laggy, etc. My Torrent app crashes, VM will hang at restart.

I leave it alone for the night, however this occurrence seems to cause  network traffic/dockers to not be accessible (SSH works fine), as I couldn't get to my MythTv backend, and it's webpage/port is inaccessible.

Try to shutdown the VM, nope. Try to kill the VM, nope error about unable to sigkill something something.

SSH .. virsh destroy VMname (same error). Grab diagnostics. Powerdown -r (powerdown initiated), nada... Hard reset.

Comes back up normal, dirty bit detected on USB, however no parity check "parity is valid".

 

Think this is a fluke, reload VM's, restart my game downloading (and torrent app).

5 minutes later, same exact thing (I was up for ~9 days previous prior to downloading this game).

Powerdown -r (powerdown initiated), nada. Shutdown -r (going down) nada.

SSH still works. Hard reset, back up, dirty bit detected, no parity check "parity is valid".

 

I've decided that downloading games are bad for me and I gave up (and have my Netbook doing it now).

I don't think the diagnostics are going to show you anything good, but attached.

 

Everything is back up without the attempt to download this game in the VM (torrent app running, nothing downloading), and all is well again.

 

I pulled 3 diagnostics while this was all happening, but nothing in the syslog that seems to point to anything.

Also, when this is happening I cannot stop dockers either, it is as if a process/thread (whatever) is holding up other things from working.

When I tried to access my flash drive through SMB during this (from within my primary VM) it barely loaded, just kept waiting to load.

 

 

server-diagnostics-20160420-1648.zip

Has anyone tested performance of the NVMe implementation in 6.2.0?

 

I have the Intel 750 and wondering if I should make it cache or make the Samsung SM951 as cache.

Due to other issues, I did not have the time yet to do perfomance tests.

As of now, there is no reason to believe these other issues are nvme related, so I can at least say its running stable.

And I can see/feel no difference from 6.1.9 where I had to use is outside of the array.

 

I dont know if there is any use-case in unRAID, that would make you see or feel any diffrence between AHCI M.2 and NVMe M.2.

At least through kvm, the latancy gain wont be as high as bare-metal.

 

I placed my old sata ssd in the array and use the 750 as cache. Write-Speed to the sata ssd is limited by the parity disk, but I enjoy a very fast, array protected read speed for my media stuff (plex is very fast now) and some rarely used VMs boot quite fast in case I do need them.

Finding that VM's freeze up after about 5-10 mins on this build. Entire array and GUI locks up and crashes at the same time.

 

Works perfectly with VM's disabled however.

Finding that VM's freeze up after about 5-10 mins on this build. Entire array and GUI locks up and crashes at the same time.

 

Works perfectly with VM's disabled however.

 

Mine are working ok -- I have 2 Windows VMs.  What do you have?

Can i please get a download link to beta 18 until the VM issue crashing everyone's server is fixed?

I'm going to hop into the "moving data breaks things in 6.2B21" camp.

I didn't expect to be, but so be it.

 

However I move things from a vdisk to array from Windows regularly and have not had an issue until this point.

 

I have a Win8VM (headless) that I use for torrents, it has a 2nd vdisk with a raw image I use as a HDD in the VM to store stuff.

I wanted to download a game I just purchased using the GOG downloader, this was ~40GB and a bunch of files.

Keep in mind I have had zero issues with 100+ GB of torrents down/up as this is a normal occurrence for this VM.

Anyhow this game downloading leads to the VM acting all kinds of weird, slow, laggy, etc. My Torrent app crashes, VM will hang at restart.

I leave it alone for the night, however this occurrence seems to cause  network traffic/dockers to not be accessible (SSH works fine), as I couldn't get to my MythTv backend, and it's webpage/port is inaccessible.

Try to shutdown the VM, nope. Try to kill the VM, nope error about unable to sigkill something something.

SSH .. virsh destroy VMname (same error). Grab diagnostics. Powerdown -r (powerdown initiated), nada... Hard reset.

Comes back up normal, dirty bit detected on USB, however no parity check "parity is valid".

 

Think this is a fluke, reload VM's, restart my game downloading (and torrent app).

5 minutes later, same exact thing (I was up for ~9 days previous prior to downloading this game).

Powerdown -r (powerdown initiated), nada. Shutdown -r (going down) nada.

SSH still works. Hard reset, back up, dirty bit detected, no parity check "parity is valid".

 

I've decided that downloading games are bad for me and I gave up (and have my Netbook doing it now).

I don't think the diagnostics are going to show you anything good, but attached.

 

Everything is back up without the attempt to download this game in the VM (torrent app running, nothing downloading), and all is well again.

 

I pulled 3 diagnostics while this was all happening, but nothing in the syslog that seems to point to anything.

Also, when this is happening I cannot stop dockers either, it is as if a process/thread (whatever) is holding up other things from working.

When I tried to access my flash drive through SMB during this (from within my primary VM) it barely loaded, just kept waiting to load.

 

You have a btrfs formatted cache drive.  Are you running the VMs from your cache drive?

 

This is from the KVM documentation at http://www.linux-kvm.org/page/Tuning_KVM

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

 

Could this be the problem with those having VM lockups?

 

You have a btrfs formatted cache drive.  Are you running the VMs from your cache drive?

 

This is from the KVM documentation at http://www.linux-kvm.org/page/Tuning_KVM

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

 

Could this be the problem with those having VM lockups?

 

It could be mine, though they've been working reliably since before 6.2.  I'm going to try to move them (though I dread touching them because I'd hate to have to rebuild them if I mess it up).

All my filesystems are XFS. (cache & array)

 

I found a very easy and fast way to reproduce the issue.

I installed an Ubuntu VM (15.04, desktop, default settings) and created a script that runs some dd commands.

The script contains variants up to 8G and 1000000 count (for i/o stress).

 

When I place the vDisk on a disk in the array and run it, the first line

dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

does not even succeed. Everything hangs after 5s - 15s.

 

It works perfectly fine while beeing placed on the cache. (850MB/s and more on NVMe cache)

 

I can now test the issue without putting my windows VMs in danger and I can rule out a windows issue.

Maybe other people want to try that and see if their system keeps running or hangs like mine.

I have no idea what the last 2 posters are doing, but unRAID is a guest VM on my ESXi. I don't think anyone in their right mind would do the reverse.

 

I tend to agree... it might be possible but it really sounds like the wrong way around... Unraid is really great with VM's, personally I have no need for esxi anymore, but -should- I have that need I would also opt for an esxi host with unraid running as a guest, probably with dockers in the unraid guests and esxi vm's for everything I would feel the need for a vm..

 

But as said: unraid with dockers and several vm's, runnin on both my primary and secundary system is running absolutely flawless.. I litterally have had -no- issues with unraid since I moved away from esxi.. But that kind of has been the case with unraid since I started using it in the beginning, no issues.. Maybe I have a very favourable hardware setup.

 

Why is it the wrong way around?

 

The sole reason for me to move from bare metal esxi to unraid was purely for passthrough of my 980ti and better support for hardware sensor monitoring without IPMI.

I can see no reason to change back to ESXi and run UNRAID KVM hypervisor inside ESXi hypervisor, it makes absolutely no sense.

 

Now that I have ESXi 6.0 and a standalone UNRAID VM running vcenter, I can simulate vmware DRS and vmotion etc capabilities for studying VCP6.

It's an opinion, not necessarily the truth, but it seems to me that if you deem unraid stable enough to run a hypervisor inside of it it should also be able to service the guests natively.. And the other way around: If you expect issues with the unraid hypervisor functionality it sounds weitd to have another hypervisor running inside of it...

 

But as said: opinion, lots of luck, does sound like a cool project..

 

 

Verzonden vanaf mijn iPhone met Tapatalk

It's an opinion, not necessarily the truth, but it seems to me that if you deem unraid stable enough to run a hypervisor inside of it it should also be able to service the guests natively.. And the other way around: If you expect issues with the unraid hypervisor functionality it sounds weitd to have another hypervisor running inside of it...

 

But as said: opinion, lots of luck, does sound like a cool project..

 

 

Verzonden vanaf mijn iPhone met Tapatalk

 

OK, i think I understand now what you're thinking. I guess my point that wasn't made very clear is that I only run ESXi inside UNRAID to set up a vmware farm/cloud environment for me to test multiple ESXi host vmotion, vcenter management and shared storage functionality through VSAN etc. I am not running Guest VMs inside ESXi inside UNRAID, now that would be pointless.

Totally get it now!

 

 

Verzonden vanaf mijn iPhone met Tapatalk

Found a fringe bug, but I believe still worth reporting, here's how to reproduce:

 

 

-start with a valid array with single parity

 

-stop array, add 2nd parity and a new disk at the same time.

 

-got an "invalid expansion" error, which after thinking about it makes sense since both operations need to use mdrecoveryd, but the error description next to the start button is not correct "You may not add new disk(s) and also remove existing disk(s).", I was not removing any disks, just adding two.

 

-So to get around this, I assigned the 2nd parity only, started array, canceled parity sync and stopped array, parity2 was showing yellow triangle as it should.

 

-Then assigned new disk and started array, clearing of new disk begun, parity2 was still showing triangle, I was expecting to be able to sync it after the clearing operation, but when the clearing finished all disks turned green, and the parity2 sync was never done, unsurprisingly resulting in many Q errors during a subsequent parity check.

You have a btrfs formatted cache drive.  Are you running the VMs from your cache drive?

 

This is from the KVM documentation at http://www.linux-kvm.org/page/Tuning_KVM

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

 

Could this be the problem with those having VM lockups?

 

Edit: I don't think this is specific to the "lockup" as UnRAID becomes very laggy and awful also, not just the VM's.

 

Hmm.. Yes, my cache drive is BTRFS, and has 5 VM's (concurrently running) and regular writes from caching recorded Tv (I don't record that much actually).

The image file I was writing to as a 2nd vdisk was on an array drive, which are all XFS.

 

That statement to me sounds a little over reaching (wouldn't you say?). I'm not questioning their guidance, but that's basically saying (for UnRAID here) don't have a cache drive as BTRFS with VM images on it. I'm pretty sure this is the opposite of the current "recommended" use case, as it seems BTRFS is primarily recommended for that exact use case (cache, with apps/VM's). I'm pretty certain most testing from JonP is with this exact same setup, and also users with cache pools have no other option than BTRFS.

 

Anyhow it could have been the issue, but the only time I've noticed it.

I've been stable since stopping the game download.

My old AMD C60 netbook handled the download just fine 

 

I may update the virtio drivers on that VM as I have not done so in a while (as I have had no reason to honestly).

All my filesystems are XFS. (cache & array)

 

I found a very easy and fast way to reproduce the issue.

I installed an Ubuntu VM (15.04, desktop, default settings) and created a script that runs some dd commands.

The script contains variants up to 8G and 1000000 count (for i/o stress).

 

When I place the vDisk on a disk in the array and run it, the first line

dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

does not even succeed. Everything hangs after 5s - 15s.

 

It works perfectly fine while beeing placed on the cache. (850MB/s and more on NVMe cache)

 

I can now test the issue without putting my windows VMs in danger and I can rule out a windows issue.

Maybe other people want to try that and see if their system keeps running or hangs like mine.

 

This is great.  I hope others can replicate it.

Finding that VM's freeze up after about 5-10 mins on this build. Entire array and GUI locks up and crashes at the same time.

 

Works perfectly with VM's disabled however.

 

Mine are working ok -- I have 2 Windows VMs.  What do you have?

 

2 VM's, both Windows.

Thought I would try my luck with 6.2. Trying to get my VM's back up and working but it would seem that with 6.2 it doesn't like the VM's hosted on the unassigned devices plugin. Is this correct?

 

I keep getting the following when I create a VM located on a mounted disk:

Warning: libvirt_domain_xml_xpath(): namespace warning : xmlns: URI unraid is not absolute in /usr/local/emhttp/plugins/dynamix.vm.manager/classes/libvirt.php on line 936 Warning: libvirt_domain_xml_xpath():

 

EDIT: hmm, even If I try to create a new VM in the /mnt/user/system/ location I get error. Diagnostics attached. Any ideas?

Thought I would try my luck with 6.2. Trying to get my VM's back up and working but it would seem that with 6.2 it doesn't like the VM's hosted on the unassigned devices plugin. Is this correct?

 

I keep getting the following when I create a VM located on a mounted disk:

Warning: libvirt_domain_xml_xpath(): namespace warning : xmlns: URI unraid is not absolute in /usr/local/emhttp/plugins/dynamix.vm.manager/classes/libvirt.php on line 936 Warning: libvirt_domain_xml_xpath():

 

EDIT: hmm, even If I try to create a new VM in the /mnt/user/system/ location I get error. Diagnostics attached. Any ideas?

 

I had/have the same problem with my Win10 VM's.

 

I ended up ditching my XML and creating a new VM and pointing it to the existing VM's Win10 disk image. I then went back and edited the XML as needed. I still get the warning but the VM's will start up.

 

Warning: libvirt_domain_xml_xpath(): namespace warning : xmlns: URI unraid is not absolute in /usr/local/emhttp/plugins/dynamix.vm.manager/classes/libvirt.php on line 936 Warning: libvirt_domain_xml_xpath():

 

I am using the Unassigned plugin and it doest seem to be a problem.

 

I had/have the same problem with my Win10 VM's.

 

I ended up ditching my XML and creating a new VM and pointing it to the existing VM's Win10 disk image. I then went back and edited the XML as needed. I still get the warning but the VM's will start up.

 

Warning: libvirt_domain_xml_xpath(): namespace warning : xmlns: URI unraid is not absolute in /usr/local/emhttp/plugins/dynamix.vm.manager/classes/libvirt.php on line 936 Warning: libvirt_domain_xml_xpath():

 

I am using the Unassigned plugin and it doest seem to be a problem.

 

My VM was also a Windows 10 VM. I went back to 6.1.9 as I have limited time currently.  I will probably try again once the next beta is released.

All my filesystems are XFS. (cache & array)

 

I found a very easy and fast way to reproduce the issue.

I installed an Ubuntu VM (15.04, desktop, default settings) and created a script that runs some dd commands.

The script contains variants up to 8G and 1000000 count (for i/o stress).

 

When I place the vDisk on a disk in the array and run it, the first line

dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

does not even succeed. Everything hangs after 5s - 15s.

 

It works perfectly fine while beeing placed on the cache. (850MB/s and more on NVMe cache)

 

I can now test the issue without putting my windows VMs in danger and I can rule out a windows issue.

Maybe other people want to try that and see if their system keeps running or hangs like mine.

Ok, I had some kind of breakthrough...

Testing the other filesystems was on the list and I finally had some spare time to move the files around.

 

Switching from XFS to BTRFS didn't change anything, but after going to ReiserFS the mentioned Ubuntu VM managed to not even just complete the first part of the script, but the whole thing.

 

Even with very decent perfomance, considering the parity and qemu overhead...

 

I need to test more, but until now, that script NEVER even once completed...

 

Btw, I forgot to mention that I changed the vDisk from "cache=writeback" to "cache=directsync" in all the tests, because it seemed to be a disk issue, not a cache issue.

So if anyone tried to reproduce, the result may be diffrent without "cache=directsync", sorry!

I have been coming across a weird issue lately which seems to have crept in with this new version, everything was working fine for ages but now i keep getting my vm's lockup and the following error message

 

Apr 23 15:49:00 Archangel kernel: pcieport 0000:00:03.0: AER: Uncorrected (Non-Fatal) error received: id=0018
Apr 23 15:49:00 Archangel kernel: pcieport 0000:00:03.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0018(Requester ID)
Apr 23 15:49:00 Archangel kernel: pcieport 0000:00:03.0:   device [8086:2f08] error status/mask=00004000/00000000

 

I'm unsure how to trace what the requester id actually goes back to but when this error appears both my vm's become paused under the gui and can not be resumed. On an attempted resume i get the bottom message:

internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

 

Not sure what to do with this as it is rendering my vm's unusable as without a forcestop they will not do anything. VM 2 (one named "Cat - SeaBios") will start up after the failure but will refuse to see any usb devices attached to it

 

No idea what to do with this one at all - i was able to take a diagnostic after this error occured

archangel-diagnostics-20160423-1555.zip