unRAID Server Release 6.2.0-beta21 Available

[JonathanM]

But it's not a network configuration issue, it's an issue with the passthrough. It's affecting other passed through devices, 6.2 doesn't see the drives on my RAID card either, which pick up just fine on 6.1.9. The network thing was just the first issue I noticed, because I couldn't even get into the system right off the bat

 

Confirmed issue: http://lime-technology.com/forum/index.php?topic=48374

 

Can limetech or jonp chime in? Hopefully something that can be fixed, but it's been around for all 6.2 betas

The last time something like this happened, the official response was, we don't support unraid as a guest, if it works fine, if not, and you can troubleshoot why and tell us what needs to be changed to fix it we will consider making the change, but we are not going to spend the time to troubleshoot it. They don't break ESXI support on purpose, they just aren't going to spend any time making it work.

 

Maybe that stance has changed, and Tom or Jon will help with ESXI support, but I seriously doubt it with all the other things going on right now.

[RobJ]

For a second time now I get the server runing 6.2b21 locked while running SyncBack between two unraid servers.

Source is running 6.2beta21 and destination is running 6.1.9.

The system GUI freezes, Dockers and VM's are not responding, ssh is still working but when launching powerdown the command freezes at: "Starting diagnostics collection..." see picture attached.

Luckily the cp /var/log/syslog /boot command worked so I can share the syslog file attached.

 

Some comments, many probably unrelated to your issues, but ... -

 

* Several devices report only 32 bit DMAR, r8169 (your Realtek NIC) and certain USB ports.  I'm not very knowledgeable in this, but I don't believe that's going to work correctly, especially virtualized.

 

* NerdPack reports installing Python 2.7.9.  I believe you need 2.7.11 for 6.2 betas.  Edit: Later 2.7.11 is installed also.

 

* The packages libevent-2.0.22-x86_64-1 and ncurses-5.9-x86_64-4 are already installed, but the Preclear beta *upgrades* them with /boot/config/plugins/preclear.disk.beta/libevent-2.0.21-x86_64-1.txz and /boot/config/plugins/preclear.disk.beta/ncurses-5.9-x86_64-2.txz.

 

Apr 19 02:36:57 Tower kernel: DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.

Apr 19 02:36:57 Tower kernel: DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.

Apr 19 02:36:57 Tower kernel: DMAR-IR: Enabled IRQ remapping in xapic mode

Apr 19 02:36:57 Tower kernel: x2apic: IRQ remapping doesn't support X2APIC mode

* Don't know if it's important but kernel people are recommending the options above.  You might also check the BIOS settings, and check for a BIOS update.

 

* Strongly recommend you switch to static IP (192.168.0.40).  That should clean up some of the errors.  The first renewal does not seem to proceed completely correctly when virtual networking is involved, not the first time I've seen that.  Can't say it caused your problems, but it would be interesting to see how it behaves when changed.

 

* USB devices were disconnected near the end.  That's fine if you did it, not good if you didn't.

 

* Powerdown started but got nowhere.  What ever the first actions, they didn't happen.  Possibly user or system kill scripts to stop VM's and Dockers?

[RobJ]

Obviously can't share any logs since I can't get into the system, but, thoughts?

So it doesn't boot at all? Do you get a local console prompt? If so, log in and run the diagnostics script at the local console, then post the resulting zip file. Just because the network doesn't appear to be working doesn't mean the system isn't booting.

 

It boots, and looks totally fine, outside of the fact that the network (and presumably other passed through devices) aren't being picked up. I can get into the console, but have no way of downloading the diagnostic tool, so best I can do is copy the normal unRAID log to the flash drive so I can access and share it

from the console,

diagnostics

  will be saved to the flash drive.

 

Oh, whoops. I was looking at the WiKi and it was mentioning I had to download something. I ran it and attached

 

I'll start by admitting I'm probably out of my depth here, this was one of the more confusing setups I've ever seen.  You are running this on a VMware base and passing everything through, and then expecting to pass through the passed through devices?  What I don't understand is why don't you remove unRAID and just run the VM's on VMware.  There are no NAS features in use that I could see, no hard drives at all, not even cache.  To this eye (not the most experienced with VM's), it looks like you are trying to run VM's on top of unRAID, on top of VMware, without any other use of unRAID?

 

There's one Intel NIC port passed through, but doesn't work very well.  Although you did get an eth0, it's broken.  You have bonding, bridging, and DHCP turned on.  Try turning off bonding, that may be confusing it (or the fact it's passed through?)  And using a static IP can help.  Not sure if turning off bridging would help or hurt.

 

The SAS controller was passed up to unRAID, and partly was set up successfully, but then failed, not sure why, but can't help wondering if it would work better bare metal.  If there are drives attached, it didn't get far enough in the initialization to find them.

 

Edit: One difference I've noticed between unRAID 6.2 and 6.1, the kernel 4.4 assigns mpt3sas, whereas 4.1 assigned mpt2sas, for cards like that.

Edit2: Sorry EGOvoruhk, I see elsewhere implied that you do have drives attached, and normal NAS usage.  Please ignore those comments.

[RobJ]

I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

[JorgeB]

I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

 

It's not possible with dual parity.

 

http://lime-technology.com/forum/index.php?topic=47408.msg454560#msg454560

[RobJ]

I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

 

It's not possible with dual parity.

 

http://lime-technology.com/forum/index.php?topic=47408.msg454560#msg454560

Thanks Johnnie, that's what I was afraid of.

 

I didn't realize the answer was already out there.  I'll make it a FAQ item, as it's a small change to the hardware freedom we've enjoyed for quite awhile, a reasonable trade-off for dual parity though.

[testdasi]

Has anyone tested performance of the NVMe implementation in 6.2.0?

 

I have the Intel 750 and wondering if I should make it cache or make the Samsung SM951 as cache.

[itimpi]

I have a question for Tom or other parity experts, knowledgeable in the new Q parity.  The P parity we're used to was a simple sum (or in our case the equivalent XOR) of the stream of bits at a specific disk offset.  As such, it didn't matter what order the bits were in, which meant that the disks could be freely rearranged without any change to parity.  This alternative way of calculating the Q parity has to use the same bits, but apply a different algorithm, and it seems likely to me that the bit order matters.  And if it does, then array drives cannot be freely rearranged without invalidating the second parity drive.

 

So there's the question, can we still swap drives around, change their array assignments, without having to redo the Q parity?  I'm asking because I'm updating the FAQ for v6, about rearranging drives within the array.

 

My apologies for never taking the time to study how the new parity drive is calculated!

i'm not knowledgeable, but I know that the 6.2 beta will not let you do so.  I am reasonably certain that Tom posted a message saying that order does now matter in response to a comment I made about this.

[bungee91]

I'm going to hop into the "moving data breaks things in 6.2B21" camp.

I didn't expect to be, but so be it.

 

However I move things from a vdisk to array from Windows regularly and have not had an issue until this point.

 

I have a Win8VM (headless) that I use for torrents, it has a 2nd vdisk with a raw image I use as a HDD in the VM to store stuff.

I wanted to download a game I just purchased using the GOG downloader, this was ~40GB and a bunch of files.

Keep in mind I have had zero issues with 100+ GB of torrents down/up as this is a normal occurrence for this VM.

Anyhow this game downloading leads to the VM acting all kinds of weird, slow, laggy, etc. My Torrent app crashes, VM will hang at restart.

I leave it alone for the night, however this occurrence seems to cause  network traffic/dockers to not be accessible (SSH works fine), as I couldn't get to my MythTv backend, and it's webpage/port is inaccessible.

Try to shutdown the VM, nope. Try to kill the VM, nope error about unable to sigkill something something.

SSH .. virsh destroy VMname (same error). Grab diagnostics. Powerdown -r (powerdown initiated), nada... Hard reset.

Comes back up normal, dirty bit detected on USB, however no parity check "parity is valid".

 

Think this is a fluke, reload VM's, restart my game downloading (and torrent app).

5 minutes later, same exact thing (I was up for ~9 days previous prior to downloading this game).

Powerdown -r (powerdown initiated), nada. Shutdown -r (going down) nada.

SSH still works. Hard reset, back up, dirty bit detected, no parity check "parity is valid".

 

I've decided that downloading games are bad for me and I gave up (and have my Netbook doing it now).

I don't think the diagnostics are going to show you anything good, but attached.

 

Everything is back up without the attempt to download this game in the VM (torrent app running, nothing downloading), and all is well again.

 

I pulled 3 diagnostics while this was all happening, but nothing in the syslog that seems to point to anything.

Also, when this is happening I cannot stop dockers either, it is as if a process/thread (whatever) is holding up other things from working.

When I tried to access my flash drive through SMB during this (from within my primary VM) it barely loaded, just kept waiting to load.

 

 

server-diagnostics-20160420-1648.zip

[dAigo]

Has anyone tested performance of the NVMe implementation in 6.2.0?

 

I have the Intel 750 and wondering if I should make it cache or make the Samsung SM951 as cache.

Due to other issues, I did not have the time yet to do perfomance tests.

As of now, there is no reason to believe these other issues are nvme related, so I can at least say its running stable.

And I can see/feel no difference from 6.1.9 where I had to use is outside of the array.

 

I dont know if there is any use-case in unRAID, that would make you see or feel any diffrence between AHCI M.2 and NVMe M.2.

At least through kvm, the latancy gain wont be as high as bare-metal.

 

I placed my old sata ssd in the array and use the 750 as cache. Write-Speed to the sata ssd is limited by the parity disk, but I enjoy a very fast, array protected read speed for my media stuff (plex is very fast now) and some rarely used VMs boot quite fast in case I do need them.

[Kuroyukihimeeee]

Finding that VM's freeze up after about 5-10 mins on this build. Entire array and GUI locks up and crashes at the same time.

 

Works perfectly with VM's disabled however.

[Adam64]

Finding that VM's freeze up after about 5-10 mins on this build. Entire array and GUI locks up and crashes at the same time.

 

Works perfectly with VM's disabled however.

 

Mine are working ok -- I have 2 Windows VMs.  What do you have?

[pyrater]

Can i please get a download link to beta 18 until the VM issue crashing everyone's server is fixed?

[dlandon]

I'm going to hop into the "moving data breaks things in 6.2B21" camp.

I didn't expect to be, but so be it.

 

However I move things from a vdisk to array from Windows regularly and have not had an issue until this point.

 

I have a Win8VM (headless) that I use for torrents, it has a 2nd vdisk with a raw image I use as a HDD in the VM to store stuff.

I wanted to download a game I just purchased using the GOG downloader, this was ~40GB and a bunch of files.

Keep in mind I have had zero issues with 100+ GB of torrents down/up as this is a normal occurrence for this VM.

Anyhow this game downloading leads to the VM acting all kinds of weird, slow, laggy, etc. My Torrent app crashes, VM will hang at restart.

I leave it alone for the night, however this occurrence seems to cause  network traffic/dockers to not be accessible (SSH works fine), as I couldn't get to my MythTv backend, and it's webpage/port is inaccessible.

Try to shutdown the VM, nope. Try to kill the VM, nope error about unable to sigkill something something.

SSH .. virsh destroy VMname (same error). Grab diagnostics. Powerdown -r (powerdown initiated), nada... Hard reset.

Comes back up normal, dirty bit detected on USB, however no parity check "parity is valid".

 

Think this is a fluke, reload VM's, restart my game downloading (and torrent app).

5 minutes later, same exact thing (I was up for ~9 days previous prior to downloading this game).

Powerdown -r (powerdown initiated), nada. Shutdown -r (going down) nada.

SSH still works. Hard reset, back up, dirty bit detected, no parity check "parity is valid".

 

I've decided that downloading games are bad for me and I gave up (and have my Netbook doing it now).

I don't think the diagnostics are going to show you anything good, but attached.

 

Everything is back up without the attempt to download this game in the VM (torrent app running, nothing downloading), and all is well again.

 

I pulled 3 diagnostics while this was all happening, but nothing in the syslog that seems to point to anything.

Also, when this is happening I cannot stop dockers either, it is as if a process/thread (whatever) is holding up other things from working.

When I tried to access my flash drive through SMB during this (from within my primary VM) it barely loaded, just kept waiting to load.

 

You have a btrfs formatted cache drive.  Are you running the VMs from your cache drive?

 

This is from the KVM documentation at http://www.linux-kvm.org/page/Tuning_KVM

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

 

Could this be the problem with those having VM lockups?

[Adam64]

 

You have a btrfs formatted cache drive.  Are you running the VMs from your cache drive?

 

This is from the KVM documentation at http://www.linux-kvm.org/page/Tuning_KVM

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

 

Could this be the problem with those having VM lockups?

 

It could be mine, though they've been working reliably since before 6.2.  I'm going to try to move them (though I dread touching them because I'd hate to have to rebuild them if I mess it up).

[dAigo]

All my filesystems are XFS. (cache & array)

 

I found a very easy and fast way to reproduce the issue.

I installed an Ubuntu VM (15.04, desktop, default settings) and created a script that runs some dd commands.

The script contains variants up to 8G and 1000000 count (for i/o stress).

 

When I place the vDisk on a disk in the array and run it, the first line

dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

does not even succeed. Everything hangs after 5s - 15s.

 

It works perfectly fine while beeing placed on the cache. (850MB/s and more on NVMe cache)

 

I can now test the issue without putting my windows VMs in danger and I can rule out a windows issue.

Maybe other people want to try that and see if their system keeps running or hangs like mine.

[mikeyosm]

I have no idea what the last 2 posters are doing, but unRAID is a guest VM on my ESXi. I don't think anyone in their right mind would do the reverse.

 

I tend to agree... it might be possible but it really sounds like the wrong way around... Unraid is really great with VM's, personally I have no need for esxi anymore, but -should- I have that need I would also opt for an esxi host with unraid running as a guest, probably with dockers in the unraid guests and esxi vm's for everything I would feel the need for a vm..

 

But as said: unraid with dockers and several vm's, runnin on both my primary and secundary system is running absolutely flawless.. I litterally have had -no- issues with unraid since I moved away from esxi.. But that kind of has been the case with unraid since I started using it in the beginning, no issues.. Maybe I have a very favourable hardware setup.

 

Why is it the wrong way around?

 

The sole reason for me to move from bare metal esxi to unraid was purely for passthrough of my 980ti and better support for hardware sensor monitoring without IPMI.

I can see no reason to change back to ESXi and run UNRAID KVM hypervisor inside ESXi hypervisor, it makes absolutely no sense.

 

Now that I have ESXi 6.0 and a standalone UNRAID VM running vcenter, I can simulate vmware DRS and vmotion etc capabilities for studying VCP6.

[Helmonder]

It's an opinion, not necessarily the truth, but it seems to me that if you deem unraid stable enough to run a hypervisor inside of it it should also be able to service the guests natively.. And the other way around: If you expect issues with the unraid hypervisor functionality it sounds weitd to have another hypervisor running inside of it...

 

But as said: opinion, lots of luck, does sound like a cool project..

 

 

Verzonden vanaf mijn iPhone met Tapatalk

[mikeyosm]

It's an opinion, not necessarily the truth, but it seems to me that if you deem unraid stable enough to run a hypervisor inside of it it should also be able to service the guests natively.. And the other way around: If you expect issues with the unraid hypervisor functionality it sounds weitd to have another hypervisor running inside of it...

 

But as said: opinion, lots of luck, does sound like a cool project..

 

 

Verzonden vanaf mijn iPhone met Tapatalk

 

OK, i think I understand now what you're thinking. I guess my point that wasn't made very clear is that I only run ESXi inside UNRAID to set up a vmware farm/cloud environment for me to test multiple ESXi host vmotion, vcenter management and shared storage functionality through VSAN etc. I am not running Guest VMs inside ESXi inside UNRAID, now that would be pointless.

[Helmonder]

Totally get it now!

 

 

Verzonden vanaf mijn iPhone met Tapatalk

[JorgeB]

Found a fringe bug, but I believe still worth reporting, here's how to reproduce:

 

 

-start with a valid array with single parity

 

-stop array, add 2nd parity and a new disk at the same time.

 

-got an "invalid expansion" error, which after thinking about it makes sense since both operations need to use mdrecoveryd, but the error description next to the start button is not correct "You may not add new disk(s) and also remove existing disk(s).", I was not removing any disks, just adding two.

 

-So to get around this, I assigned the 2nd parity only, started array, canceled parity sync and stopped array, parity2 was showing yellow triangle as it should.

 

-Then assigned new disk and started array, clearing of new disk begun, parity2 was still showing triangle, I was expecting to be able to sync it after the clearing operation, but when the clearing finished all disks turned green, and the parity2 sync was never done, unsurprisingly resulting in many Q errors during a subsequent parity check.

[bungee91]

You have a btrfs formatted cache drive.  Are you running the VMs from your cache drive?

 

This is from the KVM documentation at http://www.linux-kvm.org/page/Tuning_KVM

Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest.

 

Could this be the problem with those having VM lockups?

 

Edit: I don't think this is specific to the "lockup" as UnRAID becomes very laggy and awful also, not just the VM's.

 

Hmm.. Yes, my cache drive is BTRFS, and has 5 VM's (concurrently running) and regular writes from caching recorded Tv (I don't record that much actually).

The image file I was writing to as a 2nd vdisk was on an array drive, which are all XFS.

 

That statement to me sounds a little over reaching (wouldn't you say?). I'm not questioning their guidance, but that's basically saying (for UnRAID here) don't have a cache drive as BTRFS with VM images on it. I'm pretty sure this is the opposite of the current "recommended" use case, as it seems BTRFS is primarily recommended for that exact use case (cache, with apps/VM's). I'm pretty certain most testing from JonP is with this exact same setup, and also users with cache pools have no other option than BTRFS.

 

Anyhow it could have been the issue, but the only time I've noticed it.

I've been stable since stopping the game download.

My old AMD C60 netbook handled the download just fine 

 

I may update the virtio drivers on that VM as I have not done so in a while (as I have had no reason to honestly).

[Adam64]

All my filesystems are XFS. (cache & array)

 

I found a very easy and fast way to reproduce the issue.

I installed an Ubuntu VM (15.04, desktop, default settings) and created a script that runs some dd commands.

The script contains variants up to 8G and 1000000 count (for i/o stress).

 

When I place the vDisk on a disk in the array and run it, the first line

dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

does not even succeed. Everything hangs after 5s - 15s.

 

It works perfectly fine while beeing placed on the cache. (850MB/s and more on NVMe cache)

 

I can now test the issue without putting my windows VMs in danger and I can rule out a windows issue.

Maybe other people want to try that and see if their system keeps running or hangs like mine.

 

This is great.  I hope others can replicate it.

[Kuroyukihimeeee]

Finding that VM's freeze up after about 5-10 mins on this build. Entire array and GUI locks up and crashes at the same time.

 

Works perfectly with VM's disabled however.

 

Mine are working ok -- I have 2 Windows VMs.  What do you have?

 

2 VM's, both Windows.

[archedraft]

Thought I would try my luck with 6.2. Trying to get my VM's back up and working but it would seem that with 6.2 it doesn't like the VM's hosted on the unassigned devices plugin. Is this correct?

 

I keep getting the following when I create a VM located on a mounted disk:

Warning: libvirt_domain_xml_xpath(): namespace warning : xmlns: URI unraid is not absolute in /usr/local/emhttp/plugins/dynamix.vm.manager/classes/libvirt.php on line 936 Warning: libvirt_domain_xml_xpath():

 

EDIT: hmm, even If I try to create a new VM in the /mnt/user/system/ location I get error. Diagnostics attached. Any ideas?