May 3, 201412 yr Hey everyone, Jon here with Lime Tech! For those of you who don't know me, I just recently joined the team at Lime Tech back in April, and we have been hard at work over the course of the past month to continue development of Xen Virtualization with unRAID. Today, I want to share with you some updates on where we stand with capabilities in unRAID 6 Beta-5a, which is available now. First and foremost, let me say that we were able to reproduce all of the functionality we achieved in Beta-5a in Beta 4, but we need everyone to upgrade to 5a to confirm continuing functionality as we also enhance the product (nice way of saying I can't help you if you're lagging behind in the old beta ;-) ). For those of you who have been feverishly watching and participating in these forums, here's the short-list update on what we have tested and confirmed to work in Beta 5a: - Passing through a wide-variety of AMD-based (ATI) GPUs to Windows 7/8.1 virtual machines. - Passing through of a USB Keyboard / Mouse to an individual VM. - Successful build of an Ubuntu VM with install of Plex Media Server and XBMC (no GPU pass through yet). - Passing through of USB KB/M devices with assistance of a USB hub. - Passing through of Audio via HDMI on GPU. - Performed 3D Mark benchmark on Windows VMs with GPU pass through successfully. Impressive results… As for hardware, we have used a number of different systems (all Intel-based so far) and I will share the hardware specs with everyone in separate threads as to also grow that list as we get further confirmed support from you guys as well. I will say this much, hardware selection is pretty important as we have had different results with different motherboards and graphics cards. One of our continued efforts is to find more and more confirmed working configurations. I will be posting more threads in this forum to help describe further how we achieved these results and where we are at in current development on Beta 6, but know that our goal is to get unRAID 6 to an RC build as quickly and safely as possible. We believe we are very close in our objectives, but your support is needed to continue the effort!! Now a few goodies to share. With respect to Windows-based virtualization and GPU pass through, here is our Xen configuration file: name="vmname" builder="hvm" vcpus=1 #can set this to 1-4 safely depending on your CPU / # of VMs. memory=2048 #DO NOT EXCEED UNLESS BRAVE IF DOING GPU PASS THROUGH disk=['file:/mnt/cache/domains/vmname.img,hda,w', 'file:/your/path/to/iso/isoname.iso,hdc:cdrom,r'] #CHANGE THE PATHS TO MATCH YOUR DOMAIN vif=['mac=02:02:02:02:02:02,bridge=YOURBRIDGENAME'] boot="dc" viridian=1 #FOR WINDOWS-BASED VMs, THIS IS A MUST #xen_platform_pci=1 #ONLY NEEDED IF YOU INSTALL THE XEN PARAVIRTUALIZATION DRIVERS #VNC SETTINGS vnc=1 vnclisten='0.0.0.0' vncpasswd='ifyouwant' #COMMENT OUT IF YOU DON'T WANT A PASSWORD REQUIRED #NO USB PASSTHROUGH CONFIG FOR VNC ACCESSIBILITY usb=1 usbdevice='tablet' #WITH USB PASSTHROUGH, COMMENT OUT THE ABOVE TWO ITEMS AND UNCOMMENT THESE #usb=0 #usbdevice=['host:LOOK UP VIA lsusb COMMAND IN CONSOLE']','host:####:####'] on_poweroff="destroy" on_reboot="restart" on_crash="destroy" pci = [ '01:00.0','01:00.1' ] #CHANGE FOR YOUR CARD LOOKUP WITH lspci COMMAND This config works for Windows 7 and Windows 8.1 with a single GPU pass through. The game is "less is more" with the config file and with memory assignment. For some odd reason, if you give the VM too much RAM, even on a very saturated host (ours had 32GB), the GPU passthrough will fail to function. We were using 64-bit VMs too, so no, it's not a 32-bit limitation thing. This config was tested with an AMD R7-250, R5-230, and some other really old AMD cards (I'll get the make/models up later). What I can tell you is that the higher end R9 290 cards haven't been so successful. While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. This is a work in progress for us right now. But if your goal isn't to achieve high-end gaming, the lower-end cards work totally fine. Even some of the oldest cards we grabbed from ATI/AMD back in the hay-day seemed to work ok. If you're on Beta-4, this config MAY work for you too, but to be fair, we haven't tested all the hardware on both versions extensively. Know that we did get GPU pass through working in Beta 4 and specifically with Windows 8.1 and Windows 7, but we still recommend going to Beta 5a now as we have already moved past Beta 4 in our lab. I'll be posting more of a detailed guide on this soon, but one thing I can definitely recommend when using this configuration is that you definitely install the latest AMD Catalyst Drivers from their website (I even used the latest beta drivers). The lower end cards we tested we actually used Windows update to find the drivers and they worked fine, but in testing an R9 290, we couldn't get it to pass through without doing this (Windows update didn't work). Phew…I think that's enough for now, but I'll leave you with this: we are not stopping here. We have some exciting other developments that are on the horizon for Beta 6 that I think everyone will be excited about. Stay tuned and let us know what you've got working with what hardware so we can continue to add it to our "known configs" list. Thanks everyone!!
May 3, 201412 yr when do we get support for VMs in a different location other than /mnt/cache/domains some dont have cache drives, and I use a SSD outside of the array for trim support.... Thanks Myk
May 3, 201412 yr when do we get support for VMs in a different location other than /mnt/cache/domains some dont have cache drives, and I use a SSD outside of the array for trim support.... Thanks Myk I'm curious if there is someway to do some sort of mount -obind or do a mount -L DOMAINS /mnt/cache/domains onto /mnt/cache/domains so that it's automatic for people.
May 3, 201412 yr Author If you are comfortable with command line and building your own VMs, you can do this today by manually mounting a drive outside the array and using command line to create your VMs. Once you have the VMs created the way you like, you can add them to the web GUI for management by typing xenman register YOURXENCFGFILE.CFG from within the folder of the VM. The folder of the VM should be the same name as the .cfg name and the .img name for continuity. Adding capabilities to make this easier via the webGUI is on our roadmap.
May 3, 201412 yr Author To ellaborate, here's what you can do: from command line: lsblk This spits out all your drives. You should be able to identify your SSD by looking for it by size. Probably a lot smaller than your SATA drives. Now you want to mount the partition, not the drive, so it could be like this: cd /mnt mkdir foldername mount /dev/sda1 /mnt/foldername Voila, you have a mounted SSD outside of the array. Now I would tell you to do this via a GO script to automate this after login, but the only problem is that the letter designation could change. This will get easier as we integrate ease-of-use capabilities in future releases. We are working on a better way to do this that shouldn't require the use of an unRAID cache drive.
May 3, 201412 yr ok, I didnt know you could register VMs outside the cache drive and the best way to mount a SSD or other drive outside the array is as follows: #Mount SSD as appdisk mkdir /mnt/appdisk mount -rw -o noatime,nodiratime,discard -t ext4 /dev/disk/by-id/ata-Crucial_CT240M500SSD1_1332095464B2-part1 /mnt/appdisk then it dont matter what device it becomes Myk
May 3, 201412 yr Author ok, I didnt know you could register VMs outside the cache drive and the best way to mount a SSD or other drive outside the array is as follows: #Mount SSD as appdisk mkdir /mnt/appdisk mount -rw -o noatime,nodiratime,discard -t ext4 /dev/disk/by-id/ata-Crucial_CT240M500SSD1_1332095464B2-part1 /mnt/appdisk then it dont matter what device it becomes Myk Awesome! Thanks for sharing! I will start doing it that way. Much better I agree!! Sent from my Nexus 5 using Tapatalk
May 4, 201412 yr This config works for Windows 7 and Windows 8.1 with a single GPU pass through. The game is "less is more" with the config file and with memory assignment. For some odd reason, if you give the VM too much RAM, even on a very saturated host (ours had 32GB), the GPU passthrough will fail to function. We were using 64-bit VMs too, so no, it's not a 32-bit limitation thing. This config was tested with an AMD R7-250, R5-230, and some other really old AMD cards (I'll get the make/models up later). What I can tell you is that the higher end R9 290 cards haven't been so successful. While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. This is a work in progress for us right now. But if your goal isn't to achieve high-end gaming, the lower-end cards work totally fine. Even some of the oldest cards we grabbed from ATI/AMD back in the hay-day seemed to work ok. Jon, This sounds like what I saw here: http://lime-technology.com/forum/index.php?topic=33073.0 Adding more than 4GB of memory to a Windows 8.1 VM and adding more than 4 vCPU causes the VM to run so slow it is unusable. Thanks!
May 4, 201412 yr Author This config works for Windows 7 and Windows 8.1 with a single GPU pass through. The game is "less is more" with the config file and with memory assignment. For some odd reason, if you give the VM too much RAM, even on a very saturated host (ours had 32GB), the GPU passthrough will fail to function. We were using 64-bit VMs too, so no, it's not a 32-bit limitation thing. This config was tested with an AMD R7-250, R5-230, and some other really old AMD cards (I'll get the make/models up later). What I can tell you is that the higher end R9 290 cards haven't been so successful. While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. This is a work in progress for us right now. But if your goal isn't to achieve high-end gaming, the lower-end cards work totally fine. Even some of the oldest cards we grabbed from ATI/AMD back in the hay-day seemed to work ok. Jon, This sounds like what I saw here: http://lime-technology.com/forum/index.php?topic=33073.0 Adding more than 4GB of memory to a Windows 8.1 VM and adding more than 4 vCPU causes the VM to run so slow it is unusable. Thanks! Similar, but a little different. I was able to assign 8GB of RAM to a VM with no problem and I didn't see performance issues (so long as you have PV drivers installed). I also haven't tried giving a VM more than 4 vCPUs, so I can't say for certain that assigning more will bring it to a crawl, but that wouldn't surprise me. It's a mistake to see vCPU assignment as a "give it as much as you can and it goes faster" type thing. I actually try to keep my vCPU assignments to a minimum giving no more than 2 to any individual VM guest unless it's a gaming machine with GPU pass through (then I might give it 4). In fact, most headless VMs can probably get away with only 1 vCPU. The issue we saw with over-assignment of memory was specifically in regards to GPU pass through. If you assign more than 3GB (somewhere in between 3 and 4 is the magic #), the GPU pass through doesn't work. If you assign 3GB or less, it works fine. This applied to multiple VMs of multiple Windows operating systems.
May 4, 201412 yr I just happened to be poking through this thread, and while I'm sure some of you may have seen this in the past, figured I'd link it just in case: https://communities.vmware.com/thread/297072?start=0&tstart=0 The problems you're describing with gpu passthrough and larger amounts of assigned memory sound suspiciously familiar to issues I remember seeing when playing around with this on esxi a while back - that thread has a fair bit of empirical discussion, and I'm not even sure if any of the issues under VMware translate to Xen for this, but may provide some insight? (On the bright side, it was solvable on that hypervisor by playing with the pcihole parameter on VMware iirc)
May 4, 201412 yr While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. Have you been ejecting the GPU in the guest before shutdown / retart (or using IronicBadgers scripts to do so)? I've found this to be necessary on my 8.1 VMs using passthrough. Peter
May 5, 201412 yr I have a "lessons learned" with Xen (sorry if this was already common knowleadge). If your CPU does not support VT-x you can create and run a PV VM; however, you cannot create a HVM VM. I have tested this with 3 computers now. One of the computers had VT-x turned off in the bios and would not create a HVM until VT-x was enabled. The one computer does not support VT-x at all and will not create a HVM.
May 5, 201412 yr While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. Have you been ejecting the GPU in the guest before shutdown / retart (or using IronicBadgers scripts to do so)? I've found this to be necessary on my 8.1 VMs using passthrough. Peter I have been using the start/stop scripts and it has helped a ton. The other thing I sometimes do, is if nothing else works, I restart the entire unRAID server, resetting the video card and that does seem to help. It would be nice to find a way to reset the video card at the Xen level. Thanks, Marcus
May 7, 201412 yr While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. Have you been ejecting the GPU in the guest before shutdown / retart (or using IronicBadgers scripts to do so)? I've found this to be necessary on my 8.1 VMs using passthrough. Peter I have been using the start/stop scripts and it has helped a ton. The other thing I sometimes do, is if nothing else works, I restart the entire unRAID server, resetting the video card and that does seem to help. It would be nice to find a way to reset the video card at the Xen level. Thanks, Marcus This is my pet peeve. When a gpu pass through VM fails I have to reboot the whole server.
May 17, 201412 yr memory=2048 #DO NOT EXCEED UNLESS BRAVE IF DOING GPU PASS THROUGH <- this I've been running with 6GB memory from day one with gpu passthrough, no issues, ever *knocks on wood*
May 17, 201412 yr While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. Have you been ejecting the GPU in the guest before shutdown / retart (or using IronicBadgers scripts to do so)? I've found this to be necessary on my 8.1 VMs using passthrough. Peter I have been using the start/stop scripts and it has helped a ton. The other thing I sometimes do, is if nothing else works, I restart the entire unRAID server, resetting the video card and that does seem to help. It would be nice to find a way to reset the video card at the Xen level. Thanks, Marcus Its to do a function called FLR (function level reset). Basically means you're tricking the card into power cycling as if the box is rebooted. If you can find a way to do this, then you're golden. Sent from my Nexus 5 using Tapatalk
May 19, 201412 yr Author While we could get it to boot with pass through, reboots would crash the entire VM and Dom0. Have you been ejecting the GPU in the guest before shutdown / retart (or using IronicBadgers scripts to do so)? I've found this to be necessary on my 8.1 VMs using passthrough. Peter I have been using the start/stop scripts and it has helped a ton. The other thing I sometimes do, is if nothing else works, I restart the entire unRAID server, resetting the video card and that does seem to help. It would be nice to find a way to reset the video card at the Xen level. Thanks, Marcus Its to do a function called FLR (function level reset). Basically means you're tricking the card into power cycling as if the box is rebooted. If you can find a way to do this, then you're golden. Sent from my Nexus 5 using Tapatalk Just a heads up on this, that was my original thinking as well, but I have cards that do NOT have FLR yet are able to be rebooted without ejecting them first. Just lower end cards. FLR was a technology implemented a long time ago and not sure if that is specifically what is required in order to survive a reboot.
May 23, 201412 yr cd /mnt mkdir foldername mount /dev/sda1 /mnt/foldername Voila, you have a mounted SSD outside of the array. Now I would tell you to do this via a GO script to automate this after login, but the only problem is that the letter designation could change. This will get easier as we integrate ease-of-use capabilities in future releases. We are working on a better way to do this that shouldn't require the use of an unRAID cache drive. So, to be straight, this method would enable you to create a test USB stick of UNRAID6, with a separate external drive for the VMs, without having to buy a new registration key? I would like to test out the beta by extracting the existing USB drive and data disks and plugging in a new UNRAID 6 testing USB drive - but the cache disk requirement always prevented that. Seems that mounting a separate drive outside UNRAID is the solution. Oh, and is there anywhere here that people are talking about HP Microservers (N36L, N40L, N54L) and UNRAID 6. I know we can't get IOMMU (because the HP BIOS doesn't support it) but it would be useful to understand the support level - given how many people run UNRAID on these little boxes.
May 23, 201412 yr Author So, to be straight, this method would enable you to create a test USB stick of UNRAID6, with a separate external drive for the VMs, without having to buy a new registration key? What do you mean by "separate external drive for the VMs"? Let me put it this way. We are working on a completely improved domain manager for creating, importing, and managing VMs where you will be able to specify where you want the VM to be stored (it won't be forced to a cache drive). Oh, and is there anywhere here that people are talking about HP Microservers (N36L, N40L, N54L) and UNRAID 6. I know we can't get IOMMU (because the HP BIOS doesn't support it) but it would be useful to understand the support level - given how many people run UNRAID on these little boxes. Talking about what? I'm not sure I understand your question.
May 23, 201412 yr So, to be straight, this method would enable you to create a test USB stick of UNRAID6, with a separate external drive for the VMs, without having to buy a new registration key? What do you mean by "separate external drive for the VMs"? Let me put it this way. We are working on a completely improved domain manager for creating, importing, and managing VMs where you will be able to specify where you want the VM to be stored (it won't be forced to a cache drive). External to the UNRAID drives (data&cache), not the physical case. Basically are you able to test UNRAID 6 and VMs without having to purchase another registration key using the method of mounting you reference, at least during the testing? Oh, and is there anywhere here that people are talking about HP Microservers (N36L, N40L, N54L) and UNRAID 6. I know we can't get IOMMU (because the HP BIOS doesn't support it) but it would be useful to understand the support level - given how many people run UNRAID on these little boxes. Talking about what? I'm not sure I understand your question. [*]Lots of people run UNRAID on HP Microserver boxes - they are cheap, effective, etc. [*]However it appears that virtualisation with IOMMU is impossible, due to HP not supporting it in the BIOS. Other levels of virtualisation are workable (eg AMD-V rather than AMD-Vi) [*]Looking in this subforum, at the various threads, there seems to be little reference to Microservers, which seems strange. So the question is, is there some gotcha, some reason why it's not referenced in relation to UNRAID 6 - and how well will the two work together? Has anyone tested it? Assuming you are running it headless, is there even a need for IOMMU support? The eventual aim would be a collection of VMs onboard the headless server. Some (eg media serving) would be running all the time. Some (eg Windows VM) would only run occasionally, and would be accessed via VNC/TeamViewer etc. on a tablet.
May 24, 201412 yr Author So, to be straight, this method would enable you to create a test USB stick of UNRAID6, with a separate external drive for the VMs, without having to buy a new registration key? What do you mean by "separate external drive for the VMs"? Let me put it this way. We are working on a completely improved domain manager for creating, importing, and managing VMs where you will be able to specify where you want the VM to be stored (it won't be forced to a cache drive). External to the UNRAID drives (data&cache), not the physical case. Basically are you able to test UNRAID 6 and VMs without having to purchase another registration key using the method of mounting you reference, at least during the testing? Oh, and is there anywhere here that people are talking about HP Microservers (N36L, N40L, N54L) and UNRAID 6. I know we can't get IOMMU (because the HP BIOS doesn't support it) but it would be useful to understand the support level - given how many people run UNRAID on these little boxes. Talking about what? I'm not sure I understand your question. [*]Lots of people run UNRAID on HP Microserver boxes - they are cheap, effective, etc. [*]However it appears that virtualisation with IOMMU is impossible, due to HP not supporting it in the BIOS. Other levels of virtualisation are workable (eg AMD-V rather than AMD-Vi) [*]Looking in this subforum, at the various threads, there seems to be little reference to Microservers, which seems strange. So the question is, is there some gotcha, some reason why it's not referenced in relation to UNRAID 6 - and how well will the two work together? Has anyone tested it? Assuming you are running it headless, is there even a need for IOMMU support? The eventual aim would be a collection of VMs onboard the headless server. Some (eg media serving) would be running all the time. Some (eg Windows VM) would only run occasionally, and would be accessed via VNC/TeamViewer etc. on a tablet. OK, let's put it this way, if the hardware supports virtualization, you are good to go. Whether it is a "micro server" or something else shouldn't matter. Iommu is only required when you want to assign physical hardware resources to a VM. It is NOT required if you have an atypical need for vanilla virtualization (no hardware pass through). Now you may have to go into the BIOS on the box to turn on the virtualization tech, but that would be the only requirement. Sent from my Nexus 5 using Tapatalk
May 25, 201412 yr [*]Lots of people run UNRAID on HP Microserver boxes - they are cheap, effective, etc. [*]However it appears that virtualisation with IOMMU is impossible, due to HP not supporting it in the BIOS. Other levels of virtualisation are workable (eg AMD-V rather than AMD-Vi) [*]Looking in this subforum, at the various threads, there seems to be little reference to Microservers, which seems strange. So the question is, is there some gotcha, some reason why it's not referenced in relation to UNRAID 6 - and how well will the two work together? Has anyone tested it? Assuming you are running it headless, is there even a need for IOMMU support? The eventual aim would be a collection of VMs onboard the headless server. Some (eg media serving) would be running all the time. Some (eg Windows VM) would only run occasionally, and would be accessed via VNC/TeamViewer etc. on a tablet. I'm setting up Unraid6 on a HP N40L and so far it seems to work. At this point it is too slow to be useful since I couldn't use the SSD for the datastore (WHS2011 requires 160GB disk and my SSD isn't that big.) My end-goal was to run WHS2011 + MyMovies + Windows backups and use the Unraid disks as storage. I also wanted to setup Crashplan on the Unraid side so the box can be installed at the in-laws house as a backup site. Unraid 6 Pro Key 4GB memory 2TB + 1.5TB drvies for testing. 128GB SSD for cache drive. Questions I still have. - Can I mount a disk other than the cache drive for the datastore? - CAn I configure/present a USB drive to the WHS2011 server for Windows backups? - Should I just skip Unraid6 configure RAID5 + WHS2011 and be done? I'm just playing at this point so if you want me to try something I'm willing to give it a shot. I'm no Xen expert - or even a novice - but I usually can figure things out. Brian
May 26, 201412 yr OK, let's put it this way, if the hardware supports virtualization, you are good to go. Whether it is a "micro server" or something else shouldn't matter. Iommu is only required when you want to assign physical hardware resources to a VM. It is NOT required if you have an atypical need for vanilla virtualization (no hardware pass through). Now you may have to go into the BIOS on the box to turn on the virtualization tech, but that would be the only requirement. Thanks for that. I wasn't sure if there were other hardware passthroughs that might make a difference/be required (network, USB). However in a spirit of suck it and see, I've gone ahead a created a test setup to see how things work out. I've got as far as creating and registering the arch VM with relatively little trouble - once I've got a bit further I might bring it all together into a posting for those in a similar situation.
May 26, 201412 yr I'm setting up Unraid6 on a HP N40L and so far it seems to work. At this point it is too slow to be useful since I couldn't use the SSD for the datastore (WHS2011 requires 160GB disk and my SSD isn't that big.) My end-goal was to run WHS2011 + MyMovies + Windows backups and use the Unraid disks as storage. I also wanted to setup Crashplan on the Unraid side so the box can be installed at the in-laws house as a backup site. Unraid 6 Pro Key 4GB memory 2TB + 1.5TB drvies for testing. 128GB SSD for cache drive. Questions I still have. - Can I mount a disk other than the cache drive for the datastore? I can confirm that I have the SSD formatted as EXT4 (with TRIM support) outside the array/UNRAID, holding the VMs and registered OK with UNRAID. Since I don't need the cache drive, I can do this with just the free licence and a temp data disk for the array - making the testing phase easier. - CAn I configure/present a USB drive to the WHS2011 server for Windows backups? - Should I just skip Unraid6 configure RAID5 + WHS2011 and be done? I'm just playing at this point so if you want me to try something I'm willing to give it a shot. I'm no Xen expert - or even a novice - but I usually can figure things out. Brian Does WHS2011 actually require 160GB? I think it can actually install into 13GB (for testing purposes at least) see http://forum.wegotserved.com/index.php/topic/22356-whs-2011-160gig-hdd-minimum-workaround/ I'd also say that you can look at Bittorrent Sync as a potential alternative to Crashplan.
Archived
This topic is now archived and is closed to further replies.