1812

Look in tools>devices at your iommu groups. Anything that is in group 21 has to also be passed through. This could include 2 of the 4 ports on one Ethernet card. if you're unsure, post your complete diagnostics zip tools>diagnostics and we'll look for you.

It's there under one of the menus/sub menus. Like hidden in queue or something. I did that once and went looking for it, found it, but didn't have the problem again.

I've never done what you're trying to do. I instead use a 4 port nic and send that to pfsense. BUT This is how you change your xml to the the other virtual adapter: click on the red square button on the pfsense vm icon when it is stopped. On the dropdown menu, select edit xml. scroll down to the section that looks similar to this: <interface type='bridge'> <mac address='52:54:00:82:25:11'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> note: your mac address and address will be different. where it lists "model type=" change to <model type='e1000-82545em'/> make sure source birdge is "br0" scroll down further, click save. NOTE: you can no longer edit your vm using the vm manager as custom edits will then be discarded and you will lose the e1000. Don't be surprised that when you attempt to boot the server, and it is looking for the pfsense vm for network connectivity, that it can take a while. Docker runs an update check at boot. And if there is no network detected, each docker has to go through a preset timeout which is somewhere between 100 and 300 seconds if I remember correctly. This is on every autostart docker in order. This is before the vm's are autostarted. PRO TIP: set your unRaid server on a static ip. it's easier to find it after setting up pfsense and running the firewall on the same box. Good luck.

Pci bridges are never fun to work around. I have never directly had the issue but a couple folks on here have. First, turn acs override off. then go to syslinux.cfg and try adding the device id of the pci bridge to the two from the video card and reboot. it probably won't fix it, but worth trying. When that doesn't work, Look in your bios for an SR-IOV function. if disabled, enable and recheck iommu groupings. If that doesn't do it, then re-read the HP solution posted above and see if that is applicable to your server. And if none of that works, we may get creative and try something else. What version of unRaid are you on and what bios and date are you on?

This thread is a work in progress. Updated information will be added at the top in this post. Feel free to ask questions or post further problems in this thread. Workstation information is near the bottom. Below are common problems associated with HP servers, and where available, known working fixes. I am not a certified IT technician. The information below was either discovered by scouring the Internet and/or trial and error by myself and several others on this board. Utilize this information at your own risk, though I have done almost everything discussed below. UnRaid Versions Proliants (and HP servers) should run 6.2.4 for the time being. 6.3 caused issues with CPU core numbering and other mild problems. This was due to an update in the Linux kernel or something stupid. Preliminary tests on the 6.4 release show that most if not all the problem from 6.3 have been resolved. But I would wait to update to 6.4 until the official stable release comes out. Proliants (and HP servers) should run 6.4. Many previous issues have been resolved from 6.3.x with a few minor things still prevalent. 6.5.1+ are stable and recommended. As new versions of unRaid are produced, I will update this section if there are any known issues, otherwise assume latest available works. HP Raid Controllers (P410i) Most HP raid controllers can not do JBOD. If you intend to use one, you have to use the raid controlle to create Raid-0 volumes and those are then presented to unRaid. Recovery of data then becomes more difficult. Alternatives include using a pci-e host bus adapter that works in unraid (consult the hardware wiki for working cards.) I currently use an H220, but any compatible one will do. - UPDATE July 2018- I ran into issues using H220 HBA with HP SAS Expnders on my MD1000 (which is just a box with expanders) When using unRaid ver. 6.3.x. I swapped out several H220's and sas expanders to no avail. I also attempted to to a direct connection with and 8087 to 8088 connection. It would see the enclosure but not the disks. I then swapped in an H310 in IT mode which worked using the connection adapter. I an currently running that. 2018: I determined the fault was in the md1000 enclosure. I currently run H220 adapters in 2 different servers without issues, including using an hp expander. External connectivity can be gained via an HP SAS expander or via 8087 to 8088 adapters. HP PCI-E raid cards (P212, P411) [this section untested under 6.4 and above, relevant to 6.2.4] Under certain Bios, unraid will not play nice with some HP PCI-E raid cards. This includes unRaid hanging on boot and experiencing lockup failures on the controller. If you intend to use an HP pci-e raid card, then you must update to the latest bios available on the server. BUT doing this may lead to other issues, including "Device is ineligible for IOMMU domain attach due to platform RMRR" problems. Thread on HP raid cards: -UPDATE- I was working on setting up a temporary server (DL120 G6) using an P212 and H220, just to see if it would work. The server booted on 6.2.4, 6.3.2, and 6.3.5 and showed drives on both controllers. When I attempted to setup a vm using passthrough, I modified the syslinux.cfg as required to include: append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot From that point on, I could not get the server to complete loading unraid and kept getting errors (call traces/lockups/etc.) I gave up on combining the cards in that server and moved them to a Dell T300 that doesn't have hardware passthrough. It again booted successfully on 6.2.4, 6.3.2, and 6.3.5. So the issue does not appear to be with the cards and unraid, so much as it is an issue of needing to allow unsafe interrupts for hardware passthrough causing the issue with the cards. This was verified when I re-added the unsafe interrupts line to the syslinux.cfg and the errors returned. HP Networking Integrated gigabit ethernet seems to work in 6.2.4 when the port(s) are directly attached to the motherboard. If the port(s) are located on a riser card, drivers were implemented beginning in 6.3.x to allow functionality. VM problems - pass-through "internal error: process exited while connecting to monitor/vfio: failed to setup container for group 1/vfio: failed to get group 1/Device initialization failed" Step 1 Typically using PCIE-ACS override does not fix gpu pass-through issues. But regardless, the first step is to ensure that your video card and audio component are in their own IOMMU group. You can view the groups at Tools>System Devices, then search for the maker of your GPU. For example, my 1050 is: 09:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10) 12:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1) 12:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb9] (rev a1) 80:00.0 PCI bridge [0604]: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 [8086:3420] (rev 22) This shows both the GPU portion and audio in set as 12:00.0 & 12:00.1. Then scroll down to the IOMMU group and verify that only those 2 device are listed their own group. /sys/kernel/iommu_groups/16/devices/0000:04:00.3 /sys/kernel/iommu_groups/17/devices/0000:12:00.0 /sys/kernel/iommu_groups/17/devices/0000:12:00.1 /sys/kernel/iommu_groups/18/devices/0000:09:00.0 In my example, they are the only 2 devices in group 17, the gpu and audio portion. Your group number and device assignment may be different. If your gpu/audio are in their own IOMMU group, SKIP to STEP 3. Step 2 If you have other devices in the same IOMMU group as your GPU, you will either need to also pass those through, or isolate your GPU/Audio into a separate group. You can try enabling PCIE-ACS override and rebooting, and rechecking. If that does not work, then disable acs override, and proceed to Step 2a: Step 2a Obtain the unique device id's of the GPU and audio component. You can view find it at Tools>System Devices, then search for the maker of your GPU. For example, my 1050 is: 09:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10) 12:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1) 12:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb9] (rev a1) 80:00.0 PCI bridge [0604]: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 [8086:3420] (rev 22) The unique id is the 8 number at the end of the device in brackets. For my 1050, the GPU is, 10de:1c81 and the audio is 10de:0fb9. Once you have obtained your id's, you will need to edit your syslinux.cfg file. To do this, go to Main>Boot Device>Flash. Under Syslinux Configuration, edit the text there for the first "append initrd=/bzroot" to the following (XXXX:XXXX is your GPU id, YYYY:YYYY is your audio component): append pcie_acs_override=id:XXXX:XXXX,YYYY:YYYY initrd=/bzroot My complete edited syslinux.cfg file looks like: default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append pcie_acs_override=10de:1c81,10de:0fb9 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzim3age append initrd=/bzroot unraidsafemode label unRAID OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest Save and reboot. Verify that the GPU and audio component are either in their own IOMMU group, or more likely, they are each in their own independent groups (2 groups total with a single device in each.) You can try your pass-through now, but it probably won't work. Step 3 The server needs to allow "unsafe" interrupts. This is achieved by modifying the syslinux.cfg file. To do this, go to Main>Boot Device>Flash. Under Syslinux Configuration, edit the text there for the first "append initrd=/bzroot" to the following: append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot The full syslinux.cfg would then look like: default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest OR if you had to follow step 2, it would look like this default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append pcie_acs_override=10de:1c81,10de:0fb9 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label unRAID OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest Save and reboot. You can now try your pass-through. If the vm shows that it started on the dashboard, you're good to go. If you don't have output or distorted video, that's a non-related issue. IF if doesn't work, go to step 4. Step 4 After you have reverified that your gpu and audio component are in their own IOMMU group(s), examine your system logs after a failure to start a vm. Tools>System Log. Look at the bottom or search using your web browser for "Device is ineligible for IOMMU domain attach due to platform RMRR" If it is there, it is most likely likely your audio component listed as the problem. There are a couple ways to try to resolve it, but that depends on your hardware. According to HP, the RMRR problem stems from an upgrade to Linux above kernel version 3.16. There are discussions around the Internet about an unofficial path but with varying results. HP has a published sheet on this, with a fix for some: https://h20565.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=7271259&docId=emr_na-c04781229&docLocale=en_US If you are on Gen 8+ hardware, there is a bios fix: https://docs.hpcloud.com/hos-4.x/helion/networking/enabling_pcipt_on_gen9.html If you are pre-Gen 8 or the previous link does not work/apply you have 3 options: 1. Roll back to a previous bios. During boot, access the bios menu and select an older bios if available. It appears that bios versions that are 2011 and before do not have the RMRR issue, and pass-through of both video and audio are able to occur. 2. If you are 2012 or newer: Look online for an older bios and HP procedure to roll it back. If that is not an option, then: 3. Don't pass through the audio component of the video card. This can be achieved by following Step 2a. Then in the vm manager, only pass-through the video component to the vm, leaving out the ineligible audio device. You can get audio in your vm by utilizing a usb audio device. Not elegant, but functional. If using acs override for just the card doesn't separate out the gpu and its audio component, then try using the following acs setting instead: pcie_acs_override=multifunction 4. You may also omit the previous 3 steps and attempt to use the unRaid Proliant Edition patch located here: Further reading: Bios - Firmware Updates The 2017.04.0 SPP is the last production SPP to contain components for the G7 and Gen8 server platforms. For additional information, please refer to Reducing Server Updates. http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3f6b4074ed734dc3baf007612d#tab5 At the time of writing this info, this was true, but it appears they have issues some other updates along the way, probably for mitigations or similar. I don't follow them anymore for this older hardware, so you'll have to google it for yourself to find the latest for your particular machine. Currently there is no reason to update the server rom to a "newer" bios, but you can update controller drivers/etc using the SPP. Do not use the auto installer as it will update the server rom and introduce issues outlined above. After the system runs a scan, you can select to update your ethernet/raid/etc which have shown no unwanted effects in unRaid so far. HP ProLiant DL120 G6/ProLiant DL120 G6, BIOS O26 07/01/2013 works on 6.2.4/6.4 with pass through. ACPI Error Messages Some servers complain with an error message similar to the following: Oct 28 23:06:10 Tower kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170531/exfield-427) Oct 28 23:06:10 Tower kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170531/psparse-550) Oct 28 23:06:10 Tower kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170531/power_meter-338) Bug details are here: https://community.hpe.com/t5/ProLiant-Servers-Netservers/ACPI-Error-SMBus-or-IPMI-write-requires-Buffer-of-length-66/td-p/6943959 This is essentially a misreading and does not affect anything but spamming your system log. To remove, disable "acpi_power_meter" module from the kernel by adding this line to the /boot/config/go file: rmmod acpi_power_meter HT to @perfecblue for this fix. Further reading can be done here: https://lime-technology.com/forums/topic/59375-hp-proliant-unraid-information-thread/?do=findComment&comment=634360 HPZ420 workstation info While not technically a proliant, I picked one of these up recently and here are the important discoveries: You do not need the RMRR patch If attempting to use onboard sata ports, you must use UEFI booting with unRaid. Otherwise it just sits there and blinks the cursor at you regardless of any specified boot order (3 hours of my time on that one....) Would be nice to find a fix, as per below, legacy boot is needed for some stability. I was also getting unexpected hard server resets and the following: 928- fatal pcie error pcie error deteted slot 5 completion timeout This is on a known good GPU. It would occur when shutting down a vm the GPU was assigned to. Multiple places on the Internet said possible bad board, bios error, etc. I did change some bios settings but it still happened. Only after I applied the unsafe interrupts fix from above did it stop. CPU 0 stuck at 100% usage TLDR: add the following to your syslinux.cfg acpi=force have time to read: https://lime-technology.com/forums/topic/73537-cup-0-100-on-2-different-usb-licenses/?tab=comments#comment-676484 Weird VM issues I started getting weird crashes and inability to launch vm's when the server was booting into UEFI. Returning to Legacy appears to solve the issue. I'll retest in subsequent versions of unRaid to determine if it persists (last checked 6.5.3)

I have mine in a basement/garage. My servers are there in a 2x4 constructed, plastic wrapped clean room with filtered air. Basement gets to the 50's in winter (while being 20's outside) and 70's in the summer (when it's 90's outside.) Cost about 100 buck to build, but worth it to move the noise and heat out of the house.

Because windows 10 won't recognize more than 2 processors (need windows server for that) and apple never made a 4 processor, 48 core computer that can be upgraded to 80 cores.

One server hosts a couple vm's with physical GPU outputs to hdmi/usb over cat5e (one does photoshop editing,) pfsense firewall for the network, data storage, duckdns, managing backups via crashplan, and plex transcoding. Another one hosts a single vm for doing dedicated 4k editing/transcoding and backups. Another does off site backups with a vm for remote management.

If you want the ability to spin down the drives and access smart data, and have the ability to move them around "more" freely, then you need something like an h220 with the sas expander. This will allow you to access the drives JBOD vs. in a raid 0. If you only get the expander, then you're only creating more access ports for the P410i and it's raid configurations. I use an h220 with an sas expander. It's less difficult to move disks vs raid-0 disks run through the P410i. I'll be moving an unRaid setup from a dl380 g6 to a 580 g7 in the coming days, using the h220&expander. All I need to do is keep track of disk locations and if they "move" on the first boot, correct the location/order.

I'm assuming your other more successful builds were not Ryzen. Have you read the Ryzen thread(s) yet?

Thanks, I was confused.

Just downloaded. Gather is listed as coming soon, but described on first page of this thread. Is this part working or not?

Rebooting the serve has allowed the prelcear to start. Not sure why, but, ok.

attempted to clear 2 new discs. Neither will go past starting. most recent plugin logs look like this: Sat Jul 1 11:17:42 PDT 2017: PHP Strict Notice: Non-static method Misc::get_json() should not be called statically in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 104 Sat Jul 1 11:17:52 PDT 2017: PHP Strict Notice: Non-static method Misc::get_json() should not be called statically in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 104 Sat Jul 1 11:18:02 PDT 2017: PHP Strict Notice: Non-static method Misc::get_json() should not be called statically in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 104 Sat Jul 1 11:42:44 PDT 2017: PHP Warning: unlink(/tmp/preclear_stat_sdh): No such file or directory in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 291 Sat Jul 1 11:42:44 PDT 2017: PHP Warning: unlink(/tmp/preclear_stat_sdi): No such file or directory in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 291 Tried uninstalling/reinstalling, no change. Running 6.2.4, tried with safari/chrome. server logs at reinstall Jul 1 11:58:35 Brahms3 emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin install https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/plugins/preclear.disk.plg Jul 1 11:58:35 Brahms3 root: plugin: creating: /boot/config/plugins/preclear.disk/preclear.disk-2017.06.30b.txz - downloading from URL https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/archive/preclear.disk-2017.06.30b.txz Jul 1 11:58:35 Brahms3 root: plugin: creating: /boot/config/plugins/preclear.disk/preclear.disk-2017.06.30b.md5 - downloading from URL https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/archive/preclear.disk-2017.06.30b.md5 Jul 1 11:58:35 Brahms3 root: plugin: skipping: /boot/readvz already exists Jul 1 11:58:35 Brahms3 root: plugin: setting: /boot/readvz - mode to 755 Jul 1 11:58:35 Brahms3 root: plugin: creating: /boot/config/plugins/preclear.disk/StatsInstall - downloading from URL https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/source/statistics.sender/StatsInstall.page Jul 1 11:58:35 Brahms3 root: plugin: setting: /boot/config/plugins/preclear.disk/StatsInstall - mode to 755 Jul 1 11:58:35 Brahms3 root: plugin: running: anonymous Jul 1 11:58:42 Brahms3 root: plugin: running: anonymous Jul 1 11:58:42 Brahms3 rc.diskinfo[12817]: killing daemon with PID [3922] Jul 1 11:58:42 Brahms3 rc.diskinfo[12824]: process started. To terminate it, type: rc.diskinfo --quit Jul 1 12:04:09 Brahms3 rc.diskinfo[25737]: process started. To terminate it, type: rc.diskinfo --quit Plugin also is making the GUI hang. I can't get to the diagnostics window to post those.

Looks like potentially a great find.

This is a great find. Performs much better for me than teamviewer or splashtop.

Link to thread?

For guaranteed freshness? I set it to auto-update and forget about it.

Splashtop and teamviewer use lots of cpu. It's how it goes.

Had to use krusader to move the nvram back, but all is good now. Looking forward to the automated version in the future!

Awesome! Great things come out of my failures!

If it's not including in in this backup/restore program, then no. Unless it is stored somewhere in the array, then I have a backup via crash plan. Edit- if it's the img located in the system file, it was not a part of my crash plan backup for some reason....

I had a cache drive bite it today. Thankfully I had this great backup already setup. I followed the steps listed and got all the dockers back within minutes. But I'm not sure how to restore my vm data which is backed up in appdata. The images are still in place becase they were on unassigned disks. I could recreate the xml, but is here an easier way I'm just overlooking?

It might work, not sure. You could always run the disks as cache disks in raid10 and get good speed and redundancy without worrying about another raid controller. iscsi mightwork but I don't know anyone doing it. Doesn't mean it isn't done, just means I can't help you there.

Of course it's going to be slow on an external share. Smb/nfs/etc network communication protocols are slower in terms of latency required for running an operating system than using a single disc that is not part of the array. I played around with running vm images via smb/nfs for fun. Mostly un-useable even when running over 10gbe direct attach copper using smb/nfs to a remote ssd. the only way you can have super fast network drive images is with a San setup of some sort, allowing block level access to the disks. be aware that even when you use a spinning disk for Promox, it will be limited by the io of that single disc. And it will only get slower the more vm's you add to it. This is why many people use unassigned ssd drives for vm's. you can run a couple with little noticeable slowdown if they aren't all demanding maximum disk bandwidth.