Please Help I want to get my unRaid going full board with more than just NASing and Plexing 

 

Issue 1) I have an error with no real description and as such, I don't know if it really matters and my research into it has indicated an xfs fix or memtest may resolve it as would a bios change, but I don't even know where my exact error is seen below and I will attach my tower diagnostics if one of you Jedi Masters could help this young patawan is not sure which way to go. 

 

Call Traces found on your server

 

Issue 2)    VM creation error

internal error: qemu unexpectedly closed the monitor: 2017-09-03T23:01:15.828044Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to set iommu for container: Operation not permitted
2017-09-03T23:01:15.828070Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to setup container for group 6
2017-09-03T23:01:15.828074Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to get group 6
2017-09-03T23:01:15.828088Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: Device initialization failed

 

Issue 3)  Deluge VPN (binhex) and PIA Subscription.  Cannot enter web gui.  .ovpn file in \\TOWER\appdata\binhex-delugevpn\openvpn

 

2017-09-03 19:03:03,879 DEBG 'start-script' stdout output:
[info] Starting OpenVPN...

2017-09-03 19:03:04,048 DEBG 'start-script' stdout output:
Options error: --ca fails with 'ca.rsa.2048.crt': No such file or directory

2017-09-03 19:03:04,049 DEBG 'start-script' stdout output:
Options error: --crl-verify fails with 'crl.rsa.2048.pem': No such file or directory
Sun Sep 3 19:03:04 2017 WARNING: file 'credentials.conf' is group or others accessible
Options error: Please correct these errors.
Use --help for more information.

2017-09-03 19:03:04,049 DEBG 'start-script' stdout output:
[info] OpenVPN started

 

disk.cfg

docker.cfg

domain.cfg

go

ident.cfg

network.cfg

share.cfg

super.dat

docker.txt

libvirt.txt

syslog.txt

no qemu log files

a-----a.cfg

d-----s.cfg

D--a.cfg

i--s.cfg

M---a.cfg

s----m.cfg

Hitachi_HDS721010CLA632_JP2940J81BMYMV-20170903-1906.txt

Hitachi_HDS723020BLA642_MN1220F31U2GRD-20170903-1906.txt

SanDisk_Cruzer_Glide_4C530101500513113131-0-0-20170903-1906.txt

ST1000DM003-1CH162_S1D6QGT7-20170903-1906.txt

ST31000528AS_9VP766AJ-20170903-1906.txt

cmdline.txt

df.txt

ethtool.txt

folders.txt

ifconfig.txt

iommu_groups.txt

lscpu.txt

lsmod.txt

lsof.txt

lspci.txt

lsscsi.txt

lsusb.txt

memory.txt

ps.txt

vars.txt

unRAID-6.3.5.txt

tower-diagnostics-20170903-1906.zip

Edited September 4, 20178 yr by 88pockets

Ok I have made some headway.

 

DelugeVPN - Requires OpenVPN-as docker to be installed and configured, right?

 

I can get the VMs running if I do not try to pass through audio, if anyone can help with that it would be greatly appreciated?

 

Now I'm having a hell of a time getting openVPN setup and working as I don't think my ATT Uverse Gateway (Pace 5268ac) is actually forwarding the port to my unRaid box (UDP 1194)

 

step one: update your bios

 

------------[ cut here ]------------
Sep  3 18:18:22 Tower kernel: WARNING: CPU: 2 PID: 1 at drivers/iommu/intel-iommu.c:5480 intel_iommu_init+0xb99/0x1107
Sep  3 18:18:22 Tower kernel: Your BIOS is broken; DMA routed to ISOCH DMAR unit but no TLB space.
Sep  3 18:18:22 Tower kernel: BIOS vendor: American Megatrends Inc.; Ver: 1408   ; Product Version: System Version
Sep  3 18:18:22 Tower kernel: Modules linked in:
Sep  3 18:18:22 Tower kernel: CPU: 2 PID: 1 Comm: swapper/0 Tainted: G          I     4.9.30-unRAID #1
Sep  3 18:18:22 Tower kernel: Hardware name: System manufacturer System Product Name/P6T, BIOS 1408    09/21/2010
Sep  3 18:18:22 Tower kernel: ffffc900018abd18 ffffffff813a4a1b ffffc900018abd68 ffffffff8198a035
Sep  3 18:18:22 Tower kernel: ffffc900018abd58 ffffffff8104d0d9 00001568018abdd0 ffff880273d55000
Sep  3 18:18:22 Tower kernel: ffffffff81c792c0 0000000000013680 ffff8802765d3680 0000000000000000
Sep  3 18:18:22 Tower kernel: Call Trace:
Sep  3 18:18:22 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Sep  3 18:18:22 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Sep  3 18:18:22 Tower kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e
Sep  3 18:18:22 Tower kernel: [<ffffffff81d0332b>] intel_iommu_init+0xb99/0x1107
Sep  3 18:18:22 Tower kernel: [<ffffffff810ca7a8>] ? free_reserved_area+0xe9/0xf9
Sep  3 18:18:22 Tower kernel: [<ffffffff81cd8965>] ? memblock_find_dma_reserve+0x147/0x147
Sep  3 18:18:22 Tower kernel: [<ffffffff81cd8977>] pci_iommu_init+0x12/0x3c
Sep  3 18:18:22 Tower kernel: [<ffffffff81002171>] do_one_initcall+0x8a/0x115
Sep  3 18:18:22 Tower kernel: [<ffffffff81ccbffc>] kernel_init_freeable+0x161/0x1e2
Sep  3 18:18:22 Tower kernel: [<ffffffff816746db>] ? rest_init+0x82/0x82
Sep  3 18:18:22 Tower kernel: [<ffffffff816746e4>] kernel_init+0x9/0xf0
Sep  3 18:18:22 Tower kernel: [<ffffffff8167f785>] ret_from_fork+0x25/0x30
Sep  3 18:18:22 Tower kernel: ---[ end trace 0acd21f04f6b8bd1 ]---

 

This is causing issues with visualization, it would appear. Similar Call traces referencing IOMMU occur everal times.

i went to update the bios but it is already on the latest version

 

Might I Have issues because I swapped the unRAID USB from one build to another.  Same Drives (Array) but different Mobo CPU and RAM

34 minutes ago, 88pockets said:

i went to update the bios but it is already on the latest version

 

Might I Have issues because I swapped the unRAID USB from one build to another.  Same Drives (Array) but different Mobo CPU and RAM

 

Unlikely, I've swapped entirely different server out with the same usb and drives and had no issues because unraid reinstalls every boot. It doesn't remember how many cores, what mobo, or how much ram you have. 

If your most recent bios available is 2010, then your hardware may not properly support IOMMU and you may be SOL unless you find a workaround.

well firstly I didnt even know of IOMMU until I started the VMs on the unRAID box.  They do load but are missing the correct commands to get any hardware through to the VM.  For instance I can load the VMs as long as it is to a VNC Client and I dont attempt to bing the audio controller along as well.  

 

I feel like the mobo should support IOMMU as the unRAID System Profiler shows it as enabled.  That would be on the board or in the CPU instructions?

 

If I can't pass that sound card through it's not a terribly big deal, I can attempt to add one, down the road, as a PCI/PCI-e add in card. It does let the keyboard and mouse through (logitech unified usb reciever)

 

I have a whole nother set of problems getting openVPN to work with my pretty locked down primary Gateway from ATT (Pace 5268ac) The thing doesnt actually forward ports and DMZ+ changes my unRAID Local IP to my Public IP and I feel thats a security concern.  I have another Router behind it that I am going to attempt to set up in DMZ+ mode and then set on an alternate subnet (192.168.2.0/24) while keeping the ATT gateway on 192.168.1.0/24 (its default).  Hopefully that'll let my box communicate with the outside world (in particular) noip.com.

 

Please let me know if that enabled in system profiler really means anything.  I have reflashed the same BIOS, but I am not sure to what avail quite yet.  Thanks for your help.

heres a fresh set of diagnostic files since the BIOS reflash

tower-diagnostics-20170904-1525.zip

18 minutes ago, 88pockets said:

 

 

 

I have a whole nother set of problems getting openVPN to work with my pretty locked down primary Gateway from ATT (Pace 5268ac) The thing doesnt actually forward ports and DMZ+ changes my unRAID Local IP to my Public IP and I feel thats a security concern.  I have another Router behind it that I am going to attempt to set up in DMZ+ mode and then set on an alternate subnet (192.168.2.0/24) while keeping the ATT gateway on 192.168.1.0/24 (its default).  Hopefully that'll let my box communicate with the outside world (in particular) noip.com.

 

Please let me know if that enabled in system profiler really means anything.  I have reflashed the same BIOS, but I am not sure to what avail quite yet.  Thanks for your help.

 

 

I don't know about DelugeVPN, but if you want pfsense as a firewall/router for your network/server, then you're going to have to put your gateway into bridge mode or in the DMZ setting you've talked about. This means that the first device exposed to the internet will get that public IP (and yes, putting unraid directly on the internet is not a good idea.

 

The pfsense vm will then be your new network router assigning ip addresses. I run pfsense as my router. There are pluses and minuses. You might consider getting a cheap 20 dollar computer off craigslist (or an old one lying around,) add a 10 dollar pcie etherent card, and setting up a standalone pfsense install first before going for fully integrated.

 

Your ISP may only provision you 1 ip per gateway, so something to consider.

 

 

 

 

29 minutes ago, 88pockets said:

well firstly I didnt even know of IOMMU until I started the VMs on the unRAID box.  They do load but are missing the correct commands to get any hardware through to the VM.  For instance I can load the VMs as long as it is to a VNC Client and I dont attempt to bing the audio controller along as well.  

 

I feel like the mobo should support IOMMU as the unRAID System Profiler shows it as enabled.  That would be on the board or in the CPU instructions?

 

If I can't pass that sound card through it's not a terribly big deal, I can attempt to add one, down the road, as a PCI/PCI-e add in card. It does let the keyboard and mouse through (logitech unified usb reciever)

 

 

 

IOMMU deals with device mapping. So if your vm has no devices passed to it, then no mapping is needed.

 

are you trying to passthrough an onboard sound source? If so, i believe there are certain steps you need to take. The first maybe the following: read below about modifying your syslinux.cfg for a possible "fix." 

 

 

 

 

 

Interestingly enough I found that same thread and I just made the suggestion toward the bottom of the thread to change the syslinux config to include

 

"append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot"

 

and no error starting the VM and I actually boot into the windows 8.1 (embedded_industry_pro) (good version of 8.1 as it is very resource friendly, like 2 gb of ram used towal with chrome and multiple tabs running). Before even when I setup the VM to not include the audio card, I got nowhere, it would boot to a UEFI command prompt rather than loading the Win 8.1 Iso.

 

This may be solved, but I am not sure yet as the thread you linked to, mentions using the VM in IDE mode rather than using VIRTIO.  So I need to play around and see how this VM runs.

 

As far as the pfSense suggsetion, i have an older q6600 build that I would need to setup with an extra NIC to go from the gateway > DMZ to pfSense box > Switch or router with DHCP forwarding rather than DCHP Serving > unraid box and other devices on the network.  Am I on track with how that would be setup?

 

Seems like a lot of machines running, so I'd prefer a pfSense VM rather than the power draw of three machines (unRAID box, pfSense box, main computer) at any given time, but I do recognize the extra headaches in figuring out the VLANs and internal port forwarding to get pfSense running properly as a VM.  I have a Linksys WRT1200AC running LEDE (Open-WRT derivative) is there any way to DMZ that through the gateway and run it's firewall rather than pfSense and plug unRaid through the Linksys LEDE router to connect to the web bypassing the ATT Gateway in a sense?  Argh... so complicated just to play around with some VMs and torrent through my VPN on my myRAID box.  Whatever... I want to learn more so all of the projects serve a higher purpose than torrenting.. i guess.

20 minutes ago, 88pockets said:

 

As far as the pfSense suggsetion, i have an older q6600 build that I would need to setup with an extra NIC to go from the gateway > DMZ to pfSense box > Switch or router with DHCP forwarding rather than DCHP Serving > unraid box and other devices on the network.  Am I on track with how that would be setup?

 

Seems like a lot of machines running, so I'd prefer a pfSense VM rather than the power draw of three machines (unRAID box, pfSense box, main computer) at any given time, but I do recognize the extra headaches in figuring out the VLANs and internal port forwarding to get pfSense running properly as a VM.  I have a Linksys WRT1200AC running LEDE (Open-WRT derivative) is there any way to DMZ that through the gateway and run it's firewall rather than pfSense and plug unRaid through the Linksys LEDE router to connect to the web bypassing the ATT Gateway in a sense?  Argh... so complicated just to play around with some VMs and torrent through my VPN on my myRAID box.  Whatever... I want to learn more so all of the projects serve a higher purpose than torrenting.. i guess.

 

 

It's easier to start with more machines first, figure out how to do it all, and then condense. Trust me. Been there. Done that. What you've outlined with the q6600 would work as a starting point. I couldn't tell you about the options on the linksys, you'd have to see but my guess is yes. But I'd recommend doing it with pfsense, learning how to get it setup with what you need, then you can just redo that on the vm when ready, and eliminate half your headaches.

 

Once pfsense is integrated into unRaid, you'll learn about issues when rebooting the server, having a it searching for DHCP/internet gateway/etc, n it's not up because it's going to take 2-10 minutes to launch pfsense depending on how many dockers you have that need to timeout searching for updates...etc... and will have to decide if you want a multiport ethernet card to passthrough to unraid, which then dumps back out to a switch, which then returns to your unraid box, or if you want to go the (what some consider less secure) route of having one line in for pfsense, then having a virtual nic bridge to the machine via br0.

 

 

A year ago I started 3 machines. I took my time to learn and read. Now my server runs 3 vm's with their own GPU's, runs pfsense firewall for the house, manages local and remote backups to other servers,internal disks and external array, transcodes 3-4 plex movies, and more, all at the same time.

 

20 minutes ago, 88pockets said:

Interestingly enough I found that same thread and I just made the suggestion toward the bottom of the thread to change the syslinux config to include

 

"append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot"

 

and no error starting the VM and I actually boot into the windows 8.1 (embedded_industry_pro) (good version of 8.1 as it is very resource friendly, like 2 gb of ram used towal with chrome and multiple tabs running). Before even when I setup the VM to not include the audio card, I got nowhere, it would boot to a UEFI command prompt rather than loading the Win 8.1 Iso.

 

This may be solved, but I am not sure yet as the thread you linked to, mentions using the VM in IDE mode rather than using VIRTIO.  So I need to play around and see how this VM runs.

 

 

 

Interesting that the remapping interrupts modification fixed this issue. But as long as it's working, there you go.

 

 

at this point i think I need further recommedations for a 1366 socket mobo that with cooperate with IOMMU cause the p6t from asus has issues according to my research

 

essentially telling it to ignore the call trace errors for one device with change in the syslinux.cfg allows the VM to be made but it bricks something cause soon enough I lose all access to the unRAID server with SMB shares and to the web UI.  so problem not solved just a little tease as if it may work and then essentially a system crash.  These problems only relate to the use of VMs.  The call trace errors and the syslog errors stating "Your BIOS is broken; RMRR ends before it starts!" is likely not going to allow for a well working VM box.  I can keep plex going and likely can work out my openVPN issues with DMZ routing workarounds, but the board is simply not compatible with IOMMU apparently.

post in the hardware or general thread for recommendations.

 

sorry to hear about the "append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot" causing other issues. Many servers need it to passthrough devices (including mine) and doesn't cause other issues. BUT it's good that you've discovered that the board has iommu issues. In some ways, thats easier to fix than dealing with bios/linux/slackware/unraid issues.

good to know seems theres a lot to learn, but thats what i want to do. hoping to get into sys admin work.

 

another thing i learned is that the xeon x5679 does not support vt-d so theres a big issue there. the x5679 doesn't even have an intel ark page.  Ive found some decent deals on x58 boards with first gen i7's and now know to look further at the details.  it looks like the x5679 is very comparable to a i7-980 but has the ability to support SMP in two cpu mobos which mine is not. 

Edited September 5, 20178 yr by 88pockets
x4679 changed to actual cpu x5679