[GUIDE] Virtualizing unRAID on Proxmox 3.1


Recommended Posts

In PVE 5.4 (or lartest 6), just need passthrough (IOMMU) the booting USB and HBA, that's all ( I haven't try onboard SATA controller).

You can freely run Unraid under PVE or baremetel by boot selection, all work, just if Unraid under PVE, then no VM function available.

Edited by Benson
Link to comment
17 minutes ago, chris_netsmart said:

thanks @Benson    Dump question but, but I just want to make sure that I am reading this read     HBA = host bus adapter and PVE = Proxmox.

and is the  setting up of the passthrough (IOMMU) easy to do ?

Yes and quite easy, just need setting in PVE side. In general, Unraid nothing need to change.

Edited by Benson
Link to comment
4 hours ago, chris_netsmart said:

reading through the posts, I am liking the ideal of hosting my UNRAID within Proxmox.

but I didn't see the following question answer, or if there are then I missed them

 

  • regarding the spin down of HD's I like at @s.Oliver say, by passing through the SATA Controller, but does this work and allow the HD's to spin down. ?
  • if I build the Promox environment and I have a Unraid Server environment, will I be able to transfer over my Unraid HD's as easy as moving over the HD's.  Adding them to a VM UNraid guest and power up the guest and then set up Unraid as normal. ?
  • I see  posted that people mentioning " plopkexec " to fool Unraid, in making it think that it has a USB flash drive plug in,   I see that uldise posted a link to a ISO.  my question is. how to you install it ?

I am still at the research stage so any advice would be appreciative.

 

 

well, i answered all of your questions actually ;-)

 

when you do passthrough your hardware to unRAID (HBA, or SATA controllers, or whatever) then unRAID has full control of it's features – and yes, it spins down all HDDs, which are connected to these passthrough'ed hardware. and this includes things like hot-swap (if supported by your hardware) or other things.

 

if done in the right way, your 2nd question could be answered with yes! unRAID see's the drives as they would be on bare metal booted hardware (because you passthrough the hardware, which your drives are connected to then).

 

your 3rd question references to my hint using plopkexec to boot natively into your unRAID installation on a USB stick (this is needed, because the hypervisor can't boot any VM from a usb stick; so plopkexec is a super small VM on ISO and it's then boots unRAID from the USB stick in stage 2). this method allows for native unRAID setup (with your USB key), without any changes/modifications, whatsoever. and the bonus is, that you never need to think about it in the future. all changes which need to be written to the usb stick (where unRAID is installed on) are written to the stick (and not to the ISO in the other guys setup). so no worries in the future – and you could boot unRAID on bare metal (without modifications).

 

probably soon i'll tweak my setup again to go this virtualization route – because now i need the features of Proxmox, but i can't loose my unRAID-server – so i'll redo all this on my newer rig.

 

hope this helps for your decision making.

Edited by s.Oliver
Link to comment
1 hour ago, s.Oliver said:

because the hypervisor can't boot any VM from a usb stick

You can't specify Proxmox to boot USB stick, but if USB stick is the only bootable device in guest, then it will boot. No plopkexec step also work.

Edited by Benson
Link to comment

@Benson thanks for the post, I think I get the ideal.  basically we are doing the following steps

 

move over Unraid Hard drivers and USB Flash Drive into new case

Install a blank hard drive for Proxmox

Install Proxmox onto new blank hard drive

download Unraid VM ISO file

 

Create VM for UNraid

add USB Flash Drive and Hard drivers from old Unraid

make USB pen drive the boot drive ? - Need to look at the software

power up VM

 

once Unraid is up and running then add Hard drives to Unraid and start array.

 

if this wrong then please point it out, and again thanks for your help.

 

Link to comment
18 hours ago, Benson said:

You can't specify Proxmox to boot USB stick, but if USB stick is the only bootable device in guest, then it will boot. No plopkexec step also work.

you're using it this way? (no special prepared bootable image, pure unRAID usb-stick and it boots directly from that?)

Edited by s.Oliver
Link to comment
4 hours ago, chris_netsmart said:

@Benson thanks for the post, I think I get the ideal.  basically we are doing the following steps

 

move over Unraid Hard drivers and USB Flash Drive into new case

Install a blank hard drive for Proxmox

Install Proxmox onto new blank hard drive

download Unraid VM ISO file

 

Create VM for UNraid

add USB Flash Drive and Hard drivers from old Unraid

make USB pen drive the boot drive ? - Need to look at the software

power up VM

 

once Unraid is up and running then add Hard drives to Unraid and start array.

 

if this wrong then please point it out, and again thanks for your help. 

 

 

Step almost the same, but I think best should be prepare unraid USB stick first then unplug it before prepare Proxmox.

 

Below are some detail

 

1. Unplug well prepare Unraid USB, best ( not must ) confirm it can boot in UEFI mode.

2. Install Proxmox to harddisk, make IOMMU enable.

3. Replug Unraid USB

4. Create VM

 

1.PNG.ca60dd40d096a226c1e8491703191141.PNG

 

4a. Tweak ( Some step may not necessary )

- Click CD/DVD drive, remove it

- Click Hard Disk, detach and remove it

 

2a.PNG.c68eafa41578a10c58a77ccb2e7f5f0f.PNG

 

- Add phyical disk controller and physical Unraid USB stick

3b1.PNG.acecab05038118b61a3e3a173d8b37b2.PNG

 

3b2.PNG.605a2322eae1a4e9d37ade6e1ebea853.PNG

 

3b3.PNG.b9a879ae3c307970d154d9993dc0d09e.PNG

 

4b. Tweak

- Edit boot order, only retain CD-ROM  in boot device_1

2.PNG.cda82af0be3091bff6c7d3fe891a1c6a.PNG

 

 

That's all.

Edited by Benson
  • Upvote 1
Link to comment
29 minutes ago, Benson said:

Yes, even passthrough a NVMe as a Windows VM, freely boot under PVE or baremetel.

hmm… maybe i read the screenshots wrong...

 

but i guess, because of no other specified bootable device (boot device 1 = cd-rom with empty 'tray', device 2 & 3 are set to nothing, or have non-bootable devices) PROXMOX is looking at it's usb devices (for this VM) and boots from it, if bootable?

Link to comment
7 minutes ago, s.Oliver said:

hmm… maybe i read the screenshots wrong...

 

but i guess, because of no other specified bootable device (boot device 1 = cd-rom with empty 'tray', device 2 & 3 are set to nothing, or have non-bootable devices) PROXMOX is looking at it's usb devices (for this VM) and boots from it, if bootable?

Yes, boot device 2,3 set to none. That's why I said

 

19 hours ago, Benson said:

You can't specify Proxmox to boot USB stick, but if USB stick is the only bootable device in guest, then it will boot. No plopkexec step also work. 

 

Edited by Benson
Link to comment
  • 3 months later...

so today was the day for PROXMOX installation and virtualizing unRAID again.

most of the stuff works as expected, but some things aren't up to expectations:

 

a) booting unRAID (directly from USB stick) brings me every time into the EFI shell, even though i set all the right boot options in the PROXMOX BIOS. so with every reboot i need to enter the BIOS, set the USB stick as boot option and then it boots unRAID. looks like PROXMOX doesn't save the changes into the EFI partitions or so.

 

b) even though i enabled nested virtualization (PROXMOX) my macOS VM (under unRAID) is drastically slower than before. well, i had expected a little loss, but not that much. played around a bit with different options in the XML, but can't see any improvements.

 

has someone any ideas, how to improve?

 

well, i'll try to setup the macOS VM under PROXMOX, so no need for the nested virtualization then, but it can be a hassle ;-)

 

Link to comment
On 7/30/2019 at 8:31 PM, Benson said:

You can't specify Proxmox to boot USB stick, but if USB stick is the only bootable device in guest, then it will boot. No plopkexec step also work. 

Pls set boot device 1 be CDROM, others none.

 

I haven't run PVE now, mainly because I turn two Unraid in one, it also easy for management.

Edited by Benson
Link to comment
19 hours ago, Benson said:

Pls set boot device 1 be CDROM, others none.

 

I haven't run PVE now, mainly because I turn two Unraid in one, it also easy for management.

thx., but i did that from beginning on. it never booted automatically from the USB stick – it entered the EFI shell. when intercepted while initializing BIOS and selecting the proper shown boot device (unRAID USB stick) it boots and everything is fine.

 

not sure, why it doesn't automatically choose it. i'm running on Proxmox v6 and use UEFI/EFI boot because of passthrough'ing serveral PCI devices.

Edited by s.Oliver
Link to comment
  • 1 year later...
On 11/24/2019 at 2:50 PM, s.Oliver said:

thx., but i did that from beginning on. it never booted automatically from the USB stick – it entered the EFI shell. when intercepted while initializing BIOS and selecting the proper shown boot device (unRAID USB stick) it boots and everything is fine.

 

not sure, why it doesn't automatically choose it. i'm running on Proxmox v6 and use UEFI/EFI boot because of passthrough'ing serveral PCI devices.

 

UPDATE: since one of the latest PROXMOX updates native booting from an USB device is directly support. No hacks, tricks needed anymore.

Link to comment
  • 6 months later...


I'm looking at virtualising my unraid and opnsense under proxmox also.

 

My aim is to pass through the onboard sata controller with my drives to unraid (it sits in its own IOMMU group). But then for cache, I would like to partition my NVMe drive for the proxmox partition, opnsense, unraid cache, other guests etc. Is this possible to do? I.e. give unraid a nvme partition to play with as the cache, instead of doing direct passthrough?

Edit: answered with google

 

I understand that things like SMART etc won't be available for this partition.

 

Also how does fan control work for array drives? Is it possible to give unraid access to the fan headers, or is temperature management done in proxmox?

Edit: some google-fu tells me that fan control will need to happen on proxmox side either via a fancontrol plugin or directly from BIOS.

Edited by mishmash-
Link to comment
  • 7 months later...

I have now virtualised my unraid in proxmox. I passed through the sas falcon controller (ibm something flashed to IT mode), GTX 1060 for Emby transcoding, a couple of usb devices including the unraid usb stick, enocean usb and usb zigbee controller. I created a BOND network interface and used this as an E1000 network card. The paravirtualised network card dropped packets (as detailed before). 

It is all working great! 

 

On the same proxmox - Opnsense (in is dedicated network port, out is network BOND), MQTT LXC container, Home Assistant Virtual machine, NodeRed LXC for testing, and Zorin VM.

 

Server is a Asus Strix x570-F board, AMD 5900 with 64GB ram.

Edited by spants
Link to comment
  • 3 months later...

Timestamped - if it doesn't work, 9:08

This virtualized truenas video is the most closely representative of unraid drive usage that i could find for passing through individual drives without the entire controller.
I'm not really looking to pass through the whole controller, as i want remaining drives to be accessible to proxmox. (my unit uses a sas expander, so i'm sorta stuck).

TLDW: 
He sorta did stubbing like we would to pass a gpu through, but with the sda/sdb/sdc etc into proxmox's virtIO SCSI by id 
So, lsbk to list the drives with model/serial and attach them to the unraid VM's scsi controller.

In my case, this would be the drives that i would potentially pass to the unraid VM's virtio scsi controller.

ata-SanDisk_SDSSDH3_1T02_211135801590
ata-WDC_WD80EFBX-68AZZN0_VGKJ451G
ata-WDC_WD80EFBX-68AZZN0_VGKJ5H3G
ata-WDC_WD80EFBX-68AZZN0_VGKJD86G
ata-WDC_WD80EFBX-68AZZN0_VYGAZANM
ata-WDC_WD80EFBX-68AZZN0_VYGAZYMM



For those with some experience virtualizing unraid, is this a viable solution?


*added later* I have a new server coming tomorrow, i think i'll grab an unraid trial license and my cold spare, put some data on it, swap to proxmox and try passing that drive directly to unraid vm to see if it can 'remember' it. If it does work, i'll drop an update.

Edited by jonfive
Additional info
Link to comment

UPDATE:

It DOES work. Though, i need to test it's long term viability since there's complaints from people on the forum about i/o errors using scsiblock (see below)

 

For this test, I created another usb with a trial license on bare metal, made a few folders and stuck some isos on it.
Installed proxmox, attached the usb device, set first boot to that usb device. 

Ran the below qm set line.

Started the vm, disk showed up correctly, pre-placed in the right disk# slot without showing "missing".

Opened the 6.11 file manager, disk 1, files were there.
 

 

 

qm set vm# -scsi# /dev/disk/by-id/YourDiskInformation,scsiblock=1


Without scsiblock, it just shows up as a qemuharddisk, so it's necessary for the time being. 

The particular part of the proxmox docs that mentions the scsiblock potential issue:
 

Quote

scsiblock=<boolean> (default = 0)

whether to use scsi-block for full passthrough of host block device

! can lead to I/O errors in combination with low memory or high memory fragmentation on host


If anyone has any further information or knowledge on how high/low memory fragmentation occurs in unraid, that'd be handy :)

Link to comment
  • 2 weeks later...

Has anyone managed to reproduce @grumpybutfun unRAID vhd file? The original link is dead (given it's 8 years old, it's no real surprise).

 

I'm trying to do a similar thing, but currently, unRAID is saying "Error accessing your physical USB Flash boot device", even though the virtual USB drive is writable and boots to the unRAID GUI.

 

Steps performed:
1. Create VHD and SCP it across to Proxmox.
1a. VHD created as required by unraid instructions.
2. Run the following script:

VMNUM=1112
VMSCSINUM=0
VMDISKLOC=/mnt/prox/images
DISK_STOR=Proxmox
USBARG="args: -drive 'file=$VMDISKLOC/$VMNUM/vm-$VMNUM-disk-$VMSCSINUM.raw,if=none,id=drive-usb0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'usb-storage,drive=drive-usb0,bootindex=1,removable=on'"
cd $VMDISKLOC/$VMNUM/
qemu-img convert -O raw UNRAID.vhd unraid.img && \
rm UNRAID.vhd && \
qm importdisk $VMNUM unraid.img $DISK_STOR && \
qm set $VMNUM --scsihw virtio-scsi-pci --scsi$VMSCSINUM $DISK_STOR:$VMNUM/vm-$VMNUM-disk-$VMSCSINUM.raw && \
sed -i -e "1i${USBARG}   ${1}" /etc/pve/qemu-server/$VMNUM.conf

 

I've tried both:

args: -drive 'file=/mnt/prox/images/1112/vm-1112-disk-0.raw,if=none,id=drive-usb0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'usb-storage,drive=drive-usb0,bootindex=1,removable=on'

 

and 

 

args: -drive id=stick,if=none,format=raw,file=/mnt/prox/images/1112/unraid.img -device usb-storage,drive=stick,bootindex=1,removable=on

 

And whilst the VM boots into UNRAID with both, it throws up a warning that it cannot write to the "disk". I have confirmed that the "disk" is writable. 

Is img/raw the right way to go here?

Link to comment
On 10/10/2022 at 7:12 PM, modem7 said:

Has anyone managed to reproduce @grumpybutfun unRAID vhd file? The original link is dead (given it's 8 years old, it's no real surprise).

 

I'm trying to do a similar thing, but currently, unRAID is saying "Error accessing your physical USB Flash boot device", even though the virtual USB drive is writable and boots to the unRAID GUI.

 

Steps performed:
1. Create VHD and SCP it across to Proxmox.
1a. VHD created as required by unraid instructions.
2. Run the following script:

VMNUM=1112
VMSCSINUM=0
VMDISKLOC=/mnt/prox/images
DISK_STOR=Proxmox
USBARG="args: -drive 'file=$VMDISKLOC/$VMNUM/vm-$VMNUM-disk-$VMSCSINUM.raw,if=none,id=drive-usb0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'usb-storage,drive=drive-usb0,bootindex=1,removable=on'"
cd $VMDISKLOC/$VMNUM/
qemu-img convert -O raw UNRAID.vhd unraid.img && \
rm UNRAID.vhd && \
qm importdisk $VMNUM unraid.img $DISK_STOR && \
qm set $VMNUM --scsihw virtio-scsi-pci --scsi$VMSCSINUM $DISK_STOR:$VMNUM/vm-$VMNUM-disk-$VMSCSINUM.raw && \
sed -i -e "1i${USBARG}   ${1}" /etc/pve/qemu-server/$VMNUM.conf

 

I've tried both:

args: -drive 'file=/mnt/prox/images/1112/vm-1112-disk-0.raw,if=none,id=drive-usb0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'usb-storage,drive=drive-usb0,bootindex=1,removable=on'

 

and 

 

args: -drive id=stick,if=none,format=raw,file=/mnt/prox/images/1112/unraid.img -device usb-storage,drive=stick,bootindex=1,removable=on

 

And whilst the VM boots into UNRAID with both, it throws up a warning that it cannot write to the "disk". I have confirmed that the "disk" is writable. 

Is img/raw the right way to go here?

 

just quickly jumped here to give a hint:

if your goal is, to boot unraid as a VM inside of Proxmox see my last post (just a few upwards from here).

 

there's no need anymore, to fake/virtualize the usb-stick as vm file or such things. simple create the VM without (or delete later on) a disk associated. passthrough the original usb-stick ("UNRAID" labeled) to that VM and choose this one as boot order 1 (see Options) to boot from. i've set the machine type to Q35 with UEFI for passthrough of hardware hba controllers and other stuff.

 

works perfectly.

Link to comment
5 minutes ago, s.Oliver said:

 

just quickly jumped here to give a hint:

if your goal is, to boot unraid as a VM inside of Proxmox see my last post (just a few upwards from here).

 

there's no need anymore, to fake/virtualize the usb-stick as vm file or such things. simple create the VM without (or delete later on) a disk associated. passthrough the original usb-stick ("UNRAID" labeled) to that VM and choose this one as boot order 1 (see Options) to boot from. i've set the machine type to Q35 with UEFI for passthrough of hardware hba controllers and other stuff.

 

works perfectly.

Heya,

 

Aye, I've managed to successfully virtualise (with qemu guest agent) with no issues, I was merely looking for a way to forego the USB stick as it's a single point of failure. 

 

Would be great if there was an official vhd/iso for those who just wanted to virtualise the NAS aspect, and be able to backup/snapshot the configs as part of their standard work flows. 

 

Cheers though! 

Link to comment
6 hours ago, modem7 said:

Heya,

 

Aye, I've managed to successfully virtualise (with qemu guest agent) with no issues, I was merely looking for a way to forego the USB stick as it's a single point of failure. 

 

Would be great if there was an official vhd/iso for those who just wanted to virtualise the NAS aspect, and be able to backup/snapshot the configs as part of their standard work flows. 

 

Cheers though! 

 

well, unRAID uses the USB-stick as a way of validating your license (because it is bound to the hardware UUID of that stick). don't know how to circumvent that, even though i also don't like the single point of failure associated to that way of doing it.

 

once (quite awhile back) i had it setup with in a different way (with a synchronisation between virtual image and the USB-stick, but you had to have it mounted while booting unRAID and in the long run it was prone to failures when you forgot to sync back to the stick, before rebooting, etc.).

Edited by s.Oliver
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.