ARM VMs (Raspberry Pi emulation)

[itimpi]

Now that the QEMU included with Unraid 6.10.0-rc2 now supports emulation of ARM CPU's I am wondering if anyone has got such a VM running?  I am wondering if it is now possible to emulate a Raspberry Pi (any model) and if so running which OS?  If so please share the XML you are using for the VM.

[ghost82]

1 hour ago, itimpi said:

I am wondering if anyone has got such a VM running?

Hi, I started yesterday to play with this, not so easy if one has to start from scratch.

As far as I know we need AAVMF bios, basically it's OVMF for arm/aarch64; sources are in the same edk2 OVMF package, but we need cross compilation with a toolchain to compile the fd files on a x86_64 machine.

I was able to compile from sources with these commands:

git clone https://github.com/tianocore/edk2.git
cd edk2
git clean -ffdx
git reset --hard
git submodule deinit --force --all
git checkout edk2-stable202108
git submodule update --init --force
export GCC5_AARCH64_PREFIX=/home/xxx/Scrivania/toolchain/gcc-arm-10.3-2021.07-x86_64-aarch64-none-elf/bin/aarch64-none-elf-
source edksetup.sh
nice make -C "$EDK_TOOLS_PATH" -j $(getconf _NPROCESSORS_ONLN)
build -a AARCH64 -b RELEASE -p ArmVirtPkg/ArmVirtQemu.dsc -t GCC5

 

Note also that GCC 11 will not work, GCC 10 is ok, with GCC 11 brotli gives some errors!

Toolchain is available here:

https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads

 

About the xml template:

there are some examples in internet, they don't differ too much from a q35 template, I noticed that you need to pass the kernel too, that if I understood well must be modified to be run on qemu.

 

I'm at half way and will play with this in the following days, glad that you opened this post!

[ghost82]

Work in progress...:

This is a basic template I'm using to boot the image:

<domain type='qemu'>
  <name>rpi</name>
  <uuid>removed</uuid>
  <memory unit='KiB'>262144</memory>
  <currentMemory unit='KiB'>262144</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='armv7l' machine='versatilepb'>hvm</type>
    <kernel>/media/6TB/rasp/kernel-qemu-4.19.50-buster</kernel>
    <cmdline>root=/dev/sda2 rootfstype=ext4 rw</cmdline>
    <dtb>/media/6TB/rasp/versatile-pb-buster.dtb</dtb>
    <boot dev='hd'/>
  </os>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>arm1176</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-arm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/media/6TB/rasp/2020-02-13-raspbian-buster.img'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='0' model='lsilogic'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ed:eb:c7'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
  </devices>
</domain>

 

kernel/dtb are downloaded from here:

https://github.com/dhruvvyas90/qemu-rpi-kernel

 

Raspbian buster downloaded from here:

https://downloads.raspberrypi.org/raspbian/images/raspbian-2020-02-14/

 

For now it's giving a kernel panic on shutdown..need to investigate: seemed to be solved with qemu 6.1-RC, but it seems it's not the case..

 

I'm only able to use arm1176 with qemu-system-arm (32 bit), maybe related to the custom kernel.

Edited November 11, 2021 by ghost82

[ghost82]

I also tried the emulation of machine type raspi3 (pi 3b+) instead of the versatilepb (-M raspi3 in qemu), but this has a lot of limits because we wont have a pci bus, so we can't attach things to it, including a simple vga...

It is able to boot, also the latest raspios bullseye (2021-10-30), but only "in a console mode", with serial device configured with stdio (or mon:stdio).

Moreover qemu is not happy if the sd (note that we need sd bus, as no controller is available because of lacking of pci bus) img is not 2 Gb or a power of it, so the 2021-10-30-raspios-bullseye-armhf.img must be resized to 4 Gb (2021-10-30-raspios-bullseye-armhf.img is 3,7 Gb) with:

qemu-img resize 2021-10-30-raspios-bullseye-armhf.img 4G

 

Full qemu command:

qemu-system-aarch64 -M raspi3 -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootdelay=1" -dtb bcm2710-rpi-3-b-plus.dtb -sd 2021-10-30-raspios-bullseye-armhf.img -kernel kernel8.img -m 1G -smp 4 -serial stdio -usb -device usb-mouse -device usb-kbd -vnc 192.168.2.1:0

 

Adding -vnc ip:port and connecting to it results only in a black screen (expected).

Porting that qemu command to libvirt is quite a mess, and maybe not supported at all for raspi3.

 

For all these reasons, the versatilepb machine type is advised instead of the raspi one(s).

 

In this case, for native emulation (-M raspiX), one can extract the kernel and the dtb files from the official img with the following commands:

losetup -f --show -P 2021-10-30-raspios-bullseye-armhf.img
mount /dev/loop0p1 /path/to/mount/point
cp /path/to/mount/point/kernel* ./
cp /path/to/mount/point/*.dtb ./

 

Following screenshot refers to the output of qemu command issued through ssh:

Edited November 12, 2021 by ghost82

[stefan.tomko]

Hello @ghost82 - I was following your template, but I am unable to boot the VM, I always get the following. Any idea why? I used the template you provided, just modified local path to img/kernel/dtb files.

 

[ghost82]

9 hours ago, stefan.tomko said:

Any idea why?

Error is related to kernel, unable to mount root fs.

Check that you downloaded kernel from the source I provided (you need a custom kernel).

Check that the image you downloaded is able to be mounted by the custom kernel.

Check that you have the cmdline in your xml to specify root path, fs type and rw mode.

[stefan.tomko]

Thanks. I got confused with kernel/dtb files. I fixed that accordingly and now it is booting up. Many thanks!

[ghost82]

2 minutes ago, stefan.tomko said:

I fixed that accordingly and now it is booting up

Thanks for confirming!

Can you confirm also that the vm will panic on shutdown?

[JackSafari]

How is this emulation on Unraid progressing? I am mostly curious. I don't have any specific need for it, but it would be fun to try and useful to have.

Edited March 1, 2022 by JackSafari

[Uncle09]

Me too

[Thomas_H]

Hi,

any new experience regarding this topic?

 

I've got an Raspberry.img and I would use it as a vm.

 

Cheers Thomas 😎

 

[Eksitus]

https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/

 

I followed this tutorial and was able to get aarch64 VM running pretty well. (I needed second linux to extract files from disk image since unraid does not have the required tool)

 

Issues:

- Not sure how to best convert the quemu arguments to xml file for libvirt

- I only need to access VM via command line, so I did not even try to get VNC to work

 

Otherwise the VM runs pretty stable and so far faster than I expected, I did not run along any issue during the installation but it took couple hours for the system to install. Good luck to anyone who wants to try!

[SimonF]

3 hours ago, Eksitus said:

https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/

 

I followed this tutorial and was able to get aarch64 VM running pretty well. (I needed second linux to extract files from disk image since unraid does not have the required tool)

 

Issues:

- Not sure how to best convert the quemu arguments to xml file for libvirt

- I only need to access VM via command line, so I did not even try to get VNC to work

 

Otherwise the VM runs pretty stable and so far faster than I expected, I did not run along any issue during the installation but it took couple hours for the system to install. Good luck to anyone who wants to try!

does virsh dumpxml VMName get the XML when the VM is running?

[Eksitus]

25 minutes ago, SimonF said:

does virsh dumpxml VMName get the XML when the VM is running?

I doubt it was running through virsh at all, as far as I'm aware just using qemu, running the command from the post does not list it using "virsh list" and vm kills itself after closing the console.

However, after a lot of trial and error, I managed to create a very rough XML template, that suprisingly works! (still, I did not even test VNC)

At least now it's possible it use the VM headless as a normal unraid VM.

I hope someone can upgrade it and post the result, I expect there is huge room for improvement and it'd be nice to learn from more professional version.

Make sure to update any file locations & UUID accordingly if you want to use it.

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='qemu'>
  <name>deb_aarch64</name>
  <uuid>UUID</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='aarch64' machine='virt-6.2'>hvm</type>
    <kernel>/mnt/user/isos/arm/vmlinuz-4.9.0-19-arm64</kernel>
    <initrd>/mnt/user/isos/arm/initrd.img-4.9.0-19-arm64</initrd>
    <cmdline>root=/dev/vda2</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='2'/>
  </features>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>cortex-a72</model>
    <topology sockets='1' dies='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/user/isos/arm/hda.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:95:98:31'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <audio id='1' type='none'/>
  </devices>
</domain>

 

Edited November 8, 2022 by Eksitus

[Eksitus]

update: changing <gic version='2'/> to <gic version='3'/> allows to have more than 8 threads

[kevinpastor]

I'm on Unraid 6.11.5 and things have changed since @ghost82 posted his config back in November 2021. It seems like qemu-system-aarch64 is the new default. It includes raspi3b as an available machine. I was wondering if anybody got it to work with this new emulator.

 

I'm getting an error about the property raspi3b-machine.acpi missing when trying to start the VM. I've also tried with versatile-pb, but I'm getting a similar error. After looking at some StackOverflow question, it looks like libvirt has a bug (reference), but I'm not sure if this is the cause of the error I'm getting. I'm not sure if the bugfix on that closed issue arrived on my machine.

 

 I've arrived to my current configuration by starting with the first one on this thread and reading this article. I've obtained the kernel and the dtb file from Raspberry Pi firmware repo. Here's my configuration:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='qemu'>
  <name>rpi</name>
  <uuid>[REDACTED]</uuid>
  <memory unit='KiB'>262144</memory>
  <currentMemory unit='KiB'>262144</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='aarch64' machine='raspi3b'>hvm</type>
    <kernel>/mnt/user/domains/rpi/kernel8.img</kernel>
    <dtb>/mnt/user/domains/rpi/bcm2710-rpi-3-b-plus.dtb</dtb>
    <cmdline>root=/dev/sda2 rootfstype=ext4 rw</cmdline>
    <boot dev='hd'/>
  </os>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>arm1176</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/domains/rpi/2020-02-13-raspbian-buster.img'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='0' model='lsilogic'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ed:eb:c7'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
  </devices>
</domain>

 

Here's the full log of the failing startup: 

2023-04-13 14:53:43.017+0000: starting up libvirt version: 8.7.0, qemu version: 7.1.0, kernel: 5.19.17-Unraid, hostname: Vault
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME=/var/lib/libvirt/qemu/domain-26-rpi \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-26-rpi/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-26-rpi/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-26-rpi/.config \
/usr/bin/qemu-system-aarch64 \
-name guest=rpi,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-26-rpi/master-key.aes"}' \
-machine raspi3b,usb=off,dump-guest-core=off,memory-backend=ram \
-accel tcg \
-cpu arm1176 \
-m 256 \
-object '{"qom-type":"memory-backend-ram","id":"ram","size":268435456}' \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid [REDACTED] \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=38,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-no-acpi \
-boot strict=on \
-kernel /mnt/user/domains/rpi/kernel8.img \
-append 'root=/dev/sda2 rootfstype=ext4 rw' \
-dtb /mnt/user/domains/rpi/bcm2710-rpi-3-b-plus.dtb \
-device '{"driver":"lsi","id":"scsi0","bus":"pci","addr":"0x1"}' \
-usb \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/rpi/2020-02-13-raspbian-buster.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","scsi-id":0,"device_id":"drive-scsi0-0-0","drive":"libvirt-1-format","id":"scsi0-0-0","bootindex":1}' \
-netdev tap,fd=39,id=hostnet0 \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:ed:eb:c7","bus":"pci","addr":"0x6"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,websocket=5701,audiodev=audio1 \
-k en-us \
-device '{"driver":"cirrus-vga","id":"video0","bus":"pci","addr":"0x2"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2023-04-13T14:53:43.082319Z qemu-system-aarch64: Property 'raspi3b-machine.acpi' not found

As I'm not an expert with the setup of a VM, I'd love if anybody could help or share their current setup.

Edited April 13, 2023 by kevinpastor
Add more references

[MrAndyBurns]

Hello All,

I too wanted a piece of Pi on my Unraid. After having a good read of this, and others, I got confused so went for a play. In the end I have ended up downloading the latest "Raspberry Pi Desktop for PC and Mac" iso from https://www.raspberrypi.com/software/raspberry-pi-desktop/ . I created a new VM using the Debian template with the downloaded iso as the OS install iso drive and created a 15gb drive.  I started this VM with the consol showing. A boot screen opened, where I selected Install.  Dont just leave it as Pi will open, you will get excited but then you will realise its just in a boot loop. Work though the install, it's easy to do and you can see more information on it here https://projects.raspberrypi.org/en/projects/install-raspberry-pi-desktop/4. Once installed I stopped the VM and edited the configuration in Unraid to delete the install iso information so it boots to the 15gb drive. That was it! You start the VM, Debian pops up but you can just leave it and Pi will open. Setup a user etc, it will restart and your at the Pi home page. You can enable SSH etc as normal.

 

In summary

-Download "Raspberry Pi Desktop for PC and Mac" iso

-Create new debian VM and set the OS install iso to the "Raspberry Pi Desktop for PC and Mac" iso

-Start the VM

-Select Install option (either one)

-Stop VM

-Edit VM and remove the OS install iso information

-Start VM and enjoy a piece of Pi

 

Hope this helps ...

[gregoriko]

Hello

I'll run piKVM as VM on Unraid but I don't know how. PiKVM are based on arch Linux. I have this image file https://files.pikvm.org/images/v2-hdmiusb-rpi4-latest.img.xz (HDMI-to-USB dongle). I was tried extract kernel and rootfs img but I think I did something wrong because files are large (256 and 512 MB). Can someone help me? Thanks. (Solution can be for other linux too)

[ConnerVT]

I'm not much help with setting up PiKVM as a VM, as I have nearly no experience with Unraid's VM.  But I have mucked around with the PiKVM software on an actual RPi.  It was my first foray into the Arch distro (interesting) and the PiKVM package can be a bit difficult to navigate around, with much of the system defaulted to Read Only and files not being where one would expect them to be.  I also believe it written for armv7hf, which may not play easily with Unraid's VM.

 

It took me quite some time to get Cloudflared installed on my PiKVM.  I still have the scars.  *grin*

[OrdinaryButt]

Any updates on this topic?

[gregoriko]

No. I bought RPi. 😁

[vcadm]

So I was looking to possibly get rid of a Raspberry Pi which does a Slideshow for us on our TVs throughout the church.   I attempted to look for a docker that would give me the same functionality but couldn't find anything.  Looking at the above progress, I am guessing I need to see about going another route.