KptnKMan

That would seem to be a VERY dangerous option to implement. May not be something that gets used often either, I don't know how many people would regularly reset and wipe all disks.

You may want to look into running Kubernetes VMs across your 3 nodes for this, or I'm not sure, if its possible to run K8s on the Unraid systems themselves. K8s-on-Unraid-host might be something to aim for perhaps, if Unraid itself could be a K8s worker node. That would bring some interesting high-availability scenarios.

Please GUI enhancements for VM control, like PCIe control, ReBAR control, PCIe Address management, more VM properties, control over more things without needing to edit XML directly, which gets wiped out if GUI interface is applied anytime after XML changes. Would be the most amazing thing. I already have nightly working running VM snapshots via script and VM Backup plugin. Furthermore, there is already a working Appdata Backup plugin, these things are already available with existing plugins. Array/share data backups are easy in script (I did it personally already).

No, my backups in S3 are definitely not above 5.5TB, but Cloudberry reports that I'm exceeding the limit. This was an issue I experienced last year, and saw in this thread that it was fixed in an update, and when I updated to the new container it seemed fixed. I saw that a consistency check was performed (I can't find where to initiate this manually) and a full backup (followed by 2 incrementals of around 5.5GB the following weeks) was made halfway through January 2023. But now again the same issue has returned. I believe it's related to this comment: I'm not really sure why this is happening, my last full backup was 430GB, and I have 3 others of 381GB, 327GB, and 230GB in S3. So the 5.5TB is certainly not exceeded, but this issue has made my backup routine really unpredictable. I believe I have everything setup properly, so not sure why its behaving like this. Does this make more sense?

Hi, can someone please help? I understood this issue was solved in the version update, I had a single good backup a few weeks ago, and it was and has now returned. I'm backing upto S3, have the latest container, version in container is 4.0.2.402. Can anyone help please?

You can download the kernel in this same thread, and follow the checklist in my summarised post, you don't need to wait until a new kernel. Also, another user reported that you may not need the 6.1 kernel, but I have not had the time to confirm that as yet. But as it is, yes you need the kernel patch and xml changes.

I don't know if anyone in this thread is still waiting, but another thread with a backported kernel 6.1 patch made ReBAR work for me on my primary setup. Link here to what I did to get it working, hopefully this can be added to UnRAID in the future, I'm super stoked about it.

Is anyone using the latest drivers still experiencing the locking issue? I'm considering it again, and wondering if there's any success.

That's interesting, I've been following the ReBAR issue for some time now, trying to make it work on my setup. From what I've seen, Intel CPU/chipset platforms seem to have a much easier time getting this to work, my understanding is that kernel support is a big factor with it. Until now I've been getting the black screen issue on my passthrough 3090 no matter what I've tried until now, I should try to test without the custom kernel and see if the extra VM extensions are the ticket to getting that working.

@Trozmagon what hardware are you running? Just wondering, because I've noticed some AMD Ryzen systems have trouble.

Looks like I figured it out, I had left out the steps to add the extra lines to my Win11 VM, which enabled 64GB ReBAR support. So looks like the checklist to enable this: - Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) - Boot Unraid in UEFI Mode - VM must use UEFI BIOS - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> After that, looks like everything worked for me as well. I'm just summarising this for anyone looking for a complete idea. I'll be testing performance over the next weeks as well to see if I'm seeing any improvement. This is great, exactly what I've been waiting for! EDIT for completeness: There is a last step, that I have implemented and can confirm works, which is the bind/unbind UserScript in this comment: Specific details of the script is in the linked^ comment, but this script sets the Bar size. I would highly recommend setting this up in a userscript, and set it to run "At Startup of Array": This works for my setup, but your mileage may vary.

What platform are you using? I'm also getting the lockup/freezing issue, my details in my signature.

I'm testing Virtiofs on my main Win11 VM, and having quite some success, but with a few caveats. Firstly, while installing the latest virtio drivers, my VM IMMEDIATELY CRASHES. I found through lots of trial and crashes that if I leave "Fwcfg" DISABLED, the install finishes. Disabling it allows install of updated drivers, sets up services, and does not crash the VM. What is this Fwcfg, can anyone shed some light on that? Secondly, I'm trying to setup 2 virtiofs disks, but only 1 appears as the Z: drive If I setup 2 drives, only the second appears as Z: when the VM boots. In this example, I setup "vdata1" and "vdata2": Thirdly, the virtiofs is always set as Z: I cannot find or see any way to reassign the drive letter, or indeed use multiple drive letters. Is there a way to change the assigned drive?

I am assuming that you rebooted the server during this operation? Ie: shut down, remove disk, format in another machine, repace in original machine, startup? Since this is marked as the solution, I wonder if its the reboot that is important? Also, is there a way to reset this without rebooting?

Oh thanks for this solution, I've been getting similar issue where I replaced/upgraded my NVME cache with a larger drive, and reassigned the existing cache disk to an Unassigned disk. As a result, for some reason the "old" cache disk (now 2nd NVME) would show up with the option to FORMAT, but after that would only present the option to Preclear, which I did, but then I cannot utilise it as an Unassigned disk because the MOUNT option never becomes usable: After formatting it and Preclearing it a few times, nothing seemed to be working. I also cleared the disk a few times, deleting all partitions. After clearing the disk a final time and performing a full system reboot, it seemed to become usable again: Seems like a odd bug somewhere, maybe?

I'm using a "Gigabyte RTX3090 Turbo 24G". Full system specs are in my signature, this is on UNRAID1. I flashed this card with the updated UEFI bios some time ago, and have a dumped & hexed BIOS of the same card that I use to boot VMs.

Thanks for the response. I'm still struggling with getting any VMs or even "unraid with GUI" to start (the server starts, but the local web gui shows a black screen with blinking cursor). Booting a VM just shows a black screen. I see this in the logs when I start a VM: Oct 8 19:01:50 unraid1 kernel: vfio-pci 0000:0c:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none Oct 8 19:01:50 unraid1 kernel: br0: port 2(vnet0) entered blocking state Oct 8 19:01:50 unraid1 kernel: br0: port 2(vnet0) entered disabled state Oct 8 19:01:50 unraid1 kernel: device vnet0 entered promiscuous mode Oct 8 19:01:50 unraid1 kernel: br0: port 2(vnet0) entered blocking state Oct 8 19:01:50 unraid1 kernel: br0: port 2(vnet0) entered forwarding state Oct 8 19:01:52 unraid1 avahi-daemon[16989]: Joining mDNS multicast group on interface vnet0.IPv6 with address ipv6addresshere. Oct 8 19:01:52 unraid1 avahi-daemon[16989]: New relevant interface vnet0.IPv6 for mDNS. Oct 8 19:01:52 unraid1 avahi-daemon[16989]: Registering new address record for ipv6addresshere on vnet0.*. Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.0: vfio_ecap_init: hiding ecap 0x26@0xc1c Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.0: vfio_ecap_init: hiding ecap 0x27@0xd00 Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00 Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.0: BAR 1: can't reserve [mem 0x7000000000-0x77ffffffff 64bit pref] Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.1: enabling device (0000 -> 0002) Oct 8 19:01:53 unraid1 kernel: vfio-pci 0000:0c:00.1: vfio_ecap_init: hiding ecap 0x25@0x160 Does anyone have an idea what's going on or where I can possibly investigate? Of all of these everything is set and enabled. If I leave everything the same and boot unraid as non-uefi mode, the local GUI and VMs work, but there is no reBAR enabled.

I'm just adding my plea again if anyone knows anything at all on how to resolve this. Is there anything known in the just-released unRAID 6.11.0 that aids in this? Also, I am not aware or could find of any hardware-specific oddities with enabling ReBAR without CSM and the black screen issues, is there something that anyone might be able to highlight? Is the QEMU rea-only ReBAR disabled issue only applicable to certain hardware or AMD-only setups, or something else? If I may, @alturismo what hardware are you using? Intel CPU? Just been trying to get this to work for a long time now, and most other things work just fine, just not this. My hardware is in my sig, not sure what the issue is.

I'm just responding to report that I've been running 6.10.3 stable for some time now, and no issues have been noticed with the Connectix-3 cards swapping around and acting strange. I'm very grateful to the Unraid Developers for attention on this issue and forum mods for making this space available.

I had this strange issue today when trying to shutdown my system due to some issues. I founc that these commands worked: killall -Iv docker killall -Iv containerd umount -l /dev/loop2 Its only started happening recently, and is the /mnt/cache unable to unmount. It seems that it's only Docker-related that is causing this issue, for me at least. I'm going to put this into a UserScripts script, in case I need it again, then I can fire it off.

Hi, thanks for the info. Actually I did get a chance to upgrade to 6.10.3 on both systems, almost exactly this time yesterday. I noted a couple things: - The Network Interface Rules dialogue returned (yay!) - The nics in my systems are setup as eth0 (mlx4_core), eth1 (onboard), eth2 (mlx4_core) and that seemed to not drift to eg eth3 (yay!) - I did 3 consecutive reboots (on both systems) and the config seemed to stick (yay!) I'm not calling this "fixed" just yet, as I need to investigate a couple other things and test, but it looks good so far. 🙂 Also, my nics show up as the identical same card you have.

I've found that splitting them up as Mellanox-eth0, onboard-eth1, Mellanox-eth2 produces the most consistent results. See the screenshots I posted earlier, this seems to be working now as it did quite consistently for me in previous releases. Yeah, we'll see when that happens, I'm not trying to rush anyone. I'm just trying to work an angle that I know "reliably", rather than test a new workaround. Nothing is perfect. As an aside though, the networking issues since I upgraded to 10Gbit have put things on hold for about as long as I've had 10Gbit now. All I wanted was to setup a working 10Gbit(Mellanox)/1Gbit(Onboard) failover bond, but that seems too much to ask. These days I just want a stable server and single 10Gbit connection that will persist reboots. Details of that journey in my other long thread.

Well, I rebooted and the same issue reappeared same as if nothing happened. I hacked the network-rules.cfg manually with the correct interface IDs, kernel modules and hardware addresses... and it reboots fine now. I've rebooted the system 3 times in a row now just to see if something drifts, but its ok... for now. How mine is supposed to look in my config: I understand what you're saying, but I don't have time to verify 6.10.3 right now. I only rebooted to add something to the system, and then all hell broke loose. I know the issues with 6.10.1 at this time, and I thought I knew 6.10.2, but I was wrong there as usual. I'm sticking on 6.10.1 until a stable 6.10.3 comes out. In my experience, this doesn't match the behaviour at all. I'm using dual-port Mellanox CX-3 cards in both my servers and can verify the behaviour. What seems to happen quite consistently, and as I've documented extensively on this forum in threads, is that the first Mellanox interface seems to be fine but the second appears to be created twice. Then some kind of cleanup happens and a gap is left. THat process of creating/removing the second interface seems to mess up other assignments. If the Mellanox dual-port card is assigned last, it doesn't seem to have the issue as far as I can tell, but in unRAID if you want the Mellanox MAC as bond MAC then it needs to be the first MAC on eth0. 😐 So in my experience, on both my servers, eth0 has never been the issue if the first Mellanox is set to eth0. I could be wrong here, but I'm just saying what happened to me.

Thanks for the advice. This secondary system is my "stable" unRAID that I basically never mess with, so I'm not keen on non-stable releases here. I already downgraded to 6.10.1, and both the interface rules and interface-rules.cfg have reappeared. I saw the downgrade worked for people in this thread: However, now I have this (THe dropdown shows duplicates): I'm going to delete the interface-rules and reboot to see how that helps.

This is a nightmare, I just downgraded to 6.10.1 and the interface assignments and network-rules.cfg has reappeared but now other issues. I'll document in the linked thread, I try not to hijack other peoples support threads with my issues. Thanks for the advice.