buxel

Hi Jorge, thanks for chiming in. That's what I assumed as well but the timing is just too odd. This behavior started exactly the day i made the switch from OMV to Unraid 😒 It is hard for me to verify this after swithcing, since the former boot drive is now the ssd cache and all HDDs are in the array. Even if i boot some live distro, it would not make any use of the HDD. Could it be something driver related? My system is nothing fancy, just an old Skylake PC.

Just to be sure, - i have reset al BIOS settings and re-plugged all cables on the mainboard/drives. - I also ran two memtest passes for ~3h with no errors. - made sure i'm not using macvlan after reading the latest changelog - used a different USB port for the Unraid flash drive Still getting these crashes. Is there anything I'm missing to gather more information about the root cause?

I just realized I had "mirror to flash" accidently disabled. I have turned it on now and will provide a new set of diagnostic logs after the next crash/reboot. EDIT: here we go, another reboot happened at 14:23. It did not log much after I enabled mirroring to flash: Apr 13 14:09:01 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="12322" x-info="https://www.rsyslog.com"] start Apr 13 14:16:00 Tower root: Fix Common Problems Version 2024.03.29 Apr 13 14:16:05 Tower root: Fix Common Problems: Warning: Syslog mirrored to flash Apr 13 14:16:05 Tower root: Fix Common Problems: Other Warning: Unassigned Devices Plus not installed ** Ignored that's it 🤔 tower-diagnostics-20240413-1425.zip syslog-127.0.0.1.log

Hello, I just recently made the switch from OMV to Unraid on my small NAS. I'm still in the progress of adding all services to OMV but already hit a road block early on. I observed daily parity checks caused by "unclean shutdown". Looking at the diagnostic logs, I did not see any cause for a reboot / errors and I suspect that more than one reboot might have happened, so the "last" logs mirrored to flash are also not useful. So this morning, I set up the local syslog server and made Unraid log to itself, persisting in appdata. Judging by the uptime, the last reboot must have happened at April 13 at 11:28. During that time, only the line "wsdd2[3431]: 'Terminated' signal received." seems a little suspicious to me: .. but the following logs after that indicate a startup, rather than a shutdown. The syslog server logs indicate that my assumption of multiple reboots was wrong. I have no clue in which direction to investigate so I have attached the syslog server file and diagnostic logs. For obvious reasons the system is not used and not under any kind of load when it happens. Any pointers or tips for debugging this would be greatly appreciated. tower-diagnostics-20240413-1325.zip syslog-20240413-1325.log

Oh, i was not aware of that the order in the config matters. That is a pretty big gotcha! With the _appended_ line it actually works as expected👍. Does this mean the container inherits all capabilities from the host? If i figure out the right one, could i just add it via "lxc.cap.add"? i assume it is "net_admin" The init command is there because I just folloewed the NixOS wiki. But you are right, it works fine with

Thank you for looking into this with me. I realized that i have mixed up two different problems: nesting was enabled for NixOS. The two mounts allow it to work properly, but are unrelated to tailscale or '/dev/net/tun' I have updated the config but there is no change in behavior. The error messages are the same. For completeness sake, here is the current config: # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0 lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0 # Allow Tailscale to work lxc.cap.drop = lxc.cgroup2.devices.allow = c 10:200 rwm lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file # Distribution configuration lxc.include = /usr/share/lxc/config/common.conf lxc.arch = x86_64 # According to NixOS Wiki lxc.init.cmd = /sbin/init # Container specific configuration lxc.rootfs.path = btrfs:/mnt/cache/lxc/morbo/rootfs lxc.uts.name = morbo # Network configuration lxc.net.0.type = veth lxc.net.0.flags = up lxc.net.0.link = br0 lxc.net.0.name = eth0 lxc.net.0.hwaddr=52:54:00:72:04:38 lxc.start.auto=1

I have tried this but it results in an invalid configuration. I'm quite a noob to how LXC and apparmor/selinux relate, but from this thread i gathered that Unraid does not use apparmor -> hence the invalid configuration error when i uncomment the "lxc.include" line. # <content of /usr/share/lxc/config/nesting.conf> # Use a profile which allows nesting lxc.apparmor.profile = lxc-container-default-with-nesting # Add uncovered mounts of proc and sys, else unprivileged users # cannot remount those lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0 lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0 This is why i copied over the two other 'lxc.mount' lines over to the container's config. Thinking about it, i may just be missing the equivalent setting for SELinux... 🤔 My hunch says the /dev/net/tun device is mapped propery but the tailscale process is lacking permissions to modify it.

Hello again, i've been tinkering with the container and noticed some difference to my other server. I'm trying to run Tailscale inside the container, following these instructions: https://tailscale.com/kb/1130/lxc-unprivileged tailscaled refuses to work: is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): operation not permitted flushing log. logger closing down getLocalBackend error: createEngine: tstun.New("tailscale0"): operation not permitted `/dev/net/tun` is available inside the container but i suspect some permissions to be off. The Proxmox wiki mentions permissions, but i don't know if the mappings are the same in Unraid: https://pve.proxmox.com/wiki/OpenVPN_in_LXC Here is the container's config: # Template used to create this container: /usr/share/lxc/templates/lxc-download # Parameters passed to the template: --dist nixos --release 23.11 --arch amd64 # Template script checksum (SHA-1): 78b012f582aaa2d12f0c70cc47e910e9ad9be619 # For additional config options, please look at lxc.container.conf(5) # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0 lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0 # Allow Tailscale to work lxc.cgroup2.devices.allow = c 10:200 rwm #lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file lxc.mount.entry = /dev/net dev/net none bind,create=dir # Distribution configuration lxc.include = /usr/share/lxc/config/common.conf lxc.arch = x86_64 # According to NixOS Wiki lxc.init.cmd = /sbin/init # Container specific configuration lxc.rootfs.path = btrfs:/mnt/cache/lxc/morbo/rootfs lxc.uts.name = morbo # Network configuration lxc.net.0.type = veth lxc.net.0.flags = up lxc.net.0.link = br0 lxc.net.0.name = eth0 lxc.net.0.hwaddr=52:54:00:72:04:38 lxc.start.auto=1 Any pointers would be appreciated 😊

Great job! I can confirm the fix works on my end as well Thank you!

I can confirm everything you just said. The immutable bit seems to date back quite a while: https://github.com/NixOS/nixpkgs/commit/3877ec5b2ff7436f4962ac0fe3200833cf78cb8b#commitcomment-19100105 I assume, NixOS being special in it's ways about immutability, this was implemented to keep some declarative guarantees and prevent apps from writing where they shouldn't. I have (ab)used this bit myself to make sure no app is accidently writing to a not yet mounted share. Oh, and thanks about the note on the backing storage. 👍

Hello @ich777, thank you for the quick response. Migration is something i want to tackle later. For now, all containers are newly created through the plugin. I have not deleted (or any other way messed with) anything. I first want to see how the system works, before i break it 😉. However, i have not created any dataset myself, i just followed the instructions in the plugin and (i think) it mentioned that the non-existing paths will be created. Please find the logs attached. Before exporting, i have tried again with "test-container", which gave the same error as before. hermes-diagnostics-20240319-1929.zip

Hi, I'm quite new to Unraid but have used other Hypervisors in the past. LXC is very appealing to me because i could migrate some existing containers from my previous servers. I tried your plugin on a fresh installation of Unraid 6.12.8 and LXC plugin verison 2024.03.14. ´Creating containers worked fine but destroying them always results in this error: Destroy Container morbo, please wait until the DONE button is displayed! ERROR, failed to destroy morbo! lxc-destroy: morbo: ../src/lxc/utils.c: _recursive_rmdir: 160 Operation not permitted - Failed to delete "/mnt/cache/lxc/morbo/rootfs/var/empty" lxc-destroy: morbo: ../src/lxc/utils.c: _recursive_rmdir: 160 Directory not empty - Failed to delete "/mnt/cache/lxc/morbo/rootfs/var" lxc-destroy: morbo: ../src/lxc/utils.c: _recursive_rmdir: 160 Directory not empty - Failed to delete "/mnt/cache/lxc/morbo/rootfs" lxc-destroy: morbo: ../src/lxc/storage/dir.c: dir_destroy: 104 Directory not empty - Failed to delete "/mnt/cache/lxc/morbo/rootfs" lxc-destroy: morbo: ../src/lxc/lxccontainer.c: container_destroy: 3007 Error destroying rootfs for morbo lxc-destroy: morbo: ../src/lxc/tools/lxc_destroy.c: do_destroy: 110 Destroying morbo failed I have tried rebooting and disabling/enabling the plugin already. The folders on `/mnt/cache/lxc` are owned by `root`. Is that correct? > stat /mnt/cache/lxc/morbo/rootfs/var/empty File: /mnt/cache/lxc/morbo/rootfs/var/empty Size: 0 Blocks: 0 IO Block: 4096 directory Device: 0,43 Inode: 279359 Links: 1 Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-18 09:02:41.724269344 -0700 Modify: 2024-03-18 09:02:41.724269344 -0700 Change: 2024-03-18 09:02:41.728269369 -0700 Birth: 2024-03-18 09:02:41.724269344 -0700