buxel

I managed to pull some data from the drive. I do not recommend any of this (because evidently, I don't know what I'm doing ;) ) but will post it for any poor soul in a similar situation. Source: https://serverfault.com/questions/1001813/zpool-fails-to-import-raidz3-pool-despite-sufficient-replicas-available # Disable ZFS data verification. Flags will be reset after reboot echo 0 >/sys/module/zfs/parameters/spa_load_verify_data echo 0 >/sys/module/zfs/parameters/spa_load_verify_metadata # Mount the pool readonly zpool import -o readonly=on cache -f

Thanks for the tip, i will give it a try.

zpool clear does not find the pool. I think it needs to be imported for that? zpool import -FX cache took a while and then crashed the server. A few more questions / straws Is there any way i can get a better understanding what is going wrong? I/O error is pretty generic. Googling was inconclusive, ranging from ZFS bugs to dying hardware. I understand an SSD can die but the timing is just odd and it has not been used much. SMART is also fine. Here is where my lack of knowledge becomes obvious: could i mount the ZFS filesystem somehow manually "outside" the pool? Does "invalid label" refer to the disk label? # zpool import -d /dev/disk/by-id/ata-Crucial_CT250MX200SSD1_1529101BF7ED config: cache UNAVAIL insufficient replicas ata-Crucial_CT250MX200SSD1_1529101BF7ED UNAVAIL invalid label

I came across zpool clear in this thread: https://forum.proxmox.com/threads/zfs-i-o-error.33670/ SMART reports no errors for the drive. Are there any non-obvious consequences of running it on my pool?

You are right, same result root@Tower:~# zpool import -o readonly=on cache cannot import 'cache': I/O error Destroy and re-create the pool from a backup source.(I forgot to attach the diagnostics before. Edited my post and added them now)

Hello, After long inactivity i founds some time to tinker with my server again. Sadly, this kicked of a series of unlucky events which may have lead to data loss for me. This is a request for help to maybe still salvage some data. My ZFS knowledge is very limited and I'm most likely missing something. I have a very simple setup with two drives and a SSD as cache pool. The cache hosts the appdata of my containers. I had automated backups of appdata to the array running for a bit but it stopped working end of last year. I am solely to blame for not fixing backups in time but life sometimes just gets in the way. The server was mostly idle/unused. Short summary of what happened: The system was running fine I decided to finally tackle backups. I noticed some warnings, checked dmesg and noticed some errors regarding squashfs and xz, sounded like the USB drive was fried Got a new drive, put unraid on it and restored my config there System booted without errors in dmesgbut now my cache pool shows an error: "Unmountable: wrong or no filesystem" I searched for the error and most post led me to believe that my data is gone now. Some things i have tried from other posts: root@Tower:~# zpool list no pools available root@Tower:~# zpool import pool: cache id: 7139977111747743935 state: ONLINE status: Some supported features are not enabled on the pool. (Note that they may be intentionally disabled if the 'compatibility' property is set.) action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: cache ONLINE sdb1 ONLINE root@Tower:~# zpool import cache cannot import 'cache': I/O error Destroy and re-create the pool from a backup source. root@Tower:~# zpool import -d /dev/disk/by-id/ata-Crucial_CT250MX200SSD1_1529101BF7ED pool: cache id: 7139977111747743935 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: cache UNAVAIL insufficient replicas ata-Crucial_CT250MX200SSD1_1529101BF7ED UNAVAIL invalid labelAny advice would be appreciated. tower-diagnostics-20250604-1654.zip

@JorgeB thank you for the confirmation. I just thought some conflict with Unraid running in memory and how i installed the docker plugin. I'll do my research on rclone now. EDIT: for anyone with the same problem, this issue is tracked here: https://github.com/rclone/rclone/issues/7466

Hi @JorgeB, Thank you for the advice. I followed the steps and am back to an image based docker for now. I re-installed the plugin but adapted the config/cache paths to point to `/var/lib/docker/plugins/rclone/cache` instead of (according to the manual) `/var/lib/docker-plugins/rclone/cache`. I checked how /var/lib/docker is mounted and figured plugins should go there... right? After another reboot, the plugin seemed to be gone again, but with a different error. root@Tower:~# docker plugin ls ID NAME DESCRIPTION ENABLED 4e58ab121161 rclone:latest Rclone volume plugin for Docker false root@Tower:~# docker plugin enable rclone Error response from daemon: dial unix /run/docker/plugins/4e58ab121161290ac437b5ae49ed12822739f67e5421f2d343c389178a1019d2/rclone.sock: connect: no such file or directory The plugin somehow is still registered, but disabled. Enabling it, gives the error shown above. To my understanding my adapted paths should be relevant, once the plugin loads. So even if they are not correct, there is something else amiss.

@JorgeB sorry for pinging you directly - you seem to be knowledgeable around here. I kept trying a few more times with the same result. I'm running out if ideas. Any hints how i could get to the root of this?

Hi, I'm trying to install the docker volume plugin for rclone, as described here: https://rclone.org/docker/ If i follow all the steps, I'm also able to create the volume mentioned in the documentation with my own sftp host. So in general, the plugin seems to work fine. As soon as i reboot the system, the docker daemon will not come back and the logs are spammend with: Sep 26 21:02:14 Tower unraid-api[13420]: ⚠️ UNRAID API crashed with exit code 1 Sep 26 21:02:20 Tower unraid-api[13669]: ✔️ UNRAID API started successfully! Sep 26 21:02:21 Tower unraid-api[13669]: ⚠️ Caught exception: connect ECONNREFUSED /var/run/docker.sock The docker logs indicate a crash, but I'm not sure what to make of it: time="2024-09-26T20:07:42.215423328+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2024-09-26T20:07:42.215478433+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=i o.containerd.ttrpc.v1 time="2024-09-26T20:07:42.215492900+02:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 ty pe=io.containerd.event.v1 time="2024-09-26T20:07:42.215502680+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io .containerd.ttrpc.v1 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x263cfdb] goroutine 90 [running]: github.com/docker/docker/pkg/plugins.(*Client).callWithRetry(0x0, {0x80d939, 0x19}, {0xb65d20, 0xc00052a2a0}, 0x1, {0xc000ac4298, 0x1, 0x1c57106?}) /go/src/github.com/docker/docker/pkg/plugins/client.go:169 +0x15b github.com/docker/docker/pkg/plugins.(*Client).CallWithOptions(0x4d79a0?, {0x80d939, 0x19}, {0x455780, 0x379c380}, {0x2fc6a0, 0xc000adc820}, {0xc00 0ac4298, 0x1, 0x1}) /go/src/github.com/docker/docker/pkg/plugins/client.go:122 +0x185 github.com/docker/docker/volume/drivers.(*volumeDriverProxy).Capabilities(0xc0000322d0) /go/src/github.com/docker/docker/volume/drivers/proxy.go:244 +0xfa github.com/docker/docker/volume/drivers.(*volumeDriverAdapter).getCapabilities(0xc00052a270) /go/src/github.com/docker/docker/volume/drivers/adapter.go:93 +0x68 github.com/docker/docker/volume/drivers.(*volumeDriverAdapter).Scope(0xc00002f8c0?) /go/src/github.com/docker/docker/volume/drivers/adapter.go:85 +0x19 github.com/docker/docker/volume/drivers.validateDriver({0xb90268, 0xc00052a270}) /go/src/github.com/docker/docker/volume/drivers/extpoint.go:117 +0x2b github.com/docker/docker/volume/drivers.(*Store).lookup(0xc000ac2c30, {0xc00090f8b0, 0xd}, 0x0) /go/src/github.com/docker/docker/volume/drivers/extpoint.go:96 +0x2cc github.com/docker/docker/volume/drivers.(*Store).GetDriver(...) /go/src/github.com/docker/docker/volume/drivers/extpoint.go:149 github.com/docker/docker/volume/service.lookupVolume({0x0?, 0x0?}, 0x0?, {0xc00090f8b0?, 0x0?}, {0xc00090f8a0, 0x9}) /go/src/github.com/docker/docker/volume/service/store.go:769 +0x72 github.com/docker/docker/volume/service.(*VolumeStore).restore.func2({{0xc00090f8a0, 0x9}, {0xc00090f8b0, 0xd}, 0xc000ac3a10, 0xc000ac3a40}) /go/src/github.com/docker/docker/volume/service/restore.go:37 +0x186 created by github.com/docker/docker/volume/service.(*VolumeStore).restore /go/src/github.com/docker/docker/volume/service/restore.go:31 +0xbf panic: invalid freelist page: 2, page type is leaf goroutine 1 [running, locked to thread]: go.etcd.io/bbolt.(*freelist).read(0x0?, 0x1516f0218000) /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/freelist.go:267 +0x22e go.etcd.io/bbolt.(*DB).loadFreelist.func1() /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:415 +0xb8 sync.(*Once).doSlow(0xc000862408?, 0x10?) /usr/local/go/src/sync/once.go:74 +0xc2 sync.(*Once).Do(...) /usr/local/go/src/sync/once.go:65 go.etcd.io/bbolt.(*DB).loadFreelist(0xc000862240?) /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:408 +0x47 go.etcd.io/bbolt.Open({0xc00082c300, 0x23}, 0x83e7e0?, 0xc0009fecf8) /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:290 +0x40c github.com/docker/docker/volume/service.NewStore({0xc0004d4dc0, 0xf}, 0xc00085e690, {0xc0009fede8, 0x1, 0x0?}) /go/src/github.com/docker/docker/volume/service/store.go:108 +0x2f1 github.com/docker/docker/volume/service.NewVolumeService({0xc0004d4dc0, 0xf}, {0xb8ac08?, 0xc0005d37a0}, {0x17?, 0x7fb966?, {0x0?, 0xc000ac78b0?}}, {0xb67b80, 0xc00073ab40}) /go/src/github.com/docker/docker/volume/service/service.go:50 +0x1f3 github.com/docker/docker/daemon.NewDaemon({0xb8a000?, 0xc000ac67d0}, 0xc00070ea00, 0xc0005d37a0, 0xc000062e20) /go/src/github.com/docker/docker/daemon/daemon.go:970 +0x18a9 main.(*DaemonCli).start(0xc0004e0e40, 0xc0006c5c70) /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:232 +0xae9 main.runDaemon(...) /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14 main.newDaemonCommand.func1(0xc00070a000?, {0xc0006c5ce0?, 0x7?, 0x7?}) /go/src/github.com/docker/docker/cmd/dockerd/docker.go:39 +0x94 github.com/spf13/cobra.(*Command).execute(0xc00070a000, {0xc000052090, 0x7, 0x7}) /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:916 +0x862 github.com/spf13/cobra.(*Command).ExecuteC(0xc00070a000) /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:1044 +0x3bd I'm using docker in folder mode. To recover from this problem, i stopped the service and reverted to a previous snapshot (cache drive is ZFS). Do you have any recommendation/guides how to install docker plugins in general? I tried a forum search but "docker plugin" is quite ambiguous on this forum. tower-diagnostics-20240926-2118.zip

Hi Jorge, thanks for chiming in. That's what I assumed as well but the timing is just too odd. This behavior started exactly the day i made the switch from OMV to Unraid 😒 It is hard for me to verify this after swithcing, since the former boot drive is now the ssd cache and all HDDs are in the array. Even if i boot some live distro, it would not make any use of the HDD. Could it be something driver related? My system is nothing fancy, just an old Skylake PC.

Just to be sure, - i have reset al BIOS settings and re-plugged all cables on the mainboard/drives. - I also ran two memtest passes for ~3h with no errors. - made sure i'm not using macvlan after reading the latest changelog - used a different USB port for the Unraid flash drive Still getting these crashes. Is there anything I'm missing to gather more information about the root cause?

I just realized I had "mirror to flash" accidently disabled. I have turned it on now and will provide a new set of diagnostic logs after the next crash/reboot. EDIT: here we go, another reboot happened at 14:23. It did not log much after I enabled mirroring to flash: Apr 13 14:09:01 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="12322" x-info="https://www.rsyslog.com"] start Apr 13 14:16:00 Tower root: Fix Common Problems Version 2024.03.29 Apr 13 14:16:05 Tower root: Fix Common Problems: Warning: Syslog mirrored to flash Apr 13 14:16:05 Tower root: Fix Common Problems: Other Warning: Unassigned Devices Plus not installed ** Ignored that's it 🤔 tower-diagnostics-20240413-1425.zip syslog-127.0.0.1.log

Hello, I just recently made the switch from OMV to Unraid on my small NAS. I'm still in the progress of adding all services to OMV but already hit a road block early on. I observed daily parity checks caused by "unclean shutdown". Looking at the diagnostic logs, I did not see any cause for a reboot / errors and I suspect that more than one reboot might have happened, so the "last" logs mirrored to flash are also not useful. So this morning, I set up the local syslog server and made Unraid log to itself, persisting in appdata. Judging by the uptime, the last reboot must have happened at April 13 at 11:28. During that time, only the line "wsdd2[3431]: 'Terminated' signal received." seems a little suspicious to me: .. but the following logs after that indicate a startup, rather than a shutdown. The syslog server logs indicate that my assumption of multiple reboots was wrong. I have no clue in which direction to investigate so I have attached the syslog server file and diagnostic logs. For obvious reasons the system is not used and not under any kind of load when it happens. Any pointers or tips for debugging this would be greatly appreciated. tower-diagnostics-20240413-1325.zip syslog-20240413-1325.log

Oh, i was not aware of that the order in the config matters. That is a pretty big gotcha! With the _appended_ line it actually works as expected👍. Does this mean the container inherits all capabilities from the host? If i figure out the right one, could i just add it via "lxc.cap.add"? i assume it is "net_admin" The init command is there because I just folloewed the NixOS wiki. But you are right, it works fine with

Thank you for looking into this with me. I realized that i have mixed up two different problems: nesting was enabled for NixOS. The two mounts allow it to work properly, but are unrelated to tailscale or '/dev/net/tun' I have updated the config but there is no change in behavior. The error messages are the same. For completeness sake, here is the current config: # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0 lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0 # Allow Tailscale to work lxc.cap.drop = lxc.cgroup2.devices.allow = c 10:200 rwm lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file # Distribution configuration lxc.include = /usr/share/lxc/config/common.conf lxc.arch = x86_64 # According to NixOS Wiki lxc.init.cmd = /sbin/init # Container specific configuration lxc.rootfs.path = btrfs:/mnt/cache/lxc/morbo/rootfs lxc.uts.name = morbo # Network configuration lxc.net.0.type = veth lxc.net.0.flags = up lxc.net.0.link = br0 lxc.net.0.name = eth0 lxc.net.0.hwaddr=52:54:00:72:04:38 lxc.start.auto=1

I have tried this but it results in an invalid configuration. I'm quite a noob to how LXC and apparmor/selinux relate, but from this thread i gathered that Unraid does not use apparmor -> hence the invalid configuration error when i uncomment the "lxc.include" line. # <content of /usr/share/lxc/config/nesting.conf> # Use a profile which allows nesting lxc.apparmor.profile = lxc-container-default-with-nesting # Add uncovered mounts of proc and sys, else unprivileged users # cannot remount those lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0 lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0 This is why i copied over the two other 'lxc.mount' lines over to the container's config. Thinking about it, i may just be missing the equivalent setting for SELinux... 🤔 My hunch says the /dev/net/tun device is mapped propery but the tailscale process is lacking permissions to modify it.

Hello again, i've been tinkering with the container and noticed some difference to my other server. I'm trying to run Tailscale inside the container, following these instructions: https://tailscale.com/kb/1130/lxc-unprivileged tailscaled refuses to work: is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): operation not permitted flushing log. logger closing down getLocalBackend error: createEngine: tstun.New("tailscale0"): operation not permitted `/dev/net/tun` is available inside the container but i suspect some permissions to be off. The Proxmox wiki mentions permissions, but i don't know if the mappings are the same in Unraid: https://pve.proxmox.com/wiki/OpenVPN_in_LXC Here is the container's config: # Template used to create this container: /usr/share/lxc/templates/lxc-download # Parameters passed to the template: --dist nixos --release 23.11 --arch amd64 # Template script checksum (SHA-1): 78b012f582aaa2d12f0c70cc47e910e9ad9be619 # For additional config options, please look at lxc.container.conf(5) # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0 lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0 # Allow Tailscale to work lxc.cgroup2.devices.allow = c 10:200 rwm #lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file lxc.mount.entry = /dev/net dev/net none bind,create=dir # Distribution configuration lxc.include = /usr/share/lxc/config/common.conf lxc.arch = x86_64 # According to NixOS Wiki lxc.init.cmd = /sbin/init # Container specific configuration lxc.rootfs.path = btrfs:/mnt/cache/lxc/morbo/rootfs lxc.uts.name = morbo # Network configuration lxc.net.0.type = veth lxc.net.0.flags = up lxc.net.0.link = br0 lxc.net.0.name = eth0 lxc.net.0.hwaddr=52:54:00:72:04:38 lxc.start.auto=1 Any pointers would be appreciated 😊

Great job! I can confirm the fix works on my end as well Thank you!

I can confirm everything you just said. The immutable bit seems to date back quite a while: https://github.com/NixOS/nixpkgs/commit/3877ec5b2ff7436f4962ac0fe3200833cf78cb8b#commitcomment-19100105 I assume, NixOS being special in it's ways about immutability, this was implemented to keep some declarative guarantees and prevent apps from writing where they shouldn't. I have (ab)used this bit myself to make sure no app is accidently writing to a not yet mounted share. Oh, and thanks about the note on the backing storage. 👍

Hello @ich777, thank you for the quick response. Migration is something i want to tackle later. For now, all containers are newly created through the plugin. I have not deleted (or any other way messed with) anything. I first want to see how the system works, before i break it 😉. However, i have not created any dataset myself, i just followed the instructions in the plugin and (i think) it mentioned that the non-existing paths will be created. Please find the logs attached. Before exporting, i have tried again with "test-container", which gave the same error as before. hermes-diagnostics-20240319-1929.zip

Hi, I'm quite new to Unraid but have used other Hypervisors in the past. LXC is very appealing to me because i could migrate some existing containers from my previous servers. I tried your plugin on a fresh installation of Unraid 6.12.8 and LXC plugin verison 2024.03.14. ´Creating containers worked fine but destroying them always results in this error: Destroy Container morbo, please wait until the DONE button is displayed! ERROR, failed to destroy morbo! lxc-destroy: morbo: ../src/lxc/utils.c: _recursive_rmdir: 160 Operation not permitted - Failed to delete "/mnt/cache/lxc/morbo/rootfs/var/empty" lxc-destroy: morbo: ../src/lxc/utils.c: _recursive_rmdir: 160 Directory not empty - Failed to delete "/mnt/cache/lxc/morbo/rootfs/var" lxc-destroy: morbo: ../src/lxc/utils.c: _recursive_rmdir: 160 Directory not empty - Failed to delete "/mnt/cache/lxc/morbo/rootfs" lxc-destroy: morbo: ../src/lxc/storage/dir.c: dir_destroy: 104 Directory not empty - Failed to delete "/mnt/cache/lxc/morbo/rootfs" lxc-destroy: morbo: ../src/lxc/lxccontainer.c: container_destroy: 3007 Error destroying rootfs for morbo lxc-destroy: morbo: ../src/lxc/tools/lxc_destroy.c: do_destroy: 110 Destroying morbo failed I have tried rebooting and disabling/enabling the plugin already. The folders on `/mnt/cache/lxc` are owned by `root`. Is that correct? > stat /mnt/cache/lxc/morbo/rootfs/var/empty File: /mnt/cache/lxc/morbo/rootfs/var/empty Size: 0 Blocks: 0 IO Block: 4096 directory Device: 0,43 Inode: 279359 Links: 1 Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-18 09:02:41.724269344 -0700 Modify: 2024-03-18 09:02:41.724269344 -0700 Change: 2024-03-18 09:02:41.728269369 -0700 Birth: 2024-03-18 09:02:41.724269344 -0700