Januszmirek

So I finally solved the issue. Turns out it wasn't a corrupt docker image or a problem with docker networks or anything else that I initially suspected. One other issue I was encountering for months now (but somehow did not connect it with this one) is that every night I got notifications about cache disk space filling out (to 100%). I had no idea what was this about as in the morning everything was fine and cache disk was maybe 60% filled. I then forgot I created about 600gb VM on my array - the space i needed for it was too big for my cache. What I however forgot to change after creating the vm was to not attempt moving the VM to cache. Basically as show below: I really did not need this VM anymore, so deleted it and boom! all problems magically disappeared all together. No more btrfs or syslog errors. Doubt this would help anyone, but just wanted to let you know the issue is resolved.

You can still use your.domain. No ports forwarding on the router needed. Instead of Proxy Hosts in NPM, use Streams. Helpful link here: https://gist.github.com/Lvdwardt/707d9c3fb4581d38102399f91a21c9c6 Go straight to no 5. You don't have to set up anything else. It works like a charm. Good luck.

How about this one: Mar 26 08:58:40 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224475136 have 0 Mar 26 08:58:40 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224524288 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224507904 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224475136 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224524288 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 2 want 20224524288 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224507904 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224475136 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224524288 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 2 want 20224524288 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224507904 have 0 Mar 26 08:58:45 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 20224475136 have 0 Mar 26 09:12:57 Tower unraid-api[8774]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:04 Tower unraid-api[11408]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:11 Tower unraid-api[14508]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:18 Tower unraid-api[17221]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:25 Tower unraid-api[20009]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:32 Tower unraid-api[22508]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:39 Tower unraid-api[25349]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:46 Tower unraid-api[27752]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:13:53 Tower unraid-api[30200]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:00 Tower unraid-api[32595]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:07 Tower unraid-api[2590]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:14 Tower unraid-api[5324]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:21 Tower unraid-api[7854]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:28 Tower unraid-api[9611]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:35 Tower unraid-api[11448]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' Mar 26 09:14:42 Tower unraid-api[13675]: ⚠️ Caught exception: EIO: i/o error, scandir '/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4c765a904e781a4ced957e91ad602ca741043834bba888d3de0d59fca040f5b0/work' It's getting ridiculous now. Some containers start to behave really weird. Nginx won't generate new ssl certs. Plex web does not open. I tried to remove the container but 'Execution error Server error' pop up shows up and I am unable to remove the container. I will try to restart the unraid but this is becoming a chore and a far cry from rock solid experience I had with the machine for the last few years.

I don't need macvlan. I'm not sure why it was set up like this in a first place. Anyway, docker rebuilt didn't help. Woke up this morning to find out not only containers but also unraid web interface was not available. Hard reset later, and rebuilt of docker again, this time with ipvlan seems to work so far. At least no syslog errors in log. I will monitor those btfrs errors now. Thanks for the hint with macvlan;) EDIT: Happiness didn't last long. Now, an hour after docker rebuilt, full system crash - only reboot helped. New btfrs errors from log: Mar 24 20:19:25 Tower kernel: BTRFS info (device loop4): using crc32c (crc32c-intel) checksum algorithm Is my cache drive dying? it still shows 0 errors.

I have rebuilt docker and restarted the machine. Syslog errors came back right away Mar 23 14:10:10 Tower rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Mar 23 14:10:10 Tower rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] I wonder if this has anything to do with the network type setting in docker? Currently this is setup as follows: Docker custom network type: macvlan Could changing this have impact on these errors? For btfrs errors I will probably need to wait a day or two as these are usually happen during night hours. Hopefully the docker rebuilt helped solve this.

Thanks! I'll do that and report back on how it went. I don't suppose I have set up any custom docker network, all containers Network type is 'host' or 'bridge'.

I did. See the syslog-old.txt for btrfs errors.

I started recently to encounter lots of BTRFS and syslog errors. I would not be normally bothered but recently every few days I wake up to find out my docker containers are basically not working. Arrary restart doesn't help. Only unraid restart helps. But the issue comes back a few days later. What I usually see in log prior to restart is below. Mar 18 04:55:22 Tower rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Mar 18 04:55:22 Tower rsyslogd: file '/mnt/user/appdata/syslog-192.168.50.5.log'[2] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Mar 21 08:24:10 Tower rsyslogd: omfwd/udp: socket 5: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Mar 21 08:24:10 Tower rsyslogd: omfwd: socket 5: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Mar 15 15:54:31 Tower kernel: BTRFS error (device loop2: state EA): bad tree block start, mirror 1 want 11436949504 have 0 My array disks, cache disk and flash drive show 0 errors. I have included old log in the zip file (syslog-old.txt), maybe it will be useful for someone to help me out what's wrong with my server. I tried google both issues but nothing helpful was found. Anyway thanks in advance to anyone who has any idea what's going on. tower-diagnostics-20240321-1753.zip

Fair point. So I just go and install the same container under a different name and it becomes a completely separate instance I can play with?

Hi, not sure if this was ever asked about. I am having issues with different recognition of the credentials on a mac machine (using fire fox plugin) vs safari plugin on my iPhone. This is with services on my home server where the IP is the same but each service is using different port. On my mac this works flawlessly where each service' address is defined as 'regular impression'. So when i punch say 192.168.10.2:6060 address it only gives me a single credentials to fill in the login and password form as seen below, and not say 15 different options for every service set up under 192.168.10.2. On my iPhone this sadly is not the case. First of all the suggested credentials are NOT for the service I am trying to access. And second, I need to manually expand the drop-down list to pick up the right credentials. Not only the port is not recognised but because the drop-down menu only shows the logins, not the service address or name, it is really frustrating to actually remember by heart which credentials on the list is the right one. If anyone knows how to solve this issue on the mobile device, please share the solution;)

That's great. Can you confirm that you were actually able to recover other users account data? I have multiple accounts set up on my vaultwarden instance and am interested to know if all these accounts can be restored when needed. Thanks;)

Quick question about backup and potential restore. So i have vaultwarden set up for myself and have about 5-10 friends and family members having their accounts on this instance as well. Given the disastrous consequences of loosing access to all this data, I thought of some sort of backup solution, just in case. So the way I have configured it currently is I use Appdata Backup plugin to automagically do the copy of entire appdata/valutwarden folder on nightly basis. Then I set up rcolne to have it copy to a remote cloud every morning. Now my question basically is this: Is having the entire appdata/valutwarden folder copy enough to restore everything back to where it was from the last backup? And also if this is enough, in the event of potential rebuilt do I just go and install fresh new instance of vaultwarden container and replace all files in appdata/valutwarden from backup copy? I would really like to be sure that what I am doing now is enough and restoring in case of emergency would be a breeze. All comments and hints most appreciated.

Fair enough, tbh WG is the least of my worries atm. I practically lost all my unraid services exposed to external world;( Thanks for chip in anyway.

But id DID work previously. Are you saying that with the new update this functionality of having both worlds just disappeared?

After the update to 6.12.4 I am more than unhappy with the issues I'm facing. First noticeable issue was wireguard that just stopped working correctly. Until 6.11.x it worked flawlessly. Now I cannot make it to have a client to connect to my unraid and use internet at the same time. I mean i can browse all containers webgui but internet does not work. I had this issue previously but it was incorrect DNS configuration on my part, after I finally figured this out it and was flawless on any client ever since. After the update to 12.4 I can connect through wg to unraid but internet on any of the clients is no more. Yesterday I noticed other network related multiple issues. My plex server suddenly stopped being available remotely. Wordpress page hosted on my unraid is not available anymore, Unraid Connect doesn't work anymore (API KEY: NETWORK_ERRORNETWORK: Unable to connect to https://mothership.unraid.net), my system log is full of network errors and so on. I know there were some maclav/ipvlan changes made with this release and I did changed a few settings accordingly to release notes - probably misconfiguration on my part is what caused the issue but have no idea how to reverse these so i can get to where it was just a week ago before i updated. I hope it is all down to some configuration but no idea where to start so some steer would be much appreciated. Diagnostic log attached. Happy to provide more input if needed. Thanks. tower-diagnostics-20230916-1041.zip

Had exact same issue with my config. Tried to troubleshoot it for hours with no luck. Finally switched to binhex container and it works flawlessly.

I am experiencing similar issue with speeds. having said that I am not syncing my local Mac to cache drive on Unraid, but rather to a share set up on one of array discs (hdd). My initial syncing speeds are atrocious (20-100kbps) increasing to 25-100(occasionally) Mbps. Obviously my Mac machine and Unraid are on same 1gbit network. I am using latest 1.22 and 1.23 versions of app on mac and unraid. Both machines show TCP LAN connection type. Where do I start troubleshooting the issue?

I also have a problem with setting up this container. The problem in my case appears to be with ports not paths. For the world I am unable to change the port settings. Even though the ports in Edit appear to be set properly and don't interfere with other container ports: After saving the config, the container sets the ports on its own regardless of the settings I made: I cant use above ports for omada (8043, 8088, 8043) as I use it for a different container and don't want to mess things up with that other container. This is the first instance I see such behaviour with the container, where editing the ports does not have any effect on actual changes to container. Usually the workaround I used was to change network type from host to bridge, but in bridge mode omada webui mode is not available;( I also tried to delete the default ports in the container which use Variable config type, and set it as Ports (instead of Variable) - still no joy;( Any ideas on how to force the port change? Btw, when I turn off the other container, omada works great, so its not the paths config issue, but ports in my case.

You're a star! It worked. I can see my 'c:\' drive on VM2 and the capturing process is now working fine. Thank you once and again!

100% correct. Exactly;) See below;) <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10test1</name> <uuid>c8ab1df9-8dc9-f570-3afc-d40f7e956e84</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="Windows_10b.png" os="windows10"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='10'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c8ab1df9-8dc9-f570-3afc-d40f7e956e84_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='2' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/Dane/Windows/imagetest/Windows 10test1/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:d9:d5:39'/> <source bridge='br0'/> <model type='e1000-82545em'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> </domain>

So I don't want to use bare metal as for my needs it is not preferable and have it own issues, that are beside the point. I would want to use VM1 (Server with Windows server 2019 with WDS) and a VM2 (Client with Windows 10). Both of these are currently set up on my unraid as separate VMs. When I set up VM2, I edited the xml to define certain boot order. In my case it is 1. network boot (booting into PXE on VM1), and 2. booting into .img file with VM2. The problem is that when you boot into 1. (PXE), you don't have access to virtual disk of VM2 (.img). And therefore you can't capture that image. To put it into perspective and give you an example: when you use bootable USB drive with say linux distro on your laptop, you still have access not only to files on your USB stick but most importantly to your ssd disk in your laptop and you can install the OS onto the disk. Here I have an issue where I boot from USB but have access to USB only but not to ssd disk. Does that make more sense now? I am afraid this is actually Windows question not linux/unraid related, as once you boot into pxe you are effectively inside WDS environment. I am looking for a command that I could somehow mount/append that virtual disk of VM2 (.img file). Quite frankly I don't even know if this is even possible.

Genius, worked like a charm. Thanks!;) Ok, I won't be starting a new topic as I hope you guys can help me out with this one too. So as a part of messing around with WDS I am at the point where I finished setting up a 'windows10' (set up as a VM on unraid) and now I need to capture it into a single .wim file (using dism command) that is then used to create golden Windows 10 image than you can install on number of machines over PXE. You can usually do this using WindowsPE tool or by booting into PXE on WDS server. With bare metal machines this works like a charm. Because when I use a bare metal machine and boot to PXE I still have access to my "C:\" drive on that machine to be able to capture it into an image (.wim file). The issue I an encountering is the way VMs work on Unraid: I was able to edit the windows10 VM .xml to include booting into either PXE (Thanks @ghost82) or into WindowsPE.iso. This works just fine. However the problem is that at this point the actual virtual disk (.img file) with my windows10 is not mounted yet. And therefore I can not capture it;(. So the question is: how do I mount that .img file created in unraid so that PXE or WinPE can have access to it, so I can capture it with dism command? Hope that is roughly clear but if I did not give enough details, just let me know and I will. Thanks again;)

Genius, worked like a charm. Thanks!;)

So im trying to play with WDS setup. I have successfully set up a WDS server in a VM (lets call it Server1), which can be reached and works no problem on bare metal machines in the same network. Where im stuck is I try to set up another Windows VM which could reach Server1 and install OS from it. Based on the above advice, i have added the below code into the xml file. <interface type='bridge'> <mac address='52:54:00:d9:d5:39'/> <source bridge='br0'/> <model type='virtio'/> <boot order='1'/> </interface> The PXE is is picking up the Server1, and the install process starts.... and then it stops with the following error: "WDSclient: an error occured while trying to obtain an IP address from the DHCP server. Please check to ensure that there is an operational DHCP server on this network segment". Read on interwebs this might be a driver issue. If someone has encountered the same problem and care to share a solution, you're most welcomed. Thanks.

Finally got it working;) thank you very much. Just for anyone who stumbles upon this thread, in my case the container needed to be set to Host rather than default Bridge mode. I was no able to make it work in Bridge mode. qbit from other repos worked just fine in bridge, but not linuxserv. Not sure why.