golli53
Members
-
Joined
Everything posted by golli53
-
Ok, finally solved it. In case anyone runs into this, `umount -l /dev/loop2` worked
-
I think the docker image on /mnt/cache that's mounted on /dev/loop2 is preventing the unmount. I killed a zombie container process accessing /dev/loop2, but still cannot detach /dev/loop2 and still stuck trying to unmount. Tried everything here: https://stackoverflow.com/questions/5881134/cannot-delete-device-dev-loop0 root@Tower:/# losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop2 0 0 1 0 /mnt/cache/system/docker/docker.img 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 root@Tower:/# lsof /dev/loop2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME container 15050 root 4u FIFO 0,82 0t0 2917 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stdout.log container 15050 root 7u FIFO 0,82 0t0 2917 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stdout.log container 15050 root 8u FIFO 0,82 0t0 2918 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stderr.log container 15050 root 9u FIFO 0,82 0t0 2918 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stderr.log root@Tower:/# kill 15050 root@Tower:/# lsof /dev/loop2 root@Tower:/# losetup -d /dev/loop2 # fails silently root@Tower:/# echo $? 0 root@Tower:/# losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop2 0 0 1 0 /mnt/cache/system/docker/docker.img 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 root@Tower:/# lsof | grep loop2 loop2 12310 root cwd DIR 0,2 440 2 / loop2 12310 root rtd DIR 0,2 440 2 / loop2 12310 root txt unknown /proc/12310/exe root@Tower:/# kill -9 12310 # not sure what this is, but killing it fails root@Tower:/# lsof | grep loop2 loop2 12310 root cwd DIR 0,2 440 2 / loop2 12310 root rtd DIR 0,2 440 2 / loop2 12310 root txt unknown /proc/12310/exe root@Tower:/# modprobe -r loop && modprobe loop # try to reload the module, but it's builtin modprobe: FATAL: Module loop is builtin.
-
I tried to Stop my array and it's currently still stuck on `Retry unmounting disk share(s)...` for the last 30min. Some diagnostics from command line below (I cannot access diagnostics from GUI anymore). Prior to this, I noticed one of my dockers was having weird issues... seemingly stopped after I killed it, but kept being listed as running in `docker ps`. I was using `docker exec` to execute some commands in that container and I think some processes got stuck in the container. root@Tower:/# tail -n 5 /var/log/syslog Apr 28 14:11:36 Tower emhttpd: Unmounting disks... Apr 28 14:11:36 Tower emhttpd: shcmd (43474): umount /mnt/cache Apr 28 14:11:36 Tower root: umount: /mnt/cache: target is busy. Apr 28 14:11:36 Tower emhttpd: shcmd (43474): exit status: 32 Apr 28 14:11:36 Tower emhttpd: Retry unmounting disk share(s)... root@Tower:/# lsof /mnt/cache root@Tower:/# fuser -mv /mnt/cache USER PID ACCESS COMMAND /mnt/cache: root kernel mount /mnt/cache
-
Gotcha. It seemed aggressive for it to change remote permissions by default, as this affects permissions locally on the remote server. For instance, for any home share, this will break SSH authentication for all clients to that remote server. Any read-only permissions would also be permanently changed for all clients. I'll use a script as a workaround - thanks for the suggestion.
-
@dlandon just trying to understand whether this is a bug in unRAID or Unassigned Devices, so I can direct issue management - thanks in advance
-
When I mount an nfs share, the remote directory gets permissions changed to 0777. Since I am mounting my home folder (/home/myname), this screws up SSH authentication on my remote Ubuntu server, since it only works if the home folder is 0755. Right now I have to manually invoke chmod each time after Unassigned Devices mounts the directory.
-
+1 think this is a great idea. In addition to it being generally useful, sometimes I need to manually cancel Mover because it's slowing down my array, but if it's almost done, would prefer not to cancel.
-
Known issue - follow here:
-
Will Wireguard eventually be integrated into unRAID? I think Wireguard is very cool and definitely appreciate the work on new functionalities, but am curious about the design decision here. It's unrelated to storage (and I would personally hesitate to host my VPN server on my storage server in case I need to reboot my storage remotely). Wireguard can also be run as a docker container if the kernel supports (here's a popular one: https://github.com/cmulk/wireguard-docker). The Plugin/Community App/Docker route may be cleaner and less demanding on developer resources.
-
2 issues I've noticed through testing: [detectable on all directory sizes] Each directory listing is ~3-4x slower on average on v6.8 vs v6.72 (2.5s vs 0.7s for a 6k file directory) [detectable for very large directory sizes] Trying to open a single NONEXISTENT file in a 250k file directory on 6.8 (should be microseconds per attempt), every 100 or so attempts, it hangs for ~5s Initially, thought it had to do with concurrent calls, but it doesn't. Concurrent calls are simply additive in terms of execution time and concurrency itself doesn't seem to be the problem, but it makes it more apparent. This is the result of comparison testing of 6.8.0rc7 through stable vs 6.7.2 from a Windows 10 client on gigabit LAN. See SMB config readouts and Python code for benchmarking below. [OLD POST PRE ADDITIONAL TESTING] Noticing this on 6.8.0rc7 (previously using 6.7.2). I'm using a Windows 10 client and have a bunch of Python scripts that list files on an unRAID samba share over gigabit LAN. This slows down to a crawl (several seconds just to refresh a ~100 file directory listing, whether using Python or Windows built-in file explorer). Was running the exact same scripts on 6.7.2 without this issue. Transfer speeds seem normal. When accessing from a different Windows 10 client (at the same time that the first client has many concurrent requests), the directory listings SEEM normal. Were there any settings changed for SMB that I might be able to try tweaking?
-
Of course. In container: ssh-keygen On host: docker cp container-name:/root/.ssh/id_rsa.pub id_rsa.pub sudo sh -c "cat id_rsa.pub >> ~/.ssh/authorized_keys"
-
I got rid of the password prompts by generating an rsa key in the container (ssh-keygen) and copying it into the host's authorized_keys file
-
Thanks for looking into this! That seems to solve half of the problem. No more unknown host issues, but it still prompts me 10 times for the SSH password. Before, it was 10 prompts for each (trust host "yes", password, trust host "yes", password...)
-
Hi @dee31797 First of all, great work on these dockers! With virt-manager, I keep getting SSH known_hosts authenticity prompts (I have to type yes and my password almost 10 times each time I start a new VM). I did a bit of research and maybe it's because there is no known_hosts file in the docker container? I tried mapping a known_hosts file to /root/.ssh/known_hosts, but still have this problem. https://askubuntu.com/questions/190863/how-can-i-disable-ssh-prompt-from-kvm-remote https://stackoverflow.com/questions/48258546/how-to-add-known-hosts-for-passwordless-ssh-within-docker-container-in-docker-co
-
Wrote this after a potential nightmare when mounting a cache drive with an extra empty slot erased my LUKS header Run in ssh from whichever directory you want to store the headers. Will be automatically named with model and serial numbers. Finishes in a few seconds for i in {/dev/sd*,/dev/nvme*}; do if cryptsetup luksDump $i &>/dev/null; then dd if=$i of=`udevadm info --query=all --name=$i | sed -n 's/.*ID_SERIAL=//p'`.img bs=512 count=4096; fi; done
-
I hope the 6.8 changes won't break the ability to GUI mount an encrypted drive (with or without retyping pass) using Unassigned Devices after array start
-
Getting "no such device" and immediate exit with every version - have tried all the way back to 2.07
-
I'm having issues with my UD no longer mounting my 2-disk BTRFS RAID1 pool ("ssd" = nvme0n1p1 set to automount + "ssd2" = nvme1n1p1). The 2nd disk [mounted to "ssd2"] of the pool (not set to automount) doesn't get luksOpen'ed and therefore is "missing," so the automount of the 1st disk fails. Jul 22 12:33:02 Tower unassigned.devices: Mounting 'Auto Mount' Devices... Jul 22 12:33:02 Tower unassigned.devices: Adding disk '/dev/mapper/ssd'... Jul 22 12:33:03 Tower kernel: BTRFS: device fsid 4939602d-ea6f-4dd0-a535-252580b60aac devid 1 transid 11909816 /dev/dm-13 Jul 22 12:33:03 Tower unassigned.devices: Mount drive command: /sbin/mount '/dev/mapper/ssd' '/mnt/disks/ssd' Jul 22 12:33:03 Tower kernel: BTRFS info (device dm-13): disk space caching is enabled Jul 22 12:33:03 Tower kernel: BTRFS info (device dm-13): has skinny extents Jul 22 12:33:03 Tower kernel: BTRFS error (device dm-13): devid 2 uuid f66ef13b-8a93-45da-b73e-0d6a478729b2 is missing Jul 22 12:33:03 Tower kernel: BTRFS error (device dm-13): failed to read chunk tree: -2 Jul 22 12:33:03 Tower kernel: BTRFS error (device dm-13): open_ctree failed Jul 22 12:33:03 Tower emhttpd: Warning: Use of undefined constant luks - assumed 'luks' (this will throw an Error in a future version of PHP) in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 634 Jul 22 12:33:03 Tower unassigned.devices: Mount of '/dev/mapper/ssd' failed. Error message: mount: /mnt/disks/ssd: wrong fs type, bad option, bad superblock on /dev/mapper/ssd, missing codepage or helper program, or other error. Jul 22 12:33:03 Tower unassigned.devices: Partition 'Samsung_SSD_970_PRO_1TB_S462NF0M300357X' could not be mounted... Jul 22 12:33:03 Tower unassigned.devices: Disk with serial 'Samsung_SSD_970_PRO_1TB_S462NF0M311892X', mountpoint 'ssd2' is not set to auto mount and will not be mounted... After every array start, I have to execute luksOpen on "ssd2" and mount "ssd" manually (see below). Then after this, unmount/mount works again in the GUI. root@Tower:~# /usr/sbin/cryptsetup luksOpen /dev/nvme1n1p1 ssd2 --allow-discards --key-file /root/keyfile root@Tower:~# mkdir /mnt/disks/ssd root@Tower:~# /sbin/mount '/dev/mapper/ssd' '/mnt/disks/ssd' I think this happened after I renamed the mounts and physically changed slots, though it seemed to work after one restart, but then broke after the next restart. Have restarted a few times since and it never works anymore, even if I change the names. If I switch the nvme1n1 to automount instead of nvme0n1, it still results in the same error. tower-diagnostics-20190722-1649.zip
-
Thanks for all your work on this - has been invaluable to my unRAID setup! Would it be possible to add ledmon to control backplane/bay HDD LEDs? Would be great to be able to turn these off at night.
-
Expected: - running VM to be hibernated upon restart using GUI. the hibernation state should be preserved after restart - already hibernated VM to preserve hibernation state upon restart using GUI - possibly the same upon start and stop array using GUI What happens: - When hitting restart using GUI, unknown whether VM hibernates. Either way, hibernation state is not preserved after restart - When stopping and starting array using GUI, either 1) qemu-system-x86_64 prevents stop or 2) stop is ok but libvirt.img is locked by qemu-system-x86_64 upon restart preventing libvirt from starting (VM tab won't load) Here's the error message when libvirt.img fails to start: Tower root: /mnt/disks/ssd/system/libvirt/libvirt.img is in-use, cannot mount When I use fuser and ps to find out which processes are locking it, I see the hibernated VM: /usr/bin/qemu-system-x86_64 -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/2f4360e3-a8a0-ca29-a732-4f1191b06173_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 4096 -realtime mlock=off -smp 2,sockets=1,cores=1,threads=2 -uuid 2f4360e3-a8a0-ca29-a732-4f1191b06173 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=26,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x7 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/mnt/disks/ssd/domains/win10/vdisk1.img,format=raw,if=none,id=drive-virtio-disk2,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk2,id=virtio-disk2,bootindex=1,write-cache=on -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1a:cb:11,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,websocket=5700 -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on (Also, my docker containers were listed as locking it as well for some reason which I wasn't sure was normal, but I turned off autostart on all dockers to remove that as a source of noise) Tried: - multiple Win10 (Pro) VMs including completely clean configs - storing libvirt.img and VM domains on /mnt/cache as well as unassigned device - hibernate using 1) host "virsh dompmsuspend <domain> disk", 2) guest start menu, 3) guest PassMark PC Sleep test tool (which confirmed S4 sleep worked) Notes: - when hibernated by any method, the VM shows up as "paused" in GUI, not "stopped" - not sure if this is the norm - when hibernated by any method, virsh shows the domain as "pmsuspended" as expected - I can resume from hibernate fine - qemu-agent is installed and I confirmed the service is running using PowerShell This is a real pain point as I have multiple VMs running at any given time relied on by multiple users, and it's a real pain to make sure everything is shut off whenever I troubleshoot the array or add/change a disk tower-diagnostics-20190719-1543.zip
-
I also see many of my docker containers using libvirt.img when using fuser. I'm concerned whether it might be causing some of my Win10 hibernation issues. Can anyone confirm whether "fuser -c /mnt/user/system/libvirt/libvirt.img" should normally show docker pids?
-
No, but I just got the chance to enable it and do some tests. Now, libvirt simply fails to start up when I hibernate my VM and stop+start my array. Here's the error message: Tower root: /mnt/disks/ssd/system/libvirt/libvirt.img is in-use, cannot mount When I use fuser and ps to find out which processes are locking it, I get the hibernated VM: /usr/bin/qemu-system-x86_64 -name guest=win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/2f4360e3-a8a0-ca29-a732-4f1191b06173_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 4096 -realtime mlock=off -smp 2,sockets=1,cores=1,threads=2 -uuid 2f4360e3-a8a0-ca29-a732-4f1191b06173 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=26,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x7 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/mnt/disks/ssd/domains/win10/vdisk1.img,format=raw,if=none,id=drive-virtio-disk2,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk2,id=virtio-disk2,bootindex=1,write-cache=on -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1a:cb:11,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,websocket=5700 -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on (Also, my docker containers were listed as locking it as well for some reason which I wasn't sure was normal, but I turned off autostart on all dockers to remove that as a source of noise) Tests I've tried (all yield the same error as above): - Manually hibernating the VM - Leaving it running and letting unRAID hibernate it upon array stop - Increasing the time given to VM shutdown/hibernate to 3 minutes under VM settings - Rebooting and retrying above This was a clean Win10 install with just qemu-img installed and no pass-through devices. So, it looks like the VM hibernate option breaks libvirt when the VM resides on an Unassigned Device? PS Not sure if it matters but the Unassigned Device for libvirt.img and my VM images is formatted as btrfs-encrypted. Unraid 6.7.2. Unassigned Devices 2019.06.18
-
I'm not sure that's correct. My VMs currently run from an unassigned NVME drive. I can't stop my array without killing my VMs (the VM settings page becomes disabled and any previously started or paused VMs become stopped upon restart). Does it work differently for you?
-
I often have to restart my array for various reasons and since my family uses unRAID VMs as their go-to personal computers, this is a big hassle to close all work and shutdown. I've tried hibernation, either within the OS or using "virsh domsuspend" to disk. It appears to work, but the VMs show as "paused" not "stopped". I can resume them fine, but if I stop the array, the VMs crash and the hibernation state isn't preserved (one time, I've also encountered the qemu preventing array stop because of access to drives, in which case I had to kill the instance). I've also tested this on a brand new Win10 VM install with qemu-agent running and virtio drivers installed. [edit] Btw I can't use "virsh save" unfortunately because of GPU pass through.
