Gamberz Posted August 28, 2021 Share Posted August 28, 2021 (edited) I'd like to create a plugin that monitors VM start/stop events and then changes the allocated GPUs on a docker container. I haven't done any plugins in Unraid yet, but I've been looking through the community apps to see if there's a plugin that might be monitoring VM events. I haven't found any, but if anybody knows of any or how to monitor when a VM is started, please share. The reason I'm doing this is that I would like to mine crypto in a docker container while no VMs are in use, and manually resizing the container each time I start/stop a VM is quite annoying. Any pointers or links to plugins that implement some of these features that I can learn off would be really helpful. Thanks. Edited August 30, 2021 by Gamberz solved Quote Link to comment
SimonF Posted August 28, 2021 Share Posted August 28, 2021 (edited) 25 minutes ago, Gamberz said: I'd like to create a plugin that monitors VM start/stop events and then changes the allocated GPUs on a docker container. I haven't done any plugins in Unraid yet, but I've been looking through the community apps to see if there's a plugin that might be monitoring VM events. I haven't found any, but if anybody knows of any or how to monitor when a VM is started, please share. The reason I'm doing this is that I would like to mine crypto in a docker container while no VMs are in use, and manually resizing the container each time I start/stop a VM is quite annoying. Any pointers or links to plugins that implement some of these features that I can learn off would be really helpful. Thanks. I use QEMU hooks file in my USB Manager plugin to connect USB devices at VM startup etc. https://github.com/SimonFair/USB_Manager With 6.10 there is an easier way to use hooks files, pre 6.10 you have to modify the existing hooks file, happy to asist and provide advice as required. You may not need to write a plugin but create hooks files which are persistance in /etc/libvir/hooks/ Here is my code for installing QEMU updates. Spoiler #!/bin/bash function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; } QEMU=/etc/libvirt/hooks/qemu QEMUDFILE=/etc/libvirt/hooks/qemu.d/USB_Manager VERSION=$(sed -n 's/.*version *= *\"\([^ ]*.*\)\"/\1/p' < /etc/unraid-version) if [ $(version $VERSION) -ge $(version "6.9.9") ]; then echo "Version is for qemu.d $VERSION" # Process OS Version > 6.9.9 # Remove Previous embedded code from old QEMU File if ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then START=$(grep -n "begin USB_MANAGER" "${QEMU}" | cut -f1 -d:) END=$(grep -n "end USB_MANAGER" "${QEMU}" | cut -f1 -d:) [[ -n ${START} ]] && [[ -n ${END} ]] && sed -i "${START},${END}d" "${QEMU}" fi sed -i "s@^[^#]\(.*rc.usb_manager\)@#\1@" "${QEMU}" # Check qemu.d exists if not create. [ ! -d "/etc/libvirt/hooks/qemu.d" ] && mkdir /etc/libvirt/hooks/qemu.d # Create USB_Manager File. cat << EOF > $QEMUDFILE #!/usr/bin/env php <?php #begin USB_MANAGER if (\$argv[2] == 'prepare' || \$argv[2] == 'stopped'){ shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{\$argv[1]}' {\$argv[2]} {\$argv[3]} {\$argv[4]} >/dev/null 2>&1 & disown") ; } #end USB_MANAGER ?> EOF chmod +x $QEMUDFILE # Copy the rules file cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager6.10.rules /etc/udev/rules.d/99_persistent_usb_manager.rules chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null else # Process OS Version < 6.9.9 echo "Version is not for qemu.d $VERSION" if ! ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then FINDLINE=\<\?php NEWLINE=$(cat<<'END_HEREDOC' #begin USB_MANAGER\nif ($argv[2] == 'prepare' || $argv[2] == 'stopped'){\n shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]} >/dev/null 2>&1 & disown") ;\n}\n#end USB_MANAGER END_HEREDOC ) sed -i "/${FINDLINE}/a ${NEWLINE}" "${QEMU}" fi # Copy the rules file cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager.rules /etc/udev/rules.d/99_persistent_usb_manager.rules chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null fi Edited August 28, 2021 by SimonF 1 Quote Link to comment
Gamberz Posted August 28, 2021 Author Share Posted August 28, 2021 19 minutes ago, SimonF said: I use QEMU hooks file in my USB Manager plugin to connect USB devices at VM startup etc. https://github.com/SimonFair/USB_Manager With 6.10 there is an easier way to use hooks files, pre 6.10 you have to modify the existing hooks file, happy to asist and provide advice as required. You may not need to write a plugin but create hooks files which are persistance in /etc/libvir/hooks/ Here is my code for installing QEMU updates. Reveal hidden contents #!/bin/bash function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; } QEMU=/etc/libvirt/hooks/qemu QEMUDFILE=/etc/libvirt/hooks/qemu.d/USB_Manager VERSION=$(sed -n 's/.*version *= *\"\([^ ]*.*\)\"/\1/p' < /etc/unraid-version) if [ $(version $VERSION) -ge $(version "6.9.9") ]; then echo "Version is for qemu.d $VERSION" # Process OS Version > 6.9.9 # Remove Previous embedded code from old QEMU File if ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then START=$(grep -n "begin USB_MANAGER" "${QEMU}" | cut -f1 -d:) END=$(grep -n "end USB_MANAGER" "${QEMU}" | cut -f1 -d:) [[ -n ${START} ]] && [[ -n ${END} ]] && sed -i "${START},${END}d" "${QEMU}" fi sed -i "s@^[^#]\(.*rc.usb_manager\)@#\1@" "${QEMU}" # Check qemu.d exists if not create. [ ! -d "/etc/libvirt/hooks/qemu.d" ] && mkdir /etc/libvirt/hooks/qemu.d # Create USB_Manager File. cat << EOF > $QEMUDFILE #!/usr/bin/env php <?php #begin USB_MANAGER if (\$argv[2] == 'prepare' || \$argv[2] == 'stopped'){ shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{\$argv[1]}' {\$argv[2]} {\$argv[3]} {\$argv[4]} >/dev/null 2>&1 & disown") ; } #end USB_MANAGER ?> EOF chmod +x $QEMUDFILE # Copy the rules file cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager6.10.rules /etc/udev/rules.d/99_persistent_usb_manager.rules chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null else # Process OS Version < 6.9.9 echo "Version is not for qemu.d $VERSION" if ! ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then FINDLINE=\<\?php NEWLINE=$(cat<<'END_HEREDOC' #begin USB_MANAGER\nif ($argv[2] == 'prepare' || $argv[2] == 'stopped'){\n shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]} >/dev/null 2>&1 & disown") ;\n}\n#end USB_MANAGER END_HEREDOC ) sed -i "/${FINDLINE}/a ${NEWLINE}" "${QEMU}" fi # Copy the rules file cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager.rules /etc/udev/rules.d/99_persistent_usb_manager.rules chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null fi That's exactly what I needed, thanks a million! I've upgraded to 6.10 and am testing out the method in your install.sh file. Quote Link to comment
Gamberz Posted August 28, 2021 Author Share Posted August 28, 2021 (edited) 3 hours ago, SimonF said: I use QEMU hooks file in my USB Manager plugin to connect USB devices at VM startup etc. https://github.com/SimonFair/USB_Manager With 6.10 there is an easier way to use hooks files, pre 6.10 you have to modify the existing hooks file, happy to asist and provide advice as required. You may not need to write a plugin but create hooks files which are persistance in /etc/libvir/hooks/ Here is my code for installing QEMU updates. Reveal hidden contents #!/bin/bash function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; } QEMU=/etc/libvirt/hooks/qemu QEMUDFILE=/etc/libvirt/hooks/qemu.d/USB_Manager VERSION=$(sed -n 's/.*version *= *\"\([^ ]*.*\)\"/\1/p' < /etc/unraid-version) if [ $(version $VERSION) -ge $(version "6.9.9") ]; then echo "Version is for qemu.d $VERSION" # Process OS Version > 6.9.9 # Remove Previous embedded code from old QEMU File if ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then START=$(grep -n "begin USB_MANAGER" "${QEMU}" | cut -f1 -d:) END=$(grep -n "end USB_MANAGER" "${QEMU}" | cut -f1 -d:) [[ -n ${START} ]] && [[ -n ${END} ]] && sed -i "${START},${END}d" "${QEMU}" fi sed -i "s@^[^#]\(.*rc.usb_manager\)@#\1@" "${QEMU}" # Check qemu.d exists if not create. [ ! -d "/etc/libvirt/hooks/qemu.d" ] && mkdir /etc/libvirt/hooks/qemu.d # Create USB_Manager File. cat << EOF > $QEMUDFILE #!/usr/bin/env php <?php #begin USB_MANAGER if (\$argv[2] == 'prepare' || \$argv[2] == 'stopped'){ shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{\$argv[1]}' {\$argv[2]} {\$argv[3]} {\$argv[4]} >/dev/null 2>&1 & disown") ; } #end USB_MANAGER ?> EOF chmod +x $QEMUDFILE # Copy the rules file cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager6.10.rules /etc/udev/rules.d/99_persistent_usb_manager.rules chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null else # Process OS Version < 6.9.9 echo "Version is not for qemu.d $VERSION" if ! ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then FINDLINE=\<\?php NEWLINE=$(cat<<'END_HEREDOC' #begin USB_MANAGER\nif ($argv[2] == 'prepare' || $argv[2] == 'stopped'){\n shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]} >/dev/null 2>&1 & disown") ;\n}\n#end USB_MANAGER END_HEREDOC ) sed -i "/${FINDLINE}/a ${NEWLINE}" "${QEMU}" fi # Copy the rules file cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager.rules /etc/udev/rules.d/99_persistent_usb_manager.rules chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null fi Hey so I got the hook working thanks to your file. I'm stuck on actually restarting the docker container with the new config though, if you have any suggestions. I'm updating the template config here /boot/config/plugins/dockerMan/templates-user/my-miner.xml The changes seem to get reflected in the dashboard. If I simply restart the container though, the changes obviously don't take affect. Only if I modify another config option and apply it in the dashboard does it take effect. I could technically stop, delete and start manually like /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='miner' --net='bridge' --cpuset-cpus='0' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e 'NVIDIA_DRIVER_CAPABILITIES'='all' -e 'NVIDIA_VISIBLE_DEVICES'='REDACTED' -l net.unraid.docker.managed=dockerman --runtime=nvidia REDACTED_IMAGE REDACTED_ARGS But I'd like to avoid that, with all the params I'd need to fetch and the fact it could break at some point. My other idea was to trigger the API call http://192.168.2.156/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-miner-gpu.xml But that's also not as clean as I would like. Do you know if there's a way to trigger a container to recreate itself? EDIT: I'm going to try updating /var/lib/docker/containers/containerid/config.v2.json and restarting the container. If you've any other suggestions though that'd be great. Edited August 28, 2021 by Gamberz Quote Link to comment
SimonF Posted August 28, 2021 Share Posted August 28, 2021 (edited) 2 hours ago, Gamberz said: Hey so I got the hook working thanks to your file. I'm stuck on actually restarting the docker container with the new config though, if you have any suggestions. I'm updating the template config here /boot/config/plugins/dockerMan/templates-user/my-miner.xml The changes seem to get reflected in the dashboard. If I simply restart the container though, the changes obviously don't take affect. Only if I modify another config option and apply it in the dashboard does it take effect. I could technically stop, delete and start manually like /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='miner' --net='bridge' --cpuset-cpus='0' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e 'NVIDIA_DRIVER_CAPABILITIES'='all' -e 'NVIDIA_VISIBLE_DEVICES'='REDACTED' -l net.unraid.docker.managed=dockerman --runtime=nvidia REDACTED_IMAGE REDACTED_ARGS But I'd like to avoid that, with all the params I'd need to fetch and the fact it could break at some point. My other idea was to trigger the API call http://192.168.2.156/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-miner-gpu.xml But that's also not as clean as I would like. Do you know if there's a way to trigger a container to recreate itself? EDIT: I'm going to try updating /var/lib/docker/containers/containerid/config.v2.json and restarting the container. If you've any other suggestions though that'd be great. I dont have any idea re docker as not needed to do anything with them within code so far. There maybe others on the forums that may be able to provide more insight than me. Edited August 28, 2021 by SimonF Quote Link to comment
Squid Posted August 28, 2021 Share Posted August 28, 2021 1 hour ago, Gamberz said: But I'd like to avoid that, with all the params I'd need to fetch and the fact it could break at some point. What you'd need to do is have 2 separate XML's for the same container being installed. One with the GPU changes, and one without. Then when you need to stop one and start the other you would do a simple docker stop nameOfContainer and then start the other one via https://forums.unraid.net/topic/40016-start-docker-template-via-command-line/?tab=comments#comment-1022006 1 Quote Link to comment
Gamberz Posted August 28, 2021 Author Share Posted August 28, 2021 (edited) 20 hours ago, Squid said: What you'd need to do is have 2 separate XML's for the same container being installed. One with the GPU changes, and one without. Then when you need to stop one and start the other you would do a simple docker stop nameOfContainer and then start the other one via https://forums.unraid.net/topic/40016-start-docker-template-via-command-line/?tab=comments#comment-1022006 Thanks for the suggestion. I ended up modifying the config.v2.json for the container and was able to get it working, this is a bit more dynamic which is good. Here's the source incase anybody else finds it useful. I'd like to make it work with multiple VMs at once, but this is fine for the moment. #!/bin/bash CONTAINER_NAME="aio-miner" CONTAINER_ID=$(docker inspect --format="{{.Id}}" $CONTAINER_NAME) INPUT_VM=$1 INPUT_EVENT=$2 update_container() { local devices="$(echo "$1" | cut -d, -f"2" | xargs | sed -e 's/ /,/g')" local devices_escaped="$(echo $devices | sed 's/\-/\\-/g')" echo "cd /Container/Config[@Name='NVIDIA_VISIBLE_DEVICES'] set $devices cd /Container/Environment/Variable[2]/Value set $devices save" | xmllint --shell /boot/config/plugins/dockerMan/templates-user/my-$CONTAINER_NAME.xml docker stop $CONTAINER_ID sed -i "s/\"NVIDIA_VISIBLE_DEVICES=[A-Za-z0-9,\-]*\"/\"NVIDIA_VISIBLE_DEVICES=$devices_escaped\"/g" /var/lib/docker/containers/$CONTAINER_ID/config.v2.json /etc/rc.d/rc.docker restart } get_vm_buses() { local buses="" while read -r bus; do bus=${bus#*'"'}; bus=${bus%'"'*} buses+="$bus|" done <<< $(xmllint --xpath '//domain/devices/hostdev[@type="pci"]/source/address/@bus' --nowarning /etc/libvirt/qemu/$INPUT_VM.xml | sort | uniq) echo "${buses%?}" } if [[ "$INPUT_EVENT" == "prepare" ]]; then buses="$(get_vm_buses)" if [ -z "$buses" ]; then exit 0 fi devices="$(nvidia-smi --query-gpu="pci.bus,uuid" --format=csv,noheader | grep -Ev "$buses")" update_container "$devices" fi if [[ "$INPUT_EVENT" == "release" ]]; then buses="$(get_vm_buses)" if [ -z "$buses" ]; then exit 0 fi devices="$(nvidia-smi --query-gpu="pci.bus,uuid" --format=csv,noheader)" update_container "$devices" fi (sleep 30s && nvidia-smi -pl 280) & Edited August 29, 2021 by Gamberz Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.