VM start/stop events (SOLVED)

Gamberz

By Gamberz
in Programming

Recommended Posts

Gamberz Noob

Gamberz

Posted
Posted (edited)

I'd like to create a plugin that monitors VM start/stop events and then changes the allocated GPUs on a docker container.

 

I haven't done any plugins in Unraid yet, but I've been looking through the community apps to see if there's a plugin that might be monitoring VM events. I haven't found any, but if anybody knows of any or how to monitor when a VM is started, please share.

 

The reason I'm doing this is that I would like to mine crypto in a docker container while no VMs are in use, and manually resizing the container each time I start/stop a VM is quite annoying.

 

Any pointers or links to plugins that implement some of these features that I can learn off would be really helpful. Thanks.

Edited by Gamberz
solved
Link to comment
SimonF Experienced

SimonF

Posted
Posted (edited)
25 minutes ago, Gamberz said:

I'd like to create a plugin that monitors VM start/stop events and then changes the allocated GPUs on a docker container.

 

I haven't done any plugins in Unraid yet, but I've been looking through the community apps to see if there's a plugin that might be monitoring VM events. I haven't found any, but if anybody knows of any or how to monitor when a VM is started, please share.

 

The reason I'm doing this is that I would like to mine crypto in a docker container while no VMs are in use, and manually resizing the container each time I start/stop a VM is quite annoying.

 

Any pointers or links to plugins that implement some of these features that I can learn off would be really helpful. Thanks.

I use QEMU hooks file in my USB Manager plugin to connect USB devices at VM startup etc.

 

https://github.com/SimonFair/USB_Manager

 

With 6.10 there is an easier way to use hooks files, pre 6.10 you have to modify the existing hooks file, happy to asist and provide advice as required.

 

You may not need to write a plugin but create hooks files which are persistance in /etc/libvir/hooks/

 

Here is my code for  installing QEMU updates.

 

Spoiler 
#!/bin/bash

function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; }

QEMU=/etc/libvirt/hooks/qemu
QEMUDFILE=/etc/libvirt/hooks/qemu.d/USB_Manager

VERSION=$(sed -n 's/.*version *= *\"\([^ ]*.*\)\"/\1/p' < /etc/unraid-version)



if [ $(version $VERSION) -ge $(version "6.9.9") ]; then
    echo "Version is for qemu.d $VERSION"

# Process OS Version > 6.9.9
# Remove Previous embedded code from old QEMU File

if  ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then    
START=$(grep -n "begin USB_MANAGER" "${QEMU}" | cut -f1 -d:)
END=$(grep -n "end USB_MANAGER" "${QEMU}" | cut -f1 -d:)
[[ -n ${START} ]] && [[ -n ${END} ]] && sed -i "${START},${END}d" "${QEMU}"
fi

sed -i "s@^[^#]\(.*rc.usb_manager\)@#\1@" "${QEMU}"

# Check qemu.d exists if not create.

[ ! -d "/etc/libvirt/hooks/qemu.d" ] && mkdir /etc/libvirt/hooks/qemu.d

# Create USB_Manager File.

cat << EOF > $QEMUDFILE
#!/usr/bin/env php
<?php
#begin USB_MANAGER
if (\$argv[2] == 'prepare' || \$argv[2] == 'stopped'){
      shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{\$argv[1]}' {\$argv[2]} {\$argv[3]} {\$argv[4]}  >/dev/null 2>&1 & disown") ;
}
#end USB_MANAGER
?>
EOF

chmod +x $QEMUDFILE

# Copy the rules file
cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager6.10.rules /etc/udev/rules.d/99_persistent_usb_manager.rules
chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null

else

# Process OS Version < 6.9.9

echo "Version is not for qemu.d $VERSION"

if ! ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then

FINDLINE=\<\?php
NEWLINE=$(cat<<'END_HEREDOC'
#begin USB_MANAGER\nif ($argv[2] == 'prepare' || $argv[2] == 'stopped'){\n  shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]}  >/dev/null 2>&1 & disown") ;\n}\n#end USB_MANAGER
END_HEREDOC
)
sed -i "/${FINDLINE}/a ${NEWLINE}" "${QEMU}"

fi
# Copy the rules file
cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager.rules /etc/udev/rules.d/99_persistent_usb_manager.rules
chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null
fi

 

 

Edited by SimonF
  • Thanks 1
Link to comment
Gamberz Noob

Gamberz

Posted
  • Author
Posted
19 minutes ago, SimonF said:

I use QEMU hooks file in my USB Manager plugin to connect USB devices at VM startup etc.

 

https://github.com/SimonFair/USB_Manager

 

With 6.10 there is an easier way to use hooks files, pre 6.10 you have to modify the existing hooks file, happy to asist and provide advice as required.

 

You may not need to write a plugin but create hooks files which are persistance in /etc/libvir/hooks/

 

Here is my code for  installing QEMU updates.

 

  Reveal hidden contents 
#!/bin/bash

function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; }

QEMU=/etc/libvirt/hooks/qemu
QEMUDFILE=/etc/libvirt/hooks/qemu.d/USB_Manager

VERSION=$(sed -n 's/.*version *= *\"\([^ ]*.*\)\"/\1/p' < /etc/unraid-version)



if [ $(version $VERSION) -ge $(version "6.9.9") ]; then
    echo "Version is for qemu.d $VERSION"

# Process OS Version > 6.9.9
# Remove Previous embedded code from old QEMU File

if  ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then    
START=$(grep -n "begin USB_MANAGER" "${QEMU}" | cut -f1 -d:)
END=$(grep -n "end USB_MANAGER" "${QEMU}" | cut -f1 -d:)
[[ -n ${START} ]] && [[ -n ${END} ]] && sed -i "${START},${END}d" "${QEMU}"
fi

sed -i "s@^[^#]\(.*rc.usb_manager\)@#\1@" "${QEMU}"

# Check qemu.d exists if not create.

[ ! -d "/etc/libvirt/hooks/qemu.d" ] && mkdir /etc/libvirt/hooks/qemu.d

# Create USB_Manager File.

cat << EOF > $QEMUDFILE
#!/usr/bin/env php
<?php
#begin USB_MANAGER
if (\$argv[2] == 'prepare' || \$argv[2] == 'stopped'){
      shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{\$argv[1]}' {\$argv[2]} {\$argv[3]} {\$argv[4]}  >/dev/null 2>&1 & disown") ;
}
#end USB_MANAGER
?>
EOF

chmod +x $QEMUDFILE

# Copy the rules file
cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager6.10.rules /etc/udev/rules.d/99_persistent_usb_manager.rules
chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null

else

# Process OS Version < 6.9.9

echo "Version is not for qemu.d $VERSION"

if ! ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then

FINDLINE=\<\?php
NEWLINE=$(cat<<'END_HEREDOC'
#begin USB_MANAGER\nif ($argv[2] == 'prepare' || $argv[2] == 'stopped'){\n  shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]}  >/dev/null 2>&1 & disown") ;\n}\n#end USB_MANAGER
END_HEREDOC
)
sed -i "/${FINDLINE}/a ${NEWLINE}" "${QEMU}"

fi
# Copy the rules file
cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager.rules /etc/udev/rules.d/99_persistent_usb_manager.rules
chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null
fi

 

 

That's exactly what I needed, thanks a million! I've upgraded to 6.10 and am testing out the method in your install.sh file. 

Link to comment
Gamberz Noob

Gamberz

Posted
  • Author
Posted (edited)
3 hours ago, SimonF said:

I use QEMU hooks file in my USB Manager plugin to connect USB devices at VM startup etc.

 

https://github.com/SimonFair/USB_Manager

 

With 6.10 there is an easier way to use hooks files, pre 6.10 you have to modify the existing hooks file, happy to asist and provide advice as required.

 

You may not need to write a plugin but create hooks files which are persistance in /etc/libvir/hooks/

 

Here is my code for  installing QEMU updates.

 

  Reveal hidden contents 
#!/bin/bash

function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; }

QEMU=/etc/libvirt/hooks/qemu
QEMUDFILE=/etc/libvirt/hooks/qemu.d/USB_Manager

VERSION=$(sed -n 's/.*version *= *\"\([^ ]*.*\)\"/\1/p' < /etc/unraid-version)



if [ $(version $VERSION) -ge $(version "6.9.9") ]; then
    echo "Version is for qemu.d $VERSION"

# Process OS Version > 6.9.9
# Remove Previous embedded code from old QEMU File

if  ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then    
START=$(grep -n "begin USB_MANAGER" "${QEMU}" | cut -f1 -d:)
END=$(grep -n "end USB_MANAGER" "${QEMU}" | cut -f1 -d:)
[[ -n ${START} ]] && [[ -n ${END} ]] && sed -i "${START},${END}d" "${QEMU}"
fi

sed -i "s@^[^#]\(.*rc.usb_manager\)@#\1@" "${QEMU}"

# Check qemu.d exists if not create.

[ ! -d "/etc/libvirt/hooks/qemu.d" ] && mkdir /etc/libvirt/hooks/qemu.d

# Create USB_Manager File.

cat << EOF > $QEMUDFILE
#!/usr/bin/env php
<?php
#begin USB_MANAGER
if (\$argv[2] == 'prepare' || \$argv[2] == 'stopped'){
      shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{\$argv[1]}' {\$argv[2]} {\$argv[3]} {\$argv[4]}  >/dev/null 2>&1 & disown") ;
}
#end USB_MANAGER
?>
EOF

chmod +x $QEMUDFILE

# Copy the rules file
cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager6.10.rules /etc/udev/rules.d/99_persistent_usb_manager.rules
chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null

else

# Process OS Version < 6.9.9

echo "Version is not for qemu.d $VERSION"

if ! ( grep -q "usb_manager/scripts/rc.usb_manager" "${QEMU}" ); then

FINDLINE=\<\?php
NEWLINE=$(cat<<'END_HEREDOC'
#begin USB_MANAGER\nif ($argv[2] == 'prepare' || $argv[2] == 'stopped'){\n  shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]}  >/dev/null 2>&1 & disown") ;\n}\n#end USB_MANAGER
END_HEREDOC
)
sed -i "/${FINDLINE}/a ${NEWLINE}" "${QEMU}"

fi
# Copy the rules file
cp /usr/local/emhttp/plugins/usb_manager/99_persistent_usb_manager.rules /etc/udev/rules.d/99_persistent_usb_manager.rules
chmod 644 -R /etc/udev/rules.d/99_persistent_usb_manager.rules 2>/dev/null
fi

 

 

Hey so I got the hook working thanks to your file. I'm stuck on actually restarting the docker container with the new config though, if you have any suggestions.

I'm updating the template config here 

/boot/config/plugins/dockerMan/templates-user/my-miner.xml 

The changes seem to get reflected in the dashboard. If I simply restart the container though, the changes obviously don't take affect. Only if I modify another config option and apply it in the dashboard does it take effect. I could technically stop, delete and start manually like 

/usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='miner' --net='bridge' --cpuset-cpus='0' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e 'NVIDIA_DRIVER_CAPABILITIES'='all' -e 'NVIDIA_VISIBLE_DEVICES'='REDACTED' -l net.unraid.docker.managed=dockerman --runtime=nvidia REDACTED_IMAGE REDACTED_ARGS

But I'd like to avoid that, with all the params I'd need to fetch and the fact it could break at some point. 

My other idea was to trigger the API call 

http://192.168.2.156/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-miner-gpu.xml

But that's also not as clean as I would like. Do you know if there's a way to trigger a container to recreate itself?

 

EDIT: I'm going to try updating 

/var/lib/docker/containers/containerid/config.v2.json

and restarting the container. If you've any other suggestions though that'd be great.

Edited by Gamberz
Link to comment
SimonF Experienced

SimonF

Posted
Posted (edited)
2 hours ago, Gamberz said:

Hey so I got the hook working thanks to your file. I'm stuck on actually restarting the docker container with the new config though, if you have any suggestions.

I'm updating the template config here 

/boot/config/plugins/dockerMan/templates-user/my-miner.xml 

The changes seem to get reflected in the dashboard. If I simply restart the container though, the changes obviously don't take affect. Only if I modify another config option and apply it in the dashboard does it take effect. I could technically stop, delete and start manually like 

/usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='miner' --net='bridge' --cpuset-cpus='0' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e 'NVIDIA_DRIVER_CAPABILITIES'='all' -e 'NVIDIA_VISIBLE_DEVICES'='REDACTED' -l net.unraid.docker.managed=dockerman --runtime=nvidia REDACTED_IMAGE REDACTED_ARGS

But I'd like to avoid that, with all the params I'd need to fetch and the fact it could break at some point. 

My other idea was to trigger the API call 

http://192.168.2.156/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-miner-gpu.xml

But that's also not as clean as I would like. Do you know if there's a way to trigger a container to recreate itself?

 

EDIT: I'm going to try updating 

/var/lib/docker/containers/containerid/config.v2.json

and restarting the container. If you've any other suggestions though that'd be great.

I dont have any idea re docker as not needed to do anything with them within code so far. There maybe others on the forums that may be able to provide more insight than me. 

Edited by SimonF
Link to comment
Squid Grand Master

Squid

Posted
Posted
1 hour ago, Gamberz said:

But I'd like to avoid that, with all the params I'd need to fetch and the fact it could break at some point. 

What you'd need to do is have 2 separate XML's for the same container being installed.  One with the GPU changes, and one without.

 

Then when you need to stop one and start the other you would do a simple docker stop nameOfContainer and then start the other one via https://forums.unraid.net/topic/40016-start-docker-template-via-command-line/?tab=comments#comment-1022006

 

 

  • Thanks 1
Link to comment
Gamberz Noob

Gamberz

Posted
  • Author
Posted (edited)
20 hours ago, Squid said:

What you'd need to do is have 2 separate XML's for the same container being installed.  One with the GPU changes, and one without.

 

Then when you need to stop one and start the other you would do a simple docker stop nameOfContainer and then start the other one via https://forums.unraid.net/topic/40016-start-docker-template-via-command-line/?tab=comments#comment-1022006

 

 

Thanks for the suggestion. I ended up modifying the config.v2.json for the container and was able to get it working, this is a bit more dynamic which is good.

 

Here's the source incase anybody else finds it useful. I'd like to make it work with multiple VMs at once, but this is fine for the moment. 

#!/bin/bash

CONTAINER_NAME="aio-miner"
CONTAINER_ID=$(docker inspect --format="{{.Id}}" $CONTAINER_NAME)
INPUT_VM=$1
INPUT_EVENT=$2

update_container() {
    local devices="$(echo "$1" | cut -d, -f"2" | xargs | sed -e 's/ /,/g')"
    local devices_escaped="$(echo $devices | sed 's/\-/\\-/g')"

    echo "cd /Container/Config[@Name='NVIDIA_VISIBLE_DEVICES']
    set $devices
    cd /Container/Environment/Variable[2]/Value
    set $devices
    save" | xmllint --shell /boot/config/plugins/dockerMan/templates-user/my-$CONTAINER_NAME.xml

    docker stop $CONTAINER_ID
    sed -i "s/\"NVIDIA_VISIBLE_DEVICES=[A-Za-z0-9,\-]*\"/\"NVIDIA_VISIBLE_DEVICES=$devices_escaped\"/g" /var/lib/docker/containers/$CONTAINER_ID/config.v2.json
    /etc/rc.d/rc.docker restart
}

get_vm_buses() {
    local buses=""

    while read -r bus; do
        bus=${bus#*'"'}; bus=${bus%'"'*}
        buses+="$bus|"
    done <<< $(xmllint --xpath '//domain/devices/hostdev[@type="pci"]/source/address/@bus' --nowarning /etc/libvirt/qemu/$INPUT_VM.xml | sort | uniq)
    
    echo "${buses%?}"
}

if [[ "$INPUT_EVENT" == "prepare" ]]; then
    buses="$(get_vm_buses)"

    if [ -z "$buses" ]; then
        exit 0
    fi

    devices="$(nvidia-smi --query-gpu="pci.bus,uuid" --format=csv,noheader | grep -Ev "$buses")"
    update_container "$devices"
fi

if [[ "$INPUT_EVENT" == "release" ]]; then
    buses="$(get_vm_buses)"
    
    if [ -z "$buses" ]; then
        exit 0
    fi

    devices="$(nvidia-smi --query-gpu="pci.bus,uuid" --format=csv,noheader)"
    update_container "$devices"
fi

(sleep 30s && nvidia-smi -pl 280) &

 

Edited by Gamberz
Link to comment
  • Gamberz changed the title to VM start/stop events (SOLVED)

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing