Jump to content
Squid

Additional Scripts For User.Scripts Plugin

323 posts in this topic Last Reply

Recommended Posts

Well since no one is telling me that this is normal or not and it's been over 50 hours, I guess I'll have to see about killing the script.

 

I'm sorry, I was avoiding this thread until I had time to answer ALL of the related posts.  My apologies to you.

 

I suppose I would have considered killing it after 5 to 6 hours, definitely if it was still running after 10 hours.  I don't have any ideas yet as to what could have gone wrong.  Can you tell if there is any I/O to the drive?  Syslog may help, post your diagnostics (Need help? Read me first!)  We should take this to a separate thread.

Share this post


Link to post

Well since no one is telling me that this is normal or not and it's been over 50 hours, I guess I'll have to see about killing the script.

 

I'm sorry, I was avoiding this thread until I had time to answer ALL of the related posts.  My apologies to you.

 

I suppose I would have considered killing it after 5 to 6 hours, definitely if it was still running after 10 hours.  I don't have any ideas yet as to what could have gone wrong.  Can you tell if there is any I/O to the drive?  Syslog may help, post your diagnostics (Need help? Read me first!)  We should take this to a separate thread.

 

Thank you, that's all I needed to know. The only indication that there is something going on is the number of writes listed on the Main tab is slowly increasing for that drive and the parity drive. The syslog only shows that the process has started and nothing related since. Because this is in preparation for some significant hardware changes (new motherboard etc.), I am not really that worried about it. I've got 2 other 500GB drives I was planning on removing from the array as well and so as soon as I get my new drive precleared and the data moved off those other 2 drives, I'll just rebuild parity anyways.

Share this post


Link to post

USB Hot Plug for VM's with no passthrough

 

TLDR: The instructions are a pain for me to try and explain, but this script does work.  Don't be afraid to ask questions.  But first and foremost when experimenting with this on a running VM, set the startup delay to 1-2 seconds so you're not waiting forever for it to get up and running.  Also make sure that you're on the latest version of user.scripts prior to playing around.

 

EDIT: it appears that some devices can  get tossed into a different bus group when plugged in.  For this reason, this plugin basically is going to require a USB hub to be monitored for changes, as in my testing anything plugged into a hub will always stay in the same group as the hub

 

A simple little script to monitor a particular USB bus for changes and then automatically attach / detach the USB device to a particular VM.  Useful if you cannot pass through a USB controller to a VM  (not that I need this as I have a USB controller passed through to my main VM, but I was rather dismayed at the libvirt USB Hotplug plugin which didn't support true hotplugging without going back to the UI and manually attaching / detaching devices)

 

This started out as being a potential plugin, but there are enough caveats with the initial setup to make this impractical.

 

Couple of variables within the script need to be set:

 

$vmName  - Set to the name of the VM (must be exact)

$pollingTime  - Set to the number of seconds between looking for changes

$startupDelay - Set to the number of seconds to wait after array start to begin monitoring

 

And 2 variables of which one or the other should be set:  (Comment out the one line that's not going to get set (ie: put a # in front of it) )

 

$hubName and $bus

 

If you are using a particular USB bus (ie: 2 particular ports on the computer), the set the $bus to be the bus as listed by

lsusb

 

If you are using a USB Hub, I recommend that you set this to the exact name of the USB hub as determined by

lsusb

 

And now the caveats (which are workable IMHO)

 

Bus:  each USB bus is going to handle multiple ports on your computer.  You want to make sure that those ports are going to be dedicated to the VM in question (ie: You can't use one of those ports for say Unassigned Devices)  Also, on my system I found that when I plug a USB hub into a particular bus, then the hub gets moved into another bus which may or may not already have devices on it.  TLDR:  If you use the "bus" variable, don't plug a USB hub into it ->  it won't work.

 

hubName:  If you want to use a usb Hub, then set this variable to be the exact name of the hub.  Problem with this mode is that if the hub when plugged in gets moved from an isolated bus to one with other devices on it then everything on that particular bus either has to be permanently attached to unRaid itself (not the VM) otherwise the script will attempt to detach / attach those items from the VM.  This is the recommended way of handling USB devices

 

 

To illustrate, I have numerous empty USB busses on my system.  If I plug in a USB Hub into one of those usb's, then the hub winds up being moved from the empty bus to another bus on my system which already has my unRaid flash drive and keyboard on it.  Since those two devices are permanently attached to unRaid, this isn't a big deal to me.

 

The best way to figure out how to set things is to plugin and unplug a device from every usb port on your system and type

lsusb

in between to see how it all reacts.  Like I said, if you don't need a hub, find an unused bus and use it.  If you need a hub, then make sure that everything else on that bus is permanent to unRaid itself an be aware that any other ports on your system that are on the same bus will also be passed through to the VM.

 

Note that when I'm talking about USB Busses, I am not referring to USB 2 or USB 3.  Each motherboard will have multiple busses on the system, each of which in general will control 2 ports connected to it.

 

Additional caveat, do to how linux operates, I cannot guarantee that a particular port will always be on the same bus over resets of the system, but in my testing, it always was.

 

One final caveat.  Because this plugin monitors the bus for changes, upon startup (after the initial startup delay is finished), any device plugged into the bus or hub will not be attached to the VM.  You must pull the device and reinsert it.  Any pull and reinsert of the same device should take longer than the polling time (I'm defaulting that to 10 seconds)  Pulls and inserts of different devices can be done as fast as you want.

 

#!/usr/bin/php
<?PHP
#backgroundOnly=true
#description=A script designed to monitor a USB hub or a particular USB bus for changes and then attach or detach devices to an applicable VM

$vmName = "Windows 10";                             # must be the exact name
$pollingTime = 10;                                  # the interval between checks in seconds
$startupDelay = 300;                                # startup delay before monitoring for changes in seconds (set to enough time for the VM to get up and running)

# only use one or the other of the following lines not both  See thread for details
#$hubName = "Texas Instruments, Inc. TUSB2046 Hub";  # must be exact as listed via lsusb
$bus  = "02";                                       # see thread for details

function getDeviceList($bus) {
  exec("lsusb | grep 'Bus $bus'",$allDevicesOnBus);
  foreach ($allDevicesOnBus as $Devices) {
    $deviceLine = explode(" ",$Devices);
    $initialState[$deviceLine[5]] = $deviceLine[5];
  }
  return $initialState;
}  

function createXML($deviceID) {
  $usb = explode(":",$deviceID);
  $usbstr .= "<hostdev mode='subsystem' type='usb'>
                <source>
                  <vendor id='0x".$usb[0]."'/>
                  <product id='0x".$usb[1]."'/>
                </source>
              </hostdev>";
  file_put_contents("/tmp/USBTempXML.xml",$usbstr);
}

function logger($string) {
  shell_exec("logger ".escapeshellarg($string));
}
              
# Begin Main

logger("Sleeping for $startupDelay before monitoring $bus for changes to passthrough to $vmName");
sleep($startupDelay);

$hubBus = $bus;
if ( ! $hubBus ) {
  $hub = explode(" ",exec("lsusb | grep '$hubName'"));
  $hubBus = $hub[1];
}

logger("Monitoring $hubBus for changes");

$initialState = getDeviceList($hubBus);

while (true) {
  $unRaidVars = parse_ini_file("/var/local/emhttp/var.ini");
  if ($unRaidVars['mdState'] != "STARTED") {
    break;
  }
  $currentDevices = getDeviceList($hubBus);
  
  foreach ($currentDevices as $Device) {
    if ( ! $initialState[$Device] ) {
      logger("$Device Added to bus $hubBus  Attaching to $vmName");
      createXML($Device);
      exec("/usr/sbin/virsh attach-device '$vmName' /tmp/USBTempXML.xml 2>&1");
      $initialState[$Device] = $Device;
    }
  }
  foreach ($initialState as $Device) {
    if ( ! $currentDevices[$Device] ) {
      logger("$Device Removed from bus $hubBus  Detaching from $vmName");
      createXML($Device);
      exec("/usr/sbin/virsh detach-device '$vmName' /tmp/USBTempXML.xml 2>&1");
      unset ($initialState[$Device]);
    }
  }
  sleep($pollingTime);
}

?>

 

Technie note:  There is no way to monitor a particular port for changes, because every time you remove and insert a device, the device number associated with the bus increases.

 

Also, this script does not use udev to automatically attach / detach the usb devices.  This is because: the USB device in question may or may not be known prior to attaching (ie: a buddy brought over a USB stick for you to use), and secondly because in theory you could have multiple copies of this script running for different VMs and the same device may wind up being plugged into a different VM as you require, rather than the one udev dictated.  IE: I went for flexibility at the expense of minimal processor time.

USB_HotPlug.zip

Share this post


Link to post

Hey Squid,  just had a thought for a feature.

 

Allow user.scripts to be pulled from github.  Could add repos like docker.  Just an idea.  Would help sharing scripts etc....

 

And gives you something to do.

 

;D

Share this post


Link to post

Hey Squid,  just had a thought for a feature.

 

Allow user.scripts to be pulled from github.  Could add repos like docker.  Just an idea.  Would help sharing scripts etc....

 

And gives you something to do.

 

;D

Really!?!

Share this post


Link to post

I take it you're not keen then lol

 

Sent from my LG-H815 using Tapatalk

 

 

Share this post


Link to post

Like Endy I'm seeing a very long run of the clear disk script.   But stats are updating when I'm running so I'm certain the script didn't hang.  

 

I'm currently running the clear script on a 4 year old Hitachi 7200rpm disk on md1 and sdf.  The drive isn't failing and throws no smart errors.  There is no substantial activity on any other mdX devices.   I turned on turbo write in the GUI (but didn't reboot if that matters).  

 

My stats:
Unmounting Disk 1 ...
umount: /mnt/disk1: not mounted       [ed:  don't worry about this.  I killed it once because I was moving other data and figured I'd wait before I run it]
Clearing Disk 1 ...
66783805440 bytes (67 GB, 62 GiB) copied, 52920 s, 1.3 MB/s

 

So I'm looking at about 18 days of total run time?   

 

I see read activity for all of my array drives on sdX of about 1.3 MB/s including the drive I'm clearing.  I see write activity for md1 of the 1.3MB/s.  I also see read activity for sdf, the drive i'm clearing, of about 10MB/s.

 

Removing a cleared device and replacing it with a precleared drive and not having to lose parity (especially since I have only 1 parity disk) seems like a no brainer.

 

I'm uncertain why all sd? devices are read to maintain parity.  I always thought that unraid only did an XOR calc for parity on data change.  I guess because its block level with DD on an unmounted drive it needs a full calculation using all disks?

 

Is it possible to modify the script to write 0's to a file in the file system.  Then unmount and zero the partition info with dd using the slower method over just the partition info?

 

Thanks.

 

 

Share this post


Link to post
10 hours ago, dimes007 said:

 

Like Endy I'm seeing a very long run of the clear disk script.   But stats are updating when I'm running so I'm certain the script didn't hang.  

 

I'm currently running the clear script on a 4 year old Hitachi 7200rpm disk on md1 and sdf.  The drive isn't failing and throws no smart errors.  There is no substantial activity on any other mdX devices.   I turned on turbo write in the GUI (but didn't reboot if that matters).  

 

My stats:
Unmounting Disk 1 ...
umount: /mnt/disk1: not mounted       [ed:  don't worry about this.  I killed it once because I was moving other data and figured I'd wait before I run it]
Clearing Disk 1 ...
66783805440 bytes (67 GB, 62 GiB) copied, 52920 s, 1.3 MB/s

 

So I'm looking at about 18 days of total run time?   

 

I see read activity for all of my array drives on sdX of about 1.3 MB/s including the drive I'm clearing.  I see write activity for md1 of the 1.3MB/s.  I also see read activity for sdf, the drive i'm clearing, of about 10MB/s

 

 

Because you turned on Turbo Write, ALL of the other disks have to be read in order to determine what to write to the parity disk.  For this operation, Turbo Write is faster that the alternative, but clearly it's not helping you.  Because Turbo Write depends on ALL of the array drives, and it cannot move faster than the slowest drive, it's my suspicion you have a bad drive somewhere, that's dragging down the rest.  Your syslog may be full of drive errors, post your diagnostics.

 

10 hours ago, dimes007 said:

I'm uncertain why all sd? devices are read to maintain parity.  I always thought that unraid only did an XOR calc for parity on data change.  I guess because its block level with DD on an unmounted drive it needs a full calculation using all disks?

 

That would be correct for the Read/Modify/Write method, but you changed it to Turbo Write (which would normally be faster here).

 

10 hours ago, dimes007 said:

Is it possible to modify the script to write 0's to a file in the file system.  Then unmount and zero the partition info with dd using the slower method over just the partition info?

 

While it's possible to overwrite the space of each file with zeroes, that's not enough, you still have to zero out ALL of the deleted files, the remnants of past files, and the past versions of files, AND ALL of the sectors of the file system itself.  We don't know where they are.  For modern file systems, they can be scattered over the entire drive surface.

Share this post


Link to post

As nested vms have been disabled by default in 6.3 due to the issue with avast and windows vms when enabled.

I have made 2 scripts to turn it on and off. In script set cpu type 1 for intel 2 for amd.

Download link for both scripts https://www.dropbox.com/s/0b1tvotvl6y80uy/nested vms scripts.zip?dl=0

 

nested vm on script

#!/bin/bash

#set whether your cpu is Intel or AMD  [1-Intel] [2-Amd] 
cputype="1"

#Do not edit below this line
#turn nested on


	if [[ "$cputype" =~ ^(1|2)$ ]]; then

		if [ "$cputype" -eq 1 ]; then
			
			modprobe -r kvm_intel
				modprobe kvm_intel nested=1
				
				echo "Nested vms are enabled for intel cpus"


			elif [ "$pushnotifications" -eq 2 ]; then

			modprobe -r kvm_amd
					modprobe kvm_amd nested=1
				
					echo "Nested vms are enabled for AMD cpus"			

			fi

		else
			
				echo "invalid cpu type set please check config"
			
			fi


sleep 4
exit

and nested vm  off script

 

#!/bin/bash
#set whether your cpu is Intel or AMD  [1-Intel] [2-Amd] 
cputype="1"
#Do not edit below this line
#turn nested off

    if [[ "$cputype" =~ ^(1|2)$ ]]; then
        if [ "$cputype" -eq 1 ]; then
            modprobe -r kvm_intel
                modprobe kvm_intel nested=0
                
                echo "Nested vms are disabled for intel cpus"

            elif [ "$pushnotifications" -eq 2 ]; then
            modprobe -r kvm_amd
                    modprobe kvm_amd nested=0
                
                    echo "Nested vms are disabled for AMD cpus"            
            fi
        else
            
                echo "invalid cpu type set please check config"
            
            fi

sleep 4
exit

 

Share this post


Link to post

I'm sure this will prove useful for many users.  PIA have produced a new script to forward ONE port e.g. for running Plex behind a VPN which was been almost impossible before.

 

I'm not sure if you are limited to the same servers listed for the PIA app: https://helpdesk.privateinternetaccess.com/hc/en-us/articles/219460187-How-do-I-enable-port-forwarding-on-my-VPN-

 

The script has to be run within 2 mins of the VPN connection being made - I've set mine to run at the start of the array.  

 

In the log PIA will list what port has been forwarded that you can use in a NAT rule....Pain in the ass as a different port can get assigned each time - as you have to remember to update your router each time.

 

#!/bin/bash

error( )
{
  echo "$@" 1>&2
  exit 1
}

error_and_usage( )
{
  echo "$@" 1>&2
  usage_and_exit 1
}

usage( )
{
  echo "Usage: `dirname $0`/$PROGRAM"
}

usage_and_exit( )
{
  usage
  exit $1
}

version( )
{
  echo "$PROGRAM version $VERSION"
}


port_forward_assignment( )
{
  echo 'Loading port forward assignment information...'
  if [ "$(uname)" == "Linux" ]; then
    client_id=`head -n 100 /dev/urandom | sha256sum | tr -d " -"`
  fi
  if [ "$(uname)" == "Darwin" ]; then
    client_id=`head -n 100 /dev/urandom | shasum -a 256 | tr -d " -"`
  fi

  json=`curl "http://209.222.18.222:2000/?client_id=$client_id" 2>/dev/null`
  if [ "$json" == "" ]; then
    json='Port forwarding is already activated on this connection, has expired, or you are not connected to a PIA region that supports port forwarding'
  fi

  echo $json
}

EXITCODE=0
PROGRAM=`basename $0`
VERSION=2.1

while test $# -gt 0
do
  case $1 in
  --usage | --help | -h )
    usage_and_exit 0
    ;;
  --version | -v )
    version
    exit 0
    ;;
  *)
    error_and_usage "Unrecognized option: $1"
    ;;
  esac
  shift
done

port_forward_assignment

exit 0

 

Share this post


Link to post

Is it just me, or are none of the links in the first post working anymore?

Share this post


Link to post
43 minutes ago, Squazz said:

Is it just me, or are none of the links in the first post working anymore?

 

Share this post


Link to post
 

I take it that is a hint@trurl


Sent from my LG-D852 using Tapatalk

Share this post


Link to post

I wanted to clear a drive to remove from my array, a 2 TB WD green, using the clear_an_array_drive script.  I was unsure if I wanted to run it from the plug-in or command line.  I originally started it from the plug-in, but it appeared to be moving very slowly...  I cancelled it, and then started it from the command line in a screen.  It did not seem to be any quicker, so I changed my mind once again, and wanted to stop the script in the screen, and just remove the drive and rebuild parity.

 

I was unsure of how to close the script from within screen, and wrongly assumed if I killed the screen, the script would stop.

 

I went to power down the server, to remove the drive, and the GUI froze after stopping some services, but the system never powered down.

 

The screen I started the script in is now gone, killed.  I am no longer able to access the web GUI, although UnMenu is still working, and shows activity still going.  I can see the write number increasing for both the drive I am removing, and the parity drive ~ 680,000,000,000.  Additionally, I see the read number increasing for all other drives in the array.  (reconstruct write was turned on), and all my shares are still available.

 

I started all of this approximately 60 hours ago.

 

I was planning to just let things run, and thought the script may still complete itself, and hoping the system "might" just power itself down after.

 

Is there a simple, or "safe" way to stop all of this?  Or is my best option to let this continue for now?  I assume the script will not be "complete" until the write number approaches  2,000,000,000,000 which could take close to another 5 days?

 

I can attach a syslog from diagnostics when I get home, if that will help.

Share this post


Link to post
36 minutes ago, Turbobuickguy said:

UnMenu is still working

What version of unRAID are you using?

 

36 minutes ago, Turbobuickguy said:

I can attach a syslog from diagnostics when I get home, if that will help.

Complete diagnostics zip.

Share this post


Link to post
On 5/1/2017 at 2:28 PM, Squazz said:

Is it just me, or are none of the links in the first post working anymore?

fixed

Share this post


Link to post

About the Clear an unRAID array data drive script.

 

I started it about a day ago on a 500GB drive via the plugin, and is was half way but my browser crashed. Is there a way to check the progress again?

Edited by diditstart

Share this post


Link to post

I've automated some ripping of movies from HD to SD so I have 2 Libaries in my Plex server. One for the Big screen and the other for mobile devices. I don't like spaces in my file names so I devised a way to do some renaming prior to running through HandBrake and I wanted to create a folder for each file after ripping simply because I want a folder for each for my personal reasons. Took me a while to figure things out, but here is two little snippets of each. 

 

RemoveSpacesFromFile

#!/bin/bash

cd /SOURCE/
for f in *\ *; do mv "$f" "${f// /.}"; done

FolderfromFilename. Create folder from file name. Does not like spaces. Tis why I use above. ;-)

#!/bin/bash
#Create Folder from name and move to final location. File types editable below
for FILE in `ls /SOURCE/ | egrep "mkv|mp4"`
do
DIR=`echo $FILE | rev | cut -f 2- -d '.' | rev`
mkdir /DESTINATION/$DIR
mv /SOURCE/$FILE /DESTINATION/$DIR
done

 

Share this post


Link to post
On 8/2/2016 at 9:28 PM, Squid said:

A slightly enhanced version of the run mover at a certain threshold script.  This script additionally will skip running mover (optional) if a parity check / rebuild has already been started.

 

Only makes sense to run this script on a schedule, and disable the built-in schedule by editing the config/share.cfg file on the flash drive.  Look for a like that says something like:

 


shareMoverSchedule="0 4 * * *"
 

 

and change it to:


shareMoverSchedule="#0 4 * * *"
 

 

 

Followed by a reboot.  Note that any changes to global share settings ( or mover settings ) is probably going to wind up re-enabling the mover schedule

 

 

 


#!/usr/bin/php
<?PHP
$moveAt = 0;                 # Adjust this value to suit (% cache drive full to move at)
$runDuringCheck = false;     # change to true to run mover during a parity check / rebuild

$diskTotal = disk_total_space("/mnt/cache");
$diskFree = disk_free_space("/mnt/cache");
$percent = ($diskTotal - $diskFree) / $diskTotal * 100;

if ( $percent > $moveAt ) {
  if ( ! $runDuringCheck ) {
    $vars = parse_ini_file("/var/local/emhttp/var.ini");
    if ( $vars['mdResync'] ) {
      echo "Parity Check / Rebuild Running - Not executing mover\n";
      exec("logger Parity Check / Rebuild Running - Not executing mover");
    } else {
      exec("/usr/local/sbin/mover");
    }
  } else {
    exec("/usr/local/sbin/mover");
  }
}
?>
 

 

 

run_mover_at_threshold_enhanced.zip

 

Hi, I'd like to use this but I'm not quite sure where I can find the share.cfg file, and where I should put the php file you made. Can you please clarify? Thanks!

Share this post


Link to post
22 minutes ago, vmunich said:

 

Hi, I'd like to use this but I'm not quite sure where I can find the share.cfg file, and where I should put the php file you made. Can you please clarify? Thanks!

 

From the post you quoted:

22 minutes ago, vmunich said:

editing the config/share.cfg file on the flash drive

 

From the first post in this thread:

On 7/16/2016 at 10:00 PM, Squid said:

See the user.scripts thread for details on how to add these scripts (or any others)

 

The user scripts thread is linked in Squid's signature. If you can't see user sigs on the forum, click on the dropdown at upper right next to your name and go to your Account Settings.

Share this post


Link to post

What about a GitHub repo for all these great scripts? I can help making it.

 

(Feature Req) And have them listed in the GUI as "Community Scripts". Just like the CA plugin.

Edited by thomast_88
git -> github

Share this post


Link to post
2 hours ago, thomast_88 said:

What about a GitHub repo for all these great scripts? I can help making it.

 

(Feature Req) And have them listed in the GUI as "Community Scripts". Just like the CA plugin.

@CHBMB suggested the same thing a while ago, and I basically blew him off.  For you however my explanation would be 

- Time and effort expended in programming isn't worth it for the limited number of published scripts

- Maintenance on my end would be a nightmare due to basically having to verify compatibility / etc on any repository

 

If you really want to help and feel the effort is worth it, then feel free to fork the plugin and make whatever additions you like to it.  Myself, I can't justify doing it to myself.

Edited by Squid

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.