RobJ Posted January 28, 2017 Share Posted January 28, 2017 Well since no one is telling me that this is normal or not and it's been over 50 hours, I guess I'll have to see about killing the script. I'm sorry, I was avoiding this thread until I had time to answer ALL of the related posts. My apologies to you. I suppose I would have considered killing it after 5 to 6 hours, definitely if it was still running after 10 hours. I don't have any ideas yet as to what could have gone wrong. Can you tell if there is any I/O to the drive? Syslog may help, post your diagnostics (Need help? Read me first!) We should take this to a separate thread. Quote Link to comment
Endy Posted January 28, 2017 Share Posted January 28, 2017 Well since no one is telling me that this is normal or not and it's been over 50 hours, I guess I'll have to see about killing the script. I'm sorry, I was avoiding this thread until I had time to answer ALL of the related posts. My apologies to you. I suppose I would have considered killing it after 5 to 6 hours, definitely if it was still running after 10 hours. I don't have any ideas yet as to what could have gone wrong. Can you tell if there is any I/O to the drive? Syslog may help, post your diagnostics (Need help? Read me first!) We should take this to a separate thread. Thank you, that's all I needed to know. The only indication that there is something going on is the number of writes listed on the Main tab is slowly increasing for that drive and the parity drive. The syslog only shows that the process has started and nothing related since. Because this is in preparation for some significant hardware changes (new motherboard etc.), I am not really that worried about it. I've got 2 other 500GB drives I was planning on removing from the array as well and so as soon as I get my new drive precleared and the data moved off those other 2 drives, I'll just rebuild parity anyways. Quote Link to comment
Squid Posted February 4, 2017 Author Share Posted February 4, 2017 USB Hot Plug for VM's with no passthrough TLDR: The instructions are a pain for me to try and explain, but this script does work. Don't be afraid to ask questions. But first and foremost when experimenting with this on a running VM, set the startup delay to 1-2 seconds so you're not waiting forever for it to get up and running. Also make sure that you're on the latest version of user.scripts prior to playing around. EDIT: it appears that some devices can get tossed into a different bus group when plugged in. For this reason, this plugin basically is going to require a USB hub to be monitored for changes, as in my testing anything plugged into a hub will always stay in the same group as the hub A simple little script to monitor a particular USB bus for changes and then automatically attach / detach the USB device to a particular VM. Useful if you cannot pass through a USB controller to a VM (not that I need this as I have a USB controller passed through to my main VM, but I was rather dismayed at the libvirt USB Hotplug plugin which didn't support true hotplugging without going back to the UI and manually attaching / detaching devices) This started out as being a potential plugin, but there are enough caveats with the initial setup to make this impractical. Couple of variables within the script need to be set: $vmName - Set to the name of the VM (must be exact) $pollingTime - Set to the number of seconds between looking for changes $startupDelay - Set to the number of seconds to wait after array start to begin monitoring And 2 variables of which one or the other should be set: (Comment out the one line that's not going to get set (ie: put a # in front of it) ) $hubName and $bus If you are using a particular USB bus (ie: 2 particular ports on the computer), the set the $bus to be the bus as listed by lsusb If you are using a USB Hub, I recommend that you set this to the exact name of the USB hub as determined by lsusb And now the caveats (which are workable IMHO) Bus: each USB bus is going to handle multiple ports on your computer. You want to make sure that those ports are going to be dedicated to the VM in question (ie: You can't use one of those ports for say Unassigned Devices) Also, on my system I found that when I plug a USB hub into a particular bus, then the hub gets moved into another bus which may or may not already have devices on it. TLDR: If you use the "bus" variable, don't plug a USB hub into it -> it won't work. hubName: If you want to use a usb Hub, then set this variable to be the exact name of the hub. Problem with this mode is that if the hub when plugged in gets moved from an isolated bus to one with other devices on it then everything on that particular bus either has to be permanently attached to unRaid itself (not the VM) otherwise the script will attempt to detach / attach those items from the VM. This is the recommended way of handling USB devices To illustrate, I have numerous empty USB busses on my system. If I plug in a USB Hub into one of those usb's, then the hub winds up being moved from the empty bus to another bus on my system which already has my unRaid flash drive and keyboard on it. Since those two devices are permanently attached to unRaid, this isn't a big deal to me. The best way to figure out how to set things is to plugin and unplug a device from every usb port on your system and type lsusb in between to see how it all reacts. Like I said, if you don't need a hub, find an unused bus and use it. If you need a hub, then make sure that everything else on that bus is permanent to unRaid itself an be aware that any other ports on your system that are on the same bus will also be passed through to the VM. Note that when I'm talking about USB Busses, I am not referring to USB 2 or USB 3. Each motherboard will have multiple busses on the system, each of which in general will control 2 ports connected to it. Additional caveat, do to how linux operates, I cannot guarantee that a particular port will always be on the same bus over resets of the system, but in my testing, it always was. One final caveat. Because this plugin monitors the bus for changes, upon startup (after the initial startup delay is finished), any device plugged into the bus or hub will not be attached to the VM. You must pull the device and reinsert it. Any pull and reinsert of the same device should take longer than the polling time (I'm defaulting that to 10 seconds) Pulls and inserts of different devices can be done as fast as you want. #!/usr/bin/php <?PHP #backgroundOnly=true #description=A script designed to monitor a USB hub or a particular USB bus for changes and then attach or detach devices to an applicable VM $vmName = "Windows 10"; # must be the exact name $pollingTime = 10; # the interval between checks in seconds $startupDelay = 300; # startup delay before monitoring for changes in seconds (set to enough time for the VM to get up and running) # only use one or the other of the following lines not both See thread for details #$hubName = "Texas Instruments, Inc. TUSB2046 Hub"; # must be exact as listed via lsusb $bus = "02"; # see thread for details function getDeviceList($bus) { exec("lsusb | grep 'Bus $bus'",$allDevicesOnBus); foreach ($allDevicesOnBus as $Devices) { $deviceLine = explode(" ",$Devices); $initialState[$deviceLine[5]] = $deviceLine[5]; } return $initialState; } function createXML($deviceID) { $usb = explode(":",$deviceID); $usbstr .= "<hostdev mode='subsystem' type='usb'> <source> <vendor id='0x".$usb[0]."'/> <product id='0x".$usb[1]."'/> </source> </hostdev>"; file_put_contents("/tmp/USBTempXML.xml",$usbstr); } function logger($string) { shell_exec("logger ".escapeshellarg($string)); } # Begin Main logger("Sleeping for $startupDelay before monitoring $bus for changes to passthrough to $vmName"); sleep($startupDelay); $hubBus = $bus; if ( ! $hubBus ) { $hub = explode(" ",exec("lsusb | grep '$hubName'")); $hubBus = $hub[1]; } logger("Monitoring $hubBus for changes"); $initialState = getDeviceList($hubBus); while (true) { $unRaidVars = parse_ini_file("/var/local/emhttp/var.ini"); if ($unRaidVars['mdState'] != "STARTED") { break; } $currentDevices = getDeviceList($hubBus); foreach ($currentDevices as $Device) { if ( ! $initialState[$Device] ) { logger("$Device Added to bus $hubBus Attaching to $vmName"); createXML($Device); exec("/usr/sbin/virsh attach-device '$vmName' /tmp/USBTempXML.xml 2>&1"); $initialState[$Device] = $Device; } } foreach ($initialState as $Device) { if ( ! $currentDevices[$Device] ) { logger("$Device Removed from bus $hubBus Detaching from $vmName"); createXML($Device); exec("/usr/sbin/virsh detach-device '$vmName' /tmp/USBTempXML.xml 2>&1"); unset ($initialState[$Device]); } } sleep($pollingTime); } ?> Technie note: There is no way to monitor a particular port for changes, because every time you remove and insert a device, the device number associated with the bus increases. Also, this script does not use udev to automatically attach / detach the usb devices. This is because: the USB device in question may or may not be known prior to attaching (ie: a buddy brought over a USB stick for you to use), and secondly because in theory you could have multiple copies of this script running for different VMs and the same device may wind up being plugged into a different VM as you require, rather than the one udev dictated. IE: I went for flexibility at the expense of minimal processor time. USB_HotPlug.zip Quote Link to comment
CHBMB Posted February 6, 2017 Share Posted February 6, 2017 Hey Squid, just had a thought for a feature. Allow user.scripts to be pulled from github. Could add repos like docker. Just an idea. Would help sharing scripts etc.... And gives you something to do. Quote Link to comment
Squid Posted February 7, 2017 Author Share Posted February 7, 2017 Hey Squid, just had a thought for a feature. Allow user.scripts to be pulled from github. Could add repos like docker. Just an idea. Would help sharing scripts etc.... And gives you something to do. Really!?! Quote Link to comment
CHBMB Posted February 7, 2017 Share Posted February 7, 2017 I take it you're not keen then lol Sent from my LG-H815 using Tapatalk Quote Link to comment
dimes007 Posted February 23, 2017 Share Posted February 23, 2017 Like Endy I'm seeing a very long run of the clear disk script. But stats are updating when I'm running so I'm certain the script didn't hang. I'm currently running the clear script on a 4 year old Hitachi 7200rpm disk on md1 and sdf. The drive isn't failing and throws no smart errors. There is no substantial activity on any other mdX devices. I turned on turbo write in the GUI (but didn't reboot if that matters). My stats:Unmounting Disk 1 ...umount: /mnt/disk1: not mounted [ed: don't worry about this. I killed it once because I was moving other data and figured I'd wait before I run it]Clearing Disk 1 ...66783805440 bytes (67 GB, 62 GiB) copied, 52920 s, 1.3 MB/s So I'm looking at about 18 days of total run time? I see read activity for all of my array drives on sdX of about 1.3 MB/s including the drive I'm clearing. I see write activity for md1 of the 1.3MB/s. I also see read activity for sdf, the drive i'm clearing, of about 10MB/s. Removing a cleared device and replacing it with a precleared drive and not having to lose parity (especially since I have only 1 parity disk) seems like a no brainer. I'm uncertain why all sd? devices are read to maintain parity. I always thought that unraid only did an XOR calc for parity on data change. I guess because its block level with DD on an unmounted drive it needs a full calculation using all disks? Is it possible to modify the script to write 0's to a file in the file system. Then unmount and zero the partition info with dd using the slower method over just the partition info? Thanks. Quote Link to comment
RobJ Posted February 24, 2017 Share Posted February 24, 2017 10 hours ago, dimes007 said: Like Endy I'm seeing a very long run of the clear disk script. But stats are updating when I'm running so I'm certain the script didn't hang. I'm currently running the clear script on a 4 year old Hitachi 7200rpm disk on md1 and sdf. The drive isn't failing and throws no smart errors. There is no substantial activity on any other mdX devices. I turned on turbo write in the GUI (but didn't reboot if that matters). My stats:Unmounting Disk 1 ...umount: /mnt/disk1: not mounted [ed: don't worry about this. I killed it once because I was moving other data and figured I'd wait before I run it]Clearing Disk 1 ...66783805440 bytes (67 GB, 62 GiB) copied, 52920 s, 1.3 MB/s So I'm looking at about 18 days of total run time? I see read activity for all of my array drives on sdX of about 1.3 MB/s including the drive I'm clearing. I see write activity for md1 of the 1.3MB/s. I also see read activity for sdf, the drive i'm clearing, of about 10MB/s Because you turned on Turbo Write, ALL of the other disks have to be read in order to determine what to write to the parity disk. For this operation, Turbo Write is faster that the alternative, but clearly it's not helping you. Because Turbo Write depends on ALL of the array drives, and it cannot move faster than the slowest drive, it's my suspicion you have a bad drive somewhere, that's dragging down the rest. Your syslog may be full of drive errors, post your diagnostics. 10 hours ago, dimes007 said: I'm uncertain why all sd? devices are read to maintain parity. I always thought that unraid only did an XOR calc for parity on data change. I guess because its block level with DD on an unmounted drive it needs a full calculation using all disks? That would be correct for the Read/Modify/Write method, but you changed it to Turbo Write (which would normally be faster here). 10 hours ago, dimes007 said: Is it possible to modify the script to write 0's to a file in the file system. Then unmount and zero the partition info with dd using the slower method over just the partition info? While it's possible to overwrite the space of each file with zeroes, that's not enough, you still have to zero out ALL of the deleted files, the remnants of past files, and the past versions of files, AND ALL of the sectors of the file system itself. We don't know where they are. For modern file systems, they can be scattered over the entire drive surface. Quote Link to comment
SpaceInvaderOne Posted February 25, 2017 Share Posted February 25, 2017 As nested vms have been disabled by default in 6.3 due to the issue with avast and windows vms when enabled. I have made 2 scripts to turn it on and off. In script set cpu type 1 for intel 2 for amd. Download link for both scripts https://www.dropbox.com/s/0b1tvotvl6y80uy/nested vms scripts.zip?dl=0 nested vm on script #!/bin/bash #set whether your cpu is Intel or AMD [1-Intel] [2-Amd] cputype="1" #Do not edit below this line #turn nested on if [[ "$cputype" =~ ^(1|2)$ ]]; then if [ "$cputype" -eq 1 ]; then modprobe -r kvm_intel modprobe kvm_intel nested=1 echo "Nested vms are enabled for intel cpus" elif [ "$pushnotifications" -eq 2 ]; then modprobe -r kvm_amd modprobe kvm_amd nested=1 echo "Nested vms are enabled for AMD cpus" fi else echo "invalid cpu type set please check config" fi sleep 4 exit and nested vm off script #!/bin/bash #set whether your cpu is Intel or AMD [1-Intel] [2-Amd] cputype="1" #Do not edit below this line #turn nested off if [[ "$cputype" =~ ^(1|2)$ ]]; then if [ "$cputype" -eq 1 ]; then modprobe -r kvm_intel modprobe kvm_intel nested=0 echo "Nested vms are disabled for intel cpus" elif [ "$pushnotifications" -eq 2 ]; then modprobe -r kvm_amd modprobe kvm_amd nested=0 echo "Nested vms are disabled for AMD cpus" fi else echo "invalid cpu type set please check config" fi sleep 4 exit Quote Link to comment
DZMM Posted March 22, 2017 Share Posted March 22, 2017 I'm sure this will prove useful for many users. PIA have produced a new script to forward ONE port e.g. for running Plex behind a VPN which was been almost impossible before. I'm not sure if you are limited to the same servers listed for the PIA app: https://helpdesk.privateinternetaccess.com/hc/en-us/articles/219460187-How-do-I-enable-port-forwarding-on-my-VPN- The script has to be run within 2 mins of the VPN connection being made - I've set mine to run at the start of the array. In the log PIA will list what port has been forwarded that you can use in a NAT rule....Pain in the ass as a different port can get assigned each time - as you have to remember to update your router each time. #!/bin/bash error( ) { echo "[email protected]" 1>&2 exit 1 } error_and_usage( ) { echo "[email protected]" 1>&2 usage_and_exit 1 } usage( ) { echo "Usage: `dirname $0`/$PROGRAM" } usage_and_exit( ) { usage exit $1 } version( ) { echo "$PROGRAM version $VERSION" } port_forward_assignment( ) { echo 'Loading port forward assignment information...' if [ "$(uname)" == "Linux" ]; then client_id=`head -n 100 /dev/urandom | sha256sum | tr -d " -"` fi if [ "$(uname)" == "Darwin" ]; then client_id=`head -n 100 /dev/urandom | shasum -a 256 | tr -d " -"` fi json=`curl "http://209.222.18.222:2000/?client_id=$client_id" 2>/dev/null` if [ "$json" == "" ]; then json='Port forwarding is already activated on this connection, has expired, or you are not connected to a PIA region that supports port forwarding' fi echo $json } EXITCODE=0 PROGRAM=`basename $0` VERSION=2.1 while test $# -gt 0 do case $1 in --usage | --help | -h ) usage_and_exit 0 ;; --version | -v ) version exit 0 ;; *) error_and_usage "Unrecognized option: $1" ;; esac shift done port_forward_assignment exit 0 Quote Link to comment
Squazz Posted May 1, 2017 Share Posted May 1, 2017 Is it just me, or are none of the links in the first post working anymore? Quote Link to comment
trurl Posted May 1, 2017 Share Posted May 1, 2017 43 minutes ago, Squazz said: Is it just me, or are none of the links in the first post working anymore? Quote Link to comment
Squid Posted May 1, 2017 Author Share Posted May 1, 2017 I take it that is a hint@trurlSent from my LG-D852 using Tapatalk Quote Link to comment
Turbobuickguy Posted May 2, 2017 Share Posted May 2, 2017 I wanted to clear a drive to remove from my array, a 2 TB WD green, using the clear_an_array_drive script. I was unsure if I wanted to run it from the plug-in or command line. I originally started it from the plug-in, but it appeared to be moving very slowly... I cancelled it, and then started it from the command line in a screen. It did not seem to be any quicker, so I changed my mind once again, and wanted to stop the script in the screen, and just remove the drive and rebuild parity. I was unsure of how to close the script from within screen, and wrongly assumed if I killed the screen, the script would stop. I went to power down the server, to remove the drive, and the GUI froze after stopping some services, but the system never powered down. The screen I started the script in is now gone, killed. I am no longer able to access the web GUI, although UnMenu is still working, and shows activity still going. I can see the write number increasing for both the drive I am removing, and the parity drive ~ 680,000,000,000. Additionally, I see the read number increasing for all other drives in the array. (reconstruct write was turned on), and all my shares are still available. I started all of this approximately 60 hours ago. I was planning to just let things run, and thought the script may still complete itself, and hoping the system "might" just power itself down after. Is there a simple, or "safe" way to stop all of this? Or is my best option to let this continue for now? I assume the script will not be "complete" until the write number approaches 2,000,000,000,000 which could take close to another 5 days? I can attach a syslog from diagnostics when I get home, if that will help. Quote Link to comment
trurl Posted May 2, 2017 Share Posted May 2, 2017 36 minutes ago, Turbobuickguy said: UnMenu is still working What version of unRAID are you using? 36 minutes ago, Turbobuickguy said: I can attach a syslog from diagnostics when I get home, if that will help. Complete diagnostics zip. Quote Link to comment
Turbobuickguy Posted May 3, 2017 Share Posted May 3, 2017 20 hours ago, trurl said: What version of unRAID are you using? Complete diagnostics zip. I'm running 6.3.3 Complete diagnostics attached. tower-diagnostics-20170503-0412.zip Quote Link to comment
Squid Posted May 3, 2017 Author Share Posted May 3, 2017 On 5/1/2017 at 2:28 PM, Squazz said: Is it just me, or are none of the links in the first post working anymore? fixed 1 Quote Link to comment
Squazz Posted May 6, 2017 Share Posted May 6, 2017 On 3/5/2017 at 4:09 PM, Squid said: fixed Awesome! Quote Link to comment
diditstart Posted June 29, 2017 Share Posted June 29, 2017 (edited) About the Clear an unRAID array data drive script. I started it about a day ago on a 500GB drive via the plugin, and is was half way but my browser crashed. Is there a way to check the progress again? Edited June 29, 2017 by diditstart Quote Link to comment
kizer Posted July 2, 2017 Share Posted July 2, 2017 I've automated some ripping of movies from HD to SD so I have 2 Libaries in my Plex server. One for the Big screen and the other for mobile devices. I don't like spaces in my file names so I devised a way to do some renaming prior to running through HandBrake and I wanted to create a folder for each file after ripping simply because I want a folder for each for my personal reasons. Took me a while to figure things out, but here is two little snippets of each. RemoveSpacesFromFile #!/bin/bash cd /SOURCE/ for f in *\ *; do mv "$f" "${f// /.}"; done FolderfromFilename. Create folder from file name. Does not like spaces. Tis why I use above. #!/bin/bash #Create Folder from name and move to final location. File types editable below for FILE in `ls /SOURCE/ | egrep "mkv|mp4"` do DIR=`echo $FILE | rev | cut -f 2- -d '.' | rev` mkdir /DESTINATION/$DIR mv /SOURCE/$FILE /DESTINATION/$DIR done Obviously do some trial and error on some test files just to be sure. 😁 Quote Link to comment
vmunich Posted July 3, 2017 Share Posted July 3, 2017 On 8/2/2016 at 9:28 PM, Squid said: A slightly enhanced version of the run mover at a certain threshold script. This script additionally will skip running mover (optional) if a parity check / rebuild has already been started. Only makes sense to run this script on a schedule, and disable the built-in schedule by editing the config/share.cfg file on the flash drive. Look for a like that says something like: shareMoverSchedule="0 4 * * *" and change it to: shareMoverSchedule="#0 4 * * *" Followed by a reboot. Note that any changes to global share settings ( or mover settings ) is probably going to wind up re-enabling the mover schedule #!/usr/bin/php <?PHP $moveAt = 0; # Adjust this value to suit (% cache drive full to move at) $runDuringCheck = false; # change to true to run mover during a parity check / rebuild $diskTotal = disk_total_space("/mnt/cache"); $diskFree = disk_free_space("/mnt/cache"); $percent = ($diskTotal - $diskFree) / $diskTotal * 100; if ( $percent > $moveAt ) { if ( ! $runDuringCheck ) { $vars = parse_ini_file("/var/local/emhttp/var.ini"); if ( $vars['mdResync'] ) { echo "Parity Check / Rebuild Running - Not executing mover\n"; exec("logger Parity Check / Rebuild Running - Not executing mover"); } else { exec("/usr/local/sbin/mover"); } } else { exec("/usr/local/sbin/mover"); } } ?> run_mover_at_threshold_enhanced.zip Hi, I'd like to use this but I'm not quite sure where I can find the share.cfg file, and where I should put the php file you made. Can you please clarify? Thanks! Quote Link to comment
trurl Posted July 3, 2017 Share Posted July 3, 2017 22 minutes ago, vmunich said: Hi, I'd like to use this but I'm not quite sure where I can find the share.cfg file, and where I should put the php file you made. Can you please clarify? Thanks! From the post you quoted: 22 minutes ago, vmunich said: editing the config/share.cfg file on the flash drive From the first post in this thread: On 7/16/2016 at 10:00 PM, Squid said: See the user.scripts thread for details on how to add these scripts (or any others) The user scripts thread is linked in Squid's signature. If you can't see user sigs on the forum, click on the dropdown at upper right next to your name and go to your Account Settings. Quote Link to comment
thomast_88 Posted July 12, 2017 Share Posted July 12, 2017 (edited) What about a GitHub repo for all these great scripts? I can help making it. (Feature Req) And have them listed in the GUI as "Community Scripts". Just like the CA plugin. Edited July 12, 2017 by thomast_88 git -> github Quote Link to comment
Squid Posted July 12, 2017 Author Share Posted July 12, 2017 (edited) 2 hours ago, thomast_88 said: What about a GitHub repo for all these great scripts? I can help making it. (Feature Req) And have them listed in the GUI as "Community Scripts". Just like the CA plugin. @CHBMB suggested the same thing a while ago, and I basically blew him off. For you however my explanation would be - Time and effort expended in programming isn't worth it for the limited number of published scripts - Maintenance on my end would be a nightmare due to basically having to verify compatibility / etc on any repository If you really want to help and feel the effort is worth it, then feel free to fork the plugin and make whatever additions you like to it. Myself, I can't justify doing it to myself. Edited July 12, 2017 by Squid Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.