July 26, 20206 yr Author 13 minutes ago, mgutt said: Feature Request: I really like to see echo output inside the logs (maybe as optional setting?). At the moment only errors are logged. It already does that
July 26, 20206 yr 11 minutes ago, Squid said: It already does that Does not work for me. rclone is running and the logs are "empty": "vi /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt" returns the same.
July 26, 20206 yr Author Redirect the output of the command something like rclone ....... >> /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt
July 26, 20206 yr This would make the scripts to complicate and it's not easy to use / share them on multiple platforms. EDIT: Ok, found a solution that is more portable and works for all unraid scripts: exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1 This needs to be put in the first line after "#!/bin/bash" and now all command returns and echo's are logged! Other versions I tested that did not work (only for information purposes): # does not work #exec >> /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt #exec 3>&1 1>>/tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt 2>&1 #exec > >(tee --append --ignore-interrupts /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt) #exec 1>&1 #exec 2>&1 Example script: #!/bin/bash # logging exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1 # rclone rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -v --stats 10s Log output: Edited July 27, 20206 yr by mgutt
July 27, 20206 yr 22 hours ago, mgutt said: This would make the scripts to complicate and it's not easy to use / share them on multiple platforms. EDIT: Ok, found a solution that is more portable and works for all unraid scripts: exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1 This needs to be put in the first line after "#!/bin/bash" and now all command returns and echo's are logged! Other versions I tested that did not work (only for information purposes): # does not work #exec >> /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt #exec 3>&1 1>>/tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt 2>&1 #exec > >(tee --append --ignore-interrupts /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt) #exec 1>&1 #exec 2>&1 Example script: #!/bin/bash # logging exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1 # rclone rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -v --stats 10s Log output: It's a bit more heavyweight solution, but you can always write your own wrapper that does logging, timeouts, interlocking, and notifications. Then you can run all of your crons using this wrapper.
July 27, 20206 yr 36 minutes ago, Phoenix Down said: It's a bit more heavyweight solution, but you can always write your own wrapper that does logging, timeouts, interlocking, and notifications. Yes, but this one-liner is absolutely sufficient
July 28, 20206 yr I need to create a script that will look through a file to find a specific text and replace all the matches with other text. How can I do this? lets say the file is /mnt/user/text/mytestfile.txt I am looking for #FF8C2F and want to replace them all with #42ADFA. This is something I need to run after every reboot.
July 28, 20206 yr 6 minutes ago, almulder said: I need to create a script that will look through a file to find a specific text and replace all the matches with other text. How can I do this? lets say the file is /mnt/user/text/mytestfile.txt I am looking for #FF8C2F and want to replace them all with #42ADFA. This is something I need to run after every reboot. You could write a small Perl script to do it. Maybe something like this: #!/usr/bin/env perl use strict; use warnings; use Carp; use English; # Read the file my $file = "/mnt/user/text/mytestfile.txt"; my $fh; open $fh, "<", $file or croak("\nERROR: Cannot open '$file' for reading: $OS_ERROR\n\n"); my @rawdata = <$fh>; close $fh; # Update each line my @newdata; foreach my $line (@rawdata) { $line =~ s/#FF8C2F/#42ADFA/g; push @newdata, $line; } # Update the file with the updated data open $fh, ">", $file or croak("\nERROR: Cannot open '$file' for writing: $OS_ERROR\n\n"); foreach my $line (@newdata) { print $fh $line; } close $fh;
July 29, 20206 yr 25 minutes ago, Phoenix Down said: You could write a small Perl script to do it. Maybe something like this: #!/usr/bin/env perl use strict; use warnings; use Carp; use English; # Read the file my $file = "/mnt/user/text/mytestfile.txt"; my $fh; open $fh, "<", $file or croak("\nERROR: Cannot open '$file' for reading: $OS_ERROR\n\n"); my @rawdata = <$fh>; close $fh; # Update each line my @newdata; foreach my $line (@rawdata) { $line =~ s/#FF8C2F/#42ADFA/g; push @newdata, $line; } # Update the file with the updated data open $fh, ">", $file or croak("\nERROR: Cannot open '$file' for writing: $OS_ERROR\n\n"); foreach my $line (@newdata) { print $fh $line; } close $fh; Thanks for your help. Thats more complicated than I wanted. However if made me dig deeper and not just search for unraid but linux code and found out I could do this all in 1 line and it works great. sed -i 's/#FF8C2F/#42ADFA/gI' /mnt/user/text/mytestfile.txt Also found a way to use variables instead: old_color="#e22828" new_color="#00378F" sed -i "s/$old_color/$new_color/gI" /mnt/user/Unraid_Backup/Newlogos/hello.txt Hope this helps other one day!
July 29, 20206 yr 17 hours ago, Phoenix Down said: You could write a small Perl script to do it. Maybe something like this: #!/usr/bin/env perl use strict; use warnings; use Carp; use English; # Read the file my $file = "/mnt/user/text/mytestfile.txt"; my $fh; open $fh, "<", $file or croak("\nERROR: Cannot open '$file' for reading: $OS_ERROR\n\n"); my @rawdata = <$fh>; close $fh; # Update each line my @newdata; foreach my $line (@rawdata) { $line =~ s/#FF8C2F/#42ADFA/g; push @newdata, $line; } # Update the file with the updated data open $fh, ">", $file or croak("\nERROR: Cannot open '$file' for writing: $OS_ERROR\n\n"); foreach my $line (@newdata) { print $fh $line; } close $fh; You can achieve the same thing from the command line with PERL: perl -pi -e 's/#FF8C2F/#42ADFA/g' /mnt/user/text/mytestfile.txt Perl one-liners Edited July 29, 20206 yr by JoeUnraidUser
July 29, 20206 yr 6 minutes ago, JoeUnraidUser said: You can do the same thing from the command with PERL: perl -pi -e 's/#FF8C2F/#42ADFA/g' /mnt/user/text/mytestfile.txt Perl one-liners Good info, thanks I don't think I ever looked at the "-i" switch.
July 30, 20206 yr having issues with ssh keys i have it setup so when unraid boots it copys the ssh folder and does the keyscan stuff and chkmod so i can do rsync without a password and that works.. for a while.. it seems like it times out after 24 hours it releases and now rsync stops and public is released out of memory.. as if you drop to the terminal and ssh <servername> u gotta type yes no thing again.. if i reboot the server running the script to access the other server.. i can ssh <server> and not have to enter a password my question is how do you keep it from loosing like a network dhcp lease... as i dont wanna always reboot server just to rsync to refresh the public key so say after 1000 minutes unraid release the public key leases... is there a way to permently keep it in
July 30, 20206 yr apparenty it will loose it after 30 min also not like 1000 minutes.. i dunno how it can be loosing the public key
July 31, 20206 yr Although my last test returned the opposite, I found out that CA user scripts is not race condition safe. So I changed my script as follows: #!/bin/bash # kill all commands on exit function exitus() { exit_status=$1 sudo pkill -f rcloneorig sleep 1 sudo pkill -f rcloneorig sleep 1 sudo pkill -f rcloneorig sleep 1 sudo pkill -f rcloneorig sleep 1 rmdir "/tmp/${0///}" exit $exit_status } # make script race condition safe if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1 # do not use exitus here or it will kill the already running script! fi trap 'exitus 1' EXIT # logging exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1 # rclone rclone copy /mnt/user/software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d rclone copy /mnt/user/music owncube:music --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d rclone copy /mnt/user/movie owncube:movie --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d rclone copy /mnt/user/tv owncube:tv --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d Then I pressed "Abort Script", waited 10 seconds and checked if my tmp dir is still existing and it does: A short check with top confirmed that the rlconeorig process is still active. So I killed rclone as follows: As you can see I needed to kill every rclone command separately and you can see that "rmdir" of my script has been executed as my atomic dir "tmpuser.scriptstmpScriptsbackup_shares_owncubescript" is gone. Conclusion: The "Abort Script" button does nothing except removing the "Running"-status in the webclient. The script itself runs until the end. Edited August 1, 20206 yr by mgutt
August 3, 20205 yr Little bug: After an update the running status of a script gets lost (its still running, but the web GUI doesn't display it).
August 4, 20205 yr Looks like aborting a script does not kill everything. I'm using a user script that 'bashes' a shell script, and that shell script in turn runs a python script: #!/bin/bash #backgroundOnly=true #arrayStarted=true bash /boot/custom/kuroautomode.sh (I don't know how to use bash to call python directly? So this is how i do it. bash is needed, the .sh and .py reside on the usb stick in a subfolder under boot folder) When this is started, proceslist will show 4 processes: root 22117 0.0 0.0 104624 26196 ? SL Aug03 0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/kuroAutoMode/script root 22118 0.0 0.0 3840 2916 ? S Aug03 0:00 sh -c /tmp/user.scripts/tmpScripts/kuroAutoMode/script >> /tmp/user.scripts/tmpScripts/kuroAutoMode/log.txt 2>&1 root 22119 0.0 0.0 3840 2808 ? S Aug03 0:00 /bin/bash /tmp/user.scripts/tmpScripts/kuroAutoMode/script root 22120 0.0 0.0 3840 2916 ? S Aug03 0:00 bash /boot/custom/kuroautomode.sh root 22121 0.0 0.0 18716 14564 ? S Aug03 0:02 /usr/bin/python /boot/custom/kuroautomodecmdline.py Now, when i abort the user script from within unraid, proclist still shows: root 22119 0.0 0.0 3840 2808 ? S Aug03 0:00 /bin/bash /tmp/user.scripts/tmpScripts/kuroAutoMode/script root 22120 0.0 0.0 3840 2916 ? S Aug03 0:00 bash /boot/custom/kuroautomode.sh root 22121 0.0 0.0 18716 14564 ? S Aug03 0:02 /usr/bin/python /boot/custom/kuroautomodecmdline.py So it only aborts the script logging, nothing else. Everything else keeps running. If i restart the user.script, i get everything running 2x, 3x, 4x etc. if i don't manually remove it. Am i doing something wrong or is this an issue? Edited August 4, 20205 yr by jowi
August 4, 20205 yr 41 minutes ago, jowi said: Am i doing something wrong or is this an issue? This is a bug as stated in my post.
August 4, 20205 yr Author On 7/31/2020 at 6:08 AM, mgutt said: A short check with top confirmed that the rlconeorig process is still active. But was the script itself still running? It's a royal pain to kill off sub-processes from the main script, especially since the only thing I've got to go on is the name of the main script. The system does attempt to kill off children (pkill -TREM -P). If you can come up with a better solution, then please let me know.
August 4, 20205 yr 2 minutes ago, Squid said: But was the script itself still running? It's a royal pain to kill off sub-processes from the main script, especially since the only thing I've got to go on is the name of the main script. The system does attempt to kill off children (pkill -TREM -P). If you can come up with a better solution, then please let me know. What if you crawl the process tree and kill each child process individually? It's a bit heavy handed, but should take care of any orphaned child processes.
August 4, 20205 yr All scripts should be well behaved and handle the spawning and reaping of child processes themselves. It is their own responsibility. I suggest serious scripters put together a template using best practices to make it easier for other scripters to not fall into the common pitfalls. Edited August 4, 20205 yr by BRiT
August 4, 20205 yr Author 24 minutes ago, Phoenix Down said: What if you crawl the process tree and kill each child process individually? It's a bit heavy handed, but should take care of any orphaned child processes. And that's what the -P on the pkill is supposed to do. Kill off children from the main PID
August 4, 20205 yr 15 minutes ago, Squid said: And that's what the -P on the pkill is supposed to do. Kill off children from the main PID Right, but it doesn't appear to be working as advertised. Hence taking matters into your own hands
August 4, 20205 yr 54 minutes ago, Squid said: But was the script itself still running? Yes. I proved it in my post. The script contains 4 rclone commands. After I aborted the script through the GUI it does not kill the running rlcone process and I need to kill every following command separately (the first rclone command was already finished, so I needed to kill only 3): (The "-c" flag returns the amount of processes that were killed) And after I killed the last rclone process it executes the very last line of the script, too (the rmdir command). This proves that the script is still running (and not only one long process). How do you execute the script if the "run in background" button is pressed? Edited August 4, 20205 yr by mgutt
August 5, 20205 yr 16 hours ago, BRiT said: All scripts should be well behaved and handle the spawning and reaping of child processes themselves. It is their own responsibility. I understand there is no way to respond to a sigkill...
August 5, 20205 yr @Squid I tried many commands and by that I found out that pkill does not kill (all) child processes: root@Thoth:/tmp# pgrep -f isleep2m | xargs --no-run-if-empty ps fp PID TTY STAT TIME COMMAND 19608 ? SL 0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/isleep2m/script 19609 ? S 0:00 \_ sh -c /tmp/user.scripts/tmpScripts/isleep2m/script >> /tmp/user.scripts/tmpScripts/isleep2m/log.txt 2>&1 19610 ? S 0:00 \_ /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script root@Thoth:/tmp# pkill -9 -P 19608 root@Thoth:/tmp# pgrep -f isleep2m | xargs --no-run-if-empty ps fp PID TTY STAT TIME COMMAND 19610 ? S 0:00 /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script But I finally found two solutions to repair the "Abort Script" button.: A) Run every script in its own process group. Something similar to this should work: (set -m;sh -c /tmp/user.scripts/tmpScripts/kllme/script & pid=$!;echo $pid) Then you are able to get the pgid as follows: ps opgid= $pid Now you can kill the complete tree with the pgid kill -- -$pgid Note: It is important to execute the user script and all the other commands (like logging) in a new process group (pgid) as at the moment all running user scripts executed through PHP get the same pgid. In my case its "1733" and killing this group would kill all scripts: ("backup_shares" and "isleep2m"): root@Thoth:/tmp# pgrep -f isleep2m | xargs --no-run-if-empty ps fp PID TTY STAT TIME COMMAND 11772 ? SL 0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/isleep2m/script 11773 ? S 0:00 \_ sh -c /tmp/user.scripts/tmpScripts/isleep2m/script >> /tmp/user.scripts/tmpScripts/kllme/log.txt 2>&1 11774 ? S 0:00 \_ /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script root@Thoth:/tmp# ps opgid= 11772 1733 root@Thoth:/tmp# pgrep -g 1733 | xargs --no-run-if-empty ps fp PID TTY STAT TIME COMMAND 1733 ? Ss 0:00 /usr/sbin/atd -b 15 -l 1 26844 ? S 0:00 \_ /usr/sbin/atd -b 15 -l 1 26845 ? S 0:00 | \_ sh 26846 ? SL 346:12 | \_ /usr/bin/php /usr/local/emhttp/plugins/fix.common.problems/scripts/extendedTest.php 23911 ? S 0:00 \_ /usr/sbin/atd -b 15 -l 1 23912 ? S 0:00 | \_ sh 23913 ? SL 0:00 | \_ /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/backup_shares/script 23914 ? S 0:00 | \_ sh -c /tmp/user.scripts/tmpScripts/backup_shares/script >> /tmp/user.scripts/tmpScripts/backup_shares/log.txt 2>&1 23915 ? S 0:00 | \_ /bin/bash /tmp/user.scripts/tmpScripts/backup_shares/script 23917 ? S 0:00 | \_ /bin/bash /tmp/user.scripts/tmpScripts/backup_shares/script 23925 ? S 0:00 | | \_ tee --append --ignore-interrupts /tmp/user.scripts/tmpScripts/backup_shares/log.txt 2752 ? S 0:00 | \_ /bin/bash /usr/sbin/rclone sync /mnt/user/sharedir nextcloud:sharedir --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2753 ? Sl 10:48 | \_ rcloneorig --config /boot/config/plugins/rclone/.rclone.conf sync /mnt/user/sharedir nextcloud:sharedir --create-empty-src-dirs --i 11770 ? S 0:00 \_ /usr/sbin/atd -b 15 -l 1 11771 ? S 0:00 \_ sh 11772 ? SL 0:00 \_ /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/isleep2m/script 11773 ? S 0:00 \_ sh -c /tmp/user.scripts/tmpScripts/isleep2m/script >> /tmp/user.scripts/tmpScripts/isleep2m/log.txt 2>&1 11774 ? S 0:00 \_ /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script 11776 ? S 0:00 \_ sleep 2m Maybe you need to execute "set +m" to disable this feature after every executed user script?! B) Another solution is to get all child process ids by the following recursive function and kill all of them at once: list_descendants () { local children=$(ps -o pid= --ppid "$1") for childpid in $children do list_descendants "$childpid" done echo "$children" } kill $(list_descendants $pid) Edited August 5, 20205 yr by mgutt
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.