[Plugin] CA User Scripts


Recommended Posts

This would make the scripts to complicate and it's not easy to use / share them on multiple platforms.

 

EDIT: Ok, found a solution that is more portable and works for all unraid scripts:

exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1

This needs to be put in the first line after "#!/bin/bash" and now all command returns and echo's are logged!

 

Other versions I tested that did not work (only for information purposes):

# does not work
#exec >> /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt
#exec 3>&1 1>>/tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt 2>&1
#exec > >(tee --append --ignore-interrupts /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt)
#exec 1>&1
#exec 2>&1

 

Example script:

#!/bin/bash

# logging
exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1

# rclone
rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -v --stats 10s

 

Log output:

570562029_2020-07-2710_17_09.thumb.png.3fd4716193f47473b5d219567650d310.png

Edited by mgutt
Link to comment
22 hours ago, mgutt said:

This would make the scripts to complicate and it's not easy to use / share them on multiple platforms.

 

EDIT: Ok, found a solution that is more portable and works for all unraid scripts:


exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1

This needs to be put in the first line after "#!/bin/bash" and now all command returns and echo's are logged!

 

Other versions I tested that did not work (only for information purposes):


# does not work
#exec >> /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt
#exec 3>&1 1>>/tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt 2>&1
#exec > >(tee --append --ignore-interrupts /tmp/user.scripts/tmpScripts/backup_shares_owncube/log.txt)
#exec 1>&1
#exec 2>&1

 

Example script:


#!/bin/bash

# logging
exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1

# rclone
rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -v --stats 10s

 

Log output:

570562029_2020-07-2710_17_09.thumb.png.3fd4716193f47473b5d219567650d310.png

It's a bit more heavyweight solution, but you can always write your own wrapper that does logging, timeouts, interlocking, and notifications. Then you can run all of your crons using this wrapper.

Link to comment

I need to create a script that will look through a file to find a specific text and replace all the matches with other text. How can I do this?

 

lets say the file is /mnt/user/text/mytestfile.txt

I am looking for #FF8C2F and want to replace them all with #42ADFA.

 

This is something I need to run after every reboot.

Link to comment
6 minutes ago, almulder said:

I need to create a script that will look through a file to find a specific text and replace all the matches with other text. How can I do this?

 

lets say the file is /mnt/user/text/mytestfile.txt

I am looking for #FF8C2F and want to replace them all with #42ADFA.

 

This is something I need to run after every reboot.

You could write a small Perl script to do it. Maybe something like this:

 

#!/usr/bin/env perl

use strict;
use warnings;
use Carp;
use English;

# Read the file
my $file = "/mnt/user/text/mytestfile.txt";
my $fh;
open $fh, "<", $file
  or croak("\nERROR: Cannot open '$file' for reading: $OS_ERROR\n\n");
my @rawdata = <$fh>;
close $fh;

# Update each line
my @newdata;
foreach my $line (@rawdata)
{
  $line =~ s/#FF8C2F/#42ADFA/g;
  push @newdata, $line;
}

# Update the file with the updated data
open $fh, ">", $file
  or croak("\nERROR: Cannot open '$file' for writing: $OS_ERROR\n\n");
foreach my $line (@newdata)
{
  print $fh $line;
}
close $fh;

 

Link to comment
25 minutes ago, Phoenix Down said:

You could write a small Perl script to do it. Maybe something like this:

 


#!/usr/bin/env perl

use strict;
use warnings;
use Carp;
use English;

# Read the file
my $file = "/mnt/user/text/mytestfile.txt";
my $fh;
open $fh, "<", $file
  or croak("\nERROR: Cannot open '$file' for reading: $OS_ERROR\n\n");
my @rawdata = <$fh>;
close $fh;

# Update each line
my @newdata;
foreach my $line (@rawdata)
{
  $line =~ s/#FF8C2F/#42ADFA/g;
  push @newdata, $line;
}

# Update the file with the updated data
open $fh, ">", $file
  or croak("\nERROR: Cannot open '$file' for writing: $OS_ERROR\n\n");
foreach my $line (@newdata)
{
  print $fh $line;
}
close $fh;

 

Thanks for your help. Thats more complicated than I wanted. However if made me dig deeper and not just search for unraid but linux code and found out I could do this all in 1 line and it works great.

 

sed -i 's/#FF8C2F/#42ADFA/gI' /mnt/user/text/mytestfile.txt

 

 

Also found a way to use variables instead:

old_color="#e22828"
new_color="#00378F"
sed -i "s/$old_color/$new_color/gI" /mnt/user/Unraid_Backup/Newlogos/hello.txt

Hope this helps other one day!

 

Link to comment
17 hours ago, Phoenix Down said:

You could write a small Perl script to do it. Maybe something like this:

 


#!/usr/bin/env perl

use strict;
use warnings;
use Carp;
use English;

# Read the file
my $file = "/mnt/user/text/mytestfile.txt";
my $fh;
open $fh, "<", $file
  or croak("\nERROR: Cannot open '$file' for reading: $OS_ERROR\n\n");
my @rawdata = <$fh>;
close $fh;

# Update each line
my @newdata;
foreach my $line (@rawdata)
{
  $line =~ s/#FF8C2F/#42ADFA/g;
  push @newdata, $line;
}

# Update the file with the updated data
open $fh, ">", $file
  or croak("\nERROR: Cannot open '$file' for writing: $OS_ERROR\n\n");
foreach my $line (@newdata)
{
  print $fh $line;
}
close $fh;

 

 

You can achieve the same thing from the command line with PERL:

perl -pi -e 's/#FF8C2F/#42ADFA/g' /mnt/user/text/mytestfile.txt

Perl one-liners

Edited by JoeUnraidUser
  • Like 1
Link to comment

having issues  with ssh keys 

i have it setup so when unraid boots it copys the ssh folder and does the keyscan stuff and chkmod  so i can do rsync without a password  and that works.. for a while.. it seems like it times out after 24 hours  it releases and now rsync stops  and public is released out of memory..

as if you drop to the terminal and ssh  <servername>  u gotta type yes no  thing  again..  if i reboot the server running the script to access the other server..  i can ssh  <server>  and not have to enter a password

 

my question is how do you keep it from loosing like a network dhcp lease...  as i dont wanna always reboot  server just to rsync to refresh the public key

 

so say  after 1000 minutes unraid release the public key leases... is there a way to permently keep it in

Link to comment

Although my last test returned the opposite, I found out that CA user scripts is not race condition safe. So I changed my script as follows:

#!/bin/bash

# kill all commands on exit
function exitus() {
    exit_status=$1
    sudo pkill -f rcloneorig
    sleep 1
    sudo pkill -f rcloneorig
    sleep 1
    sudo pkill -f rcloneorig
    sleep 1
    sudo pkill -f rcloneorig
    sleep 1
    rmdir "/tmp/${0///}"
    exit $exit_status
}

# make script race condition safe
if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then
    exit 1 # do not use exitus here or it will kill the already running script!
fi
trap 'exitus 1' EXIT

# logging
exec > >(tee --append --ignore-interrupts $(dirname ${0})/log.txt) 2>&1

# rclone
rclone copy /mnt/user/software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d
rclone copy /mnt/user/music owncube:music --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d
rclone copy /mnt/user/movie owncube:movie --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d
rclone copy /mnt/user/tv owncube:tv --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s --min-age 3d

 

Then I pressed "Abort Script", waited 10 seconds and checked if my tmp dir is still existing and it does:

293622279_2020-07-3114_02_46.png.98ef9bbd72c2044ca1e05d3060d59b00.png

 

A short check with top confirmed that the rlconeorig process is still active.

 

So I killed rclone as follows:

2078881803_2020-07-3114_04_39.png.9cf890a10bba7bf490e92317a426bba9.png

 

As you can see I needed to kill every rclone command separately and you can see that "rmdir" of my script has been executed as my atomic dir "tmpuser.scriptstmpScriptsbackup_shares_owncubescript" is gone.

 

Conclusion: The "Abort Script" button does nothing except removing the "Running"-status in the webclient. The script itself runs until the end.

 

Edited by mgutt
Link to comment

Looks like aborting a script does not kill everything.

 

I'm using a user script that 'bashes' a shell script, and that shell script in turn runs a python script:

 


#!/bin/bash
#backgroundOnly=true
#arrayStarted=true
bash /boot/custom/kuroautomode.sh

(I don't know how to use bash to call python directly? So this is how i do it. bash is needed, the .sh and .py reside on the usb stick in a subfolder under boot folder)

 

When this is started, proceslist will show 4 processes:


root 22117 0.0 0.0 104624 26196 ? SL Aug03 0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/kuroAutoMode/script 

root 22118 0.0 0.0 3840 2916 ? S Aug03 0:00 sh -c /tmp/user.scripts/tmpScripts/kuroAutoMode/script >> /tmp/user.scripts/tmpScripts/kuroAutoMode/log.txt 2>&1 

root 22119 0.0 0.0 3840 2808 ? S Aug03 0:00 /bin/bash /tmp/user.scripts/tmpScripts/kuroAutoMode/script
root 22120 0.0 0.0 3840 2916 ? S Aug03 0:00 bash /boot/custom/kuroautomode.sh root 22121 0.0 0.0 18716 14564 ? S Aug03 0:02 /usr/bin/python /boot/custom/kuroautomodecmdline.py

 

Now, when i abort the user script from within unraid, proclist still shows:


root 22119 0.0 0.0 3840 2808 ? S Aug03 0:00 /bin/bash /tmp/user.scripts/tmpScripts/kuroAutoMode/script 

root 22120 0.0 0.0 3840 2916 ? S Aug03 0:00 bash /boot/custom/kuroautomode.sh 

root 22121 0.0 0.0 18716 14564 ? S Aug03 0:02 /usr/bin/python /boot/custom/kuroautomodecmdline.py

 

So it only aborts the script logging, nothing else. Everything else keeps running. If i restart the user.script, i get everything running 2x, 3x, 4x etc. if i don't manually remove it.

 

Am i doing something wrong or is this an issue?

 

Edited by jowi
Link to comment
On 7/31/2020 at 6:08 AM, mgutt said:

A short check with top confirmed that the rlconeorig process is still active.

 

But was the script itself still running?  It's a royal pain to kill off sub-processes from the main script, especially since the only thing I've got to go on is the name of the main script.  The system does attempt to kill off children (pkill -TREM -P). 

 

If you can come up with a better solution, then please let me know.

  • Thanks 1
Link to comment
2 minutes ago, Squid said:

But was the script itself still running?  It's a royal pain to kill off sub-processes from the main script, especially since the only thing I've got to go on is the name of the main script.  The system does attempt to kill off children (pkill -TREM -P). 

 

If you can come up with a better solution, then please let me know.

What if you crawl the process tree and kill each child process individually? It's a bit heavy handed, but should take care of any orphaned child processes.

Link to comment

All scripts should be well behaved and handle the spawning and reaping of child processes themselves. It is their own responsibility.

 

I suggest serious scripters put together a template using best practices to make it easier for other scripters to not fall into the common pitfalls.

Edited by BRiT
Link to comment
24 minutes ago, Phoenix Down said:

What if you crawl the process tree and kill each child process individually? It's a bit heavy handed, but should take care of any orphaned child processes.

And that's what the -P on the pkill is supposed to do.  Kill off children from the main PID

Link to comment
54 minutes ago, Squid said:

But was the script itself still running?

Yes. I proved it in my post. The script contains 4 rclone commands. After I aborted the script through the GUI it does not kill the running rlcone process and I need to kill every following command separately (the first rclone command was already finished, so I needed to kill only 3):

64947463_2020-07-3114_04_39.png.5bfbb603f8d0f63a4340c360e7d2e300.png

(The "-c" flag returns the amount of processes that were killed)

 

And after I killed the last rclone process it executes the very last line of the script, too (the rmdir command).

 

This proves that the script is still running (and not only one long process).

 

How do you execute the script if the "run in background" button is pressed?

Edited by mgutt
Link to comment

@Squid

I tried many commands and by that I found out that pkill does not kill (all) child processes:

root@Thoth:/tmp# pgrep -f isleep2m | xargs --no-run-if-empty ps fp
  PID TTY      STAT   TIME COMMAND
19608 ?        SL     0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/isleep2m/script
19609 ?        S      0:00  \_ sh -c /tmp/user.scripts/tmpScripts/isleep2m/script  >> /tmp/user.scripts/tmpScripts/isleep2m/log.txt 2>&1
19610 ?        S      0:00      \_ /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script
root@Thoth:/tmp# pkill -9 -P 19608
root@Thoth:/tmp# pgrep -f isleep2m | xargs --no-run-if-empty ps fp
  PID TTY      STAT   TIME COMMAND
19610 ?        S      0:00 /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script

But I finally found two solutions to repair the "Abort Script" button.:

 

A) Run every script in its own process group. Something similar to this should work:

(set -m;sh -c /tmp/user.scripts/tmpScripts/kllme/script & pid=$!;echo $pid)

Then you are able to get the pgid as follows:

ps opgid= $pid

Now you can kill the complete tree with the pgid

kill -- -$pgid

Note: It is important to execute the user script and all the other commands (like logging) in a new process group (pgid) as at the moment all running user scripts executed through PHP get the same pgid. In my case its "1733" and killing this group would kill all scripts: ("backup_shares" and "isleep2m"):

root@Thoth:/tmp# pgrep -f isleep2m | xargs --no-run-if-empty ps fp
  PID TTY      STAT   TIME COMMAND
11772 ?        SL     0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/isleep2m/script
11773 ?        S      0:00  \_ sh -c /tmp/user.scripts/tmpScripts/isleep2m/script  >> /tmp/user.scripts/tmpScripts/kllme/log.txt 2>&1
11774 ?        S      0:00      \_ /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script
root@Thoth:/tmp# ps opgid= 11772
 1733
root@Thoth:/tmp# pgrep -g 1733 | xargs --no-run-if-empty ps fp
  PID TTY      STAT   TIME COMMAND
 1733 ?        Ss     0:00 /usr/sbin/atd -b 15 -l 1
26844 ?        S      0:00  \_ /usr/sbin/atd -b 15 -l 1
26845 ?        S      0:00  |   \_ sh
26846 ?        SL   346:12  |       \_ /usr/bin/php /usr/local/emhttp/plugins/fix.common.problems/scripts/extendedTest.php
23911 ?        S      0:00  \_ /usr/sbin/atd -b 15 -l 1
23912 ?        S      0:00  |   \_ sh
23913 ?        SL     0:00  |       \_ /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/backup_shares/script
23914 ?        S      0:00  |           \_ sh -c /tmp/user.scripts/tmpScripts/backup_shares/script  >> /tmp/user.scripts/tmpScripts/backup_shares/log.txt 2>&1
23915 ?        S      0:00  |               \_ /bin/bash /tmp/user.scripts/tmpScripts/backup_shares/script
23917 ?        S      0:00  |                   \_ /bin/bash /tmp/user.scripts/tmpScripts/backup_shares/script
23925 ?        S      0:00  |                   |   \_ tee --append --ignore-interrupts /tmp/user.scripts/tmpScripts/backup_shares/log.txt
 2752 ?        S      0:00  |                   \_ /bin/bash /usr/sbin/rclone sync /mnt/user/sharedir nextcloud:sharedir --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers
 2753 ?        Sl    10:48  |                       \_ rcloneorig --config /boot/config/plugins/rclone/.rclone.conf sync /mnt/user/sharedir nextcloud:sharedir --create-empty-src-dirs --i
11770 ?        S      0:00  \_ /usr/sbin/atd -b 15 -l 1
11771 ?        S      0:00      \_ sh
11772 ?        SL     0:00          \_ /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/isleep2m/script
11773 ?        S      0:00              \_ sh -c /tmp/user.scripts/tmpScripts/isleep2m/script  >> /tmp/user.scripts/tmpScripts/isleep2m/log.txt 2>&1
11774 ?        S      0:00                  \_ /bin/bash /tmp/user.scripts/tmpScripts/isleep2m/script
11776 ?        S      0:00                      \_ sleep 2m

Maybe you need to execute "set +m" to disable this feature after every executed user script?!

 

B) Another solution is to get all child process ids by the following recursive function and kill all of them at once:

list_descendants ()
{
  local children=$(ps -o pid= --ppid "$1")
  for childpid in $children
  do
    list_descendants "$childpid"
  done
  echo "$children"
}
kill $(list_descendants $pid)

 

Edited by mgutt
  • Like 2
  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.