[Script] Mover Status (posts Unraid Mover status to Discord and Telegram webhooks)

[couzin2000]

So far I've been able to insert the new script with my credentials. I left it at 25% increments. 
Would be a teensy tiny bit interesting to have a "Mover starting (0% completed)" message... just so we know it's actually started.

 

I'll let you know once I get to 25% completed.

[couzin2000]

No dice. But you're right, we're getting more info now from the log. Here's what happened:
 

Sending notification...
jq: error: $TELEGRAM_CHAT_ID is not defined at , line 1:
{chat_id: $TELEGRAM_CHAT_ID, text: $text, disable_notification: "false", parse_mode: "HTML"}
jq: 1 compile error
Notification sent to Telegram.

 

I'm trying to read the corresponding lines, I can see the lines but not finding the error. My Chat ID was copied and pasted from the last script, so it's the right one.

Any ideas?

[engels74]

Yeah the script definitely needs some more work / fixing 😁. It also looks malformed in Discord.

I thought I had made it to post a beginning message, at 0%, like before, but that also doesn't work properly.

I might have some time later today or tomorrow, to look at it again.

[engels74]

On 5/11/2024 at 4:29 AM, couzin2000 said:

No dice. But you're right, we're getting more info now from the log. Here's what happened:
 

Sending notification...
jq: error: $TELEGRAM_CHAT_ID is not defined at , line 1:
{chat_id: $TELEGRAM_CHAT_ID, text: $text, disable_notification: "false", parse_mode: "HTML"}
jq: 1 compile error
Notification sent to Telegram.

 

I'm trying to read the corresponding lines, I can see the lines but not finding the error. My Chat ID was copied and pasted from the last script, so it's the right one.

Any ideas?

 

Okay, so I did a lot of work on this one :). Hopefully it'll work now.

 

Changes:

- I added a DRY_RUN option, it's set to "false" by default. If you set it to true, it'll try and post to the webhooks, to see if that part works, and exit the script. It'll also do the calculations for the /mnt/cache, but everything else it just bogus data

- I added a cooldown for the echo'ing to the logs for 10 seconds, since it would spam up the log, when it didn't find the Mover process. So now it'll look for the Mover process each 60 seconds, instead of each mili-second. I think that's.. Fine. There's probably a better way to do this, but for now, it works fine without spamming up the log too much. You should probably clear the log file at some point, as it will probably fill up the log over time. Again... I'll try and figure something out.

- I think I fixed the previous issues, but I haven't had a "real" try with the new version yet. So hopefully it'll now start at 0%, like it's supposed to.

- Probably fixed some other stuff, but I really hope it'll work now. I'm leaving mine overnight and see what happens in the morning.

 

Link to script:

https://github.com/engels74/mover-status/blob/feat-combinedVersion/moverStatus-combined.sh

 

I made it into a separate branch on the GitHub.

 

Current issues:

- If USE_DISCORD and USE_TELEGRAM are both set to true, it'll only post the first webhook to Telegram, and not any more. Meanwhile the Discord webhook(s) will keep being posted?? I added some debugging in the latest version on the GitHub repo. Hopefully that'll show something.

 

Other notes: (these are mainly for myself to remember)

- Right now, the script should not be setup with cron. The user scripts plugin will re-launch it at the cron interval, overwrite the log with the "script is already running" error message. At least that's what I noticed. But that was also with the "clearLog" variable set to "true", so that might've something to do with it.

Edited May 12 by engels74
GitHub repo link, alpha branch

[couzin2000]

Great work amigo. For the moment, I'm getting response from DRY_RUN and from running the standard script while Mover is running. So far so good. I left INCREMENT at 25 to see how well that part works, and I did get a notification at 0% which is great. You could package and sell this with Unraid, it's just that excellent.

Keep you posted on the whole move!

[couzin2000]

Completed without a hitch. Great work!
If you can get this script constantly running, it'll be incredible.

 

EDIT: Whoa snap! It DOES work on a loop!
Congrats! I'm sold!

Edited May 13 by couzin2000
new result

[couzin2000]

Ok, one bug; once you've brought the remaining percentage down to zero, the loop doesn't seem to reset. I do get a 0% notification and a Completed notification, but nothing in betwen. For that to happen, I can stop the script and run it again. Then the increments will be notified. 

[engels74]

Yeah, that's what I'm seeing as well. I'll add some more debugging, and then hopefully get some kind of useful "response" from Telegrams API. I'm guessing something's being sent malformed.

 

Especially having "USE_DISCORD" and "USE_TELEGRAM" set to true at the same time, seems to cause some weird behaviour. On my Discord server I got an odd amount of updates, nothing really consistent, and on my Telegram channel I only received the initial message, and the last completion one.

 

I'll push an update to the script soon, with some more debugging, and then I can hopefully do something with whatever response I get from Telegram and/or Discord.

 

Edit:

v0.0.5-alpha10: So I added some debugging logging all over the script, hopefully that'll show where / how it fails.

I also tried adjusting the "LAST_NOTIFICATION" logic a bit, not sure if it'll be fruitful or a failure. I've had a rough night with little sleep, but here's to hoping

v0.0.5-alpha11: Added even more debugging / logging, since I can't really reproduce why or when it's happening . I'll let it run overnight again, and take a look at the log tomorrow. Don't have too much time this week to work on it sadly.

v0.0.5-alpha14: I think we're getting there. Something was malformed being sent to both Discord and Telegram, if both were enabled. I tried to mitigate this. Also made the logging a bit smarter.. I hope. The percent increments should be more accurate now as well (again, hopefully).

v0.0.5-alpha15: The alpha14 didn't solve the issue completely, tried another approach. Also introduced a new variable, that should help the percentage calculation being more accurate. Man, I'm hoping it'll work now lol. 

Edited May 17 by engels74
v0.0.5-alpha15

[engels74]

On 5/13/2024 at 7:22 PM, couzin2000 said:

Ok, one bug; once you've brought the remaining percentage down to zero, the loop doesn't seem to reset. I do get a 0% notification and a Completed notification, but nothing in betwen. For that to happen, I can stop the script and run it again. Then the increments will be notified. 

 

Hey mate, I think the current version (v0.0.5-alpha16) is in a pretty good state:

https://github.com/engels74/mover-status/blob/feat-combinedVersion/moverStatus-combined.sh

 

I did a bunch of work on it, but it should work as intended now.

 

Can I get you to test it out, see how it works out in your end? It still has some debugging stuff enabled, I'll probably remove that for the proper "release".

 

I still haven't tested it for multiple consecutive runs (letting it run multiple loops), but hopefully that works as well.

 

Edit:

The loop seems to work. I've yet to test:

- If the script works, if only "USE_DISCORD" or "USE_TELEGRAM" is enabled

- What will happen with the log in Unraid's User Scripts plugin, if you try and run the script, while it's already running. A few versions prior, it removed the entire previous log.

Edited May 22 by engels74

[couzin2000]

Hi there,
Sorry for the delays, I've been rather busy. 
I've installed the script in its final state on my Unraid 6.12.10 about 3-4 days ago. So far, the script works exactly as intended on multiple consecutive file transfers! Outstanding work my friend, thank you! You should definitely talk about this script and make it go wide - one of the better more useful features I'd been asking for on Unraid. 
Thanks again!

[engels74]

I finally got to a proper release of the v0.0.5 of this script. A lot of work went into it.

 

If you are upgrading to v0.0.5 from v0.0.4, I recommend re-reading the installation guide on GitHub, as the script no longer supports the "cron" scheduling in the User Scripts plugin.

 

The full changelog can be found here:
https://github.com/engels74/mover-status/releases/tag/0.0.5

[engels74]

6 hours ago, couzin2000 said:

Hi there,
Sorry for the delays, I've been rather busy. 
I've installed the script in its final state on my Unraid 6.12.10 about 3-4 days ago. So far, the script works exactly as intended on multiple consecutive file transfers! Outstanding work my friend, thank you! You should definitely talk about this script and make it go wide - one of the better more useful features I'd been asking for on Unraid. 
Thanks again!

 

Very happy that it works on your end ! Spent so many hours on it, so much trial and error haha. I have a fairly basic understanding of bash, and that's it.

 

I would be glad to spread the word, but I wouldn't know where to start honestly 

Edited May 27 by engels74

[nebb00]

would love pushover/gotify or just apprise support for this

[Healzangels]

Really enjoying the script! Thanks for sharing with the community.

 

I was curious would there be any harm in setting the set the run to "At Startup of Array" to allow for it to be running after a system reboot?

Or would this not work since it needs to be running in the background?

 

Ideally would love to not have to remember to go in and start the script after restarting the system.

 

Thanks - Cheers! 

[engels74]

On 7/1/2024 at 9:19 AM, nebb00 said:

would love pushover/gotify or just apprise support for this

 

I'm not adding more providers right now. I would love Apprise, Gotify and all those as well, but currently I'm not happy with how the bash script is getting more and more "complex", with each provider having it's own way of formatting, and the script having to account for that. I've made a more lengthy answer here.

Short answer is that I'm probably going to restructure this Mover Status project into a more fully fledged, modular Python project. Hopefully I can make it work. But it'll be a while, before I get to it.

I'll gladly accept PRs

  

On 7/10/2024 at 2:11 AM, Healzangels said:

Really enjoying the script! Thanks for sharing with the community.

 

I was curious would there be any harm in setting the set the run to "At Startup of Array" to allow for it to be running after a system reboot?

Or would this not work since it needs to be running in the background?

 

Ideally would love to not have to remember to go in and start the script after restarting the system.

 

Thanks - Cheers! 

 

Are you referring to the "arrayStarted=true" variable in the User Scripts plugin? If so, then it's not how that works unfortunately.

 

The "arrayStarted" variable only creates a "check" for the User Scripts plugin, to only allow the script to be started, if the array has been started. It won't start automatically.

If you're referring to anything else, please tell me :). I mean, it should work fine, as long as it's the User Scripts plug-in that starts the script, as the plugin is the one controlling the lockfile, and the status of whether or not the script is running.

 

Sadly I've yet to find a way to circumvent or work with the User Scripts plugin's lockfile. I have tried explaining it on the GitHub page, on why I can't make it work. Trust me, I think it's annoying as well.

 

My current long-term plan is to move from a bash script to a more fully fledged Python project, that will work as a Docker image for Unraid. That should hopefully work a lot better, if I can get it working. But that will be a fair while before I have time and energy to do proper work on that sadly.

 

Edit:

Oh wait, I see what you mean! I haven't seen that option before lol, I'm apparently blind 😎. For anyone else, that is as blind as me, you can find the option here.

That could work, hopefully. I haven't tested it,. I'm going to apply that setting, and I'll check next time I reboot my system.

It can still be the case of User Scripts plugin losing track of whether or not the script is running. To check yourself, you can always do this command in the Unraid console:
 

ps aux | grep moverStatus

 

The "moverStatus" should be replaced with the name of your script in the User Scripts plugin. The output should be something like this, if it is running:
 

root@Tower:~# ps aux | grep moverStatus
root      5763  0.0  0.0   4052  2268 pts/0    S+   22:45   0:00 grep moverStatus
root     19454  0.0  0.0  95616 29964 ?        SL   Jul17   0:00 /usr/bin/php /usr/local/emhttp/plugins/user.scripts/startBackground.php /tmp/user.scripts/tmpScripts/moverStatus/script
root     19455  0.0  0.0   3980  2976 ?        S    Jul17   0:00 sh -c /tmp/user.scripts/tmpScripts/moverStatus/script  >> /tmp/user.scripts/tmpScripts/moverStatus/log.txt 2>&1
root     19456  0.0  0.0   4188  3360 ?        S    Jul17   1:48 /bin/bash /tmp/user.scripts/tmpScripts/moverStatus/script

 

If it only shows this:

root@Tower:~# ps aux | grep moverStatus
root      5763  0.0  0.0   4052  2268 pts/0    S+   22:45   0:00 grep moverStatus

 

Then the script is not running.

 

Again, I'll test it next time I reboot my system and update this post. Thanks for making me aware of the option :). Here's hoping it works!

Edited July 29 by engels74

[Healzangels]

Awesome, great to hear as well!

 

I'll adjust and report back next time I reboot my system (should be this weekend as have some maintenance planned!)

 

Thanks a lot.

[engels74]

21 hours ago, Healzangels said:

Awesome, great to hear as well!

 

I'll adjust and report back next time I reboot my system (should be this weekend as have some maintenance planned!)

 

Thanks a lot.

 

Hello again, I had to reboot my system today to update the BIOS, so I got to test it - it works great with that option. It starts as it should, and the User Scripts plugin does detect that it's running. I'll probably update the GitHub's page readme to include this option

 

Thank you for showing that little option. It makes the script easier to utilize, now that it's no longer required to manually launch it! 😁

Edited July 31 by engels74