[Script] Mover Status (posts Unraid Mover status to Discord and Telegram webhooks)

[engels74]

 

Hello!

 

I created this script (moverStatus.sh), that'll post how far along the Mover is, from moving from Cache --> Array.

Currently it supports Discord and Telegram. 

 

Description

The Bash script is designed to monitor the progress of the UnraidOS Mover utility.

 

The script checks the status of the mover process and calculates the amount of data left to move. It then formats this information into a readable format, including the percentage of the task completed and an estimated time of completion.

 

Notifications about the mover's progress are periodically sent to a Discord/Telegram channel via a webhook. These notifications include detailed progress updates and an estimated completion time, helping users track the mover process effectively. The script supports customization for notification frequency and allows users to specify directory paths to exclude from the data transfer calculation.

 

The purpose of the script is for users who want real-time updates about the Mover, can easily follow the progress on Discord and/or Telegram.

 

Image preview:

 

Installation + instructions:

The GitHub link is here:

https://github.com/engels74/mover-status

 

Kind regards,

Engels74

Edited Monday at 06:33 PM by engels74
v0.0.5

[engels74]

Changelog (YYYY-MM-DD)

 

2024-05-30 - v0.0.7

- Made the version check a bit smarter. Now checks on every loop reset, instead of only on first launch.

 

2024-05-28 - v0.0.6:

- Slight change to the copyright notice at the bottom of the script. Now includes a link to the GitHub repository.

 

2024-05-27 - v0.0.5:

Script was mostly revamped for this version. A lot of work went into it.

- Script now runs in a loop. This means cron/scheduling is no longer supported.

- Added Telegram support

- Calculations should be more accurate

- Instead of posting per "n" second, it now posts per set percentage via variable

- And a lot, lot more.. Read the rest of the changelog on GitHub.

 

2024-05-09 - v0.0.4:

- Added a small amount of error handling. The script will check if the Discord Webhook is set properly, and if the webhook responds with a HTTP 200 (OK), else it'll log the error.

- Edited the way the variables are set, so it should be easier to figure out

- Made the EXCLUSION_PATH_XX variables empty by default

 

2024-05-08 - v0.0.3:

Now you can change the message it posts with variables:

- Added MOVING_MESSAGE and COMPLETION_MESSAGE variables

 

 

2024-05-07 - v0.0.2:

- Fixed the script, in terms of calculating the right amount of data in the "Remaining Data" bit. In the previous version, only the first excluded path actually got included. Now it should work properly. Hopefully .

- Added a footer text, which reports the current version of the script.

- Added a very simple version check function. It'll compare the current version of the script, to the latest release on the GitHub repo. If there's an update, it'll be included in the footer text, e.g.: "Version: v0.0.1 (update available)".

- The version check function is only included from v0.0.2 and beyond. It won't work with the first version of the script, sorry.

- Changed the colours a bit. Previously it was "Light Red -> Light Blue -> Light Green". Now it's "Light Red -> Light Orange -> Light Green". 

 

 

GH Repo Link: 

- https://github.com/engels74/mover-status

Edited Thursday at 06:04 PM by engels74
v0.0.7

[couzin2000]

Any way this can be adapted to work with Telegram? Would be a TOTAL treat for me!

[engels74]

16 hours ago, couzin2000 said:

Any way this can be adapted to work with Telegram? Would be a TOTAL treat for me!

 

I have made something, but I haven't tested it, as I don't really use Telegram. I'll probably test it later myself.

I made a few changes, since the Unix timestamp thing that I use for Discord, is not supported in Telegram. So it posts that part a bit differently.

 

In the meantime, you can try using this:
https://logs.notifiarr.com/?cb3a4af59222b9e5#g8oTX2JN7Co9aBGjbNJsEZLWfDyAEHx3evnCcPuegVZ

 

A word of caution, it might not work at all. I haven't worked with Telegram webhooks/API before. If it does work for you, please report back in here, and I'll add it properly to the Github repository.

[couzin2000]

I saved this as "Mover Notifications", pasted the script, ran it once, and ran it in background. I check the log, here's what pops:
 

Script Starting May 09, 2024 13:01.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt

Error: Exclusion path /path/to/excluded/folder does not exist.
Script Finished May 09, 2024 13:01.02

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt

Any ideas? I don't believe I have "excluded" folders, but in theory Mover is still active as we speak.

[couzin2000]

Oh wait, did I have to comment these out if I don't have exclusion paths to enter?

 

[engels74]

1 hour ago, couzin2000 said:

I saved this as "Mover Notifications", pasted the script, ran it once, and ran it in background. I check the log, here's what pops:
 

Script Starting May 09, 2024 13:01.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt

Error: Exclusion path /path/to/excluded/folder does not exist.
Script Finished May 09, 2024 13:01.02

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt

Any ideas? I don't believe I have "excluded" folders, but in theory Mover is still active as we speak.

 

You should remove the "/path/to/excluded/folder" manually from the script, if you don't have any folders. Otherwise the script will fail / exit.

So it should say EXCLUDE_PATH_01="" without anything between the quotation marks.

 

You should be able to just restart the moverStatus script, and it should detect the Unraid mover process is running, and go from there.

 

There is not much error logging in the script, so let's see what happens 😁

I'll edit the next version, so the paths are empty by default.

 

Edited May 9 by engels74

[couzin2000]

OK. Because I was running Mover when I answered, I copied new files onto my cache and invoked Mover manually. So far, no notifications. I do have Telegram setup to give me notifications for everything. Perhaps I missed something in the script and I have some modifications to do?
The logs don't really give me anything in terms of errors. 

[engels74]

I'll probably need to do some testing myself, and set up a Telegram Bot/Channel.. Maybe I'll have time during this weekend. Sorry it didn't work out of the box.

[couzin2000]

Perhaps has to do with the curl command being not properly formatted for Telegram? 
Here's what I find online:
https://stackoverflow.com/questions/68213124/telegram-example-of-how-to-use-curl-for-windows-10-to-message-myself-with-a-bot

 

curl -s -o /dev/null -X POST -H "Content-Type: application/json" -d "{\"chat_id\": \"-100XXXXXXXXXX\", \"text\": \"test_message\", \"disable_notification\": false}" https://api.telegram.org/bot{API_TOKEN}/sendMessage

 

[engels74]

23 minutes ago, couzin2000 said:

Perhaps has to do with the curl command being not properly formatted for Telegram? 
Here's what I find online:
https://stackoverflow.com/questions/68213124/telegram-example-of-how-to-use-curl-for-windows-10-to-message-myself-with-a-bot

 

curl -s -o /dev/null -X POST -H "Content-Type: application/json" -d "{\"chat_id\": \"-100XXXXXXXXXX\", \"text\": \"test_message\", \"disable_notification\": false}" https://api.telegram.org/bot{API_TOKEN}/sendMessage

 

 

Well, you can try this modified version:

https://logs.notifiarr.com/?be6900374c58312c#F2ZesyL12fuvWpwAEczh8yxBkknWjP7joj8H98Ds99hK

 

Not sure if it'll work out, it's a quite quick and dirty version

Edited May 9 by engels74

[couzin2000]

Well, now at least it seems to be trying to send out the notification, but I get this:
 

Warning: Failed to send notification to Telegram, HTTP status 404.

I can see the script works though, because I AM getting 4 messages back to back like this. So it's just a matter of sending out the msg.

Edited May 9 by couzin2000

[engels74]

Definitely shows that there's something not working. It's probably sending the message, but Telegram responds with a 404. Could be a wide range of things.. Like the wrong bot token input to the script, to the payload being delivered via the webhook is still wrongly formatted, or something else entirely.

 

You can try executing this script, to see if the bot token is valid:
https://logs.notifiarr.com/?14191f2c8033ada0#7RLNfxFKe1XPs8KQBSCCoapci7UZtBD1oeVSX5x6SBgH

 

It should give you some kind of response, with some info of the Telegram Bot. If it does respond with basically anything other than an error, it's probably the JSON payload in the script that's acting up, or the curl thing still not working properly.

 

I would test all this myself, but I can't really at the moment, sorry. I'll try and get something done in the upcoming weekend :). Thanks for helping out though.

Edited May 9 by engels74

[couzin2000]

Result:
 

Response from Telegram API:
{"ok":false,"error_code":404,"description":"Not Found"}

Again, just to be sure, I AM getting Telegram notifications from Unraid. Just not getting these ones. 
No worries about the testing, glad to be of help. I know many who would want this.

Wait, do I have to insert my bot token in the script? Or does the script get it for me?

Edited May 9 by couzin2000

[engels74]

2 minutes ago, couzin2000 said:

Result:
 

Response from Telegram API:
{"ok":false,"error_code":404,"description":"Not Found"}

Again, just to be sure, I AM getting Telegram notifications from Unraid. Just not getting these ones. 
No worries about the testing, glad to be of help. I know many who would want this.

Wait, do I have to insert my bot token in the script? Or does the script get it for me?

 

You need to insert your bot token into that script yeah, I don't know yours .

Also yeah, it might not be your fault that the script is not working btw, I could've easily messed something up. 

Edited May 9 by engels74

[couzin2000]

ok - never mind.
Here's the results when I inserted mytoken in the script:

Response from Telegram API:
{"ok":true,"result":{"id":6*********,"is_bot":true,"first_name":"s******","username":"s*******","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false,"can_connect_to_business":false}}
DONE

 

Edited May 9 by couzin2000
for privacy

[engels74]

18 minutes ago, couzin2000 said:

ok - never mind.
Here's the results when I inserted mytoken in the script:

Response from Telegram API:
{"ok":true,"result":{"id":6031589011,"is_bot":true,"first_name":"sebunraid1","username":"sebunraid1_bot","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false,"can_connect_to_business":false}}
DONE

 

 

So at least the bot token is working.

 

And you've put both the bot token and the channel id into the script from above, right? And the bot is invited into the Telegram chat-id/channel and/or have access to the Telegram channel/chat-id? (sorry if the terminology is off, I'm not that experienced with Telegram)

 

 

Edited May 9 by engels74

[couzin2000]

I used the existing bot I use with Unraid. The token is all I needed to add in your script, the response gave me the channel ID. So in theory, I didnt need to "reinvite" my machine into the bot channel, as it's already there. 
I look into all this again, but you can take what I just gave you to the bank, I think.

[engels74]

1 hour ago, couzin2000 said:

I used the existing bot I use with Unraid. The token is all I needed to add in your script, the response gave me the channel ID. So in theory, I didnt need to "reinvite" my machine into the bot channel, as it's already there. 
I look into all this again, but you can take what I just gave you to the bank, I think.

 

Okay, so I end up at least making .. something .. work.

 

I made a bot, using the BotFather on Telegram. I created a simple channel, invited the bot as an administrator, gave it the default rights (post/delete stuff).

I copied the bot id, that BotFather gave me

I found the channel id, using web.telegram.org: simple jpg of how to or stackoverflow comment

 

I made a "dummy" version of the script, that basically just tests the same method as before - except the markdown didn't work, so I made a simple html version instead:

https://logs.notifiarr.com/?74dfa0fae8d7b2af#D6Y8i4VhoC8PTP9tBFvgQzz9tfG3FNNW7DKJHyEyZinu

 

I used the bot id and the channel id in that script, and that seems to work:

 

Bear in mind it's just a dummy script with no real data, but the posting itself seems to work, which is promising . Stil not sure how it'll deal with the formatting of the real thing.

 

 

Edit:

I tried updating the script, with the updated code, that uses the same HTML formatting, as the dummy script:
https://logs.notifiarr.com/?df0d69f2afe05113#GXhzEptFEweKj7Q3Jm6ndX4oVA3CMaiK6waqCeR2oFAG

 

But I have not tested it :). I would say it should work, but at this point, I should probably say that maybe it could work 

Edited May 9 by engels74

[couzin2000]

You did it!


Works perfectly fine! Bravo!

[engels74]

8 minutes ago, couzin2000 said:

You did it!


Works perfectly fine! Bravo!

 

Awesome ! Does the "Estimated completion time" work? Does it actually update the timestamp for each post?

I'll probably do a proper release tomorrow on GitHub.

Edited May 9 by engels74

[couzin2000]

Yes it does. However, I view this as a bit of an annoyance, because it's not on a parametered setting. 
Think there's a way to get updates every 25%?

Also, and this may just be me not knowing how to setup user scripts to permanently run, after Mover stopped, the script stopped. The next usage of Mover did not send any notifications. Which I would totally prefer on a forever basis... as long as there's a way to set notifications every x %. Otherwise, this is actually draining my Apple Watch battery!

[engels74]

9 hours ago, couzin2000 said:

Yes it does. However, I view this as a bit of an annoyance, because it's not on a parametered setting. 
Think there's a way to get updates every 25%?

Also, and this may just be me not knowing how to setup user scripts to permanently run, after Mover stopped, the script stopped. The next usage of Mover did not send any notifications. Which I would totally prefer on a forever basis... as long as there's a way to set notifications every x %. Otherwise, this is actually draining my Apple Watch battery!

 

I initially made this script for myself, so that's why it might be a bit weird

 

Currently it's just configured to post every X second, set by the "NOTIFICATION_FREQUENCY" variable, and not in percentages. This was my personal preference, as I usually move huge amounts of data (1-2TB+) per mover session, and that it can take a fair while from 0% -> 25% -> 50% and so on, so it made more sense for me, to make it post every X second instead, as the user following along, could have a better sense of things actually "moving".

 

I'm guessing I could make it an option to do either the notification pr second or by percentages. But the bash script will become increasingly complex, with all these options.. Hopefully I can figure something out.

 

The next pain point of yours is that the script exits when it's done. This was also by design, as I only run my mover once every day, at 03:00AM, and it was easy to setup a cron-job to make the script launch a few minutes before, and exit when it's done. I think I should be able to just loop the script continuously. I would probably recommend still keeping a cronjob for the script, since it'll not relaunch by itself, if you have to reboot the Unraid server or whatever.

 

I'll try and work a bit on the script. I have the following goals:
- Combining the Discord and Telegram scripts, and make it togglable if you want to use both Telegram and Discord webhooks or only one of them, e.g. "USE_TELEGRAM=true/false", "USE_DISCORD=true/false"

- Making the script use a loop so it starts anew, instead of exiting, when it has detected either the calculations have reached 100% or Unraid's Mover script has exited

- Having the option to do either a notification per second or by percentages in increments of 25: 0%/25%/50%/75%/100% - or definable by the user (the increments)

- Or.. Either that, or ditch the notification per second option completely, and make it in percentages instead only. Might be the easier approach, in terms of not having the bash script growing increasingly complex.

 

Not sure how much of it I can get working, but I'll give it a shot. Thanks for the feedback

 

Edited May 10 by engels74

[couzin2000]

3 hours ago, engels74 said:

 

I initially made this script for myself, so that's why it might be a bit weird

 

Currently it's just configured to post every X second, set by the "NOTIFICATION_FREQUENCY" variable, and not in percentages. This was my personal preference, as I usually move huge amounts of data (1-2TB+) per mover session, and that it can take a fair while from 0% -> 25% -> 50% and so on, so it made more sense for me, to make it post every X second instead, as the user following along, could have a better sense of things actually "moving".

 

I'm guessing I could make it an option to do either the notification pr second or by percentages. But the bash script will become increasingly complex, with all these options.. Hopefully I can figure something out.

Well, I can also modify the script for myself, but if you're interested in making this available to a wider audience, yes I would think it'd be interesting to add the option. Even switching between "time" or "percentage" would be a great add. 
 

 

3 hours ago, engels74 said:

The next pain point of yours is that the script exits when it's done. This was also by design, as I only run my mover once every day, at 03:00AM, and it was easy to setup a cron-job to make the script launch a few minutes before, and exit when it's done. I think I should be able to just loop the script continuously. I would probably recommend still keeping a cronjob for the script, since it'll not relaunch by itself, if you have to reboot the Unraid server or whatever.

I understand. I'm in the process of transferring over 6TB but my cache drive is only 120GB... so that becomes many MANY Mover jobs 😂But I don't mind restarting it manually. It's just that if you make it a continuously running script, then frequency does need to be adjusted -- I'm getting notifications every 45 seconds or so and my arm is numb from the vibration of my Apple Watch 😂
 

 

3 hours ago, engels74 said:

I'll try and work a bit on the script. I have the following goals:
- Combining the Discord and Telegram scripts, and make it togglable if you want to use both Telegram and Discord webhooks or only one of them, e.g. "USE_TELEGRAM=true/false", "USE_DISCORD=true/false"

- Making the script use a loop so it starts anew, instead of exiting, when it has detected either the calculations have reached 100% or Unraid's Mover script has exited

- Having the option to do either a notification per second or by percentages in increments of 25: 0%/25%/50%/75%/100% - or definable by the user (the increments)

- Or.. Either that, or ditch the notification per second option completely, and make it in percentages instead only. Might be the easier approach, in terms of not having the bash script growing increasingly complex.

 

Not sure how much of it I can get working, but I'll give it a shot. Thanks for the feedback

Great Idea to be able to toggle between Telegram and Discord. If you ever want to make this a bit more interesting, perhaps there is a way to have the system generate its own notification, and that would send it through whichever Webhook is already configured for the user. I've chosen Telegram, but there's at least 10 different options to Unraid notifications... Maybe there's an easier way?

If you need some testing or other feedback, I'm your man. I apologize as I'm not much of a programmer. I tinker, but not much else.
Thank you for your very interesting work! I appreciate your effort!

[engels74]

1 hour ago, couzin2000 said:

[...]
Thank you for your very interesting work! I appreciate your effort!

 

Thanks for the feedback and everything :).

 

I haven't tested this yet, but if you want, you can try the new version out here:

https://logs.notifiarr.com/?9eaa332e3616a02f#FYLHcQpht3cQ9QnLZC7cjnvTtfs6gGhiXbqK3qcQmZ3

 

Again, word of caution, there's a chance it'll not work at all . I literally haven't even tried running it yet. But "on paper", it could work out of the box.

 

There's a few changes, mainly:
1) Discord and Telegram versions are now combined

Combined the Telegram and Discord scripts, hopefully will be easier to maintain one script instead of several different ones.
Both variables are false by default, you should enable at least one of them.

Note: I haven't tested what happens if you enable both of them, if it actually posts to both webhooks. But again, "on paper" it could work.

 

2) Percentage Progress only

No more "notification per second", it's percentages only. I think you've made a great point. The user can always set the percentage increment lower, if they want more frequent updates posted to their webhook. So I think that should satisfy most users (including myself haha)

3) Loop instead of exit

The script will now (hopefully) loop, instead of exiting. It'll continuously monitor when the Unraid Mover runs (with a "sleep 1"), and begin anew when it detects it has started/launched.

 

4) Output to console

Made a few outputs, so the log the User Script plugin creates actually will have something semi-useful information. Like when the script starts, when it detects the mover, when it has completed, and when webhooks have been posted successfully to either Discord or Telegram.

 

Here's hoping it'll work. I might do some testing myself later, but until then, you can have fun trying it out . Made quite a few large changes, so there's a chance I broke something along the way.

 

 

Edit:

To your question regarding making this work with Unraids notification system, instead of its own script entirely.. That might be feasible, though I have absolutely no experience with Unraids plugin system. I'm better / more comfortable just using straight bash.. Not good at much else in terms of programming.

 

Edited May 10 by engels74
Small change to script