engels74 Posted April 27 Share Posted April 27 (edited) Hello! I created this script (moverStatus.sh), that'll post how far along the Mover is, from moving from Cache --> Array. Currently it supports Discord and Telegram. Description The Bash script is designed to monitor the progress of the UnraidOS Mover utility. The script checks the status of the mover process and calculates the amount of data left to move. It then formats this information into a readable format, including the percentage of the task completed and an estimated time of completion. Notifications about the mover's progress are periodically sent to a Discord/Telegram channel via a webhook. These notifications include detailed progress updates and an estimated completion time, helping users track the mover process effectively. The script supports customization for notification frequency and allows users to specify directory paths to exclude from the data transfer calculation. The purpose of the script is for users who want real-time updates about the Mover, can easily follow the progress on Discord and/or Telegram. Image preview: Installation + instructions: The GitHub link is here: https://github.com/engels74/mover-status Kind regards, Engels74 Edited May 27 by engels74 v0.0.5 Quote Link to comment
engels74 Posted May 7 Author Share Posted May 7 (edited) Changelog (YYYY-MM-DD) 2024-05-30 - v0.0.7 - Made the version check a bit smarter. Now checks on every loop reset, instead of only on first launch. 2024-05-28 - v0.0.6: - Slight change to the copyright notice at the bottom of the script. Now includes a link to the GitHub repository. 2024-05-27 - v0.0.5: Script was mostly revamped for this version. A lot of work went into it. - Script now runs in a loop. This means cron/scheduling is no longer supported. - Added Telegram support - Calculations should be more accurate - Instead of posting per "n" second, it now posts per set percentage via variable - And a lot, lot more.. Read the rest of the changelog on GitHub. 2024-05-09 - v0.0.4: - Added a small amount of error handling. The script will check if the Discord Webhook is set properly, and if the webhook responds with a HTTP 200 (OK), else it'll log the error. - Edited the way the variables are set, so it should be easier to figure out - Made the EXCLUSION_PATH_XX variables empty by default 2024-05-08 - v0.0.3: Now you can change the message it posts with variables: - Added MOVING_MESSAGE and COMPLETION_MESSAGE variables 2024-05-07 - v0.0.2: - Fixed the script, in terms of calculating the right amount of data in the "Remaining Data" bit. In the previous version, only the first excluded path actually got included. Now it should work properly. Hopefully . - Added a footer text, which reports the current version of the script. - Added a very simple version check function. It'll compare the current version of the script, to the latest release on the GitHub repo. If there's an update, it'll be included in the footer text, e.g.: "Version: v0.0.1 (update available)". - The version check function is only included from v0.0.2 and beyond. It won't work with the first version of the script, sorry. - Changed the colours a bit. Previously it was "Light Red -> Light Blue -> Light Green". Now it's "Light Red -> Light Orange -> Light Green". GH Repo Link: - https://github.com/engels74/mover-status Edited May 30 by engels74 v0.0.7 Quote Link to comment
couzin2000 Posted May 7 Share Posted May 7 Any way this can be adapted to work with Telegram? Would be a TOTAL treat for me! Quote Link to comment
engels74 Posted May 8 Author Share Posted May 8 16 hours ago, couzin2000 said: Any way this can be adapted to work with Telegram? Would be a TOTAL treat for me! I have made something, but I haven't tested it, as I don't really use Telegram. I'll probably test it later myself. I made a few changes, since the Unix timestamp thing that I use for Discord, is not supported in Telegram. So it posts that part a bit differently. In the meantime, you can try using this: https://logs.notifiarr.com/?cb3a4af59222b9e5#g8oTX2JN7Co9aBGjbNJsEZLWfDyAEHx3evnCcPuegVZ A word of caution, it might not work at all. I haven't worked with Telegram webhooks/API before. If it does work for you, please report back in here, and I'll add it properly to the Github repository. Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 I saved this as "Mover Notifications", pasted the script, ran it once, and ran it in background. I check the log, here's what pops: Script Starting May 09, 2024 13:01.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt Error: Exclusion path /path/to/excluded/folder does not exist. Script Finished May 09, 2024 13:01.02 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt Any ideas? I don't believe I have "excluded" folders, but in theory Mover is still active as we speak. Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 Oh wait, did I have to comment these out if I don't have exclusion paths to enter? Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) 1 hour ago, couzin2000 said: I saved this as "Mover Notifications", pasted the script, ran it once, and ran it in background. I check the log, here's what pops: Script Starting May 09, 2024 13:01.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt Error: Exclusion path /path/to/excluded/folder does not exist. Script Finished May 09, 2024 13:01.02 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Mover Notifications/log.txt Any ideas? I don't believe I have "excluded" folders, but in theory Mover is still active as we speak. You should remove the "/path/to/excluded/folder" manually from the script, if you don't have any folders. Otherwise the script will fail / exit. So it should say EXCLUDE_PATH_01="" without anything between the quotation marks. You should be able to just restart the moverStatus script, and it should detect the Unraid mover process is running, and go from there. There is not much error logging in the script, so let's see what happens 😁 I'll edit the next version, so the paths are empty by default. Edited May 9 by engels74 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 OK. Because I was running Mover when I answered, I copied new files onto my cache and invoked Mover manually. So far, no notifications. I do have Telegram setup to give me notifications for everything. Perhaps I missed something in the script and I have some modifications to do? The logs don't really give me anything in terms of errors. Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 I'll probably need to do some testing myself, and set up a Telegram Bot/Channel.. Maybe I'll have time during this weekend. Sorry it didn't work out of the box. 1 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 Perhaps has to do with the curl command being not properly formatted for Telegram? Here's what I find online: https://stackoverflow.com/questions/68213124/telegram-example-of-how-to-use-curl-for-windows-10-to-message-myself-with-a-bot curl -s -o /dev/null -X POST -H "Content-Type: application/json" -d "{\"chat_id\": \"-100XXXXXXXXXX\", \"text\": \"test_message\", \"disable_notification\": false}" https://api.telegram.org/bot{API_TOKEN}/sendMessage Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) 23 minutes ago, couzin2000 said: Perhaps has to do with the curl command being not properly formatted for Telegram? Here's what I find online: https://stackoverflow.com/questions/68213124/telegram-example-of-how-to-use-curl-for-windows-10-to-message-myself-with-a-bot curl -s -o /dev/null -X POST -H "Content-Type: application/json" -d "{\"chat_id\": \"-100XXXXXXXXXX\", \"text\": \"test_message\", \"disable_notification\": false}" https://api.telegram.org/bot{API_TOKEN}/sendMessage Well, you can try this modified version: https://logs.notifiarr.com/?be6900374c58312c#F2ZesyL12fuvWpwAEczh8yxBkknWjP7joj8H98Ds99hK Not sure if it'll work out, it's a quite quick and dirty version Edited May 9 by engels74 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 (edited) Well, now at least it seems to be trying to send out the notification, but I get this: Warning: Failed to send notification to Telegram, HTTP status 404. I can see the script works though, because I AM getting 4 messages back to back like this. So it's just a matter of sending out the msg. Edited May 9 by couzin2000 Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) Definitely shows that there's something not working. It's probably sending the message, but Telegram responds with a 404. Could be a wide range of things.. Like the wrong bot token input to the script, to the payload being delivered via the webhook is still wrongly formatted, or something else entirely. You can try executing this script, to see if the bot token is valid: https://logs.notifiarr.com/?14191f2c8033ada0#7RLNfxFKe1XPs8KQBSCCoapci7UZtBD1oeVSX5x6SBgH It should give you some kind of response, with some info of the Telegram Bot. If it does respond with basically anything other than an error, it's probably the JSON payload in the script that's acting up, or the curl thing still not working properly. I would test all this myself, but I can't really at the moment, sorry. I'll try and get something done in the upcoming weekend :). Thanks for helping out though. Edited May 9 by engels74 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 (edited) Result: Response from Telegram API: {"ok":false,"error_code":404,"description":"Not Found"} Again, just to be sure, I AM getting Telegram notifications from Unraid. Just not getting these ones. No worries about the testing, glad to be of help. I know many who would want this. Wait, do I have to insert my bot token in the script? Or does the script get it for me? Edited May 9 by couzin2000 Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) 2 minutes ago, couzin2000 said: Result: Response from Telegram API: {"ok":false,"error_code":404,"description":"Not Found"} Again, just to be sure, I AM getting Telegram notifications from Unraid. Just not getting these ones. No worries about the testing, glad to be of help. I know many who would want this. Wait, do I have to insert my bot token in the script? Or does the script get it for me? You need to insert your bot token into that script yeah, I don't know yours . Also yeah, it might not be your fault that the script is not working btw, I could've easily messed something up. Edited May 9 by engels74 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 (edited) ok - never mind. Here's the results when I inserted mytoken in the script: Response from Telegram API: {"ok":true,"result":{"id":6*********,"is_bot":true,"first_name":"s******","username":"s*******","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false,"can_connect_to_business":false}} DONE Edited May 9 by couzin2000 for privacy Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) 18 minutes ago, couzin2000 said: ok - never mind. Here's the results when I inserted mytoken in the script: Response from Telegram API: {"ok":true,"result":{"id":6031589011,"is_bot":true,"first_name":"sebunraid1","username":"sebunraid1_bot","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false,"can_connect_to_business":false}} DONE So at least the bot token is working. And you've put both the bot token and the channel id into the script from above, right? And the bot is invited into the Telegram chat-id/channel and/or have access to the Telegram channel/chat-id? (sorry if the terminology is off, I'm not that experienced with Telegram) Edited May 9 by engels74 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 I used the existing bot I use with Unraid. The token is all I needed to add in your script, the response gave me the channel ID. So in theory, I didnt need to "reinvite" my machine into the bot channel, as it's already there. I look into all this again, but you can take what I just gave you to the bank, I think. 1 Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) 1 hour ago, couzin2000 said: I used the existing bot I use with Unraid. The token is all I needed to add in your script, the response gave me the channel ID. So in theory, I didnt need to "reinvite" my machine into the bot channel, as it's already there. I look into all this again, but you can take what I just gave you to the bank, I think. Okay, so I end up at least making .. something .. work. I made a bot, using the BotFather on Telegram. I created a simple channel, invited the bot as an administrator, gave it the default rights (post/delete stuff). I copied the bot id, that BotFather gave me I found the channel id, using web.telegram.org: simple jpg of how to or stackoverflow comment I made a "dummy" version of the script, that basically just tests the same method as before - except the markdown didn't work, so I made a simple html version instead: https://logs.notifiarr.com/?74dfa0fae8d7b2af#D6Y8i4VhoC8PTP9tBFvgQzz9tfG3FNNW7DKJHyEyZinu I used the bot id and the channel id in that script, and that seems to work: Bear in mind it's just a dummy script with no real data, but the posting itself seems to work, which is promising . Stil not sure how it'll deal with the formatting of the real thing. Edit: I tried updating the script, with the updated code, that uses the same HTML formatting, as the dummy script: https://logs.notifiarr.com/?df0d69f2afe05113#GXhzEptFEweKj7Q3Jm6ndX4oVA3CMaiK6waqCeR2oFAG But I have not tested it :). I would say it should work, but at this point, I should probably say that maybe it could work Edited May 9 by engels74 1 Quote Link to comment
couzin2000 Posted May 9 Share Posted May 9 You did it! Works perfectly fine! Bravo! 1 Quote Link to comment
engels74 Posted May 9 Author Share Posted May 9 (edited) 8 minutes ago, couzin2000 said: You did it! Works perfectly fine! Bravo! Awesome ! Does the "Estimated completion time" work? Does it actually update the timestamp for each post? I'll probably do a proper release tomorrow on GitHub. Edited May 9 by engels74 Quote Link to comment
couzin2000 Posted May 10 Share Posted May 10 Yes it does. However, I view this as a bit of an annoyance, because it's not on a parametered setting. Think there's a way to get updates every 25%? Also, and this may just be me not knowing how to setup user scripts to permanently run, after Mover stopped, the script stopped. The next usage of Mover did not send any notifications. Which I would totally prefer on a forever basis... as long as there's a way to set notifications every x %. Otherwise, this is actually draining my Apple Watch battery! Quote Link to comment
engels74 Posted May 10 Author Share Posted May 10 (edited) 9 hours ago, couzin2000 said: Yes it does. However, I view this as a bit of an annoyance, because it's not on a parametered setting. Think there's a way to get updates every 25%? Also, and this may just be me not knowing how to setup user scripts to permanently run, after Mover stopped, the script stopped. The next usage of Mover did not send any notifications. Which I would totally prefer on a forever basis... as long as there's a way to set notifications every x %. Otherwise, this is actually draining my Apple Watch battery! I initially made this script for myself, so that's why it might be a bit weird Currently it's just configured to post every X second, set by the "NOTIFICATION_FREQUENCY" variable, and not in percentages. This was my personal preference, as I usually move huge amounts of data (1-2TB+) per mover session, and that it can take a fair while from 0% -> 25% -> 50% and so on, so it made more sense for me, to make it post every X second instead, as the user following along, could have a better sense of things actually "moving". I'm guessing I could make it an option to do either the notification pr second or by percentages. But the bash script will become increasingly complex, with all these options.. Hopefully I can figure something out. The next pain point of yours is that the script exits when it's done. This was also by design, as I only run my mover once every day, at 03:00AM, and it was easy to setup a cron-job to make the script launch a few minutes before, and exit when it's done. I think I should be able to just loop the script continuously. I would probably recommend still keeping a cronjob for the script, since it'll not relaunch by itself, if you have to reboot the Unraid server or whatever. I'll try and work a bit on the script. I have the following goals: - Combining the Discord and Telegram scripts, and make it togglable if you want to use both Telegram and Discord webhooks or only one of them, e.g. "USE_TELEGRAM=true/false", "USE_DISCORD=true/false" - Making the script use a loop so it starts anew, instead of exiting, when it has detected either the calculations have reached 100% or Unraid's Mover script has exited - Having the option to do either a notification per second or by percentages in increments of 25: 0%/25%/50%/75%/100% - or definable by the user (the increments) - Or.. Either that, or ditch the notification per second option completely, and make it in percentages instead only. Might be the easier approach, in terms of not having the bash script growing increasingly complex. Not sure how much of it I can get working, but I'll give it a shot. Thanks for the feedback Edited May 10 by engels74 Quote Link to comment
couzin2000 Posted May 10 Share Posted May 10 3 hours ago, engels74 said: I initially made this script for myself, so that's why it might be a bit weird Currently it's just configured to post every X second, set by the "NOTIFICATION_FREQUENCY" variable, and not in percentages. This was my personal preference, as I usually move huge amounts of data (1-2TB+) per mover session, and that it can take a fair while from 0% -> 25% -> 50% and so on, so it made more sense for me, to make it post every X second instead, as the user following along, could have a better sense of things actually "moving". I'm guessing I could make it an option to do either the notification pr second or by percentages. But the bash script will become increasingly complex, with all these options.. Hopefully I can figure something out. Well, I can also modify the script for myself, but if you're interested in making this available to a wider audience, yes I would think it'd be interesting to add the option. Even switching between "time" or "percentage" would be a great add. 3 hours ago, engels74 said: The next pain point of yours is that the script exits when it's done. This was also by design, as I only run my mover once every day, at 03:00AM, and it was easy to setup a cron-job to make the script launch a few minutes before, and exit when it's done. I think I should be able to just loop the script continuously. I would probably recommend still keeping a cronjob for the script, since it'll not relaunch by itself, if you have to reboot the Unraid server or whatever. I understand. I'm in the process of transferring over 6TB but my cache drive is only 120GB... so that becomes many MANY Mover jobs 😂But I don't mind restarting it manually. It's just that if you make it a continuously running script, then frequency does need to be adjusted -- I'm getting notifications every 45 seconds or so and my arm is numb from the vibration of my Apple Watch 😂 3 hours ago, engels74 said: I'll try and work a bit on the script. I have the following goals: - Combining the Discord and Telegram scripts, and make it togglable if you want to use both Telegram and Discord webhooks or only one of them, e.g. "USE_TELEGRAM=true/false", "USE_DISCORD=true/false" - Making the script use a loop so it starts anew, instead of exiting, when it has detected either the calculations have reached 100% or Unraid's Mover script has exited - Having the option to do either a notification per second or by percentages in increments of 25: 0%/25%/50%/75%/100% - or definable by the user (the increments) - Or.. Either that, or ditch the notification per second option completely, and make it in percentages instead only. Might be the easier approach, in terms of not having the bash script growing increasingly complex. Not sure how much of it I can get working, but I'll give it a shot. Thanks for the feedback Great Idea to be able to toggle between Telegram and Discord. If you ever want to make this a bit more interesting, perhaps there is a way to have the system generate its own notification, and that would send it through whichever Webhook is already configured for the user. I've chosen Telegram, but there's at least 10 different options to Unraid notifications... Maybe there's an easier way? If you need some testing or other feedback, I'm your man. I apologize as I'm not much of a programmer. I tinker, but not much else. Thank you for your very interesting work! I appreciate your effort! Quote Link to comment
engels74 Posted May 10 Author Share Posted May 10 (edited) 1 hour ago, couzin2000 said: [...] Thank you for your very interesting work! I appreciate your effort! Thanks for the feedback and everything :). I haven't tested this yet, but if you want, you can try the new version out here: https://logs.notifiarr.com/?9eaa332e3616a02f#FYLHcQpht3cQ9QnLZC7cjnvTtfs6gGhiXbqK3qcQmZ3 Again, word of caution, there's a chance it'll not work at all . I literally haven't even tried running it yet. But "on paper", it could work out of the box. There's a few changes, mainly: 1) Discord and Telegram versions are now combined Combined the Telegram and Discord scripts, hopefully will be easier to maintain one script instead of several different ones. Both variables are false by default, you should enable at least one of them. Note: I haven't tested what happens if you enable both of them, if it actually posts to both webhooks. But again, "on paper" it could work. 2) Percentage Progress only No more "notification per second", it's percentages only. I think you've made a great point. The user can always set the percentage increment lower, if they want more frequent updates posted to their webhook. So I think that should satisfy most users (including myself haha) 3) Loop instead of exit The script will now (hopefully) loop, instead of exiting. It'll continuously monitor when the Unraid Mover runs (with a "sleep 1"), and begin anew when it detects it has started/launched. 4) Output to console Made a few outputs, so the log the User Script plugin creates actually will have something semi-useful information. Like when the script starts, when it detects the mover, when it has completed, and when webhooks have been posted successfully to either Discord or Telegram. Here's hoping it'll work. I might do some testing myself later, but until then, you can have fun trying it out . Made quite a few large changes, so there's a chance I broke something along the way. Edit: To your question regarding making this work with Unraids notification system, instead of its own script entirely.. That might be feasible, though I have absolutely no experience with Unraids plugin system. I'm better / more comfortable just using straight bash.. Not good at much else in terms of programming. Edited May 10 by engels74 Small change to script Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.