Visualize storage usage over time?


hermy65

Recommended Posts

Not necessarily a support question but not sure where else to put this.

 

Is there a plugin/app of some sort that we can install in unRAID that will allow us to visualize our storage usage over time? Perhaps it would show how much additional storage we are using per month, etc? Per month storage burn might be helpful in determining when you need to get additional drives ,etc. Also, it could show you if there was a deletion of data or something of that sort as your usage would drop, etc?

  • Like 1
Link to comment
5 minutes ago, falconexe said:

+1

 

Actually, this would be pretty sweet if it logged data growth and shrinkage (deletes) over time and threw these statistics in a pretty graph. I would love to know how many TB I ate up in a given timeframe, or the last 30 days. I currently track this manually. Trending would be a nice touch...Based on current usage, here's the date you need to buy new drives, and here's when to order them ha ha. If anyone knows of a docker or plugin that does this, let us know.

 

@hermy65 Thanks for posing this question. Great stuff...

kinda a prtg ish something or 1 of the other tools.

Link to comment
3 hours ago, falconexe said:

Hey man thanks for responding. I've been working on this all day and night. Just woke up from a 5 hour "nap". I am getting pretty good with it (I come from a SQL/Data Science background). However, I do have some questions, and have a few oddities in some of the queries. Perhaps what you are already working on... Hopefully you are down for and open to some questions. I'm really looking forward to picking your brain. Can't thank you enough! I haven't been this geeked out on my server for a while ha ha.

 

The major item is I can't get CPU temps to show up. I'm thinking I'm missing a core plugin or something. Until I installed the HDDTemp docker, my HD temps looked empty as well (now working). When I dive into these queries, none of the values auto populate. Looks like yours are hard coded. I'm inferring that it can't query the database for these values because they don't exist in some array. Any thoughts?

 

image.thumb.png.f2047427cb8d3453d539abc5b4325d2d.png

 

EDIT:

 

@GilbN Here is the Query.

 

The values for "sensors", "chip", and "feature", all do not have a drop down with auto populated values like every single other FROM statement. I am guessing the "sensors" table/plugin is missing, and therefore the WHERE clauses are not intelli-sensed. Let me know your thoughts. I appreciate it!

 

image.thumb.png.d212b685cfbb90fa2c315b493d0e211c.png

 

EDIT 2:

 

Not sure if it has anything to do with this.

 

image.png.0d4d9dd0dd0770e83284308e9f0eb65a.png

 

image.thumb.png.62add406e3aa10d6db7586545024f161.png

 

Bottom Right of GUI:

 

The white values show up with it on. If I turn it off, only the green values show. In fact, this looks redundant, since with it off the native GUI already polls these correctly (shows 2 CPUs), where the plugin only has the ability to show 1 CPU at a time.

 

image.png.1538ba8e6078979225d55ba4e7f421ea.png

 

Np. Heh, I don't have any database background so don't be surprised if you find some wierd queries in there. 

What happens if you run the `sensors` command in unraid terminal?

 

Many of the panels have hard-coded stuff in them that I'm removing so they can be more dynamic.

Edited by GilbN
Link to comment
2 hours ago, falconexe said:

Not much ha ha.

 

image.png.fce4b5ddb031570cf67912041a1f3529.png

 

 

So I am getting more and more deep into this tool and I have found a number of issues and I've made some improvements to existing panels. I am most excited about the more custom stuff I have developed. Hope to share it with you all soon.

Hmm, which cpu do you have? AMD?

 

Heres, what I have so far btw: https://cdn.discordapp.com/attachments/397193259476058113/753738500217765969/Unraid_System_Dashboard_NEW-1599775585538.json

Link to comment
7 hours ago, falconexe said:

UPDATED IN VERSION 1.1

 

I FIXED THE BROKEN disk trending queries. They had some wacky hard coded values for DAY/WEEK/MONTH/YEAR. So I dove into the documentation and figured it out.

 

Documentation:

image.thumb.png.d5ab67d3bc4e8607645006e336aeaf29.png

 

Link: https://grafana.com/docs/grafana/latest/panels/queries/

 

The other fun thing was that where you edit this in the query window is a little hidden. It is actually in the "Query Options" area in light blue.

 

image.png.db4dbda90d88f6420d509e549aa14bac.png

 

The relative time interval field shows "now" (current date/time) divided by the time frame (d/w/M/y) variable. This is then calculated when the "Calculation" field is set to "Difference" (Right Side Panel of Query Editor). This basically says show me the difference between today (now) and all data points up until the last data point in that time frame interval.

 

image.png.38616fbb17ac3801103923d2d275c750.png

 

In order get all 4 date ranges synced up, I had to calculate the total seconds in each date range and then divide by my interval (10 seconds). The original dashboard was using a mean() function to aggregate the average. This was very bad because it breaks the dashboard pulling so much data. I was using 4.5GB of RAM just on the 1 tab. Once I changed the selection to distinct(), it effectively said take the very last record from the aggregate table. This is the very last "difference" between the last 20 seconds. Once I made this change, these calculations were instantaneous! No more lag!

 

Day:

image.thumb.png.f374fb865038d466c32b17c1a3ce4c93.png

 

Week:

image.thumb.png.69372eb75b6c0be8d636bfc9ed74b19a.png

 

Month:

image.thumb.png.d8549ddd28e7758f3be37aacac9738a0.png

 

Year:

image.thumb.png.bbc88084ceb7c80678f5aa6e6924e4a5.png

 

Fun Note: The disk path of "user0" is equivalent to your entire array storage minus parity/cache/appdata (shares only).

 

And For the Results...

 

 

image.thumb.png.4fac4eafe50d368ed6e10578dea0c389.png

Great job! I was having issues with those panels and couldn't figure out how to do them correctly. Looking forward to take a look at your dash 😁

Link to comment
10 minutes ago, FreeMan said:

@falconexe & @GilbN If the two of you figure out how to package this as a docker with some "easy" to follow instructions on setup & config (understanding that everyone's setup is different), you'll be the coolest kids at the playground for at least a month!!

Well all this is setup by using 3 containers. Telegraf, InfluxDB and Grafana. You can take a look here https://technicalramblings.com/blog/how-to-setup-grafana-influxdb-and-telegraf-to-monitor-your-unraid-system/

 

Though I'm gonna rewrite parts of that guide when I'm finished with the dashboard. But the installation and setup of the containers will be the same. 

Link to comment

OK, I got the dockers set up.

 

@GilbN thanks for your guide! Easy as pie, thouh you may want to make a note in your blog about setting it up that when you install telegraf, since it doesn't find the telegraf.conf file, it creates a directory named that. The directory has to be removed & replaced with the telegraph.conf file which then needs to be edited. This kept me quite confused for a bit.

 

@falconexe - There's probably something just as simple for fixing this... I downloaded the v1.1 JSON file you posted. I was able to upload it just fine and everything comes up, except it throws an error that says it can't find MassEffect. That makes perfect sense, since that's your server name, not mine. Where the heck do I go to tweak that? There's nothing listed in the "host" drop-down menu: image.png.8f56ebeadf07f2df8972c9e46e324e20.png and I haven't (yet) been able to figure that one out.

 

It does, though, list my server (NAS) for all the other options: image.png.b57ba4bec52ce01d6102dcbfaf23c312.png

Edited by FreeMan
Link to comment
11 hours ago, FreeMan said:

OK, I got the dockers set up.

 

@GilbN thanks for your guide! Easy as pie, thouh you may want to make a note in your blog about setting it up that when you install telegraf, since it doesn't find the telegraf.conf file, it creates a directory named that. The directory has to be removed & replaced with the telegraph.conf file which then needs to be edited. This kept me quite confused for a bit.

 

You didn't see the big red warning box? 😛

 

I'll add a note on how to fix if you skip that step, thanks 👍🏻

  • Like 1
Link to comment
8 hours ago, falconexe said:

 

@FreeMan Simply open the drop down, and in the blank line, type "NAS" and then it will automatically make everything work instantly. 

 

image.png.cf6611a4405daecd82c1a0d646ef0a03.png

 

That changes the following lines in the JSON with that variable...

image.png

Yeah I noticed that too. I think you missed to select the datasource in the variable. Set it to the other variable where you select the database

Link to comment
11 hours ago, falconexe said:

Simply open the drop down, and in the blank line, type "NAS" and then it will automatically make everything work instantly. 

Well, not quite "everything", but a fair chunk! I'd hit that drop down and saw that the box was empty - never occurred to me to just type the server name in. I'd been doing construction - working on fixing the porch ceiling - earlier yesterday, so my brain wasn't fully in IT mode. That's my excuse and I'm sticking with it! 🤣

 

Oddly, I'm not getting any disk space usage, which was the whole point of this exercise. I'm getting disk temps and I/O, but not for the cache drives. I've got a bit of tweaking to do, but I love it so far.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.