Ultimate UNRAID Dashboard (UUD)

[grumpy]

I have had no luck installing Varken; I think it may be a dead project. Github is 4-6 years old, discord link both at Grafanna and GitHub dead, Wiki 304...

Any suggestions?

[falconexe]

1 hour ago, grumpy said:

I have had no luck installing Varken; I think it may be a dead project. Github is 4-6 years old, discord link both at Grafanna and GitHub dead, Wiki 304...

Any suggestions?

 

Varken is no longer being developed, but I still see it in Community Applications, and it doesn't appear to be officially deprecated. Do you see this? This is the way it is installed.

 

Edited April 2 by falconexe

[falconexe]

While I was researching how to currently install Varken, I noticed that it only supports InfluxDB 1.8.

 

 

From the Varken GitHub: https://github.com/Boerderij/Varken

 

 

 

So, if we were to move the UUD to Influx 3.X using SQL, it would require someone to pick up the mantel on Varken, and branch it. If anyone is serious about it, and willing to continue that project, I'm willing to try and get the UUD to Influx 3.X in tandem, once they release it and we can get our hands on it.

 

Once Varken's repository is gone gone, the main part of the UUD that I use (Plex) would need something else to get the Tautulli data into InfluxDB. I'm open to suggestions. We don't necessarily need Varken, we just need the code equivalent that moves the data. Perhaps a direct API call from Tautulli to InfluxDB? Let me know your thoughts (anyone).

 

 

@Jufy111 @viktortras

 

 

Edited April 2 by falconexe

[Scheev]

Need some advice on here, Ive been bashing my head for awhile trying to get this all working.

 

I am finding that all the guides are fairly outdated. When I use telegraf 1.20.2 it doesn't gather metrics cause the only way ive found to get the config provides the latest config. Using an older config wouldn't exactly solve my problem either as explained below.

Switching to the latest telegraf branch gives me permissions errors and won't start. Ive tried to do some chmod antics but can't seem to get the hang of where or what this is supposed to be. In my post arguments I have it running apt install arguments that have been outlined in guides and such. Would that be the culprit? how do I elevate permissions for this?

E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)

I imagine if telegraf started I could likely configure it to give metrics to influx.

Next problem, my influxDB docker is running influxdb v2 because I have a OPNsense instance that is posting metrics to Influx and requires it. However, without telegraf starting I can't even use the influxdb_v2 inputs in the config.

 

As well, how would I configure influx if I have two different instances of telegraf providing metrics to the database?

 

Thanks for any and all help!

 

[Jufy111]

5 hours ago, falconexe said:

While I was researching how to currently install Varken, I noticed that it only supports InfluxDB 1.8.

 

 

From the Varken GitHub: https://github.com/Boerderij/Varken

 

 

 

So, if we were to move the UUD to Influx 3.X using SQL, it would require someone to pick up the mantel on Varken, and branch it. If anyone is serious about it, and willing to continue that project, I'm willing to try and get the UUD to Influx 3.X in tandem, once they release it and we can get our hands on it.

 

Once Varken's repository is gone gone, the main part of the UUD that I use (Plex) would need something else to get the Tautulli data into InfluxDB. I'm open to suggestions. We don't necessarily need Varken, we just need the code equivalent that moves the data. Perhaps a direct API call from Tautulli to InfluxDB? Let me know your thoughts (anyone).

 

 

@Jufy111 @viktortras

 

 

It's possible to enable the influxdb v1 API on v2, but it requires some terminal work set up the backwards compatible bucket and endpoint.

https://docs.influxdata.com/influxdb/v2/reference/api/influxdb-1x/

 

Hopefully the backwards compatibility remains with the release of V3. I'll have a look at it and see how hard it would be to modify the post functions for v2/sql support. But I'm no dev, so I'm probably  out of my depth.

 

**edit

It actually doesn't look too hard to modify the payload  in varken to make it compatible with influxdb2 or SQL. If I find some spare time someday I might fork it and make a version that is natively compatible with both influxdb v1 and v2.

Edited April 2 by Jufy111

[Jufy111]

On 4/3/2024 at 5:12 AM, falconexe said:

While I was researching how to currently install Varken, I noticed that it only supports InfluxDB 1.8.

 

 

From the Varken GitHub: https://github.com/Boerderij/Varken

 

 

 

So, if we were to move the UUD to Influx 3.X using SQL, it would require someone to pick up the mantel on Varken, and branch it. If anyone is serious about it, and willing to continue that project, I'm willing to try and get the UUD to Influx 3.X in tandem, once they release it and we can get our hands on it.

 

Once Varken's repository is gone gone, the main part of the UUD that I use (Plex) would need something else to get the Tautulli data into InfluxDB. I'm open to suggestions. We don't necessarily need Varken, we just need the code equivalent that moves the data. Perhaps a direct API call from Tautulli to InfluxDB? Let me know your thoughts (anyone).

 

 

@Jufy111 @viktortras

 

 

Also I don't think it is dead. The dockerhub shows a nightly build updated 18 hours prior to this post.

https://hub.docker.com/r/boerderij/varken/tags

[Kilrah]

2 hours ago, Jufy111 said:

Also I don't think it is dead. The dockerhub shows a nightly build updated 18 hours prior to this post.

https://hub.docker.com/r/boerderij/varken/tags

That's just because there's still an automated action building the code every night even though it hasn't changed in 4 years.

Edited April 4 by Kilrah

[1471]

On 4/2/2024 at 11:32 PM, Jufy111 said:

It's possible to enable the influxdb v1 API on v2, but it requires some terminal work set up the backwards compatible bucket and endpoint.

https://docs.influxdata.com/influxdb/v2/reference/api/influxdb-1x/

 

Hopefully the backwards compatibility remains with the release of V3. I'll have a look at it and see how hard it would be to modify the post functions for v2/sql support. But I'm no dev, so I'm probably  out of my depth.

 

**edit

It actually doesn't look too hard to modify the payload  in varken to make it compatible with influxdb2 or SQL. If I find some spare time someday I might fork it and make a version that is natively compatible with both influxdb v1 and v2.

 

Its not something I have the skills to contribute to, but thanks for looking, Varken has been really useful over the years. Sounds like updating Varken at some point would probably be the easiest option for the Ultimate UNRAID Dashboard?

 

If not prometheus-plex-exporter might be another route, would also need some adapting though. Blog post about it HERE, github repository HERE for info.

Edited April 4 by 1471

[PJsUnraid]

For some reason my Plex statistics shows N/A until i restart Varken, then it reoccurs every 5ish minutes.. any ideas?

Edit: more specifically the total values (total movies, total seasons, total shows etc). The rest seems to be fine

Editx2: Looks like it only updates when tautulli starts streams, then goes back to N/A after a few mins

Edited April 4 by PJsUnraid

[grumpy]

My original post was poorly worded, I had Varken installed; but it would not start. I got past that by changing owner of the config.example from root to nobody, why it interfered with the required config.ini beats me. Every time I edit the config and remove some stuff when I restart it replaces the removed stuff.

 

There needs to be an updated How to! Which is true of many contributions.

[grumpy]

So now that I have all components running as expected (I think) I have more questions.

Did you setup a Varken database in influx as your Dashboard variable tab picture suggests you did. As the guides I followed does not suggest that but to use the Telegraph database. 

 

I have no Plex (no media) information showing on my Dashboard but as far as I can tell Telegraph, Taututlli and Varken are all processing the data as seen through the log of each.

 

I do not have IPMI so how do I get that information? (TEMPS)

Array Drive, System Temps/Power,  CPU, Ramm DIMM, Fan Speeds

I use NUTs(slave) for the UPS; the information is shown in Uraid but not in the dashboard. How to get that?

SSD Temp, Life, Power On, Lifetime reads/writes, are missing.

 

[Jufy111]

5 hours ago, PJsUnraid said:

For some reason my Plex statistics shows N/A until i restart Varken, then it reoccurs every 5ish minutes.. any ideas?

Edit: more specifically the total values (total movies, total seasons, total shows etc). The rest seems to be fine

Editx2: Looks like it only updates when tautulli starts streams, then goes back to N/A after a few mins

I have come across issues with varken on consistent writing data to the database. It is suppose to write points every 30s, but it does not.

 

In the in influx V2 version of UUD that I've been working I've modified the queries so that it gets the the most recent result from the last 10 minutes.

 

You could try and do the same to the v1 queries.

[Jufy111]

54 minutes ago, grumpy said:

So now that I have all components running as expected (I think) I have more questions.

Did you setup a Varken database in influx as your Dashboard variable tab picture suggests you did. As the guides I followed does not suggest that but to use the Telegraph database. 

 

I have no Plex (no media) information showing on my Dashboard but as far as I can tell Telegraph, Taututlli and Varken are all processing the data as seen through the log of each.

 

I do not have IPMI so how do I get that information? (TEMPS)

Array Drive, System Temps/Power,  CPU, Ramm DIMM, Fan Speeds

I use NUTs(slave) for the UPS; the information is shown in Uraid but not in the dashboard. How to get that?

SSD Temp, Life, Power On, Lifetime reads/writes, are missing.

 

The dashboard variable just points to a database, not a bucket.

 

The system stats are collected by telegraf and stired in the 'system' bucket. You will need to configure the telegraf config file to collect all the system stats.

 

The ups info is also configured in the telegraf config.

 

Do you get hdd stats (smart data, not dusk usage and IO? My guess is telegraf config is not configured properly. Or you haven't installed smartmontools in the telegraf container.

 

The telegraf config is quite long, but most of it is just comments. Once I'm back at my computer, I'll post a shortened version for you.

Edited April 4 by Jufy111

[Jufy111]

12 hours ago, Kilrah said:

That's just because there's still an automated action building the code every night even though it hasn't changed in 4 years.

Ah bummer, I might have a go at forking varken and updating it to be compatible with influx 2.0.

 

There is an outstanding pull request on GitHub that apparently enables it and works, so I might just see if I can fork it and manually merge the PR.

[grumpy]

@Jufy111

Thank You, I'm looking forward to your example, as far as the smartmontools. I think it is installed with the docker image post argument; which is how I did it.

/bin/sh -c 'apt update && apt install -y smartmontools && apt install -y lm-sensors && telegraf' --user 0 

 

I do get the panels:  Drive SMART Health Overview, Drive Life

[grumpy]

I'm an idiot; ok things I do not know.

 

I needed to add a varken data source connection, not just telegraph now I see Plex stuff; not everything yet, but at least moving forward.

[Jufy111]

16 minutes ago, grumpy said:

@Jufy111

Thank You, I'm looking forward to your example, as far as the smartmontools. I think it is installed with the docker image post argument; which is how I did it.

/bin/sh -c 'apt update && apt install -y smartmontools && apt install -y lm-sensors && telegraf' --user 0 

 

I do get the panels:  Drive SMART Health Overview, Drive Life

 

Hmm, not sure about the SSD data then.

 

Here is my telegraf config file (all though I'm using influxdb2 so that section will look different for you). It works for me, but still may not work for you. Full disclosure too, I've not actually tested UUD1.7, as it's all in InfluxDB v1 queries. I'm slowly changing them over to v2.

 

Spoiler

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = false
  quiet = false
  logfile = ""
  hostname = ""
  omit_hostname = false

 

[[outputs.influxdb_v2]]    
  urls = ["http://192.168.100.100:8086"]
  token = "your-token-here"
  organization = "your-org-here"
  bucket = "system"

 

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

 

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]

 

[[inputs.diskio]]

 

[[inputs.docker]]    
  endpoint = "unix:///var/run/docker.sock"
  gather_services = false
  container_names = []
  container_name_include = []
  container_name_exclude = []
  timeout = "5s"
  perdevice = true
  total = true
  docker_label_include = []
  docker_label_exclude = []

 

[[inputs.mem]]

[[inputs.net]]
#  interfaces = ["eth0","br-0d215e2e2923","br-8be3c4b3369e"]

[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.smart]]
 use_sudo = false
# path_smartctl = "/usr/bin/smartctl"
 attributes = true

 

[[inputs.sensors]]
[[inputs.apcupsd]]

 

Edited April 4 by Jufy111

[Jufy111]

On 4/3/2024 at 8:58 AM, Scheev said:

Need some advice on here, Ive been bashing my head for awhile trying to get this all working.

 

I am finding that all the guides are fairly outdated. When I use telegraf 1.20.2 it doesn't gather metrics cause the only way ive found to get the config provides the latest config. Using an older config wouldn't exactly solve my problem either as explained below.

Switching to the latest telegraf branch gives me permissions errors and won't start. Ive tried to do some chmod antics but can't seem to get the hang of where or what this is supposed to be. In my post arguments I have it running apt install arguments that have been outlined in guides and such. Would that be the culprit? how do I elevate permissions for this?

E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)

I imagine if telegraf started I could likely configure it to give metrics to influx.

Next problem, my influxDB docker is running influxdb v2 because I have a OPNsense instance that is posting metrics to Influx and requires it. However, without telegraf starting I can't even use the influxdb_v2 inputs in the config.

 

As well, how would I configure influx if I have two different instances of telegraf providing metrics to the database?

 

Thanks for any and all help!

 

 

 

I'm running influx v2 as well.

Telegraf 1.20.2 was the last version before the devs locked down the permission an made it  impossible to install smartmontools and other packets in the container. Any release newer than this, and you will have to remove the install argument for smartmontools and remove [inputs.smart]  out of the telegraf config. This will sacrifice any smart data.

 

I don't use OPNsense, but I just have my pfsense configured to write to a bucket called 'pfsense'. Pfsense has it's of telegraf instance. I assume that OPNsense is the same

 

I'm still not quite sure why you need the newer version of telegraf. OPNsense should have it's own telegraf instance pointed to separate bucket in your influxdb 2 instance. This should leave the docker telegraf free to be installed as v 1.20.2

 

See my post above for my telegraf :1.20.2 config that is configured for infludb v2

[Scheev]

24 minutes ago, Jufy111 said:

 

 

I'm running influx v2 as well.

Telegraf 1.20.2 was the last version before the devs locked down the permission an made it  impossible to install smartmontools and other packets in the container. Any release newer than this, and you will have to remove the install argument for smartmontools and remove [inputs.smart]  out of the telegraf config. This will sacrifice any smart data.

 

I don't use OPNsense, but I just have my pfsense configured to write to a bucket called 'pfsense'. Pfsense has it's of telegraf instance. I assume that OPNsense is the same

 

I'm still not quite sure why you need the newer version of telegraf. OPNsense should have it's own telegraf instance pointed to separate bucket in your influxdb 2 instance. This should leave the docker telegraf free to be installed as v 1.20.2

 

See my post above for my telegraf :1.20.2 config that is configured for infludb v2

I was able to figure it out, but my only hurdle is related to the use of InfluxQL for the dashboard instead of Flux. Is there a setting that will alleviate this? I started rewriting the dashboard in Flux...but realized theres probably another way around it.

[Jufy111]

12 minutes ago, Scheev said:

I was able to figure it out, but my only hurdle is related to the use of InfluxQL for the dashboard instead of Flux. Is there a setting that will alleviate this? I started rewriting the dashboard in Flux...but realized theres probably another way around it.

Give it a few days. I'm currently working on updating all the queries to Flux for UUD1.7.

 

I havent been able to find a way around it.

Edited April 4 by Jufy111

[Scheev]

1 hour ago, Jufy111 said:

Give it a few days. I'm currently working on updating all the queries to Flux for UUD1.7.

 

I havent been able to find a way around it.

Oh awesome, I spent like 4 hours today converting some of it already, happy to help if you want.

[Jufy111]

2 hours ago, Scheev said:

Oh awesome, I spent like 4 hours today converting some of it already, happy to help if you want.

Thanks, I'm almost done I think. Could definitely use your help for testing though. I also have a few questions for @falconexe too.

Edited April 5 by Jufy111

[PJsUnraid]

16 hours ago, Jufy111 said:

I have come across issues with varken on consistent writing data to the database. It is suppose to write points every 30s, but it does not.

 

In the in influx V2 version of UUD that I've been working I've modified the queries so that it gets the the most recent result from the last 10 minutes.

 

You could try and do the same to the v1 queries.

Yeah my knowledge with all this is very limited lol. I'll wait and monitor the progress of the v2 influx for the time being

 

[grumpy]

On 4/4/2024 at 9:23 AM, PJsUnraid said:

For some reason my Plex statistics shows N/A until i restart Varken, then it reoccurs every 5ish minutes.. any ideas?

Edit: more specifically the total values (total movies, total seasons, total shows etc). The rest seems to be fine

Editx2: Looks like it only updates when tautulli starts streams, then goes back to N/A after a few mins

@PJsUnraid

I do not believe it is Varken causing the issue, I think it is a timing issue. If you look at the influx database you will see all of plex data being updated constantly; not sure if it is 30sec or not but those metrics you are missing are done hourly.

 

So you no longer see N/A go to the panels that show Total

3 dots on top right

drop down - edit

now in edit mode for the panel

top left side click on query options

Relative time change to 62m or 1h

top right of page Apply to close panel

 

Now it will show the number in a persistent state till it changes on the next query.

 

Edited April 5 by grumpy
SP

[grumpy]

Now after watching the panel for 16 hours, and correcting some issues for myself I have noticed some other issues that go beyond my understanding.

 

Stream Log (Overview) misses some entries. In music playing from Plex it can keep up sometimes, skip a couple of songs, or even skip 10's of songs. Probably not noticed during movies or tv shows; I do not know as I haven't made it that far during testing.

Stream Log (Detailed) Keeps up even though there are delays, to be expected Plex -> Tautulli -> Varken -> Influx, same with Current Streams. During Music play percentage is of no use, while the song can have 72% remaining when in fact the next song has started playing.

Even though most of my posts seem like complaints I really like to thank @falconexe for all of his hard work, it is functional and visual pleasing.

I know I wouldn't have been able to do this on my own accord.