Ultimate UNRAID Dashboard (UUD)

[falconexe]

Just now, GilbN said:

Yeah you should really run telegraf as host, as you want it to get all the host metrics

Agreed. Added to Install Instructions.

[falconexe]

18 minutes ago, falconexe said:

@FreeMan

 

Actually add this. It will load both IPMI and S.M.A.R.T. when the docker automatically starts. I believe this is why it was not finding it in your path. Without these extra arguments, you would have to manually bash into the docker and manually load them each time the Docker updates/restarts.

 

/bin/sh -c 'apk update && apk upgrade && apk add ipmitool && apk add smartmontools && telegraf'

 

Report back and let me know if your log is no long spamming pink.

@FreeMan If the above still does not work, you may also need to install the S.M.A.R.T. plugin as it may have never been initially loaded. Right click the Telegraf docker, select Console, and run "apk add smartmontools".

 

Please report back and let us know.

Edited September 14, 2020 by falconexe

[FreeMan]

16 minutes ago, falconexe said:

@FreeMan

 

Actually add this. It will load both IPMI and S.M.A.R.T. when the docker automatically starts. I believe this is why it was not finding it in your path. Without these extra arguments, you would have to manually bash into the docker and manually load them each time the Docker updates/restarts.

 

/bin/sh -c 'apk update && apk upgrade && apk add ipmitool && apk add smartmontools && telegraf'

 

Report back and let me know if your log is no long spamming pink.

Thanks for digging into this. Unfortunately, even after adding that to the Post Arguments, I'm still getting

2020-09-14T21:32:35Z I! Starting Telegraf 1.15.3
2020-09-14T21:32:35Z I! Using config file: /etc/telegraf/telegraf.conf
2020-09-14T21:32:35Z I! Loaded inputs: kernel mem docker netstat sensors disk cpu processes system smart diskio swap apcupsd net
2020-09-14T21:32:35Z I! Loaded aggregators:
2020-09-14T21:32:35Z I! Loaded processors:
2020-09-14T21:32:35Z I! Loaded outputs: influxdb
2020-09-14T21:32:35Z I! Tags enabled: host=NAS
2020-09-14T21:32:35Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"NAS", Flush Interval:10s
2020-09-14T21:32:40Z E! [inputs.smart] Error in plugin: failed to run command '/usr/bin/smartctl --scan': fork/exec /usr/bin/smartctl: no such file or directory -

Just to be sure, I copy pasted what you quoted above for the Post Arguments. Here's a copy/paste back from the config page. Looks right to me, but just in case I can't read this afternoon...

/bin/sh -c 'apk update && apk upgrade && apk add ipmitool && apk add smartmontools && telegraf'

Also, I added the --restart=always to the Extra Parameters setting. Obviously, it sets it to always restart, but at what point is it restarting?

 

[falconexe]

8 minutes ago, FreeMan said:

Thanks for digging into this. Unfortunately, even after adding that to the Post Arguments, I'm still getting

2020-09-14T21:32:35Z I! Starting Telegraf 1.15.3
2020-09-14T21:32:35Z I! Using config file: /etc/telegraf/telegraf.conf
2020-09-14T21:32:35Z I! Loaded inputs: kernel mem docker netstat sensors disk cpu processes system smart diskio swap apcupsd net
2020-09-14T21:32:35Z I! Loaded aggregators:
2020-09-14T21:32:35Z I! Loaded processors:
2020-09-14T21:32:35Z I! Loaded outputs: influxdb
2020-09-14T21:32:35Z I! Tags enabled: host=NAS
2020-09-14T21:32:35Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"NAS", Flush Interval:10s
2020-09-14T21:32:40Z E! [inputs.smart] Error in plugin: failed to run command '/usr/bin/smartctl --scan': fork/exec /usr/bin/smartctl: no such file or directory -

Just to be sure, I copy pasted what you quoted above for the Post Arguments. Here's a copy/paste back from the config page. Looks right to me, but just in case I can't read this afternoon...

/bin/sh -c 'apk update && apk upgrade && apk add ipmitool && apk add smartmontools && telegraf'

Also, I added the --restart=always to the Extra Parameters setting. Obviously, it sets it to always restart, but at what point is it restarting?

 

I'll defer to @GilbN on that one.

 

Your syntax appears correct:

 

Mine:  /bin/sh -c 'apk update && apk upgrade && apk add ipmitool && apk add smartmontools && telegraf'

Yours: /bin/sh -c 'apk update && apk upgrade && apk add ipmitool && apk add smartmontools && telegraf'

 

After running that, my log is clean.

 

You can see it installing/updating each of the plugins.

 

 

The --restart=always command just tells the docker to restart after a server boot or array stoppage. It is just another way of setting it to restart instead of using the button on the Docker page within the native UNRAID dashboard. Just a style thing. I like the code because I can see it and I know it works 100% of the time.

Edited September 14, 2020 by falconexe

[GilbN]

6 minutes ago, FreeMan said:

Thanks for digging into this. Unfortunately, even after adding that to the Post Arguments, I'm still getting

2020-09-14T21:32:35Z I! Starting Telegraf 1.15.3
2020-09-14T21:32:35Z I! Using config file: /etc/telegraf/telegraf.conf
2020-09-14T21:32:35Z I! Loaded inputs: kernel mem docker netstat sensors disk cpu processes system smart diskio swap apcupsd net
2020-09-14T21:32:35Z I! Loaded aggregators:
2020-09-14T21:32:35Z I! Loaded processors:
2020-09-14T21:32:35Z I! Loaded outputs: influxdb
2020-09-14T21:32:35Z I! Tags enabled: host=NAS
2020-09-14T21:32:35Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"NAS", Flush Interval:10s
2020-09-14T21:32:40Z E! [inputs.smart] Error in plugin: failed to run command '/usr/bin/smartctl --scan': fork/exec /usr/bin/smartctl: no such file or directory -

 

My telegraf smart plugin looks like this:

 

# # Read metrics from storage devices supporting S.M.A.R.T.
 [[inputs.smart]]
#   ## Optionally specify the path to the smartctl executable
#   # path = "/usr/bin/smartctl"
#   #
#   ## On most platforms smartctl requires root access.
#   ## Setting 'use_sudo' to true will make use of sudo to run smartctl.
#   ## Sudo must be configured to to allow the telegraf user to run smartctl
#   ## with out password.
#     use_sudo = false
#   #
#   ## Skip checking disks in this power mode. Defaults to
#   ## "standby" to not wake up disks that have stoped rotating.
#   ## See --nocheck in the man pages for smartctl.
#   ## smartctl version 5.41 and 5.42 have faulty detection of
#   ## power mode and might require changing this value to
#   ## "never" depending on your disks.
#   # nocheck = "standby"
#   #
#   ## Gather detailed metrics for each SMART Attribute.
#   ## Defaults to "false"
#   ##
     attributes = true
#   #
#   ## Optionally specify devices to exclude from reporting.
#   # excludes = [ "/dev/pass6" ]
#   #
#   ## Optionally specify devices and device type, if unset
#   ## a scan (smartctl --scan) for S.M.A.R.T. devices will
#   ## done and all found will be included except for the
#   ## excluded in excludes.
#   # devices = [ "/dev/ada0 -d atacam" ]

I only have  [[inputs.smart]] and attributes = true uncommented

[falconexe]

4 minutes ago, GilbN said:

My telegraf smart plugin looks like this:
 

I only have  [[inputs.smart]] and attributes = true uncommented

 

 

@FreeMan Oh yeah, we had you try to place the path in there as a way to test. Please remove that by commenting it out. My config matches @GilbN's. Just those 2 lines are ON.

Edited September 14, 2020 by falconexe

[FreeMan]

23 minutes ago, falconexe said:

Just a style thing. I like the code because I can see it and I know it works 100% of the time.

Gotcha!

 

For some reason, I can't get the web-based console to work for me, so I did `docker exec -i -t telegraf bash` from an SSH session. From there:

bash-5.0# which smartctl
/usr/sbin/smartctl
bash-5.0# ls -la /usr/sbin/sma*
-rwxr-xr-x    1 root     root        761448 Feb 22  2020 /usr/sbin/smartctl
-rwxr-xr-x    1 root     root        617416 Feb 22  2020 /usr/sbin/smartd
bash-5.0# smartctl -i /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Compute
<snip>

bash-5.0# /usr/sbin/smartctl -i /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

so, within the docker itself, it can find smartctl on the path, it sees it at /usr/sbin, it works when finding it on the path and it works when giving it an absolute path. However, I'm still getting my log spammed with:

2020-09-14T21:45:50Z E! [inputs.smart] Error in plugin: failed to run command '/usr/bin/smartctl --scan': fork/exec /usr/bin/smartctl: no such file or directory -

Then...

19 minutes ago, GilbN said:

I only have  [[inputs.smart]] and attributes = true uncommented

So I adjusted my telegraf.conf to almost match - I commented out the `path = "/usr/bin/smartctl"` portion and uncommented `attributes = false` and forgot to set it to `true`.

 

No errors and some HDD data!!!

 

I set `attributes = true` and still only some joy...

but...

 

So, I'm getting some HDD data now, but not everything. Interestingly, /dev/sdb is the drive I used for testing access to smartctl earlier and it responded just fine. It's a reasonably new 8TB drive. Odd that it's not responding here when all the others are responding without issue now.

Edited September 14, 2020 by FreeMan

[GilbN]

6 minutes ago, FreeMan said:

Gotcha!

 

For some reason, I can't get the web-based console to work for me, so I did `docker exec -i -t telegraf bash` from an SSH session. From there:

bash-5.0# which smartctl
/usr/sbin/smartctl
bash-5.0# ls -la /usr/sbin/sma*
-rwxr-xr-x    1 root     root        761448 Feb 22  2020 /usr/sbin/smartctl
-rwxr-xr-x    1 root     root        617416 Feb 22  2020 /usr/sbin/smartd
bash-5.0# smartctl -i /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Compute
<snip>

bash-5.0# /usr/sbin/smartctl -i /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

so, within the docker itself, it can find smartctl on the path, it sees it at /usr/sbin, it works when finding it on the path and it works when giving it an absolute path. However, I'm still getting my log spammed with:

2020-09-14T21:45:50Z E! [inputs.smart] Error in plugin: failed to run command '/usr/bin/smartctl --scan': fork/exec /usr/bin/smartctl: no such file or directory -

Then...

So I adjusted my telegraf.conf to almost match - I commented out the `path = "/usr/bin/smartctl"` portion and uncommented `attributes = false` and forgot to set it to `true`.

 

No errors and some HDD data!!!

 

I set `attributes = true` and still only some joy...

but...

 

So, I'm getting some HDD data now, but not everything. Interestingly, /dev/sdb is the drive I used for testing access to smartctl earlier and it responded just fine. It's a reasonably new 8TB drive. Odd that it's not responding here when all the others are responding without issue now.

Probably because of the hardcoded paths @falconexe uses on those panels. Falc, you should switch those to some regex. /.*user0/ will match /rootfs/mnt/user0 and /mnt/user0. And your host variable is broken. It should list the host in that drop down. 

[falconexe]

3 minutes ago, FreeMan said:

Gotcha!

 

For some reason, I can't get the web-based console to work for me, so I did `docker exec -i -t telegraf bash` from an SSH session. From there:

bash-5.0# which smartctl
/usr/sbin/smartctl
bash-5.0# ls -la /usr/sbin/sma*
-rwxr-xr-x    1 root     root        761448 Feb 22  2020 /usr/sbin/smartctl
-rwxr-xr-x    1 root     root        617416 Feb 22  2020 /usr/sbin/smartd
bash-5.0# smartctl -i /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda Compute
<snip>

bash-5.0# /usr/sbin/smartctl -i /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-4.19.107-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

so, within the docker itself, it can find smartctl on the path, it sees it at /usr/sbin, it works when finding it on the path and it works when giving it an absolute path. However, I'm still getting my log spammed with:

2020-09-14T21:45:50Z E! [inputs.smart] Error in plugin: failed to run command '/usr/bin/smartctl --scan': fork/exec /usr/bin/smartctl: no such file or directory -

Then...

So I adjusted my telegraf.conf to almost match - I commented out the `path = "/usr/bin/smartctl"` portion and uncommented `attributes = false` and forgot to set it to `true`.

 

No errors and some HDD data!!!

 

I set `attributes = true` and still only some joy...

but...

 

So, I'm getting some HDD data now, but not everything. Interestingly, /dev/sdb is the drive I used for testing access to smartctl earlier and it responded just fine. It's a reasonably new 8TB drive. Odd that it's not responding here when all the others are responding without issue now.

 

OK a lot to unpack here ha ha.

 

#1. Glad to see you see something. Looks like that original issue is resolved.

 

#2. So here is where the fun begins. You need to go into each panel and adjust the queries/overrides to match your exact architecture. So let's start with the array size. That uses mnt/User0. So you should not need to adjust it. Have you tried refreshing manually in the upper right hand corner? Or change the time line to last 5 minutes then back to last 24 hours? Sometimes upon initially loading the database, the GUI will fail to show data. You can also just close your browser, clear your cookies, and try again.

 

#3. On the S.M.A.R.T. dashboard I created, you will need to go in there an adjust he overrides where we are mapping the drive device to the drive name. My kernal mounts will not necessarily match yours (sd*). So you will need to modify the value mappings. Same thing with each of the 4 "Storage" panels.

 

[falconexe]

3 minutes ago, GilbN said:

Probably because of the hardcoded paths @falconexe uses on those panels. Falc, you should switch those to some regex. /.*user0/ will match /rootfs/mnt/user0 and /mnt/user0. And your host variable is broken. It should list the host in that drop down. 

@GilbN

 

For the User0 path, aren't they all the same in that they are "rootfs/mnt/user0"? In that case, why would we need to Regex it? I only hard coded paths where they are the same for all users to my knowledge...

 

 

 

"And your host variable is broken. It should list the host in that drop down. "

 

Also, do you mean that the variable in MY version 1.2 is broken? Or his?

 

Mine shows the following:

 

 

So it appears to be working and picking it up dynamically.

 

 

[falconexe]

@GilbN

 

In fact, I actually made no changes to the variables from your original dashboard. They all worked so great LOL.

 

[GilbN]

I dont have rootfs there and I know other users that dont get the /rootfs paths prepended. Not sure why. 

Yea, yours. I just downloaded the lastest, and if I try and edit the variable it doesnt display the query when you click edit on it. 

 

https://selfhosted.app/f3c10/GIwIGOSa20.mp4

[FreeMan]

7 minutes ago, falconexe said:

You need to go into each panel and adjust the queries/overrides to match your exact architecture.

Yeah... I started figuring that out. I, somewhat naively it seems, ass-u-med that it would figure out how many drives I've got and report on 'em. Maybe V2.0.

 

I'm editing one of the panels and set Parity01 to sdd. Can't that assignment change with every boot? Doesn't that mean that the next time I reboot my machine, these may all need to be changed?

[GilbN]

1 minute ago, FreeMan said:

Yeah... I started figuring that out. I, somewhat naively it seems, ass-u-med that it would figure out how many drives I've got and report on 'em. Maybe V2.0.

 

I'm editing one of the panels and set Parity01 to sdd. Can't that assignment change with every boot? Doesn't that mean that the next time I reboot my machine, these may all need to be changed?

Yeah, I stopped doing it that way. Got tired of updating it everytime.. lol 

[falconexe]

Just now, GilbN said:

Yeah, I stopped doing it that way. Got tired of updating it everytime.. lol 

How did you get around that dynamically? What did you use? Screenshot please...😁

Edited September 14, 2020 by falconexe

[falconexe]

4 minutes ago, GilbN said:

I dont have rootfs there and I know other users that dont get the /rootfs paths prepended. Not sure why. 

Yea, yours. I just downloaded the lastest, and if I try and edit the variable it doesnt display the query when you click edit on it. 

 

https://selfhosted.app/f3c10/GIwIGOSa20.mp4

Oh wow. Not sure what's going on there. I didn't mess with any of that. You want to see if you can fix it and I can merge it into version 1.3?

[FreeMan]

Just now, GilbN said:

Yeah, I stopped doing it that way. Got tired of updating it everytime.. lol 

A recommended better way of doing it?

 

I can see assigning parity a name and leaving the rest labeled by sdx - that won't change, but, what was sdd on last boot may be sdb on this boot and that would cause all sorts of bizarre reporting next time 'round.

[FreeMan]

Also, odd that even though it's got my server name in the dash variables, I still get that big red error about not being able to find MassEffect

 (and for some odd reason, the forum wouldn't let me embed the image in the post...)

 

[GilbN]

@falconexe @FreeMan Take a look at https://grafana.com/grafana/dashboards/7233 I updated it 1 hour ago. I have switched about all panels to use regex where I can. 

 

I heading to bed now, but you should be able to recreate the variable from my dash. 
>A recommended better way of doing it? - Dont hardcode anything

 

@falconexe Since you're the db expert, isn't this more effective than the 20 queries to the left? The result is the same. 

[falconexe]

Can you try these settings and let me know if it still does that? Change your host to "NAS".

 

[falconexe]

1 minute ago, GilbN said:

@falconexe @FreeMan Take a look at https://grafana.com/grafana/dashboards/7233 I updated it 1 hour ago. I have switched about all panels to use regex where I can. 

 

I heading to bed now, but you should be able to recreate the variable from my dash. 
>A recommended better way of doing it? - Dont hardcode anything

 

@falconexe Since you're the db expert, isn't this more effective than the 20 queries to the left? The result is the same. 

Yeah that makes sense, but I wanted to split my CPU cores between the 2 CPUs that I have. So CPU1 is 0-15 and CPU2 is 16-31. Guess I could have regexed that but I was trying to make it as "transparent" as possible for anyone coming in behind us to they can adjust where needed to their architecture. Guess we'll have to call your dynamic version the UUUD. The Ultimate Ultimate UNRAID Dashboard ha ha. 😂

[FreeMan]

6 minutes ago, falconexe said:

Can you try these settings and let me know if it still does that? Change your host to "NAS".

 

still no love

[falconexe]

2 minutes ago, FreeMan said:

still no love

Alrighty. Prob has everything to do with the variables that GilbN was talking about. I've grabbed his latest version. If I can get it fixed, I will merge and release version 1.3.

[falconexe]

6 minutes ago, FreeMan said:

still no love

 

2 minutes ago, falconexe said:

Alrighty. Prob has everything to do with the variables that GilbN was talking about. I've grabbed his latest version. If I can get it fixed, I will merge and release version 1.3.

Hey what do you have in this window?

 

 

 

 

You'll need to add YOUR datasource here (remove mine). Let me know if that works and I'll update the instructions to include this bit.

Edited September 14, 2020 by falconexe

[falconexe]

25 minutes ago, FreeMan said:

Yeah... I started figuring that out. I, somewhat naively it seems, ass-u-med that it would figure out how many drives I've got and report on 'em. Maybe V2.0.

Above my pay-grade LOL. I wish I had the time to figure that out. I'm sure GilbN will figure it out (if he hasn't already).