Ultimate UNRAID Dashboard (UUD)

[grumpy]

Is there a way to limit the display of duplicates in Grafana? I realize it is a new input from Varken but it is the same data over and over.

 

Stream Log (Detailed)

 

[PJsUnraid]

4 hours ago, grumpy said:

@PJsUnraid

I do not believe it is Varken causing the issue, I think it is a timing issue. If you look at the influx database you will see all of plex data being updated constantly; not sure if it is 30sec or not but those metrics you are missing are done hourly.

 

So you no longer see N/A go to the panels that show Total

3 dots on top right

drop down - edit

now in edit mode for the panel

top left side click on query options

Relative time change to 62m or 1h

top right of page Apply to close panel

 

Now it will show the number in a persistent state till it changes on the next query.

 

Perfect! that seemed to update it, hopefully theres a minor revision that addresses this

[Scheev]

20 hours ago, Jufy111 said:

Thanks, I'm almost done I think. Could definitely use your help for testing though. I also have a few questions for @falconexe too.

Yeah for sure, just send me a DM here on the forums and I can give it a shot

[Jufy111]

20 hours ago, grumpy said:

Now after watching the panel for 16 hours, and correcting some issues for myself I have noticed some other issues that go beyond my understanding.

 

Stream Log (Overview) misses some entries. In music playing from Plex it can keep up sometimes, skip a couple of songs, or even skip 10's of songs. Probably not noticed during movies or tv shows; I do not know as I haven't made it that far during testing.

Stream Log (Detailed) Keeps up even though there are delays, to be expected Plex -> Tautulli -> Varken -> Influx, same with Current Streams. During Music play percentage is of no use, while the song can have 72% remaining when in fact the next song has started playing.

Even though most of my posts seem like complaints I really like to thank @falconexe for all of his hard work, it is functional and visual pleasing.

I know I wouldn't have been able to do this on my own accord.

 

Take the following with a grain of salt because my DB is influxDB v2 and I have not used the queries from falconexe's dashboard. The following is based on my experience updating all the queries to influxdb v2. unfortunately a lot of the queries are just missing, so I haven't been able look at the influxql query and convert to to flux.

 

I think the stream log misses entries because the filter is using tag "session_id", if you change this to "_value" it should fix it.

I believe the session_id stays the same for a single session, even if different media is played (songs or consecutive episode). This will mean if a stream is stopped and restarted it will still have multiple entries in the history

 

I agree, the percentage is kind of a little useless. This is because varken (maybe tautulli too) only update at most every 30 seconds. If a song finishes between DB writes the last percentage reading is left in the history. I don't believe that there is any way around this at this stage (short of coding in some logic to the query that updates the 100 percent if conditions are met.),

Edited April 6 by Jufy111
changed "hash" to "_value"

[grumpy]

There is no session _value from Varken to Influx, unless that is an Influx built-in. But that is moat as @falconexe is not using session_id for these queries any way. He is using player_state.

 

            hash_id = hashit(f'{session.session_id}{session.session_key}{session.username}{session.full_title}')
            influx_payload.append(
                {
                    "measurement": "Tautulli",
                    "tags": {
                        "type": "Session",
                        "session_id": session.session_id,
                        "friendly_name": session.friendly_name,
                        "username": session.username,
                        "title": session.full_title,
                        "product": session.product,
                        "platform": platform_name,
                        "product_version": product_version,
                        "quality": quality,
                        "video_decision": video_decision.title(),
                        "transcode_decision": decision.title(),
                        "transcode_hw_decoding": session.transcode_hw_decoding,
                        "transcode_hw_encoding": session.transcode_hw_encoding,
                        "media_type": session.media_type.title(),
                        "audio_codec": session.audio_codec.upper(),
                        "audio_profile": session.audio_profile.upper(),
                        "stream_audio_codec": session.stream_audio_codec.upper(),
                        "quality_profile": session.quality_profile,
                        "progress_percent": session.progress_percent,
                        "region_code": geodata.subdivisions.most_specific.iso_code,
                        "location": location,
                        "full_location": f'{geodata.subdivisions.most_specific.name} - {geodata.city.name}',
                        "latitude": latitude,
                        "longitude": longitude,
                        "player_state": player_state,
                        "device_type": platform_name,
                        "relayed": session.relayed,
                        "secure": session.secure,
                        "server": self.server.id
                    },
                    "time": now,
                    "fields": {
                        "hash": hash_id
                    }
                }
            )

        influx_payload.append(
            {
                "measurement": "Tautulli",
                "tags": {
                    "type": "current_stream_stats",
                    "server": self.server.id
                },
                "time": now,
                "fields": {
                    "stream_count": int(get['stream_count']),
                    "total_bandwidth": int(get['total_bandwidth']),
                    "wan_bandwidth": int(get['wan_bandwidth']),
                    "lan_bandwidth": int(get['lan_bandwidth']),
                    "transcode_streams": int(get['stream_count_transcode']),
                    "direct_play_streams": int(get['stream_count_direct_play']),
                    "direct_streams": int(get['stream_count_direct_stream'])
                }
            }
        )

        self.dbmanager.write_points(influx_payload)

 

 

[Jufy111]

10 hours ago, grumpy said:

There is no session _value from Varken to Influx, unless that is an Influx built-in. But that is moat as @falconexe is not using session_id for these queries any way. He is using player_state.

 

The code above looks like it is the write data to influxdb code from varken.

 

Sorry, my apologies, the "_value" is an influxDB v2.X field. In v2 "_value" is the data column associated with the "hash" field.

 

Unfortunately not all of the v1 queries were displayed in grafana when I imported the dashboard using a v2 data source, so I'm not able to see how falconexe has structured the queries using influxql. If you post the query here I'll see if I can work it out.

 

For the stream history, I've done my the following way. I grouped by the "hash" value and pick the last value with some data cleanup. My Flux (influxdb v2) query is below.

 

Stream Log (Detailed)

Spoiler

// groups by hash value and passes data from last time value.

from(bucket: ${bucketVarken})

  |> range(start: -7d)

  |> filter(fn: (r) => r["_measurement"] == "Tautulli")

  |> filter(fn: (r) => r["session_id"] != "") //data cleanup

  |> group(columns: ["_value"]) //groups by the "hash value"

  |> last()  //gets the  last value i group

  |> filter(fn: (r) => r["username"] != "") //data cleanup

 

 

Edited April 7 by Jufy111

[MrCrispy]

is there a simpler version of this? and does stream history only work with Plex streams?

[Bili]

On 4/5/2024 at 8:52 AM, Jufy111 said:

谢谢，我想我快完成了。不过，绝对可以使用您的帮助进行测试。我也有几个问题要 @falconexe too.

I am also using influx2, and I spent several hours converting only a few queries. It is really difficult to complete them all. I wonder if your project has been released? Maybe I can help a little bit.

[BVD]

On 4/8/2024 at 2:54 AM, MrCrispy said:

is there a simpler version of this? and does stream history only work with Plex streams?

 

You can always trim down the number of panels to suit your specific requirements. It is Plex only - if someone wanted to pull emby/jellyfin/(etc), you'd need to have a separate set of requirements since the API calls necessary to collect the data (and the applications which are responding to those calls) would be separate for each.

 

Kind of an aside, but I think it's worth at least mentioning for anyone coming in to this fresh -

 

I'd be willing to bet @falconexe has invested several hundred hours in to this over the years, so even if it only takes someone ~5% of the total "from-scratch" creation time to tweak it to match their specific combination of server + storage + containers + share config + (all the things)... Most should probably expect a needing a handful of evenings of dedicated effort to get the dashboard up to a fully functional state.

 

(extra random thoughts, hoping 'spoiler'-ing it keeps this post from being too long unless someone clicks on it lol)

 

Spoiler

The UnRAID Ultimate Dashboard is *NOT* meant to be plug-and-play. While it's an extremely well built and meticulously designed data visualization 'system' (for lack of a better term), and has been made in such a way that one can deploy it on virtually any server with a fraction of the effort it took to build from scratch, that fraction of effort can still be quite significant.

 

While all the most difficult parts have been sorted by the author, there's unfortunately no getting around the fact that every user's deployment is unique, and getting everything installed and running is certainly the easiest/quickest part of this, and only the first step.

 

I don't want to discourage anyone who's interested, as there's HUGE value here, not just for the metrics/troubleshooting aspect, but in learning Grafana as well. More just hoping folks can step into this with their eyes open, knowing it's going to take some time, and hopefully not getting discouraged if it's still not quite working the way they want even after 4-5 hours being sunk in to it 👍

 

[Jufy111]

4 hours ago, Bili said:

I am also using influx2, and I spent several hours converting only a few queries. It is really difficult to complete them all. I wonder if your project has been released? Maybe I can help a little bit.

 I do have a copy, but I've been waiting on falconexe to discuss some things with him. Here is a the dashboard, that is has most of the queries complete

 

This is still a draft and a lot of the queries still need optimization.

 

Ultimate_UNRAID_Dashboard_v1.7_influxDB2.0_v0.1.json

Edited April 27 by Jufy111

[Bili]

On 4/27/2024 at 7:35 PM, Jufy111 said:

 I do have a copy, but I've been waiting on falconexe to discuss some things with him. Here is a the dashboard, that is has most of the queries complete

 

This is still a draft and a lot of the queries still need optimization.

 

Ultimate_UNRAID_Dashboard_v1.7_influxDB2.0_v0.1.json 796.05 kB · 3 downloads

Great work, very useful to me.

Some datasource uid need to be changed to "${Datasource_Telegraf}"，Such as “VxKS81dnz”.
There are also a few queries that can use aggregateWindow to improve query speed:

|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)

 

[Jufy111]

Glad to here it works for you.

I'm curious to see if any panels didn't work for you. I've only got one system to test on so other system may not work. Let me know if there are any panels that don't work.

As far as query speed goes, yes some queries defiantly need improvement. I'll send you a PM to discuss some things.