grumpy

@Gazeley You back yet? Have you built your server? I'm in need of the same and read your thought process and liked it; very complete. Will probably copy a few of those thoughts for my own rebuild. I like VM's and dockers, so I have struggled with what platform to go with. Also availably of MB's that I think I want. Not sure I would use Pro Art in a server more aimed at PC (I have one with 13900K that had the microcode problem)

So drive order does not matter because it is saved on the USB flash drive. No matter how I plug in to the new MB. I'm understanding this correctly? Also I'm going from AMD to Intel on the donor board, so Unraid should not care?

My 2950 Thread Ripper has passed on. My it rest in peace. Well during a 4 day backup it decided it had enough, and completely died. MB or CPU not sure which error was C2 on MB. Got it to do 16 I think but no go and then back to C2. (using Synology Business active backup) So time for an upgrade as far as CPU and motherboard and memory go. But unsure what to replace with, my use case is large storage and multiple VM's and docker containers. I like testing out new things that catch my interest. Any suggestions of what to get would be appreciated. For now I need to install a underpowered MB, CPU, Ram just to get it running to finish backup but the disk picture I have does not include the two newer 24TB HD's. So unsure of order. And do not want to wipe out a data drive as mistake. So how would I go about trying to put them in the correct order?

There is no session _value from Varken to Influx, unless that is an Influx built-in. But that is moat as @falconexe is not using session_id for these queries any way. He is using player_state. hash_id = hashit(f'{session.session_id}{session.session_key}{session.username}{session.full_title}') influx_payload.append( { "measurement": "Tautulli", "tags": { "type": "Session", "session_id": session.session_id, "friendly_name": session.friendly_name, "username": session.username, "title": session.full_title, "product": session.product, "platform": platform_name, "product_version": product_version, "quality": quality, "video_decision": video_decision.title(), "transcode_decision": decision.title(), "transcode_hw_decoding": session.transcode_hw_decoding, "transcode_hw_encoding": session.transcode_hw_encoding, "media_type": session.media_type.title(), "audio_codec": session.audio_codec.upper(), "audio_profile": session.audio_profile.upper(), "stream_audio_codec": session.stream_audio_codec.upper(), "quality_profile": session.quality_profile, "progress_percent": session.progress_percent, "region_code": geodata.subdivisions.most_specific.iso_code, "location": location, "full_location": f'{geodata.subdivisions.most_specific.name} - {geodata.city.name}', "latitude": latitude, "longitude": longitude, "player_state": player_state, "device_type": platform_name, "relayed": session.relayed, "secure": session.secure, "server": self.server.id }, "time": now, "fields": { "hash": hash_id } } ) influx_payload.append( { "measurement": "Tautulli", "tags": { "type": "current_stream_stats", "server": self.server.id }, "time": now, "fields": { "stream_count": int(get['stream_count']), "total_bandwidth": int(get['total_bandwidth']), "wan_bandwidth": int(get['wan_bandwidth']), "lan_bandwidth": int(get['lan_bandwidth']), "transcode_streams": int(get['stream_count_transcode']), "direct_play_streams": int(get['stream_count_direct_play']), "direct_streams": int(get['stream_count_direct_stream']) } } ) self.dbmanager.write_points(influx_payload)

Is there a way to limit the display of duplicates in Grafana? I realize it is a new input from Varken but it is the same data over and over. Stream Log (Detailed)

Now after watching the panel for 16 hours, and correcting some issues for myself I have noticed some other issues that go beyond my understanding. Stream Log (Overview) misses some entries. In music playing from Plex it can keep up sometimes, skip a couple of songs, or even skip 10's of songs. Probably not noticed during movies or tv shows; I do not know as I haven't made it that far during testing. Stream Log (Detailed) Keeps up even though there are delays, to be expected Plex -> Tautulli -> Varken -> Influx, same with Current Streams. During Music play percentage is of no use, while the song can have 72% remaining when in fact the next song has started playing. Even though most of my posts seem like complaints I really like to thank @falconexe for all of his hard work, it is functional and visual pleasing. I know I wouldn't have been able to do this on my own accord.

@PJsUnraid I do not believe it is Varken causing the issue, I think it is a timing issue. If you look at the influx database you will see all of plex data being updated constantly; not sure if it is 30sec or not but those metrics you are missing are done hourly. So you no longer see N/A go to the panels that show Total 3 dots on top right drop down - edit now in edit mode for the panel top left side click on query options Relative time change to 62m or 1h top right of page Apply to close panel Now it will show the number in a persistent state till it changes on the next query.

I'm an idiot; ok things I do not know. I needed to add a varken data source connection, not just telegraph now I see Plex stuff; not everything yet, but at least moving forward.

@Jufy111 Thank You, I'm looking forward to your example, as far as the smartmontools. I think it is installed with the docker image post argument; which is how I did it. /bin/sh -c 'apt update && apt install -y smartmontools && apt install -y lm-sensors && telegraf' --user 0 I do get the panels: Drive SMART Health Overview, Drive Life

So now that I have all components running as expected (I think) I have more questions. Did you setup a Varken database in influx as your Dashboard variable tab picture suggests you did. As the guides I followed does not suggest that but to use the Telegraph database. I have no Plex (no media) information showing on my Dashboard but as far as I can tell Telegraph, Taututlli and Varken are all processing the data as seen through the log of each. I do not have IPMI so how do I get that information? (TEMPS) Array Drive, System Temps/Power, CPU, Ramm DIMM, Fan Speeds I use NUTs(slave) for the UPS; the information is shown in Uraid but not in the dashboard. How to get that? SSD Temp, Life, Power On, Lifetime reads/writes, are missing.

My original post was poorly worded, I had Varken installed; but it would not start. I got past that by changing owner of the config.example from root to nobody, why it interfered with the required config.ini beats me. Every time I edit the config and remove some stuff when I restart it replaces the removed stuff. There needs to be an updated How to! Which is true of many contributions.

My experiment was in an array with each disk being a standalone ZFS drive. Hard to do a pool with 3*3TB, 1*2TB, 2*1TB I could do a pool with the original 4*16TB plus the 2*16TB, just not sure of the game plan needed and how to move the original data around. Obviously in place will not work as there would not be enough free HD's for any kind of a real pool.

Current Unraid Server a 10 core Intel® Xeon® W-2155 CPU @ 3.30GHz with 16 GB ram, 4 16TB spinning rust and 4TB SSD cache Moving to: MB X399 AORUS XTREME-CF, AMD Ryzen Threadripper 2950X 16-Core @ 3500 MHz with 32 GB ram, 2 new 16TB spinning rust plus 4 current 16TB spinning rust, current 4TB SSD cache could add another 2TB SSD to cache Both 10GB Only reason of changing hardware is the case of the Dell does not support more drives. Also runs hot and does not support 6.12 Unraid (last I checked). I tried ZFS on the replacement board Cpu and it works; but I'm not impressed. If I run more than one task to drives I get CPU_IOWAIT errors. Caused by memory shortage (1/8) I'm sure as the CPU has room to spare and no other tasks running. So I do not understand why ZFS is so in demand will it really be a benifit or just more headaches. I'm all for cutting edge but do not see the benifit. My usage is media/document server with VM's and Docker containers running 24-7. Is Mover really that slow, 9 hours about; in each direction: to move about 30GB from ZFS SSD to XFS HD, ZFS HD to ZFS SSD Unbalance is slow but way faster then this. Network copy starts of great as expected to array but during the move it eventually times out at 0 bytes from Windows machine. So do I stay with the devil I know XFS spinning rust, BRTFS SSD cache or move to ZFS Full or a Hybrid? I would like to achieve greater transfer speeds more akin to my current Synology which uses BRTFS. If if I move away from XFS how do I go about that?

I have had no luck installing Varken; I think it may be a dead project. Github is 4-6 years old, discord link both at Grafanna and GitHub dead, Wiki 304... Any suggestions?

Are you not using a SSD cache drive for file transfers? You may want to read my quest to get faster writes. Simple answer @JorgeB suggested Turbo Writes and Cache drive makes it livable. Turbo Drive It improves writes and reads. My Post

I do not think your issue is the same as the OP. You are doing a backup with encryption and the highest level of deduplication that Duplicacy can do. I would think if you increased the resources to this container it would speed it up. (no experience) I was amazed how long the new PC I setup took to backup to Synology Active Backup compared to the original PC it was replacing. This was caused by the 90% CPU usage and transfer rates were dismal even though the Synology transfer rates are faster then my Unraid on testing file transfers.

solution

@JorgeB I marked it as a solution as this maybe the best I can expect from Unraid. Synology is still faster but they write to all disks sharing the love evenly. I have no cache in the Synology. Test: 151 GB dir with 10 video files is my test now. Before turbo write MB/s 180 read 75 write After turbo write 230 read 220 write Cached enabled (4 TB SSD) 490 Read 380 Write Iperf-3 7.79 Gbits/second

Its been awhile since I have done tech support. (mind is leaving me) But until you can figure what layer is causing you issues you will not be able to fix it. Unraid is many layers and you need to identifie the problem layer, so time is needed to do this or you may introduce other issues in trying to guess what the issue is. Let Unraid run with just the array started and see what it is doing, no issues then move on to VM's no issues then start a docker container ... rinse and repeat till you find the issue. Then you can ask for more direct help or a light bulb turns on and you figure it out yourself. I do understand not wanting to stop everything, I have been working on my own Unraid issues and time has found me a solution. Somebody smarter than me too. Personally I would let the parity job finish, unless all your data is backed up or of no consequence if you lose it.

Why not stop everything, and let Unraid run by itself with the array loaded. Then after turn on the parity and let it finish, then docker containers one by one to see where the issue is happening. Then finally the vm's.

@JorgeB Unraid without cache (NC), Write 14MB - Read 598MB is the speed directly to the array. (Which would be 1 drive bypassing cache but not parity) But I do not know what or how to enable turbo speed!

@Tzekang You are correct that there is a LARGE penalty for using parity, and of course the cache drive will improve transfer speed up to a point. But I just did another test between PC, Synology and UNRAID (with and without cache) and I'm still not satisfied with the results. Instead of using a benchmark that I do not understand I went with a real world copy and paste. A directory containing 12 video clips 4K mp4. And the results were better but Synology (with no cache, weak cpu, less memory) still kicked Unraid's butt with and without cache. The whole reason to use Unraid over Synology is power. (Yea for some it is dollars) Any CPU any motherboard allows you to do more than just a NAS, which was my use case. But the NAS part is not performing up to par, read and write speeds are below my needs/expectations. If Synology and maybe Qnap can do it, why not Unraid? I thought of changing hardware, but in reality the Xeon CPU rarely goes over 10%, 32 GIG of ram is more than enough and the 7200RPM enterprise class HD are fast enough. The problem is software, somewhere there is a bottle neck. In the short term I will enable cache (4TB SSD) on my Media share and get some improvement. I just haven't done that as SSDs have a habit of no warning dying and I download a bunch of ahhh! ISO's. Maybe I should put a cache drive in the Synology just to see what it really can do, but the CPU, memory is it weak point. The Synology just mirror the Unraid data currently.

Nobody have any ideas/suggestions?

Diagnostics File tower-diagnostics-20230721-1452.zip

Original post Not sure with almost 24 hours and 70 views with no response if others are having the same issue as me not seeing the text unless you click the button Reveal hidden contents. I have no idea why it is hidden. But here is the text portion of the post. I have just discovered that my network slowdown is my Unraid. (I have been ignoring for long time) Using BlackMagic Speed test between local and the 2 NAS's really shows the poor performance from UNRAID to the point I'm reconsidering my decision to continue with Unraid. Just put in a second Synology and be done with it. But first I thought I would ask for help/pointers in correcting this major slowdown. There are 4 images. Local PC Samsung 860 EVO M.2 2TB, Write 478MB - Read 520MB Synology populated with 8 Iron Wolf 10 TB, Write 564MB - Read 602MB Unriad share with cache and share Write 109MB - Read 588MB Unraid without cache (NC), Write 14MB - Read 598MB 4 (1 as Parity) Seagate 16TB HDD Exos X16 7200 RPM 512e/4Kn SATA 6Gb/s 256MB Cache 3.5-Inch Enterprise Hard Drive (ST16000NM001G) Cache Drive Western Digital 4TB WD Red SA500 NAS 3D NAND Internal SSD The Synology is extremely close to the local speed (beating it) But the Unraid is extremely slow Iperf3 is about 7 GBs to Unraid ; not optimum but not the problem. Unraid system: Unraid server Pro, version 6.11.5 Motherboard: Dell Inc. 002KVM, Version: A01, s/n: /GH1V9Z2/CNFCW0098C00GC/ Processor: Intel® Xeon® W-2155 CPU @ 3.30GHz HVM: Enabled IOMMU: Enabled Cache: L1-Cache = 0 KiB (max. capacity 0 KiB) L2-Cache = 0 KiB (max. capacity 0 KiB) L3-Cache = 0 KiB (max. capacity 0 KiB) Memory: 32 GiB DDR4 Single-bit ECC (max. installable capacity 3072 GiB) DIMM1: Hynix HMA82GR7JJR8N-VK, 16 GiB DDR4 @ 2666 MT/s DIMM2: Hynix HMA82GR7JJR8N-VK, 16 GiB DDR4 @ 2666 MT/s Network: bond0: fault-tolerance (active-backup) eth0: 10000 Mbps, full duplex, mtu 1500 eth1: interface down Kernel: Linux 5.19.17-Unraid x86_64 OpenSSL: 1.1.1s P + Q algorithm: 23849 MB/s + 44160 MB/s