DaveDoesStuff

Members
  • Posts

    71
  • Joined

  • Last visited

Everything posted by DaveDoesStuff

  1. Hmm, didn't know that. To be fair it would be counter-intuitive for me to assume it's a load of crap lol but thank you for the info. So obviously there *was* an issue that lead to these two devices racking up 58.5TB in writes in just 4 months since I began using them...and it seems like possibly something I did either earlier or in the last 72 hours has actually fixed it but I was focusing on metrics that don't actually represent the TBW or possibly even misrepresent it. Now I'll just have to re-enable Qbtorrent and monitor it closely, then if that checks out Duplicati (which I feel is possibly the culprit) and take some additional actions depending on the outcome. Thank you for the fresh set of eyes on this Jorge! I had complete tunnel vision until you came along lol! Do you use any particular tools or commands on your own setup(s) to monitor this kind of thing? Or anything you would recommend over what I've utilised in this thread?
  2. So after running overnight... Cache 1: 114,313,326 [58.5 TB] Cache 2: 114,312,943 [58.5 TB] Thats a delta of roughly 35,000 on each drive. No this does not seem like a lot...but unRAID main reporting 3.9 million when it was around 2.6 million yesterday evening is still pretty bad so a server that was essentially only running a handful of low IO dockers and a pfSense VM.
  3. I believe what I posted is actually for the total time this "server" has been in operation/powered on. That's probably a bit useless come to think of it, I picked it up in a bug report Currently the 2 cache drives report: Cache 1: 114,276,540 [58.5 TB] Cache 2: 114,276,147 [58.5 TB]
  4. Yeah for sure its not adding up. I've got Qbtorrent disabled now, at time of writing this the TBW is: Let's see where it sits tomorrow morning. Thank you for the assit by the way!
  5. Nothing beyond the standard Radarr/Sonarr > Qbtorrent > Plex setup used by many. Here is a screenshot of QB at present along with a fresh iotop, the QB screenshot effectively represents my last 72 hours of torrenting. As you can see, not a ton. From what I've seen on the forums here I'd say it's a standard amount even. EDIT: I've just noticed that iotop, despite being the same window running for the last 6 hours seems to be giving inconsistant stats...haven't used it outside of troubleshooting this issue so not sure if thats normal. For example in the screenshot in this post taken at roughly 6hrs run time, compared to the previous 4hrs one the entries at the top for highest write are missing. EDIT2: Added share settings to the OP.
  6. That's what I've been thinking...but not sure how to narrow it down further, or possibly go beyound whatever bounds iotop has. Here is an SS from 4 hours, not much difference.
  7. Hi all, I've taken so many different actions to try and resolve this issue that it's hard to collate everything and to know where to start but here goes... (Diagnostics attached) ...so essentially since 6.8.3 I've been struggling with the excessive cache writes issue. Due to personal circumstances I was not able to do any meaningful troubleshooting until the January or so after moving to 6.9.0 It should also be noted that I not sure if I was experiancing excessive writes prior to December of 2020 when my SSD Cache Pool was 4x 250GB SATA SSDs in Parity (BTRFS) mode, but I have since upgraded to 2x WD Blue SN550 1TB NVME (WDS100T2B0C) where the issue first came to my attention or possibly even started. Since they've been installed (to this date) they have 114,252,802 [58.4 TB] data units written each. I've poured over every forum/reddit post about this issue and tried everything in them except for moving my cache over to XFS/Unencrpyted as I strictly require the encryption and redundency. I know I'm making life hard on myself with this...but I need it and prior to I'll have more specifics on writes below. My Specs are: Ryzen 2700X TUF B450-PLUS GAMING 32GB RAM (4x Team Group Vulcan Z T-Force 8GB DDR4 3000MHz) Silverstone Essential 550W 80 Plus Gold nVidia Geforce GT710 1GB My Array Configuration: Share Settings: After upgrading to 6.9.0 I've: Reformating cache pool to 1MiB partition layout as described by limetech HERE. Switching from Official Plex docker to Binhex Plex Pass container. Tried toggling various dockers/vms on and off to find a culprit, no joy. After upgrading to 6.9.1 I've: Switching Docker to use a directory instead of an image. Moved Docker back to a fresh btrfs image after the above didn't work. Tried toggling various dockers/vms on and off to find a culprit, no joy. After upgrading to 6.9.2 on 18/04 I've: Moved docker back to using a directory instead of an image again. Disabled my duplicati docker and anything else I don't utilise often (bazarr, krusader...etc...) Disabled all my W10 VMs, only a pfSense VM running now. Tried toggling various dockers/vms on and off to find a culprit, no joy. Following my upgrade to 6.9.1 and subsequent actions I let it run for 1 month without my interfering and in that time the cache had over 60 million writes...and the USB key failed. Which is actually what precipitated my upgrade to 6.9.2 this past Sunday and another round of troubleshooting this issue. TBW according to the cli: cat /etc/unraid-version; /usr/sbin/smartctl -A /dev/sdb | awk '$0~/Power_On_Hours/{ printf "Days: %.1f\n", $10 / 24} $0~/LBAs/{ printf "TBW: %.1f\n", $10 * 512 / 1024^4 }' version="6.9.2" Days: 2261.2 TBW: 81.6 TBW: 217.8 Cache Settings showing alignment: Current Docker Settings: Currently Running Dockers and their mappings: Screenshot of Main 14hrs after a stats resest (after moving docker back to directory mode from img): As you can see from the above screenshots, the writes are excessive. This is despite the cache re-alignment, move back to using docker in directory mode and the disabling of the only docker that should be doing large writes because of how I have it set up (duplicati). The only VM I'm running is for my pfSense and I've disabled any intensive logging I had running there. 14 hours before the time of the screenshots (after moving docker back to directory mode) I cleared the stats on Main and left an iotop -ao running on my laptop. Unofrtunately the laptop decided to reboot for updates (damn) during this time so I don't have the output from iotop but as you can see in that period the cache had over 2.3 million writes with not a whole lot going on/running. I've run another iotop -ao for the last 3hours before writing this post (without changing anything else) to give some sort of idea of what that overnight output would look like if I had it: I should also mention that yesterday I ran each docker 1 by 1 (with all others disabled) and ran iotop -ao for 10 minutes each but the results were inconclusive. As I've mentioned I have to have the cache encrypted and redundant. I do understand there is a write aplificiation expected from the encryption etc...but it shouldn't be this high. I've tried to be thourough and include as much current information and background as possible. But if I've missed anything please don't hesitate to ask! I can't help but feel like I'm missing something...any help/different perspectives would be very very welcome at this point. ibstorage-diagnostics-20210420-1106.zip
  8. I was able to get into the webUI and do it. However as of the 6.9.1 update this is broken for me again. No combination of installing and re-installing will fix it. So I've simply stopped using GUI mode completely.
  9. That's actually a really cool website, bookmarked. It never occured to me that the media, coupled with lack of 265 support was the problem...but it does make total sense thanks for the steer. I'll try a different format and lower the quality and report back! EDIT: So the HW transcoding kicks in when playing this second show and transcoding to h.264...but it seems to be using CPU not GPU...or am I misreading this? Hmm, does the source also have to be h.264 or could a GT710 transcode HVEC Main 10 to h.264 at all? I clearly need to improve my knowledge in this area
  10. Firstly, great plugin/work! Unfortunately I'm having some issues getting plex transcoding to work on my GT710. The very first time I installed the plugin and set everything up it worked fine until my next system restart. Since then every combo of troubleshooting steps I've tried failed to get it to ever work again. I'm running binhex-plexpass with a lifetime plex pass. Rest of the info is in the screenshots, I've tried to include any/everything I've seen in the other posts in this thread that seems to be of help. Sorry if I've overdone it It should be noted I have no errors in the systemlog or the container log for plex. None whatsoever. EDIT: For clarity I'm doing the PCI override because I have a dual intel nic for a dedicated pfSense VM, so it's not really optional.
  11. This was resolved by installing the Nvidia Driver plugin (and possibly limetechs underlying nvidia driver stuff).
  12. Hi team, Upgraded to 6.9.0-rc2 last night after I was forced to look into some issues with a parity drive (unrelated) and decided I was sick of the warnings about the mover tuning plugin being incompatible with stable branch (I'm easily annoyed). I rely on booting unRaid in GUI mode since I run a pfSense VM and 2 dedicated NICs as my home router via unRaid...and obviously this means if the array is down there is no webUI...hence my need for a working GUI mode on boot. After upgrading however all I get after going through the boot sequence is a black screen with a blinking cursor in the top left instead of the GUI. I've tried various ways of troubleshooting this myself such as: Enabling CSM Enabling CSM but allowing UEFI only/first Disabling CSM and forcing UEFI only Reseating my GT 710 BIOS defaults and re doing my settings one at a time GUI safe mode (issue persists/no change) I can boot in normal OS mode (cli) no problem, I was able to work around my pfSense going down by plugging a laptop directly into the unRaid NIC (pfSense NICs are on a seperate card) on the mobo and accessing the webUI but this isn't a practical/long term solution. Interesting to note that while the "system profiler" tool does not seem to detected the GT 710 that it appears as an option for GPU if I go into the create VM wizard (currently no VM is using the GT 710 so there shouldn't be anything "claiming it" and it has not been isolated in anyway). ibstorage-diagnostics-20210215-1138.zip
  13. A restart and a BIOS update seems to have fixed this. After running benchmarks it looked like the controller on my ASUS TUF B450 was having issues, hence why I decided to try a BIOS update.
  14. The original diagnotics I posted were from straight after the crash/shutdown and it just occurred to me that a recent set might be more useful (attached). ibstorage-diagnostics-20210112-0922.zip
  15. So yesterday around 12:30pm (ish, should have noted it down) my entire unRAID server went unresponsive. It was apparent since I run a 2x GBE card on it with pfSense...kinda easy to take notice when your internet goes down with it lol. Went to my home rack and tried to login via the GUI, which was frozen and wouldn't accept any inputs. So made the tough call to power off. Upon powering on the array started smoothly, all dockers, all vms on...no problems/everything seemed fine. Unfortunately there was no way for me to retrieve my syslog prior to rebooting 😪 However as of lastnight/this morning my CPU (AMD 2700X has been going wild doing the parity check, so much so that I can't stream anything from plex. Based on this I powered off or paused most if not all VMs/Dockers/etc that could be writing to the array. CPU usage/temps went down but no improvement on the plex front. It was then I noticed that the parity check has been running for over 18 hours with 2+ days estimated remaining...speed crawling along at around 16mb/sec. Pretty big yikes there. Hit pause on the check, turned all my dockers and resumed and speeds stayed the same. Decided to take this as a bad omen and reach out for some second opinions on possible causes here. Diagnostics attached. It should be noted that there is 1 drive with some SMART errors and its "Parity 1" a 4TB WD drive, it has a Reallocated sector count of 8 and a Reported uncorrect count of 24. However when I manually run the extended test it comes back as having no errors...which is strange. This has been the case for almost a year and I've never had parity problems before after a shutdown/reboot be it clean or unclean. I'm not saying that's not a candidate for the cause...just saying that it seems unlikely when its been like that for so long. Any and all help/pointers appreciated. ibstorage-diagnostics-20210111-1307.zip
  16. Thanks for the analysis and workaround, I appreciate that you provided it even tho its a UD issue.
  17. Curious what you make of this @doron, I've been having an issue whereby my UD NVME drive won't auto mount on system startup after a reboot. I posted over in the UD thread and supplied my diagnostics and dlandon suggested the issue below is my problem. Seems a bit coincidental right? What do you think? Possible sidefffect of the sript or unrelated and a 6.8.3 issue? (dlandon didn't know I'd run a script to replace my keys)
  18. Of course, thank you (didn't want to bombard you with diagnostics straight away incase it was a common issue my searches had just failed to find). Attached. ibstorage-diagnostics-20201019-0854.zip
  19. First off, awesome plugin! Been using it for close to a year now without issue. Really, really useful. Currently I'm using it for an NVME drive to run my VMs (and a couple of dockers) off of. Everything was fine until I changed my array (and UD) encryption to a keyfile, now the array starts on reboot but the UD NVMe drive will no longer auto mount (it's unlocked successfully via the keyfile but just isn't mounted). Has anyone come accross this before and/or have a fix?
  20. So it seems that this issue was caused by a pending restart/update after I installed unraidDVB and it patched the kernal. On a whim I decided to try and re-enable wireguard after I did the restart today and it worked straight out of the box! For any other PIA people getting wireguard working with a NextGen server configured for it has restored my PIA speeds to the point where I can max out my connection again. Although it does bounce around a lot.
  21. Bizzarre indeed! I can clearly see in the CMD output that when I toggle it and save the changes it is indeed running with it set to true (which you probably confirmed in the logs also): root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='binhex-qbittorrentvpn' --net='bridge' --cpuset-cpus='1,9' --privileged=true -e TZ="Europe/London" -e HOST_OS="Unraid" -e 'VPN_ENABLED'='yes' -e 'VPN_USER'='XXXXX' -e 'VPN_PASS'='XXXXX' -e 'VPN_PROV'='pia' -e 'VPN_OPTIONS'='' -e 'STRICT_PORT_FORWARD'='yes' -e 'ENABLE_PRIVOXY'='yes' -e 'WEBUI_PORT'='8080' -e 'LAN_NETWORK'='10.1.1.0/24' -e 'NAME_SERVERS'='209.222.18.222,84.200.69.80,37.235.1.174,1.1.1.1,209.222.18.218,37.235.1.177,84.200.70.40,1.0.0.1' -e 'ADDITIONAL_PORTS'='' -e 'DEBUG'='true' -e 'UMASK'='000' -e 'PUID'='99' -e 'PGID'='100' -e 'VPN_CLIENT'='wireguard' -p '6881:6881/tcp' -p '6881:6881/udp' -p '8080:8080/tcp' -p '8118:8118/tcp' -v '/mnt/user/Downloads/':'/data':'rw' -v '/mnt/disks/UD_SSD/AppData/binhex-qbittorrentvpn':'/config':'rw,slave' 'binhex/arch-qbittorrentvpn' 9b76b47381e4fb14f762b1b87757888056b7aed2e9e4ae044bfc7113e0808968 Would there be any value to just blowing away the whole thing and re-installing from CA Apps? It's not ideal with 75 torrents in my queue so I was holding this option way back in reserve Hmm, not likely to be related (or possibly already addressed) but I found someone with a similar wireguard issue from April from a different container...might be something to it. But probably not https://github.com/linuxserver/docker-wireguard/issues/17
  22. The slider was still set to on, I toggled it on/off then restarted twice and no change. Same issue. I then toggled it off and re-added the extra parameter --privileged=true and the issue persists. I followed those instructions to the letter, log file attached. Info removed/replace was username, pass and the wireguard public and private key (replaced with X's). supervisord.log
  23. Fair enough. The plot thickens however: 2020-10-14 11:31:48,391 DEBG 'start-script' stderr output: [#] ip link add wg0 type wireguard 2020-10-14 11:31:48,393 DEBG 'start-script' stderr output: RTNETLINK answers: Operation not supported 2020-10-14 11:31:48,395 DEBG 'start-script' stderr output: Unable to access interface: Protocol not supported 2020-10-14 11:31:48,395 DEBG 'start-script' stderr output: [#] ip link delete dev wg0 2020-10-14 11:31:48,397 DEBG 'start-script' stderr output: Cannot find device "wg0" 2020-10-14 11:31:48,397 DEBG 'start-script' stdout output: [warn] WireGuard interface failed to come 'up', exit code is '1' Thats with the default wg0.conf, I originally had input france.privacy.network:1198 with which it also failed before deleting the conf and restarting as I assumed I had messed it up. The above in the logs was preceeded by: 2020-10-14 11:30:18,310 DEBG 'start-script' stdout output: -P INPUT DROP -P FORWARD DROP -P OUTPUT DROP -A INPUT -s 172.17.0.0/16 -d 172.17.0.0/16 -j ACCEPT -A INPUT -i eth0 -p udp -m udp --sport 1337 -j ACCEPT -A INPUT -i eth0 -p tcp -m tcp --dport 8080 -j ACCEPT -A INPUT -i eth0 -p tcp -m tcp --sport 8080 -j ACCEPT -A INPUT -s 10.1.1.0/24 -i eth0 -p tcp -m tcp --dport 8080 -j ACCEPT -A INPUT -s 10.1.1.0/24 -d 172.17.0.0/16 -i eth0 -p tcp -j ACCEPT -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -i wg0 -j ACCEPT -A OUTPUT -s 172.17.0.0/16 -d 172.17.0.0/16 -j ACCEPT -A OUTPUT -o eth0 -p udp -m udp --dport 1337 -j ACCEPT -A OUTPUT -o eth0 -p tcp -m tcp --dport 8080 -j ACCEPT -A OUTPUT -o eth0 -p tcp -m tcp --sport 8080 -j ACCEPT -A OUTPUT -d 10.1.1.0/24 -o eth0 -p tcp -m tcp --sport 8080 -j ACCEPT -A OUTPUT -s 172.17.0.0/16 -d 10.1.1.0/24 -o eth0 -p tcp -j ACCEPT -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT -A OUTPUT -o lo -j ACCEPT -A OUTPUT -o wg0 -j ACCEPT Not sure if this helps but my allocations for the container are: