atconc

Bumping this - I just temporarily reinstalled Beta 30 for another reason and took the opportunity to check if this was still happening - it is, very slow web ui and apps in docker containers, extremely high cpu usage for shfs while this is happening. Reverting to b25 again solves this for me. Any idea what's going on? Trying to avoid ending up with this issue on the next stable

I rolled back to beta 25 and the apps in docker containers are noticeably significantly more responsive again so there's definitely a regression in beta29 and beta30 for me. Happy to help troubleshoot, let me know if there's anything I can do.

Is there anything else I can try to troubleshoot this (issue with very slow apps in docker and high SHFS cpu use)? or if i roll back to beta 25 where I didn't have this issue will the new partition layout be recognized or will I have to rebuild my cache again?

Here you go, thanks for looking. bb8-diagnostics-20201014-1637.zip

Has anyone else also noticed slower performance from cache pools since the partition layout changed in beta 29? This is really noticeable for me using applications in docker containers - plex loading thumbnails in the web interface, tautulli loading history, sonarr v3 showing it's witty quotes while it loads are all noticeably much much slower since repartitioning with beta 29 - Sonarr takes about 15 seconds to load now when it was a second or 2 before and I don't remember any noticeable lag loading the plex thumbs or tautulli history before this change. At first I thought that the combination of the write amplification issue and several rebalances had finally killed my 2 480gb sandisk ssds (they had 3+years power on time and showing several hundred bad blocks) so I replaced them with new samsung pro drives but haven't seen any improvement. I also tried switching from a docker img file to a directory on the share which also doesn't seem to help. I also noticed that there's a lot of SHFS processes that often are using the most cpu of anything, one has had 48hrs Cpu time on a machine with 6 days uptime (filtered htop screenshot attached) After reading the earlier posts in this thread I was wondering if this might be related. My app data and docker folders are both on cache only shares if that matters. Array is single parity with 5x8tb and 5x3tb, cache is 2x500gb sata ssds in btrfs raid 1, I also have a single 3tb hd defined as another pool and an old 128gb ssd as an unassigned device.

The work around seems to have worked for me - the only difference being in step 2 no balance was triggered. I carried on and everything else worked as expected and my partitions now show start at 2048.

I tried again, this time no balance seems to have been triggered at all - I can switch to the mover way and recreate the cache contents but wanted to help troubleshoot this first. Diags attached, let me know what else I can do. Edit to add - I'm using the Nvidia build in case that's relevant here. bb8-diagnostics-20201001-1605.zip

Both return 0 I'll try removing and re-adding again and post drags

How do I check it the partition alignment actually worked / happened? I tried the 2nd method (removing the drives from the pool 1 by 1) but when I re-added them it didn't seem that a balance was automatically triggered so I manually triggered one. from my quick bit of research this seems to be a way to check but I'm not sure how to interpret the output:

I've been running unraid on this hardware for a few years and it's been mostly rock solid but recently I've been getting some lockups where the system becomes completely unresponsive (stops responding to ping, ssh sessions die, apps in docker containers stop working). I haven't noticed a particular pattern for when these crashes occur, to have idea whether this is load related for example, timingwise I think this corresponds with upgrading to 6.8.0. I turned on the syslog writing to usb feature and caught the logs which i have attached (edited to redact my email address from notifications but otherwise untouched). I just saw 6.8.1 is out with some kernel updates so have upgraded to see if things improve but in the meantime thought I'd also post here with the syslog. There's a couple of "kernel bug error" lines and lots like: rcu_sched self-detected stall on CPU Hardware is: Gigabyte Z97-D3H-CF motherboard 32gb DDR3 ram CPU Xeon e3-1230 LSI SAS controller Array drives 4x 8tb wd red , 6 x 3tb wd red - single parity with one of the 8tb reds Cache 2x 480gb sandisk ssds 2 other drives unassigned (2tb wd green and 128gb crucial ssd used as scratch drives) I'm running various apps in docker: Plex, sonarr, radarr, nzbget, unifi controller etc Everything is running headless but I do have a graphics card arrived today so I can check bios settings etc if this persists. syslog-email-removed.log