Jump to content
CowboyRedBeard

6.8.3 Disk writes causing high CPU

44 posts in this topic Last Reply

Recommended Posts

Posted (edited)

Not sure where to even start looking. But now if I for example have sabnzdb in docker download a file of say 8G it puts my system into pretty heavy CPU usage and from there, a lot of the other services experience interruption. I just recently upgraded from 6.7.0 to 6.8.3, I was hoping that would fix it but did not. The system is dual E5 2690v2 @ 3Ghz ... 20 cores with 128G of RAM so it's decently powered.

 

What am I looking at here? What information is helpful in tracking this down? I assume it's happening because of operations happening on the cache drives, which are SSD connected on SATA3. At times I can see writes in excess of 500MBs

 

image.thumb.png.e48b0618ceed8e72fcef7cc87abbe40b.png

 

 

 

 

 

 

Edited by CowboyRedBeard

Share this post


Link to post

OK... might have found some part of an answer to my question. Not sure why this would have not been needed before. My guess is that it started during my upgrade to 6.7.

 

 

This seems to make some difference when only sab is running. However, if I'm streaming Plex at the same time it then starts to be an issue again. I'm wondering if I need this for Plex too?

 

I'm don't understand what the -c does or how to put the right number there. I also have the Plex transcode folder on the cache drive so I suppose that compounds whatever the issue is here.
 

I have 128G of RAM, so maybe there's a way to use it?

 

I found this thread also:

 

Which suggests adjusting the "dirty ratio" within tips and tweaks....

 

I'm afraid of doing that as I have just read the explanation here:
https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

 

And have no idea what it's saying....

 

This is the kind of action on the cache SSD during this, and worse actually... So maybe this "dirty ratio" through tips and tweaks is the right place to be looking?

image.thumb.png.fd8d4b91b9fa2c786f8f5fe95bd985df.png

 

When this happens nothing seems to work until it's done doing whatever it's doing. Even my pi-hole docker isn't able to serve DNS queries.

 

Help a n00b!

Edited by CowboyRedBeard

Share this post


Link to post

What is your SSD in the cache pool?

Generally a good idea to attach your diagnostics as well (Tools -> Diagnostics -> attach full zip file).

 

It's not uncommon for a failing drive and/or slow drive in cache causing everything to grind to a halt (since docker and VM need IO from cache).

The high CPU usage is a red herring in a sense because it's not really CPU usage but rather high IO wait (think of it like your CPU core freezes while waiting for the storage device to respond).

 

The dirty ratio tweaks and memory parameter are just ways to reduce IO and/or make IO less random and more sequential.

However, they are like putting a bandage over COVID-19 if you have a more fundamental problem with your cache drive(s).

 

Docker -c is the same as --cpu-shares which limit the share of CPU processing power when there are multiple competing processes.

I highly doubt it would have any impact on your situation since high IO wait isn't processing share.

In fact, the reddit you quoted -c=1024 is rather pointless since the default value, if not provided, is 1024.

Share this post


Link to post

I had originally thought the SSD drives might be going bad, and I frequently ran out of cache space anyway... so I upgraded them last night to brand new (qty 2) 1TB Samsung EVO 860 QVO (MZ-76Q1T0). I probably should have mentioned that. The problem existed with the Crucial 500G ones I had in before, but I don't think I had ever seen it happen before upgrading to 6.7 but I'm not certain. I mean, unraid has been so freaking rock solid, and I'm not really a systems/linux guy in my day job to be honest I've not really given it much attention until this started happening. It just sat on the shelf and worked...

 

Attached is diag

 

What else can I provide?

 

 

tower-diagnostics-20200325-0950.zip

Share this post


Link to post

Following this thread because I've had this exact same issue for some time now. I noticed this as soon as I set up my server about 14 months ago. I added a 2nd SSD in hopes of alleviating this issue, but it had no effect.

Share this post


Link to post
11 minutes ago, CowboyRedBeard said:

Attached is diag

Why do you have 40G allocated to docker image? 20G is usually much more than enough, but I see you have already used that much. This is often because the user has applications misconfigured so they are writing into the docker image instead of mapped storage.

 

Also, your system share, where you docker image is stored, has files on the array. So does your domains share, in fact, it is set to have files moved to the array.

Share this post


Link to post
9 minutes ago, trurl said:

Why do you have 40G allocated to docker image? 20G is usually much more than enough, but I see you have already used that much. This is often because the user has applications misconfigured so they are writing into the docker image instead of mapped storage.

 

Also, your system share, where you docker image is stored, has files on the array. So does your domains share, in fact, it is set to have files moved to the array.

Should the system share be set to "only" ?

 

Docker image size, I have no idea... I probably messed that up. Is it bad to have it be 40G?

 

Like I said, I'm not a systems/linux guy... I know enough to be dangerous. I follow directions well though! ;-)

Edited by CowboyRedBeard

Share this post


Link to post
17 minutes ago, CowboyRedBeard said:

Should the system share be set to "only" ?

 

Docker image size, I have no idea... I probably messed that up. Is it bad to have it be 40G?

 

Like I said, I'm not a systems/linux guy... I know enough to be dangerous. I follow directions well though! ;-)

It isn't necessarily bad to have 40G, but that is not the recommended setting and it is very unlikely necessary. Making it larger will not fix the problem with it filling or growing, it will just make it take longer to fill. Only fixing your applications will fix this problem. The recommended setting is 20G, but you have already used all of that.

 

If you have an application configured to write to a path that is not mapped to the host, then that path is inside the docker image. Linux is case-sensitive, so if you specify the path inside the application with a different upper/lower case than that specified in the mapping, then that is a different path and will be inside the docker image instead of mapped storage on the host. Also, any path that is not absolute (beginning with /) will also be inside the docker image.

 

The system share, if it was already all on cache, could be set to only. But mover won't move cache-no or cache-only shares. To get it moved to cache it would have to be cache-prefer. And mover can't move open files, so you would have to disable Docker and VM Manager (not the individual dockers and VMs, the whole thing) in Settings before they could be moved.

 

Same applies to your domains share.

 

If you have appdata (yours is OK), domains, or system shares on the array, your docker and VM performance will be impacted by the slower parity writes, and they will keep array disks spinning.

Share this post


Link to post

OK, so if I understand correctly I should:

 

1) shut down all the dockers and VMs

2) disable both Docker and VM in settings

3) change the shares to "only"

4) invoke mover

5) re-enable docker and VM

6) turn them back on

?

 

Also, I know there's some cleanup of appdata that can be done due to dockers that I no longer have, how else can I see what is eating up all the docker image size? I think this was when I was fighting the image running out of space. That ended up being the Plex docker "/transcode" mapping, which I have since fixed.

 

Do you think this could be the issue with the high IO when Sab is unpacking a file?

Share this post


Link to post
3 minutes ago, CowboyRedBeard said:

3) change the shares to "only"

11 minutes ago, trurl said:

But mover won't move cache-no or cache-only shares. To get it moved to cache it would have to be cache-prefer.

 

Before doing this

4 minutes ago, CowboyRedBeard said:

5) re-enable docker and VM

post new diagnostics so we can see if there is more that needs to be done. Mover also won't move duplicates so if there are files with the same path on cache and array we will have to figure out which to keep.

Share this post


Link to post

Looks like you ignored my response to your earlier step 3.

4 hours ago, trurl said:

mover won't move cache-no or cache-only shares. To get it moved to cache it would have to be cache-prefer.

Try again.

Share this post


Link to post

And I guess my next question would be, does it make sense to create a share for just sab that is cache only. Then sonarr / radarr can move the files off of that share to the media share once unpacking is done, which would also happen on cache but in a "yes" share?

Share this post


Link to post
28 minutes ago, CowboyRedBeard said:

And I guess my next question would be, does it make sense to create a share for just sab that is cache only. Then sonarr / radarr can move the files off of that share to the media share once unpacking is done, which would also happen on cache but in a "yes" share?

Yes.  But, I'd set it as cache "prefer"  that way it stays on the cache drive (unless it fills up in which case it overflows to the array).  If you have it set to "yes", then at 4am (or whenever mover runs), it's going to move everything in the downloads share over to the array, where unpacking will be slow.

Share this post


Link to post
3 hours ago, CowboyRedBeard said:

Next try, did I get it right this go?

system share still has files on disk1.

 

Assuming you really did set it to cache-prefer and ran mover, then it must have some duplicates.

 

Did you actually have any VMs? If not you might as well disable VMs and delete libvirt image. That will simplify getting system share all on cache.

 

Since I am going to suggest you delete and recreate docker image at the more reasonable size of 20G, we won't worry about that.

Share this post


Link to post

I do have VM's... I disabled everything, did prefer, ran the mover, then changed it to only and started it back it up.

 

What do I do now?

 

Ugh... I guess it's time I do some RTFM.... unraid is too powerful, look at what I've created and now I cannot control THE BEAST !  🤣

Share this post


Link to post

OK, let's just go ahead and deal with docker image and see if that is all from system share that is taking space on the array.

 

Go to Settings - Docker, disable, then delete the docker image (don't worry it is easy to get your dockers back just as they were).

 

Then post a new diagnostic.

Share this post


Link to post

Did you actually delete the docker image? There is a checkbox on that same page where you disable dockers.

Share this post


Link to post

Yes, I did the check box and delete

 

root@Tower:~# ls -lah /mnt/disk1/system
total 0
drwxrwxrwx 4 nobody users  35 Feb  6  2019 ./
drwxrwxrwx 8 nobody users 125 Mar 24 22:24 ../
drwxrwxrwx 2 root   root   24 Feb  6  2019 docker/
drwxrwxrwx 2 root   root   25 Feb  6  2019 libvirt/
root@Tower:~# ls -lah /mnt/cache/system
total 16K
drwxrwxrwx 1 nobody users 26 Feb  6  2019 ./
drwxrwxrwx 1 nobody users 82 Mar 25 17:42 ../
drwxrwxrwx 1 root   root   0 Mar 26 09:00 docker/
drwxrwxrwx 1 root   root  22 Feb  6  2019 libvirt/
root@Tower:~#

 

Share this post


Link to post

Looks like you must have duplicates. What do you get with these?

ls -lah /mnt/cache/system/docker
ls -lah /mnt/cache/system/libvirt

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.