6.6.6 Memory and CPU Spike - Can't Access Dockers


Recommended Posts

This issue started a few weeks ago and it happens every few days. I would notice that automations in my homeassistant docker weren't running. When I'd check, I'd find that none of my dockers were accessible. On the dashboard page of Unraid, the CPU and memory would be spiked. The dockers page of Unraid would not load at all. Forgive me if I'm not describing this correctly or providing the right diagnostic data. This is my first real issue with Unraid in 5 years of use.

tower.status.png

tower-diagnostics-20190313-0935.zip

Link to comment

Maybe not but it couldn't hurt.

 

But before going there, and probably more likely:

 

Why do you have 150G for your docker image? This often indicates the user has problems with docker image filling up. Making docker image larger doesn't fix the problem, it just makes it take longer to fill. If you have things setup correctly it is extremely unlikely you would ever get close to filling 20G.

 

The typical cause of filling docker image is application(s) that are writing to a path that isn't in mapped storage. This isn't about the mappings, it is about settings with the application itself. The application must not write to any path that isn't in mapped storage. Common mistakes are not using the same upper/lower case as in the container volume mappings, or using a relative instead of an absolute path (must begin with /).

 

 

There is a whole section of the Docker FAQ on "Docker Image Filling Up",

 

https://forums.unraid.net/forums/topic/57181-real-docker-faq/

 

but probably you have the problem I described above.

 

Delete and recreate docker image as only 20G then you can reinstall your dockers exactly as before using the Previous Apps feature on the Apps page. Then examine each one to figure out where the problem is. Possibly some downloading application, or maybe you have Plex transcoding setup wrong. Anything that writes a lot of data is a good place to start.

 

Probably a good idea to just take them one at a time and turn all the others off. We can help you look at each one at a time if you need it.

Link to comment

Also, your appdata share is set cache-only, but it has files on the array. Go to Settings - Docker and disable docker service. This is the same page where you will delete and recreate your docker image.

 

With docker services disabled, go to Shares - User Shares, click on the appdata share, and set it to cache-prefer. Then go to Main - Array Operations and click the Move Now button. After it finishes, go back to the User Shares and click Compute... for the appdata share to make sure it all got moved to cache.

 

 

Link to comment

Your screenshot and your diagnostics don't actually indicate a full docker image at the time they were taken, though, so perhaps you already took care of that problem and just never reduced to a more reasonable size. You should still reduce docker image to 20G and get your appdata moved to cache where it belongs.

 

Link to comment
10 minutes ago, bcbgboy13 said:

Most of your disks are also old and you are running them at very high temperature - above 50 and even 55 degC (and even higher in previous power cycles).

Keep this in mind.

Didn't notice that. Cache is nearly 10 years old, and a ReiserFS spinner.

 

You should definitely try to do something about the cooling.

Link to comment
44 minutes ago, trurl said:

When did you upgrade to V6?

Is V6 when Docker was added? If so, it's been several years.

 

1 hour ago, trurl said:

Your screenshot and your diagnostics don't actually indicate a full docker image at the time they were taken, though, so perhaps you already took care of that problem and just never reduced to a more reasonable size. You should still reduce docker image to 20G and get your appdata moved to cache where it belongs.

 

Yes, I was having issues with my docker image filling up, but I was never able to pinpoint the problem. Maybe the issue was resolved with one of my dockers being updated. I'll reduce the docker image back to 20G.

 

1 hour ago, trurl said:

Also, your appdata share is set cache-only, but it has files on the array. Go to Settings - Docker and disable docker service. This is the same page where you will delete and recreate your docker image.

 

With docker services disabled, go to Shares - User Shares, click on the appdata share, and set it to cache-prefer. Then go to Main - Array Operations and click the Move Now button. After it finishes, go back to the User Shares and click Compute... for the appdata share to make sure it all got moved to cache.

 

 

I'll try this when I get home. I'm not sure what files it would have on the array.

 

56 minutes ago, trurl said:

Didn't notice that. Cache is nearly 10 years old, and a ReiserFS spinner.

 

You should definitely try to do something about the cooling.

Yes, some of my disks are very old. I've just been replacing the smallest disk whenever I add more storage.

 

I'll work on the cooling. Honestly I've never really paid too much attention to temperature because I've never had problems. I guess I have just been lucky.

Link to comment
  • 4 weeks later...

I have an update on this post. Since I originally posted I have done the following per your suggestions:

  • Added storage
  • Converted all disks to XFS
  • Reduced the size of my Docker image file
  • Replaced the extremely old cache HDD with a new SSD
  • Replaced a fan and moved my server to improve temperatures

After making all of these changes, it ran for a week straight without issue. However, last night my dockers crashed again. The Dashboard showed the same CPU and Memory spike. This time I had Fix Common Problems installed and it said I was getting out of memory errors. I have attached the diagnostics I downloaded during the spike last night.

 

Any ideas of what I should try next?

tower-diagnostics-20190409-1951.zip

Link to comment

I like your improvements. Other than your oom I don't see anything.

 

Not likely your problem, but you should upgrade to latest stable. Maybe one of your apps has a memory leak. I seem to recall some speculation about that on some docker thread. Maybe you could try without some of them to try to narrow down the possible culprit.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.