As title suggest, my system is only 6 or 7 months old and I've got some sort of memory leak with something, I'm thinking I configured something wrong in a Docker App but I haven't found it yet.  Its a slow creep, fresh reboot in system Docker will show about 5gb in use (32gb total system RAM). My up-time right now is nearly 50 days and Docker is now showing over 10gb in use. I haven't changed a thing. 

 

The first time I noticed this leak was when some Docker apps stopped working.... logged in after 3 or 4 months uptime and not only did some Apps stop, some were uninstalled completely. Just poof, gone not even listed in the Docker area as started or stopped. After a hard reboot 1 or 2 reappeared as stopped needing updates. The other 1 or 2 I had to reinstall but all my files and settings were still present.  I don't want this to happen again so looking for some assistance.  I've been slowly trying to reboot or update Docker Apps and configs to see if it clears the in use RAM but nothing has helped, I'm out of ideas and need some help.   

Diagnostics attached! 

OS 6.12.14 Unleashed
32GB Ram

Apps in screenshot, blacked out WAN IP
Storage I've got 2 Arrays; Array (50TB) for Storage & a Cache_nvme Array (1TB) for downloads, appdata, docker apps etc. I tried setting it up so all my Docker apps and their temp files are all saved in that Cache nvme within Appdata but I feel like I suck and didn't do the volume mappings correctly or something

Any feedback or ideas are appreciated. IF there is something I should attach or provide please let me know and I can assist. Thank you for your help in advance if you do end up replying to my post; honestly appreciate it.

 

 

 

tower-diagnostics-20250414-2152.zip

8 hours ago, brandon13ke said:

Docker is now showing over 10gb in use

I think you must be talking about docker.img usage and not memory. That seems fine to me, you have 20G allocated.

 

There are some things about your shares that could be better.

 

https://docs.unraid.net/unraid-os/manual/shares/user-shares/#default-shares

 

Ideally, these shares would have all files on cache or other pool, with nothing on the array, so Docker/VM performance won't be impacted by slower array reads and especially writes, and so array disks can spin down since these files are always open.

 

appdata                           shareUseCache="only"    # Share exists on cache_nvme, disk1
domains                           shareUseCache="only"    # Share exists on disk1
d-------s                         shareUseCache="only"    # Share exists on cache_nvme, disk1
isos                              shareUseCache="only"    # Share exists on disk1
system                            shareUseCache="only"    # Share exists on disk1

These shares are configured to only put files on cache_nvme. As you can see, they have files on the array, and they won't be moved to cache_nvme since they have no Mover action.

 

All of these are discussed in that link I gave, except...

Your downloads share (if that is its name). Seems to me you would want that written to cache_nvme and moved to the array. What do you want for that share?

 

You should install Fix Common Problems. It would be telling you that your shares locations do not match their settings.

 

What do you get from command line with this?

du -h -d 1 /mnt/disk1/appdata

 

Thank you so much for such a detailed response, seriously --- thank you.  As for the link and Shares setup, this does not surprise me 😥. I read that article plus like 4 or 5 other related ones + watched all the Space Invaders vids on the topics before doing the first time setup and I was still a bit shaky.  First time using Unraid, spun the system up in Sept/Oct so still learning the ways.

 

As for the Shares:

 

Appdata - I would love for all the Appdata for all my Docker apps and really anything that requires anything "Fast" to be stored purely on the cache_nvme. That was always the plan and if I'm being honest, I thought I did that 😅, so it wasn't for lack of trying.  If you could assist or help clean that up for one Docker app I think I could mirror it for the others?

 

Downloads - I thought it was only on cache_nvme, not the array/both. The reason is because of the Resilio-sync App, which I could not figure out or configure for the life of me during setup. I would love for it to live solely on Disk1 or Disk2 or w/e on the main Array. This is basically a folder sync App between another remote system and it mirrors the folder contents from that remote system syncing it to this one. I used Syncthing for years but also could not get the dang app working but could get Resilio working so here I am.  Only thing that needs to monitor the Downloads area are some other apps like Sonarr and Radarr.

 

FixProblems - Downloaded and going through a few of the things. You are correct it shows the Appdata shares thing you talked about.     

 

 

For the command I get the following response:

16M     /mnt/disk1/appdata/audiobookshelf
87M     /mnt/disk1/appdata/radarr
110M    /mnt/disk1/appdata/sonarr
213M    /mnt/disk1/appdata

 

 

Here is my Resilio app settings if you could take a look or point out what I've done wrong: 


Really appreciate your help with this. Ready for next steps or recommendations! In System it shows Docker continues to climb, now at 10.2GB so I need to figure this out before it starts uninstalling and stopping Dockers apps

Edited April 16, 20251 yr by brandon13ke

I don't use Resilio, but I think we can deal with a lot of this more generally.

 

You said "Downloads", but as you can see in your screenshot, the share is really named "downloads". You need to be careful with that, linux is case-sensitive, so "Downloads" and "downloads" are different.

 

As mentioned, those "default" shares files are always open. Nothing can move open files.

 

Before we can move those, we need to make sure all of those files are not open:

Go to Settings - Docker and disable Docker.

Go to Settings - VM Manager and disable VMs.

Leave them disabled until we get everything where it belongs.

 

Then post new diagnostics so I can make sure I have the latest information.

 

Docker disabled.

VM Manager disabled (Didn't have any VMs so should have no impact). 

 

Left the system running for about 10-15mins after those changes then ran a new diagnostics, see attached. 

 

tower-diagnostics-20250416-0925.zip

You have a few .cfg files in config/shares that don't correspond to any user shares. Probably you created some shares (perhaps accidentally) and then deleted them.

 

Go to the User Shares page and use the CLEAN UP button to clean those up (and clean up your diagnostics). That can sometimes also help if a share setting doesn't seem to be working, since they may be loading from those instead if the names are similar (usually upper/lowercase mismatch).

 

Set downloads share to

Primary:cache_nvme; Secondary:array; Mover action:cache_nvme->array.

After we get that one moved we can change it so it only puts files on the array.

 

Set appdata, domains, isos, system shares to

Primary:cache_nvme; Secondary:array; Mover action:array->cache_nvme.

 

What do you get from command line with this?

du -h -d 1 /mnt/cache_nvme

And this?

du -h -d 1 /mnt/disk1

 

Also post new diagnostics after completing those steps.

• User Shares Cleanup button complete!
• Set downloads share primary cache_nvme, secondary array
  • Set Mover action from cache_nvme-->array
  • Also set/added one extra thing, to only use disk4 and disk5, exclude disk1 and disk2
• Set appdata, domains, isos, system shares to Primary cache_nvme with Secondary as the array

  • Set Mover action from array-->cache_nvme

• Command line for the cache_nvme results are:

16G     /mnt/cache_nvme/appdata
673G    /mnt/cache_nvme/downloads
688G    /mnt/cache_nvme

 

• Command line for the disk1 results are:

21G     /mnt/disk1/system
213M    /mnt/disk1/appdata
0       /mnt/disk1/domains
0       /mnt/disk1/isos
0       /mnt/disk1/downloads
0       /mnt/disk1/audiobooks
8.4T    /mnt/disk1/plex
8.4T    /mnt/disk1

 

• re-ran diagnostics after all the steps were completed above. See attached! 

tower-diagnostics-20250416-1435.zip

3 hours ago, brandon13ke said:

Also set/added one extra thing, to only use disk4 and disk5, exclude disk1 and disk2

You should never set both include and exclude, and no good reason to want to do that.

 

Include already excludes all disks not in the include list

Exclude already includes all disks not in the exclude list

 

I usually use Include, that way I know what disks the share will use, even if I add additional disks later they are automatically excluded because they aren't in the include list.

 

Also, disk1 is the only array disk that currently has downloads. Making a setting for it that doesn't include disk1 (or that excludes disk1) will not move any of those disk1 files to other disks. That setting only affects new files.

Gotcha, wasn't aware of that so that's good to know. I just updated downloads share to not have any disks selected for include or exclude, will leave it all default. 

This is probably a good time to talk about Minimum Free, just to make sure those settings don't get in the way of moving things.

 

Each user share has a Minimum Free setting. Each pool also has a Minimum Free setting.

 

In the general case, Unraid has no way to know how large a new file will become when it chooses a disk or pool to write it to.

 

If a new file is supposed to go to a pool (cache_nvme is Primary), but the pool has less than Minimum Free for the pool, the new file will instead go to Secondary, if that is set. This is known as overflow.

 

If a new file is supposed to go to the array, whether because it overflows, or because array is Primary, if an array disk has less than Minimum Free for the share, another disk will be chosen.

 

Note that Minimum Free doesn't say how much free space will be kept on a disk or pool. It says how much free space a disk or pool must have before it can be chosen for a new file.

 

Example1: Pool is Primary, Minimum Free for pool is 20G, pool has 25G free, new file is 15G. The pool will be chosen because it has more than Minimum. After the file is written, the pool will have 10G remaining, which is less than Minimum, so it won't be chosen again.

 

Example2: Pool is Primary, Minimum Free for pool is 20G, pool has 25G free, new file is 30G. The pool will be chosen because it has more than Minimum. The write will fail when it runs out of space.

You must set Minimum Free for a pool to larger than the largest file you expect to write to the pool. Currently, cache_nvme has no Minimum. You should set Minimum Free for cache_nvme.

 

You must set Minimum Free for a share to larger than the largest file you expect to write to the share. Currently, it looks like many of your shares have a calculated value for Minimum, and it may be larger than necessary. You should review this setting for each share.

 

For completeness, here are the links to the documentation about this:

 

https://docs.unraid.net/unraid-os/manual/storage-management/#minimum-free-space-for-a-pool

 

https://docs.unraid.net/unraid-os/manual/shares/user-shares/#minimum-free-space

I should be OK on space for the minimum free just went through all my shares and settings confirmed the following:

• array
  • My most 'full' drives disk1 and disk2 have 4.6TB and 3.6TB space remaining on them
  • 2 other disks are nearly completely empty
• cache_nvme
  • It's got 259 GB free of the 1TB
  • I move on average maybe 75GB a month incoming and outgoing so not a ton of movement there, so no chance it just "fills up" 

 

As for the Shares minimum free space, I just went through all of them and they are a range from 30GB at the smallest to 146GB at the largest. 

• All of them are set to high-water for allocation method and auto split only the top level dir as required

It doesn't let me set a minimum free space on the cache_nvme pool, its grayed out.  

• I have 2 NVMe drives in my system, a 1.5TB and a 1TB basically in mirror (1TB usable which is expected)

1 minute ago, brandon13ke said:

It doesn't let me set a minimum free space on the cache_nvme pool, its grayed out.  

I think you have to stop the array to change that.

4 hours ago, brandon13ke said:

0       /mnt/disk1/domains
0       /mnt/disk1/isos
0       /mnt/disk1/downloads
0       /mnt/disk1/audiobooks

Are these folders empty? If so, you should delete them. Did you ever use Mover Tuning plugin? I think some versions of it left empty folders behind when it moved all files for the share from a disk or pool.

In theory --- with the new settings changed for the downloads share, the Mover should kick-off tonight/early tomorrow morning and start moving everything from the cache_nvme to the array?   If it does that, then the 1TB cache_nvm will go from something like 250GB free to more like 800GB free so minimum won't be an issue.

 

I looked up the Mover Tuning Plugin and installed it, reviewing it all now. 

 

For those 4 shares on disk1 they are empty, just folders. I went ahead and deleted them!

9 minutes ago, trurl said:

Are these folders empty?

Maybe domains and isos were created when you enabled VM Manager, but then you never put anything in them. I guess they are OK, since those shares are going to be moved anyway. Or if you do delete them, they will just be recreated if you enable VM Manager.

I guess those other empty folders are OK too, it just means disk1 will always indicate those shares have contents on disk1. And it's fine for them to be there, that is where they belong anyway. And it's fine to delete them.

 

1 minute ago, brandon13ke said:

I looked up the Mover Tuning Plugin and installed it, reviewing it all now. 

Please uninstall it for now, it may complicate things. Built-in mover should be fine for what we want to do. If for some reason you think you need some of its features, you can try it out later.

 

4 hours ago, brandon13ke said:

21G     /mnt/disk1/system
213M    /mnt/disk1/appdata

Looks like only this much to be moved TO cache_nvme

 

4 hours ago, brandon13ke said:

673G    /mnt/cache_nvme/downloads

But a lot more to be moved FROM cache_nvme

 

I guess you're ready to go.

 

You can wait for the scheduled run, or run mover now. Will probably take quite a while to get downloads moved.

 

When it's finished, post new diagnostics. I'll check back tomorrow.

Empty Folders - Yeah I likely created them during my initial setup process. Then moved it once cache_nvme pool was added so it was legacy, they are now deleted. Good to know those folders may be recreated if I re-enable VM Manager, I might want future VM's to always have everything on the cache_nvme in the future if I spin something up but I'll look into that when I get there. No future plans as of now. 

 

Moving Tuning - I removed it when I read your post yesterday, so it did not run. I'll try it out later if needed in the future. 

 

Moving TO cache_nvme -  

• /mnt/disk1/system only has a docker folder in it, which has docker.img for the 21.5gb
• /mnt/disk1/appdata has 3 folders, audiobookshelf, radarr, and sonarr each with their db files

None of these were moved last night via Mover because I turned Docker back on (I know, my bad). I disabled Docker this morning, went to Main > Array Operation and then ran the "Move" option to kick it off manually.  After just a few mins, Mover finished. I rechecked the /mnt/disk1/ folders to see what was there and both the /system and /appdata folders were successfully moved! 

 

 

Moving TO array -  

• /mnt/cache_nvme/downloads was fully moved last night, it is no longer on the cache_nvme at all. cache_nvme only has 2 folders in it, appdata and system as of now. 

 

Confirmed via:

 

Attached are recent diagnostics after running the Mover this morning just a bit ago.   Just want to say, really appreciate your assistance with everything getting my system  dialed in. Its been a huge help and I feel wayyyy more confident in the system now long-term. Thank you. 

 

tower-diagnostics-20250417-1043.zip

Looks good!

 

You can run your dockers now.

 

Might as well leave VM Manager disabled until you are ready to do VMs.

 

I have played with VMs from time to time, but I find docker does everything I want.

Dockers are now back online!

I'm actually going to switch my HA from a Docker instance to a VM for HAOS sooooo I'll be toying around with the VMs tonight for that migration. Tired of trying to get this Zigbee USB stick to pass through 😅. Z-Wave was easy but this Zigbee one is a PITA. 

I really appreciate your help and support trurl, been a huge help guiding me along. If there are any resources or things you recommend I should look into and learn based on our brief interactions I'm all ears! 

5 minutes ago, brandon13ke said:

any resources

SpaceinvaderOne is always fun, but you probably already know about that.

 

What are you doing about remote access? I was using Wireguard, but Tailscale is easier and more versatile.

Had VPN via Unifi, but this weekend just phased that out and installed and setup Tailscale.

 

Within Tailscale added in local IP's for my firewall, Unraid, and HA server only, nothing more.  Also got my Zigbee dongle working almost instantly when phasing out of HA docker doing HAOS via the VM. That also let me phase out like 4 or 5 docker apps now that I'm running HAOS via VM.