46 minutes ago, jeff.lebowski said:

Is the new one v 2.2.8?

Yes, cache_dirs 2.2.8.  The only changes are to the GUI.

I'm now having the spiking CPU issue with the latest version of Cache Dirs. The below is from a 30 second window of CPU activity.

 

I didn't have this problem with the old 'incompatible' one. 

 

 

My settings are whatever defaults come with the plugin (including /mnt/user scanning turned off), except I have removed any directories on my cache, so only have array drives cached by the plugin, none of which are written to at the time (the disks are all spun down). These are the settings I've always had and it's been fine.

 

Disabling the plugin completely stops this behavior.

 

Any ideas?

Edited April 28, 20242 yr by xreyuk

1 hour ago, xreyuk said:

I'm now having the spiking CPU issue with the latest version of Cache Dirs. The below is from a 30 second window of CPU activity.

 

I didn't have this problem with the old 'incompatible' one. 

 

 

My settings are whatever defaults come with the plugin (including /mnt/user scanning turned off), except I have removed any directories on my cache, so only have array drives cached by the plugin, none of which are written to at the time (the disks are all spun down). These are the settings I've always had and it's been fine.

 

Disabling the plugin completely stops this behavior.

 

Any ideas?

Nothing in the cache_dirs script has changed.  Setup cache_dirs to cache only one share and then watch the CPU activity.  Then add one at a time to see which one spikes the CPU.

35 minutes ago, dlandon said:

Nothing in the cache_dirs script has changed.  Setup cache_dirs to cache only one share and then watch the CPU activity.  Then add one at a time to see which one spikes the CPU.

Thanks, I think it may have been my 'snapshots' dir causing it. I use SpaceInvaderOne's replication script to replicate my ZFS cache snapshots to the array. It's possible I didn't have this in the previous caching as it would have been setup after Cache Dirs was setup, however I didn't think adding a share to cache dirs would have caused such a cpu spike!

 

I'm seeing smaller cpu jumps over the same time period, which is only minor, but is there a setting I can use to adjust these? I don't write to the array very often so probably don't cache dirs updating this often.

Edited April 28, 20242 yr by xreyuk

48 minutes ago, xreyuk said:

Thanks, I think it may have been my 'snapshots' dir causing it. I use SpaceInvaderOne's replication script to replicate my ZFS cache snapshots to the array. It's possible I didn't have this in the previous caching as it would have been setup after Cache Dirs was setup, however I didn't think adding a share to cache dirs would have caused such a cpu spike!

The idea behnid the recent changes to the GUI was to keep cache_dirs under control.  If a person was to install the old cache_dirs and not make any include or exclude share settings, cache_dirs would cache anything that showed up at /mnt/.  That includes UD disks and remote shares.  Caching those would cause some errors with CIFS mounts.  I'm not surprised certain shares would cause CPU spikes.  I guess it depends on the file structure.  It should never be setup to cache shares if not necessary.  For example, the 'appdata' and 'system' shares are not even choices because they should reside on SSD drives and caching those is not necessary.

 

57 minutes ago, xreyuk said:

I'm seeing smaller cpu jumps over the same time period, which is only minor, but is there a setting I can use to adjust these? I don't write to the array very often so probably don't cache dirs updating this often.

There are a lot of settings to control the cache_dirs script operation.  You should read the documentation and determine if there are any setting changes appropriate to your situation.

Thank you for creating this script.  I've tried searching the history here and saw a couple of very old posts saying that the -e command for excluding directories won't work on subdirectories.  Is that still the case?  I have a couple of subdirectories with hundreds of thousands of small files that I really don't want to include in the scanning and am not sure if putting them in the user defined options like this will have any effect when /mnt/user/backup is an included directory?

 

-a '-e /mnt/user/backup/directory-to-avoide -e /mnt/user/backup/another-directory-to-avoid'

 

Thank you.

On 4/28/2024 at 11:49 PM, xreyuk said:

I'm now having the spiking CPU issue with the latest version of Cache Dirs. The below is from a 30 second window of CPU activity.

 

I didn't have this problem with the old 'incompatible' one. 

 

 

My settings are whatever defaults come with the plugin (including /mnt/user scanning turned off), except I have removed any directories on my cache, so only have array drives cached by the plugin, none of which are written to at the time (the disks are all spun down). These are the settings I've always had and it's been fine.

 

Disabling the plugin completely stops this behavior.

 

Any ideas?

 

I activated disk spin down on my array a couple of days ago since i never really access the array other than to archive stuff a couple of times a year. However the drives would spin up at random times for no reason so i installed this plugin and it's been working great in that aspect, however, since i installed it i'm getting these exact cpu spikes and i don't have any replication script running and when i turn the script off the spikes stopp, any ides?

On 12/19/2024 at 9:27 PM, Fifthy said:

 

I activated disk spin down on my array a couple of days ago since i never really access the array other than to archive stuff a couple of times a year. However the drives would spin up at random times for no reason so i installed this plugin and it's been working great in that aspect, however, since i installed it i'm getting these exact cpu spikes and i don't have any replication script running and when i turn the script off the spikes stopp, any ides?

 

Same situation, i disabled the plugin and cpu came back to 0.7,

 

enabled, then comes back to 170%

 

changed setting scan /mnt/user to no, and came back to 0.7 % CPU

 

 

 

 

 

 

Have we had confirmation that this works in Unraid 7? It doesn't seem to be for me.

I just checked with my 7.0.0-rc.2 and it works fine. I don't have it set to scan user shares, and even despite that it didn't spin up disks when browsing user share via samba or directly in ssh terminal for folders that only were on that spun down disk.

It also seems to work fine on 7.0.0

Edited January 14, 20251 yr by Alex R. Berg

On 1/14/2025 at 2:07 PM, Alex R. Berg said:

I just checked with my 7.0.0-rc.2 and it works fine. I don't have it set to scan user shares, and even despite that it didn't spin up disks when browsing user share via samba or directly in ssh terminal for folders that only were on that spun down disk.

It also seems to work fine on 7.0.0

Also running it here on 7.0 without issues so far. Also running the default settings/never changed anything. 

On 1/6/2025 at 1:25 PM, Duc said:

 

Same situation, i disabled the plugin and cpu came back to 0.7,

 

enabled, then comes back to 170%

 

changed setting scan /mnt/user to no, and came back to 0.7 % CPU

 

 

 

 

 

 

I reanabled it (/mnt/user) today, and got again high cpu usage from 80% to 170%

 

i did go in 

Included folders: unselected all folders, close, apply

 

looks like good now @2%cpu, the ui shows everything selected

 

while having 2 folders not selected i got high cpu% usage , and in logs excluded dirs was empty.

 

for now it looks good ?!

 

 

Edited January 20, 20251 yr by Duc

Duc, just to be sure, you waited some minutes to see if CPU would drop right? It's natural to go high CPU initially as it scans the disks. You can enable logging and follow what is logged to see what is going on, and compare it with cpu stats from top or similar.

I have this in my .bash_aliases so its a breeze to check up on it, when I think it misbehaves:

alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log'

Horrible Noob-like question here, but is there a recommended set of default settings that mostly just work?

Asking because I had tried cache-dirs a number of years ago, fiddled with it quite a bit based on suggestions from this forum thread, and it just never worked (actually increased my CPU usage and disk spinups). My assumption is "user error" was at fault

Would be interested to revisit the plugin if there was a base-level default settings that just works.

Thanks,

Ari

This is my cfg file, f:\config\plugins\dynamix.cache.dirs\dynamix.cache.dirs.cfg 
```
options="-p 1 -S -i "appdata" -i "backup" -i "md5" -i "media" -i "secure" -i "storage" -i "sync" -e "system" -e "ubuntu" -e "windowsbackup" -c 4 -W 61 -Z 149 -U 100000 -l on -a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'"
service="1"
suspend="-S"
include="appdata,backup"
exclude="app,appbackup"
adaptive="1"
depth=""
scanTimeoutIdle=""
scanTimeoutBusy=""
scanTimeoutStable="149"
minimum=""
maximum=""
log="on"
other="-a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'"
diskIdleTimer="61"
cachePressure="1"
ulimit="100000"
mulithreaded=""
scan_user_share="-u"
mindepth="2"
minDepth="4"
shares=""
```

The important one is 
1) Don't cache more than you need, start small, so there's not so many files
2) Avoid user share, named as `Scan user shares (/mnt/user)` in the gui

In my tests on unraid 7.0 it didn't spin up disks even when accessing user shares, even though only the disk shares were cached, which is good and expected. Scanning user shares is more expensive cpu wise. 

And check my comment above about checking the log, it gives you lots of good information about files on your system and the time it took to scan.

Edited February 11, 20251 yr by Alex R. Berg

On 6/14/2023 at 6:40 PM, dopeytree said:

The default 50000 memory is that Kb / Mb or MB??

Was this ever answered?  Tried to search for it, but didn't find anything. 

 

Also for folder depth, is /mnt/disk3/ count as 2?  Or is it 0 and starts after the disk?

16 hours ago, masterkaj said:

Also for folder depth, is /mnt/disk3/ count as 2?  Or is it 0 and starts after the disk?

Zero means only scan content of /mnt/disk3/ not subfolders /mnt/disk3/*/

Regarding the memory question, you are probably refering to ulimit:

https://phoenixnap.com/kb/ulimit-linux-command

You can also check the script it self, its probably still this version of the script that's the active, I certainly haven't updated it since:

https://github.com/bergware/dynamix/blob/master/source/cache-dirs/scripts/cache_dirs

The script above runs ulimit.

On my system cache_dirs is installed here, but it might have been moved in the public plugin:

/usr/local/bin/cache_dirs

On a simmilar topic, i had a weird idea and i wonder if someone did take the time to benchmark it.
Instead of using the vfs_cache_pressure variable of the linux os to force it to keep directory structure in cache and that spam find on the mount point to keep it in memory, could we just use rclone union provider with --dir-cache-time 1000h --poll-interval 5h ?

 

Like in the config :
 

[testunion]
type = union
upstreams = hdd1: hdd6:

[hdd1]
type = alias
remote = /mnt/data/drive/by-slot/hdd1

[hdd6]
type = alias
remote = /mnt/data/drive/by-slot/hdd6

And command like rclone mount testunion:/ /mnt/data/drive/union/ --dir-cache-time 1000h --poll-interval 5h --allow-other

Hi guys, 

 

My /var/log is getting full. And the culprit is is the cache_dir plugin, I think as per the picture below. 

 

 

So how can I fix this issue so it doesn't occur anymore in the future? Thanks guys. 

1 hour ago, Yannoviche said:

Hi guys, 

 

My /var/log is getting full. And the culprit is is the cache_dir plugin, I think as per the picture below. 

 

 

So how can I fix this issue so it doesn't occur anymore in the future? Thanks guys. 

Disable logging in the Folder Caching settings

10 hours ago, Squid said:

Disable logging in the Folder Caching settings

Thank you Squid, I found the setting. 

I am on Unraid 7.1.2 and having a problem when trying to exclude some directories.

options="-p 1 -i "backups" -W 90 -X 1200 -M 60 -U -U 0 -l on -a '-noleaf -name backups_new -prune -o -print'"

service="1"

cachePressure="1"

suspend=""

shares=""

include="backups"

adaptive="1"

minDepth=""

depth=""

diskIdleTimer="90"

scanTimeoutIdle="1200"

scanTimeoutBusy=""

scanTimeoutStable=""

minimum=""

maximum="60"

mulithreaded=""

ulimit="-U 0"

log="on"

other="-a '-noleaf -name backups_new -prune -o -print'"

I have 2 million of files in backups share, which like 1.9 million of them are under backups_new directory. Tried with exclude and without this, still job is taking the same time to do it's work. Also, when checking for find command in process list, all what I can find is:

root 2613525 0.0 0.0 2704 1556 ? S 11:56 0:00 /bin/timeout 30 find /mnt/disk1/backups -noleaf

root 2613526 0.0 0.0 2704 1332 ? S 11:56 0:00 /bin/timeout 30 find /mnt/disk2/backups -noleaf

root 2613527 0.0 0.0 2704 1332 ? S 11:56 0:00 /bin/timeout 30 find /mnt/disk3/backups -noleaf

root 2613528 107 0.0 5200 3492 ? R 11:56 0:00 find /mnt/disk1/backups -noleaf

root 2613529 107 0.0 8992 7476 ? R 11:56 0:00 find /mnt/disk2/backups -noleaf

root 2613530 103 0.0 4332 2796 ? R 11:56 0:00 find /mnt/disk3/backups -noleaf

Also, even when I have an option log="on" it is not creating that cache_dirs.csv file, just the typical .log.

I'll check it out. But you do have to also bear in mind that what this plugin does is execute the find command in an attempt to keep the directory cached in memory. Million things can cause the cache to drop it, and with 2M files I wouldn't be surprised that it is.

that's what the log is there for. Its very good for figuring out what's going on in these cases... It also counts number of files and shows scan times...

14 hours ago, Alex R. Berg said:

that's what the log is there for. Its very good for figuring out what's going on in these cases... It also counts number of files and shows scan times...

No, it is not helpful in my situation. Like I said, it doesn't matter what kind of folder (not share) I am gonna exclude, the number on the filecount is not going to get lowered, also the time between building the 'cache' with each depth is the same.

I would love to verify this with .csv log file, but that one doesn't appear on my system

Edited May 23, 20251 yr by silverkin