Alex R. Berg

@dlandon Could you help us with link to the new limetech repo, if its public? @Freender Thank for the details and the log. The problem is that the file counting phase will not include the find-options with filters given with -a. But this file counting phase only runs on the initial scans at each depth. Of cause that is rather unfortunate as that means the files listed is incorrect, plus its not nice that it scans a huge amount of files on reboot. The adaptive runs the counts at each adaptive depth (initially), which is by default 20 levels (depth_max_incremental_depth=20) + infinite (9999). So shouldn't scan 30 but 21 times I think. But maybe you just rounded or guestimated which is fine. Indeed the non-adaptive does it in one go, and does not repeat it. The adaptive also does not repeat the count once initial 21 scans is complete. I have experimented with a fix. What worries me a bit is changing other peoples options by fixing them for the file count. But I think this works I've created a fix here. I have temporarily placed it in a gist. We would have to reapply the changes to the real current version, rather than the one i had lying: https://gist.github.com/arberg/08adf6259f09c76274db7feaf38fc81e The 1st revision is the original version I applied my changes to. I ran it with sudo cache_dirs -q; sudo rm /var/log/cache_dirs.log; sudo ./cache_dirs -p 1 -i appdata -i backup -i md5 -i secure -i storagehdd -i sync -i windowsbackup -e app -e appbackup -e docker -e home -e muaddibhdd -e pil -e svn -e svnbackup -e system -e ubuntu -e windowsbackup -c 4 -W 61 -Z 149 -U 100000 -l on -a '-name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'; psg cache_dirs my alias: alias psg='ps -e x -o ppid,pid,pgid,user,group,uid,gid,%cpu,%mem,tty,vsz,rss,etime,cputime,rgroup,ni,fname,args | body grep' use 'psg cache_dirs', to obain the full command you use for cache_dirs before you stop the original running script, so you can copy the arguments you use.

@silverkin @dlandon freeender Its rather time consuming to investigate for me and I don't have the details to detect what you are seeing and why. I really need to see the non-short version of the log file written. It contains lots of important information. You cannot just update args inside the script and add filters, it breaks other stuff such as determining which folders are under backup. Ah, maybe you are trying to just avoid backup_news via find filters? You cannot do that, an exclude is required. I didn't remember that before spending some hours reading the script, to review the change request... The '-type f' you found only affects initial counts, and it writes the find statement itself, because it has already built a list of relevant dirs. Removing the -type f, wont make it scan less, as the dirs to scan is in a for outside. These things will probably break if you hack the script, and enter overwrite the default args intead of using the proper settings. See build_dir_list function, which need to work for count_files to work. I think you broke build_dir_list by the manual edit of the file. Best Alex

I've added a feature request now: https://forums.unraid.net/topic/191994-common-virtual-pool-for-all-pools-array-cache-mounted-devices-regarding-split-level-and-choice-for-placement-of-new-files/

I would like to control whether folders are stored on SSD or HDD storage, and/or possible my tmpssd storage from separate device without mirror. This way I can move folders from hdd to ssd, as my ssd storage increases over time, and/or my needs change. I can do that move without having to change location used by scripts and people for my files, because unRaid as this cool /mnt/user view of everything. Currently in Shares I can choose Primary storage / secondary storage. Here I want Primary storage as the Common virtual pool, and then all disks should appear in include/exclude, and split level / new-files affect all regardless of source pool. I think this is a more generically useful abstraction that abstracts away the pool setup from the shares. I don't care about the mover anymore, so for this I wouldn't need a mover, and if that should change, I would just create a script to do the moving, which is easy compared to changing how unraid selects new file location, which I think is a black box I cannot change. See more details about my use case here: https://forums.unraid.net/topic/191966-howto-use-common-share-for-2-separate-pools-one-with-ssds-and-one-with-hdds-where-i-control-what-goes-where/

In these days of expanding SSD storage, I'm considering consolidating eg. music on ssd-storages, but keeping videos on hdd's. I access shares via samba from windows, and would like samba .Recycle.Bin to work for deletions. The .Recycle.Bin does not work if I use softlinks from my hdd-mounted share to ssd-storage, because the samba rename() when across different file-systems. Same goes when mounting ssd folder with 'mount --bind'. Currently I have m: which is /mnt/user/media, and I want m:\music on ssd and m:\video on hdd. I also prefer not to move musik to different share, because then I have to teach people and update programs to where the new location is. Its not as elegant as just changing a pointer in the filesystem, when I want to change storage-medium hdd <> ssd. It seems like an obvious think for unraid using its pools and is fuser which sums share from across multiple disks. However if I create share which have only primary storage cache-pool, then it always writes new files to that primary storage, even if folder I write is already on the disk-pool. Likewise if I add secondary storage as disk-pool, I get same behaviour. I could just let all new files be written to cache, but I dislike that, because then when writing huge files they have to be moved, and writing can fail because of lack of space on the cache ssd-storage. I have learned the hard way not to use the built in mover, and just write directly to the pool where I want it. So the question is: Can I somehow get control over in unraid which pool it chooses for new files, by simply selecting the pool the parent folder is already on? Might this be a desirable feature request for unraid for others than me ? Are the other ways to achieve the virtual layout of my files that I want, with recycle bin still working, but me regaining control over which (root)-folders go on SSD vs HDD? Best Alex

This is what my log is showing: ``` $ alias tailcachelog alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log' $ tailcachelog cache_dirs version 2.2.7 Setting Memory ulimit to 100000 Setting cache_pressure=1 Arguments=-p 1 -S -i appdata -i backup -i md5 -i media -i secure -i storage -i sync -i windowsbackup -e app -e appbackup -e cacheonly -e docker -e home -e muaddibhdd -e nanite -e pil -e svn -e svnbackup -e system -e ubuntu -e windowsbackup -c 4 -W 61 -Z 149 -U 100000 -l on -a -noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print Max Scan Secs=10, Min Scan Secs=1 Scan Type=adaptive Min Scan Depth=4 Max Scan Depth=none Use Command='find -noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print' ---------- Caching Directories --------------- appdata backup md5 media secure storage sync ---------------------------------------------- Setting Included dirs: appdata,backup,md5,media,secure,storage,sync,windowsbackup Setting Excluded dirs: app,appbackup,cacheonly,docker,home,muaddibhdd,nanite,pil,svn,svnbackup,system,ubuntu,windowsbackup min_disk_idle_before_restarting_scan_sec=61 scan_timeout_sec_idle=150 scan_timeout_sec_busy=30 scan_timeout_sec_stable=149 frequency_of_full_depth_scan_sec=604800 ERROR: excluded directory 'cacheonly' does not exist. ERROR: excluded directory 'nanite' does not exist. cache_dirs started 2025.05.23 19:24:23 Executed find in (72s) 72.40s, wavg=72.40s Idle____________ depth 4(timeout 30s) slept 0s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=2 scan_tmo=30s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=16%, filecount[4]=52198 2025.05.23 19:25:36 Executed find in (51s) 51.23s, wavg=58.29s Idle____________ depth 5 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=10%, filecount[5]=130628 2025.05.23 19:26:28 Executed find in (22s) 22.75s, wavg=40.52s Idle____________ depth 6 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=12%, filecount[6]=203844 2025.05.23 19:26:52 Executed find in (19s) 19.19s, wavg=31.99s Idle____________ depth 7 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU= 9%, filecount[7]=297086 2025.05.23 19:27:12 Executed find in (21s) 21.86s, wavg=28.61s Idle____________ depth 8 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU= 6%, filecount[8]=356281 2025.05.23 19:27:35 Executed find in (15s) 15.68s, wavg=24.92s Idle____________ depth 9 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU= 9%, filecount[9]=410859 2025.05.23 19:27:52 Executed find in (4s) 04.29s, wavg=19.76s Idle____________ depth 10 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=16%, filecount[10]=430518 2025.05.23 19:27:57 Executed find in (3s) 03.98s, wavg=16.25s Idle____________ depth 11 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=18%, filecount[11]=448640 2025.05.23 19:28:02 Executed find in (5s) 05.83s, wavg=14.17s Idle____________ depth 12 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=14%, filecount[12]=465810 2025.05.23 19:28:09 Executed find in (8s) 08.79s, wavg=13.19s Idle____________ depth 13 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=13%, filecount[13]=485555 2025.05.23 19:28:19 Executed find in (8s) 08.55s, wavg=12.42s Idle____________ depth 14 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=13%, filecount[14]=505355 2025.05.23 19:28:28 Executed find in (5s) 05.67s, wavg=11.38s Idle____________ depth 15 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=16%, filecount[15]=518085 2025.05.23 19:28:35 Executed find in (2s) 02.66s, wavg=10.13s Idle____________ depth 16 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=26%, filecount[16]=526342 2025.05.23 19:28:39 Executed find in (2s) 02.55s, wavg=09.12s Idle____________ depth 17 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=28%, filecount[17]=533061 2025.05.23 19:28:42 Executed find in (2s) 02.28s, wavg=08.27s Idle____________ depth 18 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=33%, filecount[18]=540454 2025.05.23 19:28:46 Executed find in (2s) 02.27s, wavg=07.56s Idle____________ depth 19 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=31%, filecount[19]=544809 2025.05.23 19:28:49 Executed find in (2s) 02.27s, wavg=06.97s Idle____________ depth 20 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=29%, filecount[20]=546436 2025.05.23 19:28:52 Executed find in (2s) 02.30s, wavg=06.48s Idle____________ depth 9999 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=29%, filecount[9999]=546900 I'm running v2.2.7: see head -n 150 /usr/local/bin/cache_dirs ``` # Version 2.2.7 - Fixed -a option which was broken in 2.2 # arberg #################################################################################### version=2.2.7 program_name=$(basename "$0") arg_count=$# run_log="/var/log/cache_dirs.log" csv_log="/var/log/cache_dirs.csv" lost_cache_log="/var/log/cache_dirs_lost_cache.csv" ``` Notice this at the top of the log file: ``` ---------- Caching Directories --------------- appdata backup md5 media secure storage sync ``` My cfg looks like this: config\plugins\dynamix.cache.dirs\dynamix.cache.dirs.cfg ``` options="-p 1 -S -i "appdata" -i "backup" -i "md5" -i "media" -i "secure" -i "storage" -i "sync" -i "windowsbackup" -e "app" -e "appbackup" -e "cacheonly" -e "docker" -e "home" -e "muaddibhdd" -e "nanite" -e "pil" -e "svn" -e "svnbackup" -e "system" -e "ubuntu" -e "windowsbackup" -c 4 -W 61 -Z 149 -U 100000 -l on -a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" service="1" suspend="-S" include="appdata,backup,md5,media,secure,storage,sync,windowsbackup" exclude="app,appbackup,cacheonly,docker,home,muaddibhdd,nanite,pil,svn,svnbackup,system,ubuntu,windowsbackup" adaptive="1" depth="" scanTimeoutIdle="" scanTimeoutBusy="" scanTimeoutStable="149" minimum="" maximum="" log="on" other="-a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" diskIdleTimer="61" cachePressure="1" ulimit="100000" mulithreaded="" scan_user_share="-u" mindepth="2" minDepth="4" shares="" ``` If you play around with it, you can probably solve your problem. Play around with the cfg and start and stop the script manually and watch the log file. Or rather that's probably what I would do. You can also find the command it runs with ps -ef | grep cache And check ps -ef | grep find to see what it is calling find on. Though you have to catch. You can also ask deepseek for how to get process tree from the cache_dirs execution.

I also had problem logging in with SSH after updating from 7.0.1 to 7.1.2. For me it turned out I was missing an /etc/rc.d/rc.sshd restart after my plugin-script had copied my personalized version of the sshd_config to /etc/ssh. Apparently things ran in a different order before, or for some other reason I didn't have to restart th sshd before .

that's what the log is there for. Its very good for figuring out what's going on in these cases... It also counts number of files and shows scan times...

Zero means only scan content of /mnt/disk3/ not subfolders /mnt/disk3/*/ Regarding the memory question, you are probably refering to ulimit: https://phoenixnap.com/kb/ulimit-linux-command You can also check the script it self, its probably still this version of the script that's the active, I certainly haven't updated it since: https://github.com/bergware/dynamix/blob/master/source/cache-dirs/scripts/cache_dirs The script above runs ulimit. On my system cache_dirs is installed here, but it might have been moved in the public plugin: /usr/local/bin/cache_dirs

This is my cfg file, f:\config\plugins\dynamix.cache.dirs\dynamix.cache.dirs.cfg ``` options="-p 1 -S -i "appdata" -i "backup" -i "md5" -i "media" -i "secure" -i "storage" -i "sync" -e "system" -e "ubuntu" -e "windowsbackup" -c 4 -W 61 -Z 149 -U 100000 -l on -a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" service="1" suspend="-S" include="appdata,backup" exclude="app,appbackup" adaptive="1" depth="" scanTimeoutIdle="" scanTimeoutBusy="" scanTimeoutStable="149" minimum="" maximum="" log="on" other="-a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" diskIdleTimer="61" cachePressure="1" ulimit="100000" mulithreaded="" scan_user_share="-u" mindepth="2" minDepth="4" shares="" ``` The important one is 1) Don't cache more than you need, start small, so there's not so many files 2) Avoid user share, named as `Scan user shares (/mnt/user)` in the gui In my tests on unraid 7.0 it didn't spin up disks even when accessing user shares, even though only the disk shares were cached, which is good and expected. Scanning user shares is more expensive cpu wise. And check my comment above about checking the log, it gives you lots of good information about files on your system and the time it took to scan.

Duc, just to be sure, you waited some minutes to see if CPU would drop right? It's natural to go high CPU initially as it scans the disks. You can enable logging and follow what is logged to see what is going on, and compare it with cpu stats from top or similar. I have this in my .bash_aliases so its a breeze to check up on it, when I think it misbehaves: alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log'

I just checked with my 7.0.0-rc.2 and it works fine. I don't have it set to scan user shares, and even despite that it didn't spin up disks when browsing user share via samba or directly in ssh terminal for folders that only were on that spun down disk. It also seems to work fine on 7.0.0

I had the same problem, with github error as Joe mentioned and the hanging > Performing actions > Please keep this window open while we perform some actions In my case it was uMatrix browser extension that blocked my unraid from getting data from unraid.net and blocked unraid from contacting github.

I would probably check if auto-start is enabled, and if so disable it. It seems to be in disk.cfg (probably: startArray="no"), but since you have access to the webgui, you can just disable it there, restore your super.dat, and reboot. I think super.dat contains your disk spec, and it would appear so on your experience. > If I reassign all of the disks to their original spot in the array, should everything come up as expected? Yes. But I would definitely mount as read only and not let in recreate the parity, but instead validate the parity is correct not writing corrections, at least for some gigabytes. I'm not quite sure what the options are. We wouldn't want it to overwrite your good data with parity if you mount incorrectly. You can mount the disks in read-only mode safely manually in a terminal if you so desire, to check the data is there. If you only have one parity disk, the order of your array disks shouldn't matter for the parity. Though of cause it might very well matter for your shares, if some shares are on specific disks. Do you have stored information elsewhere about which disks were array-disks and which were parity? Probably not... I checked my disks ids and I don't see them in my syslog. Remember to check your syslog for disk errors before and after recreating the array. Alex.

well, be that as it may, is it possible?