Alex R. Berg

@dlandon Could you help us with link to the new limetech repo, if its public? @Freender Thank for the details and the log. The problem is that the file counting phase will not include the find-options with filters given with -a. But this file counting phase only runs on the initial scans at each depth. Of cause that is rather unfortunate as that means the files listed is incorrect, plus its not nice that it scans a huge amount of files on reboot. The adaptive runs the counts at each adaptive depth (initially), which is by default 20 levels (depth_max_incremental_depth=20) + infinite (9999). So shouldn't scan 30 but 21 times I think. But maybe you just rounded or guestimated which is fine. Indeed the non-adaptive does it in one go, and does not repeat it. The adaptive also does not repeat the count once initial 21 scans is complete. I have experimented with a fix. What worries me a bit is changing other peoples options by fixing them for the file count. But I think this works I've created a fix here. I have temporarily placed it in a gist. We would have to reapply the changes to the real current version, rather than the one i had lying: https://gist.github.com/arberg/08adf6259f09c76274db7feaf38fc81e The 1st revision is the original version I applied my changes to. I ran it with sudo cache_dirs -q; sudo rm /var/log/cache_dirs.log; sudo ./cache_dirs -p 1 -i appdata -i backup -i md5 -i secure -i storagehdd -i sync -i windowsbackup -e app -e appbackup -e docker -e home -e muaddibhdd -e pil -e svn -e svnbackup -e system -e ubuntu -e windowsbackup -c 4 -W 61 -Z 149 -U 100000 -l on -a '-name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'; psg cache_dirs my alias: alias psg='ps -e x -o ppid,pid,pgid,user,group,uid,gid,%cpu,%mem,tty,vsz,rss,etime,cputime,rgroup,ni,fname,args | body grep' use 'psg cache_dirs', to obain the full command you use for cache_dirs before you stop the original running script, so you can copy the arguments you use.

@silverkin @dlandon freeender Its rather time consuming to investigate for me and I don't have the details to detect what you are seeing and why. I really need to see the non-short version of the log file written. It contains lots of important information. You cannot just update args inside the script and add filters, it breaks other stuff such as determining which folders are under backup. Ah, maybe you are trying to just avoid backup_news via find filters? You cannot do that, an exclude is required. I didn't remember that before spending some hours reading the script, to review the change request... The '-type f' you found only affects initial counts, and it writes the find statement itself, because it has already built a list of relevant dirs. Removing the -type f, wont make it scan less, as the dirs to scan is in a for outside. These things will probably break if you hack the script, and enter overwrite the default args intead of using the proper settings. See build_dir_list function, which need to work for count_files to work. I think you broke build_dir_list by the manual edit of the file. Best Alex

I've added a feature request now: https://forums.unraid.net/topic/191994-common-virtual-pool-for-all-pools-array-cache-mounted-devices-regarding-split-level-and-choice-for-placement-of-new-files/

I would like to control whether folders are stored on SSD or HDD storage, and/or possible my tmpssd storage from separate device without mirror. This way I can move folders from hdd to ssd, as my ssd storage increases over time, and/or my needs change. I can do that move without having to change location used by scripts and people for my files, because unRaid as this cool /mnt/user view of everything. Currently in Shares I can choose Primary storage / secondary storage. Here I want Primary storage as the Common virtual pool, and then all disks should appear in include/exclude, and split level / new-files affect all regardless of source pool. I think this is a more generically useful abstraction that abstracts away the pool setup from the shares. I don't care about the mover anymore, so for this I wouldn't need a mover, and if that should change, I would just create a script to do the moving, which is easy compared to changing how unraid selects new file location, which I think is a black box I cannot change. See more details about my use case here: https://forums.unraid.net/topic/191966-howto-use-common-share-for-2-separate-pools-one-with-ssds-and-one-with-hdds-where-i-control-what-goes-where/

In these days of expanding SSD storage, I'm considering consolidating eg. music on ssd-storages, but keeping videos on hdd's. I access shares via samba from windows, and would like samba .Recycle.Bin to work for deletions. The .Recycle.Bin does not work if I use softlinks from my hdd-mounted share to ssd-storage, because the samba rename() when across different file-systems. Same goes when mounting ssd folder with 'mount --bind'. Currently I have m: which is /mnt/user/media, and I want m:\music on ssd and m:\video on hdd. I also prefer not to move musik to different share, because then I have to teach people and update programs to where the new location is. Its not as elegant as just changing a pointer in the filesystem, when I want to change storage-medium hdd <> ssd. It seems like an obvious think for unraid using its pools and is fuser which sums share from across multiple disks. However if I create share which have only primary storage cache-pool, then it always writes new files to that primary storage, even if folder I write is already on the disk-pool. Likewise if I add secondary storage as disk-pool, I get same behaviour. I could just let all new files be written to cache, but I dislike that, because then when writing huge files they have to be moved, and writing can fail because of lack of space on the cache ssd-storage. I have learned the hard way not to use the built in mover, and just write directly to the pool where I want it. So the question is: Can I somehow get control over in unraid which pool it chooses for new files, by simply selecting the pool the parent folder is already on? Might this be a desirable feature request for unraid for others than me ? Are the other ways to achieve the virtual layout of my files that I want, with recycle bin still working, but me regaining control over which (root)-folders go on SSD vs HDD? Best Alex

This is what my log is showing: ``` $ alias tailcachelog alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log' $ tailcachelog cache_dirs version 2.2.7 Setting Memory ulimit to 100000 Setting cache_pressure=1 Arguments=-p 1 -S -i appdata -i backup -i md5 -i media -i secure -i storage -i sync -i windowsbackup -e app -e appbackup -e cacheonly -e docker -e home -e muaddibhdd -e nanite -e pil -e svn -e svnbackup -e system -e ubuntu -e windowsbackup -c 4 -W 61 -Z 149 -U 100000 -l on -a -noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print Max Scan Secs=10, Min Scan Secs=1 Scan Type=adaptive Min Scan Depth=4 Max Scan Depth=none Use Command='find -noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print' ---------- Caching Directories --------------- appdata backup md5 media secure storage sync ---------------------------------------------- Setting Included dirs: appdata,backup,md5,media,secure,storage,sync,windowsbackup Setting Excluded dirs: app,appbackup,cacheonly,docker,home,muaddibhdd,nanite,pil,svn,svnbackup,system,ubuntu,windowsbackup min_disk_idle_before_restarting_scan_sec=61 scan_timeout_sec_idle=150 scan_timeout_sec_busy=30 scan_timeout_sec_stable=149 frequency_of_full_depth_scan_sec=604800 ERROR: excluded directory 'cacheonly' does not exist. ERROR: excluded directory 'nanite' does not exist. cache_dirs started 2025.05.23 19:24:23 Executed find in (72s) 72.40s, wavg=72.40s Idle____________ depth 4(timeout 30s) slept 0s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=2 scan_tmo=30s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=16%, filecount[4]=52198 2025.05.23 19:25:36 Executed find in (51s) 51.23s, wavg=58.29s Idle____________ depth 5 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=10%, filecount[5]=130628 2025.05.23 19:26:28 Executed find in (22s) 22.75s, wavg=40.52s Idle____________ depth 6 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=12%, filecount[6]=203844 2025.05.23 19:26:52 Executed find in (19s) 19.19s, wavg=31.99s Idle____________ depth 7 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU= 9%, filecount[7]=297086 2025.05.23 19:27:12 Executed find in (21s) 21.86s, wavg=28.61s Idle____________ depth 8 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU= 6%, filecount[8]=356281 2025.05.23 19:27:35 Executed find in (15s) 15.68s, wavg=24.92s Idle____________ depth 9 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU= 9%, filecount[9]=410859 2025.05.23 19:27:52 Executed find in (4s) 04.29s, wavg=19.76s Idle____________ depth 10 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=16%, filecount[10]=430518 2025.05.23 19:27:57 Executed find in (3s) 03.98s, wavg=16.25s Idle____________ depth 11 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=18%, filecount[11]=448640 2025.05.23 19:28:02 Executed find in (5s) 05.83s, wavg=14.17s Idle____________ depth 12 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=14%, filecount[12]=465810 2025.05.23 19:28:09 Executed find in (8s) 08.79s, wavg=13.19s Idle____________ depth 13 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=13%, filecount[13]=485555 2025.05.23 19:28:19 Executed find in (8s) 08.55s, wavg=12.42s Idle____________ depth 14 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=13%, filecount[14]=505355 2025.05.23 19:28:28 Executed find in (5s) 05.67s, wavg=11.38s Idle____________ depth 15 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=16%, filecount[15]=518085 2025.05.23 19:28:35 Executed find in (2s) 02.66s, wavg=10.13s Idle____________ depth 16 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=26%, filecount[16]=526342 2025.05.23 19:28:39 Executed find in (2s) 02.55s, wavg=09.12s Idle____________ depth 17 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=28%, filecount[17]=533061 2025.05.23 19:28:42 Executed find in (2s) 02.28s, wavg=08.27s Idle____________ depth 18 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=33%, filecount[18]=540454 2025.05.23 19:28:46 Executed find in (2s) 02.27s, wavg=07.56s Idle____________ depth 19 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=31%, filecount[19]=544809 2025.05.23 19:28:49 Executed find in (2s) 02.27s, wavg=06.97s Idle____________ depth 20 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=29%, filecount[20]=546436 2025.05.23 19:28:52 Executed find in (2s) 02.30s, wavg=06.48s Idle____________ depth 9999 slept 1s Disks idle before/after scan 9998s/9998s Scan completed/timedOut counter cnt=0/1/0 mode=1 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=0 CPU=29%, filecount[9999]=546900 I'm running v2.2.7: see head -n 150 /usr/local/bin/cache_dirs ``` # Version 2.2.7 - Fixed -a option which was broken in 2.2 # arberg #################################################################################### version=2.2.7 program_name=$(basename "$0") arg_count=$# run_log="/var/log/cache_dirs.log" csv_log="/var/log/cache_dirs.csv" lost_cache_log="/var/log/cache_dirs_lost_cache.csv" ``` Notice this at the top of the log file: ``` ---------- Caching Directories --------------- appdata backup md5 media secure storage sync ``` My cfg looks like this: config\plugins\dynamix.cache.dirs\dynamix.cache.dirs.cfg ``` options="-p 1 -S -i "appdata" -i "backup" -i "md5" -i "media" -i "secure" -i "storage" -i "sync" -i "windowsbackup" -e "app" -e "appbackup" -e "cacheonly" -e "docker" -e "home" -e "muaddibhdd" -e "nanite" -e "pil" -e "svn" -e "svnbackup" -e "system" -e "ubuntu" -e "windowsbackup" -c 4 -W 61 -Z 149 -U 100000 -l on -a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" service="1" suspend="-S" include="appdata,backup,md5,media,secure,storage,sync,windowsbackup" exclude="app,appbackup,cacheonly,docker,home,muaddibhdd,nanite,pil,svn,svnbackup,system,ubuntu,windowsbackup" adaptive="1" depth="" scanTimeoutIdle="" scanTimeoutBusy="" scanTimeoutStable="149" minimum="" maximum="" log="on" other="-a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" diskIdleTimer="61" cachePressure="1" ulimit="100000" mulithreaded="" scan_user_share="-u" mindepth="2" minDepth="4" shares="" ``` If you play around with it, you can probably solve your problem. Play around with the cfg and start and stop the script manually and watch the log file. Or rather that's probably what I would do. You can also find the command it runs with ps -ef | grep cache And check ps -ef | grep find to see what it is calling find on. Though you have to catch. You can also ask deepseek for how to get process tree from the cache_dirs execution.

I also had problem logging in with SSH after updating from 7.0.1 to 7.1.2. For me it turned out I was missing an /etc/rc.d/rc.sshd restart after my plugin-script had copied my personalized version of the sshd_config to /etc/ssh. Apparently things ran in a different order before, or for some other reason I didn't have to restart th sshd before .

that's what the log is there for. Its very good for figuring out what's going on in these cases... It also counts number of files and shows scan times...

Zero means only scan content of /mnt/disk3/ not subfolders /mnt/disk3/*/ Regarding the memory question, you are probably refering to ulimit: https://phoenixnap.com/kb/ulimit-linux-command You can also check the script it self, its probably still this version of the script that's the active, I certainly haven't updated it since: https://github.com/bergware/dynamix/blob/master/source/cache-dirs/scripts/cache_dirs The script above runs ulimit. On my system cache_dirs is installed here, but it might have been moved in the public plugin: /usr/local/bin/cache_dirs

This is my cfg file, f:\config\plugins\dynamix.cache.dirs\dynamix.cache.dirs.cfg ``` options="-p 1 -S -i "appdata" -i "backup" -i "md5" -i "media" -i "secure" -i "storage" -i "sync" -e "system" -e "ubuntu" -e "windowsbackup" -c 4 -W 61 -Z 149 -U 100000 -l on -a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" service="1" suspend="-S" include="appdata,backup" exclude="app,appbackup" adaptive="1" depth="" scanTimeoutIdle="" scanTimeoutBusy="" scanTimeoutStable="149" minimum="" maximum="" log="on" other="-a '-noleaf -name .Recycle.Bin -prune -o -name log -prune -o -name temp -prune -o -name .sync -prune -o -print'" diskIdleTimer="61" cachePressure="1" ulimit="100000" mulithreaded="" scan_user_share="-u" mindepth="2" minDepth="4" shares="" ``` The important one is 1) Don't cache more than you need, start small, so there's not so many files 2) Avoid user share, named as `Scan user shares (/mnt/user)` in the gui In my tests on unraid 7.0 it didn't spin up disks even when accessing user shares, even though only the disk shares were cached, which is good and expected. Scanning user shares is more expensive cpu wise. And check my comment above about checking the log, it gives you lots of good information about files on your system and the time it took to scan.

Duc, just to be sure, you waited some minutes to see if CPU would drop right? It's natural to go high CPU initially as it scans the disks. You can enable logging and follow what is logged to see what is going on, and compare it with cpu stats from top or similar. I have this in my .bash_aliases so its a breeze to check up on it, when I think it misbehaves: alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log'

I just checked with my 7.0.0-rc.2 and it works fine. I don't have it set to scan user shares, and even despite that it didn't spin up disks when browsing user share via samba or directly in ssh terminal for folders that only were on that spun down disk. It also seems to work fine on 7.0.0

I had the same problem, with github error as Joe mentioned and the hanging > Performing actions > Please keep this window open while we perform some actions In my case it was uMatrix browser extension that blocked my unraid from getting data from unraid.net and blocked unraid from contacting github.

I would probably check if auto-start is enabled, and if so disable it. It seems to be in disk.cfg (probably: startArray="no"), but since you have access to the webgui, you can just disable it there, restore your super.dat, and reboot. I think super.dat contains your disk spec, and it would appear so on your experience. > If I reassign all of the disks to their original spot in the array, should everything come up as expected? Yes. But I would definitely mount as read only and not let in recreate the parity, but instead validate the parity is correct not writing corrections, at least for some gigabytes. I'm not quite sure what the options are. We wouldn't want it to overwrite your good data with parity if you mount incorrectly. You can mount the disks in read-only mode safely manually in a terminal if you so desire, to check the data is there. If you only have one parity disk, the order of your array disks shouldn't matter for the parity. Though of cause it might very well matter for your shares, if some shares are on specific disks. Do you have stored information elsewhere about which disks were array-disks and which were parity? Probably not... I checked my disks ids and I don't see them in my syslog. Remember to check your syslog for disk errors before and after recreating the array. Alex.

well, be that as it may, is it possible?

I want to have my music on always ready SSD storage, while keeping my movies on slow moving ssd spinning disks. I don't want to wait for spinups should I add new music files. Is this possible? It was possible, I believe, in the older unRaid version before the primary/secondary storage change to configuration. Back then I think I could just manually move folder to cache, 'disable' split-levels and tell it never to use cache ('prefer: no', or something) When using primary/secondary storage, then new files will always be written to primary storage (ie. cache). I kind of prefer not to have that, but the worse part is that I cannot disable mover in this setting, and thus my files get moved back to disks by mover. If I use primary=array, and no secondary storage, new files will be added to array, which I don't want. The only solution I found for now is to create a separate ssd share on my cache drive with cache only storage (primary), and then moves files here and create links from original share. This works, but isn't really super UnRaid niceness. I really like unRaid's /mnt/user that gives me a shared perspective of where files are, and then my /mnt/disk*, /mnt/cache which shows me what files are there, and where files are the expected locations with the only question being which disk contains the files, not where they are linked from. Is this on the potential future feature-list? I saw some mover improvements here, though I don't know what those were:

Install NerdTools, and in it under Settings > NerdTools, install lftp

I'm running unraid 6.12.4. This is likely not unRaid specific, but I don't know. TLDR: I can mount a LUKS drive but sometimes when on luksClose it won't close, but so far waiting 15 minutes helps. I tried creating a LUKS encrypted file using this guide: https://blog.tian.it/create-encrypted-luks-sparse-file/ When I unmount the mounted filesystem and then try to close the LUKS using ``` umount /cryptoarchive/stuff cryptsetup luksClose "/dev/mapper/stuff" ``` then I get the following error: ``` Device /dev/mapper/stuff.img is still in use. ``` Running `df` I can see the unmount worked, and also the `umount` didn't fail. `lsof` shows me that the loop-device file is (of cause) still in use, but no files on the mounted dir `/cryptoarchive/stuff` is shown . Once after waiting an hour or so, I tried remounting and then umounting immediately afterwards, and that one time luksClose succeeded. I have shared the mount point via samba, but as mentioned umount works. I have attached my mount and unmount scripts. Thx, Alex luks-unmount.sh luks-mount.sh

Ah, I had forgotten that part of the mover semantics, that the mover will leave any file currently in use. That kind of makes my hypothesis less likely. Though it might have started moving a file that then becomes used. I haven't seen the problem at all since changing my duplicati + Resilio Sync dockers to use /mnt/disk* and /mnt/cache instead of /mnt/user. Before it was there nearly once a month or so. But now I just reliably reproduced the issue. Though the reliable way is unfortunately just by calling the move script incorrectly. But maybe you can find something. This makes it way more likely that the actual bug is in the move script or how the move-script uses shfs (or something...), since it still manages to crash the shfs process Note that this thing I reproduced is not something I have done in the previous error-reports, previously it seemed quite reasonable that it was the actual mover causing the error, not my custom move-script, due to the time-of-day was exactly when the actual mover ran. --- I configured on of my shares to be cache-only. Currently the shares files are on disk1. This caused the issue immediately each time I run my custom mover which use Limetechs move-command, (but use it incorrectly, as I discovered) I now know that I can reproduce the issue when neither of these were running: Docker stopped Parity check not running Mover disabled using touch /var/run/mover.pid, and check its not running To crash the server I simply run: echo /mnt/user0/home/svnbackup/dead.letter | /usr/local/bin/move -e 0xfffffffe Other files on same share give the same result, when using /mnt/user0 `0xfffffffe` should have been the magic word for move to cache (many years ago). However I can see that the mover script has changed and it no longer uses the new 'move' command in this way, so maybe this problem is something that has nothing to do with the other problem. Though it seems to crash in a similar way (different code, but also different unRaid version). If I run this (which seems to be what mover does) then I get: > echo /mnt/user0/home/svnbackup/dead.letter | /usr/local/bin/move "" move_object: /mnt/user0/home/svnbackup/dead.letter File exists > find /mnt/*/home/svnbackup -maxdepth 0 /mnt/disk1/home/svnbackup /mnt/user/home/svnbackup /mnt/user0/home/svnbackup If I run it CORRECTLY then I get - using disk1 instead of user0: > echo /mnt/disk1/home/svnbackup/dead.letter | /usr/local/bin/move "" then it just moves the file to cache as it should. I'm running unRaid 6.12.4 now. PS: I'm now following the topic, so maybe that should help me discover late comments. tower-diagnostics-20231105-1706-shfs-crash-5-anon.zip

Right that fits with my experience. So it seems that when a unRaid process moves a file away from a disk via disk share /mnt/disk1 (such as the mover process) while a docker container accesses (reads?) the file via the '/mnt/user'-share, then it'll crash the shfs. It's probably the remove operation 'rm /mnt/disk1/someFile' that conflics with shfs reading that file, possible it could also be related to that same file appearing in shfs on another disk. It's vague understanding that it is shfs that creates the virtual image /mnt/user as the sum of /mnt/disk*. So I would say if someone wants to reproduce this issue, like just create a docker that reads a large GB sized file from /mnt/user and then have a mover process running directly on unraid that moves that file back and forth between two disks. The reason why I was only seeing it in the night and not from my custom mover is almost certainly that my custom mover runs every 5 minutes checking disks are idle and only then moving files. */5 * * * * /home/alex/bin/move_if_idle > /dev/null See attached files, for content of my move_if_idle move_if_idle are_disks_idle

Thank you, I havn't noticed that line as being the first line, or I haven't previously followup on it. It does (did) keep happening. Now I have changed my dockers which access lots of files (backup+Resilio sync) and only allow those to access my files via disk-shares and so far it looks good. Its a bit to early to tell though. I'll report back in a few weeks, with whether it has been stable by then. It was happening maybe once-twice a week when my backup was processing all my files checking checksums through /mnt/user. I've been exporting diagnostics when it happened many times back last year, I just checked previous logs. 2022: Aug 22 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup/.sync/Archive Aug 22 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup/.sync Aug 22 03:40:02 Tower kernel: shfs[23214]: segfault at 0 ip 00000000004043a4 sp 0000152d0c670780 error 4 in shfs[402000+c000] Aug 22 03:40:02 Tower kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 43 8b 05 8f df 00 00 85 c0 78 2f e8 16 e0 ff ff Aug 22 03:40:02 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup Aug 22 03:40:02 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync Aug 29 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup/.sync/Archive Aug 29 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup/.sync Aug 29 03:40:01 Tower kernel: shfs[1963]: segfault at 0 ip 00000000004043a4 sp 0000147af70f3780 error 4 in shfs[402000+c000] Aug 29 03:40:01 Tower kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 43 8b 05 8f df 00 00 85 c0 78 2f e8 16 e0 ff ff Aug 29 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup Aug 29 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync Sep 2 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup/.sync/Archive Sep 2 03:40:01 Tower move: error: move, 392: No such file or directory (2): lstat: /mnt/cache/sync/AppBackup/.sync Sep 2 03:40:01 Tower kernel: shfs[16416]: segfault at 0 ip 00000000004043a4 sp 0000152977cfb780 error 4 in shfs[402000+c000] Sep 2 03:40:01 Tower kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 43 8b 05 8f df 00 00 85 c0 78 2f e8 16 e0 ff ff Sep 16 03:40:02 Tower kernel: shfs[27859]: segfault at 0 ip 00000000004043a4 sp 000014caa2cb2780 error 4 in shfs[402000+c000] Sep 16 03:40:02 Tower kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 43 8b 05 8f df 00 00 85 c0 78 2f e8 16 e0 ff ff 2023: Jun 18 03:40:02 Tower kernel: shfs[4635]: segfault at 0 ip 000055c0d617359c sp 0000150db53fa810 error 4 in shfs[55c0d6171000+c000] Jun 18 03:40:02 Tower kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 4d 8b 05 67 dd 00 00 85 c0 78 39 e8 9e da ff ff Aug 10 03:40:02 Tower kernel: shfs[8688]: segfault at 0 ip 000055a8e226f59c sp 000014bf269d9810 error 4 in shfs[55a8e226d000+c000] Aug 10 03:40:02 Tower kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 4d 8b 05 67 dd 00 00 85 c0 78 39 e8 9e da ff ff I picture presents it self. I have lots of things running during the day, but I also log whenever run-parts execute something, and it seems in the above examples that the previous onces finished long before. Like lines like these: Aug 29 03:24:49 Tower /etc/cron.daily/userScripts/run_vps_up_test: Executing Aug 29 03:24:49 Tower /etc/cron.daily/userScripts: run-parts completed I also checked all my other crontab-jobs and userScripts, and I'm fairly certain I have nothing else running at that specific time. Hmm, so I can just disable the mover and then my problems will probably be fixed? That does sound kind of very strange given that I'm using my own custom mover that is based on mover-codebase and runs the mover codebase regularly throughout the day. I run this 0:40 every day, so if it was just the mover script then this should also have triggered the issue some times (unless other script of mine have turned off the docker to make backups, but I don't think I have that any longer): #!/usr/bin/bash $ALEXBIN/custom_mover /usr/local/sbin/mover > /dev/null # Completion check was previously moved to appBackup which turns off docker and services: \\towerAlex\appbackup\backupRsync # but now back here as there's no longer an appbackup mover_completion_check I don't know if the scheduled mover does anything different that running '/usr/local/sbin/mover'. --- I'll leave my docker changes so stuff run on /mnt/disk* for the next month or two. I'll also change the mover schedule to run at another time, so if it crashes again I would expect a new timestamp. If it runs smoothly for a couple of months, I'll try changing by dockers to use /mnt/user again, and see it the problem the resurfaces. Will it be helpful to track the issue down to anyone else if I report back and try to recreate the issue? PS: I updated the title to include 'shfs segfault', to make it easier for others to find this issue.

Thank you itimpi. Here's my diagnostics. I have further anonymized the syslog. It echoed all my sudo-lines, and lots of scripts I was running, I have removed those lines that seemed private, but kept those at the end around where the problem arises, in the hope of not removing anything important for the diagnostic-process. Best Alex tower-diagnostics-20230729-0846-anonymized-extra-anonymized.zip

I have this problem and have had it for probably a couple of years now. $ ls /mnt/user /bin/ls: cannot access '/mnt/user': Transport endpoint is not connected There have been many others telling of the same problem: https://forums.unraid.net/topic/102568-transport-endpoint-is-not-connected-with-mntuser/ https://forums.unraid.net/topic/115300-binls-cannot-access-user-transport-endpoint-is-not-connected/ - The first mentions it could be docker accessing /mnt/user. This could very well fit my usecase. Because I use Duplicati (running via docker) to backup my /mnt/disk* and /mnt/user, and let Duplicati deduplicati the double backup. So if a disk+parity fails I can restore files from that disk, and if I lose a /mnt/user/ folder I can restore that across all disks. Has any found a solution to this problem? Is there a different way of using docker, so that it can access /mnt/user without the issue? Has anyone determined that docker IS the cause of this issue (at least in their case) ?

There's a bug in the detection of whether WORK_DIR is on persistent storage, so it always says persistent, unless its on 'ramfs' (the first match in the array). This code fixes it: Adding 'tmpfs' to array, and don't break on first mismatch (in .plg file) denyhosts_datacheck() { array=( ramfs proc tempfs sysfs tmpfs ) fs=$( stat -f -c '%T' $WORK_DIR ) if [ "$fs" = "msdos" ]; then echo "<p style="color:red\;"><b>WARNING:</b> Your WORK_DIR is located on your flash drive. This can decrease the life span of your flash device!</p>" else found=0 for i in "${array[@]}" do if [[ "'$i'" = "'$fs'" ]]; then echo "<p style="color:red\;"><b>WARNING:</b> Your WORK_DIR is not persistent and WILL NOT survive a reboot. The WORK_DIR maintains a running history of past DenyHosts entries and ideally should be maintained across reboots. Please locate your WORK_DIR on persistent storage. eg. cache/array disk</p>" found=1 break fi done if (( ! $found )) ;then echo "<p style="color:green\;">WORK_DIR located on persistent storage. Your data will persist after a reboot :-)</p>" fi fi } I'm not really sure whether its a good idea to put it on /boot due to spamming writes on USB, and also I would prefer it to be not dependent on /mnt being available. So I suspect the best option would be copying to/from /boot on start/stop or mount or something like that. What do others do to persist the data, and what is the data? I would be fine with moving the deny-lists to /boot as I expect those are not written frequently, unless of cause I'm spammed from unlimited number of ipv6 addresses... (if that can happen...)