tcharron

Great - thanks for that detail.

I came across an article saying that docker will be moving to limit pulls: https://www.theregister.com/2025/02/22/docker_hub_pull_limits/ For non-paying customers, unverified users will be limited to 10 pulls per IP per hour, while verified users will have 40. It might be useful to allow users to inform Unraid of credentials that should be used.

The clear drive script from this thread is from 2016, and the author hasn't been to this board since 2017. I found 3 edits needed to clean it up and have it work well for me: 1) The script does not work for unraid >6.9 since it parses the unraid version # wrong. If you are running >=6.10 then change the detection code to this: # check unRAID version v1=`cat /etc/unraid-version` v2="${v1:9:1}" if [[ $v2 -ge 7 ]] then v=" status=progress" else v2="${v1:9:1}${v1:11:2}" if [[ $v2 -ge 610 ]] then v=" status=progress" else v="" fi fi #echo -e "v1=$v1 v2=$v2 v=$v\n" 2) With the larger arrays that are common now, the search for drives is slower than it needs to be. Rather than check for the marker and emptiness at once, splitting that up makes the check much faster: for d in /mnt/disk[1-9]* do #echo -e "ls -A d:$d\n" x=`ls -A $d` #echo $x #echo -e "d:"$d "x:"${x:0:20} # the test for marker if [ "$x" == "$marker" ] then z=`du -s $d` y=${z:0:1} #echo -e "d:"$d "x:"${x:0:20} "y:"$y "z:"$z # the test for emptiness if [ "$y" == "0" ] then found=1 #echo -e "d:"$d "is empty and has the 'clear-me' marker" break fi fi let n=n+1 done 3) The device names have changed from /dev/mdX to /dev/mdXp1. So change the script references from md${d:9} to md${d:9}p1 . There are 5 such references in the script (2 of which are commented out). I'm hesitating to share the full edited script here as I would rather see others validate the above.

You're right - I will just disable it until I need it. However, I think that whenever I do go to use it down the road, it will still cause a long pause while it counts directories. I had removed cache_dirs many years ago - I see that you've improved it since then and I will install again.

The "slow array start" problem is due to a problem with the file.activity plugin. That plugin was updated November 25 2024, but users will not be affected until the array is restarted. I described the issue here: https://forums.unraid.net/topic/54808-file-activity-plugin-how-can-i-figure-out-what-keeps-spinning-up-my-disks/page/17/#findComment-1512201

The November 25 change has caused my array start to take 35 minutes longer than it used to. I had not noticed it until now as I did not reboot my server or bring the array down then up. The problem code is this: # Loop through each mount point in the file while IFS= read -r mount_point; do if [ -d "$mount_point" ]; then # Count directories in the mount point count=$(find "$mount_point" -type d 2>/dev/null | wc -l) dir_count=$((dir_count + count)) fi done < "$disks_file" The above creates a series of 'find' commands on each of the drives. On my array, this takes 34 minutes. What happens at startup is that /mnt/cache and /mnt/user are 'up', but until the above code finishes, samba shares are not available to clients (the webgui also pauses if the array start was done via the gui). This code is a good idea, but given the above problem it should be included as an option that is by default disabled.

My notifications are set correctly. The point of my message here was to highlight that the unraid updater is sending messages differently than I expected. here is the header for the udpater message: from: Console and webGui login account <[email protected]> to: root bcc: [email protected] date: Jan 14, 2025, 12:39 PM subject: Output from your job 6 mailed-by: gmail.com I receive the above message since my chosen destination is on the bcc line. I agree that the sender is 'root', but the sender is not '[email protected]' as was captured in the syslog. It's a bit odd to me that the udpater does that, when every other app that sends notifications puts the proper "to:" field instead of a bcc field. Here is the syslog entry for the message above, and the next notification that was sent... Jan 14 12:39:06 Tower sSMTP[3695]: Sent mail for [email protected] (221 2.0.0 closing connection 6a1803df08f44-6dfade72a80sm55481256d6.81 - gsmtp) uid=0 username=root outbytes=405 Jan 14 12:41:25 Tower sSMTP[17049]: Sent mail for [email protected] (221 2.0.0 closing connection af79cd13be357-7bce3248c9fsm626070685a.50 - gsmtp) uid=0 username=root outbytes=797

Sorry but I don't follow your point. Yes I have an SMTP email configured. The existing SMTP settings work fine for notifications from other applications. How do I avoid syslog trying to use [email protected]? The account on unraid sending the message is root (root@localhost), but that should not translate to [email protected].

The syslog server page (Settings->Syslog) does not have notification settings. I have notifications configured in the unraid Settings->notifications page, but that is using a gmail address of my own, not [email protected].

By explaining how I thought this should work (as I outlined above), I was able to get ChatGPT to admit that it was wrong !

I am running 6.12.10. When the array is brought online, it still takes forever. The array and cache are up, but something is causing a huge pause in activity, and the SMB shares are not available for around 30 minutes or more. I looked at the "main" page in a browser, and can see the 'reads' count rising slowly for each of my drives (sequentially, not all at once). I was able to use 'ps' and figured out that it is caused by these processes: UID PID PPID C STIME TTY TIME CMD root 4277 4276 0 23:34 ? 00:00:01 find /mnt/disk4 -type d root 4276 22055 0 23:34 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/file.activity/scripts/rc.file.activity start root 22055 22054 0 23:14 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/file.activity/scripts/rc.file.activity start root 22054 21657 0 23:14 ? 00:00:00 /bin/bash /usr/local/emhttp/plugins/file.activity/event/disks_mounted disks_mounted root 21657 10575 0 23:14 ? 00:00:00 /bin/bash /usr/local/sbin/emhttp_event disks_mounted root 10575 1 0 Jan13 ? 00:04:13 /usr/local/bin/emhttpd I could see the 'find' command above progressing through each of my drives, and the 'read' counts risin gin the web guii. Once that finished with the last drive, the array status in the web gui changed from "starting..." to "started", and the SMB shares become available. My syslog showed a 34 minute gap just now as I brought the array up: Jan 14 23:14:15 Tower emhttpd: Starting File Activity... Jan 14 23:14:16 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="22046" x-info="https://www.rsyslog.com"] start Jan 14 23:48:07 Tower file.activity: File Activity inotify starting Jan 14 23:48:07 Tower inotifywait[25916]: Setting up watches. Beware: since -r was given, this may take a while! Jan 14 23:48:09 Tower unassigned.devices: Mounting 'Auto Mount' Devices... Jan 14 23:48:09 Tower emhttpd: Starting services... Jan 14 23:48:09 Tower emhttpd: shcmd (202405): /etc/rc.d/rc.samba restart Jan 14 23:48:11 Tower root: Starting Samba: /usr/sbin/smbd -D I wonder if the problem is file.activity and the related inotify? That message didn't show up in the log until after the 34 minute pause, but that seems like the kind of thing that would be scanning files.

I think that was related to an an unbalanced pool (which needed a btrfs balance to correct). I removed my 240GB drive from the pool. Unraid now reports usuable space on the pool of 511GB (formerly 512GB). Note that the pool size went down (from 589GB to 477GB), but used and available did not change whatsoever, despite removal of the 240GB drive ! : # btrfs fi usage -T /mnt/cache Overall: Device size: 953.88GiB Device allocated: 426.06GiB Device unallocated: 527.82GiB Device missing: 0.00B Device slack: 0.00B Used: 422.18GiB Free (estimated): 264.25GiB (min: 264.25GiB) Free (statfs, df): 264.25GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 264.42MiB (used: 0.00B) Multiple profiles: no Data Metadata System Id Path RAID1 RAID1 RAID1 Unallocated Total Slack -- --------- --------- --------- -------- ----------- --------- ----- 1 /dev/sdf1 211.00GiB 2.00GiB 32.00MiB 263.91GiB 476.94GiB - 2 /dev/sdj1 211.00GiB 2.00GiB 32.00MiB 263.91GiB 476.94GiB - -- --------- --------- --------- -------- ----------- --------- ----- Total 211.00GiB 2.00GiB 32.00MiB 527.82GiB 953.88GiB 0.00B Used 210.66GiB 444.53MiB 48.00KiB # df /mnt/cache Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdf1 500107080 221614288 277087344 45% /mnt/cache # df /mnt/cache -h Filesystem Size Used Avail Use% Mounted on /dev/sdf1 477G 212G 265G 45% /mnt/cache This is a big win from my perspective -- one less drive in the system with no downside. Also worth mentioning for anyone who finds this... the carfax calculator I linked to above is wrong (since it showed 0 unusable space) !

i figured it out -- It doesn't work like I thought. I originally thought it should all 'fit' like this: Drive 1 (512GB): Mirrors 120GB to Drive 3 (240GB) + 392GB to Drive 2 (512GB) = 512GB used. Drive 2 (512GB): Mirrors 120GB to Drive 3 (240GB) + 392GB to Drive 1 (512GB) = 512GB used. Drive 3 (240GB): Mirrors 240GB to each of the 512GB drives = 240GB used. --> Total usable space (excluding overhead): 512+240/2 = 632GB It turns out that btrfs does not support that kind of 'creative' block distribution. It can't 'plan' to have the 240GB split across two drives. For some fun, I tried asking ChatGPT. I'm pretty sure it lied to me. It said I would only have 480GB of usable space and said that this is how my data would be allocated: Drive 1 (512GB): Mirrors 240GB to Drive 3 (240GB) + 240GB to Drive 2 (512GB) = 480GB used. Drive 2 (512GB): Mirrors 240GB to Drive 3 (240GB) + 240GB to Drive 1 (512GB) = 480GB used. Drive 3 (240GB): Mirrors 240GB to each of the 512GB drives = 240GB used. --> Total usable space (according to chatgpt): 480GB The problem with the above is that it suggests that each of drive 1 and drive 2 are mirroring 240GB to drive 3... but drive 3 can't store 480GB ! I think that the real 'answer' that matches my data above (ie 512GB of usable storage) is this: Drive 1 (512GB): Mirrors 240GB to Drive 3 (240GB) + 272GB to Drive 2 (512GB) = 512GB used (0 unused) Drive 2 (512GB): Mirrors 272GB to Drive 1 (512GB) = 272GB used (and 240GB unused) Drive 3 (240GB): Mirrors 240GB to Drive 1 (512GB) = 240GB used (0 unused) --> Total usable space (excluding overhead): 512GB What I've learned is that my choice of drives is poor. The 240GB drive can be removed entirely from this pool, with the only impact being that the metadata would have to be backed up only once instead of twice. I would then have 512GB of mirrored data, which after overhead should still be over 500GB of accessible storage! Drive 1 (512GB): Mirrors 512GB to Drive 2 (512GB) = 512GB used (0 unused) Drive 2 (512GB): Mirrors 512GB to Drive 1 (512GB) = 512GB used (0 unused) --> Total usable space (excluding overhead): still 512GB

I did configure the pool's metadata to Raid1c3, so there is 3 copies of metadata instead of the usual 2 that would correspond to raid1. The table above confirms this -- it shows 1G of metadata on each drive. However, that's still only a total of 3G. Even with 3G metadata and another 1% for other overhead, I still have over 90G 'missing'. The carfax btrfs calculator shows that I shouldn't have any unusable space -- https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=0&d=240&d=512&d=512 Maybe as the drive fills up the 'missing' data will shrink. I might do a test for that.

I have 3 drives in a cache pool. They are 512GB, 512GB, and 240GB. I would have expected that I would have just under half of the total of this as usuable cache space -- ie ~630GB. However, I end up with a lot less. # btrfs fi usage -T /mnt/cache Overall: Device size: 1.15TiB Device allocated: 425.09GiB Device unallocated: 752.36GiB Device missing: 0.00B Device slack: 0.00B Used: 423.11GiB Free (estimated): 376.27GiB (min: 250.88GiB) Free (statfs, df): 265.00GiB Data ratio: 2.00 Metadata ratio: 3.00 Global reserve: 264.48MiB (used: 0.00B) Multiple profiles: no Data Metadata System Id Path RAID1 RAID1C3 RAID1C3 Unallocated Total Slack -- --------- --------- --------- -------- ----------- --------- ----- 1 /dev/sdf1 210.00GiB 1.00GiB 32.00MiB 265.91GiB 476.94GiB - 2 /dev/sdj1 211.00GiB 1.00GiB 32.00MiB 264.91GiB 476.94GiB - 3 /dev/sdl1 1.00GiB 1.00GiB 32.00MiB 221.54GiB 223.57GiB - -- --------- --------- --------- -------- ----------- --------- ----- Total 211.00GiB 1.00GiB 32.00MiB 752.36GiB 1.15TiB 0.00B Used 210.90GiB 444.69MiB 48.00KiB # df /mnt/cache Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdf1 617322596 222101324 277876812 45% /mnt/cache # df /mnt/cache -h Filesystem Size Used Avail Use% Mounted on /dev/sdf1 589G 212G 266G 45% /mnt/cache The 589G from df corresponds to 617322596*1024 bytes, and that is exactly 50% of my total SSD capacity. My question is why am I losing close to 20% of that? The Used+Avail space is only 478GB, or 501219328*1024 bytes. The difference is 111GB... Even with the overhead of the filesystem, that's more of a haircut than I expected. I would think that all of my space is usable within btrfs raid1. It should use half of the 240GB (ie 120GB eaach) with each of the 512GB Drives, and then the remaining 389GB on those two drives could be paired to use all of the other space. What am I missing?