On 3/17/2024 at 9:24 PM, warwolf7 said:

(I wish I could create a pullrequest, but I couldn't find the repo on github, the only one I found was 8years old and the code did not match the one in unraid current release)

 

The 6.12 branch of the Unraid webgui repo is here if you would like to submit a PR:

https://github.com/unraid/webgui/tree/6.12

The exact file is:

https://github.com/unraid/webgui/blob/6.12/emhttp/plugins/dynamix/scripts/monitor

 

On 3/17/2024 at 9:24 PM, warwolf7 said:

we can see here that it only looks for 'missing' but when a filesystem is not found, it actually return ERROR:
ERROR: not a valid btrfs filesystem: 6790d0c5-bc2f-499f-a9cb-5b5bf6f03d0z

This should be changed to 'ERROR'

 

Nice troubleshooting. This is not my area of expertise, but your research helps a lot.

 

Do you think this:

grep -c 'missing'

should be changed to:

grep -c 'ERROR'

Or should it check for both?

grep -Ec 'missing|ERROR'

 

 

On 3/17/2024 at 9:24 PM, warwolf7 said:

But since this script execute every minute, and the next section creates a warning on an error notification. it could potentially flood an unraid admin with warning email.

 

Would you mind testing locally by adding '- x' near the end of line 196:

exec("$notify -l '/Main' -e ".escapeshellarg("Unraid $text message")." -s ".escapeshellarg("Warning [$server] - Cache pool BTRFS missing device(s)")." -d ".escapeshellarg("$info")." -i \"warning\" -x 2>/dev/null");

That will give you one notification for this subject per reboot (or until you go to Tools -> Archived Notifications and delete the notification)

 

 

On 4/2/2024 at 5:59 PM, ljm42 said:

 

The 6.12 branch of the Unraid webgui repo is here if you would like to submit a PR:

https://github.com/unraid/webgui/tree/6.12

The exact file is:

https://github.com/unraid/webgui/blob/6.12/emhttp/plugins/dynamix/scripts/monitor

 

 

Nice troubleshooting. This is not my area of expertise, but your research helps a lot.

 

Do you think this:

grep -c 'missing'

should be changed to:

grep -c 'ERROR'

Or should it check for both?

grep -Ec 'missing|ERROR'

 

HI, first of all thank you for looking at this. (I know very very little about programming, so I'm hoping you or someone from unraid would actually look at this and propose something, it would be probably a lot better than me making a pullrequest. I'm somewhat scared to create more problems)

 

I would guess third option, I don't know all the errors that can be thrown. but running this command "btrfs filesystem show invalideuuid" does return "ERROR"

So I re-added the problematic uuid in the pool config file, rebooted, added your third option, but it did not triggered the notification. 

Just to be sure notification were working I created a dummy notification that I worked immediatly. /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "TEST" -d "TEST"

 

this line

if (exec("/sbin/btrfs filesystem show "._var($disk,'uuid')." 2>/dev/null|grep -c 'missing'")>0) {

seems to be removing stderr ( 2>/dev/null). So maybe the intent was something else. Otherwise it would have been something like this no?

if (exec("/sbin/btrfs filesystem show "._var($disk,'uuid')." 2>&1>/dev/null|grep -Ec 'missing|ERROR'")>0) {

and that one triggered notification and works properly. I would absolutly not known whether this should be changed to this or not. I guess the author would know. But I believe that there should be a notification, otherwise the line that calls btrfs filesystem show <uuid> will wake up all drive and leaves no trace to help find the problem.

 

 

On 4/2/2024 at 5:59 PM, ljm42 said:

 

Would you mind testing locally by adding '- x' near the end of line 196:

exec("$notify -l '/Main' -e ".escapeshellarg("Unraid $text message")." -s ".escapeshellarg("Warning [$server] - Cache pool BTRFS missing device(s)")." -d ".escapeshellarg("$info")." -i \"warning\" -x 2>/dev/null");

That will give you one notification for this subject per reboot (or until you go to Tools -> Archived Notifications and delete the notification)

 

 

It seems the script does not trigger a notification every minute even if it runs every minutes and encounter this error with the modified line. So I did not add the "-x". I looked a bit more and I think that the script stores that notification status in this file "/boot/config/plugins/dynamix/monitor.ini" and reads it every minute preventing multiple notification for the same problem.

 

 

 

Now the uuid problem in the config file

This is strange to me. Why would the UUID in the config file "nano /boot/config/pools/poolname.cfg" contains a UUID but that UUID field is not populated in all my new pool configs. Is that uuid only used when the pool contains more than 1 drive/ssd? 

Why would the monitor script look for that specific config uuid information and have a code section just for that if that UUID field is not used anymore.

 

Nonetheless, that uuid that was in my config file created a situation where every "btrfs filesystem show <pooluuid>"  would wake up all drive to find it. To me it seems like the notification should be different to help the admin understand that it could create this sorts of situation.

I think the description should read like this 

"the command btrfs filesystem show <insert uuid> returned an error instead of returning info for this disk <insertpool diskinfo> This error can spin up all your drives. This command is called every minute " something like this, but that works. I don't know how to insert the uuid in the string.

Shouldn't this be an alert instead of a warning?

 

exec("$notify -l '/Main' -e ".escapeshellarg("Unraid $text message")." -s ".escapeshellarg("Warning [$server] - Cache pool BTRFS missing device(s)")." -d ".escapeshellarg("the command btrfs filesystem show <insert uuid> returned an error instead of returning info for this disk [$info] This error can spin up all your drives. This command is called every minute")." -i \"warning\" -x 2>/dev/null");

 

Let me know what you think. 

Thank you

Edited April 16, 20242 yr by warwolf7
more details

I know this is quite some time later, but just wanted to thank @warwolf7 for detailing the steps that they had taken for this.

 

I recently replaced my cache drive, though in my case it was a 1 for 1 replacement, and afterwards was seeing the same disk spin up issues that Warwolf was despite the likes of Docker being disabled, appdata etc being purely on the cache drive.

I stumbled upon this thread though and was able to see the same issue with calls for a btrfs uuid that wasn't valid would occur when the disks would spin up. Likewise the uuid that was being called showed in the config file for the cache pool, since updating the config file to remove that everything is now back to running as expected with drives spinning down, and staying spun down until actually in use.

@Slystuff I'm very glad that it solved your problem and saved you time. 

On 3/18/2024 at 12:24 AM, warwolf7 said:

SOLVED : CONCLUSION : 

my feb25th hypothesis was correct

The command "btrfs filesystem show 32f96308-aa9a-4c86-b799-d222f4af4a75" was causing all disks to spinup because it couldn't find the UUID requested. That command was issued by /usr/local/emhttp/plugins/dynamix/scripts/monitor that took that information from the "/var/local/emhttp/disks.ini" file that probably gets populated from the files in this directory /boot/config/

The dynamix monitor is called every minute from a cron job in "/etc/cron.d/root"

 

The UUID in the config file in /boot/config/pools/ are left empty, why? That would be my question. In my case, my older pool config had an uuid that became invalid (or always have been I don't know) when the array failed due to bad ram and I reformatted 1 of the 2mirror drive in that pool and removed the other drive. That UUID was left in the config file of the pool. When I removed that uuid, the monitor script stopped waking all disk every minutes.

 

It is my understanding that unraid created that problem somewhere during one of those manipulation because I have never played with any of those files prior to this investigation 🔍

Proposition 1 to unraid

Look at what happens when you create a pool array with 2 devices. remove one device, reformat the other one and look at the UUID in the config file. (I can't recreate that, I don't have unused disk right now) Why is there not UUID in the btrfs pool config file? Why did I have one before but not my new one. Was there a changes in the code from previous versions?

Also look for other cases that can create that type of wrong UUID, there might be more that just mine.

Proposition 2 to unraid

The monitor script also fails to recognized the error and send a notification regarding that error. 

 

if (exec("/sbin/btrfs filesystem show "._var($disk,'uuid')." 2>/dev/null|grep -c 'missing'")>0) {

we can see here that it only looks for 'missing' but when a filesystem is not found, it actually return ERROR:
ERROR: not a valid btrfs filesystem: 6790d0c5-bc2f-499f-a9cb-5b5bf6f03d0z

This should be changed to 'ERROR'

But since this script execute every minute, and the next section creates a warning on an error notification. it could potentially flood an unraid admin with warning email. Which could be too much.

(I wish I could create a pullrequest, but I couldn't find the repo on github, the only one I found was 8years old and the code did not match the one in unraid current release)

However, proposition1 has to be implemented first, otherwise, a lot of users might get wrong warnings.

notes:

@foo_fighter

do you use the unassigned plugin? if so, I suggest you update it. This is in the changelog

>Fix: Zpool operation that kept disks from spinning down.

 

 

Thank you

Thank you everyone that took a look at my thread and for help and guidance along my route. 

 

finally

Big hurray to me, I have very very basic knowledge of any of this. So congrats to me for finding it. It can potentially affect a lot of users. Please unraid acknowledge in this thread that something will be done.

This bug is still live in unraid 7.2.3. Took me FOREVER to find this forum post and fix the issue manually. Cheers to @warwolf7 for getting it sorted. My disks can spin down and snooze now.