File Activity Plugin - How can I figure out what keeps spinning up my disks?

[DataCollector]

52 minutes ago, dlandon said:

Post diagnostics.  Also look at the output of this command:

cat /tmp/file.activity/file.activity.disks

and see if the share in queston is in that output.

root@UR-TESSA:~# cat /tmp/file.activity/file.activity.disks
/mnt/disk1
/mnt/disk2
/mnt/disk3
/mnt/disk4
/mnt/disk5
/mnt/disk6
/mnt/disk7
/mnt/disk8
/mnt/disk9
/mnt/disk10
/mnt/disk11
/mnt/disk12
/mnt/disk13
/mnt/disk14
/mnt/disk15
/mnt/disk16
/mnt/disk17
/mnt/disk18
/mnt/disk19
/mnt/disk20
/mnt/disk21
/mnt/disk22
/mnt/disk23
/mnt/disk24
/mnt/cache
/mnt/cachenvmeb
/mnt/poolhdda
/mnt/poolssda
root@UR-TESSA:~#

 

 

Yes, poolhdda is listed.

 

ur-tessa-diagnostics-20230402-0043.zip

[dlandon]

4 hours ago, DataCollector said:

Yes, poolhdda is listed.

Install the Tips and Tweaks plugin and check the inotify watches.  Be sure the 'Available User Watches' is a positive value.  If not, then increase the 'Max User Watches' until it is a positive value.

[JudasD]

This plugin solved a new issue for me.  Out of nowhere drives were not spinning down and there had been no updates to unraid.  Using this plug-in, I found a VM was causing the keep-alive.  An app in the VM did an auto-update (even with auto-update turned off in the app).  I would have never been able to diagnose this without the plug-in!  Greatly appreciated! 

 

JD

 

Edited April 13, 2023 by JudasD

[dopeytree]

I've got some small writes happening to the array but not showing up in this plugin.

[dlandon]

Posting diagnostics can help with diagnosing your issue.

[dopeytree]

moulin-rouge-diagnostics-20230703-1256.zip

[thany]

It's not working. I *know* that I'm accessing a files from the main array, I *know* which share I'm accessing it on, and I can *see* with my own two eyeballs my access is producing disk I/O activity. All three monitor are enabled. Even though I might be seeing hundreds of MB/s being written and/or read, the File Activity plugin happily shows no activity whatsoever.

 

Then why do I need File Activity plugin? Because sometimes I also see some kind of ghost file activity of about 5MB/s being written that for whatever reason blocks up most other I/O. And that too doesn't show up, even though that's actually the important bit that I need to know about.

 

So how do I make it actually show activity that is definitely occurring?

[dlandon]

1 hour ago, thany said:

It's not working. I *know* that I'm accessing a files from the main array, I *know* which share I'm accessing it on, and I can *see* with my own two eyeballs my access is producing disk I/O activity. All three monitor are enabled. Even though I might be seeing hundreds of MB/s being written and/or read, the File Activity plugin happily shows no activity whatsoever.

 

Then why do I need File Activity plugin? Because sometimes I also see some kind of ghost file activity of about 5MB/s being written that for whatever reason blocks up most other I/O. And that too doesn't show up, even though that's actually the important bit that I need to know about.

 

So how do I make it actually show activity that is definitely occurring?

I can offer better help if you post your diagnostics.

 

Here is some information on how File Activity works.  These are the file events that ae set up with inotify to watch:

open,attrib,move,create,delete,modify

 

File reads are not logged because the log would be horrendously large.  Generally the cache_dirs plugin will keep drectory accesses from spinning up disks.

 

There are some automatically ignored shares because watching those shares would also generate a large amount of logging:

appdata, syslogs, and system

File activity in these shares will be ignored.

 

If you have the Tips and Tweaks plugin installed, you can check the intotify watches available.  Not having enough will definitely cause File Activity to not work.

 

[DevanteWeary]

Quick question...

 

IS there any kind of log for this besides the one on the settings page? 

Like something I can go back and search the last several days?

[alturismo]

3 hours ago, DevanteWeary said:

IS there any kind of log for this besides the one on the settings page? 

Like something I can go back and search the last several days?

may look 1 post upper yours ... ?

 

 

so no  there is nothing more ... this should be also only used when you have issues ... to track it down, solve it, and then turn it off.

[DevanteWeary]

6 minutes ago, alturismo said:

may look 1 post upper yours ... ?

 

 

so no  there is nothing more ... this should be also only used when you have issues ... to track it down, solve it, and then turn it off.

I'm such an idiot I swear I read! Well more like skimmed ha.... big failed on my part. 

 

I wouldn't mind the option of having an external log running and maybe a setting to prune it however you want. Such as size or life of log.

 

Well anyway just glad such a good plugin was made!

 

Thank you! 

[ConnerVT]

Wouldn't writing this log be file activity?  If you didn't have an issue with drives not spinning down, you certainly could have it here.  🤨

[DevanteWeary]

6 hours ago, ConnerVT said:

Wouldn't writing this log be file activity?  If you didn't have an issue with drives not spinning down, you certainly could have it here.  🤨

 

Options are always nice. 

 

For instance, I looked into this plugin because I wanted to know what was deleting the movies out of my completed torrents folder sporadically.

 

Since I can't predict WHEN the movies will get deleted, by the time I wake up and notice the movie is gone, this plugin's momentary log is filled with other stuff.

 

If I'm being honest, while I am eternally grateful that people put the hard work in to make these things, I don't see how having a plugin whose purpose is to track down file activity... not log file activity.

 

That's just me! Not negativity intended. I love this community and am grateful for this plugin. 

Edited September 3, 2023 by DevanteWeary

[KnifeFed]

19 hours ago, ConnerVT said:

Wouldn't writing this log be file activity?  If you didn't have an issue with drives not spinning down, you certainly could have it here.  🤨

Not if the log is written to RAM like with most Unraid plugins.

[dopeytree]

For me it was a plugin called Dashdot

 

Quote

Culprit seems to be Dashdot container that is spinning up the main array and stopping it staying spun down.

If you edit the container and turn 'privileged' to off it should stop the constant array spinning up.

 

[dlandon]

On 9/2/2023 at 7:34 PM, DevanteWeary said:

Quick question...

 

IS there any kind of log for this besides the one on the settings page? 

 

The file activity log is written to ram in '/var/log/file.activity.log' and stays until the server is rebooted.  The events logged are file events that can potentially cause a disk spinup.  That's what this plugin was intended to do - help understanding why disks are spinning up.  It was not intended to be run continuously.  The log file is limited to 20,000 events so it does not fill ram.

 

File activity is logged with inotify events and does not log every file event such as file access so important events do not roll off the log.

 

Quote

Like something I can go back and search the last several days?

You are looking for this plugin to be something it is not.  I don't believe there is anything in a plugin that would do what you are looking for.

[Autchirion]

Can this plugin be extended to read activities as well (and probably limit it to only some disks of the array)? I've got some reads that prevent disks from spinning down, but I have no Idea where they are comming from.

[kfb]

Hey there, I installed the plugin but it seems like it won't start. When I click in the webgui, it always says "Stopped" and the system logs for the Unraid server shows this:

 

Sep 13 13:10:34 Crushinator  ool www[17699]: /usr/local/emhttp/plugins/file.activity/scripts/rc.file.activity 'start'
Sep 13 13:10:34 Crushinator file.activity: Starting File Activity
Sep 13 13:10:34 Crushinator file.activity: File Activity inotify starting
Sep 13 13:10:34 Crushinator file.activity: File Activity inotify exiting

 

I tried pulling apart the script a little and found the command. I tried running that manually and got this:

 

root@Crushinator:~# /usr/local/emhttp/plugins/file.activity/scripts/inotify.file.activity

Warning: array_diff(): Expected parameter 1 to be an array, null given in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 71

Warning: array_merge(): Expected parameter 2 to be an array, null given in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 78

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 81
No files specified to watch!

 

I'm not that great at PHP, so I wasn't able to get much further than that. It looks to me like it's not finding the disks that it wants to watch, but that's just a guess.

 

Any help on fixing or debugging this would be appreciated.

[dlandon]

3 minutes ago, kfb said:

I'm not that great at PHP, so I wasn't able to get much further than that. It looks to me like it's not finding the disks that it wants to watch, but that's just a guess.

 

Any help on fixing or debugging this would be appreciated.

This is generally an issue with not enough iotify watches.  It should log that.  Pleasae post diagnostics.

[kfb]

Ah, sorry, meant to post that:

 

Current Inotify Settings

Max User Watches = 524288

Current User Watches = 342

Available User Watches = 523946

 

Here's the diagnostics:

crushinator-diagnostics-20230913-1306.zip

[dlandon]

2 hours ago, kfb said:

Ah, sorry, meant to post that:

 

 

Current Inotify Settings

Max User Watches = 524288

Current User Watches = 342

Available User Watches = 523946

 

 

Here's the diagnostics:

crushinator-diagnostics-20230913-1306.zip 136.15 kB · 0 downloads

 

Go to a command line and run this script:

/usr/local/emhttp/plugins/file.activity/scripts/inotify.file.activity

Post any output from the command.

[kfb]

Here you go

 

root@Crushinator:~# /usr/local/emhttp/plugins/file.activity/scripts/inotify.file.activity

Warning: array_diff(): Expected parameter 1 to be an array, null given in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 71

Warning: array_merge(): Expected parameter 2 to be an array, null given in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 78

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 81
No files specified to watch!

 

Looking at configure_file_activity, I don't really know enough PHP to debug what's going on there.

[dlandon]

36 minutes ago, kfb said:

Here you go

 

 

root@Crushinator:~# /usr/local/emhttp/plugins/file.activity/scripts/inotify.file.activity

Warning: array_diff(): Expected parameter 1 to be an array, null given in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 71

Warning: array_merge(): Expected parameter 2 to be an array, null given in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 78

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/file.activity/scripts/configure_file_activity on line 81
No files specified to watch!

 

 

Looking at configure_file_activity, I don't really know enough PHP to debug what's going on there.

I see the problem.  I'll have a fix for this today.

[kfb]

Got the fix and was able to figure out what was stopping that disk from spinning down. Thanks for the help!

[Holle88]

hello,

i have trouble with installing inotify, because it says my Unraid Version is too high (6.12.4).

i have random disk writes all 30 seconds (VM and Docker Service NOT running, as UrbackUp uses the array for backups, the appdata is on cache so no docker is storing anything on array except for backups), and i wanna know which plugin keeps the disk alive instead of spin down...

 

since i can not install inotify i wonder how i can use this plugin...

 

can someone help me out? 

how can i set up inotify in /boot/extra to install it by restart by unraid itself?

 

 

iotop (installed with nerdpack) says only shfs is writing on it (it's jsut a second present and then goes away another 30seconds

 

Edited October 5, 2023 by Holle88