File Activity Plugin - How can I figure out what keeps spinning up my disks?

[dlandon]

3 hours ago, tuxbass said:

Note data share doesn't show any modifications as of late. The activity above is expected to be under the backup share alone.

Any idea as to why above activity is listed also for /mnt/user/data share?

No, but looking into it.

[tuxbass]

Looking at source, isn't the problematic bit caused by
 

$share_dev = basename($share)."/";
$files = shell_exec("cat /var/log/file.activity.log 2>/dev/null | grep ".escapeshellarg($share_dev)."

i.e. we take basename of a share (resulting in "data" in my case) and then grep it through activity log; as the backup share node has "/data/" in event file path, those events get erroneously included?

 

On an unrelated note - in this particular case there's no real need to escapeshellarg - can use grep's --fixed-strings option instead. Also, that is a great example of useless use of cat, why not directly grep the file?

Edited April 1 by tuxbass

[dlandon]

4 hours ago, tuxbass said:

i.e. we take basename of a share (resulting in "data" in my case) and then grep it through activity log; as the backup share node has "/data/" in event file path, those events get erroneously included?

I have a fix for this.  I just haven't released it yet.

 

4 hours ago, tuxbass said:

On an unrelated note - in this particular case there's no real need to escapeshellarg - can use grep's --fixed-strings option instead. Also, that is a great example of useless use of cat, why not directly grep the file?

Different styles.  If you think you can do a better job, you are free to fork this project and do it yourself.

[dlandon]

Update released.

[dlandon]

6 hours ago, tuxbass said:

On an unrelated note - in this particular case there's no real need to escapeshellarg - can use grep's --fixed-strings option instead. Also, that is a great example of useless use of cat, why not directly grep the file?

Just as a clarificaton.  When AI is asked about which approach is better, the response is:

"It depends on the size of the file /var/log/file.activity.log and the frequency of the script execution.

 

If the file is relatively small and the script is not executed frequently, directly grepping the file would likely be more efficient and result in lower disk I/O. This is because reading the file directly with grep avoids the overhead of piping the output of cat into grep.

 

However, if the file is large or the script is executed frequently, it may be more efficient to use a combination of cat and grep as you have done, especially if you're only interested in specific lines matching a pattern. This can help avoid repeatedly scanning the entire file with grep, potentially reducing overall disk I/O."

 

The line of code in question is executed for every disk device and the log can get large (it is limited to 20,000 lines though), so based on the AI response, it appears the " useless use of cat" may not be useless after all.

 

Maybe before criticizing the code, you need to understand it better?

[tuxbass]

17 minutes ago, dlandon said:

Just as a clarificaton.  When AI is asked about which approach is better, the response is:

"It depends on the size of the file /var/log/file.activity.log and the frequency of the script execution.

 

If the file is relatively small and the script is not executed frequently, directly grepping the file would likely be more efficient and result in lower disk I/O. This is because reading the file directly with grep avoids the overhead of piping the output of cat into grep.

 

However, if the file is large or the script is executed frequently, it may be more efficient to use a combination of cat and grep as you have done, especially if you're only interested in specific lines matching a pattern. This can help avoid repeatedly scanning the entire file with grep, potentially reducing overall disk I/O."

 

The line of code in question is executed for every disk device and the log can get large (it is limited to 20,000 lines though), so based on the AI response, it appears the " useless use of cat" may not be useless afterall.

 

Maybe before criticizing the code, you need to understand it better.

 

Truly thought I might be helpful here tracking down the possible cause of the problem affecting me, and just mentioned the UUOC and a different grep option - I personally don't know everything and oftentimes learn from others in similar manner. Didn't realize I'm doing something awful here.

 

In my professional developer career I haven't reached the place where I ask "AI" (whatever that means) how to solve problems, I'll try to improve, sorry.

 

Also wasn't aware this thread is essentially /r/Conservative where people get hurt over literally nothing, am sorry again.

 

And it's really cool to slide into people's DMs and vaguely threaten them with reporting their horrible posts like mine. Just wanted to reiterate it's completely OK and I invite everyone to report such content.

[dlandon]

1 minute ago, tuxbass said:

"AI" (whatever that means)

Artificial Intelligence.  If you aren't using it, do.  It will do wonders for your coding.  You can get a free ChatGPT account.

[Bovin]

After upgrade from 6.11.X to 6.12.X, the disk spin up more frequently. I enable monitor, but I did not find any record during disk spin up timestamp. 

 

Any suggestion?

[dopeytree]

Do you have:

• ZFS master plugin installed? swap to lazy load.
• Other monitoring containers like netdata etc? try unticking priveled mode when you edit the container settings.
• Bazarr doing subtitle searches on all media?
• Disk Location plugin? check the smart health check settings.

Edited April 3 by dopeytree

[furian]

On 4/3/2024 at 11:50 AM, dopeytree said:

Do you have:

• ZFS master plugin installed? swap to lazy load.
• Other monitoring containers like netdata etc? try unticking priveled mode when you edit the container settings.
• Bazarr doing subtitle searches on all media?
• Disk Location plugin? check the smart health check settings.

 

i have the same issue as @Bovin.

no zfs, no netdata, bazarr is disabled since a couple of days *bazarr was visible in file activity monitor, no disk location plugin.

 

the logs show:
Jul 9 15:58:13 Plex emhttpd: read SMART /dev/sdb

Jul 9 15:58:22 Plex root: Total Spundown: 1

Jul 9 16:03:22 Plex root: Total Spundown: 1

Jul 9 16:08:22 Plex root: Total Spundown: 1

Jul 9 16:13:22 Plex root: Total Spundown: 1

Jul 9 16:18:22 Plex root: Total Spundown: 1

Jul 9 16:23:22 Plex root: Total Spundown: 1

Jul 9 16:27:46 Plex emhttpd: spinning down /dev/sdc

Jul 9 16:28:22 Plex root: Total Spundown: 2

Jul 9 16:29:37 Plex emhttpd: read SMART /dev/sdc

Jul 9 16:30:04 Plex emhttpd: read SMART /dev/sdg

Jul 9 16:30:14 Plex emhttpd: read SMART /dev/sdf

 

when the disks finally spin down they spin up quite fast again.

looking at the file activity monitor i can see stuff happening on the cache and nvme drives (home assistant and vm's). but the disks show no activity.. yet they still spun up..

 

keeping record now:
Jul 10 14:12:02 Plex emhttpd: spinning down /dev/sdb
Jul 10 14:12:04 Plex emhttpd: spinning down /dev/sdg

Jul 10 14:28:43 Plex emhttpd: spinning down /dev/sde

Jul 10 14:37:49 Plex emhttpd: spinning down /dev/sdc

Jul 10 16:24:49 Plex kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 10 16:24:59 Plex kernel: ata4.00: configured for UDMA/133
Jul 10 16:24:59 Plex emhttpd: read SMART /dev/sde
yet nothing in the share or disk activity

Edited July 10 by furian

[dopeytree]

The path forward is one has to go through each docker 1 by 1 and see which is causing it. 

Stop all containers.

Start 1 by 1.

Then keep adding more till you find the problem one.

Tedius but not the end of the world.

Edited July 11 by dopeytree

[JonathanM]

1 hour ago, dopeytree said:

The path forward is one has to go through each docker 1 by 1 and see which is causing it. 

Or, if the cause is truly just one container, start roughly half of the containers, check, if the problem is found, start half of the those, if the problem isn't found, start half of the remainder.

 

Depending on how many items you have to check, elimination by halves can be much quicker than 1 by 1.

[coldjim]

Hi, until recently file activity worked great, however no everytime I start up the plugin it goes into a 'stopped' state straight away. I have searched the thread and found a suggestion to turn off unassigned devices and cache & pool disks, which I have, but still no luck.

Any advice on this?

Thanks

[dlandon]

3 hours ago, coldjim said:

Hi, until recently file activity worked great, however no everytime I start up the plugin it goes into a 'stopped' state straight away. I have searched the thread and found a suggestion to turn off unassigned devices and cache & pool disks, which I have, but still no luck.

Any advice on this?

Thanks

Post diagnostics.

[coldjim]

1 hour ago, dlandon said:

Post diagnostics.

Here they are:

tower-diagnostics-20240813-1428.zip

[dlandon]

1 hour ago, coldjim said:

Here they are:

tower-diagnostics-20240813-1428.zip 171.31 kB · 0 downloads

You are getting an inotify watches error:

Aug  3 20:44:07 Tower inotifywait[12274]: Failed to watch /mnt/disk3; upper limit on inotify watches reached!
Aug  3 20:44:07 Tower inotifywait[12274]: Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.
Aug  3 20:44:08 Tower file.activity: File Activity inotify exiting

 

Install the Tips and Tweaks plugn and increase the number of inotify watches allowed.  The T&T plugin will show you what you currently have.  Increase the number of watches as needed to get file activity to run.

[coldjim]

13 hours ago, dlandon said:

You are getting an inotify watches error:

Aug  3 20:44:07 Tower inotifywait[12274]: Failed to watch /mnt/disk3; upper limit on inotify watches reached!
Aug  3 20:44:07 Tower inotifywait[12274]: Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.
Aug  3 20:44:08 Tower file.activity: File Activity inotify exiting

 

Install the Tips and Tweaks plugn and increase the number of inotify watches allowed.  The T&T plugin will show you what you currently have.  Increase the number of watches as needed to get file activity to run.

 

Thank you! I have increased the number from 524288, to 720000 but i'm still stopping. Is there a limit to the inotify number? what are the downsides to keep increasing this?

[dlandon]

6 hours ago, coldjim said:

 

Thank you! I have increased the number from 524288, to 720000 but i'm still stopping. Is there a limit to the inotify number? what are the downsides to keep increasing this?

There's really no limit, but memory is consumed so eventually you might have memory issues.  Inotify doesn't use much memory though and new servers have a lot of memory.  Just keep increasing it until File Activity finally works for you.

[coldjim]

21 hours ago, dlandon said:

There's really no limit, but memory is consumed so eventually you might have memory issues.  Inotify doesn't use much memory though and new servers have a lot of memory.  Just keep increasing it until File Activity finally works for you.

Thank you for your help! I finally got it working again, had to increase the the number to 4000000. Its running great now!

Edited August 15 by coldjim

[mrpops2ko]

 

plugin: installing: file.activity.plg
Executing hook script: pre_plugin_checks
plugin: downloading: file.activity.plg ... done

Executing hook script: pre_plugin_checks
plugin: run failed: '/bin/bash' returned 2
Executing hook script: post_plugin_checks

 

noticing this error, wlll back search now to see if others have it too

[fr500]

hello, thanks for the plugin, it's great.

one question, I'm trying to optimize energy usage, so, is there any way to make this plugin only work when the browser tab is open? I have a cpu spike every few seconds when the plugin is enabled, the process is lsof