Why won't inotifywait detect anything?

[TexasUnraid]

So I am trying to figure out which dockers are causing writes and in the process it was suggested to use inotifywait to log the modified files.

 

The problem is that no matter what I do, inotifywait never logs a single thing.

 

I have the most basic command now

 

inotifywait /mnt/user/Temp/Temp/Test.txt 
Setting up watches.
Watches established.

 

Doesn't matter what I do to that file, nothing is output from inotifywait.

 

Any ideas?

[Squid]

Does it ever exit back to the command prompt?  Works 100% for me.

[TexasUnraid]

36 minutes ago, Squid said:

Does it ever exit back to the command prompt?  Works 100% for me.

No, it just sits there waiting forever and never detects anything. I can't figure it out, the server was just restarted this morning as well.

 

Really strange problem indeed. The file activity plugin was working fine last time I used it a few weeks ago, it uses the inotifywait I am pretty sure?

[dlandon]

1 hour ago, TexasUnraid said:

The file activity plugin was working fine last time I used it a few weeks ago, it uses the inotifywait I am pretty sure?

Yes it does.  Why not use that plugin to find what you are looking for?

[TexasUnraid]

23 minutes ago, dlandon said:

Yes it does.  Why not use that plugin to find what you are looking for?

 

I would love to but it didn't scan the system share last I checked?

 

At least I never remember seeing any activity from the docker files before (I have around 10 million files, so it takes a LONG time for file activity to start, so not something I can just try out real quick).

Edited June 20, 2021 by TexasUnraid

[dlandon]

1 hour ago, TexasUnraid said:

 

I would love to but it didn't scan the system share last I checked?

 

At least I never remember seeing any activity from the docker files before (I have around 10 million files, so it takes a LONG time for file activity to start, so not something I can just try out real quick).

The file activity plugin does not detect file activity on the syslogs or appdata shares.  All others are monitored.

[TexasUnraid]

Ok,

7 minutes ago, dlandon said:

The file activity plugin does not detect file activity on the syslogs or appdata shares.  All others are monitored.

 

Cool, I never remember seeing the docker folder included but I also didn't look for it since I was using an image before.

 

I just started the plugin so in an hour or 2 I should start to see results.

[TexasUnraid]

Ok, I started the file activity plugin but strangely it is not working either. I know it worked a few weeks ago and the only changes I remember making since then was:

 

A: I switched from a docker image to a folder along with changing the drive they were stored on, doubt this has any effect

 

B: I had to uninstall atop from the nerdpack as it was filling my log all of the sudden. I have no idea what atop is and particularly have no idea why the log would fill the partition in a few days.

 

What would cause atop to log? Other then updating plugins I don't remember adding any new ones recently and no other config changes come to mind since it was last working.

[Squid]

8 minutes ago, TexasUnraid said:

What would cause atop to log?

It's a known thing with atop.  Generally speaking, if you don't know what something is and how to use it, don't install it via NerdPack.  Many people simply install everything offered up by NerdPack when it's unlikely they need anything at all (or at best, only need Perl once to setup System Temp plugin)

[TexasUnraid]

Then I am at a loss on what could be causing the inotifywait to not work.

 

I have been meaning to switch my USB drive for awhile, maybe this is a sign I should reinstall and start fresh. What config files would be safe to transfer to the new usb if I do this considering these issues I am having?

[TexasUnraid]

Ok, did some troubleshooting.

 

I rebooted into safe mode and inotifywait worked in root.

 

I guess you can't start the array in safe mode? I was not given the option for that for some reason. So I rebooted again and tried inotifywait in root again, it once again worked.

 

So I started the array and tried it again, also worked.

 

I then tried it on a share, worked.

 

I then think I might of found part of the problem, I tried inotifywait using the user share and I edited the file using a direct path. This did NOT work.

 

Editing the file via the user share worked. Including in the docker folder.

 

Is it expected that changes done via user / direct path will not be detected?

 

Although docker is supposed to use the user share, so not sure why that would have an issue.

 

I restarted the file activity plugin, in a few hours I should hopefully start seeing some activity.

[Squid]

Makes sense, since even though the file is identical, inotifywait doesn't actually look at changes made to files only requests to open / close / write on the particular path / file you specify

[TexasUnraid]

1 hour ago, Squid said:

Makes sense, since even though the file is identical, inotifywait doesn't actually look at changes made to files only requests to open / close / write on the particular path / file you specify

Interesting, good to know.

 

So file activity seems to be started now. I am getting some logs from my array disks but nothing from the system / docker drives or folders.

 

Is this an issue with it not working with custom pools (system is not on "cache" but a custom "write-cache" I am using for testing right now), or it not scanning the system share?

 

It would be cool if file activity could have an option added to select which shares / folders / drives to monitor. For example I have ~6 million files in a folder that I do not need to monitor and it would save a lot of time to not have it scanned. Just an idea.

 

Just tried running a manual command to monitor the docker folder, it is setting up the watches now:

 

inotifywait -e create,modify,attrib,moved_from,moved_to --timefmt %c --format '%T %_e %w %f' -mr /mnt/user/system/docker/docker > /mnt/user/system/recent_modified_files_$(date +"%Y%m%d_%H%M%S").txt

 

[TexasUnraid]

Ok, so that last command finished and the watches are established.

 

I manually tested it by touching some files in the docker folder and it logged these correctly.

 

Yet I stopped and then started all the dockers yet not a single thing was logged?

 

EDIT: After some research it seems the docker folder has a symlink to /var/lib/docker. If I touch the files from that path, nothing is logged.

 

I am running the same command now using that path instead. Hopefully that will work.

Edited June 22, 2021 by TexasUnraid

[TexasUnraid]

Looks like we have a winner, finally getting the log populated from the docker files!

 

Now to figure out how to correlate the really long container names in the docker folder with the dockers that are using them.