Large Reads & Writes to Array and Cache - 6.12.3

[bhman79]

As the title states I'm seeing a lot going on with the array and cache drive.  I've shut down all dockers and VM's and it doesn't solve the issue. These numbers are for the last 24 hours.  I didn't get the cache in the screenshot, but it has 2.5M reads and writes.

 

Any ideas?

 

 

 

 

 

i

 

 

[JorgeB]

That suggests something is writing to disk2, start by posting the diagnostics.

[JorgeB]

None of the usual suspect shares are on disk2 and there were no writes when the diags were saved, assuming no transfer to the array manually or the mover, is it still going up? 

[bhman79]

Yes, disk 2 writes are up to 80M.  I have more free space on disk 2 now.  It's almost like it's moving files around from disk to disk.

 

I'm running a parity check to see if it finds anything.

[JorgeB]

20 minutes ago, bhman79 said:

It's almost like it's moving files around from disk to disk.

That would cause writes to another disk, are you sure it's not the mover? Enable mover logging and check after the scheduled time.

[bhman79]

I have the mover scheduled to run once at night.  It shows that it's completed every morning, so I don't think that it's constantly writing to the array.  I'll enable the logging to verify.

[JorgeB]

28 minutes ago, bhman79 said:

so I don't think that it's constantly writing to the array

When you posted the diags nothing was writing to the array, so it's not constant.

[bhman79]

Parity check finished with no errors.  I enabled mover logging.  Ran mover.  It moved over security camera files and completed.  I reset the stats, so this is as of 2 hours ago.

 

 

[JorgeB]

Post new diags.

[JorgeB]

Also install the open files plugin and see if that shows anything.

[bhman79]

Looking at open files.  Is it normal to have 15 nginx processes running?

[bhman79]

hurstserver-diagnostics-20230802-1228.zip

[JorgeB]

Again there's nothing writing to disk2 at the moment the diags were saved, are the stats always increasing or it's every few minutes or so?

[bhman79]

The cache is writing constantly.  Disk 2 seems to be all at once.

[bhman79]

Also, my IPCam files are on a separate cache.

[JorgeB]

I cannot find anything in the diags, if open files doesn't help boot in same mode and stop docker and VMs services, not just the VMs/containers, failing that redo the flash drive.

[bhman79]

It seems that something is constantly writing to the cache drive, then mover is moving it to Disk 2.  Looking at what data is on the disk, doesn't show anything modified today, so I can't figure out what it's moving.

[JorgeB]

On 8/1/2023 at 5:09 PM, JorgeB said:

Enable mover logging and check after the scheduled time.

 

[bhman79]

I checked the mover log for activity and didn't see anything unusual.  I downloaded the File Activity plugin.  It looks like some media files are showing an "Attrib" line.  I'm also showing some .tmp files that are created, opened, modified, and then deleted.  Not sure if that's normal.