Access to anything in /mnt/user is very slow

[plarsson]

I recently starting seeing some issues in my server that access to anything in the /mnt/user directory is slow, using command line things like just pressing tab to finish the command, or ls in a folder can take up to 30 seconds. First time I saw it was earlier this week I did a reboot and it seemed to have cleaned up, now 2 days later I had the same issue, this time a reboot did not seem to really fix it.

I do see issues in other places than just the terminal, most notably in Machinaris, which slows to a crawl when this happens (the reason I started to look into it)

Not sure where to start troubleshooting this. I'm attaching the diagnostic log

using the terminal in other folders does not seem slower than normal

 

Update:

Just after typing in the text above I went back to the terminal and things was fast again, behaved just as it should, then a few seconds later navigation in the terminal got slow again, followed by fast.... It seems that for the moment it's intermittent, before the reboot (at least before the first reboot) it was all the time I spent an hour trying to resolve it before rebooting that time.

 

Further updates:

I was monitoring CPU utilization when the slowdown happens, and I can see that some CPU cores are always maxed out the times it happens. Once the CPU utilization goes down it's good again. The utilization is not extreme about 20% of total CPU is used (saw 25% once).

It's a 2x 12c/24t Xeon @1.8Ghz, so in general not too concerned to see 3-4cores maxed out

share-diagnostics-20230304-0622.zip

Edited March 4, 2023 by plarsson

[plarsson]

As a test I downgraded to the previous version (6.10.3) and the system seems much snappier - I still see spikes in cpu utilization and actually higher cpu utilization than before, but anything I do seems fast even during those spikes, everything else was left the same

 

[trurl]

Your appdatassd and system shares both have files on the array. Also domains is completely on the array but VM Manager is disabled.

[plarsson]

Not sure that I understand, the impact of this?

However,

appdatassd is set to "only: pooltwo" , looking at the array for appdatassd files, I did see that two now unused docker containers are for some reason on the array. These containers have been deleted, probably 3-4months ago

System is a bit strange; it's set to "Prefer: pooltwo" however one file are staying at the array ("docker-xfs.img")

Domains is set to use the array, should that be set to prefer, to move things back the the cache?

I don't use any VM's, used a VM a while ago for a bit, but removed it and turned of VM support

[plarsson]

I think I have located the problem.

In this case the problem was Storj - it was trying to write data to the disk faster than the disk could handle (Was writing to the array)

After moving Storj to cash pool the problem is resolved