I have had this issue a few times where everything on my server stops working and the top menu shows the command `arc_evict` at 100% on the CPU. At the same time of this happening everything else appears to shutdown or at least cease to respond including the dashboard, all my docker containers, VMs, SAMBA, etc. The only thing that appears to work is ssh but sometimes that freezes up too. In the past i've been able to shutdown via ssh access or in the past had to restart physically after hours of unresponsiveness from the server.

 

For context all my drives do use ZFS cache pools together and individual hard drives are individual ZFS pools.

 

I attached the diagnostics from my server after reboot. 

scarif-diagnostics-20240403-2127.zip

Edited April 4, 20242 yr by cenzar

Based on top Plex with the one using the most CPU, does it improve if you stop the container?

Yeah plex was streaming and recording live TV. I was unable to interact with the dashboard and didn't try turning it off but i'm fairly certain it was caused by the arc_evict process that stopped all containers and other processes.

 

I attached the ouput of the top command from when I used ssh into the system.

That's very different from the one in the diags, but don't remember arc_evict ever being an issue, so not sure what it could be

That would make sense the diagnostics I got after rebooting so I'm realizing now they may not be relevant to solving this arc_evict issue

18 minutes ago, cenzar said:

I got after rebooting so I'm realizing now they may not be relevant to solving this arc_evict issue

Yep, I though they were from that time, you can try enabling the syslog server and post that if it happens again, there may be some zfs related crashing/call traces.

What exactly is arc_evict and what would cause it to run, and what would lead to it running at 100% for hours on end. These days my server isn't crashing but it's spending more time maxed out on running arc_evict then anything else. I'm currently away so currently i'm not in a position that make it easy to copy logs but when I return to the server I can, and is it ok to post the logs here or do I need to check anything to anonymise it.

I was looking at the logs is it possible unraid is spinning down disks and arc_evict then has no where to write to? Because the logs appear to stop outputting right after it lists drives it's spinning down then no longs until I ssh in.

 

My current set up is 2 SSD cache pools, 1 is in a raid setup (i forget which number) but it is 3 drives all ZFS, then the second is raid 0 with ZFS (snapshotting it to HDD ZFS pools), and each drive in my system is ZFS formatted to be the hybrid ZFS system.

Edited April 8, 20242 yr by cenzar

16 hours ago, cenzar said:

What exactly is arc_evict

I would expect it's an ARC related zfs process, a quick search on Google show someone complaining of a similar issue with a different OS when running a smallish ARC and many files, try doubling or quadrupling the ARC max size and retest.

This ended up fixing the issue I quadrupled the size of the arc and it is no longer having this issue. Hopefully once they release future updates allowing it to be dynamic there can be an easier solution.