High CPU usage when writing to ZFS cache/Pausing during long Writes

[Marc_G2]

I set up a new separate cache for one of my shares and went with ZFS.  It's a single drive cache.  Is relatively high CPU usage normal when writing to a ZFS cache?  Apparently, my CPU can't keep up to the point where my large file transfers regularly halt for 5-10 seconds every 30 seconds or so.

My other cache is setup as 2 drive BTRFS.  I haven't had this issue with it.  

 

 

Edited August 10, 2023 by Marc_G2

[Marc_G2]

Diagnostics attached in case that helps any

nas-ng-diagnostics-20230808-1921.zip

[JorgeB]

Check top during a write, dash graph includes i/o wait.

[Marc_G2]

5 hours ago, JorgeB said:

Check top during a write, dash graph includes i/o wait.

 

I'm not sure graph you're referring to.   

 

I did some more tests and found the issue only happens after around 25GB of continuous transfer.  Here's a graph of the network activity (1 large file transfer).

During those periods of zero activity, there's 100% cpu usage.  So there appears to be some computations happening that's causing the delays

 

Edited August 9, 2023 by Marc_G2

[JorgeB]

Type top or htop in the CLI

[Marc_G2]

Is this what you're looking for? Interestingly when the system pauses, this graphs shows no cpu activity unlike the one on the dash

Edited August 9, 2023 by Marc_G2

[JorgeB]

Like mentioned the dash includes i/o wait, it's just waiting for the disks, so not an issue.

[Marc_G2]

That doesn't explain why there's frequent pauses during large sequential writes.  This write performance isn't normal for any SSD.  Is it due to ZFS overhead? 

[JorgeB]

22 minutes ago, Marc_G2 said:

That doesn't explain why there's frequent pauses during large sequential writes.

The device cannot keep up, you'd need a faster SSD, note that not all SSDs can keep constant high write speed, most infact can't, especially entry level models, an MX500 for example should perform better.

 

 

[Marc_G2]

I'm writing to the SSD at only 100 MB/s.  Under normal desktop usage, the BX500 write speed will drop to 40 MB/s when it runs out of cache (which is 48GB).   It's not normal for it to completely halt.  

I have other SSDs so I'll try a different one.  I'll also see if there's any difference between btrfs and zfs.

[TCMapes]

Could the ZFS Pools and Array becausing my high CPU spikes and the hard reboots?

 

I notice every time my unraid system spikes my system reboots.  Usually when I am downloading a game on my VM.  In this case Baldurs Gate 3.

 

Is there something I should do?  I feel like my system is rebooting to protect itself just don't know why.

 

[TCMapes]

2 minutes ago, TCMapes said:

Could the ZFS Pools and Array becausing my high CPU spikes and the hard reboots?

 

I notice every time my unraid system spikes my system reboots.  Usually when I am downloading a game on my VM.  In this case Baldurs Gate 3.

 

Is there something I should do?  I feel like my system is rebooting to protect itself just don't know why.

 

 

Forgot to add I am not overclocking anything system speeds are default.  I have all DIMMS populated with a total RAM give 128GB.  

[JonathanM]

40 minutes ago, TCMapes said:

system spikes my system reboots.

Usually that's a symptom of a power supply issue.

[Marc_G2]

52 minutes ago, TCMapes said:

Is there something I should do?  I feel like my system is rebooting to protect itself just don't know why.

Yeah reboots like that are usually either due to either power or overheating issues.  If your psu is spec'd plenty high, then I would double check all your connections.

[Marc_G2]

I changed the file system of my BX500 SSD to btrfs instead of ZFS.  I then tested the performance by transferring a 50GB file.  There were fluctuations in the transfer speed but it never completely halted like it did when was formatted as ZFS.  The total transfer time was probably about cut in half.  So I'm going to assume the issue has been mitigated.

 

My advice to anyone else to only use drives with good sustained write performance if using ZFS.  Though I haven't confirmed yet whether that would also fix the issue on my system.