Constant write IO on Cache


Rune

Recommended Posts

Ran with a single XFS-encrypted cache disk for a while, then moved to a 3-disk BTRFS-encrypted cache pool.

 

Since that change, I've noticed a constant write IO on the cache disks of about 2-3MB/s (per the Main page).  IOStat on the unraid host confirms the IO is happening.  No other tool can give me any idea on what is doing the writing (eg: iotop points at shfs).

 

I have 3 VMs with disks on the cache pool (that were there prior to the change).  Monitoring on the VMs, iostat says there is maybe 10kB/s average write IO on each.  I NFS-mount in a bunch of stuff, but monitoring nfsiostat tells me that there's not much going on there, and mostly read traffic.

 

Since Btrfs is copy-on-write, I checked, and made sure that the VM disks, libvirt, docker, and other random-write stuff is COW-disabled (chattr +C, and verify with lsattr).

 

With no IO coming from SMB clients, an only minor IO coming from VMs and NFS mounts, and COW disabled...  how can I tell where all the extra IO is coming from?  (I can account for, say, 100kB/s, but I'm seeing 3MB/s (1.5MB/s when you account for btrfs writing redundant blocks).

Link to comment

Did some more testing on this, and there is definitely some write amplification going on here.

 

I transferred everything off of the cache, and marked it as 'No' for all shares, and that dropped the write IO to zero on cache.  At that point, there was reasonably constant 50-200kB/s write IO on the array (including parity).  That IO only appeared on the disk that I moved the VM disks to.  I moved everything except the VMs back, one at a time, and the write IO stayed on the array.  Shut down the VMs, the write IO stopped, moved the disks to cache pool, set the 'No COW' flag (chattr +C) on the disks, and when I started up one, the IO was already above the 50-200kB/s that I saw as an average on the array.  On cache, it's about 900kB/s-3.8MB/s

 

...which is about a 18-19x write amplification for being on the cache pool instead of the array.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.