Jump to content

A proposal for smart read/write caching


Recommended Posts

I've used Synology, TrueNAS (core and scale), and Unraid and overall Unraid is my favorite.

 

But I continue to be surprised by the lack of sophistication in the cache system. I'm constantly choosing between cache: yes, where the caches sit mostly empty unless I just wrote a lot to the share, or cache: prefer, where the mover will happily try to shove everything into the cache, bringing the NVMEs or SSDs to almost completely full and taking data off the parity protected drives. Now, I can fix some of the above with tuning and with the mover plugin. But it would be fairly easy to have this all work a lot better.

 

So my proposal would be to implement cache: smart.

 

In this mode Unraid will monitor file opens/reads/writes and populate a small database. This can either be done by actually watching for open/read/write system calls, or can periodically poll processes for open file descriptors. Then the mover can periodically update a priority queue and create a "plan" for what files it prefers to have on the cache, how much room it wants to preserve for incoming writes, etc. Once a plan is created, the mover can move data to the proper place.

 

In my ideal world, data on the cache is also replicated to the array, since it worries me to have cached data not parity protected. Maybe this is hard to do in a consistent way, but a COW filesystem should make this quite a bit easier.

  • Like 1
  • Upvote 3
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...