Write Cache filling up too fast


Recommended Posts

Hi All,

 

So i took the plunge and purchased a license and now i'm copying over my legally acquired collection of movies and music.

 

Transfer speeds were brutally slow around the 250ish Mbps mark so I decided to set the share to write cache the data.  Perfect, write speeds pop up to 1.1 Gbps woo hoo!  

 

Not so fast, despite having the mover tool set up to flush the cache hourly, my SSDs can't keep up with the network.

 

Every hour or so my cache fills up and Rsync fails and kicks back a message similar to the following:  rsync error: error in file IO (code 11) at receiver.c (378) [receiver=3/2/3] rsync: [sender] write error: Broken pipe (32) 

 

My question; aside from purchasing a larger SSD (current SSD is 480 GB) is there a way to throttle the connection so the mover has time to push the cash to the HDD or have rsync simply wait its turn to continue writing?

 

..... or am I talking EMC Netapp type enterprise features?

 

Thanks in advance. 

Edited by zetabax
Link to comment
On 4/14/2021 at 12:10 AM, ChatNoir said:

It is advised not to use cache for the initial data transfer since you cannot empty the cache as fast as you can fill it.

Fair point.  Unfortunately though, i'm only dealing with 12x 10 TB 7.2K SAS drives so white performance is bound to suck.  I've quote quite a bit of data and based on my calculations, I'd be waiting a long ass time waiting for the transfer to complete.  So I went ahead and put 2x 1TB WD Blue SSDs and set them as a write cache.  The cache still fills up about only about once a day which I can live with.  RSync will fail and I wait for the mover to flush the cache, then restart RSync.  Seems to be working. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.