Write speed to array slows to a crawl over time


Recommended Posts

Hi I just setup unRaid a couple of weeks ago and I'm still in the process of getting all my disks added to the array and copying over data from my previous NAS.  I've been running into an issue with write speed for disks in the array.  After a reboot things seem normal, I see between 50-120MB/s write speeds, as time goes on and especially as I've been doing file transfers from my old NAS to unRaid, the write speeds decline eventually ending up in the KB/s range.  I've had this happen as I mentioned during file copies (using Krusader for that, and copying direct to the array /mnt/user0 to bypass cache), but I've also had this behavior while running a preclear (using the plugin).  I remember when the system was first setup and I precleared my very first disk that the zero phase took about 16hours on an 8TB disk, write speed around 120MB/s.  The last disk I precleared the zero phase took about 40hours and write speed started out in the 120MB/s area but declined down into the KB/s area over time.  

 

If I reboot everything goes back to normal and starts the decline in speed all over again.

 

I'm running unRaid 6.8.3.  Diagnostics file is attached

 

Thanks in advance for help.

diagnostics-20201222-0851.zip

Edited by justme123
Link to comment

@JorgeB  thanks for the suggestion.  So I had a file transfer that has been running for a couple of hours, the speed had slowed to 350KB/s, when I turned off the VM the speed momentarily went back to 50-60MB/s for a few seconds and then went back down to about 500KB/s.

 

After that I went into the main VM manager and disabled VMs all together just to see if that made any diff, but no change.  However now when I try to re-enable VMs I get an error that the libvert service failed to start. 

Seeing this in the syslog - 

emhttpd: shcmd (996): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1

root: /mnt/user/system/libvirt/libvirt.img is in-use, cannot mount

 

The VM that was running is my main system running blue iris so I do need to have that running.  I have an unassigned device that is being passed thru to it as it is always recording.

Edited by justme123
Link to comment
3 hours ago, justme123 said:

the zero phase took about 16hours on an 8TB disk, write speed around 120MB/s.  The last disk I precleared the zero phase took about 40hours and write speed started out in the 120MB/s area but declined down into the KB/s area over time.  

Pls check does any slow/abnormal disk in array, this problem quite common. You may need docker diskspeeed apps to identify it.

Link to comment

Ok I will perform some speed tests.  I can say though that I've seen the same issues on all of the disks where speed is good at first and then it declines over time.  Would SMR drives show this type of behavior?  Right now I've only been copying video files that are 10+GB in size.

 

thanks :)

Edited by justme123
Link to comment

Did some more testing, this time I rebooted the system and left the vm manager completely disabled.  Over time ended up with the same result file transfers slowed to under 300KB/s.  

 

This is a screen shot from grafana.  I was copying a bunch of large movies and it was fast for a long time then suddenly dumped down to 5MB/s and trailed off to the speed it is now while I'm posting this under 300KB/s.

image.thumb.png.c5e4cb63d77a6db9cb44fd556f9926c6.png

 

Not sure what to try next, I guess I could uninstall plugins, or maybe upgrading to 6.9.x?  

Link to comment

Tried writing to disk1 and still ran into problems.

 

I hope I don't jinx myself but I've since upgraded to 6.9-RC2 and so far haven't been able to reproduce the issue.  I did make one other change and that was disabling folder caching, but I had done that pre upgrade and it hadn't made a change in behavior.

 

Hopefully the issue is resolved with the upgrade!

 

Thanks for the help :)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.