Jump to content

[SOLVED] 6.5.2 Very Slow Mover (bad splits and slow archive drive)


DZMM

Recommended Posts

I've been having problems over the last couple of weeks with a very slow mover. 

 

My download connection is 200Mbps i.e. 25MB/s, yet files are being added too fast for my mover to shift them or just about the same speed so my cache drive never empties.  I've even tried to reduce the strain on my cache drive by downloading torrents direct to the array, but even still the mover can't shift files fast enough.

 

For instance I started a manual move 11mins ago with 342GB used and it's now running at 344GB used.  The problem this is causing is my cache drive keeps filling up, stopping new writes, rather than the mover script working.

 

In terms of other file activity, there's a bit but not a lot going on at the moment:

 

  1. transferring files from disk 2 to 3 - going at around 5MB/s i.e. very slow
  2. nzbget downloading to cache at around 7-8MB/s
  3. deluge checking 136 torrents on disk 5 and downloading to disk 5 at 6MB/s (no seeds) with about 270 connections open
  4. rclone doing an upload from disk 6 at around 5MB/s (upload then remove)

 

Any ideas what's wrong?  Thanks in advance for any help

 

PS cache now risen to 356GB used in the time taken to write this post................

highlander-diagnostics-20180721-1831.zip

Link to comment

The parity drives can only do one thing at a time - so multiple writes to the array will result in a very large slowdown of the total write capacity.

 

Not sure if you have turned on turbo write which means unRAID doesn't need to do read/modify/write whenever it performs writes to the array.

Link to comment
27 minutes ago, pwm said:

The parity drives can only do one thing at a time - so multiple writes to the array will result in a very large slowdown of the total write capacity.

 

Not sure if you have turned on turbo write which means unRAID doesn't need to do read/modify/write whenever it performs writes to the array.

I forgot to say turbo write is on as well.

 

I've had more concurrent activity than what's going on right now before and things have never slowed down like this before.

 

I've worked out what the problem is.  For a test, I paused nzbget, cancelled the upload, reduced deluge to one 1 download etc so I could see the mover write activity better and I can see that the mover is transferring files to disk6, my seagate archive drive.  This drive has been a pain in the ass in my system - originally I bought it as a parity drive, but the write performance killed my whole server so I replaced it and moved it to the array.

 

I managed to 'hide' the problem by only writing certain shares to that drive, but a couple of weeks ago I changed my share/split-level strategy to support better hardlining and fewer disk-2-disk transfers, and I included that drive in the majority of my shares.  Now, because it's got the most free space the mover is moving files to that drive - hence the poor performance.

 

To fix, I'm going to manually remove it from shares and then in the background fill it up so that I'm never waiting in the foreground for new files to be added to it.

 

I wish I'd never bought that drive!

Link to comment

Ok, not out of the woods yet.  I've removed disk6 from most of the shares but the mover is still moving some files to that share at around 22MB/s i.e. pitiful.  However, when the mover has been transferring to other disks I'm not getting faster speeds.

 

I've tried running 'mover stop' but it hasn't worked, so I'm waiting for the kids to go to bed so I can reboot the server and stop all array activity to better pinpoint if it just disk 6 playing up

Link to comment

Solved for now. 

 

My problem was a combination of split levels and the poor performance of my seagate archive drive.  I stopped all array activity so I could just experiment with the mover/cache drive performance:

 

- mover to single HDD (not archive) - 150-180MB/s

- mover to just archive drive - 90-135MB/s (peaked at 167)

- mover to 2+ HDDs (including archive) - =35-55MB/s

 

My split levels were causing the mover to have to move files to more than one drive at times.  I've removed non-essential splits so that my mover can run as fast as possible by only transferring to one drive at a time e.g my movie folders now only have the movie file, so splits aren't important. 

 

I've also removed the archive drive from shares as it doesn't like multiple read/writes and I'm just going to fill it manually.

 

Link to comment

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...