NZBget and Radarr/Sonarr workflow to ease the pain on array


Fredrick

Recommended Posts

Hi guys,

 

I've got these apps set up in dockers, and things are moving along nicely. The problem is I'm seeing a lot of iowait (15-30%) during periods where I'm downloading, and wondered if there is anything I can do with my settings to ease it up a bit. I've got a Proliant ML350 G6 with 42gb of ram and dual X5650 Xeons and a 500mbit connection

 

I've got a 240gb SSD with about 190gb free for activities after the mover has run. NZBget is downloading and extracting to this drive, and Radarr/Sonarr is then moving the files to the array. I cant have Radarr/Sonarr move to cache as that would easily fill up my cache before the mover can run. Having a much larger cache would obviously solve my problems, but spinners cant keep up with my connection during extraction (i'm pretty sure), and SSDs are too expensive atm.

 

My settings:

59bf716e06e39_ScreenShot2017-09-18at09_09_40.png.67fdd8f46cc58e9970e9ff3b6be32622.png59bf716f5245c_ScreenShot2017-09-18at09_09_14.png.0a0cca4bc4b783733b1b3f0d98b7a62e.png59bf716fe4ec7_ScreenShot2017-09-18at09_09_04.png.2709a8a9c6f1c7ae9d8b3f9200d85897.png

 

So the question is simple. Can I improve this "workflow" to ease the strain on my system, particularily the array/iowait during downloads? Paging @neilt0 who seem to have some knowledge :) about this

 

Thanks a lot :)

Link to comment

Trying to bump this for visibility.

 

One option would perhaps be to switch my cache from my current SSD, and rather pull 2 Ironwolf 4tb from the array and set it up as Raid0 for performance. This way I can also have my Media share set to use cache, and the array would stay calm during most writes. 

 

I could use my current SSD-cache as an unassigned drive, and have my VMs and dockers run from that. Additionally, I could have NZB-get use it as a download-folder, and then extract to cache. 

 

I know this would reduce redundancy on my media-share as files would be unprotected until mover runs, but the way I see it would give me some significant benefits over todays solution. Having a raid0 cache would increase risk of the files on the cache, but as long as this is easily replaced files I'm willing to take that risk. Any other drawbacks of this solution?

Link to comment
  • 1 month later...

NSP? Newsgroup? I'm mainly on Ninja, main server is their DE-server.

 

I havent changed that setting, and I fully saturate my 500mbit connection. Under each provider (I have a few block-providers aswell) I've set how many connection to each provider, are you saying this is limited by the total number from "Urlconnections = 4"? The setting information seem to indicate this amount of connections is just to download the actual NZB-file.

Link to comment
1 hour ago, Fredrick said:

NSP? Newsgroup? I'm mainly on Ninja, main server is their DE-server.

 

I havent changed that setting, and I fully saturate my 500mbit connection. Under each provider (I have a few block-providers aswell) I've set how many connection to each provider, are you saying this is limited by the total number from "Urlconnections = 4"? The setting information seem to indicate this amount of connections is just to download the actual NZB-file.

 

I am on frugaluse and they provide 40 connections. I set my sab to allow for 38 connections. Setting to 40 sometimes gives me too many connection errors. Not sure if nzbget functions different than sab, but you wanna set your total connections available to at or just below your providers given stats.

 

As for your performance, I DL to my cache(1 TB SSD), extract to it, and my mover moves once a week. I do notice when a large file is extracting and im topping my DL speed @250mbps, things slow down a bit. Perhaps would be better if I separated my DL/extract locations, but it doesnt bother me just yet.

Edited by NotYetRated
Link to comment
57 minutes ago, NotYetRated said:

I am on frugaluse and they provide 40 connections. I set my sab to allow for 38 connections. Setting to 40 sometimes gives me too many connection errors. Not sure if nzbget functions different than sab, but you wanna set your total connections available to at or just below your providers given stats.

 

AFAIK this is the "wrong" way to set amount of connections as each connection brings a bit of overhead "cost" in CPU-cycles. You should set your connections to the least amount that still saturates your 250mbps connection. I'm using 25 connections to my main provider, and it fills my 500mbps connection just fine. Having anything more than 25 connections is actually less efficient.

 

1 hour ago, NotYetRated said:

As for your performance, I DL to my cache(1 TB SSD), extract to it, and my mover moves once a week. I do notice when a large file is extracting and im topping my DL speed @250mbps, things slow down a bit. Perhaps would be better if I separated my DL/extract locations, but it doesnt bother me just yet.

I'm getting a 2tb Barracuda tomorrow which will serve as my unassigned-downloads+extraction-disk. Hopefully this will prove a better solution than downloading to my 240gb cache-SSD. (Downloads might not extract as fast, but it will put less strain on my SSD with all docker appdata+VM etc located there. 240gb (minus docker.img, appdata and VMs) is simply not enough when downloading several items consecutively or when grabbing larger 4K-content where the .rars and extracted mkv can total about 150gb alone.

Link to comment

Just to update, I'm very pleased with downloading and extracting to an unassigned drive. The extraction speed is noticeably slower (especially when still downloading), but I was able to have two transcodes of Bluray Remuxes + one Remux direct stream going while downloading and extracting. The WebUI and all streams went flawlessly :)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.