Best way to set up a "download/temp" drive?

[FrostyWolf]

I'm still wrapping my head around how linux/unraid does things.  In my old set up, I had a extra drive in the rig but not apart of the array for downloads (sab/torrents/etc) to go to.  That way the download/unrar/etc operations were not taking place on the array or the OS drive.  I also had a "temp" folder set up on the drive so that anything I just need to pass around my local network could get tossed into it.

 

In unRaid, the OS drive is a flash drive (well, stored in RAM while running).  Still, the SSD cache I have in the rig is holding all my plex metadata, docker stuff, etc and I rather not download and etc to it if possible.

 

Now, I've been playing around and I think I figured out a solution, but I wanted to ask if what I was doing was "best practice" or if I was missing a better way to handle things.

I was thinking about mounting a drive with "unassigned devices", and pointing sab/torrents to download and handle the files there, before radarr/sonarr/etc pick them up and move them to the array.  I could also create a temp directory on the drive as before.  I was thinking about making a "temp" share that used cache, so that it stored whatever files I threw in it on the SSD, then moved it to the unassigned drive that night via mover.  I'm pretty sure I understand how to set that up on my own.  My question is...should I be handling it differently? I also was going to make a Backup directory on the drive to back up the flash drive and other relevant stuff using CA Appdata Backup / Restore.  

 

I'm just trying to minimize the amount of time the drives in the array need to be used. 

[trurl]

mover only works with user shares, and unassigned devices are not part of user shares.

 

Personally I would just do it all on cache

[FrostyWolf]

2 hours ago, trurl said:

mover only works with user shares, and unassigned devices are not part of user shares.

 

Personally I would just do it all on cache

Besides my worry about stressing the SSD with all the extra read/write/etc, I'm not sure I actually have enough space on it.  Would it be possible to add say a 4TB drive as a second cache for downloads/temp/CA Backup? I push 3-4 TBs of downloads a month, and I don't want it killing off a drive in the array or my main cache when it fails.  Because of what's stored on it (downloads/etc) I don't need it backed up cause I won't really lose anything I need if it dies.

 

EDIT: Honestly,  I'm not sure if I am even asking the right questions here.  I'm still wrapping my head around everything while I transfer all my data over to the array.  I previously used a Drobo, then Windows + Flexraid and then Windows + stablebit drivepool + snapraid.  So having a disk off the array dor downloads and temp junk to throw around the network and stuff like that was pretty trivial.   It's entirely possible I'm asking for something that doesn't even need to be done, so don't be afraid to just tell me I don't know what I'm talking about.  I'm here to learn, so let me just try to explain one more time.

My main array is going to be ~80TB across 13 drives.
My SSD cache is 240GB.
I'll have one drive left in the server (probably 3~4TB, not done testing the drives from the old build yet) not part of the array.

I'm afraid to do my downloading directly to the array.  When I first started I did this, and I ended up killing a drive in the array in a month, and then again in 6 months.  Both times it was the drive with the download folders on it.  Because of that, I moved it to a separate, off array disk.

I'm also afraid to put it on the main cache drive for the same reason.  If that drive dies, there goes my plex metadata, and whatever else I will be storing on it (via Prefer option for cache).  While this isn't as big of a pain in the ass as rebuilding the array, it's still not something I want to worry about having to do.

If it is off array, and it dies, its as simple as just slapping a new drive in.  Any data (unfinished downloads) on it is easily replaceable, cause sonarr/radarr will just redownload it.  

Then, because I have a drive already that is kind of a "throw away" disk, I also keep a folder on it called "Temp" which I throw files and stuff in to easily share on my local network.  I have 8-9 computers in the house and countless cell phones/laptops/other devices, and I allow temp to be open to everyone to easily transfer items between devices without having to set up shares on each device.  "Temp" works the same way for me, if I lose everything in it, no big deal. Same thing with my CA Backup.  Unless the download/temp/backup drive dies at the same time as the flash drive or whatever other thing I am backing up, I'm not losing anything I can just easy replace.

 

Since this is what I'm used to, I'm trying to recreate something similar in unRaid.  If there is a better way to do it then unassigned devices, please just point me in the right direction.

Sorry for the long post, but I'm just trying to explain it as best I can.

Edited March 5, 2018 by FrostyWolf

[trurl]

How big is your cache? You can have a cache pool which allows you to choose from several btrfs raid configurations, including raid0 which will add their size together. There is no way to treat cache as separate disks.

 

You might also reconsider how you are currently using cache. I only use it for cache-only shares, such as dockers or temporary files. There is no requirement to cache any user share writes, and most of mine go directly to the array. They are usually written to by automated processes such as downloads and backup, so it is unattended and I am not really waiting for them to complete.

 

Each user share has cache settings. Here is a good reference of the effect of each of these settings:

 

https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

[trurl]

1 hour ago, FrostyWolf said:

I'm afraid to do my downloading directly to the array.  When I first started I did this, and I ended up killing a drive in the array in a month, and then again in 6 months.  Both times it was the drive with the download folders on it.  Because of that, I moved it to a separate, off array disk.

I think it likely that you have misdiagnosed the problem. In what way was the disk "killed"? There are a number of ways to have disk problems, and most of them don't mean there is anything actually wrong with the disk itself. If you have a situation in the future please ask for help.

[FrostyWolf]

33 minutes ago, trurl said:

I think it likely that you have misdiagnosed the problem. In what way was the disk "killed"? There are a number of ways to have disk problems, and most of them don't mean there is anything actually wrong with the disk itself. If you have a situation in the future please ask for help.

As in the drive died, and was no longer usable at all.  There wasn't anything to misdiagnose, flex raid and drive pool store the files in a way where it's trivial to see what files are on what disk, and both times the disk died it was the one handling torrents/usenet downloads/unpacks.  It could be a coincidence I guess, but after that I just moved the download drive to its own drive.  I've burned out 2 more since then over the course of 5 years.  Both times the download drive.  As far as the specifics behind the exact failure, all the drives failed their respective diag tools (seatools, etc).  2 I was able to send out for RMA.  I have had 4 drives die over the course of 10 years or so, and all 4 were drives that where handling download/unpacking.  I download about 4TB a month to it, and that 4TB gets unpacked on it, processed (renamed/etc) before being moved to its final location in the array.

 

 

39 minutes ago, trurl said:

How big is your cache? You can have a cache pool which allows you to choose from several btrfs raid configurations, including raid0 which will add their size together. There is no way to treat cache as separate disks.

 

You might also reconsider how you are currently using cache. I only use it for cache-only shares, such as dockers or temporary files. There is no requirement to cache any user share writes, and most of mine go directly to the array. They are usually written to by automated processes such as downloads and backup, so it is unattended and I am not really waiting for them to complete.

 

Each user share has cache settings. Here is a good reference of the effect of each of these settings:

 

https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

I  edited post with that info, but 240gb.  Right now I am only using cache for docker/plex/appdata.  My plex metadata is 70GB and 2 million+ files.  It is a massive pain to backup and restore and I would like to have to do that as little as possible, which is why I don't want to use it for downloads. (also sometimes I'm downloading more than free space is available on the cache drive, so I don't want it to spill over to the array).  I'm not done setting up unraid yet tho (no sonarr/jacket/radarr/plexrequests/torrents/etc/etc) so I'm sure more will end up on it.

As far as putting a Temp share as cache-only/prefered that could work. Sometimes I'm tossing data between computers that is bigger than available space on the cache, but if that spilled over to the array in those situations it would be ok.  The only reason I am talking about backing up and putting the temp on a unassigned drive though is cause I am assuming I will have one for downloads.  So it's safe to ignore the temp/backup part of why I want to use it, the main reason is just I'm afraid to download directly to the array or cache drive for fear of shorting its life span.

If this isn't actually a problem and has just been one major coincidence in my life since I started data hoarding, I can just get a 500/1TB SSD and replace the cache and use it for downloads and the like, which I would honestly prefer because not only is it easier to set up it frees up a slot in the server for another array drive.
 

Edited March 5, 2018 by FrostyWolf

[trurl]

It is certainly possible to use an Unassigned Device for your downloads. You just can't use mover to get files to/from it.

 

I think some of the apps you may be using have ways to move files around, or at least download to one place and unpack to another, for example. So maybe some of these apps could do some of the work depending on how you configure them.

 

There is also User Scripts plugin which will allow you to schedule a script. Maybe something for you in the sample scripts there or the Unassigned Devices sample scripts.

 

And, you might even make a copy of mover and modify it to your purposes. It is just a bash script at /usr/sbin/local/mover.

 

 

[FrostyWolf]

43 minutes ago, trurl said:

It is certainly possible to use an Unassigned Device for your downloads. You just can't use mover to get files to/from it.

 

I think some of the apps you may be using have ways to move files around, or at least download to one place and unpack to another, for example. So maybe some of these apps could do some of the work depending on how you configure them.

 

There is also User Scripts plugin which will allow you to schedule a script. Maybe something for you in the sample scripts there or the Unassigned Devices sample scripts.

 

And, you might even make a copy of mover and modify it to your purposes. It is just a bash script at /usr/sbin/local/mover.

 

 

I could do the unassigned drive for downloads, which will allows sonnar/radarr/etc to do there thing to them on that drive, then move them to their final location on the array.

Then, I could do a Temp share with cache=prefer, which should keep it on the SSD unless its full.  After some more thought and your comments there doesn't seem to be a reason to need to move the temp data to anywhere else, I can just delete it if I need more room cause that's why it is temp in the first place.

Editing mover is a cool idea, I am going to have to build a test rig so I can mess with stuff like that.

 

I'm sure you have extremely more experience in data storage then I do.  While I promise you I am not lying about how unlucky I have been with my download disks in the past, I understand that my person experience is not fact, and it could entire be something else other than "too much downloading is killing them".  I'm just trying to explain things as I had them happen, not contradict anything you are saying.

 

Thank you for all your help so far.  I think, at least for now, Temp on cache, unassigned disk for downloading.

Do you think it matters what format the download disk is? And if so, do you have a recommendation? I'm really only familiar with FAT, FAT32, xFAT and NTFS myself.

Edited March 5, 2018 by FrostyWolf

[trurl]

7 minutes ago, FrostyWolf said:

I could do a Temp share with cache=prefer, which should keep it on the SSD till once a night when mover runs.

Cache-prefer doesn't work the way you seem to think. Be sure you check out the link I gave earlier.

[FrostyWolf]

Cache prefer keeps the data in the cache unless the cache is full, then writes it to the array.  When mover runs, it moves anything it on the array back to the cache, provided there is space, no?  I would like my Temp directory to be on the SSD for speed, when possible. 

 

EDIT: I just read my own quote, I'm doing a lot of things at once and I miss typed there, I'll go back and edit it. 

Edited March 5, 2018 by FrostyWolf

[trurl]

Yeah I just realized that cache-prefer might be exactly what you intend. Many expect that setting to result in files getting moved from cache to the array for some reason. All of those settings aren't easily understood from their one-word specifiers.

[FrostyWolf]

2 minutes ago, trurl said:

Yeah I just realized that cache-prefer might be exactly what you intend. Many expect that setting to result in files getting moved from cache to the array for some reason. All of those settings aren't easily understood from their one-word specifiers.

 

Hey no worries, I'm still glad you said something.  I probably reread the help description for cache options 10 times before it clicked for me when I first started going through the options.  I still have 6 more drives to empty to the array before I start messing more with plugins/shares/dockers and all that, including the download drive.  Thank you for your help, now I have some stuff to think about in the mean time.

[casperse]

This is exactly what I have been struggling with "Temp & Share on cache? vs UA-drive? vs Array?"

I have tried many different things, if you need to seed and store data then cache is quickly not big enough...

 

I came to this conclusion:

1) Seeding for a long time needing lots of storage! = UAD --> set movie/tv shows folder on array cache: yes

2) Temp download to cache? (torrent to move) completed download to --> UAD? --> Movie folder (cache: Yes) move to --> Array (much moving files around?)

3) All on cache drive --> Array? (Not enough space and needed time for seeding!)

4) All on array - so many drives spinning up 24/7

 

Cache 2TB M.2 usage: Appdata (Plex is huge!)/VM/System - and cache setting to (Yes on important shares)

 

I went with option 1 - Eager to hear what others are doing? 

 

[Yivey_unraid]

On 2/20/2020 at 3:45 PM, casperse said:

This is exactly what I have been struggling with "Temp & Share on cache? vs UA-drive? vs Array?"

I have tried many different things, if you need to seed and store data then cache is quickly not big enough...

 

I came to this conclusion:

1) Seeding for a long time needing lots of storage! = UAD --> set movie/tv shows folder on array cache: yes

2) Temp download to cache? (torrent to move) completed download to --> UAD? --> Movie folder (cache: Yes) move to --> Array (much moving files around?)

3) All on cache drive --> Array? (Not enough space and needed time for seeding!)

4) All on array - so many drives spinning up 24/7

 

Cache 2TB M.2 usage: Appdata (Plex is huge!)/VM/System - and cache setting to (Yes on important shares)

 

I went with option 1 - Eager to hear what others are doing? 

 

I'm looking into this as well.

 

Since I don't have any free slots in the server I'm thinking of setting up an external USB drive via UAD. To minimize array spin-up. 

Then point my qBittorrent containers completed torrents folder to that USB drive and handle all seeding on that. The incomplete folder could still be on the SSD cache but I'm not sure if there is any benefit? It would maybe create a bottleneck since it's transfering over USB2.0 (yes, very old hardware).

 

I could keep a backup of the .torrent files on the array in case the USB drive bites the dust, then I could just download what I was sharing again with a new drive.

 

Sonarr/Radarr handles the moving of the completed downloads to the array. As long as the seeding requirements aren't met by the torrents they keep seeding from the USB drive. When the seeding time is up, Sonarr/Radarr removes the torrent and all associated files from the USB drive.

 

Should work, but is there something I'm missing?

 

I'm still setting upp my docker containers and migrating all downloading from my desktop so I can't test yet.