Considering a Migration to UnRaid - Questions and Existing User Thoughts


JesterEE

Recommended Posts

Hello UnRaid Community!

 

1st Post!  I'm excited!

 

As my storage needs evolve and I find myself more and more on /r/DataHoarder/ I'm strongly considering upping my OS game.  I've read, and watched a lot about UnRaid but I still have a few questions as it may or may not pertain to my use case.  I'd also like anyone's input that is willing to share personal experiences with the OS, particularly those that use the OS like I intend to.  Thanks for reading and sharing in advance!

 

Currently, my build (pcpartpicker) is reasonable for most low-mid level tasks considering it's core is based on 8 year old hardware.  File storage, web server, media server, database hosting ... the basics.  I am using a duplication based storage strategy based on StableBit software.  I have been operating this way since a previous migration for Windows Home Server v1 and haven't looked back as everything is simple (and beautiful) to manage and works well in the background.  Heck, I really love the StableBit software ... it has saved my bacon a few times!

 

Though, my needs are evolving.  I run more services for media/PVR management than I ever thought I would.  I started running a 2nd low power machine to handle many tasks for caching internet data just so I didn't have to task my main server.  I find myself adding at least 1 HDD/year to my storage pool just to keep up with my endless hoarding of "stuff".  And ... I'm running out of physical space, making file duplication untenable as a long term solution.

 

So, it's time to think differently!  Ideally, I want to bring everything all under one roof.  Have a machine that operates as my "Windows daily driver" for office level/programming tasks and light gaming for when I want to relax, as well as my file storage and daemon host in one box without server grade hardware (or at least as little as possible).  Up until now, I think this has been all but impossible, but with recent CPUs pushing more and more cores, virtualization and containerization becoming more mature, and hardware support being more tightly integrated, I think we are entering the Goldilocks zone of enthusiast computing.

 

So, I went searching and found that UnRaid seems to check all the boxes!  While this is exciting, and I'm eager to jump ... I want to be cautious.  Sure, UnRaid will let me try it for 30 days before I buy, but my time is more valuable than the cost of the license.  I don't want to migrate all my data just to find it's not for me ... and I don't have a box of spare parts to play with to find out.  So, I'm here on the forums asking advise :).  I was also considered sticking with Windows and using SnapRaid with StableBit, but this is more kludge than I'd like for all my important data.  Having to stop writing to the pool to get a valid parity snapshot seems hard to do in practice as a server with lots of daemon tasks may (or may not) be accessing data at any given time.  I found a script that uses BTRFS snapshots in conjunction with SnapRaid to solve this ... but, obviously, not Windows compatible!  And if I'm going to migrate to a Linux or a Linux-like system, I might as well not kludge my solution.

 

I was inspired by the Linus Tech Tips: Use your Gaming PC's Extra Power as a NAS Ultimate Guide video.  This is pretty much what I have in mind, but I'm not sure my current system is up for the task.  To meet my requirements I envision needing to run 6-10 docker containers at all times (probably 1-2 cores worth of processing power) with UnRaid as the storage backbone, and boot a Windows 10 VM on an as needed basis when I feel like using desktop instead of laptop.  So here are my questions/concerns:

 

  1. If I want to keep all my dockers up while operating a Windows 10 VM with gaming as the intent (high CPU/GPU and moderate RAM utilization) is a Sandy Bridge i7-2600 and 16GB DDR3 going to cut it?
    • I could/would shut down my media server docker processes, but would rather UnRaid deal with provisioning and prioritizing the scheduling (in this case, VM over plex docker)
      • I've read some on the forum about CPU pinning for both VMs and dockers ... but with only 4 cores, I think I may under-power the whole setup doing so. 
    • I eventually want to upgrade to a 6/8 core CPU and DDR4 (when hardware prices permit), but until I do, I don't want to lose functionality in my setup.
  2. I need my GTX 1060 GPU to work with both an always running Plex docker and an on-demand Windows 10 VM.  I know Nvidia GPU passthrough is a thing and works with UnRaid which is fantastic!  Probably the main reason I'm truly considering making this move.  I read the LinuxServer.io forum post and watched SpaceInvader One's Youtube video on the topic.  Does anyone do this switching/sharing effectively?  It seems that there are a lot of opportunities for locking the server while passing the GPU around.
  3. One thing I like using StableBit software is that running periodic surface scans (StableBit Scanner) directly keeps my data storage pool (StableBit Drivepool) safe as they communicate.  I know this is a debated topic, and some say this is not necessary, though I disagree.  Not too long ago, I had a drive that passed a week long pre-clear, operated for 3 months, then started dying with bad sectors popping up in one drive location.  StableBit Scanner found it, told StableBit Drivepool, immediately migrated all the data that wasn't corrupted off the drive and made sure there was adequate duplication present on the pool, and told me what was unreadable (corrupt) for manual counter-measures.  No man-in-the-middle, no muss, no fuss ... easy!  Does UnRaid do anything like this?  I know I can check S.M.A.R.T. with smartctl, and there is bitrot protection with the Dynamix file Integrity plugin (integrated in v6?).  Is this the same thing?  Are there other health monitoring tools in UnRaid I overlooked?  Is action, or rather, reaction to these outputs, automated by UnRaid?
  4. Upon disk failure, if there is enough "free space" on other arrayed drives to reconstruct the lost data from parity, will UnRaid reconstruct the data and place it on the array using the "free space" and rebuild parity?  For example, if there is a 1x 1TB parity drive, and 3x 50% full 1TB data drives in the array (each drive has 500GB used and 500GB free), if 1 of the 1TB data drives fail, will UnRaid use the 1TB of "free space" left on the array to reconstruct the 500GB of "lost" data from parity?  From what I've read, I don't think it will, but I wanted to ask. 
    • If not, why would I not want to fill all the drives in the array before adding an additional drive?  Best case all the drives continue to work and I add a drive to get more storage.  Worst case, a drive fails and I have another to pop in and reconstruct the array.  Should I always have a spare (hot or not) on reserve with UnRaid?

 

I know this is a lot ... but thanks again for reading and commenting!

 

-JesterEE

Link to comment

4. Say you have the following.

 

1 TB Parity

1 TB Data

1 TB Data

 

and loose one of your Data drives unRAID simulates the data on the fly so you can use it. I've even written to a down drive and unRAID did its magic behind the curtain. When you replace the down drive it rebuilds the down drive and upon completion you have all your data back physically.  (NOTE) You can only loose one drive at a time unless you are running Dual Parity.

 

When I first started using unRAID I did a test. I threw some data on some drives and reached in and unplugged a drive while the system was running. The video I was watching on the newly unplugged drive Glitched and then I hit play again and it continued on. 😀 Shortly afterwards I rebuilt the drive and watched the same video while it was rebuilding. After that I bought my license and 9years later here I am.

 

You could have a Spare ready to go. Actually I'd kinda advise you to do so, but I've had a failure in the past and simply drove to the store and picked up another. Just keep in mind whatever drives you use the Parity has to be the largest or equal to your largest Data drive.

 

Link to comment

I would suggest you look into a dual CPU motherboard but consider your budget, a board will run you around $200 on the low end and they go up from there, your CPU choices are many and you could start off with an 8 core XEON and add another one down the road. Now Xeon's are not great for gaming, but they can provide enough horse power for your needs, and with an 8 core CPU you would have a total of 16 threads, add a second one and you are up to 16 cores and 32 threads. unRAID doesn't do some of the fancy stuff that stablebit does in terms of migrating data around etc, unRAID is simple, you have a failed or failing drive, the data is simulated until you replace the drive, done. I run four unRAID servers, two have 30 drives in them and the other two are smaller. Once of my large servers have dual 8 core Xeons and runs Plex along with a bunch of other dockers. I seldom have to do anything to it, it just does it's thing and hums along. You can certainly build an all in one unRAID server and run a Windows vm and have the server be a media/storage/file server, you just needs to decide what kind of case you want, if you want hot swap drives, how many you want etc, but it's totally doable.

Link to comment

@kizer: Somewhere along the line I read that ... and then instantly forgot it.  Thanks for reminding me and for the great story of it actually working as intended!  The parity drive being the largest in the array I understand, and I'm sure over time people likely cycle their parity drives into data drives as they upgrade.

 

@ashman70: The homelab setup ... nooooooo ... I'm not there yet ... and if I am, I'm resisting :D.  My main issue with going this route is more physical than anything else.  Good chassis's (rack mounts) are fracking big ... and fracking noisy ... and I don't have a good place to put it where it can be both and not be a nuisance!  If I avoid going the rack mount route, the dual processor Xeon boards are all E-ATX, so that means a large full sized tower.  This I can deal with, but with a nice Phanteks case to support what I already have and provide a reasonable amount of future-proofing and keeping it cool with some quiet Noctua fans it's going to cost $200+.  It seems like a silly patch just to not have to get a solid used 4U with great power supplies that's probably better at the job.  Then add in mountains of ECC memory to support the CPUs, and some adequate CPU coolers for both processors and I'm looking at a ~$800-$1000 build even when speccing out reasonable ~2011 hardware similar to my i7-2600 but with more processing power (OT: Xeon E5-2650 v2s are a great value right now at ~$65!).  With that budget I could get a lower end consumer grade i9 or threadripper setup in an ATX form factor.  I'm not convinced server grade hardware is needed for my intended workload, but also, I'm not looking to spend $1000 here ... I was thinking more on the order of an i7-8700 and 32GB DDR4 which I can probably find for ~$500 with a good combo deal or part out.  Another time, another home ... probably, but not right now because I know I can't do it "right".  Unless I'm thinking about it wrong ... I've never built an enterprise grade computer.  Am I missing something?  Are my preconceived notions of enterprise hardware incorrect?  Set me straight! :D

 

I appreciate your insights though ... you're making me feel better about moving forward in this direction.

 

-JesterEE

Edited by JesterEE
grammar fail
Link to comment

Well you want to a lot of things, so you have to scale accordingly. If you go with an i7-8700 that is only six cores. You want to run a VM for work and gaming, so lets say that should at least be four cores, that leaves you with two to run unRAID and whatever else you want to do. You could probably get away with running your VM with only two cores, that leaves you with four, my point is, you are limited here, no room for growth. You want to run Plex, you will need CPU cores for transcoding, is it only for internal use or to server external users too? I understand your physical concerns, rack mount chassis are loud for sure, you can get by with a quiet tower more easily but you have to be smart about planning for future growth, so maybe consider a tower that you can use 5 in 1 hot swap bays with. If you went AMD a Ryzen 7 2700 has 8 cores and 16 threads, that would give you a lot more room to do what you want and grow, but it's not a cheap CPU. How much storage space do you want to start with and then have to grow into and then have room to grow to?

Link to comment

My personally. I have my machine setup to do well what it does. Runs some Basic Dockers, Plex and various others and I'll be adding Zoneminder so I can run a cheap security system.

 

I would of gone the route of having my machine do everything. But I thought about it this way. Its the same reason I didn't ever by one of those TVS long time ago that had a Built in DVD player / VCR. I figured if one of them stopped working then the TV would only function half way.

 

Of course its totally up to you to have an all in one box, but the level of complexity grows and grows and honestly my Octacore gaming rig does just fine sitting here in the office while my kids beat up the server which sits in another room. I've upgraded both now and then and neither affects the other. 🤣

Link to comment

Both @kizer and @ashman70 have great valid points, for me to grow from my amd A8 chip what i did was buy one thing at time and sell to recoup the funds, took me a while but i an happy now with what i got. its old server grade equipment but its bullet proof. On the flip side since it older hardware replacement parts are cheaper on the ebay's

 

 

Link to comment

Thanks for the recommendations everyone.  I'll have to think about it some more I guess.  95% of the time my existing hardware should do everything I need it to do and more ... it's that pesky 5% :(.

 

Right now this system is mostly just for me.  A I share my Plex with a few friends, but after monitoring the usage, sparse external use is an understatement.  That's kinda why it's hard to pull the trigger on server hardware.  I just don't need it today if I am smart about how I manage the system resources.  For example, hibernating the work VM and closing the Plex docker when I want to open a game VM.  At that point the only real motivation to jump to UnRaid at all is the parity storage solution ... which is important!  Though, I can see how this might change in the next year as I continue throwing more and more HDDs at the problem.  At that point it might be the best move to get server grade stuff.  I may start budgeting for one and keeping an eye out for deals.  Maybe I'll get lucky...

 

As for multiple machines, I really want to get away from that!  Granted, I'm in a Windows ecosystem right now, and that doesn't bode well for low resource servers ... it was just way easier to set up that way.  Microsoft stripped out all the great little nice things it had for small business servers in Server 2012 R2 with the release of Server 2019, which coincidentally, also made that OS great for home use too. So, they have really made the choice for me to move on from the Server builds very easy.  Moving to UnRaid on my storage server was really going to force my hand at migrating away from using Windows in this capacity.  I figured I might as well move it all in house at the same time, but really I can UnRaid the storage server and keep my old Atom based net top 2nd machine doing all sorts of useful stuff.  I mean, 7W power draw at full load is pretty amazing, it's just not meant to do all the stuff Windows wants to do in the background as well :D.  What is out of the question is 2 beefy computers ... one for storage/media serving and one for "other".  Those tasks need to be carried by same machine because I'm not buying 2 powerful machines to just sit dormant most of the time.  This is the reason my question #2 above is so important to me.  I'm not going to UnRaid my main machine if I can't provision it to switch between demanding tasks well.  And I know the platform is not meant to do that ... but if it can't, it's not really the droid I'm looking for.

 

If that's the case, I can just keep the heavy lifting on Windows (10 Pro) where it all gets provisioned well on bare metal and deal with better storage when I have absolutely no choice ... and I'm not quite there yet.  Maybe the right solution at that point is truly go 2 machines and build a puny low power UnRaid server that does nothing but manage the storage array and potentially a few dockers.  I'm thinking minimum TDP CPU ... maybe re-purposed laptop internals or home NAS hardware ... but that just seems like a waste of potential.

 

So with all that said, question #2 and the health monitoring/action parts of question #3 are still quite important if anyone has insights!  Outside of bare metal hardware decisions, these are going to make or break it for me.

 

Thanks again!  If nothing else, I am really enjoying the community here!

-JesterEE

Link to comment

As you seem to be a Windows data-hoarder, I recommend a few things first;

Get rid of the hoard and doubles;

http://www.joerg-rosenthal.com/en/antitwin/

Use a good file-manager like TotalCommander to move your lower importance directories in a \deep\deeper\deepest\ dir and then run anti-twin on your pooled driveletter, have it tag based on deeper dir depth. Trust me, this is an amazingly well written piece of freeware, it's gonna save you terabytes of BS doubles and moot backups you still had lying around.

 

Then, while you're cleaning up, handy tools I often use as well:

http://www.jonasjohn.de/red.htm

https://portableapps.com/apps/utilities/revo_uninstaller_portable

https://portableapps.com/apps/utilities/wise-registry-cleaner-portable

https://sdi-tool.org/download/

 

and, since you're running Windows 10, I don't know, consider these:

https://github.com/Sycnex/Windows10Debloater

https://github.com/madbomb122/Win10Script

https://github.com/madbomb122/BlackViperScript

 

Realize you didn't ask for any of these, but still, what you wrote made me think of those. Some take a level of tech that you seem to fall under, so that's OK. And I stand behind all of the above mentioned.

 

I'm at the point of buying the 12 drive Unraid license myself. And my main PC also runs Covecube's DrivePool, so there..

Link to comment

@fluisterben Thanks!  Much appreciated!  Most of these I have never seen ... I'll have to take a closer look!  So, your looking to migrate too?  Thoughts on your own migration decision?

 


So I looked a bit at dual CPU server grade hardware and/or using server grade components in consumer housing solutions.  Man, really not liking the options here!  It seems you either get a cheapish 2nd hand (ebay) large capacity LOUD rack mount chassis that you have to modify to make work in a home setting (and still have to mount ... somewhere), deal with the loud server and buy an expensive acoustic mounting enclosure, buy some crazy expensive amalgamation case with lots of expansion slots and modification options, or a less expensive case with extras to support additional expansion slots and be left with minimal room for routing/air flow (also still rather expensive).  I see a common thread here of expensive, and I can not justify any of it :|.

 

Going the cheapest possible option which will leave only minimal future proofing with insufficient cooling (IMO) and will still cost ~$150 new with an additional ~$120 to upgrade to capacity ... and getting one used is not too easy from some quick searches.  And that doesn't even take into consideration the extra fans and CPU coolers likely needed to prevent it from suffocating.  At that point, buying the 2nd hand server chassis is the cheapest, but it's, so, LOUD!

 

I see no good economical home solutions here.

 

-JesterEE

Link to comment

The great thing about unRAID is you can spend as much or as little as you want to get started. I had dreams of a Rack in my Office and a bunch of gear in it. I literally picked up a full rack and had it in my office. 3Months later that rack sits in my garage and I can't get rid of it.

 

  • I built a single core Sempron 140 which lasted me for a couple of years costing me $300 which includes case, PSU and Ram - (700CpuBenchMmark)
  • Then I upgraded the CPU to an Athlon Dual core which lasted me a couple of years costing me $35 - (2000CPUBenchMark)
  • Now I'm running a Single Xeon QuadCore which does everything I need, but I had to buy a new motherboard and the processor. I think I paid $60 for the Processor, $180 for the 10 Port Sata Motherboard and $80 for the Ram. - (7000CpuBenchMark)

Sure that's money I'm throwing in the wind, but hey its been 9Years on what I have now and upgrades here and there aren't to bad and when I need a new drive I just pop it in. My Family has never cared about anything seeing everything has always worked and sitting in my Game Room my machine is literally silent serving up 17TB. I could easily get 12 Drives in my Case or 18 with an LSI Sata Card Expander, but it would be a mess. Lol

 

Link to comment
7 hours ago, kizer said:

The great thing about unRAID is you can spend as much or as little as you want to get started.

Since I posted this topic I've done a lot of reading on the forum, and searching for good solutions, and I think this is the most true statement I've seen yet! Even though I haven't gotten all my concerns addressed I think I have enough to go on to know that even with some flaws, UnRaid seems to still be the best solution currently available. Worst case, I start with the hardware I have (which is pretty decent IMO) and build up as I go and have need, and possibly have another machine for the heavier tasks

 

I was concerned about having the NAS and the media server(s) on 2 different PCs on the same LAN, but the internet says it works fine over a wired gigabit connection, so I'm less concerned about it actually functioning if it doesn't work well all on my current hardware.

 

Hopefully in the next few weeks I can find the time to get my data in a place to migrate. I'm a little paranoid about the move because to get 1 parity drive up and start the array, I need to remove almost all the redundancy I currently have on my pool (14TB of 30TB). Not going to feel great about it till every drive is added back in. Also, I need to figure out a write cache RAID 1 of SSDs. I have a 1TB SATA SSD, but I think getting another would be too much for this task. I'm thinking 2x500GB drives would be more appropriate and then I can use the 1TB for a VM drive or something.

 

-JesterEE

Link to comment
5 hours ago, JesterEE said:

I need to remove almost all the redundancy I currently have on my pool (14TB of 30TB).

Redundancy is about uptime availability, not data backup. You need to reconsider your storage strategy if it's currently possible to lose important data during this operation. All important data must be backed up elsewhere, RAID or Unraid is NOT a substitute for backups.

Link to comment
4 hours ago, jonathanm said:

Redundancy is about uptime availability, not data backup. You need to reconsider your storage strategy if it's currently possible to lose important data during this operation.

Haha, you're not wrong 😁. I have backups of the really important, cry for a week if you lost it, stuff. That's not an issue. It's all those darn Linux isos ... They so big 🤣!

 

In all seriousness though, cloud backup is on the list of things to do when I migrate.

 

-JesterEE

Edited by JesterEE
I'm my own grammar Nazi
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.