Is unRAID for me?

[METDeath]

So right now I use a Windows 7 Home Premium installation for my file server, is equipped with 14 drives (27.5 TB physical space), and a BD-RW (for removal of media when I'm done with it, but want to watch it again. So here's a bit about my setup, and the questions I have.

 

Build specs:

ASUS M4A89GTD PRO/USB

AMD Athalon II X2 215 (2.7GHz)

4GB RAM

14 HDDs (totalling 27.5 TB physical)

two Highpoint Rocket Raid cards (two RAID5 arrays of 2 TB and 3 TB drives, with hot spares)

an ASUS USB 3.0/SATA addon card

Orico USB 3.0 expansion card (internal 20-pin header ftw!)

 

 

I use it to run the following:

uTorrent 2.2 (3+ is loaded with garbage not related to torrenting, can GTFO)

No-IP.com Dynamic Update Client

iTunes

Ember Media Manager (scrape meta data for XBMC)

MySQL (central XBMC database)

MakeMKV (ripping of my discs)

VPN Server

VNC Server

XBMC (for database management such as clean library)

Putty (terminal for OpenELEC machine)

 

I also do the following on this machine:

light web browsing

burn BD-Rs for "offline" storage of things I don't actively need

direct copy of files to travel storage devices (USB drives, portable playback devices, phones)

 

About my shares:

I currently just use SAMBA shares to allow a Win7 (WMC/XBMC hybrid HTPC), and an OpenELEC Fusion machine, as well as for backups of my gaming PC, with the possibility of a third XBMC machine (Raspberry Pi or another E-350 machine)

 

Reasons to stay on Win 7:

It's all done already, and I'm familiar with Windows environment. (not scared of CLI, used to use C64s and x86 DOS machines, just never ran a full time *nix machine)

 

Reasons to switch to unRAID:

parity protection for ALL drives

stop having hot spares and just run my RAID cards in JBOD (prior to change over move data out of arrays and onto normal NTFS drives)

 

Questions about unRAID:

Can I specify a hot spare drive for unattended rebuilds?

Does it support NTFS?

Is there an FTP server or other way to easily access all my files from the public internet?

How large of a USB stick should I use?

Is there a torrent plugin? (Should be based on Deluge, KTorrent, rTorrent, qBittorrent, Transmission, uTorrent, Vuze for use with a torrent monitoring app, also those are the whitelisted clients)

 

Absolute minimum for my server:

Torrent program

MySQL Server (central XBMC database)

 

[garycase]

Relative to what you have now ...

 

=>  UnRAID will be slower, since it doesn't stripe the data

=>  Potential data loss is a lot less with UnRAID, since at most you lose the data on the failed drives in the event of a dual drive failure -- with RAID-5 you'd lose all data on the array in that case

=>  UnRAID is more "power friendly", as it only spins up the drives you need when accessing data (typically only one drive), and only two drives for writes (the drive you're writing to plus parity)

=>  UnRAID does not support NTFS.  Everything is stored using ReiserFS

=>  There are many plugins available for UnRAID ... I don't use plugins, so I'll let other comment on whether or not the specific ones you want are available with the latest v5.0 release (some have not yet been updated for v5)

=>  UnRAID would indeed provide "parity protection for all drives" ... but be aware that the more drives you have, the higher the probability of dual drive failures.    On the other hand, the damage done with a dual drive failure is much lower with UnRAID than with traditional arrays.    If additional protection is one of your goals, you may want to consider simply switching your arrays to RAID-6 instead of a complete switch in operating environments.

=>  UnRAID is NOT a backup ... nor is any RAID array.  If you don't have backups of your data, THAT is what you should be taking care of before switching environments.  In fact, if you're comfortable with your current setup, you may want to build an UnRAID server simply as a backup for your current data.  [i.e. just use it as a NAS and backup all your data to it]

 

[METDeath]

Relative to what you have now ...

 

=>  UnRAID will be slower, since it doesn't stripe the data

=>  Potential data loss is a lot less with UnRAID, since at most you lose the data on the failed drives in the event of a dual drive failure -- with RAID-5 you'd lose all data on the array in that case

=>  UnRAID is more "power friendly", as it only spins up the drives you need when accessing data (typically only one drive), and only two drives for writes (the drive you're writing to plus parity)

=>  UnRAID does not support NTFS.  Everything is stored using ReiserFS

=>  There are many plugins available for UnRAID ... I don't use plugins, so I'll let other comment on whether or not the specific ones you want are available with the latest v5.0 release (some have not yet been updated for v5)

=>  UnRAID would indeed provide "parity protection for all drives" ... but be aware that the more drives you have, the higher the probability of dual drive failures.    On the other hand, the damage done with a dual drive failure is much lower with UnRAID than with traditional arrays.    If additional protection is one of your goals, you may want to consider simply switching your arrays to RAID-6 instead of a complete switch in operating environments.

=>  UnRAID is NOT a backup ... nor is any RAID array.  If you don't have backups of your data, THAT is what you should be taking care of before switching environments.  In fact, if you're comfortable with your current setup, you may want to build an UnRAID server simply as a backup for your current data.  [i.e. just use it as a NAS and backup all your data to it]

 

Since most of it is simply media (not created by me, something I can get again if need be) data loss is more an annoyance (file renaming, meta data scraping, etc), and the hassle of keeping the data while migrating to UnRAID (NTFS to ReiserFS), sounds like I should just stick to what I have. Granted, I may certainly consider it for a torrent box build in the future (currently a 1TB drive on the current server. The primary reason I use RAID5s is for large volumes, I just have the hot spares because I had issues with the PSU not being quite up to the task (somehow a 750w single rail wasn't cutting it, 1000w fixed it)

[JonathanM]

I also do the following on this machine:

light web browsing

burn BD-Rs for "offline" storage of things I don't actively need

direct copy of files to travel storage devices (USB drives, portable playback devices, phones)

There is no gui available at the unraid local console, so light web browsing would be limited to text only, like lynx or similar.

Unraid does not have optical drivers built in, and I am unaware of any plugins to add that functionality.

If a drive can be mounted RW, you can copy to it directly. NTFS is read only in the stock distribution, so you would need to add a plugin package to enable writing to an NTFS volume.

 

Unraid is not meant to be a machine you actively sit in front of and operate, it's meant to be a server you place in an out of the way corner and access from the rest of your networked client machines.

[doorunrun]

You may have already seen this series on YouTube about unRAID; if not it's a great introduction (although a little dated):

  is the starting episode.

 

Compared to your current work flow, setting up unRAID would be better with a fresh build; or upgrade your Windows hardware and re-purpose your current hardware to unRAID.

 

The attraction of parity redundancy, incremental growth and unified shares are the strong points for unRAID.

 

You can't just physically move your data drives into unRAID but you can migrate data off the existing HD's to unRAID and integrate the now un-needed drives into the unRAID array incrementally. To do it would mean starting unRAID with a few 3 or 4TB drives.

 

Good luck!

 

[vl1969]

Relative to what you have now ...

 

=>  UnRAID will be slower, since it doesn't stripe the data

=>  Potential data loss is a lot less with UnRAID, since at most you lose the data on the failed drives in the event of a dual drive failure -- with RAID-5 you'd lose all data on the array in that case

=>  UnRAID is more "power friendly", as it only spins up the drives you need when accessing data (typically only one drive), and only two drives for writes (the drive you're writing to plus parity)

=>  UnRAID does not support NTFS.  Everything is stored using ReiserFS

=>  There are many plugins available for UnRAID ... I don't use plugins, so I'll let other comment on whether or not the specific ones you want are available with the latest v5.0 release (some have not yet been updated for v5)

=>  UnRAID would indeed provide "parity protection for all drives" ... but be aware that the more drives you have, the higher the probability of dual drive failures.    On the other hand, the damage done with a dual drive failure is much lower with UnRAID than with traditional arrays.    If additional protection is one of your goals, you may want to consider simply switching your arrays to RAID-6 instead of a complete switch in operating environments.

=>  UnRAID is NOT a backup ... nor is any RAID array.  If you don't have backups of your data, THAT is what you should be taking care of before switching environments.  In fact, if you're comfortable with your current setup, you may want to build an UnRAID server simply as a backup for your current data.  [i.e. just use it as a NAS and backup all your data to it]

 

Since most of it is simply media (not created by me, something I can get again if need be) data loss is more an annoyance (file renaming, meta data scraping, etc), and the hassle of keeping the data while migrating to UnRAID (NTFS to ReiserFS), sounds like I should just stick to what I have. Granted, I may certainly consider it for a torrent box build in the future (currently a 1TB drive on the current server. The primary reason I use RAID5s is for large volumes, I just have the hot spares because I had issues with the PSU not being quite up to the task (somehow a 750w single rail wasn't cutting it, 1000w fixed it)

 

In my experience its not watts you need to be worry about , its amps.

On 12v rail that need to be high enough.

 

I now have 760w psu with 72amp on a rail. Shoulb be more than enough for at least 24 consumer graid hdds

 

Sent from my SGH-T889 using Tapatalk 4

 

 

[METDeath]

You may have already seen this series on YouTube about unRAID; if not it's a great introduction (although a little dated):

  is the starting episode.

 

Compared to your current work flow, setting up unRAID would be better with a fresh build; or upgrade your Windows hardware and re-purpose your current hardware to unRAID.

 

The attraction of parity redundancy, incremental growth and unified shares are the strong points for unRAID.

 

You can't just physically move your data drives into unRAID but you can migrate data off the existing HD's to unRAID and integrate the now un-needed drives into the unRAID array incrementally. To do it would mean starting unRAID with a few 3 or 4TB drives.

 

Good luck!

 

Well, I have a gaming PC, I could pull one of the GPUs out and use that to move my RAID data back to the current file server, granted, it'd be a lot of data over ethernet, which even with Gigabit would be a time consuming process. Granted, it also means I could get some cheaper controller cards instead of my two RAID cards that I have.

 

I also realized I can set up a Raspberry Pi as a VPN server since I haven't gotten around to installing a custom firmware on my router. Heck, I think I could even get the Pi to be a torrent client and point it at the unRAID... but prefer to do that much data thrashing "locally" instead of doing it basically twice on the network (once off the server, once out to the web). With a VPN I could then access the shares, so it's certainly feasible... just not sure if I want to do that much leg work... it'd probably take me about three or four days to get it fully converted.

 

Is there a place where I can look at all the plug ins prior to installing unRAID?

[garycase]

Is there a place where I can look at all the plug ins prior to installing unRAID?

 

Browse through the various threads here:

http://lime-technology.com/forum/index.php?board=6.0

 

Not a complete list, but it captures most of them -- and the threads let you see some of the issues they have in certain situations.

 

[chickensoup]

Is there a place where I can look at all the plug ins prior to installing unRAID?

 

Keep in mind that unRAID is an operating system, not an application which runs on Windows or *nix. It has a minimal package set for stability and basically, it 'does what it says on the box.' Once you start getting involved with plugins you do so at your own risk, it can be a little involved to get everything setup though once it is- many people here don't touch their servers again until they need to power them down for some reason, or to add/replace a drive. There are plenty of plugins on here which do work great but often almost all issues people have with unRAID are attributed in one way or another to plugins, rather than the core product.

 

Another solution you might also want to consider is an ESXi/unRAID build. There is a bit more setup work with this type of setup and you would need to ensure your hardware is supported. There are a few threads about virtualizing unRAID with ESXi or even XenServer. This way you can run a barebones unRAID VM alongside a Windows 7/8/Server VM or whatever you feel will work for you.

 

unRAID is a great NAS/Media server solution and overall I wouldn't even consider any of the alternatives out there at the moment. A cache drive is a great solution for improving write speeds if you can handle your data not being protected for a few hours.

[henris]

unRAID can run the centralised XBMC database (see here for the plugin).

 

Depending on your needs I would recommend you to at least take a look on Plex. unRAID with Plex is an awesome combination. You get an officially supported Plex Media Server running in unRAID and can then use any of the multiple client platforms supported by Plex. Since Plex is built on top of the XBMC code base you will be pleased to see that almost all the XBMC goodies are there with some very nice additions. Plex is built right from the start to be a true server / client solution. If you wish to try it out, it should take no more than 15 minutes to setup following this guide.

 

If you have a lot of customised meta data, you can utilise this Plex meta data agent to re-use it fully.

[snowboardjoe]

Regarding your flash drive question, you only need a small device.  I have a San Disk Cuzer Fit 8GB flash drive and only using 300MB with several add-ons included. 

 

http://www.sandisk.com/products/usb/drives/cruzer-fit/

 

Very small and has a tiny LED for activity.

[METDeath]

Looked at what I really would want the server to do, narrowed it down to the MySQL server and torrents. Everything else I can offload to either a Raspberry Pi or my main PC. Plus I can then move the BD-RW back to my main desktop and use it to rip movies (currently only has a DVD-RW), just have to move the 25 GB of stuff I want to burn over to the the main PC first.

 

I'm not really interested in Plex, as I have a perfectly functional XBMC setup (three to four devices all using the MySQL database, I know the few quirks, and am an old hand at setting them up... plus it's FREE) I tried it a while back, didn't really care for it. Hell, technically I can make a RPi a torrent client (use a separate one) however, with as many large torrents as I run I'd rather just have it on a real machine.

[METDeath]

Bumping my own thread, 'cause I can.

 

So, new questions:

 

Largest drive needs to be Parity, correct?

 

If I want to have one VERY large folder that spans multiple drives, is that possible? I have a large collection of media that is currently on RAID 5 arrays, but am moving away from that in favor of unRAID (looked at ZFS, it doesn't feel like it's quite the solution for me.

 

Can I attach say... a USB drive directly to my unRAID box, and copy to/from using the unRAID box itself (or the web GUI)?

 

As I understand it, if my USB drive that unRAID is loaded on dies I've effectively lost my license for unRAID and have to buy it again?

[trurl]

Largest drive needs to be Parity, correct?

Parity must be as large as or larger than any other drive in the array.

If I want to have one VERY large folder that spans multiple drives, is that possible?

Yes

Can I attach say... a USB drive directly to my unRAID box, and copy to/from using the unRAID box itself (or the web GUI)?

Yes, USB2. You can also connect drives by SATA outside the array for copying. And you can use NTFS and other file systems supported by Linux on drives outside the array, but array drives must be ReiserFS.

As I understand it, if my USB drive that unRAID is loaded on dies I've effectively lost my license for unRAID and have to buy it again?

Limetech will usually give you a new license for a new USB if your old one dies.

[JonathanM]

If I want to have one VERY large folder that spans multiple drives, is that possible? I have a large collection of media that is currently on RAID 5 arrays, but am moving away from that in favor of unRAID

Folders can span all array drives or be limited to which specific drives you want. However, individual files cannot span drives, each drive is complete in and of itself. So, you can't store a file larger than the free space left available on any single drive. This is generally not an issue if you keep a reasonable amount of free space available.

Can I attach say... a USB drive directly to my unRAID box, and copy to/from using the unRAID box itself (or the web GUI)?

Yes, but USB as it's currently implemented in unRaid is generally slower than a properly configured gigabit network, so it's typically just as fast to copy from your network machines. ESata might be a little faster than the network, depends on the drives.

[Kryspy]

METDeath,

 

My advice, keep your current set up and use snapRAID or Flexraid for redundancy.  If you want a pooled drive set up then on windows I think Flexraid is your best bet.  I am not a windows guy so I am unsure of other windows pooling programs.

 

Kryspy

[SSD]

METDeath,

 

My advice, keep your current set up and use snapRAID or Flexraid for redundancy.  If you want a pooled drive set up then on windows I think Flexraid is your best bet.  I am not a windows guy so I am unsure of other windows pooling programs.

 

Kryspy

 

I don't agree with this.

 

RAID-5 is a somewhat scary setup. If one drive fails, you can rebuild it. But if something goes wrong in the rebuild you loose all of the data on all of the disks. With a full backup (RAID-5, unRAID, or one of its competitors), you would certainly be better protected, but that is a lot of redundancy. Let the value of your data versus the cost of redundancy be your guide.

 

If you are looking for a more robust primary media storage platform, I think that unRAID offers several advantages. If a drive fails, you can rebuild it. But if something goes wrong on the rebuild, you may lose data on one or two disk, but you will not lose data on unaffected disks. And because even the 1 or 2 disks that failed are each formatted as autonomous disks, the chances of using a tool like reiserfsck to recover data from them has been extremely successful. Also, unRAID's real-time parity protection does not rely on configuration updates, nightly processes or user ad hoc processes to provide its redundancy features - from the moment you copy data to the protected array it is protected. So when I copy a file from my Windows box to my unRAID box, I am confident to delete the Windows copy.

 

unRAID provides a feature called "user shares" that somewhat mimics your single folder spanning multiple drives. You can read about this feature. There is an excellent section in wiki "unoffical documentation" area. But, at a high level, root directories of the same name on multiple drives appear as a single unified "user share" folder.

[BobPhoenix]

METDeath,

 

My advice, keep your current set up and use snapRAID or Flexraid for redundancy.  If you want a pooled drive set up then on windows I think Flexraid is your best bet.  I am not a windows guy so I am unsure of other windows pooling programs.

 

Kryspy

Drive Bender and StableBit DrivePool are two windows Pooling programs that I use(or have heard of).  I use StableBit on my WHS2011 VM.  I'm sure there are many others.

[Kryspy]

BobPhoenix,

 

SnapRAID isn't raid.  It's a parity based redundancy just like we have now.

 

Kryspy

[BobPhoenix]

BobPhoenix,

 

SnapRAID isn't raid.  It's a parity based redundancy just like we have now.

 

Kryspy

Think you meant bjp999.  I just posted two pooling options in Windows for you.

[SSD]

BobPhoenix,

 

SnapRAID isn't raid.  It's a parity based redundancy just like we have now.

 

Kryspy

 

I was responding to your comment ...

 

My advice, keep your current set up ...

 

and the OP said ...

 

I have a large collection of media that is currently on RAID 5 arrays ...

 

I thought you were advising he stay with the RAID-5 and get a backup system. Maybe that was not your intent.

[Kryspy]

Oh yes sorry. I meant bjp999.

 

Sent from my Nexus 5 using Tapatalk

 

 

[METDeath]

I've thought about it a fair bit these past few weeks... particularly last week as one of my larger RAID5 array had a soft fail (parity mis-match?, not sure...) and a hard drive malfunction (extended read failure). So I seem to have most of the data after replacing the faulty drive and performing a verify/rebuild. However, I got lucky this time, other wise would have lost a Windows report 5.5 TB of data. I need to begin the long process of doing random spot checks, and possibly a complete CRC check of all the video files on that array (all publicly available).

 

I looked into ZFS... and didn't want to lose a bunch of storage to parity... because I would have done two 8 drive RAIDZ2 arrays. One would have been 3 TB drives, and the other 2 TB drives.

I also didn't want to get into something that involved and want something that will just work.

 

Due to internet usage changes don't need nearly so much from my storage machine, I no longer run torrents locally, I have enough web browsing devices and machines that I really don't need to use it for web browsing and light computer projects.

 

So my plan now is to do the following:

-Set up an unRAID free box to preclear two <1 TB drives

-Test smb share access from Windows and Linux for XBMC

-Either run a VM to host a central XBMC MySQL database or get a Raspberry Pi setup for the MySQL db*

-Should the above all work as I hope, I will then change the long process of converting my data and drives to unRAID

--in the test box preclear a 3 TB trio

--prep my file server box for conversion (remove all the drives and my current RAID cards, drop the CPU down from the current Phenom II x6 unless I decide to do one or more VMs on the machine, install 4 GB ECC RAM).

--copy my smaller (4 x 2 TB) array to unRAID

--move, and preclear the four 2 TB drives

--copy my larger (4 x 3 TB) array to unRAID

--move, and preclear the four 3 TB drives

--copy my remaining JBOD drives (one 3 TB, one 2 TB, two 1 TB)

--shuffle data internally on unRAID for best data spread

 

I'm posting my intended build in another thread, found here as it should see a wider audience for additional feedback.

 

I'm leaning towards getting the Pi set up to be my XBMC central server simply because all my XBMC machines are SSD (Windows based) or thumb drives (OpenELEC), so they will be in the gui in a matter of seconds so having the MySQL server up before them will help them load more smoothly.

[cirkator]

Would the pi be fast enough to serve the DB to XBMC smoothly? Interesting usecase for a PI.

[METDeath]

Would the pi be fast enough to serve the DB to XBMC smoothly? Interesting usecase for a PI.

 

It's been done, but I had no luck with it. I asked in this thread on XBMC forums... however, it is from nearly a year ago. I may drive the clock all the way to the max available in the config and not go over that. Since it's headless I may stream the packages down (remove X and other GUI elements after it's working 100% and I've verified Linux access with my unRAID setup.

 

Update: I got it working under ArchLinux for the Pi. It seems to be as snappy as it was running on my Win 7 box. Granted, I did do the "High" overclock profile, so the one just below "Turbo".