[Solved] Copying 4tb data -- where to put it so spanning happens?


Recommended Posts

Hey all. I am a long-time lurker of unraid and after a recent drive failure in my WHS setup which caused massive data loss I have decided it is time to give something else a try. :)

 

I have done a ton of reading on wiki, this forum, etc, etc, etc, but there is one question I have that I haven't been able to find an answer to. I have tried googling and searching the forums and no luck.

 

I will be setting up unraid with a certain number of drives and then physically hooking up additional drives one at a time and manually copying the data on to unraid. Due to the size of these drives (up to 2tb) it will be very difficult for me to just pick a drive and dump it on to that one as I may not end up with enough free space to copy a whole drive on to another drive.

 

I said all that to say this: I have created a share called "unraid." I have the allocation method set to high-water. I have left everything else blank. I do not have anything in min. free space, split level, included/excluded disks, etc.

 

When I install one of my old harddrives and mount it, can I simply copy the data to /mnt/user/unraid? And then that way it will automatically span it across the drives for me?

 

Thanks!

 

Edit: If anyone would like to give input on which split level I should use, my folder structure is:  ./Movies/Movie Name/Movie.avi. However my tv shows are: ./TV/Show name/Season #/Episodes.avi . I believe that split level should be set to 1, is that correct? I'll never be playing the tv shows one right after another (like the vob file example) so waiting for the drive to spin up is no big deal. And my movies are all single files (avi or mkv) so that's no problem either. Should I even put it as split level 1 or should I just leave it blank like I have it?  Edit 2: After a little more research it looks like I may actually want to set it to a really high number like 999 as it doesn't much matter to me where the files end up.

Link to comment

That's the general idea. However, you might want to put more consideration into the user shares and the settings so you don't end up with a jumbled mess of data spread out over all the drives. With the random storage method, if something bad did happen (you lost a disk of data) then how would you ever figure out what you did lose? If you actually put some logical thought into your storage then you know what's stored where.

 

Peter

 

Link to comment

That's the general idea. However, you might want to put more consideration into the user shares and the settings so you don't end up with a jumbled mess of data spread out over all the drives. With the random storage method, if something bad did happen (you lost a disk of data) then how would you ever figure out what you did lose? If you actually put some logical thought into your storage then you know what's stored where.

 

Peter

 

 

Hey Peter,

 

That does make total sense as in the case of me having just lost one of my WHS drives (accounting for 25% of my total storage space) I don't actually know specifically what I lost because WHS handles all the pooling and spanning. However my thought with unraid is that should I lose 1 drive then I can rebuild it and it's no big deal. Should I lose more than 1 at a time well then I'm kind of out of luck anyways. I have so many movies and such on here that even knowing which ones I lost wouldn't help me any. Unless there is another point that I'm missing?

 

Thanks for the quick response!

Link to comment

Hello,

 

I have unRaid storing over a thousand movies and even if I lost two drives it would be a big help knowing which movies I would have to re-burn. I make it a habit to put any files I copy to unRaid on a specific disk and when I am done I always print out a file listing for each disk and place it in safe spot. I do not use any of the spanning features. Of course it would be a pain to re-burn 300 - 600 movies but at least I would know which ones I needed to replace.

 

Just a suggestion.

 

Have a good one,

 

Jim

Link to comment

Hey all. I am a long-time lurker of unraid and after a recent drive failure in my WHS setup which caused massive data loss I have decided it is time to give something else a try. :)

 

Just out of curiousity were you using duplication on WHS? Just wondering how the data loss occured. I just recently came back to UnRaid after several years on WHS after a drive went bad. I did not lose any data, but due to other problems I was going to have to do a reinstall and decided as long as I was gonna start from scratch, I was coming back to UnRaid.

 

To echo what has already been said give some thought to how the data is organized, it makes it a lot easier to manage. I copied my all of my shares  over from WHS and use those as my top level shares so 11 shares at this point. I have set my split levels on my media shares, Movies, TV, Music, ect. for some of my other shares I still have to work split levels out. What I am finding is if I do not enter a split level it acts like a very high number and files end up all over the place, which I personally don't like.

Link to comment

Hello,

 

I have unRaid storing over a thousand movies and even if I lost two drives it would be a big help knowing which movies I would have to re-burn. I make it a habit to put any files I copy to unRaid on a specific disk and when I am done I always print out a file listing for each disk and place it in safe spot. I do not use any of the spanning features. Of course it would be a pain to re-burn 300 - 600 movies but at least I would know which ones I needed to replace.

 

The more that I think about it, the more that this makes sense. I guess for me the main factor is that I can read everything from 1 share which unraid does perfectly, no matter how I have data stored on the drives. It certainly would be nice to know what I lost if something were to happen, and it won't be too hard to simply fill up a drive and then move on to the next. I've always kind of preferred that I don't have 1 full drive and then 1 empty drive, but this fear is calmed a bit knowing that I have that parity drive.

 

Just out of curiousity were you using duplication on WHS? Just wondering how the data loss occured.

 

It was a combo of two things, one that is my fault and one that isn't. First no, I did not have duplication turned on for my movies. It was on for my music and pictures as those are both harder (or impossible) to replace, but not my movies. Reason being is I have about 3tb worth of movies (lots of high-def stuff) and only had 4tb of space. I knew that this was one day a possibility, but I just hoped it was never happen.  :-\ Hindsight tells me that having spent the few hundred dollars on extra drives would be worth it compared to the time I'll be spending redoing it all, but that's how life is.

 

The more catastrophic loss though is that I lost every data backup after that drive died. WHS wanted to run a database repair on the backups and upon finishing, every single backup except one was gone. Typically not a big deal except that I JUST had a laptop die on me (go me, I back up every single night! nothing to worry about right?) and the backups for that computer were lost too. So just terrible timing. It just goes to show that even when you think you are doing things right, things can go terribly wrong. With that said I was under the impression that data backups that were stored on WHS were always duplicated. The problem comes in when you have data that is identical on multiple computers. To make things more efficient, WHS only stores 1 copy of any particular files that are the same across any backups. Lose that 1 file and the entire backup is apparently wiped out. Lesson learned.

 

36YfH.jpg

 

Thanks for the tips and input all!

Link to comment

I lost all of my backups when the backup db became corruped. I set it up after that so that I could back up the backup db.....worked for a while, after several months when nothing bad ever happened I stopped backing up the backup db. So guess what I lost......all my backups. So I know that feeling. What I miss most is the image of my fresh installs with my selected software and config.

 

In the end the fact that I no longer had room with duplication on to add new movies that got me thinking back toward UnRaid.

Link to comment

I would suggest you even just think about seperate shares for movies, music and TV shows. For example, I use split level 1 on my TV shows so a whole series has to stay on a single drive. That way, I know I either have it or it's all gone and I won't have part of a series where I'm trying to piece it back together. Music would be the same way for each artist, except I would suggest considering just keeping music on a single disk. Same would go for movies if you keep more than a single file per movie.

 

If you use a number of shares and set the split levels correctly then things will be somewhat organized and you'll know how the data is on the drives.

 

A collegue at work is also using unRAID (he copied my unRAID basically and also uses MediaBrowser after I recommended it) except he just let the server place the files wherever as it filled. He goofed during a disk failure and hit the Restore button wiping out 1T of data. He was still finding files missing 6 months later. He'd go to watch a TV series he downloaded and find missing eposides in the middle of the series. Or he'd go to play an artist and find songs missing.

 

Peter

 

Link to comment

A collegue at work is also using unRAID (he copied my unRAID basically and also uses MediaBrowser after I recommended it) except he just let the server place the files wherever as it filled. He goofed during a disk failure and hit the Restore button wiping out 1T of data. He was still finding files missing 6 months later. He'd go to watch a TV series he downloaded and find missing eposides in the middle of the series. Or he'd go to play an artist and find songs missing.

 

Sometimes it takes someone laying it out like this and giving you a good example to realize the whole benefit of it. As obvious as it is, I didn't even think about the fact that I may now have tv shows that are missing random episodes, or the same with my music. It makes perfect sense, it just didn't even cross my mind. You're all very right. :) My whole collection is very well organized but I absolutely need to setup the actual shares themselves better.

 

Thanks again Peter, and everyone else. Love these forums. :)

Link to comment

I would suggest you even just think about seperate shares for movies, music and TV shows. For example, I use split level 1 on my TV shows so a whole series has to stay on a single drive. That way, I know I either have it or it's all gone and I won't have part of a series where I'm trying to piece it back together. Music would be the same way for each artist, except I would suggest considering just keeping music on a single disk. Same would go for movies if you keep more than a single file per movie.

 

If you use a number of shares and set the split levels correctly then things will be somewhat organized and you'll know how the data is on the drives.

 

A collegue at work is also using unRAID (he copied my unRAID basically and also uses MediaBrowser after I recommended it) except he just let the server place the files wherever as it filled. He goofed during a disk failure and hit the Restore button wiping out 1T of data. He was still finding files missing 6 months later. He'd go to watch a TV series he downloaded and find missing eposides in the middle of the series. Or he'd go to play an artist and find songs missing.

 

Peter

 

 

That would suck, I think I am going to have to revisit split levels and if I only want to allow certian things to occupy certian drives. I think I read Joe L posting about this too and how he only puts certian shares on certian drives.

Link to comment

Hello,

 

I have unRaid storing over a thousand movies and even if I lost two drives it would be a big help knowing which movies I would have to re-burn. I make it a habit to put any files I copy to unRaid on a specific disk and when I am done I always print out a file listing for each disk and place it in safe spot. I do not use any of the spanning features. Of course it would be a pain to re-burn 300 - 600 movies but at least I would know which ones I needed to replace.

 

Just a suggestion.

 

Have a good one,

 

Jim

 

do you use a app for file listings or just the command line?

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.