what's the best way to move 24tb of media files to an unraid current build system?

 

I've been kick'n around the idea of using a usb3 ethernet from the motherboard to the internet, and using the onboard NIC for drobo5N to unraid data transfer...

 

I am a total noob with droboports addin apps, so rcopy(?) is not a solution, as well as unraid that's not drag and drop and gui happy,  I've been playing with unraid and need to utilize the dedicated windows plex box to an unraid plex box. when it's not plexxing I need to spin up a windows vm for rdp sessions and gradually learn the inner workings of unraid and docker containers, etc.

 

 I was thinking about using KRUSADER DOCKER container, but I don't know how to mount the drobo5n to the unraid for best throughput and file integrity, etc etc etc....

 

this project is imperavtive that I complete it, I need my plex.tv and then I need to learn docker, VMs and linux down the road

 

I'm kicking around in my head wether to use a multi-TB spinning drive for my cache drive for when I'm moving data then when the data is moved, preclear it, add it to the pool and then add my ssds onto the unraid for normal operation.

 

I realize there is about 17 questions in this message but I really need to get this project futher along than it is...   If someone reccommends that we get on a skype fee for services rendered to set things up for me, I would need to break out a credit card, I live on disability and I've used it up for this month and it's only  the 8th.

 

Thank yuo for all your help Lime-tech community!

-TinkerToyTech

 

 

 

 

Hi -

 

A little more information would help.  Do you have a new set of drives in the unRAID box, or are you scavenging from the Drobo?  Maybe describe your unRAID setup...  Do you have a Windows machine to help facilitate the transfer if necessary?  Have you loaded the Community Applications plugin, and from there tried the Unassigned Devices plugin to mount the Drobo via SMB? 

Sorry, it's a virgin unraid box which is a follows:

 

I will be buying the Unraid 12 drive license:  4 ports on mobo, 8 on card

the 6GB ports on the mobo will get the ssds, a 128 dedicated to a windows 10 vm, the other will get the 960gb sandisc Ulta

 

my Unraid box

Z77 EVGA STINGER Motherboard - mitx

Unraid IT mode LSI 2008 card

8gb ram for now, 16gb corsair when it comes back

iNtel i7 2600K cpu

Hyper 212 evo cooler with 120MM fan

Fractal Design's Node 804 case

3 front intake 120MM fans

2 rear exhaust fans

5 tb parity drive

1 potential 960tb for cache

26tb of scavanged drives from around the shop

16gb USB 3.1/3.0 Trasnscend JetFlash 710S thumb Drive

1 potential usb3 dongle w/3 usbs and a gigabit port

 

I mention the second gigabit port with an idea:

if I can add the drobo5N to one port, tied to the windows vm, use another port, on a switch, hub or crossover cable to not congest my home's network connection while a transfer all this data

 

I've been reading and watching untold hours of youtube video

right now there are 8 hds mounted on the drive brackets, I have not hooked anything to use the motherboard yet, the plan is to put a 128 crucial M4 for dedicated win10pro vm and then I can use my license to GoodSync portable on windows to a network share

 

what's the best way to build this up, the parity, 1 data hd and 1 hard-drive for the cache, start it building and use the drive zeroing tool to run the other drives individually as I needed or should I just zero all the drives at once?  If I have a 3tb drive, move 2.5tb to it off the drobo5N to the cache drive and then let it move the data when it's gone idle

 

phew, ok,  next set of questions so I can get a clue?

 

2 hours ago, TinkerToyTech said:

26tb of scavanged drives from around the shop

This line jumped out at me.

 

Before you assign drives to the array, be sure they are healthy, from end to end. That means clean smart reports, and preferably at least a long smart test and full pass with manufacturers diagnostic software, or at least 1 preclear pass.

 

The fault tolerance feature of unraid requires that all drives other than the failed drive to be rebuilt be perfectly read from end to end, so any dodgy drives in the array even if they are totally empty can cause data loss if you need to rebuild a different drive.

 

DO NOT USE DRIVES THAT ARE IN ANY WAY UNTRUSTWORTHY. I learned this the hard way many years ago, my thought process at the time was, "hey, if a drive fails I can just rebuild it, so who cares if the drives I use are sketchy". Then a drive full of valuable data outright failed, and one of my empty array drives started acting up during the rebuild, and I ended up losing data.

 

2 hours ago, TinkerToyTech said:

what's the best way to build this up, the parity, 1 data hd and 1 hard-drive for the cache, start it building and use the drive zeroing tool to run the other drives individually as I needed

 

Yes, only add drives to the array as you need the capacity, having lots of empty spindles increases the chances of problems without giving you any real benefit. I wouldn't use the cache feature for migrating mass data to the array, just use turbo write mode and copy directly to the array disks until you have everything migrated. If you take the time to absorb how user shares work, and how they interact with disk shares, you can turn on disk shares and write directly to the target disk you want. In many cases turbo write combined with copying directly to a disk share will be as fast or faster overall than filling and emptying the cache drive while copying many TB's of data.

 

Don't enable and use disk shares until you understand how they participate in the user share system, because failure to understand why the same data shows up in 2 locations can cause data loss if you copy from one location to another on the array. If you just want to use unraid without digging in to the backbone of how it works, user shares by themselves are perfectly safe, just a little slower in some instances.

On 4/8/2017 at 11:37 PM, jonathanm said:

This line jumped out at me.

 

Before you assign drives to the array, be sure they are healthy, from end to end. That means clean smart reports, and preferably at least a long smart test and full pass with manufacturers diagnostic software, or at least 1 preclear pass.

 

The fault tolerance feature of unraid requires that all drives other than the failed drive to be rebuilt be perfectly read from end to end, so any dodgy drives in the array even if they are totally empty can cause data loss if you need to rebuild a different drive.

 

DO NOT USE DRIVES THAT ARE IN ANY WAY UNTRUSTWORTHY. I learned this the hard way many years ago, my thought process at the time was, "hey, if a drive fails I can just rebuild it, so who cares if the drives I use are sketchy". Then a drive full of valuable data outright failed, and one of my empty array drives started acting up during the rebuild, and I ended up losing data.

 

 

Yes, only add drives to the array as you need the capacity, having lots of empty spindles increases the chances of problems without giving you any real benefit. I wouldn't use the cache feature for migrating mass data to the array, just use turbo write mode and copy directly to the array disks until you have everything migrated. If you take the time to absorb how user shares work, and how they interact with disk shares, you can turn on disk shares and write directly to the target disk you want. In many cases turbo write combined with copying directly to a disk share will be as fast or faster overall than filling and emptying the cache drive while copying many TB's of data.

 

Don't enable and use disk shares until you understand how they participate in the user share system, because failure to understand why the same data shows up in 2 locations can cause data loss if you copy from one location to another on the array. If you just want to use unraid without digging in to the backbone of how it works, user shares by themselves are perfectly safe, just a little slower in some instances.

 

phew, ok,  next set of questions so I can get a clue?

 

ok,  I worded that poorly, the drives are drives I filled and swapped out from my Windows Home Server to drobo5N migration, starting out with 3TB drives only to discover I needed more room, then more room, swapped out for 6tb, then another, next etc my de-duping wasn't going as well as hoped, but drobo says the volume limit is 64TB - problem.  so I bought a 5N and filled it with 3TB WD REDS.  Filled that, put in a 6TB, filled another 6TB, got fed up with that...

 

I look to buy an 8tb WD RED drive, $424 at Amazon, then I see that a 16TB MyBook was $599, so for 150 dollars more I get another 8TB of storage and 25 dollars I get a little box to toss two 3tb drives into!  So I stuffed the first one into the drobo, filled as all data expands to space available, about 15% or so free, drobo, prompts me for a bigger drive, popped in the next drive, and 3 days later I come to find that I have nearly 6tb of unusable space and I need to move the data off the drobos, one at a time, to get it better sorted, then all the drives reclaimed in the big server which some of will be spelled out in the next paragraph.

 

Which brings me to why I went to put the little server together.  I have this data sitting on 2 drobo5Ns full of more expensive RED drives that I can access built I had the parts to put this box together, then I can put the 2x8TB drives for dual parity drives in the old WHS case, and move it to the i7-3700K with full ATX motherboard and all the bays I can find to shove the 10 Bays in the Fractal Designs XL rev-1 10 bay in the case, plus 5x3.5s in a cage and 4x2.5" cage. 2x256GB for my dockers, and other neccesities in raid1 and the Sandisc 960GB ultra for caching, the 1tb dedicated Windows 10 VM drive for my rdp stuff that need installed in running order, when it's not being a fulltime Plex box for the house.  When I outgrow this the intel ti-ock tock tock, will have gone through another set of internations and I'll saved enough to buy a new motherboard and processor.

 

so now you should have a better idea of that I want to do, the 2600K based server is just to move the data off the overfull drobo, pull those drives and those to the final destination 3700K plex sever with dual parity and all my newest WD RED Hard drives, which is 8+8+6+6+4TB worth of drives, then move the 15TB of known great from the other drobo's, fill them up with all the seagates I have laying around, include a spare and sell the drobos off

Regardless of what you think about the reliability of your drives, you would still do well to test them before trusting them in the array.

 

Other than that, I think the answer to your initial question rests with the Unassigned Devices plugin. It will let you mount the drobo and copy its data to your unRAID.

11 hours ago, trurl said:

Regardless of what you think about the reliability of your drives, you would still do well to test them before trusting them in the array.

 

Other than that, I think the answer to your initial question rests with the Unassigned Devices plugin. It will let you mount the drobo and copy its data to your unRAID.

 

ok, I'll bite, if I have a base image of unraid/unbuntu/etc current setup, if there something in nerdtools or in an another respostory that I can test a batch of discs and then toss out any that are the least bit suspect?  My knowledge of things linux are very limited.  I'll start looking in my limited familar areas such as sourceforge, and other repos until I find something.  I understand* how critical the discs need to be, believe me. I, Get, the POINT.  ok?

Edited April 11, 20179 yr by TinkerToyTech
early morning fumble fingers

People usually use the Preclear plugin.

 

When you add a disk to a new slot in the parity array, it must be clear to be consistent with existing parity. A clear disk is all zeros, which has no impact on parity.

 

If you add a disk to a new slot in the parity array, unRAID will clear it if it isn't already clear. Older versions of unRAID would take the array offline while it cleared a disk, so preclear was invented. Preclear allows you to clear a disk before (pre) adding it to the array. Preclear clears a disk then writes a special "clear signature" to the disk so unRAID will recognize it is clear.

 

Preclear also tests a disk, It reads all of the disk, writes zeros to the whole disk (clearing it), reads back all of those zeros to confirm that the disk is clear, then writes the clear signature.

 

Even if you don't need a clear disk, such as when replacing a disk (instead of adding one), people often use preclear to test a disk before trusting it in their array.

 

The drive manufacturers also have free diagnostics which will allow you to test a disk on another computer before using it in unRAID.

 

Preclear support thread:

 

And here is Unassigned Devices:

 

I planned on preclearing the drives, my question and frustrration came from my 

question getting derailed to drive stability when I have had a house full of people

that are missing their Plex server.  as I think I mentioned I harvest stuff from two

TiVo's with 4TB drives in each and princess and the queen missing their shows

tha were on the Plex server. I need to get out of the doghouse!

 

@TinkerToyTech

 

[ticket closed]

31 minutes ago, TinkerToyTech said:

frustrration came from my 

question getting derailed to drive stability

Just trying to help without assuming too much about your prior knowledge since we don't know you well.

 

34 minutes ago, TinkerToyTech said:

[ticket closed]

We're all just fellow users here.

 

ok, I have a question referring to all the 100's of the others I had, my addon adding problem came from the fact that the only mainstream Windows browser that I don't normally have loaded is firefox.  Once I loaded it, opened my server up and BiNGO, the add button magically shows up!  So now I just need to learn how to run each of the plugins and dockers in all their splendor and glory.  Gridrunner's videos are excellent, in fact, next month if I have any money over from teprevious month I'll send him a tip even if sometimes it takes me a week to watch and implement more than 10% of it. couple questions:

 

What about running "Synergry".  The program lets you share a monitor by simply scrolling the mouse to the side of the screen over the network?  It supports Windows, PC and multiple flavors of Linux

 

Is there a notepad that is available on the UNRAID server system?  I see it seemingly done on many youtube videos, or those with Mac laptops, but all my other notepads / wordpads / etc are all on Windows

 

Does it matter if I copy data from inside UNRAID with Krusader, or if I copy the data ONTO the server, where I want it using a Windows utility GoodSync?  The external utility appears to be considerably faster and I can tell it to use ftp, could I use the ftp server inside to transfer all of all this drobo5N data directly to the directories within my /multimedia directories such as /music/albums /music/VA (various artists) /music/dance that I want them to go to?

 

And finally: Can Precleared drives from one unraid system be placed in another?  My understanding is one XFS drive can be read on any system that can read XFS drives, is XFS available in Windows?  Perhaps under HyperV?

 

Thank you for your answers.

@TinkerToyTech

 

 

 

 

Edited May 15, 20179 yr by TinkerToyTech
late night spelling errors