Have decided to move away from unRAID...

[mikeybunting]

Greetings all...Unfortunately, I have decided to move away from unRAID...as I need to go with a much smaller footprint and may even just move to USB based external storage for any media I want to keep on hand in the future.  That is still to be decided.  I will be retiring soon (~1.5 yrs) to the Philippines and will not want to trust running a massive 24 disk array (even on an UPS) with the power issues in the Philippines (brownouts galore!). unRAID has been fantastic...but my upcoming situation calls for something a bit easier to manage for my family down the road.

 

My questions as I get ready to part out my Norco 4224 MONSTER build:

 

Is there anyway to directly copy files from each individual disk in my array to another computer or external drive? 

 

Can I stop the array, remove the disks and then use carbon copy cloner (or something similair) to copy the drives?  I don't think so, due to the file system differences...so if not, what is the best method to go about doing what I need to do?

 

Thanks!

 

[itimpi]

Greetings all...Unfortunately, I have decided to move away from unRAID...as I need to go with a much smaller footprint and may even just move to USB based external storage for any media I want to keep on hand in the future.  That is still to be decided.  I will be retiring soon (~1.5 yrs) to the Philippines and will not want to trust running a massive 24 disk array (even on an UPS) with the power issues in the Philippines (brownouts galore!). unRAID has been fantastic...but my upcoming situation calls for something a bit easier to manage for my family down the road.

 

My questions as I get ready to part out my Norco 4224 MONSTER build:

 

Is there anyway to directly copy files from each individual disk in my array to another computer or external drive? 

 

Can I stop the array, remove the disks and then use carbon copy cloner (or something similair) to copy the drives?  I don't think so, due to the file system differences...so if not, what is the best method to go about doing what I need to do?

 

Thanks!

One of the features of unRAID is that each disk is its own self-contained file system.  Each disk can therefore easily be put into a USB adapter/box and attached to another box for accessing the files it contains.  The requirement is tha the system you want to attach it to should be capable of handling the reiserfs format (assuming that you are still using that since on v6 there are other options).  Tools for reiserfs exist for both PC and Mac.  If tunning Linux on the system then reiserfs supp[ort is a standard feature so there is no special actin required.

[Fireball3]

It is also not clear if your files are organized in shares or on specific drives in unRAID.

You could end up having files in a share distributed on multiple disks.

Merging them from single drives might be a little time consuming.

 

[DanielCoffey]

And you might also want to look at a good UPS which will protect you from the power issues.

[dgaschk]

 

Greetings all...Unfortunately, I have decided to move away from unRAID...as I need to go with a much smaller footprint and may even just move to USB based external storage for any media I want to keep on hand in the future.  That is still to be decided.  I will be retiring soon (~1.5 yrs) to the Philippines and will not want to trust running a massive 24 disk array (even on an UPS) with the power issues in the Philippines (brownouts galore!). unRAID has been fantastic...but my upcoming situation calls for something a bit easier to manage for my family down the road.

 

My questions as I get ready to part out my Norco 4224 MONSTER build:

 

Is there anyway to directly copy files from each individual disk in my array to another computer or external drive? 

 

Can I stop the array, remove the disks and then use carbon copy cloner (or something similair) to copy the drives?  I don't think so, due to the file system differences...so if not, what is the best method to go about doing what I need to do?

 

Thanks!

One of the features of unRAID is that each disk is its own self-contained file system.  Each disk can therefore easily be put into a USB adapter/box and attached to another box for accessing the files it contains.  The requirement is tha the system you want to attach it to should be capable of handling the reiserfs format (assuming that you are still using that since on v6 there are other options).  Tools for reiserfs exist for both PC and Mac.  If tunning Linux on the system then reiserfs supp[ort is a standard feature so there is no special actin required.

 

I have found no Mac reiserfs tools. Where are they?

[WeeboTech]

I have found no Mac reiserfs tools. Where are they?

 

You can probably run unRAID with vmware fusion/fusion pro, then mount the drive in a USB shell, attach it to the unRAID virtual machine.

Set up a 1 disk array and access it via internal network.  Yeah. I get that it's a pain.

 

Another option is a Dell E series laptops (a used one) can run unRAID, with an eSATAp port in a USB/eSATA shell.

There are some trayless eSATA shells (with fans) by startech that work great in this scenario.

Attach it to the network in mini unRAID mode. 

 

But if you are going to go that far, then a cheap HP Mini proliant microserver can help too.

[garycase]

I'd be inclined to just do the following:

 

(1)  Buy 2 or 3 cases (20-30 of the storage boxes) of these:

http://www.amazon.com/DriveBox-3851-0000-11-Hard-Disk-Case/dp/B004UALLPE

 

(2)  Store all of your data drives in these boxes.  But as you're doing so, first create a PDF file with the directory contents of each drive, so you can easily find anything you want without physically accessing the drives.  You can do this in Windows using the excellent "Directory Printer" [ http://www.galcott.com/dp.htm ] and "printing" the directory to a PDF file with CutePDF [ http://www.cutepdf.com/ ]

 

(3)  Install the free LinuxReader [ http://www.diskinternals.com/linux-reader/ ] on a Windows box (perhaps a laptop in your new situation) and simply attach whichever drive you need at the moment via a USB bridge device.

 

At your leisure, you could go through the disks one-at-a-time and decide just what you really want to keep, and copy that off to a large external drive ... I suspect that this would notably reduce the drive count you'll have [just how much depends on (a) how much of a "pack rat" you are  , and (b) what size your current drives are vs. the size of the externals you're using. ].

 

 

[WeeboTech]

I wouldn't use all those windows tools when the most basic stuff is already there on unRAID.

Since you mentioned carbon copy cloner, I'm assuming mac os is the front end that's planned.

You can save text files of the drive contents on your flash after it is consolidated.

 

This gets you the drive model/serial

 

root@unRAID1:/boot/log# ls -l /dev/disk/by-id | grep -v part

total 0

lrwxrwxrwx 1 root root  9 2014-11-23 19:25 scsi-1ATA_Hitachi_HDS5C3030ALA630_MJ1321YNG0EEXA -> ../../sdg

lrwxrwxrwx 1 root root  9 2014-11-23 19:16 scsi-1ATA_Hitachi_HDS5C3030ALA630_MJ1321YNG0GBPA -> ../../sdd

lrwxrwxrwx 1 root root  9 2014-11-23 19:16 scsi-1ATA_ST3000DM001-1CH166_W1F1GWZK -> ../../sdc

lrwxrwxrwx 1 root root  9 2014-11-23 19:16 scsi-1ATA_ST3000DM001-9YN166_W1F191JR -> ../../sdb

lrwxrwxrwx 1 root root  9 2014-11-23 19:16 scsi-1ATA_WDC_WD20EARX-00PASB0_WD-WCAZAC993717 -> ../../sde

lrwxrwxrwx 1 root root  9 2014-11-23 19:17 usb-Generic-_USB3.0_CRW_-0_000000002318-0:0 -> ../../sdf

 

This matches it with the disk number, In this case. 1.

 

root@unRAID1:/boot/log# grep sdc /proc/mdcmd 

rdevName.1=sdc

 

root@unRAID1:/boot/log# grep 'disk.*\.1=' /proc/mdcmd     

diskNumber.1=1

diskName.1=md1

diskSize.1=2930266532

diskState.1=7

diskId.1=ST3000DM001-1CH166_W1F1GWZK

 

You can also do this from the webGui to find the Model/serial

 

When the drive is mounted it is on /mnt/disk1in my case

In a terminal session do

 

find /mnt/disk1 -type f -fprintf /mnt/disk1/ST3000DM001-1CH166_W1F1GWZK.txt "%P\r\n"

 

This makes a dos.txt file (cr\lf) in the root of the drive that is relative from that drive downward. "%P"

For standard unix take out the cr '\r' character.

 

If you want the full path use %p instead of %P

 

%p    File's name.

%P    File's name with the name of the command line argument under which it was found removed.

 

Now you have a dos txt file of all the files on the filesystem that can be loaded into an editor or copied to the flash drive.

It will have the serial number of the drive and the list of files on the drive.

 

Output is like this.

 

root@unRAID1:/mnt/disk1# head ST3000DM001-1CH166_W1F1GWZK.txt

md5sums.nonexist

md5sums.relative

md5sums

md5sums.log

md5sums.new

ST3000DM001-1CH166_W1F1GWZK.txt

filelist

filedb/disk2.filelist.2014-11-01.txt

filedb/disk2.filelist.2014-11-02.txt

filedb/disk2.filelist.2014-11-04.txt

 

If you wanted to go a step further you can make an md5sums file of the whole drive to detect corruption.

That is still searchable, just with an extra field.

 

cd /mnt/disk1 # This is important to make sure the md5sum command works correctly with relative files.

find /mnt/disk1 -type f -printf "%P\0"  | xargs -0 md5sum > /mnt/disk1/ST3000DM001-1CH166_W1F1GWZK.md5sums

 

output is like this.

 

root@unRAID1:/mnt/disk1# head ST3000DM001-1CH166_W1F1GWZK.md5sums

90aae0712ad50a0255690e003969bbd3  md5sums.nonexist

fd50229dfc3a694bebd358101195651e  md5sums.relative

b3689f18fdd389775bec465d8d6f739d  md5sums

fa665c6001ecb8a8c4a18eccab349d9a  md5sums.log

8fba62d641ab5f30e8d1230030b74abc  md5sums.new

e702fa77e084e85e0a2ec931f87135d1  ST3000DM001-1CH166_W1F1GWZK.txt

dc10aebdd08866dbe8643b772a1b9d57  filelist

6b70e900973ad24fb90427e6dd029aeb  ST3000DM001-1CH166_W1F1GWZK.md5sums

8bc3988079b29097ecd5cbd6f1c3dd82  filedb/disk2.filelist.2014-11-01.txt

c02b20eb606a567d7961883ffc0fbfa6  filedb/disk2.filelist.2014-11-02.txt

 

Now you can quickly grep your filelist with

 

root@unRAID1:/mnt/disk1# grep md5sums /mnt/disk1/ST3000DM001-1CH166_W1F1GWZK.md5sums

90aae0712ad50a0255690e003969bbd3  md5sums.nonexist

fd50229dfc3a694bebd358101195651e  md5sums.relative

b3689f18fdd389775bec465d8d6f739d  md5sums

fa665c6001ecb8a8c4a18eccab349d9a  md5sums.log

8fba62d641ab5f30e8d1230030b74abc  md5sums.new

6b70e900973ad24fb90427e6dd029aeb  ST3000DM001-1CH166_W1F1GWZK.md5sums

 

and check the integrity with.

 

root@unRAID1:/mnt/disk1# cd /mnt/disk1 && md5sum -c ST3000DM001-1CH166_W1F1GWZK.md5sums

md5sums.nonexist: OK

md5sums.relative: OK

md5sums: OK

md5sums.log: OK

md5sums.new: OK

ST3000DM001-1CH166_W1F1GWZK.txt: OK

filelist: OK

ST3000DM001-1CH166_W1F1GWZK.md5sums: FAILED

 

or search the md5sum with

 

root@unRAID1:/mnt/disk1# grep md5sums /mnt/disk1/ST3000DM001-1CH166_W1F1GWZK.md5sums

90aae0712ad50a0255690e003969bbd3  md5sums.nonexist

fd50229dfc3a694bebd358101195651e  md5sums.relative

b3689f18fdd389775bec465d8d6f739d  md5sums

fa665c6001ecb8a8c4a18eccab349d9a  md5sums.log

8fba62d641ab5f30e8d1230030b74abc  md5sums.new

6b70e900973ad24fb90427e6dd029aeb  ST3000DM001-1CH166_W1F1GWZK.md5sums

 

you can also use my older method of just calling it

 

disk1.filelist.txt and disk1.md5sums.

If they are in the root of all disks you can search your whole system for a file with.

 

grep /mnt/disk*/disk*.filelist.txt

 

root@unRAID1:/boot# grep -n md5sums /mnt/disk*/disk*.filelist.txt

/mnt/disk1/disk1.filelist.txt:1:md5sums.nonexist

/mnt/disk1/disk1.filelist.txt:2:md5sums.relative

/mnt/disk1/disk1.filelist.txt:3:md5sums

/mnt/disk1/disk1.filelist.txt:4:md5sums.log

/mnt/disk1/disk1.filelist.txt:5:md5sums.new

/mnt/disk2/disk2.filelist.txt:1:md5sums.nonexist

 

if you move them to the same directory off your root filesystem (in my case I use filedb)

Then they will all be collected in the same user share folder called filedb.

 

In my case I used full paths.

 

root@unRAID1:/boot# grep md5sums /mnt/user/filedb/disk*.filelist.txt 

/mnt/disk1/md5sums

/mnt/disk2/md5sums.nonexist

/mnt/disk2/md5sums

 

However what I've found useful in the archiving aspect is to copy these files to the drivemodel-serialnum file on the boot flash somewhere.

Now you have a searchable index pointing to the drivemodel-serialnum.

 

If you use common extensions such as filelist and md5sums you can do global grep searches or load each individually into an editor for searching.

 

If this is saved on your flash in let

Let's say a /boot/filedb folder.

 

You can boot up the scraptop with the unraid flash, do a search via command line.

Get the drive, mount it in some eSATA external unit and make it available for copying over the network or to another drive.

Say some internal SSD or spinning 2.5" drive. ala cache.

 

Point here is, you don't need external software when the tools are there for you.

Since you mentioned carbon copy cloner, I'm assuming mac os is the front end that's planned.

 

vmware fusion lets you load unRAID locally, albeit with a lil complexity and dependancy on the mac.

a Dell E series with eSATA (or any model with eSATA) lets you load it like a mini temporary fileserver (since you're already familiar with unRAID).

 

Given the webGui and a lil PHP programming, a plugin can be created to grep files like a mini internal google or locate command.

Find the disk.

Load it.

do a new config on the array.

go to main screen

configure new drive

Start the single drive array.

 

You can't beat having native ReiserFS on unRAID since it was already created there.

 

The cost for a used Dell E series with eSATA can range from 250-400 depending on quality and CPU model

Then there is the cost of an eSATA/USB box (This is a given in all scenarios).

USB 2.0 is limited to 30MB/s, eSATA is 90-190 depending on drive model, Ethernet transfer is 90MB/s max.

 

Vmware fusion route is about $75.

But there's slight complexity there until you know what you are doing, then it gets easy.

 

If you really think about it though, by the time this is all said and done, you can get an HP microserver for approx $200.

any ICY Dock removable unit that lets you pop a drive in and out with ease like a floppy for approx $50.

You already have unRAID on a flash and you are familiar with it.

 

So the most used media can be on the unRAID microserver and the less often media on a removable hard drive

 

I have this unit. (but the fan can get noisy after a long time of use)

http://www.newegg.com/Product/Product.aspx?Item=N82E16817707169

 

This lets you load two.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817707369

 

The hp micro servers are cheap these days as the model is going to be retired. I have two and love them.

They are great utility servers.

http://www.newegg.com/Product/Product.aspx?Item=N82E16859108164

 

This is the icy dock that can be used on top of the micro server.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817994143

 

You would have space for an SSD cache drive also, but that will require a controller or some external to internal wiring.

 

With the dell E series with eSATA you can find a model that can load OS/X and have it do double duty.

Check out here

http://www.osxlatitude.com/

 

The important part is to have a craptop with eSATA so you have decent speed and unRAID can load it from the webGui.

you'll get tired of entering the commands after a while.

 

After hurricane sandy, having half my server under water and two boxes of hard drives I was happy to have serial number indexed filelists and a quick trayless insert/remove tool.

You can also get one of those desktop docks, but then you have to think about cooling. The startech units have fans.

The microserver moves enough air that the top doesn't get all that hot and the ICYdock is solid metal which acts like a heat sink.

 

Here is what I built after hurricane sandy to recover some important data.

http://lime-technology.com/forum/index.php?topic=25136.msg218357#msg218357

In that case I used an old Dell D series, a dock capable of PCI with a silicone image chipset for port multiplier capability.

It worked for me at that time and was cost effective to recover critical records.

The E series has an eSATA port on the machine and another on the dock if you go that route.

I would still suggest the microserver unless you double duty the scraptop with other functions.

 

Hope this helps.