Is it possible (via mount points or already existing Linux features) to combine two unRAID boxes? The idea is to use the shares of one machine and these shares already include mount points to a second additional machine. Is this possible?

 

Thanks in advance.

Harald

 

Yes, I have done this with NFS many times under Linux (although not specifically on unRAID) where the second one is mounted under the first.

 

There are even some overlay file systems that will let you see NFS mounts as merged.

I thought that unRAID has this feature already. Call me stupid but mnt/user (with cache enabled) seems to work similar. The cache drive is joined with individual shares.

 

I will try to find an "overlay file system".

 

Thanks

Harald

 

I am interested in this as well...any new info?

I have been searching around - and it seems this has been discussed before and at one point there was a mention of testing features coming out in a beta release. (http://lime-technology.com/forum/index.php?topic=298.0)

 

There may be a better way to do this - so I am going to throw this out there and see what happens.

 

We are using unraid to back up a massive amount of files.  Last time I searched, there were about 15 million.  I'm sure we are close to 25 million now.  These are image files that we scan for government agencies.  My boss told me we couldn't afford a backup - after our production server failed and I was throwing a fit.  I went out and found the cheapest backup solution available - we ordered an unRaid box from a few generations ago, and filled it with 1TB green drives.

 

The solution has been slow, and it's taken some babying on my part because our hardware department doesn't want to deal with it - but I sleep MUCH better at night.  We have maxed out the unraid box, and now need a second.  The problem is, I'd really like to keep all of these images on one share.  We are up to 13.5TB - and I'm about to start migrating data to the second lime box.

 

Has there been any more testing with the idea of a share that spans multiple servers?  Is there some other type of solution that I'm missing?

I hate to toot my own horn - but we're down to 200GB on our 14TB box & I'd really like to know if this is\will be a possibility in the near future 

I hate to toot my own horn - but we're down to 200GB on our 14TB box & I'd really like to know if this is\will be a possibility in the near future 

With all due respect for limetech,  I would not expect this to be in the near future given we're still on beta's for the 4.5 release.

This feature seems like it would be a fairly lengthy beta in it's own.

I hate to toot my own horn - but we're down to 200GB on our 14TB box & I'd really like to know if this is\will be a possibility in the near future 

With all due respect for limetech,   I would not expect this to be in the near future given we're still on beta's for the 4.5 release.

This feature seems like it would be a fairly lengthy beta in it's own.

You can do this on your own, as long as you don't mind the second machine's files being in a sub-directory on the first machine's share.

 

If the first machine has a share on disk1 named "images" then in that folder, (or anywhere under it) create a new empty directory to use as a mount-point.  (Change the "disk1" to be any disk you desire.)

 

The linux command is:

mkdir /mnt/disk1/images/images2

 

Then, you should be able to mount the remote share on that mount-point.

mount -t cifs //tower2/images /mnt/disk1/images/images2

 

Then, you should be able to see all the files under the one share for "tower"

 

I've not done this, but there is no reason it should not work...  You will need to un-mount the remote machine BEFORE you can shut-down the unRAID server.  If you try to shut down, you will see all your other drives that could be un-mounted as un-formatted... Do not panic, just log in via telnet, type

umount /mnt/disk1/images/images2

(Note... no "u" in umount)

 

Then press the stop button once more, the remaining disk1 can then be un-mounted since it will no longer be busy.

 

Then you can shut down or reboot.

 

Let us know if it works for you.

 

Joe L.

Thank you for the answer.  I've been thinking about it - I don't think it will work for what I need...but - I do appreciate the time you took to answer - thanks!

Thank you for the answer.  I've been thinking about it - I don't think it will work for what I need...but - I do appreciate the time you took to answer - thanks!

I know you want to have a single share accessible from PCs on your LAN.  Do you need a single directory with all your files? (Somehow I doubt all your files are in a single folder...but I've been wrong before)

 

There are a few assumptions I've made...

1. Access through one server to the next will be less efficient than if only accessing a single server, and you recognize that.

2. The second server's file-system will somehow need to be mounted on the first server to get the data to appear in one view

 

Can you share how you would like the data to appear in the consolidated view?  Perhaps there is a way...

 

Joe L.

 

JoeL,

 

for me it should be a transparent solution for the user/client.

 

I did the "mount way" first but this scares me a lot because of the nearly-heart-ache during every stop of the array. I always forget the mounted shares so the array always shows unformatted disks whenever I stop it. This solution would be transparent but the system doesn't allow a comforrtable way to archive it.

 

Currently I'm using Vistas softlinks to combine the shares in a local directory and use subst to make a letter out of it. As there is only a handful of clients left (and they are all mine) I can go that way. I know, this is not the wanted transparent solution but I don't get "unformatted disks" when I forget to unmount the next time.

 

What I did is to create a directory struct like the following:

 

c:

cd \

md unRAID

cd unRAID

md Data

mklink /d "c:\unRAID\Data\2009" "\\Tower2\Data\2009" (latest unraid box)

mklink /d "c:\unRAID\Data\2008" "\\Tower\Data\2008" (first unraid box)

subst q: C:\unRAID\Data

...

Now issue "dir q:".

 

As you see I usually have entry points (e.g. 2008) below the root directories of the shares (Data). So I can switch these entry points between unraid boxes to combine them within the main folders.

 

Drawbacks: You need to give the DOS console administrator rights to issue the mklink command. You need to issue 'rmdir /s /q "c:\unRAID\Data\2008"' on a local directory to remove the link. This one scares me too so I double-check and double-check before switching directories from one unraid server to another one. It's not transparent to the end-user (client).

 

Pros: Creating subdirectories and files within this struct works perfect. During next boot Windows recreates the links.

 

Regards

Harald

 

I was thinking about this on and off over the last couple of weeks, without knowing this thread was here.  I'd assumed - and this is purely theoretical, there's no fancy technical solution coming up - that you could in some way make unRaid scalable to suit this purpose.

 

Obviously not in its current format, but if, once you filled up your first box, you started a second (tower2, or whatever), and ran a third box that would act as a master.  This third box would treat the two unRaid boxes as singular - alberit external - drives, and would look upon it simply as an unRaid system but without parity.  This could potentially work up to a limit of - say 16 or 20 boxes - after which time you introduce another tiered level.  If you've got a silly amount of data, of course.

 

What I guess I mean is, you copy stuff to an unRaid master system (that could be an empty machine, or could double as one of the shares), and then configure it so that all of your other towers appear as singular drives in unRaid.  The unraid on these 'child' systems would handle the file allocations in the same way that they currently do, and the master would work out which machine to allocate the data to based on the existing share logic.

 

I appreciate that this is not without its problems at all, but if unRaid could mount a network drive in its share, it could be potentially viable.  A master unRaid system could control 16 unRaid systems and show up as one big share.  After a lot of work, obviously.  And 17 licences.  And it sort of sounds like quite a lot of work.....

Joe,

  I didn't realize you'd responded to my request.  Currently - we have two lime boxes we are trying to use to backup images.  We scan images for people, and our production server crashed once before which was painful enough for me to convince the bosses we couldn't afford not to have a backup.  The Lime's were a low cost solution.

 

  We currently have two "production" servers - one is a 12TB and another is 6TB.  Soon we will be consolidating into one new 25TB server.  My idea was to try to back everything up to one lime box.  At the time of this post, I was wanting one machine to back up everything up to one server.  We have one share, (jobs).  Inside this we have dozens of jobs and add new ones on a weekly basis.  A job could consist of 20GB to a few TB's.  I don't want to have to manually assign jobs to disks\servers, so the idea was to have one backup routine that used one share for the job.

 

  Unfortunately, I keep having problems.  I may be wrong on this, but I just don't think the lime file system was designed to have a 12TB share with 25,000,000 files.  On any given night, I could add or change 200GB of data that needs to be backed up. 

 

  At this point, I think we are going to be moving to a faster backup.  I am thinking I will use them to archive older jobs since we like to keep all the images we scan.  After the jobs are on the lime boxes, they will not change they will only be there for the occasional retrieval when something goes wrong and we need to pull images from a year ago.  If anybody has any ideas on how I could make the backup viable & fast - I am still interested in using them. 

 

Here is some stuff I've tried

 

1 - Used SyncBackPro so I don't have to scan the lime box, (destination backup is stored in a database).  Unfortunatly SyncBackPro clogs out the system @ about 15,000,000 files.

2 - Used a cache drive.  Helps - but not enough. 

3 - I have started replacing the 1TB Green Drives with faster 1.5TB drives.

4 - Upgraded to 4GB ram

   Unfortunately, I keep having problems.  I may be wrong on this, but I just don't think the lime file system was designed to have a 12TB share with 25,000,000 files.  On any given night, I could add or change 200GB of data that needs to be backed up. 

 

   At this point, I think we are going to be moving to a faster backup.  I am thinking I will use them to archive older jobs since we like to keep all the images we scan.  After the jobs are on the lime boxes, they will not change they will only be there for the occasional retrieval when something goes wrong and we need to pull images from a year ago.  If anybody has any ideas on how I could make the backup viable & fast - I am still interested in using them. 

 

What exactly is the problem?

If it's speed, then writing to a parity protected drive is going to be as fast as the parity protected drive.

In this case.

 

Here is some stuff I've tried

 

3 - I have started replacing the 1TB Green Drives with faster 1.5TB drives.

 

Don't skimp out on the parity drive, I replaced my 1TB green drive with a 1.5TB Seagate 32MB Cache drive and noticed the difference right away.

Also, one of the pitfalls I've noticed with ReiserFS. When you add a new file to a fairly full disk, it takes a long time to add the new directory entry because the disk and directories need to be searched. The only way to "assist" with this is constant directory scanning (with the one of the tools posted here).

 

Frankly, if you have a cache drive and want the fastest transfer possible,

consider the quality of drive you are using as a cache drive. where it is located on your bus and what type of bus you are using.

 

I would not say you need a 1.5TB drive as a cache drive, but at least a 7200RPM drive with w/32MB cache would help.

I've heard the WD 640's with 32MB cache are very fast.

 

After that you could consider a program called unison which synchronizes directories.

I believe unison keeps databases on both machines which speeds up synchronization.

There is also rsync which can synchronize and/or move files between directories and/or machines.

 

What exactly is the problem?

 

First there is the speed issue.  When backing up new files, it takes a LONG time.  Part of this is the nature of the beast when you are doing jillions of files.  I'm not much of a hardware person, but from what I understand RAID controllers have ram that help cache the files, then they stripe the information across 14 disks, so it's just faster.  Lime is slower because it writes to one disk AND writes to the parity disk.  On some level this is just going to be slower and you are going to have to live with it or cough up the money for a more expensive solution.

 

I have had a variety of problems, but here are some of the highlights I'm struggling with now.

 

1 - When deleting a large number of files, unraid eventually quits responding and the share is no longer available.  I have to restart my deletes.  It took me two days to clear out about 6TB's of data (kinda long story on why).  Several times I had to stop the array and restart it because it thought my disks were unformatted. 

2 - When it runs out of ram, copies get canceled and the shares disappear until it can catch up.  Sometimes massive copies can make it do this.  Luckily it seems to play nicer with rSync than Robocopy, and the Cache drive has helped tremendously.  UnRAID seems to do much better at moving data off of the cache disk onto a share than copying directly to a share.

3 - When browsing a directory on a share it can take 15 - 20 seconds to open.  Perhaps related is scanning the directories.  If I run a dir against my production machine it is, boy I don't know, 8 - 10 times as fast as my unRaid box?

4 - I am a little bit concerned about how long it would take if I did have a failure on my production box.  Will it struggle with copying files off like it does deleting files and in some cases copying them?

 

I guess I assumed that what was slowing it down was the billions of files and thousands of directories.  Anytime it does a copy to a share, I assume it has to check all 14 disks to make sure the file doesn't exist, then it has to write to parity and the disk.  After using a cache drive, it is still fairly slow, (8-9mbs) but MUCH faster than it was, so I'm guessing it was the parity slowing it down.  The strange part - and I can do some testing to confirm this, but it seems to like writing off of the cache drive much better than from a robocopy.

 

How does the filesystem work anyway?  Does it have a master table with the disks\files for each share?  There isn't any way this could be sped up by being stored in memory or on a SSD is there?

 

Anyway - those are just some random thoughts.  I do have some pretty fast cache drives, (500GB Barracuda's) and the copy isn't bad at about 9mbs.  I thought I was upgrading the parity with 7200rpm 1.5TB's, but they wound up being 5900RPM's.  I will be switching those out early next week and we'll see if that bumps my performance.  Overall I do like unRaid and think that in a less demanding environment it would be awesome.  I've about decided on it for storage on a home theater project I will start at some point.  But, it just seems like I'm always "massaging" my backups.  Thanks for the tip - I will check out unison.

Lime is slower because it writes to one disk AND writes to the parity disk.  On some level this is just going to be slower and you are going to have to live with it or cough up the money for a more expensive solution.

Actually, that statement is not accurate at all.

 

First, ALL raid arrays that involve parity MUST write to the parity drive whenever any writes to a data drive occur.  This is not unique to unRAID.

 

The HUGE difference is that on most RAID implementations,  whenever a "write" to a disk occurs, first the data is split into X number of pieces, one per data disk involved.  Then parity is calculated on those pieces.  This is all done in memory and no disk access is needed at all.  Then, that set of pieces representing what is known as a "stripe" of data is written in parallel to X+1 number of drives. (the +1 is the parity drive)  We have X+1 writes in parallel.  Unless limited by the PCI/PCIe bus, this is very fast.

 

This process repeats as the next "write" to a data disk occurs.  Notice that all the writes are done in parallel, and there is no need to know what was on the "stripe" disks beforehand, as it is written in its entirety.    It forces all the disks to be identical in size, since 1/X of the data is stored on each.   

 

You can therefore "write" to disks as fast as you can calculate parity and physically write to the disks in parallel over the PCI/PCIe bus on most normal raid protected arrays.

 

In unRAID, to allow each disk to be an independent file system, writes are made only to a single data disk, with parity to another dedicated disk.  To "write" those, unRAID must first know what was on a given data disk in the same "stripe" position. since it is not writing to all the disks it has no way to know what they contain. 

 

Fortunately, it does not need to know what is on all the disks, just the one being written, and the same sector on the parity disk.   It must first read those and then, it must do some exclusive -or operations on the existing data, the existing parity, and the new data being written.  This is very fast and takes very little CPU.

 

Finally, it must then write the block on the data disk it had just read, and also write the block on the parity disk it had just read. It must make two write operations.   To do those writes, the physical disk must spin around at least once to get the disk head over the same sector on the disk it just finished reading.   That is the reason for the slowdown in unRAID. It must wait for the disk to spin at least one full rotation for each sector written after first reading it.

 

unRAID will always be limited by the rotational speed of the disks.   Notice we are not limited by bandwidth of the BUS, as only two reads, or two writes are occurring (in most cases) and they are performed in parallel.  The eventual speed of a "write" to a disk in the protected array is dictated by the slowest rotating drive involved.

 

SO... to get an unRAID array to write faster, install 7200 RPM disks instead of 5400 RPM disks.  Remember, it must wait at least one revolution to be able to write a sector a disk just finished reading... no way around it.

 

If you want raw writing speed, get hardware raid, and normal raid5.  But, you will need to keep all drives spinning when reading and writing data...  unRAID is not the solution for you.

 

If you can trade off the raw speed of writing to the array, or can "write" to the cache drive at full speed, and later have it transfer its contents at the slower speed to the protected array, unRAID is a perfect solution.  It was never intended for performance... it can't, it has to wait between  a read and a subsequent write of the same sector for a complete revolution of a disk platter to occur. 

 

The difference in writing speed has nothing to do with writing parity.  all raid arrays do that.  It has to do with waiting for a disk to spin a platter at least one full revolution to get the same sector under the read/write head a second time. 

 

Joe L.

Excellent analysis Joe.

I think Tom should write on the homepage that

"It is recommended that you use 7200 or 10000 rpm disks in order to achieve the highest possible speed in an unraid box"

After using a cache drive, it is still fairly slow, (8-9mbs) but MUCH faster than it was, so I'm guessing it was the parity slowing it down.

 

I get 8-12mb/s using rsync directly to the disk. Perhaps the whole user share or rotational speed is coming into play.

 

How much RAM is in the machine?

 

 

 

What exactly is the problem?

 

First there is the speed issue.  When backing up new files, it takes a LONG time.  Part of this is the nature of the beast when you are doing jillions of files.  I'm not much of a hardware person, but from what I understand RAID controllers have ram that help cache the files, then they stripe the information across 14 disks, so it's just faster.  Lime is slower because...

 

"some" raid controllers have cache ram (mostly for read). The higher end ones with battery backup will have write caching.

Lime is slower.... Because it is not striped, your hard drives are individual file systems accessed through a virtual file system.

It's meant for write a little, read allot.

The cache drive should help a great deal.

Is your rsync direct or over ssh. (ssh will be yet another protocol overhead).

 

 

1 - When deleting a large number of files, unraid eventually quits responding and the share is no longer available.  I have to restart my deletes.  It took me two days to clear out about 6TB's of data (kinda long story on why).  Several times I had to stop the array and restart it because it thought my disks were unformatted. 

 

The nature of this problem is reiser filesystem overhead. The first write of any filesystem that has been spun down will take a long time.

Then the removing command has to traverse the directory tree to remove the files selected.

What may be happening is the split level and spun down disks are causing all disks to be spun up, then read(a long time), then the writes (for remove) occur.

You should not have to stop and restart the array during this operation.

Somewhere the command connection timed out, when you stopped the array, the command (or SAMBA) was still busy on the disk causing it to report unformatted.  It does take time to traverse a large directory structure.

 

2 - When it runs out of ram, copies get canceled and the shares disappear until it can catch up.  Sometimes massive copies can make it do this.  Luckily it seems to play nicer with rSync than Robocopy, and the Cache drive has helped tremendously.  UnRAID seems to do much better at moving data off of the cache disk onto a share than copying directly to a share.

You should not be running out of ram. Unless perhaps your syslog is growing too large too fast.

Check it. Run a top in a telnet to see if it is really running out of ram.

I would look more into something timing out because the machine is busy.

 

 

3 - When browsing a directory on a share it can take 15 - 20 seconds to open.  Perhaps related is scanning the directories.  If I run a dir against my production machine it is, boy I don't know, 8 - 10 times as fast as my unRaid box?

 

Different architecture, 14 separate disks, which spin down and/or the directory information gets flushed.

If you want something to perform like your production box, you should build another like that.

You could "try" Joe's cache_dirs script, then again, you have so many files, I'm not sure it would help.

Directory information does not get cached forever, there is a finite size and time limit. This is the nature of Linux.

 

4 - I am a little bit concerned about how long it would take if I did have a failure on my production box.  Will it struggle with copying files off like it does deleting files and in some cases copying them?

 

If you set up an rsync server on unRAID, you would be bypassing a machine in the middle.

If you are using a Windows machine to access unRAID and copying to the production server, then the data gets copied twice.

Learn and test how to use rsync.

I've moved terabytes with it. It takes a while, however it can go directly from one machine to another if done properly.

 

I guess I assumed that what was slowing it down was the billions of files and thousands of directories.

 

How does the filesystem work anyway?  Does it have a master table with the disks\files for each share?  There isn't any way this could be sped up by being stored in memory or on a SSD is there?

 

Read up on reiserfs, an SSD will not help, more ram may help a little, Joe's cache_dirs command may help.

Make sure you spin up all disks before major search, backup, restore operations, that should save you some time.

 

Anyway - those are just some random thoughts.  I do have some pretty fast cache drives, (500GB Barracuda's) and the copy isn't bad at about 9mbs.  I thought I was upgrading the parity with 7200rpm 1.5TB's, but they wound up being 5900RPM's.  I will be switching those out early next week and we'll see if that bumps my performance. 

 

This wont help much unless you are doing direct writes to the array. Then the 7200 32MB cache will help allot.

If you want high performance on the other drives, put high performance drives in place.

The green drives are great archive drives, but if speed is what you need, then those are not the correct drives.

 

 

What I would do in this case.

Change my spin down timer to 2-4 hours.

Write a cron job to spin up all drives before any search, restore, backup operation.

scan the whole filesystem (use joe's cache_dirs utility)

 

Then see how it performs.

 

on my busy rtorrent system, whenever I write to the drives that have movies (which have spun down).

There is a long delay and a very busy drive light as it spins up, then searches the whole directory tree for free blocks (I suppose), then the write begins.

I am NOT writing to user shares, I would expect this situation to be magnified if I were writing to a user share with all drives spun down and all directory information out of the cache.

 

 

 

Thanks for the ideas.  I did get RSync working, and it is still running.  I am in the process of replacing my HD's. 

 

To answer your questions - the syslog is not growing - it is staying fairly small, (<100K).  I have 3GB ram in one box & 4 in the other.  Since I am working on catching up with Rsync & rebuilding drives I am leaving the disks spun up all the time.  For whatever reason, Rsync is working much better than robocopy.  It ran all weekend without a problem.