coppit

Let's say I have a layout like this: //disk1/Movies //disk2/TV Shows and a read/write user share that hides the details of which directories are on which disk, like this: //Share/Movies //Share/TV Shows When I write new files into Movies or TV Shows, I want unraid to put them first on disk1 or disk2, where those directories already exist. Only when a disk is full do I want it to create a new directory on a different disk. And only when that new directory is full should it create a similar directory on a third disk. This kind of allocation method helps minimize spin up of lots of disks if I'm browsing movies and not TV shows. I don't think I can do this with any of the allocation methods we have now, can I?

Hi everyone, I tried setting up rsync for automatic backups of my Mac prior to Leopard's Time Machine. Here are my notes: Setup rsync on the server: 1. In /boot/config/go, add: rsync --daemon --config=/boot/config/rsyncd.conf 2. Edit /boot/config/rsyncd.conf: gid = root uid = root log file = /var/log/resyncd.log [backups] path = /mnt/disk1/Backups comment = read only = no list = yes Also, I ported a C program to do exponential backup rotation to bash, since I didn't have an easy way to compile it for unraid: #!/bin/sh VERSION=2.20 # This is a bash version of John C. Bowman's C++ program powershift, # distributed with rlbackup. There are two functional differences: This # version includes the delimiter with the directory or file base, and this # version does not retain the access and modification times of rotated files # and directories. The original C++ code is copyright John C. Bowman. This # code is copyright David Coppit. # Syntax: powershift directory_or_file_base [n] # # Generate a geometric progression of backup files or directories by # retaining, after the first n (>= 2) backups, only every every second backup, # then every fourth backup, and so on, for each successive power of 2 (except # for intermediate files necessary to accomplish this). # # Path are constructed from the first argument concatenated with a number # generated by this program. New backups should be placed into the rotation as # a concatenated file ending with "0". # # EXAMPLE #1: DIRECTORIES # # For the default value of n=2, successive applications of # # mkdir -p dir/0; powershift 'dir/' # # will shift dir/i to dir/i+1 for i=0,1,2,... while retaining (along with # necessary intermediate files) only the sequence # # dir/1 dir/2 dir/4 dir/8 dir/16 dir/32 dir/64... # # EXAMPLE #2: FILES # # The command # # echo > name.log0; powershift name.log 4 # # will retain (disregarding intermediate files) # # name.log1 name.log2 name.log3 name.log4 name.log6 name.log10 name.log18 # name.log34 name.log66... # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License version 2 as published by # the Free Software Foundation. # # This program is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., 675 Mass # Ave, Cambridge, MA 02139, USA. function stagger() { let RESULT="$1 + ( $1 >> 1 )"; } function file_name() { local SUM; let SUM=$1+$OFFSET; RESULT="$FILESTEM$SUM"; } # First arg is the message # Second arg is a file path relative to the current directory function fatal_error() { local MESSAGE=$1; if (( $# == 2 )); then MESSAGE="$MESSAGE "`pwd`; if [ -n $2 ]; then MESSAGE="$MESSAGE/"; fi MESSAGE="$MESSAGE$2"; fi echo "$MESSAGE: $?" 1>&2; exit 1; } function recursive_delete() { if ! rm -rf $1 2>/dev/null; then fatal_error "Cannot remove" $1; fi; } # The original C++ version fixed up the modified time so that it didn't change # after the rename. We can't do that in bash, unfortunately function rename() { mv $1 $2; } function power() { local I=$1; local STAGGER_I_FILENAME; local N; local N_FILENAME; local I_FILENAME; stagger $I; file_name $RESULT; STAGGER_I_FILENAME=$RESULT; if [ -e $STAGGER_I_FILENAME ]; then let "SHIFTED=$1 << 1"; power $SHIFTED; N=$RESULT; let "I=$N >> 1"; stagger $I; file_name $RESULT; STAGGER_I_FILENAME=$RESULT; if [ -e $STAGGER_I_FILENAME ]; then file_name $N; N_FILENAME=$RESULT; rename $STAGGER_I_FILENAME $N_FILENAME; fi file_name $I; recursive_delete $RESULT; else file_name $I; I_FILENAME=$RESULT; if [ -e $I_FILENAME ]; then rename $I_FILENAME $STAGGER_I_FILENAME; fi fi RESULT=$I; } function powershift() { FILESTEM=$1; if (( $# > 1 )); then N=$2; else N=2; fi if (( $N < 2 )); then echo "n must be >= 2" 1>&2; exit 1; fi let OFFSET=$N-2; file_name -$OFFSET; OFFSET_FILENAME=$RESULT; if [ -e $OFFSET_FILENAME ]; then touch $OFFSET_FILENAME; # Timestamp the latest backup; power 2; for (( I=2; I >= -$OFFSET; I-- )); do file_name $I; I_FILENAME=$RESULT; let "IPLUSONE=$I+1"; file_name $IPLUSONE; IPLUSONE_FILENAME=$RESULT; if [ -e $I_FILENAME ]; then rename $I_FILENAME $IPLUSONE_FILENAME; fi done fi } if (( $# == 0 )) || (( $# > 2 )); then echo "Usage: $0 directory [n]" 1>&2; exit 1; fi; powershift $@ I hope this helps someone. I got it working except I was having errors during the rsync, which I reported. I didn't follow up because I went with Time Machine because it "just works".

I use the hard drive on the HTPC to save HighDef streams. Does anyone send the video over the network to the unraid box? Does it work well?

Bill, You're right about the initial backup. But incrementals are very fast with rsync. It computes diffs and can send them compressed. I've been holding off on posting my code and HOWTO document because of two things: (1) rsh on unRAID still doesn't work, (2) there's a bug in rsync regarding hard links. This means my current solution requires you to telnet into the unRAID server to manually run some commands, and the backups take up a lot of space and time because they are not incremental. David

Hopefully it isn't some script registering dummy users for a spam attack.

I'll write up a blog post that explains how you can get space-optimized snapshot backups using hard links. I'm waiting on sshd or rshd support so that I can automate the "mv" command from the client machine. I've written some scripts to help with the backup, including one that rotates the backups using powers-of-2 retention.

But, again, how can you possibly give the memory and CPU more data than it can handle? On writes you're pulling from multiple drives, so your I/O bus is going to be busy. Even when you get the bits in memory, doing a simple XOR on them takes no effort. Then the data is never looked at again. With DMA reads don't even involve the CPU. By the way, I agree with you that a more real-world metric would be useful. Unfortunately I've found that a million things factor into hiccups, not just speed. But even single-channel memory is much much faster than your network could ever be. Here are some numbers from the net: DDR2 400: 6400 MB/s PCIe: 250 MB/s SATA II: 300 MB/s Gig-Ethernet: 125 MB/s We would need to be doing something pretty heavy-duty like encryption to tax the main memory. Simple parity computations aren't going to do it. Like I said, I suspect that any buffering problems will be from the I/O bus and disks not being able to fill the buffer fast enough, not main memory being too slow. All that said, I'm looking forward to your experiments. I hope you prove me wrong. I'd much rather upgrade my memory than my motherboard and network.

I'm having a hard time imagining how an unRAID server could be memory bound. It's disk-intensive, and disk I/O is a few orders of magnitude slower than memory. I'm guessing that a faster bus would help the most.

Yeah... I VNC into a windows box, open a terminal, type "ping tower" and use the reported IP address on my Mac. If you have a decent router (e.g. Netgear, but not Linksys) you can assign the IP so that it won't change. Then in your Mac's /etc/hosts you can set the IP to "storage" so that you don't have to remember it.

But for things like parity checks the network isn't involved. In this case I believe PCI is the #1 bottleneck. I read that in practice you only get 50% of the theoretical limit. I think that this is why people with PCIe are seeing so much better performance. Now, whether PCIe removes the bus as the #1 bottleneck I don't know. Perhaps some people with PCIe can do a few memory/CPU tests for us?

I agree on the price, open-ness, and capacity benefits of unRAID. But lots of people would pay a premium for ease of use, UI, and minimal hassles. My motherboard's NIC died, for example. If this were a Drobo box I'd send it back, and you can bet that V2 of their box would not have chosen the wrong component like I did. For a long time people had to pay extra for Macs, and were happy to do so.

Mine died a couple of weeks ago too. I'm limping along with a 100MB PCI NIC for now. I'm thinking about upgrading my motherboard to one with PCIx. Can anyone recommend a good upgrade board with PCIx and on-board Gigabit ethernet?

I thought this device from Drobo was interesting in that it seems to follow the same non-RAID strategy as unRAID. It also has "pooled virtualized storage" which sounds a lot like unRAID's user shares.

I used to keep lots of copies of pictures and such for backups. Now I don't, so there's no need for syncing. Now I just do syncs for backups and music on my laptop. I've found, believe it or not, Microsoft's SyncToy to be the fastest and most cross-platform solution. I export my Mac drive via Samba and sync using network paths.

I built a server that uses an Asus P5PE-VM. My performance numbers aren't as good as the ones cited here. I think the problems might be related to PCI versus PCI-e. I heard that a gigabit NIC can saturate PCI, resulting in a little better than 1/2 of full network performance. This would certainly coincide with what I'm seeing. So maybe the moral is get a m/b with PCI-e?

Here's my unRAID server setup: CPU: Intel Celeron D 331 Prescott 2.66GHz Case: Raidmax Ares ATX-801WBP MB: Asus P5PE-VM Memory: 1 GB PC 3200 (1 stick) PCI SATA II Controller: Promise SATA300 TX4 PCI SATA II USB Stick: Sony Micro Vault Tiny 512MB Flash Drive Parity Drive: WD5000AAKS 500GB SATA II (on Promise controller) Data Drives: Two Maxtor 6V300F0 300GB SATA II (on Promise controller), One Western Digital WD2500JB 250GB drive on motherboard PATA controller The system probably cost $320 from Newegg (I had some parts already). I'm shipping data across gigabit ethernet from an Athlon 64 3500, Abit AV8 machine across 2 Netgear Gigabit switches to my HTPC. My 64K results: SATA II READSATA II WRITEIDE READIDE WRITE 32MB33.91331.43637.87929.381 64MB35.00833.13737.03025.420 128MB35.57326.57838.86325.832 256MB36.47525.21141.69025.724 512MB42.17125.49644.17325.488 1GB8.35616.2918.29016.930 As with others, I'm seeing faster writes for some reason...

Thanks for the reply. Is there any risk that writing to two drives will overload the PCI bus? If so does it make sense to put the parity drive on the motherboard? (I think that controller runs straight to the southbridge rather than the PCI bus.) I tried informally putting the parity drive on the MB controller and it didn't seem to make a big difference.

Let's say I have a SATA II 4-port IDE controller, a SATA I 4-port PCI IDE controller, an on-board 2-port SATA controller. If I have 4 SATA II disks, do I want to put them all on the SATA II controller, or is it better to spread them across the different controllers? Is the on-board SATA controller free of IDE contention? i.e. Would it be better to put the parity drive on the on-board controller, separate from the other data drives? If I have to I'll do some experiments, but I was hoping someone else with more hardware knowledge than me would already know the right answer.

To answer my own questions (please correct me if I'm wrong): All the RAID is implemented in software, which requires some CPU to execute lots of XORs compared to hardware RAID solutions. Plus 2.5 GHz CPUs are about $60, so why not? We can get by with 512MB. Two sticks of 512MB for DDR is really not necessary. (Right?) Some people have had success writing two HDTV streams at once while reading another. Try it and see. I ended up grabbing all the parts (minus drives) from Newegg for a few hundred. Not too expensive...

So did you do it? Any problems?

Hi all, I'm considering building an unRaid server and have some questions: I don't understand why a 2+ GHz CPU is recommended. I would think that an I/O intensive system like this could get by with < 1 GHz CPU? Ditto for memory. Why is more than 512MB needed? I'm was hoping to stream 2 HDTV recordings from SageTV to it, but people on the SageTV forums are saying that I'll get dropped frames. Is there any hope for better write performance in the future? (Is it just a matter of waiting for faster disks, or making sure I buy fast disks now?) BTW, I plan on using gigabit ethernet between the machines. Has anyone found a suitable pre-built OEM system, rather than assembling it themselves? I was hoping for a Walmart or Newegg machine that would be cheaper than buying all the parts separately.