February 23, 200818 yr ROBJ BINGO, you picked up on it... As I re-iterate, if the zero of the drive is only done on the parity drive (faked or not really executed on the data drive). then as it sweeps through the data drive block by block, it gets slowly removed from parity up until it reaches the end of the drive. Thereafter it is removed from the array totally. This eliminates a total rescan of the array as it's only a rescan of the drive to be removed. For those interested, this is what I use. http://www.newegg.com/Product/Product.aspx?Item=N82E16817182112 Frankly, if I'm transporting a drive or not, removing it from an array should be as described. remove it block by block out of parity (but do not really touch the drive).
February 23, 200818 yr Author I'm just thinking: There is an outstanding enhancement request to be able to expose a disk that is NOT part of the array via unRAID. Right now all data drives are always supposed to be protected by parity (or else unRAID will complain). Let's call that configuration "A". When that feature request mentioned by bjp999 is implemented, we can have data drives which are unprotected by the parity but still available on the network. Let's call that configuration "B". Now once unRAID supports configuration "B" it would make sense to offer ways to transform configuration "A" to "B" on the fly and also back from "B" to "A". There might be buttons like "exclude this drive from parity" (A -> B) and "include this drive to parity" (B -> A). These buttons would result in a full surface reading of the single data drive and updating of the parity. Both actions (A -> B, B -> A) could be done without ever losing parity protection for the other drives. This solution would happen to make both WeeboTech's and my original wish possible. What do you guys think?
February 23, 200818 yr You still need to be able to deal with the fact that the process of updating parity will probably take an hour or more. If the process is interrupted by anything (power failure, loose cable, drive error) then parity across the entire array is invalidated since you do not know where in the parity drive you need to use the extra drive and where to use the original set of drives to calculate parity. My idea of using a mirrored pair of drives that could be added or removed from the array in a single operation, since parity would not need to be calculated, has a huge of advantage... it is really fast. Seconds (if that) instead of hours. The cost is that of a second data drive for the mirrored data, a second enclosure, and a bit more weight to carry between the two locations. (I like the enclosure you linked to... inexpensive too) We would need to find a "mirror" driver for the work location, to keep the two drives data in exact sync with each other but I'm guessing that will not be too hard.... basically Raid-0. Joe L.
February 23, 200818 yr Joe - Although you are right, of course, that the RAID 1 (mirrored) pair could be added and removed from the array without impacting parity, it is also true that there are no benefits to having such a RAID pair in the array at all. All you are really doing is lowering the performance of that RAID pair by subjecting it to a meaningless parity calculation that will always cancel out. Wouldn't it be better to just be able to add a RAID 1 pair to the unRAID server as a separate sharable disk? You could move them to other servers without affecting unRAID at all. Your write performance would be much better and no lengthy parity calculations would be required. If you didn't care about the redundancy, you would just need a single (non-RAID) disk. Tom would need to implement the enhancement to allow a separate sharable disk to be added to the server. (I am curious, if you know what you're doing could you expose a separate disk via a telnet session today? Is the feature just a way to have this feature simply administered inside the Web GUI?) In WeboTech's request I read nothing about a desire to maintain a RAID pair - he just wanted to be able to eject some random disk from his array, take it with him somewhere else (where he would presumably WRITE to the disk), and then put it back in his array with no loss of parity protection. I don't think that's possible unless you follow MadShi's model. The way to accomplish this - and you could do this today, is to copy his data off the array to another drive, and then take that drive away. On return, remount the drive and copy the delta data back to the array. Madshi - Perhaps at some future time, Tom may have the ability to keep all that straight (knowing that a drive was "half added" to the array and that parity for the first half would be calculated WITH the other drive, and the parity on the other half would be calculated WITHOUT the other drive). I have no idea how hard that would be, and whether it would violate Tom's KISS model. Maybe 2 years from now when he's got 5000 users, he may want to add such a feature. In think the concensus opinion is that you could do a lot with the two basic features that I think we're all supporting - the ability to have a non-array disk exposed as a sharable disk (already on the list), and the ability to remove a disk from the array while maintaining parity (needs to be added to the list). -Brian
February 23, 200818 yr It is very easy to add and remove an external USB drive to be visible in user-shares. VERY IMPORTANT... This procedure to mount an NTFS drive to be visible through user-shares DOES NOT PROTECT THE CONTENTS OF THE NTFS DRIVE with your parity drive. The NTFS drive is not used for parity, nor is it part of the unRaid array protected by the parity drive. If you want the files to be protected, copy the files to one of your data drives for editing and then back to the USB drive when done to transport once the USB drive is unplugged With that warning out of the way. Here is how I just did it, without even re-booting the unRaid server, using an external USB enclosure on my array.. I have an external USB enclosure with a hard disk that has an NTFS file system. I use it for backups of some PCs, but let's say that is the data I want to move back and forth from home to work. We can actually use any file-system that unRaid understands, but FAT and FAT32 have file size limitations, and reiserfs does not to my knowledge have a read/write driver available for windows. That leaves NTFS. so... Get yourself an external USB enclosure, add a hard disk, format it as NTFS. In the past, Linux had partial support of NTFS file-systems. In fact, unRaid comes with the ntfs loadable module that can read an NTFS file system, unfortunately, it is not able to properly write to NTFS file systems. The read-only-capable drive is not too much help in this case, where I want to be able to move data to and from work, so we will NOT use the one Tom supplied. (Note to Tom... might want to include the new ntfs-3g module in an upcoming release... hint-hint ... please) Instead, we can use the newer NTFS-3G driver that has full NTFS capability and has been stable since February 2007. You can read about it here: http://www.ntfs-3g.org/ You can download the package you will need to install from here: http://www.linuxpackages.net/download.php?id=11248 The actual package you will download is : ntfs-3g-1.2216-i486-1McD.tgz I made a folder on my flash drive for packages and put it there. root@Tower:/~# ls -l /boot/packages total 224 -rwxr--r-- 1 root root 221030 Feb 23 15:26 ntfs-3g-1.2216-i486-1McD.tgz You must then install it to your unRaid box. installpkg /boot/packages/ntfs-3g-1.2216-i486-1McD.tgz You could add a line to the "go" script to do this install every time you reboot. Once it is installed, you must make a mount point (an empty directory) to mount the new drive. since I already had a user-share folder named "data" in my user-share file-system I made my mount point there. You can name the empty directory anything you want, I used "usb" (you only need to do this once) mkdir /mnt/user/data/usb Next step is to plug in the drive into a spare USB port on your unRaid server (do not unplug your flash drive), and mount the NTFS file system resident on the USB hard-disk on the mount point. Now, on my all IDE array, the USB hard-disk appears as /dev/sdb and /dev/sdb1 You will need to find the appropriate device on your system. If you type the following command before you plug in your USB drive, and after, you should be able to find the added drive partitions. cat /proc/partitions my results are here, the new USB drive is detected as type 8 (scsi) and assigned sdb and sdb1 ... root@Tower:~# cat partitions major minor #blocks name 8 0 1000944 sda 8 1 1000912 sda1 3 0 488386584 hda 3 1 488386552 hda1 3 64 488386584 hdb 3 65 488386552 hdb1 22 0 245117376 hdc 22 1 245117344 hdc1 22 64 245117376 hdd 22 65 245117344 hdd1 33 0 390711384 hde 33 1 390711352 hde1 33 64 390711384 hdf 33 65 390711352 hdf1 34 0 390711384 hdg 34 1 390711352 hdg1 34 64 488386584 hdh 34 65 488386552 hdh1 56 0 488386584 hdi 56 1 488386552 hdi1 9 1 488386552 md1 9 2 245117344 md2 9 3 245117344 md3 9 4 390711352 md4 9 5 390711352 md5 9 6 390711352 md6 9 7 488386552 md7 9 8 488386552 md8 8 16 245117376 sdb 8 17 245111706 sdb1 Now, sdb represents the entire raw drive... most of you will not want it, as it does not contain a file-system, but instead you will want the first partition, it will be the NTFS partition and it will be the one with the "1" at the end of its name. In my case, sdb1 If you try to use the wrong partition, the mount -t ntfs-3g command will fail, as it will not find an NTFS file-system. Now, all that is left to do is to mount the file system on the mount point. To do that type: mount -t ntfs-3g /dev/sdb1 /mnt/user/data/usb/ Now... You can see if the your NTFS file-system was mounted correctly... ls -l /mnt/user/data/usb -rwxrwxrwx 1 root root 61724 Jan 31 2007 APC.CSV* drwxrwxrwx 1 root root 0 Jan 31 2007 LaptopBackup/ drwxrwxrwx 1 root root 0 Apr 14 2005 BACKUP-IMAGES/ drwxrwxrwx 1 root root 0 Aug 16 2005 RECYCLER/ drwxrwxrwx 1 root root 0 Apr 23 2005 RRUbackups/ -rwxrwxrwx 1 root root 41906 Jan 31 2007 STC.CSV* drwxrwxrwx 1 root root 4096 Apr 4 2007 System\ Volume\ Information/ -rwxrwxrwx 2 root root 3726 Jan 31 2007 business_objects.CSV* -rwxrwxrwx 1 root root 56494 Jan 31 2007 ecam.CSV* -rwxrwxrwx 2 root root 780157952 Oct 26 15:23 email_backup.pst* -rwxrwxrwx 2 root root 101598208 Jan 31 2007 email_backup2.pst* drwxrwxrwx 1 root root 0 Mar 25 2005 grisoft\ antivirus/ -rwxrwxrwx 1 root root 19172769 Jan 31 2007 inbox.CSV* -rwxrwxrwx 2 root root 6755567 Jan 31 2007 sent\ items.CSV* -rwxrwxrwx 2 root root 853151 Jan 31 2007 sent-items1.CSV* -rwxrwxrwx 2 root root 3615993 Jan 31 2007 tmp-inbox.CSV* drwxrwxrwx 1 root root 0 Mar 25 2005 true-image-update350/ -rwxrwxrwx 1 root root 5767 Jan 31 2007 volt.CSV* The contents of the drive will also be visible from the lan, in the user-share folder you created the mount point. When you are done with the NTFS files in the USB enclosure YOU MUST unmount the drive before you can unplug it, otherwise, a file may only be partially written to it. To unmount the drive, first close all programs using it... including file-explorer on windows. then type: sync umount /dev/sdb1 Note: the command above is umount (without the first letter "n") not unmount. This should get you to where you can move a drive from home to work and add it to be visible on your LAN through user-shares. VERY IMPORTANT... This procedure to mount an NTFS drive to be visible through user-shares DOES NOT PROTECT THE CONTENTS OF THE NTFS DRIVE with your parity drive. The NTFS drive is not used for parity, nor is it part of the unRaid array protected by the parity drive. If you want the files to be protected, copy the files to one of your data drives for editing and then back to the USB drive when done to transport once the USB drive is unplugged. Have fun... Use a file that is not mission critical until you are comfortable with mounting and un-mounting the drive. Joe L. Notice... I did not need to reboot the unRaid server, I did not need to stop or re-start the array. Try that on MS-Windows where it seems that anything you do requires a reboot. ;D
February 23, 200818 yr You are a fountain of knowledge, kind sir! Would this also work the same way with an attached eSATA NTFS drive (except maybe for naming the mountpont something besides USB)? I could see a kind of buffering use for a drive like this - where you would write your data to the unprotected disk instead of directly to the array (so it would be fast) and a background process would move the files to the array during idle time. That way you could quickly disconnect the Windows machine. (This probably will be unnecessary with 4.3 comes out with the faster writes). Thanks again!
February 23, 200818 yr This should work with any drive, internal or external, as long as you can figure out the device partition that has the NTFS partition (/dev/???1) So, you can format an NTFS drive in your windows machine and then install it in the unRaid array... DO NOT ASSIGN IT TO YOUR ARRAY. IF YOU DO, IT WILL APPEAR AS UNFORMATTED AND WILL BE CLEARED when you start your array... not what you want... Leave it un-assigned. But if you just want a disk outside of the array on the share-drives, No need for an NTFS file system... partition an un-assigned disk, use mkreiserfs on it, then mount it with "mount -t reiserfs" on an empty folder on a shared drive disk. The need for NTFS occurs when trying to move data to/from other locations. This technique might be very useful in combination with the midnight-commander add-on when initially populating the unRaid server from existing NTFS formatted drives taken from other machines. When doing that, I would add the -o ro option to the mount command so it is mounted as read-only and no writes occur to it at all. That way, you could copy the data off the old drive to drives protected by the unRaid array. the mount command would then be: mount -o ro -t ntfs-3g /dev/sdb1 /mnt/user/data/usb/ Joe L.
February 27, 200818 yr What i have suggested elsewhere is a possibly quicker version, similar, except to skip the write to the drive being removed. On selecting the option to remove a data drive, unRAID will loop as long as there are blocks in the data drive, reading them and updating the corresponding blocks of the parity drive. Plus, only data blocks with binary ones require modifications to the parity drive. Once the loop reaches the end of blocks that have ever been written to, then no accesses of the parity drive will occur, and it should finish fast. Blocks that do require parity updating will require a single read from the data drive and a read then write to the parity drive, which should run fairly fast. In addition, you end up with an unassigned data drive that is still untouched, any data is still intact, which may or may not be useful. I don't see any useful reason to zero out the drive, plus it takes longer I was just re-reading this thread and found this from RobJ that I somehow missed. If I understand what you are suggesting, it sounds like you would drop the drive from the array and then use it to as the basis of an optimized parity update. That will work faster than zeroing out, and generally accomplish the same thing. It also has the benefit of keeping the data disk intact - a very useful thing! But there are still a couple of benefits for the zero out option: 1 - Parity is kept intact throughout the process, so the user can shutdown and restart the array IN THE MIDDLE of the operation (without rebuilding parity). For similar reasons, the array can operate during the process. I'm not sure how this would work in your optimized approach. 2. The zero out logic is already being done for format, and would therefore likely be easy to implement. -Brian
Archived
This topic is now archived and is closed to further replies.