gundamguy

Agree ==> HOWEVER, the yellow triangle is NOT an "error" condition. It simply means "Some or all files are on unprotected storage." In the case of a cache-only share being stored on an unprotected cache drive that is the normal condition ... NOT an error. The user should certainly KNOW if he's storing data on a cache-only share ... and this indicator simply reminds him that the share is unprotected. Don't like looking at it? ... update the cache to a pool so it's protected I hear you and agree files on the cache are not inherently an error condition. However most of us have been conditioned to see yellow triangles as signifiers of that there is an non-critical issue that should be looked at. If this is normal behavior and performing as intended I don't understand why it's shown with a symbol that we are conditioned to believe signifies that something non-critical needs investigating. Right now it basically says "Caution! Everything is working properly!" What I would like to see is a better system which only shows a warning symbol if something went wrong, like mover failed or files were not moved properly according to the mover schedule. Perhaps even something that said the time of the last move, and time till next move? I don't know it seems like something that can be improved to give users better information so they know if they need to act on something or not.

jonathanm just summerized my thoughts right as I was about to post. I agree. If things are working as intended it shouldn't really be showing an "error" condition IMO.

I think there is a plugin for crashplan floating out there some where (try searching the plugin design forum), but I and some other people here used the following tutorial to create custom scripts that can run a daily (or Hourly, Weekly, Monthly, etc.) backup.

Hmm... Maybe I'm missing something. I was not planning on rebuilding a drive from parity. My thought was to empty out one drive by moving it's contents to another existing drive, remove the empty drive and replace it with a larger drive, pre-clear the new larger drive, format it with XFS, and then move data to it from another existing drive. Once that is done, I would continue a similar process until the data has been migrated to drives formatted with XFS. In particular here is the layout: Current state: Future State: Parity: 2 TB (keep drive) Parity: 2 TB Disk1: 1 TB RFS (60% full) (replace drive) Disk1: 2 TB XFS Disk2: 1 TB RFS (40% full) (replace drive) Disk2: 2 TB XFS Disk3: 2 TB RFS (20% full) (keep drive) Disk3: 2 TB XFS My thought process is as follows: Move all data from Disk1 to Disk3, remove current Disk1. Install replacement for Disk 1 Move all data from Disk2 to new Disk 1, remove Disk2. Install replacement for Disk 2 Move all data from Disk3 to new Disk 2, re-format Disk3. Move some data from Disks 1 and 2 to Disk 3 to even out the disks somewhat. Does that make sense? Thanks, John Your process does make sense, and will work, but as Trurl explained when you remove a data disk from the array you are going to invalidate parity and be prompted to rebuild your parity. So you might have more downtime as you do parity rebuilds during this process. Also since you plan to replace two disks that means you'll be prompted to rebuild parity twice. Not sure if there is a better way to sequence this, but you can view this as two seperate tasks, 1) Converting from RFS to XFS 2) Replacing to hard drives. If I were trying to do what you want to do, I would first replace the drives and then convert from XFS to RFS, but you can sequence this as well so that you replace drives and convert in alternating steps. You can run without a parity disk for awhile and still have access to your data on array data drives, if you want to. I'm not sure that pre-clearning has much value for you here since the disk has already been stress tested before. I'm not an expert on pre-clearning but I would think that you should be fine just re-formating it. However again I am not the best on pre-clearing so there might be some benifit I am missing here.

Greetings, I've been reading and re-reading this thread over the last couple of weeks. I have two questions: 1) Before I can add a new, larger drive, I need to move data around so that I can remove a drive. I figure it's about about 600 GB of data. Would it be best to shut down the array to move that data? I assume in that case I'm safe to just telnet to the unRAID server and use the following rsync commands to move the data into the existing hierarchy, without having to create a temporary subdirectory that isn't part of the user share, correct? rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ At that point, I should be able to start following the guide, I think. 2) Where does pre-clear of the new drive get done? Is it before the steps in bjp999's guide? It is mentioned, but it is above the steps, so i wasn't sure. 3) Also - in the steps above, as the array is running, are you at risk of duplicate files, and confusing things between steps 10 and 11? Thanks, John You've got to different issues, replacing a disk with a bigger disk, and converting your file systems. The thing to keep in mind is that if you rebuild from parity your array will rebuild that disks with the current filesystem, you can't switch file systems and rebuild at the same time. So what you will need to do is either convert the disk to XFS first then swap for the bigger disk, or swap for the bigger disk then convert to XFS. If you've got space issues I think the second path makes more sense... Answers to questions: 1) If you've got space on the destination disk for the files on the source disk that should work just fine. 2) This depends on how you want to / are able to do this process. You might be better off, pre-clearning your disk, replacing the smaller disk, having it rebuild, then start the conversion process to XFS. In that case you'll have wanted to pre-clear before moving any data. 3) You should be fine at step 10 / 11 since you will have already formated the disk you moved the data off of.

I don't know for sure, but I do know that trailing / are important to rsync. Perhaps it's a missmatch issue, where you did /mnt/disk3/ the first time and /mnt/disk3 the second time. I'd run it again like rsync -nrcv /mnt/disk3/ /mnt/disk14/t >/boot/verify_disk14.txt see if that comes back with an empty list (since the files are already in disk14/t) Thanks, I will try that. EDIT:It worked, the file ended up containing: sending incremental file list sent 382,411 bytes received 813 bytes 6.83 bytes/sec total size is 3,928,060,848,066 speedup is 10,250,038.75 (DRY RUN) That is great news. Glad to hear it!

I don't know for sure, but I do know that trailing / are important to rsync. Perhaps it's a missmatch issue, where you did /mnt/disk3/ the first time and /mnt/disk3 the second time. I'd run it again like rsync -nrcv /mnt/disk3/ /mnt/disk14/t >/boot/verify_disk14.txt see if that comes back with an empty list (since the files are already in disk14/t)

I think (not tested the -R use of ls before myself) you can check if the files are actually still there using the following command. ls -R /mnt/disk4 ls lists the contents of a directory, the -R option does it recursively, and /mnt/disk4 is the disk I assume you want to check, if it's not disk 4 substitute for the disk number it is... I suspect this will only return the directories you had but no files. As to why Windows would think your files are still on disk4... well it doesn't. When trurl asked you to v /mnt/user the results that you posted show that you have a user share "mnt/user/disk4" that appears to be a disk share but is actually a user share. Because disk4 is actually a user share and user shares aggregate across all disks (unless the share is set to not do that) it's showing the data on the other disks when you look at the disk4 folder in Windows. Unless creating a user share named disk4 was intentional I suggest you change that.

Additional thought, are you looking at a disk share or a user share? If you are looking at a disk share the share should be called diskX. User shares by default aggragate files and directories from accross all the disks in the array (unless set differently) so even though you moved files from DiskX to DiskY they are on DiskY so they show up in the user share. If you look at the DiskX share (by sharing that disk under the disk settings) it only shows what's on DiskX and should show empty folders. If this isn't it, and those files really are still there I am really confused by what's going on, though I don't know that it's a huge problem.

Did you dig down below the top level into folders where the data you wanted to copy was stored? I think --remove-source does not delete folders just files, so you should have a bunch of empty folders left over. I suspect that it only appeard that the data was still there because the (now empty) folders were still there. If that's not it we can try something else to verify.

I think in that cause you have to manually split and copy the directories over. I do not think there is anyway to specify to rsync to only copy a set amount of data.

On rsync you need to include the -H option, or else it won't do anything about them... If you want more info about rsync -H you can find it here. Rsync Doc

Yes, it can be done with rsync but it's takes a bit of command line set up first.

How exctly is that accomplished, and would that remove one of the best features of unRAID, namely you can take a drive out of your unRAID array and plug it into a linux distro and get the data off should your hardware fail? I'm just having nightmares again about the day my PS3 died. You might not know this but the PS3 has a built in hard drive which is replacable by individuals. So clearly when the PS3 hardware fails you just pull out the PS3 hard drive plug it into a new PS3 and get on with your life right.... WRONG, each hard drive was individually formated with an onboard chip to only be readable by that particluar PS3... so when the hardware fails even though the hard drive is good and the data is there you can't recover it... (Good job Sony!) Anyway, I'm only mentioning this becuase this is a concern of mine, not becuase I beleive that this concern will be realized.

I believe the way that user shares work this isn't truely an issue. If you copy disk1 to disk16 data that was stored under /mnt/disk1/Movies and is now at /mnt/disk16/Movies will still show up under /mnt/user/Movies. I think this is true even if say under Movies share settings you exclude disk16. The exclude include options are for controlling what disks data is writen to, and doesn't affect the propagation of that data to /user/share or at least that is my understanding. That said if you are using the exlcude / include options for your shares, you will need to go back and change which disk you are excluding / including to ensure that any new data added to the array goes to the right disk.

I'm not 100% sure why rsync is having trouble, but here's a few things you can try. You can add -n to do a dry run this will tell you what needs to be transfered still. You can also make rsync more verbose which should help you troubleshoot it. -vvv will give you more information (too much IMO, but might help you figure out what is causing rsync to hang)

Admittedly we learn from these user errors and hopefully modify the GUI to help prevent people from making the same mistake. You can't 100% prevent it, but maybe adding some warning text or adding a check-box or something will make it harder for users to mess it up. As it stands now it's way to easy to mess up.

This is a drive that you've copied teh data from and are ready to format correct. If not do not format. Here is what I did on 6b12. 1) Verify that your disk is empty and make sure you know which disk you want to format. 2) Stop the array 3) On list of disks tab, click on the disk name (Disk1, Disk 2, etc..) (Make sure you picked the right one.) 4) On that Disk's settings page there should be a drop down for formating, select xfs 5) Start the array 6) Hit format (Which will appear below where you start the array) 7) Wait a bit, and it should be formated and part of the array.

I used rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ to convert my disks to XFS and one of them red balled during rsync, in my case it corrupted one file. That is intresting behavior that I would not have expected. I am suprised it kept a transfered file that it didn't verify, which the documentation suggested would not happen. I think the best course of action is to run two passes then. rsync -av --progress /mnt/diskX/ /mnt/diskY/ This should transfer all the data. rsync -avc --progress -remove-source-files /mnt/diskX/ /mnt/diskY/ This second pass should not transfer any data, but it'll check the checksums of files at both locations, and then delete the source file if they match. This will take longer and is more IO intensive since it has to generate checksums. Of coruse not you have the file duplication issue for a little while.

I never thought that corruption was possible. I would have thought rsync would be smart enough to know not to copy / remove the source file if that file was currently being written to. When I did this, I disabled mover and my backup program while I was converting. I just didn't want any files to be possibly fragmented to death by multiple writes to the drive at the same time. I would have thought so too, but the documentation says to be warry of this. This might be a disqualifier and why you shouldn't use rsync as I originally suggested. Perhaps a two step process is in order. First rsync -avP /mnt/diskX/ /mnt/diskY/ then rsync -avP --remove-source-files /mnt/diskX/ /mnt/diskY/ You could also throw in a checksum comparison on the second one if your concenred by sending rsync the -c attribute. Also I'm pretty sure that -P is the same as --progress.

I just finished my process of converting from rfs to xfs last night using your described process and rsync to move the data. I had no problems. I do want to say after a bit more reading about rsync I added to my prior post to add a note about not using --remove-source-files if there are active writes to the directories your moving.

After a bit of reading I think the answer is yes. Quotes from the Rsync Documentation. Under the -c (use Checksum) it says. And under --remove-source-files it says. So it seems that corruption would set off a redflag and the source file will not be deleted, unless I am missing something.

I just started this process last night. I want to say thanks for the intro in how to make this change. With out your post I doubt I would have taken the effort to make the change to xfs. I asked a few questions in another thread and someone made a suggestion which I want to share because I believe it reduces the steps, and is a more powerful approach. To my knowledge the following code represents a simplified method to accomplish steps 6, 7 & 8 in one pass while also maintaining permissions, timestamps, owners, groups, symlinks, and device files using rsync. rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ Where disk X is the rfs disk you are copying to the newly formated xfs disk (disk Y). For example I ran this last night on as /mnt/disk1/ /mnt/disk4/ copying disk1 to disk4 -a means it's in "archive mode" which equals -rlptgoD (Recursive, Symlinks, Permissions, Timestamps, Groups, Owners, Device Files) Note if you want to Preserve extended attributes add a capital X to your call of rsync. -v puts it in verbose mode (verbose is not required) --progress shows you the progress of the copies as files are moving (Progress is not required) --remove-source-files : After the copy is done rsync will verify and then remove the source files. IMPORTANT do not use --remove-source-files if you have files currently being writen to the disk, this will result in corruption of the that file. You should do this anyway since your moving data from one disk to the other, but disable any plugins - dockers which write to that disk, and it's best to not write to shares that could be put on that disk while rsync is running. The end result should be empty directories on diskX and a duplication of diskX on diskY. Rsync is very powerful and there are way more things that can be done with it, so if you have questions or want to know more about what it can and can't do you can read more here. I think this is a much more elegant solution.

Thanks again for the advice, I feel like I'm moving in the right direction now. I guess I need to figure out if I want to have a second unraid box or if I don't have enough data to warrent that yet and back up via an USB3 HDD attached outside of the array. Oddball question. Should you backup \flash ? In the event of a usbdrive failure I realize that a new key will be required (I have a backup of that anyway... because why not...) but is there any other settings or files which having a backup would help the process of getting up and running again? If not the whole flash are there important files we should be looking at backing up?

I want to say thank you so much for this! Really good stuff! The link is also awesome and set me on the right path. My only problem is that My wife and I live in a small apartment, (just starting out), so I'm not sure that I can convince her that a second backup server is what we need. She's onboard with the primary one. I could possibly put a remote server in at my parent’s house... but we'll have to see about that as well. Also I assume there are a few additional steps when performing back-ups across the internet?