gundamguy

I believe the way that user shares work this isn't truely an issue. If you copy disk1 to disk16 data that was stored under /mnt/disk1/Movies and is now at /mnt/disk16/Movies will still show up under /mnt/user/Movies. I think this is true even if say under Movies share settings you exclude disk16. The exclude include options are for controlling what disks data is writen to, and doesn't affect the propagation of that data to /user/share or at least that is my understanding. That said if you are using the exlcude / include options for your shares, you will need to go back and change which disk you are excluding / including to ensure that any new data added to the array goes to the right disk.

I'm not 100% sure why rsync is having trouble, but here's a few things you can try. You can add -n to do a dry run this will tell you what needs to be transfered still. You can also make rsync more verbose which should help you troubleshoot it. -vvv will give you more information (too much IMO, but might help you figure out what is causing rsync to hang)

Admittedly we learn from these user errors and hopefully modify the GUI to help prevent people from making the same mistake. You can't 100% prevent it, but maybe adding some warning text or adding a check-box or something will make it harder for users to mess it up. As it stands now it's way to easy to mess up.

This is a drive that you've copied teh data from and are ready to format correct. If not do not format. Here is what I did on 6b12. 1) Verify that your disk is empty and make sure you know which disk you want to format. 2) Stop the array 3) On list of disks tab, click on the disk name (Disk1, Disk 2, etc..) (Make sure you picked the right one.) 4) On that Disk's settings page there should be a drop down for formating, select xfs 5) Start the array 6) Hit format (Which will appear below where you start the array) 7) Wait a bit, and it should be formated and part of the array.

I used rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ to convert my disks to XFS and one of them red balled during rsync, in my case it corrupted one file. That is intresting behavior that I would not have expected. I am suprised it kept a transfered file that it didn't verify, which the documentation suggested would not happen. I think the best course of action is to run two passes then. rsync -av --progress /mnt/diskX/ /mnt/diskY/ This should transfer all the data. rsync -avc --progress -remove-source-files /mnt/diskX/ /mnt/diskY/ This second pass should not transfer any data, but it'll check the checksums of files at both locations, and then delete the source file if they match. This will take longer and is more IO intensive since it has to generate checksums. Of coruse not you have the file duplication issue for a little while.

I never thought that corruption was possible. I would have thought rsync would be smart enough to know not to copy / remove the source file if that file was currently being written to. When I did this, I disabled mover and my backup program while I was converting. I just didn't want any files to be possibly fragmented to death by multiple writes to the drive at the same time. I would have thought so too, but the documentation says to be warry of this. This might be a disqualifier and why you shouldn't use rsync as I originally suggested. Perhaps a two step process is in order. First rsync -avP /mnt/diskX/ /mnt/diskY/ then rsync -avP --remove-source-files /mnt/diskX/ /mnt/diskY/ You could also throw in a checksum comparison on the second one if your concenred by sending rsync the -c attribute. Also I'm pretty sure that -P is the same as --progress.

I just finished my process of converting from rfs to xfs last night using your described process and rsync to move the data. I had no problems. I do want to say after a bit more reading about rsync I added to my prior post to add a note about not using --remove-source-files if there are active writes to the directories your moving.

After a bit of reading I think the answer is yes. Quotes from the Rsync Documentation. Under the -c (use Checksum) it says. And under --remove-source-files it says. So it seems that corruption would set off a redflag and the source file will not be deleted, unless I am missing something.

I just started this process last night. I want to say thanks for the intro in how to make this change. With out your post I doubt I would have taken the effort to make the change to xfs. I asked a few questions in another thread and someone made a suggestion which I want to share because I believe it reduces the steps, and is a more powerful approach. To my knowledge the following code represents a simplified method to accomplish steps 6, 7 & 8 in one pass while also maintaining permissions, timestamps, owners, groups, symlinks, and device files using rsync. rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ Where disk X is the rfs disk you are copying to the newly formated xfs disk (disk Y). For example I ran this last night on as /mnt/disk1/ /mnt/disk4/ copying disk1 to disk4 -a means it's in "archive mode" which equals -rlptgoD (Recursive, Symlinks, Permissions, Timestamps, Groups, Owners, Device Files) Note if you want to Preserve extended attributes add a capital X to your call of rsync. -v puts it in verbose mode (verbose is not required) --progress shows you the progress of the copies as files are moving (Progress is not required) --remove-source-files : After the copy is done rsync will verify and then remove the source files. IMPORTANT do not use --remove-source-files if you have files currently being writen to the disk, this will result in corruption of the that file. You should do this anyway since your moving data from one disk to the other, but disable any plugins - dockers which write to that disk, and it's best to not write to shares that could be put on that disk while rsync is running. The end result should be empty directories on diskX and a duplication of diskX on diskY. Rsync is very powerful and there are way more things that can be done with it, so if you have questions or want to know more about what it can and can't do you can read more here. I think this is a much more elegant solution.

Thanks again for the advice, I feel like I'm moving in the right direction now. I guess I need to figure out if I want to have a second unraid box or if I don't have enough data to warrent that yet and back up via an USB3 HDD attached outside of the array. Oddball question. Should you backup \flash ? In the event of a usbdrive failure I realize that a new key will be required (I have a backup of that anyway... because why not...) but is there any other settings or files which having a backup would help the process of getting up and running again? If not the whole flash are there important files we should be looking at backing up?

I want to say thank you so much for this! Really good stuff! The link is also awesome and set me on the right path. My only problem is that My wife and I live in a small apartment, (just starting out), so I'm not sure that I can convince her that a second backup server is what we need. She's onboard with the primary one. I could possibly put a remote server in at my parent’s house... but we'll have to see about that as well. Also I assume there are a few additional steps when performing back-ups across the internet?

I've seen some people suggest using rsync to set an automated backup of your unraid box to another unraid box (or other linux box). If you use an automated rsync process isn't there potential for corruption to be mirrored into the backup? If there are goals in having a backup are to protect against the following right? [*]Software Errors – File system corruption or bit rot or some other sort of internal error which silently (or not so silently) corrupts your data. [*]Hardware Failure – Protect against hard disk or other mechanical error [*]Human Error – Protect against accidental deletion or other human initiated mistakes which destroy your data. My concern with automated rsync processes is that that rsync will update files if they have changed from an earlier rsync run. If the reason the file has changed is silent corruption won’t that silent corruption be copied into the backup diminishing your ability to recover? Maybe rsync doesn't work that way... or maybe this can be avoided by properly configuring rsync? Any advice on how to properly do this would be much appreciated, because I've def seen people use the --delete command (which deletes files on the destination which don't exist on the source) Which totally removes the ability to recover from human error should you not catch your mistake before your automated process runs. Any other pitfalls to avoid?

Just want to say I did a bit more digging on this, and it seems that there are a lot of dropped packets early on (after a reboot, maybe I should have mentioned that before, but the earlier numbers were after a reboot) and sense then has been really solid driving the ratio way down. Is there a reason why I would expect a higher percentage of drops early with a lower and lower number as time goes on?

A direct jump to the plugin's configuration page makes a great deal of sense. I am not a plugin developer (Yet) but this make absolute sense to me. I got really confused at first when I installed my first plugin, only to have to hunt though the menus to descover where it was located. Clicking the plugin icon or link seems way more intuitive.

6% is a bit high, but if they are only coming through the WAN and not the LAN there may not be anything you can do about it. Most of my traffic should be within my LAN... I'm only running Plex, and APCUPSD (networked with a windows pc on the same UPS but that's still within the lan...). Oh well, I'm not super worried but I am going to look into this more. It's possible something is configured incorrectly, or there is a bad connection somewhere.

Fair enough... I might delve into this a bit more later, right now I'm showing Received: 992427 Drops: 59817 which is 6.02%... slightly higher then I expected... I guess it's not a huge concern. When i pinged Google 20 times I got 0% loss though... i suppose that 20 isn't really a great sample... but I digress.

I also just noticed that I have a lot of dropped packets listed for eth0 receive, is there a way to troubleshoot why or how these packets are getting dropped? I'm only running plex when it comes to plugins.