gundamguy

I mean that should work, but to be honest with you, I avoid doing that too, for the simple reason that I don't want to get in the habit of doing something that with a few minor changes can have major consequences. Also I wasn't trying to be pedantic earlyer, really wasn't sure how much you knew. All this said, I'm not really sure what why you would want to do a move like mv /mnt/disk1/media/stuff to /mnt/user/media/stuff The only time I use user share in my paths is when I'm doing config on an application that I want to read the data from whatever disk it's on (like giving plex /mnt/user/movies instead of giving it /mnt/disk1/movies /mnt/disk2/movies /mnt/disk3/movies and so on...) because this is more flexible, if I add a disk 4 at some point and I write to the movies share and files wind up on disk4 plex wouldn't need to be reconfigured to find them...) Now I do write to user shares a lot via Windows, but only when I don't care what disk in the share the data actually goes too. So when you say you want to move data around in your share, I just wonder why you don't just go to /mnt/disk to /mnt/disk? Would that cause problems because of balance reasons or because you are trying to move data from a share where you don't know what disk it's on to a specific disk, or move it off that disk? Just trying to get an understanding of what the goal of your planned move is, because if it's to consolidate data in a share that is on multiple disks to one disk or something we might be able to give you a better safer process.

Ok, now that I have a better understanding of what you are asking. At a basic level user shares work by aggregating the content of the multiple disks into a user share. From a Linux prospective it's as simple as this. /mnt/user/Movies Is just the sum of ALL /mnt/diskX/Movies folders for disks1-diskX where X is all your disks + Cache /mnt/user0/XXXX Is the same as /mnt/user/XXXX but without your Cache. Fill settings such as "Most Free", "Fill Up" or "Min Free Space" apply logic that helps decide which disk from 1 to X is written to when you write to /mnt/user/Movies because a write to /mnt/user/Movies is actually just a write to /mnt/diskX/Movies (where X is the desk that the logic decides is best.) Include / Exclude (don't use both at the same time) is logic that says never write to the selected disks, or only write to the selected disks when you write to /mnt/user/Movies Split Level is also logic that affects writes, but split level is more about keeping sets of folders and subfolders on the same disk.

The first thing you should know is that doing this puts you at a huge risk for Data Loss. Hence the reason why you want to exclude the disk it's currently on to prevent the copying from causing the glitch... (it's not a glitch exactly as much as a known issue) But my question is... is there a better way to accomplish what you are trying to accomplish? What is the end game you are trying to achieve that has you making this move?

You don't have to do it though Windows Explorer, if you are comfortable with Linux you can move the "t" folder using mv (or another command that works) as well. Going to say this again, because it can't be said too many times. NEVER copy from /mnt/user/Share to /mnt/disk/Share or vice versa. In windows explorer this would be copying from a "Share" to "DiskX" (Where X is the disk number). This causes a file system error that will result in data loss. mnt/user0/share is a special case but I am 99% sure it suffers from the same user share to disk problem.

to the best of my knowledge this is not possible but can I get an example of what you plan to accomplish if you could? From a hypothetical stand point, I'm pretty sure this is possible.... but having said that... I have no idea how to do it.

A couple of points / questions. I am using the other Checksum tools, but I like the style of this plugin. I am 100% sure that Corz will not work with the files + extended attributes approach, but it might work with the exported hash files if that file / files are in the right format. I'm not an expert on what that should be but it sound like Squid can help a lot on that if you choose to go down that path. Second Point: If you want to maintain your checksums while copying from one unRAID device to another, I am pretty sure you can use rsync with the extended attributes flag. -X or --xattrs should preserve extended attributes and not require rehasing the file...

That makes no sense to me based on documented ZFS requirements. The general rule of thumb is 1GB of RAM per TB of storage space. Are you sure it's actually ZFS?

Yes, the -c option invokes the generation of checksums for both source and destination. It's a really good way to be sure they are actually the same, but it takes a lot more time.

Both? If the file doesn't exist on diskY it'll copy the file, if the file does exist and when checked has the same time stamp and size, it won't transfer that file as it already exists, if the file exists but the time stamp or size are different it will update the file on diskY to match.

I'm not sure that this step was ever really necessary, but doing it won't hurt. I'm not exactly sure how Teracopy validates but I would trust its methods. Peforming a rsync -c check will take awhile and is wasted time if you've already validated the data IMO. Other then what has already been pointed out, looks good.

Fully support this. Currently using a custom script that's started via the go file to run a daily trim event. Since Cron is built is included in unraid now it seems like this should be an easy thing to support?

Also, it is NOT in any way an "error" condition ... it's simply an informative status indicating some data isn't fault-tolerant. For cache-only shares without a pool, that's always going to be the case; for cached shares, that will automatically change when they're moved to the array. I generally agree with what you've said but here are my comments. If the goal is to be informative perhaps don't use the same symbol that on a different tab means there are problems you need to fix. This requires people to understand that in one context this symbol is bad news, but in another context is nothing serious at all. I think cache pools are great, but there are legit reasons (limited drive bays, limited ports, need for all of your array slots) why people won't want or be able to do cache pools some times. An ideal build would have a cache pool IMO.

I agree with you in that context. I'm not sure this needs to be a persistant warning when it comes to Cache-only shares, because ostensbily you are using your cache only share for plugins/docker apps and it's only housing data you are ok with losing. I agree with Gary that this shouldn't exactly be a Green Cricle either, as it's not the same condition.

Agree ==> HOWEVER, the yellow triangle is NOT an "error" condition. It simply means "Some or all files are on unprotected storage." In the case of a cache-only share being stored on an unprotected cache drive that is the normal condition ... NOT an error. The user should certainly KNOW if he's storing data on a cache-only share ... and this indicator simply reminds him that the share is unprotected. Don't like looking at it? ... update the cache to a pool so it's protected I hear you and agree files on the cache are not inherently an error condition. However most of us have been conditioned to see yellow triangles as signifiers of that there is an non-critical issue that should be looked at. If this is normal behavior and performing as intended I don't understand why it's shown with a symbol that we are conditioned to believe signifies that something non-critical needs investigating. Right now it basically says "Caution! Everything is working properly!" What I would like to see is a better system which only shows a warning symbol if something went wrong, like mover failed or files were not moved properly according to the mover schedule. Perhaps even something that said the time of the last move, and time till next move? I don't know it seems like something that can be improved to give users better information so they know if they need to act on something or not.

jonathanm just summerized my thoughts right as I was about to post. I agree. If things are working as intended it shouldn't really be showing an "error" condition IMO.

I think there is a plugin for crashplan floating out there some where (try searching the plugin design forum), but I and some other people here used the following tutorial to create custom scripts that can run a daily (or Hourly, Weekly, Monthly, etc.) backup.

Hmm... Maybe I'm missing something. I was not planning on rebuilding a drive from parity. My thought was to empty out one drive by moving it's contents to another existing drive, remove the empty drive and replace it with a larger drive, pre-clear the new larger drive, format it with XFS, and then move data to it from another existing drive. Once that is done, I would continue a similar process until the data has been migrated to drives formatted with XFS. In particular here is the layout: Current state: Future State: Parity: 2 TB (keep drive) Parity: 2 TB Disk1: 1 TB RFS (60% full) (replace drive) Disk1: 2 TB XFS Disk2: 1 TB RFS (40% full) (replace drive) Disk2: 2 TB XFS Disk3: 2 TB RFS (20% full) (keep drive) Disk3: 2 TB XFS My thought process is as follows: Move all data from Disk1 to Disk3, remove current Disk1. Install replacement for Disk 1 Move all data from Disk2 to new Disk 1, remove Disk2. Install replacement for Disk 2 Move all data from Disk3 to new Disk 2, re-format Disk3. Move some data from Disks 1 and 2 to Disk 3 to even out the disks somewhat. Does that make sense? Thanks, John Your process does make sense, and will work, but as Trurl explained when you remove a data disk from the array you are going to invalidate parity and be prompted to rebuild your parity. So you might have more downtime as you do parity rebuilds during this process. Also since you plan to replace two disks that means you'll be prompted to rebuild parity twice. Not sure if there is a better way to sequence this, but you can view this as two seperate tasks, 1) Converting from RFS to XFS 2) Replacing to hard drives. If I were trying to do what you want to do, I would first replace the drives and then convert from XFS to RFS, but you can sequence this as well so that you replace drives and convert in alternating steps. You can run without a parity disk for awhile and still have access to your data on array data drives, if you want to. I'm not sure that pre-clearning has much value for you here since the disk has already been stress tested before. I'm not an expert on pre-clearning but I would think that you should be fine just re-formating it. However again I am not the best on pre-clearing so there might be some benifit I am missing here.

Greetings, I've been reading and re-reading this thread over the last couple of weeks. I have two questions: 1) Before I can add a new, larger drive, I need to move data around so that I can remove a drive. I figure it's about about 600 GB of data. Would it be best to shut down the array to move that data? I assume in that case I'm safe to just telnet to the unRAID server and use the following rsync commands to move the data into the existing hierarchy, without having to create a temporary subdirectory that isn't part of the user share, correct? rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/ At that point, I should be able to start following the guide, I think. 2) Where does pre-clear of the new drive get done? Is it before the steps in bjp999's guide? It is mentioned, but it is above the steps, so i wasn't sure. 3) Also - in the steps above, as the array is running, are you at risk of duplicate files, and confusing things between steps 10 and 11? Thanks, John You've got to different issues, replacing a disk with a bigger disk, and converting your file systems. The thing to keep in mind is that if you rebuild from parity your array will rebuild that disks with the current filesystem, you can't switch file systems and rebuild at the same time. So what you will need to do is either convert the disk to XFS first then swap for the bigger disk, or swap for the bigger disk then convert to XFS. If you've got space issues I think the second path makes more sense... Answers to questions: 1) If you've got space on the destination disk for the files on the source disk that should work just fine. 2) This depends on how you want to / are able to do this process. You might be better off, pre-clearning your disk, replacing the smaller disk, having it rebuild, then start the conversion process to XFS. In that case you'll have wanted to pre-clear before moving any data. 3) You should be fine at step 10 / 11 since you will have already formated the disk you moved the data off of.

I don't know for sure, but I do know that trailing / are important to rsync. Perhaps it's a missmatch issue, where you did /mnt/disk3/ the first time and /mnt/disk3 the second time. I'd run it again like rsync -nrcv /mnt/disk3/ /mnt/disk14/t >/boot/verify_disk14.txt see if that comes back with an empty list (since the files are already in disk14/t) Thanks, I will try that. EDIT:It worked, the file ended up containing: sending incremental file list sent 382,411 bytes received 813 bytes 6.83 bytes/sec total size is 3,928,060,848,066 speedup is 10,250,038.75 (DRY RUN) That is great news. Glad to hear it!

I don't know for sure, but I do know that trailing / are important to rsync. Perhaps it's a missmatch issue, where you did /mnt/disk3/ the first time and /mnt/disk3 the second time. I'd run it again like rsync -nrcv /mnt/disk3/ /mnt/disk14/t >/boot/verify_disk14.txt see if that comes back with an empty list (since the files are already in disk14/t)

I think (not tested the -R use of ls before myself) you can check if the files are actually still there using the following command. ls -R /mnt/disk4 ls lists the contents of a directory, the -R option does it recursively, and /mnt/disk4 is the disk I assume you want to check, if it's not disk 4 substitute for the disk number it is... I suspect this will only return the directories you had but no files. As to why Windows would think your files are still on disk4... well it doesn't. When trurl asked you to v /mnt/user the results that you posted show that you have a user share "mnt/user/disk4" that appears to be a disk share but is actually a user share. Because disk4 is actually a user share and user shares aggregate across all disks (unless the share is set to not do that) it's showing the data on the other disks when you look at the disk4 folder in Windows. Unless creating a user share named disk4 was intentional I suggest you change that.

Additional thought, are you looking at a disk share or a user share? If you are looking at a disk share the share should be called diskX. User shares by default aggragate files and directories from accross all the disks in the array (unless set differently) so even though you moved files from DiskX to DiskY they are on DiskY so they show up in the user share. If you look at the DiskX share (by sharing that disk under the disk settings) it only shows what's on DiskX and should show empty folders. If this isn't it, and those files really are still there I am really confused by what's going on, though I don't know that it's a huge problem.

Did you dig down below the top level into folders where the data you wanted to copy was stored? I think --remove-source does not delete folders just files, so you should have a bunch of empty folders left over. I suspect that it only appeard that the data was still there because the (now empty) folders were still there. If that's not it we can try something else to verify.

I think in that cause you have to manually split and copy the directories over. I do not think there is anyway to specify to rsync to only copy a set amount of data.

On rsync you need to include the -H option, or else it won't do anything about them... If you want more info about rsync -H you can find it here. Rsync Doc