Freddie

The file modified time stamps do not change, but the directory time stamps do change. The consld8 script uses diskmv which in turn uses rsync to copy files with the -t option flag. The -t option flag preserves the modification times on all files, however, when files are copied to the destination disk and removed from the source disk, the parent directory's modification time is updated by the filesystem.

So my question is what if there isn't enough space on disk8? I have a couple of tasks I would like to perform. [*]Move a bunch of TV shows I have spread across disks 4, 5, 8 & 10 to disks 1, 2, & 3, according to the allocation method I have chosen (high water.) [*]Move a bunch of movies from disks 4, 5, 6 & 7 to disks 5 & 6, again, according to the high water allocation method. Is there a way to achieve this using your tools? Or is there a better way? Sorry, no, my tools completely bypass the allocation method and you have to specify a single destination disk.

I'm sorry you are having troubles Dave. I read through your thread and it looks painful. The next time I mess with consld8 I will consider adding an option to keep the source files.

Thanks Freddie, I really appreciate the info. My 5TB drive should accomodate all of my Movies (at the moment) and then some. As presently configured w/o "usr" directories/shares would the command be "consld8 -f /mnt/Movies disk5" or do I need the "/usr/" in the the command line? Thanks so much for your response and wonderful work! Dave Both diskmv and consld8 are designed to act on user share directories. You must specify a valid user share path. I expect you can simply enable user shares, then /mnt/user/Movies will be a valid user share path and you should be able to run [color=blue]diskmv -f -k /mnt/user/Movies disk1 disk5[/color] .

No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example: [color=blue]consld8 -f /mnt/user/Movies disk8[/color] will move the Movies user share from all other disks to disk8 (if there is enough space).

I came to the conclusion they are totally harmless. See here for discussion: https://github.com/trinapicot/unraid-diskmv/issues/2

Yes, file attributes are preserved. That syntax should work.

There is no built-in log file mechanism in diskmv. You can re-direct standard output to a file diskmv /user/share/video disk1 disk2 > logfile.log Or use tee to keep standard output on your terminal and get a log file diskmv /user/share/video disk1 disk2 | tee logfile.log

diskmv will skip any files that are in use. Is there any chance the left-behind files were being accessed by something else when you where trying to move them?

A single slash can make a big difference. It should be: rsync -nrcv /mnt/disk1/ /mnt/disk10/t >/boot/verify.txt Without the slash, rsync would copy the directory. With the slash rsync would copy the contents of the directory. @bjp999: Perhaps your instruction post should be edited to include the slash:

I'm not sure diskmv is the best tool to use for switching file systems, especially if you want to follow the recommendation to verify files on the destination disk before deleting files from the source disk. I've updated the first post to try and address this issue. I think it would be good to have more testing before making a plugin. I only know of the few users that have posted in this thread. Has anyone else out there used these scripts? Any problems? New users may want to wait until there's been more testing, but if you want to jump in thestewman, these are just bash scripts that you run from the commandline (telnet, ssh or a keyboard/monitor attached to the server). It works much like the preclear script if you used that. There are many ways to set it up, the easiest may be: [*]Download the latest release zip from github, currently https://github.com/trinapicot/unraid-diskmv/archive/v0.2.0.zip [*]Extract the zip and copy diskmv and consld8 to the root of your unRAID flash drive [*]At the command prompt, change to the directory that contains the scripts: cd /boot/ [*]Run the script: diskmv -h

Yes I will consider it. I have encountered that type of error in the past with the mover script. I traced it back to cache_dirs but never really understood what was going on. Are you running cache_dirs? For me, running 'ps 7924' (where the number matches the number in the error message) showed it was a cache_dirs process. That's about as far as my understanding of the proc filesystem goes. Maybe it's time for me to do some reading.

Thanks for testing and providing feeback switchman. Yep, that should happen, but it doesn't. I'll work on that. That is a "feature" of bash (the shell you are using to enter the commands). Spaces and parens are special characters in bash and they must be quoted or escaped when they are in file names. You get the same cryptic error for the same problem in very basic bash commands: root@tower:~/testdata# ls Archer (2009) -bash: syntax error near unexpected token `(' root@tower:~/testdata# ls "Archer (2009)" Archer\ (2009) root@tower:~/testdata# ls Archer\ \(2009\) Archer\ (2009) It's just something you have to deal with when working at the command line. The "\" tells bash to ignore the special meaning of the next character. The tab key can be very helpful in these situations, it will autocomplete file names when possible. Type in the first part of the file name then hit tab and it will autocomplete if there is a single file that matches. Hit tab twice and it will list all the possible matches. When it autocompletes it includes the necessary backslashes to escape the special characters.

Yes, consld8 calls diskmv and diskmv only moves files from one disk to another. No moves are done to or from the user share. I was trying to follow your flowchart, but I am not a flowcharter. I think a lot of the logic involving disk usage and free space is already included in consld8. A relatively simple script could be made to loop through all first level subdirectories and perform consld8 on each of them, letting consld8 determine the destination disk. You could even use 'find' and do it with one line (probably). An alternative solution might be to add a recursive option to consld8. If a disk is not found with enough free space for the specified directory, consld8 is performed on all of the first level subdirectories. I came at these tools from a perspective that day-to-day dispersal of files is controlled by split level settings. If a new season of a tv show starts then a new season folder is created on the same disk as the other seasons of that show. This can be accomplished with split level settings. But sometimes mistakes are made or disks fill up and things need to be rearranged. That is when I intended for diskmv and consld8 to be used. That being said, if you want to run them on a cron schedule, I won't try to stop you.

Based on the comments and suggestions made earlier, I have released updated scripts here: https://github.com/trinapicot/unraid-diskmv/tree/v0.2.0 This is very much a "use at your own risk" kind of thing, but I am tagging releases that I trust and have used on my own data. But I definitely make sure my data is fully backed up before using it. Anyone interested enough to test it out? @NAS & @WeeboTech: The master branch has an option in diskmv to move only small files from one disk to another. If you are interested.

'diskmv' is a utility to that moves files. 'consld8' is one of many possible utilities that determine how files should be moved depending on the goal. These many possible utilities then call 'diskmv' to get the job done. mbfrosty's 'rebalance' / 'pressure relief' and lboregard's 'unBalance' / 'disk packer' are other examples that could use 'diskmv' to get their job done. Another I've seen suggested is 'empty disk' which is very similar to 'unBalance', just with more available space.

I agree. The error message is not good and that may be a reason to have a separate check for a null value. I will make it better and add some usage output on error. I will also add the DEBUG snippet. Thank you.

Thanks. All your ideas are under consideration. I especially like the dry run default for diskmv. I was planning to remove the test option as default in consld8, but I now think having them both be "opt in" for write would be better.

Yep, I failed to consider a call like: diskmv "" disk1 disk2 , but that could be valid usage if you want to move everything from one disk to another, so I think I'll leave it. consld8 "" disk1 could also be valid, but I don't know if anyone would find that useful. $2 is not tested with null, but the subsequent variable $SRCDISK is tested to be a valid disk, so a null value in $2 is caught. Is there some reason I need to test $2 with null at first reference? Word splitting is not performed in case statements and assignment statements so case $1 in ... and SRCDISK=$1 are OK. But I will add some more quoting. Thanks for looking and commenting. I welcome more comments if you look deeper.

I've created a couple of bash scripts to facilitate moving files between disks. As these utilities are touching user data, I'd like to have some experienced users review and/or test them. If there is interest and no major problems, I will look into making them easier to use (first a plugin for command line usage, then maybe gui) 'diskmv' will move a user share directory from one disk to another. It uses a find/rsync command similar to what is found in the standard mover script. It is suitable for merging a user share directory onto a disk that already contains that directory. Files that have duplicate file names on the destination disk will not moved. By default, 'diskmv' will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced. 'consld8' can consolidate a user share directory from multiple disks onto one disk. If a destination disk is not specified, it will pick the best disk based on max usage and available space. By default, consld8 will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced. 'diskmv' is required to actually move files. Example usage: diskmv "/mnt/user/video/tv/Pushing Daisies" disk4 disk1 consld8 /mnt/user/video/tv/Wonderfalls disk3 To get get a help message: diskmv -h I am tagging releases for code that I trust and have used on my real data. Releases are here: https://github.com/trinapicot/unraid-diskmv/releases The master branch has code that I have tested on sample data, but I have not necessarily used it on my real data. These utilities move files around on an unRAID server. I have done my best to prevent any data loss, but there is always a chance something can go wrong. Use at your own risk and please ensure you have everything backed up. Note for those wishing to use diskmv in the process of changing filesystems: The diskmv script will not verify that copied files are the same as the source files. With the default options, diskmv will rsync each file and if the rsync is successful, that file is then deleted before moving to the next file. There is no opportunity to compare the source and destination files. The script is relying on the OS, rsync and whatever else is involved to correctly read data from one disk and write it to another. If you still want to use diskmv to move all the files and directories from one disk to another, you can use this syntax: diskmv -f "" disk1 disk2 Or to keep source files (copy instead of move): diskmv -f -k "" disk1 disk2

The issue with cache_dirs not woking properly when invoked from the go file on unRAID v6 is due to the first few lines in the cache_dirs script: #!/bin/bash if test "$SHELL" = "/bin/sh" && test -x /bin/bash; then exec /bin/bash -c "$0" "$@" fi The SHELL environment variable starts out as "/bin/sh" and does not change when executing /bin/bash. So it gets stuck in a loop. When invoked from a standard shell, the SHELL environment variable is "/bin/bash" and it works just fine. The test can be modified like this: if test "$BASH"x = "x" && test -x /bin/bash; then This works on unRAID 6.0-beta5a booted with Xen. I was thinking of other required test cases, but I'm not sure what this chunk of code is supposed to be doing. It appears to ensure the script is invoked with bash, but doesn't the first line (#!/bin/bash) do that on its own? I would think the other three lines could just be removed.