huladaddy Posted January 31, 2016 Share Posted January 31, 2016 No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example: [color=blue]consld8 -f /mnt/user/Movies disk8[/color] will move the Movies user share from all other disks to disk8 (if there is enough space). So my question is what if there isn't enough space on disk8? I have a couple of tasks I would like to perform. [*]Move a bunch of TV shows I have spread across disks 4, 5, 8 & 10 to disks 1, 2, & 3, according to the allocation method I have chosen (high water.) [*]Move a bunch of movies from disks 4, 5, 6 & 7 to disks 5 & 6, again, according to the high water allocation method. Is there a way to achieve this using your tools? Or is there a better way? Quote Link to comment
Freddie Posted January 31, 2016 Author Share Posted January 31, 2016 No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example: [color=blue]consld8 -f /mnt/user/Movies disk8[/color] will move the Movies user share from all other disks to disk8 (if there is enough space). So my question is what if there isn't enough space on disk8? I have a couple of tasks I would like to perform. [*]Move a bunch of TV shows I have spread across disks 4, 5, 8 & 10 to disks 1, 2, & 3, according to the allocation method I have chosen (high water.) [*]Move a bunch of movies from disks 4, 5, 6 & 7 to disks 5 & 6, again, according to the high water allocation method. Is there a way to achieve this using your tools? Or is there a better way? Sorry, no, my tools completely bypass the allocation method and you have to specify a single destination disk. Quote Link to comment
switchman Posted February 1, 2016 Share Posted February 1, 2016 @huladaddy, The excel sheet I put together may help you. Basically, it read your drives, builds a grid showing drives where the various directories are located. You can elect to "move" a single directory from say disk2 to disk1 or you can move the directories from say disk2, disk3 and disk7 to disk1. In this case you use the stacked rsync command. Based on the selections you make, it builds a rsync command you copy and paste into a telnet session you you open to your Unraid server. Every time you run the data import scripts, it resets the rsync command to "dryrun". You have to change it in the drop-down. You really want to read through the entire thread, but at least these two links. http://lime-technology.com/forum/index.php?topic=33689.msg310759#msg310759 http://lime-technology.com/forum/index.php?topic=33689.msg415201#msg415201 Also, unraid allocations are not involved, you make the decision on the file locations and how to get them there. Quote Link to comment
jademonkee Posted July 19, 2016 Share Posted July 19, 2016 I ran consld8 to consolidate my music directory, and now SyncToy thinks heaps of files have changed that shouldn't have. Can you confirm that the time modified stamp doesn't change when using consld8? I've posted my problem - in excruciating detail - here: https://lime-technology.com/forum/index.php?topic=50458.0 Your help is appreciated. Quote Link to comment
Freddie Posted July 19, 2016 Author Share Posted July 19, 2016 I ran consld8 to consolidate my music directory, and now SyncToy thinks heaps of files have changed that shouldn't have. Can you confirm that the time modified stamp doesn't change when using consld8? I've posted my problem - in excruciating detail - here: https://lime-technology.com/forum/index.php?topic=50458.0 Your help is appreciated. The file modified time stamps do not change, but the directory time stamps do change. The consld8 script uses diskmv which in turn uses rsync to copy files with the -t option flag. The -t option flag preserves the modification times on all files, however, when files are copied to the destination disk and removed from the source disk, the parent directory's modification time is updated by the filesystem. Quote Link to comment
SpeedyVV Posted August 3, 2016 Share Posted August 3, 2016 new user of unRAID coming from Windows... Would it be possible to explain to me how to get the script and "install it"? Usually, I just try till it works, but this could be a bit destructive considering the functionality. Thanks in advance. Quote Link to comment
trurl Posted August 3, 2016 Share Posted August 3, 2016 new user of unRAID coming from Windows... Would it be possible to explain to me how to get the script and "install it"? Usually, I just try till it works, but this could be a bit destructive considering the functionality. Thanks in advance. Might be simpler for you to just use the unBalance plugin which is based on this. Quote Link to comment
SpeedyVV Posted August 4, 2016 Share Posted August 4, 2016 Might be simpler for you to just use the unBalance plugin which is based on this. Great suggestion. Got it installed in a flash. Unfortunately, it does not seem to be able to move a folder from Unassigned Devices :-( And that is what I am trying to do. Currently trying to move my old Windows (NTFS) data onto the array. I figured it would be quicker that doing a move from a Windows VM. Quote Link to comment
trurl Posted August 4, 2016 Share Posted August 4, 2016 Might be simpler for you to just use the unBalance plugin which is based on this. Great suggestion. Got it installed in a flash. Unfortunately, it does not seem to be able to move a folder from Unassigned Devices :-( And that is what I am trying to do. Currently trying to move my old Windows (NTFS) data onto the array. I figured it would be quicker that doing a move from a Windows VM. Google Midnight Commander (mc). It is built-in to unRAID and it gives you a sort of text-based, two-panel, file explorer that is easy to figure out and use. Just navigate to /mnt and look around, you will be able to see your disks, including the unassigned device. Then you can copy/move from one panel to the other with some function keys. Quote Link to comment
korpo53 Posted August 4, 2016 Share Posted August 4, 2016 new user of unRAID coming from Windows... Would it be possible to explain to me how to get the script and "install it"? Usually, I just try till it works, but this could be a bit destructive considering the functionality. Thanks in advance. Use Putty to remote into your server mkdir /boot/scripts wget https://raw.githubusercontent.com/trinapicot/unraid-diskmv/master/diskmv -O /boot/scripts/diskmv wget https://raw.githubusercontent.com/trinapicot/unraid-diskmv/master/consld8 -O /boot/scripts/consld8 echo "ln -s /boot/scripts/diskmv /usr/sbin" >> /boot/config/go echo "ln -s /boot/scripts/consld8 /usr/sbin" >> /boot/config/go That'll: Create a directory to hold the scripts that survives reboots. Download the scripts to that directory. Add lines to your startup script to create symlinks (shortcuts, sort of) to those permanent files in a non-permanent space that makes them more usable. After doing that, if you don't want to reboot, you can just run the below to create the symlinks right now. They'll get recreated every boot by the go script, so you only have to do this part the first time. ln -s /boot/scripts/diskmv /usr/sbin ln -s /boot/scripts/consld8 /usr/sbin Quote Link to comment
twg Posted August 8, 2016 Share Posted August 8, 2016 is there a way to preserve the directory timestamps ? Quote Link to comment
tokra Posted December 21, 2016 Share Posted December 21, 2016 I've created a couple of bash scripts to facilitate moving files between disks. As these utilities are touching user data, I'd like to have some experienced users review and/or test them. If there is interest and no major problems, I will look into making them easier to use (first a plugin for command line usage, then maybe gui) 'diskmv' will move a user share directory from one disk to another. It uses a find/rsync command similar to what is found in the standard mover script. It is suitable for merging a user share directory onto a disk that already contains that directory. Files that have duplicate file names on the destination disk will not moved. By default, 'diskmv' will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced. 'consld8' can consolidate a user share directory from multiple disks onto one disk. If a destination disk is not specified, it will pick the best disk based on max usage and available space. By default, consld8 will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced. 'diskmv' is required to actually move files. Example usage: diskmv "/mnt/user/video/tv/Pushing Daisies" disk4 disk1 consld8 /mnt/user/video/tv/Wonderfalls disk3 To get get a help message: diskmv -h I am tagging releases for code that I trust and have used on my real data. Releases are here: https://github.com/trinapicot/unraid-diskmv/releases The master branch has code that I have tested on sample data, but I have not necessarily used it on my real data. These utilities move files around on an unRAID server. I have done my best to prevent any data loss, but there is always a chance something can go wrong. Use at your own risk and please ensure you have everything backed up. Note for those wishing to use diskmv in the process of changing filesystems: The diskmv script will not verify that copied files are the same as the source files. With the default options, diskmv will rsync each file and if the rsync is successful, that file is then deleted before moving to the next file. There is no opportunity to compare the source and destination files. The script is relying on the OS, rsync and whatever else is involved to correctly read data from one disk and write it to another. If you still want to use diskmv to move all the files and directories from one disk to another, you can use this syntax: diskmv -f "" disk1 disk2 Or to keep source files (copy instead of move): diskmv -f -k "" disk1 disk2 Thank you so much for this work !!! I think this script should be pinned on forum for other guys to use - I took me a while when I found this great tool. I also think it would be to have it as WebUI plugin, if someone wants to do it I would but I never done any plugin Quote Link to comment
Fireball3 Posted December 21, 2016 Share Posted December 21, 2016 No need for a full-quote! Similar tool. 1 Quote Link to comment
tokra Posted December 21, 2016 Share Posted December 21, 2016 BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM. Quote Link to comment
JorgeB Posted December 21, 2016 Share Posted December 21, 2016 BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM. That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight. Quote Link to comment
tokra Posted December 21, 2016 Share Posted December 21, 2016 BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM. That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight. How much RAM your machine have ? This happens only when some big files are moving.. I think it has to do with low RAM...4GB ? what ya think ? Quote Link to comment
JorgeB Posted December 21, 2016 Share Posted December 21, 2016 BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM. That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight. How much RAM your machine have ? This happens only when some big files are moving.. I think it has to do with low RAM...4GB ? what ya think ? Now I have 8 but at the time I used it I had 4GB. Quote Link to comment
tokra Posted December 21, 2016 Share Posted December 21, 2016 BTW. just wanted to notice that script causes reboot when moving big files (HD movies etc.). I have 4GB of RAM. That probably means you have a hardware problem, I used the script to move TBs of data without any issue, in one of my servers it worked for 48 hours straight. How much RAM your machine have ? This happens only when some big files are moving.. I think it has to do with low RAM...4GB ? what ya think ? Now I have 8 but at the time I used it I had 4GB. I had one reply in different post where guy told me his 6.x system was unstable with 4gb of RAM, and after he upgraded to 8GB everything is fine now. I have been using this HW for 4 years stable with UNRAID. Quote Link to comment
slushieken Posted June 28, 2017 Share Posted June 28, 2017 I just wanted to say this script is just... amazing. I have been working on cleaning up nearly 20TB of movies and tv shows for several years now split incorrectly over time due to this or that, settings errors on splits, etc. I have been having success over that time with many different methods and a hit and miss effort, but yours wrapped it all up in one fell swoop. Thank you. Quote Link to comment
FreeMan Posted December 1, 2017 Share Posted December 1, 2017 Jumping on the THANK YOU bandwagon! I was using unBalance, but its one big drawback is resolved with @Benni-chan's excellent commands here. I used that to straighten up my Movie share in about 2 hours. I'd spent a couple of hours at a time using unBalance one movie at a time because uB won't automatically pick a disk to move everything to. I did note that it left a few files behind, but when I went back through the log ( | tee movielog.txt to the rescue!) to investigate, I discovered that it was skipping duplicate files. Somehow, I'd gotten different versions of the <movie>.nfo file. I deleted about a dozen of them and all is good! Off to clean up my TV shows next. On 11/22/2015 at 1:10 PM, Archivist said: My moving file/directory is very large and it's still testing after 15 minutes with no way that I'm aware to stop the process. Also, Ctrl-C is a great way to stop a shell script without killing the telnet session and losing all your command history. /late to the party... Quote Link to comment
FreeMan Posted December 2, 2017 Share Posted December 2, 2017 On 11/30/2017 at 7:57 PM, FreeMan said: I did note that it left a few files behind I've also noted that it's left quite a few empty directories behind. This is probably something in diskmv since unBalance does it too, and I understand that it uses diskmv in the background. Quote Link to comment
FreeMan Posted December 2, 2017 Share Posted December 2, 2017 Once again, @Benni-chan has come to the rescue and gotten my TV share about 98% cleaned up. However, what I'm finding is this: A season has been nicely consolidated to a single disk, but the various show-level metadata is still scattered about. So I ran a combo script that, when I ran it in test mode seemed to have taken care of things nicely: find "/mnt/user/Media/TV" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 consld8 -f | tee TVlog-season.txt However, when I started running it in real mover mode, it started to move season folders. I read the find man page and changed -type d to -type f in the hopes of finding only the files at the <show> level. That worked, but... it ended up passing file names to consld8, instead of directory names. It looks like the issue I'm running into is that consld8 will, by default, move all files and sub-directories to the specified destination disk, and I want it to only move files. Is my understanding of this issue correct? If so, is there a possibility of adding a switch to consld8 that would have it move files only and leave the sub-directories where they are? Still really a fan of these two tools! Even with the OCD cleanup of the <show>/<files> I'm certain that this will end up saving me many, many, many hours of manual work vs what I was doing before. I've accomplished more clean up by letting two command lines run on their own than I had in a week of sessions with mc or even unBalance! Quote Link to comment
Urlryn Posted December 17, 2017 Share Posted December 17, 2017 (edited) Is there a way to expand on this command to find a group of folders starting with the letter M...something like this for example: find "/mnt/user/Media/Movies" -name "M*" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 consld8 -f disk1 Well obviously that didn't work since consld8 requires a share specified! Right now having to do one folder at a time sucks since i'm trying to move stuff to Disk1 or Disk2 for organizational purposes. So I don't want to do a full disk4 to disk1. Not enough room even if I did! Thanks Urlryn Edited December 18, 2017 by Urlryn Quote Link to comment
Urlryn Posted December 18, 2017 Share Posted December 18, 2017 (edited) Figured it out.... Did a couple test runs for some of my anime stuff and everything transferred nicely! --Finds any anime folder starting with Kidou and merges them to disk1 find "/mnt/user/media/anime" -name "Kidou*" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 -I {} consld8 -t {} disk1 --Finds any folder starting with A or B and merges them to disk1. This will omit any folder whose name is only a single letter A or b find "/mnt/user/media/anime" \( -name '[ab]*' \) -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 -I {} consld8 -t {} disk1 --If you also want to include those above, you will need to specify another pattern: the name must match either [ab]* or [ab] find "/mnt/user/media/anime" \( -name '[ab]*' -o -name '[ab]' \) -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 -I {} consld8 -t {} disk1 *Note...the commands are set with the -t switch. So will need to change this to -f when you are ready to do the actual move/merge. Please test before changing it! Urlryn Edited December 18, 2017 by Urlryn Quote Link to comment
deagle Posted April 20, 2018 Share Posted April 20, 2018 Thank you @Freddie @Benni-chan! When I first migrated to unRAID, even though I had researched split levels and set them up accordingly, rsync sprayed folders all over the array and the files ended up scattered. Finally fixed! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.