March 10, 201511 yr I'm not sure diskmv is the best tool to use for switching file systems, especially if you want to follow the recommendation to verify files on the destination disk before deleting files from the source disk. I've updated the first post to try and address this issue. You can see why people get confised ... the manpage about -c is a bit misleading but if you read it all Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum more debate here http://unix.stackexchange.com/questions/30970/does-rsync-verify-files-copied-between-two-local-drives I am sure its possible in some world that rsync fails and deletes the original but ive never heard of anyone seeing it. has anyone? I am still going to recommend diskmv over cp/mv and manual rsync as I think the real list of data loss is because new users with no experience of this stuff start learing it by using/messing with rsync --delete-after.
July 16, 201510 yr the diskmv script is great - but i've noticed that when moving a lot of files (e.g. >10k) sometimes it leaves a few behind and you need to re-run it.
July 18, 201510 yr Author the diskmv script is great - but i've noticed that when moving a lot of files (e.g. >10k) sometimes it leaves a few behind and you need to re-run it. diskmv will skip any files that are in use. Is there any chance the left-behind files were being accessed by something else when you where trying to move them?
July 19, 201510 yr no, they weren't in use - the only client that would be (kodi) was switched off. The failure rate is something like 0.05% (i.e. i had about 5 files in the 10k i was moving). Is there a way to capture a detailed log other than piping the output to |& logger (as i want to be able to keep track)?
July 19, 201510 yr Author Is there a way to capture a detailed log other than piping the output to |& logger (as i want to be able to keep track)? There is no built-in log file mechanism in diskmv. You can re-direct standard output to a file diskmv /user/share/video disk1 disk2 > logfile.log Or use tee to keep standard output on your terminal and get a log file diskmv /user/share/video disk1 disk2 | tee logfile.log
July 22, 201510 yr Hi, I'd like to use this to consolidate my season folders, rather than the entire tv shows... Would I have to use the mv command along with the path to each season, for each show? I have approx 7000 episodes, and the season folders are scattered among 6 drives so I presume this might be impossible if in fact I need to run it at season level each time... I'm also looking to consolidate my movies folder, as the various movie folders are split between drives (sub on one, mkv on another etc). Would I need to run this for each movie folder, or can it be run for the entire movie folder? Sorry for all the question :-/
July 22, 201510 yr @markswift: i've used consld8 recently for my Movies Collection for the same reason you wish. Since I have a few thousand movie folders, i used following command: (USE AT OWN RISK) find "/mnt/user/Media/Movies" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -n 1 consld8 -f the find command goes through the submitted path (in this case /mnt/user/Media/Movies) and only lists directories (-type d). with help of maxdepth, i tell find to not include any subdirectories in the listing. mindepth 1 tells find to not include the parent directory. it could be modified for your season problem (i used the same script for my tv shows, so every season of a show is on the same disk). if you want to have it consolidate on a season and not show level, you could use this command: find "/mnt/user/Media/TV" -mindepth 2 -maxdepth 2 -type d -print0 | xargs -0 -n 1 consld8 -f (not tested, but should work. mindepth and maxdepth are modified, so only the subdirectories of the shows (season folders) are seen. If you want to test it without moving anything, replace the -f at the end with -t. (of course you have to change the path at the beginning, pointing to the correct user share) this runs consld8 in auto mode, so the script will try to determine the best target disk.
July 22, 201510 yr Hi Benni, Firstly, I really appreciate you taking the time to help me, I've run the movie command you gave me and it work perfectly! Regarding TV shows, I think I'm looking to do the same as you did? I want to keep each season of a particular show on the same disk (To prevent unrequired spin ups)... Is the command you gave me correct for that usage? My share for TV is mnt/usr/TV/Show Name/Season XX/S01E01 - Episode Name - Quality.mkv
July 22, 201510 yr maybe I misunderstood you in your first post on my system, i have every season of a show on the same disk and not only the files of one season. for the first case, you can use the same command as with the movies. for the second case (i thought, you wanted to do this), use the other command I posted above (with mindepth and maxdepth set to 2). try to use the command with -t at the end instead of -f first, to see, if it does, what you want... case 1 (move every season of a show to the same disk): disk1/TV/show1/season 01/... disk1/TV/show1/season 02/... disk1/TV/show1/season 03/... disk2/TV/show2/season 01/... disk2/TV/show2/season 02/... case 2 (consolidate on a season level, so all files of a season are on the same disk, but seasons of a show could be on different disks): disk1/TV/show1/season 01/... disk3/TV/show1/season 02/... disk2/TV/show1/season 03/... disk1/TV/show2/season 01/... disk4/TV/show2/season 02/...
August 3, 201510 yr I am thinking of using this script to assist in my move to XFS, but I wanted to know does this method preserve all of the file attributes or will things like the time/date stamps change? I was going to use this command: Disk1 is the currently full and disk4 is the newly formatted XFS disk and I wanted it to save a log. diskmv -f "" disk1 disk4 > disk1move.log
August 3, 201510 yr Author I am thinking of using this script to assist in my move to XFS, but I wanted to know does this method preserve all of the file attributes or will things like the time/date stamps change? Yes, file attributes are preserved. I was going to use this command: Disk1 is the currently full and disk4 is the newly formatted XFS disk and I wanted it to save a log. diskmv -f "" disk1 disk4 > disk1move.log That syntax should work.
August 12, 201510 yr Hey Freddie, Ive been using the scripts and getting this intermittently using consld8: Cannot stat file /proc/23098/fd/51: No such file or directory It appears that all files are moved/copied correctly but I'm not sure why these errors would occur. Any thoughts?
August 12, 201510 yr Author I came to the conclusion they are totally harmless. See here for discussion: https://github.com/trinapicot/unraid-diskmv/issues/2
August 12, 201510 yr Awesome. Small feature suggestion. It would be nice to run the consld8 command with a switch that gives you a summary, i.e.: consld8 -s /mnt/user/media/tvshows/Alphas would output (if consolidated): /mnt/user/media/tvshows/Alphas Total usage: 32099028 (Consolidated: disk2) If not consolidated: /mnt/user/media/tvshows/Alphas Total usage: 32099028 (Not consolidated)
October 5, 201510 yr @Freddie, wonderful work ! For an "older user", it would be great to have this as a Plug-in similar to Pre-Clear. I'm in the process of upgrading an older server (its run out of gas) and have added a new 5TB Parity Drive + 5TB storage drive (XFS) to my Array. I still have several 1 & 2 TB's that I need to consolidate the data on, most of which will fit on the "new" 5TB drive & perhaps 1 x 2 TB or another new drive (TBD). My ? is do I run diskmv first then consld8? I have lots of /Movies, /TV, /Music, & Data Files spread between various disks. Currently the Array is started but w/o any Global Share Settings even though all disks were set-up with those Folders under the old UnRaid (V1 - whatever Tom named the first version before V4! - Yep I'm that "old" ). FYI - Most of the Movies files are in separate "name".TS Folders though a few have been converted to /apple/"name".MKV and the Music files are in both separate artist as well as combined folders. Thanks in advance for any assistance or help. Dave
October 8, 201510 yr Author My ? is do I run diskmv first then consld8? I have lots of /Movies, /TV, /Music, & Data Files spread between various disks. Currently the Array is started but w/o any Global Share Settings even though all disks were set-up with those Folders under the old UnRaid (V1 - whatever Tom named the first version before V4! - Yep I'm that "old" ). No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example: [color=blue]consld8 -f /mnt/user/Movies disk8[/color] will move the Movies user share from all other disks to disk8 (if there is enough space).
October 9, 201510 yr No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example: [color=blue]consld8 -f /mnt/user/Movies disk8[/color] will move the Movies user share from all other disks to disk8 (if there is enough space). Thanks Freddie, I really appreciate the info. My 5TB drive should accomodate all of my Movies (at the moment) and then some. As presently configured w/o "usr" directories/shares would the command be "consld8 -f /mnt/Movies disk5" or do I need the "/usr/" in the the command line? Thanks so much for your response and wonderful work! Dave
October 10, 201510 yr Many thanks to Freddie for these scripts, I didn’t plan my split levels as I should and I would spend days trying to do this by hand. Thanks as well to Benni-chan, I have over 1000 tv folders to consld8 and that helps a lot. Is it possible to output the test run for de line below to a text file? find "/mnt/user/TV" -mindepth 2 -maxdepth 2 -type d -print0 | xargs -0 -n 1 consld8 -t
October 17, 201510 yr Hi Freddie, Sorry to be a bother, but I am trying desperately to both move & consolidate my files onto some larger disks. I have been able to download and install both diskmv and consld8 to my flash and can invoke either, but all I get back is 'disk1' is not a valid user share path. My command line is diskmv -f -k disk1 disk5 and I've tried putting it into "" and even adding Disk 1 as a Share, but to no avail. I'm sure that this is basic but it's driving me crazy. I need to move my disks 1-3 to my 5TB drive so that I can take them out of the array and replace with larger disks. Help from anyone else would also be appreciated. Thanks! Dave
October 17, 201510 yr Author No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example: [color=blue]consld8 -f /mnt/user/Movies disk8[/color] will move the Movies user share from all other disks to disk8 (if there is enough space). Thanks Freddie, I really appreciate the info. My 5TB drive should accomodate all of my Movies (at the moment) and then some. As presently configured w/o "usr" directories/shares would the command be "consld8 -f /mnt/Movies disk5" or do I need the "/usr/" in the the command line? Thanks so much for your response and wonderful work! Dave Hi Freddie, Sorry to be a bother, but I am trying desperately to both move & consolidate my files onto some larger disks. I have been able to download and install both diskmv and consld8 to my flash and can invoke either, but all I get back is 'disk1' is not a valid user share path. My command line is diskmv -f -k disk1 disk5 and I've tried putting it into "" and even adding Disk 1 as a Share, but to no avail. I'm sure that this is basic but it's driving me crazy. I need to move my disks 1-3 to my 5TB drive so that I can take them out of the array and replace with larger disks. Help from anyone else would also be appreciated. Thanks! Dave Both diskmv and consld8 are designed to act on user share directories. You must specify a valid user share path. I expect you can simply enable user shares, then /mnt/user/Movies will be a valid user share path and you should be able to run [color=blue]diskmv -f -k /mnt/user/Movies disk1 disk5[/color] .
October 17, 201510 yr @markswift: it could be modified for your season problem (i used the same script for my tv shows, so every season of a show is on the same disk). if you want to have it consolidate on a season and not show level, you could use this command: find "/mnt/user/Media/TV" -mindepth 2 -maxdepth 2 -type d -print0 | xargs -0 -n 1 consld8 -f (not tested, but should work. mindepth and maxdepth are modified, so only the subdirectories of the shows (season folders) are seen. Just to report that this worked perfectly for me to consolidate my TV series by season on the same disk, it took about 48 hours to complete, can’t even imagine how long it would take by hand. Thanks again.
October 28, 201510 yr edit: I moved my issue here, as it appears to be issue with the unbalance and not diskmv. http://lime-technology.com/forum/index.php?topic=39707.msg417146#msg417146
November 1, 201510 yr Hi Freddie, I'd like to offer a Word of Caution to others using your wonderful diskmv and consld8 programs. They work exactly as planned but consld8 removes the data from a disk which could be used as a Backup for possible failure or Stupid actions, like mine. [see HELP PLEASE? by Archivist link=topic=43674.msg416774#msg416774 date=1445968491] Simply put, I consolidated all my Movies, Music, TV Shows on a new larger drive formatted in XFS and by accident started the Array in wrong order resulting in a total disaster with all of my 4+ years of work! Consld8 needs a command that would allow it to copy & combine files (eliminate duplicates) but retain the data on the old disks. As Garycase has noted, Backups are a Necessity not a Luxury (my paraphrase) http://lime-technology.com/forum/index.php?topic=31020.0 Dave
November 1, 201510 yr Author Hi Freddie, I'd like to offer a Word of Caution to others using your wonderful diskmv and consld8 programs. They work exactly as planned but consld8 removes the data from a disk which could be used as a Backup for possible failure or Stupid actions, like mine. [see HELP PLEASE? by Archivist link=topic=43674.msg416774#msg416774 date=1445968491] Simply put, I consolidated all my Movies, Music, TV Shows on a new larger drive formatted in XFS and by accident started the Array in wrong order resulting in a total disaster with all of my 4+ years of work! Consld8 needs a command that would allow it to copy & combine files (eliminate duplicates) but retain the data on the old disks. As Garycase has noted, Backups are a Necessity not a Luxury (my paraphrase) http://lime-technology.com/forum/index.php?topic=31020.0 Dave I'm sorry you are having troubles Dave. I read through your thread and it looks painful. The next time I mess with consld8 I will consider adding an option to keep the source files.
November 22, 201510 yr Freddie: Quick Update ? I did finally get some of my data back and am setting up my Backup Server per Gary. I do need to Consolidate/Move some of the Data around in order to have enough space for the Backup Server and am running a Test Move. Is there a command to halt the process/test and if not could that be added in your next iteration to both "diskmv & consld8". My moving file/directory is very large and it's still testing after 15 minutes with no way that I'm aware to stop the process. I just wanted to see if it would do what was planned! Thanks, Again GREAT PROGRAMS!!! Dave SOLVED: Closed my Telnet Session
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.