Freddie
-
Posts
99 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by Freddie
-
-
No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example:
[color=blue]consld8 -f /mnt/user/Movies disk8[/color]
will move the Movies user share from all other disks to disk8 (if there is enough space).
So my question is what if there isn't enough space on disk8?
I have a couple of tasks I would like to perform.
[*]Move a bunch of TV shows I have spread across disks 4, 5, 8 & 10 to disks 1, 2, & 3, according to the allocation method I have chosen (high water.)
[*]Move a bunch of movies from disks 4, 5, 6 & 7 to disks 5 & 6, again, according to the high water allocation method.
Is there a way to achieve this using your tools? Or is there a better way?
Sorry, no, my tools completely bypass the allocation method and you have to specify a single destination disk.
-
Hi Freddie, I'd like to offer a Word of Caution to others using your wonderful diskmv and consld8 programs. They work exactly as planned but consld8 removes the data from a disk which could be used as a Backup for possible failure or Stupid actions, like mine. [see HELP PLEASE?
by Archivist link=topic=43674.msg416774#msg416774 date=1445968491]Simply put, I consolidated all my Movies, Music, TV Shows on a new larger drive formatted in XFS and by accident started the Array in wrong order resulting in a total disaster with all of my 4+ years of work!
Consld8 needs a command that would allow it to copy & combine files (eliminate duplicates) but retain the data on the old disks.
As Garycase has noted, Backups are a Necessity not a Luxury (my paraphrase) http://lime-technology.com/forum/index.php?topic=31020.0
Dave
I'm sorry you are having troubles Dave. I read through your thread and it looks painful. The next time I mess with consld8 I will consider adding an option to keep the source files.
-
No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example:
[color=blue]consld8 -f /mnt/user/Movies disk8[/color]
will move the Movies user share from all other disks to disk8 (if there is enough space).
Thanks Freddie, I really appreciate the info. My 5TB drive should accomodate all of my Movies (at the moment) and then some. As presently configured w/o "usr" directories/shares would the command be "consld8 -f /mnt/Movies disk5" or do I need the "/usr/" in the the command line?
Thanks so much for your response and wonderful work!
Dave
Hi Freddie,
Sorry to be a bother, but I am trying desperately to both move & consolidate my files onto some larger disks. I have been able to download and install both diskmv and consld8 to my flash and can invoke either, but all I get back is 'disk1' is not a valid user share path.
My command line is diskmv -f -k disk1 disk5 and I've tried putting it into "" and even adding Disk 1 as a Share, but to no avail. I'm sure that this is basic but it's driving me crazy. I need to move my disks 1-3 to my 5TB drive so that I can take them out of the array and replace with larger disks.
Help from anyone else would also be appreciated. Thanks!
Dave
Both diskmv and consld8 are designed to act on user share directories. You must specify a valid user share path. I expect you can simply enable user shares, then /mnt/user/Movies will be a valid user share path and you should be able to run
[color=blue]diskmv -f -k /mnt/user/Movies disk1 disk5[/color]
.
-
My ? is do I run diskmv first then consld8? I have lots of /Movies, /TV, /Music, & Data Files spread between various disks. Currently the Array is started but w/o any Global Share Settings even though all disks were set-up with those Folders under the old UnRaid (V1 - whatever Tom named the first version before V4! - Yep I'm that "old"
).No need to run diskmv first. When you run consld8, it will call diskmv as many times as needed to consolidate your source directory to the destination disk. For example:
[color=blue]consld8 -f /mnt/user/Movies disk8[/color]
will move the Movies user share from all other disks to disk8 (if there is enough space).
-
Any one else have their cache drive show up as an "Unassigned Drive"?
Other points of interest regarding this, every reboot of unraid my assigned cache drive isn't automatically selected since changing from spinner to ssd.
I had the same symptoms after changing cache drive from one SSD to another. Seems like unRAID doesn't want to forget about the first cache device. I had to stop array, un-assign cache device, start array, stop array, assign new cache, and start array again.
I don't think I had this plugin installed, but I'm not absolutely sure.
-
I came to the conclusion they are totally harmless. See here for discussion: https://github.com/trinapicot/unraid-diskmv/issues/2
-
I am thinking of using this script to assist in my move to XFS, but I wanted to know does this method preserve all of the file attributes or will things like the time/date stamps change?
Yes, file attributes are preserved.
I was going to use this command:
Disk1 is the currently full and disk4 is the newly formatted XFS disk and I wanted it to save a log.
diskmv -f "" disk1 disk4 > disk1move.log
That syntax should work.
-
Is there a way to capture a detailed log other than piping the output to |& logger (as i want to be able to keep track)?
There is no built-in log file mechanism in diskmv. You can re-direct standard output to a file
diskmv /user/share/video disk1 disk2 > logfile.log
Or use tee to keep standard output on your terminal and get a log file
diskmv /user/share/video disk1 disk2 | tee logfile.log
-
the diskmv script is great - but i've noticed that when moving a lot of files (e.g. >10k) sometimes it leaves a few behind and you need to re-run it.
diskmv will skip any files that are in use. Is there any chance the left-behind files were being accessed by something else when you where trying to move them?
-
This is a very odd behavior; the lsblk and the blockdev commands work fine when reading the disk, but not when reading the partition. I'm unable to debug this right now, since I don't have a spare 2TB+ hdd around. If anyone found a solution, please let me know.
A precleared disk >2.2TB doesn't really have a partition on it. It has a "GPT Protective MBR" but it doesn't have a GPT. The block tools interpret it as a MBR partition of maximum size.
In reviewing this plugin code I see the partition size is reported in the disk row when there is only one partition. I would change it so that the disk row always reports the disk size. This would fix the disk size, but would leave a confusing 2.2TB partition listed.
You could also get really fancy and detect a valid preclear signature, note it in the FS field and not show any partition.
-
A newly installed WD30EFRX is only being reported as having 2.2TB capacity.
Now I want to nail this. Does df -h show the right size?
Out of town for the weekend. Will let you know asap.
Only waiting for you.... [emoji28]
Right... sorry, i already added the drive to my array
I have a precleared 4TB drive that also shows a disk size of 2.2TB. Like mr-hexen wrote in another post, df -h does not show any information on the disk because it is not mounted. Here is some other information on the disk:
root@uwer:~# lsblk /dev/sdd NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdd 8:48 0 3.7T 0 disk ??sdd1 8:49 0 2T 0 part root@uwer:~# blockdev --getsize64 /dev/sdd 4000787030016 root@uwer:~# blockdev --getsize64 /dev/sdd1 2199023255040 root@uwer:~# /boot/custom/preclear_disk.sh -t /dev/sdd Pre-Clear unRAID Disk /dev/sdd ################################################################## 1.15 Model Family: Seagate Desktop HDD.15 Device Model: ST4000DM000-1CD168 Serial Number: Z300119S LU WWN Device Id: 5 000c50 04f1e88ef Firmware Version: CC43 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes 255 heads, 63 sectors/track, 486401 cylinders, total 7814037168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdd1 1 4294967295 2147483647+ 0 Empty Partition 1 does not start on physical sector boundary. ######################################################################## ========================================================================1.15 == == DISK /dev/sdd IS PRECLEARED with a GPT Protective MBR == ============================================================================
Also saw some strange behavior when testing preclear signature. Disk was in standby when preclear test command was issued. While drive was spinning up, unassigned devices tab was dimmed except for spinning wheel. Then unassigned devices tab refreshed with correct drive size. After another refresh, the unassigned devices tab again shows incorrect drive size.
-
Now I want to check if all files are okay so I type this command in the command line:
rsync -nrcv /mnt/disk1 /mnt/disk10/t >/boot/verify.txt
But all it does was a lot of reading on disk1, no reading on disk10 so disk10 went to sleep after a while. After the process has finished I have a verify.txt file. It seems to contain a list of all files that are an disk1.
A single slash can make a big difference. It should be:
rsync -nrcv /mnt/disk1/ /mnt/disk10/t >/boot/verify.txt
Without the slash, rsync would copy the directory. With the slash rsync would copy the contents of the directory.
@bjp999: Perhaps your instruction post should be edited to include the slash:
(edited)
From a screen session, run this command:
rsync -nrcv /mnt/[source]/ /mnt[dest]/t >/boot/verify.txt
-
I'm not sure diskmv is the best tool to use for switching file systems, especially if you want to follow the recommendation to verify files on the destination disk before deleting files from the source disk. I've updated the first post to try and address this issue.
I think it would be good to have more testing before making a plugin. I only know of the few users that have posted in this thread. Has anyone else out there used these scripts? Any problems?
New users may want to wait until there's been more testing, but if you want to jump in thestewman, these are just bash scripts that you run from the commandline (telnet, ssh or a keyboard/monitor attached to the server). It works much like the preclear script if you used that. There are many ways to set it up, the easiest may be:
[*]Download the latest release zip from github, currently https://github.com/trinapicot/unraid-diskmv/archive/v0.2.0.zip
[*]Extract the zip and copy diskmv and consld8 to the root of your unRAID flash drive
[*]At the command prompt, change to the directory that contains the scripts: cd /boot/
[*]Run the script: diskmv -h
-
would you consider adding an option to exclude a disk to consld8 .
Yes I will consider it.
So I pretty consistently receive the following error. Any thoughts?
I have encountered that type of error in the past with the mover script. I traced it back to cache_dirs but never really understood what was going on. Are you running cache_dirs? For me, running 'ps 7924' (where the number matches the number in the error message) showed it was a cache_dirs process. That's about as far as my understanding of the proc filesystem goes. Maybe it's time for me to do some reading.
-
Thanks for testing and providing feeback switchman.
if you issue the quite "-q" flag to consld8, it does not pass it diskmv when it is called.
Yep, that should happen, but it doesn't. I'll work on that.
consld8 does not like a paren "(" in the subdirectory name. It returns the following error:
-bash: syntax error near unexpected token `('
To get around it, I just escaped the entire sub-directory like this.
consld8 -f -q /mnt/user/Test_Shows/"Archer (2009)" disk3
That is a "feature" of bash (the shell you are using to enter the commands). Spaces and parens are special characters in bash and they must be quoted or escaped when they are in file names. You get the same cryptic error for the same problem in very basic bash commands:
root@tower:~/testdata# ls Archer (2009) -bash: syntax error near unexpected token `(' root@tower:~/testdata# ls "Archer (2009)" Archer\ (2009) root@tower:~/testdata# ls Archer\ \(2009\) Archer\ (2009)It's just something you have to deal with when working at the command line. The "\" tells bash to ignore the special meaning of the next character.
The tab key can be very helpful in these situations, it will autocomplete file names when possible. Type in the first part of the file name then hit tab and it will autocomplete if there is a single file that matches. Hit tab twice and it will list all the possible matches. When it autocompletes it includes the necessary backslashes to escape the special characters.
-
I was trying to follow your code, I am not a coder, and was trying to understand how you use "user, as in "consld8 /mnt/user/video/tv/Wonderfalls disk3". I am assuming you tear it apart and use the actual disk(n) values so that the data is not lost by using the /user paths.
Yes, consld8 calls diskmv and diskmv only moves files from one disk to another. No moves are done to or from the user share.
Any thoughts about the flowchart?
I was trying to follow your flowchart, but I am not a flowcharter.
I think a lot of the logic involving disk usage and free space is already included in consld8. A relatively simple script could be made to loop through all first level subdirectories and perform consld8 on each of them, letting consld8 determine the destination disk. You could even use 'find' and do it with one line (probably).
An alternative solution might be to add a recursive option to consld8. If a disk is not found with enough free space for the specified directory, consld8 is performed on all of the first level subdirectories.
...this could be a totally automated solution that could be scheduled from cron.
I came at these tools from a perspective that day-to-day dispersal of files is controlled by split level settings. If a new season of a tv show starts then a new season folder is created on the same disk as the other seasons of that show. This can be accomplished with split level settings. But sometimes mistakes are made or disks fill up and things need to be rearranged. That is when I intended for diskmv and consld8 to be used.
That being said, if you want to run them on a cron schedule, I won't try to stop you.
-
Based on the comments and suggestions made earlier, I have released updated scripts here:
https://github.com/trinapicot/unraid-diskmv/tree/v0.2.0
This is very much a "use at your own risk" kind of thing, but I am tagging releases that I trust and have used on my own data. But I definitely make sure my data is fully backed up before using it.
Anyone interested enough to test it out?
@NAS & @WeeboTech: The master branch has an option in diskmv to move only small files from one disk to another. If you are interested.
-
'diskmv' is a utility to that moves files. 'consld8' is one of many possible utilities that determine how files should be moved depending on the goal. These many possible utilities then call 'diskmv' to get the job done. mbfrosty's 'rebalance' / 'pressure relief' and lboregard's 'unBalance' / 'disk packer' are other examples that could use 'diskmv' to get their job done. Another I've seen suggested is 'empty disk' which is very similar to 'unBalance', just with more available space.
-
So while the error condition is reported, there is no usage condition that says, there are 3 required parameters and this is how you use the program.
I agree. The error message is not good and that may be a reason to have a separate check for a null value. I will make it better and add some usage output on error.
I will also add the DEBUG snippet.
Thank you.
-
Congratulations these really caught my attention as being genuinely useful and powerful.
I have some ideas for consideration:
Since these tools are so unRAID specific check for the existence of "/etc/unraid-version" as a sanity check before doing anything.
It is not beyond the realms of possibility that a version of unRAID comes out with a missing dependency or a user simply breaks a dependency. To ensure this doesn't end up in unpredictable behavior check before execution i.e. rsync, shopt etc
Consider by default having the rsync "-n, --dry-run " option set in diskmv so that a user essentially has to opt in to a write action (the same logic as you have put in place with -t in consld8).
Consider more script commenting. e.g. i doubt most users will know what this is without googling "-i -dIWRpEAXogt --numeric-ids --inplace" and in general it will help users get a better feel for how it works.
Thanks. All your ideas are under consideration. I especially like the dry run default for diskmv. I was planning to remove the test option as default in consld8, but I now think having them both be "opt in" for write would be better.
-
test for empty arguments. In one script I see $1 and $2 referenced before they are tested with null
Yep, I failed to consider a call like:
diskmv "" disk1 disk2
, but that could be valid usage if you want to move everything from one disk to another, so I think I'll leave it.
consld8 "" disk1
could also be valid, but I don't know if anyone would find that useful.
$2 is not tested with null, but the subsequent variable $SRCDISK is tested to be a valid disk, so a null value in $2 is caught. Is there some reason I need to test $2 with null at first reference?
"$1" should always be encapsulated in quotes to handle spaces.
you could have some disastrous results without it. Especially since you are running as root.
Word splitting is not performed in case statements and assignment statements so
case $1 in ...
and
SRCDISK=$1
are OK. But I will add some more quoting.
Thanks for looking and commenting. I welcome more comments if you look deeper.
-
I've created a couple of bash scripts to facilitate moving files between disks. As these utilities are touching user data, I'd like to have some experienced users review and/or test them. If there is interest and no major problems, I will look into making them easier to use (first a plugin for command line usage, then maybe gui)
-
'diskmv'
will move a user share directory from one disk to another. It uses a find/rsync command similar to what is found in the standard mover script. It is suitable for merging a user share directory onto a disk that already contains that directory. Files that have duplicate file names on the destination disk will not moved. By default, 'diskmv' will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced.
-
'consld8'
can consolidate a user share directory from multiple disks onto one disk. If a destination disk is not specified, it will pick the best disk based on max usage and available space. By default, consld8 will run in test mode and display some information about how directories would be moved, but will not actually move files unless forced.
'diskmv'
is required to actually move files.
Example usage:
diskmv "/mnt/user/video/tv/Pushing Daisies" disk4 disk1 consld8 /mnt/user/video/tv/Wonderfalls disk3
To get get a help message:
diskmv -h
I am tagging releases for code that I trust and have used on my real data. Releases are here: https://github.com/trinapicot/unraid-diskmv/releases
The master branch has code that I have tested on sample data, but I have not necessarily used it on my real data.
These utilities move files around on an unRAID server. I have done my best to prevent any data loss, but there is always a chance something can go wrong. Use at your own risk and please ensure you have everything backed up.
Note for those wishing to use diskmv in the process of changing filesystems:
The diskmv script will not verify that copied files are the same as the source files.
With the default options, diskmv will rsync each file and if the rsync is successful, that file is then deleted before moving to the next file. There is no opportunity to compare the source and destination files. The script is relying on the OS, rsync and whatever else is involved to correctly read data from one disk and write it to another.
If you still want to use diskmv to move all the files and directories from one disk to another, you can use this syntax:
diskmv -f "" disk1 disk2
Or to keep source files (copy instead of move):
diskmv -f -k "" disk1 disk2
-
1
-
-
Thanks for posting bjp999.
The script I downloaded a couple days ago has a preread_skip_pct parameter that is hard coded to 47. This results in almost half of the disk being skipped in the preread stage. I'm guessing that was used for testing and maybe the posted version should be revised to avoid any skipping.
-
...blah blah blah... umask... blah blah blah...
I wasnt sure how to answer this and then I realised why. I dont think this is a best practice recommendation I think it is a feature request and perhaps even a bug report.
Let me raise it as such. Thanks
Thanks NAS. I'm curious why you concluded it was a feature request or bug report.
The permissions of files created from inside a docker container are controlled by settings inside that docker container. If the container creates a file on a volume mapped from unRAID, the umask inside the container sets the permissions, not the unRAID umask. That is why I thought this should be included in the best practices.
I've been playing around a bit and now have a specific example. Running a needo/couchpotato docker with an empty directory mapped to the /config directory results in these new files:
-rw-r--r-- 1 nobody users 8822 Jul 31 13:50 config.ini drwxr-xr-x 1 nobody users 62 Jul 31 13:50 data/
These files are not writable by a user setup in the unRAID GUI.
I cloned the needo/couchpotato git and added a "umask 000" line to the couchpotato.sh script. The results of running that image with another empty config directory:
-rw-rw-rw- 1 nobody users 8822 Jul 31 13:35 config.ini drwxrwxrwx 1 nobody users 62 Jul 31 13:35 data/
These files have the same permissions as those that have been through the New Permissions utility.
by Archivist link=topic=43674.msg416774#msg416774 date=1445968491]
).
diskmv -- A set of utilities to move files between disks
in User Customizations
Posted
The file modified time stamps do not change, but the directory time stamps do change.
The consld8 script uses diskmv which in turn uses rsync to copy files with the -t option flag. The -t option flag preserves the modification times on all files, however, when files are copied to the destination disk and removed from the source disk, the parent directory's modification time is updated by the filesystem.