[Plugin] unbalanced


Recommended Posts

  • 2 weeks later...

I started a Gather operation for about 6TB of data. 

It is showing 105hrs remaining and the transfer speed is continuously slowing down (from 105Mbps to now 14.5Mbps)

 

So what happens if i stop the process?

Also the disks support much faster transfer speeds so why is it so slow?

 

I have no docker containers and those disks aren't being used at all over the network.

 

 

 

Link to comment
  • I have three disks + parity and set Unbalance to move about 300GB of files from Disk 1 to Disk 3.
  • I confirmed that I ran the New Permissions script immediately before executing, and I received no permissions warnings when running Plan in Unbalance

  • I have my rsync settings at the default, which seems to just populate the field with -X. I confirmed that during execution it uses rsync -avPR -X "source" "dest" so I think this is set up as intended

  • I have a share called Files that pulls from both disks.

  • Disk 2 is untouched unmodified in this process

  • Unbalance runs, and notes in its email that it has moved about, 300GB of data

  • The free space on Disk 1 goes from 772GB to 928GB, about 156GB removed

  • Disk 3's free space goes from 1417GB down to 1116GB, about 300GB of data added

  • The File share's free space goes from 2441GB to 2295GB, or about 146GB less free space after the move

 

image.thumb.png.b1e5ef943941de17ac92a866a7f1427d.png

 

This seems to me as if Unbalance successfully copied the full 300GB set of data from Disk 1 to Disk 3, but was unable to delete 146GB of data from Disk 1 after moving. I would have expected the share to have the same amount of free space before and after moving, but this does not seem to be the case.

I've confirmed that the syslog doesn't output any errors or other logs during the Move, and I scanned through /boot/logs/unbalance.log to confirm everything looked OK there.

 

I just had a thought that perhaps this has something to do with the sparse files of my VM disk backups. For example, I have a 100GB .vdmk file that only occupies 20GB on disk, and perhaps after rsync it occupies the full 100GB. Put a couple of VM backups together and we have our missing 146GB. In other words, the disk space estimate for Disk 1 was using the occupied space and not the total space, while the rsync move and the files that resulted have matching occupied and total space.

 

I've done some additional testing and confirmed this is what was happening. I had to update the Unbalance rsync settings from the default -X to -X --sparse as I found in the Rsync man page and a forum discussion https://communities.vmware.com/t5/Technical-Community-Resources/copying-virtual-disk-images-with-rsync/m-p/848647 https://linux.die.net/man/1/rsync

 

After making that change in the Unbalance settings and running the transfer again, everything worked exactly as expected. A huge amount of space was freed up on the share, meaning that the VM backups were able to regain their sparseness and only occupy part of the disk. Nice!

 

image.thumb.png.9acf8900b005f49f551445b2130784ca.png

 

So as a warning to people out there: If you use Unraid for virtual machine images or other sparse files, make sure to use --sparse as an option when running Rsync.

  • Like 1
Link to comment
On 11/16/2020 at 3:15 AM, swamiforlife said:

It is showing 105hrs remaining and the transfer speed is continuously slowing down (from 105Mbps to now 14.5Mbps)

Is it possible that one of the disks involved in the operation is not quite healthy ?

 

Slowing down that much feels to me like some disks is having i/o issues.

 

In the unbalance UI, I added the link to a plugin by a fellow forum member that looks at the health of a disk.

 

Can you check that ?

Link to comment
  • 2 weeks later...

Does anyone know of a way to gather all the items for subfolders in a share onto a single drive?

 

Example;

Media/shows/shows folder here

I would like for all the subfolders under shows to be gathered together onto a single disk. This way worst comes to worst and a drive was lost, I know I have all the files for a particular show on a single disk.

 

I have a script that runs regularly that gives me a list of all the folders on each disk, so if this happened I could re-rip any lost shows but this only really works if the shows are stored together on a single disk. If not then I have to go through a much more tedious process.

 

This is also good for drive spin up, as generally we watch 1 or 2 shows at a time and by having them stored on a single disk, less drives will need to be spun up on a regular basis.

 

Yes, I have folder splitting setup now but trying to clean up some items from when it was set incorrectly by mistake (miscalculated the depth).

Edited by TexasUnraid
Link to comment
On 11/17/2020 at 7:43 PM, jbrodriguez said:

Is it possible that one of the disks involved in the operation is not quite healthy ?

 

Slowing down that much feels to me like some disks is having i/o issues.

 

In the unbalance UI, I added the link to a plugin by a fellow forum member that looks at the health of a disk.

 

Can you check that ?

Can you provide the name or a link to the app concerning disk health? Thanks

Link to comment

Is there any critical reason why running UnBalance at the same time as Dockers could cause issues?

 

I'm waiting to move around several TB's of Photography to essentially have "archival" HDD's of older photography I access less often. So I'm not moving any system critical data. So could I potentially still run UnBalance with dockers and VM's on?

Link to comment
On 9/18/2020 at 7:34 AM, titanarchon said:

Thanks jbrodriquez! 

 

Is there a tool for that then and maybe a feature request for Unraid? It doesn't seem far fetched nor substantially difficult to add a "Balance" option as good as this software works and the demand for it has been pretty ongoing from what i have seen in the forums. I would even go as far as to rename the "Scatter" option to "Evacuate" to better illustrate it's function. 

I have the same question. I'm not sure the word "scatter" is the best fit for what the plug-in actually does, more like re-locate, imo. I'd love to see an option like this in your plug-in, no idea if it interests you but +1 from me anyways.

Link to comment
  • 2 weeks later...

So far unbalance did a great job - but since the hard coded and preferred https:// connection requirement is in most main stream browsers things got a bit inconvenient to say the least.

 

Only solution for me so far :

 

1   run the Unraid-Gui and Unbalance in two separate browser tabs in the browsers private mode by right click the UNB button in the Unraid-Gui or

2  use a bookmark for the UNB part.

 

In both cases one needs to click twice for the browser to use the "unsafe" web page ... as it is in private mode one needs to to this in every "new" browser session.

 

 

Bug or feature in UNB ? The elapsed time starts always with some 8 to 12 hours offset rather than with 0:00:00

 

 

At the moment i am "rescuing" a failing 3 TB disk and I observed another strange phenomenon in the Unraid Main view :

 

Take a look at the reads and writes for the 2 parity disks, then the read and writes for disk 10 (source) and disk 01 (target) and all the other disks.

In a perfect world all drives but 10 and 01 should be and stay spun down and the transfer speeds should be between 120 and 180 MByte per second rather than 30-50 MB/sec average.

 

Another nice thing would be nice if the read errors defective files or HDD segments would not be UNB "moved" and one could find out the name of the "lost" files.

 

I attached 2 screen shots to explain what i mean ...

 

cheers

UnRaid.jpg

UNB.jpg

Link to comment
On 9/25/2020 at 5:43 PM, jbrodriguez said:

If Music was a top-level folder (share), per design, the app doesn't delete it.

I have the same problem but not top level share.  /Media/Movies/Movie_name_folder

 

I was only moving the Movie_name_folder and contents from Disk 1 to Disk 3 but Movie_name_folder is still on Disk 1 empty.

 

Edit: I can go back into Disk 1 after to delete the empty Disk 1 /Media/Movies/Movie_name_folder?

Edit2: Went and used Krusader to delete the leftover empty folders on disk 1 and 2.

Edited by Paul_Ber
Link to comment
  • 3 weeks later...
39 minutes ago, BWS001 said:

Likely already know the answer.. but can I run multiple instances of unbalance.. I have just installed a couple 8TB drives... I'm going to retire a number of 2 and 3 TB drives... ( I think at coming up 8 years and 90K hours some of them have been spinning long enough)

 

Even if you could (I am not sure) I wouldn't recommend it.  Specifically, if you have any parity drive in your system then the speed of transfer of files is going to be limited anyway by the need to read both the destination drive and the parity drive before the data is written and the parity updated.  If you run two instances, then the parity drive will spend a lot of time moving the heads between the two areas that are being updated when you are unBalancing moving data to two separate data drives.  The time lost would be much greater than when running the two sessions separately     

Link to comment
On 11/4/2020 at 3:10 AM, Matthew_K said:

QQ or maybe a feature request or maybe I am missing something. Is there a way for Unbalance to automatically rebalance disks based upon the shares what is defined by Include/Exclude Disks? I could probably script some decent logic in bash but I am not sure I want to learn go at this time.

I would love this as an option... from what i understand (which in this as all things is very little) if you could identify things that are not where they should be from the include/exclude you could move them to cache and let the mover sort it out... so it would just be a case of scanning then initiating the copy, and maybe monitor cache usage and pause the move at say 90% to let the mover catch up and restart at say 60%.... it doesn't feel that complex but i'm probably wrong.  

Link to comment
  • 2 weeks later...
On 12/13/2020 at 4:11 AM, kjac said:

I have the same question. I'm not sure the word "scatter" is the best fit for what the plug-in actually does, more like re-locate, imo. I'd love to see an option like this in your plug-in, no idea if it interests you but +1 from me anyways.

I agree with this.  I recently went back to unBALANCE and went to move half of my videos folder (just movies) from my current Videos drive to the other three, with an expectation it would do what unRAID shares does - try to balance how much of a disk is used (depending on settings of course).  It is, really, a "fill up each disk in sequence" effort rather than a re-balance.

So, now I'm gathering everything back onto my Videos drive and will have to manually balance my existing content by creating a new share and copying everything to the new share to let unRAID manage the disk usage balancing.  When finished, I'll have to rename the shares.

Edited by Kaldek
Link to comment
  • 2 weeks later...

Can anyone help with this problem. I am unable to move the data because of these errors. 

Thank you

 

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk1):no-items:currentFree(1.19 TB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk2):no-items:currentFree(504.92 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk3):no-items:currentFree(350.33 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk4):no-items:currentFree(488.64 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk5):no-items:currentFree(782.85 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk6):no-items:currentFree(516.84 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk7):no-items:currentFree(1.17 TB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk8):no-items:currentFree(1.99 TB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk9):no-items:currentFree(1.66 TB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk10):no-items:currentFree(1.81 TB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/disk11):no-items:currentFree(992.71 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:497: =========================================================

I: 2021/02/12 18:45:12 planner.go:498: disk(/mnt/disk12):items(2)-(2.81 TB):currentFree(3.11 TB)-plannedFree(298.37 GB)

I: 2021/02/12 18:45:12 planner.go:499: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:502: [1.51 TB] /mnt/disk1/ServerPlus/./Video Projects

I: 2021/02/12 18:45:12 planner.go:502: [1.30 TB] /mnt/disk1/ServerPlus/./New File System

I: 2021/02/12 18:45:12 planner.go:505: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:506:

I: 2021/02/12 18:45:12 planner.go:508: =========================================================

I: 2021/02/12 18:45:12 planner.go:509: disk(/mnt/cache):no-items:currentFree(450.94 GB)

I: 2021/02/12 18:45:12 planner.go:510: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:511: ---------------------------------------------------------

I: 2021/02/12 18:45:12 planner.go:512:

I: 2021/02/12 18:45:12 planner.go:516: =========================================================

I: 2021/02/12 18:45:12 planner.go:517: Bytes To Transfer: 2.81 TB

I: 2021/02/12 18:45:12 planner.go:518: ---------------------------------------------------------

I: 2021/02/12 18:45:50 core.go:687: Running Move operation ...

I: 2021/02/12 18:45:50 core.go:710: Command Started: (src: /mnt/disk1) rsync -avPR -X "ServerPlus/Video Projects" "/mnt/disk12/"

I: 2021/02/12 18:45:50 core.go:185: Sending operation

I: 2021/02/12 18:45:58 core.go:835: monitor:transfer:(/mnt/disk1/ServerPlus/Video Projects)

W: 2021/02/12 18:46:09 shell.go:51: flag:(rsync: get_xattr_data: lgetxattr("/mnt/disk1/ServerPlus/Video Projects/Music/Patriotic March","user.com.apple.quarantine",67) failed: Input/output error (5)

)

W: 2021/02/12 18:46:10 core.go:763: command:end:error(exit status 13)

I: 2021/02/12 18:46:10 core.go:767: command:retcode(0):exitcode(13)

I: 2021/02/12 18:46:10 core.go:1028: Command Finished

I: 2021/02/12 18:46:10 core.go:1041: Current progress: 53.62% done ~ 17s left (71731.32 MB/s)

W: 2021/02/12 18:46:10 core.go:960: skipping:deletion:(rsync command was flagged):(/mnt/disk12/ServerPlus/Video Projects)

I: 2021/02/12 18:46:10 core.go:710: Command Started: (src: /mnt/disk1) rsync -avPR -X "ServerPlus/New File System" "/mnt/disk12/"

W: 2021/02/12 18:46:10 shell.go:51: flag:(rsync: get_xattr_data: lgetxattr("/mnt/disk1/ServerPlus/New File System/Cellular One","user.DosStream.com.apple.metadata:_kMDItemUserTags:$DATA",1) failed: Input/output error (5)

)

W: 2021/02/12 18:46:10 core.go:763: command:end:error(exit status 13)

I: 2021/02/12 18:46:10 core.go:767: command:retcode(0):exitcode(13)

I: 2021/02/12 18:46:10 core.go:1028: Command Finished

I: 2021/02/12 18:46:10 core.go:1041: Current progress: 100.00% done ~ 0s left (130523.86 MB/s)

W: 2021/02/12 18:46:10 core.go:960: skipping:deletion:(rsync command was flagged):(/mnt/disk12/ServerPlus/New File System)

I: 2021/02/12 18:46:10 core.go:1107:

unBALANCE - MOVE operation completed

Started: Feb 12, 2021 18:45:50

Ended: Feb 12, 2021 18:46:10

Elapsed: 20.531s

Move operation has finished

Transferred 2.81 TB at ~ 130523.70 MB/s

 

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.