[Plugin] unbalanced


Recommended Posts

Upgraded to 6.7.0 and unbalance is running normally.

root@lucy:~# uname -a
Linux lucy 4.19.41-Unraid #1 SMP Wed May 8 14:23:25 PDT 2019 x86_64 Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz GenuineIntel GNU/Linux
root@lucy:~# ps aux | grep unbalance
nobody    4808  0.0  0.3 107864 10324 ?        Sl   10:06   0:00 /usr/local/emhttp/plugins/unbalance/unbalance -port 6237
root      5199  0.0  0.0   3908  2228 pts/0    S+   10:10   0:00 grep unbalance

So, for those having issues, try this on the command line

killall unbalance
/usr/local/emhttp/plugins/unbalance/unbalance -port 6237

This is to check the output.

Post the result.

Link to comment
4 hours ago, jbrodriguez said:

Upgraded to 6.7.0 and unbalance is running normally.


root@lucy:~# uname -a
Linux lucy 4.19.41-Unraid #1 SMP Wed May 8 14:23:25 PDT 2019 x86_64 Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz GenuineIntel GNU/Linux
root@lucy:~# ps aux | grep unbalance
nobody    4808  0.0  0.3 107864 10324 ?        Sl   10:06   0:00 /usr/local/emhttp/plugins/unbalance/unbalance -port 6237
root      5199  0.0  0.0   3908  2228 pts/0    S+   10:10   0:00 grep unbalance

So, for those having issues, try this on the command line


killall unbalance
/usr/local/emhttp/plugins/unbalance/unbalance -port 6237

This is to check the output.

Post the result.

killall unbalance
unbalance: no process found
 ✘  root@TheiaHD  /mnt/user  /usr/local/emhttp/plugins/unbalance/unbalance -port 6237
configuration variable provided but not defined: notifyCalc
Usage of /usr/local/emhttp/plugins/unbalance/unbalance:
  -apiFolders="/var/local/emhttp": folders to look for api endpoints
  -checkForUpdate=1: checkForUpdate: 0 - dont' check; 1 (default) - check
  -config="/boot/config/plugins/unbalance/unbalance.conf": config location
  -dryRun=true: perform a dry-run rather than actual work
  -folders="": deprecated - do not use
  -logdir="/boot/logs": pathname where log file will be written to
  -notifyPlan=0: notify via email after plan operation has completed (unraid notifications must be set up first): 0 - No notifications; 1 - Simple notifications; 2 - Detailed notifications
  -notifyTransfer=0: notify via email after transfer operation has completed (unraid notifications must be set up first): 0 - No notifications; 1 - Simple notifications; 2 - Detailed notifications
  -port="6237": port to run the server
  -refreshRate=250: how often to refresh the ui while running a command (in milliseconds)
  -reservedAmount=512: Minimun Amount of space to reserve
  -reservedUnit="Mb": Reserved Amount unit: Mb, Gb or %
  -rsyncArgs="-X": custom rsync arguments
  -verbosity=0: include rsync output in log files: 0 (default) - include; 1 - do not include
2019/05/20 15:11:39 Unable to start the app: configuration variable provided but not defined: notifyCalc

 

Link to comment

Thanks for the reply d2dyno.

 

I recently removed from the code a flag that I had deprecated a long time ago, but it looks like it's still present in users' config file.

 

Try either one of these:

- edit /boot/config/plugins/unbalance/unbalance.conf, remove the line that starts with notifyCalc

- remove /boot/config/plugins/unbalance/unbalance.conf altogether, but you'll lose any custom settings.

 

Then try to stop/start the plugin from the webUI.

 

Let me know if that works.

Link to comment

I opted to remove the whole file, because just removing notifyCalc still left me with an error about 2019/05/20 16:38:26 Unable to start the app: configuration variable provided but not defined: notifyMove.

 

After removing the conf file, unbalance now starts.

Link to comment

You're absolutely right and thanks for the confirmation of the solution !

 

So for anyone with this issue, there are 2 options:

  • edit /boot/config/plugins/unbalance/unbalance.conf, to remove lines that start with notifyCalc, notifyMove and rsyncFlags
  • remove /boot/config/plugins/unbalance/unbalance.conf altogether, but you'll lose any custom settings
Link to comment

I just ran into a bit of a snag with this plugin. When moving a hardlinked file with Gather, I'm essentially still copying the data, as it continues to exist on the source drive in form of the other (but same) hardlinked file elsewhere. Does Unbalance have an option to check if files are hardlinked, and if it finds that a file is indeed hardlinked, to opt to move all these linked files to the target drive? I can't find it in the web UI, so I fear this is not an option at this time. If it's added, perhaps this could be a regularly scheduled task so as to fix the Mover creating the same problem.

Edited by thatsthefrickenlightning
Link to comment

I had a similar issue, but with non hard-linked files. I think it had more to do with rsync itself. I ended up using rclone move to finish the task of removing orphaned/ already copied files from my cache drive (which was a remarkable 2TB).

Link to comment
12 hours ago, jbrodriguez said:

I know very little about hard links and their effect, but rsync has a hard-link option, that you can add as a custom rsync argument in the settings page.

 

I just don't know if that would fix your use case.

Sadly, I in turn know very little about Linux and rsync. I came from Windows just a week ago. ;)

 

I do know hardlinks, though. A hardlink is a sort of false copy. A file that points to the same data on the disk that another file is also pointing to. So you've got the same amount of data taking up space, but twice the amount of files. Or even more than twice, of course. What is an application for this? Well, I keep one hardlink in my torrent seeding folder and one in my plex library, for example. That way I can seed, but I also adhere to Plex's rules conventions regarding folder structures. Radarr and Sonarr have options to use hardlinks instead of copies.

 

So what happens when you delete hardlink A, but not B? Then file A is gone, but the data is not. You've gained no hard disk space. If you delete B but not A? Same deal, one less file, no space gain. Delete both, you gain the extra free space.

 

In my use case, my Plex library is all hardlinks. I can delete my entire library and not bat an eye. Because I'll always have the torrent folders where the data still resides. Conversely, if I wanted to, I could also delete my torrents folder and my data would live on in my Plex library. Hardlinks are super useful.

 

Now here's my problem.

 

As you know, moving with Unbalance is essentially 'copy to other drive and delete source'. But as I've explained, if you delete only one of two hardlinked files, you're not gaining any drive space. But you ARE losing drive space on the other drive because you're writing new data to it. This is how Unbalance's 'move' essentially becomes 'copy' when you're dealing with a hardlinked file.

 

I'm no programmer, but I think I can already envision the solution here. There's a command that you can use to easily check if a file is hardlinked. In a terminal in the folder, type 'stat file.ext'. It'll tell you how many links the data has. 2 links = 2 files pointing to the same data. Second, the size will be exactly the same. I think modifying one of these files means modifying the other as well, but I haven't tried that. No need in my use case. I read-only. Filenames can differ. Aside from that, there are perhaps more ways to check for hardlinks. I know dupeGuru does, but idk how.

 

Anyway, when Unbalance moves a file, it could check the file it's trying to move for hardlinks. And if it finds hardlinks, it could opt to move those along with it.

 

So hardlink /user/mediaserver/torrents/file.ext gets moved to another disk, but hardlink /user/mediaserver/plex/film/file.ext gets moved as well. And of course just as with the former, the exact location, directory-wise, is preserved.

 

Perhaps to simplify the process, in reality Unbalance should just create a new hardlink instead of trying to move an existing one, to avoid the risk of the process tripping up and creating a dupe instead of moving hardlinks.

 

So to summarize, I think Unbalance needs to...

 

Check if /disk1/mediaserver/torrents/file.ext has alter-egos in the form of hardlinks

If yes (and in this case, it does, in the plex folder), remember the location (folder-wise, not disk-wise) and filename of that hardlink alter-ego

Delete this alter-ego/hardlink

Copy /disk1/mediaserver/torrents/file.ext to /disk2/mediaserver/torrents/file.ext, then delete /disk1/mediaserver/torrents/file.ext (this is Move as it already exists now)

Recreate the hardlink deleted off disk1 on disk2, referencing the newly created file /disk2/mediaserver/torrents/file.ext

 

That way, the pitfall of using Move but not freeing up space on the source disk is avoided.

 

I hope this helps explain my situation. I'm sincerely grateful for this cool tool you've made and I hope you can implement an option that'll help me solve my problem. I'd love to hear what you think. Thanks.

 

EDIT: to add, it appears the Mover respects and conserves hardlinks. I just tested this myself. Downloaded a torrent onto the cache drive, hardlink was automatically created on the cache drive, ran the mover, it moved the two files to disk1, and they are still hardlinks. Perhaps you could simply copy Mover's implementation of hardlink conservation?

Edited by thatsthefrickenlightning
  • Upvote 1
Link to comment
  • 2 weeks later...
44 minutes ago, jbrodriguez said:

It doesn't 'balance' data across drives.

 

Scatter will use all the free space on the drive with the most available space, if there's still content left to move, uses all the free space from the drive with the second most available space and so on.

 

I see.  So if i wanted to move 7.5TB off one drive and scatter it across 8 new drives, I'd have to select subfolders and do a different move job for each subfolder?

Link to comment
1 hour ago, jbrodriguez said:

It doesn't 'balance' data across drives.

 

Scatter will use all the free space on the drive with the most available space, if there's still content left to move, uses all the free space from the drive with the second most available space and so on.

 

Ok so even trying to do multiple scatter jobs doesn't seem to work.  The behavior you describe above is not happening.  No matter what, the data is getting moved to disk8 even though it does not have the most free space.  So I'm not sure if I'm doing something wrong or if Unbalance will just always start with the first chosen disk regardless of free space available.

Link to comment

I can't get unBALANCE to start from the WebGUI, but I can get it to start running the following, but then I have to leave a command line window open.

 

/usr/local/emhttp/plugins/unbalance/unbalance -port 6237

 

Linux 5.0.10-Unraid.
Last login: Thu Jun  6 06:55:06 -0700 2019 on /dev/pts/0.
root@Tower:~# /usr/local/emhttp/plugins/unbalance/unbalance -port 6237
I: 2019/06/06 07:03:20 app.go:51: unbalance v5.5.0-1104-b9678b5-v2019.02.12b starting ...
I: 2019/06/06 07:03:20 app.go:59: No config file specified. Using app defaults ...
I: 2019/06/06 07:03:20 server.go:77: Starting service Server ...
I: 2019/06/06 07:03:20 server.go:94: Serving files from /usr/local/emhttp/plugins/unbalance
I: 2019/06/06 07:03:20 array.go:46: starting service Array ...
I: 2019/06/06 07:03:20 planner.go:52: starting service Planner ...
I: 2019/06/06 07:03:20 core.go:101: starting service Core ...
I: 2019/06/06 07:03:20 server.go:145: Server started listening http on :6237
W: 2019/06/06 07:03:20 core.go:116: Unable to read history: open /boot/config/plugins/unbalance/unbalance.hist: no such file or directory
I: 2019/06/06 07:03:20 app.go:73: Press Ctrl+C to stop ...

 

Link to comment
3 hours ago, IamSpartacus said:

No matter what, the data is getting moved to disk8 even though it does not have the most free space

The logs shows the chosen disk(s) and which data is being sent to each disk, in the planning step.

 

It's located at /boot/logs/unbalance.log

 

I'll take a look at it if you send it over (pm if you prefer).

Link to comment
  • 3 weeks later...

I seem to be having an issue with unbalance...  Whenever I run it to move files across my disks, it always ends with an error...

After checking the disk the files were supposed to be moved to, I see it made the directory and sub-directories but there are

no files within them.  It doesn't matter if I use scatter or gather, I end up with the same error...  I have tried deleting the .conf

file, as well as uninstalling and reinstalling the plugin...  

unbalance error.jpg

unbalance.hist

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.