[Plugin] unbalanced


Recommended Posts

Any thoughts about this error and have to complete the operation, it no longer wants to...

I: 2018/02/15 10:13:16 core.go:655: Running Move operation ...
I: 2018/02/15 10:13:16 core.go:678: Command Started: (src: /mnt/disk2) rsync -avPRX --append-verify "personal/Documents" "/mnt/disk8/"
I: 2018/02/15 10:13:16 core.go:158: Sending operation
W: 2018/02/15 10:13:17 shell.go:171: rsync:exit:(exit status 23)
W: 2018/02/15 10:13:17 core.go:818: Command Interrupted: rsync -avPRX --append-verify "personal/Documents" "/mnt/disk8/" (exit status 23 : Partial transfer due to error)
I: 2018/02/15 10:13:17 core.go:980:
unBALANCE - MOVE operation INTERRUPTED

I tried the --append-verify switch to no avail.

 

EDIT: Never mind, it seems to be a file with the user.com.apple.quarantine attribute. BTW the log didn't indicate that this way the case, it was only until I ran a rsync from the command line did i finally get to the bottom of this. In any case, when it bombed it ended up creating duplicates, I'm assuming because it couldn't complete - which was a bit of a mess - otherwise a cool app and much needed functionality in unRAID.

Edited by joelones
Link to comment

Glad it 'kind' of worked ! O.o

 

Do you happen to still have the file that caused you trouble ? Maybe a similar file with no personal data ?

 

I'm testing a new approach to invoke rsync from the plugin, which should help with the error you had, so having data that is known to currently fail would be helpful.

 

If anyone else has something similar that can share, I'd appreciate it.

 

Pm if your prefer.

 

I guess I'll have time to work on this in a week or two.

 

Link to comment
  • 2 weeks later...

v2018.02.27 (5.0.0) is out !

 

This release improves rsync execution handling.

 

Any rsync command that generates a code 23 error will be flagged and the source files/folders it refers to, will not be deleted.

 

Use the History page and the log file (/boot/logs/unbalance.log), to check which commands had issues and act accordingly.

 

Once the issue is solved, you can remove the source files/folders, using the 'rmsrc' button in the History page.

 

Also, although the rsync default flags are the same (-avPRX), you can now remove the 'X' flag, if you want.

 

Additional changes:
- Implement /proc based rsync monitoring
- Enable https support
- Fix floating footer issue
- Use date-based version
 

For some more context, this is the log where a command has been flagged:

I: 2018/02/26 15:25:13 core.go:682: Running Move operation ...
I: 2018/02/26 15:25:13 core.go:705: Command Started: (src: /mnt/disk1) rsync -avPRX "anime/Chaos;Child (2017)" "/mnt/disk2/"
W: 2018/02/26 15:25:13 shell.go:51: flagged:(rsync: opendir "/mnt/disk1/anime/Chaos;Child (2017)" failed: Permission denied (13)
)
W: 2018/02/26 15:25:13 core.go:746: command:end:error(exit status 23)
I: 2018/02/26 15:25:13 core.go:750: command:retcode(0):exitcode(23)
I: 2018/02/26 15:25:13 core.go:995: Command Finished
I: 2018/02/26 15:25:13 core.go:1008: Current progress: 28.21% done ~ 1s left (293.21 MB/s)
W: 2018/02/26 15:25:13 core.go:927: skipping:deletion:(rsync command was flagged):(/mnt/disk2/anime/Chaos;Child (2017))
I: 2018/02/26 15:25:13 core.go:705: Command Started: (src: /mnt/disk1) rsync -avPRX "anime/Apocrypha (2017)" "/mnt/disk2/"
W: 2018/02/26 15:25:21 core.go:795: getReadBytes:err(open /proc/7011/io: permission denied):xfer(0)
I: 2018/02/26 15:25:21 core.go:750: command:retcode(0):exitcode(0)
I: 2018/02/26 15:25:21 core.go:995: Command Finished
I: 2018/02/26 15:25:21 core.go:1008: Current progress: 100.00% done ~ 0s left (65.05 MB/s)
I: 2018/02/26 15:25:21 core.go:939: removing:(rm -rf "/mnt/disk1/anime/Apocrypha (2017)")
W: 2018/02/26 15:25:21 core.go:984: skipping:prune:(/mnt/disk1/anime)
I: 2018/02/26 15:25:21 core.go:1074:
unBALANCE - MOVE operation completed


Started: Feb 26, 2018 15:25:13
Ended: Feb 26, 2018 15:25:21

Elapsed: 8.077s

Move operation has finished

Transferred 546.31 MB at ~ 64.50 MB/s

The rsync command was flagged because of permission issues. So, the source folder (/mnt/disk1/anime/Chaos;Child (2017)) wasn't deleted.

 

You could solve permission issues and run the operation again, or deal with it however you find appropriate.

 

The history page will show an yellow warning color for the operation and an 'rmsrc' (remove source) button for the flagged rsync command (shown in the attached image)

 

Once you have solved the issue, you can remove the source folder using the button.

 

Let me know how this works for you.

 

On another note, I'd like to thank gridrunner for his unBALANCE converage in one of his awesome videos.

 

I can't find the forum thread at the moment, but the video is located at

 

Must Have unRAID Plugins - Part 3 Disk Utility Plugins

 

(the discussion specific to unBALANCE starts here).

 

I'm linking to it in the main page.

 

Finally, a small technical note:

 

One of the internal changes in this release, was changing how to monitor rsync progress.

 

Previously, it was: start rsync, parse stdout for progress, end rsync.

Now, it's: start rsync, monitor rsync via /proc, end rsync

 

/proc has tons of info about a process, so it only made sense to do this.

 

The idea was heavily inspired by https://github.com/Xfennec/progress.

 

history.png

  • Like 1
  • Upvote 1
Link to comment

Did something change with this release regarding how the Plan pops up? I used to be able to use this from my iPad but after the last release hitting the Plan button starts the churning in the upper right but I never see the plan and cannot advance to begin the move. I am sad! Lol

Link to comment

Not much to send honestly. I select a drive, it expands, I select a folder (expands fine), select a target drive, Plan is lit, I press Plan and it goes dark while the colored bars in the top right begin “playing”. I never see a dark box displaying progress with the planning as before, the Plan button stays dim, and the page is unresponsive until I refresh which starts it all over again. Does that help? Certainly willing to test if you need the help! I’ll check it on my desktop when I get home too.

Link to comment

It has an air of a crash O.o.

 

I'm not sure if you can open the browser's console to look for errors.

 

Can you check the log (/boot/logs/unbalance.log) ?

 

The last couple of lines should provide some insights.

 

Also, I guess you're running the app from Safari, which is known to have some issues with websockets (used by unbalance). Probably an iOS update made it more incompatible :)

 

Is it possible you test on your ipad with chrome ?

Link to comment

I can test on Chrome but it’ll use the same engine I believe. I don’t know how to access the debug console on a tablet browser. Honestly it doesn’t feel like a crash just an inability to display the dialog. I’ll be home tonight and can check and troubleshoot it further. My tablet is an iOS update behind, my phone is current so I can check both for you. Once I kick it off from my desktop I can access it from anywhere, it’s starting that’s the issue. When I refresh I do see the following message:

{"message":"Not Found"}

But  removing the /transfer from the URL brings the page up fine ?

 

I’ll post an update tonight with what I see. I have a big move from this morning running for another hour or so as I clean drives for crypto. Your tool is far superior to trying to do this with Krusader as that container tends to freeze and move data slowly, rsync rocks! I look forward to using the Gather soon too when this is done too...

Link to comment

Okay, with Chrome on iPad no change. Likewise on my iPhone with an up to date iOS. Logs show:

 

I: 2018/03/09 21:26:07 core.go:170: Sending config
I: 2018/03/09 21:26:07 core.go:175: Sending state
I: 2018/03/09 21:26:07 core.go:185: Sending storage
I: 2018/03/09 22:32:53 core.go:170: Sending config
I: 2018/03/09 22:32:53 core.go:175: Sending state
I: 2018/03/09 22:32:53 core.go:185: Sending storage

 

When my phone attempts to make a plan to Scatter and goes off into the weeds, I think it's not crashing just iOS isn't handling it well. Would love to be able to handle this remotely but I could use Chrome Remote Desktop in a pinch so if no one else is interested I'll understand or help you solve it if you want :)

Link to comment

Unfortunately, I can't provide any additional logs and/or specifics but I twice tried to use the scatter feature to move all data off of one drive and disperse it across several other drives and once ran into the issue of unbalance filling up one drive first ( and quitting due to an out of space error ) and the second attempt,  I needed to kill the rsync processes before it did the same. Which begs the question, how does one kill the current rsync process (or the entire operation) from the GUI, is there a way, or I'm missing something? I tried the little icons before the source field, it didn't do it.

 

To elaborate on my second attempt, I had about 1TB to move off of one drive, so I selected it as the source drive (with all folders selected), I then proceeded to select two drives (with about 900GB free on each) as targets. I did the plan and move and ended up killing unbalance as I got an unRAID low disk warning (>90% full) on the first target drive and nothing got allocated to the second. So when I manually killed unbalance, the first target drive had 175GB left and the second still had ~900GB, I'm sure had I let it run it's course, I would have end the same out of space error as on my first attempt.

 

Perhaps it's just my setup but I'm assuming it's suppose to adequately calculate total moveable GB and plan accordingly taken into consideration the space available on the target disks selected, no? If I do it piecemeal, where the target is one drive then it's fine.

 

Also I notice the footer breaking when source directories are uncollapsed and list is quite long.

 

And I also notice the 

{"message":"Not Found"}

output when I reload the "/transfer" URL from some reason.

 

Thanks for your work. 

Edited by joelones
Link to comment
14 hours ago, joelones said:

Perhaps it's just my setup but I'm assuming it's suppose to adequately calculate total moveable GB and plan accordingly taken into consideration the space available on the target disks selected, no?

 

Yes, but this is an approximation due to this 

My suggestion is for you to increase the minimum reserved space to something 'appropriate'.

 

How far into the copy was the 'out of space' error thrown? That should help you decide how much space to reserve.

 

I'd guess you have a lot of 'small' files ? By that I mean one 4k file occupies 4k on disk, but four 1k files occupy 16k on disk (assuming 4k sectors).

 

14 hours ago, joelones said:

Also I notice the footer breaking when source directories are uncollapsed and list is quite long.

 

I thought I had fixed that :) .. What browser are you using ? I assume you're on the latest unbalance version (5.0.0)

 

14 hours ago, joelones said:

And I also notice the 


{"message":"Not Found"}

 

That's harmless. It's just an issue with the internal app navigation, I'll eventually fix it, but it's just a nuisance right now.

Link to comment
8 hours ago, jbrodriguez said:

 

Yes, but this is an approximation due to this 

My suggestion is for you to increase the minimum reserved space to something 'appropriate'.

 

How far into the copy was the 'out of space' error thrown? That should help you decide how much space to reserve.

 

I'd guess you have a lot of 'small' files ? By that I mean one 4k file occupies 4k on disk, but four 1k files occupy 16k on disk (assuming 4k sectors).

 

 

I thought I had fixed that :) .. What browser are you using ? I assume you're on the latest unbalance version (5.0.0)

 

 

That's harmless. It's just an issue with the internal app navigation, I'll eventually fix it, but it's just a nuisance right now.

 

Thanks, concerning the footer, I'm on Chrome and yes, using v5.

Link to comment

On my Chrome equipped desktop I also see where expanding a large number of directories expands into the disks listed below if that's what you're describing above. I generally try to move at the top level to avoid this. I have seen the size issue as well and increased my minimum space. It doesn't seem consistent though and  appears better with this last release? I've been careful to move drive to drive and not give it much space to have to calculate on it's own :)

Link to comment

Can you describe how is your content organized ?

 

If you have something like /mnt/user/movies as a share, with individual movie folders below it, if you mark movies, it should use each movie folder as a separate unit to be allocated to different disks.

 

I'm just trying to better understand your scenario.

Link to comment
4 hours ago, jbrodriguez said:

Can you describe how is your content organized ?

 

If you have something like /mnt/user/movies as a share, with individual movie folders below it, if you mark movies, it should use each movie folder as a separate unit to be allocated to different disks.

 

I'm just trying to better understand your scenario.

 

I have a share called media, and under that I have 2 sub folders called Movies and Tv Shows. When I select "media" I will get an "not enough space" error because the total value is bigger than the max avalible space on any drive. If you instead pick "Movies" or even the individual folders inside it, it works, because then it's smart enough to understand it can move it to multiple disks. This is fine but if I had to mark every single movie it would be very tedious.

 

Spaceinvader One touched in this subject in his most recent video between 4:00-5:10: 

 

Edit: Bonus question, is there any single button to cancel the current job?

Edited by huffsper
Link to comment

@gridrunner does make awesome videos ! ?

 

In the video, he selected each separate tvshow folder, which is definitely tedious.

 

But he could have selected the parent TVShows folder and it would have worked (similar to what you mentioned).

 

That is:

- 1st attempt - Select /Media

unbalance looks for all /Media subfolders, finds only TVshows, which occupies, I don't know, 3.5Tb, uses it as an allocation unit and finds that no disk has 3.5Tb free. That's the reason for the message

 

- Hypothetical 2nd attempt - Select /Media/TVShows

unbalance looks for all /Media/TVShows subfolders, finds Billions (40Gb), The Walking Dead (35GB), etc. Each of these tv shows folders are treated as an allocation unit, and it's a lot easier to find space for them on the different disks.

 

I'm pretty sure I'm not presenting this concept correctly in the app's onboarding/docs (if at all?), but .. I haven't found a better way to do it yet O.o :D

 

Any suggestions will be appreciated.

 

As for the stop button, there isn't one. It's on the to-do list though.

Link to comment
37 minutes ago, jbrodriguez said:

Any suggestions will be appreciated.

 

Does the utility take into account the split level of each share?  If not, rather than looking at the next level, could it drill down to the correct split level?

 

In your example above, 1st attempt would automatically read the same data as 2nd attempt.

 

Appreciate that might complicate things quite a bit though.

Link to comment

Allright thanks for the answer! Just wanted to report it to see if it was something you were working on for a future release.

 

Final question: you say that 5.0 supports https but I must be missing some setting. Going to https://UNRAIDIP:6237/ gives me ERR_SSL_PROTOCOL_ERROR. I'm using a self signed certificate for unRAID instead of Let's encrypt if that matters.

Edited by huffsper
Link to comment
58 minutes ago, huffsper said:

Allright thanks for the answer! Just wanted to report it to see if it was something you were working on for a future release.

 

Final question: you say that 5.0 supports https but I must be missing some setting. Going to https://UNRAIDIP:6237/ gives me ERR_SSL_PROTOCOL_ERROR. I'm using a self signed certificate for unRAID instead of Let's encrypt if that matters.

6237 is an unusual port for https (it is normally 443 by default).    Is that what you have set under Settings->Identification?

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.