[Plugin] unbalanced


Recommended Posts

4 hours ago, SavellM said:

@jbrodriguez quick question, if I only have 1 user account (root) would running this as nobody have any issues?

 

'nobody' is present by default in linux, so it shouldn't be an issue.

 

4 hours ago, SavellM said:
4 hours ago, AnnabellaRenee87 said:


I often wondered this too. I just run with "nobody" and run Docker Safe Permissions after moving things and wonder if using root would be a good idea or not to avoid running Docker Safe Permissions

Sent from my SM-G955U using Tapatalk
 

Ye thats my thoughts too

 

'root' arguably has full access to the entire filesystem, so it should be able to read/copy any file and also delete them, if doing a move operation.

 

I run unBALANCE as nobody and have run it with warnings, but this is because I knew the issues were about files/folder permissions not being exactly as expected from Docker Safe New Perms, not about owner/groups issues.

 

The warning to run Docker Safe New Perms is general, since there are potentially many different scenarios.

 

Link to comment
3 hours ago, Phastor said:
6 hours ago, jbrodriguez said:

 

No, it works at the disk level (/mnt/disk*), not at a user shares level (/mnt/user), so it doesn't know about allocation method nor split levels.

 

Any plans for it in the future, if it's even possible?

 

Not sure if it makes sense, to be honest. Will look into it.

 

Link to comment
14 hours ago, jbrodriguez said:

 

Not sure if it makes sense, to be honest. Will look into it.

 

 

Thanks!

 

Here's a scenario I was thinking of. I currently have a share called movies. Under that is a folder for each movie where the movie itself, subs, posters, etc reside. I have split levels set to only split the top level directory. This way each movie would reside on the same disk as its subs and whatnot. I wouldn't want a movie and it's related files stretched across multiple disks.

 

If I felt the need to scatter the movies folder, say after I've gathered it to one disk for whatever reason, I would want scatter to follow the split levels so again the individual moves would stay together with their respective files. Or am I misunderstanding how this works?

Link to comment
21 hours ago, Phastor said:

so again the individual moves would stay together with their respective files

 

If you have 

movies
movies/avatar/avatar.mkv
movies/avatar/avatar.sub
movies/avatar/poster.jpg
movies/it/it.mkv
movies/it/it.sub
movies/it/poster.jpg

and you select the movies share in the scatter page, it would move the 'avatar' and 'it' folders whole to the target disk.

 

Probably 'avatar' would end in disk1 and 'it' in disk2 (as an example), depending on available space but each folder is copied whole.

 

 

Link to comment
15 hours ago, jbrodriguez said:

 

If you have 


movies
movies/avatar/avatar.mkv
movies/avatar/avatar.sub
movies/avatar/poster.jpg
movies/it/it.mkv
movies/it/it.sub
movies/it/poster.jpg

and you select the movies share in the scatter page, it would move the 'avatar' and 'it' folders whole to the target disk.

 

Probably 'avatar' would end in disk1 and 'it' in disk2 (as an example), depending on available space but each folder is copied whole.

 

Gotcha. I thought it would recursively scatter all individual files. Thanks for the clarification!

Edited by Phastor
Link to comment
  • 2 weeks later...

Unbalance is stopping a couple of hours into a move.  I get an error like this:

 

I: 2017/10/19 00:17:58 core.go:944: Command Started: (src: /mnt/disk4) rsync -avPRX "TV/Angel" "/mnt/disk1/"
W: 2017/10/19 00:17:58 shell.go:176: rsync:exit:(exit status 23)
W: 2017/10/19 00:17:58 core.go:1016: Command Interrupted: rsync -avPRX "TV/Angel" "/mnt/disk1/" (exit status 23 : Partial transfer due to error)

 

rsync man page is unhelpful:

23     Partial transfer due to error

Anyone have any idea where I could look next on this?  

I'm starting to fear a disk issue.

Link to comment

Hi. I installed this quite a while ago, had it working, but haven't played with it for a long time. Now, I can't seem to get it to start.

59efcbc14e9b4_URunBalance.thumb.PNG.3552481ab5a07504670327872c361eb6.PNG

As you can see in the screen shot, the server is running, the "Apply" button is disabled (because I just clicked it), but it still says the server is stopped. I've enabled & disabled several times, refreshed the page, and attempted to go to tower:6237. Clicking on Settings > Utilities > unBalance brings me to the settings screen you see here, as does Plugins > Installed Plugins > unBalance. Neither of these gives me a docker-style drop-down menu with an option to Open Web UI.

 

On a whim, I decided to uninstall/reinstall, and now it's started properly. I know I've had this installed for probably a year or so, and it seems that something got a little wonky during all the updates.

 

Nothing really for you to "fix", just something to be aware of. Let it rattle around at the back of the brain, and maybe you'll notice something at some point when you're digging in the code.

Link to comment

OK, so now I seem to be having a real issue.

 

For the most part, this seems to be working very well, however, it's leaving some files behind on the disk(s) it's supposed to be moving files off of.

 

Here's a snippet of my Movies share showing poor disk organization

59efd907ad348_URunBalancegather1start.thumb.PNG.08621fc2caf969487eb926e462cc2d98.PNG

 

I selected these files to be Gathered. It showed what it was going to process, but the display didn't seem quite right (note single disks - some of which don't match what was listed in the unRAID GUI share locations - listed in Location, and some Locations are blank). I boldly went forward.

59efd9394333f_URunBalancegather2setupissue.PNG.32ef640c48a36816fecb3741142414dc.PNG

 

I selected disk5 as my target - it looks like there will be plenty of space left there when it's done. (I've adjusted the min-free to be 5GB, just for some breathing room.)

59efd96ee1ad6_URunBalancegather3target.thumb.PNG.f8d88deb55de039d52f36cfab2b060ff.PNG

and hit move.

 

It stopped with this error

 W: 2017/10/24 20:14:02 core.go:1016: Command Interrupted: rsync -avPRX "Movies/2010 The Year We Make Contact (1984)/.actors" "/mnt/disk5/" (exit status 23 : Partial transfer due to error) 

which I noticed was mentioned by someone else, above. I've not researched it yet, but I'll be fair and not include that one in this "complaint". Oddly, though, the ../2010.../.actors directory doesn't exist on disk11 any more. The directory Movies/2010 does exist there, but it's empty (according to MC). Movies/2001 did get completely moved from disk11 to disk5, but it left the empty top-level directory behind so unRAID still shows that there's usage on disk11 (since I failed to include that in my screen shot). For the directories it was supposed to have moved ("13 Going on 30" through "1969"), here's what the unRAID GUI is showing for share disk usage now.

59efda17c156a_URunBalancegather4finish.PNG.b95aebe7c4e4797d2e515af45f88cda4.PNG

 

4 out of the 5 it should have completed are showing that they have data still on 2 disks. I've confirmed that in MC by going to the /mnt/diskx directories and noted that these folders exist, though they are empty now. (After the first shot at this, there were some files left in some of them, though I don't recall which ones.)

 

Here is the complete log as shared via the GUI when unBalance completed its tasks


I: 2017/10/24 20:12:05 unraid.go:471: Id(11); Name(disk11); Path(/mnt/disk11); Device(sdd); Type(Data); FsType(xfs); Free(1.95 TB); NewFree(1.90 TB); Size(4.00 TB); Serial(HGST_HMS5C4040ALE640_PL2331LAG6W5WJ); Status(DISK_OK); Bin(&{43043129538 [0xc4202c2100 0xc4202c2140 0xc4202c2180 0xc4202c21c0 0xc4202c2200 0xc4202c22c0 0xc42004c280 0xc42004c2c0 0xc42004c300 0xc42004c340 0xc42004c380 0xc42004c400 0xc42004c440 0xc42004c480 0xc42004c4c0 0xc42004c500 0xc42004c540 0xc42004c580 0xc42004c5c0 0xc42004c600 0xc42004c640 0xc42004c700 0xc42004c740 0xc42004c780 0xc42004c7c0 0xc42004c840 0xc42004c880 0xc42004c8c0 0xc42004c980 0xc42004c9c0 0xc42004ca40 0xc42004ca80 0xc42004cb00 0xc42004cb40 0xc42004cb80 0xc42004cbc0 0xc42004cc00 0xc42004ccc0 0xc42004cd00 0xc42004cd80 0xc42004cdc0 0xc42004cec0 0xc42004cf00 0xc42004cf40 0xc42004d040 0xc42004d0c0 0xc42004d100 0xc42004d200 0xc42004d240 0xc42004d280 0xc42004d300 0xc42004d340 0xc4202c2440 0xc4202c24c0 0xc4202c2540 0xc4202c2580 0xc42004d4c0 0xc42004d500 0xc42004d580 0xc42004d5c0 0xc42004d600 0xc42004d680 0xc42004d6c0 0xc42004d700 0xc42004d740 0xc42004d780 0xc42004d7c0 0xc42004d800 0xc42004d840 0xc42004d880 0xc42004d8c0 0xc42004d900 0xc42004d940 0xc42004d980 0xc42004d9c0 0xc42004da00 0xc42004da80 0xc42004dac0 0xc4202c2680 0xc4202c26c0 0xc4202c2700 0xc4202c2740 0xc4202c2800 0xc4202c2880 0xc4202c28c0 0xc4202c2900 0xc4202c2940 0xc4202c2980 0xc4202c29c0 0xc4202c2a00 0xc4202c2a40 0xc4202c2a80 0xc4202c2ac0 0xc4202c2b00 0xc4202c2b40 0xc4202c2040 0xc4202c2080 0xc4202c20c0 0xc42004c1c0 0xc42004c200]})

I: 2017/10/24 20:12:05 unraid.go:471: Id(12); Name(disk12); Path(/mnt/disk12); Device(sdf); Type(Data); FsType(xfs); Free(2.00 TB); NewFree(1.96 TB); Size(4.00 TB); Serial(HGST_HMS5C4040ALE640_PL1331LAHE2R0H); Status(DISK_OK); Bin(&{43043129610 [0xc4202c2100 0xc4202c2140 0xc4202c2180 0xc4202c21c0 0xc4202c2200 0xc4202c22c0 0xc42004c280 0xc42004c2c0 0xc42004c300 0xc42004c340 0xc42004c380 0xc42004c400 0xc42004c440 0xc42004c480 0xc42004c4c0 0xc42004c500 0xc42004c540 0xc42004c580 0xc42004c5c0 0xc42004c600 0xc42004c640 0xc42004c700 0xc42004c740 0xc42004c780 0xc42004c7c0 0xc42004c840 0xc42004c880 0xc42004c8c0 0xc42004c980 0xc42004c9c0 0xc42004ca40 0xc42004ca80 0xc42004cb00 0xc42004cb40 0xc42004cb80 0xc42004cbc0 0xc42004cc00 0xc42004ccc0 0xc42004cd00 0xc42004cd80 0xc42004cdc0 0xc42004cec0 0xc42004cf00 0xc42004cf40 0xc42004d040 0xc42004d0c0 0xc42004d100 0xc42004d200 0xc42004d240 0xc42004d280 0xc42004d300 0xc42004d340 0xc4202c2440 0xc4202c24c0 0xc4202c2540 0xc4202c2580 0xc42004d4c0 0xc42004d500 0xc42004d580 0xc42004d5c0 0xc42004d600 0xc42004d680 0xc42004d6c0 0xc42004d700 0xc42004d740 0xc42004d780 0xc42004d7c0 0xc42004d800 0xc42004d840 0xc42004d880 0xc42004d8c0 0xc42004d900 0xc42004d940 0xc42004d980 0xc42004d9c0 0xc42004da00 0xc42004da80 0xc42004dac0 0xc4202c2680 0xc4202c26c0 0xc4202c2700 0xc4202c2740 0xc4202c2800 0xc4202c2880 0xc4202c28c0 0xc4202c2900 0xc4202c2940 0xc4202c2980 0xc4202c29c0 0xc4202c2a00 0xc4202c2a40 0xc4202c2a80 0xc4202c2ac0 0xc4202c2b00 0xc4202c2b40 0xc4202c2040 0xc4202c2080 0xc4202c20c0 0xc42004c1c0 0xc42004c200 0xc4202c2300 0xc4202c2380 0xc4202c23c0 0xc4202c2400 0xc4202c2600 0xc4202c2640 0xc4202c2840 0xc42004d400]})

I: 2017/10/24 20:12:05 unraid.go:471: Id(30); Name(cache); Path(/mnt/cache); Device(sdh); Type(Cache); FsType(btrfs); Free(150.06 GB); NewFree(107.01 GB); Size(240.06 GB); Serial(PNY_CS1311_240GB_SSD_PNY3916219312020272B); Status(DISK_OK); Bin(&{43043129610 [0xc4202c2100 0xc4202c2140 0xc4202c2180 0xc4202c21c0 0xc4202c2200 0xc4202c22c0 0xc42004c280 0xc42004c2c0 0xc42004c300 0xc42004c340 0xc42004c380 0xc42004c400 0xc42004c440 0xc42004c480 0xc42004c4c0 0xc42004c500 0xc42004c540 0xc42004c580 0xc42004c5c0 0xc42004c600 0xc42004c640 0xc42004c700 0xc42004c740 0xc42004c780 0xc42004c7c0 0xc42004c840 0xc42004c880 0xc42004c8c0 0xc42004c980 0xc42004c9c0 0xc42004ca40 0xc42004ca80 0xc42004cb00 0xc42004cb40 0xc42004cb80 0xc42004cbc0 0xc42004cc00 0xc42004ccc0 0xc42004cd00 0xc42004cd80 0xc42004cdc0 0xc42004cec0 0xc42004cf00 0xc42004cf40 0xc42004d040 0xc42004d0c0 0xc42004d100 0xc42004d200 0xc42004d240 0xc42004d280 0xc42004d300 0xc42004d340 0xc4202c2440 0xc4202c24c0 0xc4202c2540 0xc4202c2580 0xc42004d4c0 0xc42004d500 0xc42004d580 0xc42004d5c0 0xc42004d600 0xc42004d680 0xc42004d6c0 0xc42004d700 0xc42004d740 0xc42004d780 0xc42004d7c0 0xc42004d800 0xc42004d840 0xc42004d880 0xc42004d8c0 0xc42004d900 0xc42004d940 0xc42004d980 0xc42004d9c0 0xc42004da00 0xc42004da80 0xc42004dac0 0xc4202c2680 0xc4202c26c0 0xc4202c2700 0xc4202c2740 0xc4202c2800 0xc4202c2880 0xc4202c28c0 0xc4202c2900 0xc4202c2940 0xc4202c2980 0xc4202c29c0 0xc4202c2a00 0xc4202c2a40 0xc4202c2a80 0xc4202c2ac0 0xc4202c2b00 0xc4202c2b40 0xc4202c2040 0xc4202c2080 0xc4202c20c0 0xc42004c1c0 0xc42004c200 0xc4202c2300 0xc4202c2380 0xc4202c23c0 0xc4202c2400 0xc4202c2600 0xc4202c2640 0xc4202c2840 0xc42004d400]})

I: 2017/10/24 20:12:59 core.go:1366: {"id":5,"name":"disk5","path":"/mnt/disk5","device":"sdi","type":"Data","fsType":"xfs","free":11572387840,"newFree":9141918081,"size":3998833471488,"serial":"HGST_HDN724040ALE640_PK1338P4GT2D9B","status":"DISK_OK","src":false,"dst":false}

I: 2017/10/24 20:12:59 core.go:811: Running Move operation ...

I: 2017/10/24 20:12:59 core.go:827: dry-run OFF

I: 2017/10/24 20:12:59 core.go:944: Command Started: (src: /mnt/disk1) rsync -avPRX "Movies/321 Penguins Save The Planets/321 Penguins Save The Planets-fanart.jpg" "/mnt/disk5/"

I: 2017/10/24 20:12:59 core.go:1026: Command Finished

I: 2017/10/24 20:12:59 core.go:1032: Current progress: 0.00% done ~ 10h38m36s left (0.06 MB/s)

I: 2017/10/24 20:12:59 core.go:1054: Removing: rm -rf "/mnt/disk1/Movies/321 Penguins Save The Planets/321 Penguins Save The Planets-fanart.jpg"

I: 2017/10/24 20:12:59 core.go:1072: Running find "/mnt/disk1/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:12:59 core.go:944: Command Started: (src: /mnt/disk1) rsync -avPRX "Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.fanart.jpg" "/mnt/disk5/"

I: 2017/10/24 20:12:59 core.go:1026: Command Finished

I: 2017/10/24 20:12:59 core.go:1032: Current progress: 0.00% done ~ 5h18m14s left (0.12 MB/s)

I: 2017/10/24 20:12:59 core.go:1054: Removing: rm -rf "/mnt/disk1/Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.fanart.jpg"

I: 2017/10/24 20:12:59 core.go:1072: Running find "/mnt/disk1/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:12:59 core.go:944: Command Started: (src: /mnt/disk1) rsync -avPRX "Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.mkv" "/mnt/disk5/"

I: 2017/10/24 20:14:01 core.go:1026: Command Finished

I: 2017/10/24 20:14:01 core.go:1032: Current progress: 99.99% done ~ 0s left (37.02 MB/s)

I: 2017/10/24 20:14:01 core.go:1054: Removing: rm -rf "/mnt/disk1/Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.mkv"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk1/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk1) rsync -avPRX "Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.tbn" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 99.99% done ~ 0s left (36.67 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk1/Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.tbn"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk1/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk1) rsync -avPRX "Movies/321 Penguins Save The Planets/Thumbs.db" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 99.99% done ~ 0s left (36.65 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk1/Movies/321 Penguins Save The Planets/Thumbs.db"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk1/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk3) rsync -avPRX "Movies/321 Penguins Save The Planets/.actors" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.60 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk3/Movies/321 Penguins Save The Planets/.actors"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk3/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk6) rsync -avPRX "Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.nfo" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.51 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk6/Movies/321 Penguins Save The Planets/321 Penguins Save The Planets.nfo"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk6/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk6) rsync -avPRX "Movies/321 Penguins Save The Planets/fanart.jpg" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.48 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk6/Movies/321 Penguins Save The Planets/fanart.jpg"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk6/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk6) rsync -avPRX "Movies/321 Penguins Save The Planets/movie.tbn" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.46 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk6/Movies/321 Penguins Save The Planets/movie.tbn"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk6/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk9) rsync -avPRX "Movies/321 Penguins Save The Planets/backdrop.jpg" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.43 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk9/Movies/321 Penguins Save The Planets/backdrop.jpg"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk9/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk9) rsync -avPRX "Movies/321 Penguins Save The Planets/movie.xml" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.42 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk9/Movies/321 Penguins Save The Planets/movie.xml"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk9/Movies/321 Penguins Save The Planets" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk11) rsync -avPRX "Movies/17 Again (2009)/.actors" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.40 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk11/Movies/17 Again (2009)/.actors"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk11/Movies/17 Again (2009)" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk11) rsync -avPRX "Movies/2010 The Year We Make Contact (1984)/extrafanart" "/mnt/disk5/"

I: 2017/10/24 20:14:02 core.go:1026: Command Finished

I: 2017/10/24 20:14:02 core.go:1032: Current progress: 100.00% done ~ 0s left (36.32 MB/s)

I: 2017/10/24 20:14:02 core.go:1054: Removing: rm -rf "/mnt/disk11/Movies/2010 The Year We Make Contact (1984)/extrafanart"

I: 2017/10/24 20:14:02 core.go:1072: Running find "/mnt/disk11/Movies/2010 The Year We Make Contact (1984)" -type d -empty -prune -exec rm -rf {} \;

I: 2017/10/24 20:14:02 core.go:944: Command Started: (src: /mnt/disk11) rsync -avPRX "Movies/2010 The Year We Make Contact (1984)/.actors" "/mnt/disk5/"

W: 2017/10/24 20:14:02 shell.go:176: rsync:exit:(exit status 23)

W: 2017/10/24 20:14:02 core.go:1016: Command Interrupted: rsync -avPRX "Movies/2010 The Year We Make Contact (1984)/.actors" "/mnt/disk5/" (exit status 23 : Partial transfer due to error)

I: 2017/10/24 20:14:02 core.go:1162:

unBALANCE - MOVE operation INTERRUPTED

Started: Oct 24, 2017 20:12:59

Ended: Oct 24, 2017 20:14:02

Elapsed: 1m3.909s

Command Interrupted: rsync -avPRX "Movies/2010 The Year We Make Contact (1984)/.actors" "/mnt/disk5/" (exit status 23 : Partial transfer due to error)

Transferred 2.43 GB at ~ 36.27 MB/s

I: 2017/10/24 20:14:04 unraid.go:458: Unraid Box Condition: &{NumDisks:13 NumProtected:0 Synced:2017-10-01 02:30:01 -0400 EDT SyncErrs:0 Resync:0 ResyncPos:0 State:STARTED Size:28231945736192 Free:4832802639872 NewFree:4832802639872}

I: 2017/10/24 20:14:04 unraid.go:471: Id(1); Name(disk1); Path(/mnt/disk1); Device(sdn); Type(Data); FsType(xfs); Free(31.60 GB); NewFree(31.60 GB); Size(999.72 GB); Serial(WDC_WD10EARS-00Y5B1_WD-WCAV5U070002); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(2); Name(disk2); Path(/mnt/disk2); Device(sdo); Type(Data); FsType(xfs); Free(23.24 GB); NewFree(23.24 GB); Size(2.00 TB); Serial(ST32000542AS_5XW27ZVJ); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(3); Name(disk3); Path(/mnt/disk3); Device(sdj); Type(Data); FsType(xfs); Free(12.01 GB); NewFree(12.01 GB); Size(999.72 GB); Serial(Hitachi_HDT721010SLA360_STF604MH0EN8XB); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(4); Name(disk4); Path(/mnt/disk4); Device(sdl); Type(Data); FsType(xfs); Free(10.19 GB); NewFree(10.19 GB); Size(2.00 TB); Serial(ST32000542AS_5XW2EWWX); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(5); Name(disk5); Path(/mnt/disk5); Device(sdi); Type(Data); FsType(xfs); Free(9.14 GB); NewFree(9.14 GB); Size(4.00 TB); Serial(HGST_HDN724040ALE640_PK1338P4GT2D9B); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(6); Name(disk6); Path(/mnt/disk6); Device(sdc); Type(Data); FsType(xfs); Free(16.94 GB); NewFree(16.94 GB); Size(2.00 TB); Serial(WDC_WD20EARX-00PASB0_WD-WCAZAJ031431); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(7); Name(disk7); Path(/mnt/disk7); Device(sde); Type(Data); FsType(xfs); Free(591.87 GB); NewFree(591.87 GB); Size(2.00 TB); Serial(ST2000DM001-1CH164_W2418WLL); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(8); Name(disk8); Path(/mnt/disk8); Device(sdk); Type(Data); FsType(xfs); Free(8.65 GB); NewFree(8.65 GB); Size(2.00 TB); Serial(WDC_WD20EFRX-68AX9N0_WD-WMC1T0990614); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(9); Name(disk9); Path(/mnt/disk9); Device(sdm); Type(Data); FsType(xfs); Free(4.75 GB); NewFree(4.75 GB); Size(999.72 GB); Serial(ST1000DM003-1CH162_S1DE5H8K); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(10); Name(disk10); Path(/mnt/disk10); Device(sdg); Type(Data); FsType(xfs); Free(29.20 GB); NewFree(29.20 GB); Size(3.00 TB); Serial(WDC_WD30EZRX-00D8PB0_WD-WMC4N0533589); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(11); Name(disk11); Path(/mnt/disk11); Device(sdd); Type(Data); FsType(xfs); Free(1.95 TB); NewFree(1.95 TB); Size(4.00 TB); Serial(HGST_HMS5C4040ALE640_PL2331LAG6W5WJ); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(12); Name(disk12); Path(/mnt/disk12); Device(sdf); Type(Data); FsType(xfs); Free(2.00 TB); NewFree(2.00 TB); Size(4.00 TB); Serial(HGST_HMS5C4040ALE640_PL1331LAHE2R0H); Status(DISK_OK); Bin(<nil>)

I: 2017/10/24 20:14:04 unraid.go:471: Id(30); Name(cache); Path(/mnt/cache); Device(sdh); Type(Cache); FsType(btrfs); Free(150.06 GB); NewFree(150.06 GB); Size(240.06 GB); Serial(PNY_CS1311_240GB_SSD_PNY3916219312

 

Again, I'll read the posts on the "Exit status 23", but the rest of this is a bit confusing. If there's anything else I can provide to help trouble shoot, please let me know.

 

Edited by FreeMan
add details after the error message
Link to comment
11 hours ago, jbrodriguez said:

Pm if you prefer.

 

Done. Thanks!

 

Also, when you get a chance, you may want to take a look at the floating bottom bar - it doesn't seem to float too well. ;)

 

59f10c959ed36_URunBalanceUIissue.PNG.c8b66e1542aa2dcf323ab4fd97f6889a.PNG

 

Firefox 56.0.1 64-bit on Win10 64-bit. AdBlock Plus installed but disabled on entire unRAID IP address

Edited by FreeMan
Link to comment
12 hours ago, jbrodriguez said:

Just as a quick check: does it work on Edge ?

 

NO: Edge 40.15063.674.0 (ugh, I felt dirty using that. Had to rummage around in the Start menu for a while to find it...)

YES: Chrome 61.0.3163.100 (Official Build) (64-bit)

YES: Chrome 62.0.3202.75 (Official Build) (64-bit)  - since it updated itself while I was checking the version

 

Neither Edge nor Firefox even show the scroll bar that Chrome shows for the "Shares Explorer". When I scroll down the Selected & Location portions roll off the screen, too.

Link to comment

Ok, I know what happened with the issue you reported, FreeMan.

 

The bottom line is that no files/folders were left behind by the app, it simply didn't even consider them as part of the job.

 

Let me explain.

 

Given a folder selected to be transferred (either for scatter or gather), the app creates a list of its first level children and later executes an rsync command for each one of them.

If there are no children (the folder is empty), the app ignores the folder: there's no work to be done here.

 

But, especially in the case of gather, where one would reasonably want the (empty) folder to be removed, this is not the correct behavior.

 

It's an edge case that I hadn't considered and needs to be looked into.

 

Let me show you how this applies to your case.

 

Just for context (and based on the logs), you made 3 runs of the app, the first one successful, the other two interrupted by rsync's error 23.

 

This is how it played out for some of the movies you were gathering.


- Movies/13 Going on 30

 

I'll break it down by run (it wasn't included in the first run, so I'll start with the second, gather to disk5)

 

You had files/folders in disk1, disk3, disk6 and disk11. Particularly disk11, had two 6 bytes folders, one of which caused the error 23.

Second run
I: 2017/10/24 19:51:30 core.go:656: getFolders:Readdir(9)
I: 2017/10/24 19:51:30 core.go:666: getFolders:Executing find "/mnt/disk1/Movies/13 Going on 30/." ! -name . -prune -exec du -bs {} +
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 747502	/mnt/disk1/Movies/13 Going on 30/./13 Going on 30-fanart.jpg
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 168352	/mnt/disk1/Movies/13 Going on 30/./13 Going on 30.fanart.jpg
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 904917533	/mnt/disk1/Movies/13 Going on 30/./13 Going on 30.mp4
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 170020	/mnt/disk1/Movies/13 Going on 30/./13 Going on 30.tbn
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 38400	/mnt/disk1/Movies/13 Going on 30/./Thumbs.db
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 120161	/mnt/disk1/Movies/13 Going on 30/./fanart.jpg
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 170020	/mnt/disk1/Movies/13 Going on 30/./folder.jpg
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 143184	/mnt/disk1/Movies/13 Going on 30/./movie.tbn
I: 2017/10/24 19:51:30 core.go:669: getFolders:find(/mnt/disk1/Movies/13 Going on 30/.): 170020	/mnt/disk1/Movies/13 Going on 30/./poster.tbn

I: 2017/10/24 19:51:31 core.go:656: getFolders:Readdir(8)
I: 2017/10/24 19:51:31 core.go:666: getFolders:Executing find "/mnt/disk3/Movies/13 Going on 30/." ! -name . -prune -exec du -bs {} +
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 9374657	/mnt/disk3/Movies/13 Going on 30/./.actors
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 1346357	/mnt/disk3/Movies/13 Going on 30/./extrathumbs
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 287008	/mnt/disk3/Movies/13 Going on 30/./13 Going on 30-poster.jpg
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 83335	/mnt/disk3/Movies/13 Going on 30/./banner.jpg
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 355158	/mnt/disk3/Movies/13 Going on 30/./clearart.png
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 879830	/mnt/disk3/Movies/13 Going on 30/./disc.png
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 410197	/mnt/disk3/Movies/13 Going on 30/./landscape.jpg
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk3/Movies/13 Going on 30/.): 38371	/mnt/disk3/Movies/13 Going on 30/./logo.png

I: 2017/10/24 19:51:31 core.go:656: getFolders:Readdir(2)
I: 2017/10/24 19:51:31 core.go:666: getFolders:Executing find "/mnt/disk6/Movies/13 Going on 30/." ! -name . -prune -exec du -bs {} +
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk6/Movies/13 Going on 30/.): 5320	/mnt/disk6/Movies/13 Going on 30/./13 Going on 30.nfo
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk6/Movies/13 Going on 30/.): 12451	/mnt/disk6/Movies/13 Going on 30/./movie.xml

I: 2017/10/24 19:51:31 core.go:656: getFolders:Readdir(2)
I: 2017/10/24 19:51:31 core.go:666: getFolders:Executing find "/mnt/disk11/Movies/13 Going on 30/." ! -name . -prune -exec du -bs {} +
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk11/Movies/13 Going on 30/.): 6	/mnt/disk11/Movies/13 Going on 30/./.actors
I: 2017/10/24 19:51:31 core.go:669: getFolders:find(/mnt/disk11/Movies/13 Going on 30/.): 6	/mnt/disk11/Movies/13 Going on 30/./extrathumbs

 

The app did its thing, then was interrupted

W: 2017/10/24 20:03:36 core.go:1016: Command Interrupted: rsync -avPRX "Movies/13 Going on 30/extrathumbs" "/mnt/disk5/" (exit status 23 : Partial transfer due to error)

 

On the third run, you had this

I: 2017/10/24 20:12:04 core.go:656: getFolders:Readdir(19)
I: 2017/10/24 20:12:04 core.go:666: getFolders:Executing find "/mnt/disk5/Movies/13 Going on 30/." ! -name . -prune -exec du -bs {} +
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 9378446	/mnt/disk5/Movies/13 Going on 30/./.actors
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 747502	/mnt/disk5/Movies/13 Going on 30/./13 Going on 30-fanart.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 168352	/mnt/disk5/Movies/13 Going on 30/./13 Going on 30.fanart.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 904917533	/mnt/disk5/Movies/13 Going on 30/./13 Going on 30.mp4
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 170020	/mnt/disk5/Movies/13 Going on 30/./13 Going on 30.tbn
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 38400	/mnt/disk5/Movies/13 Going on 30/./Thumbs.db
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 120161	/mnt/disk5/Movies/13 Going on 30/./fanart.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 170020	/mnt/disk5/Movies/13 Going on 30/./folder.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 143184	/mnt/disk5/Movies/13 Going on 30/./movie.tbn
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 170020	/mnt/disk5/Movies/13 Going on 30/./poster.tbn
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 1346357	/mnt/disk5/Movies/13 Going on 30/./extrathumbs
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 287008	/mnt/disk5/Movies/13 Going on 30/./13 Going on 30-poster.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 83335	/mnt/disk5/Movies/13 Going on 30/./banner.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 355158	/mnt/disk5/Movies/13 Going on 30/./clearart.png
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 879830	/mnt/disk5/Movies/13 Going on 30/./disc.png
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 410197	/mnt/disk5/Movies/13 Going on 30/./landscape.jpg
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 38371	/mnt/disk5/Movies/13 Going on 30/./logo.png
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 5320	/mnt/disk5/Movies/13 Going on 30/./13 Going on 30.nfo
I: 2017/10/24 20:12:04 core.go:669: getFolders:find(/mnt/disk5/Movies/13 Going on 30/.): 12451	/mnt/disk5/Movies/13 Going on 30/./movie.xml

I: 2017/10/24 20:12:05 core.go:656: getFolders:Readdir(0)
I: 2017/10/24 20:12:05 core.go:659: getFolders:No subdirectories under /mnt/disk11/Movies/13 Going on 30

All files/folders were already gathered into disk5 (as a result of the second run), and disk11 was left with an empty folder, after the operation was interrupted.

 

That's why the webgui shows the movie is still present in disk11 (as per the above explanation).

 

I'm definitely not sure about the semantics of the rsync error 23 on that 6 byte folder, it's a bit strange, to be honest.

 

Other examples:

 

- Movies/2001 A Space Odyssey (1968)

First Run
(wasn't included)

Second Run
I: 2017/10/24 19:51:31 core.go:656: getFolders:Readdir(0)
I: 2017/10/24 19:51:31 core.go:659: getFolders:No subdirectories under /mnt/disk11/Movies/2001 A Space Odyssey (1968)

Third Run
I: 2017/10/24 20:12:05 core.go:656: getFolders:Readdir(0)
I: 2017/10/24 20:12:05 core.go:659: getFolders:No subdirectories under /mnt/disk11/Movies/2001 A Space Odyssey (1968)

 

- Movies/13 Hours The Secret Soldiers of Benghazi (2016)

First Run
(wasn't included)

Second Run
I: 2017/10/24 19:51:31 core.go:656: getFolders:Readdir(0)
I: 2017/10/24 19:51:31 core.go:659: getFolders:No subdirectories under /mnt/disk11/Movies/13 Hours The Secret Soldiers of Benghazi (2016)

Third Run
I: 2017/10/24 20:12:05 core.go:656: getFolders:Readdir(0)
I: 2017/10/24 20:12:05 core.go:659: getFolders:No subdirectories under /mnt/disk11/Movies/13 Hours The Secret Soldiers of Benghazi (2016)

and so on, similarly with the other folders (movies) you mentioned.

 

I hope this explains what happened, the "don't do anything if it's empty" issue is something I'll get fixed.

 

Edited by jbrodriguez
  • Like 2
Link to comment
11 hours ago, jbrodriguez said:
Ok, I know what happened with the issue you reported, FreeMan.
 

 

Thanks for looking into this! Looking forward to seeing the fix soon.

 

annnnd... that's how people end up quoting entire long posts for no logical reason... Tapatalk!


Sent from Tapatalk
 

Edited by FreeMan
removed unnecessary quote of previous post
Link to comment

As I'm going through my rather large collection of movies, trying to clean things up and get each movie's folder to reside on a single disk, a lovely new feature for this plug-in came to mind. It would be great if I could select the entire Movies share, then (based on some settings parameters) have it move each movie to the lowest numbered disk that it's currently occupying that also has enough space to hold it.

 

For example, if I have

-- 2001 A Space Odyssey    occupies space on: disk1, disk3, disk5, disk11

Check each disk in increasing number order to see where there's enough space for all of it, then move it there. If there's enough space on disk1, move all the pieces there, if there's not, check disk3, etc.

 

Configuration options could include

  • disks to exclude from consideration (maybe disk5 is getting pretty old and I don't want anything moved there)
  • disks to include for consideration (I want to move all movies to disks 1-5 and 10-12, but skip 6-9)
  • search disks from lowest number to highest number (or vice-versa)
  • move to the disk that will require the smallest amount of data movement
    • disk1: 100MB, disk3: 5GB, disk5: 950MB - move it to disk3 if there's enough room, otherwise move to disk5 if there's enough room, etc.
  • move to the disk that has the least free space that will hold the files to be moved no matter how many GB have to be moved
    • move to the disk that has the least free space even if all the files have to be moved to get it there. (Disk packing)
  • move to the disk that has the most free space available no matter how many GB have to be moved (optionally even if it's moving all the files).

I realize that there's a fair bit of logic involved there, but it seems that all the pieces exist, they just need to be called in a different order, and, potentially, multiple times (make all the checks on a per-directory basis, instead of a per-move basis as it does now).

 

This could work for TV shows, as well. If I select "Seinfeld", then each sub-folder would be processed the same way "2001" was described above. That would put each season on a single disk. If I wanted an entire show on a single disk, I could select the "TV" share and it would attempt to put "Seinfeld" on a single disk somewhere.

 

There would, of course, have to be some logic to handle the case where the directory (Seinfeld), in its entirety, won't fit on any existing disk. That could be skipped, or cause an abort (that could possibly be a configurable option). If it's skipped, I might just clear up enough space moving the rest of the data around that Seinfeld will now fit on a single disk.

 

Yes, I can absolutely do this now, but it takes me manually moving and selecting for each movie or TV show/season. Having an automated process is precisely what computers are for, isn't it?

 

Just a suggestion and an item for the Christmas wish list - it you say it's not doable or you're not interested, no worries - it's your baby, and I'm glad to be able to use it as is.

Edited by FreeMan
  • Upvote 1
Link to comment
14 hours ago, jbrodriguez said:

Hi, thanks for the suggestion.

 

It doesn't seem to be much different from what the app currently does, I'll give it some thought when I work on it again.

 

Glad to hear it! The way I figure it (says the guy who hasn't poked at your code at all:) it should be reasonably easy! ;)

 

 

Link to comment
  • 2 weeks later...
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.