[Plugin] unbalanced

[BF90X]

 
Obviously if there was no parity drive then parallel operations would be faster provided there was no overlap inthe drives being used by each of the move operations.  Risk could be reduced by using something like the File Integrity Plugin and checking everything afterwards.  Another considerationis backups - I wouldn't even attempt this without a backup of all files being moved.  Parity or File Integrity can help you identify whether there have been problems with the move, but neither can fix the data if a file was lost or became corrupted for some reason.

Thanks S80_UK

That was a fantastic recommendation, I didnt even know that plugin existed.

I am currently running the build process.

Sent from my SM-G955U using Tapatalk

[Gico]

I ran unBALANCE today after a long time without using it.

I ran into the http-->https problem and manually changed the address.

Is there a permanent solution? I didn't follow the entire thread to find out.

 

Than I had a permission error when unBALANCE ran as "nobody". Why doesn't the "Plan" mode check for/alert about this error?

 

Anyway I changed it to run as some user and now it can delete the source files,

however when I use the rmsrc button, all the other lines turn into little un-clickable yellow v status (image attached),

and I have to reload the page in order to click another rmsrc button,

but refresh button doesn't work (I'm getting {"message":"Not Found"} ),

so I have to re-enter the plugin in order to run one more rmsrc button.

Don't tell my wife, but cleaning dishes is more rewarding task than this one ?

Edited June 3, 2018 by Gico

[jbrodriguez]

On 6/3/2018 at 12:55 PM, Gico said:

however when I use the rmsrc button, all the other lines turn into little un-clickable yellow v status (image attached),

 

Hadn't hit this bug, I'll take a look.

[jbrodriguez]

v2018.06.05 (5.2.1) is out !

 

- Fix calculation of space in the planning stage
- Fix wrong command icon in history screen

 

jj666's issue was a bug I introuced switching from int64 to uint64 in the code: at one point it subracted min. reserved space from disk's free space, which resulted in a negative number that wrapped around uint64's capacity, thus EB.

 

The solution was switch back to int64: they're still capable in the foreseeable future

 

As for Gico's, I made a small change: only the most recent operation in the history screen can be replayed, validated or have one of its flagged command be able to remove the source.

 

There was some logic that I think was causing confusion as to what icon (or button) to show.

 

As always, I'll appreciate the feedback.

[jj666]

It works perfectly now, thank you!!!!

 

Cheers,

 

-jj-

[shEiD]

I think, the latest version (v2018.06.05) introduced a bug. At least, this never happened before, and I've been using unBALANCE for about 6 months.

 

The problem is with a Gather mode:

1. I select a folder to be moved - it is currently on two disks: disk 1 and disk 26
2. After planning is finished - disk 26 is missing from the list of potential destinations
3. Actually disk 26 is exactly the disk that I wanted to move files to, because it already has the majority of files in it. basically, it should have been the first/preferred choice in the suggested list - but as I said, it is missing completely.
4. And no, it is not out of empty space. The size of the files to be moved would have been ~32GB and disk 26 still has 3.28TB of free space.

Basically, it kinda feels like a bug, like the "top item" was removed from the list by mistake. Sorry, I do not know Go language so I'm not able to check out the latest commits and find an answer.

[jbrodriguez]

Hi, could you send me /boot/logs/unbalance.log ?

 

I'll take a look.

[KeithG]

Hello all,

 

I'm planning on moving  about 750GB of data from one driver to another.  Knowing that this is going to take a while, my question is, can I close my browser window where unBALANCE is running, then reconnect at a later time to see the move progress?  Or do I have to leave the unBALANCE browser window open and undisturbed the entire time?

 

Thanks!

[shEiD]

@jbrodriguez I went through the unbalance logs and found that it was probably my own mistake. The files at that moment were only on disk 26, and the folder on disk 1 was empty, because I forgot to move the files on disk 1 to the appropriate folder before running unbalance.

Long story short, I have a routine for moving/gathering files between disks. This time, I missed a step, it seems, and thought it was unbalance's fault, but it was mine. Sorry, for making a hasty false alarm.

 

Thanks for an awesome, script, dude. I use it all the time.

[_Shorty]

Keith, the web interface is just a remote way to control/start a process that is local to the server.  Once you start it, the remote portion of the job is finished.  Your client computer is no longer involved with what the server is doing.  Close it and be on your way.  Check later to make sure things finished if you wish.

[jbrodriguez]

14 hours ago, shEiD said:

Thanks for an awesome, script, dude. I use it all the time.

 

Thanks for the kind comments ! Glad it wasn't an issue with the plugin

[jbrodriguez]

KeithG, what _Shorty said

 

You can also set up notifications in the plugin settings (enabling unRAID notifications first).

 

Notifications are sent for end of planning stage, start/end of transfer operations.

[Cessquill]

4 minutes ago, jbrodriguez said:

You can also set up notifications in the plugin settings (enabling unRAID notifications first).

 

Notifications are sent for end of planning stage, start/end of transfer operations.

Ooh!  I didn't know that - that's handy.

[KeithG]

Thanks for the replies _Shorty and jbrodriguez!

 

I just wanted to be sure I did things right as I've seen processes at other websites where if you close the browser window the process is interrupted / stopped.

 

Have a great day!

[kizer]

I'm getting a Page not Found all of the sudden when I click on the link Open Web UI ( v5.2.1 )

Mine points to http://192.168.7.127:6237/ which is my servers IP and the port defaulted in the settings. 

 

I don't see anything odd in the system log. 

 

********************

Updating to the latest OS release then I'll follow up just in case I needed a Reboot or something. 

 

***********************

6.5.3-rc2 and I'm still seeing the can not connect message. Basically Page not found

 

 

[AnnabellaRenee87]

4 hours ago, kizer said:

I'm getting a Page not Found all of the sudden when I click on the link Open Web UI ( v5.2.1 )

Mine points to http://192.168.7.127:6237/ which is my servers IP and the port defaulted in the settings. 

 

I don't see anything odd in the system log. 

 

********************

Updating to the latest OS release then I'll follow up just in case I needed a Reboot or something. 

 

***********************

6.5.3-rc2 and I'm still seeing the can not connect message. Basically Page not found

 

 

 

Same here, running on 6.5.3-rc2.

Tried removing and reinstalling the add on, didn't help ?

Attaching logs.

 

EDIT: Just tried adding https:// to the address and after accepting the security warning about a certificate it will load the page. I'm not using https on the server btw.

 

PS. Sorry for all the edits.

 

server-diagnostics-20180611-1609.zip

Edited June 11, 2018 by AnnabellaRenee87

[kizer]

I was having issues on 6.5.2-rc2 before I even updated to 6.5.3-rc2

 

I just happened to need it today and figured it might of needed a system reboot or an update to the plugin did it. Not a 100%, but figured I'd mention it. 

[jbrodriguez]

well, glad to know it's working for both of you 

[kizer]

3 hours ago, jbrodriguez said:

well, glad to know it's working for both of you 

 

No its not working for us two. Lol 

[zhulick]

Second time using this plugin.  First time was like half a year ago, and it worked flawlessly then.  This time, however, i believe i may've run into a problem.

 

I have a 9 drives plus 2 parity array (11 total) all 8tb each.  3 of the drives were recently added, and i wanted to empty out disk 1 to disk 7 and 8.   I setup a scatter/move (not copy) operation to that.  Everything seemed to be working ok,  i could see drive 7 slowly filling up etc.

 

After about 2tb of files transferred, i noticed that the free space on drive 1 wasn't changing.  It was 6tb full, with 2 tb empty when  started, and after transferring over 2tb worth of files it was still 6tb full and 2 tb empty etc.  I also noticed that now both disk 1 and disk 7 contained the same files across multiple directories, so i had dupes.

 

In other words, it looks like Unbalance has copied files (not moved) from disk 1 to disk 7 and now i have 2 tb worth of dupes on both drives.

 

How do i "de-dupe" ?

[itimpi]

Just a guess, but it is likely that the source files only get removed on completion of the move.

[Cessquill]

11 minutes ago, itimpi said:

Just a guess, but it is likely that the source files only get removed on completion of the move.

This.  From my experience, the files don't get removed until the end.

 

Moving is technically a copy and delete, and is so in case of mid-move failure (that's my understanding at least).  If there are several stages listed in the unbalance list, it might clear at the end of each one - can't recall.

[zhulick]

Yeah, job didn't complete.  

 

I had to move the tower over to it's normal spot, which means i had to shut it down etc.  The job was about a third of the way through a long move, and i couldn't wait.

 

Ok, so that explains how i got dupes, and i'll have to keep in mind that this happens for next time.   

 

Now i need to figure out the best way to clean this up.   About 2tb worth of data was transferred, so i have 2tb of dupes i need to get rid of.  

 

All the dupes are between drives 1 and 7.  7 had an additional 3tb already on it before the move began.  

 

What's my best option for cleaning this up ?

 

 

 

[JorgeB]

26 minutes ago, zhulick said:

What's my best option for cleaning this up ?

You can use rsync with --remove-source-files, this way it will remove the already existing files on the destination, it would work best if the share or folder you're syncing fits on the destination, or rsync will abort possibly leaving a couple of dupes.

[Cessquill]

56 minutes ago, zhulick said:

What's my best option for cleaning this up ?

 

Not sure whether it's the right thing to do, but when I did similar I opened up Midnight Commander from the command line and manually moved all those which this plugin was going to do.  When it warns about existing files, overwrite all.  My logic was that the masters of the file still live on the from disk, and I couldn't be sure which duplicate(s) were unfinished).

 

That is, if you still want to move from and to the same drives.