v3.1.2 is available !

 

2017-07-06 - 3.1.2

Before removing a file/folder on the source disk, make sure it exists on the destination disk.
 

This is for the MOVE operation, as an additional check.

v3.2.0 is out !

 

----------------------

2017-07-07 - 3.2.0

VALIDATE operation is now implemented !!

 

Available after a COPY operation, it performs a checksum copy, thus validating the integrity
of the data.

 

It works by replacing the rsync flags from

 

-avPRX (default flags)

 

to

 

-rcPRX

 

If you set custom rsync flags, "-a" MUST still be the first one.

 

Also note that this operation takes a long time to complete.
--------------

 

I'd like to stress how long this operation will take. I will take a LOT of time

 

I'll now tackle the GATHER operation.

 

I would like to thank all the people that has contributed to the beer/coffee/soft drinks fund

 

I have enjoyed reading the notes on how unBALANCE helped you.

 

Thanks a lot !

 

Great tool!

 

I tried to do a move yesterday from one data disk to another and it seemed to work (a part from seeming very slow...)

However when I looked today, it never deleted the source files.  They all got copied, but never deleted...

 

I basically used all the stock settings..

 

Jim

Thanks for the comments !

 

It should run at approx the same speed as copying data to a parity protected drive.

 

Can you check what the log says (/boot/logs/unbalance.log) ?

 

Send a pm if you prefer.

Here is the relevant stuff from the log file..

I: 2017/07/10 20:54:50 core.go:596: _calc:End:srcDisk(/mnt/disk4)
I: 2017/07/10 20:55:06 core.go:782: Running transfer operation ...
I: 2017/07/10 20:55:06 core.go:814: Command Started: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/"
I: 2017/07/10 23:22:17 core.go:143: Sending config
W: 2017/07/11 04:58:25 core.go:886: Command Interrupted: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" (exit status 23 : Partial transfer
due to error)
I: 2017/07/11 04:58:25 core.go:991:
unBALANCE - TRANSFER operation INTERRUPTED


Started: Jul 10, 2017 20:55:06
Ended: Jul 11, 2017 04:58:25

Elapsed: 8h3m18.906s

Command Interrupted: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" (exit status 23 : Partial transfer due to error)

Transferred 962.2G at ~ 33.98 MB/s

And I don't see anything in the syslog around 4:58...   What error?

 

hmm...

Ok, you're not running the latest unBALANCE version (which is 3.2.0, it would have logged which transfer operation you ran: move/copy).

 

Yeah, rsync's error 23 (Partial transfer due to error) is almost like a catch-all 

 

But it's generally associated with permission errors or date-time issues with some of your files/folders.

 

Did you get any permission issues during the calculate phase ?

 

 

P.S.:

 

Turbo Write really helps with write speeds, at the expense of having all of your disks spun up.

7 minutes ago, jbrodriguez said:

Ok, you're not running the latest unBALANCE version (which is 3.2.0, it would have logged which transfer operation you ran: move/copy).

 

Yeah, rsync's error 23 (Partial transfer due to error) is almost like a catch-all 

 

But it's generally associated with permission errors or date-time issues with some of your files/folders.

 

Did you get any permission issues during the calculate phase ?

 

 

I don't recall if I had any permission issues.  I think I might have.

 

And I just installed unbalance several days ago!  I didn't think to check for an update so soon! :-)

23 hours ago, jbuszkie said:

Great tool!

 

I tried to do a move yesterday from one data disk to another and it seemed to work (a part from seeming very slow...)

However when I looked today, it never deleted the source files.  They all got copied, but never deleted...

 

I basically used all the stock settings..

 

Jim

You need to fix the permission issues (run "newperms path_to_folder") - on the source AND on the target directory!

Then run exactly the same move command again that you used yesterday.

The sources should get removed then.

Edited July 12, 20178 yr by Fireball3

I have a new disk coming online soon.  I'll probably have to move more file around.  I'll check the permissions before I run it next time.

 

Thanks for the help guys,

 

Jim

Just after some direction here, and what im looking for might be the gather function but im not sure what exactly that is planned to provide.

 

Basically i have just moved all my data using unbalance in order to move from reinfs to xfs, now i have one completely empty disk.

 

I would like to distribute the data on the other disks to this one, so that i end up with all drives having relatively the same amount of free space.

 

Whats the besr way to accomplish this?

I can tell you what GATHER will do with an example.

 

Let's say you have Breaking Bad's season 1/2 on disk2, season 3/4 on disk3, season 5 on disk4.

 

GATHER will help you consolidate all 5 seasons in a single disk. It will let you choose to consolidate on disk2, disk3 or disk4 (provided there's enough available space) or any other disk that has enough space.

 

What it won't do is  fill an empty disk to "rebalance" the space among all available disks.

 

Hope it makes it a bit clearer.

 

 

16 minutes ago, jbrodriguez said:

I can tell you what GATHER will do with an example.

 

Let's say you have Breaking Bad's season 1/2 on disk2, season 3/4 on disk3, season 5 on disk4.

 

GATHER will help you consolidate all 5 seasons in a single disk. It will let you choose to consolidate on disk2, disk3 or disk4 (provided there's enough available space) or any other disk that has enough space.

 

What it won't do is  fill an empty disk to "rebalance" the space among all available disks.

 

Hope it makes it a bit clearer.

 

 

Definitely clears that up, obviously a useful feature.

Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time?

 

Any suggestions to speed up manually doing this?

54 minutes ago, kingy444 said:

Definitely clears that up, obviously a useful feature.

Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time?

 

Any suggestions to speed up manually doing this?

 

Not the best way... but could you move it all to the cache disk and then let the mover "rebalance" it with the share set to the correct setting (I forget which one it is..)?

 

56 minutes ago, kingy444 said:

Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time?

Depending on your use, rebalancing may be counter productive. As you fill a disk, the speeds get lower and lower. Many people would rather fill a disk to capacity and leave it alone, so new writes and fresh data are written to the newest and fastest disks. Older less used data is left on the slower full disks. The more you erase and move data around on a disk, the more fragmented it gets, and modern filesystems don't really have a good method of defragmenting. They are designed in such a way that fragmentation doesn't have as much of an effect as in the past, but it's still noticeable on a badly thrashed disk.

 

The default fill method is high water for a reason, it's the best compromise between spreading data all over the disks evenly and filling each disk fully before moving to the next one.

 

tl;dr Rebalancing may hurt performance.

Since it's coming along pretty well, I figured I might as well post a teaser screenshot for the GATHER op 

 

There's still some work to be done, but it's not that far away now !

I've been using unbalance to move data off two disks to convert them to btrfs from rfs. My process was to move the data after calculating (no issues in calculation), and then stop the array and format the old drive. I noticed that it had copied the data not moved it after I did the second drive. 

 

The problem is, some of my shares are not showing the data that was moved... They are showing missing folders and files. I checked the share with my important folders and it's not showing all data, in crashplan it shows files are on the disk when you navigate through /mnt/user/share (pretty sure that's what it's saying, not showing as deleted from a backup) but the share isn't showing them.

 

I'm running a parity sync now. Is there something that needs to be done for unraid to recognize the files moved disks? It seems like that should have been happening realtime with unbalance but it looks like the share is looking for data on disk 1 when it was moved to disk 2. All my share information has "all disks" and no excluded disks.

Edited August 7, 20178 yr by ptirmal

3 minutes ago, ptirmal said:

All my share information has "all disks" and no excluded disks.

How about your Global Share Settings?

41 minutes ago, trurl said:

How about your Global Share Settings?

Same, set to default, nothing is excluded.

 

Just want to reiterate that I'm pretty sure the data has been copied correctly, because the drive usage percentage increased respective to the data I moved (which was copied) prior to formatting (which then went back to where it was before any file moving).

 

Edit: So, looking at this thread: 

 

 

I think this is a samba issue. I'll have to check when I get home to be certain. I've never had to fix permissions before, is there a way I can go about doing that?

Edited August 7, 20178 yr by ptirmal

Going from memory since not near machine. Tools tab. Docker safe permission (think part of fix common problem plugin)

BTW. Looking forward to the Gather function. Been meaning to consolidate some shares to specific drives but have not done it yet due to time constraints. How close are you for it?

1 hour ago, ptirmal said:

I'm running a parity sync now.

Why?

2 hours ago, ptirmal said:

it shows files are on the disk when you navigate through /mnt/user/share

What about taking a look at the disk itself ? 

$ ls -al /mnt/disk2/share

I understand disk2 is where data is supposed to be now.

 

You could also check the log (/boot/logs/unbalance.log) or send it over (pm if you want).

1 hour ago, graywolf said:

BTW. Looking forward to the Gather function. Been meaning to consolidate some shares to specific drives but have not done it yet due to time constraints. How close are you for it?

 

It should be ready sometime next week, if there are no issues in the mean time

33 minutes ago, jbrodriguez said:

What about taking a look at the disk itself ? 

$ ls -al /mnt/disk2/share

I understand disk2 is where data is supposed to be now.

 

You could also check the log (/boot/logs/unbalance.log) or send it over (pm if you want).

Thanks, I'll check when I get back.

1 hour ago, jonathanm said:

Why?

At the time I thought maybe the parity disk wasn't updating during the transfer.