Hi, unbalance works at disk level not user share level.

 

If you check the disks individually, is the data there ?

1 hour ago, jbrodriguez said:

Hi, unbalance works at disk level not user share level.

 

If you check the disks individually, is the data there ?

Yes.

 

How do I move that data back to its user share if the data is already on that drive?  Not sure what the next steps would be to relink.

 

Edited August 4, 20205 yr by waymon

Have you tried simply enabling those disks in the Global,Share settings?    A User Share is simply an amalgamation of all the top level folders on all drives where the share name corresponds to the folder name.

I have not.  I'm nervous to overwrite data by mistake.

 

Edited August 4, 20205 yr by waymon

21 hours ago, waymon said:

Yes.

 

How do I move that data back to its user share if the data is already on that drive?  Not sure what the next steps would be to relink.

 

Try a

cd /mnt/disk4
du -sh *

should give you an idea of how much data is there, at a minimum

 

Enabling shares shouldn't be an issue, at worst some files/folders will be 'hidden' if present in the source and destination disks (not sure if you did a move or copy)

 

Not sure what you mean by "a lot of data is missing", that's something you need to figure out first.

32 minutes ago, jbrodriguez said:

Try a

cd /mnt/disk4
du -sh *

should give you an idea of how much data is there, at a minimum

 

Enabling shares shouldn't be an issue, at worst some files/folders will be 'hidden' if present in the source and destination disks (not sure if you did a move or copy)

 

Not sure what you mean by "a lot of data is missing", that's something you need to figure out first.

 

Sorry. No data is missing, its just not under the user share. I did a move. Ill try enabling the share...hopefully you guys are right!

 

That looked to work. Thanks all. I just wanted to double check.  Thanks for the responses.

Edited August 5, 20205 yr by waymon

Hi, I'm using Unbalance to copy files from a disk I am going to remove from the array (and keep the data for archiving).

Using Scatter, the planning says there is an issue with permissions and recommends that I run "Docker Safe Permissions" to fix.

I also got email with this:

There are some permission issues:

1 file(s)/folder(s) with an owner other than 'nobody'

0 file(s)/folder(s) with a group other than 'users'

1 folder(s) with a permission other than 'drwxrwxrwx'

0 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'

Check the log file (/boot/logs/unbalance.log) for additional informatio

I looked into the log file but it does not say which files need to be fixed.  If it did, I could just run chmod on it.

I have used UnBalance to do this on three other disks and I have already run Docker Safe New Permissions.  Shouldn't it have fixed all the problems on all the disks? 

 

unbalance_log.txt

Edited August 10, 20205 yr by CaptainTivo
add some info

Those warnings are very general because there may be hundreds (if not thousands) of differences in ownership/permissions.

 

The idea of the warning is a heads up, in case you run unbalance as 'nobody' user, there may be issues moving files (in the delete part of the process).

 

unbalance runs as root starting with 6.8 (iirc), due to the new default permissions on /boot (the usb drive), so you should generally be fine.

15 hours ago, jbrodriguez said:

Those warnings are very general because there may be hundreds (if not thousands) of differences in ownership/permissions.

 

The idea of the warning is a heads up, in case you run unbalance as 'nobody' user, there may be issues moving files (in the delete part of the process).

 

unbalance runs as root starting with 6.8 (iirc), due to the new default permissions on /boot (the usb drive), so you should generally be fine.

OK.  How do I determine which user is running UnBalance?  I am simply running it from the Plugins page.  In addition, I have tried to find out how to use Linux style logins with UnRaid but have been unsuccessful.   I have created new users but have only been able to login as root.  I have been told elsewhere on this forum that "users do not really work on UnRaid" which does not make sense to me since UnRaid is built on top of Linux, but hey...

1 hour ago, CaptainTivo said:

OK.  How do I determine which user is running UnBalance?  I am simply running it from the Plugins page.  In addition, I have tried to find out how to use Linux style logins with UnRaid but have been unsuccessful.   I have created new users but have only been able to login as root.  I have been told elsewhere on this forum that "users do not really work on UnRaid" which does not make sense to me since UnRaid is built on top of Linux, but hey...

 

It's always running on behalf of "root" since that's the only user that exists for unRaid on the GUI or able to login.

Hello,

 

My disk4 has been disabled (2 To), but content is still emulated thanks to parity.

I added a fresh new disk into the array (4 To) called disk12.

 

I plan to move all datas from disk4 (emulated content) to disk12, but nothing is transferred.

 

 

 

 

Let me show you the logs :

 

I: 2020/09/01 21:33:51 planner.go:509: disk(/mnt/disk9):no-items:currentFree(121.25 GB)

I: 2020/09/01 21:33:51 planner.go:510: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:511: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:512:

I: 2020/09/01 21:33:51 planner.go:508: =========================================================

I: 2020/09/01 21:33:51 planner.go:509: disk(/mnt/disk10):no-items:currentFree(157.66 GB)

I: 2020/09/01 21:33:51 planner.go:510: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:511: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:512:

I: 2020/09/01 21:33:51 planner.go:508: =========================================================

I: 2020/09/01 21:33:51 planner.go:509: disk(/mnt/disk11):no-items:currentFree(219.88 GB)

I: 2020/09/01 21:33:51 planner.go:510: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:511: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:512:

I: 2020/09/01 21:33:51 planner.go:497: =========================================================

I: 2020/09/01 21:33:51 planner.go:498: disk(/mnt/disk12):items(12)-(1.82 TB):currentFree(3.84 TB)-plannedFree(2.02 TB)

I: 2020/09/01 21:33:51 planner.go:499: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:502: [1.26 TB] /mnt/disk4/Home Cinema/./-=[ BluRay Disc ]=-

I: 2020/09/01 21:33:51 planner.go:502: [184.78 GB] /mnt/disk4/Home Cinema/./-=[ HD ]=-

I: 2020/09/01 21:33:51 planner.go:502: [162.79 GB] /mnt/disk4/Home Cinema/./-=[ 4K ]=-

I: 2020/09/01 21:33:51 planner.go:502: [66.57 GB] /mnt/disk4/Home Cinema/./-=[ Download ]=-

I: 2020/09/01 21:33:51 planner.go:502: [52.46 GB] /mnt/disk4/Home Cinema/./-=[ Documentaires ]=-

I: 2020/09/01 21:33:51 planner.go:502: [43.78 GB] /mnt/disk4/Home Cinema/./-=[ 3D ]=-

I: 2020/09/01 21:33:51 planner.go:502: [27.95 GB] /mnt/disk4/Home Cinema/./-=[ Series TV ]=-

I: 2020/09/01 21:33:51 planner.go:502: [21.78 GB] /mnt/disk4/Home Cinema/./-=[ Animation ]=-

I: 2020/09/01 21:33:51 planner.go:502: [949.12 MB] /mnt/disk4/Boite à Outils/./-=[ AppZ ]=-

I: 2020/09/01 21:33:51 planner.go:502: [6.50 MB] /mnt/disk4/Home Cinema/./-=[ DVD ]=-

I: 2020/09/01 21:33:51 planner.go:502: [49.36 KB] /mnt/disk4/Mes Documents/./Nicolas

I: 2020/09/01 21:33:51 planner.go:502: [344.00 B] /mnt/disk4/system/./docker

I: 2020/09/01 21:33:51 planner.go:505: ---------------------------------------------------------

I: 2020/09/01 21:33:51 planner.go:506:

I: 2020/09/01 21:33:51 planner.go:516: =========================================================

I: 2020/09/01 21:33:51 planner.go:517: Bytes To Transfer: 1.82 TB

I: 2020/09/01 21:33:51 planner.go:518: ---------------------------------------------------------

I: 2020/09/01 21:34:02 core.go:687: Running Move operation ...

I: 2020/09/01 21:34:02 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ BluRay Disc ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:02 core.go:185: Sending operation

I: 2020/09/01 21:34:02 core.go:835: monitor:transfer:(/mnt/disk4/Home Cinema/-=[ BluRay Disc ]=-/Chronique de Narnia - Le Lion, la Sorcière blanche et l'Armoire magique,)

I: 2020/09/01 21:34:06 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:06 core.go:1028: Command Finished

I: 2020/09/01 21:34:06 core.go:1041: Current progress: 69.26% done ~ 1s left (267673.28 MB/s)

I: 2020/09/01 21:34:06 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ HD ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:07 core.go:835: monitor:transfer:(/mnt/disk4/Home Cinema/-=[ HD ]=-/12 Year a Slave)

I: 2020/09/01 21:34:12 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:12 core.go:1028: Command Finished

I: 2020/09/01 21:34:12 core.go:1041: Current progress: 79.38% done ~ 2s left (138070.07 MB/s)

I: 2020/09/01 21:34:12 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ 4K ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:12 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:12 core.go:1028: Command Finished

I: 2020/09/01 21:34:12 core.go:1041: Current progress: 88.30% done ~ 1s left (146271.62 MB/s)

I: 2020/09/01 21:34:12 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ Download ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:13 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:13 core.go:1028: Command Finished

I: 2020/09/01 21:34:13 core.go:1041: Current progress: 91.95% done ~ 0s left (145389.54 MB/s)

I: 2020/09/01 21:34:13 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ Documentaires ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:13 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:13 core.go:1028: Command Finished

I: 2020/09/01 21:34:13 core.go:1041: Current progress: 94.82% done ~ 0s left (143414.67 MB/s)

I: 2020/09/01 21:34:13 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ 3D ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:14 core.go:835: monitor:transfer:(/mnt/disk4/Home Cinema/-=[ 3D ]=-/Le Chasseur et la Reinde de Glace)

I: 2020/09/01 21:34:19 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:19 core.go:1028: Command Finished

I: 2020/09/01 21:34:19 core.go:1041: Current progress: 97.22% done ~ 0s left (98031.05 MB/s)

I: 2020/09/01 21:34:19 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ Series TV ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:19 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:19 core.go:1028: Command Finished

I: 2020/09/01 21:34:19 core.go:1041: Current progress: 98.75% done ~ 0s left (96768.83 MB/s)

I: 2020/09/01 21:34:19 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ Animation ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:20 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:34:20 core.go:1028: Command Finished

I: 2020/09/01 21:34:20 core.go:1041: Current progress: 99.95% done ~ 0s left (95254.28 MB/s)

I: 2020/09/01 21:34:20 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Boite à Outils/-=[ AppZ ]=-" "/mnt/disk12/"

I: 2020/09/01 21:34:21 core.go:835: monitor:transfer:(/mnt/disk4/Boite à Outils/-=[ AppZ ]=-/DVDFab 11.0.6.0 Complet & Portable/dvdfab_11.0.6.0_x64/DVDFab11/App)

I: 2020/09/01 21:37:09 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:37:09 core.go:1028: Command Finished

I: 2020/09/01 21:37:09 core.go:1041: Current progress: 100.00% done ~ 0s left (9301.15 MB/s)

I: 2020/09/01 21:37:09 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Home Cinema/-=[ DVD ]=-" "/mnt/disk12/"

I: 2020/09/01 21:37:09 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:37:09 core.go:1028: Command Finished

I: 2020/09/01 21:37:09 core.go:1041: Current progress: 100.00% done ~ 0s left (9276.38 MB/s)

I: 2020/09/01 21:37:09 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "Mes Documents/Nicolas" "/mnt/disk12/"

I: 2020/09/01 21:37:10 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:37:10 core.go:1028: Command Finished

I: 2020/09/01 21:37:10 core.go:1041: Current progress: 100.00% done ~ 0s left (9251.70 MB/s)

I: 2020/09/01 21:37:10 core.go:710: Command Started: (src: /mnt/disk4) rsync -avPR -X --dry-run "system/docker" "/mnt/disk12/"

I: 2020/09/01 21:37:10 core.go:767: command:retcode(0):exitcode(0)

I: 2020/09/01 21:37:10 core.go:1028: Command Finished

I: 2020/09/01 21:37:10 core.go:1041: Current progress: 100.00% done ~ 0s left (9227.14 MB/s)

I: 2020/09/01 21:37:10 core.go:1107:

unBALANCE - MOVE operation completed

Started: Sep 1, 2020 21:34:02

Ended: Sep 1, 2020 21:37:10

Elapsed: 0s

Move operation has finished

Transferred 1.82 TB at ~ 9227.14 MB/s

Could you please help me?

 

Thanks

5 hours ago, neuk34 said:

Could you please help me?

The usual way of dealing with a disabled disk would be to rebuild to the replacement disk. I'm not even sure how you convinced unraid to allow you to add a disk while one was disabled, but beyond that, you are running mostly ReiserFS disks, so you eventually want to migrate everything to XFS anyway.

 

Your new almost empty disk is ReiserFS, which is unfortunate, I'd have preferred any new disks be XFS.

 

If I were in your shoes, I'd get another new 4TB now, rebuild disk4 on it, that way you are back to being parity protected. Then empty out disk12 on to the new disk4, reformat disk12 as XFS, and start the process of copying ReiserFS disks onto XFS disks and formatting the ReiserFS disks to XFS after their contents are safely on an XFS disk.

 

Running for an extended period of time with a disabled disk is very risky, especially with disks that old. How confident are you in the health of the rest of your disks?

3 minutes ago, jonathanm said:

Running for an extended period of time with a disabled disk is very risky, especially with disks that old. How confident are you in the health of the rest of your disks?

And do you have good backups.

Thanks for your help. I'm on my way to execute your process because my disk are old and array is clearly at risk.

unbalance does not work presently.

How can I move data (fast and easy) from one disk to another?

Edited September 2, 20205 yr by neuk34

8 hours ago, neuk34 said:

How can I move data (fast and easy) from one disk to another?

I recommend copying, not moving.

1. It's faster, ReiserFS is VERY slow to delete files, especially on very full or very well used volumes, and moving involves writing to both the source and destination where copying only involves writing on the destination, so parity is less busy.

2. After the copy is done, you can verify the results before you format the source to XFS.

 

You can use any method to copy that you are comfortable with, I personally use rsync at the command line, one pass for a quick copy, second pass with checksums for verification that the copy is complete.

These are the commands I used.

To copy

rsync -arv /mnt/disk(source)/ /mnt/disk(destination)

To verify

rsync -narcv /mnt/disk(source)/ /mnt/disk(destination)

 

Where (source) and (destination) are the literal numbers, like /mnt/disk12/

Be careful to include the slashes at the end where needed, otherwise you will end up with a root folder of disk12 with all your shares inside it, which can get VERY confusing since it will automatically show up as a share "disk12", but it will be on the destination disk, which is another disk number.

 

To use the command line for Unraid, there are MANY different ways to access it, but for your purposes my first choice would be the actual keyboard and monitor attached to the Unraid tower. 2nd choice would be a remote SSH session, and start a "screen" command before doing the copy and verify, that way if the session gets disconnected you aren't killing the copy.

 

Normally unbalance would work, but I suspect ReiserFS is causing your issues.

 

If you had pointed to your current thread where you were discussing your array failure with @johnnie.black, I would never have gone through the trouble of typing this up.

 

You really need to stop trying to move data around on the array, and copy anything important to good drives.

 

21 hours ago, neuk34 said:

Could you please help me?

Just as an additional comment on top of what jonathanm and trurl have suggested, you ran unbalance with --dry-run, that's why it didn't transfer any file.

thanks, it ran perfectly. Thank you all for your help much appreciated

Thanks for the plugin, very cool.  I just did a gather operation moving 714 folders and 7TB and ended up with 216 moves that resulted in "rmsrc button" appearing.  I did have to do the docker safer permissions before I started.  I show a log snippet below.  Couple questions:

• Is it normal to get that many; anything I can do to minimize it?
• I basically moved all my data to disk 4.  When I go and look for the data that genererated the "RSMC" I do not see it the original on Disk1, 2, or 3 anymore.  Is there a chance it really did delete the source file already?  Do I still need to click the RSMC button?
• Is there a faster way to click the RSMC button?  Currently I go to the History tab of UnBalance, scroll down to find "RSMC", click it, then the webpage reloads.  Navigate back to History tab, scroll down to find the next RSMC, click it.  Rinse and repeat.  Lost of clicking and scrolling.

 

Here is a an example of an RSMC generated for Guardians of the Galaxy in the unbalance log.  It processes the file then moves on to the next one:

 

However about 1000 lines (and 4 hours) later in the log I get this entry:

 

Edit: I see in the screen shots the 2nd failed operation is referencing Disk2, vs Disk1 in the first operation. 

Edit2: When I do click the RSMC button it seems to be confirming that the data was in fact deleted? 

Edited September 7, 20205 yr by drjUnraid

Hi, the general idea of rmsrc is this

- you transfer from diskX to diskY

- rsync logs a warning (most of the times, err code 23 which is basically a blanket for anything that moves :))

- unbalance informs you about this issue via rmsrc in the history tab

 

If you checked that data is already available on disk4, then it's not an issue to  leave the source there (if you're going to wipe out the disk for example)

 

I should probably add a checkbox to rmsrc all, something like that, but I currently have zero cycles to spare due to work.

Hello everyone,

 

is it nessesary to exclude other shares than appdata from the "Docker Safe New Permissions command"? I just want to move a share that is splitted to 3 harddrives to a new member of the array.

 

Thanks and stay safe

1 hour ago, Bob@unraid said:

Hello everyone,

 

is it nessesary to exclude other shares than appdata from the "Docker Safe New Permissions command"? I just want to move a share that is splitted to 3 harddrives to a new member of the array.

 

Thanks and stay safe

The regular New Permissions page can be used on a single specific share.

Hey community! I have a quick question and im sure its user error. First what im working with is 8x8tb drives that i have moved from a windows system. All my drives are in and my share "Data" is set to most free space and split the contents however. What i am running into is even though i have selected 1 drive to scatter and all the others for it to distribute the files to, it only mass moves the files to another drive. Am i missing something? Below is a few snips to illustrate. Btw my use case is yes it benefits me to have my files split this way. All of it is media and most over 10-15GB files. Like it is now it saturates a single drives I/O and is creating buffering as there are over 25 streams at a time hitting these drives. Having the data split across the array helps immensely with I/O. As you can see after the "Plan" it just moves all the data to Disk5 instead of actually scattering. Any info would be awesome and thanks! 

Before Plan

After Plan

The idea of scatter is to empty a disk by transferring data to other disk, and it does so eagerly: find the disk with most free space, transfer as much data as possible, if there's still data, move it to the next most free disk and so on.

 

I'm guessing what you want is to balance the data into the disks, but this is not the tool for that.

10 hours ago, jbrodriguez said:

The idea of scatter is to empty a disk by transferring data to other disk, and it does so eagerly: find the disk with most free space, transfer as much data as possible, if there's still data, move it to the next most free disk and so on.

 

I'm guessing what you want is to balance the data into the disks, but this is not the tool for that.

Thanks jbrodriquez! 

 

Is there a tool for that then and maybe a feature request for Unraid? It doesn't seem far fetched nor substantially difficult to add a "Balance" option as good as this software works and the demand for it has been pretty ongoing from what i have seen in the forums. I would even go as far as to rename the "Scatter" option to "Evacuate" to better illustrate it's function. 

Edited September 18, 20205 yr by titanarchon
Added another quick idea

It's not something on my radar, to be honest. 🤷‍♂️