Replace 8 drives as fast as possible.

[Waffle]

The story.

I bought 10 12TB drives, but I clicked the wrong ones and got cheap drives. Like an idiot I tried to use them any way. As you would expect they started to fail quickly since they were not meant to run this way. I have over 20 TB of data moved onto them already.  I  ordered new drives of the right kind to replace them. But replacing 2 at a time (dual parity) to very slow with 7+ days of rebuilding on each. And the return process is started on the cheap ones.

 

The question.

Is their a way (even if it means taking the array offline) to move the data from all 8 drives to the new drives at once. So it is not running at supper slow speeds. I only have about a week to do this and I would rather not re copy all the data over again (but if I have to I will). 

 

All drives are in system at the same time.

[trurl]

44 minutes ago, Waffle said:

As you would expect they started to fail quickly since they were not meant to run this way. I

No I wouldn't expect. What makes you think there is anything wrong with them? 

[Waffle]

3 minutes ago, trurl said:

No I wouldn't expect. What makes you think there is anything wrong with them? 

Hundreds of bad sectors

[itimpi]

I think the only way to do what you want is to use the New Config tool and reset the array to only include the drives you want to keep.     You can then mount the ‘suspect’ drives using Unassigned Devices to copy the data off them.    This will save time but if any of the suspect drives now fail you will have lost their contents.   However if the drives are suspect this may be the best approach anyway as any read errors on them would stop the rebuild using the other approach being error free.

[Waffle]

5 minutes ago, itimpi said:

I think the only way to do what you want is to use the New Config tool and reset the array to only include the drives you want to keep.     You can then mount the ‘suspect’ drives using Unassigned Devices to copy the data off them.    This will save time but if any of the suspect drives now fail you will have lost their contents.   However if the drives are suspect this may be the best approach anyway as any read errors on them would stop the rebuild using the other approach being error free.

You just said something that gives me hope. You can copy data of drives that are not part of the array? I'm less than a month into unraid could you maybe break that down for me.

 

31 minutes ago, trurl said:

 

 

[itimpi]

Every array drive in UnRAID is a self-contained file system so can be read when not part of the array.   This means that they can be individually mounted when removed from the array and as long as they are not actually failing their contents read.  The easiest way to mount drives that are not part of the array is the Unassigned Devices plugin.

 

once mounted you can use your favoured method for copying files.  There are a wide variety of choices such as cp, rsync, or mc from the command line, Krusader docker container for an UnRAID hosted graphical file manager, or over the network from a client PC/Mac using its file manager.

[Waffle]

4 minutes ago, itimpi said:

Every array drive in UnRAID is a self-contained file system so can be read when not part of the array.   This means that they can be individually mounted when removed from the array and as long as they are not actually failing their contents read.  The easiest way to mount drives that are not part of the array is the Unassigned Devices plugin.

 

once mounted you can use your favoured method for copying files.  There are a wide variety of choices such as cp, rsync, or mc from the command line, Krusader docker container for an UnRAID hosted graphical file manager, or over the network from a client PC/Mac using its file manager.

Firstly, thank you so much. I think you just saved me over a month of teadio tedious rebuilds. Secondly, this community is so much better than any linux community I've ever tried to get help from. As I have no formal education in this field and this is just a hobby of mine. Your previous posts on other people's threads have helped me a great deal as well thank you again.

[sota]

additional suggestion:

if you go this route, mount the disks as READ ONLY under UD.  that way if one or two do decide to shuffle off their mortal coils, you could stand a better chance of recovering data.

 

in fact, I'll suggest a different course of action:

DO NOT take apart the current array, which while having problems is protected by dual parity, according  to you.

Mount the new disks as Unassigned Devices, copy data directly to them from the array, THEN once all of your data is copied off the failing array, you can pull it down and install the new disks and perform a New Config.

You'll have duplicate arrays then.

[Waffle]

3 minutes ago, sota said:

additional suggestion:

if you go this route, mount the disks as READ ONLY under UD.  that way if one or two do decide to shuffle off their mortal coils, you could stand a better chance of recovering data.

 

in fact, I'll suggest a different course of action:

DO NOT take apart the current array, which while having problems is protected by dual parity, according  to you.

Mount the new disks as Unassigned Devices, copy data directly to them from the array, THEN once all of your data is copied off the failing array, you can pull it down and install the new disks and perform a New Config.

You'll have duplicate arrays then.

I actually really like that suggestion. I will try that way once the current rebuild

 

1 hour ago, itimpi said:

 

is done (in about 20 hours) thank you.

[sota]

you're welcome.

in fact, my second action is most likely preferred.

 

if you create a new array with the new disks, then copy the data to it, you'll be "abusing" the parity disks, since you'll most likely be writing to storage disks one at a time.  By copying data to the new drives first, then creating the array around them, you'll only be building the parity a single time.

 

I have some other thoughts, but I don't want to "weed" up the discussion.
If you have any disks that are looking pretty "bad", I can type those thoughts out, at your request.

[trurl]

4 hours ago, Waffle said:

Hundreds of bad sectors

Is that based on SMART reports for the disks?

[sota]

The following is based on a request by the OP @Waffle

I'll be editing this post throughout the day, as I complete my thoughts, so until I put a big "DONE" at the top, consider it an incomplete work in progress.

I just don't want to lose my work by accident.

 

Copying all data from current array, to new disks as Unassigned Devices.

 

"Tools" used (hyperlinks):

Community Applications

Unassigned Devices and UD+

binhex - krusader

 

When I make reference to any of the above, use the hyperlinks to get specific instructions on how to install or manage them.  They are a treasure trove of information.

 

We'll need Docker to work, so if you haven't already, go to your 'SETTINGS' tab and pick 'DOCKER' under System Settings.

Make sure it's enabled.

 

You'll need to install Community Applications first.  Once you have that, everything else is installable from there, by search for them by name.

With all of them installed, and assuming you have plugged in drives that are NOT part of the array, you should be able to see drives listed in 'Unassigned Devices'.

 

If format is greyed out, you'll need to go into UD's settings and enable 'Destructive Mode'.

Click 'FORMAT' to format the disk. Make sure to pick XFS (it's default) as you're turning these new disks into an array later.

Follow the prompts to complete formatting.

'MOUNT' the disk by pressing the button.

Repeat as needed for all of the Unassigned Devices you have listed that need formatting.

 

Let's get ready to copy some data.

You'll have the krusader docker available on the 'DOCKER' tab.

If it's not 'started', click on the icon for it, and in the menu click 'start'.

Once it's running, same menu, click 'WebUI'.

 

Krusader is just a GUI file manager.  It's easy enough to use, but take note it only has a single folder 'tree' on the left; make sure you have clicked into the correct target window if you want to change your left and right views, otherwise you'll wonder what you've done wrong.

Also, the 'noVNC' call out on the left, has a right-pointing chevron; click it. It'll get that menu mostly out of the way.

 

The folder you'll care about, is the 'HostMedia' folder.  You'll find everything important in there.

 

Here, we need to make a decision:

The 'disk#' folders are the individual data disks in your array.  You can access your data through them, but only for the specific disk in question.

The 'user' folder contains all of your shares.  From there you can access your data without regard to which disk it's actually on.

*IF* you have a known failing/problematic disk in the array, I'd suggest going directly to that disk to copy the data on it.

The 'disks' folder contains all of the Unassigned Devices drives you've added/formatted above.

 

From this point on it's just a matter of using the krusader file manager to get your source and target set up, and clicking 'F5 Copy.'

 

Edited January 8, 2021 by sota

[JonathanM]

47 minutes ago, sota said:

From this point on it's just a matter of using the krusader file manager to get your source and target set up, and clicking 'F5 Copy.'

I get why you want to use krusader, but wouldn't it save a bunch of time and knowledge transfer to just use midnight commander (mc) ?

No docker config necessary, and mc seems to me to be faster than krusader.

[Waffle]

6 minutes ago, jonathanm said:

I get why you want to use krusader, but wouldn't it save a bunch of time and knowledge transfer to just use midnight commander (mc) ?

No docker config necessary, and mc seems to me to be faster than krusader.

They are most likely trying to make the overall process as familiar as possible. Since I am very new to an operating system that is not windows/dos. But if you would like to type up a very detailed step by step post on how to do it the way you see fit then this thread might become very very useful to future users.

[sota]

24 minutes ago, jonathanm said:

I get why you want to use krusader, but wouldn't it save a bunch of time and knowledge transfer to just use midnight commander (mc) ?

No docker config necessary, and mc seems to me to be faster than krusader.

I honestly, don't have any "seat time" with MC, so I can't give instruction on how to use it.

 

[itimpi]

13 minutes ago, sota said:

I honestly, don't have any "seat time" with MC, so I can't give instruction on how to use it.

 

Maybe mention it as an alternative to Krusader and provide a link like this one might be worth doing?

[sota]

It might, but the OP PM'ed me and asked if I wouldn't mind do a detailed write up about how I would handle his situation, and based on his expressed knowledge level. So, that's exactly what I did.  I'm certain others will do it differently, and in fact I might do it differently for myself (multiple terminal windows and command line sequences), but that wouldn't, in my opinion, give him the immediate solution he's in need of.

[sota]

@Waffle let me know if where I left off is good enough, or if you have any other questions.  I'll fill in what I can based on that.

[Waffle]

I will be home in about 3-4 hours to try this. The first question I see (and perhaps it will be self evident when I start doing it and can see everything in person). Is: "do I have the array on our off on the first step?

[sota]

I would say, on.  That way if during the copy activities, if one of your data disks dies, it'll get picked up by the parity and you can finish.

Also, I don't think the 'disk#' folders exist until the array is started anyways.

So, definitely ON.

Edited January 8, 2021 by sota

[Waffle]

Ok, well, i had a moment so i thought i would remote in to start this. And the first issue I have found is the "format" option is greyed out on all of the unassigned drives. I have installed everything you have requested as well as rebooted the system. All the drives show up, just the format option is grey and unclickable. Both when the array is up and down.

[JonathanM]

1 hour ago, Waffle said:

the first issue I have found is the "format" option is greyed out on all of the unassigned drives.

Either UD+ isn't installed, or destructive mode hasn't been enabled in UD settings.

[Waffle]

I am home now, and that was correct thank you. "destructive mode" was not on thank you!  I have however reached a new issue.  'HostMedia' does not seem to be present in my system. I did however go into each folder looking for 'driveX' and did not see that either. Maybe I set unraid up wrong? I have included pictures of krusader and my 'Main' tab for inspection.

 

 

[sota]

Look in the docker tab, for Krusader, and see what the volume mappings are.

 

If you edit the container, mine says:

 

[Waffle]

Here is the information you requested. I looked in the '/media' folder and did not see the drives in their as well. Included is a picture of that folder.