Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

[dlandon]

I've issued an update to UD that turns the diskinfo from preclear back on.  When the preclear plugin is installed, there is a background daemon that preclear uses for disk information to cut down on disk queries.  UD also takes advantage of this diskinfo.  This was done because a preclearing operation can appear hung, or stop completely when there is too much disk query activity on the preclearing disk.

 

UD now checks to be sure the rc.diskinfo daemon is running before using the diskinfo.json file for disk status.  If the daemon is not running the diskinfo.json file will not be current and shouldn't be relied on.

 

There were also a duplicate mounting attempt on ntfs disks with issues like an unclean shutdown, or a hibernated disk.  The ntfs mount is successful in these cases, but the disks will be mounted as read only.  This has apparently changed in the latest release of Linux and/or the ntfs driver.  In the past when there was an error from these situations, the disk would not be mounted and UD would have to mount the disk read only.  UD now checks that the disk is actually mounted and won't try another mount unless it actually did not mount.  Let me know if you have any issues with mounting ntfs disks.

[SSD]

Forgive if this has been asked before.

 

But would it be possible to make unassigned devices partition and format disks in an unRAID compatible format? There are a lot of use cases where being able to create a compatible disk would be very handy, and instructions to do so can get complicated (e.g., booting a trial version).

[dlandon]

10 minutes ago, bjp999 said:

Forgive if this has been asked before.

 

But would it be possible to make unassigned devices partition and format disks in an unRAID compatible format? There are a lot of use cases where being able to create a compatible disk would be very handy, and instructions to do so can get complicated (e.g., booting a trial version).

This has been asked before.  Why would someone want to format a disk in UD for the array?  This is not what UD was designed to do.

[SSD]

When helping people with some tricky array problems, being able to have a get them to prepare an unRAID compatible disk would be useful. This is particularly needed with older arrays demonstrating multiple drive issues. I'd rather not have them do multiple rounds of rebuilds, but instead copy the data to new disks and then form a new array when complete.

 

I also have a relatively frequent need. I will often buy a set of new drives (say 2 8T drives), that I want to use to replace several smaller drives (say 3 4T drives). I want to install the 8T drives outside the array, copy the data to the new drives, run md5 comparisons and ensure all is good, and then do a new config excluding the 4T drives and including the 8T drives with a single parity build.
 

The I/O to the unassigned drives is fast. And the array only has to do one rebuild cycle. And I am free to shuffle the content as desired. Any disk fragmentation is taken care of with the fresh copies (unlike a rebuild that "inherits" the fragmentation).

 

And I am left with the 3 4T drives with data intact that I can use for backups.

 

Doing this in other ways is more complicated and time consuming, and overall more disruptive to my usage.

 

(I do take add'l precautions to monitor smart reports and run a parity check soon before the exchange to protect against failure during the rebuild, and have other techniques to recover even if there is a failure during the build.)

 

Formatting in unRAID format is not THAT difficult without this feature. The cache slot can be used - but if you have a cache drive or cache pool supporting VMs and Dockers it is not so obvious that you can do so without creating a problem for yourself. But I recently did exactly that with my single cache drive that contained a set of Dockers. Replacing it with a real disk that I wanted to format disabled docker functionality. After formatting it, I stopped the array and install the next disk to format, until all disks are formatted, And re-installing the correct cache drive everything was put right. I have not tried with cache pools or with VM feature enables, and would not wholeheartedly recommend this method if you are not confident in your ability to recover if things went wrong.

 

Another way is to use a trial key. This is foolproof but a bit of a PITA.

 

A good way, IMO, is to create a temporary config folder on your real USB key, and boot with that. With the array stopped, rename the config folder (e.g, "config-real") and create a new "config" folder. Copy the primary config files, like ident and network and disk, as well as your USB-specific ".key" file. No "super.dat" file. But have no plugins or other stuff that might confuse things. And no super.dat (although that is not really a problem if you include it). You can boot, create an "array" of the few disks to format, format the disks, stop the array, rename the config folder to "config-format" or something, and rename the "config-real" to "config". When you reboot all is as it was, except the disks are formatted. UD works great with the disks formatted like this. And unRAID is happy with them as well.

 

I believe this would be a nice enhancement to UD to avoid all this rebooting and renaming, and make these techniques easier for users.

[dlandon]

25 minutes ago, bjp999 said:

When helping people with some tricky array problems, being able to have a get them to prepare an unRAID compatible disk would be useful. This is particularly needed with older arrays demonstrating multiple drive issues. I'd rather not have them do multiple rounds of rebuilds, but instead copy the data to new disks and then form a new array when complete.

 

I also have a relatively frequent need. I will often buy a set of new drives (say 2 8T drives), that I want to use to replace several smaller drives (say 3 4T drives). I want to install the 8T drives outside the array, copy the data to the new drives, run md5 comparisons and ensure all is good, and then do a new config excluding the 4T drives and including the 8T drives with a single parity build.
 

The I/O to the unassigned drives is fast. And the array only has to do one rebuild cycle. And I am free to shuffle the content as desired. Any disk fragmentation is taken care of with the fresh copies (unlike a rebuild that "inherits" the fragmentation).

 

And I am left with the 3 4T drives with data intact that I can use for backups.

 

Doing this in other ways is more complicated and time consuming, and overall more disruptive to my usage.

 

(I do take add'l precautions to monitor smart reports and run a parity check soon before the exchange to protect against failure during the rebuild, and have other techniques to recover even if there is a failure during the build.)

 

Formatting in unRAID format is not THAT difficult without this feature. The cache slot can be used - but if you have a cache drive or cache pool supporting VMs and Dockers it is not so obvious that you can do so without creating a problem for yourself. But I recently did exactly that with my single cache drive that contained a set of Dockers. Replacing it with a real disk that I wanted to format disabled docker functionality. After formatting it, I stopped the array and install the next disk to format, until all disks are formatted, And re-installing the correct cache drive everything was put right. I have not tried with cache pools or with VM feature enables, and would not wholeheartedly recommend this method if you are not confident in your ability to recover if things went wrong.

 

Another way is to use a trial key. This is foolproof but a bit of a PITA.

 

A good way, IMO, is to create a temporary config folder on your real USB key, and boot with that. With the array stopped, rename the config folder (e.g, "config-real") and create a new "config" folder. Copy the primary config files, like ident and network and disk, as well as your USB-specific ".key" file. No "super.dat" file. But have no plugins or other stuff that might confuse things. And no super.dat (although that is not really a problem if you include it). You can boot, create an "array" of the few disks to format, format the disks, stop the array, rename the config folder to "config-format" or something, and rename the "config-real" to "config". When you reboot all is as it was, except the disks are formatted. UD works great with the disks formatted like this. And unRAID is happy with them as well.

 

I believe this would be a nice enhancement to UD to avoid all this rebooting and renaming, and make these techniques easier for users.

This appears to be an edge case that would not be worth the investment in time on my part to implement.  I feel that it is outside the scope of what UD was designed to do.  Using UD is not the proper way to prepare a disk for the array.

 

I could see a lot of support come up when users try to perform the odd gyrations you describe here.  Your process might be fine for you, but it is rife with potential errors by the casual user.  When replacing a disk in unRAID, just install it and let it rebuild.  Why use unRAID and then don't trust it to rebuild a disk?  Use dual parity for a multiple disk failure situation.

 

If you need this capability desperately, ask LT to add a feature to format a new disk not in the array to be array compatible.

[SSD]

1 hour ago, dlandon said:

This appears to be an edge case that would not be worth the investment in time on my part to implement.  I feel that it is outside the scope of what UD was designed to do.  Using UD is not the proper way to prepare a disk for the array.

 

I could see a lot of support come up when users try to perform the odd gyrations you describe here.  Your process might be fine for you, but it is rife with potential errors by the casual user.  When replacing a disk in unRAID, just install it and let it rebuild.  Why use unRAID and then don't trust it to rebuild a disk?  Use dual parity for a multiple disk failure situation.

 

If you need this capability desperately, ask LT to add a feature to format a new disk not in the array to be array compatible.

 

I never said that I didn't trust unRAID to rebuild a disk. Or I was desperately in need of this feature (I explained three ways to do it!) Please reread my post.

 

I think I articulated my use cases pretty well. And made a reasonable suggestion. You can take it or leave it, but I'd prefer you didn't insult the idea and tell me my its not worth your time.

 

And I asked not on behalf of myself. And I am not even a UD user! I just think it belongs in UD. Your users do not know of your decision to partition disks incompatible with unRAID disks. A number of users have used UD to format disks, and then copied data to them, only to find out they can't and having to redo a lengthy set of operation. They are mystified and blame unRAID because they don't know enough to trace it back to you. And I believe for no good reason except it wasn't worth your time to understand the unRAID partitioning and formatting commands.

 

How would you have a user with a 2011 vintage array with 4 disks showing reallocated sectors, one drive dropped from the array, and priceless (from their perspective) data to recover? Add dual parity? (ROFL) Scenarios like this has happened 2-3 times in the last couple months. My suggestion would help those users who you refer to as edge cases.

 

Later.

[BRiT]

I'd tell them the exact same thing everyone here would tell them, Restore from backups.

[SSD]

56 minutes ago, BRiT said:

I'd tell them the exact same thing everyone here would tell them, Restore from backups.

 

You're a hard man @BRiT!  

 

But it is a fair point and I agree with you this far - users should have backups of their critical data. But problem is, many do not have backups at all. And even those that do, may not backup voluminous data that is "recoverable in other ways", that they take a calculated risk they'd never have to recover. Does that mean the support forum should not promote an option to maximize the value of unRAID and offer the very best chance of recovering the data? And give the user the control to decide which data is most valuable and direct that it be the priority to recover?

 

Offering such a user a way to put in a new disk, partition and format it so that it could be added into unRAID later, and using Krusader, mc, cp, or something else to copy the data over in a selective manner, is what I would be advising. And today, in order to give that option, I need to explain a complicated multi-reboot process to format their new 8T drive for unRAID and mount it outside the array. I'd rather explain to install UD, do these three clicks, and they are ready to go.

 

If the array is too far gone, and the user doesn't have backups, shame on them. But if the array is recoverable to at least some degree, and we as the support forum do not make the option available to give them the best chance of recovery, shame on us. That's how I feel anyway.

[saarg]

1 hour ago, bjp999 said:

 

I never said that I didn't trust unRAID to rebuild a disk. Or I was desperately in need of this feature (I explained three ways to do it!) Please reread my post.

 

I think I articulated my use cases pretty well. And made a reasonable suggestion. You can take it or leave it, but I'd prefer you didn't insult the idea and tell me my its not worth your time.

 

And I asked not on behalf of myself. And I am not even a UD user! I just think it belongs in UD. Your users do not know of your decision to partition disks incompatible with unRAID disks. A number of users have used UD to format disks, and then copied data to them, only to find out they can't and having to redo a lengthy set of operation. They are mystified and blame unRAID because they don't know enough to trace it back to you. And I believe for no good reason except it wasn't worth your time to understand the unRAID partitioning and formatting commands.

 

How would you have a user with a 2011 vintage array with 4 disks showing reallocated sectors, one drive dropped from the array, and priceless (from their perspective) data to recover? Add dual parity? (ROFL) Scenarios like this has happened 2-3 times in the last couple months. My suggestion would help those users who you refer to as edge cases.

 

Later.

 

Not quite fair to blame this on @dlandon is it? As you are aware he makes this plugin on his free time and it's up to him to decide if it's worth it or not to implement. He is after all not paid to develop this plugin.

Better to push this on LT to make it simpler if you mean this is an important feature, in my opinion.

[JorgeB]

10 minutes ago, bjp999 said:

And today, in order to give that option, I need to explain a complicated multi-reboot process to format their new 8T drive for unRAID and mount it outside the array. I'd rather explain to install UD, do these three clicks, and they are ready to go.

 

I made the same request earlier basically for the same reasons, so unknowing users don't shoot themselves in the foot, dlandon said he didn't plan to change that and I respect that since he's the one doing the work, but in case you need to help a user it's very easy to partition a GPT disk (any disk >2TB) outside the array, you can use:

 

sgdisk -o -a 64 -n 1:64:0 /dev/sdX

 

This is the same command unRAID uses and the user can then use the UD plugin to format the disk and it can be used as an array disk later.

 

MBR disks (any disk up to 2TB) are different, unRAID  generates the MBR contents in a buffer and then writes it to sector 0 of the device, so I don't know a single line command to use, but by now most new disks should be GPT anyway.

[dlandon]

When I said it was not worth my time, I meant that I invest what little spare time I have to features that are useful to a wider crowd.  I don't have the time to invest in features that a small group of users will benefit from when there are other answers - i.e. proper backups.  I do not get paid for any of this work, and I get no benefit from the license fee that LT charges.  If there is a shortcoming in unRAID, then you should be lobbying LT for the feature.

 

I'm sorry you feel that I didn't take your ideas seriously, but trying to sell me on this with all your reasoning does not change the fact that I just don't think it appropriate for UD to format disks for the array.

[dlandon]

2 hours ago, bjp999 said:

And I asked not on behalf of myself. And I am not even a UD user! I just think it belongs in UD. Your users do not know of your decision to partition disks incompatible with unRAID disks. A number of users have used UD to format disks, and then copied data to them, only to find out they can't and having to redo a lengthy set of operation. They are mystified and blame unRAID because they don't know enough to trace it back to you. And I believe for no good reason except it wasn't worth your time to understand the unRAID partitioning and formatting commands.

UD Help text:

Note: A disk formatted in UD is not compatible with the array disk format and cannot be added to the array.

Users are forewarned.  Yea, I know you'll tell me no one reads the help.

 

UD formats the disks in the format most compatible for the disk and file system.  It is not doing some bizarre format.  And, yes I do not care to invest the time in understanding the unRAID partitioning and formatting commands.

[SSD]

32 minutes ago, saarg said:

 

Not quite fair to blame this on @dlandon is it? As you are aware he makes this plugin on his free time and it's up to him to decide if it's worth it or not to implement. He is after all not paid to develop this plugin.

Better to push this on LT to make it simpler if you mean this is an important feature, in my opinion.

 

Hey - we all give our time. Have any idea how many hours I put into unmenu/mymain? The faster preclear script? The wiki? I've handled some of the most complicated data recovery scenarios in the forum taking hours of research and extensive step-by-step instructions. I'm not demanding anything of anyone. I just made a suggestion - anyone should be able to do that - and expect a minimum of a friendly response thanking for the insight.

 

@johnnie.black also has requested the use of the unRAID disk format. It would be quite useful and users would benefit.

 

@dlandon - thanks for your excellent contribution on UD. Putting it all together into this plugin is quite a lot of benefit to unRAID users. On this topic, we'll just agree to disagree.

 

Peace.

[dlandon]

4 minutes ago, bjp999 said:

 

Hey - we all give our time. Have any idea how many hours I put into unmenu/mymain? The faster preclear script? The wiki? I've handled some of the most complicated data recovery scenarios in the forum taking hours of research and extensive step-by-step instructions. I'm not demanding anything of anyone. I just made a suggestion - anyone should be able to do that - and expect a minimum of a friendly response thanking for the insight.

 

@johnnie.black also has requested the use of the unRAID disk format. It would be quite useful and users would benefit.

 

@dlandon - thanks for your excellent contribution on UD. Putting it all together into this plugin is quite a lot of benefit to unRAID users. On this topic, we'll just agree to disagree.

 

Peace.

Adding this to the format dialog:

Couldn't be any clearer than that.

 

It's not a matter of disagreeing.  I think the feature would be useful, just not in the UD plugin.

[JorgeB]

14 minutes ago, dlandon said:

Adding this to the format dialog:

 

Thanks for adding that, technically the problem is only when partitioning the disk, formatting is OK, but probably best to leave it like that or it may be confusing for some users.

[dlandon]

1 minute ago, johnnie.black said:

 

Thanks for adding that, technically the problem is only when partitioning the disk, formatting is OK, but probably best to leave it like that or it may be confusing for some users.

Correct, but the uninformed user is warned ahead of time that the disk is not going to be compatible with the array.  Whether or not it's technically the format or the partition is not really important because the format operation in UD does the format and partitioning in one operation and the user is given notice ahead of time.

[dlandon]

28 minutes ago, johnnie.black said:

 

Thanks for adding that, technically the problem is only when partitioning the disk, formatting is OK, but probably best to leave it like that or it may be confusing for some users.

Better?

[JonathanM]

Honestly, ever since @limetech made the decision to base licensing on all attached devices instead of assigned array devices, it became their responsibility to properly manage unassigned devices. I think it's very unfair to @dlandon to expect him to support UD at all. I suspect the only reason he does is the overwhelming need, but I would totally understand if he decided to just abandon it so limetech HAS to pick up the slack.

 

This has become one of the core functions of unraid. It doesn't belong in community support, it belongs in limetech's realm.

[dlandon]

1 minute ago, jonathanm said:

Honestly, ever since @limetech made the decision to base licensing on all attached devices instead of assigned array devices, it became their responsibility to properly manage unassigned devices. I think it's very unfair to @dlandon to expect him to support UD at all. I suspect the only reason he does is the overwhelming need, but I would totally understand if he decided to just abandon it so limetech HAS to pick up the slack.

 

This has become one of the core functions of unraid. It doesn't belong in community support, it belongs in limetech's realm.

I agree.  One of the main reasons I continue with it is because it is so damn useful to me.  I keep a disk mounted to do daily backups (using a scheduled User Scripts script) and I have USB drives I plug in to create on and off site backups.  Quite nice to just plug in a USB drive and have it fire off a script to do a backup and then let me know with a notification it is done.

 

I know a lot of people are mounting UD disks for VMs and Docker use.  I don't do that, but I can see the value of UD in that case.

 

I won't abandon UD, but eventually I expect LT will have pick up the UD functionality and include it in unRAID.  It has become a "must have" plugin.

 

I finally got to that point with the powerdown script.  It was becoming very difficult for me to keep up with the changes, and it was time for LT to clean up the powerdown process.  They did it and it works much better than the powerdown plugin.

[dlandon]

I finally settled on this:

It's more descriptive of the operation being performed - "Format and Partition", not just "Format".  And the warning is about the partition not being compatible.  The format is not really the problem as @johnnie.black posted earlier.  Technically correct now.

[Squid]

1 hour ago, jonathanm said:

Honestly, ever since @limetech made the decision to base licensing on all attached devices instead of assigned array devices,

Well, in their defense with more and more users simply using unRaid as a VM manager and passed-through devices and drives you can't really blame them for how they handle things...

[BRiT]

2 minutes ago, Squid said:

Well, in their defense with more and more users simply using unRaid as a VM manager and passed-through devices and drives you can't really blame them for how they handle things...

 

There is never a need to involve logic when assigning blame.

[limetech]

2 hours ago, dlandon said:

won't abandon UD, but eventually I expect LT will have pick up the UD functionality and include it in unRAID.  It has become a "must have" plugin.

 

With your permission we'd be happy to integrate into 'webGui'.  You can still work on it if you wish by submitting Pull Requests against our code base on github.

[dlandon]

10 minutes ago, limetech said:

 

With your permission we'd be happy to integrate into 'webGui'.  You can still work on it if you wish by submitting Pull Requests against our code base on github.

I think that is a great idea.  Permission granted.

[ruthan]

Hello i just want to ask, if Its possible to mount partition from unassigned devices into unraid VM, and if yes how?