(Solved) unRAID Array Rebuild - What can I do and what not during rebuild?

[FlorinB]

Hi,

 

Let's assume I have a broken disk in array and I am replacing it. What I am allowed to do during rebuild and what not?

For example:

1. May I copy new files or do I need to wait until rebuild is completed.

2. May I read the files from array? It will be slower?

3. May I change the shares pemissions and add/delete users?

4. May I install/uninstall plugins?

 

Thank you.

 

Best regards,

Florin

 

[JonathanM]

In general, you are allowed to do any and all of what you listed.

 

However, any reads or writes will slow down the rebuild significantly. Any operation that requires stopping the array can't be done during the rebuild, once the rebuild is started if you stop the array the rebuild will have to start over from scratch.

 

It is in your best interest to allow the rebuild to complete ASAP, so you are protected from another failure sooner rather than later.

 

If this is an actual scenario and not a theoretical question, I recommend you post the diagnostic zip file so any show stoppers can be found before you proceed. In general, many disabled disks aren't actually bad, it's a cabling or controller error that caused the failed write, and if that's the case, rebuilding to a new disk may very well fail, and could cause another drive to drop, causing data loss.

[FlorinB]

Hi Jonathan,

 

Thanks for your answer. It is just a theoretical question.

 

At this moment I've just build the appliance, precleared all disks including the ssd. The motherboard X11SSM-F have 8 SATA connectors (the powersource is Be Quiet. Pure Power 10 ATX) . The disks which are installed are salvaged 2.5 inch disks (various brands: HGST, Samsung, Toshiba, Hitachi) from old notebooks plus a 250GB Samsung SSD for cache. I have also available 2 disks of 3.5 inch WD Caviar Green WD15EARD of 1.5 TB. Should I install those ones also into array, one as parity and the second as storage? Are the 3.5 inch disks which I have better than the 2.5 inch?

 

Additionally I will install a Syba SATA III 4 Port PCI-e x1 Controller to have in tota 12 SATA connectors. I know that is not the best option, but this is what I afford at this moment.

 

What looks strange for me is that already 3 of disks have the counter "199 UDMA CRC error count" bigger than 0. I had build previously a test unRAID on a Bqeel Z83V MINI PC, using an USB hub and 4 HDDs, without any UDMA CRC error. That looks somehow odd on the X11SSM-F, since the SATA connections should be far more better than SATA to USB adaptors.

 

Please comment my configuration and give me some tips and hints.

 

Thank you in advance.

 

 

Best regards,

Florin

 

 

[JonathanM]

45 minutes ago, FlorinB said:

Additionally I will install a Syba SATA III 4 Port PCI-e x1 Controller to have in tota 12 SATA connectors. I know that is not the best option, but this is what I afford at this moment.

That controller uses a Marvell chipset, which is no good for linux. Much better to get an LSI based controller.

 

46 minutes ago, FlorinB said:

The disks which are installed are salvaged 2.5 inch disks

This sounds like a recipe for disaster. When I first started using unraid many years ago, I thought "hey, no big deal if a drive goes bad, it's protected by parity". However, what you need to realize is that when a drive fails, ALL the other drives have to be read perfectly to reconstruct the failed drive. So, I ended up losing a whole drives worth of data because when it failed, one of my other drives wouldn't read properly.

 

Don't use any drives that you aren't sure are perfect, because when a drive fails, the rest of them must be perfect.

 

52 minutes ago, FlorinB said:

What looks strange for me is that already 3 of disks have the counter "199 UDMA CRC error count" bigger than 0.

 

Just make sure the error count doesn't increase.

[pwm]

1 hour ago, jonathanm said:

So, I ended up losing a whole drives worth of data because when it failed, one of my other drives wouldn't read properly.

 

Which is also the important reason why parity is intended to improve availability, but will not remove the need to backup important files. And even with an arbitrary number of parity disks, only an off-site backup will protect from a major fire.

[FlorinB]

2 hours ago, jonathanm said:

That controller uses a Marvell chipset, which is no good for linux. Much better to get an LSI based controller.

Ok - No way  for Syba SATA III 4 Port PCI-e x1 Controller. I will try to find a used LSI based controller.

 

2 hours ago, jonathanm said:

Don't use any drives that you aren't sure are perfect, because when a drive fails, the rest of them must be perfect.

 All the disks are thoroughly tested, but are just normal disks, not NAS designed like WD Red, Seagate IronWolf or HGST Deskstar.

 

3 hours ago, jonathanm said:

Just make sure the error count doesn't increase.

I will note down the counters for the "199 UDMA CRC error count" errors of each disk and keep an eye on them.

 

1 hour ago, pwm said:

3 hours ago, jonathanm said:

So, I ended up losing a whole drives worth of data because when it failed, one of my other drives wouldn't read properly.

 

Which is also the important reason why parity is intended to improve availability, but will not remove the need to backup important files. And even with an arbitrary number of parity disks, only an off-site backup will protect from a major fire.

I just hope I will not end up in losing more than a disk at a time.

Of course, I am aware that it is allowed by the system to lose no more than one disk at a time and one parity disk (only if there are 2 installed). In german speaking countries there is a saying in IT: "No backup, no compassion!".

 

I will setup my array with XFS filesystem, as according to UnRAID_v6_FileSystems "ReiserFS seems to be reaching end-of-life" and BTRFS  "it is not as well-tried as XFS and the recovery tools are not as good."

Beside the situation when I have a power failure, the system is not gracefully shut down or a filesystem check is required, is there any case when i need to start the array in maintenance mode?

 

On Array Status-Maintenance Mode it states that "When operating in Maintenance Mode, the array is Started as usual; however, the array disks and cache disk (if present) are not Mounted, and hence not exported either". That means the shares and everything are available as usual, in read only mode or not available at all?

[JonathanM]

8 minutes ago, FlorinB said:

not Mounted, and hence not exported

This

8 minutes ago, FlorinB said:

not available at all

 

 

10 minutes ago, FlorinB said:

I am aware that it is allowed by the system to lose no more than one disk at a time and one parity disk (only if there are 2 installed).

2 parity disks can rebuild any 2 disks, including 2 data disks.

 

10 minutes ago, FlorinB said:

I will setup my array with XFS filesystem, as according to UnRAID_v6_FileSystems "ReiserFS seems to be reaching end-of-life" and BTRFS  "it is not as well-tried as XFS and the recovery tools are not as good."

I agree with that generally, but BTRFS for data drives in unraid is highly regarded as well for some things, notably the ability to do snapshots (command line only right now) and native checksum error detection (NOT CORRECTION). Personally I use XFS, but others here have good luck with BTRFS. Do NOT use ReiserFS.

 

Maintenance mode is only needed for XFS or ReiserFS file system checks and corrections. Light BTRFS maintenance can be done with the drives mounted.

[pwm]

5 minutes ago, FlorinB said:

All the disks are thoroughly tested, but are just normal disks, not NAS designed like WD Red, Seagate IronWolf or HGST Deskstar.

 

Desktop-type disks can work fine. But do your best to see if you can keep down vibrations, because without a vibration sensor they will not abort a write operation if there is too much vibration - so there is a larger chance of the drive writing data that can't be read back and will end up flagged as "offline uncorrectable".

 

Disks performs the writes completely blind - they do not go back and check the result of the write operation, so if vibrations have made the head miss the alignment, the disk will not notice until a much later time when you either perform a surface scan or request the data to be read back.

 

8 minutes ago, FlorinB said:

I will note down the counters for the "199 UDMA CRC error count" errors of each disk and keep an eye on them.

 

No need to. unRAID can keep track of the "high water mark" for the counter for each disk. So unRAID will notify you again if the counter later changes.

 

10 minutes ago, FlorinB said:

Of course, I am aware that it is allowed by the system to lose no more than one disk at a time and one parity disk (only if there are 2 installed). In german speaking countries there is a saying in IT: "No backup, no compassion!".

 

With one parity drive, it's allowed to lose any disk in the array - data or parity.

With two parity drives, it's allowed to lose any two disks in the array - data or parity. So two data disks, or one data and one parity or both parity disks.

 

By the way - I like that german saying. Having backup is all about how much the user cares about the data.

 

12 minutes ago, FlorinB said:

ReiserFS seems to be reaching end-of-life" and BTRFS  "it is not as well-tried as XFS and the recovery tools are not as good."

 

I don't really know how good/bad the BTRFS recovery tools are - I have never had a BTRFS volume needing recovery.

But BTRFS is very stable when used with single-disk volumes (as in the unRAID array).

I have a huge number of BTRFS volumes in embedded equipment installed in vehicles with no separate battery support taking care of a hard power loss. Until now, they have worked very well without any failed volumes.

 

It's when moving to multi-disk volumes that you'll enter into the unknown - there are probably quite a lot of bugs left to catch. Especially if moving away from simple mirrors/stripes and trying something like a BTRFS RAID-5 volume.

[FlorinB]

21 hours ago, jonathanm said:

BTRFS for data drives in unraid is highly regarded as well for some things, notably the ability to do snapshots (command line only right now) and native checksum error detection

 

The ability to do snapshots sounds interesting for me as well as the native checksum error detection, but...

"More susceptibile to corruption if you have dirty shutdowns" does not sound too good. "Must people here, i'd say 95%+ are on XFS." - I will go with the crowd...use XFS.

 

22 hours ago, pwm said:

With one parity drive, it's allowed to lose any disk in the array - data or parity.

With two parity drives, it's allowed to lose any two disks in the array - data or parity. So two data disks, or one data and one parity or both parity disks.

Now I know for sure that I need 2 parity disks. However it is unclear for me how 2 parity disks are able to recover 2 data disks, even after reading the documentation from here and watching the 2 youtube videos: unRAID Parity Made Simple, as well as How does unRAID Parity works and why did I use it...?

 

The same question is also here unanswered:

Can someone explain me how the 2 parity disks works that 2 data disks are recovered?

 

[pwm]

2 minutes ago, FlorinB said:

"More susceptibile to corruption if you have dirty shutdowns" does not sound too good.

 

I'm not sure this really is true for the last year or two when it comes to single-volume BTRFS.

 

I have laboratory results from tens of thousands of hard power cuts while running software that performs lots of file writes before the power cuts. The equipment was running day after day with the power connected to a relay that regularly cut the power.

 

After every single reboot, the machine did roll back to the most recent stable state for all volumes and booted correctly with no volume corruption. Obviously, any open file being written to during the time of power loss will lose the most recent file writes - how much depending on the configured file system sync speed and the speed the data is written at.

 

1.5-2 years ago, the above test constantly resulted in broken file systems but then quite a number of BTRFS file system fixes was introduced in the kernel.

 

The state of multi-disk volumes is still very much a gray area for BTRFS - but since most other file systems don't support multi-disk volumes it's hard to compare stability for such usage. But there are quite a number of people starting support threads with broken multi-disk BTRFS caches, so I'm pretty sure the mirroring/striping code still has some juicy bugs that needs flushing.

 

14 minutes ago, FlorinB said:

Now I know for sure that I need 2 parity disks. However it is unclear for me how 2 parity disks are able to recover 2 data disks, even after reading the documentation from here and watching the 2 youtube videos: unRAID Parity Made Simple, as well as How does unRAID Parity works and why did I use it...?

 

It's all about math.

It's easy to describe the math for a single parity drive.

The math for a second or third or fourth parity are way more complicated.

 

But see it as a question of equations.

With one unknown, you can solve having only one equation:

1 + x = 3

x = 3 - 1 = 2

With two unknowns, you need two independent equations to be able to solve x an y.

1 + x + y = 3
1 + y = 9

y = 9 - 1 = 8
x = 3 - 9 - 1 = -7

The more unknowns (broken disks) you need to solve, the more independent equations (parity disks with independent parity computation formulas) you need.

So one parity using one math formula can solve one unknown - one broken disk.

And two parity disks using two independent formulas can solve two unknown - two broken disks.

 

Obviously, if it's a parity that is broken you just recompute the content of that disk.

While if it's a data disk, you solve the math problem of what value the disk needs to have so the full set of data disks will result in the expected parity.

[FlorinB]

1 hour ago, pwm said:

I have laboratory results from tens of thousands of hard power cuts while running software that performs lots of file writes before the power cuts. The equipment was running day after day with the power connected to a relay that regularly cut the power.

 

After every single reboot, the machine did roll back to the most recent stable state for all volumes and booted correctly with no volume corruption. Obviously, any open file being written to during the time of power loss will lose the most recent file writes - how much depending on the configured file system sync speed and the speed the data is written at.

Those are really good news. The only fear I had was just to not end up in a broken array due to a banal power failure. I understand and expect to have data loss on file writing in progress, that is normal. 

 

I have to do some research for BTRFS, the ability to make snapshots and the native checksum error detection is quite appealing for me. It does not matter too much if at this moment there is not gui support for snapshots.

 

1 hour ago, pwm said:

With two unknowns, you need two independent equations to be able to solve x an y.

This means that the two parity disks does not hold the same data? actually it is doing a kind of party for parity. On short if there are 2 data disks broken one parity disk will help to recover the first broken data disk and the second parity disk will help to recover the second data disk. As you,` @pwm said, the more parity disks the more complicated formula to recover the data.

 

 

 

[pwm]

11 minutes ago, FlorinB said:

This means that the two parity disks does not hold the same data? actually it is doing a kind of party for parity. On short if there are 2 data disks broken one parity disk will help to recover the first broken data disk and the second parity disk will help to recover the second data disk. As you,` @pwm said, the more parity disks the more complicated formula to recover the data.

 

Exactly - the two disks using different formulas to compute the parity.

 

But you can't use one of them to recover the first disk and then the second parity to recover the second disk. You use the combination of values from the two parity disks to figure out which of the four combinations on the data disks that are correct - if you slice four combinations into two twice, you'll end up with a single combination. And that's the correct answer of what data to write to the two missing disks.

 

With two lost data disks, you will have four possible values at every bit position 0+0, 0+1, 1+0 and 1+1

It's easiest to visualize it as two-dimensional problem where we place the four combinations into a 2 by 2 array.

 

One parity disk helps figure out if the two missing bits are in the top or bottom row.

And the other disk helps figure out if the two missing bits are in the left or right column.

So with both parity disks you can figure out the expected bit values for the two missing disks.

 

The first disk (P parity) is just a simple XOR operation and the order of the disks doesn't matter.

 

With the second disk (Q parity), the order of the disks does matter. That's required to make the two formulas independent so that they, when combined, can be used to solve two unknowns - i.e. the contents of two arbitrary disks lost.

 

The main complexity increase happens going from first to second parity. The math for a third or fourth parity disk basically adds dimensions to the problem.

 

But if you run with more than one parity disk, it's very important that you keep track of the array location of all disks - i.e. which disk is data1, data2, data3, ... parity1, parity2.

With just the first parity, it's enough to know which disk is parity and which of the remaining disks are data disks.

[FlorinB]

23 minutes ago, pwm said:

But if you run with more than one parity disk, it's very important that you keep track of the array location of all disks - i.e. which disk is data1, data2, data3, ... parity1, parity2.

With just the first parity, it's enough to know which disk is parity and which of the remaining disks are data disks.

Do i need to make some screenshots with the disks order or there is somewhere a config/log file which I can save with a timestamp?

 

 

[JonathanM]

31 minutes ago, FlorinB said:

Do i need to make some screenshots with the disks order or there is somewhere a config/log file which I can save with a timestamp?

If you set up the recommend email status messages you will get 1 message a day with the list of disks, and the status. Very handy.

[Frank1940]

3 hours ago, FlorinB said:

Now I know for sure that I need 2 parity disks. However it is unclear for me how 2 parity disks are able to recover 2 data disks, even after reading the documentation from here and watching the 2 youtube videos: unRAID Parity Made Simple, as well as How does unRAID Parity works and why did I use it...?

 

 

The reason is that dual parity is NOT simple to explain!  You might want to read this thread:

 

          https://lime-technology.com/forums/topic/39334-lets-talk-about-raid-6/

 

 

While this discussion was way back when dual parity was being considered, there are links in this thread to papers that discuss the math behind the calculation of Q.  As I recall, Reed-Solomon was the one that was actually implemented in the final code.  You can google that term for a lot of papers that will make your head swim unless you are working on a PHD in Mathematics.  

[FlorinB]

Thank you very much @jonathanm, @pwm and @Frank1940 for your answers.

 

Now, thanks to you,  is all clear for me.

[tr0910]

@pwm regarding your labratory results with cutting power to btrfs during write operations. Was this to spinning media, SSD, or USB? Would you expect the results to differ depending on media?

 

 

 

 

 

[pwm]

21 minutes ago, tr0910 said:

@pwm regarding your labratory results with cutting power to btrfs during write operations. Was this to spinning media, SSD, or USB? Would you expect the results to differ depending on media?

 

All the tests were done with an embedded device equipped with industrial eMMC flash media.

With the latest patches available last summer, all power cycles went well with no broken file system.

With an older kernel, the file systems did regularly break badly.

One possible caveat is that unRAID versions older than 12-24 months may lack important BTRFS bug fixes. But I haven't followed up exactly when the BTRFS patches were released  and exactly when they got merged into the Slack kernel and when unRAID picked them up.

 

It's my belief BTRFS would be best tested for HDD media and worst for flash media (SSD/USB) given that it's quite hard to know exactly what decisions the flash controller chips does to optimize performance while keeping down the amount of wear. BTRFS like most other file systems is designed based on HDD use with native LBA support, while flash media have way larger blocks and has to implement a virtualization layer just to pretend that there are 512 byte or 4096 byte sectors.

 

One implication of this is that a lot depends on how well the flash media itself handles power losses. A bad flash memory controller that can't handle the removal of power while an erase or write operation is ongoing or that may sometime lose the mapping between the LBA the OS sees and regions on the raw flash blocks would obviously give bad results. But that wouldn't be a limitation of BTRFS but a limitation of the used storage device.

[FlorinB]

On 6/3/2018 at 9:44 PM, FlorinB said:

Can someone explain me how the 2 parity disks works that 2 data disks are recovered?

Tested myself the above scenario as follows:

On my test UnRAID there are 2 parity disks and 2 data disks. I had intentinally removed the 2 data disks and replaced them with other 2 new disks. You need to assign manually the new disks to the Array then the rebuild is starting automatically.

 

While the rebuild was in progress the 2 missing disks were emulated by the 2 parity disks. See below image Disk3 and Disk4 "Device Contents Emulated"

 

After 3h, 15 min and 50 sec the Array returned to normal.

 

 

[JonathanM]

27 minutes ago, FlorinB said:

Tested myself the above scenario as follows:

On my test UnRAID there are 2 parity disks and 2 data disks. I had intentinally removed the 2 data disks and replaced them with other 2 new disks. You need to assign manually the new disks to the Array then the rebuild is starting automatically.

Trust but verify, eh? ?

[FlorinB]

37 minutes ago, jonathanm said:

Trust but verify, eh? ?

In theory, theory and practice are the same. In practice, they are not.
Albert Einstein

 

I think is important not only to undertsand how it is working, but also to know what to do when it happens.

 

By the way, I am still not decided which filesystem to use. For BTRFS everybody is saying just that you need to do some command line to manage the snapshots, but I could not find, on our forum, a tutorial with how to do a snapshot, restore a snapshot, see how much of disk is occupied by a snapshot, by all snapshots and so on...

[JonathanM]

1 hour ago, FlorinB said:

For BTRFS everybody is saying just that you need to do some command line to manage the snapshots, but I could not find, on our forum, a tutorial with how to do a snapshot, restore a snapshot, see how much of disk is occupied by a snapshot, by all snapshots and so on...

@johnnie.black is the fellow that is active on the forum about BTRFS, perhaps a google search with his user id, unraid, and btrfs snapshot may yield some answers.

[JorgeB]

9 hours ago, FlorinB said:

By the way, I am still not decided which filesystem to use. For BTRFS everybody is saying just that you need to do some command line to manage the snapshots, but I could not find, on our forum, a tutorial with how to do a snapshot, restore a snapshot, see how much of disk is occupied by a snapshot, by all snapshots and so on...

Some info here on how to use snapshots to backups VMs, but principle is the same, I also snapshot all the disks on my backups servers before a sync, to restore you just need to copy the data over, how much disk space snapshots are using is currently not easy with btrfs, you'd need to enable quotas, and that isn't 100% stable and has a performance impact.

[FlorinB]

And the filesystem is ... BTRFS

Already installed a Windows 10, Lubuntu 18.04 and one Docker application.

 

Thanks @johnnie.black for the info and tips. I was very happy when I read that I can assign quotas as well, but since is not stable enough, I will not use that feature.

 

The caching disk option is quite cool! Most of the time the other disks are stopped, the CPU temperature is at 35 C and case at 29.8 C, with the case fan at lower speed.

 

[pwm]

One thing with snapshots is that they, themselves, do not consume any disk space at all except for a tiny amount for book keeping.

 

But since BTRFS uses CoW (Copy-on-Write), any changes performed after the snapshot is made will make the file system deviate between snapshot and "main" view. So the more changes that are introduced, the more additional disk space will be consumed to remember the older and now overwritten data within the snapshots.

 

So the disk space costs of using snapshots will depend on how long they are kept, and how much changes are introduced between two snapshots.