Skip to content
View in the app

A better way to browse. Learn more.

Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

Need help: rebuilding Array after controller failure - two array disks seem to have a broken BTRFS file system

Featured Replies

  • Community Expert

Try the other backup but most likely will be the same:


 

btrfs-select-super -s 2 /dev/md1p1

btrfs-select-super -s 2 /dev/md3p1

 

  • Community Expert
37 minutes ago, fusselnerd said:

Can you recommend a read about btrfs and Unraid array? I'm missing the fundamentals, obviously, so I'd like to dig into it a bit.

Not sure if it would be easy to find specif btrfs info for the array, but if you search for btrfs in general there are lots of resources on the net, Unraid uses regular btrfs.

  • Author

Yeah, it's the same result:

root@GrayBigBerta:~# btrfs-select-super -s 2 /dev/md1p1
ERROR: superblock checksum mismatch
ERROR: superblock checksum mismatch
No valid Btrfs found on /dev/md1p1
ERROR: open ctree failed
root@GrayBigBerta:~# btrfs-select-super -s 2 /dev/md3p1
ERROR: superblock checksum mismatch
ERROR: superblock checksum mismatch
No valid Btrfs found on /dev/md3p1
ERROR: open ctree failed

 

  • Author
Just now, JorgeB said:

Not sure if it would be easy to find specif btrfs info for the array, but if you search for btrfs in general there are lots of resources on the net, Unraid uses regular btrfs.

Thank you! 🙏

  • Community Expert
6 minutes ago, fusselnerd said:
No valid Btrfs found on /dev/md1p1
ERROR: open ctree failed

I'm afraid the backup superblocks are also damaged, best option now would be to use something like UFS explorer to scan the disks and see if it can recover the data, the free trial should allow you to see if that's possible, before having to pay for for it.

  • 1 month later...
  • Author

Hi @JorgeB

 

Hi finally had time to do the recovery: with UFS I managed to restore almost all data and backed it up. Thank you very much for that tip.

 

What will be the next step?

  • Community Expert

Post a screenshot of main to refresh my memory as to the current array status.

  • Author

Sure

 

grafik.thumb.png.a284711d6b7ba95a3ea7616321265d42.png

[...left out pool and single disks...]

grafik.thumb.png.f066a1ba470ec94eca01d104ce7696ed.png

  • Community Expert

And both the emulated disks 1 and 3 are not mounting correct? Same for the actual disks using UD?

  • Author

Correct. If I mount them or start the array with emulated disks I get the error "Unmountable... missing file system" or so. In syslog it's shown that the BTRFS filesystem ist corruped (ERROR: superblock checksum mismatch). We tried to restore the superblock but it was not possible because I started the rebuild by mistake.

 

Now, I restored the actual data by hand (UFS).

I didn't assign the disks or started the array since then.

 

 

PS: You also mentioned in a previous post that I should scrub the other disks and reset the errors.

  • Community Expert

If you have already recovered the data, you can do a new config to assign the disabled disks, then start the array to re-sync parity, you can then format the disks and restore the data.

  • Author

Allright, I did a new config with all disks assigned the same as before except the two disabled disks. Parity is syncing now. Then I'll format the two disks, assign them and recover the backed up data.

This might take a couple of days.

I'll get back to you when it's finished for the last steps (I guess), scrubbing and error-reset.

 

Thank you so far.

  • Community Expert
1 hour ago, fusselnerd said:

except the two disabled disks. Parity is syncing now. Then I'll format the two disks, assign them

This is the wrong order to do things.

 

If you did New Config without the disks, then they are not part of parity. If you assign the disks after parity builds, Unraid will have to clear them so parity remains valid. After the disks have been assigned and cleared, then you can format them.

  • Community Expert
1 hour ago, fusselnerd said:

This might take a couple of days.

I'll get back to you when it's finished

@fusselnerd

See my post above

  • Author

Hi @trurl

Oh I see, thank you for correcting me!

 

I stopped the sync and did a New Config, again, and assigned the two disks.

I would now start the array to start the parity-sync.

 

Is this the correct order?

  • Community Expert
1 minute ago, fusselnerd said:

Is this the correct order?

yes.

 

You can format them after parity builds, or even while parity builds, and the format (filesystem creation) will be part of parity just like any other write to the array.

 

Be sure to notice the list of disks that will be formatted. DO NOT enable and click FORMAT if any other disks are listed.

  • Author

Thanks a lot!

 

I started the sync and formated both disks. Also, no other disks were listed.

 

grafik.thumb.png.3a9fed552ab97816c5c0a722a1121947.png

  • Community Expert

That screenshot is clipped on the right, so doesn't include enough information to reassure me about your disk contents (USED FREE).

  • Author

oh sorry, here we go

 

grafik.thumb.png.1e454c08f59483b82a7e387feb9bcdaa.png

  • Community Expert

Looks good, let the sync finish, and then restore the data.

  • Author

Hi,

Sync and data restore finally finished without errors. Also no errors in syslog and drive logs. Respective shares are shown as protected.

  • Author

As you suggested here

 

On 7/4/2024 at 12:49 PM, JorgeB said:

You will also need to scrub all the disks that have corruption detected, then reset the errors and monitor to see if new ones come up, but this is for later.

 

I checked the other disks for errors with

btrfs dev stats /mnt/<disk>

 

and it turned out that all disks have errors:

root@GrayBigBerta:/mnt# btrfs dev stats /mnt/disk2
[/dev/md2p1].write_io_errs    0
[/dev/md2p1].read_io_errs     0
[/dev/md2p1].flush_io_errs    0
[/dev/md2p1].corruption_errs  22
[/dev/md2p1].generation_errs  0
root@GrayBigBerta:/mnt# btrfs dev stats /mnt/disk4
[/dev/md4p1].write_io_errs    359
[/dev/md4p1].read_io_errs     1
[/dev/md4p1].flush_io_errs    0
[/dev/md4p1].corruption_errs  71
[/dev/md4p1].generation_errs  0
root@GrayBigBerta:/mnt# btrfs dev stats /mnt/disk5
[/dev/md5p1].write_io_errs    0
[/dev/md5p1].read_io_errs     0
[/dev/md5p1].flush_io_errs    0
[/dev/md5p1].corruption_errs  287
[/dev/md5p1].generation_errs  0
root@GrayBigBerta:/mnt# btrfs dev stats /mnt/disk6
[/dev/md6p1].write_io_errs    0
[/dev/md6p1].read_io_errs     0
[/dev/md6p1].flush_io_errs    0
[/dev/md6p1].corruption_errs  12
[/dev/md6p1].generation_errs  0

 

except the recovered disks:

root@GrayBigBerta:/mnt# btrfs dev stats /mnt/disk1
[/dev/md1p1].write_io_errs    0
[/dev/md1p1].read_io_errs     0
[/dev/md1p1].flush_io_errs    0
[/dev/md1p1].corruption_errs  0
[/dev/md1p1].generation_errs  0
root@GrayBigBerta:/mnt# btrfs dev stats /mnt/disk3
[/dev/md3p1].write_io_errs    0
[/dev/md3p1].read_io_errs     0
[/dev/md3p1].flush_io_errs    0
[/dev/md3p1].corruption_errs  0
[/dev/md3p1].generation_errs  0

 

 

Questions:

I intend to run scrub in a small script to scrub all disks after one after another.

What parameters should I use?

Will this do (for each disk)?

btrfs scrub start -B /dev/disk#

 

As I understand array disks have single disk file systems. Therefore automatic repair should not be possible, right?

Should I use the -r option?

  • Community Expert
51 minutes ago, fusselnerd said:

As I understand array disks have single disk file systems. Therefore automatic repair should not be possible, right?

It can still be possible for metadata, but not for data, but always run a correcting scrub anyway, since the other one doesn't have much point.

 

52 minutes ago, fusselnerd said:
btrfs scrub start -B /dev/disk#

That should work, but you can also use the built in scheduler.

  • Author

Scrub of all disks resultet in four currupted files.

I deleted the files and resetted the btrfs scrub stats, then run scrub again: 0 errors.

 

Is there anything else I should check/do before bringing the server online?

  • Community Expert

Should be OK, then just make sure to monitor/scrub for any new errors in a month or so.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

Account

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.