After 3 years Unraid is being wonky, dockers are not working, drives keep failing


Recommended Posts

  • 3 weeks later...

Here's my update. I let the rebuild and parity checks finish, and I still had a dead drive. I decided to just remove the 6 TB drive and replace it with a larger one. I got a good deal on a 16 TB seagate, so I swapped out the 14 TB drive in parity for the larger 16 TB drive after a preclear. Both parity drives are now 16tb. I let parity rebuild onto the new drive, and swapped out the dead 6 TB for the 14 TB drive (that has always worked fine). The data rebuilt and I had all drives green! But, the last 2 drives still show up as unmountable.

 

I did some digging into the Marvell raid controller. I initially thought it had to be the motherboard since it says it has 4 ports, but nope, it's the expansion card which has 6 ports on it. I decided to stop the array and see if I could troubleshoot, and this is what I saw next. Only 2 drives showed up as attached and the rest were missing.

 

image.thumb.png.6f3e883241be40c26d9bcc570883dd7e.png

 

I downloaded the diagnostics and rebooted, now the drives are all back.

 

image.thumb.png.5024fd26d6786f574718d4f1bfe04d00.png

 

Disks 6 and 7 still show up as unmountable.

 

My thoughts are that I should replace the sata controller and see if that helps.

My question at this point is do I need to do something with the unmountable drives? I know disk 6 is a good disk since it's been in use as a parity drive and I did not unplug or move the port it is connected to, just assigned it in the UI.

 

I also attached the diagnostics for after I rebooted if that helps. It's the one that ends in "-2".

 

Thanks!

orthanc-diagnostics-20220807-2206.zip orthanc-diagnostics-20220807-2219-2.zip

Link to comment

Well, I just reviewed the thread after all these weeks, and though a lot was said about unmountable disks and filesystem corruption, I don't think anything was said about what to do about it.

 

Fixing hardware won't fix filesystem corruption, since the corruption already happened. And rebuild won't fix it, because parity is in sync with the disk that is corrupt.

 

Disable Docker and VM Manager in Settings until things are fixed, and try not to write anything to your server.

 

Check filesystem on disk 6 and 7

 

Also, lots of syslog entries about Unassigned Device sdh, looks like it might be a drive problem. Disable spindown on that disk and run an extended self-test

Link to comment

Running xfs_repair from the UI with the `-n` flag and it has been hanging for about 20 minutes.

 

Here is the output so far:

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200
agf has bad CRC for ag 4
block (2,184178199-184178280) multiply claimed by bno space tree, state - 1
block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2
block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2
block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2
block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2
block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2
block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2
block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2
block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2
block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2
block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2
block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2
block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2
block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2
block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2
block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2
block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2
block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2
block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2
block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2
block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2
block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2
block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2
block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2
block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2
block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2
block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2
block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2
block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2
block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2
block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2
block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2
block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2
block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2
block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2
agf_freeblks 2038004, counted 2048802 in ag 2
agf_freeblks 2116416, counted 2106811 in ag 4
agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168)
sb_fdblocks 793886, counted 11056513
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
Link to comment

xfs_repair for disk 6 finished:

 

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200
agf has bad CRC for ag 4
block (2,184178199-184178280) multiply claimed by bno space tree, state - 1
block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2
block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2
block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2
block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2
block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2
block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2
block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2
block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2
block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2
block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2
block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2
block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2
block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2
block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2
block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2
block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2
block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2
block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2
block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2
block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2
block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2
block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2
block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2
block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2
block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2
block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2
block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2
block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2
block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2
block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2
block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2
block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2
block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2
block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2
agf_freeblks 2038004, counted 2048802 in ag 2
agf_freeblks 2116416, counted 2106811 in ag 4
agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168)
sb_fdblocks 793886, counted 11056513
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
data fork in ino 1841992 claims free block 722810940
data fork in ino 1841992 claims free block 722818433
data fork in ino 1841992 claims free block 722766205
data fork in ino 1841992 claims free block 722745607
        - agno = 1
data fork in ino 2358538759 claims free block 721347041
data fork in ino 2358538759 claims free block 721149027
        - agno = 2
data fork in ino 4296520813 claims free block 556498438
data fork in ino 4296520813 claims free block 578099252
data fork in ino 4296520813 claims free block 773182227
data fork in ino 4296520813 claims free block 773182467
data fork in ino 4296520817 claims free block 573284307
data fork in ino 4296520817 claims free block 639394219
data fork in ino 4296520817 claims free block 639395162
data fork in ino 4296733237 claims free block 616345471
data fork in ino 4296733237 claims free block 639394507
data fork in ino 4296733237 claims free block 639395362
data fork in ino 4296739755 claims free block 641669917
data fork in ino 4296739755 claims free block 699854846
data fork in ino 4296739755 claims free block 699839130
data fork in ino 4296739755 claims free block 632622970
data fork in ino 4296739755 claims free block 632630486
data fork in ino 4296739755 claims free block 643598034
data fork in ino 4296739755 claims free block 632630142
data fork in ino 4296739755 claims free block 585572404
data fork in ino 4296739755 claims free block 585563566
data fork in ino 4296739755 claims free block 585563626
data fork in ino 4296744535 claims free block 711244085
data fork in ino 4296744537 claims free block 578099040
data fork in ino 4296744541 claims free block 596876767
data fork in ino 4296744541 claims free block 763252191
data fork in ino 4296744541 claims free block 763255439
data fork in ino 4296744541 claims free block 763255426
data fork in ino 4296744541 claims free block 763255442
data fork in ino 4296744541 claims free block 641708205
data fork in ino 4296744541 claims free block 584836169
data fork in ino 4296744541 claims free block 643965222
data fork in ino 4296744541 claims free block 643962849
data fork in ino 4296744541 claims free block 643962780
data fork in ino 4296744541 claims free block 643958036
data fork in ino 4296744541 claims free block 643962552
data fork in ino 4296744541 claims free block 639248129
data fork in ino 4296744541 claims free block 715506878
data fork in ino 4296744541 claims free block 660075640
data fork in ino 4296744541 claims free block 660051040
data fork in ino 4296744541 claims free block 584913266
data fork in ino 4296744541 claims free block 722709888
data fork in ino 4296744541 claims free block 722710684
data fork in ino 4296744541 claims free block 584912986
data fork in ino 4296744544 claims free block 546384767
data fork in ino 4296744546 claims free block 546619004
data fork in ino 4296744546 claims free block 546619106
data fork in ino 4296744546 claims free block 546619200
data fork in ino 4296744547 claims free block 577095121
data fork in ino 4296744547 claims free block 577095239
data fork in ino 4296744547 claims free block 577095345
data fork in ino 4296744547 claims free block 577096441
data fork in ino 4296744547 claims free block 577096535
data fork in ino 4296744547 claims free block 577108880
data fork in ino 4296744547 claims free block 585468837
data fork in ino 4296744553 claims free block 546619278
data fork in ino 4296744553 claims free block 598007436
        - agno = 3
data fork in ino 6446982893 claims free block 721391881
        - agno = 4
data fork in ino 8590740692 claims free block 664165116
        - agno = 5
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
free space (2,36216381-36216733) only seen by one free space btree
free space (2,40529923-40530269) only seen by one free space btree
free space (2,47525503-47525835) only seen by one free space btree
free space (2,48366886-48367196) only seen by one free space btree
free space (2,82534521-82534831) only seen by one free space btree
free space (2,100630810-100631194) only seen by one free space btree
free space (2,106722377-106722817) only seen by one free space btree
free space (2,119468999-119469400) only seen by one free space btree
free space (2,119830899-119831347) only seen by one free space btree
free space (2,187768370-187768613) only seen by one free space btree
free space (2,187768626-187768701) only seen by one free space btree
free space (2,187785086-187785185) only seen by one free space btree
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 1
        - agno = 3
        - agno = 4
        - agno = 5
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 2492166168, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 2492166168 nlinks from 0 to 1
No modify flag set, skipping filesystem flush and exiting.

 

Link to comment
13 minutes ago, thebedivere said:

pre-clear on it last night which failed, so it's not looking good

SMART attributes for it look OK, but it has never had an extended self-test so you might try that. You should remove it if for no other reason than it is cluttering your syslog and we will wonder about it next time.

22 minutes ago, thebedivere said:

xfs_repair for disk 6 finished

23 minutes ago, thebedivere said:
No modify flag set, skipping filesystem flush and exiting

 

Go ahead and do check filesystem on emulated disk6 without -n (no modify) so it will do the repair. Since it is emulated, it won't actually repair the physical disk, but the result is what would be rebuilt onto disk6. If the repair of emulated disk6 looks good then you can rebuild it. If not, then we can take a look at the physical disk as Unassigned Device and see if it is any better.

 

Disk7 later

Link to comment

Ok, xfs_repair on disk 6 finished:

 

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200
agf has bad CRC for ag 4
block (2,184178199-184178280) multiply claimed by bno space tree, state - 1
block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2
block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2
block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2
block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2
block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2
block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2
block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2
block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2
block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2
block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2
block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2
block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2
block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2
block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2
block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2
block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2
block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2
block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2
block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2
block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2
block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2
block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2
block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2
block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2
block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2
block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2
block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2
block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2
block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2
block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2
block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2
block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2
block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2
block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2
agf_freeblks 2038004, counted 2048802 in ag 2
agf_freeblks 2116416, counted 2106811 in ag 4
agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168)
sb_fdblocks 793886, counted 11056513
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
data fork in ino 1841992 claims free block 722810940
data fork in ino 1841992 claims free block 722818433
data fork in ino 1841992 claims free block 722766205
data fork in ino 1841992 claims free block 722745607
        - agno = 1
data fork in ino 2358538759 claims free block 721347041
data fork in ino 2358538759 claims free block 721149027
        - agno = 2
data fork in ino 4296520813 claims free block 556498438
data fork in ino 4296520813 claims free block 578099252
data fork in ino 4296520813 claims free block 773182227
data fork in ino 4296520813 claims free block 773182467
data fork in ino 4296520817 claims free block 573284307
data fork in ino 4296520817 claims free block 639394219
data fork in ino 4296520817 claims free block 639395162
data fork in ino 4296733237 claims free block 616345471
data fork in ino 4296733237 claims free block 639394507
data fork in ino 4296733237 claims free block 639395362
data fork in ino 4296739755 claims free block 641669917
data fork in ino 4296739755 claims free block 699854846
data fork in ino 4296739755 claims free block 699839130
data fork in ino 4296739755 claims free block 632622970
data fork in ino 4296739755 claims free block 632630486
data fork in ino 4296739755 claims free block 643598034
data fork in ino 4296739755 claims free block 632630142
data fork in ino 4296739755 claims free block 585572404
data fork in ino 4296739755 claims free block 585563566
data fork in ino 4296739755 claims free block 585563626
data fork in ino 4296744535 claims free block 711244085
data fork in ino 4296744537 claims free block 578099040
data fork in ino 4296744541 claims free block 596876767
data fork in ino 4296744541 claims free block 763252191
data fork in ino 4296744541 claims free block 763255439
data fork in ino 4296744541 claims free block 763255426
data fork in ino 4296744541 claims free block 763255442
data fork in ino 4296744541 claims free block 641708205
data fork in ino 4296744541 claims free block 584836169
data fork in ino 4296744541 claims free block 643965222
data fork in ino 4296744541 claims free block 643962849
data fork in ino 4296744541 claims free block 643962780
data fork in ino 4296744541 claims free block 643958036
data fork in ino 4296744541 claims free block 643962552
data fork in ino 4296744541 claims free block 639248129
data fork in ino 4296744541 claims free block 715506878
data fork in ino 4296744541 claims free block 660075640
data fork in ino 4296744541 claims free block 660051040
data fork in ino 4296744541 claims free block 584913266
data fork in ino 4296744541 claims free block 722709888
data fork in ino 4296744541 claims free block 722710684
data fork in ino 4296744541 claims free block 584912986
data fork in ino 4296744544 claims free block 546384767
data fork in ino 4296744546 claims free block 546619004
data fork in ino 4296744546 claims free block 546619106
data fork in ino 4296744546 claims free block 546619200
data fork in ino 4296744547 claims free block 577095121
data fork in ino 4296744547 claims free block 577095239
data fork in ino 4296744547 claims free block 577095345
data fork in ino 4296744547 claims free block 577096441
data fork in ino 4296744547 claims free block 577096535
data fork in ino 4296744547 claims free block 577108880
data fork in ino 4296744547 claims free block 585468837
data fork in ino 4296744553 claims free block 546619278
data fork in ino 4296744553 claims free block 598007436
        - agno = 3
data fork in ino 6446982893 claims free block 721391881
        - agno = 4
data fork in ino 8590740692 claims free block 664165116
        - agno = 5
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 2
        - agno = 0
        - agno = 3
        - agno = 1
        - agno = 5
        - agno = 4
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 2492166168, moving to lost+found
Phase 7 - verify and correct link counts...
done

 

Link to comment

lost+found folder is created on a disk when filesystem repair can't figure out what to do with some of the data.

 

Reviewing this thread, only (emulated) disk6 has been repaired in this thread.

 

But your screenshot is telling us that you have previously repaired all those other disks as well. And some of them have a lot of data in lost+found. Have you ever looked in that share?

Link to comment
3 minutes ago, thebedivere said:

Looking in the share I just see a bunch of folder and files with alpha-numeric names that don't have any meaning to me.

That is typical for lost+found, filenames couldn't be associated with the data, and neither could folders. So you just get a lot of stuff that you can either forget about, or try to figure it out one at a time. Often not worth the trouble. Disks 1, 2, 3 have a large amount of data in lost+found.

 

Linux 'file' command can sometimes tell you what kind of data is in a file, which might give you some idea of how to open them.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.