July 18, 20223 yr Author 1 hour ago, trurl said: Might as well let rebuild complete first. Gotcha, I'll report back in once the rebuild is done.
August 8, 20223 yr Author Here's my update. I let the rebuild and parity checks finish, and I still had a dead drive. I decided to just remove the 6 TB drive and replace it with a larger one. I got a good deal on a 16 TB seagate, so I swapped out the 14 TB drive in parity for the larger 16 TB drive after a preclear. Both parity drives are now 16tb. I let parity rebuild onto the new drive, and swapped out the dead 6 TB for the 14 TB drive (that has always worked fine). The data rebuilt and I had all drives green! But, the last 2 drives still show up as unmountable. I did some digging into the Marvell raid controller. I initially thought it had to be the motherboard since it says it has 4 ports, but nope, it's the expansion card which has 6 ports on it. I decided to stop the array and see if I could troubleshoot, and this is what I saw next. Only 2 drives showed up as attached and the rest were missing. I downloaded the diagnostics and rebooted, now the drives are all back. Disks 6 and 7 still show up as unmountable. My thoughts are that I should replace the sata controller and see if that helps. My question at this point is do I need to do something with the unmountable drives? I know disk 6 is a good disk since it's been in use as a parity drive and I did not unplug or move the port it is connected to, just assigned it in the UI. I also attached the diagnostics for after I rebooted if that helps. It's the one that ends in "-2". Thanks! orthanc-diagnostics-20220807-2206.zip orthanc-diagnostics-20220807-2219-2.zip
August 8, 20223 yr Community Expert Well, I just reviewed the thread after all these weeks, and though a lot was said about unmountable disks and filesystem corruption, I don't think anything was said about what to do about it. Fixing hardware won't fix filesystem corruption, since the corruption already happened. And rebuild won't fix it, because parity is in sync with the disk that is corrupt. Disable Docker and VM Manager in Settings until things are fixed, and try not to write anything to your server. Check filesystem on disk 6 and 7 Also, lots of syslog entries about Unassigned Device sdh, looks like it might be a drive problem. Disable spindown on that disk and run an extended self-test
August 8, 20223 yr Community Expert 1 minute ago, trurl said: Check filesystem on disk 6 and 7 Be sure to capture the output so you can post it
August 8, 20223 yr Author Running xfs_repair from the UI with the `-n` flag and it has been hanging for about 20 minutes. Here is the output so far: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0
August 8, 20223 yr Author 35 minutes ago, trurl said: Also, lots of syslog entries about Unassigned Device sdh, looks like it might be a drive problem. Disable spindown on that disk and run an extended self-test That drive isn't in use, and I am fairly certain that the drive is bad.
August 8, 20223 yr Community Expert 39 minutes ago, thebedivere said: That drive isn't in use, and I am fairly certain that the drive is bad. Why is it in the server then?
August 8, 20223 yr Author 8 hours ago, trurl said: Why is it in the server then? It's the one that I just replaced with the 14 TB drive. I was hoping to verify that the drive was dead before I removed it from the server. I ran a pre-clear on it last night which failed, so it's not looking good.
August 8, 20223 yr Author xfs_repair for disk 6 finished: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1841992 claims free block 722810940 data fork in ino 1841992 claims free block 722818433 data fork in ino 1841992 claims free block 722766205 data fork in ino 1841992 claims free block 722745607 - agno = 1 data fork in ino 2358538759 claims free block 721347041 data fork in ino 2358538759 claims free block 721149027 - agno = 2 data fork in ino 4296520813 claims free block 556498438 data fork in ino 4296520813 claims free block 578099252 data fork in ino 4296520813 claims free block 773182227 data fork in ino 4296520813 claims free block 773182467 data fork in ino 4296520817 claims free block 573284307 data fork in ino 4296520817 claims free block 639394219 data fork in ino 4296520817 claims free block 639395162 data fork in ino 4296733237 claims free block 616345471 data fork in ino 4296733237 claims free block 639394507 data fork in ino 4296733237 claims free block 639395362 data fork in ino 4296739755 claims free block 641669917 data fork in ino 4296739755 claims free block 699854846 data fork in ino 4296739755 claims free block 699839130 data fork in ino 4296739755 claims free block 632622970 data fork in ino 4296739755 claims free block 632630486 data fork in ino 4296739755 claims free block 643598034 data fork in ino 4296739755 claims free block 632630142 data fork in ino 4296739755 claims free block 585572404 data fork in ino 4296739755 claims free block 585563566 data fork in ino 4296739755 claims free block 585563626 data fork in ino 4296744535 claims free block 711244085 data fork in ino 4296744537 claims free block 578099040 data fork in ino 4296744541 claims free block 596876767 data fork in ino 4296744541 claims free block 763252191 data fork in ino 4296744541 claims free block 763255439 data fork in ino 4296744541 claims free block 763255426 data fork in ino 4296744541 claims free block 763255442 data fork in ino 4296744541 claims free block 641708205 data fork in ino 4296744541 claims free block 584836169 data fork in ino 4296744541 claims free block 643965222 data fork in ino 4296744541 claims free block 643962849 data fork in ino 4296744541 claims free block 643962780 data fork in ino 4296744541 claims free block 643958036 data fork in ino 4296744541 claims free block 643962552 data fork in ino 4296744541 claims free block 639248129 data fork in ino 4296744541 claims free block 715506878 data fork in ino 4296744541 claims free block 660075640 data fork in ino 4296744541 claims free block 660051040 data fork in ino 4296744541 claims free block 584913266 data fork in ino 4296744541 claims free block 722709888 data fork in ino 4296744541 claims free block 722710684 data fork in ino 4296744541 claims free block 584912986 data fork in ino 4296744544 claims free block 546384767 data fork in ino 4296744546 claims free block 546619004 data fork in ino 4296744546 claims free block 546619106 data fork in ino 4296744546 claims free block 546619200 data fork in ino 4296744547 claims free block 577095121 data fork in ino 4296744547 claims free block 577095239 data fork in ino 4296744547 claims free block 577095345 data fork in ino 4296744547 claims free block 577096441 data fork in ino 4296744547 claims free block 577096535 data fork in ino 4296744547 claims free block 577108880 data fork in ino 4296744547 claims free block 585468837 data fork in ino 4296744553 claims free block 546619278 data fork in ino 4296744553 claims free block 598007436 - agno = 3 data fork in ino 6446982893 claims free block 721391881 - agno = 4 data fork in ino 8590740692 claims free block 664165116 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... free space (2,36216381-36216733) only seen by one free space btree free space (2,40529923-40530269) only seen by one free space btree free space (2,47525503-47525835) only seen by one free space btree free space (2,48366886-48367196) only seen by one free space btree free space (2,82534521-82534831) only seen by one free space btree free space (2,100630810-100631194) only seen by one free space btree free space (2,106722377-106722817) only seen by one free space btree free space (2,119468999-119469400) only seen by one free space btree free space (2,119830899-119831347) only seen by one free space btree free space (2,187768370-187768613) only seen by one free space btree free space (2,187768626-187768701) only seen by one free space btree free space (2,187785086-187785185) only seen by one free space btree - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 - agno = 4 - agno = 5 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2492166168, would move to lost+found Phase 7 - verify link counts... would have reset inode 2492166168 nlinks from 0 to 1 No modify flag set, skipping filesystem flush and exiting.
August 8, 20223 yr Community Expert 13 minutes ago, thebedivere said: pre-clear on it last night which failed, so it's not looking good SMART attributes for it look OK, but it has never had an extended self-test so you might try that. You should remove it if for no other reason than it is cluttering your syslog and we will wonder about it next time. 22 minutes ago, thebedivere said: xfs_repair for disk 6 finished 23 minutes ago, thebedivere said: No modify flag set, skipping filesystem flush and exiting Go ahead and do check filesystem on emulated disk6 without -n (no modify) so it will do the repair. Since it is emulated, it won't actually repair the physical disk, but the result is what would be rebuilt onto disk6. If the repair of emulated disk6 looks good then you can rebuild it. If not, then we can take a look at the physical disk as Unassigned Device and see if it is any better. Disk7 later
August 8, 20223 yr Author Ok, xfs_repair on disk 6 finished: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1841992 claims free block 722810940 data fork in ino 1841992 claims free block 722818433 data fork in ino 1841992 claims free block 722766205 data fork in ino 1841992 claims free block 722745607 - agno = 1 data fork in ino 2358538759 claims free block 721347041 data fork in ino 2358538759 claims free block 721149027 - agno = 2 data fork in ino 4296520813 claims free block 556498438 data fork in ino 4296520813 claims free block 578099252 data fork in ino 4296520813 claims free block 773182227 data fork in ino 4296520813 claims free block 773182467 data fork in ino 4296520817 claims free block 573284307 data fork in ino 4296520817 claims free block 639394219 data fork in ino 4296520817 claims free block 639395162 data fork in ino 4296733237 claims free block 616345471 data fork in ino 4296733237 claims free block 639394507 data fork in ino 4296733237 claims free block 639395362 data fork in ino 4296739755 claims free block 641669917 data fork in ino 4296739755 claims free block 699854846 data fork in ino 4296739755 claims free block 699839130 data fork in ino 4296739755 claims free block 632622970 data fork in ino 4296739755 claims free block 632630486 data fork in ino 4296739755 claims free block 643598034 data fork in ino 4296739755 claims free block 632630142 data fork in ino 4296739755 claims free block 585572404 data fork in ino 4296739755 claims free block 585563566 data fork in ino 4296739755 claims free block 585563626 data fork in ino 4296744535 claims free block 711244085 data fork in ino 4296744537 claims free block 578099040 data fork in ino 4296744541 claims free block 596876767 data fork in ino 4296744541 claims free block 763252191 data fork in ino 4296744541 claims free block 763255439 data fork in ino 4296744541 claims free block 763255426 data fork in ino 4296744541 claims free block 763255442 data fork in ino 4296744541 claims free block 641708205 data fork in ino 4296744541 claims free block 584836169 data fork in ino 4296744541 claims free block 643965222 data fork in ino 4296744541 claims free block 643962849 data fork in ino 4296744541 claims free block 643962780 data fork in ino 4296744541 claims free block 643958036 data fork in ino 4296744541 claims free block 643962552 data fork in ino 4296744541 claims free block 639248129 data fork in ino 4296744541 claims free block 715506878 data fork in ino 4296744541 claims free block 660075640 data fork in ino 4296744541 claims free block 660051040 data fork in ino 4296744541 claims free block 584913266 data fork in ino 4296744541 claims free block 722709888 data fork in ino 4296744541 claims free block 722710684 data fork in ino 4296744541 claims free block 584912986 data fork in ino 4296744544 claims free block 546384767 data fork in ino 4296744546 claims free block 546619004 data fork in ino 4296744546 claims free block 546619106 data fork in ino 4296744546 claims free block 546619200 data fork in ino 4296744547 claims free block 577095121 data fork in ino 4296744547 claims free block 577095239 data fork in ino 4296744547 claims free block 577095345 data fork in ino 4296744547 claims free block 577096441 data fork in ino 4296744547 claims free block 577096535 data fork in ino 4296744547 claims free block 577108880 data fork in ino 4296744547 claims free block 585468837 data fork in ino 4296744553 claims free block 546619278 data fork in ino 4296744553 claims free block 598007436 - agno = 3 data fork in ino 6446982893 claims free block 721391881 - agno = 4 data fork in ino 8590740692 claims free block 664165116 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 2 - agno = 0 - agno = 3 - agno = 1 - agno = 5 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2492166168, moving to lost+found Phase 7 - verify and correct link counts... done
August 8, 20223 yr Community Expert So emulated disk6 mounts and has plenty of data. Go to User Shares, click Compute... for lost+found share, post a screenshot
August 8, 20223 yr Community Expert lost+found folder is created on a disk when filesystem repair can't figure out what to do with some of the data. Reviewing this thread, only (emulated) disk6 has been repaired in this thread. But your screenshot is telling us that you have previously repaired all those other disks as well. And some of them have a lot of data in lost+found. Have you ever looked in that share?
August 8, 20223 yr Author I have not looked in that share. I have probably repair the other disks at some point in time, yes. What should I do to restore that data? Looking in the share I just see a bunch of folder and files with alpha-numeric names that don't have any meaning to me.
August 8, 20223 yr Author Scratch that, after clicking into more folders I found some audiobook files.
August 8, 20223 yr Author The audio book files I found are also on the correct share for Media/audiobook files.
August 8, 20223 yr Community Expert 3 minutes ago, thebedivere said: Looking in the share I just see a bunch of folder and files with alpha-numeric names that don't have any meaning to me. That is typical for lost+found, filenames couldn't be associated with the data, and neither could folders. So you just get a lot of stuff that you can either forget about, or try to figure it out one at a time. Often not worth the trouble. Disks 1, 2, 3 have a large amount of data in lost+found. Linux 'file' command can sometimes tell you what kind of data is in a file, which might give you some idea of how to open them.
August 8, 20223 yr Author Anything important is backed up to another machine and to backblaze, so I'm not too worried about missing files at this point. Is it safe to ignore for now, or do I need to address it before I move on? Edited August 8, 20223 yr by thebedivere
August 8, 20223 yr Community Expert If you don't want it you can delete it and recover the space. You can decide later. Check filesystem on disk 7
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.