January 9, 20251 yr Hello, I recently acted without caution and believe I have destroyed parity for my array. I was preparing to start the process of manually rebuilding the lost data (which would be a long and tedious process), but after discovering one of the drives still has the original data when mounting as an unassigned disk device I thought I should reach out to the experts here to see if there is any hope of recovering the data without that amount of work. I recently looked to expand the number of drives in my array which included replacing both parity drives with larger capacity drives. Here is what happened: I powered down my server and added the new larger capacity drives, then started the server back up. At this time all drives (new & current array) showed in the UI without issues I stopped the array and reassigned Parity 1 to one of the new larger drives. Then initiated a parity rebuild At some point During the parity rebuild of Parity 1, Parity 2 showed the Red X next to it. I allowed rebuild to finish since I was replacing Parity 2 next anyway After Parity rebuild completed for Parity 1, I began the same process for Parity 2 by reassigning another one of the new larger drives. At some point during this parity rebuild, one of the data drives developed a Red X also (grrr). I continued to monitor the rebuild every few hours and everything seemed fine. When I returned to check on the server after it should have completed I found it had completed the parity build but now a second data drive was showing the Red X. This is when I probably should have come here, but I did not. I did the following. I then reassigned one of the original (now replaced) parity drives as well as one of the two data drives that had the Red X to the two data drive slots in the array with Red Xs and initiated a data-rebuild for them. They rebuild did finally complete and now they are listed as "Unmountable: Unsupported or no file system". I'll admit that I'm confused by that since they were just part of the array prior and are listed as xfs like all the other drives in the array. Maybe that's to be expected? So then I mounted the data disks that developed a Red X during the Parity 2 rebuild and can see that it has the data I would expect to see on it. If I had taken my time and thought to do this for the other Red X data drive instead of using it for a data-rebuild in the array then I might of found the same, but now I'll never know it seems. Is there any chance of saving my data or did I damage it beyond repair already? Thank you.
January 10, 20251 yr Author Hello, and thank you for the quick response. This is my first time posting, so sorry if that was something I should have done by default. I have also turned on notifications so I should see these responses right away. Diagnostics are now attached, but I have to let you know that I have done a reboot since completing the rebuild. calculon-diagnostics-20250110-1004.zip
January 10, 20251 yr The best option is probably to do a New Config with the old drives and resync parity, you can try to sync with the new parity drives, but let us know which disks were the old disks just to make sure SMART looks OK.
January 10, 20251 yr Author Can I assume by "old drives" you mean the ones specifically that were moved around? In this case it is the following: current Disk 5 - formerly Parity 1 (underwent data rebuild in current configuration) current Disk 6 - formerly Disk 5 (underwent data rebuild in current configuration) former Disk 6 - no in unassigned disk pool Currently both parity drives are from the new batch. All other drives unchanged with no known issues
January 10, 20251 yr 9 minutes ago, beatmurph said: Can I assume by "old drives" you mean the ones specifically that were moved around? The original disks 5 and 6, are they still available and untouched?
January 10, 20251 yr Author No, Disc 5 was moved to Disk 6 in the array and after that I initiated the data-rebuild.
January 10, 20251 yr So former disk6 is available, but not disk5 correct? And when you did this: 22 hours ago, beatmurph said: So then I mounted the data disks that developed a Red X during the Parity 2 rebuild and can see that it has the data I would expect to see on it. Which disks do you mean?
January 10, 20251 yr Author Correct, Disk 6 was only moved from the array to unassigned disk devices. I can mount it and see all its original contents and have not done anything with it but a SMART check. 16 minutes ago, JorgeB said: Which disks do you mean? This was Disk 6
January 10, 20251 yr So for disk6 a new config using the old disk6 should be the best option, as for disk5, does the current disk5 show the correct data?
January 10, 20251 yr Author The Disks currently in the array: current Disk 5 - formerly Parity 1 (underwent data rebuild in current configuration) current Disk 6 - formerly Disk 5 (underwent data rebuild in current configuration) These are for some reason showing as "Unmountable: Unsupported or no file system". This is the part that confused me the most. They are both disks that were in the array before I started the re-build and are still listed as xfs which is the same as the rest of the array. Because of that I don't know of a way to see what is on them. Edited January 10, 20251 yr by beatmurph
January 10, 20251 yr Author Actually, I might be in worse shape. After the data-rebuild of Disks 5 & 6 there was a warning presented which stated something to the affect. "Warning, when the array is started all data on these disks will be deleted" because of that I had been avoiding starting the array. I'm not sure if in some brain dead state I eventually started the array for some reason, or if maybe I did a system restart and that automatically started the array. Either way, I'm just now looking and registering that the array has been started. I just stopped it and that warning no longer exists for disks 5 & 6. Edited January 10, 20251 yr by beatmurph
January 11, 20251 yr Check filesystem on current disks 5 and 6, run it without -n, but if the rebuilds were incomplete, possibly they will have some missing data, but from the description, don't see any other options for disk5, disk6 you can then recover from the old disk, if the current one is worse.
January 11, 20251 yr 17 hours ago, beatmurph said: "Warning, when the array is started all data on these disks will be deleted" .... the array has been started. I just stopped it and that warning no longer exists for disks 5 & 6. Did you tell it to format the disks? Or are the still unmountable? Attach new Diagnostics to your NEXT post in this thread.
January 12, 20251 yr Author I completed the filesystem check for Disk 5. I executed with the -n as I was a little nervous jumping straight into repair, though I was unaware it would take a full day. I was going to proceed with your original recommendation to do the check and repair without -n, but given the time it takes I figured I better pause any action for now and await your recommendation. On 1/11/2025 at 8:16 AM, trurl said: Did you tell it to format the disks? Or are the still unmountable? I have not formatted the disks. They still show as unmountable in the array. I have attached the latest diagnostics after performing a Check Filesystem on Disk 5. calculon-diagnostics-20250112-1022.zip
January 12, 20251 yr 5 hours ago, beatmurph said: I completed the filesystem check for Disk 5. I executed with the -n Can you post the output for that?
January 12, 20251 yr Author I couldn't find if there was a way to export this so here is a copy/paste: "Phase 1 - find and verify superblock... couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!! attempting to find secondary superblock... .found candidate secondary superblock... unable to verify superblock, continuing... ........found candidate secondary superblock... unable to verify superblock, continuing... unable to verify superblock, continuing... . . ..Sorry, could not find valid secondary superblock Exiting now." There might be something missing where the space is, but the amount of dots was crashing everything I tried to copy it over to. I get the sense that doesn't matter.
January 12, 20251 yr Doesn't sound like a repair will be successful, but don't know what else to try.
January 13, 20251 yr If the emulated disk5 can't be fixed, don't see any other options for that one, you can recover disk6 from the old one.
January 13, 20251 yr Author So I have still not tried the file system check for the current Disk 6 (which is the former Disk 5). Would it make sense to try that first. Maybe without -n as originally suggested?
January 13, 20251 yr Author OK, so just to clarify, I'm not positive if parity was finish rebuilding for Parity 2 before both disks 5 & 6 developed the red X, so its possible the red Xs developed while Parity 2 was rebuilding. You still think a resync is the is the best path forward in this case given the original disk 6 seems to be mountable with all the original data? I have tried looking for a guide on parity resync. Is there one I should follow?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.