Jump to content

  • [6.6.6] Unmountable filsystem resulted in data loss

    Mikki

    By Mikki

      • Closed Urgent

    Hi

     

    I have an external sata controller that do not like ssd's in unraid, it will just start kicking out disk's running on that controller, im not sure it is unarid to blame or not, but it is not the problem the problem is what follow from that.

     

    I have 2 Parity disks and 6 data disk so 8 disk i total, i also have a cache (SSD) disk in the array, all running BTRFS.

     

    I have had some problems with unraid do a kernel panic like 4-6 times in the last 2-3 months, installed Fix Common Problems, and fixed all the problems, but have not been able to se if that worked, as i by mistake added a SSD more to the system, and added it in the external controller.

    So after a day or two it started to kick out 2 disk from the array, (Parity 2 and disk 5), and the new SSD (set as an Unassigned Device) after some high load on the SSD. 

    Okay fine, i will just stop the array and fix the problem as done before np. Well stoped the array and tryed to powered down the system to be able to moved the SSD to the motherboard, but the system froze like the kernal panic's i had before. Killed the system to turn it on again, when booted problems started, everything looked god i started the array, but then all the disk dipasered from the web gui, all dynamic elements was gone. (i have tryed to make a bug report on that from the gui but im not sure it got sendet as the gui was buged as f. rebooted a few times, and i was able to get the disk to show sometimes and was able to remap the Parity 2 to rebuild that disk as disk 5 and the SSD cache was "Unmountable filsystem" in status, so just before the rebuild i removed disk 5 from the array so i just was emulated during the rebuild. 

    When the Parity 2 was back to normal i rebooted the system. Got the GUI back to normal... humm.. okay...

    So disk 5 and the ssd cache was still "Unmountable filsystem", so stated maintenance mode to do a check on the SSD cache using the "Check Filesystem Status" with readonly to start with. okay it had some errors, so started the repair paramter and had i running for 2+ days did not finish a single bit on the filesystem at all.. okay i have a week old copy for some odd reason on a external disk, so stoped the repair, to work on disk 5, and here the real problem startes, the data loss. 

     

    I got unraid to get it to know that is was a new disk, and started the array as normal, just like a normal rebuild when you loss a disk, it still was "Unmountable filsystem" in status, but i did not think much of it as it stated the rebuild as normal. When the rebuild was done it still had the status "Unmountable filsystem" and the array was still 8TB short of total size.

    Unraid still asked me to do a reformat of the disk 5 and cache, so as stupid i was i pressed format, and bam, disk 5 was now working but empty, dam it.. 

     

    Why did unraid not fix the "Unmountable filsystem" on the rebuild, and why did it start the rebuild if it was not mountable, or suitable for data yet?

     

    But okay, i should have a backup of must of it i think time will show, but how to firure out what is missing is gonna be "fun"....

     

    So okay i wanted to get to work restoring the data, just to get stoped by that the unraid config was partiel reset, the shares was gone, and the disk settings was set to xfs not btrfs - encrypted.

     

    Did i do someting totaly wrong that i should have known not to do during the rebuild of disk 5 that would have prevented my data loss on disk 5, i am back to why did unraid not fix the "Unmountable filsystem" on the rebuild?

     

    My data is lost and i will try to recover it, but i hope we can help each other so no one else lose data to and Unmountable filsystem i have added the diagnostics and and old photo of the output of kernel panic as i do not have anything else of it sry.

     

    (btw sdm1 is not to blame in the logs, it is a faulty external disk that have the cache backup please look "round" it sry.)

     

    UPDATE: the cfg's for there shares is still there how do it get then to show in the gui agning to work?

    UPDATE: Reboot with at mountable cache fixed the shares not showing

     

    nas-diagnostics-20190119-1513.zip

    2018-11-11 18.34.17.png

    Go to reports Stable Releases

    User Feedback

    Recommended Comments

    JorgeB

    I didn't follow everything you did, and your log is very difficult to analyze since it's spammed with various unrelated errors, I did see cache filesystem was corrupt, and I also see you're using a Marvell 9230 controller, those controllers are a known problem with Linux, they tend to drop disks without a reason, btrfs is particularly sensible to dropped disks, and corruption beyond repair is possible, even likely if it keeps happening, I suggest you replace that controller with for example an LSI HBA.

     

    Also, btrfs --check repair should only be used if told so by a btrfs maintainer or as a last resort, since many times it can do more harm than good, more info about btrfs recovery here.

    Link to comment
    itimpi

    It is worth pointing out that a rebuild never fixes an ‘unmountable disk’ status.   A rebuild is intended to rebuild a failed drive onto a new drive in the same state as it was at the point of failure.   If a disk is flagged as unmountable before starting the rebuild it will always be unmountable afterwards.   Since rebuild works at the physical sector level it has no idea if the file system is good or even if the disk has even been formatted.

     

    The recommended procedure is to try and fix the file system on the emulated drive before doing the rebuild as if it cannot be fixed at that point you will have the same corruption present after the rebuild.

    Link to comment
    trurl

    In future I hope you will consider asking for advice when you are uncertain about how to proceed.

     

    Your post doesn't really follow the guidelines for a report

    so it should probably be moved to the General Support section of the forum. But I don't see any way for me to move it.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    ×

  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.

    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.

×
×
  • Create New...