• Upgrade from 6.9.2 to 6.10 - lost 1 disk in array, cache drive went offline


    Monteroman
    • Closed Minor

    Hey there.

    While this may be a huge coincidence, but my server was running fine on 6.9.2 with all drives running normally including the cache drive. When I upgraded and rebooted, the system came up, but the cache drive was in read only mode and 1 of my 4 drives was unreadable (btrfs on all of my drives). I also noticed that the /var/log folder filled up (syslog filled the volume). I deleted the file, then rebooted the system again.  When it came back up, Drive 2 was still offline and then the cache drive was dead.  It ended up having to get reformatted and restored as well as the disk that wouldn't mount. I tried running a btrfs check on it, but it wasn't working. There was something about the log on the disk being corrupted so I formatted both the cache and that drive.  

     

    I thought the array would rebuild the drive, but it isn't rebuilding it. Before I formatted, I tried to remove the drive from the array, but couldn't so the only option left to me was to reformat it.  Is there a command to force an array rebuild or at this point, did I lose all of my data on that drive?

     

    The diagnostics attached are before I formatted the drive.

     

    Any help would be appreciated.

     

    Thanks!

    tower-diagnostics-20220519-0703.zip




    User Feedback

    Recommended Comments

    Format is NEVER part of rebuild. Format is a write operation. When you format a disk in the parity array, that write operation is treated just like any other, parity is updated. So after formatting a disk in the parity array, the only thing that can result from rebuilding that disk is a formatted disk.

     

    Your diagnostics indicated filesystem corruption on disk2, cache, and the unassigned disk. None of these disks were missing or offline in your diagnostics, just corrupt.

     

    Seems extremely unlikely upgrade related.

     

    You should have asked for help in General Support before doing anything. Rebuilding was entirely the wrong approach to fixing your issues, and formatting any drive with data you want to keep is always a mistake.

     

    Link to comment

    There's constant crashing on the log, including references to the Unraid driver, there's another user that recently had similar issues with the same HP Microserver Gen8, including some data loss due to fs corruption with both btrfs and xfs devices:

     

     

    For now I would recommend going back to v6.9.2, looks like these serves don't get along with the newer Kernel, unclear for now if all Microserver gen8 will have the same issues or just a specific CPU/BIOS combo.

    Link to comment
    45 minutes ago, trurl said:

    Format is NEVER part of rebuild. Format is a write operation. When you format a disk in the parity array, that write operation is treated just like any other, parity is updated. So after formatting a disk in the parity array, the only thing that can result from rebuilding that disk is a formatted disk.

     

    Your diagnostics indicated filesystem corruption on disk2, cache, and the unassigned disk. None of these disks were missing or offline in your diagnostics, just corrupt.

     

    Seems extremely unlikely upgrade related.

     

    You should have asked for help in General Support before doing anything. Rebuilding was entirely the wrong approach to fixing your issues, and formatting any drive with data you want to keep is always a mistake.

     

    So does this mean that I will indeed have to restore the array from a backup?

    Link to comment
    1 hour ago, Monteroman said:

    So does this mean that I will indeed have to restore the array from a backup?

    If you formatted disk2 and cache that is gone, but each of the other disks still has its data.

    Link to comment

    So I'll have to restore whatever was on disk2 then.  Cache, I knew I'd have to restore from backup. I've gone through that before. Well, it's going to be a long restore from crashplan then.. fun times ahead.

     

    I've reverted back to 6.9.2. It's been stable on the MicroServer Gen8.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.