Unraid flash drive corrupt - how to proceed


Recommended Posts

Woke up this morning to a message when clicking on the Apps tab in Unraid stating the Flash Drive may be corrupt or unreadable and to post on the forums (little did I know...).  I then noticed the Unraid webui was acting a bit weird, although the array was functional.  I decided to reboot since it had been a few weeks since a fresh reboot.  The reboot happened but the server didn't come back up.  I checked and the flash drive was unbootable.  Great.  

 

I shut down completely and booted back up, also unplugged and replugged the flash drive in for good measure.  No change.  Double-checked the boot sequence in BIOS, no issues.

 

Plugged the flash drive into my Win10 machine, the USB drive appears but is not readable and Windows requests to format it.  I didn't.  

 

About a month back I made a backup of the flash drive.  I believe I put it back on the Unraid array in my 'Backups' share.  Probably not the best idea to put it there since at this point I can't get the array back to access this backup.  I'm a little wary about re-creating the flash drive.  I know I need the /config folder for a complete restore, but I guess at this point I have no choice but to format the USB drive and start 'fresh'.  Then mount the array and restore the backup from there and reboot again to restore it completely.  Am I correct in this train of thought?

 

Thanks in advance for your input.

Edited by quadrant6
Link to comment

No, I don't have them off the top of my head.  

 

I did find a different backup (maybe slightly older) from another machine I did with the /config folder intact, so I may go this route.  Only problem now is I attempted to re-create the USB from the Unraid USB Creator and it's unable to create it, producing a cryptic error message:

 

euK6UGY.png

 

I'll have to create a new USB drive.

Link to comment
22 minutes ago, quadrant6 said:

did find a different backup (maybe slightly older) from another machine I did with the /config folder intact, so I may go this route. 

Let's discuss this before you actually do anything with this backup.    I am concerned about what you mean by the "another machine"  phase in the above quote.   The    /config   folder has information on the drive assignments and things can rapidly move from 'bad' to 'catastrophic' If these are not correct!!!!

Edited by Frank1940
Link to comment
3 minutes ago, Frank1940 said:

Let's discuss this before you actually do anything with this backup.    I am concerned about what you mean by the "another machine"  phase in the above quote.   The    /config   folder has information on the drive assignments and things can rapidly move from 'bad' to 'catastrophic' If these are not correct!!!!

Apologies for being unclear - "another machine" meaning the backup for my existing Unraid config was stored on "another machine", not meaning it was "another machine" as in a different Unraid config.

 

After reading a bit from https://wiki.unraid.net/UnRAID_6/Changing_The_Flash_Device, I created a new USB flash drive and will work on getting the license transferred when I'm back up and running.  

 

I was able to restore the /config folder from that backup on to the new flash drive and booted it up.  Now I'm getting somewhere, although the drive assignments are messy...  Some appear to be auto-assigned from before, but the rest I'm not entirely sure of.

 

 

20201123_083051.jpg

Link to comment
3 minutes ago, trurl said:

You absolutely MUST NOT assign a disk with data on it to any parity slot or it will be overwritten with parity!!!

 

If you don't know which disks are parity and which parity is which assign all disks to data slots. That will get you going enough to get the real flash backup.

Gotcha.  

 

I got the license key situation sorted and it's licensed on the replacement flash drive.  Now the first parity drive (it was initially set to dual parity) was auto-assigned as well as a few other drives (see screenshot).  I have at least 9 more drives that are unassigned in the array. 

 

I can either #1) start the array with the 1 parity drive that was auto-assigned, although I don't know if I want to. 

 

Or #2) if I change the 1 parity drive to 'no device', the option to start the array states 'Start will disable the missing disk and bring the array on-line.  Install a replacement disk as soon as possible'.   

 

If I assign any other of the 9 unassigned drives to any open slots, I'm unable to start the array with the message 'You may not add new disks and also remove existing disks'.  

 

I guess I'll choose option 2.  I believe the flash backup on the array is more current than the one on my 'other machine' which is from Sept 30 this year.

 

unnamed.jpg

Link to comment

The main danger of using an old backup is assuming it knows what disk is parity. More than one person has replaced parity with a new disk, reused old parity as a data disk, then later tried to use an old backup of flash that thought the old parity (now data) disk was parity, and it began overwriting the data on that disk.

Link to comment
21 minutes ago, quadrant6 said:

any other of the 9 unassigned drives to any open slots,

WOW!!  Nine MIA disks from the drive configuration that you loaded!   Give us a history (as complete as you can remember it) about this server.  How many drives were in the array?  Did you have any cache drives?  Did you have any disks that were being mounted using the Unassigned Devices plugin.   When was the last time you added, changed or moved any of the disks that are currently in the server?  

Link to comment
8 minutes ago, trurl said:

The main danger of using an old backup is assuming it knows what disk is parity. More than one person has replaced parity with a new disk, reused old parity as a data disk, then later tried to use an old backup of flash that thought the old parity (now data) disk was parity, and it began overwriting the data on that disk.

Understood.  I went ahead and unassigned the Parity 1 drive and started the array with just the 5 disks it had auto-assigned.  Since the cache array wasn't setup, it's now building the BTRFS cache and won't let me stop the array for now.  Will let that run for now.

 

I'm able to access my Backups share, but not finding the flash drive backup I created.  I don't know for sure if I saved it there, or if it was created somewhere else.  Is there a default location for the Flash backup when it's created from the Unraid webui?  

 

From what I understand, I can assign all disks to disk slots (non-parity) and then the two that don't mount properly should be my parity drives.  The remaining disks are all data and can be assigned freely and don't have to be assigned exactly as they were before, correct?

Link to comment
19 minutes ago, Frank1940 said:

WOW!!  Nine MIA disks from the drive configuration that you loaded!   Give us a history (as complete as you can remember it) about this server.  How many drives were in the array?  Did you have any cache drives?  Did you have any disks that were being mounted using the Unassigned Devices plugin.   When was the last time you added, changed or moved any of the disks that are currently in the server?  

At the time of the backup (Sept 30 of this year) I want to say I had most/all of the drives installed properly, so it's a bit puzzling to me as to why there are so many that aren't being assigned correctly again.  One of the last disk changes I did was a month ago or so that I can recall that I added the 2nd parity drive.  To answer your other questions, there are 14 drives total in the array, not including 2 cache/NVME drives.  No unassigned devices when everything was working (UA plugin was installed but not really used).  

 

Going forward I want to obviously prevent this from happening again - the possibility of losing this much data is pretty terrifying especially for something so trivial as backing up your flash drive.  I'll be looking more into periodically backing up the flash drive via cronjob or plugin or something.

Edited by quadrant6
Link to comment
35 minutes ago, quadrant6 said:

Is there a default location for the Flash backup when it's created from the Unraid webui?

If you use the button at Main - Boot Device - Flash - Flash Backup, you get a downloaded zip file so that isn't on the server. If you were using the CA Backup plugin, then it would be wherever you configured it to save your flash backup.

 

Just because some of your shares are showing up doesn't mean they are complete. Probably some of the files for the shares are on disks you haven't assigned yet.

38 minutes ago, quadrant6 said:

From what I understand, I can assign all disks to disk slots (non-parity) and then the two that don't mount properly should be my parity drives.  The remaining disks are all data and can be assigned freely and don't have to be assigned exactly as they were before, correct?

Parity2 depends on the order of the disks, so if you don't get the order correct you will have to rebuild parity2. But if you find that backup and it has all your disk assignments you shouldn't have to worry about that.

Link to comment
37 minutes ago, quadrant6 said:

From what I understand, I can assign all disks to disk slots (non-parity) and then the two that don't mount properly should be my parity drives.  The remaining disks are all data and can be assigned freely and don't have to be assigned exactly as they were before, correct?

With Dual Parity, data drives position does matter.  (Parity2 is calculated using a matrix math operation.)   Parity1 is position independent. 

 

2 minutes ago, trurl said:

If you use the button at Main - Boot Device - Flash - Flash Backup, you get a downloaded zip file so that isn't on the server.

To clarify this a bit more, using the flash drive backup option, the default location for this zip file in the 'Downloads' directory that you have setup for your browser on your PC.

Link to comment
Just now, trurl said:

Are you talking about the time of the backup that you haven't found yet?

No, that is the date of the backup I did find.  I'm still unsure of the date (probably more recent) that I haven't found yet.

 

Still waiting on the BTRFS cache setup.  Then from there I will stop the array and see if I can assign more (or all) drives again.

Link to comment

Ok, the BTRFS parity check finally finished and the cache pool is in place (didn't know it'd take that long or else I would have waited on that for later).  

 

I found a fairly recent screenshot of my Unraid Dashboard just before I added the 2nd parity drive - the 2nd parity drive was the last drive I added and was showing as an unassigned device 'sdm'.  This is pretty much an undeniable fact based on the dashboard screenshot.  Unfortunately I don't have any other identifying factors in the screenshot for all the other disks.  The other parity 1 drive I have reason to believe is 'sdf' (from the screenshot above) when it auto-assigned from an old backup. 

 

EDIT:  Just got confirmation by starting the array with all data drives that sdf and sdm devices were unmountable file systems, so they are our clear winners for the parity drives.

 

Assuming it's safe to assign sdf as Parity 1 and sdm as Parity 2... 

 

Do I still need to know the other drive positions also?

Edited by quadrant6
Link to comment
31 minutes ago, trurl said:

The letter designation is not a reliable indicator since they are just assigned in the order they are detected. 

Understood.

 

So, I was finally able to locate the most recent flashdrive backup I made, from Oct 13th.  It was in fact in the Backups share, just not shown from when only 5 disks were assigned earlier.  I copied the config over to the flash drive (overwriting everything) and rebooted the server.  All of my drives are now detected in their proper places (except the 2nd parity drive, which is obviously the only remaining unassigned 'sdm' drive).  I believe I am saved now.  Thank you so much for all of your assistance!  You guys are awesome!

 

Last question - If I start the array without Parity 2 assigned, it says it will perform a parity-check as expected.  If I assign Parity 2, it says it will start Parity-Sync and/or Data-rebuild.  Does it really matter if I assign Parity 2 at this point, or should I 'be safe' and assign Parity 1, wait for the parity check, then assign parity 2?  I may just be overly cautious.

 

 

 

Edited by quadrant6
Link to comment
20 minutes ago, ChatNoir said:

On the screenshot you mention, do you have the drives serial number visible ?

If so just use this to assign your drives. As you said, Parity 2 will be the last remaining drive.

Nope, it was just a screenshot of the Dashboard which doesn't have S/N info or any other identifying characteristics.

 

Thankfully I am back up and running - I decided to add the 2nd parity and it's now doing its parity check on 2.  All is functional except for a few minor changes I made after the last backup.

 

Thanks again.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.