Jump to content
Sign in to follow this  
dogfluffy

Unformatted disk after data rebuild

115 posts in this topic Last Reply

Recommended Posts

Version 5.05 and I was attempting to update to version 6 prior to this but was unable due to a driver issue with one of my disk controllers.

 

Anyway I'm not entirely sure what happened as it was a routine swap and rebuild. I may have had the disks in maintenance mode during the rebuild, can't recall but after completion I had an empty disk and a new prompt asking me to format. I rebooted a few times and did the format, although I'm not sure why in retrospect. I also have the original pulled disk, I ran Seatools for Windows and repaired some bad sectors, drive tests Good now. After the format as you might expect it shows a blank disk. I also later ran reiserfsck on one of the /dev/sdg1 which after test asked to rebuild-tree which I also did successfully. After checking some disks I restarted the array and one of the WD drives froze and subsequently was re-added to the array, and now it says it has an unformulated partition. I finally dug up my lost account info and figured I better ask for some help. Thanks in advance.

Share this post


Link to post

I hope you have backups. I think you are about 3 steps too deep to expect comprehensive recovery at this point. Can you post a screenshot of main as it is currently? It's really unclear from your description where you have ended up.

 

Which disk controller is a problem with V6?

Share this post


Link to post

I'm not sure which controller it was. I installed 6.0 and a whole series of disks was missing so I used the email an issue feature which said it was uploading a log file along with my email address and the bug description, but I haven't heard back. I'm also unclear where I ended up. I'm still not sure what happened to the original rebuild and have been having dropped disks and rebuild issues every few weeks for awhile now, which prompted the upgrade attempt. Basically unraid would report disks as bad or disabled, I would pull and swap, rebuild and carry on. The disk would then test OK outside unraid so I would cycle them back through again until...

Share this post


Link to post

Sounds like you should have asked when you first started having issues. Drive connection issues are by far the most common problem, actual bad drives much less so. Restoring from your backups is probably the simplest since it would avoid having to get into much greater detail about what you done and what you have to work with.

Share this post


Link to post

There are no backups. It's a 42 TB array full of 3TB disks. I have the original disk that redballed in both cases, although the first may have been precleared before I realized what was happening.

Share this post


Link to post

Pretty much everything you described in your first post was the wrong way to go. By repairing an sd device instead of an md device, for example, parity is invalid. And of course, formatting a disk with data on it is always a bad idea. And parity is no substitute for backups.

 

Are any of the files on your server important and irreplaceable?

 

I think we need a better picture of where you are now before making any recommendations. Post a screenshot of Main for starters.

 

Share this post


Link to post

Go ahead and start the array, and post another screenshot after it's started.

Share this post


Link to post

You mentioned reiserfsck on sdg1. Your screenshot says sdg is currently disk3, but that is not necessarily always the case. Whichever disk it was, if you did reiserfsck on the md device, then parity would have been maintained, but since you said you did it on sdg1, then I think we have to assume that parity is invalid.

 

Do you know which disk slot you ran reiserfsck on? Can you tell us more about the unformatted disk10? What about the missing disk13? Which slot was the "original pulled disk"?

 

Please review your original post in this thread, and see if you can rewrite it in such a way that we can clearly identify which specific disks you were referring to in each part of your post.

Share this post


Link to post

I meant something like /sdg1, I didn't recall at the time which one exactly. I ran it on /dev/sdh1 or disk 11, the Toshiba, while the array was offline if I remember correctly, but also after I had formatted that same disk. I believe again I had performed the rebuild with the array in maintenance mode which left me with a blank disk after the Toshiba rebuild. I first ran resiserfssck --check which prompted for the --rebuild-tree /dev/sdh1 which I also ran with the array offline. I have the Seagate 3TB which I pulled and replaced with the Toshiba. The WD that now reports as unformatted had been disabled and rebuilt...but I believe while the array was in maintenance mode. I hope that makes sense. I also have 2 more 3 TB disks which have been in the array at some point. I can't recall their exact state currently but a safe guess is they have been precleared.

Edited by dogfluffy

Share this post


Link to post

I don't really remember V5 well enough. Why does it say disk13 is missing before the array is started, then says it is not installed after the array is started? And everything else is green? If this was V6 I would ask if you can read the contents of the emulated disk13, but the screenshot seems to indicate it is not being emulated.

 

Simplest thing might just be to New Config with only the known (?) good (?) disks, rebuild parity, then see what if anything can be read from the assortment of other disks you have laying around.

 

What about this?

19 hours ago, trurl said:

Are any of the files on your server important and irreplaceable?

 

Anybody else have any ideas?

Share this post


Link to post

Emulated disk13 appears to be mounting fine, it shows the free space on the GUI.

 

He can also try running reiserfsck on disk10.

Share this post


Link to post
5 hours ago, johnnie.black said:

it shows the free space on the GUI.

I noticed that. The V5 GUI is a bit ambiguous compared to V6. And if he repaired an sd device, parity is likely out-of-sync to some extent, so rebuild of disk13 may be compromised.

 

So, dogfluffy, can you?

5 hours ago, trurl said:

read the contents of the emulated disk13

???

Share this post


Link to post

OK, it's weird in that some of disk 13 is actually there, the directory structure seems mostly good with 8 reported missing, but a bunch of files are missing as well. I'm clicking via the browser window, folder icon and the 8 directories missing list on console as: /usr/local/emhttp/mnt/disk13/...missing ...due to locked reiserfs recordtype error message. Also disk 13 and 9 still appear in the listed disks for shares, Disk 10 is not listed as browsable and 10 and 11 have been removed from the shares disks list entirely while 11 is browsable as blank. Disk 11 is again the one I have the original Seagate which ran Seatools and repaired a few sectors, then passed a long generic test.

 

There is also a /mnt/user/lost+found on disk 9 which contains data under coded names and directories. I can browse further via smb shares if that helps.

Edited by dogfluffy

Share this post


Link to post
2 hours ago, dogfluffy said:

but a bunch of files are missing as well

 

Not surprising since you ran reiserfsck outside the array, so parity is out of sync, you can rebuild to a new disk and salvage as much data as you can between that and the old disk.

Share this post


Link to post

And disk10 is unmountable so isn't browsable. Disk11 as you said is newly formatted so has no files in any share.

 

Do you by any chance have the SNAP plugin installed? It would be useful if you could mount the old disk11 and old disk13 outside the array. If not we can mount them from the command line.

 

I think johnnie.black has a good plan for starters. Rebuild disk13 to a new disk. Then we can go on to possibly fixing that rebuild if necessary, and seeing what can be done with the old disk11 and old disk13.

Share this post


Link to post

Yeah this project went down the rabbit hole in a hurry. I can't tell what I was trying to do in reverse, and I've forgotten what happened and why exactly. What really caused my problem was a weirdly bad UPS, so when I was getting disk errors I would disconnect and move the server to test on my workbench, everything would pass, connect back to the UPS and start getting errors again. I thought I had some errors, but they just needed to sync up.  Made it all worse, obviously. Finished my remodel project and I dusted off the old server to try and salvage data, but get it up and running again. I had tried upgrading from the version 5, but one of the controllers on my motherboard wasn't recognized. Board is Intel Bx975XBX2 and it has 2 different Marvell controllers I think. I also have an add-in controller, but when I upgraded I was missing 4 disks so it had to be one from the main board. I fired up the server and I'm back where I was before, I also have 2 disks with unknown or no content left. I don't remember where I was or why. I would need to brush up the upgrade and backup process to try a newer version. I'm not sure what is the best idea at this point. Thanks for all your help. I know this doesn't make much sense.

Share this post


Link to post

I don't know what the SNAP plugin is, but I have the unmenu I think? I also have a USB to SATA controller I could connect to try and read what is on these other disks. I don't know which disk is number 13 until I can read the serial number inside the case, it's a tight stacked horizontal layout. And I suppose with a sharpie, I've got a new project next time I'm yanking disks.

Screenshot 2019-01-29 at 1.41.50 PM.png

Edited by dogfluffy

Share this post


Link to post

I think maybe we'd be able to make more sense of what you have if you left unMenu out of it. So go back to giving us screenshots of just the regular unRAID UI as you had in your previous screenshots.

Share this post


Link to post

Since you let this go for another year and a half before getting back on it, maybe you aren't in any hurry to get this done.

 

The simplest and best long-term solution in my opinion would be to simply set this aside, build a new server with new disks and the new Unraid, then try to copy the data from these old disks to the new server.

Share this post


Link to post

I don't have a new server to build. The screenshots look identical to the original ones, as the server sat idle. I added the unmenu as it had different options because someone mentioned 5.0.5 was vague, but it's showing the same thing. I'm ready to try to recover and rebuild this system now.

Edited by dogfluffy

Share this post


Link to post
On 8/31/2017 at 7:20 AM, trurl said:

I think johnnie.black has a good plan for starters. Rebuild disk13 to a new disk. Then we can go on to possibly fixing that rebuild if necessary, and seeing what can be done with the old disk11 and old disk13.

 

 

Share this post


Link to post

So my new disk 13 has at least a data structure on it, when I browsed it. I'm not keen on the idea of blasting data by formatting to begin. I'd like to find out what I actually have and what is rebuildable if possible. I'm not sure if it's useful data, it only seems to have the folder structure, but maybe that's because the array was out of sync? I'm trying to find the blank spots in the array, but ultimately would like to see what data is still on these disks, if any. My memory is fuzzy but I think I may at least have the original disk 10 or 11, which I think is now added to the array by empty. Does that make sense? Ultimately what I think is happening, or was, I thought I had failing disks because they kept redballing. It was only when I was connected to the UPS they were failing though, I would yank, rebuild on my bench all fine for weeks, then fail again. So...I think all 3 of my disks are ultimately sound, but likely dirty, data wise. I was hoping there was a way to ID a disk to the software raid assignment, because I'm sure I got mixed up somewhere.

Edited by dogfluffy

Share this post


Link to post

Nobody ever suggested formatting anything. Format is not and has never been part of the rebuild process.

 

So do you have the original disk13? And also a new disk13 with something on it? Or are you actually talking about the emulated disk13? 

 

Actually, before doing anything at all let's go back to this unanswered question:

On 8/29/2017 at 9:10 PM, trurl said:

Are any of the files on your server important and irreplaceable?

 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this