missing data after disk recovery

[smakovits]

I had a failing disk, all parity was good.  Prior to replacing, I thought I would be smart and actually move the data off the disk with unbalance.  As the time kept going up, I realized the disk was gone and I proceeded to stop the process and stop the array.  Replaced the failed disk with a new disk and re-built the drive.  However, the disk is empty.

 

Did I screw something up in my process?  If I think about the steps I did, the only thing I did extra was switch from riser to xfs.  Did this recovery drop all the data?  I know it formatted, but I did not think about it as anything different from any other time.

 

As I type, I am thinking I jumped the gun on my conversion.  I should have rebuilt reiser, moved the data off and then gone xfs.  Is this correct?  Is this where I went wrong?  Assume there is no going back, some 2.5TB gone.

 

Is there any way to possibly see what was there that is gone?  I have backups of my music and pictures, which is most important.  Other stuff is replaceable, however, I am just looking for a way to know where to start.

[itimpi]

Your mistake is that you tried to change the file system type and did a format as that created a new empty file system on the drive and updated parity to reflect this.   You should have had a warning about a format never being part of a data recovery process.   The rebuild process cannot be used to change a file system type.

 

The reiserfsck program is very good at recovering extremely corrupted file systems so there is a faint chance that if you now stop the array; explicitly set the file system back to reiserfs and then try to repair the file system it might be able to recover most of the data.   Be warned, though, that reiserfsck can take many hours to run.

[smakovits]

6 hours ago, itimpi said:

Your mistake is that you tried to change the file system type and did a format as that created a new empty file system on the drive and updated parity to reflect this.   You should have had a warning about a format never being part of a data recovery process.   The rebuild process cannot be used to change a file system type.

 

The reiserfsck program is very good at recovering extremely corrupted file systems so there is a faint chance that if you now stop the array; explicitly set the file system back to reiserfs and then try to repair the file system it might be able to recover most of the data.   Be warned, though, that reiserfsck can take many hours to run.

Since the drive is empty and new, are you suggesting to change the new drive back? 

 

Or are you suggesting putting the failed drive back instead. 

 

Or are you suggesting something completely different? I am game to try, just need to understand the steps to try what you described. 

[trurl]

24 minutes ago, smakovits said:

putting the failed drive back

Put it back in the server, but not back in the array. See if you can mount the failed drive as an Unassigned Device.

[smakovits]

17 minutes ago, trurl said:

Put it back in the server, but not back in the array. See if you can mount the failed drive as an Unassigned Device.

Is it enough to do it as a external USB mount or do I want it as a sata mount? 

[itimpi]

1 hour ago, smakovits said:

Since the drive is empty and new, are you suggesting to change the new drive back? 

That is what I was thinking may work as Unraid will have restored the drive at the bit level including any corruption of the reiserfsck file system by the format action.

 

Of course if you can get the data off the original drive that is even better.

[smakovits]

9 minutes ago, itimpi said:

That is what I was thinking may work as Unraid will have restored the drive at the bit level including any corruption of the reiserfsck file system by the format action.

 

Of course if you can get the data off the original drive that is even better.

are you suggesting there is corruption restored to the new disk or that by it is better off since I formatted it off for the new disk?

 

As for getting the original data back, can we do a bit level recovery if I mount it as a external USB?  They are crc rewrites and not a clicking disk, so not sure what is possible.

 

I know when I was trying to get it all off with unbalance, it started at like 30 hours and 192 errors and when I stopped it it was 50 hours in with 75 hours remaining and over 1200 errors, so not sure what is possible with it.

[trurl]

20 minutes ago, smakovits said:

not sure what is possible with it

Can't find out without trying.

 

20 minutes ago, smakovits said:

corruption restored to the new disk

The disk is formatted XFS and is probably not corrupt as an XFS filesystem. From the point of view of ReiserFS, XFS formatting is corruption.

[trurl]

1 hour ago, smakovits said:

external USB mount or do I want it as a sata mount

SATA connection might be more dependable but I use external USB dock all the time for reading/writing Unassigned Devices.

[smakovits]

21 minutes ago, trurl said:

SATA connection might be more dependable but I use external USB dock all the time for reading/writing Unassigned Devices.

I'll try this to try and get my most critical data first and see where it gets me. 

[itimpi]

1 hour ago, smakovits said:

are you suggesting there is corruption restored to the new disk or that by it is better off since I formatted it off for the new disk?

I am suggesting that if you explicitly tell Unraid the drive should be reiserfs then the reiserfsck program might be able to recover most of the contents from the corruption caused by formatting it to an (empty) xfs file system.

[trurl]

20 minutes ago, itimpi said:

reiserfsck program might be able to recover most of the contents from the corruption caused by formatting it to an (empty) xfs file system.

1 hour ago, trurl said:

From the point of view of ReiserFS, XFS formatting is corruption.

which might be repairable.

[smakovits]

As a side topic. I thought we want xfs these days.  I know I started going xfs for that reason with new disks. But this now makes me question my doings. 

[JonathanM]

13 hours ago, smakovits said:

  I should have rebuilt reiser, moved the data off and then gone xfs.

This, except copy vs. move.

13 minutes ago, smakovits said:

As a side topic. I thought we want xfs these days.  I know I started going xfs for that reason with new disks. But this now makes me question my doings. 

ReiserFS performance is horrible over 2TB, especially when deleting things, worse when the file system is mostly full. It's never going to get better, the project has been largely abandoned and the author of ReiserFS is incarcerated in California for murdering his wife.

 

It's a pity, because for 1TB and smaller it was an extremely robust filesystem that could recover from almost any corruption. It just hasn't aged well.

[itimpi]

19 minutes ago, smakovits said:

As a side topic. I thought we want xfs these days.  I know I started going xfs for that reason with new disks. But this now makes me question my doings. 

Reiserfs is great for recovering from corruption but it has several downsides (which is why is now deprecated) such as:

• it cannot handle disks larger than 16TB (and you can now get drives larger than this)
• performance degrades badly as disks get anywhere near full
• although recovery from corruption is very good it can be very slow (whereas xfs often completes in seconds) on large drives.
• it is no longer being maintained or developed at the Linux level (the main developer is in prison for murdering his wife) so bugs on newer kernels will probably never get fixed.

There may be some other downsides I have not mentioned but the ones already mentioned are not insignificant  

[JonathanM]

2 minutes ago, itimpi said:

although recovery from corruption is very good it can be very slow

That's an understatement. I remember waiting over 24 hours for a drive to mount after an unclean shutdown back in 4.7 days. Didn't lose anything though.

[smakovits]

4 hours ago, itimpi said:

I am suggesting that if you explicitly tell Unraid the drive should be reiserfs then the reiserfsck program might be able to recover most of the contents from the corruption caused by formatting it to an (empty) xfs file system.

This perhaps is a misunderstanding. The drive that is now xfs is the new 8tb drive I put in to replace the failing 3tb drive. The failed/failing drive is still reiserfs and has my data, it is simply sitting on the counter. 

 

The 8tb drive is xfs, new and empty and never had anything on it. 

 

All which makes me think maybe I try usb mounting and start copying folders. Question is, is this where I user reiserfsck knowing unbalance was not cutting it prior to swapping the disks. 

[trurl]

Don't think we ever saw a SMART report for that failed disk. Often nothing wrong with a disk and some other problem is to blame for the disk getting disabled. See if you can get a SMART report for that disk after you plug it in and post it.

[itimpi]

1 hour ago, smakovits said:

This perhaps is a misunderstanding. The drive that is now xfs is the new 8tb drive I put in to replace the failing 3tb drive. The failed/failing drive is still reiserfs and has my data, it is simply sitting on the counter. 

If you rebuilt the 8TB drive to replace the old reiserfs drive then Unraid would have been emulating the missing drive and the rebuild process will have restored the bit pattern that was on the original drive (as parity is not aware of file systems or data - just bit patterns).   That is why I think it might be worth trying to run reiserfsck against it.   

[trurl]

See if you can get your files from the original disk first. If you "repair" that new disk it might recover the ReiserFS filesystem, but then it would be ReiserFS again.

[smakovits]

4 hours ago, trurl said:

Don't think we ever saw a SMART report for that failed disk. Often nothing wrong with a disk and some other problem is to blame for the disk getting disabled. See if you can get a SMART report for that disk after you plug it in and post it.

mounted as USB, but I believe that limits the SMART capabilities.  I can try other things if needed.

 

smartctl -H /dev/sdr
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.10.28-Unraid] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   001   001   005    Pre-fail  Always   FAILING_NOW 2005

 

[trurl]

The drive says it is failing now so that settles it. Maybe you can still read something from it though.

[smakovits]

27 minutes ago, trurl said:

The drive says it is failing now so that settles it. Maybe you can still read something from it though.

yeah, so the question is, should it be possible as an unassigned device or do I want to put it in another system?

 

As an unassigned drive it only shows 802GB, as size vs 3TB.  and it is not mounted or shown mounted.  The only immediate option in the UI is format which I will not do.

 

I assume there might be a way to access the data via CLI or do we just consider it dead

 

[trurl]

If the unassigned drive isn't mountable you could try to repair its filesystem.