Disabled Drive rebuilt shows tons of corruption (SOLVED)

[gnollo]

Got home a found a disabled drive. Bought a new one, popped it in, rebuilt the drive.

Then I realised that the rebuilt drive was read only.

Run reiserfsck which showed 136 found corruptions. Fixed the broken directories, lost a ton of data.

So i used the Unassigned Devices plugin to mount the previously disabled drive. The data is still there, so I used the following command to send the missing files from the drives

 

rsync -avrth --progress //mnt/disks/WDC_WD20EADS-00S2B0_WD-WCAVY1112779-part1/ //mnt/disk10/

 

Once completed, I checked the two disk sizes. The newly rebuilt drive according to the UI is using 1.34TB. The old drive is using 1.86TB.

 

Run the command again, this is what I get:

 

"sending incremental file list

 

sent 951.58K bytes  received 4.47K bytes  637.36K bytes/sec

total size is 1.86T  speedup is 1,946,793.25"

 

What gives? Are the files there? If so are they different size? Shouldn't they copy over if they are not?

How do I know if the data on the old drive is on the new one, and that is IDENTICAL?

I'm stumped. Short of deleting ALL of the data on the new drive, and repeat the command, I am not sure what to do.

[gnollo]

root@Tower:~# df -h

Filesystem      Size  Used Avail Use% Mounted on

tmpfs          128M  2.3M  126M  2% /var/log

/dev/sda1      486M  117M  369M  25% /boot

/dev/md1        1.9T  1.7T  174G  91% /mnt/disk1

/dev/md2        466G  33M  466G  1% /mnt/disk2

/dev/md3        466G  403G  64G  87% /mnt/disk3

/dev/md4        466G  420G  47G  90% /mnt/disk4

/dev/md5        466G  433G  34G  93% /mnt/disk5

/dev/md6        1.9T  1.7T  124G  94% /mnt/disk6

/dev/md7        466G  415G  51G  90% /mnt/disk7

/dev/md8        3.7T  2.6T  1.2T  70% /mnt/disk8

/dev/md9        1.9T  1.5T  344G  82% /mnt/disk9

/dev/md10      3.7T  1.3T  2.5T  34% /mnt/disk10

/dev/md11      1.9T  1.7T  185G  91% /mnt/disk11

shfs            17T  12T  5.0T  71% /mnt/user

/dev/sdj1      1.9T  1.7T  129G  94% /mnt/disks/WDC_WD20EADS-00S2B0_WD-WCAVY1112779-part1

[gnollo]

I decided to experiment. Started a new configuration. Replaced the rebuilt drive with the old drive.

Restarted the array, waiting for parity to be rebuilt.

I will then replace the drive and re- rebuild the data using the newest drive. Let's see if this avoids the data corruption.

[trurl]

Did you check the old drive for corruption? Most likely the reason the new drive was corrupt is because the old drive was. Rebuilding just rebuilds what was there except for possibly the final failed write which disabled the disk and any subsequent writes to the emulated disk. Since you fixed corruption on the new drive but not the old it is not surprising they have different contents.

[gnollo]

Did you check the old drive for corruption? Most likely the reason the new drive was corrupt is because the old drive was. Rebuilding just rebuilds what was there except for possibly the final failed write which disabled the disk and any subsequent writes to the emulated disk. Since you fixed corruption on the new drive but not the old it is not surprising they have different contents.

 

That would be a bummer, if parity just updates including the drive corruption. This would mean I really have no way of getting my data back!

So I started calculating parity on the array with the old drive back in, Tower notified me that there were two disks with errors during parity calculation, and then confirmed that parity calculation was complete, but with a ton of errors "Parity sync: finished (14456654 errors)"..

After that, it decided to put Disk 10 in error state (disk dsbl) [this is the drive I have been trying to rebuild]

 

Disk 9 - WDC_WD20EARS-00J2GB0_WD-WCAYY0067586 (sdk) (errors 33)

Disk 10 - WDC_WD20EADS-00S2B0_WD-WCAVY1112779 (sdj) (errors 14457636)

 

Now I am going to try the other approach, add the new drive into the array, make the array new again, calculate parity, then try to copy the files over from the busted drive.

If that fails too, I will reisercheck the old drive and fix it, and see how much data I can restore.

Does it sound like a plan? How do I preclear that drive with the new UI?

 

 

 

 

[gnollo]

So, rebuilt array without the new and the old drive 10.

No problems whatsoever, including no read errors from disk 9 (which were caused by the errors on disk 10?).

I will now add the new 4TB drive. Then I will use the command:

rsync -avrth --progress //mnt/disks/WDC_WD20EADS-00S2B0_WD-WCAVY1112779-part1/ //mnt/disk10/

 

and see how much of the reported 1.8TB of data moves across.

 

Anyone has a better idea to get all the data off the drive?

[gnollo]

After I used the rsync command below, the system started to copy files across.

When it completed it gave me this message

 

"sent 1.87T bytes  received 546.10K bytes  26.64M bytes/sec

total size is 1.86T  speedup is 1.00

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]"

 

1.86Tbytes is the size of data on the old drive, so this is a better result than the one I achieved by rebuilding the drive.

Now how do I find out what the errors were?

[gnollo]

So I rerun the command, it tried to update a 4GB mkv rip across to the other drive and reported read errors mapping file xyz input/output error (5) warning file xyz failed verification (will try again).

It then tried it again, failed and then the update was discarded.

So it would look like I only lost a file, which I can re-rip.

[gnollo]

Concerned by the fact that unraid parity was unable to rebuild parity, and there was no warning that parity was corrupted. If I had a total failure with the drive, I would have lost a ton of data.

[gnollo]

Does anyone know what is the best way to compare the two drives and confirm with absolute certainty which files are not IDENTICAL across the two drives?

Thanking you...

[Squid]

Via md5 checksums.  (bonienl file integrity plugin)

[gnollo]

WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS.

 

The disk is a reiserfs disk...

 

[Squid]

WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS.

 

The disk is a reiserfs disk...

There are a great many apps for Windows / Mac in addition to FIP that will also generate checksums.

[gnollo]

So checksums tells me if the data on the drive is corrupted, but it doesn't tell me if drive b is missing data that is on drive a right?

[gubbgnutten]

So checksums tells me if the data on the drive is corrupted, but it doesn't tell me if drive b is missing data that is on drive a right?

 

Save a list of all files and their corresponding checksums from each system and compare the lists. Both corrupted and missing files will show up that way.

[gnollo]

So checksums tells me if the data on the drive is corrupted, but it doesn't tell me if drive b is missing data that is on drive a right?

 

Save a list of all files and their corresponding checksums from each system and compare the lists. Both corrupted and missing files will show up that way.

 

So I have been using Corz checksum for windows, and it works fine on the new drive I copied the data from (I tested it on my bluray folder first). When I try to build a single file for the same folder on the old drive with pending sectors, the program crashes.

 

Any other suggestions are welcome!

[gnollo]

I compared the two drives using the cmd window and "dir /b /s", then verified which files were missing on the new drive. It consists of a movie in the bluray folder, which explains why the check fails in that folder. Deleted the folder in question, and now using Checksum Compare to compare the two directories across the drives. Let's see if that works.

[gnollo]

Checksum compare is awesome, compares directories easily calculating and comparing checksums for every file in the subdirectory too.

Problem solved.