Help! 2 disk with "Wrong or no file system"

[jcarre]

Hello,

 

Today I received a message that disk 4 had errors on it. I looked at the smart report and it had a bunch of UDMA CRC errors, so I thought it was just a bad connection and was no big deal. I tooked the array down, to take it out, but it got stucked, so I did a normal reboot (with the GUI). When it came back online it now shows that another disk has an unmontable file system, and the errors of the first one disappeared.

 

As you can see both drives are Seagate drives. All the other drives are connected to a HBA card, but those two where having problems with it so I connected them directly with sata cables.

What's the best approach I can take dealing with this situation? Do I try changing the cables, the port and just start rebuilding? I cannot understand why another drive went unmontable.

Thank you!

[trurl]

Attach diagnostics to your NEXT post in this thread

[jcarre]

Yes, sorry I forgot!

ivpiter-diagnostics-20220607-1941.zip

[trurl]

Do you have ECC RAM?

[jcarre]

No I have regular DDR4 sticks

[trurl]

Have you done memtest?

[jcarre]

No, I haven't. Had the system for +1 year. Do you suggest me doing it?

[trurl]

Memtest might be a good idea just to make sure it isn't the cause of your unmountable disks. You can't fix anything if RAM isn't perfect.

 

 

[jcarre]

Doing the test right now, so far so good (nearly 1 pass done). Will leave it overnight to make sure.

What should I do if memory turns out ok?

[trurl]

Rebuilding won't fix unmountable.

 

https://wiki.unraid.net/Manual/Storage_Management#Drive_shows_as_unmountable

[jcarre]

Ok, ram seems to be ok!
Ran a test run of xfs_repair on disk8 the results are as follows:

 

Do I now run the check without the "-n" option, should I add any additional option?

Thanks!

ivpiter-syslog-20220608-0937.zip

Edited June 8, 2022 by jcarre

[JorgeB]

1 hour ago, jcarre said:

Do I now run the check without the "-n" option

Yes, and only if it asks for it use the -L option.

[jcarre]

Ok, just did (had to add the -L option) and this is the result. The disk still shows as unmontable

 

 

[JorgeB]

6 minutes ago, jcarre said:

The disk still shows as unmontable

Did you re-start the array in normal mode? If yes post new diags.

[jcarre]

9 minutes ago, JorgeB said:

Did you re-start the array in normal mode? If yes post new diags.

Restarting in normal made made disk8 come back! Thanks!

I now have disk4 disabled with the wrong filesystem. Do I try the same procedure? Or I format it and start a rebuild?

[JorgeB]

4 minutes ago, jcarre said:

Or I format it

Never format, that's not part of a rebuild.

 

First run xfs_repair and depending on the result decide if you rebuild on top or not, i.e., if the emulated disk is fixable and contents look correct and there's nothing or not much in a lost+found folder.

[trurl]

57 minutes ago, jcarre said:

Or I format it and start a rebuild?

If you format the only thing it can rebuild is a formatted disk. Parity is updated on all write operations, write, move, copy, delete, format are all write operations. Format writes an empty filesystem to the disk, parity is updated, and so then rebuild can only rebuild an empty filesystem.

[trurl]

54 minutes ago, JorgeB said:

run xfs_repair

on the disabled / emulated disk. It works just like the other repair you did, but doesn't really change the physical disk itself. It just repairs the emulated disk, same as when reading or writing a disabled disk, its contents are emulated by reading all other disks and parity, and updating parity for writes.

After the emulated disk is repaired, if the results look good then you can rebuild those results. If not then you can consider if the actual contents of the physical disk are better.

 

If you do the rebuild to another disk, you have both the original and the rebuilt repair and can compare and get files from either.

[jcarre]

The repair is done (it took a long time!) I uploaded the results in a file below, but it looks that a lot of things happened.
Don't know if it would be wise to just start the array with this drives. What do you think?

Other option is taking the drive out and putting it back to let it rebuild from parity right?

Thanks!.

Edit: uploaded file seems to not be working. Use: https://sharetext.me/mnzyojsjlt

xfs_repair results.txt

Edited June 8, 2022 by jcarre

[JorgeB]

Not looking great, but just start the array as is in normal mode and check the contents of the emulated disk4, look for a lost+found folder and see if there are many things there.

[jcarre]

Disk4 just have the lost+found folder which has 990 files on it:

 

root@ivpiter:/mnt/disk4/lost+found# ls | wc -l
990

 

The device is disabled and the contents emulated.

[JorgeB]

Probably best to go with the actual disk then, unless you have a spare you could use, to check the actual disk contents stop the array, unassign disk4, start the array (leave disk4 unassigned), stop the array, then you can mount actual disk4 with the UD plugin to see if it mounts and contents look OK, note that the array can't be started or the disk won't mount without first changing its UUID.

[trurl]

48 minutes ago, jcarre said:

Other option is taking the drive out and putting it back to let it rebuild from parity right?

Rebuilding from parity will give the same results as the repair. The parity array is emulating the repair you did, and rebuild always results in the same as the emulated disk. (how could it be otherwise?)

2 hours ago, trurl said:

After the emulated disk is repaired, if the results look good then you can rebuild those results. If not then you can consider if the actual contents of the physical disk are better.

 

[jcarre]

The parity rebuild finished succesfully, the array Is now working properly.

 

Thank you very much for you help!

[trurl]

How much data in lost+found?