Solved - Drive marked Unmountable Wrong or no filesystem

[Drezzden865]

Hello i have had my Unraid running for 2-3 years now and check my drive health quite often and yesterday when i checked it their where ZERO errors and temps where good at 95. Today with out the device crashing or power loss all my docker apps stopped working and i logged in remotely to see what wast up. in the Docker section i have no apps and it stated No Docker Containers Installed. I then checked the main tab and found that Drive 2 stated Unmountable Wrong or no filesystem ( this is a 6 TB drive that had about 4TB on it) i attempted to resolve by restarting the device. I then attempted a repair via starting the array in maintenance mode this failed and did nothing that i could tell. I then removed the drive and tried to re-add the drive which now shows formatted. I was able to get the array started again with out the drive.
My issue is what could have caused this and how do i rebuild the drive from the parity.

Note- I have a larger drive that i plan swapping for the Parity drive and rotate the 8tb parity drive to replace the 6tb i just haven't done it. it just came in this weekend.demonreachunrai-diagnostics-20230802-1548.zip

Attached are what I'm seeing and I am really worried i just lost 2-3 years of setting up programs and tools.

 

Please help. Going crazy here.

Edited November 30, 2023 by Drezzden865

[trurl]

You mention starting in maintenance mode, but you don't specifically mention running a repair on the disk while in maintenance mode. Wish you had asked for help sooner.

 

Since you formatted disk2 in the array, the only thing that can be rebuilt from parity is a formatted drive. Format is never part of rebuild.

 

Do you have backups?

 

[Drezzden865]

So i that was a dumb idea to format and i shouldnt have but i did start the rebuild and it ran for a couple hours and started showing the same error on the disk. So i replaced the disk with a new larger disk last night. it started the rebuild. Today when i woke up i received notice that the disk failed and was disabled. i now have the same 

Disk 2ST8000VN004-3CP101_WWZ2NL03 - 8 TB (sde) Unmountable: Wrong or no file system

I don't know what to do. I'm thinking I might change the drive port on the motherboard maybe the port has gone bad. I feel I'm grasping at straws at this point. 

Edited August 12, 2023 by Drezzden865

[trurl]

post new diagnostics

[Drezzden865]

This is RESOLVED im pretty sure. I finally found this article and was able to get the the sever in maintenance Mode run. the script 

xfs_repair -v /dev/md2

 it ran supper fast a completed on the new disk. 

I wish i had been calmer and found this the day it happened. 

[Drezzden865]

I thought i had this fixed and this issue might be because it not showing Unmountable. but this morning it is now showing that Disk 2 is disabled. What is going on. 

[JorgeB]

On 8/12/2023 at 10:38 PM, trurl said:

post new diagnostics

 

[Drezzden865]

demonreachunrai-diagnostics-20230817-0710.zip

[JorgeB]

Disk was disabled at boot, so we cannot see what happened, but SMART looks OK and assuming the emulated disk contents look correct you can rebuild on top:

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

[Drezzden865]

Thank you that looks like it was the solution

 

[Drezzden865]

The disk is showing disabled again. Same day as the following the steps you instructed. 

 

demonreachunrai-diagnostics-20230817-2301.zip

[JorgeB]

Looks more like a power/connection problem, replace both power and SATA cables and try again.

[Drezzden865]

So Update.

I have replaced the PDU with a 1050w unit but in the ordeal i ended up having to replace the motherboard and CPU.

Which caused my Lic to fall off and i lost alot of my configs as well and my server name. I was able to reach out to UNRAID and get my License readded.

While i was doing this i added another 32g of RAM and a 2tb M.2. I also attempted to swap out the 8tb Parity drive to a 12tb as well as purchased 2 additional 12 tb to change out the drives causing issue.

 

I put in the new M.2 and copied everything over and moved all the shares to the new catche and i was pretty sure Docker. When i changed the PDU i took the old Cache out of the tower and installed one of the 12tb drives to replace the parity with the original still attached.

Upon power on, in the software i swapped the parity drive from the current 8th to the new 12 tb and moved the 8tb into the place of disk1 that was not showing at all in the raid anymore. Both rebuilt and it all 8 disks were green.

When finished my Docker wouldn't start, when checking my system the docker was still pointing to the old Parity drive. The system wouldn't let me remap it, every time i would go to apply it would just revert back. Disks one and two are still showing Unmountable: Wrong or no file system. Also all of my shares are missing in the software.  i took the system down removed the disk one a plugged to old Cache drive in its place. I then turned the tower back on, it is now showing that the old cache drive is back installed and the new M.2 is showing as well. The the old 8 tb Parity drive (that should be the new disk1) is showing up as showing up as the the parity drive and the 12 tb drive is not showing up at all.

I attempted to run the script from before and was informed that the file did not exist.

 

my shares are still not showing up and Docker is still not starting.

I have turned the tower off and checked all connections. and the 12tb device is still now showing back up, but it turns out not all the sata connectors on the Motherboard work. and i dont have any available ports. When i plug the 12tb in again the system started rebuilding the drives again.

 

Edited November 23, 2023 by Drezzden865

[JorgeB]

Post new diags.

[Drezzden865]

tower-diagnostics-20231123-1738.zip I did schedule a support session for Sunday. because i'm not getting anywhere on this.

Edited November 24, 2023 by Drezzden865

[Drezzden865]

Support Canceled my appointment 10 mins after it should have stared and i have already paid for it.

[trurl]

I don't see any disk for disk1 in those latest diagnostics. Is disk1 attached?

[JorgeB]

Like mentioned disk1 is not being detected, check the board/controller BIOS, if it's not detected there it also won't be detected by Unraid.

[Drezzden865]

Yes those where the Diagnostics from before i removed disk one and started the rebuild of the parity and disk one again. I can pull some more tonight and post them.

 

[Drezzden865]

tower-diagnostics-20231128-1009.zip

@JorgeB and @trurl He are the most recent Diagnostics. The rebuild of the Parity drive completed again and the disk1 rebuilt again as well. Both Disks 1 and 2 are still showing  the filing system error. My Docker still wont start or update the file path and the shares are still missing i worry that i have just lost all my data.

Edited November 28, 2023 by Drezzden865

[JorgeB]

Check filesystem on disks 1 and 2, run it without -n, you should also update Unraid to latest stable, or at least to latest v6.11.x release, which is 6.11.5

[trurl]

Check filesystem on disk1 and disk2, be sure to capture the output so you can post it.

 

1 hour ago, Drezzden865 said:

worry that i have just lost all my data

The other disks seem to have plenty of data, user shares will probably work again after you get disk1 and disk2 filesystems repaired.

 

You must always have another copy of anything important and irreplaceable. Parity is not a substitute for backup.

[Drezzden865]

@JorgeB I will run that tonight and if it works ill do the update. Or should i do the update first?

@trurl All the important data is backed up in other places and i have a record of everything on there is just alot of data to replace.

What about getting the shares back and access to the data again?
What do i do about the Docker not letting me change the path to the new cache drive?
 

Thank you Both for assisting

[trurl]

1 hour ago, Drezzden865 said:

What about getting the shares back and access to the data again?

 

4 hours ago, trurl said:

user shares will probably work again after you get disk1 and disk2 filesystems repaired

 

1 hour ago, Drezzden865 said:

Docker not letting me change the path to the new cache drive?

Disable Docker and VM Manager in Settings until user shares are working again. We can take another look then 

[Drezzden865]

Any idea how long it will take to Check the Filesystem on a tb drive. I triggered the scan without the -n on disk1 and it instantly told me

{

Phase1 - Find and verify superblock...

Bad Primary Superblock - -Bad macgic number !!!

Attempting to find secondary superblock....

}

It has been sitting ther stating running for about 4 hours now

attached is the result and nothing seems to have changed

 

Disk 1 and 2 Check logs.txt

Edited November 28, 2023 by Drezzden865