Unraid 6.9.2 two drives failed at same time or..?

[KTgsxr]

my system was running fine a few hours ago and just notice it was offline. went and checked and I have two drives that say Unmountable: not mounted one of them says device is disabled drive is emulated the other is still green. Just seems odd that two drives failed at the same time. I have not tried replacing sata cables yet just also seems odd that two cables failed at the same time? I have attached my diagnostics not sure if anyone can take a look and give any suggestions?

 

Thank you in advance.

tower-diagnostics-20210620-2032.zip

[limawaken]

coincidentally i was in the same situation very recently (and not the first time)

my advice is stay calm and wait for the experts to chime in if you're not sure what to do.

in my experience the disks and the data could be recovered.

[JorgeB]

There's filesystem corruption detected on disks 1 and 5, disk6 is complaining about a duplicate UUID which is strange, if you're not using ECC RAM first thing to do is to run memtest, then reboot and if disk6 is still unmountable post the output of

blkid

 

[KTgsxr]

Thank you Jorge

 

So last night I removed disk 1 and re-added it and it rebuilt from parity and appears ok... but disk6 is still showing unmountable but green..not sure if it should just do the same? I am running ECC so not sure what to do there.

 

We have been having some power glitches at our house that are being investigated randomly lights will dim can also hear box fans slow for a second (new place). I put a UPS on the server but not sure maybe these power issues are still getting to the server. I also was on the fence about ordering a few new drives so the 1tb disk5 will be the first to go. and will probably disk 1 or 6.

 

I have uploaded another diagnostic from this morning after the rebuild completed.

 

 

 

tower-diagnostics-20210621-0717.zip

[JorgeB]

You still need to run xfs_repair on disk5, also:

4 hours ago, JorgeB said:

then reboot and if disk6 is still unmountable post the output of

blkid

 

[KTgsxr]

5 minutes ago, JorgeB said:

You still need to run xfs_repair on disk5, also:

 

I ran xfs_repair on disk5 and rebooted. everything appears to be back up and running now...I still did the blkid and attached a new diagnostic. All drives are showing mounted now....

 

 

/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/sda1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="2732-64F5" BLOCK_SIZE="512" TYPE="vfat"
/dev/sdb1: UUID="1f322d5c-d0b2-4516-bdcf-d9b50cc0d3ed" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="9f6521b3-cb56-4241-a2fc-ed5263329587"
/dev/sdd1: UUID="f446b92d-cdda-4ff6-8526-8096e352e254" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="c7bab035-f047-48a3-9833-a151c6aba19f"
/dev/sde1: UUID="fcae4422-e265-4567-a8ba-3fa6251cb7bf" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="870ed786-f782-4f2f-ae9b-e3ce95644653"
/dev/sdi1: UUID="16faf95a-2795-46f2-bf38-9c438231209e" UUID_SUB="052c73e3-b2cd-4be4-9850-09cd8fe2e429" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdf1: UUID="a17183b1-8c25-48e0-885e-51a3f6231fca" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdh1: UUID="b583d3a9-92a2-42e4-97d4-797830338e95" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdg1: UUID="e72fe6ff-d3fc-4a87-8352-ec90c5264abe" BLOCK_SIZE="512" TYPE="xfs"
/dev/md1: UUID="a17183b1-8c25-48e0-885e-51a3f6231fca" BLOCK_SIZE="512" TYPE="xfs"
/dev/md2: UUID="1f322d5c-d0b2-4516-bdcf-d9b50cc0d3ed" BLOCK_SIZE="512" TYPE="xfs"
/dev/md3: UUID="f446b92d-cdda-4ff6-8526-8096e352e254" BLOCK_SIZE="512" TYPE="xfs"
/dev/md4: UUID="fcae4422-e265-4567-a8ba-3fa6251cb7bf" BLOCK_SIZE="512" TYPE="xfs"
/dev/md5: UUID="b583d3a9-92a2-42e4-97d4-797830338e95" BLOCK_SIZE="512" TYPE="xfs"
/dev/md6: UUID="e72fe6ff-d3fc-4a87-8352-ec90c5264abe" BLOCK_SIZE="512" TYPE="xfs"
/dev/loop2: UUID="e2531b82-dbb4-4047-96ba-80cdbd1c2a8b" UUID_SUB="f466264f-9dfd-4bf4-bd92-bb88414b4f28" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdc1: PARTUUID="adbad1f0-a333-47dd-87d1-20f59b504708"

 

and thank you again for your help

[JorgeB]

Strange about disk6, don't see how there could be a duplicate filesystem and now there isn't, but if all is working now...

[KTgsxr]

17 minutes ago, JorgeB said:

Strange about disk6, don't see how there could be a duplicate filesystem and now there isn't, but if all is working now...

well I have a few drives on the way will get these old drives replaced soon. We also have someone looking at the power issues so hopefully this is the end of it. Thank you again for your help

[JorgeB]

8 minutes ago, KTgsxr said:

well I have a few drives on the way will get these old drives replaced soon

That wasn't a disk problem, it was a filesystem problem.

[KTgsxr]

9 hours ago, JorgeB said:

That wasn't a disk problem, it was a filesystem problem.

ah ok. they were just my two oldest/smallest drives that I had planned to replace.

 

so everything has appeared to be ok today like my plex runs fine but when I try to browse to my media folder tower\media its blank. if I use the unraid gui it says "too many files" but my Plex is playing videos from that folder so something is still amiss.

 

I opened a command prompt and found this...

root@Tower:/mnt/user# cd Media
root@Tower:/mnt/user/Media# ls
/bin/ls: reading directory '.': Structure needs cleaning
root@Tower:/mnt/user/Media#

Edited June 22, 2021 by KTgsxr

[KTgsxr]

ran nfs_check on all my drives. it found a few issues. it appears things are ok. Gonna look around some more.

[JorgeB]

Would need current diags.

[KTgsxr]

I feel like things are fixed and there is no more corruption... I have attached my recent diagnostics in case there is something someone else can see other than me.

 

Power company came and picked up their line monitor yesterday for someone to review the logs. Hopefully we will get some answers. As now I'm thinking these power glitches might have caused this.

tower-diagnostics-20210622-1237.zip

[JorgeB]

Diags are just after rebooting so not much to see, post new ones if you get more issues.