I recently had a series of electricity outages and it seems to have taken a toll on my array. My parity drive had read errors, but all my other drives seemed to be fine. So I shut down the array, tried to rebuild it (which started a parity sync) but now all of a sudden, my Disk 2 in the array has read errors, and shows up as "Unmoumtable filesystem: wrong or no filesystem" in the array tab, even though it clearly says "xfs" to the left.

 

Worse yet: literally all my docker apps disappeared. I don't understand why, aren't docker containers stored on the cache drive? My appdata folder is still accessible, yet my apps randomly got deleted. Are they gone forever? I'm pretty sure I backed my server up a few months back but not sure how the newer system works.

 

I attached the diagnostics .zip. Also on my server's log I see these two errors:

Aug 21 23:55:44 Tower kernel: ata9.00: exception Emask 0x10 SAct 0x1c0 SErr 0x90202 action 0xe frozen

Aug 22 00:03:13 Tower root: Fix Common Problems: Error: disk2 (WDC_WD20EFRX-68EUZN0_WD-WCC4M2ZHRNXF) has read errors

 

tower-diagnostics-20220822-0024.zip

Edited August 21, 20223 yr by Stubbs

I'm rebuilding parity again, but my apps are still gone. When I try to install a new container, its inaccessible from the WebUI.

 

I'm losing my mind over this. I just don't understand wtf is going on.

21 minutes ago, Stubbs said:

rebuilding parity again

 

Was your parity disabled? Was it disabled when you did the first rebuild? Or were you just rebuilding it because you thought it might help something?

 

Those diagnostics show connection problems with disks 2,3 so parity rebuild may not be going well. After you get those connection problems fixed maybe disk2 will be mountable. If not you will have to try to repair its filesystem.

 

3 hours ago, Stubbs said:

electricity outages

Get an UPS

 

3 hours ago, Stubbs said:

aren't docker containers stored on the cache drive

Depends on how you have things setup. With your current setup, your docker.img should be on cache in your system share, but it might be a new docker.img. If you enabled docker before you installed cache, then your original docker.img would have gotten created on the array.
 

 

7 minutes ago, trurl said:

 

Was your parity disabled? Was it disabled when you did the first rebuild? Or were you just rebuilding it because you thought it might help something?

 

Those diagnostics show connection problems with disks 2,3 so parity rebuild may not be going well. After you get those connection problems fixed maybe disk2 will be mountable. If not you will have to try to repair its filesystem.

 

Get an UPS

 

Depends on how you have things setup. With your current setup, your docker.img should be on cache in your system share, but it might be a new docker.img. If you enabled docker before you installed cache, then your original docker.img would have gotten created on the array.
 

 

It said something like parity was invalid. The parity disk wasn't coming back on even after a server restart, and the only option seemed to be a parity sync, which lead to disk 2 showing an error. At first it showed all disks were fine but then disk 2 started returning an error. 

 

I'm doing a parity sync right now. I haven't noticed any data missing in my network share so I'm hoping it will all work out, except my apps seem to be gone for good.

 

I reinstalled jellyfin, and when opening the webUI, it looked exactly like it was before, so I assume its retaining everything from the appdata directory. My docker is definitely on my cache drive too, so I don't know how it got wiped simply because the parity disk stopped working.

As seen here, its rebuilding with no disk 2 or 3 errors.

 

If I hover over the parity exclamation point widget, it says "parity is invalid", but I assume that will go away once it rebuilds.

1 minute ago, Stubbs said:

docker is definitely on my cache drive

I know it is now. I was just wondering if it might have been created on unmountable disk2 at some time in the past.

 

2 minutes ago, Stubbs said:

said something like parity was invalid

It is invalid now since you are rebuilding it.

 

A disabled disk would have a red X beside it, and you must rebuild a disabled disk.

 

An invalid disk would have a yellow triangle beside it. A rebuilding disk is invalid, but you can make it rebuild a disk even if it was never disabled.

 

So that is the reason I asked

23 minutes ago, trurl said:

were you just rebuilding it because you thought it might help something?

 

Please do screenshots instead of photos. Can't see whether or not disk2 is unmountable, since you didn't get that part in your photo, but diagnostics showed it not mounted.

1 minute ago, trurl said:

I know it is now. I was just wondering if it might have been created on unmountable disk2 at some time in the past.

 

It is invalid now since you are rebuilding it.

 

A disabled disk would have a red X beside it, and you must rebuild a disabled disk.

 

An invalid disk would have a yellow triangle beside it. A rebuilding disk is invalid, but you can make it rebuild a disk even if it was never disabled.

 

So that is the reason I asked

 

Please do screenshots instead of photos. Can't see whether or not disk2 is unmountable, since you didn't get that part in your photo, but diagnostics showed it not mounted.

Sorry I'm kind of stuck with photos right now because my internet isn't working (currently posting from mobile data).

 

Here is my current diagnostics though.

tower-diagnostics-20220822-0512.zip

4 minutes ago, trurl said:

I know it is now. I was just wondering if it might have been created on unmountable disk2 at some time in the past.

 

It is invalid now since you are rebuilding it.

 

A disabled disk would have a red X beside it, and you must rebuild a disabled disk.

 

An invalid disk would have a yellow triangle beside it. A rebuilding disk is invalid, but you can make it rebuild a disk even if it was never disabled.

 

So that is the reason I asked

 

Please do screenshots instead of photos. Can't see whether or not disk2 is unmountable, since you didn't get that part in your photo, but diagnostics showed it not mounted.

And sorry, I can't really remember exactly what it said when the parity disk was not functioning, but I was getting a barrage of errors/warnings like this in my server log:

 

Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 349

Warning: file_put_contents(/var/lib/docker/unraid/images/sonarr-icon.png): failed to open stream: Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 93

Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 349

Warning: file_put_contents(/var/lib/docker/unraid/images/swag-icon.png): failed to open stream: Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 93

Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 349

Warning: file_put_contents(/var/lib/docker/unraid/images/tautulli-icon.png): failed to open stream: Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 93

Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 349

Warning: file_put_contents(/var/lib/docker/unraid/images/wikijs-icon.png): failed to open stream: Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 93

 

4 hours ago, trurl said:

I was just wondering if it might have been created on unmountable disk2 at some time in the past.

These latest diagnostics shows disk2 mounted, and system share with files on disk2.

 

What do you get from the command line with this?

ls -lah /mnt/cache/system

and this?

ls -lah /mnt/disk2/system

 

On 8/21/2022 at 11:43 PM, trurl said:

These latest diagnostics shows disk2 mounted, and system share with files on disk2.

 

What do you get from the command line with this?

ls -lah /mnt/cache/system

and this?

ls -lah /mnt/disk2/system

 

 

root@UnraidServer:~# ls -lah /mnt/cache/system
total 16K
drwxrwxrwx 1 nobody users  26 Dec 21  2019 ./
drwxrwxrwx 1 nobody users 102 Aug 31 14:40 ../
drwxrwxrwx 1 nobody users  20 Aug 21 23:53 docker/
drwxrwxrwx 1 nobody users  22 Dec 21  2019 libvirt/
root@UnraidServer:~# ls -lah /mnt/disk2/system
total 0
drwxrwxrwx  3 nobody users  20 Nov 23  2019 ./
drwxrwxrwx 10 nobody users 137 Aug 31 14:40 ../
drwxrwxrwx  2 nobody users  24 Nov 23  2019 docker/
root@UnraidServer:~# 

 

Edited August 31, 20223 yr by Stubbs

2 weeks later...

 

Did you get parity resynced?

 

Attach new diagnostics to your NEXT post in this thread.

5 hours ago, trurl said:

2 weeks later...

 

Did you get parity resynced?

 

Attach new diagnostics to your NEXT post in this thread.

Yeah sorry about that. Parity did sync and I sort of forgot about this thread. I came back and replied because I was also asking about a VM-related issue on the VM forum.

 

I did have a "disk2 has read errors error" yesterday, although I haven't noticed anything wrong besides that. They all look fine on the array menu.

tower-diagnostics-20220831-2114.zip

 What do you get from the command line with this?

ls -lah /mnt/cache/system/docker

and this?

ls -lah /mnt/disk2/system/docker

 

10 hours ago, trurl said:

 What do you get from the command line with this?

ls -lah /mnt/cache/system/docker

and this?

ls -lah /mnt/disk2/system/docker

 

root@Tower:~# ls -lah /mnt/cache/system/docker
total 25G
drwxrwxrwx 1 nobody users  20 Aug 21 23:53 ./
drwxrwxrwx 1 nobody users  26 Dec 21  2019 ../
-rw-rw-rw- 1 nobody users 25G Sep  1 22:40 docker.img
root@Tower:~# ls -lah /mnt/disk2/system/docker
total 25G
drwxrwxrwx 2 nobody users  24 Nov 23  2019 ./
drwxrwxrwx 3 nobody users  20 Nov 23  2019 ../
-rw-rw-rw- 1 nobody users 25G Aug 21 23:53 docker.img
root@Tower:~# 

 

I guess I had a docker image on disk 2 after all.

Not that it's of big consequence, after all the persistent data is fine.

Looks like the one on cache is current.

 

Since you're on 6.10 you should install Dynamix File Manager plugin and use it to delete that system folder from disk2