My Unraid server has been working fine for a while, but recently it seems to be having problems with the Docker Containers.  All of them show up with version 'not available' and as a result i can't update them.

 

I've attached the diagnostics simply because I have to assume it is networking in some way.  

 

Before today it was set up with Eth0 bonded to eth0/1/2/3.  I changed that to only bonding with eth0/1 and plugged in eth1.  I did that hoping it would resolve the issue.  previously none of 1/2/3 were plugged in.

 

Hopefully someone has some insights

tower-diagnostics-20200802-1129.zip

Looks like issues on the cache drive, so I'll let @johnnie.black advise.

appreciate any help I can get. Seems like a mess.  Some of the dockers appear not to work properly and also seems like networking making them not work.  Like Plex is essentially inaccessible.

Cache filesystem is severely corrupt, best bet it to backup, re-format and restore any important data there.

3 hours ago, johnnie.black said:

Cache filesystem is severely corrupt, best bet it to backup, re-format and restore any important data there.

What would be the method to do this? Looks like all the files are:

• Docker Files - in /appdata/{Container-Name}
• libvirt.img

 

You can first try using the command line, or your favorite copy tool like midnight commander or krusader, note that it's good practice making regular backups of those files, both the appdata folder and libvirt.img (if you have VMs).

12 minutes ago, johnnie.black said:

You can first try using the command line, or your favorite copy tool like midnight commander or krusader, note that it's good practice making regular backups of those files, both the appdata folder and libvirt.img (if you have VMs).

I have Krusader on there, but it is a docker and no longer working due to this corruption issue.    I will have to figure out command line I guess.

ok, so I ran:

 

cp -r /mnt/cache /mnt/user/Backup

 

Looks like every file got an error as shown in the attached image.  Unsure if this is because of permissions or a general read error on the cache drive?

 

Those errors are likely the result of data corruption, btrfs will give an i/o error if corruption is detected so that you're not fed bad data without knowing, would need diags to confirm, if it's that you can use btrfs restore to bypass it, but data will still be corrupt.

1 minute ago, johnnie.black said:

Those errors are likely the result of data corruption, btrfs will give an i/o error if corruption is detected so that you're not fed bad data without knowing, would need diags to confirm, if it's that you can use btrfs restore to bypass it, but data will still be corrupt.

 

I really don't think I need to worry about that data right?  I believe all that data is located at /mnt/user/appdata anyway.  I have no VMs, just Docker containers.

/mnt/cache/appdata is in /mnt/user/appdata

1 minute ago, johnnie.black said:

/mnt/cache/appdata is in /mnt/user/appdata

So you are saying if I want my dockers to retain their settings - I need to recover that.  Sound about right?

 

Sorry for all the questions - I'm not all that familiar with Unraid yet.  I've only had it running for 6 months and mostly left it alone after setup.

1 minute ago, Convington said:

So you are saying if I want my dockers to retain their settings - I need to recover that.  Sound about right?

Yes, unless you had duplicate appdata on more than one device, it would still all be under /mnt/user

Still struggling with this.  How does the cache get corrupted anyway?  Is this a disk issue and I should replace the disk?

 

 

4 minutes ago, Convington said:

How does the cache get corrupted anyway?

Most times result of a bad shutdown or bad hardware.

ok, makes sense.

on the BTRFS, I assume I have to shut stuff down before trying any of that?  or at least unmount the cache drive?  Trying to figure out how to do that and I am not succeeding.  Sorry - I feel like such a newbie with Unraid.

ok, I did the following and I don't think the copy worked very well:

 

 

Like mentioned best bet is the btrfs restore option (2nd one).

I was going to go there next, just checking to make sure I'm on the right track here.  

It is in the process... Man does Plex have a lot of files in there   I would have thought most would be in the array, something I will have to look at after I finally get this back up and running

 

After using that process to copy the files over - I have a lot of these errors.  What is the next step in the process?

7 minutes ago, Convington said:

What is the next step in the process?

Nothing as long as the data fits on disk1, if it doesn't you'll need to use another disk with more space.

 

Also good idea to cancel/pause the parity check for nw.

15 minutes ago, johnnie.black said:

Nothing as long as the data fits on disk1, if it doesn't you'll need to use another disk with more space.

 

Also good idea to cancel/pause the parity check for nw.

 

Don't I have to do something with the cache drive?  Like, format, replace, something?  I'm not real sure where to do that since the cache drive is live.  

I do have a second drive, but not sure what I do first.. I would think I need the following things to happen:

 

• Copy that data somehow from the backup to the new cache drive
• start up the array after this is done

Those are guesses, but I have no idea lol

1 hour ago, Convington said:

Man does Plex have a lot of files in there   I would have thought most would be in the array, something I will have to look at after I finally get this back up and running

Keeping appdata (and system) on cache is the usual way. If you have those on the array dockers will have performance impacted by the slower parity writes, and will also keep array disks spinning since those files will be open.