Good Morning,

 

Hoping someone will be as kind to get me straightened out...

 

This morning noticed on Zoneminder it wasn't working correctly. Stopped the container, restarted Zoneminder and it and got a "403 error" and wouldn't start. Noticed Log File on main screen was full. Stopped all containers and used "Reboot" on main screen in GUI. Server restarted, stuff came back on and was fine...for a minute...until it wasn't. I did research and concluded that it's like my docker.img file got corrupt when logfile got full. I followed the information on going through to delete docker.im and restart it. Then, I hit issues about unable to write to cache and Docker failing to start as a result. Tried leaving docker stopped and restarting, and despite that I "deleted docker.img" all my containers come back and auto start just like I had not deleted docker.img. My cache drive(s) do not ever appear to have gotten full. I admittedly don't know what I am looking for but skimming logs saw the BTFRS error stuff?

 

Not sure what else to do but my diagnostics are attached below. I would be super grateful and thankful for anytime anyone could lend me a hand with!

 

Thanks!

tower-diagnostics-20200814-0843.zip

Edited August 14, 20205 yr by blaine07

Cache filesystem is corrupt, best bet is to backup anything important and re-format the pool.

24 minutes ago, johnnie.black said:

Cache filesystem is corrupt, best bet is to backup anything important and re-format the pool.

So Would I stand a chance in migrating app data off cache, formatting and moving back I suppose mate?

Edited August 14, 20205 yr by blaine07

Yes, but you won't know until you try it, also some recovery options here if needed.

15 minutes ago, johnnie.black said:

Yes, but you won't know until you try it, also some recovery options here if needed.

I have got it mounted and got to this step; could you be as kind to help me with copying data off cache pool?

 

I got here successfully: If it mounts copy all the data from /x to another destination, like an array disk, you can use Midnight Command (mc on the console/SSH) or your favorite tool, after all data is copied format the device or pool and restore data.

 

These are commands I ran to get this far, no errors so I assume I am good this far:

Linux 4.19.107-Unraid.
root@Tower:~# mkdir /x
root@Tower:~# mount -o usebackuproot,ro /dev/sde1 /x
root@Tower:~# 

 

I just need help with copying data from /x to another destination, please?

 

 

On the console type "mc" to open midnight commander, then navigate to /x on the left side and to where you want to copy on the right, then just copy from one side to the other.

On the console type "mc" to open midnight commander, then navigate to /x on the left side and to where you want to copy on the right, then just copy from one side to the other.

Should I stop the array first? “Try” to stop docker or anything special before copying over mate?

Thank you, so much. Gonna win this war lol [emoji3]

7 minutes ago, blaine07 said:

Should I stop the array first?

no

 

7 minutes ago, blaine07 said:

“Try” to stop docker

wouldn't hurt

wouldn't hurt

Moving everything off now; thank the both of you! Will report back...

I’m assuming once I get cache formatted I will try to move app data over but then I want to recreate docker.img and not move it? :-)

9 minutes ago, blaine07 said:

I want to recreate docker.img

Yes, but why did you have 35G allocated to docker.img? Had you had problems filling it? 20G should be more than enough and if it grows then you have application misconfigured.

 

Its contents will be downloaded again when you reinstall dockers. Be sure to use the Previous Apps feature on the Apps page.

Yes, but why did you have 35G allocated to docker.img? Had you had problems filling it? 20G should be more than enough and if it grows then you have application misconfigured.
 
Its contents will be downloaded again when you reinstall dockers. Be sure to use the Previous Apps feature on the Apps page.

I don’t think anything currently was misconfigured but at one point something was writing logs to it and I admittedly must’ve changed docker size not knowing at time what other issues that causes... I was super newbs when I had changed it LOL. I believe docker had been staying at 57% full which I assume is probably fine?

2 minutes ago, blaine07 said:

docker had been staying at 57% full which I assume is probably fine?

Yes

 

I have 20G docker.img, only 44% used with 17 dockers running.

Yes
 
I have 20G docker.img, only 44% used with 17 dockers running.

Do I move libvirt back over or will it rebuild too or?

If you have VMs keep existing file, also good practice to have a backup of that file.

If you have VMs keep existing file, also good practice to have a backup of that file.

Thank you!

 

Yes had it set to backup on schedule.

 

I mean good news is all I had was appdata and docker.img on cache. Got both copied. When I get a second gonna who’s, move app data back over and get docker.jmg recreated without a hitch... hopefully [emoji1695]

Can’t get array to stop; do I need to use command line to unmount the X above or?

Array should still stop, make sure mc is closed and close the ssh session, but you can unmount with

umount /x

 

19 minutes ago, johnnie.black said:

Array should still stop, make sure mc is closed and close the ssh session, but you can unmount with

umount /x

 

Got it; now the only way I could find to format cache drive(s) was to change the file type, save, start array, format, then stop array, change to BRFRS. At any rate, it doesn't appear that I can change cache drive file type to anything BUT btfrs? What am I missing here on formatting cache drives?

Pools can only be btrfs, with the array stopped run:

 

blkdiscard /dev/sdX

 

Replace X by the correct letters for both devices and run it one at a time, this will do a full trim on the devices complete wiping them, then start the array and you'll have the option to format.

Pools can only be btrfs, with the array stopped run:
 

blkdiscard /dev/sdX

 
Replace X by the correct letters for both devices and run it one at a time, this will do a full trim on the devices complete wiping them, then start the array and you'll have the option to format.

Got it!

Moving app data over now [emoji3] [emoji1695]

Okay, got all that stuff recreated and core containers restarted. Issue now is that I need my "proxynet" network back...Don't remember how to re-create that sigh 😞

1 hour ago, blaine07 said:

Okay, got all that stuff recreated and core containers restarted. Issue now is that I need my "proxynet" network back...Don't remember how to re-create that sigh 😞

Yes, that is a side effect of deleting docker.img. From command line

 

docker network create proxynet

 

Alright guys, got everything going. Network deal gave me some qualms but finally got it figured out. I can’t figure out what I did or didn’t do/what is necessary to get my VMs to come back but 🤷🏼‍♂️
 

Thank you BOTH so so so much! I really appreciate it. Have learned a lot; thanks! You guys the real MVPs! 🎉😀