blaine07 Posted August 14, 2020 Share Posted August 14, 2020 (edited) Good Morning, Hoping someone will be as kind to get me straightened out... This morning noticed on Zoneminder it wasn't working correctly. Stopped the container, restarted Zoneminder and it and got a "403 error" and wouldn't start. Noticed Log File on main screen was full. Stopped all containers and used "Reboot" on main screen in GUI. Server restarted, stuff came back on and was fine...for a minute...until it wasn't. I did research and concluded that it's like my docker.img file got corrupt when logfile got full. I followed the information on going through to delete docker.im and restart it. Then, I hit issues about unable to write to cache and Docker failing to start as a result. Tried leaving docker stopped and restarting, and despite that I "deleted docker.img" all my containers come back and auto start just like I had not deleted docker.img. My cache drive(s) do not ever appear to have gotten full. I admittedly don't know what I am looking for but skimming logs saw the BTFRS error stuff? Not sure what else to do but my diagnostics are attached below. I would be super grateful and thankful for anytime anyone could lend me a hand with! Thanks! tower-diagnostics-20200814-0843.zip Edited August 14, 2020 by blaine07 Quote Link to comment
JorgeB Posted August 14, 2020 Share Posted August 14, 2020 Cache filesystem is corrupt, best bet is to backup anything important and re-format the pool. Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 (edited) 24 minutes ago, johnnie.black said: Cache filesystem is corrupt, best bet is to backup anything important and re-format the pool. So Would I stand a chance in migrating app data off cache, formatting and moving back I suppose mate? Edited August 14, 2020 by blaine07 Quote Link to comment
JorgeB Posted August 14, 2020 Share Posted August 14, 2020 Yes, but you won't know until you try it, also some recovery options here if needed. Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 15 minutes ago, johnnie.black said: Yes, but you won't know until you try it, also some recovery options here if needed. I have got it mounted and got to this step; could you be as kind to help me with copying data off cache pool? I got here successfully: If it mounts copy all the data from /x to another destination, like an array disk, you can use Midnight Command (mc on the console/SSH) or your favorite tool, after all data is copied format the device or pool and restore data. These are commands I ran to get this far, no errors so I assume I am good this far: Linux 4.19.107-Unraid. root@Tower:~# mkdir /x root@Tower:~# mount -o usebackuproot,ro /dev/sde1 /x root@Tower:~# I just need help with copying data from /x to another destination, please? Quote Link to comment
JorgeB Posted August 14, 2020 Share Posted August 14, 2020 On the console type "mc" to open midnight commander, then navigate to /x on the left side and to where you want to copy on the right, then just copy from one side to the other. Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 On the console type "mc" to open midnight commander, then navigate to /x on the left side and to where you want to copy on the right, then just copy from one side to the other.Should I stop the array first? “Try” to stop docker or anything special before copying over mate?Thank you, so much. Gonna win this war lol Quote Link to comment
trurl Posted August 14, 2020 Share Posted August 14, 2020 7 minutes ago, blaine07 said: Should I stop the array first? no 7 minutes ago, blaine07 said: “Try” to stop docker wouldn't hurt 1 Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 wouldn't hurtMoving everything off now; thank the both of you! Will report back... I’m assuming once I get cache formatted I will try to move app data over but then I want to recreate docker.img and not move it? :-) Quote Link to comment
trurl Posted August 14, 2020 Share Posted August 14, 2020 9 minutes ago, blaine07 said: I want to recreate docker.img Yes, but why did you have 35G allocated to docker.img? Had you had problems filling it? 20G should be more than enough and if it grows then you have application misconfigured. Its contents will be downloaded again when you reinstall dockers. Be sure to use the Previous Apps feature on the Apps page. 1 Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 Yes, but why did you have 35G allocated to docker.img? Had you had problems filling it? 20G should be more than enough and if it grows then you have application misconfigured. Its contents will be downloaded again when you reinstall dockers. Be sure to use the Previous Apps feature on the Apps page.I don’t think anything currently was misconfigured but at one point something was writing logs to it and I admittedly must’ve changed docker size not knowing at time what other issues that causes... I was super newbs when I had changed it LOL. I believe docker had been staying at 57% full which I assume is probably fine? Quote Link to comment
trurl Posted August 14, 2020 Share Posted August 14, 2020 2 minutes ago, blaine07 said: docker had been staying at 57% full which I assume is probably fine? Yes I have 20G docker.img, only 44% used with 17 dockers running. 1 Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 Yes I have 20G docker.img, only 44% used with 17 dockers running.Do I move libvirt back over or will it rebuild too or? Quote Link to comment
JorgeB Posted August 14, 2020 Share Posted August 14, 2020 If you have VMs keep existing file, also good practice to have a backup of that file. Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 If you have VMs keep existing file, also good practice to have a backup of that file.Thank you! Yes had it set to backup on schedule. I mean good news is all I had was appdata and docker.img on cache. Got both copied. When I get a second gonna who’s, move app data back over and get docker.jmg recreated without a hitch... hopefully Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 Can’t get array to stop; do I need to use command line to unmount the X above or? Quote Link to comment
JorgeB Posted August 14, 2020 Share Posted August 14, 2020 Array should still stop, make sure mc is closed and close the ssh session, but you can unmount with umount /x Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 19 minutes ago, johnnie.black said: Array should still stop, make sure mc is closed and close the ssh session, but you can unmount with umount /x Got it; now the only way I could find to format cache drive(s) was to change the file type, save, start array, format, then stop array, change to BRFRS. At any rate, it doesn't appear that I can change cache drive file type to anything BUT btfrs? What am I missing here on formatting cache drives? Quote Link to comment
JorgeB Posted August 14, 2020 Share Posted August 14, 2020 Pools can only be btrfs, with the array stopped run: blkdiscard /dev/sdX Replace X by the correct letters for both devices and run it one at a time, this will do a full trim on the devices complete wiping them, then start the array and you'll have the option to format. Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 Pools can only be btrfs, with the array stopped run: blkdiscard /dev/sdX Replace X by the correct letters for both devices and run it one at a time, this will do a full trim on the devices complete wiping them, then start the array and you'll have the option to format.Got it! Moving app data over now Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 Okay, got all that stuff recreated and core containers restarted. Issue now is that I need my "proxynet" network back...Don't remember how to re-create that sigh 😞 Quote Link to comment
trurl Posted August 14, 2020 Share Posted August 14, 2020 1 hour ago, blaine07 said: Okay, got all that stuff recreated and core containers restarted. Issue now is that I need my "proxynet" network back...Don't remember how to re-create that sigh 😞 Yes, that is a side effect of deleting docker.img. From command line docker network create proxynet Quote Link to comment
blaine07 Posted August 14, 2020 Author Share Posted August 14, 2020 Alright guys, got everything going. Network deal gave me some qualms but finally got it figured out. I can’t figure out what I did or didn’t do/what is necessary to get my VMs to come back but 🤷🏼♂️ Thank you BOTH so so so much! I really appreciate it. Have learned a lot; thanks! You guys the real MVPs! 🎉😀 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.