Random Docker issues

[scythe944]

Hi all, I went to visit one of my docker app pages today and found several of them not working, even though they said they were, "started".

 

I figured I just needed a reboot of the server.  I rebooted and it took an extra long time to come back up and after logging in, I had a parity check running and the docker service wasn't started.

 

I see all my drives and my cache disk, nothing is full, all drives say that they're "healthy".  Could someone assist me with what might have happened?

 

I will post diagnostics.

tower900-diagnostics-20240807-1309.zip

[scythe944]

I suppose that I'm victim of a corruption bug as referenced in this reddit post. Just going to wait a day for the parity check to finish and go from there I suppose. Might be rebuilding my docker image I guess?

 

 

[JorgeB]

There are issues with the pool filesystem, it's going read-only, suggest backing it up and then re-formatting.

[scythe944]

12 minutes ago, JorgeB said:

There are issues with the pool filesystem, it's going read-only, suggest backing it up and then re-formatting.

Thanks, when you say "pool" you mean my cache, right?  I'll have to see what the best way is to back it up. Just connect via UNC path and copy it somewhere, then copy it back after a re-format? I haven't seen any documentation yet. I'll try to google

[JorgeB]

Yes, that works 

[scythe944]

I didn't have permission to copy a lot of the files and it was fairly large. My parity check is still running, waiting until it finishes before I screw anything else up.

 

I found a vid and was going to have my appdata / domains / etc moved from cache to the array, but for domains, the option is greyed out.  Once this parity check is done, I'll stop as many services as I can and try to get everything moved off cache and try to reformat my cache as ZFS I guess, then move it back. Don't have much experience with this though. I pretty much set it up and forget it.  Hopefully it works

 

 

[scythe944]

Never mind. Domains looked to be empty, not sure if it's supposed to contain data or not, but mine didn't. Attempting to move my Appdata and system shares to the array now, then will attempt to move them back when I'm done reformatting this disk. Hopefully I don't have to rebuild all my dockers and their settings. 😞

 

Crap, can't "move" anything, since it's read-only.  Would be nice if I could get krusader up to copy the files from the cache to the array.

 

Looks like I'm just rebuilding all my dockers at this point. ugh.

 

Edited August 8 by scythe944

[strike]

Just use the terminal and midnight commander (file manager in the terminal: Type mc in the terminal and hit enter. Then copy all your files to the array. All your shares and drives (if disk shares are enabled) will show up under /mnt/user/ or /mnt/diskX or the name of your pool 

 

Edit: You can also use the built-in file manager in the unraid webui

Edited August 8 by strike

[scythe944]

I guess my problem is that I'm trying to figure out why this happened to begin with. My drive says that it's fine, at least according to the SMART test. It's not that old, probably less than 2 years at this point, since my server isn't even that old.  Did the file system just corrupt the data or is the drive bad?  Should I try a different file system?  Not sure I want to re-format and set up all my stuff all over again just for it to get corrupted again.

[scythe944]

15 minutes ago, strike said:

Just use the terminal and midnight commander (file manager in the terminal: Type mc in the terminal and hit enter. Then copy all your files to the array. All your shares and drives (if disk shares are enabled) will show up under /mnt/user/ or /mnt/diskX or the name of your pool 

 

Edit: You can also use the built-in file manager in the unraid webui

Well, shoot, I might have to give that a go and see if it's any better than copying over to my Windows machine as a backup. I'm finding a lot of things that I don't have access to and not even sure if it's worth trying to copy what I can if I'm missing critical files that I don't have the ability to copy.  Thanks for the suggestion

[strike]

If you use MC you shouldn't have any issues copying the files. I didn't look at your diags but the drive is probably good or I think @JorgeB would have suggested replacing it. File system corruption can happen some times. And there can be several things that can cause it. One of them is bad ram. A power outage could also cause corruption, so if you don' have a UPS, get one. if you encounter this issue again I would suggest running memtest. But for now I would just get the data of it and re-format. 

[scythe944]

17 minutes ago, strike said:

If you use MC you shouldn't have any issues copying the files. I didn't look at your diags but the drive is probably good or I think @JorgeB would have suggested replacing it. File system corruption can happen some times. And there can be several things that can cause it. One of them is bad ram. A power outage could also cause corruption, so if you don' have a UPS, get one. if you encounter this issue again I would suggest running memtest. But for now I would just get the data of it and re-format. 

Sounds good. I do have a UPS and I have about an hour of runtime with it, so it wasn't a power outage.  I was using the server (watching a movie while I worked a couple days ago) and then I went to look something up from one of my dockers, it wasn't running.  Checked a few others and they weren't running either.  Anywho, I made a copy using MC, still copying to my windows machine and also did a backup using "appdata backup" that I forgot I even installed but never really configured. Hopefully one of the 3 backups will save my ass from having to reconfigure everything.