Docker will not start

[ninjaneer68]

I have already deleted the docker.img file and started over and about 2 hours into setting up containers, I started having issues and when I tried to stop docker, I couldn't start it up again. Not sure what is going on. 

tower-diagnostics-20231216-2129.zip

[trurl]

Corrupt cache pool. Have you done memtest?

 

Usual advice for btrfs would be backup and reformat.

 

Why do you think you want 60G docker.img?

 

Your appdata and domains shares have files on the array. Currently VMs are disabled so maybe you don't need those domains files.

 

Ideally, appdata, domains, system shares would have all files on fast pool such as cache with nothing on the array for Docker/VM performance, and so array disks can spin down since these files are always open.

 

But you can't do anything with these for now until cache is working again.

 

Disable Docker (VM Manager already disabled) and leave these disabled until your cache is fixed, then get those shares moved back to cache.

 

Mover may not be able to help get files off the corrupt cache since it may not be writable, so you will probably have to copy the cached shares to the array then reformat cache.

 

Install Dynamix File Manager plugin to help work with files directly on the server.

 

These shares all have files on cache:

appdata                           shareUseCache="no"      # Share exists on disk4, disk5, cache
d--a                              shareUseCache="no"      # Share exists on disk1, disk2, disk3, disk4, disk5, cache
P--------e                        shareUseCache="only"    # Share exists on cache
system                            shareUseCache="no"      # Share exists on cache

 

[ninjaneer68]

My cache drive are two pooled m.2 drives. 

 

I recently bought bigger m.2 to replace the current ones. I will work on getting the files off and replace the drives.

 

I moved everything off of cache cause I was running my Plex and sabnzd in cache. 

 

14 minutes ago, trurl said:

Ideally, appdata, domains, system shares would have all files on fast pool such as cache with nothing on the array for Docker/VM performance, and so array disks can spin down since these files are always open

 

Do I went the option that moves things in and out of cache ? 

 

What are domains ? 

 

I am assuming miss using the cache like you mentioned I am writing to my spinning disk to much ? 

[ninjaneer68]

16 minutes ago, trurl said:

60G docker.img

I did it in the past when I couldn't figure out why my image kept getting full, just never shrank it back down. 

[ninjaneer68]

18 minutes ago, trurl said:

Have you done memtest?

Not yet didn't realize bad cache till now. 

 

Would love to know the best way to search through this diag files to find the issues like you did. 

Edited December 17, 2023 by sstretchh

[trurl]

20 minutes ago, sstretchh said:

Would love to know the best way to search through this diag files to find the issues like you did. 

Cache corruption is apparent in your syslog entries. I encourage everyone to look at their diagnostics. Some of it is webUI settings, some of it is the output of simple commands...

 

23 minutes ago, sstretchh said:

Do I went the option that moves things in and out of cache ?

Nothing moves thing in and out of cache. There are settings for each user share that can move from cache to array, or move from array to cache, but it never goes both ways. And, as mentioned

39 minutes ago, trurl said:

Mover may not be able to help get files off the corrupt cache since it may not be writable, so you will probably have to copy the cached shares to the array then reformat cache

 

23 minutes ago, sstretchh said:

What are domains ?

domains is the default user share for storing VM vdisks.

 

 

23 minutes ago, sstretchh said:

using the cache like you mentioned I am writing to my spinning disk to much ?

Not entirely sure I understand your question, but maybe read what I said again here:

39 minutes ago, trurl said:

Ideally, appdata, domains, system shares would have all files on fast pool such as cache with nothing on the array for Docker/VM performance, and so array disks can spin down since these files are always open.

 

23 minutes ago, sstretchh said:

I moved everything off of cache cause I was running my Plex and sabnzd in cache. 

All of your dockers should be on cache as mentioned, and it looks like you have plenty of space for that.

 

Here is something from the documentation about the "default shares" which might help explain what I said above about appdata, domains, system:

https://docs.unraid.net/unraid-os/manual/shares/user-shares/#default-shares

 

[trurl]

Wouldn't hurt to run memtest before anything else just to eliminate that. You shouldn't even attempt to run any computer unless memory is working perfectly. Everything goes through RAM, the OS and other executable code, your data, everything. The CPU can't do anything with anything until it is loaded into RAM.

 

Then

49 minutes ago, trurl said:

Disable Docker (VM Manager already disabled) and leave these disabled until your cache is fixed

 

[trurl]

I'll check back tomorrow. We can work through all this in stages with more diagnostics along the way to see how things are progressing.

[trurl]

Any progress?

[ninjaneer68]

2 hours ago, trurl said:

Any progress?

I haven't had a chance to work on it yet, I might get to it today or later this week, depending on the WAF .. LOL 

[ninjaneer68]

Here is the only thing I have on the cache. I believe the only thing I need to move is the plexmediaserver folder under appdata, the rest doesn't need moved ? ( maybe stuff under system ) 

Plex-cache/  
  - plex/  temp/
appdata/  

  - PlexMediaServer/
data/  
  - usenet/ ( Folder is empty) 
system/
  - docker/  libvirt/

So far I have moved appdata folder and system folder off the cache. But to memtest the RAM, and going to replace the m.2 drives

Edited December 17, 2023 by ninjaneer68

[ninjaneer68]

15 hours ago, trurl said:

Corrupt cache pool

Where did you see corrupt cache pool at ? 

[trurl]

15 hours ago, ninjaneer68 said:

will work on getting the files off and replace the drives

Nothing indicates the drives need to be replaced. You just have a corrupt filesystem on the pool.

 

If you're going to reformat cache to fix the corrupt filesystem, you need to get everything off it you want to keep.

 

1 hour ago, ninjaneer68 said:

Where did you see corrupt cache pool at ? 

In your diagnostics, logs/syslog.txt, starting with these entries:

Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 4096
### [PREVIOUS LINE REPEATED 47 TIMES] ###
Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 16384
Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 4096
Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 16384

Also, just after those entries (and also somewhat before) you have dumps related to macvlan. These have been known to cause crashes. The solution is to change Docker from macvlan to ipvlan. But we can deal with that later since you should

16 hours ago, trurl said:

Disable Docker (VM Manager already disabled) and leave these disabled until your cache is fixed

 

[ninjaneer68]

I just replaced the cache drive with a 2 TB SSD. 

 

a long time ago, I don' know know, I think i was struggling with docker, I have been setting each container to a static IP from my DHCP pool and instead of using the internal  pool, but I am open to switching it back

[ninjaneer68]

18 hours ago, trurl said:

Why do you think you want 60G docker.img?

Is this bad ? 

I am moving things over to the new cache drive now, about to turn everything back on.
should I switch to ipvlan, and move all the containers over to the docker DHCP pool ?

[trurl]

Post new diagnostics

[ninjaneer68]

I have everything up and running except Traefik

[ninjaneer68]

tower-diagnostics-20231217-1851.zip
Updated Diag

[ninjaneer68]

how can i see if the cache drive went to read only mode again ? I have temp download folders located on cache, and my containers are telling me the drive is full that they can't write to the locations that I mapped to the container

[trurl]

Your appdata share still has files on the array.

 

2 hours ago, ninjaneer68 said:

everything up and running except Traefik

As the eror message says, host port 8080 already being used.

 

1 hour ago, ninjaneer68 said:

containers are telling me the drive is full

What host path is specified in the mapping?

[ninjaneer68]

here is how its mapped

That usenet folder, is where the completed and incomplete folders are.

You can see from the container terminal window, its mapped, but when I try and make a test directory, it says no space left. 

It should be able to write to the location

[ninjaneer68]

On 12/17/2023 at 9:28 PM, trurl said:

As the eror message says, host port 8080 already being used.

I just finished setting up CF Tunnel for ZT, and decommissioned traefik

[ninjaneer68]

if I change the path from user to cahce, it works. why is that ? Does it have anything to do with don't split he directory settings ?

[trurl]

The screenshot with the command lines looks like you have created folders in /, the root of the OS, which is all in RAM. You shouldn't be working there at alll.

 

Attach new Diagnostics to your NEXT post in this thread.

[ninjaneer68]

Here is the new Diag, I am getting errors that /config is not writable. Am I working out of the cache incorrectly ? 

tower-diagnostics-20231223-1723.zip