ninjaneer68 Posted December 17, 2023 Share Posted December 17, 2023 I have already deleted the docker.img file and started over and about 2 hours into setting up containers, I started having issues and when I tried to stop docker, I couldn't start it up again. Not sure what is going on. tower-diagnostics-20231216-2129.zip Quote Link to comment
trurl Posted December 17, 2023 Share Posted December 17, 2023 Corrupt cache pool. Have you done memtest? Usual advice for btrfs would be backup and reformat. Why do you think you want 60G docker.img? Your appdata and domains shares have files on the array. Currently VMs are disabled so maybe you don't need those domains files. Ideally, appdata, domains, system shares would have all files on fast pool such as cache with nothing on the array for Docker/VM performance, and so array disks can spin down since these files are always open. But you can't do anything with these for now until cache is working again. Disable Docker (VM Manager already disabled) and leave these disabled until your cache is fixed, then get those shares moved back to cache. Mover may not be able to help get files off the corrupt cache since it may not be writable, so you will probably have to copy the cached shares to the array then reformat cache. Install Dynamix File Manager plugin to help work with files directly on the server. These shares all have files on cache: appdata shareUseCache="no" # Share exists on disk4, disk5, cache d--a shareUseCache="no" # Share exists on disk1, disk2, disk3, disk4, disk5, cache P--------e shareUseCache="only" # Share exists on cache system shareUseCache="no" # Share exists on cache Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 My cache drive are two pooled m.2 drives. I recently bought bigger m.2 to replace the current ones. I will work on getting the files off and replace the drives. I moved everything off of cache cause I was running my Plex and sabnzd in cache. 14 minutes ago, trurl said: Ideally, appdata, domains, system shares would have all files on fast pool such as cache with nothing on the array for Docker/VM performance, and so array disks can spin down since these files are always open Do I went the option that moves things in and out of cache ? What are domains ? I am assuming miss using the cache like you mentioned I am writing to my spinning disk to much ? Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 16 minutes ago, trurl said: 60G docker.img I did it in the past when I couldn't figure out why my image kept getting full, just never shrank it back down. Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 (edited) 18 minutes ago, trurl said: Have you done memtest? Not yet didn't realize bad cache till now. Would love to know the best way to search through this diag files to find the issues like you did. Edited December 17, 2023 by sstretchh Quote Link to comment
trurl Posted December 17, 2023 Share Posted December 17, 2023 20 minutes ago, sstretchh said: Would love to know the best way to search through this diag files to find the issues like you did. Cache corruption is apparent in your syslog entries. I encourage everyone to look at their diagnostics. Some of it is webUI settings, some of it is the output of simple commands... 23 minutes ago, sstretchh said: Do I went the option that moves things in and out of cache ? Nothing moves thing in and out of cache. There are settings for each user share that can move from cache to array, or move from array to cache, but it never goes both ways. And, as mentioned 39 minutes ago, trurl said: Mover may not be able to help get files off the corrupt cache since it may not be writable, so you will probably have to copy the cached shares to the array then reformat cache 23 minutes ago, sstretchh said: What are domains ? domains is the default user share for storing VM vdisks. 23 minutes ago, sstretchh said: using the cache like you mentioned I am writing to my spinning disk to much ? Not entirely sure I understand your question, but maybe read what I said again here: 39 minutes ago, trurl said: Ideally, appdata, domains, system shares would have all files on fast pool such as cache with nothing on the array for Docker/VM performance, and so array disks can spin down since these files are always open. 23 minutes ago, sstretchh said: I moved everything off of cache cause I was running my Plex and sabnzd in cache. All of your dockers should be on cache as mentioned, and it looks like you have plenty of space for that. Here is something from the documentation about the "default shares" which might help explain what I said above about appdata, domains, system: https://docs.unraid.net/unraid-os/manual/shares/user-shares/#default-shares Quote Link to comment
trurl Posted December 17, 2023 Share Posted December 17, 2023 Wouldn't hurt to run memtest before anything else just to eliminate that. You shouldn't even attempt to run any computer unless memory is working perfectly. Everything goes through RAM, the OS and other executable code, your data, everything. The CPU can't do anything with anything until it is loaded into RAM. Then 49 minutes ago, trurl said: Disable Docker (VM Manager already disabled) and leave these disabled until your cache is fixed Quote Link to comment
trurl Posted December 17, 2023 Share Posted December 17, 2023 I'll check back tomorrow. We can work through all this in stages with more diagnostics along the way to see how things are progressing. Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 2 hours ago, trurl said: Any progress? I haven't had a chance to work on it yet, I might get to it today or later this week, depending on the WAF .. LOL Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 (edited) Here is the only thing I have on the cache. I believe the only thing I need to move is the plexmediaserver folder under appdata, the rest doesn't need moved ? ( maybe stuff under system ) Plex-cache/ - plex/ temp/ appdata/ - PlexMediaServer/ data/ - usenet/ ( Folder is empty) system/ - docker/ libvirt/ So far I have moved appdata folder and system folder off the cache. But to memtest the RAM, and going to replace the m.2 drives Edited December 17, 2023 by ninjaneer68 Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 15 hours ago, trurl said: Corrupt cache pool Where did you see corrupt cache pool at ? Quote Link to comment
Solution trurl Posted December 17, 2023 Solution Share Posted December 17, 2023 15 hours ago, ninjaneer68 said: will work on getting the files off and replace the drives Nothing indicates the drives need to be replaced. You just have a corrupt filesystem on the pool. If you're going to reformat cache to fix the corrupt filesystem, you need to get everything off it you want to keep. 1 hour ago, ninjaneer68 said: Where did you see corrupt cache pool at ? In your diagnostics, logs/syslog.txt, starting with these entries: Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 4096 ### [PREVIOUS LINE REPEATED 47 TIMES] ### Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 16384 Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 4096 Dec 16 21:17:00 Tower kernel: BTRFS critical (device sdb1): unable to find logical 9836280312331550720 length 16384 Also, just after those entries (and also somewhat before) you have dumps related to macvlan. These have been known to cause crashes. The solution is to change Docker from macvlan to ipvlan. But we can deal with that later since you should 16 hours ago, trurl said: Disable Docker (VM Manager already disabled) and leave these disabled until your cache is fixed Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 I just replaced the cache drive with a 2 TB SSD. a long time ago, I don' know know, I think i was struggling with docker, I have been setting each container to a static IP from my DHCP pool and instead of using the internal pool, but I am open to switching it back Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 18 hours ago, trurl said: Why do you think you want 60G docker.img? Is this bad ? I am moving things over to the new cache drive now, about to turn everything back on. should I switch to ipvlan, and move all the containers over to the docker DHCP pool ? Quote Link to comment
trurl Posted December 17, 2023 Share Posted December 17, 2023 Post new diagnostics Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 I have everything up and running except Traefik Quote Link to comment
ninjaneer68 Posted December 17, 2023 Author Share Posted December 17, 2023 tower-diagnostics-20231217-1851.zip Updated Diag Quote Link to comment
ninjaneer68 Posted December 18, 2023 Author Share Posted December 18, 2023 how can i see if the cache drive went to read only mode again ? I have temp download folders located on cache, and my containers are telling me the drive is full that they can't write to the locations that I mapped to the container Quote Link to comment
trurl Posted December 18, 2023 Share Posted December 18, 2023 Your appdata share still has files on the array. 2 hours ago, ninjaneer68 said: everything up and running except Traefik As the eror message says, host port 8080 already being used. 1 hour ago, ninjaneer68 said: containers are telling me the drive is full What host path is specified in the mapping? Quote Link to comment
ninjaneer68 Posted December 19, 2023 Author Share Posted December 19, 2023 here is how its mapped That usenet folder, is where the completed and incomplete folders are. You can see from the container terminal window, its mapped, but when I try and make a test directory, it says no space left. It should be able to write to the location Quote Link to comment
ninjaneer68 Posted December 19, 2023 Author Share Posted December 19, 2023 On 12/17/2023 at 9:28 PM, trurl said: As the eror message says, host port 8080 already being used. I just finished setting up CF Tunnel for ZT, and decommissioned traefik Quote Link to comment
ninjaneer68 Posted December 19, 2023 Author Share Posted December 19, 2023 if I change the path from user to cahce, it works. why is that ? Does it have anything to do with don't split he directory settings ? Quote Link to comment
trurl Posted December 19, 2023 Share Posted December 19, 2023 The screenshot with the command lines looks like you have created folders in /, the root of the OS, which is all in RAM. You shouldn't be working there at alll. Attach new Diagnostics to your NEXT post in this thread. Quote Link to comment
ninjaneer68 Posted December 23, 2023 Author Share Posted December 23, 2023 Here is the new Diag, I am getting errors that /config is not writable. Am I working out of the cache incorrectly ? tower-diagnostics-20231223-1723.zip Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.