manolodf Posted June 10, 2019 Share Posted June 10, 2019 Hi All, I noticed a Docker was not working, so I tried to restart it, and it gave a 403 error. I thought that was strange enough, so I went to Dashboard to find the Log at 100%. I can't seem to get the docker to restart, and was going to reboot the whole system but I pulled the diagnostics first. Is it something where I need to increase the log size on a specific docker parameter? Or is this a system wide thing? That is what caught my eye. Here are Diagnostics. tower-diagnostics-20190610-1628.zip Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 So I rebooted and the Log is back under control, a lot of my start-up dockers did not start up and trying to start those manually dockers gives me the 403 error, ironically the one that gave me the error before did start up on boot. Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 The syslog is reporting that you have file system corruption on disk5. You want to start the array in Maintenance mode and run a file system check on that drive, Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 6 minutes ago, itimpi said: The syslog is reporting that you have file system corruption on disk5. You want to start the array in Maintenance mode and run a file system check on that drive, Ok running that now, so this has nothing to do with cache being btrfs instead of xfs? I read about doing the balance on cache: btrfs balance start -dusage=75 /mnt/cache It just came out of nowhere because it was relatively stable past 13 days. I had some issues few weeks ago but then it stabilized. Quote Link to comment
Squid Posted June 10, 2019 Share Posted June 10, 2019 Disk 5 has file system errors. You need to Check Disk File System On It Also, I hope that you currently live in St. Kitts Quote May 28 21:36:59 Tower vsftpd[9198]: connect from 41.216.186.87 (41.216.186.87) and Florida Quote May 29 01:35:03 Tower vsftpd[19626]: connect from 50.115.20.70 (50.115.20.70) And are just commuting back and forth. Otherwise, stop forwarding ports directly to the server for FTP if you don't need to (and can't use a VPN) The actual problem though is May 29 18:19:38 Tower rc.diskinfo[5089]: PHP Warning: file_put_contents(): Only 4096 of 7851 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Which is the Preclear plugin going haywire. Not in a position to look at the plugin, so can't say where it's trying to write to Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 (edited) Hi Squid, thanks for the help! The florida IP is right, its a whm backup that is FTPing to server, the other one I'm not so sure about need to see about somehow masking it, not sure what options I have... So should I stop/remove the preclear plugin? I am not running it, nor have I ran it in ages, my system has been intact for well over a year, no disks in/out or anything of the sort. Should I uninstall and re-install? Or Uninstall and simply leave it as is? The only thing I ever did years back to the preclear was install the fast preclear plugin, but that is all. Attached is the Check Disk File System output. Disk5Check.txt Edited June 10, 2019 by manolodf Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 That shows quite serious corruption so you may well get some data loss (do you have backups?). You need to rerun with the -n option (the no modify flag) removed so that it can repair as much as possible. Some files are going to end up in the lost+found folder for files where the name cannot be identified. Those are always a nuisance to try and work out what they are. Quote Link to comment
Squid Posted June 10, 2019 Share Posted June 10, 2019 20 minutes ago, manolodf said: So should I stop/remove the preclear plugin? Because its one of the rare plugins that does stuff in the background all of the time, since it was having problems (not sure why), I'd get rid of it. Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 22 minutes ago, manolodf said: So should I stop/remove the preclear plugin? When you install the Pre-Clear plugin it also (optionally) installs the statistics plugin. That is the one that is currently 'spamming' the syslog. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 3 minutes ago, Squid said: Because its one of the rare plugins that does stuff in the background all of the time, since it was having problems (not sure why), I'd get rid of it. Ok I popped it. Though I still cannot seem to do anything else to get the dockers to work without the 403 error. I can't get that balance command to work either. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 2 minutes ago, itimpi said: When you install the Pre-Clear plugin it also (optionally) installs the statistics plugin. That is the one that is currently 'spamming' the syslog. Thanks for the help itimpi! Should I remove the "Dynamix System Statistics" Plugin as well? Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 1 minute ago, manolodf said: Thanks for the help itimpi! Should I remove the "Dynamix System Statistics" Plugin as well? I would. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 (edited) 12 minutes ago, itimpi said: That shows quite serious corruption so you may well get some data loss (do you have backups?). You need to rerun with the -n option (the no modify flag) removed so that it can repair as much as possible. Some files are going to end up in the lost+found folder for files where the name cannot be identified. Those are always a nuisance to try and work out what they are. Running now with -n option, attached is the result of -n . Disk5Check.txt Edited June 10, 2019 by manolodf Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 3 minutes ago, itimpi said: I would. I have removed that as well, and attached the result of disk check with -n option. Is there anything else I should do now? How can i check what files possibly were corrupt? Is this a cache disk error? Do i need to change the filesystem or anything of the sort? Disk5Check.txt Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 4 minutes ago, manolodf said: I have removed that as well, and attached the result of disk check with -n option. Is there anything else I should do now? How can i check what files possibly were corrupt? Is this a cache disk error? Do i need to change the filesystem or anything of the sort? Disk5Check.txt 53.39 kB · 0 downloads You have to run without -n to get anything fixed. This nothing to do with the cache disk - it is specific to disk5. I do not know how you would identify which files are missing/corrupt. Normally this is done by comparing against your backups (if you have any) or be running something like the File Integrity plugin that can help you identify both what files are missing and which ones might be corrupt. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 Just now, itimpi said: You have to run without -n to get anything fixed. This nothing to do with the cache disk - it is specific to disk5. I do not know how you would identify which files are missing/corrupt. Normally this is done by comparing against your backups (if you have any) or be running something like the File Integrity plugin that can help you identify both what files are missing and which ones might be corrupt. Ok I want to tread slowly here since this is my first time in well a decade with Unraid that I have had any issues as scary. So right now I have array on, to run the above screenshot I posted. Do you recommend I install that File Integrity Plugin before or after doing the file check without the -n? Not sure which one should go first, second, third etc. Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 The File Integrity plugin cannot help at this point with your current problems. It has to be running when files are being written to the array so that it can generate appropriate checksums. It could only help you, therefore, if this type of problem occurs again. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 Ok, I will run the check without the -n, so the 403 error on the dockers has nothing to do with my cache drive? Thats a relief because I thought I was going to have to do something like switch it to xfs. It just seems so weird some dockers start, and when stopped they give that 403 error and are just useless and no dockers boot up, so I can't run the dolphin/krusader that are listed on fix common problems even. Not sure what is going on, if my docker img is got corrupt because of all of this? Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 16 minutes ago, manolodf said: Ok, I will run the check without the -n, so the 403 error on the dockers has nothing to do with my cache drive? Not what I was saying The disk5 errors are nothing to do with your cache drive. It is possible that that there is something wrong with the cache, although I did not spot anything saying so. There definitely seems to be something wrong with the docker.img file. The normal way to handle that is: Stop the docker service Delete the existing docker.img file Restart the docker service to create a new empty docker.img file Restore your docker containers using the Previous Apps section on the Apps tab. This will restore your docker containers with their settings intact. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 Ahhh balls, thanks for the clarification!! Ok do you recommend I run the check filesystem before doing the docker image restore, or does it not matter which one I do first? After it is back, do I need to do that brtfs balance command I have seen when others have 403 errors? Quote btrfs balance start -dusage=75 /mnt/cache Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 10 minutes ago, manolodf said: Ahhh balls, thanks for the clarification!! Ok do you recommend I run the check filesystem before doing the docker image restore, or does it not matter which one I do first? After it is back, do I need to do that brtfs balance command I have seen when others have 403 errors? It would not do any harm to do a file system check first on the cache drive as if it reports any errors you want to get that fixed first before trying to fix the docker image. As to the BTRFS balance I am not sure - I use exclusively XFS so have no experience of this. My guess is that it cannot really do any harm but may be superflous. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 Sorry I was asking about Disk5 File System check without the -n, should I do that before or after the cache docker image process? Does it matter? I ran check on Cache without any errors, should I take this opportunity to switch my Cache to XFS? I couldn't tell you why its btrfs, I suppose it just was like that at some point and stayed that way. But I am unaware if there are limitations that prevent me from switching to it, or if there is a correct detailed procedure that there may be a link to? Considering I have dockers like Plex app data and such, just want to be careful to not lose any of that. If this is the right time and better for future, then I am open to switching to xfs. Quote Link to comment
itimpi Posted June 10, 2019 Share Posted June 10, 2019 No - the disk5 fix order is not important unless you have something docker related on it. Probably best to get it out of the way. Quote Link to comment
manolodf Posted June 10, 2019 Author Share Posted June 10, 2019 Do you have a link to a trusted step by step procedure to switch cache to XFS by chance? I just want to make sure I don't risk any appdata in the process. I found this one, but not sure if it is the latest/best option, with Format instead of Replace in this case. Quote This procedure assumes that there are at least some dockers and/or VMs related files on the cache disk, some of these steps are unnecessary if there aren't. Stop all running Dockers/VMs Settings -> VM Manager: disable VMs and click apply Settings -> Docker: disable Docker and click apply Click on Shares and change to "Yes" all cache shares with "Use cache disk:" set to "Only" or "Prefer" Check that there's enough free space on the array and invoke the mover by clicking "Move Now" on the Main page When the mover finishes check that your cache is empty (any files on the cache root will not be moved as they are not part of any share) Stop array, replace cache device, assign it, start array and format new cache device (if needed), check that it's using the filesystem you want Click on Shares and change to "Prefer" all shares that you want moved back to cache On the Main page click "Move Now" When the mover finishes re-enable Docker and VMs Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.