Server Crashing

[prytzen]

Dont really know where to begin on troubleshooting so any assistance would be very helpful. 

 

Currently, my server will not boot properly in normal mode. I can start the server, it will get to the login prompt and be pingable on the network but the web interface will not load. After about 1-2 minutes, the server will no longer be pingable. During the time that the server is available, i can SSH to it without issue. 

 

If i boot the server in GUI mode, the same symptoms occur and i am still unable to get to the web interface directly on the server via "localhost"

 

I have started the server in Safe Mode GUI No Plugins and it will start up no problem and be stable with the web interface available. 

 

This lead me to believe that it is an issue with a plugin. I looked on the forms and saw that you could remove the files in Config/Plugins so i have (Before and after shots attatched). After doing this, the server is still responding with the same behavior. What i did notice, is after turning the server back on, the folder Config/plugins/dynamix was recreated somehow.

 

What do i do?!

[trurl]

You shouldn't delete the whole config/plugins folder. Just delete or rename any .plg files in that folder and the plugins won't load.

[prytzen]

I have renamed all the PLG files in the Config/Plugin folder and the symptoms are the same. 

 

The server will start up, become pingable, but the web interface will never load. Eventually the server becomes unpingable. 

 

How can i find out what is going on here?

[trurl]

Post the contents of your go file on flash, config/go.

[trurl]

13 hours ago, prytzen said:

I have started the server in Safe Mode GUI No Plugins and it will start up no problem and be stable with the web interface available. 

Is this still true?

[prytzen]

Config/go file attached. 

 

Yes - SafeMode GUI No Plugins works fine still. 

go

[trurl]

Boot in SAFE mode then go to Tools - Diagnostics and attach the complete Dagnostics ZIP file to your NEXT post.

[prytzen]

Diagnostics Attached

unraid-diagnostics-20200528-0816.zip

[trurl]

Go to Settings - Docker and disable dockers. Then see what happens if you start the array. Post new diagnostics.

[prytzen]

I disabled Docker, and started the array. 

 

the message Array Starting.  Mounting disks is displayed, but it does not seem to be bringing the array online. 

 

I have tried to open a new browser window to the server to pull the diagnostics again, but it is unavailable. 

 

Pinging the server, it responded for about a minute, but now is not pinging. 

[prytzen]

Is there a way to run the diagnostics from a the command line directly on the server and save it to the USB stick so i can provide this?

[itimpi]

3 hours ago, prytzen said:

Is there a way to run the diagnostics from a the command line directly on the server and save it to the USB stick so i can provide this?

Yes, use the ‘diagnostics’ command which writes the results to the logs folder on the flash drive.

[prytzen]

32 minutes ago, itimpi said:

Yes, use the ‘diagnostics’ command which writes the results to the logs folder on the flash drive.

I tried this and let it go for 10 minutes but didnt seem to complete. I turned off the server and inspected the logs folder and nothing was there. Plugged the USB back into the computer and now it appears the USB is no longer bootable.

 

Do i need to re-create my boot disk?

[itimpi]

I would definitely think that recreating the flash drive might be a good idea.  If the system had problems reading and/or writing to the USB stick that might explain your symptoms.   Make sure you first back up the ‘config’ folder on the USB drive as that contains your licence key and all your settings type information.    

[prytzen]

Ok. So i have created a new flash drive and gotten it licensed properly. I copied the configuration files necessary (following this: https://wiki.unraid.net/Files_on_v6_boot_drive) leaving out plugins or docker files. 

 

The server boots, and webpage displays fine. However, i am unable to bring the array online. I have let it run for about an hour and it has not finished. I tried to run "diagnostics" from the command line, but it as well has not finished (as i suspect the array mounting is getting in the way). 

 

So i would suspect there is something up with my array.... or my disks? I did notice that when i started the server my Disk4 had a SMART error. I acknowledged the error and tried to bring the array online. Would this be causing it all? Is Disk4 just dead? How can i test this theory. 

[prytzen]

I looked at it again this morning. I started the machine in safemode and was able to bring the array online in MAINTENANCE mode. I looked at the log and didnt see anything and with the log open i took the array offline and then tried to bring it online in normal mode (with mounting the drives). In the log (attached) it looks like all the drives mount fine, but there are errors btrfs with the CACHE drive. 

 

I had to restart the server and i brought the array back on in MAINTENANCE mode. and then opened the CACHE drive to do the btfrs Check status and here is the output. 

 

 

[1/7] checking root items
[2/7] checking extents
ref mismatch on [15157989376 159744] extent item 1099503797234, found 1
incorrect local backref count on 15158149120 root 5 owner 1597264 offset 950272 found 1 wanted 3827212289 back 0x3d892c0
backpointer mismatch on [15158149120 8192]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/sdh1
UUID: 02bb4712-df62-48cc-9654-1e32a7629104
found 14323052544 bytes used, error(s) found
total csum bytes: 5855024
total tree bytes: 96059392
total fs tree bytes: 69353472
total extent tree bytes: 15826944
btree space waste bytes: 22216609
file data blocks allocated: 91474538496
 referenced 8596598784

log.txt

[prytzen]

I swapped the CACHE drive with another on hand and was able to get the array to mount properly. 

 

 

Now i need to understand how to restore my Docker images. I use CA Backup/Restore and have a backup of the appdata and have restored that backup, but none of the containers are showing up under the Docker tab. What am i not doing right or missing?

[trurl]

The appdata is only one piece of the puzzle. The docker image contains the executable code for your dockers. According to your settings in previous diagnostics that docker image was probably on cache. But docker image isn't important since the executable code can be downloaded again.

 

The templates which contained all the settings you made for each docker (mappings, etc.) was in the config folder on flash. Were you able to restore that config folder to your new flash?

[prytzen]

Yes i was able to restore the AppData, and after watching a few videos i was able to learn i could just redownload the Docker executable code and make sure it was pointing to my AppData and everything is working. 

 

Odd that a server crashing and having to essentially rebuild it has given me more confidence in the product and how it works. 

 

Thank you all for your help.