Unraid startup failure - cannot get to GUI

[risenfromthedust]

Everything was known working last night. This morning, I noticed a couple of my docker apps werent responding so I logged on my server. On the Dashboard, I saw that several docker apps had question mark logos, instead of their respective logos. When I tried to go to the Docker page, I got a message that docker failed to start. I then went to Settings>Docker to see if i could restart the service. I changed the yes to no and back, and then hit apply. The page reloaded, but still said service was stopped. Then I went to tools to look at the syslog. I saw the following error at the bottom:

Jan 24 08:06:11 unraid kernel: lo_write_bvec: 29 callbacks suppressed
Jan 24 08:06:11 unraid kernel: loop: Write error at byte offset 7246594048, length 4096.
Jan 24 08:06:11 unraid kernel: blk_print_req_error: 29 callbacks suppressed
Jan 24 08:06:11 unraid kernel: I/O error, dev loop2, sector 14153504 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2
Jan 24 08:06:11 unraid kernel: btrfs_dev_stat_inc_and_print: 29 callbacks suppressed
Jan 24 08:06:11 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 904, rd 0, flush 0, corrupt 0, gen 0
Jan 24 08:06:11 unraid kernel: loop: Write error at byte offset 7515029504, length 4096.
Jan 24 08:06:11 unraid kernel: I/O error, dev loop2, sector 14677792 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2
Jan 24 08:06:11 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 905, rd 0, flush 0, corrupt 0, gen 0
Jan 24 08:06:12 unraid kernel: loop: Write error at byte offset 3662774272, length 4096.
Jan 24 08:06:12 unraid kernel: I/O error, dev loop2, sector 7153856 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Jan 24 08:06:12 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 906, rd 0, flush 0, corrupt 0, gen 0
Jan 24 08:06:12 unraid kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction)
Jan 24 08:06:12 unraid kernel: BTRFS info (device loop2: state E): forced readonly
Jan 24 08:06:12 unraid kernel: BTRFS warning (device loop2: state E): Skipping commit of aborted transaction.
Jan 24 08:06:12 unraid kernel: BTRFS: error (device loop2: state EA) in cleanup_transaction:1992: errno=-5 IO failure

 

My next idea was the 'the old standby' - turn it off and on. I went to 'Main' and told it to reboot. Note: at this point I could still navigate around the webGUI, the array was still showing green and no errors on any drives.

 

 

After reboot, the login page showed and I attempted to login. The login page hung and never loaded. Eventually I got this:

 

When I tried to ssh into the box, I got the login, but after typing in pw, it never dumped me at the prompt. Luckily I'm running on an old Dell r720xd, so I logged into iDRAC, and pulled up the console. I was able to login again, and again I wasnt dumped at the prompt. Frustrated I hit <CTRL> +C and that dumped me at the prompt. Something caught my eye. Normally when I ssh into the box, the username@servername part before the command prompt is usually green, but it stayed the black and white prompt colors, so already I knew somethings amiss. I went to /var/log/syslog and found more of the error above. I also saw my failure to load the 'Main' page after logging in:

Jan 24 08:40:08 unraid nginx: 2024/01/24 08:40:08 [error] 8620#8620: *3494 upstream timed out (110: Connection timed out) while reading response header from upstream, client: [MY LAPTOP IP], server: , request: "GET /Main HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "[MY SERVER IP]"

I figured I was at a loss. I was seeing errors in my logs and I had no way to mess with the GUI. I knew I needed help, so I googled how to get a diagnostics file from command line and ran the command ("diagnostics" is the command, in case someone sees this). When I went to the location of the file, I saw that the system had auto generated two more, likely from failures at reboot. 

I am including all 3 just in case. The latest file is the one I generated. 

 

 

 

unraid-diagnostics-20240124-0743.zip unraid-diagnostics-20240124-0759.zip unraid-diagnostics-20240124-0855.zip

[JorgeB]

Docker image is corrupt and needs to be recreated, but that would not cause the GUI issue, does it work if you boot in safe mode?

[risenfromthedust]

I rebooted using 'powerdown -r' and via iDRAC, chose to boot into safemode. Now I can get into the webGUI and click around. Of course the array isnt started and no plungins are available. But the basics are working. What do I need to do from here to hopefully not lose my data and fix my server?

[JorgeB]

You can recreate the docker image now and and if all is still good reboot in normal mode, then if it still doesn't work in normal mode it would suggest a plugin issue.

[risenfromthedust]

Recreating the docker img seems to have resolved my issues. While still in safemode, I started the array, went to Settings>Docker, turned Docker off, deleted the 'vDisk', then turned Docker back on. Then I rebooted back to regular mode. From there I reinstalled my apps from 'Previous apps' and all appears to be working. Thanks