My unRAID system has been working reliably since April. I left for a short trip to the US on Monday. On Tuesday, I noticed errors when attempting to save to Vaultwarden. I logged into the server in the evening to find that there was an error where none of the Docker containers were running. I tried restarting them, but I kept getting a message that they could not restart. The array looked healthy, but one CPU showed 80% usage, while the rest showed none.

 

Finally, I decided it was best to restart. The system never came back up.

 

Last night when I got home, I connected a monitor and keyboard to see why it would not start. I saw the checksum on the firmware file on the USB was not matching. I ran a memory check and dosfsck. I could find no evidence of failing hardware. So I downloaded Unraid's latest zip file and extracted that onto the USB.

 

It then restarted and came up for the internet, but I could not log in as root. Not being sure what was wrong, I erased the shadow and smbpasswd. This allowed me to set a new password. But afterward, I still could not log in.

 

I tried the password reset twice more, this time erasing the USB and essentially doing a fresh install.

 

I simply cannot log in to my unRAID system to finish restoring it.

 

I ordered a replacement USB. But it is a reputable brand from a reputable source and shows no signs of failure. I don't really have any good guesses as to what could have corrupted the file, what caused the NAS to crash in that manner, nor why I cannot log in with corrected files.

 

So you can connect to the GUI but not login?

Yes.  I am thinking the next steo is just to go throygh all the steps for a freah install.   I might aa well wait for the usb stick i ordered to arrive tomorrow to avoid that variable.

 

Only three hypothesis make sense:

1. Usb drive failure.

2. Failed update

3. Hackers.

 

The replacememt usb will remove the first possibility.  The fresh install removes the second.  The third seema unlikely, but i can search for unusual activity with my router.

Edited October 17, 20241 yr by docbillnet

You ca do a fresh install, test that it works, and then restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.    

I did a complete test of the usb using btrfs scrub of random data.  No errors.  So then i did a fresh install complete with the vfat formatting.

 

Then to avoid browser cache i did the reconfigure in igncognito mode.  This resulted in dialog for installing a license.  The net result is everything worked.

 

So it looks like both the bzfirmware and files related to licensing were corrupted.

 

This leaves likely caused interrupted update or hackers.  Hackers could overwrite files as sort of root kit, which would likely work until i rebooted.  A failed update could leave corrupted files.  So either cause fits with what i know.   Only i was away, and i don't see any updates that were released in the last couple of days that would somehow auto-install.

 

Presumably i need to restore my users and passwords.   But all the docker services are running so that is not urgent.

It seems part of the problem is the cache. The redirects for HTTP to HTTPS are permanent redirects, meaning your browser will continue to try and use them when you do a reinstall. I bypassed that problem on my final procedure by using incognito mode. But as soon as I tried connecting without using incognito mode, I hit the "cannot log in" problem. This is only truly solved by either clearing the browser cache or re-enabling HTTPS and rebooting the NAS.

 

Edited October 18, 20241 yr by docbillnet

It seems the most likely fundamemtal cause would have been one of the plugins i had installed.  One of them was to do auto updates, and amy of the plugins could have had code that either deliberately or accidently changed the files that caused the failure.

 

Since it really is not possible to carefully peer review every plugin in the community, it is probably best to only install ones i have reviewed and compiled myself.