Unraid keeps crashing after USB reset at same time

[Skylinar]

Hi folks,

 

I'm facing, out of nowhere, Server crashes at the same time on random days.
I have been watching the problem since 01.12.22 and could not find the problem until now. It seems that the issue occurs always at 4:40 AM - the Unraid USB stick gets reset and is after that no longer accessible. The log then fills up and Unraid crashes. Attached are the last three crashes over the last 2 weeks. After the last crash (tonight) the syslog was unfortunately empty.

Can anyone help me with this problem? Can it have something to do with the latest Unraid version or is my USB stick broken?

 

Thanks!

 

unraid-diagnostics-20221201-1840.zip unraid-diagnostics-20221208-1532.zip unraid-diagnostics-20221212-1149.zip

[JorgeB]

Make sure you are using a USB 2.0 port, if it keeps dropping replace the flash drive.

[Skylinar]

The Mainboard doesn't have a USB 2.0 port. The usb stick has been plugged into the same port for 1.5 years without any problems. I switched it to different usb ports after mentioned crashes but it seems not to resolve it.

[Kilrah]

Do you have anything scheduled at that time?

 

Might want to try installing the "flash remount" plugin too. 

Edited December 12, 2022 by Kilrah

[Frank1940]

35 minutes ago, Skylinar said:

The Mainboard doesn't have a USB 2.0 port.

 

I would be very surprised if it does not have a UBS header on the MB which would allow you to attach a USB port to it.  Here is a link to one:

 

https://www.amazon.com/Duttek-2-Pack-Female-Dupont-Motherboard/dp/B06Y5C7DKH/ref=sr_1_4?crid=17CEBL462XND4&keywords=USB+A+header&qid=1670848029&sprefix=usb+a+header%2Caps%2C77&sr=8-4

 

EDIT:  Some folks like having the boot drive inside the case for security and protection from damage.

Edited December 12, 2022 by Frank1940

[Skylinar]

4 hours ago, Kilrah said:

Do you have anything scheduled at that time?

 

Might want to try installing the "flash remount" plugin too. 

 

that was also the same thing I thought of first because it always happens at 4:40 AM. I've already searched quite a bit, but haven't found a specific job that does anything at 4:40 AM for me.

 

3 hours ago, Frank1940 said:

 

I would be very surprised if it does not have a UBS header on the MB which would allow you to attach a USB port to it.  Here is a link to one:

 

https://www.amazon.com/Duttek-2-Pack-Female-Dupont-Motherboard/dp/B06Y5C7DKH/ref=sr_1_4?crid=17CEBL462XND4&keywords=USB+A+header&qid=1670848029&sprefix=usb+a+header%2Caps%2C77&sr=8-4

 

EDIT:  Some folks like having the boot drive inside the case for security and protection from damage.

 

Agreed @Frank1940. I was only talking about the USB Type A headers... But your idea is not a bad one. I have indeed such adapters here and will install it immediately and report back in a couple of days whether this was a solution. 

[Kilrah]

3 hours ago, Skylinar said:

that was also the same thing I thought of first because it always happens at 4:40 AM. I've already searched quite a bit, but haven't found a specific job that does anything at 4:40 AM for me.

 

Out of chance I just noticed that this is the default time at which Unraid runs its daily tasks, as well as user scripts set to daily...

 

[Skylinar]

UPDATE:

My system crashed again this night at 4:40 AM. Same error - USB Stick got reset and is not available after that. As mentioned I switched USB Stick to USB 2.0 header, but this does not seem to work out.

 

14 hours ago, Kilrah said:

Out of chance I just noticed that this is the default time at which Unraid runs its daily tasks, as well as user scripts set to daily...

 

That is a great hint. I have two user scripts that run on the "daily" dropdown... Those scripts are backing up the USB drive like mentioned here: 

 

I have those scripts active at least one year but I've deactivated them on the schedule to test if the scripts are responsible for the freezes.

 

I'm attaching the custom script, maybe someone has an idea what could cause the USB reset when running it? I'm confused about it because it ran all time without any issues, so maybe this is an issue that came up with the new unraid version?

 

#!/bin/bash

#### SECTION 1 ####------------------------------------------------------------------------------------------------------
#dir = WHATEVER FOLDER PATH YOU WANT TO SAVE TO
dir="/mnt/user/Backup/Unraid/flash_backup"

echo 'Executing native unraid backup script'
/usr/local/emhttp/webGui/scripts/flash_backup

#### SECTION 2 ####------------------------------------------------------------------------------------------------------
echo ' Remove symlink from emhttp'
find /usr/local/emhttp/ -maxdepth 1 -name '*flash-backup-*.zip' -delete

sleep 5

#### SECTION 3 ####------------------------------------------------------------------------------------------------------
if [ ! -d "$dir" ] ; then
 
			echo "making directory as it does not yet exist"

			# make the directory as it doesnt exist
			mkdir -vp "$dir"
		else
			echo "As $dir exists continuing."
			fi


#### SECTION 4 ####------------------------------------------------------------------------------------------------------
echo 'Move Flash Zip Backup from Root to Backup Destination'
mv /*-flash-backup-*.zip "$dir"

sleep 5

#### SECTION 5 ####------------------------------------------------------------------------------------------------------
echo 'Deleting Old Backups'

#ENTER NUMERIC VALUE OF DAYS AFTER "-MTIME +"
find "$dir"* -mtime +30 -exec rm -rfv {} \;

echo 'All Done'

#### SECTION 6 ####------------------------------------------------------------------------------------------------------

#UNCOMMENT THE NEXT LINE TO ENABLE GUI NOTIFICATION UPON COMPLETION 
/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Flash Zip Backup" -d "A copy of the Unraid flash disk has been backed up" -i "normal"

exit

 

[Kilrah]

Doesn't look like it should cause anything to fail, the USB drive is probably simply on the way out and sometimes crashing when it's being read out...

 

[Skylinar]

Interesting about this is that I can run this backup script 20 times in a row without any issue at all. But today it seems like I can't even reboot into the system - no boot drive anymore but shows up when inserting it into a different PC - checked USB health -  all good as well.. maybe only data corrupted?

[Kilrah]

Possible...

[Skylinar]

UPDATE:

I restored my USB flash drive and it seems like the data on it was corrupted. As mentioned I had running scripts on daily schedule (4:40am) and one of them was a flash backup script. In fact none of the scripts was causing this issues but the corrupted data on the flash drive. After the complete data restore on the usb stick the server is stable again and the running scripts are not causing any freezes anymore. (did not changed anything on the scripts)