Think i'm in biiiig trouble :S

[Brydezen]

Hello, 

 

have been away for some time on a samll vacation. And my server has been offline. I booted it up today and saw it had an update from 6.5.3 -> 6.3.5 so like I always do, I install the update. And everything seems just fine. After some time I come back to the WebGUI (never closed the tab) and it keeps saying Syncing or something. I did that for about 2 - 3 hours. Thought it was weird so I refresh the tab. And everything seemed fine except I could not load the Update OS section at all. And some of my cores was running at 100% even without any dockers or VM's running on them. So after some time (about 4 hours - after the first 2/3 hours - so about 6 or 7 in total) I wanted to try and reboot the server.. I have tried many times to access the webgui, but with no luck. So I just hit the power down button, and it made it's shutdown sound. But it keeps running. And is still running after 20 minutes now. Have no idea what to do. And really don't want to hold down the power button for a force shutdown

 

Any ideas what to do? 

Best regards,

Mads

Edited November 13, 2018 by Brydezen

[Brydezen]

here is a picture of the console screen. Don't look like their is anything wrong in that screen. But not sure

[jonp]

Hi,

 

It will be difficult for us to diagnose unless you can send us your diagnostics.  Can you get the system in a similar state, then go to Tools > Diagnostics tab and click Download.  Then upload that Zip file in an attachment here on the forums so we can review.

[John_M]

The

cat: write error: Broken pipe

line doesn't look normal. It suggests that some output can't be written to a file. I strongly suspect that your boot flash device has dropped offline. If it's completely unresponsive I think you're going to have to bite the bullet and force a hard shutdown. Then unplug the USB flash and plug it into a PC and check it for file system errors.

[itimpi]

4 hours ago, John_M said:

The

cat: write error: Broken pipe

line doesn't look normal. It suggests that some output can't be written to a file. I strongly suspect that your boot flash device has dropped offline. If it's completely unresponsive I think you're going to have to bite the bullet and force a hard shutdown. Then unplug the USB flash and plug it into a PC and check it for file system errors.

This has been reported before and said to be harmless.    I think it is caused by one of the frequently used plugins.   I know I get it every time my system boots and everything is working as expected.

[Brydezen]

Hello, thanks for the responses. I bit the bullet and hard shutdown the server this morning. After booting it up again it had for some reason reset the boot order. After changing it back to UEFI usb boot. It loads the unraid boot menu. But ends up just giving me a

Loading /bzimage.... failed: I/O error

 

What to do now? Can load the usb stick in my mac and browse the files. Can I maybe just replace the bzimage file? Or what. 

 

Best regards,

Mads

Edited November 13, 2018 by Brydezen

[bonienl]

Try a different USB port, preferably USB2.

[Brydezen]

Are you sure that would work? I think the update never fully completed as it was saying Syncing for so long. But will try when I get home from school. Got the USB with me. Is their a way to check if it is complete or maybe revert back to 6.3.3?

 

best regards,

Mads

[bonienl]

If you suspect the download was incomplete. Do the following:

 

1. Download the zip file

2. Copy all bz* files to your flash device

3. Reboot your Unraid system

[Brydezen]

I cross checked the files on the usb vs the zip file. And can se the files on my usb states they are null bytes. So that must be the problem for me. 

All the bz* files are null bytes but also: make_bootable* and memtest - I should also copy those over, right?

 

Best regards,

Mads

[bonienl]

Yes, your flash device got corrupted.

I also miss the "syslinux" folder on your USB. You need to copy that folder + content too.

Run "make_bootable" afterwards to ensure the USB is bootable.

[Brydezen]

No the syslinux is their, and it seems to be alright. I just cut the image short, so it would not be to big. Here is a complete list of the files. And the syslinux folder too. 

[itimpi]

You need to copy all the files at the root level and the syslinux folder (and contents as well) from the ZIP file.    There also appear to be some redundant files (e.g. ldlinux.c32) from earlier Unraid releases where the syslinux files were at the root level rather than in the syslinux folder, nd these can be deleted.

 

What is not clear is whether there is also corruption inside the other folders (e.g. the config folder) that are not part of the release.   Hopefully not and it was only the ones the Unraid update was trying to change that got affected.

 

It is not clear if this corruption was a one-off or part of a potential evolving problem with the USB drive.   I would keep an eye on it and see whether you get more of the FSCK* type files appearing a these are caused by trying to correct corruption.   In my experience they are quite likely to happen if you have unscheduled poweroffs when writes might be in progress but should not otherwise happen in normal use.

[Brydezen]

Alright I'm home from school now. And have been looking a bit at the flash drive and what has come up here. I have made the changes to the drive you have described @itimpi Here are some screenshots. Do they look alright? 

NOTE: It's just made in a temp folder on my desktop. Don't wanna edit the files on the USB stick before I'm 100% sure. But I have made a copy of all the content on the stick, so I should always be able to return back.. But after copying the files over, I just just run the make_bootable_mac version in the terminal and boot the server up.

 

Best regards,

Mads

Edited November 13, 2018 by Brydezen

[trurl]

You could just prepare the flash as a new install and copy the config folder from the backup you made. That is the simplest and most foolproof method of getting things going again. Everything unique to your config is in the config folder, and everything else is in a new install.

[Brydezen]

I could do that too. But do I need to use my transfer license then? Don't wanna waste it if I don't have to 🙂

Edited November 13, 2018 by Brydezen

[itimpi]

Just now, Brydezen said:

I could do that too. But do I need to use my transfer license then? Don't wanna waste it if I don't have to 🙂

If you are using the same USB stick then there is no need to use the License transfer process.   The key file is tied to the GUID of the USB drive which will not change if you have reformatted it. 

[Brydezen]

Think I will try and just replace the files first. And then if that don't work I will install unraid from fresh and just copy over my config folder.

[Brydezen]

Think maybe the USB drive is bad... When trying to copy the big bz* files over. It for some reason eject's itself from my mac and stop the file transfer... So maybe I should find me a new usb drive instead...

[jonp]

35 minutes ago, Brydezen said:

Think maybe the USB drive is bad... When trying to copy the big bz* files over. It for some reason eject's itself from my mac and stop the file transfer... So maybe I should find me a new usb drive instead...

Definitely.  Your USB is likely past the point of recoverability so just get the config folder copied off there so you don't lose any settings and set up a new USB flash.  Follow this guide:

 

https://wiki.unraid.net/UnRAID_6/Changing_The_Flash_Device

[Brydezen]

Just really a bad time for this to happen. I have some USB sticks. But most of them are 16, 32 and 64 GB. And need them for other things than to host a small OS like unraid. But I think I need to bid the dust, and just order one off amazon or something asap. So I can transfer my license to that one.

[itimpi]

I use Sandisk Cruzer Fit 16GB USB2 drives which have proved reliable and are also cheap.

[Brydezen]

UPDATE: I reinstalled unraid 6.6.5 to the same usb stick, and copied over all my config files. And the command line boot for unraid is successful, and I get an ip from my DHCP server. But I cant access the web ui. Not even on the machine itself if I boot into the WebGUI menu.  - not even in GUI safemode can I connect to the webinterface

 

EDIT: It even shows up under network drives on my mac. But can't connect to it.

 

UPDATE2: I looked at the config folder again and it seems that some of the apache files got corrupted too. But the flash creator, don't make them from scratch. So can't copy any new ones over  @jonp & @bonienl one of you guys much know where I can get my hands on some new ones? - sorry for the ping btw. 

Edited November 13, 2018 by Brydezen

[Brydezen]

Lol @itimpi was JUST looking at that usb drive. I think I will order one tonight. I really think I need it. But this thing is slowly killing me by now. Don't get why I'm having so much troubles

[trurl]

Can you access it by IP?

 

Your OP said you were updating from pre 6.4 so I have to wonder if you took care of everything in the Update Notes. In particular, maybe you have given your server a name that isn't allowed.

 

You could try to get us a diagnostic by using the command line method explained in the Need Help sticky.