waiting up to 30 seconds for device with label UNRAID to come online...

[Hoopster]

I know this was reported with past versions of Unraid but I have not seen it be an issue for a while.  A few days ago, I started seeing this message when the server boots: waiting up to 30 seconds for device with label UNRAID to come online...

 

Pressing the ENTER key to reboot does a reboot and then it comes back to the same message.  Lather, rinse, and repeat.

 

At first, the message would appear on one boot and then with a reboot, it started fine with no error.  Now it happens on every boot attempt (reboot or cold boot) and I can't get the server to boot.  Fortunately, it is my backup server.  I was a rebooting a lot working through some IPMI issues.

 

Everything was fine until it wasn't.  The only thing I have done recently that would affect /config is update plugins and docker containers.

 

I have tried the following:

- reformat USB, download 6.11.5 from Unraid, net, manually install on flash drive, restore /config folder from a backup

- try a completely different flash drive

- run chckdsk in WIndows on the flash drive (no errors found)

- USB Creator does not recognize the flash drives I have tried (admittedly just generic ones to see if I could get something to boot)

 

Flash drives are USB3 but in a USB2 port in the server.

 

Edited March 15, 2023 by Hoopster

[JorgeB]

Do you have a different PC to try booting Unraid? To see if the flash drive or the server is the problem. 

[Hoopster]

On 3/15/2023 at 3:46 AM, JorgeB said:

Do you have a different PC to try booting Unraid? To see if the flash drive or the server is the problem. 

I have two Unraid servers.

 

The USB flash drives I use in both are identical even though the Unraid configuration is slightly different.

 

In server #1, both flash drives boot up Unraid with no problems.

 

In server #2 (the one that prompted OP) neither flash drive boots up Unraid and both hang at "waiting up to 30 seconds for device with label UNRAID to come online..." and eventually prompt to press ENTER to reboot as an UNRAID device is not found.

 

It looks like a hardware problem (USB controller apparently).  I also tried to boot a Linux Live USB flash drive in both systems.  It works in server #1 and fails to boot in server #2.

[itimpi]

12 hours ago, Hoopster said:

It looks like a hardware problem (USB controller apparently).  I also tried to boot a Linux Live USB flash drive in both systems.  It works in server #1 and fails to boot in server #2.

I would think confirms that there is some sort of USB issue with the hardware on server #2.

[Hoopster]

4 hours ago, itimpi said:

I would think confirms that there is some sort of USB issue with the hardware on server #2.

That's my conclusion.  Interestingly, I connected to the motherboard USB 3.x header a cable with two USB 3.0 ports on the other end and the flash drive successfully boots Unraid from that.  It just will not boot from any of the 6 USB ports (4 USB 3.x and 2 USB 2.0) on the rear panel.  The "front" panel (cable) ports work fine.

 

According to the motherboard block diagram, all USB ports (rear and front) are controlled by the C246 PCH.  They must have different controllers internally.

 

Once booted into Unraid from the front panel cable, I can attach USB flash drives, USB HDDs, etc, to the rear USB ports and they are all recognized as unassigned devices.  It is just booting from USB that appears to be the problem.

 

UPDATE: and it now refuses to boot from the front panel cable as well.

Edited March 17, 2023 by Hoopster

[Hoopster]

UPDATE 2:  I took the motherboard out of the case and set it up as a breadboard system on my desk with a couple of disk drives connected.  I created a new USB flash with an Unraid trial and it boots just fine from the rear USB ports.  The Unraid flash drive that belongs to this system also boots (although the config shows a bunch of missing disks).  A DOS boot disk also works from the rear ports.

 

Outside the case, things work as they should.  The only things not connected to the breadboard setup that were connected in the case are the case fans, six disk drives (via SATA and OCuLink cables) and the front panel power switch and LED. 

 

I'll keep experimenting for a couple of days and then put it back in the case and see what happens.  Can't see anything that would cause an MB short while in the case and it would be odd it that only affected the ability to boot from rear panel USB.

[Hoopster]

On 3/15/2023 at 3:46 AM, JorgeB said:

Do you have a different PC to try booting Unraid? To see if the flash drive or the server is the problem. 

So, is this totally hardware related?  Despite some success indicated above, I put the board back in the server, connected everything to it and it was working fine.  I then disconnected the LAN and power cables and moved the server back where it belongs.  Now the same old problem is back and I cannot successfully get into Unraid with any of four USB flash drives as they are all waiting to be found.

 

The Unraid flash drive is recognized as the boot device, I see the menu and it goes through loading the bz* files.  Eventually, the error above appears about waiting for a device with the label UNRAID which it never finds.

 

Looking back on past threads where this was reported, it was recommended to make sure the boot flash had only one partition (it does) and that it in fact was labelled UNRAID; it is.  Limetech asked some users to press CTRL-C when the reboot message appeared, the enter root at the login and run 'blkid'.  Yep, only /dev/sdi1 appears and it is lableled correctly as seen below. 

 

It seems to me that this is a problem with Unraid enumerating the devices and finding the UNRAID USB flash drive to mount at /boot.  Or is the problem that the motherboard does not let Unraid see it?  I suppose there must be some sort of hardware issue if the flash drives boot fine on my other Unraid server.

 

What little hair I have left is coming out fast.

 

Edited March 23, 2023 by Hoopster

[Hoopster]

In reading through the threads discussing this issue when it first surfaced (with 6.10.0) it was apparent that 6.9.2 seemed to work just fine and that something had changed in 6.10.0 (Limetech confirmed as much) with the way the UNRAID flash device was identified and mounted at /boot.

 

I have been fighting this issue for a couple of weeks with a "new" motherboard (ASRock E3C246D2I) that I got as an RMA replacement for one on which the BMC chip had failed.  Since I have another Unraid server (with a different model ASRock server MB), I test booted four flash drives on it and all came up fine without the "waiting up to 30 seconds for device with label UNRAID to come online..."  They even worked on a couple of laptops I had laying around.  Must be totally a hardware problem with the E3C246D2I, right?  Not so fast!

• I took one of the flash drives that will not boot up on the E3C246D2I with Unraid 6.11.5, reformatted it and installed a clean version of 6.9.2.  Boots right up in Unraid with no problems.
• I then did the same with a clean Unraid 6.10.3 which supposedly has a fix for this problem when it was discovered in 6.10.0.  Fail with "waiting up to 30 seconds for device with label UNRAID to come online..."  message appearing and the boot process eventually failing to complete.
• I tried Unraid 6.11.5 again.  Same waiting... message.
• I then went back to 6.9.2.  Again, success and it boots up with no problems.
• Next I upgraded the OS to 6.11.5 from within the GUI of 6.9.2 and rebooted and it again failed to boot with the "waiting..." mesage

My conclusion is that there is some combination of Linux kernel version in anything after 6.9.2 and some unknown hardware issue (the board has the latest available BIOS) that simply will not let Unraid boot successfully on the E3C246D2I.  The frustrating thing is that 6.11.5 worked without issue on the same model of board and with the same BIOS for months before I had to RMA it because of the BMC chip failure.

 

At one point, I got it to boot successfully several times in a row with Unraid 6.11.5 and even transferred the license to that flash drive.  Once I moved the server back down to the basement, it has never again booted up successfully with Unraid 6.11.5.

 

To say this is frustrating is a huge understatement.

 

Edited March 23, 2023 by Hoopster

[JorgeB]

9 minutes ago, Hoopster said:

My conclusion is that there is some combination of Linux kernel version in anything after 6.9.2 and some unknown hardware issue (the board has the latest available BIOS) that simply will not let Unraid boot successfully on the E3C246D2I.  The frustrating thing is that 6.11.5 worked without issue on the same model of board and with the same BIOS for months before I had to RMA it because of the BMC chip failure.

That is very strange, possibly an issue with current Unraid and that specif board, if the previous one worked it cannot be a general problem with that board model, and I would expect more reports if it affected more users using the same one, do you remember if previous board was using default BIOS settings of if anything was changed? Only suggestion I have would be to try and find another member using one with v6.11.5 and ask for photos of all the BIOS settings pages.

 

P.S. did you try v6.12?

[Hoopster]

1 minute ago, JorgeB said:

That is very strange, possibly an issue with current Unraid and that specif board, if the previous one worked it cannot be a general problem with that board model, and I would expect more reports if it affected more users using the same one, do you remember if previous board was using default BIOS settings of if anything was changed? Only suggestion I have would be to try and find another member using one with v6.11.5 and ask for photos of all the BIOS settings pages.

 

P.S. did you try v6.12?

I have not yet tried 6.12 RC but it is worth a shot.

 

I have changed several things in BIOS settings.  Using the onboard video as default, enabling iGPU, some IPMI settings changes and, of course, boot from the Unraid flash drive.  Same changes existing in same BIOS version on prior board.

[Hoopster]

20 minutes ago, JorgeB said:

did you try v6.12?

Just tried 6.12.  Same problem, "waiting up to 30 seconds for device with label UNRAID to come online..." and it never does. 

 

Looks like I am stuck with Unraid 6.9.2 on this board or I convince ASRock to RMA the RMA board.

[Hoopster]

1 hour ago, BRiT said:

are you using legacy boot or that EUFI boot mode

I am booting UEFI.  I tried Legacy a while ago, only with 6.11.5, and it made no difference but it has been a few days since I tried it (not that time should make any difference).

[Squid]

Complete shot in the dark.

 

What if you rename the flash drive to something else and then try adding this to the syslinux config append line

 

unraidlabel=NEWFLASHDRIVENAME

 

[Hoopster]

59 minutes ago, Squid said:

Complete shot in the dark.

Well, shots in the dark are all I have left now.

1.  Reformatted flash USB with the label SQUIDUSB
2.  Could not make bootable as it was looking for a USB label UNRAID
3.  Edited make_bootable.bat and set the tag to SQUIDUSB
4.  Ran make_bootable.bat successfully
5.  Edited syslinux.cfg append line and added unraidlabel=SQUIDUSB
6.  Booted up Unraid 6.11.5 from the SQUIDUSB

Result:  Same problem with a different name.

 

 

Booted up Unraid 6.9.2 successfully again.

Edited March 25, 2023 by Hoopster

[Squid]

Is that a card reader?  Does a real flash drive boot?

[Frank1940]

1 hour ago, Squid said:

Is that a card reader? 

@Hoopster, if this is a memory-card-to-USB device, does it have two memory slots?  (I have two different readers that I have used for years.  One is dual slot one and the other has a single slot.  The dual slot reader will work properly with only type of card--- basically, if the other slot is used, Unraid does not recognize it properly!!!)

[Hoopster]

5 hours ago, Squid said:

Is that a card reader?

Yes.  Kingston MobileLite G2 USB 2.0 with SD and Micro SD card slots.

 

5 hours ago, Squid said:

Does a real flash drive boot?

No.  Have tried three different USB drives in rear USB 2.0 and USB 3.x ports all with Unraid 6.11.5 installed.

• Very old 8GB Kingston Data Traveler 2.0 - Not even recognized in BIOS
• 16 GB Lexar USB 3.0 - "waiting up to 30 seconds..." message
• 32 GB PNY USB 3.0 - "waiting up to 30 seconds..." message

4 hours ago, Frank1940 said:

does it have two memory slots?

Yes, SD and Micro SD with UNRAID labeled SD card and SYSLOG labeled MicroSD card.

 

It makes no difference if the second slot is empty and only the UNRAID SD card is inserted in the reader.

 

The same model Kingston MobileLite G2 boots Unraid 6.11.5 just fine in my other Unraid server.  It is from the same Manufacturer (ASRock) and has the same chipset (C246) as the problematic system on which I am unable to boot anything greater than version 6.9.2 of Unraid.

 

Prior to returning the E3C246D2I motherboard for RMA due to a BMC chip failure, I was booting into Unraid 6.11.5 on this system with a Kingston MobileLite G3 (USB 3.0) card reader with the same two cards installed.

Edited March 26, 2023 by Hoopster

[Frank1940]

53 minutes ago, Hoopster said:

It makes no difference if the second slot is empty and only the UNRAID SD card is inserted in the reader.

Just checked my card reader and it is using the SD card slot.  (I can't remember at this point if I am using a Micro-SD card in a Micro-to-SD adapter...)  

 

The only other possibility is the BIOS.  I would assume that you have previously thought of this and are on the latest version. However, a later version might have changed something that impacted USB booting.  (I almost hate to mention this as I can remember that many long years ago, flashing the MB BIOS could result in converting the MB into a doorstop-- very rarely, I admit.) 

[Hoopster]

11 minutes ago, Frank1940 said:

The only other possibility is the BIOS. 

BIOS is the latest version available for the board and the same version I had on the board I had to RMA.

 

Actually, I have reflashed the BIOS on this board three times already just to make sure.

[Frank1940]

32 minutes ago, Hoopster said:

BIOS is the latest version available for the board and the same version I had on the board I had to RMA.

 

Actually, I have reflashed the BIOS on this board three times already just to make sure.

 

Actually, I was suggesting trying earlier versions of the BIOS...

[Hoopster]

33 minutes ago, Frank1940 said:

Actually, I was suggesting trying earlier versions of the BIOS...

Yeah, I misunderstood.  I will try the previous version but the current version is the same version I had on the prior board when all was working (except the BMC).

[Hoopster]

1 hour ago, Frank1940 said:

I was suggesting trying earlier versions of the BIOS

That was a no go.  The prior BIOS version (two years older than current version) would not even successfully load.  It kept hitting 55 percent or so and then dropped down to 20% loaded, got up to 50 something percent again, dropped back down, etc.  It would never finish loading so it could be flashed.

 

Fortunately, I was able to bail out of that without negative consequences as it never got the the BIOS flashing process.  The server still boots albeit with the same problems as BIOS version has not changed (if that is the issue).

 

I may try it again by flashing from a USB drive instead of via the IPMI Maintenance page.

Edited March 26, 2023 by Hoopster

[Hoopster]

Here is a difference I see in the system that can boot Unraid 6.11.5.

 

Results of blkid show LABEL_FATBOOT="UNRAID" before LABEL="UNRAID"  I do not see the LABEL_FATBOOT entry for blkid on the system that cannot boot Unraid 6.11.5.  I am not sure what that means. 

 

System that cannot boot to Unraid 6.11.5 show this (booted to Unraid 6.9.2):

 

According to Limetech, 6.9.2 used udev to identify the device labeled UNRAID and 6.11.5 uses blkid.

Edited March 28, 2023 by Hoopster

[Hoopster]

I have made some progress.

 

After trying at least 5 flash drives in my possession, none of which could boot into anything beyond Unraid version 6.9.2, on a whim I purchased this flash drive from a nearby Office Depot.  It was mentioned by another user in the Sandisk PSA discussion.  What the heck, for $5 it was worth a try.  USB 2.0, 16GB (although really it is 32GB as it shows >28GB when formatted) and appears to have a unique GUID.

 

I installed Unraid 6.11.5 on it and it booted right up and has continued to boot up Unraid 6.11.5 through several reboots (warm and cold).  I got a trial license to make sure it will continue to work over the next 30 days.

 

And, yes, this one when successfully booted also has the LABEL_FATBOOT="UNRAID" in blkid

 

Unraid did not seem to like it when I copied over the /config folder from a backup.  However, I reassigned the same disks in the same slots and started the array indicating parity was valid and it came right up.  This is a backup server so not a lot was installed and configured on it.  I will just slowly redo the config over the next few days while doing several reboots just to make sure the server and Unraid really like this USB flash drive.

 

I will admit, I am puzzled that a random flash drive boots Unraid 6.11.5 when none of the others would; especially since all of the others booted just fine in another Unraid server.  Of course, I'll take it even if I do not fully understand what is happening. 😀

Edited March 30, 2023 by Hoopster

[Hoopster]

So far, so good after two days.  Unraid 6.11.5 is all configured the way I had it before and the server has booted up at least 10 times with the HP flash drive with no problems.

 

Since the flash drive I was hoping to use is the Kingston MobileLite G2 reader with SD and MicroSD slots, the latter being used for the syslog, I purchased this small microsd reader and put the syslog card in it.  Both the HP flash drive for Unraid and the Sandisk reader for the syslog are working great in the "problematic" server.  I'll give it a few more days and will likely end up licensing the HP flash drive for Unraid.

 

It is still strange to me how I had the Kingston Reader (both USB 2.0 and 3.0 versions) working with this server and Unraid 6.11.5 and then no flash drive (other than the HP) would boot anything greater than Unraid 6.9.2.

 

UPDATE:  As all continues to work well, I bought another Unraid license for the HP USB flash drive rather than transfer the license.  I don't really need three servers now but you never know.  I now have a license in reserve.

Edited April 3, 2023 by Hoopster