I recently purchased a HP DL380e gen8 with 14 LFF drive bays. I am having a really strange issue that I cannot seem to track down.

 

Unraid will not boot more than once. The first boot goes fine and all of my drives show up, but if I reboot after that it hangs at "starting rsyslog daemon". See attached screenshot for what the console looks like.

 

When I tried moving over my USB stick from my old server I had the same problem.

 

Things I have tried:
Two flash drives, one of which is basically brand new and the other worked fine in my other server.

Disconnect all storage devices

New flash drive with old config folder(from different server)

New flash drive with untouched config folder

Removed PCI-E HBA(server has onboard HP B120i that I haven't tried disabling and has no attached storage devices)

Safe mode with old config folder(from different server)

Safe mode with untouched config folder

 

At this point I don't know what else to try. If anyone has any suggestions I'd be happy to try them out.

 

System Specs:

HP DL380e G8 System

2x Xeon E5-2420v2 CPUs

16GB DDR3 1333Mhz ECC ram

 

Thanks!

Edited December 18, 20196 yr by soja

Have you tried booting from USB2? 

 

Have you done memtest? 

2 hours ago, trurl said:

Have you tried booting from USB2? 

 

Have you done memtest? 

Thank you for the suggestions.

 

I will try forcing USB 2.0 or finding a USB 2 only port when I get home from work tonight. I read in another post that it may be a waste of time to run memtest on ECC ram, is that the case?

Edited December 18, 20196 yr by soja

1 minute ago, soja said:

I read in another post that it may be a waste of time to run memtest on ECC ram, is that the case?

Yes no point since memtest won't find anything with self-correcting RAM.

2 minutes ago, soja said:

I read in another post that it may be a waste of time to run memtest on ECC ram, is that the case?

I believe that is the case unless the BIOS on the motherboard has an option to disable the EEC capability.

If it only fails after a reboot, I assume it boots correctly after a power cycle, to me it suggests something hardware/bios related, not a failing part per se, looks for a bios update, restore bios to defaults, remove everything you can from the server to test, etc.

7 minutes ago, johnnie.black said:

If it only fails after a reboot, I assume it boots correctly after a power cycle, to me it suggests something hardware/bios related, not a failing part per se, looks for a bios update, restore bios to defaults, remove everything you can from the server to test, etc.

It does not boot correctly after a power cycle. It appears if there has ever been a successful boot of the USB(on another system, or the same system) it will refuse to boot again. 

 

I do intend to try to get as much firmware/bios updates as I can but annoyingly HP tries to lock that stuff so only people with a valid support contract can access it.

14 minutes ago, itimpi said:

I believe that is the case unless the BIOS on the motherboard has an option to disable the EEC capability.

I do have spare RAM of the same type I can try anyway but it really seems like a farfetched symptom of always being able to boot on a USB once and never again.

2 minutes ago, soja said:

It appears if there has ever been a successful boot of the USB(on another system, or the same system) it will refuse to boot again. 

Very strange, don't remember seeing anyone with a similar issue before.

8 minutes ago, soja said:

It appears if there has ever been a successful boot of the USB(on another system, or the same system) it will refuse to boot again. 

Maybe getting corrupted by whatever it boots on?

 

Try USB2 port as already suggested.

4 minutes ago, trurl said:

Maybe getting corrupted by whatever it boots on?

 

Try USB2 port as already suggested.

I will absolutely try that later today. Until then I am open to getting ideas and feedback for other things I can try.

 

EDIT: According to this spec sheet the internal USB port I was using is USB 2 only.

Edited December 18, 20196 yr by soja

Maybe a power issue and all the other symptoms are just random manifestations of that?

35 minutes ago, trurl said:

Maybe a power issue and all the other symptoms are just random manifestations of that?

It came with two 750w 80+ Platinum PSUs but I was only using one for ease. I will try with both and see if it likes that any better.

 

2 hours ago, soja said:

t does not boot correctly after a power cycle. It appears if there has ever been a successful boot of the USB(on another system, or the same system) it will refuse to boot again. 

If this is the case, I would suspect that the flash drive is corrupted.  I have highlighted and unlined a section of the quote.  Could you expand on this a bit more? 

 

I don't see that you have run a 'chkdsk' on the Flash Drive after it has failed to boot after the first successful boot.  Can we assume that you can 'repair' the flash drive by reinstalling Unraid?   

1 minute ago, Frank1940 said:

 

If this is the case, I would suspect that the flash drive is corrupted.  I have highlighted and unlined a section of the quote.  Could you expand on this a bit more? 

 

I don't see that you have run a 'chkdsk' on the Flash Drive after it has failed to boot after the first successful boot.  Can we assume that you can 'repair' the flash drive by reinstalling Unraid?   

Initially I wanted to transfer my array from an existing installation in a Dell R710 which only had 6 LFF drive bays to the new server. When I gracefully shut down the old server and moved everything over I had the result in the screenshot and the server wouldn't complete booting. I then tried creating a new flash drive with the official unraid USB tool and copying the config folder from my existing USB and it resulted in the same experience. When that didn't work I tried a different USB drive with the same result. 

 

Essentially the sequence of events is this:

 

Existing installation -> Boot on new server fails

New Installation -> Successful boot -> next boot always fails

 

I have not tried a chkdisk on a USB that fails to boot. I have also not tried to boot the USB stick on a different server after it fails on the HP. I will try these things when I get home today.

 

Thank you all for your suggestions so far

Okay I have tried a few of the suggestions.

 

I tried other USB ports, none allow a proper boot.

I swapped in 2 4GB sticks of ram from another server, same issue.

The USB stick that will not boot on the HP DL380e gen8 does complete booting on my old Dell r710 so the problem does seem to be something in the HP system

I have ran chkdsk and chkdsk /f on the USB drive and Windows reported no errors.

I reset the server BIOS to defaults.

 

I am going to try to find which files specifically are changed during the first boot to see if I can narrow down the issue. 

Edited December 18, 20196 yr by soja

It appears that when the file config/network-rules.cfg exists it causes the boot to hang. Anyone have any idea why that would be and how I could fix it?

 

network-rules.cfg

Edited December 19, 20196 yr by soja

Based on troubleshooting tips I have seen on other threads I enabled "mirror syslog to flash" and then rebooted. Attached is the syslog of a failed boot(I think, does it write the existing syslog when you enable that option, or only new entries?).

 

I also downloaded the syslog of a successful boot from the web gui. If there is anything else I can get to help troubleshooting please let me know!

 

EDIT: Just deleted the syslog file from USB and tried boot again and it did re-create it. I have attached it as syslog-new.

syslog tower-syslog-20191219-0147.zip

syslog-new

Edited December 19, 20196 yr by soja

Can you post your diagnostics after a successful boot

2 hours ago, soja said:

It appears that when the file config/network-rules.cfg exists it causes the boot to hang. Anyone have any idea why that would be and how I could fix it?

You can always go back to default network settings by deleting (or renaming) network.cfg and network-rules.cfg from the config folder.

22 minutes ago, Squid said:

Can you post your diagnostics after a successful boot

Yep, attached

 

11 minutes ago, trurl said:

You can always go back to default network settings by deleting (or renaming) network.cfg and network-rules.cfg from the config folder.

I am repeatedly deleting network-rules.cfg to allow me to boot.

tower-diagnostics-20191218-2202.zip

Ok I *think* I have found the issue. When I disable the "Intel SATA controller" in bios(it is listed as a PCI-E device but it is onboard) unraid appears to work normally. Not sure why this Intel device causes unraids boot to hang, I'd be happy to get some more information if you can tell me what to look at.

BIOS update?

6 minutes ago, trurl said:

BIOS update?

I'll mess with it more later. As of right now it is working as expected and I have a few TB's waiting to be written to the newly expanded array.