Downloaded 6.11.0.r4, power-outtage caused reboot/install - can only boot in safemode.

[OneMeanRabbit]

Captured the error which flashed very quickly:

Loading /bzroot: ...ok

Loading amd_iommu=pt...failed: No such file or directory

 

Didn't change any other unRAID options or BIOs settings, besides downloading r4 update.  Storms must have knocked out power long enough to cause UPS to kickoff shutdown.  

 

My BIOs has SV-IOV enabled, and I have diagnostics files 6 days ago after issue showed up and today.  Things I tried:

 

1. Rollback to 6.11.0.r3.
2. Ran Memtest - 8 runs, no errors
   1. syslog in safemode keeps showing "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/emhttp/plugins/dynamix/include/Syslog.php on line 18"
3. Renamed plugins & rebooted.
4. Removed all plugins & rebooted.
5. Manually edit out ohptions under regular boot option - PCIE passtrough & amd_iommu=pt, leaving only bzimage & bzroot
6. Booted in safemode:
   1. reinstalled zfs plugin & others, zfs pools (all data) is fine and accessible after installation.  
   2. SMART errors on 2 mirrored cache drives (reallocated sector counts 24) and several on my spinning rust (mirrored vdevs - still showing healthy)...

      1. UDMA CRC error count 2

      2. Reallocated sector count 568

      3. offline uncorrectable

 

Thinking maybe reset CMOS or replace CMOS battery?  Happy to post entire .zip or specific ones if this helps.

[Hoopster]

4 minutes ago, OneMeanRabbit said:

Captured the error which flashed very quickly:

Loading /bzroot: ...ok

Loading amd_iommu=pt...failed: No such file or directory

Power outages have been known to occasionally cause corruption on the unRAID USB flash drive.  You can try running checkdisk on the flash drive from a Windows computer.

 

If you have a recent full backup of the flash drive (always recommended), you can reformat the USB flash drive and restore the backup.

 

In a worst-case scenario (physically-damaged USB flash drive), you can restore a backup to a new flash drive and transfer the license/registration to the new flash drive. 

 

If you have no flash drive backup, copy everything you can from the current flash drive (corrupted files will likely not be copied) and restore them to either the same or another flash drive.

[OneMeanRabbit]

@HoopsterThank you for the quick reply.  I haven't made a backup in sometime, but will try this this evening.  Anyone else here with an AMD Ryzen 5 CPU - does your normal unRAID boot option show that option?  Most things I'm seeing online says it should say amd_iommu=on iommu=pt or that it's unnecessary. 

 

Worst case, is it possible to download 6.10.3 and "start over"?  I can save my docker templates, and my config files are on a separate zfs nvme pool.  Probably just need to redo my cache ssd's which hold my docker subsystem (not using .img, but folders on the SSDs).  Would that work as well?

 

Thanks again to @Hoopster for advice!  Used an old backup and looked up syslinux.cfg - didn't have the offending amd_iommu option.  Deleted it from current cfg and voila!  Not sure how it got there, but great lesson in using backups to validate changes and make them more often...

Edited September 9, 2022 by OneMeanRabbit