Kaput!

[tucansam]

Upgraded to Version: 6.10.2 yesterday.  Less than 24 hours later the server hung.  I couldn't even get Num Lock to work on an attached KB, or a an attached monitor to wake up.  Totally brain dead, power only lighting lights and fans.

 

Power cycle, stopped the parity check.  Now totally unable to ssh in ("Couldn't agree on a host key algorithm" was the error from Putty) and copying files to the server is going between 256Kb/s (not a typo) and 1.74Mb/s.  

 

Any suggestions?

[tucansam]

Has now hung a second time, as of this moment, 22 minutes after I posted my last message.

ffs2-diagnostics-20220530-1741.zip

Edited May 31, 2022 by tucansam

[tucansam]

Here's a photo of what the screen looked like before I had to power off/on.  Totally frozen, no movement, KB dead, and showed an uptime of 17min when it froze.

[tucansam]

Up 18 mins this time, then hung.

 

Can I roll back to the previous version of unraid without destroying everything?  Hopefully its better than a Windows rollback..... 

ffs2-diagnostics-20220530-1830.zip

Edited May 31, 2022 by tucansam

[tucansam]

Well its been up for a while, knock on wood.  But copying 80MB of files to a share took 22 minutes......

 

[JorgeB]

If it keeps crashing enable the syslog server and post that after a crash.

[tucansam]

Froze again overnight.  I set up the syslog server, but the last time that was suggested, for another problem, it generated zero useful info.

 

I've re-started a file copy from a local host to unraid that died during the freeze.

 

355KB/s.

 

 

[tucansam]

Came home to a crashed server.

 

--

 

pi@pihole:/var/log $ more ffs2.log
May 31 04:32:29 ffs2 rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="15169" x-info="https://www.rsyslog.com"] start
May 31 04:36:00 ffs2 root: Fix Common Problems Version 2022.04.14
May 31 04:43:48 ffs2 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin update fix.common.problems.plg
May 31 04:43:48 ffs2 root: plugin: creating: /boot/config/plugins/fix.common.problems/fix.common.problems-2022.05.30-x86_64-1.txz - downloading from URL https://raw.githubusercontent.com/Sq
uidly271/fix.common.problems/master/archive/fix.common.problems-2022.05.30-x86_64-1.txz
May 31 04:43:48 ffs2 root: plugin: checking: /boot/config/plugins/fix.common.problems/fix.common.problems-2022.05.30-x86_64-1.txz - MD5
May 31 04:43:48 ffs2 root: plugin: running: /boot/config/plugins/fix.common.problems/fix.common.problems-2022.05.30-x86_64-1.txz
May 31 04:43:49 ffs2 root: plugin: running: anonymous
May 31 07:00:11 ffs2 crond[1022]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
May 31 07:07:46 ffs2 emhttpd: spinning down /dev/sdc
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdg
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdw
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sde
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdu
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdb
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdx
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdv
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdo
May 31 07:07:50 ffs2 emhttpd: spinning down /dev/sdi
May 31 07:07:54 ffs2 emhttpd: spinning down /dev/sdm
May 31 07:07:54 ffs2 emhttpd: spinning down /dev/sdj
May 31 07:07:54 ffs2 emhttpd: spinning down /dev/sdd
May 31 07:07:54 ffs2 emhttpd: spinning down /dev/sdf
May 31 07:07:54 ffs2 emhttpd: spinning down /dev/sds
May 31 07:07:58 ffs2 emhttpd: spinning down /dev/sdk
May 31 07:07:58 ffs2 emhttpd: spinning down /dev/sdh
May 31 07:07:58 ffs2 emhttpd: spinning down /dev/sdl
May 31 07:34:06 ffs2 emhttpd: read SMART /dev/sdh
May 31 10:14:01 ffs2 emhttpd: spinning down /dev/sdt
May 31 10:14:01 ffs2 emhttpd: spinning down /dev/sdr
May 31 10:14:01 ffs2 emhttpd: spinning down /dev/sdq
May 31 10:26:25 ffs2 emhttpd: read SMART /dev/sdt
May 31 10:26:25 ffs2 emhttpd: read SMART /dev/sdr
May 31 10:26:25 ffs2 emhttpd: read SMART /dev/sdq
May 31 10:52:49 ffs2 emhttpd: spinning down /dev/sdh
May 31 12:04:09 ffs2 emhttpd: read SMART /dev/sds
May 31 12:19:30 ffs2 emhttpd: read SMART /dev/sde
pi@pihole:/var/log $
 

[tucansam]

That's the only stuff in the syslog.

 

I'm going to look into downgrading back to 6.9.2.

 

 

[JorgeB]

Unfortunately there's nothing relevant logged, you can try is to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes could be a hardware problem, or some compatibly issue with the newer kernel, if it doesn't start turning on the other services one by one.

[tucansam]

6.9.2 is back and now has going on 24 hours of uptime.

 

Which is sad because I may never be able to upgrade the unraid version again.

[tucansam]

Going on two days of uptime.

 

So, incompatible hardware with the new kernel, then???

[JorgeB]

Possibly, also do you use the iGPU? I believe it loads the driver by default with v6.10, you can blacklisted if you don't need it, it has been known to cause some issues.

[tucansam]

Do I use the integrated GPU, like for Plex? No, not that I know of.  I'm not sure how to black list it.

 

I'm on an i7-2600 and have an i5-9400 + 300 series MB on the way.  Will upgrade the system to more modern hardware and then try the unraid update again.  

[JorgeB]

8 hours ago, tucansam said:

I'm not sure how to black list it.

 

Add to the append line in syslinux.cfg, e.g:

 

append initrd=/bzroot modprobe.blacklist=i915

 

[SimonF]

9 hours ago, tucansam said:

Do I use the integrated GPU, like for Plex? No, not that I know of.  I'm not sure how to black list it.

 

I'm on an i7-2600 and have an i5-9400 + 300 series MB on the way.  Will upgrade the system to more modern hardware and then try the unraid update again.  

or from release notes

 

Linux Kernel

Upgraded to Linux 5.15.x LTS kernel which includes so-called Sequoia and Dirty Pipe vulnerability mitigations.

In-tree GPU drivers are now loaded by default if corresponding hardware is detected:

amdgpu

ast

i915

radeon

These drivers are required mostly for motherboard on-board graphics used in GUI boot mode.  Loading of a driver can be prohibited by creating the appropriate file named after the driver:

 

echo "blacklist i915" > /boot/config/modprobe.d/i915.conf

 

Alternately, the device can be isolated from Linux entirely via the System Devices page.  Note that in Unraid OS 6.9 releases the in-tree GPU drivers are blacklisted by default and to enable loading a driver you need to create an empty "conf" file.  After upgrading to Unraid OS 6.10 you may delete those files, or leave them as-is.  This change was made to greatly improve the Desktop GUI experience for new users.

[tucansam]

Thank you.

 

This explains why my console, which I have probably used ten time in 12 years, went from regular old text like we've had since the 80s to some high-resolution colorful screen config with fancy fonts.  I still didn't get a GUI, but rather just the typical shell, but it looked nice, I'll give ya that.

 

I'll wait til my new hardware gets here and plan an upgrade once its in place.  The little i7-2600 has served me well, but adding family members to Plex for remote streaming is showing the system's age a bit.

 

I am running a 16-port SATA controller and an expander to an external storage system.  I hope those drivers aren't the problem.