Extreme instability since upgrade to 6.12.2

[BrokenHalo]

I've upgraded to Unraid 6.12.2 a few days ago and since then I'm having severe stability issues, bordering on the system being almost unusuable.

Since I'm not sure if this is only related to the update or another issue I hope for some help.

The only thing I have been changing along with the upgrade was creating a new ZFS cache pool insted of the old btrfs one.

Unraid locks up multiple times a day with different behaviors, sometimes its a kernel panic, sometimes the web UI is not reachable anymore (in general the UI is very slow) but every time a hard reset is needed.

 

Other stuff I've noticed:

• Docker service fails to start every other time I start the array (docker image is located on the ZFS-cache pool)
  I've removed the docker image and created everything from scratch multipe times now, I even switched from XFS to Directory for the docker image
• while the web UI is sometimes completely unreachable, some dockers can still be reached through the Cludflaredd-Tunnel
• even when idle, sooner or later the system will lock up, even in safe mode, even with everything stopped and only the array running

 

What I've tried so far:

• running system in safe mode / de-activating every plugin/docker
• checking every cable connection on the disk and making sure all cache-SSDs are connected to the HBA card and not the internal SATA-ports
• reducing RAM speeds to 1886 MT/s and only using two sticks of RAM instead of four
• making sure C-states are de-activated (as per this topic)

 

I've attached the most recent syslog and diagnostics. Hope someone can point me in the right direction.

 

Thanks a lot in advance!

nrc-syslog-20230709-1504.zip nrc-diagnostics-20230709-1700.zip

[ashman70]

Did you run the upgrade assistant before upgrading?

What version did you upgrade from?

Are you able to roll back to the version you upgraded from?

I have not looked at your logs yet.

[BrokenHalo]

1 hour ago, ashman70 said:

Did you run the upgrade assistant before upgrading?

What version did you upgrade from?

Are you able to roll back to the version you upgraded from?

I have not looked at your logs yet.

 

Yes I did, I came from 6.11.5. I'm afraid rolling back will not be easy due to the ZFS cache pool that has not existed before... 

[ashman70]

Yes you would have to move everything from the cache back to the array, reformat the cache and move it back,

 

Was everything stable under 6.11.5?

What are you doing on the server, dockers, vm's?

[BrokenHalo]

51 minutes ago, ashman70 said:

Was everything stable under 6.11.5?

What are you doing on the server, dockers, vm's?

 

Let's say it wasn't that bad. Ryzen 1st and 2nd Gen really don't go well with Unraid for me but at least it was usable. I'm doing nothing special, Emby and selfhosted cloud stuff. Everything else is put on hiatus until I get things stable again.

Edited July 9, 2023 by BrokenHalo

[ashman70]

When you ran the upgrade assistant what did it tell you?

From your logs it appears you have nerd tools and several dynamix plugins installed, or did you remove these prior to upgrading?

[BrokenHalo]

Thanks for taking the time!

 

I've re-installed most plugins after the upgrade.  But since I have the same instability when running Unraid in Safe Mode, with all plugins being deactivated, I doubt it is plugin-related. 

Yesterday I removed Mover Tuning altogether since it caused some problems when booting.

[JorgeB]

Make sure this is done then enable the syslog server and post that after a crash.

[BrokenHalo]

9 minutes ago, JorgeB said:

Make sure this is done then enable the syslog server and post that after a crash.

 

Sorry if I miss something here but I literally linked to this thread in OP and attached log/diagnostics from after the last crash?

[JorgeB]

5 minutes ago, BrokenHalo said:

from after the last crash?

Those won't help since the log starts over after every reboot.