[6.9.x - 6.11.x] intel i915 module causing system hangs with no report in syslog (not alder lake)

When these system hangs happen for me (started on 6.9.2 and have continued with 6.10.0 RC), there is nothing meaningful in the syslog around the time of the hang but this is sometimes reported in the IPMI log.

 

 

An OS Stop/Shutdown sounds like a kernel/driver issue to me.  It's like the rest of the system is working but the OS just decided to shutdown.

 

FWIW, I have never seen these shutdowns as a result of hardware transcoding.

Edited January 18, 2022 by Hoopster

Another thread detailing similar issues.
 

 

Is there any extra info the devs need for this? I assume you are going to set up a test intel system with the i915 module loaded to see if you can replicate the fault?

Edit:

Looks like my issue was caused by a faulty CPU. Replaced RAM/Mobo and PSU with known good hardware and it still persisted. Intel agreed to RMA the 11500 I had an issued me with a refund. Purchased a 11700k as a replacement and it's been working fine ever since.

My CPU would only ever crash my system when transcoding, p95/furmark were stable, which led me to the confusion of a software level issue.

Edited January 5, 2022 by CaptainPeech

I have been using Unraid since 6.1.8, and I believe I've been experiencing a similar issue with 6.9.2

This year in June I upgraded my motherboard & CPU (built in 2015: Asrock H97 Pro4S, Xeon E3-1231v3 24GB 1600 ram, upgraded this year to: Asrock Z170S, i5-7600k, 16GB 3200 ram) and I had upgraded specifically for the Intel QuickSync hardware transcoding. On the Xeon hardware, the server was perfectly stable, but it was at its limit in terms of software transcoding, hence the upgrade.

After the upgrade, I started noticing frequent crashes occurring more & more frequently -- at first once per month, then once per week, and now every few days. 

 

I thought I had successfully troubleshooted (troubleshot?) the issue when I noticed the NTP server was causing a kernel panic. I found this after I set up a syslog server on a raspberry pi I had lying around. Unfortunately, even after disabling NTP entirely, the crashes persisted, but now there was no trace of any issues in the syslog. My raspberry pi is also a pihole, so I am able to see when the DNS requests cease, and that the server crashes most frequently in the early hours of the morning, but beyond that I'm stumped. 

 

This morning, I found the threads posted by @Tristankin, and rather than downgrade back to 6.8.3, I decided to experiment by commenting out the

# modprobe i915

line in my go file. 

 

I'm hoping that the server continues to function, and I think I'll be able to make do with software transcoding for the time being, but I would like very much for this issue to be resolved. Happy to provide any details as necessary. 

Edited December 27, 2021 by bearcat2004

Changed Status to Open

Changed Priority to Minor

UPDATE:  I rolled back to 6.9.2 from 6.10.0 rc2 and still had random system hangs.  16 days ago I did the following and have not had an issue since then:

• recreated i915.conf with the 'touch' method
• uninstalled the Intel GPU Top plugin
• uninstalled the GPU Statistics plugin
• enabled Turbo Boost in the Tips and Tweaks plugin (it had been disabled because heavy sustained workloads had been causing crashes with Turbo Boost)

 

UPDATE: A few days ago, I also uninstalled the CoreFreq  plugin.  I had not had problems with it installed but I see that it has been recommended to others with server lockups to uninstall that plugin.

 

I am not declaring victory yet nor do I think the above is necessarily a solution.  This just happens to be the longest I have ever gone without a system hang since I started having the problems last July.

 

- Over five months of running 6.92 and 6.10.x with no hangs and many QSV transcodes.  The only thing that appears to cause a hang is when Turbo Boost is enabled and there is high CPU load.  This only happened again after upgrading to 6.10.0.  I have Turbo Boost off for now.

Edited June 15, 2022 by Hoopster

 

I've had two crashes -- about one per week -- since 12/27. Admittedly, despite my previous post where I had mentioned I could make do with software transcoding, I had pretty quickly re-enabled hardware transcoding after hitting a limit with my patience while watching a 4K HDR movie on Plex, and forgot to disable it when I was done. 

 

I am going to try uninstalling the Intel GPU Top & GPU Statistics plugins as well to see if that changes things. I didn't realize that they automatically configure the i915 kernel driver when they're installed. I suppose I'll have to reboot and reconfigure my Plex docker to reconfigure the kernel driver/module.

 

2 hours ago, Hoopster said:

• recreated i915.conf with the 'touch' method

Where would I create this file if I were to create this file the same way?

 

Lastly, after my most recent crash, I did see one odd line in my logs when the system came back online, something i915/GPU-related that I hadn't noticed before:

Jan  9 23:38:15 Lighthouse kernel: i915 0000:00:02.0: [drm] HPD interrupt storm detected on connector HDMI-A-1: switching from hotplug detection to polling

The only thing I could think of to try and remedy this was to connect a HDMI dummy device to the motherboard. I had an old HDMI-to-VGA converter lying around, so I connected that as well to see if that might have any effect. 

 

Still wish we had answers to this puzzle!

Edited January 11, 2022 by bearcat2004

Just now, bearcat2004 said:

Where would I create this file if I were to create this file the same way?

From the terminal type 'touch /boot/config/modprobe.d/i915.conf'

 

Intel GPU Top tries to load i915 as well and I thought it might be interfering so I removed it.

Ok, so I loaded the i915 kernel driver using the method from @Hoopster above, and interestingly the 'HPD interrupt storm detected' line in my previous post is absent from the system log.

I googled that error and it led to a bunch of bug reports from 2018 for the Linux kernel wherein the X server would freeze after loading the i915 module (sounds familiar).

Some examples:

https://issues.hyperbola.info/index.php?do=details&task_id=741

https://bugs.freedesktop.org/show_bug.cgi?id=106675

 

I think it's a good sign that this error has not appeared in my logs thus far.  

 

I hope that I get at least 16 days of stable uptime too!

 

Hi, I appear to have the same issue.

@Hoopster What did you put in the i915.conf file ?

10 minutes ago, Arkhad said:

Hi, I appear to have the same issue.

@Hoopster What did you put in the i915.conf file ?

Nothing, the touch command will create an empty file.

Unraid checks on startup if a file is present or not. The content is irrelevant.

Well Hoopster method doesn't work for me. Server still crashes shortely after the array is up. The weird thing is the crash doesn't happen if the parity check is running.

Edited January 13, 2022 by Arkhad

I believe I'm experiencing this same issue.  I have been discussing it in the 12th gen Alder Lake thread here:

I touched the i915.conf file, have Intel GPU Top and GPU statistics installed. The hangs will lockup the system entirely, but the terminal is still accessible direct on the machine itself.  Shutdown commands don't shut the system down. Syslog says nothing.  Crashes would happen whether people were streaming or not.  Sometimes several times an hour, others 4-6 hours apart.  I'm on 6.10.0 RC2

The fix for me was to remove the /dev/dri device from the Plex container.  I still have the i915.conf, Intel GPU Top, and GPU Statistics installed, but the crashes stopped immediate when I removed /dev/dri from the container.

I was about to test if pinning the container to the performance cores would help, but based on the people experiencing the same issue here it appears the issue is not limited to 12th gen Intel CPUs

Edited January 15, 2022 by Earendur

3 hours ago, Earendur said:

the crashes stopped immediate when I removed /dev/dri from the container.

When you remove /dev/dri from the plex container, you switch to software transcoding though, right? Or are you able to confirm it is using hardware transcoding?

I’ve had this issue when transcoding, when not transcoding, when running handbrake, and when saving a simple change to the app running in a docker.

 

Asus Z690-P D4 (latest BIOS)

i5-12600k

4x 16GB DDR4-3200 CL16

4 hours ago, muzo178 said:

When you remove /dev/dri from the plex container, you switch to software transcoding though, right? Or are you able to confirm it is using hardware transcoding?

Originally, I simply turned off the hardware transcoding in Plex itself, but the crashes would still occur, which is consistent with the reported problem here - that the drivers don't even need to be actively running to cause the issue.

I tried a number of different troubleshooting steps to figure out the issue, from Memtest to see if it was a ram issue, to updating the BIOS.  It wasn't until I deleted the /dev/dri device from the docker container configuration - effectively making it impossible for plex to access the drivers for hardware transcoding - that the crashes stopped.

There is definitely an instability in the iGPU drivers.  I tested the hardware transcoding and was using it for several hours in some cases before it would crash the system.  Other times, it would happen 3 times in a hour, and when no one was streaming from the plex server at all. 

I knew the hardware transcoding was working because plex dashboard was showing (hw) and so was tautulli.  Since the removal of the device, no transcoding operations are showing (hw).

Edited January 15, 2022 by Earendur

well it seems i also got this issue..

i miss 2 hours as you can see in the log i write to app pool:

 

Jan 18 08:02:41 Plex emhttpd: read SMART /dev/sdg
Jan 18 08:31:09 Plex webGUI: Successful login user root from 
Jan 18 10:02:17 Plex kernel: mdcmd (36): set md_write_method 1
Jan 18 10:02:17 Plex kernel: 

 

so the only fix seems to be disabling the i915 in total?

I have same program.

 

Motherboard is asrcock j3455-itx.

Looks like there are lots of issues with i915 driver in the 5.x kernels, not sure where there will be a window for a stable one that happens to align with one of the unriad releases.

https://askubuntu.com/questions/1288395/system-is-unusable-after-upgrade-to-20-10-i915-gpu-hang

 

https://gitlab.freedesktop.org/drm/intel/-/issues/4530

Devs, is the plan to wait till there is a stable kernel? Or a patched one?

27 minutes ago, Tristankin said:

Devs, is the plan to wait till there is a stable kernel? Or a patched one?

 

I did some research some time ago and the i915 driver is very troublesome.  There has been some work done in 6.10 with this, but there just isn't a lot that Limetech can do since it appears to be a driver issue,  I've been using it on 6.10 and have not had any issues.  On 6.10, the driver is loaded by Unraid and you don't need to put it in your go file.

 

Of course this is the problem with the i915 driver.  Certain things work for some, and not for others.  It's all pretty hit and miss.

 

There was recently an issue with Plex not handling the driver correctly, but they released an updated version that fixed the problem with Plex.

 

So I can't say it has been definitely fixed, but it has been worked on.

Once 6.10 is released, let's revisit this.  6.10rc3 (not released yet) is currently on Linux Kernel 5.15.15.  This latest Kernel may see this fixed.  Feel free to PM me if I don't get back to this issue.

Sounds reasonable.  Does rc3 have any release timeframe, other than "when it's ready"?

Unfortunately, I have no information about the release of rc3.

When rc3 is released do the following before rebooting:

• Remove any references to i915 from your go file.
• Remove any i915 files from the /flash/modprobe.d folder.

 

The driver is loaded by Unraid without you having to do anything.  I wanted to give you a heads up because you would probably upgrade before I could post here.