Unraid Crashes and Errors - Plex, Parity, and General Crashes

[BRS]

Hello.

 

I've been battling several errors for several months. I've completely reinstalled the dockers, attempted multiple versions of unraid to no avail. I've read the forums, double checked settings, and I'm at my wits end.

 

Error 1: Unraid crashes when running parity check. When I run parity check, the server will eventually prevent remote access through the UI. The server also completely freezes and refuses to allow interface of any kind - even with the command line. This happens every single time parity check is run.

 

Error 2: Unraid crashes randomly. Sometimes once every few minutes, sometimes it is stable for days and weeks at a time. I have set it up to log errors - but the logs do not show anything before and after these crashes. No warning or error messages are visible. During the crash, the UI freezes, and I'm unable to pull any error logs during the crash.

 

Error 3: After recovering from a crash state, Plex forgets the old server exists. The data on the array remains ok, but plex forgets that a server exists. All of my settings remain, but it will not connect or run properly unless I set up a new server via the webUI. I have attempted to rectify this by migrating app data to my cache. This has not resolved this issue. I have run memtest, fix common problems, etc.

 

Specs:

AMD Ryzen 5 3600 32 GB ECC DDR4

Asrock X570 Phantom Gaming 4

4x8 gig nasred hdd

1x samsung 500 gig NVME SSD

 

Dockers:

Krusader

plex - linux

valheim

 

Note: I have attached the syslog and plex log for the past few days. All of these errors have reoccured recently. Also note - my internet crashed during the night between 4/7 - early morning 4/8 (around 9:00 AM). 

 

The array itself is still stable, I have reinstalled all of the dockers several times. My last thought of what may be broken is that unraid itself may be corrupted.

  

syslog-192.168.50.50.log Log for_ plex.html

[trurl]

Go to Tools-Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread 

[BRS]

Diagnostics ZIP attached.

tower-diagnostics-20210409-1024.zip

[trurl]

 

[BRS]

I literally don't have an option to disable C6 in my Bios. I just upgraded it to the latest version to see if there would be an option. No difference in the lack of C6 menu options between version 1.70 and version 3.90 that I just installed. 

[trurl]

And what about the RAM part of that link I gave?

[BRS]

I'm not running any overclock - on either the ram or the CPU.

 

2666 MHZ for the memory speed - well within the advised frequency band.

 

This server and hardware was stable for well over a year. Crashes started happening periodically in May 2020. The server would be stable for 2-3 months, then crash. Now it is crashing more frequently. Sometimes daily.

 

Forgive my tone - I'm just frustrated. I'm glad the array itself is stable. 

[BRS]

Trurl, I also wanted to thank you for your time and assistance on this. I really do appreciate it.

[BRS]

I went ahead and dug through more BIOS settings. I think I may have identified a setting affecting the sleep mode. I was able to login and run parity check for a good 6-7 hours (normally, it stalls out after 15 minutes). Unfortunately, it crashed around hour 7, close to 70-80% completion.

 

I was getting a lot of print_req_error's - indicating bad sectors.

Apr 10 21:05:30 Tower kernel: print_req_error: I/O error, dev sda, sector 2006984
Apr 10 21:05:30 Tower kernel: print_req_error: I/O error, dev sda, sector 2006984
Apr 10 21:05:30 Tower kernel: Buffer I/O error on dev sda, logical block 250873, async page read

 

I'm sorry - I'm still a noob at this. How do I check which drive has the sector issues? I can try a different sata cable or sata connection on the M/B to see if that resolves, or even replace the drive if push comes to shove.

I also got a kernal bug error - when the parity check crashed.

Apr 10 21:05:33 Tower kernel: kernel BUG at drivers/md/unraid.c:477!
Apr 10 21:05:33 Tower kernel: invalid opcode: 0000 [#1] SMP NOPTI
Apr 10 21:05:33 Tower kernel: CPU: 9 PID: 9756 Comm: mdrecoveryd Tainted: G O 4.19.107-Unraid #1

 

Please see the attached log ZIP for further information.

tower-syslog-20210411-0322.zip

[JorgeB]

sda is a flash drive, but since it-s spamming the log disconnect it if not in use, enable this and then post that log and the complete diagnostics after a crash.

[BRS]

JorgeB,

 

Thanks for the response. I have the syslog enabled for a few months now (actually attached to the first post on this thread.) Note that I separated the files into 2.  syslog.old was logging for many months, and it was becoming a bit of a bear to sift through, so I renamed it to have it to create a fresh log to look through tackling the most recent issues. The second syslog has logged the most recent issues discussed in this thread occuring in the most recent weeks.

You're definitely right about the sda - that's the flash drive that I used to update the bios, I left it in, and it was spamming in the log. I have removed it.

 

So I have good news and bad news. Parity is still crashing the UI access - but before, it was crashing the entire server. Now I seem to still have terminal access despite the crash. It crashes the UI every time I run parity, and parity never finishes successfully. I just ran parity for almost 12 hours with no success.

 

Also - despite having successful access to terminal, when I run the powerdown script for a graceful shutdown, the command hangs up and never finishes. Odd. I have to hard reboot every time - which leads to a parity check, which leads to a hangup and crash for the UI. Quite infuriating.

 

 

 

 

 syslog-192.168.50.50-old.log

syslog-192.168.50.50.log

[BRS]

Also - here's the complete diagnostics file from the most recent crash:

 

tower-syslog-20210411-1605.zip

[trurl]

6 hours ago, BRS said:

the complete diagnostics file

You only posted syslog

[BRS]

My apologies. Here it is.

tower-diagnostics-20210411-1921.zip

[JorgeB]

Syslog is full of CPU NMI events, looks for a BIOS update, also try v6.9.x

[BRS]

JorgeB - thanks for the feedback.

 

I've tried to upgrade to v6.9.2 and v6.9.3, but hit a wall with the dockers.

 

The dockers fail to start and I get an error "unraid docker failed to start".

 

Every time I hit this wall, I revert, and the dockers are functional again.

 

Any ideas on Parity issues? That's the most consistent and universal error I can see literally every time.

[JorgeB]

4 hours ago, BRS said:

The dockers fail to start and I get an error "unraid docker failed to start".

Post diags after this.

[BRS]

Jorge B, 

 

Sorry for the delay on this. Here you go.

 

tower-diagnostics-20210414-1934.zip

syslog-192.168.50.50.log

[trurl]

Why is your docker.img path "/mnt/disk1/Share 1/docker.img" ? Probably the space in that path is breaking something. Can't imagine why you would want a user share named "Share 1" anyway.

 

The conventional place for docker.img is in the system user share at /mnt/user/system/docker/docker.img or similar. And system share should be configured to stay on cache.

 

Also, your appdata has some files on the array. Since docker isn't currently working and appdata is cache-prefer, you should be able to get those moved to cache by going to Main - Array Operation and click Move.

[BRS]

Trurl,

 

You're absolutely right. That did the trick and the update worked.

 

Share 1 was a legacy of my very first effort setting up the array - when I was clueless and making many errors. I have moved the docker.img to the system share address you specified. I also renamed Share 1 to Share_1. It's mostly filled with useless junk anyway - now that the docker img is out of there.

 

I set the system share to prefer cache, and I also disabled the dockers and I'm running mover now. Thanks for the advice.

 

I'm going to be out for a few days, but when I get back, I'll run parity check again and see if it works.

[BRS]

I just had another crash for no apparent reason. I did not run parity check, simply left the server running over the weekend. Here are the logs and diagnostics after this crash.                                                tower-diagnostics-20210420-1907.zip

syslog-192.168.50.50.log

Edited April 21, 2021 by BRS

[JorgeB]

Syslog is still full of call traces, this can be a hardware issue or your hardware doesn't get along with current Linux kernel.

[BRS]

JorgeB - Any ideas to what hardware may be an issue?

 

My hardware passes Memtest - even if I run it for 24+ hours.

My only thought at the moment is motherboard. I doubt that the CPU is the root cause. The power supply is also decent and essentially brand new.

 

I could troubleshoot by removing a ram stick at a time as well. Any suggestions would be useful.

[JorgeB]

10 minutes ago, BRS said:

JorgeB - Any ideas to what hardware may be an issue?

Not really, but board/CPU combo would be the first things I would try changing, there have been other Ryzen users with similar issues.