Unraid problems

[Kjetil]

Hello everyone,

 

First, I just want to say that this is my first forum post as I am not a tech guy. I wrote to Unraid support but every time they refer me here.

I have searched the forums to see if others have posted similar problems but I have not found help there - maybe I am searching using the

wrong terminology (also I don´t understand some of the vocabulary here, its too technical for me).

 

My problems:

 

Had Unraid for 3-4 years, worked very good until now. My problems started a few days ago, the first thing that happened was that I lost my 

Plex server. Worked fine one evening, the next day there was no contact to the web UI (:32400). I had not done any changes in Unraid, there

was suddenly just no contact. And I found no way of correcting this.

 

Then the server became unresponsive and I had to do a dirty closedown, and boot it up again. Unraid then did a full parity check and came

back with zero errors. But still no Plex. The day after (which is yesterday) there was a software update. I hoped this would help bring Plex

back so I updated the software. After this things have gone for the worse. Now I only get IP from Unraid upon boot up. I can log in and access

the Unraid menues, but if I try to do a diagnostic it just says something like "it might take a few minutes" - and then nothing happens for hours.

 

When I try to refresh the website (to access the ip adress) it then says "cant connect". And using the FING app (to see IP adresses on my

network) tells me the connection to the Unraid is lost (the IP adress of the Unraid has disappeared). 

 

Only way to get it back is a dirty closedown and start it up again. Which I already done once before, and dont want to do again.

 

I made diagnostics the day after I first lost Plex, thats the only diagnostic I have available, not sure if it will help but I add it here anyway.

As it seems I cant get a diagnostic from Unraid now.

 

So I am lost, and dont know what to do. Should I just let the Unraid be powered up and wait ? Who knows whats its doing now... OR should I do another

dirty closedown ? 

 

And how to proceed from here ?

 

Any help and tip is much appreciated, I am at a loss and quite nervous as the Unraid is 28 discs and about 100TB.

Please write as detailed explanation as possible as I am not tech savvy or familiar to programming language.

 

Kind regards, Kjetil

 

 

federalreserve-diagnostics-20191209-1901.zip federalreserve-syslog-20191209-1832.zip Log for: PlexMediaServer.pdf

[Frank1940]

Do you have a monitor and keyboard attached to your server?  (connect them if you do not.)  Log into the server using   root    for the user name and your Unraid password when prompted for it.  At the command line type:

diagnostics 

That should write a current Diagnostics file to the    logs    directory/folder of your Unraid boot drive.  Next, you will type the following command:

poweroff

That will safety shut your server down.  You can then remove the boot drive, put it in your PC and upload that Diagnostics file in a new post.  (Please never go back and modify (or add to) an earlier post.  If you do, that new (or changed) information will probably never be read by those trying to help you!)  You can then plug the boot drive into your Unraid server and restart it. 

[trurl]

While we wait to see if you can get additional diagnostics I will work with what you have here.

 

One thing puzzling in your diagnostics but probably unrelated to any problem. Why is your cache and only cache reiserFS?

 

The syslog you attached separately is earlier than the one in the diagnostics, and it ends with a dump so possibly a crash. Then the syslog in the diagnostics are after rebooting after that.

 

In future, please don't attach pdfs when a simple text file will do. The simple text file is easier to work with. I don't notice anything particularly meaningful in your plex log.

 

There are several things I would like to change about the way you have your user shares configured but we can save that for later.

 

You have way more disks than I want to wade through SMART reports for. We can look at those later also. Nothing obvious in the syslogs about disk problems.

 

Go ahead and try what Frank1940 suggested about getting more diagnostics. Possibly it won't work but maybe.

 

If you can't shutdown cleanly then you will have to do it the hard way.

 

Then when you reboot, select Memtest from the boot menu and let that run for a while.

 

 

[Kjetil]

Thank you so much for helping guys, highly appreciated !!

 

Frank1940: Clear instructions, thank you. I have found a monitor and keyboard (server has old PS/2 keyboard plug), its now doing the diagnostics. Looks like I will be able to powerdown safely after. Will update here when its finished.

 

trurl:  Thanks for kind suggestions. This got a bit technical for me though.  What is cache reiserFS ? I attached the files I got from Unraid unchanged, Unraid downloaded the reports this way. I dont know if there is a setup to have all files as text ? Shall I do a Memtest directly on the server (as I did with diagnostics and powerdown) ?

 

 

Edited December 12, 2019 by Kjetil

[itimpi]

1 hour ago, trurl said:

One thing puzzling in your diagnostics but probably unrelated to any problem. Why is your cache and only cache reiserFS

This could be because Unraid was originally on v5 when ReiserFS was the only option for the cache file system, and it was never changed when the system was upgraded to v6?

[trurl]

1 hour ago, Kjetil said:

I attached the files I got from Unraid unchanged, Unraid downloaded the reports this way. I dont know if there is a setup to have all files as text ? Shall I do a Memtest directly on the server (as I did with diagnostics and powerdown) ?

I was referring to your plex log which you attached as a pdf. Unraid does not produce any pdf files so I don't know what you did to get it in that form. The other attachments were fine and as expected.

 

Memtest is on the boot menu. You have to select it using the keyboard during the bootup process instead of booting Unraid.

[trurl]

40 minutes ago, itimpi said:

This could be because Unraid was originally on v5 when ReiserFS was the only option for the cache file system, and it was never changed when the system was upgraded to v6?

I guess it's possible but he doesn't seem to be aware there is a filesystem by that name, so how did his array disks wind up as XFS if they also came from V5?

[Kjetil]

Hi again to all. I was out of the house some hours and now back. The Unraid after 3-4 hours still says "Starting diagnostics collection...".

Has it collected diagnostics now for hours, or has it just not started yet ? And how should I proceed ?

[Frank1940]

IT should have finished within twenty minutes at the most.  Do a   <CTRL>-c  and see if you get a command line prompt back.  If you do, try the poweroff command.  The next step would be to push the power button/switch on the case for about ten seconds.  That will force an unclean shutdown of the server.  IF you are reluctant to do that, wait for a few hours  and see if anyone else has any other ideas.  

 

When you restart the server, go to   Settings   >>>   Network Service (section)   >>>   Syslog Server    In the     "Mirror syslog to flash:"    area change the option to 'Yes' and click on the "Apply" button.  That will save your syslog to the logs folder/directory on the flash drive.  Wait for the problem to happen again and up load that syslog file. 

[Kjetil]

CTRL-c gave me command line prompt back and I could enter poweroff command. On the screen it looks like it accepted this command, but nothing seems to be happening though. Looks like the commands are not executed. Its been 10 min already. Will let it sit a few more minutes and then do the unclean shutdown. Settings >>> Network service >>> Syslog Server.. do I find this directly on the server (on the screen connected directly to the Unraid) after restarting the server ? I have only used web browser access before, so I am not so familiar with the interface directly on the server. After rebooting it might get an IP adress and be available from the web browser. I now basically have problems using the Unraid at all, so a bit unsure what to do after

 

Also,

trurl asked me to do a Memtest. Should I do that before the Syslog server - mirror to flash?? He said let it run for a while.. if I let it run for 15 min, how do I end it and go to the Settings >>> Network service .... etc ? Sorry for asking very detailed questions, just was hoping not to do any more unclean shutdowns.

 

Could it be the usb stick boot drive be the problem ? 

 

[Frank1940]

5 minutes ago, Kjetil said:

trurl asked me to do a Memtest. Should I do that before the Syslog server - mirror to flash?? He said let it run for a while.. if I let it run for 15 min,

You should run the memtst first.  It should actually run for about 24 hours and a passed test is zero failures.  (I just looked and you have a Xeon CPU.  Do you have ECC memory as the builtin memtst will not test ECC memory.  In that case, wait for @trurl to get back to you.)

 

12 minutes ago, Kjetil said:

Settings >>> Network service >>> Syslog Server.. do I find this directly on the server

You find these options in the GUI on your PC.  I understand that your Unraid boots up normally and then has problems at a later time. 

[Kjetil]

How can I check if its ECC ? In the browser after restart ?

[Kjetil]

Do I automatically arrive at the boot menu where I find the Memtest, or do I need to do anything during startup ?

[Frank1940]

I do have one more thought, you have a massive amount number of hard drives-- As I Remember--- 31 of them at least.  What are you using as a power supply in this server and how old is it.  Power supplies do not last forever and, when they fail, they often casue wierd problems.  Collecting the information for the diagnostics file will spin up every one of those hard drives and that could be triggering this type of issue if the power supply is near its-end-of-life.

Edited December 12, 2019 by Frank1940

[Frank1940]

2 minutes ago, Kjetil said:

Do I automatically arrive at the boot menu where I find the Memtest

It is in the boot menu which is displayed during the boot process and displayed on the monitor screen and you 'arrow-down' to it while the menu is displayed and select it by hitting the <ENTER> key.

9 minutes ago, Kjetil said:

How can I check if its ECC ? In the browser after restart ?

There is a system information icon on the toolbar of the GUI.  I know it shows Memory but I am not sure if it will indicate if that memory is ECC.   (Most folks know what they purchased when they built their server  as ECC memory is priced at a premium.  Plus, the MB must also support it and the MB information is in that icon.  You will have either read your MB manual or google for the information.)

 

 

[Kjetil]

Its a Coraid server. I bought it 2nd hand, ready for use - I just added drives and a friend setup Unraid. Its a 2012 server. It has two powersupplies and two powercords. Its running power from a APC STM1000 UPS.

[Kjetil]

Should I open it and have a look at the mem/ram ? I think its 96gb ram.

 

[Frank1940]

Yes.  And while you are in there, clean it out.  You will need a compressed air and do it in a garage/outside unless you are unmarried (or looking to get divorced!)   😈

[Kjetil]

I opened it and its superclean inside. Its stored in a clean technical room of the house with other equiptment. No need for compressed air it seems. I blew inside it and no dust. The ram is 48gb Micron (4gb a piece, number MICRON 4GB 2Rx4 PC3-10600R-9-10-J0 MT36JSZF51272PZ-1G4H1HG). So it looks like its ECC ram (?) Does this mean we can not do a Memtest ?

[Kjetil]

So 12 x 4gb ram = 48gb

[Kjetil]

And if it cant do a Memtest, how do you guys think I should proceed ?

 

[trurl]

Memtest won't detect anything with that RAM since it is self-correcting. And I didn't see anything specific that made me think RAM was an issue.

 

Looks like your only plugin is preclear, and it is not up-to-date. Remove that plugin just to get it out of the picture.

 

Setup Syslog Server as directed and wait for another crash then post syslog.

[Kjetil]

Ok, thank you. Will do sir

[Kjetil]

Plugin removed, and server rebooted and I am back in. Changed syslog to "mirror to syslog to flash". All good there.

 

@trurl    Should I/we do something with the cache ReiserFS ? You also mentioned some changes to my user shares configuration ?

 

Also, my first problem was that I lost Plex. Should I ask about this issue on the Plex forum, or is it something you fellas would have any suggestions

about ?

 

Edited December 12, 2019 by Kjetil

[Squid]

Not necessarily your problem, but it looks like you've got 3x Marvel based HBA controllers.  These tend to be very problematic and ideally should get replaced with LSI based ones.