Hello all: my unraid web gui is un responsive. I can log in one time which takes me to Main. After that I cannot do anything, cannot go to different menu like Docker, Dashboard. I can go to Settings and tools to get the diagnostics. Can you point me in the right direction? I have attached the diagnostics from the unraid.

 

Thank you.

unraid-diagnostics-20240529-0437.zip

The syslog shows that you have corruption on disk2:

May 28 08:42:26 unraid kernel: XFS (md2p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x30e48b5eb dinode
May 28 08:42:26 unraid kernel: XFS (md2p1): Unmount and run xfs_repair

If you can you should run a check filesystem on that drive.

 

It may also be a by-product of an underlying hardware issue such as RAM playing up.

I am trying to stop the array in maintenance mode from GUI, but the proceed doesn't do anything. Is there option to stop it via SSH?

 

I was able to go to maintenance mode by rebooting the system to safe mode. I ran >> xfs_repair -v /dev/md2p1. Looks like it cleared reflink flags on inode during Phase 4. Not sure what it meant but the diagnostic in syslog no longer shows any issue. I will keep the system on for few days and see if the issue revisits. I will update again in few days with the progress. Thank you @itimpi

When you rebooted in normal mode did the drive mount OK?    Was there a lost+found folder created which is where the repair process puts files for which it does not know the name.

Yes the drive mounted OK. There is no lost+found folder in the root of boot. Is that the location I should check?

Just now, 062bel313 said:

Is that the location I should check?

No - it would be on the drive you repaired (and show up as a share).    Not having it is a good sign as it means that there was probably no data loss at all.

I checked and disk 2 doesn't have any Lost+Found. Web UI is accessible so far. Some how Parity check ran, may be because there was some issue in reboot. Parity check going about 230MB/sec will take few more hours. Hopefully things will get to normal. 🙂

Now the xfs corruption shifted to disk 1 😢. Going to do same thing for disk 1.

In case you missed it there's also fs corruption on disk2.

I noticed that, so I ran xfs_repair for both disk 2 before and still see that. I could not find lost+found in disk root directory.

 

Am I using the right syntax: 

xfs_repair -V -L /dev/md2p1

xfs_repair -V -L /dev/md1p1

 

My server crashed this morning. When checking in the UDM (router), I was not able to see unRAID IP in the clients, but server was running; running in the sense that it had power but was not able to access it via SSH and WebUI. so had to force restart. Not sure what is causing this crash. Here is the diagnostics after the restart, if you can find something.

unraid-diagnostics-20240601-1108.zip

Edited June 1, 20242 yr by 062bel313

16 hours ago, 062bel313 said:

xfs_repair -V -L /dev/md2p1

Should be first

xfs_repair /dev/md2p1

Only use -L if it asks for it

 

16 hours ago, 062bel313 said:

Not sure what is causing this crash. Here is the diagnostics after the restart, if you can find something.

Syslog starts over after every boot, you can enable the syslog server and post that after a crash to see if it catches something.

Unfortunately, when I was planning to enable syslog server, I noticed that the server is not accessible from web UI as well as ssh. I did hard reboot and unraid is seen in the DHCP clients of my router but cannot access webui and ssh. Looks like the server is dead. 😭 What should be my next step, any guidance?

Edited June 4, 20242 yr by 062bel313
wrong information

2 hours ago, 062bel313 said:

Looks like the server is dead. 😭 What should be my next step, any guidance?

Do you still get the CLI or is it dead dead?

no cli. cannot ssh to the machine. it is headless, may be try to hook the monitor?

5 minutes ago, 062bel313 said:

may be try to hook the monitor?

Yep.

in the monitor i see:

unraid login: sed: couldn’t open temporary file /boot/config/sedwmnHkH: Read-only file system

 

but this time i was able to login the ui, and i see red notification

USB flash drive failure: time

Alert[UNRAID] - USB drive is not read-write Cruzer_Fit(sda)

 

does this mean my usb is failing?

 

I will enable syslog server this time.

Did a chkdsk of a boot drive on a windows machine fixed the read-only issue. I was also able to enable the syslog mirror on the usb drive. Hopefully I will be able to share the diagnostic now if server crashes.

I am back, but server hasn't crashed 🤩..... yet. So I am in an aggressive monitoring mode. I am checking the syslog to see if I encounter any lines with red color or mention of error. I see there is some lines with kernel tainted in CPU 3 with some proccess ID lsof. I have no idea what it means, a quick google and my speculation suggests that the kernel is not in pristine condition. Can you check with your experienced eyes if you see any warnings or errors, that I need to worry? The diagnostics are attached. 

unraid-diagnostics-20240605-2147.zip

There's a call trace, start by running memtest, or re-test with just one stick of RAM, if the same try the other one, that will basically rule out a RAM issue.

Did a memtest from unraid boot option with both RAM. passed the test for 6 passes.

Since memtest is only definitive if it finds errors, 

On 6/6/2024 at 9:23 AM, JorgeB said:

re-test with just one stick of RAM, if the same try the other one, that will basically rule out a RAM issue.

 

As suggested, I did memtest. I have two RAM sticks. I did the test again for individual RAMs and all tests are passed. It took long. What I did was after 6 successfull passes I stopped and exchanged the RAM and did memtest again for 6 passes. Is that the process, or should I let it continue? I don't know how many passes it needs to confirm that RAMs are fine. After that I rebooted the machine to unRAID and I see there are trace calls (like before) as attached. Any suggestions?

The point is that running memtest is only definitive if it finds errors.   What was suggested was that you run the server with less RAM sticks plugged in to see if you still get errors. not that you run memtest with less sticks plugged in.

Oh got it, I will give it another try with less RAM and run unraid for a while.