Unresponsive server - no Web UI, no SSH, no Ping

[PLAY3R]

Hey all.

 

Been having an issue off and on for a bit where the server will get into a "loop" and i have to cold boot it, 3 times so far, not happy doing this, but when i have no access at all ( even the GUI on the servers VGA is frozen, there is not much more to do )

 

Happened again tonight, with a twist, it came out of it. after failing to figure it out the first 2 times, i left the Syslog window open, and watched it till it happened again, tonight is when it decided to "act up". You can always tell, because the server starts acting like its under some load and the fans kick in to cool it all down. After about 3-4 min, everything came back on its own, shares, GUI, Ping etc. so i quickly downloaded the Diagnostics and came here, i also have screenshots of the log as this was happening, if it helps.

 

Anymore information, please ask'

 

Edit #1: Only errors Fix Common Problems is finding are "macvlan call traces found" can this be the cause?

 

blueberry-diagnostics-20230615-1907.zip

Edited June 15, 2023 by PLAY3R

[JorgeB]

9 hours ago, PLAY3R said:

Problems is finding are "macvlan call traces found" can this be the cause?

It can, try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

You also need to check filesystem for disk1.

[PLAY3R]

Running the check now, I use macvlan because i use custom MACs on some docker containers, if i switch to ipvlan, i cannot use custom MACs, i'll try that as a second option

 

Thanks so far though!

[JorgeB]

26 minutes ago, PLAY3R said:

if i switch to ipvlan, i cannot use custom MACs

You still can.

[PLAY3R]

Thanks for the reply, finally have time to look into this. All my "critical" data is backed up and the drive removed, checked disk 1 file system, please see below. I know two of the disks have CRC errors (one per drive) but they have been like that for months without issue, this locking up started almost a month ago. It has locked up 2 more times since my post.

 

I have also done some searching and not found how to use custom MAC addresses with ipvlan, can you explain more please.

[itimpi]

You need to remove the -n option or nothing will be fixed.  If xfs_repair asks for it use the -L option.

[PLAY3R]

Thought so, but wasn't sure. Below is the output without options, and below that, it ran again, from what i see, it fixed it?

 

Phase 1 - find and verify superblock...

Phase 2 - using internal log

        - zero log...

        - scan filesystem freespace and inode maps...

        - found root inode chunk

Phase 3 - for each AG...

        - scan and clear agi unlinked lists...

        - process known inodes and perform inode discovery...

        - agno = 0

        - agno = 1

        - agno = 2

        - agno = 3

bad CRC for inode 7108726930

bad CRC for inode 7108726930, will rewrite

cleared inode 7108726930

        - agno = 4

        - agno = 5

        - process newly discovered inodes...

Phase 4 - check for duplicate blocks...

        - setting up duplicate extent list...

        - check for inodes claiming duplicate blocks...

        - agno = 0

        - agno = 1

        - agno = 5

        - agno = 2

        - agno = 3

        - agno = 4

clearing reflink flag on inodes when possible

Phase 5 - rebuild AG headers and trees...

        - reset superblock...

Phase 6 - check inode connectivity...

        - resetting contents of realtime bitmap and summary inodes

        - traversing filesystem ...

        - traversal finished ...

        - moving disconnected inodes to lost+found ...

Phase 7 - verify and correct link counts...

done

 

------------------------------------------------------------------------------------------------------------------------------------------------------------

 

Phase 1 - find and verify superblock...

Phase 2 - using internal log

        - zero log...

        - scan filesystem freespace and inode maps...

        - found root inode chunk

Phase 3 - for each AG...

        - scan and clear agi unlinked lists...

        - process known inodes and perform inode discovery...

        - agno = 0

        - agno = 1

        - agno = 2

        - agno = 3

        - agno = 4

        - agno = 5

        - process newly discovered inodes...

Phase 4 - check for duplicate blocks...

        - setting up duplicate extent list...

        - check for inodes claiming duplicate blocks...

        - agno = 0

        - agno = 4

        - agno = 1

        - agno = 3

        - agno = 2

        - agno = 5

Phase 5 - rebuild AG headers and trees...

        - reset superblock...

Phase 6 - check inode connectivity...

        - resetting contents of realtime bitmap and summary inodes

        - traversing filesystem ...

        - traversal finished ...

        - moving disconnected inodes to lost+found ...

Phase 7 - verify and correct link counts...

done

[JorgeB]

Filesystems should be fixed now.