Unsure where to go. Unraid dockers were unresponsive, but I was able to ssh in, although VERY slow. Nothing seemed out of the ordinary so I issued a reboot command. The reboot got hung up on "Starting diagnostic collection". Having felt defeated, I pressed the reboot button on the computer. It just gave me a black screen from HDMI. Tried again, same thing.

 

It wasn't until I killed power to the outlet that I was finally able to get the server back up. Any ideas? Thanks!

syslog-192.168.1.65.log unraid-server-diagnostics-20241121-1423.zip

during shutdown unriad will have timeout (up to 15 min) to collet logs and other data to stop the array.

at grub boot run memtest.

@bmartino1thank you. I thought I had waited long enough but might have been a little early to call it. My biggest concern is why it's going unresponsive to begin with. I'll check memtest.

@bmartino1it passed memtest 3 times

I don't know then could be anything for faulty hardware to plugin / docker misconfiguration. 

@bmartino1anything in the logs give you a clue? It will suck but I think I will shut down my cloudflare tunnels, maybe I'm getting DDOS'd?

not really...

 

The portion of the system log provided shows a boot sequence with hardware details, BIOS RAM mappings, and kernel initialization steps. While these lines indicate the system is booting correctly, they don't immediately point to issues such as high CPU usage, resource contention, or network anomalies.

To diagnose possible DDOS attacks or high resource utilization from services like Cloudflare tunnels, we should look deeper into:

 

Network Logs - Check for unusual spikes in network traffic or repeated connection attempts.

Docker Logs - Examine for issues or anomalies in containerized services.

CPU and Memory Usage - Any logs indicating unusually high resource usage.

Other System Errors - Check for repetitive error messages or warnings

 

Let me review the logs again.

 

The docker logs you provided appear to show routine operations, such as loading plugins during startup. There's no immediate evidence of errors or issues causing high resource usage or unresponsiveness.

To continue diagnosing:

Check for Network Anomalies:

Look at logs from services or containers that use Cloudflare tunnels.

Inspect network traffic to see if there are unusual spikes or repeated connection attempts.

System Resource Usage:

Investigate logs related to memory or CPU contention. These could be in the syslog or performance monitoring logs.

 

I'm not seeing anything obvious or out of the norm....
 

The ifconfig output shows several network interfaces on the Unraid system:

br0 (bridge interface): Assigned to 192.168.1.65/24, likely the main IP for local access.

docker0: Docker's default bridge network with IP 172.17.0.1/16.

eth0 and eth1: Both are marked as UP, possibly bonded or used in different network setups.

Various veth interfaces: Virtual Ethernet interfaces, typically associated with Docker containers.

Observations:

Network Status: All critical interfaces (eth0, br0, docker0) are marked UP. This suggests the network is operational.

Cloudflare Tunnel: If your Cloudflare tunnel uses specific virtual interfaces (like veth), we should investigate logs from services using these interfaces.

 

Key Findings:

Unraid API Logs:

The API status is [running] and authenticated with a valid key.

Cloud services and mini-graphs are connected and marked as [ok] and [CONNECTED].

No errors or warnings in the API status report, which suggests proper functioning of Unraid's integration with My Servers and related services.

Network Hardware Details (ethtool):

Bond0:

Speed: 10000Mb/s (10 Gbps), Full Duplex.

Link detected: Yes.

Auto-negotiation: Off.

Eth0:

Supports multiple speeds, including 10000Mb/s, 5000Mb/s, 2500Mb/s, and 1000Mb/s in full duplex.

Current speed: 10000Mb/s, Full Duplex.

Auto-negotiation: On.

Observations:

The network hardware is functioning correctly, with the main interfaces operating at 10 Gbps.

The Cloudflare tunnel or network interruptions are unlikely due to hardware issues.

The API logs show no disconnection or service failure, so connectivity with Unraid's My Servers appears stable.

If you suspect external interference like a DDOS attack, we should check for:

Unusual spikes in network activity.

Errors or repetitive patterns in Docker container logs or syslog entries.

if you can get terminal access, run htop and see if their is something maxing your device's resources.

 

Sorry all i ahve is general torubleshooting steps to check:

 

When Unraid is mostly unresponsive, troubleshooting can involve a mix of reviewing logs, system monitoring, and isolating potential problem areas. Here's a structured approach:

 

1. Check Basic System Health

CPU and Memory Usage: Ensure the system isn't overwhelmed.

Review top.txt and memory.txt logs in the diagnostics.

Storage Health:

Check for full drives or failing disks using smart reports or disk utilization logs.

Networking: Confirm the system isn't being flooded with requests (e.g., potential DDOS).

 

2. Review Logs for Errors

Syslog (syslog.txt):

Look for errors or warnings, such as I/O errors, kernel panics, or other anomalies.

Docker Logs (docker.txt):

Verify containers aren't causing excessive load or memory leaks.

Unraid API Logs (unraid-api.txt):

Ensure no API disconnections or errors.

Networking (ethtool.txt, ifconfig.txt):

Confirm interfaces are stable with no frequent up/down transitions.

 

3. Test Isolated Services

Stop Docker: If Docker containers are suspected, stop them temporarily to see if the system becomes responsive.

Stop VMs: Virtual Machines can sometimes consume excessive resources or cause conflicts.

Disable Plugins: Certain plugins might cause unresponsiveness. Boot Unraid in Safe Mode to test without plugins.

 

4. Check System Hardware

Power Supply: Insufficient power or failing PSU can cause instability.

RAM Errors: Run a memory test to rule out bad RAM modules.

Cooling: High temperatures can throttle performance or crash the system.

 

5. Investigate Network Traffic

Monitor inbound and outbound traffic for unusual patterns.

Use the br0 interface or Cloudflare tunnel logs to detect possible DDOS attacks.

Limit exposure of Unraid services to the internet; disable unnecessary port forwarding or use a VPN.

 

6. Run Tools for More Insights

htop or iotop (CLI): To identify high CPU or disk I/O usage.

Netstat or Wireshark: Analyze network connections for anomalies.

 

7. Take Preventive Actions

Backup Important Data: Before making major changes, ensure critical data is backed up.

Update Unraid: Ensure the system is running the latest stable version.

Reboot Safely: If no critical tasks are running, reboot might temporarily resolve the issue.

@bmartino1 some developments. My plex and all other services were down again. I couldn't immediately connect to local IP so I thought disabling one of the NICs might at least let me access it. I was able to get onto the dashboard and pull diagnostics (CPU was pinned to 100% on the dashboard). Rather than reboot I tried HTOP and saw usage was mostly normal but still pinned on the dashboard. very odd (see pic). I tried stopping a couple dockers I thought might be the culprit like cloudflared but CF didn't stop. I then stopped docker and started again and it resolved the issue w/o a reboot. So there is either a docker or something in the docker service tying it up.

 

Happy thanksgiving!

 

 

 

 

unraid-server-diagnostics-20241128-AFTER DOCKER RESTART.zip unraid-server-diagnostics-20241128-BEFORE DOCKER RESTART.zip

pics.zip

Edited November 29, 20241 yr by spdelope
change pictures to zip file

htop shows current metrics.
Dashboard shows prolonged metrics.

Unriads webUI can look and be pegged due to ram usage as the ram disk buffer is processed and exchanged.

It apears that you have a few dockers that are very resource intensive. per pictures.

lets dig into the diag and see if we see anything unusal.
I run: https://forums.unraid.net/topic/178033-bmartino1-user-scripts/#findComment-1478662

and other to scan thorugh the daig files... its a pain when people edit and change syslog and other data. I get it privatize security...
^Posted as example as i'm looking for keywords...

So here are some Repeated Issues in Both Log:

Multiple entries like:

rsyslogd: omfwd/udp: socket 8: sendto() error: Network is unreachable

the remote syslog up and working? 

depending on your bios and board looks like pcie resizeable bar needs adjusted enabled...
PCI BAR Assignment Failures

Quote

pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x001c0000 64bit]
pci 0000:02:00.0: BAR 7: failed to assign [mem size 0x00040000 64bit]
 

 

Docker or System Services Issues:

The pinned CPU on the dashboard not matching htop suggests a bug in reporting, potentially in Unraid's GUI or Docker interactions

*unraid refresh is behid the actual live veiw due to mutiple docker interaction at full request 100% before htop show the actual resource they got.

 

Some side Suggested Steps to Troubleshoot

 

Networking Configuration:

Review your NIC configuration, especially the one that was disabled. Ensure that all NICs have valid IP configurations.

Check your rsyslog settings to ensure it's not trying to forward logs to an unreachable network or host.

 

PCI Device Conflicts:

Examine the hardware associated with PCI devices mentioned in the logs. Consider updating your motherboard BIOS or checking for compatibility issues with installed hardware.

 

Docker Investigation:

Since stopping and restarting Docker resolved the issue:

Check logs for individual containers (e.g., Cloudflare and others).

Ensure no containers are over-utilizing resources or misconfigured.

docker ps
docker inspect <contianer>

 

System Updates:

Verify if you're using the latest stable version of Unraid.

Update Docker, plugins, and other related components.

Unraid had an update 6.12.14

 

The Docker logs from both "before" and "after" the restart do not contain any explicit warnings, errors, or notable issues based on filtering for terms like "error," "failed," or "warning."

This indicates that the root cause might not be logged directly by Docker itself but could be due to resource contention or other underlying issues affecting Docker indirectly.

so we will need to get data taht the daig doent' collect using docker commands.
https://docs.docker.com/reference/cli/docker/inspect/