Hung with xfs_log_force error in syslog

[renobles]

The current status of my server is that I can't get in to the WebGUI or browse shares. I can, however, get into SSH/SCP.

 

So I'm running Unraid v6, but I'm not certain of the exact version. I don't believe that there were any system updates available, so that should make it 6.1.7 I think. It's on whitebox hardware - AMD FX-6300, MSI AM3 MB, 4xSATA HDDs. As far as add-ons and plugins, it is mainly docker templates, with speedtest and powerdown. I wish that I could be more descriptive here, but without access to the webgui, I'm running off of memory.

 

Here's the somewhat longer version of the issue, and what I've done. First, I was moving some files around within a share, and noticed that some files disappeared suddenly. I then went and checked the webgui, which showed disk 2 being offline. From here, I attempted to stop the array, with the intent to reboot to see if the drive came back. The array began stopping, but the hung at something like "Stopping User Shares...". I tried going back to the main webgui page, but it would never load. Nor would any other part of the webgui load.

 

From here, I went in via SSH, and ran the diagnostic command (link below). I went ahead and issued the powerdown -r command, which stated that the system was going down for reboot. The funny thing is that it never actually did go down. I can still open SSH sessions to it without issue. Just not the webui or SMB shares.

 

This is the message that appears to be repeating multiple times per second in the syslog: Feb  9 14:55:02 Nobles-Home kernel: XFS (md2): xfs_log_force: error -5 returned.

 

The archive was too large to attach, so here is a link to it on Google Drive: https://drive.google.com/file/d/0B7qWXJTelq0LellaMEVnVy1tTTg/view?usp=sharing

 

Before I go nuclear and hard power cycle the box, what else can I do?

 

Thanks for any assistance!

[renobles]

So this is resolved now. When I got home last night, I hard powered down the server because nothing else would shut it down. Afterwards, it came up OK with the array stopped (by design). Started it without the failed drive, stopped, and re-added the drive. It started to rebuild, but the speed was all over the place. Looked at the syslog, and noticed this a few times every second:

 

Feb 10 09:48:44 Nobles-Home kernel: ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Feb 10 09:48:46 Nobles-Home kernel: ata3.00: configured for UDMA/33
Feb 10 09:48:46 Nobles-Home kernel: ata3: EH complete
Feb 10 09:48:46 Nobles-Home kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x10200 action 0xe frozen
Feb 10 09:48:46 Nobles-Home kernel: ata3.00: irq_stat 0x00400000, PHY RDY changed
Feb 10 09:48:46 Nobles-Home kernel: ata3: SError: { Persist PHYRdyChg }
Feb 10 09:48:46 Nobles-Home kernel: ata3.00: failed command: READ DMA EXT
Feb 10 09:48:46 Nobles-Home kernel: ata3.00: cmd 25/00:40:10:1b:c6/00:05:04:00:00/e0 tag 11 dma 688128 in
Feb 10 09:48:46 Nobles-Home kernel:         res 50/00:00:8f:ee:09/00:00:05:00:00/e0 Emask 0x10 (ATA bus error)
Feb 10 09:48:46 Nobles-Home kernel: ata3.00: status: { DRDY }
Feb 10 09:48:46 Nobles-Home kernel: ata3: hard resetting link

 

The rebuild did eventually complete though. I then started a parity check, which was going, but extremely slowly. Also, mid-morning, my cache drive simply dropped out of Unraid. At this point I began to suspect the power supply. Replaced the PSU tonight, and the errors are all gone. Running a parity check now in any event.

[RobJ]

The PHYRdyChg flag would generally indicate either loose connections or power issues, so it's good you found power issues and corrected it!

 

Once the rebuild is done, I would also run Check Disk File systems on that disk, just in case.  I haven't seen that xfs_log_force error before, just want to make sure the XFS file system is clean.