March 17, 20233 yr Hi all, I've got a pushover notification a few minutes ago telling me there was 1 read error, but it appeared on all my drives at once. I stupidly decided to start a parity sync and noticed non of my drives were reachable anymore : https://app.screencast.com/3rYhKTK4EHjzQ I also tried to access /mnt/user -> root@Tower:~# cd /mnt/user/ -bash: cd: /mnt/user/: Input/output error here is an extract of what syslog tells me: ``` Mar 17 10:54:51 Tower root: ddns-updater: Could not download icon https://github.com/DiamondPrecisionComputing/unraid-templates/blob/main/templates/img/ddnsgopher.png?raw=true Mar 17 11:16:54 Tower kernel: md: disk1 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk2 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk3 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk4 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk5 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk6 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk7 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk8 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk9 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk10 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk11 read error, sector=0 Mar 17 11:16:54 Tower kernel: md: disk0 read error, sector=0 Mar 17 11:16:54 Tower kernel: XFS (md1): metadata I/O error in "xfs_alloc_read_agfl+0x84/0xc3 [xfs]" at daddr 0x3 len 1 error 5 Mar 17 11:16:54 Tower kernel: md: disk1 read error, sector=12886351208 Mar 17 11:16:54 Tower kernel: md: disk0 read error, sector=12886351208 Mar 17 11:16:54 Tower kernel: md: disk1 read error, sector=12886351216 Mar 17 11:16:54 Tower kernel: md: disk0 read error, sector=12886351216 Mar 17 11:16:54 Tower kernel: XFS (md1): log I/O error -5 Mar 17 11:16:54 Tower kernel: XFS (md1): Corruption of in-memory data (0x8) detected at xfs_defer_finish_noroll+0x479/0x503 [xfs] (fs/xfs/libxfs/xfs_defer.c:573). Shutting down filesystem. Mar 17 11:16:54 Tower kernel: XFS (md1): Please unmount the filesystem and rectify the problem(s) Mar 17 11:17:33 Tower kernel: mdcmd (36): check correct Mar 17 11:17:33 Tower kernel: md: recovery thread: check P ... Mar 17 11:17:33 Tower kernel: md: recovery thread: multiple disk errors, sector=0 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=8 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=16 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=24 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=32 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=40 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=48 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=56 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=64 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=72 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=80 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=88 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=96 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=104 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=112 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=120 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=128 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=136 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=144 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=152 Mar 17 11:17:33 Tower kernel: md: disk1 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk2 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk3 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk4 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk5 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk6 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk7 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk8 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk9 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk10 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk11 read error, sector=160 Mar 17 11:17:33 Tower kernel: md: disk0 read error, sector=160 ``` Can someone help me out figuring out what's happening and what broke ? It's pretty urgent, all my dockers are running from there I've attached full diagnostics zip. thanks ! tower-diagnostics-20230317-1141.zip
March 17, 20233 yr Author I did reboot, everything is back up. For which reason could the HBA have been reset ? electrical problem ? anyway, the new zip is attached ! thanks ! tower-diagnostics-20230317-1227.zip
March 17, 20233 yr Difficult to say, make sure it's well seated and sufficiently cooled, you can also try a different PCIe slot.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.