Random crashes..no access to gui or ssh. Have to hard reboot.


mkono87

Recommended Posts

Turn on save syslog to flash so we can see the reason the server is crashing.

Check C states, turn off ram overclocking (anything above the spec on ark is overclocking) make sure psu sleep state is turned off in bios. Failing those look at downgrading to 6.8.3, this fixed mine, hard lock every 2 days on 6.9.x, today, 34 days uptime.

Link to comment
5 hours ago, Tristankin said:

Turn on save syslog to flash so we can see the reason the server is crashing.

Check C states, turn off ram overclocking (anything above the spec on ark is overclocking) make sure psu sleep state is turned off in bios. Failing those look at downgrading to 6.8.3, this fixed mine, hard lock every 2 days on 6.9.x, today, 34 days uptime.

Syslog to flash is on. This is where the log from that pastebin came from. Downgrsding I have not tried but I will give that a shot.

Link to comment
On 9/7/2021 at 10:46 AM, fmp4m said:

We are awaiting the full Syslog, however you are running V1 of the AsRockRack motherboard bios from 2015.  You need to update to atleast 2.6 of the bios as that fixed a lot of known issues.  2.7 is current and I know of no reason not to update to 2.7 so I suggest it.

Okay I have no updated to the latest bios and attached the latest logs from when it went down yesterday.

 

Please note: These logs are from before I updated the Bios.

syslog.zip nas-diagnostics-20210908-1252.zip

Edited by mkono87
Link to comment

@mkono87 you have XFS corruption,  hopefully @JorgeB can assist with that.

 

Thats the last entry before the crash:

 

Sep  7 14:22:55 NAS kernel: XFS (sdb1): Metadata corruption detected at xfs_dinode_verify+0xa7/0x567 [xfs], inode 0xe997421 dinode
Sep  7 14:22:55 NAS kernel: XFS (sdb1): Unmount and run xfs_repair
Sep  7 14:22:55 NAS kernel: XFS (sdb1): First 128 bytes of corrupted metadata buffer:
Sep  7 14:22:55 NAS kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Sep  7 14:22:55 NAS kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Sep  7 14:22:55 NAS kernel: 00000020: f0 be 68 03 81 88 ff ff 60 d6 04 ab 27 03 36 41  ..h.....`...'.6A
Sep  7 14:22:55 NAS kernel: 00000030: 60 d6 04 ab 27 03 36 41 00 00 00 00 00 08 de c1  `...'.6A........
Sep  7 14:22:55 NAS kernel: 00000040: 00 00 00 00 00 00 00 8e 00 00 00 00 00 00 00 01  ................
Sep  7 14:22:55 NAS kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 5f 57 70 8c  ............_Wp.
Sep  7 14:22:55 NAS kernel: 00000060: ff ff ff ff be bc 40 17 00 00 00 00 00 00 00 06  ......@.........
Sep  7 14:22:55 NAS kernel: 00000070: 00 00 1d dd 00 00 f5 f6 00 00 00 00 00 00 00 00  ................
Sep  7 16:13:34 NAS kernel: XFS (sdb1): Metadata corruption detected at xfs_dinode_verify+0xa7/0x567 [xfs], inode 0xe997421 dinode
Sep  7 16:13:34 NAS kernel: XFS (sdb1): Unmount and run xfs_repair
Sep  7 16:13:34 NAS kernel: XFS (sdb1): First 128 bytes of corrupted metadata buffer:
Sep  7 16:13:34 NAS kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Sep  7 16:13:34 NAS kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Sep  7 16:13:34 NAS kernel: 00000020: f0 be 68 03 81 88 ff ff 60 d6 04 ab 27 03 36 41  ..h.....`...'.6A
Sep  7 16:13:34 NAS kernel: 00000030: 60 d6 04 ab 27 03 36 41 00 00 00 00 00 08 de c1  `...'.6A........
Sep  7 16:13:34 NAS kernel: 00000040: 00 00 00 00 00 00 00 8e 00 00 00 00 00 00 00 01  ................
Sep  7 16:13:34 NAS kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 5f 57 70 8c  ............_Wp.
Sep  7 16:13:34 NAS kernel: 00000060: ff ff ff ff be bc 40 17 00 00 00 00 00 00 00 06  ......@.........
Sep  7 16:13:34 NAS kernel: 00000070: 00 00 1d dd 00 00 f5 f6 00 00 00 00 00 00 00 00  ................
Sep  7 17:25:13 NAS kernel: XFS (sdb1): Metadata corruption detected at xfs_dinode_verify+0xa7/0x567 [xfs], inode 0xe997421 dinode
Sep  7 17:25:13 NAS kernel: XFS (sdb1): Unmount and run xfs_repair
Sep  7 17:25:13 NAS kernel: XFS (sdb1): First 128 bytes of corrupted metadata buffer:
Sep  7 17:25:13 NAS kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Sep  7 17:25:13 NAS kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Sep  7 17:25:13 NAS kernel: 00000020: f0 be 68 03 81 88 ff ff 60 d6 04 ab 27 03 36 41  ..h.....`...'.6A
Sep  7 17:25:13 NAS kernel: 00000030: 60 d6 04 ab 27 03 36 41 00 00 00 00 00 08 de c1  `...'.6A........
Sep  7 17:25:13 NAS kernel: 00000040: 00 00 00 00 00 00 00 8e 00 00 00 00 00 00 00 01  ................
Sep  7 17:25:13 NAS kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 5f 57 70 8c  ............_Wp.
Sep  7 17:25:13 NAS kernel: 00000060: ff ff ff ff be bc 40 17 00 00 00 00 00 00 00 06  ......@.........
Sep  7 17:25:13 NAS kernel: 00000070: 00 00 1d dd 00 00 f5 f6 00 00 00 00 00 00 00 00  ................

 

Link to comment
24 minutes ago, fmp4m said:

@mkono87 you have XFS corruption,  hopefully @JorgeB can assist with that.

 

Thats the last entry before the crash:

 

Sep  7 14:22:55 NAS kernel: XFS (sdb1): Metadata corruption detected at xfs_dinode_verify+0xa7/0x567 [xfs], inode 0xe997421 dinode
Sep  7 14:22:55 NAS kernel: XFS (sdb1): Unmount and run xfs_repair
Sep  7 14:22:55 NAS kernel: XFS (sdb1): First 128 bytes of corrupted metadata buffer:
Sep  7 14:22:55 NAS kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Sep  7 14:22:55 NAS kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Sep  7 14:22:55 NAS kernel: 00000020: f0 be 68 03 81 88 ff ff 60 d6 04 ab 27 03 36 41  ..h.....`...'.6A
Sep  7 14:22:55 NAS kernel: 00000030: 60 d6 04 ab 27 03 36 41 00 00 00 00 00 08 de c1  `...'.6A........
Sep  7 14:22:55 NAS kernel: 00000040: 00 00 00 00 00 00 00 8e 00 00 00 00 00 00 00 01  ................
Sep  7 14:22:55 NAS kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 5f 57 70 8c  ............_Wp.
Sep  7 14:22:55 NAS kernel: 00000060: ff ff ff ff be bc 40 17 00 00 00 00 00 00 00 06  ......@.........
Sep  7 14:22:55 NAS kernel: 00000070: 00 00 1d dd 00 00 f5 f6 00 00 00 00 00 00 00 00  ................
Sep  7 16:13:34 NAS kernel: XFS (sdb1): Metadata corruption detected at xfs_dinode_verify+0xa7/0x567 [xfs], inode 0xe997421 dinode
Sep  7 16:13:34 NAS kernel: XFS (sdb1): Unmount and run xfs_repair
Sep  7 16:13:34 NAS kernel: XFS (sdb1): First 128 bytes of corrupted metadata buffer:
Sep  7 16:13:34 NAS kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Sep  7 16:13:34 NAS kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Sep  7 16:13:34 NAS kernel: 00000020: f0 be 68 03 81 88 ff ff 60 d6 04 ab 27 03 36 41  ..h.....`...'.6A
Sep  7 16:13:34 NAS kernel: 00000030: 60 d6 04 ab 27 03 36 41 00 00 00 00 00 08 de c1  `...'.6A........
Sep  7 16:13:34 NAS kernel: 00000040: 00 00 00 00 00 00 00 8e 00 00 00 00 00 00 00 01  ................
Sep  7 16:13:34 NAS kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 5f 57 70 8c  ............_Wp.
Sep  7 16:13:34 NAS kernel: 00000060: ff ff ff ff be bc 40 17 00 00 00 00 00 00 00 06  ......@.........
Sep  7 16:13:34 NAS kernel: 00000070: 00 00 1d dd 00 00 f5 f6 00 00 00 00 00 00 00 00  ................
Sep  7 17:25:13 NAS kernel: XFS (sdb1): Metadata corruption detected at xfs_dinode_verify+0xa7/0x567 [xfs], inode 0xe997421 dinode
Sep  7 17:25:13 NAS kernel: XFS (sdb1): Unmount and run xfs_repair
Sep  7 17:25:13 NAS kernel: XFS (sdb1): First 128 bytes of corrupted metadata buffer:
Sep  7 17:25:13 NAS kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Sep  7 17:25:13 NAS kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Sep  7 17:25:13 NAS kernel: 00000020: f0 be 68 03 81 88 ff ff 60 d6 04 ab 27 03 36 41  ..h.....`...'.6A
Sep  7 17:25:13 NAS kernel: 00000030: 60 d6 04 ab 27 03 36 41 00 00 00 00 00 08 de c1  `...'.6A........
Sep  7 17:25:13 NAS kernel: 00000040: 00 00 00 00 00 00 00 8e 00 00 00 00 00 00 00 01  ................
Sep  7 17:25:13 NAS kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 5f 57 70 8c  ............_Wp.
Sep  7 17:25:13 NAS kernel: 00000060: ff ff ff ff be bc 40 17 00 00 00 00 00 00 00 06  ......@.........
Sep  7 17:25:13 NAS kernel: 00000070: 00 00 1d dd 00 00 f5 f6 00 00 00 00 00 00 00 00  ................

 

 

Hmm, thats no good. I dont even see a sdb1 entry when using lsblk or ls /dev. Im not sure what that would be.

Link to comment
1 hour ago, fmp4m said:

SDB is usually Cache.  SDB1 is usually cache disk one.

 

49 minutes ago, trurl said:

sdb1 is the first partition on /dev/sdb. Most disks would be listed that way in syslog unless they were array devices. Those entries were before reboot and no sdb attached currently. My guess is it referred to cache also.

 

Yes this is what I have now. Just to let you know. My docker containers and vms are on their own separate drive. Yes its small but I have have vms atm.

 

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0  13.5M  1 loop /lib/modules
loop1    7:1    0 101.1M  1 loop /lib/firmware
loop2    7:2    0    40G  0 loop /var/lib/docker/btrfs
                                 /var/lib/docker
sda      8:0    1   7.3G  0 disk
└─sda1   8:1    1   7.3G  0 part /boot
sdc      8:32   0 223.6G  0 disk
└─sdc1   8:33   0 223.6G  0 part /mnt/disks/docker-vm
sdd      8:48   0   3.6T  0 disk
└─sdd1   8:49   0   3.6T  0 part
sde      8:64   0   3.6T  0 disk
└─sde1   8:65   0   3.6T  0 part
sdf      8:80   0 465.8G  0 disk
└─sdf1   8:81   0 465.8G  0 part /mnt/cache
sdg      8:96   0   3.6T  0 disk
└─sdg1   8:97   0   3.6T  0 part
sdh      8:112  0   3.6T  0 disk
└─sdh1   8:113  0   3.6T  0 part
md1      9:1    0   3.6T  0 md   /mnt/disk1
md2      9:2    0   3.6T  0 md   /mnt/disk2
md3      9:3    0   3.6T  0 md   /mnt/disk3

 

Edited by mkono87
Link to comment
32 minutes ago, mkono87 said:

I did receive an error notification for the docker driver with an uncorrected 1 status. I did a smart test after it finished the parity check. (I know not related). I did get a successful smart extended test, is this not to be trusted in this case?

Sent from my Mi 9T using Tapatalk
 

 

SMART checkes drives health - not data health.  You have corruption and will need to repair the corruption.

Link to comment
On 9/9/2021 at 10:27 AM, trurl said:

If you don't repair the corrupt filesystem you will get those log entries again when it is accessed

 

It did crash again and I finally got around to doing xfs_repair on my appdata drive. Seems to have fixed itself correctly.

root@NAS:~# xfs_repair /dev/sdb1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
bad CRC for inode 244937761
bad CRC for inode 244937761, will rewrite
Bad atime nsec 2173239295 on inode 244937761, resetting to zero
cleared inode 244937761
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 2
        - agno = 3
        - agno = 1
        - agno = 0
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

No lost and found folder created. So I guess at this point is reset the syslog and see what happens.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.