tmchow Posted December 23, 2016 Share Posted December 23, 2016 My unraid 6.2 box seems frozen or stuck somehow. I can ping it but can't access web UI nor SSH into the box. What's the remedy for this situation? Link to comment
Frank1940 Posted December 23, 2016 Share Posted December 23, 2016 Try a quick push of the power button on the server and see if that will start a normal shutdown. Hopefully, if that does work, you will be able to find a syslog in the logs folder of your flash drive. Link to comment
tmchow Posted December 23, 2016 Author Share Posted December 23, 2016 Try a quick push of the power button on the server and see if that will start a normal shutdown. Hopefully, if that does work, you will be able to find a syslog in the logs folder of your flash drive. I power cycled the server through IPMI.. will post syslog shortly. Link to comment
tmchow Posted December 23, 2016 Author Share Posted December 23, 2016 Unfortunately looks like the power cycle via IPMI (a "Powercycle") did not generate logs that I can see in /boot/logs Link to comment
Frank1940 Posted December 23, 2016 Share Posted December 23, 2016 I would suggest hooking up a monitor and see what is going on during the boot process. If it does not finish with a login prompt, then take a picture of what you see. (Make sure that the picture is in focus, readable and any flash reflection does not obscure any information.) Link to comment
tmchow Posted December 23, 2016 Author Share Posted December 23, 2016 Well it booted up fine, but is now in parity check. Looking at Syslog, I'm seeing all sorts of errors. The latest tail is: Dec 22 20:13:01 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 0 csum 2206272318 expected csum 0 Dec 22 20:13:01 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 4096 csum 2819881197 expected csum 0 Dec 22 20:13:01 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:13:01 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 0 Dec 22 20:13:01 Tower kernel: BTRFS warning (device loop0): csum failed ino 274 off 0 csum 2977712167 expected csum 0 Dec 22 20:13:01 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:13:01 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 4096 Dec 22 20:13:01 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:13:01 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 8192 Dec 22 20:13:11 Tower emhttp: shcmd (1838): umount /var/lib/docker |& logger Parity check is running (which I expected because of unclean shutdown) but oddly my docker service doesn't seem to want to go up. It's enabled in the settings, but UI says "Status: Stopped" and there is no Docker tab in the Web UI. If I toggle Docker from "Yes" to "no" then back to "yes" in the Web UI settings, my syslog spits out this: Dec 22 20:15:14 Tower emhttp: shcmd (1891): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/cache/docker/docker.img' /var/lib/docker 20 |& logger Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): disk space caching is enabled Dec 22 20:15:15 Tower kernel: BTRFS: has skinny extents Dec 22 20:15:15 Tower root: Resize '/var/lib/docker' of 'max' Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): new size for /dev/loop0 is 21474836480 Dec 22 20:15:15 Tower emhttp: shcmd (1893): /etc/rc.d/rc.docker start |& logger Dec 22 20:15:15 Tower root: starting docker ... Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4569 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4569 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4569 start 8192 Dec 22 20:15:15 Tower kernel: __readpage_endio_check: 2 callbacks suppressed Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4569 off 0 csum 511918521 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4569 off 4096 csum 790941476 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4569 off 8192 csum 4283786149 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 2165 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 2165 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 2165 off 0 csum 3553516629 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 2165 off 4096 csum 886249706 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4837 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4837 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4837 off 0 csum 3428491842 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4837 off 4096 csum 494999041 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4689 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4689 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 0 csum 2206272318 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 4096 csum 2819881197 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 274 off 0 csum 2977712167 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 8192 Dec 22 20:15:25 Tower emhttp: shcmd (1895): umount /var/lib/docker |& logger Link to comment
trurl Posted December 23, 2016 Share Posted December 23, 2016 Unfortunately looks like the power cycle via IPMI (a "Powercycle") did not generate logs that I can see in /boot/logs You won't get a syslog with an unclean shutdown. Link to comment
trurl Posted December 23, 2016 Share Posted December 23, 2016 Well it booted up fine, but is now in parity check. Looking at Syslog, I'm seeing all sorts of errors. The latest tail is: Dec 22 20:13:01 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 0 csum 2206272318 expected csum 0 Dec 22 20:13:01 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 4096 csum 2819881197 expected csum 0 Dec 22 20:13:01 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:13:01 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 0 Dec 22 20:13:01 Tower kernel: BTRFS warning (device loop0): csum failed ino 274 off 0 csum 2977712167 expected csum 0 Dec 22 20:13:01 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:13:01 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 4096 Dec 22 20:13:01 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:13:01 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 8192 Dec 22 20:13:11 Tower emhttp: shcmd (1838): umount /var/lib/docker |& logger Parity check is running (which I expected because of unclean shutdown) but oddly my docker service doesn't seem to want to go up. It's enabled in the settings, but UI says "Status: Stopped" and there is no Docker tab in the Web UI. If I toggle Docker from "Yes" to "no" then back to "yes" in the Web UI settings, my syslog spits out this: Dec 22 20:15:14 Tower emhttp: shcmd (1891): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/cache/docker/docker.img' /var/lib/docker 20 |& logger Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): disk space caching is enabled Dec 22 20:15:15 Tower kernel: BTRFS: has skinny extents Dec 22 20:15:15 Tower root: Resize '/var/lib/docker' of 'max' Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): new size for /dev/loop0 is 21474836480 Dec 22 20:15:15 Tower emhttp: shcmd (1893): /etc/rc.d/rc.docker start |& logger Dec 22 20:15:15 Tower root: starting docker ... Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4569 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4569 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4569 start 8192 Dec 22 20:15:15 Tower kernel: __readpage_endio_check: 2 callbacks suppressed Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4569 off 0 csum 511918521 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4569 off 4096 csum 790941476 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4569 off 8192 csum 4283786149 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 2165 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 2165 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 2165 off 0 csum 3553516629 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 2165 off 4096 csum 886249706 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4837 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4837 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4837 off 0 csum 3428491842 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4837 off 4096 csum 494999041 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4689 start 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 4689 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 0 csum 2206272318 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 4689 off 4096 csum 2819881197 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 0 Dec 22 20:15:15 Tower kernel: BTRFS warning (device loop0): csum failed ino 274 off 0 csum 2977712167 expected csum 0 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 4096 Dec 22 20:15:15 Tower kernel: BTRFS critical (device loop0): corrupt leaf, bad key order: block=550535168,root=1, slot=11 Dec 22 20:15:15 Tower kernel: BTRFS info (device loop0): no csum found for inode 274 start 8192 Dec 22 20:15:25 Tower emhttp: shcmd (1895): umount /var/lib/docker |& logger Post diagnostics. Looks like your docker.img is corrupt. With docker service stopped, delete docker.img, recreate it, and add your previous apps back with Community Application or from the "my_templates". Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.