MvL Posted April 25, 2019 Share Posted April 25, 2019 Hi, After an forced shutdown my cache drive isn't mountable. Is it possible to repair the file system? Till now I'm not sure why my server was hanging. I will attach the diagnostic file. Please some help. rackserver-diagnostics-20190425-0624.zip Quote Link to comment
MvL Posted April 25, 2019 Author Share Posted April 25, 2019 Interesting.. I found this in the syslog: Quote Apr 25 08:20:06 RackServer kernel: BTRFS critical (device sdb1): corrupt leaf: root=2 block=159318016 slot=36, unexpected item end, have 2430846058 expect 14375 ### [PREVIOUS LINE REPEATED 1 TIMES] ### Apr 25 08:20:06 RackServer kernel: BTRFS: error (device sdb1) in btrfs_replay_log:2277: errno=-5 IO failure (Failed to recover log tree) Apr 25 08:20:06 RackServer kernel: BTRFS error (device sdb1): pending csums is 57344 Apr 25 08:20:06 RackServer kernel: WARNING: CPU: 18 PID: 7098 at fs/btrfs/extent-tree.c:107 btrfs_put_block_group+0x1a/0x44 Apr 25 08:20:06 RackServer kernel: Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod dax nfsd lockd grace sunrpc md_mod bonding igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd ipmi_ssif cryptd glue_helper mpt3sas intel_cstate intel_uncore ahci intel_rapl_perf libahci i2c_i801 i2c_core raid_class scsi_transport_sas wmi ipmi_si acpi_power_meter pcc_cpufreq button acpi_pad [last unloaded: i2c_algo_bit] Apr 25 08:20:06 RackServer kernel: CPU: 18 PID: 7098 Comm: mount Not tainted 4.19.33-Unraid #1 Apr 25 08:20:06 RackServer kernel: Hardware name: Supermicro Super Server/X10SRi-F, BIOS 3.1 06/06/2018 Apr 25 08:20:06 RackServer kernel: RIP: 0010:btrfs_put_block_group+0x1a/0x44 Apr 25 08:20:06 RackServer kernel: Code: ff 48 83 c4 50 5b 5d 41 5c 41 5d 41 5e 41 5f c3 53 f0 ff 8f 08 01 00 00 75 38 48 83 7f 48 00 74 02 0f 0b 48 83 7f 50 00 74 02 <0f> 0b 48 8b 87 e8 01 00 00 48 85 c0 74 02 0f 0b 48 89 fb 48 8b bf Apr 25 08:20:06 RackServer kernel: RSP: 0018:ffffc900071cbb00 EFLAGS: 00010206 Apr 25 08:20:06 RackServer kernel: RAX: 0000000000000000 RBX: ffff888ff0022000 RCX: 0000000000000048 Apr 25 08:20:06 RackServer kernel: RDX: 0000000000000048 RSI: 0000000000004000 RDI: ffff888fed5dbc00 Apr 25 08:20:06 RackServer kernel: RBP: ffff888fed5dbce0 R08: 0000000000000001 R09: ffffffff81243d00 Apr 25 08:20:06 RackServer kernel: R10: ffffea003fb422c0 R11: ffff889033878001 R12: ffff888ff0022090 Apr 25 08:20:06 RackServer kernel: R13: ffff888fed5dbc00 R14: dead000000000100 R15: ffff888ff00220a0 Apr 25 08:20:06 RackServer kernel: FS: 000014adb5be5780(0000) GS:ffff88903f680000(0000) knlGS:0000000000000000 Apr 25 08:20:06 RackServer kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 25 08:20:06 RackServer kernel: CR2: 000014adb5c7d200 CR3: 0000000fedcf0003 CR4: 00000000001606e0 Apr 25 08:20:06 RackServer kernel: Call Trace: Apr 25 08:20:06 RackServer kernel: btrfs_free_block_groups+0x17a/0x2c8 Apr 25 08:20:06 RackServer kernel: open_ctree+0x1bb2/0x1c61 Apr 25 08:20:06 RackServer kernel: btrfs_mount_root+0x382/0x4f5 Apr 25 08:20:06 RackServer kernel: ? cpumask_next+0x16/0x17 Apr 25 08:20:06 RackServer kernel: ? pcpu_alloc+0x366/0x471 Apr 25 08:20:06 RackServer kernel: mount_fs+0x10/0x6b Apr 25 08:20:06 RackServer kernel: vfs_kern_mount+0x62/0xfa Apr 25 08:20:06 RackServer kernel: btrfs_mount+0x163/0x7f0 Apr 25 08:20:06 RackServer kernel: ? pcpu_block_update_hint_alloc+0x9c/0x1a0 Apr 25 08:20:06 RackServer kernel: ? pcpu_chunk_relocate+0x8/0x5a Apr 25 08:20:06 RackServer kernel: ? pcpu_alloc_area+0xc7/0xd7 Apr 25 08:20:06 RackServer kernel: ? pcpu_next_unpop+0x31/0x3c Apr 25 08:20:06 RackServer kernel: ? cpumask_next+0x16/0x17 Apr 25 08:20:06 RackServer kernel: ? pcpu_alloc+0x366/0x471 Apr 25 08:20:06 RackServer kernel: ? mount_fs+0x10/0x6b Apr 25 08:20:06 RackServer kernel: mount_fs+0x10/0x6b Apr 25 08:20:06 RackServer kernel: vfs_kern_mount+0x62/0xfa Apr 25 08:20:06 RackServer kernel: do_mount+0x7b3/0xa2f Apr 25 08:20:06 RackServer kernel: ? __kmalloc_track_caller+0x65/0x122 Apr 25 08:20:06 RackServer kernel: ? _copy_from_user+0x2f/0x4d Apr 25 08:20:06 RackServer kernel: ? memdup_user+0x39/0x55 Apr 25 08:20:06 RackServer kernel: ksys_mount+0x6d/0x92 Apr 25 08:20:06 RackServer kernel: __x64_sys_mount+0x1c/0x1f Apr 25 08:20:06 RackServer kernel: do_syscall_64+0x57/0xe6 Apr 25 08:20:06 RackServer kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Apr 25 08:20:06 RackServer kernel: RIP: 0033:0x14adb5d21fca Apr 25 08:20:06 RackServer kernel: Code: 48 8b 0d c9 7e 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 96 7e 0c 00 f7 d8 64 89 01 48 Apr 25 08:20:06 RackServer kernel: RSP: 002b:00007ffcb1aeb568 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 Apr 25 08:20:06 RackServer kernel: RAX: ffffffffffffffda RBX: 000000000040d2b0 RCX: 000014adb5d21fca Apr 25 08:20:06 RackServer kernel: RDX: 000000000040d4c0 RSI: 000000000040d540 RDI: 000000000040d520 Apr 25 08:20:06 RackServer kernel: RBP: 000014adb5eaef10 R08: 0000000000000000 R09: 0000000000000000 Apr 25 08:20:06 RackServer kernel: R10: 0000000000000c00 R11: 0000000000000202 R12: 0000000000000000 Apr 25 08:20:06 RackServer kernel: R13: 0000000000000c00 R14: 000000000040d520 R15: 000000000040d4c0 Apr 25 08:20:06 RackServer kernel: ---[ end trace 62ba4894ff66ba3e ]--- Apr 25 08:20:06 RackServer kernel: WARNING: CPU: 18 PID: 7098 at fs/btrfs/extent-tree.c:9747 btrfs_free_block_groups+0x233/0x2c8 Apr 25 08:20:06 RackServer kernel: Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod dax nfsd lockd grace sunrpc md_mod bonding igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd ipmi_ssif cryptd glue_helper mpt3sas intel_cstate intel_uncore ahci intel_rapl_perf libahci i2c_i801 i2c_core raid_class scsi_transport_sas wmi ipmi_si acpi_power_meter pcc_cpufreq button acpi_pad [last unloaded: i2c_algo_bit] Apr 25 08:20:06 RackServer kernel: CPU: 18 PID: 7098 Comm: mount Tainted: G W 4.19.33-Unraid #1 Apr 25 08:20:06 RackServer kernel: Hardware name: Supermicro Super Server/X10SRi-F, BIOS 3.1 06/06/2018 Apr 25 08:20:06 RackServer kernel: RIP: 0010:btrfs_free_block_groups+0x233/0x2c8 Apr 25 08:20:06 RackServer kernel: Code: 93 38 0e 00 00 48 39 c2 0f 84 a4 00 00 00 48 8b ab 38 0e 00 00 48 83 7d 90 00 75 0e 48 83 7d 98 00 75 07 48 83 7d a0 00 74 15 <0f> 0b 31 c9 31 d2 48 89 df 48 8d b5 78 ff ff ff e8 4f 97 ff ff 48 Apr 25 08:20:06 RackServer kernel: RSP: 0018:ffffc900071cbb10 EFLAGS: 00010206 Apr 25 08:20:06 RackServer kernel: RAX: ffff888fed5c8488 RBX: ffff888ff0022000 RCX: 0000000000000000 Apr 25 08:20:06 RackServer kernel: RDX: ffff888ff0022e38 RSI: 0000000000000001 RDI: ffff888fed5c8200 Apr 25 08:20:06 RackServer kernel: RBP: ffff888fed5c8488 R08: 0000000000000000 R09: 0000000000000000 Apr 25 08:20:06 RackServer kernel: R10: 0000000000000fc0 R11: ffff889033878080 R12: ffff888ff0022090 Apr 25 08:20:06 RackServer kernel: R13: ffff889034575400 R14: dead000000000100 R15: ffff888ff00220a0 Apr 25 08:20:06 RackServer kernel: FS: 000014adb5be5780(0000) GS:ffff88903f680000(0000) knlGS:0000000000000000 Apr 25 08:20:06 RackServer kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 25 08:20:06 RackServer kernel: CR2: 000014adb5c7d200 CR3: 0000000fedcf0003 CR4: 00000000001606e0 Apr 25 08:20:06 RackServer kernel: Call Trace: Apr 25 08:20:06 RackServer kernel: open_ctree+0x1bb2/0x1c61 Apr 25 08:20:06 RackServer kernel: btrfs_mount_root+0x382/0x4f5 Apr 25 08:20:06 RackServer kernel: ? cpumask_next+0x16/0x17 Apr 25 08:20:06 RackServer kernel: ? pcpu_alloc+0x366/0x471 Apr 25 08:20:06 RackServer kernel: mount_fs+0x10/0x6b Apr 25 08:20:06 RackServer kernel: vfs_kern_mount+0x62/0xfa Apr 25 08:20:06 RackServer kernel: btrfs_mount+0x163/0x7f0 Apr 25 08:20:06 RackServer kernel: ? pcpu_block_update_hint_alloc+0x9c/0x1a0 Apr 25 08:20:06 RackServer kernel: ? pcpu_chunk_relocate+0x8/0x5a Apr 25 08:20:06 RackServer kernel: ? pcpu_alloc_area+0xc7/0xd7 Apr 25 08:20:06 RackServer kernel: ? pcpu_next_unpop+0x31/0x3c Apr 25 08:20:06 RackServer kernel: ? cpumask_next+0x16/0x17 Apr 25 08:20:06 RackServer kernel: ? pcpu_alloc+0x366/0x471 Apr 25 08:20:06 RackServer kernel: ? mount_fs+0x10/0x6b Apr 25 08:20:06 RackServer kernel: mount_fs+0x10/0x6b Apr 25 08:20:06 RackServer kernel: vfs_kern_mount+0x62/0xfa Apr 25 08:20:06 RackServer kernel: do_mount+0x7b3/0xa2f Apr 25 08:20:06 RackServer kernel: ? __kmalloc_track_caller+0x65/0x122 Apr 25 08:20:06 RackServer kernel: ? _copy_from_user+0x2f/0x4d Apr 25 08:20:06 RackServer kernel: ? memdup_user+0x39/0x55 Apr 25 08:20:06 RackServer kernel: ksys_mount+0x6d/0x92 Apr 25 08:20:06 RackServer kernel: __x64_sys_mount+0x1c/0x1f Apr 25 08:20:06 RackServer kernel: do_syscall_64+0x57/0xe6 Apr 25 08:20:06 RackServer kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Apr 25 08:20:06 RackServer kernel: RIP: 0033:0x14adb5d21fca Apr 25 08:20:06 RackServer kernel: Code: 48 8b 0d c9 7e 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 96 7e 0c 00 f7 d8 64 89 01 48 Apr 25 08:20:06 RackServer kernel: RSP: 002b:00007ffcb1aeb568 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 Apr 25 08:20:06 RackServer kernel: RAX: ffffffffffffffda RBX: 000000000040d2b0 RCX: 000014adb5d21fca Apr 25 08:20:06 RackServer kernel: RDX: 000000000040d4c0 RSI: 000000000040d540 RDI: 000000000040d520 Apr 25 08:20:06 RackServer kernel: RBP: 000014adb5eaef10 R08: 0000000000000000 R09: 0000000000000000 Apr 25 08:20:06 RackServer kernel: R10: 0000000000000c00 R11: 0000000000000202 R12: 0000000000000000 Apr 25 08:20:06 RackServer kernel: R13: 0000000000000c00 R14: 000000000040d520 R15: 000000000040d4c0 Apr 25 08:20:06 RackServer kernel: ---[ end trace 62ba4894ff66ba3f ]--- Apr 25 08:20:06 RackServer kernel: BTRFS info (device sdb1): space_info 1 has 32111308800 free, is not full Apr 25 08:20:06 RackServer kernel: BTRFS info (device sdb1): space_info total=126709923840, used=94598602752, pinned=0, reserved=12288, may_use=0, readonly=0 Apr 25 08:20:06 RackServer kernel: BTRFS error (device sdb1): open_ctree failed Apr 25 08:20:06 RackServer root: mount: /mnt/cache: can't read superblock on /dev/sdb1. Apr 25 08:20:06 RackServer emhttpd: shcmd (222): exit status: 32 Apr 25 08:20:06 RackServer emhttpd: /mnt/cache mount error: No file system Apr 25 08:20:06 RackServer emhttpd: shcmd (223): umount /mnt/cache Apr 25 08:20:06 RackServer root: umount: /mnt/cache: not mounted. Apr 25 08:20:06 RackServer emhttpd: shcmd (223): exit status: 32 Apr 25 08:20:06 RackServer emhttpd: shcmd (224): rmdir /mnt/cache Apr 25 08:20:06 RackServer emhttpd: shcmd (225): sync Apr 25 08:20:06 RackServer emhttpd: shcmd (226): mkdir /mnt/user0 Apr 25 08:20:06 RackServer emhttpd: shcmd (227): /usr/local/sbin/shfs /mnt/user0 -disks 510 -o noatime,big_writes,allow_other |& logger Apr 25 08:20:06 RackServer shfs: stderr redirected to syslog Apr 25 08:20:06 RackServer emhttpd: shcmd (228): mkdir /mnt/user Apr 25 08:20:06 RackServer emhttpd: shcmd (229): /usr/local/sbin/shfs /mnt/user -disks 511 2048000000 -o noatime,big_writes,allow_other -o remember=330 |& logger Apr 25 08:20:06 RackServer shfs: stderr redirected to syslog Apr 25 08:20:06 RackServer emhttpd: shcmd (231): /usr/local/sbin/update_cron Apr 25 08:20:06 RackServer root: Creating custom tabs Apr 25 08:20:06 RackServer emhttpd: Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/customtab/scripts/createTab.php on line 33 Apr 25 08:20:06 RackServer cache_dirs: Arguments=-l off Apr 25 08:20:06 RackServer cache_dirs: Max Scan Secs=10, Min Scan Secs=1 Apr 25 08:20:06 RackServer cache_dirs: Scan Type=adaptive Apr 25 08:20:06 RackServer cache_dirs: Min Scan Depth=4 Apr 25 08:20:06 RackServer cache_dirs: Max Scan Depth=none Apr 25 08:20:06 RackServer cache_dirs: Use Command='find -noleaf' Apr 25 08:20:06 RackServer cache_dirs: ---------- Caching Directories --------------- Apr 25 08:20:06 RackServer cache_dirs: h..e Apr 25 08:20:06 RackServer cache_dirs: m..) ### [PREVIOUS LINE REPEATED 6 TIMES] ### Quote Link to comment
MvL Posted April 25, 2019 Author Share Posted April 25, 2019 (edited) Found this excellent post. I'm trying to recover the cache drive! https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-543490 Edited April 25, 2019 by MvL Quote Link to comment
MvL Posted April 25, 2019 Author Share Posted April 25, 2019 So cache drive is recovered thanks to the post that I linked above. I'm only not sure why my server was hanging.. Something to investigate further. If you have any idea then please leave a comment behind. Quote Link to comment
MvL Posted April 26, 2019 Author Share Posted April 26, 2019 No problems last night! It runs smoothly.. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.