I have an ASrock C2750D4I server running 6.1.8 that seems to have encountered a fault at 3:40am, when the mover started up.  The GUI and CLI are still accessible, but I'm hitting errors accessing the array via SMB, and when I go to the GUI to shutdown the array, I can't because it says that 'Mover is running'.  This happened about a week ago as well and I had to do a restart of the server via IPMI, which led to a dirty shutdown and parity check.

 

Anyone else seen this happen or have an idea what to check?  I'll go down the path of upgrading BIOS/unRaid/etc shortly but is there a way I can terminate the mover and have the array cleanly shutdown?

 

Relevant snippet from syslog below.  I don't see anything obvious otherwise, but can provide more if needed.

 

Apr  8 03:40:01 Tower kernel: general protection fault: 0000 [#1] PREEMPT SMP 
Apr  8 03:40:01 Tower kernel: Modules linked in: cdc_acm xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables kvm_intel kvm vhost_net vhost macvtap macvlan tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod igb i2c_algo_bit i2c_i801 ptp pps_core ahci libahci mvsas libsas scsi_transport_sas ipmi_si acpi_cpufreq
Apr  8 03:40:01 Tower kernel: CPU: 1 PID: 2272 Comm: shfs Not tainted 4.1.17-unRAID #1
Apr  8 03:40:01 Tower kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./C2750D4I, BIOS P2.80 12/04/2014
Apr  8 03:40:01 Tower kernel: task: ffff8804694a90c0 ti: ffff880453518000 task.ti: ffff880453518000
Apr  8 03:40:01 Tower kernel: RIP: 0010:[<ffffffff81153948>]  [<ffffffff81153948>] __discard_prealloc+0x98/0xb3
Apr  8 03:40:01 Tower kernel: RSP: 0018:ffff88045351bb68  EFLAGS: 00010246
Apr  8 03:40:01 Tower kernel: RAX: ffff88011699f9e8 RBX: ffff88011699f9c0 RCX: bd64cb7ef318fc19
Apr  8 03:40:01 Tower kernel: RDX: f52578f4ad8d3960 RSI: ffff88011699f9c0 RDI: ffff88045351bcb0
Apr  8 03:40:01 Tower kernel: RBP: ffff88045351bb98 R08: 00000000000005da R09: 0000000000012548
Apr  8 03:40:01 Tower kernel: R10: 00000000ffffffff R11: ffff880463739d20 R12: ffff88045351bcb0
Apr  8 03:40:01 Tower kernel: R13: ffff88011699fa60 R14: ffff88045351bcb0 R15: 0000000033c6363c
Apr  8 03:40:01 Tower kernel: FS:  00002b39177af700(0000) GS:ffff88047fc40000(0000) knlGS:0000000000000000
Apr  8 03:40:01 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr  8 03:40:01 Tower kernel: CR2: 00002ab0ad452000 CR3: 0000000455cde000 CR4: 00000000001006e0
Apr  8 03:40:01 Tower kernel: Stack:
Apr  8 03:40:01 Tower kernel: ffff880463739c00 ffff88045351bcb0 ffffc900034b1000 ffffc900034d11e8
Apr  8 03:40:01 Tower kernel: ffff88045351bcb0 ffff8800761f5000 ffff88045351bbc8 ffffffff811539c7
Apr  8 03:40:01 Tower kernel: ffff88045351bcb0 ffff8804694a90c0 ffffc900034b1000 ffffc900034b1000
Apr  8 03:40:01 Tower kernel: Call Trace:
Apr  8 03:40:01 Tower kernel: [<ffffffff811539c7>] reiserfs_discard_all_prealloc+0x44/0x4e
Apr  8 03:40:01 Tower kernel: [<ffffffff81170240>] do_journal_end+0x4e7/0xc78
Apr  8 03:40:01 Tower kernel: [<ffffffff81170f30>] journal_end+0xae/0xb6
Apr  8 03:40:01 Tower kernel: [<ffffffff8116161e>] reiserfs_dirty_inode+0x6c/0x7c
Apr  8 03:40:01 Tower kernel: [<ffffffff8104d340>] ? ns_capable+0x3a/0x4f
Apr  8 03:40:01 Tower kernel: [<ffffffff8111ce45>] __mark_inode_dirty+0x2f/0x265
Apr  8 03:40:01 Tower kernel: [<ffffffff8115d373>] reiserfs_setattr+0x262/0x297
Apr  8 03:40:01 Tower kernel: [<ffffffff810ff608>] ? __sb_start_write+0x9a/0xce
Apr  8 03:40:01 Tower kernel: [<ffffffff811099f2>] ? putname+0x46/0x4b
Apr  8 03:40:01 Tower kernel: [<ffffffff8111399a>] notify_change+0x1db/0x2dc
Apr  8 03:40:01 Tower kernel: [<ffffffff811167f8>] ? __mnt_want_write+0x4a/0x61
Apr  8 03:40:01 Tower kernel: [<ffffffff81121010>] utimes_common+0x114/0x18b
Apr  8 03:40:01 Tower kernel: [<ffffffff81121172>] do_utimes+0xeb/0x125
Apr  8 03:40:01 Tower kernel: [<ffffffff811212fb>] SyS_futimesat+0x7f/0x9a
Apr  8 03:40:01 Tower kernel: [<ffffffff8112132a>] SyS_utimes+0x14/0x16
Apr  8 03:40:01 Tower kernel: [<ffffffff815f74ee>] system_call_fastpath+0x12/0x71
Apr  8 03:40:01 Tower kernel: Code: 1c 75 bb 0f 0b 85 c0 74 12 48 8b 93 e8 00 00 00 4c 89 ee 4c 89 e7 e8 be 6e 00 00 48 8b 4b 28 44 89 7b 1c 48 8d 43 28 48 8b 53 30 <48> 89 51 08 48 89 0a 48 89 43 28 48 89 43 30 58 5b 41 5c 41 5d 
Apr  8 03:40:01 Tower kernel: RIP  [<ffffffff81153948>] __discard_prealloc+0x98/0xb3
Apr  8 03:40:01 Tower kernel: RSP <ffff88045351bb68>
Apr  8 03:40:01 Tower kernel: ---[ end trace ea9896d35b985763 ]---

Since 'reiserfs' is mentioned, try checking all data disks that are Reiser formatted, see Check Disk File systems.

I ran through a 'reiserfsck --check' on all disks and /dev/md10 came back with corruption issues.  Smart report looks fine on the disk, so I'm letting a 'reisersck --rebuild-tree' run overnight as suggested by the output of the check function.  This would have been a candidate disk for the mover to place things on in this case, so that very well could have been the issue.  Now I'm just hoping the recovery works with little loss.

 

Any ideas how this might have happened?  First time I've seen it after years of use, but it makes me a little concerned because it's not something parity can correct.

It's an extremely rare thing, but most likely you had a power cut or spike or nearby lightning strike while the drive was being written to, probably while the file system structures were being updated.  ReiserFS is known for being extraordinarily resilient, but nothing's perfect.

Fair enough, and thanks very much for the information.  The tree rebuild seems to have gone through with nothing in lost+found, so it seems like I may have escaped anything nasty.  Still poking around but haven't seen anything corrupted yet.  Whew.