Hi guys this crashing has me pretty stumped and not sure where to start to trouble shoot.

 

About a week ago the Mover process on my server started crashing the linux kernel, the Server goes unresponsive and I must physically hit the reset button on it. I cannot think of anything that has changed apart from the adding of OSX mavericks onto the network.

 

I have disabled all plugins and just run 5.0 and Simple features. If I clear everything on my cache disk mover will run and move nothing but it starts to build up with files again and then will crash.

 

Any help would be appreciated.

 

Cheers

Will

 

Oct 28 20:26:53 Storageking emhttp_event: svcs_restarted

Oct 28 20:28:50 Storageking emhttp: shcmd (74): /usr/local/sbin/mover |& logger &

Oct 28 20:28:50 Storageking logger: mover started

Oct 28 20:28:50 Storageking logger: skipping Apps/

Oct 28 20:28:50 Storageking logger: moving TV Shows/

Oct 28 20:28:50 Storageking logger: ./TV Shows/insert TV show name here.avi

Oct 28 20:28:52 Storageking logger: .d........x TV Shows/

Oct 28 20:29:58 Storageking logger: rsync: get_xattr_names: llistxattr("TV Shows",1024) failed: Input/output error (5)

Oct 28 20:29:58 Storageking logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

Oct 28 20:29:58 Storageking kernel: BUG: unable to handle kernel paging request at 2df67000

Oct 28 20:29:58 Storageking kernel: IP: [] mntget+0xa/0xf

Oct 28 20:29:58 Storageking kernel: *pdpt = 000000002b601001 *pde = 0000000000000000

Oct 28 20:29:58 Storageking kernel: Oops: 0002 [#1] SMP

Oct 28 20:29:58 Storageking kernel: Modules linked in: md_mod bonding pata_jmicron sg coretemp hwmon r8168(O) ata_piix i2c_i801 jmicron i2c_core mvsas libsas scsi_transport_sas ahci libahci

Oct 28 20:29:58 Storageking kernel: Pid: 12150, comm: fuser Tainted: G          O 3.9.6p-unRAID #23 Gigabyte Technology Co., Ltd. X58A-UD7/X58A-UD7

Oct 28 20:29:58 Storageking kernel: EIP: 0060:[] EFLAGS: 00010206 CPU: 2

Oct 28 20:29:58 Storageking kernel: EIP is at mntget+0xa/0xf

Oct 28 20:29:58 Storageking kernel: EAX: 50ef3105 EBX: eeb50051 ECX: 80c15605 EDX: 00000000

Oct 28 20:29:58 Storageking kernel: ESI: f7493e20 EDI: eeb93600 EBP: f7493df0 ESP: f7493df0

Oct 28 20:29:58 Storageking kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068

Oct 28 20:29:58 Storageking kernel: CR0: 8005003b CR2: 2df67000 CR3: 2bbd1000 CR4: 000007f0

Oct 28 20:29:58 Storageking kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000

Oct 28 20:29:58 Storageking kernel: DR6: ffff0ff0 DR7: 00000400

Oct 28 20:29:58 Storageking kernel: Process fuser (pid: 12150, ti=f7492000 task=f6598a20 task.ti=f7492000)

Oct 28 20:29:58 Storageking kernel: Stack:

Oct 28 20:29:58 Storageking kernel:  f7493dfc c109d1c1 00000000 f7493e14 c10d32fb f7022520 eef4a9c0 eee3f900

Oct 28 20:29:58 Storageking kernel:  f7493e20 f7493e34 c10cec9e f7493ea8 50ef3105 80c15605 f7493ea8 eee3f900

Oct 28 20:29:58 Storageking kernel:  f6598a20 f7493e7c c109ee04 f7493ea8 f7493e6c f7493e0a f6598a20 f7493ebc

Oct 28 20:29:58 Storageking kernel: Call Trace:

Oct 28 20:29:58 Storageking kernel:  [] path_get+0xd/0x26

Oct 28 20:29:58 Storageking kernel:  [] proc_fd_link+0x79/0x9a

Oct 28 20:29:58 Storageking kernel:  [] proc_pid_follow_link+0x2d/0x45

Oct 28 20:29:58 Storageking kernel:  [] path_lookupat+0x28a/0x558

Oct 28 20:29:58 Storageking kernel:  [] filename_lookup+0x20/0x52

Oct 28 20:29:58 Storageking kernel:  [] user_path_at_empty+0x41/0x6c

Oct 28 20:29:58 Storageking kernel:  [] ? generic_block_fiemap+0x58/0x58

Oct 28 20:29:58 Storageking logger: find: `fuser' terminated by signal 9

Oct 28 20:29:58 Storageking kernel:  [] user_path_at+0x1a/0x1f

Oct 28 20:29:58 Storageking kernel:  [] vfs_fstatat+0x49/0x8c

Oct 28 20:29:58 Storageking kernel:  [] vfs_stat+0x1b/0x1d

Oct 28 20:29:58 Storageking kernel:  [] sys_stat64+0x14/0x28

Oct 28 20:29:58 Storageking kernel:  [] ? vfs_readdir+0x67/0x78

Oct 28 20:29:58 Storageking kernel:  [] ? generic_block_fiemap+0x58/0x58

Oct 28 20:29:58 Storageking kernel:  [] ? sys_getdents64+0x70/0xbe

Oct 28 20:29:58 Storageking kernel:  [] syscall_call+0x7/0xb

Oct 28 20:29:58 Storageking kernel: Code: 8b 5b 04 39 f3 75 04 31 c0 eb 11 8b 7b 08 83 c7 10 39 f8 75 e3 39 53 0c 75 de 89 d8 5b 5e 5f 5d c3 55 85 c0 89 e5 74 06 8b 50 0c <64> ff 02 5d c3 55 89 c1 89 e5 83 ea 2c ff 51 54 5d c3 55 89 e5

Oct 28 20:29:58 Storageking kernel: EIP: [] mntget+0xa/0xf SS:ESP 0068:f7493df0

Oct 28 20:29:58 Storageking kernel: CR2: 000000002df67000

Oct 28 20:29:58 Storageking kernel: ---[ end trace ab13e3e7d93b450e ]---

Oct 28 20:30:58 Storageking kernel: INFO: rcu_sched self-detected stall on CPU { 3}  (t=6000 jiffies g=772 c=771 q=3388)

Oct 28 20:30:58 Storageking kernel: Pid: 12152, comm: fuser Tainted: G      D    O 3.9.6p-unRAID #23

Oct 28 20:30:58 Storageking kernel: Call Trace:

Oct 28 20:30:58 Storageking kernel:  [] print_cpu_stall+0xbc/0x107

Oct 28 20:30:58 Storageking kernel:  [] __rcu_pending+0x4f/0x12a

Oct 28 20:30:58 Storageking kernel:  [] rcu_check_callbacks+0x73/0x9b

Oct 28 20:30:58 Storageking kernel:  [] update_process_times+0x2d/0x53

Oct 28 20:30:58 Storageking kernel:  [] tick_sched_timer+0x77/0xa1

Oct 28 20:30:58 Storageking kernel:  [] ? __remove_hrtimer+0x25/0x7a

Oct 28 20:30:58 Storageking kernel:  [] __run_hrtimer+0x45/0xaf

Oct 28 20:30:58 Storageking kernel:  [] hrtimer_interrupt+0xf1/0x1e7

Oct 28 20:30:58 Storageking kernel:  [] ? __do_softirq+0x12f/0x151

Oct 28 20:30:58 Storageking kernel:  [] smp_apic_timer_interrupt+0x6d/0x7f

Oct 28 20:30:58 Storageking kernel:  [] apic_timer_interrupt+0x2d/0x34

Oct 28 20:30:58 Storageking kernel:  [] ? _raw_spin_lock+0x19/0x1f

Oct 28 20:30:58 Storageking kernel:  [] proc_fd_link+0x51/0x9a

Oct 28 20:30:58 Storageking kernel:  [] proc_pid_follow_link+0x2d/0x45

Oct 28 20:30:58 Storageking kernel:  [] ? touch_atime+0x71/0x106

Oct 28 20:30:58 Storageking kernel:  [] path_lookupat+0x28a/0x558

Oct 28 20:30:58 Storageking kernel:  [] filename_lookup+0x20/0x52

Oct 28 20:30:58 Storageking kernel:  [] user_path_at_empty+0x41/0x6c

Oct 28 20:30:58 Storageking kernel:  [] ? generic_block_fiemap+0x58/0x58

Oct 28 20:30:58 Storageking kernel:  [] user_path_at+0x1a/0x1f

Oct 28 20:30:58 Storageking kernel:  [] vfs_fstatat+0x49/0x8c

Oct 28 20:30:58 Storageking kernel:  [] vfs_stat+0x1b/0x1d

Oct 28 20:30:58 Storageking kernel:  [] sys_stat64+0x14/0x28

Oct 28 20:30:58 Storageking kernel:  [] ? vfs_readdir+0x67/0x78

Oct 28 20:30:58 Storageking kernel:  [] ? generic_block_fiemap+0x58/0x58

Oct 28 20:30:58 Storageking kernel:  [] ? sys_getdents64+0x70/0xbe

Oct 28 20:30:58 Storageking kernel:  [] syscall_call+0x7/0xb

Oct 28 20:33:58 Storageking kernel: INFO: rcu_sched self-detected stall on CPU { 3}  (t=24003 jiffies g=772 c=771 q=24545)

I would put the array into maintenance mode, and then run reiserfsck checks on all of your disks.  I have seen file system corruption have unexpected side-effects.

 

e.g. for each drive run the following command from a telnet/console session

reiserfsck /dev/md??

where ?? corresponds to each of your disk.  If running from telnet then you may want to use screen to allow you to run multiple checks in parallel and not have them abort if you lose the telnet session.

See here: http://lime-technology.com/forum/index.php?topic=28484.0

I would put the array into maintenance mode, and then run reiserfsck checks on all of your disks.  I have seen file system corruption have unexpected side-effects.

 

e.g. for each drive run the following command from a telnet/console session

reiserfsck /dev/md??

where ?? corresponds to each of your disk.  If running from telnet then you may want to use screen to allow you to run multiple checks in parallel and not have them abort if you lose the telnet session.

 

I've checked all 9 disks and reported no issues.

 

See here: http://lime-technology.com/forum/index.php?topic=28484.0

 

I went through each of my disks back when I had R13 and at that time it did report issues and fix them from memory. It going back a few months now though!  It is something I should run again?

 

Thanks for your input guys

 

I would put the array into maintenance mode, and then run reiserfsck checks on all of your disks.  I have seen file system corruption have unexpected side-effects.

 

e.g. for each drive run the following command from a telnet/console session

reiserfsck /dev/md??

where ?? corresponds to each of your disk.  If running from telnet then you may want to use screen to allow you to run multiple checks in parallel and not have them abort if you lose the telnet session.

 

I've checked all 9 disks and reported no issues.

 

See here: http://lime-technology.com/forum/index.php?topic=28484.0

 

I went through each of my disks back when I had R13 and at that time it did report issues and fix them from memory. It going back a few months now though!  It is something I should run again?

 

Thanks for your input guys

 

It will do no harm.

Running a Parity check is crashing the Kernel now, my monitor attached to unraid is saying "Fixing recursive fault reboot is needed!"

 

anyone got any other ideas? I did replace a disk recently, it rebuilt successfully thought.

 

 

I have re-done all the file attribute commands on all disks and directory's, I have noticed this soon as I evoked the mover in telnet.

 

Does this indicate something is wrong with a disk or smartctl .

 

Thanks

Will

 

 

Message from syslogd@Storageking at Thu Oct 31 16:13:10 2013 ...

Storageking kernel: Process smartctl (pid: 23076, ti=e61fe000 task=f66f1e60 task

.ti=e61fe000)

 

Message from syslogd@Storageking at Thu Oct 31 16:13:10 2013 ...

Storageking kernel: Stack:

 

Message from syslogd@Storageking at Thu Oct 31 16:13:10 2013 ...

Storageking kernel: Call Trace:

 

Run memtest overnight.

See here: http://lime-technology.com/forum/index.php?topic=28484.0

 

Thanks dgaschk,

This ended up being the problem, I found 1 directory on  my cache i missed. OSX mavericks did some weird stuff with AFP so i have disabled that in unraid now.

 

Cheers

Will

 

thank you for posting this, I would never have found my mover crash solution if not for you.