unRAID OS Version: 6.0b12 with 15 (1-4TB) ReiserFS drives and a 500GB ReiserFS cache disk.

 

Description: Repeated CPU stall on netatalk in syslog, seem to be caused by ReiserFS. Unable to unmount drives and shut down cleanly. The syslog entries repeat every 3 minutes and look like this:

 

Jan 24 11:54:01 FileServer kernel: INFO: rcu_sched self-detected stall on CPU { 1}  (t=6000 jiffies g=13394302 c=13394301 q=35368)
Jan 24 11:54:01 FileServer kernel: Task dump for CPU 1:
Jan 24 11:54:01 FileServer kernel: afpd            R  running task        0  2838   2271 0x0000000c
Jan 24 11:54:01 FileServer kernel: 0000000000000000 ffff88019bc83de8 ffffffff8105cc09 0000000000000001
Jan 24 11:54:01 FileServer kernel: 0000000000000001 ffff88019bc83e00 ffffffff8105f2c4 ffffffff81822d00
Jan 24 11:54:01 FileServer kernel: ffff88019bc83e30 ffffffff810766a5 ffffffff81822d00 ffff88019bc8e0c0
Jan 24 11:54:01 FileServer kernel: Call Trace:
Jan 24 11:54:01 FileServer kernel: <IRQ>  [<ffffffff8105cc09>] sched_show_task+0xbe/0xc3
Jan 24 11:54:01 FileServer kernel: [<ffffffff8105f2c4>] dump_cpu_task+0x34/0x38
Jan 24 11:54:01 FileServer kernel: [<ffffffff810766a5>] rcu_dump_cpu_stacks+0x6a/0x8c
Jan 24 11:54:01 FileServer kernel: [<ffffffff81078ead>] rcu_check_callbacks+0x1e1/0x4ff
Jan 24 11:54:01 FileServer kernel: [<ffffffff81086659>] ? tick_sched_handle+0x34/0x34
Jan 24 11:54:01 FileServer kernel: [<ffffffff8107ac1a>] update_process_times+0x38/0x60
Jan 24 11:54:01 FileServer kernel: [<ffffffff81086657>] tick_sched_handle+0x32/0x34
Jan 24 11:54:01 FileServer kernel: [<ffffffff8108668e>] tick_sched_timer+0x35/0x53
Jan 24 11:54:01 FileServer kernel: [<ffffffff8107b149>] __run_hrtimer.isra.29+0x57/0xb0
Jan 24 11:54:01 FileServer kernel: [<ffffffff8107b634>] hrtimer_interrupt+0xd9/0x1c0
Jan 24 11:54:01 FileServer kernel: [<ffffffff8102ea78>] local_apic_timer_interrupt+0x4f/0x52
Jan 24 11:54:01 FileServer kernel: [<ffffffff8102ee4a>] smp_apic_timer_interrupt+0x3a/0x4b
Jan 24 11:54:01 FileServer kernel: [<ffffffff815ead9d>] apic_timer_interrupt+0x6d/0x80
Jan 24 11:54:01 FileServer kernel: <EOI>  [<ffffffff81147b9a>] ? reiserfs_discard_all_prealloc+0x3b/0x4c
Jan 24 11:54:01 FileServer kernel: [<ffffffff81163ed6>] do_journal_end+0x4e1/0xc57
Jan 24 11:54:01 FileServer kernel: [<ffffffff811648cf>] ? do_journal_begin_r+0x283/0x2b7
Jan 24 11:54:01 FileServer kernel: [<ffffffff811155e8>] ? do_fsync+0x48/0x48
Jan 24 11:54:01 FileServer kernel: [<ffffffff81164d7e>] journal_end_sync+0x60/0x65
Jan 24 11:54:01 FileServer kernel: [<ffffffff81155445>] reiserfs_sync_fs+0x2f/0x4c
Jan 24 11:54:01 FileServer kernel: [<ffffffff81115603>] sync_fs_one_sb+0x1b/0x1d
Jan 24 11:54:01 FileServer kernel: [<ffffffff810f63b2>] iterate_supers+0x70/0xbd
Jan 24 11:54:01 FileServer kernel: [<ffffffff811156d1>] sys_sync+0x3e/0x7c
Jan 24 11:54:01 FileServer kernel: [<ffffffff815e9fa9>] system_call_fastpath+0x16/0x1b

 

How to reproduce: Have some almost-full reiserFS disks. Setup a Timemachine share (The TimeMachine share is confined to disk1 with 392GB free and uses reiserfs Cache disk with 213GB free) . Start Timemachine backup on mac.

 

Expected results: Just backup the ~100GB worth of mac data.

 

Actual results:

 

- Timemachine never finishes.

- Webinterface stopped responding.

- Syslog has multiple CPU stall entries quoting ReiserFS (see attached).

- System load avg is 55 and climbing with no user tasks (all system and the occational kworker blip seen in top).

- Unable to unmount /dev/md1 - drive busy..

- fuser -mv /mnt/disk* hangs the console window (same with fuser -mv /mnt/disk1)

- Unable to unmount /dev/md2 - console window hangs (but the disk actually did unmount - /mnt/disk2 is empty).

- Unable to kill a screen MC instance with kill -9 [PID] (the process persists after kill -9).

- Accessing /mnt/user or /mnt/disk14 hangs the MC instance.

- "rc.docker stop" fails, says unable to unmount image.

- lsof /var/lib/docker produces no output and hangs the console session (ctrl-C soedn't work).

 

Other information:

 

I'm at a loss here. Looks like I won't be able to shot down cleanly after this one.

 

Prior to this situation, I have had some trouble with full reiserfs disks - I tried to move more files onto a drive than the space available, and it hung the server. I could ping it, but not establish contact with webinterface/unmenu/etc or start a terminal session. Had to hard power off (fortunately parity sync completed without errors, but did I suffer reiserfs corruption??)

 

My theory is that in the current situation disk14 had become too full for reiserfs to handle. I can't check it now, but I thought it had 100+GB free, but I could be wrong. In any case, I think due to some kind of write to that disk (that I'm not sure what caused), reiserFS started looking for free space and hung the kernel. Afpd then keeps stalling due to reiserfs not cooperating.

 

Webinterface stopped working, so I can't disable afp from there, and I can't figure out how to kill it per command line. Too dumb, I guess (well, in my defense, the afpd processes do magically keep respawning after I kill'em..).

 

Out of ideas. Help?

 

How to shut down cleanly in a situation like this?

 

I'm down to my last terminal window. Have already increased to 9 pts seats with echo "pts/8" >>/etc/securetty (and pts/9).

 

Will wait a little while, hoping for some good advice..

syslog-2015-01-24_ReiserFS_crashes.zip

Followup: I ended up installing the powerdown plugin manually from console, and executed that. Server did not reboot, but became unresponsive. Could not connect telnet. Could still ping it, but appeared dead.

 

After hard reseting it, server booted fine and parity check completed without errors (well, it's at 98% now with 0 so far).

 

Turns out my disk14 has 185GB free. Should not have been a problem with that drive. I can write to disk14 fine following the reboot.

 

I'm really hesitant to start another timemachine backup..