Only thing happening on Unraid server was a lengthy mover process (moving 3TB of data from cache to array), but this has been running for at least 12 hours without any problems. Latest 6.x (b14). Array web interface is now unresponsive - only the log window keeps updating with more errors.

 

Apr 5 08:24:53 StorageArray kernel: INFO: rcu_sched self-detected stall on CPU { 1} (t=6001 jiffies g=1536126 c=1536125 q=41119)

Apr 5 08:24:53 StorageArray kernel: Task dump for CPU 1:

Apr 5 08:24:53 StorageArray kernel: shfs R running task 0 3897 1 0x00000008

Apr 5 08:24:53 StorageArray kernel: 0000000000000000 ffff880231043c08 ffffffff8105e0b5 0000000000000001

Apr 5 08:24:53 StorageArray kernel: 0000000000000001 ffff880231043c28 ffffffff81060780 0000000000000002

Apr 5 08:24:53 StorageArray kernel: ffffffff81834400 ffff880231043c58 ffffffff8107845f ffffffff81834400

Apr 5 08:24:53 StorageArray kernel: Call Trace:

Apr 5 08:24:53 StorageArray kernel: [] sched_show_task+0xbe/0xc3

Apr 5 08:24:53 StorageArray kernel: [] dump_cpu_task+0x35/0x39

Apr 5 08:24:53 StorageArray kernel: [] rcu_dump_cpu_stacks+0x6a/0x8c

Apr 5 08:24:53 StorageArray kernel: [] rcu_check_callbacks+0x1db/0x4f9

Apr 5 08:24:53 StorageArray kernel: [] ? tick_sched_handle+0x34/0x34

Apr 5 08:24:53 StorageArray kernel: [] update_process_times+0x3a/0x64

Apr 5 08:24:53 StorageArray kernel: [] tick_sched_handle+0x32/0x34

Apr 5 08:24:53 StorageArray kernel: [] tick_sched_timer+0x37/0x61

Apr 5 08:24:53 StorageArray kernel: [] __run_hrtimer.isra.29+0x57/0xb0

Apr 5 08:24:53 StorageArray kernel: [] hrtimer_interrupt+0xd9/0x1c0

Apr 5 08:24:53 StorageArray kernel: [] xen_timer_interrupt+0x2b/0x108

Apr 5 08:24:53 StorageArray kernel: [] ? add_interrupt_randomness+0x37/0x198

Apr 5 08:24:53 StorageArray kernel: [] handle_irq_event_percpu+0x29/0xf2

Apr 5 08:24:53 StorageArray kernel: [] ? info_for_irq+0x9/0x18

Apr 5 08:24:53 StorageArray kernel: [] handle_percpu_irq+0x39/0x4d

Apr 5 08:24:53 StorageArray kernel: [] generic_handle_irq+0x1a/0x27

Apr 5 08:24:53 StorageArray kernel: [] evtchn_fifo_handle_events+0x12d/0x156

Apr 5 08:24:53 StorageArray kernel: [] __xen_evtchn_do_upcall+0x4a/0x77

Apr 5 08:24:53 StorageArray kernel: [] xen_evtchn_do_upcall+0x2f/0x41

Apr 5 08:24:53 StorageArray kernel: [] xen_do_hypervisor_callback+0x1e/0x30

Apr 5 08:24:53 StorageArray kernel: [] ? __discard_prealloc+0xb2/0xb3

Apr 5 08:24:53 StorageArray kernel: [] ? reiserfs_discard_all_prealloc+0x44/0x4e

Apr 5 08:24:53 StorageArray kernel: [] ? do_journal_end+0x4e7/0xc78

Apr 5 08:24:53 StorageArray kernel: [] ? journal_end+0xae/0xb6

Apr 5 08:24:53 StorageArray kernel: [] ? reiserfs_do_truncate+0x2e2/0x425

Apr 5 08:24:53 StorageArray kernel: [] ? reiserfs_truncate_file+0x1b2/0x2cb

Apr 5 08:24:53 StorageArray kernel: [] ? truncate_pagecache+0x4e/0x56

Apr 5 08:24:53 StorageArray kernel: [] ? reiserfs_setattr+0x242/0x297

Apr 5 08:24:53 StorageArray kernel: [] ? notify_change+0x1dc/0x2d0

Apr 5 08:24:53 StorageArray kernel: [] ? do_truncate+0x64/0x89

Apr 5 08:24:53 StorageArray kernel: [] ? SyS_ftruncate+0x117/0x12e

Apr 5 08:24:53 StorageArray kernel: [] ? system_call_fastpath+0x12/0x17

 

Apr 5 08:27:53 StorageArray kernel: INFO: rcu_sched self-detected stall on CPU { 1} (t=24004 jiffies g=1536126 c=1536125 q=83482)

Apr 5 08:27:53 StorageArray kernel: Task dump for CPU 1:

Apr 5 08:27:53 StorageArray kernel: shfs R running task 0 3897 1 0x00000008

Apr 5 08:27:53 StorageArray kernel: 0000000000000000 ffff880231043c08 ffffffff8105e0b5 0000000000000001

Apr 5 08:27:53 StorageArray kernel: 0000000000000001 ffff880231043c28 ffffffff81060780 0000000000000002

Apr 5 08:27:53 StorageArray kernel: ffffffff81834400 ffff880231043c58 ffffffff8107845f ffffffff81834400

Apr 5 08:27:53 StorageArray kernel: Call Trace:

Apr 5 08:27:53 StorageArray kernel: [] sched_show_task+0xbe/0xc3

Apr 5 08:27:53 StorageArray kernel: [] dump_cpu_task+0x35/0x39

Apr 5 08:27:53 StorageArray kernel: [] rcu_dump_cpu_stacks+0x6a/0x8c

Apr 5 08:27:53 StorageArray kernel: [] rcu_check_callbacks+0x1db/0x4f9

Apr 5 08:27:53 StorageArray kernel: [] ? tick_sched_handle+0x34/0x34

Apr 5 08:27:53 StorageArray kernel: [] update_process_times+0x3a/0x64

Apr 5 08:27:53 StorageArray kernel: [] tick_sched_handle+0x32/0x34

Apr 5 08:27:53 StorageArray kernel: [] tick_sched_timer+0x37/0x61

Apr 5 08:27:53 StorageArray kernel: [] __run_hrtimer.isra.29+0x57/0xb0

Apr 5 08:27:53 StorageArray kernel: [] hrtimer_interrupt+0xd9/0x1c0

Apr 5 08:27:53 StorageArray kernel: [] xen_timer_interrupt+0x2b/0x108

Apr 5 08:27:53 StorageArray kernel: [] ? add_interrupt_randomness+0x37/0x198

Apr 5 08:27:53 StorageArray kernel: [] handle_irq_event_percpu+0x29/0xf2

Apr 5 08:27:53 StorageArray kernel: [] ? info_for_irq+0x9/0x18

Apr 5 08:27:53 StorageArray kernel: [] handle_percpu_irq+0x39/0x4d

Apr 5 08:27:53 StorageArray kernel: [] generic_handle_irq+0x1a/0x27

Apr 5 08:27:53 StorageArray kernel: [] evtchn_fifo_handle_events+0x12d/0x156

Apr 5 08:27:53 StorageArray kernel: [] __xen_evtchn_do_upcall+0x4a/0x77

Apr 5 08:27:53 StorageArray kernel: [] xen_evtchn_do_upcall+0x2f/0x41

Apr 5 08:27:53 StorageArray kernel: [] xen_do_hypervisor_callback+0x1e/0x30

Apr 5 08:27:53 StorageArray kernel: [] ? __discard_prealloc+0x62/0xb3

Apr 5 08:27:53 StorageArray kernel: [] ? reiserfs_discard_all_prealloc+0x44/0x4e

Apr 5 08:27:53 StorageArray kernel: [] ? do_journal_end+0x4e7/0xc78

Apr 5 08:27:53 StorageArray kernel: [] ? journal_end+0xae/0xb6

Apr 5 08:27:53 StorageArray kernel: [] ? reiserfs_do_truncate+0x2e2/0x425

Apr 5 08:27:53 StorageArray kernel: [] ? reiserfs_truncate_file+0x1b2/0x2cb

Apr 5 08:27:53 StorageArray kernel: [] ? truncate_pagecache+0x4e/0x56

Apr 5 08:27:53 StorageArray kernel: [] ? reiserfs_setattr+0x242/0x297

Apr 5 08:27:53 StorageArray kernel: [] ? notify_change+0x1dc/0x2d0

Apr 5 08:27:53 StorageArray kernel: [] ? do_truncate+0x64/0x89

Apr 5 08:27:53 StorageArray kernel: [] ? SyS_ftruncate+0x117/0x12e

Apr 5 08:27:53 StorageArray kernel: [] ? system_call_fastpath+0x12/0x17

 

Apr 5 08:30:53 StorageArray kernel: INFO: rcu_sched self-detected stall on CPU { 1} (t=42007 jiffies g=1536126 c=1536125 q=92332)

Apr 5 08:30:53 StorageArray kernel: Task dump for CPU 1:

Apr 5 08:30:53 StorageArray kernel: shfs R running task 0 3897 1 0x00000008

Apr 5 08:30:53 StorageArray kernel: 0000000000000000 ffff880231043c08 ffffffff8105e0b5 0000000000000001

Apr 5 08:30:53 StorageArray kernel: 0000000000000001 ffff880231043c28 ffffffff81060780 0000000000000002

Apr 5 08:30:53 StorageArray kernel: ffffffff81834400 ffff880231043c58 ffffffff8107845f ffffffff81834400

Apr 5 08:30:53 StorageArray kernel: Call Trace:

Apr 5 08:30:53 StorageArray kernel: [] sched_show_task+0xbe/0xc3

Apr 5 08:30:53 StorageArray kernel: [] dump_cpu_task+0x35/0x39

Apr 5 08:30:53 StorageArray kernel: [] rcu_dump_cpu_stacks+0x6a/0x8c

Apr 5 08:30:53 StorageArray kernel: [] rcu_check_callbacks+0x1db/0x4f9

Apr 5 08:30:53 StorageArray kernel: [] ? tick_sched_handle+0x34/0x34

Apr 5 08:30:53 StorageArray kernel: [] update_process_times+0x3a/0x64

Apr 5 08:30:53 StorageArray kernel: [] tick_sched_handle+0x32/0x34

Apr 5 08:30:53 StorageArray kernel: [] tick_sched_timer+0x37/0x61

Apr 5 08:30:53 StorageArray kernel: [] __run_hrtimer.isra.29+0x57/0xb0

Apr 5 08:30:53 StorageArray kernel: [] hrtimer_interrupt+0xd9/0x1c0

Apr 5 08:30:53 StorageArray kernel: [] xen_timer_interrupt+0x2b/0x108

Apr 5 08:30:53 StorageArray kernel: [] ? add_interrupt_randomness+0x37/0x198

Apr 5 08:30:53 StorageArray kernel: [] handle_irq_event_percpu+0x29/0xf2

Apr 5 08:30:53 StorageArray kernel: [] ? info_for_irq+0x9/0x18

Apr 5 08:30:53 StorageArray kernel: [] handle_percpu_irq+0x39/0x4d

Apr 5 08:30:53 StorageArray kernel: [] generic_handle_irq+0x1a/0x27

Apr 5 08:30:53 StorageArray kernel: [] evtchn_fifo_handle_events+0x12d/0x156

Apr 5 08:30:53 StorageArray kernel: [] __xen_evtchn_do_upcall+0x4a/0x77

Apr 5 08:30:53 StorageArray kernel: [] xen_evtchn_do_upcall+0x2f/0x41

Apr 5 08:30:53 StorageArray kernel: [] xen_do_hypervisor_callback+0x1e/0x30

Apr 5 08:30:53 StorageArray kernel: [] ? __discard_prealloc+0x13/0xb3

Apr 5 08:30:53 StorageArray kernel: [] ? reiserfs_discard_all_prealloc+0x44/0x4e

Apr 5 08:30:53 StorageArray kernel: [] ? do_journal_end+0x4e7/0xc78

Apr 5 08:30:53 StorageArray kernel: [] ? journal_end+0xae/0xb6

Apr 5 08:30:53 StorageArray kernel: [] ? reiserfs_do_truncate+0x2e2/0x425

Apr 5 08:30:53 StorageArray kernel: [] ? reiserfs_truncate_file+0x1b2/0x2cb

Apr 5 08:30:53 StorageArray kernel: [] ? truncate_pagecache+0x4e/0x56

Apr 5 08:30:53 StorageArray kernel: [] ? reiserfs_setattr+0x242/0x297

Apr 5 08:30:53 StorageArray kernel: [] ? notify_change+0x1dc/0x2d0

Apr 5 08:30:53 StorageArray kernel: [] ? do_truncate+0x64/0x89

Apr 5 08:30:53 StorageArray kernel: [] ? SyS_ftruncate+0x117/0x12e

Apr 5 08:30:53 StorageArray kernel: [] ? system_call_fastpath+0x12/0x17

 

Apr 5 08:33:53 StorageArray kernel: INFO: rcu_sched self-detected stall on CPU { 1} (t=60011 jiffies g=1536126 c=1536125 q=101114)

Apr 5 08:33:53 StorageArray kernel: Task dump for CPU 1:

Apr 5 08:33:53 StorageArray kernel: shfs R running task 0 3897 1 0x00000008

Apr 5 08:33:53 StorageArray kernel: 0000000000000000 ffff880231043c08 ffffffff8105e0b5 0000000000000001

Apr 5 08:33:53 StorageArray kernel: 0000000000000001 ffff880231043c28 ffffffff81060780 0000000000000002

Apr 5 08:33:53 StorageArray kernel: ffffffff81834400 ffff880231043c58 ffffffff8107845f ffffffff81834400

Apr 5 08:33:53 StorageArray kernel: Call Trace:

Apr 5 08:33:53 StorageArray kernel: [] sched_show_task+0xbe/0xc3

Apr 5 08:33:53 StorageArray kernel: [] dump_cpu_task+0x35/0x39

Apr 5 08:33:53 StorageArray kernel: [] rcu_dump_cpu_stacks+0x6a/0x8c

Apr 5 08:33:53 StorageArray kernel: [] rcu_check_callbacks+0x1db/0x4f9

Apr 5 08:33:53 StorageArray kernel: [] ? tick_sched_handle+0x34/0x34

Apr 5 08:33:53 StorageArray kernel: [] update_process_times+0x3a/0x64

Apr 5 08:33:53 StorageArray kernel: [] tick_sched_handle+0x32/0x34

Apr 5 08:33:53 StorageArray kernel: [] tick_sched_timer+0x37/0x61

Apr 5 08:33:53 StorageArray kernel: [] __run_hrtimer.isra.29+0x57/0xb0

Apr 5 08:33:53 StorageArray kernel: [] hrtimer_interrupt+0xd9/0x1c0

Apr 5 08:33:53 StorageArray kernel: [] xen_timer_interrupt+0x2b/0x108

Apr 5 08:33:53 StorageArray kernel: [] ? add_interrupt_randomness+0x37/0x198

Apr 5 08:33:53 StorageArray kernel: [] handle_irq_event_percpu+0x29/0xf2

Apr 5 08:33:53 StorageArray kernel: [] ? info_for_irq+0x9/0x18

Apr 5 08:33:53 StorageArray kernel: [] handle_percpu_irq+0x39/0x4d

Apr 5 08:33:53 StorageArray kernel: [] generic_handle_irq+0x1a/0x27

Apr 5 08:33:53 StorageArray kernel: [] evtchn_fifo_handle_events+0x12d/0x156

Apr 5 08:33:53 StorageArray kernel: [] __xen_evtchn_do_upcall+0x4a/0x77

Apr 5 08:33:53 StorageArray kernel: [] xen_evtchn_do_upcall+0x2f/0x41

Apr 5 08:33:53 StorageArray kernel: [] xen_do_hypervisor_callback+0x1e/0x30

Apr 5 08:33:53 StorageArray kernel: [] ? __discard_prealloc+0x1b/0xb3

Apr 5 08:33:53 StorageArray kernel: [] ? reiserfs_discard_all_prealloc+0x44/0x4e

Apr 5 08:33:53 StorageArray kernel: [] ? do_journal_end+0x4e7/0xc78

Apr 5 08:33:53 StorageArray kernel: [] ? journal_end+0xae/0xb6

Apr 5 08:33:53 StorageArray kernel: [] ? reiserfs_do_truncate+0x2e2/0x425

Apr 5 08:33:53 StorageArray kernel: [] ? reiserfs_truncate_file+0x1b2/0x2cb

Apr 5 08:33:53 StorageArray kernel: [] ? truncate_pagecache+0x4e/0x56

Apr 5 08:33:53 StorageArray kernel: [] ? reiserfs_setattr+0x242/0x297

Apr 5 08:33:53 StorageArray kernel: [] ? notify_change+0x1dc/0x2d0

Apr 5 08:33:53 StorageArray kernel: [] ? do_truncate+0x64/0x89

Apr 5 08:33:53 StorageArray kernel: [] ? SyS_ftruncate+0x117/0x12e

Apr 5 08:33:53 StorageArray kernel: [] ? system_call_fastpath+0x12/0x17

What filesystem are your disks?

At least one of them is RFS (ReiserFS) based on the stack traces.

Yep, they're all ReiserFS. I had to forcibly restart the server and so far everything is ok. It's possibly a hardware problem with the CPU (Xeon) because there's been some overheating. I'll keep an eye on it thanks.

Yep, they're all ReiserFS. I had to forcibly restart the server and so far everything is ok. It's possibly a hardware problem with the CPU (Xeon) because there's been some overheating. I'll keep an eye on it thanks.

Its not a hardware issue per se (although the issue only surfaces under specific hardware combinations).  What's happening is that under the right circumstances (heavy I/O) the OS's watchdog timer is missing its "beat"  So long as there are no major ill effects, I'd wait until the next beta / RC is released and see if that fixes it.  (according to one of jonP's recent posts it should be "any day now")

Yep, they're all ReiserFS. I had to forcibly restart the server and so far everything is ok. It's possibly a hardware problem with the CPU (Xeon) because there's been some overheating. I'll keep an eye on it thanks.

Its not a hardware issue per se (although the issue only surfaces under specific hardware combinations).  What's happening is that under the right circumstances (heavy I/O) the OS's watchdog timer is missing its "beat"  So long as there are no major ill effects, I'd wait until the next beta / RC is released and see if that fixes it.  (according to one of jonP's recent posts it should be "any day now")

 

Yup, its been in the "soon" territory for a few months now.

Yup, its been in the "soon" territory for a few months now.

You do mean the whole 41 days since 14b was released

Yup, its been in the "soon" territory for a few months now.

You do mean the whole 41 days since 14b was released

Yup. I recall seeing "soon" posted shortly there after, perhaps even back in February. Its now April.

 

That's actually good news because I was thinking I might need to replace my CPU :-)  I'm happy to wait for the fix to surface.

I suffered this a lot and made the transition to XFS.

I haven't had a problem since.

 

The process is very simple, as long as you have (or can clean up) a drive to enable you to shuffle data around.

 

I followed this guide: http://lime-technology.com/forum/index.php?topic=37490.msg346739#msg346739

While there are a number of ways of moving data around, I was in no hurry so used rsync.

Actually, I was thinking of changing filesystems now that unraid supports xfs and btrfs. Are there any pros and cons to the different filesystems compared to ReiserFS?

The topic of which filesystem to use has been highly discussed here in the forums.  Search is your friend ;-)

 

With respect to the stalls, there is a good chance the next release will resolve your issue.  It has for a few select test users to which we rolled out an internal development build.

There haven't been any further stalls since the mover finished copying all the cache contents to the array, so that's good. Yes I'll do some searching because I'm very interested about the differences between the various filesystems. I'm hoping that one of the new filesystems will offer some sort of file corruption/crc checking. Thanks.

There haven't been any further stalls since the mover finished copying all the cache contents to the array, so that's good. Yes I'll do some searching because I'm very interested about the differences between the various filesystems. I'm hoping that one of the new filesystems will offer some sort of file corruption/crc checking. Thanks.

 

btrfs will be your friend here.