Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

CPU stall error causing high load and unresponsive system

Ascii227
in General Support

Featured Replies

Hi unraiders, not having fun today. Syslog is reporting a cpu stall on docker, which in turn is causing some sort of IO disk errors. In top, Docker is reporting extremely high cpu usage, and I cant kill it. I think its due to the mounted share it is trying to access being unavailable due to the cpu error.

 

Unraid web gui is completely unresponsive

Netdata is working and reporting extremely high load and iowait times

"ls /mnt" just hangs the terminal

"umount -f /var/lib/docker" just returns "Target is busy"

"reboot" and "powerdown -r" just do nothing

 

I have included what my syslog is looping on below. Any advice on what to try is much appreciated. Thankyou!

 

Dec 22 20:22:50 AsQ-NAS kernel: INFO: rcu_bh self-detected stall on CPU
Dec 22 20:22:50 AsQ-NAS kernel: INFO: rcu_bh detected stalls on CPUs/tasks:
Dec 22 20:22:50 AsQ-NAS kernel:         6-....: (1 GPs behind) idle=912/1/4611686018427387906 softirq=3811864/3811866 fqs=15825140
Dec 22 20:22:50 AsQ-NAS kernel:          (t=63601766 jiffies g=-185 c=-186 q=130)
Dec 22 20:22:50 AsQ-NAS kernel:         6-....: (1 GPs behind) idle=912/1/4611686018427387906 softirq=3811864/3811866 fqs=15825140
Dec 22 20:22:50 AsQ-NAS kernel:         (detected by 1, t=63601769 jiffies, g=-185, c=-186, q=130)
Dec 22 20:22:50 AsQ-NAS kernel: NMI backtrace for cpu 6
Dec 22 20:22:50 AsQ-NAS kernel: CPU: 6 PID: 22098 Comm: docker Tainted: G    B D           4.18.17-unRAID #1
Dec 22 20:22:50 AsQ-NAS kernel: Hardware name: System manufacturer System Product Name/ROG STRIX Z370-G GAMING (WI-FI AC), BIOS 1601 10/29/2018
Dec 22 20:22:50 AsQ-NAS kernel: Call Trace:
Dec 22 20:22:50 AsQ-NAS kernel: <IRQ>
Dec 22 20:22:50 AsQ-NAS kernel: dump_stack+0x5d/0x79
Dec 22 20:22:50 AsQ-NAS kernel: nmi_cpu_backtrace+0x71/0x83
Dec 22 20:22:50 AsQ-NAS kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
Dec 22 20:22:50 AsQ-NAS kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7
Dec 22 20:22:50 AsQ-NAS kernel: rcu_dump_cpu_stacks+0x91/0xbb
Dec 22 20:22:50 AsQ-NAS kernel: rcu_check_callbacks+0x23f/0x5ca
Dec 22 20:22:50 AsQ-NAS kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
Dec 22 20:22:50 AsQ-NAS kernel: update_process_times+0x23/0x45
Dec 22 20:22:50 AsQ-NAS kernel: tick_sched_timer+0x36/0x64
Dec 22 20:22:50 AsQ-NAS kernel: __hrtimer_run_queues+0xb1/0x105
Dec 22 20:22:50 AsQ-NAS kernel: hrtimer_interrupt+0xf4/0x20d
Dec 22 20:22:50 AsQ-NAS kernel: smp_apic_timer_interrupt+0x79/0x89
Dec 22 20:22:50 AsQ-NAS kernel: apic_timer_interrupt+0xf/0x20
Dec 22 20:22:50 AsQ-NAS kernel: </IRQ>
Dec 22 20:22:50 AsQ-NAS kernel: RIP: 0010:filemap_map_pages+0x98/0x2b8
Dec 22 20:22:50 AsQ-NAS kernel: Code: 89 f8 83 e0 03 74 23 48 ff c8 0f 85 77 01 00 00 48 c7 44 24 20 00 00 00 00 48 8b 44 24 10 31 db 48 89 44 24 18 e9 89 01 00 00 <49> 8b 47 08 4c 89 ff a8 01 74 04 48 8d 78 ff 8b 47 34 85 c0 74 b3
Dec 22 20:22:50 AsQ-NAS kernel: RSP: 0000:ffffc9000da2bd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Dec 22 20:22:50 AsQ-NAS kernel: RAX: 0000000000000000 RBX: ffff8808259c75f8 RCX: 0000000000000000
Dec 22 20:22:50 AsQ-NAS kernel: RDX: 0000000000000001 RSI: 000000000096a000 RDI: ffffea0020621280
Dec 22 20:22:50 AsQ-NAS kernel: RBP: ffffc9000da2be10 R08: 000000000000056a R09: 0000000818849000
Dec 22 20:22:50 AsQ-NAS kernel: R10: 0000000000000580 R11: 0000000000000000 R12: ffff88081d69eaa0
Dec 22 20:22:50 AsQ-NAS kernel: R13: ffff880718c86a00 R14: 000000000000056a R15: ffffea0020621280
Dec 22 20:22:50 AsQ-NAS kernel: __handle_mm_fault+0xda0/0x10aa
Dec 22 20:22:50 AsQ-NAS kernel: handle_mm_fault+0x159/0x1a8
Dec 22 20:22:50 AsQ-NAS kernel: __do_page_fault+0x271/0x40b
Dec 22 20:22:50 AsQ-NAS kernel: ? page_fault+0x8/0x30
Dec 22 20:22:50 AsQ-NAS kernel: page_fault+0x1e/0x30
Dec 22 20:22:50 AsQ-NAS kernel: RIP: 0033:0x966ef0
Dec 22 20:22:50 AsQ-NAS kernel: Code: 4c 24 08 48 89 44 24 10 e8 dd 89 b0 ff e9 73 fe ff ff e8 53 cd ae ff e9 fe fd ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc <64> 48 8b 0c 25 f8 ff ff ff 48 3b 61 10 0f 86 b3 00 00 00 48 83 ec
Dec 22 20:22:50 AsQ-NAS kernel: RSP: 002b:000000c420205f08 EFLAGS: 00010202
Dec 22 20:22:50 AsQ-NAS kernel: RAX: 0000000000000002 RBX: 0000000000000040 RCX: 000000c420000180
Dec 22 20:22:50 AsQ-NAS kernel: RDX: 0000000001962020 RSI: 000000000158ad00 RDI: 000000c4202512a8
Dec 22 20:22:50 AsQ-NAS kernel: RBP: 000000c420205f30 R08: 000000c420251270 R09: 0000000000000047
Dec 22 20:22:50 AsQ-NAS kernel: R10: 0000000000000010 R11: 000000c420251270 R12: 0000000000000068
Dec 22 20:22:50 AsQ-NAS kernel: R13: 0000000000000018 R14: 0000000000000057 R15: 0000000000000100
Dec 22 20:22:50 AsQ-NAS kernel: Sending NMI from CPU 1 to CPUs 6:
Dec 22 20:22:50 AsQ-NAS kernel: NMI backtrace for cpu 6
Dec 22 20:22:50 AsQ-NAS kernel: CPU: 6 PID: 22098 Comm: docker Tainted: G    B D           4.18.17-unRAID #1
Dec 22 20:22:50 AsQ-NAS kernel: Hardware name: System manufacturer System Product Name/ROG STRIX Z370-G GAMING (WI-FI AC), BIOS 1601 10/29/2018
Dec 22 20:22:50 AsQ-NAS kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x103/0x16d
Dec 22 20:22:50 AsQ-NAS kernel: Code: 36 c1 e9 12 83 e0 03 ff c9 48 c1 e0 04 48 63 c9 48 05 c0 17 02 00 48 03 04 cd 00 17 da 81 48 89 10 8b 42 08 85 c0 75 04 f3 90 <eb> f5 48 8b 0a 48 85 c9 74 c9 0f 0d 09 8b 07 66 85 c0 74 04 f3 90
Dec 22 20:22:50 AsQ-NAS kernel: RSP: 0000:ffff880826383e90 EFLAGS: 00000046
Dec 22 20:22:50 AsQ-NAS kernel: RAX: 0000000000000000 RBX: 0000000000000046 RCX: 0000000000000008
Dec 22 20:22:50 AsQ-NAS kernel: RDX: ffff8808263a17c0 RSI: 00000000001c0000 RDI: ffffffff81e38ac0
Dec 22 20:22:50 AsQ-NAS kernel: RBP: ffff8808263a1840 R08: 000000000000000f R09: ffff88081fb53b00
Dec 22 20:22:50 AsQ-NAS kernel: R10: 0000000000000001 R11: 000000000000002b R12: 0000000000000046
Dec 22 20:22:50 AsQ-NAS kernel: R13: ffff8808263a1840 R14: 0000000000000000 R15: ffffffff81e38ac0
Dec 22 20:22:50 AsQ-NAS kernel: FS:  00000000022e9a90(0000) GS:ffff880826380000(0000) knlGS:0000000000000000
Dec 22 20:22:50 AsQ-NAS kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 22 20:22:50 AsQ-NAS kernel: CR2: 0000000000966ef0 CR3: 0000000718ad0003 CR4: 00000000003606e0
Dec 22 20:22:50 AsQ-NAS kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 22 20:22:50 AsQ-NAS kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 22 20:22:50 AsQ-NAS kernel: Call Trace:
Dec 22 20:22:50 AsQ-NAS kernel: <IRQ>
Dec 22 20:22:50 AsQ-NAS kernel: _raw_spin_lock_irqsave+0x28/0x2f
Dec 22 20:22:50 AsQ-NAS kernel: rcu_check_callbacks+0x247/0x5ca
Dec 22 20:22:50 AsQ-NAS kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
Dec 22 20:22:50 AsQ-NAS kernel: update_process_times+0x23/0x45
Dec 22 20:22:50 AsQ-NAS kernel: tick_sched_timer+0x36/0x64
Dec 22 20:22:50 AsQ-NAS kernel: __hrtimer_run_queues+0xb1/0x105
Dec 22 20:22:50 AsQ-NAS kernel: hrtimer_interrupt+0xf4/0x20d
Dec 22 20:22:50 AsQ-NAS kernel: smp_apic_timer_interrupt+0x79/0x89
Dec 22 20:22:50 AsQ-NAS kernel: apic_timer_interrupt+0xf/0x20
Dec 22 20:22:50 AsQ-NAS kernel: </IRQ>
Dec 22 20:22:50 AsQ-NAS kernel: RIP: 0010:filemap_map_pages+0x98/0x2b8
Dec 22 20:22:50 AsQ-NAS kernel: Code: 89 f8 83 e0 03 74 23 48 ff c8 0f 85 77 01 00 00 48 c7 44 24 20 00 00 00 00 48 8b 44 24 10 31 db 48 89 44 24 18 e9 89 01 00 00 <49> 8b 47 08 4c 89 ff a8 01 74 04 48 8d 78 ff 8b 47 34 85 c0 74 b3
Dec 22 20:22:50 AsQ-NAS kernel: RSP: 0000:ffffc9000da2bd90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Dec 22 20:22:50 AsQ-NAS kernel: RAX: 0000000000000000 RBX: ffff8808259c75f8 RCX: 0000000000000000
Dec 22 20:22:50 AsQ-NAS kernel: RDX: 0000000000000001 RSI: 000000000096a000 RDI: ffffea0020621280
Dec 22 20:22:50 AsQ-NAS kernel: RBP: ffffc9000da2be10 R08: 000000000000056a R09: 0000000818849000
Dec 22 20:22:50 AsQ-NAS kernel: R10: 0000000000000580 R11: 0000000000000000 R12: ffff88081d69eaa0
Dec 22 20:22:50 AsQ-NAS kernel: R13: ffff880718c86a00 R14: 000000000000056a R15: ffffea0020621280
Dec 22 20:22:50 AsQ-NAS kernel: __handle_mm_fault+0xda0/0x10aa
Dec 22 20:22:50 AsQ-NAS kernel: handle_mm_fault+0x159/0x1a8
Dec 22 20:22:50 AsQ-NAS kernel: __do_page_fault+0x271/0x40b
Dec 22 20:22:50 AsQ-NAS kernel: ? page_fault+0x8/0x30
Dec 22 20:22:50 AsQ-NAS kernel: page_fault+0x1e/0x30
Dec 22 20:22:50 AsQ-NAS kernel: RIP: 0033:0x966ef0
Dec 22 20:22:50 AsQ-NAS kernel: Code: 4c 24 08 48 89 44 24 10 e8 dd 89 b0 ff e9 73 fe ff ff e8 53 cd ae ff e9 fe fd ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc <64> 48 8b 0c 25 f8 ff ff ff 48 3b 61 10 0f 86 b3 00 00 00 48 83 ec
Dec 22 20:22:50 AsQ-NAS kernel: RSP: 002b:000000c420205f08 EFLAGS: 00010202
Dec 22 20:22:50 AsQ-NAS kernel: RAX: 0000000000000002 RBX: 0000000000000040 RCX: 000000c420000180
Dec 22 20:22:50 AsQ-NAS kernel: RDX: 0000000001962020 RSI: 000000000158ad00 RDI: 000000c4202512a8
Dec 22 20:22:50 AsQ-NAS kernel: RBP: 000000c420205f30 R08: 000000c420251270 R09: 0000000000000047
Dec 22 20:22:50 AsQ-NAS kernel: R10: 0000000000000010 R11: 000000c420251270 R12: 0000000000000068
Dec 22 20:22:50 AsQ-NAS kernel: R13: 0000000000000018 R14: 0000000000000057 R15: 0000000000000100

Edited by Ascii227
Added more symptoms for clarity

  • Quote
    • 4 years later...

    Shot in the dark considering how old this thread is, but were you able to determine what the cause of this was? I can't seem to replicate it on demand so it's hard to tell what causes dockerd to cause CPU stalls/spinlocks.

  • Quote

    • Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Reply to this topic...
    Go to topic listing

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.