Unraid freezes when trying to copy large file from cache drive to array

[Lake-end]

Hello, I´ve been using this unraid setup for few years now but I am now facing problem I need some outside help with.

 

I was trying to download my VM image file from my SSD cache drive (where I run all my VMs) to backup storage on my regulard HDD array.

I am loggin on the Unraid via SSH and tried few things just regular cp, rsync etc operations to move this 100gb image file. I am getting consistent system crash trying to do this operation and from outside it looked like system just froze. When I had tail syslog running on second SSH window while doing this I could actually see the error happening. Problem is I dont understand what is happening. (log attached below)

 

I´ve searched around for solution and so far ran the xfs repair for the Array drive and scrubbing for the SSD cache pool. Also I ran SMART tests for drives involved and they showed green and this doesnt "feel" like failing drive to me.

 

One further issue I now understand might be related is following that started happening last week:

I run unraid as my daily gaming/work computer by running a Windows 10 VM on it and using it for pretty much all my computing needs. Last week I noticed my VM started straight out crashing without any message (like someone switched of power from the VM) whenever Steam was trying to download large updates for games. This is also 100% reproducable, I need to pause all downloads in Steam or Ill get a crash. My VMs run on SSD cache pool.

 

Anyone have any clues on what could be happening? Also I realize I am not running latest version and I am not asking for any fixes to be made to old version, just some support on figuring what is wrong in my system that used to run stable.

 

Cheers!

 

May 19 14:47:44 Megathron dnsmasq-dhcp[5182]: DHCP, sockets bound exclusively to interface virbr0
May 19 14:47:44 Megathron dnsmasq[5182]: reading /etc/resolv.conf
May 19 14:47:44 Megathron dnsmasq[5182]: using nameserver 8.8.8.8#53
May 19 14:47:44 Megathron dnsmasq[5182]: using nameserver 8.8.4.4#53
May 19 14:47:44 Megathron dnsmasq[5182]: using nameserver 192.168.0.1#53
May 19 14:47:44 Megathron dnsmasq[5182]: read /etc/hosts - 2 addresses
May 19 14:47:44 Megathron dnsmasq[5182]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
May 19 14:47:44 Megathron dnsmasq-dhcp[5182]: read /var/lib/libvirt/dnsmasq/default.hostsfile
May 19 14:47:44 Megathron kernel: virbr0: port 1(virbr0-nic) entered disabled state
May 19 14:47:46 Megathron sshd[4792]: Accepted password for root from 192.168.0.210 port 51339 ssh2
May 19 14:48:02 Megathron sshd[5327]: Accepted password for root from 192.168.0.210 port 51374 ssh2
May 19 14:48:45 Megathron kernel: general protection fault: 0000 [#1] PREEMPT SMP
May 19 14:48:45 Megathron kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod x86_pkg_temp_thermal coretemp e1000e mxm_wmi i2c_i801 ptp i2c_smbus i2c_core pps_core kvm_intel ahci libahci kvm video wmi backlight [last unloaded: md_mod]
May 19 14:48:45 Megathron kernel: CPU: 1 PID: 335 Comm: kworker/1:1 Not tainted 4.9.30-unRAID #1
May 19 14:48:45 Megathron kernel: Hardware name: System manufacturer System Product Name/MAXIMUS VIII GENE, BIOS 2202 09/19/2016
May 19 14:48:45 Megathron kernel: Workqueue: xfs-data/md1 xfs_end_io
May 19 14:48:45 Megathron kernel: task: ffff88085a199a80 task.stack: ffffc90003920000
May 19 14:48:45 Megathron kernel: RIP: 0010:[<ffffffff8107dd9f>]  [<ffffffff8107dd9f>] native_queued_spin_lock_slowpath+0x12d/0x17e
May 19 14:48:45 Megathron kernel: RSP: 0018:ffffc90003923d60  EFLAGS: 00010282
May 19 14:48:45 Megathron kernel: RAX: 0000000000000775 RBX: ffff8807798bdaf8 RCX: ff79afd600018686
May 19 14:48:45 Megathron kernel: RDX: ffff88087ec58600 RSI: 0000000000080000 RDI: ffff8807798bdb70
May 19 14:48:45 Megathron kernel: RBP: ffffc90003923d60 R08: 0000000000000001 R09: ffff88087eff8000
May 19 14:48:45 Megathron kernel: R10: ffff8807798bdaf8 R11: ffffffffffffffe5 R12: 0000000000000001
May 19 14:48:45 Megathron kernel: R13: 0000000000000000 R14: 00000000000000dc R15: ffff88082dd13dc0
May 19 14:48:45 Megathron kernel: FS:  0000000000000000(0000) GS:ffff88087ec40000(0000) knlGS:0000000000000000
May 19 14:48:45 Megathron kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 19 14:48:45 Megathron kernel: CR2: 00002b9ee126b570 CR3: 0000000001c09000 CR4: 00000000003406e0
May 19 14:48:45 Megathron kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 19 14:48:45 Megathron kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 19 14:48:45 Megathron kernel: Stack:
May 19 14:48:45 Megathron kernel: ffffc90003923d70 ffffffff8167f0c0 ffffc90003923d90 ffffffff811490fd
May 19 14:48:45 Megathron kernel: ffff88082dcaa700 ffff88082dcaac00 ffffc90003923df0 ffffffff81295df7
May 19 14:48:45 Megathron kernel: 00000000812b2f4b ffff8807798bdaf8 ffff8807798bdaf8 00000fff00001000
May 19 14:48:45 Megathron kernel: Call Trace:
May 19 14:48:45 Megathron kernel: [<ffffffff8167f0c0>] _raw_spin_lock+0x21/0x25
May 19 14:48:45 Megathron kernel: [<ffffffff811490fd>] attach_nobh_buffers+0x32/0x92
May 19 14:48:45 Megathron kernel: [<ffffffff81295df7>] xfs_destroy_ioend+0xac/0xe2
May 19 14:48:45 Megathron kernel: [<ffffffff81295ebf>] xfs_end_io+0x92/0x99
May 19 14:48:45 Megathron kernel: [<ffffffff8105ed53>] process_one_work+0x192/0x295
May 19 14:48:45 Megathron kernel: [<ffffffff8105f752>] worker_thread+0x27d/0x369
May 19 14:48:45 Megathron kernel: [<ffffffff8105f4d5>] ? rescuer_thread+0x2b1/0x2b1
May 19 14:48:45 Megathron kernel: [<ffffffff81063939>] kthread+0xdb/0xe3
May 19 14:48:45 Megathron kernel: [<ffffffff8110c273>] ? kfree+0xdf/0xfa
May 19 14:48:45 Megathron kernel: [<ffffffff8106385e>] ? kthread_park+0x52/0x52
May 19 14:48:45 Megathron kernel: [<ffffffff8105c16c>] ? umh_complete+0x20/0x20
May 19 14:48:45 Megathron kernel: [<ffffffff8105c28b>] ? call_usermodehelper_exec_async+0x11f/0x128
May 19 14:48:45 Megathron kernel: [<ffffffff8167f785>] ret_from_fork+0x25/0x30
May 19 14:48:45 Megathron kernel: Code: e8 10 66 87 47 02 c1 e0 10 74 6b 48 89 c1 c1 e8 12 48 c1 e9 0c ff c8 83 e1 30 48 98 48 81 c1 00 86 01 00 48 03 0c c5 60 62 9b 81 <48> 89 11 8b 42 08 85 c0 75 04 f3 90 eb f5 48 8b 0a 48 85 c9 74
May 19 14:48:45 Megathron kernel: RIP  [<ffffffff8107dd9f>] native_queued_spin_lock_slowpath+0x12d/0x17e
May 19 14:48:45 Megathron kernel: RSP <ffffc90003923d60>
May 19 14:48:45 Megathron kernel: ---[ end trace 8c1a697b0d7af270 ]---
May 19 14:48:45 Megathron kernel: note: kworker/1:1[335] exited with preempt_count 1
May 19 14:48:45 Megathron kernel: BUG: unable to handle kernel paging request at ffffffffffffffd8
May 19 14:48:45 Megathron kernel: IP: [<ffffffff81064153>] kthread_data+0xc/0x11
May 19 14:48:45 Megathron kernel: PGD 1c0c067
May 19 14:48:45 Megathron kernel: PUD 1c0e067
May 19 14:48:45 Megathron kernel: PMD 0
May 19 14:48:45 Megathron kernel:
May 19 14:48:45 Megathron kernel: Oops: 0000 [#2] PREEMPT SMP
May 19 14:48:45 Megathron kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod x86_pkg_temp_thermal coretemp e1000e mxm_wmi i2c_i801 ptp i2c_smbus i2c_core pps_core kvm_intel ahci libahci kvm video wmi backlight [last unloaded: md_mod]
May 19 14:48:45 Megathron kernel: CPU: 1 PID: 335 Comm: kworker/1:1 Tainted: G      D         4.9.30-unRAID #1
May 19 14:48:45 Megathron kernel: Hardware name: System manufacturer System Product Name/MAXIMUS VIII GENE, BIOS 2202 09/19/2016
May 19 14:48:45 Megathron kernel: task: ffff88085a199a80 task.stack: ffffc90003920000
May 19 14:48:45 Megathron kernel: RIP: 0010:[<ffffffff81064153>]  [<ffffffff81064153>] kthread_data+0xc/0x11
May 19 14:48:45 Megathron kernel: RSP: 0018:ffffc90003923e80  EFLAGS: 00010002
May 19 14:48:45 Megathron kernel: RAX: 0000000000000000 RBX: ffff88085a199a80 RCX: 0000000000000001
May 19 14:48:45 Megathron kernel: RDX: ffff88085a199a80 RSI: ffff88085a199b00 RDI: ffff88085a199a80
May 19 14:48:45 Megathron kernel: RBP: ffffc90003923e88 R08: 0000000000000113 R09: 0000000000044c00

 

megathron-diagnostics-20190519-1838.zip

[Lake-end]

Would appreciate any pointers if available.

 

More info: I am gettin the above freeze also when I SCP a 4gb ISO file to Array from my laptop. So it seems that either writing to the HDD array is hosed somehow or I might have RAM thats gone bad? 

[JorgeB]

Running memtest is always a good idea.

[Lake-end]

41 minutes ago, johnnie.black said:

Running memtest is always a good idea.

Yeah I´ve just noticed the unraid bootmenu has this by default (I dont really boot this machine often).

Seems that one of my RAM sticks has broken, so there we go.