my95z34

I have about 500 miles of the first two errors in the logs before the server hard locked. What am I looking at here? I have additional complete log files for the last day or so if those are needed as well. Just let me know. Additional diag attached. Jan 25 13:32:54 Goliath kernel: critical medium error, dev sda, sector 5110141 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Jan 25 13:32:54 Goliath kernel: FAT-fs (sda1): Directory bread(block 5108093) failed Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current] Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0 Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 4d f9 7e 00 00 01 00 Jan 25 13:32:54 Goliath kernel: critical medium error, dev sda, sector 5110142 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Jan 25 13:32:54 Goliath kernel: FAT-fs (sda1): Directory bread(block 5108094) failed Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current] Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0 Jan 25 13:32:54 Goliath kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 4d f9 7f 00 00 01 00 Jan 25 13:32:54 Goliath kernel: critical medium error, dev sda, sector 5110143 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Jan 25 13:32:54 Goliath kernel: FAT-fs (sda1): Directory bread(block 5108095) failed Jan 25 13:33:08 Goliath kernel: Invalid SPTE change: cannot replace a present leaf Jan 25 13:33:08 Goliath kernel: SPTE with another present leaf SPTE mapping a Jan 25 13:33:08 Goliath kernel: different PFN! Jan 25 13:33:08 Goliath kernel: as_id: 0 gfn: 2c5c33 old_spte: ffff88814b626a48 new_spte: 6000004a9033b77 level: 1 Jan 25 13:33:08 Goliath kernel: ------------[ cut here ]------------ Jan 25 13:33:08 Goliath kernel: kernel BUG at arch/x86/kvm/mmu/tdp_mmu.c:485! Jan 25 13:33:08 Goliath kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI Jan 25 13:33:09 Goliath kernel: CPU: 2 PID: 19792 Comm: CPU 4/KVM Tainted: P W O 6.1.64-Unraid #1 Jan 25 13:33:09 Goliath kernel: Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/21/2019 Jan 25 13:33:09 Goliath kernel: RIP: 0010:__handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: Code: a8 01 74 28 80 7c 24 3e 00 74 21 48 8b 0c 24 45 89 e1 49 89 d8 48 c7 c7 92 7d aa a0 48 8b 54 24 10 8b 74 24 24 e8 46 56 dd e0 <0f> 0b 48 39 1c 24 0f 84 94 03 00 00 66 90 eb 4d 65 8b 3d cb b2 58 Jan 25 13:33:09 Goliath kernel: RSP: 0018:ffffc9000bc17aa8 EFLAGS: 00010246 Jan 25 13:33:09 Goliath kernel: RAX: 00000000000000c2 RBX: 06000004a9033b77 RCX: 0000000000000000 Jan 25 13:33:09 Goliath kernel: RDX: 0000000000000000 RSI: ffffffff820d7e01 RDI: 00000000ffffffff Jan 25 13:33:09 Goliath kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000031 Jan 25 13:33:09 Goliath kernel: R10: 0000000000000010 R11: ffff88907ff8eb0c R12: 0000000000000001 Jan 25 13:33:09 Goliath kernel: R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000800 Jan 25 13:33:09 Goliath kernel: FS: 000014648f7ff6c0(0000) GS:ffff88885f880000(0000) knlGS:ffffc500dd1e9000 Jan 25 13:33:09 Goliath kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 25 13:33:09 Goliath kernel: CR2: 000000008c6ce000 CR3: 00000008d8c00005 CR4: 00000000001726e0 Jan 25 13:33:09 Goliath kernel: DR0: 00000000042e1e67 DR1: 0000000000000000 DR2: 0000000000000000 Jan 25 13:33:09 Goliath kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 25 13:33:09 Goliath kernel: Call Trace: Jan 25 13:33:09 Goliath kernel: <TASK> Jan 25 13:33:09 Goliath kernel: ? __die_body+0x1a/0x5c Jan 25 13:33:09 Goliath kernel: ? die+0x30/0x49 Jan 25 13:33:09 Goliath kernel: ? do_trap+0x7b/0xfe Jan 25 13:33:09 Goliath kernel: ? __handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: ? __handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: ? do_error_trap+0x6e/0x98 Jan 25 13:33:09 Goliath kernel: ? __handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: ? exc_invalid_op+0x4c/0x60 Jan 25 13:33:09 Goliath kernel: ? __handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: ? asm_exc_invalid_op+0x16/0x20 Jan 25 13:33:09 Goliath kernel: ? __handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: ? __handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: tdp_mmu_set_spte_atomic+0x4b/0x64 [kvm] Jan 25 13:33:09 Goliath kernel: kvm_tdp_mmu_map+0x27c/0x3c9 [kvm] Jan 25 13:33:09 Goliath kernel: direct_page_fault+0x39f/0x581 [kvm] Jan 25 13:33:09 Goliath kernel: ? try_to_wake_up+0x20e/0x248 Jan 25 13:33:09 Goliath kernel: kvm_mmu_do_page_fault+0xcf/0x12d [kvm] Jan 25 13:33:09 Goliath kernel: kvm_mmu_page_fault+0x392/0x4a0 [kvm] Jan 25 13:33:09 Goliath kernel: ? vmx_vmexit+0x79/0x8bd [kvm_intel] Jan 25 13:33:09 Goliath kernel: ? vmx_vmexit+0x73/0x8bd [kvm_intel] Jan 25 13:33:09 Goliath kernel: ? vmx_vmexit+0x8d/0x8bd [kvm_intel] Jan 25 13:33:09 Goliath kernel: vmx_handle_exit+0x58d/0x667 [kvm_intel] Jan 25 13:33:09 Goliath kernel: kvm_arch_vcpu_ioctl_run+0x133f/0x15db [kvm] Jan 25 13:33:09 Goliath kernel: ? kvm_vm_ioctl_irq_line+0x23/0x37 [kvm] Jan 25 13:33:09 Goliath kernel: kvm_vcpu_ioctl+0x192/0x5b4 [kvm] Jan 25 13:33:09 Goliath kernel: ? __note_gp_changes+0x12f/0x13b Jan 25 13:33:09 Goliath kernel: ? _raw_spin_unlock_irqrestore+0x24/0x3a Jan 25 13:33:09 Goliath kernel: ? note_gp_changes+0x62/0x78 Jan 25 13:33:09 Goliath kernel: ? __seccomp_filter+0x89/0x2ff Jan 25 13:33:09 Goliath kernel: ? rcu_core+0x27f/0x2ac Jan 25 13:33:09 Goliath kernel: vfs_ioctl+0x1e/0x2f Jan 25 13:33:09 Goliath kernel: __do_sys_ioctl+0x52/0x78 Jan 25 13:33:09 Goliath kernel: do_syscall_64+0x6b/0x81 Jan 25 13:33:09 Goliath kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce Jan 25 13:33:09 Goliath kernel: RIP: 0033:0x146894cad4e8 Jan 25 13:33:09 Goliath kernel: Code: 00 00 48 8d 44 24 08 48 89 54 24 e0 48 89 44 24 c0 48 8d 44 24 d0 48 89 44 24 c8 b8 10 00 00 00 c7 44 24 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 07 89 d0 c3 0f 1f 40 00 48 8b 15 f9 e8 0d Jan 25 13:33:09 Goliath kernel: RSP: 002b:000014648f7fdc48 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 Jan 25 13:33:09 Goliath kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 0000146894cad4e8 Jan 25 13:33:09 Goliath kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000001a Jan 25 13:33:09 Goliath kernel: RBP: 0000146492738fc0 R08: 00005636dc7c9fe0 R09: 000000000000ffff Jan 25 13:33:09 Goliath kernel: R10: 0000000000000007 R11: 0000000000000246 R12: 0000000000000000 Jan 25 13:33:09 Goliath kernel: R13: 0000000000000002 R14: 0000000000000001 R15: 0000000000000000 Jan 25 13:33:09 Goliath kernel: </TASK> Jan 25 13:33:09 Goliath kernel: Modules linked in: veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_nat xt_tcpudp macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nvidia_uvm(PO) xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nvidia_modeset(PO) video nvidia(PO) ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls tg3 ipmi_ssif intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 drm_shmem_helper i2c_algo_bit crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd drm cryptd rapl intel_cstate hpsa Jan 25 13:33:09 Goliath kernel: backlight i2c_i801 syscopyarea cp210x acpi_ipmi i2c_smbus sysfillrect sysimgblt intel_uncore i2c_core usbserial fb_sys_fops scsi_transport_sas wmi ipmi_si acpi_tad button unix [last unloaded: acpi_power_meter] Jan 25 13:33:09 Goliath kernel: ---[ end trace 0000000000000000 ]--- Jan 25 13:33:09 Goliath kernel: RIP: 0010:__handle_changed_spte+0x11b/0x4ce [kvm] Jan 25 13:33:09 Goliath kernel: Code: a8 01 74 28 80 7c 24 3e 00 74 21 48 8b 0c 24 45 89 e1 49 89 d8 48 c7 c7 92 7d aa a0 48 8b 54 24 10 8b 74 24 24 e8 46 56 dd e0 <0f> 0b 48 39 1c 24 0f 84 94 03 00 00 66 90 eb 4d 65 8b 3d cb b2 58 Jan 25 13:33:09 Goliath kernel: RSP: 0018:ffffc9000bc17aa8 EFLAGS: 00010246 Jan 25 13:33:09 Goliath kernel: RAX: 00000000000000c2 RBX: 06000004a9033b77 RCX: 0000000000000000 Jan 25 13:33:09 Goliath kernel: RDX: 0000000000000000 RSI: ffffffff820d7e01 RDI: 00000000ffffffff Jan 25 13:33:09 Goliath kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000031 Jan 25 13:33:09 Goliath kernel: R10: 0000000000000010 R11: ffff88907ff8eb0c R12: 0000000000000001 Jan 25 13:33:09 Goliath kernel: R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000800 Jan 25 13:33:09 Goliath kernel: FS: 000014648f7ff6c0(0000) GS:ffff88885f880000(0000) knlGS:ffffc500dd1e9000 Jan 25 13:33:09 Goliath kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 25 13:33:09 Goliath kernel: CR2: 000000008c6ce000 CR3: 00000008d8c00005 CR4: 00000000001726e0 Jan 25 13:33:09 Goliath kernel: DR0: 00000000042e1e67 DR1: 0000000000000000 DR2: 0000000000000000 Jan 25 13:33:09 Goliath kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 25 13:33:09 Goliath kernel: note: CPU 4/KVM[19792] exited with preempt_count 1 Jan 25 13:33:48 Goliath nmbd[13543]: [2024/01/25 13:33:48.081639, 0] ../../source3/nmbd/nmbd_become_lmb.c:398(become_local_master_stage2) Jan 25 13:33:48 Goliath nmbd[13543]: ***** Jan 25 13:33:48 Goliath nmbd[13543]: Jan 25 13:33:48 Goliath nmbd[13543]: Samba name server GOLIATH is now a local master browser for workgroup WORKGROUP on subnet 192.168.1.207 Jan 25 13:33:48 Goliath nmbd[13543]: Jan 25 13:33:48 Goliath nmbd[13543]: ***** goliath-diagnostics-20240125-1604.zip

I was just wondering if you ever got this to work. I've been fighting it for 2 days now... lol

Well, that makes sense. I'll keep an eye and see if my mover will properly run once it reaches the capacity requirement and I'll post back. Thanks!

Hmm, I've never noticed that button there! I've been using the button on the main array page in Unraid. I could have sworn that the previous behaviour was that button calling the original script... unless I'm crazy, lol.

I'm getting the same log output with that file. Assuming I don't need to reboot or anything after replacing. I also ensured the permissions were the same as the original as well. //edit// I lied. I enabled the standard Unraid mover logging and I got some output. Mar 27 22:25:06 Goliath root: Manually starting mover Mar 27 22:25:06 Goliath root: Forcing turbo write on Mar 27 22:25:06 Goliath kernel: mdcmd (95): set md_write_method 1 Mar 27 22:25:06 Goliath kernel: Mar 27 22:25:06 Goliath root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' '' Mar 27 22:25:06 Goliath root: *********************************MOVER START******************************* Mar 27 22:25:06 Goliath root: mover: started Mar 27 22:25:06 Goliath root: Share Name Only: ((omitted)) Mar 27 22:25:06 Goliath root: Cache Pool Name: Mar 27 22:25:06 Goliath root: No shareCachePool entry found in config file, defaulting to cache Mar 27 22:25:06 Goliath root: cache Threshold Pct: Mar 27 22:25:06 Goliath root: OVERALL Threshold: 65 Mar 27 22:25:06 Goliath root: Share Path: /mnt/cache/((omitted)) Mar 27 22:25:06 Goliath root: Pool Pct Used: 54 % Mar 27 22:25:06 Goliath root: DFTPCT LIMIT USED FOR SETTING: 65 Mar 27 22:25:06 Goliath root: Threshold Used: 65 Mar 27 22:25:06 Goliath root: Mover not Needed. Getting that for every share. So it seems that instead of moving anyway, it's still obeying the cache space used requirement. If I remember correctly, that didn't use to be the case. But I'm not positive.

I am having the same issue on 6.9.1. I had noticed the other day that my cache was filling up so I tried to manually run the mover. Nothing was happening. So I uninstalled this plugin and was able to get files to move correctly. I just tried to reinstall it, but I'm back to having the same issues. I did verify that I have the config file /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg. If there's anything you need me to test, I'm more than happy to. Thanks in advance! Mar 24 16:18:12 Goliath root: Manually starting mover Mar 24 16:18:12 Goliath root: Forcing turbo write on Mar 24 16:18:12 Goliath kernel: mdcmd (75): set md_write_method 1 Mar 24 16:18:12 Goliath kernel: Mar 24 16:18:12 Goliath root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' '' Mar 24 16:18:13 Goliath root: Restoring original turbo write mode Mar 24 16:18:13 Goliath kernel: mdcmd (76): set md_write_method auto Mar 24 16:18:13 Goliath kernel: Mar 24 16:18:57 Goliath emhttpd: shcmd (15981): /usr/local/sbin/mover &> /dev/null & Mar 24 16:18:57 Goliath root: Manually starting mover Mar 24 16:18:57 Goliath root: Forcing turbo write on Mar 24 16:18:57 Goliath kernel: mdcmd (77): set md_write_method 1 Mar 24 16:18:57 Goliath kernel: Mar 24 16:18:57 Goliath root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' '' Mar 24 16:18:58 Goliath root: Restoring original turbo write mode Mar 24 16:18:58 Goliath kernel: mdcmd (78): set md_write_method auto Mar 24 16:18:58 Goliath kernel:

I'm going to say that the cause for my lockups seemed to be the log filling up. Since I fixed the ACPI error I've gotten tons of uptime with no issues. Thanks for the help!

Alrighty, done. Now to wait another 6-7 days. XD

So, for the past few weeks I've been having an issue where the server is completely locked up after 6-7 days. I posted a bug report last week when it happened but I didn't have active logs, just the ones after the reboot. I have since set up the syslog server so I have the logs from the past 6 days. Attached are both, the current diag, plus the syslog from the last week. Today I was able to catch it before it fully locked up and was able to issue the command through the terminal to reboot. (I've always had to do parity checks after, so that was kind of nice, lol) But, when it came up the docker service wouldn't start. I had to delete the image and reinstall my apps. I'm not sure if docker has something to do with my lockups or not, but figured it was a data point. Thank you in advance for any help!

I mean, the server had been up for days. So it wasn't an unclean shutdown. Is there anything in the diagnostics that points to a hardware issue?

Wow... What caused it to spontaneously self destruct?

So, out of nowhere my docker applications were acting funny. I was able to watch stuff on Plex, for example, but updating the watched statuses wasn't working. So I attempted to restart the plex docker. When restarting, it would show "execution error 403". I attempted to stop the service, and recreate the docker image, but deleting the image fails. It just refreshes the settings page. After rebooting the server, still same outcome. Also, when I try to write data to cache enabled shares, I get write protected errors from Windows. I ran the btrfs check while in maintenance mode and the output is literally a million lines long. I've truncated it here: https://hastebin.com/tahofejalo.sql On the display for the server I'm seeing corrupt leaf errors, and IO failure errors for sdk1. Thanks for any help in advance! goliath-diagnostics-20200208-2351.zip

I never got a notification for that comment. Ugh, lol. I ended up just nuking the drive. I remembered that Syncthing had backed up everything that I thought I'd lose, to my other server. Plus, I run CA Appdata backup nightly, so I only lost a small amount of Plex watched statuses. I can attach it now, but I doubt that'd do any good. Also, just for clarification, was removing drives while the array was stopped, but leaving the server powered up, a bad practice?

So, long story. I've got an HP DL380 G9. I've had my 2 SSD's just sitting in 2 bottom drive trays since I didn't have an SSD drive adapter. (The server is outfitted for 3.5" drives) Well, I purchased adapters and they just came in today. I stopped the array, popped out the SSD's, put them into the adapters, and then put them into the server. Ended up having to use different slots, since the adapters didn't want to fit right for some reason... Anyway. Started the array back up and now it's saying that the cache is unmountable. If I have to start format the cache, it's not 100% the end of the world. My appdata backs up every night to the array. But I do have some shares that use the cache that files got copied to today. When starting the array in maintenance mode and checking the file system on the cache drive I get this output: No valid Btrfs found on /dev/sdk1 ERROR: cannot open file system Opening filesystem to check... Are there any steps I should take before nuking the cache? Thanks in advance! //edit// Reading some other threads I've been throwing some commands at the server. Ran btrfs check -s 1 /dev/sdk1 and got: using SB copy 1, bytenr 67108864 Opening filesystem to check... bad tree block 298118643712, bytenr mismatch, want=298118643712, have=0 Couldn't read tree root ERROR: cannot open file system