I upgraded 2 my cache pool with brand new NVME drives. I attempted this task without watching a video on how it's done because how hard could it be right? Now I'm paying for it because I am having a multitude of issues due to my stupidity "Troubleshooting". I just hope I am overthinking things, and this is a simple fix I am missing. I don't want to have to start from square one and lose all my data. If anyone could throw some help my way it'd be much appreciated. I've added a lot of information below as well as my diagnostics zip.

 

Unraid Version 7.0.1

System Board: MSI Z790 Tomahawk Max Wifi

CPU: Intel i9 14900KS

Memory: 32gb x4 Vengeance DDR5 5200MHz

 

My Purpose:

 

My 2 Intel SSDs were failing so I upgraded them with 2 1tb Crucial NVMEs. I also wanted to add another 500gb nvme for my /system/* and /appdata folders to live on.

 

My process:

 

• I left in the 2 failing disks and added the NVMEs
• I created a mirrored BTRFS cache pool named Nvmecache with the 2 1tb NVMEs to use as main cache
• I create a BTRFS cache pool named appdatacache with the 1 500gb nvme to use for my /system and /appdata folders
• I changed the appdata and system shares storage cache => appdatacache
• I change all other shares from cache => nvmecache
• I invoked mover. After letting it go several hours it didn't look like mover was doing anything, so I started moving things manually through terminal (Big Mistake on my part).
  • /mnt/user/system => /mnt/appdatacache/system
  • /mnt/user/appdata => /mnt/appdatacache/appdata
• Changed docker from macvlan to ipvlan after having issues with random system freezes.

 

Issues I'm now facing:

• My docker containers immediately stopped working.
  • After having docker issues I recreated the image file. This has been done multiple times in an attempt to troubleshoot issues.
• I installed an initial container (Plex-Media-Server) and started library scans before bed last night and when I checked it this morning the container was unresponsive and become unhealthy after I attempted to restart it. 
• I am unable to remove/stop/purge, when ever I try to do anything with this container in CLI I get "Error response from daemon: cannot remove container "/Plex-Media-Server": could not kill: tried to kill container, but did not receive an exit event" When I try from gui I get an execution error. It takes rebooting the system to get stop or remove this docker.
• Parity sync seems like it is running and starts out fast. After it's been running for a short time, it slows down considerably It's been running 2 hours and has only completed 0.7%
• Host refuses to reboot/shutdown via CLI or Webgui when parity is "running" and the Docker container is unstable. The only way to reboot is to poke the power button.

 

Things I've tried:

• I moved folders back to their original locations
  • /mnt/user/system
  • /mnt/user/appdata
• I've recreated the docker.img file multiple times
• I've tried using a docker folder instead of an image file
• I use new permissions to set defaults on disks and shares.
• I've booted into safe mode and get all the same issues there
• I've tried remove and recreate containers in live and safe environments
• deleted the docker container manually from /var/lib/docker/containers/

 

 As I think of more stuff that I tried I will add them to this thread. Also please let me know if more information is needed.

 

Thanks in advance,

 

Dabbler455

 

 

star-command-diagnostics-20250502-1050.zip

Parity check issue is unrelated to the pool, the Unraid driver is crashing, that is almost always a hardware problem, most often bad RAM or, since you have a 14900K it can also be the Intel 13/14 gen issue, since you have multiple RAM sticks, try running a check again with just one pair, if the same try with the other one, that will basically rule out bad RAM.

I will try this, but I'd had the same configuration up and running without issues for months. Also, one of the things I left out in my troubleshooting steps was that I ran memtest before I did anything else, which passed. 

 

It would be coincidental for ram or CPU to fail at the exact time I replaced a couple of drives. May I ask where in the provided logs were you able to identify that the Unraid driver crashed. This information would be useful in the future.

Edited May 2, 20251 yr by dabbler455
added more information

1 hour ago, dabbler455 said:

May I ask where in the provided logs were you able to identify that the Unraid driver crashed.

 

 

May  2 09:07:42 Star-Command kernel: ------------[ cut here ]------------
May  2 09:07:42 Star-Command kernel: kernel BUG at drivers/md/unraid.c:1617!
May  2 09:07:42 Star-Command kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
May  2 09:07:42 Star-Command kernel: CPU: 11 PID: 5521 Comm: unraidd2 Tainted: P           O       6.6.78-Unraid #2
May  2 09:07:42 Star-Command kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E25/MAG Z790 TOMAHAWK MAX WIFI (MS-7E25), BIOS A.80 09/27/2024
May  2 09:07:42 Star-Command kernel: RIP: 0010:unraidd+0x1189/0x1278 [md_mod]
May  2 09:07:42 Star-Command kernel: Code: 00 83 3d 04 d0 ed ff 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 b3 7a a0 48 8b 73 20 e8 68 b5 92 e0 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 18 41 c7 46 b0 00 10 00 00 49 8b 56 10
May  2 09:07:42 Star-Command kernel: RSP: 0018:ffffc9000fd43da8 EFLAGS: 00010246
May  2 09:07:42 Star-Command kernel: RAX: 0000000000000000 RBX: ffff8881811db0a0 RCX: 0000000000000000
May  2 09:07:42 Star-Command kernel: RDX: 0000000000000000 RSI: ffffffff82cbf420 RDI: ffff8881018e7c38
May  2 09:07:42 Star-Command kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
May  2 09:07:42 Star-Command kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881468a5108
May  2 09:07:42 Star-Command kernel: R13: ffff8881811db1e8 R14: ffff8881811db260 R15: ffff88816f769220
May  2 09:07:42 Star-Command kernel: FS:  0000000000000000(0000) GS:ffff889fff2c0000(0000) knlGS:0000000000000000
May  2 09:07:42 Star-Command kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  2 09:07:42 Star-Command kernel: CR2: 00001459afe1c6a8 CR3: 0000000106e90000 CR4: 0000000000750ee0
May  2 09:07:42 Star-Command kernel: PKRU: 55555554
May  2 09:07:42 Star-Command kernel: Call Trace:
May  2 09:07:42 Star-Command kernel: <TASK>
May  2 09:07:42 Star-Command kernel: ? __die_body+0x1a/0x5c
May  2 09:07:42 Star-Command kernel: ? die+0x30/0x49
May  2 09:07:42 Star-Command kernel: ? do_trap+0x7b/0xfe
May  2 09:07:42 Star-Command kernel: ? unraidd+0x1189/0x1278 [md_mod]
### [PREVIOUS LINE REPEATED 1 TIMES] ###
May  2 09:07:42 Star-Command kernel: ? do_error_trap+0x6e/0x98
May  2 09:07:42 Star-Command kernel: ? unraidd+0x1189/0x1278 [md_mod]
May  2 09:07:42 Star-Command kernel: ? exc_invalid_op+0x4c/0x60
May  2 09:07:42 Star-Command kernel: ? unraidd+0x1189/0x1278 [md_mod]
May  2 09:07:42 Star-Command kernel: ? asm_exc_invalid_op+0x16/0x20
May  2 09:07:42 Star-Command kernel: ? unraidd+0x1189/0x1278 [md_mod]
May  2 09:07:42 Star-Command kernel: ? unraidd+0x1117/0x1278 [md_mod]
May  2 09:07:42 Star-Command kernel: ? preempt_latency_start+0x2b/0x46
### [PREVIOUS LINE REPEATED 1 TIMES] ###
May  2 09:07:42 Star-Command kernel: md_thread+0xf7/0x127 [md_mod]
May  2 09:07:42 Star-Command kernel: ? __pfx_autoremove_wake_function+0x10/0x10
May  2 09:07:42 Star-Command kernel: ? __pfx_md_thread+0x10/0x10 [md_mod]
May  2 09:07:42 Star-Command kernel: kthread+0xf1/0xfc
May  2 09:07:42 Star-Command kernel: ? __pfx_kthread+0x10/0x10
May  2 09:07:42 Star-Command kernel: ret_from_fork+0x21/0x36
May  2 09:07:42 Star-Command kernel: ? __pfx_kthread+0x10/0x10
May  2 09:07:42 Star-Command kernel: ret_from_fork_asm+0x1b/0x30
May  2 09:07:42 Star-Command kernel: </TASK>
May  2 09:07:42 Star-Command kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter md_mod zfs(PO) spl(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet cfg80211 8021q garp mrp bridge stp llc intel_rapl_common x86_pkg_temp_thermal intel_powerclamp i915 coretemp kvm_intel kvm iosf_mbi drm_buddy ttm i2c_algo_bit crct10dif_pclmul crc32_pclmul drm_display_helper crc32c_intel ghash_clmulni_intel btusb sha512_ssse3 sha256_ssse3 btrtl sha1_ssse3 btbcm btintel drm_kms_helper aesni_intel crypto_simd bluetooth cryptd drm ecdh_generic mei_pxp mei_hdcp ecc rapl input_leds led_class joydev rfkill intel_cstate mpt3sas wmi_bmof mxm_wmi intel_gtt nvme i2c_i801 mei_me intel_uncore agpgart i2c_smbus igc raid_class nvme_core mei ahci i2c_core scsi_transport_sas libahci thermal fan tpm_crb video tpm_tis tpm_tis_core tpm wmi backlight acpi_tad acpi_pad button
May  2 09:07:42 Star-Command kernel: ---[ end trace 0000000000000000 ]---
May  2 09:07:42 Star-Command kernel: pstore: backend (efi_pstore) writing error (-5)
May  2 09:07:42 Star-Command kernel: RIP: 0010:unraidd+0x1189/0x1278 [md_mod]
May  2 09:07:42 Star-Command kernel: Code: 00 83 3d 04 d0 ed ff 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 b3 7a a0 48 8b 73 20 e8 68 b5 92 e0 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 18 41 c7 46 b0 00 10 00 00 49 8b 56 10
May  2 09:07:42 Star-Command kernel: RSP: 0018:ffffc9000fd43da8 EFLAGS: 00010246
May  2 09:07:42 Star-Command kernel: RAX: 0000000000000000 RBX: ffff8881811db0a0 RCX: 0000000000000000
May  2 09:07:42 Star-Command kernel: RDX: 0000000000000000 RSI: ffffffff82cbf420 RDI: ffff8881018e7c38
May  2 09:07:42 Star-Command kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
May  2 09:07:42 Star-Command kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881468a5108
May  2 09:07:42 Star-Command kernel: R13: ffff8881811db1e8 R14: ffff8881811db260 R15: ffff88816f769220
May  2 09:07:42 Star-Command kernel: FS:  0000000000000000(0000) GS:ffff889fff2c0000(0000) knlGS:0000000000000000
May  2 09:07:42 Star-Command kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  2 09:07:42 Star-Command kernel: CR2: 00001459afe1c6a8 CR3: 0000000106e90000 CR4: 0000000000750ee0
May  2 09:07:42 Star-Command kernel: PKRU: 55555554
May  2 09:07:42 Star-Command kernel: ------------[ cut here ]------------

 

md_mod is the Unraid driver

Do failing flash drives cause the issue called out in the call trace? Would it be beneficial to try and replace the flash drive to see if that helps?

8 hours ago, dabbler455 said:

Do failing flash drives cause the issue called out in the call trace?

It shouldn't, typically is bad RAM (or CPU).

 

 

I took apart my server fully and noticed that I had bent pins in the CPU slot of my motherboard. I had installed an LGA 1700 Thermalright contact frame at the beginning of my build. After I started having these issues, I did a little bit of research. What I found was these frames can cause issues. One of which is they really easy to overtighten and in doing so can bend pins in your CPU slot. If anyone is running into issues like this and you use a contact frame check your CPU slot because that could be the cause of your grief.