FQs19

So one of my nvme drives (nvme0n1) went missing again. This happened right after the backup/restore appdata plugin finished running a backup. The last time is went missing after running TRIM. I'm at work, unable to do anything but grab the diagnostic zip. I'm attaching it to this post. Could someone please help me figure out why I keep losing an nvme disk. Do I need to wipe that cache pool and start over? Remove that nvme disk, put it in a windows machine, test it, then erase it and put it back in unraid and try again? Just return it to Corsair if it's still under warranty? Or is there a problem with this motherboard? I just can't seem to win with unraid. It's been nothing but problem after problem after problem. I can't believe that I'm having this many hardware problems. In one year I've replaced the case, the power supply, the motherboard, the flash drive, the memory, the sata cables, the hba cables, and even changed out the nvme drives. Beyond frustrated at this point. threadripper19-diagnostics-20220908-0253.zip

Curious why I had the nvme drop offline if I already had that code in there. I'll try rebooting and enabling TRIM again, to see what happens.

@JorgeB I already had that line in my boot folder, but I think I messed up the position of it. Which is probably why is wasn't working. Can you please correct my boot folder settings. I have it under Unraid OS and Unraid OS GUI MODE. Thanks

Thanks for the reply JorgeB. I did have that line in my original unraid server, but don't remember why I didn't put it in my fresh install of unraid. I'll try and find out why I didn't put it back in, then add it and turn TRIM back on and see what happens. Thanks again.

I found the cause of the Errors on Cache Pool. I forgot I made a user script to run a btrfs check hourly and that after an error is seen, it will continue to notify you of the error every time it runs, which for me is every hour. I had to force reset the stats on the cache pool to stop the error. I just assumed the stats would reset after correcting the errors. I still haven't figured out why the nvme drive dropped offline in the first place after a scheduled TRIM.

After starting my Unraid server, I started the array in maintenance mode then ran a file system check on both of my cache pools. Neither of them had any errors. I then stopped the array and started it normally. I ran a scrub with repair corrupted blocks checked on my arraycache pool. The scrub repaired a lot of corruption since that had the nvme drop offline. I ran a scrub on my plexcache pool and there were no errors. I thought everything was good until a couple hours ago. Every hour at exactly the 47th min of the hour, I get an email stating: unraid status: errors on cache pool Event: Unraid Status Subject: ERRORS on cache pool Description: No description Importance: warning It doesn't tell me what cache pool has the errors and I'm not seeing any errors on either cache pool. I've ran another scrub on each cache pool and there are no errors. I looked in my logs and there's no mention of cache pool errors except the system sending out an email. What is going on here? threadripper19-syslog-20220904-0003.zip

If anyone has time to explain to me what happened on my server, I would really appreciate it. While at work last night, I received an email from my unRAID server that stated the following: cron for user root /sbin/fstrim -a -v | logger &> /dev/null fstrim: /mnt/arraycache: FITRIM ioctl failed: Input/output error Immediately after that I received another email from my unRAID server that stated: Unraid Status: Warning [THREADRIPPER19] - Cache pool BTRFS missing device(s) Event: Unraid Arraycache disk message Subject: Warning [THREADRIPPER19] - Cache pool BTRFS missing device(s) Description: Force_MP600_20238284000130053009 (nvme0n1) Importance: warning I wasn't able to log in remotely to unRAID even though I had My Servers setup to do so, but that's a problem for another day. When I got home and logged in, I found that the Disk in question, Disk nvme0n1, was still shown on the Main tab and I was able to view it by clicking on the Arraycache link. I've had nvme disks drop offline before, but they were gone from the Main tab. Why was unRAID saying it was offline, but still visible? My Plex docker was still working perfectly fine, which I assume is because the other disk in the cachepool never dropped offline. I did some googling and read that I shouldn't be using TRIM on nvme devices especially if they are in a btrfs pool. That's the first I ever heard of that. So I will turn off TRIM on my server. I haven't had time to turn my unRAID server back on and see if there's any other problems. I just wanted to post here and get some feedback as to why it happened, how I can keep it from happening again, and what steps should I take before rebooting to make sure no data is lost? Attached is my Diagnostic file and a screenshot from after getting the Diagnostic file and before shutting my server down. Thank you in advance for any help given. threadripper19-diagnostics-20220903-0504.zip

Thanks for responding. I'm going to get support from AMD. I'll see what they say.

I finally switched to a new motherboard. This time I'm using the MSI TRX40 Pro 10G. I thought everything was working well until I had two issues pop up. The first is more of an annoyance. When booting my system, I can not change the default Boot Selector Mode. I have unRAID set to boot into GUI mode, but when the Boot Selector screen comes up I can't change the selection to anything else like Memtest or No GUI. The keyboard won't recognize any inputs from any keys. I've tried everything including different keyboards, different USB ports including USB2 and USB3, changing the BIOS USB settings, and I even tried flashing my boot drive with a clean version of 6.10.3. Nothing helped. So I gave up and will live with it. Here's the forum post for that issue: The second problem is exactly like this post about not being able to Disable Global C-State Control. If I set Global C-State Control to Disabled on this MSI motherboard, the parity check runs very slow around 10MB/s. Once I change Global C-State Control to Auto, the parity check runs at 180MB/s. So my question is, do you think that there's actually a problem with my AMD 3960x CPU? Because I can not believe that two separate motherboards would have the same issue of not being able to Disable Global C-State control, especially since there are other unRAID Threadripper users that have it disabled with no issues. I don't have Diagnostics to post since I'm way to tired of dealing with this and working. Plus, I tried to remove a drive from my array following the Preserve Parity Method, but obviously did it wrong because when I checked Parity with Corrections Enabled, it started running slow at only 89MB/s because it was writing corrections only to the second Parity disk. So right now I'm running a Parity Sync after running a New Config and don't want to stop the Parity Sync to grab Diagnostics showing the problem.

That's something I always disable on the first boot. I also tried changing the USB boot options in the BIOS, like LEGACY USB, to different configurations with no change.

After getting unRAID to boot back up into the GUI, I did extensive testing and I still can't change the boot selection during booting. I've got no clue and hope someone here could help solve this problem. I've tried multiple keyboards, different USB3.0 ports, a USB2.0 header port, and I still couldn't change the boot selection during the booting process. The three keyboards I've tried all work perfectly fine on other computers. They also work in the bios and in the GUI, once unRAID starts, on the unRAID server. As a last resort, I even flashed a new USB flash drive, and I still couldn't change the boot selection option. I've checked the USB flash drive with CHKDSK and didn't find any errors. I've cleared the motherboard CMOS multiple times with no change. I'm wondering if there's some configuration on my flash drive that is causing this issue or if someone can spot something in my diagnostics that could be the culprit. Any help is appreciated. Thanks threadripper19-diagnostics-20220814-1947.zip

@trurl Nevermind, I found the file on the flash drive.

The motherboard has one USB2 header inside the case. Can't attach the keyboard outside the case to it. The keyboard works in every situation except during the blue unRAID boot selection screen. And yes it works fine on other computers as well. Right now I'm stuck with unRAID always booting to memtest and have no idea how to edit the flash drive so it boots into GUI mode. Can you tell me what file on the flash I need to edit to change the default unRAID boot mode from memtest to GUI?

The motherboard only has USB3 ports and yes I've tried all of them. USB2 ports are rare to find on new motherboards so it would be nice to see full support of USB3.

The keyboard is a Kensington wired keyboard model M01181 It's plugged into a USB 3.0 motherboard port.

I did notice the tests were different so I'll make a memtest86 boot drive and test them, but what about not being able to change the boot selection?

I'm currently running 6.10.3 with a Threadripper 3960X and an MSI TRX40 Pro 10G motherboard with a wired keyboard plugged into one of the usb3.0 motherboard headers. I'm able to use the motherboard in the Bios, but when the unRAID boot selection screen comes up I can't move the selection. I've tried the cursor keys and the tab key. No key on the keyboard does anything. The keyboard is still receiving power since I can see the numlock key light lit up. I've also tried changing the USB Configuration in the Bios with no change. All I was trying to do was select the MemTest boot to test some new ECC memory I purchased. Has anyone ever experienced this before?

I couldn't find anything that was similar to Power Supply Idle Control nor did I find anything that said typical current idle. I have a new motherboard that I will test in the next couple weeks. Hopefully it'll work properly. I'll also look for the same settings on that board which is an MSI AMD Pro Series TRX40 Pro10G.

Excellent. Maybe I'll start replacing my array disks with 7200rpm disks when I need to. Thanks again for the help. I'm ready to get this server back up and running after several months of troubleshooting and a case change. Just waiting for my new power supply to replace my current one, then I can start working on rebuilding my Plex docker and all the damn metadata. That'll take me weeks to do, maybe even months if I don't get time off from work. hahaha

@Vr2Io @JorgeB My parity-sync finally finished after 25hrs 32mins 13secs with an average speed of 130.5 MB/s. I believe my previous parity checks were around 26~28 hours. I forget. Would you say that's an average speed for two WD Red Pro drives as parity disks? I do have a couple 5400 rpm disks in the array. @JorgeB I'll look for that "Power Supply Idle Control" setting in my BIOS now that my parity-sync completed.

Will do. Thanks

@Vr2Io So it looks like I have to keep Global C-State control set to Auto for proper performance. I had it set that way for the prior two years until I had issues this year with UDMA CRC errors and was told to change that to Disabled. Does this setting really need to be Disabled for Unraid?

Here is my BIOS set to the following: - Global C-State Control set to Auto - IOMMU set to Auto performance is good threadripper19-diagnostics-20220712-1918.zip

Sorry, I configured the BIOS wrong in the last test. Here's it correctly configured for - Global C-State Control set to Disabled - IOMMU set to Enabled performance is terrible threadripper19-diagnostics-20220712-1910.zip