Pri

I think having a Mover Stop button on the GUI where the current "Move" button is (perhaps only showing while Mover is actually running) would be useful to users. I know currently you can run the command: mover stop from the console but having it in the GUI I think makes sense and would be useful. I have created this mockup to illustrate what I mean.

Small update to the server. So I've had the RAM in there a while and I did solve the overheating issue a long time ago by cutting some plastic packaging with scissors to make an air baffle. I wasn't really happy with that solution long term so I have designed and printed a proper baffle at home. The left baffle is a bit thinner than the right side so that it fits with a PCIe card in the top slot. But of course this may differ based on your motherboard. Link to the model for printing: https://makerworld.com/en/models/1712778-arctic-freezer-4u-m-ram-air-guide-baffle And I've included some images of the baffle installed too. Temperature wise I now see both sides of memory at 61-64c where as before the right side would regularly hit 71c with the janky cut-up plastic baffle I made from some packaging material. Quite an improvement and a lot more sturdy and dependable. Included in the model is a mount for a 120mm fan so I could in the future mount a fan back on the heatsink in the middle, I may do that and use two rows of 40mm fans on either side for the RAM baffles but right now I'm happy with this as-is, CPU temps around 60c under load aswell.

Thought I'd update on the CPU situation. So the 7443p I was originally using in this build was stable for 11 days after I removed the 7773X which kept crashing. That was definitely confirmation to me that the problem was the 7773X. After that 11 days of uptime my replacement 7773X arrived and I put that in one week ago. Since then the system has been completely stable and I'm able to enjoy the new processor to its full potential. So the upgrade was a success but that first chip definitely suffered from infant mortality. For those curious I bought the chip brand new from tugm on ebay, he's quite a well known seller in the Homelab community and with good reason. He was happy to replace the chip for me and even pre-paid for the return shipping with FedEX express which only took 3 days all the way back to Hong Kong. Really good seller and I just wanted to give him a shout out for the excellent service I received

Little update on this CPU upgrade. So 4 days after I installed the 7773X the system began to randomly shutdown (complete power off) then start again on its own. The logs in unRAID revealed absolutely nothing. Never any errors before or afterwards, just completely random restarts. The only common denominator between the restarts is that the system only restarted when it was idle. Even being completely stressed for 40 minutes it wouldn't restart but when left idle for a while there was a high probability it would shut off. In-fact I was able to induce this shutdown several times by switching from a high-load to low-load scenario on the CPU. So I tried the following: 1. Disabling C-States 2. Updating BIOS to latest (I was 1 version behind) 3. Reducing chip TDP from 280 Watts to 225 Watts 4. Reducing RAM speed from 3200MHz to 2933MHz 5. Re-seating the CPU in the socket 6. Disabling unRAID power states (trying to eliminate low-energy idle states to rule things out). 7. Removing the USB cord I had my unRAID drive loading from so it was plugged into the mobo directly. None of this worked. The maximum amount of uptime I was able to achieve apart from the initial first 4 days was about 2 days. Most of the time it shutdown within 2 to 6 hours. Sometimes I even had it shut down immediately upon reaching the unRAID GUI. Final thing I've tried, I put the old 7443p CPU back in the system. It has been stable ever since, that's almost three days uptime and prior to this the day before I changed CPU's it crashed three times in one day with the 7773X. So this all leads me to believe the CPU is faulty, doesn't appear to like going idle. Possibly it degraded quickly after that first initial four days of uptime, it is a brand new retail chip and these things do happen especially with X3D chips that are lets say a bit more fragile. With regards to the IPMI on the motherboard, it only ever gave me a single log entry when these shutdowns occurred and it said the CPU was "Out of Spec Definition" which again points to the CPU, I would receive this error every single time the system started back up after shutting off and no other errors were ever reported. Right now I'm just going to wait until I have I think 5 or 6 days uptime on the 7443p to fully rule out anything else and then I'll RMA the 7773X and try again with a replacement. It's a real shame but I've only lost some time to this problem and luckily I have the old chip to use in the meantime. I really love the performance the 7773X has delivered for my workloads and it actually runs about the same temperature as the 7443p even while having almost 50% higher TDP requirements.

Sure yeah, the PCI ID. I found when updating my BIOS both changed sometimes. I'm using EPYC which has 102 different IOMMU groups and when updating my BIOS recently half of my devices changed ID's and it has done that before on previous BIOS updates aswell. One other issue I found is one of my Network card ports I had passed through to one VM had swapped ID's with an AMD CPU feature, when the VM started it took control of this: Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA just because it had occupied the PCI ID of the network card previously. My fault for not checking before I started the VM but if you have a lot of VM's with passed through stuff this would definitely get complicated quickly.

I've not read the thread to see if someone already mentioned this but: When you have PCIe devices set to be passed through to a VM and then perform a BIOS update to your motherboard that may (not always, but may) change the IOMMU group ID of some or all of your passed through devices. This presents an issue where the VM Managers GUI for a VM lists the devices you can pass through to the VM like normal, but the XML config still has the old entries which are not presented in the GUI because the group ID's changed vs what was in the XML previously. This stops the VM from being started with an error message of missing PCI devices. The only fix is to modify the XML configuration by hand which I think many won't know what to look for or remove or even believe that's the issue due to the GUI not corresponding to what the XML has inside of it. I think a fix for this would be to update the GUI to recognize missing devices that are present in the XML and ask the user and have buttons to remove missing devices.

That's an excellent score. I was thinking about going Threadripper myself in the future due to the clock speed advantage they have. Hopefully AMD continues to release Threadripper Pro

Today I upgraded the processor from the 7443p 24-core to the 7773X 64-core with 3D V-Cache. I've been surprised by the performance, the cores actually do want to hit 3.5GHz with regularity (with turbo-boost enabled of course). I performed only a couple benchmarks from within Windows due to the extremely slow loading time of the benchmarks from a USB 2.0 thumb drive that I was running Windows from. Neither of these benchmarks will take advantage of the 3D V-Cache and so this is more just a raw core compute test than anything. Obligatory Windows Task Manager shot for the hilarity that is displaying 128 threads. As this was only Windows 10 Home it did detect the 1TB of RAM but only lets you use 128GB of it: unRAID's main dashboard does actually struggle a little bit to keep the information refreshing smoothly with this many threads showing. Feels like it's missing some frames when it renders the animations on the bar charts. But everything is working well, very happy with this chip already as you can see it is singing with my software and there's still headroom left left for more activities. So what is the next upgrade for the server? - Today I don't have anything planned but I think in the far off future (few years time) I'll likely buy a 100Gb NIC to finish it off, but only if I end up getting faster internet at home that would warrant it, currently the multiple 10Gb connections the server has is perfectly fine for my needs.

I would say heatsinks would help for sure, especially just to add some thermal mass that the incidental airflow can hit. Honestly it surprises me these kinds of high-powered modules don't come with heatsinks like FB-DIMM's used to back in the day which also ran very hot. I've actually put the 7773X upgrade on pause as the seller I was buying from tried a bait and switch for an engineering sample version after I'd already paid for a brand new retail 7773X. Then a second seller informed me their 7773X's are locked to Dell motherboards so sourcing the chip has become complicated. But I don't think there will be any issues cooling wise, the 7443p I have now is a 200 Watt chip and it remains around 35-45c (idle-load). This heatsink from Arctic is quite good. Mhm it is surprising. Those are some great temperatures and indeed I like them to stay as cool as possible for longevity.

So with this memory upgrade from my last post I have actually had a lot of problems with the temperatures. With my default cooling setup they immediately (within 5 minutes of booting) reached 98-100c when the system was idle and I had to shut the system down and figure out a solution. By contrast the previous memory only hit 62c or so, way below the 85c warning and 90c thermal shutdown alerts in the Supermicro IPMI. So at first I tried of course increasing fan speeds, that didn't do anything. I tried inserting an extra 120mm fan pointed just slightly below the CPU heatsink so it hit the memory and also left the lid of the chassis slightly open to let more air in. This didn't resolve the overheating issue and the memory still hit 82-83c under load and was still climbing in temperature before I shut the system down to try something else. Here is a photo showing the setup (which didn't work) with the 120mm fan added in front of the CPU heatsink assembly, pardon the dust. As you can see I had a stack of U.3 drives positioned in-front of the CPU heatsink in the top right corner and I thought that was perhaps part of the issue as they do get quite warm, consuming up-to 19 Watts each and I have 4 of them for 76 Watts total. However, the CPU was still very cool only 35c idle and 45c under a high load with the fan RPM's at 900. So I purchased a PCIe to 4 x U.2/U.3 card and moved those U.3 drives out of there and in doing so removed that rats nest of cables. This surprisingly did not change the temperatures of the RAM or the CPU at all. I tried positioning two 92mm fans connected together side-by-side aimed directly at the RAM below the CPU heatsinks intake fan, this did have an effect but the memory at idle was still 72c and would likely go over 80c in the summer when the room is hotter or under a high load. Back to the drawing board. I thought about the airflow situation and what exactly is happening, well that big CPU heatsink is sucking in all the air before it has a chance to reach the RAM modules underneath. Ordinarily not a problem with low-to-medium sized UDIMM & RDIMM's as they consume so little energy but these 128GB LRDIMM's are a different beast entirely, I would estimate they are consuming somewhere in the range of 120 Watts under load across the full 8 sticks vs about 40 Watts for the 32GB RDIMM's I had previously. So the solution to the problem, firstly I moved around a few of my PCIe cards, moved the U.3 drives to a PCIe card as mentioned previously and then I propped up two 120mm Noctua fans off-set from each other, one in front and one behind the current CPU heatsink and this has actually reduced the RAM temperatures to between 58c-62c idle and 64-66c under a stressed load. Very far away from the 85c warning / 90c shutoff temperatures I was seeing before. Here is how that looks in the system: I was afraid that removing the fans from the heatsink would harm the CPU temperatures due to the loss of static pressure but to my surprise the temperature has remained about the same 40c or so. So some lessons here, firstly I knew before I got this RAM that these are intended for very high airflow server chassis. But since I work in the same room as this server I needed it to be quiet, in-fact the build with these 7 fans seen in the photo (3 black Noctuas in the middle, 5 brown ones in the main chamber) have had the needed result, every component is now running cool and the build is still whisper quiet. It's a little janky.. but it works and honestly if it didn't the RAM would have to go. If I were to do another build in the future I would probably just go for the largest RDIMM's perhaps 8 x 64GB for 512GB total to not have such a complicated cooling setup necessary for LRDIMM power consumption. But the RAM is really good, the performance (latency and throughput) is identical to the RDIMM's I had before and apart from the tinkering I had to do with the cooling, works great. Perhaps someone else will stumble across this post and find it useful, my clear warning here is, you need high airflow or a large case where you can pepper multiple fans to cool these high capacity LRDIMM's and perhaps even if I used a less-wide CPU heatsink that allowed for a RAM-specific cooling apparatus that would be a viable solution too.

Have a new update today a prelude to another update coming in a couple weeks time. Today I've installed 1TB of DDR4 3200MHz CL22 LRDIMM ECC memory (Model: M386AAG40AM3-CWE). That's 8 x 128GB modules from Samsung. The upgrade went without a hitch even though I had read online about the H12SSL series of boards having some trouble posting when LRDIMM's are installed I didn't have any such problems, the system posted after a minute and all was good. This may be due to me using EPYC Milan and not EPYC Rome combined with much newer modules, I think first-generation DDR4 LRDIMM's that were around 2400MHz are the problematic ones when it comes to EPYC. I've included some photos of the memory and thought I'd start with a humorous one where unRAID is displaying that I have 1024 TiB of memory installed. No doubt a niche visual bug. The eight modules: The information from a single module: And I thought showing how they look in the Supermicro IPMI may also be interesting: Unlike the previous 32GB RDIMM's I was using the IPMI isn't able to read the speed of these 128GB modules, but I did check them from within Linux and they are all running at 3200 MHz. Also the IPMI is able to read the temperature sensors from all the modules and boy do they get hot. In-fact I had to install an extra fan in my system just in front of the modules to keep them from overheating. So that's the update for today, I have another one in a few weeks when my new CPU arrives which is an EPYC 7773X 64-core Milan-X processor with 3D-VCache. I decided to pretty much max out this build and get many more years out of it instead of switching to a newer socket, the main reason for that is the lack of PCIe slots on the newer systems combined with the performance of the Milan-X3D chips actually being around 6% higher than the same core-count Genoa chips that lack X3D so it was a good upgrade for my use case considering I didn't need to change motherboards, cooler or lose 2 PCIe slots.

It has been a great system for sure. I'm very happy with it and with some of the additions I made like the RTX A4000 and the 7.68TB SSD's it has been able to fulfill more roles that I have for the server. Most of the CPU usage on it is actually from my own software that I write so there has been some situations where more cores would have been beneficial to me but that can be remedied, I'll probably pick up an EPYC 7773X (64 Core with 3D V-Cache) when they're cheaper and swap that in as another upgrade. The SSD's perform well. I'm hitting up against the FUSE overhead on unRAID quite often so using these SSD's while bypassing FUSE to get maximum performance is basically required or I'm only getting 1/10th what they can do. Similarly I've found ZFS to be quite hard on the processor so I think that would also benefit from the 7773X if and when I do that upgrade. Looking back at the system in hindsight I think the only things I would have changed is I wouldn't have purchased those M.2 PCIe cards from Amazon the basic looking ones. They struggled with higher powered M.2's. I would have just bought 2 x Hyper M.2 from Asus right from the start. And also the power supply I bought while quite efficient (80+ platinum rated) isn't as efficient as some other units that cost a bit more. I would have definitely gone for a titanium power supply from the get-go to get that extra few percent of efficiency. Also just to touch on stability, the system has been rock solid. I encountered a few bugs with Docker that broke unRAID a few times, but those are not hardware issues they're just software bugs which I was able to work around. The motherboard from Supermicro has been very dependable and everything has "just worked" including those U.3 SSD's which make use of the SlimSAS x8 connectors on the board. Whenever you use cables for high speed niche connectivity like that it's a question mark about whether it really works properly and I'm happy to say it does, I've not had any issues with it or ZFS since adding those drives. Looking towards the future apart from the potential 7773X upgrade I'm looking forward to future linux kernels which apparently will include directed I/O for FUSE shares so that reads and writes can head directly to the disks which should significantly reduce overhead, that'll likely be a huge efficiency win for my usage.

I've found it idles around 45c in my server and a 41% fan duty cycle. When transcoding it gets up to 55-60c. When I'm doing any inference on it, if it's something that takes a few minutes I've seen it hit 90c and begin to throttle due to temperatures. So far I really like the card, it's quite fast especially for LLM's. I've not had it crash or stop working due to temperatures at all it seems to throttle itself quite well when it needs to. For short loads say under a minute sometimes the power limit kicks in before the temperature limit.

Congratulations 19 years is an amazing achievement and unRAID has a huge amount of momentum even after all this time. I've only been using it for two years but I'm really happy how you've been developing the software, the website here, the community forums and more ❤️