mavrrick

I have been running unraid for a while on a more traditional setup with spinning fisk and a few SSD's for caching. Recently i aquired a Mini PC based on a N305 intel chip that can run 5 NVME drives at PCIe 3x1 speeds. I have 4 drives now but will eventually populate the 5 slot. Currently the drives are 2 crucial P3Plus 4Tb drives, a WD SN770 1TB drive and a WD SN560e 2 Tb drive. The question is what is the most reasonable way to not kill the drives performance any more, but also maintain minimum impact on write endurance. Right now i have thems as a simple 4 drive array with no parity. Then backing them up to my older unraid system with spinning disk. I have read some fairly negative things about ZFS when it comes to NVME drives and performance, and i would prefer not to loose 50% capacity with BTRFS. So how are others doing this with there setups. Right now i am leaning toward the best option being to go ahead and get 3 more of the crucial 4TB drives and use ZFS with it being the best option.

You need to check the BIOS under the CPU performance and power section. In there under a few menu options you will find CPU Package and CP Power Limit settings. Your BIOS for your board may have those settings set to something other then 0 which mean it i not following default cpu settings. You can also set it to 9 watts limit the power used. You also want to go through the bios and turn off anything in the bios that isn't actually used. Any part on the board that isn't being used but powered can add to the idle power envelop. In my case i turned off things like the Audio, or Sata controller. Remember my setup isn't one of the NAS boards so i can't speak exactly for your setup. If you add "i915.alpha_support=1" to your boot config on for unraid it enables the video driver Then you can add these plugins to your install Once the plugin is configured you will have a dashboard item like this When configuring the GPU Statistics plugin to get the power usage make sure you set the "Power Draw Selection" value to "Max of GPU or Package"

I tried the precompiled kernals from the links in that reddit post and it didn't work and nothing changed. I tried both the 6.6.9 and 6.7.0 kernals. Hopefully when the official release is done whatever work is completed will fix it, but the lack of results with that gave me a l little pause. Hopefully that is all that is needed and nothing else.

Keep in mind that isn't the amount of Power the processor will draw exactly, but more like guidelines when designing the cooling solution for the processor. That just means you need to have a solution that can dissipate that much heat. I would really suggest checking the bios as I found some strange settings that didn't make sense for processor package and CPU power limits. You can also potentially be loosing boost if your cooling solution is better then what the processor spec is designed for. With the bios default settings my CPU would ramp down to a Configured power of 10 watts when all cores were being pushed. and stay that way. It was hard set that way. That would limit the CPU to 1.8Ghz when all 8 cores were under heavy load. Just by removing the limits now my cpu will boost to 3.x ghz and then after about 30 seconds it will drop to 2.3ghz instead of 1.8ghz. With full load it cruises around 57c. So that is a boost of 500mhz simply by removing that setting. All of that ofcourse doesn't help your idle power though.

Keep in mind many of those folks reporting 2-7 watts of power usage are getting that with bare minimal hardware configured just to test with. You mention 2 NVME and 3 hard drives. I would also bet you included a fan or two. To test your power usage power down and disconnect those 3 hard drives and any fans you have running. Then when you power on go into the bios and turn off anything that isn't needed like HD audio, or any other devices. Also disconnect any USB devices that are not needed to boot. The more you can simply turn off the better. Then power it on and let it stabilize that would be your comparable power draw to those 2-7 watt systems. I also found recently found that my cwwk N305 4x2.5gbps mini pc had some some strange power settings for the CPU and Package power limits. I would suggest you go into there and set them to "0" so that the cpu can run as designed unless. That may help with performance. I have found now i can achieve better high end clocks and complete work faster. i also cruise along with 4 NVME drives at around 15 watts. If you load the get the intel driver loaded and use GPU Stats you can see the processor package power on the Unraid Dashboard. The processor does power way down to under 1 watt at times so it is the rest of the gear that is driving the power usage.

It looks like getting Kernel 6.5 or greater is actually is being discussed on reddit here. The first response links to a walkthrough that discusses how to apply the kernel change with a link to compiled kernels. Looks like the last release is actually on kernel 6.7. Think I will try the 6.6.9 kernel version and see how it goes. It would be nice if we could get a moderator or someone to look at this and see about getting this into the regular build before to long if this works.

That is great to hear that there is a solution in site. Now we just need to get there.

I have been researching a bit about ZFS lately because I have setup a all NVME MiniPC Unraid server recently. From my research it seems well documented that when you use drives that are not spinning drives with ZFS, it tends to leave allot of performance on the table. The general consensus I have found is that you don't want to run ZFS unless you want to use some of the file systems advanced features like De-Duplication, Snapshots, ect. When those features become a factor it is very difficult to replace ZFS as the best option. I have yet to find a anyone that claims they can get anywhere near max performance form NVME drives with ZFS As far as your memory question, you may want to read the initial release info for the feature. I believe it stated that it is going to be configurable eventually, but for now it is set to 1/8th the total memory of the system. ZFS can be very memory intensive in a large scale deployment. There are also allot of features that will be coming with ZFS in the next release based on that blog post. It would be interesting if we could get some benchmark numbers between ZFS and BTRFS.

The benefit would probably be more of a future thing. Think of it like this. DDR5 simply means more memory throughput potential at a slightly higher latency. If you had enough multitasked things happening it could be a bonus, but if your memory isn't taxed much now it likely won't make much of a difference. Generally speaking right near the memory change between generations good optimize previous gen memory is just as fast/efficient as the latest and greatest of the next gen in over 95% of stuff. That will likely change in the future though. There is also a slightly lower power requirement for the new stuff as well that may help low power systems. That said You aren't running the best DDR4 memory currently since it is 2666. Personally, I would do DDR5 in one stick probably to reduce power usage.

This one. Though in August when i ordered it i got the V1 version. Now they appear to have move To V2 which from what i can tell just added 2 Superspeed USB ports and two USB 2.0 ports to the front. Everything else is the same. From that page though you would need to select the N305 chip one and the option with no ram/no SSD + 4 m.2 nvme. The last option gets the daughter board.

May be true, but i am very impressed with it so far. My unit from CWWK with the adapter board to enable 4 NVME m.2 drives has been functioning great for about 3 months. I now have 11TB of storage in it and it idles around 13 watts when not busy. That is with a 2TB WD 650E, 1TB WD 770, and 2 4TB Crucial P3 plus drives in it. So i have that expansion board filled. with drives. Only thing i wish it did was have proper guid from the built in TF card slot so i could git rid of the USB Flash drive. One thing i possibly should consider is a more efficient power brick as that is something that has been called out in some reviews as a potential efficiency loss.

Be careful with this adapter as i believe that is the 2.5Gbps adapter that is blacklisted. You may need to do some extra stuff to make it work. I would be interested to know your complete BOM. Also how this goes as you grow it out. The 2-3 watt power draw is impressive, but with so few devices connected i wonder how it will scale. I am very curious how i twill scale once you add a controller and such to the unit. Either way great job on a low power solution. I really wish there were more options to house many NVME SSD's

Something else you may want to consider as well is simply going with a Low Power N305 or similar MB/cpu/mem combination. Not sure what your Memory situation is like, but it wouldn't take much of a new mb and Processor to almost pay for itself when the power bill considered most likely.

Each core runs between 800Mhz and 3.9Ghz If we are talking about raw size of the content, it isn't 24TB. I recently went through and converted everything to H.265(HVEC) to recover space. Prior to that it was close to probably 16TB. Now that collection because of cleanup and such is down to around 3TB. Keep in mind the MiniPC i am using would never to a 24TB setup unless you used 8TB NVME drives. I doubt the 3 NVME drives i have are sleeping and i am sure when they are busy the power draw goes up, but i think most of the time they are idle. Keep in mind most of the time this system is idle doing nothing. When that is happening it hangs between 12 and 16 watts depending on what i look at is measuring it. I kind of doubt the 12 watts a bit now since that is coming from the UPS. The 16 watt measurement is from a Smartplug with power monitoring. The Smartplug tends to vary between 14 and 16 watts. I kind of think this is dependent on the gear. I have watched a few youtube videos and it is clear that the power usage depending on the gear can vary a fair amount. Is it possible it is higher sure, but so far i haven't seen any significant impact. The system i have is really only good with the NVME internal storage, but the V2 of the hardware does have 2 usb 3 10gbps ports on it. That may work, but without knowing for sure, it is hard to tell Very true. The N305 only has 9 PCIe Gen3 lanes. The question I ask though is how fast do you need the NVME's to be. Even if you have 10Gbps in your network and have 1 lane of PCIe Gen3 which each of those drives has will do a decent job of filling that pipe. I have seen my drives reach speeds up to close to 800MB/s on this little guy. Well beyond the the 2.5GB limit of the internal ports. Don't get me wrong I would never pass up the higher number of lanes, or faster drive access if it was possible, but is it needed for this kind of use case. The use case is important and this kind of rig is certainly not for everyone. My point is that the u300e is a little bit of a odd ball. Because it has 1 P-core and 4 E-Cores it technically can't do as much work as the n305 in the same interval. It does have the benefit of faster and more lanes, but is that much of a benefit in these smaller footprints. It doesn't matter if my drive can do 5GB/s if my network connection can only handle 300MB or 1.2GB/s. The higher number of lanes becomes a benefit for things that are local to the box. So maybe for a large influxDB database you could retrieve data faster. But as far as plex goes it is unlikely to make much of a difference. One use case I do see a benefit would be doing video editing from it. But then I suspect storage capacity would become a issue quickly. Personally i am just amazed at this little box. The box that is currently my main Unraid box is a AMD Ryzen 9 5950X with 128GB of ram, Asus ROG X570 Gaming MB, Nvidia 3050 Video card, 2x 3x5.25 Hotswap bay kits with Array disks: 10TB Segate Ironwolf, 10TB WD White, 8TB WD White, 6TB WD Red, Cache: 160GB Intel G2, 120GB OZ Vortex3, Pool1: 2TB Samsung 870 Evo Pool2: 2TB Samsung 980 Pro. Other then the raw processing power the little mini pc is handeling much of the same workload without flinching except for my desktop experience VM. If i end up moving unraid fully to it. i will just revert that hardware back to a desktop. It is kind of a surreal experience walking into the room and it is silent after all these years. Ontop of that because of all of the fans, and such in a Lian Li Armor Suite Case it idles around 110 Watts. Getting that down to 15 is huge from a idle perspective.

That question really depends on your workload and gen of the hardware. If your work load really likes to multithread out it is possible the higher core count will be a benfit if it is capable of doing more work then the lower core processor. If it isn't because of generational difference then stick with the lower core count. Part of the question is do you have anything that can actually consume a fair amount of cores. I would probably go with the 1620 and see if it can stand the load you want to run on it. If so it will probably be more friendly to your power bill when it isn't busy.

I am very curious about this item from CWWK with a i7 1240P cpu. This looks like another option to get 5 NVME drives. The difference between the U300E and the N305 is basically a trade between better single core performance or more overall work capacity with more threads. Everything I have seen between those two puts the N305 above the U300 once you add in threaded applications I would expect once you virtualize or load much in the way of dockers on the those two cpu's the N305 will actually win easily as long as it isn't hitting a single threaded speed limit. A few more observations. 1. I managed to get plex to work with hardware acceleration with the N305. So at this point I am really considering moving my license to it perm. So far it can 100% support my environment. That said I did considerably shrink my setup over the last year with considerable consolidation of old data and improved compression of content I had. 2. When using Unraids dashboard it is showing a low 12.6 watts idle power draw that is with a 140 mm fan running on low speed from the usb port, 3 nvme drives, and wireless keyboard receiver powered and 1 of the 4 nics active. 3. I confirmed it can do 2.5gbps to the NVME drives no problem. 4. It can run nested vm's with multiple instances of VMWare ESXi 8.x no problem. I read through the linked thread about the 7 watt system. It also shows how much harder things get once you include storage. All the stuff that comes with including Spinning disk add power. Hot swap disk bays, fans, the drives themselves. Every little things adds a few watts here and there. I am really liking the idea of complete NVME storage.

A few months ago I picked up one of these CWWK N305 4 Port Firewall devices. And then added their 4xm.2 board that converts one PCIe 3.0x4 slot to 4 slots that are 3.0x1 The key with it is that it provides essentially 8 12th gen Efficiency cores. For a all NVME flash setup it isn't bad. Yes each NVME slot is limited to 1 lane of PCIe Gen3, but there are a total of 5 slots in it. With 4TB drives it you can have 16TB of storage. Being Gen3 that slot allows for about 1GB/s on each drive. You also can't beat the footprint. Also handles VM's and Docker like a champ. I loaded Unraid on it just about a week ago and am testing it now. I can't give idle numbers but it does seem to go pretty low. I have seen values on my UPS that show usage as low as 13 watts with other gear on it as well.

AMD and NVidia encoding engines are very robust and can do allot more then video. Also with the AMD one I am not sure if it is much less efficient. The AMD RDNA 3 cores can be used for several tasks like GPU based Acceleration as well for many other tasks. I wouldn't discount the AMD RDNA3 GPU simply because it doesn't support QuickSync. This is purely subjective. AMD built correctly are very stable and efficient. Also I think this largely depends on the work load being run on them. If you are putting much load on system AMD's seem to have an efficiency edge, but it is heavily dependent on the work load being put on them. If your load isn't much though i do think Intel runs at lower power when mostly idle.

All the new Zen 3 AMD cpu's include integrated graphics so not sure why that would be the case. Some Intel chips don't though.

Is there someone we can attach or link to this thread to for review of this issue in development.

Yea.. I just upgraded from Asus 4403 to 4408 a few days ago hoping it may help. I was running a earlier version as well. I just checked with running virt-host-validate on my older test system and it seems that it gets a similar message. It is different since it is Intel, but this may indicate the concern about IOMMU and ACPI may not be the cause.

@afc_rich What is the bios of your MB. I wonder if something changed in the Ages AM4 part of the bios. I am on the latest bios for my main board which is 4408 Just to make sure it is the same board it is a "Asus Rog Strix X570-E Gaming" with bios 4408 dated on 10/27/22 but I also tried a few back. Your comment about IMMOU is interesting. I do use a Window 10 VM that is attached to a few devices that are passed through. @ryanm91 In my research trying to get this working earlier i did see allot of references to VMWare Workstation having problems when HyperV was turned on. that may be related to what you are experiencing. I think you will need HyperV turned off if you are nesting VMware from what i found.

So I got out my old Unraid server hardware and got it in a working state, then got Unraid running on it. Tested it with trying to run VMware ESXi7.0 on it as this older hardware doesn't support ESXi8.0 and it seems to have worked. It doesn't say much though as that older hardware is a Sandybridge Intel Chip that is rather dated at this point. I think this is likely something between KVM, and the AMD Ryzen CPU and what features are being allowed to the nested hypervisor. I tried forcing the CPU Profile to EPYC and EPYC-IBM and both of these profiles were found to automatically disable the Monitor CPU Feature for the lower VM.

I have not made any progress. I have attempted again today to try to get this working manipulating a few things. Now I am just going to focus on ESXI 8.0. It generates that strange message about: AMD-V is supported by the platform, but is implemented in a way that is incompatible. What is strange is that i can't get ESXi 7.x working either which was working a while back. I have confirmed the platform works as expected if using ESXI on Bare Metal instead of Unraid. For testing i switched my box to boot into ESXi first and then run Unraid underneath ESXi. This kind of worked for testing, but complications with PCIe Passtrough requirements caused me to move away from running ESXi as the main hypervisor. My suspicion is that some of the updates that were applied to KVM in the last few releases may have borked something with VMWare compatibility as a nested hypervisor. VMWARE ESXi 8 flat out says it can't access AMD-V features when being installed. Clearly unraid does based on what it is reporting. It is almost acting like the virtual bios the VM is using has it disabled. Not sure if that is possible but that is what it kind of looks like. When i ran esxcfg-info | grep "HV Support" i get a line that says |----HV Support .....................................................1 From what I can find that means it is supported but disabled in bios. For reference my machine hardware is below. Ryzen 5950X Asus ROG Strix MB with latest bios 128GB Ram I have tried both SEABIOS and OVMF bios in the settings. I have also tried adding SVM as required for the profile as well.

I have tried a few more things since I last posted. 1. I tried loading vmware 8.x ESXI since it is available now. It is giving two error messages: 1. module "MonitorMode" failed to start. 2. AMD-V is supported by the platform, but is implemented in a way that is incompatible. Where there any changes with QEMU that could effect how the CPU's are passed to underlying environments. This seems to me like a feature isn't being passed and causing the Nested environment to not work. I have also tried a few of the options i have seen to enable nested Virtualization.