ufopinball

I like the idea of building a wiki page. I bought my Ryzen on launch day (March 2nd), and put it in an unRAID box early on. The other thread is ~20 pages long, and while it has a lot of good information, it's likely a bit daunting. I'll see what I can do about making a wiki section, but you can start with the previous thread and see what you think. My build has been running for ~3 months now. There are a few gotchas, but the feeling is that those things will be sorted out eventually ... either by BIOS upgrades, or unRAID & Linux Kernel upgrades. Do you have a list of specs on what you have bought or plan to buy? My specs are in my signature line, if that helps. - Bill

Last week, I had upgraded to the ASUS Prime X370-PRO beta BIOS 0803 with AGESA 1.0.0.6 ... so far, the server is running smoothly. Today, I tried a test run with unRAID 6.4-rc2 (sorry, it's what I had on the thumb drive), but the later Linux kernel didn't include any new support for the sensor chip on the ASUS Prime X370-PRO motherboard. Since I was only curious, I built a temp thumb drive to run the test. Afterwards, I booted back on my regular thumb drive that's running unRAID 6.3.5. If you're interested in using the Dynamix System Temp plugin on the ASUS Prime X370-PRO, check my previous post for details. Bill

Okay, so I have loaded the ASUS Prime X370-PRO beta-test BIOS 0803 with AMD/Ryzen AGESA 1.0.0.6. I am still running unRAID 6.3.5, though. The BIOS upgrade seemed to take longer this time. As usual, I had to go in and redo all my preferred settings. The system accepted them and rebooted, then did nothing for like 10 minutes. I finally powered down, counted to 10, and started it up again and this time it worked. No idea what was up with the weird hang. I've rebooted about 3-4 times since then trying different things, and have not encountered it again. Just be-aware that this may occur, at least the first time after the BIOS upgrade. Otherwise, my system temperatures are a bit higher, but we're in the middle of a heat wave and that may have something to do with it. My memory is rated at 2133, so I'm not testing for the 3000-4000 speeds that others are looking for. I did turn off all the ACS override stuff, but the "new" IOMMU table doesn't look all that impressive: IOMMU group 0 [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 1 [1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 2 [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 3 [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 4 [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 5 [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 6 [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454 [1022:145a] 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a [1022:1456] 29:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456 [1022:145c] 29:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c IOMMU group 7 [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454 [1022:1455] 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455 [1022:7901] 2a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) [1022:1457] 2a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457 IOMMU group 8 [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 9 [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467 IOMMU group 10 [1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02) [1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02) [1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02) [1022:43b4] 1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1000:0072] 21:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) [1b21:1343] 25:00.0 USB controller: ASMedia Technology Inc. Device 1343 [8086:1539] 26:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 11 [1002:677b] 28:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450] [1002:aa98] 28:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] I ended up switching the ACS override back on, since my Windows 10 VM wouldn't start without it. Unfortunately, that's about all the time I have at the moment. I'd still like to switch from the USB audio to the onboard audio, and will hopefully find some time to poke at that soon. Dunno if anyone is monitoring the unRAID 6.4 board, but they're up to RC3 now. No idea how close they are to a final cut. That's still on my list, though. Anyone else give it a try? - Bill

Some minor updates. A new "system stability" BIOS has been posted for the ASUS Prime X370-PRO motherboard, version 0612. I've been on it for about a week and have not encountered any issues. Likewise, I upgraded to unRAID 6.3.5 in a similar time frame, and things are going smoothly. I know there's an unRAID 6.4 pre-release RC available, but I haven't had a chance to load it yet. Might have better support for Ryzen, since it's based on a Linux 4.11.* kernel. Also there appears to be a ASUS Prime X370-PRO beta-test BIOS with the AMD/Ryzen AGESA 1.0.0.6. If I have time I may load that as well. Just throwing it out there for those who are looking for a weekend project. :-) - Bill

For what it's worth, this has always been the behavior that I have experienced. Even when I was doing pre-testing with an Asus F2A85V-PRO motherboard and AMD A10-5800K Trinity 3.8GHz FM2 Quad CPU. The three GPUs I tested were the ASUS Radeon HD 3450, ASUS Radeon HD 4350 and Asus Radeon HD 6450. Dunno if it's something specific to AMD CPUs, ASUS motherboards or ASUS Radeon GPUs ... or even the Windows OS? Are others seeing this same sort of behavior where you shut down the Windows VM and it doesn't return you to the unRAID console? Or is this just how it's expected to work? Thanks... - Bill

It's certainly worth trying out the other options, but failing that I would reboot and go into the BIOS and try to record nominal values for the CPU/MB temp and CPU fan. Then boot into unRAID and see which of the items in your "sensors" outputs comes closest and use those. Afterwards you can always run a Prime95 or similar test and see if the output values follow the expected progression. - Bill

When you say "onboard audio", do you mean the audio off the motherboard, or off one of the GPUs? I'm presently using a USB audio device. Looking at my "System Devices", the only audio device is the HDMI audio output from my GPU. I'd love to reduce another device and work with the motherboard audio chip. Would you share the necessary details? Thanks! - Bill

That's disappointing. With the ACS patch on unRAID, I am able to pass in one of three USB controllers built into the motherboard. I ended up returning my USB 3.0 PCIe card because it wasn't necessary anymore. It was less than $10, so I suppose it's no big deal either way. Still, it would be nice to be able to plug the USB 3.0 card into a PCIe x1 slot, than having to use one of the larger x16 slots ... that's just how the IOMMU groupings broke up. - Bill

Good article on the Ryzen AGESA 1.0.0.6 firmware update, which includes much more flexibility (BIOS settings) when it comes to expanded memory support. Surprise, there are also some details on expanded IOMMU/ACS support! AMD must have listened when this was brought up in an earlier AMA meeting. Glad to see them stepping up items for virtualization, as this will become more important moving forward into Naples/Epyc. https://community.amd.com/community/gaming/blog/2017/05/25/community-update-4-lets-talk-dram All that said, it looks like it's not out yet, just in beta test. Actual finalized BIOS updates aren't expected till mid/late June. Wonder what else will be included? - Bill

Okay, so I took another stab at this today. For the record, my board is the ASUS Prime X370-PRO. If I open a putty window and run "sensors-detect", here's what I get (skipping 'No" items): Trying family `ITE'... Yes Found unknown chip with ID 0x8665 (logical device 4 has address 0x290, could be sensors) Unfortunately, the ITE 0x8665 driver is not included in the unRAID 6.3.4 build ... however, this led me to this thread: https://github.com/groeck/it87/issues/10 Following their instructions, I tried this command: modprobe it87 force_id=0x8620 and then ran 'sensors', which now produced this output: it8620-isa-0290 Adapter: ISA adapter in0: +0.88 V (min = +2.22 V, max = +2.42 V) in1: +2.74 V (min = +0.97 V, max = +1.39 V) in2: +2.20 V (min = +2.06 V, max = +1.81 V) in3: +2.18 V (min = +1.34 V, max = +2.27 V) in4: +0.04 V (min = +1.57 V, max = +0.49 V) in5: +0.04 V (min = +1.46 V, max = +2.76 V) in6: +0.04 V (min = +2.11 V, max = +0.16 V) 3VSB: +3.60 V (min = +3.24 V, max = +4.51 V) Vbat: +3.46 V in11: +0.41 V Array Fan: 1053 RPM (min = 11 RPM) fan3: 795 RPM (min = 40 RPM) CPU Temp: +44.0°C (low = -31.0°C, high = +78.0°C) sensor = thermistor MB Temp: +32.0°C (low = +84.0°C, high = +13.0°C) sensor = thermistor temp3: +34.0°C (low = -13.0°C, high = -14.0°C) sensor = thermistor temp4: +34.0°C temp6: +34.0°C intrusion0: ALARM Since we don't have the proper driver, we're using a 'compatible' driver, so the above information has some issues. Notably the min/max values are incorrect ... however for my purposes, I am only interested in obtaining enough data to populate the Dynamix System Temperature plugin. This works as follows: To complete the setup, I added the command "modprobe it87 force_id=0x8620" to my 'go' file and rebooted. The system loads the drivers at boot time, and the plugin now displays the CPU/MB temperature and the CPU fan speed on the status line. Hopefully in unRAID 6.4, they'll include the necessary sensors packages to support the ITE 0x8665 chipset. If someone has one of the other ASUS motherboards and find that this also works for you, let us know? - Bill PS: I did not test this with an earlier version of unRAID, so I don't know if version 6.3.4 (or later) is required. You may wish to upgrade prior to attempting this.

I've been checking every few days, but so far there's no new BIOS for the ASUS Prime X370-PRO motherboard. Last update was on April 10th. Otherwise, I was away for the weekend, but upgraded to unRAID 6.3.4 this morning. So far, no problems. Can't comment on stability, I'm leaving my Windows 10 VM because that was my original use case. This effectively prevents me from testing stability without a VM running. Again, the C-States setting doesn't appear to make a difference on this board, so it's currently at the default setting. - Bill

Upgraded from 6.3.3 to 6.3.4, went smoothly with no issues. Thanks Limetech!

Are you running any VMs? It might be worth setting one up. I have the ASUS Prime X370-PRO motherboard, and last I checked (albeit not recently), disabling C-States didn't actually do anything. My guess was that the BIOS setting exists, but the underlying feature was unimplemented. My system stays alive (16 days, 7 hours, 55 minutes) by virtue of having a Windows 10 VM running all the time. This keeps the processor busy enough to keep the sleep states from occurring, and thus it works around the crash issue. Since you have the ASUS Prime B350M-A board, I am guessing that the C-States setting may not be performing any action on your board, either. You can test power consumption with the C-States enabled/disabled and see if there is a difference. Not sure how you got a week of uptime, maybe you have a VM but it's not always running? Anyway, something to try out... - Bill

Sounds good, where did you find the details on this setting? I'm curious to know when it is recommended... - Bill

Interesting, though I'd argue that Test 3 should be a repeat of Test 1. I'm still expecting that the file is being cached after the first copy. Also, for Test 4 ... I do not my cache drive to buffer writes to the array, all writes to the array are done directly to the array drives (and of course, parity) ... I'm not sure if Test 4 is writing directly to your 7200RPM, or is being cached by your SSD to be moved to the array at a later time. If I did my math correctly, my Test 1 achieved 63.5MB/s, so maybe that's not out of line ... though my drives are all 5400RPM. Your numbers *should* be faster. Can we try this? Pick a different 4GB+ file and run the test again, but this time run Test 2 before Test 1? Can probably skip 3 & 4 for now. - Bill

Okay, so here are my test parameters: The source file comes from a SAMBA share on UNRAID (\\cortex\disk2) and is 4.37 GB (4,695,246,094 bytes). The first destination is C:\TEMP on my Windows 10 VM, which lives in \\cortex\cache\vDisk The second destination is also a SAMBA share on UNRAID (\\cortex\cache\inbox) In my opinion, the two destinations should behave similarly. All tests were performed using "TeraCopy" running on the Windows 10 VM. Test 1: Copying the file from \\cortex\disk2 to C:\TEMP took 70.50 seconds Test 2: Copying the file from \\cortex\disk2 to \\cortex\cache\inbox took 32.81 seconds Test 3: Copying the file from \\cortex\disk2 to C:\TEMP (again) now takes 35.38 seconds My UNRAID system has 64GB installed, but 16GB is reserved for the Windows 10 VM. My guess is Test 1 takes longer because the file is read from a HDD. Test 2 & 3 run faster because the file is now cached in RAM (either in UNRAID or the Windows 10 VM). The difference between Test 2 and 3 is not wholly insignificant, but hardly seems worth worrying about. Finally, as I understand it, file copies here do not really involve the NIC since none of the data needs to go over the network? My VM is running the Red Hat VirtIO Ethernet Adapter. The hardware NIC is the Intel I211AT Gigabit LAN, integrated into the motherboard. I did not change the MTU settings. The \\cortex\cache drive runs on dual SAMSUNG 850 EVO 2.5" 1TB SATA SSDs, configured in a cache pool. The \\cortex\disk2 drive is a Seagate ST4000DM000 4TB SATA HDD. Let me know if you'd like me to run additional tests. - Bill

This is the example I followed: https://forums.lime-technology.com/topic/55150-anybody-planning-a-ryzen-build/?do=findComment&comment=553895 - Bill

Good to know the Dell H310 and SuperMicro SAS2LP 8-port controllers are comparable in power consumption ... at least at idle. I was running on the SAS2LP when I was on unRAID 5.x and never had a problem. I upgraded to unRAID 6.x and ran into the same problems that others were seeing (drives would suddenly disappear during parity checks, etc.) LimeTech never offered a fix, but did suggest that the older motherboard (ASUS M3A78-T w/ AM2 socket, purchased in 2008) might be to blame. So, I moved over to the Dell H310 controllers which worked fine in my existing setup. Much later, after finishing the Ryzen build, I freed up my Win7 workstation (ASUS F2A85-V PRO w/ FM2 socket, purchased in 2012) and ran a test with the SAS2LP board. I didn't run into problems, so it seems that LimeTech was right on the money for this issue. If/when I decide to sell the SAS2LP boards, I'll feel less guilty since they appear to be solid so long as your existing setup isn't as ancient as mine was. - Bill

My raw notes read as follows: Power per SATA controller: 92.2w - Bare machine (Asus M3A78-T MB) w/ 4GB RAM and no other peripherals 99.4w - Dell H310 (8 ports) [7.2 watts x 2 = 14.4 watts] 101w - AOC-SAS2LP (8 ports) [8.8 watts x 2 = 17.6 watts] 97.4w - Adaptec 430A (4 ports) [5.2 watts x 3 = 15.6 watts] This was taken some months ago. I don't know what power supply I was using, but the wattage was measured using a Kill-A-Watt device. How did you perform your tests? - Bill

Okay, so if the overclock sleep bug is previously fixed, is the new AGESA 1.0.0.6 all about memory compatibility? Is there something specific you're looking for in this release? Regarding power consumption ... I was actually looking at your HighPoint 2760A. If I upgraded to a 20 or 24-drive case, I'm going to need more ports. Right now I have 8 off the motherboard, and 8 off the Dell Perc H310 controller. I guess at best I'd only need 8 more ports, but it's nice to know what's available. Interestingly enough, I did some power testing on the various controllers I have in-house. Here's what I recorded: Dell HV52W PERC H310 (8 ports) -- 7.2 watts SuperMicro AOC-SAS2LP-MV8 (8 ports) -- 8.8 watts Adaptec RAID 1430A (4 ports) -- 5.2 watts At the time my motherboard had only five SATA ports ... so to support 15 drive slots (three 5x3 cages), I would either need two 8-ports, or three 4-ports (plus MB ports). The two Dell H310s worked out best at 14.4 watts, the two SuperMicro SAS2LPs come in at 17.6 watts, and the three Adaptec 1430As total 15.6 watts. Considering your power report I'll probably spend the upgrading money moving to 8TB drives, instead of moving to a whole new case and add/upgrade drive controllers, just to support more 4TB drives. Also, since I just dropped a lot of $$$ on the Ryzen build to begin with, I figure the next update will have to wait for a bit. My server is full-ish, but there's still a fair amount of space available. I'm not adding content *that* fast, and there are a few things I can totally get rid of and reclaim that space. - Bill

Glad to hear there are new updates. I'm not having any issues with my RAM, but expansion on memory compatibility is still a plus for the platform. The second article did mention: We resolved the “overclock sleep bug” where an incorrect CPU frequency could be reported after resuming from S3 sleep. Dunno if that will help solve the C-State issue at all? BTW, do you keep your drives spinning all the time? My system idles at 54 watts (drives spun down, Win10 VM active but idle), which is substantially better than the 90 watts you're seeing with C-states enabled. Considering our system configurations are reasonably similar, I'd expect the power usage to be roughly the same? I'm measuring off the UPS, and the unRAID tower is the only active item plugged into it. My C-States setting is the default value, enabled. - Bill

If I knew then what I know now, I might have just stuck with the 1700 or 1700X. I'm not an overclocker, but I have dealt with somewhat unstable systems. I guess I wanted a CPU that was binned as one of the best, just to be sure and hopefully save myself some headaches down the road. I did vet it pretty severely with things like Memtest and Prime95. Everything passed, so I'm pretty happy. My motherboard is the ASUS Prime X370-PRO (full system specs for 'Cortex' are in my signature). This arguably wasn't the top-of-the-line since it's not the gaming version of the X370 board. This one came with 8 SATA ports, plus an M.2 slot. Since I can only fit 15 HDDs in my system, I was able to drop a SATA controller. Now I have 8 ports on the MB, and 8 ports on the H310 controller. I don't know if/when I will buy an M.2 drive. I bought dual 1TB Samsung SSDs to use as a cache pool, so I should probably stick with that for now. Seems to perform quickly enough. This motherboard also comes with 3 PCIe x16 slots (I think one at x16 and two at x8), which I would need for a GPU and/or SATA controller(s). So I picked this one since there was a fair amount of support for things that were important to me. My GPU is pretty run-of-the-mill, a Radeon 6450 that's mainly used for desktop graphics. Again, I'm not really a gamer ... should that change someday, it should be no big deal to upgrade the GPU as needed. At any rate, the Radeon 6450 passes through just fine to my Windows 10 VM. I have also tested with a couple of older cards, the Radeon 3450 and the Radeon 4350. I upgraded in order to test GPU pass through with things like Linux (CentOS7), but apparently the driver support just isn't there yet. In theory the newer Radeon 6450 card will run cooler and use less power, so I'm hoping it wasn't a completely pointless purchase. - Bill PS: I don't know what 'trick' is involved with not using a secondary card. I've only ever had one GPU in this system.

That's pretty much what I did. I had an existing unRAID build based on the AMD Phenom II 1100T (6-core CPU) and upgraded to the AMD Ryzen 1800X (8-core/16-thread CPU). I dropped in my new motherboard as the upgrade and had minimal issues. Somewhere I read to just configure your VMs and Dockers so that Autostart=OFF, that way you can start just unRAID and then check settings, CPU pinnings, etc. prior to starting anything else up. My LAMP VM (CentOS 7) had no issues whatsoever. I don't remember running into issues with the existing Windows 10 VM, but I ended up starting a fresh install with a larger vDisk size. When I had upgraded my cache drive, I recall having issues with Plex, so I may have nuked that and started over ... I don't remember now, it was a few weeks ago. Not really a big deal anyway, since most of it runs in the background. What are the specs of the new (Ryzen) rig? - Bill

Earlier this month, Tom's Hardware held an AMA (Ask Me Anything) with some folks from AMD. The IOMMU grouping issue was discussed, but not much was thrown our way: Presentato: As someone interested in doing virtualization and PCI pass-through of a GPU early reports of IOMMU groupings don't look promising for the consumer motherboards. Is that something AMD can address or are any improvements reliant on motherboard manufacturers? DON WOLIGROSKI: This is something I've personally started to look at recently as a pet project. I'm playing with VM-Ware on my Ryzen system at home because, really, Ryzen's highly-threaded CPUs bring a lot of virtualization potential to the table in price segments where it hasn't been before. The sub-$300 segment has been limited to 4-thread processors on the Intel side, while Ryzen 5 ratchets that up to 12 threads. Boom. Having said that, we're in launch mode right now, and virtualization isn't a top priority at the moment. We're laser focused on making the platform as fast as we can in the near future. I anticipate we'll look harder at virtualization as time goes on. I guess this doesn't really answer the question about how much the BIOS is involved in dividing up the IOMMU groupings. Anyway, at the very least, we're not the only ones asking this question, and AMD now knows it's an open issue. Link: http://www.tomshardware.com/reviews/amd-ryzen-ama,5018-2.html - Bill

I think that it will. I have the ASUS Prime X370-PRO motherboard paired with a Ryzen 1800X CPU. I have my own multi-threaded image processing program that just screams on a Windows 10 VM, hosted by unRAID but only using half the cores/threads (8) that are available. If it's CPU bound, chances are it will perform quite well. Pretty much everything on the Windows 10 VM (with GPU passthrough) runs very smoothly, though one thing I did notice is some minor delay when I try to play a video file. I'm using VLC and the first few seconds will stutter a bit, then the system sorts itself out and the rest of the video runs fine. I'm not sure if this is an issue with VLC, or some conflict with WIndows 10 itself (kinda new to this OS) ... or something specific to VMs or Ryzen. It honestly is a minor issue, so I'm not overly concerned and I haven't done much testing with other players. As of now, I have 11+ days of uptime, and am pretty happy with the current setup. When I have more time I'll dig deeper into the ACS override stuff and see if I can re-arrange my IOMMU groups. If I can dispense with the USB PCIe card, that would be nice ... maybe save a few watts on power. Experimenting with the Global C-States is still down the road a bit. - Bill