Anybody planning a Ryzen build?


Recommended Posts

For anyone interested, here are the default IOMMU Groups for the ASRock X370 Fatal1ty Professional Gaming.

 

I was surprised that even with nVidia GeForce GTX670 in PCIe slot #5, it still ended up in IOMMU Group 0.

 

IOMMU group 0
	[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
	[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
	[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
	[144d:a804] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
	[1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
	[1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
	[1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
	[1022:43b4] 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1d6a:d108] 05:00.0 Ethernet controller: Device 1d6a:d108 (rev 02)
	[1b21:0612] 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
	[1b21:1184] 07:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
	[1b21:1184] 08:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
	[1b21:1184] 08:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
	[1b21:1184] 08:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
	[1b21:1184] 08:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
	[8086:24fb] 09:00.0 Network controller: Intel Corporation Device 24fb (rev 10)
	[8086:1539] 0b:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
	[10de:1189] 0d:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1)
	[10de:0e0a] 0d:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
IOMMU group 1
	[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 2
	[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 3
	[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 4
	[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
	[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
	[1022:145a] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
	[1022:1456] 11:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
	[1022:145c] 11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
IOMMU group 5
	[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
	[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
	[1022:1455] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
	[1022:7901] 12:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
	[1022:1457] 12:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457
IOMMU group 6
	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
	[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 7
	[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
	[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
	[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
	[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
	[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
	[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
	[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
	[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467

 

Link to comment
On 3/11/2017 at 0:51 AM, phbigred said:

Found this in another board for IOMMU for a B350, ASUS appears to possibly have it in a different menu than documented. 

Yeah, found it. ASUS is lying in the manual. It was in Advanced>AMD CBS submenu, and not in CPU Configuration. One has to wonder which side is to blame - the writers of the manual, or the BIOS developers.

 

Thanks, I found my IOMMU setting, but still don't know why it's disabled by default?  It also turns out that "Auto" is the necessary setting.  "Enabled" didn't work for me on the Asus Prime X370-PRO.

 

So now I have a Windows 10 VM running with the GPU pass-through.  Thus far, I also have the USB keyboard, mouse and a sound device being passed through.  The USB sound is kinda crackly, so I may have to replace it with a PCIe card.  Also, I don't seem to be able to find a happy combination of IOMMU groups and the available USB hubs, so I'm looking at getting a USB PCIe card as well.

 

Specs:  AMD 1800X CPU, Asus Prime X370-PRO MB, Crucial CT16G4DFD8213.16FA1 DDR4 2133 (4x16G), Asus Radeon 6450 GPU (desktop, not for gaming)

 

At some point, I'll add a Dell HV52W PERC H310 8-Port controller card, so combined with the 8 SATA ports on the MB, I can still support 16 drives.

 

Hope all those PCIe cards work smoothly together (GPU, SAS/SATA, USB and sound).  Need to dig through the manual to see if there are any gotchas.

Link to comment

I've got mine more or less up and running at the moment with an 1800x, a new RX 460 and a GTX 560 on the Crosshair VI Hero (which I seem to be about the only person not having issues with, albeit with bios 0902).  Seems to be stable on its own, but I've yet to wrangle any GPU into actually initializing with IOMMU and did encounter Naiqus' issue an Ubuntu install bringing down the whole system..  Going to try a manual bios dump on IOMMU and start from scratch on a Linux VM (first attempt was pre bios update, but kernal seems more likely...) tomorrow.  Will report back.

Link to comment
4 hours ago, Bureaucromancer said:

I've got mine more or less up and running at the moment with an 1800x, a new RX 460 and a GTX 560 on the Crosshair VI Hero (which I seem to be about the only person not having issues with, albeit with bios 0902).  Seems to be stable on its own, but I've yet to wrangle any GPU into actually initializing with IOMMU and did encounter Naiqus' issue an Ubuntu install bringing down the whole system..  Going to try a manual bios dump on IOMMU and start from scratch on a Linux VM (first attempt was pre bios update, but kernal seems more likely...) tomorrow.  Will report back.

Please do report back, I am considering this board as my next UNRAID build. I stripped down my Intel 2660v3 build since I was not using its full potential, got myself a 1800x, gskill 2x32GB - just deciding on a board and cooler now.

Link to comment

I've found something odd related to power consumption.

 

I'm running an 1800X on the ASRock X370 Fatal1ty Professional Gaming and a GeForce GTX 670.

 

It is idling at 46w in Windows 10 ("Performance" power profile), but 56w in unRAID (Safe Mode).  No array drives configured or even installed.

 

Any ideas why unRAID would be consuming 10w more, just sitting there idle?

 

I'm also having stability issues in unRAID (again, just sitting there idle with no array), with GUI hangs typically within an hour, followed by telnet and console hangs, and at least one crash/reboot that resulted in machine check exceptions.  Haven't been able to capture any data yet.  I'm running in Safe Mode now to see if it was one of the many plugins I installed to evaluate the motherboard.

 

Windows 10 running stress tests non-stop overnight works like a champ, so it doesn't seem to be a hardware issue.

 

-Paul

Link to comment

Just got another machine check exception crash&reboot.  This happened from Safe Mode, but on reboot it returned to the default plug-in mode. Diagnostics attached.

 

I enabled mcelog, but in the log file I see this:

Mar 15 12:10:08 TESTTower root: mcelog: ERROR: AMD Processor family 23: mcelog does not support this processor. Please use the edac_mce_amd module instead.

 

edac_mce_amd is not in the NerdPack.  I would appreciate some assistance for getting this installed manually, not something I do everyday.

 

-Paul

Ryzen_ASRock_X370_Fatal1ty_Professional_Gaming_testtower-diagnostics-20170315-1210.zip

Link to comment

Yikes, getting EDAC going is hard, I'm lost.  From what I'm reading, EDAC is built into the Kernel, and needs to be configured, but I'm not finding it in our unRAID build.  Did Lime-Tech take it out?

 

There's obviously a lot of interest in Ryzen, and many unRAID users have already jumped onboard, including Lime-Tech themselves.  We all need a solution for this.

 

I did find in the /sys/devices/system/machinecheck folder a whole bunch of machinecheck folders:  Looking inside the folders, these seem to be snapshots updating regularly, and may not represent actual issues, but I honestly don't know.

 

root@TESTTower:/sys/devices/system/machinecheck# ls -l
total 0
drwxr-xr-x 25 root root    0 Mar 15 11:58 machinecheck0/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck1/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck10/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck11/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck12/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck13/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck14/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck15/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck2/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck3/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck4/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck5/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck6/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck7/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck8/
drwxr-xr-x  9 root root    0 Mar 15 11:58 machinecheck9/
drwxr-xr-x  2 root root    0 Mar 15 12:48 power/
-rw-r--r--  1 root root 4096 Mar 15 12:48 uevent

Not sure what to make of these, if anything.

 

-Paul

Link to comment
1 hour ago, Pauven said:

Just got another machine check exception crash&reboot.  This happened from Safe Mode, but on reboot it returned to the default plug-in mode. Diagnostics attached.

 

I enabled mcelog, but in the log file I see this:

Mar 15 12:10:08 TESTTower root: mcelog: ERROR: AMD Processor family 23: mcelog does not support this processor. Please use the edac_mce_amd module instead.

 

edac_mce_amd is not in the NerdPack.  I would appreciate some assistance for getting this installed manually, not something I do everyday.

 

-Paul

Ryzen_ASRock_X370_Fatal1ty_Professional_Gaming_testtower-diagnostics-20170315-1210.zip

You're late on the draw....  

Except that I never followed up and PM'd tom to add the module....

Link to comment
44 minutes ago, Pauven said:

There's obviously a lot of interest in Ryzen, and many unRAID users have already jumped onboard, including Lime-Tech themselves.  We all need a solution for this.

 

I believe Ryzen support is much better on kernel 4.10, so it should improve a lot on next unRAID release (maybe v6.4-rc?)

Link to comment
4 hours ago, Pauven said:

I've found something odd related to power consumption.

 

I'm running an 1800X on the ASRock X370 Fatal1ty Professional Gaming and a GeForce GTX 670.

 

It is idling at 46w in Windows 10 ("Performance" power profile), but 56w in unRAID (Safe Mode).  No array drives configured or even installed.

 

Any ideas why unRAID would be consuming 10w more, just sitting there idle?

 

 

Perhaps it is the GPU? Unlike Windows 10, unRAID doesn't have graphics drivers with power managmentby default.

 

My power consumption drops by ~12W when I start my Win10 VM with Nvidia WHQL drivers and my GTX1060 passed through.

 

Thanks for the numbers by the way. Quite a bit higher than I hoped they would be. I wonder if the full featured mainboard with a 5-Gbit NIC, Soundblaster and other fancy features is party to blame. 

 

As for EDAC I don't think Ryzen supports it. I remember reading that ECC and single error bit correction is silently functional but not validated and unsupported.

Two-bit error correction with reporting and kernel halt is not implemented and won't work with standard desktop Ryzen.

I guess that's where they draw the line for Naples or some other dedicated workstation SKU (Ryzen Pro?) 

Edited by lionceau
Link to comment
1 hour ago, johnnie.black said:

 

I believe Ryzen support is much better on kernel 4.10, so it should improve a lot on next unRAID release (maybe v6.4-rc?)

 

From everything I've read, the Ryzen fixes in 4.10 were backported to 4.9.10.  Since 6.3.2 is already on 4.9.10, I'm anticipating almost no benefit from 4.10.

 

While future kernels will improve Zen optimizations, in general Ryzen is fully supported on 4.9.10.  Kudos to Lime-Tech for getting us there before Ryzen hit the market!

 

I am waiting on 4.11 for driver updates needed for my motherboard.  Both the sound (Realtek ALC1220 Audio Codec) and 5Gb LAN (Aquantia AQC108) are being added to 4.11.

 

On the other hand, the ability to use EDAC MCE logging has pretty much nothing to do with 4.9.10 or 4.10.  It is supposed to be built into the kernel since the 2.6 days.  While EDAC itself may need updating to support Ryzen (and the EDAC code support seems pretty stagnant), that's a separate issue.  I see no reason EDAC shouldn't be runable on unRAID, regardless of whether is supports Ryzen or not, there's a lot of non-Ryzen AMD CPU's in unRAID boxes too.  

 

It almost seems like Lime-Tech pulled EDAC support out somehow.  I think Tom strips a lot of the packages and modules out to make unRAID lightweight.  But this is just a guess on my part, Lime-Tech would have to confirm.

 

I also see a note that mcelog is supposedly being maintained for AMD again, which would be the best possible solution.  However, a version with AMD support isn't available at this time.

 

-Paul

Link to comment
11 minutes ago, lionceau said:

 

Perhaps it is the GPU? Unlike Windows 10, unRAID doesn't have graphics drivers with power managmentby default.

 

My power consumption drops by ~12W when I start my Win10 VM with Nvidia WHQL drivers and my GTX1060 passed through.

 

Thanks for the numbers by the way. Quite a bit higher than I hoped they would be. I wonder if the full featured mainboard with a 5-Gbit NIC, Soundblaster and other fancy features is party to blame. 

 

As for EDAC I don't think Ryzen supports it. I remember reading that ECC and single error bit correction is silently functional but not validated and unsupported.

Two-bit error correction with reporting and kernel halt is not implemented and won't work with standard desktop Ryzen.

I guess that's where they draw the line for Naples or some other dedicated workstation SKU (Ryzen Pro?) 

 

That's a great point about it being the GPU, I didn't think about that.  I've never installed GPU drivers on a Linux box before, guess I need to look into it.  I suppose GPU issues could even be the root cause of the crashes I'm experiencing.  

 

Sorry to further burst your bubble, at 46w in Windows I had already disabled almost everything on the motherboard that I didn't need.  Sound, wi-fi, serial ports, and almost all SATA ports have been disabled.  I wasn't able to disable Bluetooth, though, the BIOS doesn't give me the option, but I might be able to disable the USB 3.1 ports which is what the BT module is connected to internally.  And no matter how hard I tried, 3 SATA ports keep showing up.  Immature BIOS.  47w is also with my dual-140mm CPU fans on the Silent setting, cranking them up to max adds another 3-4 watts.

 

With everything turned on, I was in the low to mid 50w range at idle in Windows.  Oh, and changing the Power Profile from Performance (recommended by AMD) to Balanced saves another watt at idle, and Power Saver maybe another watt on top of that.  On Performance, the CPU never idles below 3.16GHz (that I can see).  Both Balanced and Power Saver drop minimum speeds to about 2.1GHz.

 

Oh, and on EDAC, unfortunately this acronym seems to have more than one application.  There is a module called EDAC, that I believe taps into the CPU's error correction capabilities (also called EDAC) to report on Machine Check Exceptions.  Yes, Ryzen supports ECC, but it is up to the motherboard manufacturer to enable support.

 

-Paul

 

 

Link to comment

Got my hands on a Ryzen just for today and while doing some other bandwidth tests also took some power readings.

 

This is not trying to be as low as possible since I have a bunch of SSDs connected but mostly for comparison to a Kabylake i5, both system have the exact same things connected, only difference is the board and CPU, UnRAID v6.3.2 idle:

 

Asus PRIME B350M-A + Ryzen 1700 + GPU: 62w

Asus PRIME B250M-K + i5-7400 + GPU: 50w

Asus PRIME B250M-K + i5-7400 (iGPU): 38w

 

GPU used AMD 3450

Edited by johnnie.black
Link to comment

This article on PC Perspective reports an idle wattage of 37.6w, on an ASUS Crosshair VI Hero motherboard and with a GTX 1080 GPU.

 

https://www.pcper.com/reviews/Processors/AMD-Ryzen-7-1800X-Review-Now-and-Zen/Power-Consumption-and-Conclusions

 

That seems pretty good, for a true 8-core CPU, and they show it as besting all of the competition.

 

My GTX 670 supposedly consumes about 6-7w more than a GTX 1080, so swapping GPU's would bring me down to about 40w.  Also, I'm running 4 sticks of dual-rank 16GB DIMMS (64GB total), and in my testing I found that each DIMM added a watt or two.  

 

With the right components, I think it would be realistic to see mid to high 30's on idle wattage.  I think that's best case.

 

My Celeron G1610 idles at 17-18 watts, which is real nice, but it would take probably 6-8 of those systems combined to come close to the total performance of the 1800X, so from that perspective the Ryzen is very energy efficient.

 

-Paul

Link to comment
9 minutes ago, Pauven said:

This article on PC Perspective reports an idle wattage of 37.6w, on an ASUS Crosshair VI Hero motherboard and with a GTX 1080 GPU.

 

That looks much better, I'd be happy with a similar idle consumption to Kabylake, maybe Linux is not yet using all the power saving features, I didn't check if the CPU was throttling down.

Link to comment
2 minutes ago, johnnie.black said:

 

That looks much better, I'd be happy with a similar idle consumption to Kabylake, maybe Linux is not yet using all the power saving features, I didn't check if the CPU was throttling down.

 

Yeah, 62w is a bit on the high side.  

 

The GTX 1080 idles at about 7w

 

My GTX 670 idles at about 14w.

 

Your Radeon HD 3450 idles at about 17w.

 

Factor in that my system idles 10w higher in unRAID vs. Win10, if (and that's a big IF) that is caused by Ryzen on Linux, then your 62w reading could be about 20w lower with a modern GPU and some OS optimizations.

 

Compare that back to the i5-7400 running 38w with a weak iGPU, and all of a sudden 42w for a 1800x + GTX 1080 seems super efficient.

 

-Paul

Link to comment

So I'm still digging into the idle power consumption, and my findings don't make sense.

 

Using the "Tips and Tweaks" plugin, I'm toggling the CPU Scaling Governor between Performance and Power Save.

 

Using the command "lscpu | grep MHz", on Performance I get:

root@TESTTower:/boot# lscpu | grep MHz
CPU MHz:               3600.000
CPU max MHz:           3600.0000
CPU min MHz:           2200.0000

with the processor never going below 3.6GHz.  That's right, it is idling at 3.6GHz.

 

 

On Power Save I get:

root@TESTTower:/boot# lscpu | grep MHz
CPU MHz:               2200.000
CPU max MHz:           3600.0000
CPU min MHz:           2200.0000

with the processor idling nicely at 2.2GHz.  So it appears the CPU Scaling Governor settings are working.

 

But on my Kill-a-Watt power meter, the consumption stays at 56w for BOTH settings.

 

How in the world can the CPU downclock from 3.6GHz to 2.2GHz, and not save a single watt?

 

I tried various methods to check CPU frequency (like /proc/cpuinfo, lshw, /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq) and they report the same as above.

 

I also tried dmidecode, and it always reports 3.6GHz even on Power Save.  Perhaps this is not accurate?

root@TESTTower:/boot# sudo dmidecode -t processor | grep MHz
        External Clock: 100 MHz
        Max Speed: 3600 MHz
        Current Speed: 3600 MHz

The same with the dmesg log:

root@TESTTower:/boot# cat /var/log/dmesg | grep "MHz processor"
[    0.000000] tsc: Detected 3600.316 MHz processor

 

My gut is telling me that the 10w power increase I'm seeing on unRAID vs. Win10 is related to this.  Perhaps the voltage scaling is not working, even though frequency scaling is?

 

-Paul

Link to comment
1 hour ago, Pauven said:

the consumption stays at 56w for BOTH settings.

 

I see 1w difference between those settings, still seems very little, freq on the 1700 is 1550Mhz for power save and 3000GHz for performance, a few more readings (+/- 1w):

 

# of cores/threads @ 100% utilization - power consumption:

0 (idle) - 62w

1 - 75w

2 - 83w

4 - 88w

6 - 102w

8 - 114w

16 - 115w

Link to comment

Just a quick update from me.  Looked through the Lime Tech docs and for the first time realized that they've only calidated back the GTX 600s, on trying my 970 passthrough in the second slot is suddenly working with the ACS override but no need to choose a manual BIOS.  I dont have it fully running, but things are looking better on the linux front as well, no systemwide crashes on a fresh attempted install, although the installer itself died right at the end (think it might have been a fluke tbh).

 

Next project will be seeing if I can manage to get both GPUs passed through simultaneously. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.