Anybody planning a Ryzen build?


Recommended Posts

Thought I'd chime in with some feedback here on our Ryzen test build that we put together last night and are continuing to test with today.  First and foremost:  YES!  IT WORKS!  Booting up unRAID OS, assigning some storage, starting up a VM, passing through a GPU...all working as intended!  For those of you who were biting your nails on this, you can give them a rest.

 

Now our test rig is based on this mobo / CPU combo:

 

M/B: ASRock - X370 Killer SLI/ac

CPU: AMD Ryzen 7 1700X Eight-Core @ 3400

 

This motherboard does NOT have on-board video. This means that the GPU installed in the top PCI slot will be the "primary display adapter".  For those wondering the importance of that, NVIDIA GPUs have very limited support for pass through when installed as the "primary display adapter" on a system that does not have an embedded graphics adapter.  It will not work by default, and only some have success manually passing through a ROM file.  For AMD-based GPUs, however, this doesn't seem to be a problem, though AMD-based GPUs can be problematic in other ways.  If you have the option, always buy a motherboard with an embedded graphics adapter to avoid this problem.

 

The second thing worth mentioning is the way the IOMMU groups are laid out.  Here is my lspci info from the test build:

 

IOMMU group 0
    [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
    [1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
    [1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
    [1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
    [1022:43b4] 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [8086:24fb] 05:00.0 Network controller: Intel Corporation Device 24fb (rev 10)
    [8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 1
    [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 2
    [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
    [1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
    [10de:1401] 0c:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
    [10de:0fba] 0c:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)
    [10de:11c6] 0d:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] (rev a1)
    [10de:0e0b] 0d:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
IOMMU group 3
    [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 4
    [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
    [1022:145a] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
    [1022:1456] 11:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
    [1022:145c] 11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
IOMMU group 5
    [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
    [1022:1455] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
    [1022:7901] 12:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
    [1022:1457] 12:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457
IOMMU group 6
    [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
    [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 7
    [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
    [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
    [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
    [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
    [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
    [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
    [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
    [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467

This system has two GPUs installed in it right now (a GTX 960 and a GTX 650 Ti).  The remainder of the devices listed are all built-into the motherboard chipset.  The first thing to note is that the PCIe ACS Override does not work here.  I turned it on yet devices remain in their respective groups.  The second thing to note is that both GPUs are in the same IOMMU group.  That means that this particular combination of hardware will not work for a dual GPU pass through setup (e.g. two gaming VMs).  The third thing to note is that the on-board devices (USB, audio, network, and storage controllers) are also in groups together, except for one of the USB controllers which may be ok to pass through (group 4).

 

So is this a good system for unRAID?  Knowing what I know now, I would probably think a 6-core (12 thread) variant would be a better choice for this motherboard.  When both your full-length PCI slots for the motherboard share the same IOMMU group, you really only have one GPU pass through VM use-case, and that's a localized virtual desktop.  I would probably try to find a decent AMD GPU (or wait for Vega and test that) for the top slot and try to snag a dedicated USB controller add-on card for the second slot to pass both through to my VM.  Alternatively you could just do USB assignment and go with a cheapo card in the top slot and a GTX of your choice in the second.  Either way, I would allocate 4 of the 6 cores to the VM (so 8 threads) and leave the 2 cores (4 threads) remaining for the host.  I would CPU pin my Docker VMs and use Isol-cpus so that my VM didn't share physical CPU resources with the host.  The motherboard features a 6-port sata controller, but also features two M.2 slots.  The highest-end configuration i could think of would involve two M.2 SSDs configured in a cache pool and then use the 6 SATA ports for HDDs to build an array.  Alternatively you could skip the M.2's and go with dual SATA SSDs for the cache pool and 4 HDDs for the array.  This would be an ideal setup for a desktop gaming PC + NAS combo system.

 

For those with other motherboards out there (especially those with embedded graphics), please let us know your experiences as well!

Link to comment

So there is no way to separate the 2 full length PCI express slot that are meant to run 2 vid cards in order to pass them through? :(  Is it possible that the IOMMU groupings are setup this way due to motherboards shipping with very early bios revisions?  Is this something that could be fixed with later bios revisions or fixed with another workaround?

Edited by azteca25
Link to comment
59 minutes ago, Pauven said:

Let's see...

 

Ryzen 1700X = 16 threads x 3.9GHz (if OC) = 62.4GHz total processing capacity

E5-2683 v3 = 28 threads x 2.5GHz (max all-core boost) = 70.0GHz total processing capacity

 

The Ryzen performance is indeed impressive, but a few "nits" r.e. the above ...

 

(a)  I wouldn't multiply clock by threads ... the actual processing power is in the cores.   Hyperthreading allows very efficient use of the cores, since all of the registers are replicated; but it doesn't actually add additional processing units.   The outcome is the same (with smaller numbers) ... i.e. Ryzen still looks very good relative to the far-more expensive E5.   But processing power [if you measure it in GHz (see next comment)] would more accurately be 8 x 3.9  and  14 x 2.5

 

(b)  GHz isn't necessarily a good measure these days ... the CPU designs include a lot of parallelism, and many times a CPU with a low clock rate can notably outperform one with a much higher clock rate.   A far better measure is to look at the PassMark scores -- in this case 17627 for the E5, 14492 for the Ryzen.   The Ryzen is still very impressive -- that's a superb score for a 8-core CPU.   Almost as good as the 8-core i7-6900k (17732) for less than half the price.   I suspect the i7 would be notably faster due to the 2011-3 bus and quad-channel DDR4, but the Ryzen is still very impressive for the cost.

 

... I have to wonder what Intel's going to do this fall :D

(Cutting the price of a 6900k in half would be a good start -- doing the same with a few E5's would be even better >:(.   An 8-core E5-2687W v4 (PassMark 20203) for about 1/3rd of it's current price ($2100) would be a VERY attractive processor.

 

I AM tempted by the Ryzen -- the performance of the 1800x is very impressive both for the price and the relatively low TDP.   I'll wait for the next Intel refresh before committing to it, however, as I wouldn't be surprised to see some movement there as well.

Link to comment
3 hours ago, jonp said:

Thought I'd chime in with some feedback here on our Ryzen test build that we put together last night and are continuing to test with today.  First and foremost:  YES!  IT WORKS! 

 

Thanks so much for sharing your detailed findings!  Very encouraging that Lime-Tech is already working with Ryzen in-house!

 

If you don't mind, a few clarifications:

  • While some Ryzen motherboards include video out, they require CPU support to work.  No Ryzen CPU's at this time have this capability.  And as far as I know, no AM4 motherboards with video out have a graphics adapter built into the motherboard.  Thus, for now, it is impossible to use integrated graphics with Ryzen.  This may change with Naples (server version of Zen), but that is not Ryzen or AM4.
  • The reason some motherboards include video out is that the AM4 socket also works with the latest Bristol Ridge APU's (legacy Bulldozer architecture, not Zen), and in the second half of 2017 AMD will release Zen/Ryzen APU's (most likely 4core/8thread + Vega graphics).
  • It's probably not a great idea to put the cheapo/low end GPU in the primary x16 slot, as it is the ONLY slot wired for PCIe 3.0 x 16.  The second x16 slot (on all X370 motherboards) will be wired for PCIe 3.0 x 8, and the third x16 slot will most likely be wired for PCIe 3.0 x 4 PCIe 2.0 x 4.  You can visually confirm this by looking inside the slots (you see the missing contact wires in the second and third slots) or even easier, flip the motherboard over and look at the bottom, where it is very easy to see the missing solder joints on the other two slots.
  • A workaround might be to put a cheap GPU in the small x1 slot, that is typically PCIe slot 1 (the primary x16 slot is typically slot 2).  Not sure if that would be able to be set as the primary graphics adapter, though that is probably a BIOS limitation/feature and nothing to do with Ryzen itself.  There are other PCIE x1 slots on these boards that may also be good for installing a cheap GPU, but it will still be a BIOS limitation to flag it as a primary.  My advice is to work with the vendor on BIOS updates to enable this functionality.  I've done this in the past, and while it may take a few months, I got what I needed. 
  • The IOMMU groups will vary from motherboard to motherboard, not just chipset to chipset.  For example, the Asus ROG Crosshair VI Hero has 4 USB ports from the CPU, 13 USB ports from the X370 chipset, and an additional 2 USB ports from an ASMedia USB 3.1 controller.  I would hope that these would show up in different IOMMU groups, and may eliminate the need for an extra PCIe USB adapter (which is a great idea, by the way, for motherboards that are not as capable as the Crosshair).
  • It sounds like Alex Williamson's ACS override patch needs to be updated to support Ryzen and the new AMD chipsets.  A quick google search didn't turn up any activity on this front.  Hopefully this is not a hardware limitation, and merely just another example of a completely new architecture that requires the software ecosystem to be updated around it.
  • Though you reasoning is sound, it is probably too early to conclude that this motherboard is only suitable for a single gaming PC.  This brand new architecture has only been on the market a week, the motherboards are very rushed to market and the BIOS's are very immature, and the supporting software needs time to be updated for Ryzen.
  • hblockx got dual gaming pc's working on the Gigabyte X370 Gaming 5 (his post is earlier in this thread).  From a physical hardware standpoint, there shouldn't be anything special about that Gigabyte board that makes it superior to the ASRock - X370 Killer SLI/ac.  Primarily, they both have a similar physical PCIe arrangement. This is either a BIOS limitation (fixable) or hblockx found a workaround.  That said, there is one physical difference, the Gigabyte X370 Gaming 5 has a third x16 slot, which may be in a different IOMMU group.  Perhaps hblockx used this third slot.  The problem here is that the third slot is only wired for x4, so it is bandwidth constrained.

 

-Paul

Edited by Pauven
typos, and corrected third x16 slot speed to PCIe 2.0 x4
Link to comment
17 minutes ago, Pauven said:

The problem here is that the third slot is only wired for x4, so it is bandwidth constrained.

 

True ... HOWEVER => since it's a PCIe v3 system, that "constraint" is at almost 4GB/s ... i.e. essentially the same as an x16 slot on a PCIe v1 board (3.938GB/s vs. 4GB/s).   Yes, it's a constraint, but I'd think the performance would still be just fine :D

Link to comment
7 hours ago, hblockx said:

Actually everything except boost / xfr works fine atm (just set cpu-clock to 3,8ghz). My system is working fine with lots of Dockers and VMs (also with passthroughed graphic cards and so on). I just miss an option for a second passthroughed USB-Card (cause of unlucky IOMMU). Anyways awesome for the money :D

 

What motherboard are you using?

Link to comment

I thought it would be helpful to do a separate post on Ryzen PCIe lanes, since it is a bit confusing right now.  Here's a crash course.

 

DISCLAIMER:  I may be wrong on a detail or two, but if so I blame the tech sites, many of which have posted bad info.  Also keep in mind that with ~80 different motherboards on the market, in different configurations, there may be some exceptions to the info I post below.

 

Ryzen has 24 PCIe 3.0 lanes.  Total.  That's it.

 

On X370, B350 and A320 chipset motherboards, 4 of those lanes are dedicated to the chipset.  That leaves 20 PCIe 3.0 lanes for add-in cards.

 

The X300/B300 small form-factor chipsets are unique in that they hardly do anything, and supposedly don't use PCIe lanes to communicate with the Ryzen CPU, instead they use a dedicated SPI link.  This should leave all 24 PCIe 3.0 lanes available directly from the CPU.  Sounds interesting, though IOMMU groups may be bad, and all USB ports will be going directly to the CPU, so again IOMMU groups may be bad.  Plus, since the X300/B300 series chipsets are intended for small form factor only (ITX), dual or triple x8 slots are not likely to be available with X300, even though it certainly seems possible.  No X300/B300 boards have been released at this time - the expectation is for the second half of 2017.

 

Back to those 20 remaining PCIe 3.0 lanes:

  • 16 of those 20 lanes will go to the primary x16 slot, which will be wired for x16
  • The secondary x16 slot will be wired for x8, and will "steal" those 8 lanes from the primary slot.
  • That means, with dual GPU's in the primary and secondary slots, both operate at PCIe 3.0 x8.
  • This also means that, since those 16 lanes are being split, typically both of those slots will be in the same IOMMU group - unless the BIOS can somehow fix this

 

The last 4 of the 20 lanes:

  • On most motherboards, the remaing PCIe 3.0 x4 lanes will be dedicated to the M.2 slot.  Done and done. 
  • But that isn't a hard and fast rule.
  • Sorry, I don't have any examples of other uses of the remaining PCIe 3.0 x4 lanes, but here's some ideas:
  • It could go to a third x16 slot, running at PCIe 3.0 x4 - but this is not likely.
  • If there is a third x16 slot, most likely it will be running at PCIe 2.0 x4, not x16 or x8, a strong indication that this is wired to the X370 chipset, and is NOT using the 4 remaining PCIe 3.0 lanes.
  • If there are additional x1 slots, these will often be shared with the third x16 @ PCIe 2.0 x4 slot and go through the X370 chipset as well.
  • In that case, if you plug anything into an x1 slot, the third x16 slot will downgrade from x4 to x1 as well.

 

The X370 chipset offers an additional 8 lanes of PCIe 2.0.  These get routed through the X370 back to the Ryzen CPU over 4 lanes of PCIe 3.0 (along with all the other communications that are probably going through the X370, like audio and USB).  This likely means that these 8 lanes of PCIe 2.0 perform below what would be expected from dedicated PCIe 2.0 x8.  In some cases, these 8 lanes are being used to create the third x16 slot, which would most likely be wired as PCIe 2.0 x4.

 

The B350 chipset offers an additional 6 lanes of PCIe 2.0, instead of the 8 that X370 offers.  The A320 (which for the most part none of use should even consider) only offers 4 PCIe 2.0 lanes.  X300/B300 chipsets offer 0 additional lanes, since they connect via SPI link and not PCIe.

 

On some motherboards, like the ASRock Fatal1ty X370 Professional Gaming, using the 2nd M.2 slot will completely disable the third PCIe x16 @x4 slot. You may see similar compromises on other motherboards, where using lanes in one place prevents you from using them in another.

 

Ryzen could have really used another 8 lanes, and this shortcoming is forcing motherboard manufacturers to make compromises.

 

The only potential solution, in the near-term, is for manufacturers to add a PLX chip to their high-end motherboards.  If you don't know what a PLX chip is, think of it like a 4-port USB hub that allows a single USB connection to become 4.  A PLX chip could easily allow 3 or 4 fully wired x16 slots, possibly each in their own IOMMU group, though their traffic will be shared with each other like a carpool using the 16 lanes going back to the CPU.  A PLX chip will probably add about $100 to the cost of a motherboard.  The most expensive motherboard (that I've seen) is the MSI X370 XPower Gaming Titanium, at about $300, and it does not appear to have a PLX chip.  So at this time, it appears no AM4 motherboards are implementing PLX, though that may change in the future.  I'm really, really, really surprised no manufacturer has done this yet.

 

 

Hopefully this information provides useful, and doesn't scare anyone away from Ryzen.  Ryzen is great, but it is smart to know about the nitty gritty details like this before you commit.

 

-Paul

 

  • Upvote 1
Link to comment
1 hour ago, garycase said:

 

True ... HOWEVER => since it's a PCIe v3 system, that "constraint" is at almost 4GB/s ... i.e. essentially the same as an x16 slot on a PCIe v1 board (3.938GB/s vs. 4GB/s).   Yes, it's a constraint, but I'd think the performance would still be just fine :D

 

That's probably not quite correct.  AND APOLOGIES, I mistakenly wrote that the third PCIe x16 slot would be wired for PCI 3.0 x4, so you got your misinformation directly from me!  O.o  I corrected that post now.

 

The third PCIe slot is most likely routed through the X370 chipset, so it is PCIe 2.0 x4, not PCIe 3.0 x4.  So cut that value in half, 2GB/s.  That's PCIe 1.0 x8 speeds.

 

Plus, that slot may share its lanes with any x1 slots, and if you plug anything into one of them, it will downgrade from PCIe 2.0 x4 to PCIe 2.0 x1.  So reduce the bandwidth again to 0.5GBs.  This comes into play if you're thinking of adding a PCIe x1 USB card to get USB ports into a separate IOMMU group.

 

Worse, on my motherboard that PCIe 2.0 x16 @ x4 slot is shared with the 2nd M.2 port, which also runs at PCIe 2.0 x4.  You can only run one or the other.

 

-Paul

Edited by Pauven
Apology...
Link to comment
3 hours ago, garycase said:

 

The Ryzen performance is indeed impressive, but a few "nits" r.e. the above ...

 

(a)  I wouldn't multiply clock by threads ... the actual processing power is in the cores.   Hyperthreading allows very efficient use of the cores, since all of the registers are replicated; but it doesn't actually add additional processing units.   The outcome is the same (with smaller numbers) ... i.e. Ryzen still looks very good relative to the far-more expensive E5.   But processing power [if you measure it in GHz (see next comment)] would more accurately be 8 x 3.9  and  14 x 2.5

 

(b)  GHz isn't necessarily a good measure these days ... the CPU designs include a lot of parallelism, and many times a CPU with a low clock rate can notably outperform one with a much higher clock rate.   A far better measure is to look at the PassMark scores -- in this case 17627 for the E5, 14492 for the Ryzen.   The Ryzen is still very impressive -- that's a superb score for a 8-core CPU.   Almost as good as the 8-core i7-6900k (17732) for less than half the price.   I suspect the i7 would be notably faster due to the 2011-3 bus and quad-channel DDR4, but the Ryzen is still very impressive for the cost.

 

... I have to wonder what Intel's going to do this fall :D

(Cutting the price of a 6900k in half would be a good start -- doing the same with a few E5's would be even better >:(.   An 8-core E5-2687W v4 (PassMark 20203) for about 1/3rd of it's current price ($2100) would be a VERY attractive processor.

 

I AM tempted by the Ryzen -- the performance of the 1800x is very impressive both for the price and the relatively low TDP.   I'll wait for the next Intel refresh before committing to it, however, as I wouldn't be surprised to see some movement there as well.

 

You're right on all accounts.  Thanks for keeping me honest.  :$

 

Though as a counterpoint, the GHz estimation I did suggested that the 1700X (overclocked to 3.9GHz) would be about 89% of the E5.  While the PassMark scores indicate that the 1700X (at stock speeds) would be about 82%.  Factoring in the overclock on the 1700X, I would estimate a PassMark score ~10% higher,  ~15940, which is ~90% of the E5.  Crazy enough, my GHz estimation method appears to be really close, at least to PassMark.  This is only because Ryzen IPC is really close to Intel now.  You're still right, I'm just a little less wrong...

 

Need someone like planetwilson to test PassMark on his 1700X @ 3.9GHz and see if it scores about 15940 (and maybe higher with the DDR4 running at 3200!).

 

On a related note, it appears that the Windows 10 Scheduler has multiple bugs in how it interacts with the new Ryzen CPU.  It thinks all threads are real, full performance cores, and doesn't recognize that the virtual SMT cores are lower performance.  It also incorrectly identifies the amount of cache.  Applied to gaming (for which most games currently don't scale beyond 3 or 4 threads, and can even slow down with extra threads) this means that Windows 10 is significantly hurting performance on workloads that don't fully utilize all threads.  This explains why many games see a big performance boost (up to 17% reported) from disabling SMT on Ryzen.  Gaming benchmark scores will improve in the future, it will just take time for these issues to be worked out. B|

 

http://wccftech.com/amd-ryzen-performance-negatively-affected-windows-10-scheduler-bug/

 

I wish Microsoft would go on the record as to when these various issues will be addressed.

 

This also makes me wonder if there are still similar issues in Linux?  I know 4.10 brings Ryzen support, but that doesn't mean everything is perfect.  My gut tells me odds are, improvements are to be had.

 

 

Edited by Pauven
Link to comment
1 minute ago, garycase said:

I suspect that simply disabling hyperthreading in the BIOS would likely improve gaming performance, since there wouldn't be any "fake" cores to assign :D

Not sure if that's also true with Intel CPU's ... I'm not a gamer so don't really have anything to test that with.

 

 

It's also true for Intel, but to a much lesser extent, and not for as many games.  The Windows Scheduler knows how to avoid the fake cores on Intel CPU's.  And developers have been optimizing for Intel for the past decade, somewhat ignoring AMD.

Link to comment

For those wanting to use ECC RAM, support seems to vary by motherboard, and if I'm interpreting it correctly, the Gigabyte GA-AX370 Gaming K7 may limit ECC to 32GB, while non-ECC maxes out at 64GB

 

CORRECTION: 16GB modules come as 2Rx8, so you can install up to 64GB of ECC memory.  Should have checked before I posted this warning.  Odlly enough, on Gigabyte's "Supported Memory List", they don't list any 1Rx16 memory sticks at all, so not sure what the big deal is.  Maybe a future memory arrangement?  For those wondering, 1Rx8 means single rank, and 2Rx8 means dual rank - I think this is the modern terminology for single-sided and double-sided.  The reason to care is that 2R memory puts a higher load on Ryzens memory controller, and requires lower memory speeds.  You can get 8GB modules at 1Rx8, but 16GB modules only come as 2Rx8.  In theory, you could also have a 16GB module as 1Rx16, but these may not even exist yet.

 

From the specs:

  1. 4 x DDR4 DIMM sockets supporting up to 64 GB of system memory
  2. Dual channel memory architecture
  3. Support for DDR4 3600(O.C.) / 3400(O.C.) / 3200(O.C.) / 2933(O.C.) / 2667* / 2400 / 2133 MHz memory modules
    * Support for higher than DDR4 2667 MHz may vary by CPU.
  4. Support for ECC Un-buffered DIMM 1Rx8/2Rx8 memory modules
  5. Support for non-ECC Un-buffered DIMM 1Rx8/2Rx8/1Rx16 memory modules

      (Please refer "Memory Support List" for more information.)

 

Hopefully I'm wrong in my interpretation, but just wanted to put a warning out there for buyers to do their homework before plunking down cash.  Anyone that understand the terminology better than I, please correct me.

 

-Paul

Edited by Pauven
Correction
Link to comment

Has anybody measured idle power consumption at the wall with Ryzen?

I read that typical Ryzen systems idle at 60W (without harddisks or with HDDs spun down). My current Haswell only sips 30W with a 1060-6GB. Perhaps this has to do with the fact that most benchmarks were done using Windows and AMD strongly recommends "High Performance" power saving setting? In any case it looks promising, my ideal setup would be an R5 1600X with 3200 Mhz ECC RAM but neither product exists at the moment. :)

Link to comment
48 minutes ago, lionceau said:

Has anybody measured idle power consumption at the wall with Ryzen?

I read that typical Ryzen systems idle at 60W (without harddisks or with HDDs spun down). My current Haswell only sips 30W with a 1060-6GB. Perhaps this has to do with the fact that most benchmarks were done using Windows and AMD strongly recommends "High Performance" power saving setting? In any case it looks promising, my ideal setup would be an R5 1600X with 3200 Mhz ECC RAM but neither product exists at the moment. :)

 

I am idling with around 70 Watts, but:

Watercooled (pump is around 10 watts) , overclocked to 3.9, ram set to 3,5V, several harddrives, 980ti, 1050ti, 6 fans and just bronze psu (@wall)

So i guess 60 watt is a bit too much without that all?! I would say 40W are achievable...

 

Link to comment
1 hour ago, hblockx said:

 

I am idling with around 70 Watts, but:

Watercooled (pump is around 10 watts) , overclocked to 3.9, ram set to 3,5V, several harddrives, 980ti, 1050ti, 6 fans and just bronze psu (@wall)

So i guess 60 watt is a bit too much without that all?! I would say 40W are achievable...

 

 

Yes, thanks for the data point. I'm very curious what a "bare" system would idle at.

10W for the pump, 7W per idle graphics card, 1W per fan would bring it down to 40W OC before any minor "tweaks" like deactivating the soundcard or any unused controllers, etc.

That's not as bad as I thought it would be.

Link to comment
On 3/8/2017 at 1:13 PM, jonp said:

Thought I'd chime in with some feedback here on our Ryzen test build that we put together last night and are continuing to test with today.  First and foremost:  YES!  IT WORKS!  Booting up unRAID OS, assigning some storage, starting up a VM, passing through a GPU...all working as intended!  For those of you who were biting your nails on this, you can give them a rest.

 

Now our test rig is based on this mobo / CPU combo:

 

M/B: ASRock - X370 Killer SLI/ac

CPU: AMD Ryzen 7 1700X Eight-Core @ 3400

 

This motherboard does NOT have on-board video. This means that the GPU installed in the top PCI slot will be the "primary display adapter".  For those wondering the importance of that, NVIDIA GPUs have very limited support for pass through when installed as the "primary display adapter" on a system that does not have an embedded graphics adapter.  It will not work by default, and only some have success manually passing through a ROM file.  For AMD-based GPUs, however, this doesn't seem to be a problem, though AMD-based GPUs can be problematic in other ways.  If you have the option, always buy a motherboard with an embedded graphics adapter to avoid this problem.

 

The second thing worth mentioning is the way the IOMMU groups are laid out.  Here is my lspci info from the test build:

 


IOMMU group 0
    [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
    [1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
    [1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
    [1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
    [1022:43b4] 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [1022:43b4] 04:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
    [8086:24fb] 05:00.0 Network controller: Intel Corporation Device 24fb (rev 10)
    [8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 1
    [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 2
    [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
    [1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
    [10de:1401] 0c:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
    [10de:0fba] 0c:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)
    [10de:11c6] 0d:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] (rev a1)
    [10de:0e0b] 0d:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
IOMMU group 3
    [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 4
    [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
    [1022:145a] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
    [1022:1456] 11:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
    [1022:145c] 11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
IOMMU group 5
    [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
    [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
    [1022:1455] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
    [1022:7901] 12:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
    [1022:1457] 12:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457
IOMMU group 6
    [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
    [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 7
    [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
    [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
    [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
    [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
    [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
    [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
    [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
    [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467

This system has two GPUs installed in it right now (a GTX 960 and a GTX 650 Ti).  The remainder of the devices listed are all built-into the motherboard chipset.  The first thing to note is that the PCIe ACS Override does not work here.  I turned it on yet devices remain in their respective groups.  The second thing to note is that both GPUs are in the same IOMMU group.  That means that this particular combination of hardware will not work for a dual GPU pass through setup (e.g. two gaming VMs).  The third thing to note is that the on-board devices (USB, audio, network, and storage controllers) are also in groups together, except for one of the USB controllers which may be ok to pass through (group 4).

 

So is this a good system for unRAID?  Knowing what I know now, I would probably think a 6-core (12 thread) variant would be a better choice for this motherboard.  When both your full-length PCI slots for the motherboard share the same IOMMU group, you really only have one GPU pass through VM use-case, and that's a localized virtual desktop.  I would probably try to find a decent AMD GPU (or wait for Vega and test that) for the top slot and try to snag a dedicated USB controller add-on card for the second slot to pass both through to my VM.  Alternatively you could just do USB assignment and go with a cheapo card in the top slot and a GTX of your choice in the second.  Either way, I would allocate 4 of the 6 cores to the VM (so 8 threads) and leave the 2 cores (4 threads) remaining for the host.  I would CPU pin my Docker VMs and use Isol-cpus so that my VM didn't share physical CPU resources with the host.  The motherboard features a 6-port sata controller, but also features two M.2 slots.  The highest-end configuration i could think of would involve two M.2 SSDs configured in a cache pool and then use the 6 SATA ports for HDDs to build an array.  Alternatively you could skip the M.2's and go with dual SATA SSDs for the cache pool and 4 HDDs for the array.  This would be an ideal setup for a desktop gaming PC + NAS combo system.

 

For those with other motherboards out there (especially those with embedded graphics), please let us know your experiences as well!

 

Hi Jon,

 

Were you able to pass-through the wifi-bluetooth m.2 adapter?  I am thinking of implementing a desktop gaming PC + NAS combo as you suggested at the end of your article.  This would allow me to use an xbox one controller via bluetooth.

 

Thank you,

Chad

Link to comment
21 minutes ago, chadjj said:

 

Hi Jon,

 

Were you able to pass-through the wifi-bluetooth m.2 adapter?  I am thinking of implementing a desktop gaming PC + NAS combo as you suggested at the end of your article.  This would allow me to use an xbox one controller via bluetooth.

 

Thank you,

Chad

 

That wouldn't be possible on this system because the wifi adapter is in the same IOMMU group as the normal network adapter, a USB controller, and the storage controller.

Link to comment
5 minutes ago, jonp said:

 

That wouldn't be possible on this system because the wifi adapter is in the same IOMMU group as the normal network adapter, a USB controller, and the storage controller.

 

Thanks Jon, I wasn't sure if "[8086:24fb] 05:00.0 Network controller: Intel Corporation Device 24fb (rev 10)" was the Intel wifi adapter.

Link to comment

Greetings all, I'm late to the party.  Glad to hear the Ryzen tests are looking good.  I actually picked up my Ryzen build on launch day, and then started running things like MemTest or Prime95.

 

My rig:

  • AMD Ryzen 1800X CPU
  • ASUS Prime X370-PRO MB
  • 64GB RAM (Crucial 16GB Single DDR4 2133 x 4)
  • Asus Radeon 6450 1GB Graphics Card (for VM support, not gaming)

Right now I have unRAID Pro 6.3.2 running with some older 4TB drives (dual parity, two data drives) and a 128GB SSD cache drive.

 

I have configured a Windows 7 and Windows 10 VM, but not a lot else.

 

So far, HVM is Enabled but IOMMU is Disabled.  In the BIOS, I had to enable SVN (?) to get HVM working, by default is is disabled for some reason.  I didn't see anything else obvious to enable IOMMU.  The BIOS still feels pretty early, so I don't know if there are problems.  The two Windows VMs seem pretty happy.  Current BIOS for this mobo is 0504.

 

Here are the IOMMU groupings from lspci:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1450
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
25:00.0 USB controller: ASMedia Technology Inc. Device 1343
26:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
28:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450]
28:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
29:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
29:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
2a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

Will keep tabs for updates on the unRAID software, if that helps iron out any odd behavior.  Thanks for all the hard work, LimeTech!

Link to comment
1 hour ago, ufopinball said:

Greetings all, I'm late to the party.  Glad to hear the Ryzen tests are looking good.  I actually picked up my Ryzen build on launch day, and then started running things like MemTest or Prime95.

 

My rig:

  • AMD Ryzen 1800X CPU
  • ASUS Prime X370-PRO MB
  • 64GB RAM (Crucial 16GB Single DDR4 2133 x 4)
  • Asus Radeon 6450 1GB Graphics Card (for VM support, not gaming)

Right now I have unRAID Pro 6.3.2 running with some older 4TB drives (dual parity, two data drives) and a 128GB SSD cache drive.

 

I have configured a Windows 7 and Windows 10 VM, but not a lot else.

 

So far, HVM is Enabled but IOMMU is Disabled.  In the BIOS, I had to enable SVN (?) to get HVM working, by default is is disabled for some reason.  I didn't see anything else obvious to enable IOMMU.  The BIOS still feels pretty early, so I don't know if there are problems.  The two Windows VMs seem pretty happy.  Current BIOS for this mobo is 0504.

 

Here are the IOMMU groupings from lspci:


00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1450
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1d:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
25:00.0 USB controller: ASMedia Technology Inc. Device 1343
26:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
28:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450]
28:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
29:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
29:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
2a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

Will keep tabs for updates on the unRAID software, if that helps iron out any odd behavior.  Thanks for all the hard work, LimeTech!

Found this in another board for IOMMU for a B350, ASUS appears to possibly have it in a different menu than documented. 

Yeah, found it. ASUS is lying in the manual. It was in Advanced>AMD CBS submenu, and not in CPU Configuration. One has to wonder which side is to blame - the writers of the manual, or the BIOS developers.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.