Pauven

Just a quick followup on this (predicting your next rebuttal): The motherboard vendor's QVL list includes overclocked speeds. They are simply communicating that they have successfully tested the memory at that speed, and provided the necessary BIOS support to enable it. But in the eyes of AMD, if it exceeds the limits I listed above, it is overclocked. Paul

[Edited to address John_M's complaint that I was making unfounded assumptions on his setup. Bolded blue sentences below have been edited to clarify my original helpful intent. I never mean to insinuate John_M was in fact overclocking, I had no way to know for sure and thought I had said as much later in my post, so that was a lazy choice of wording on my part.] They are not suggesting that you're overclocking the memory. They are suggesting that you're overclocking the built-in memory controller in your Ryzen CPU. Depending upon your actual memory configuration, you might be overclocking it by quite a bit, or not at all. For example, on my Ryzen 7 1800X, I'm running my memory at 2400, which sounds low but it is definitely overclocked. The RAM itself is rated for 2400, so the RAM isn't overclocked. But my memory, 4x 16GB sticks, is Dual Rank, so I'm presenting the highest possible memory load on the 1800X. Here's AMD's non-overclocked supported memory speeds for Ryzen 1st generation: 2666 MHz for 2 Single Rank DIMMs in Dual Channel 2400 MHz for 2 Dual Rank DIMMs in Dual Channel 2133 MHz for 4 Single Rank DIMMs in Dual Channel 1866 MHz for 4 Dual Rank DIMMs in Dual Channel As you can see, the non-overclocked speed for my memory configuration is 1866. This is a memory controller limitation, not a RAM limitation. By running them at 2400MHz, I have a 533MHz overclock on the memory controller. 2666 may or may not be an overclock on your system, depending upon the number of sticks of RAM and whether they are Single Rank or Dual Rank. But odds are that if you have anything other than 2 sticks of 8GB RAM (which are often Single Rank but not always), you are overclocked. Paul

I look forward to it. I really appreciate the excellent follow-up. As a long-time customer, the Lime-Tech transformation is nothing short of magical. Paul

Thank you, very much appreciated!

At that point, it has already been loaded, and is no different than me unloading it from the command line. I want a way to prevent it from ever loading in the first place.

Interesting results. After loading the linux scaling driver, does the behavior revert to the Ryzen PBoost curve when you unload the driver, or do you have to reboot. I don't see where my driver is getting loaded, which makes me think it is automatic by the kernel. Not sure how I would block it during the boot. I just read that Ryzen microcode updates were finally added to the linux-firmware.git collection. This makes me think my Ryzen 1800X has never received any microcode firmware updates. Not sure if that would affect the boost or not, but regardless I posted a request for including the new microcode updates here: https://lime-technology.com/bug-reports/stable-releases/652-amd-cpu-microcode-updates-r84/

I feel that I have essentially already done this test. With everything unloaded and nothing running, unRAID reports that all 16 cores are idle at 0%. I can then run a command to load a single core, and it never exceeds 3.7GHz. I have also repeatedly seen 14 cores idling at 2.2GHz, and the other 2 at 3.7GHz, which seems to corroborate that I am loading only a single core. I have also used other commands to test frequency, besides grepping /proc/cpuinfo. So far, the other commands essentially matched the /proc/cpuinfo, and if anything they were slightly lower. Sorry, I don't have all the commands in front of me, I found them during web searches (supposedly they were more accurate for Ryzen), gave them a try, and moved on. I certainly agree that, based upon that graphic, the Ryzen 2xxx series will hit higher frequencies with more cores active, which is really awesome. But that shouldn't change the fact that with only a single core loaded, a Ryzen 1800X should hit 4.0 GHz minimum, and 4.1 GHz with XFR. Mine never goes over 3.7GHz, which is the same frequency that I can easily hit with all cores using the Performance governor. Perhaps I am mistaken, and that even when it appears that only a single core is loaded, in reality more cores are active. In which case, I agree that the graphic perfectly explains what is going on. Paul

That worked, thanks! Now I get this: root@Tower:~# cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to [email protected], please. analyzing CPU 0: no or unknown cpufreq driver is active on this CPU maximum transition latency: 4294.55 ms. for all 16 CPU's. I also see "Driver: * no driver *" for CPU Frequency Scaling in Tips & Tweaks, and changing from On Demand to Performance no longer has any impact, where before it would push all cores to 3.7GHz. So the behavior is now very similar to react's system. But I'm not seeing anything above 3.7GHz. Top speeds looks the same. I wonder if it has to boot that way. Perhaps stripping the driver out of a running system doesn't have the same result as booting without a driver. And I still need to look in my BIOS again to see if there is a setting related to giving the OS control over frequency scaling. Paul

Very interesting! Thanks react! I might be reading between the lines, but it looks like you don't have a software cpu frequency driver running at all, so it is defaulting to hardware frequency control. I have the acpi-cpufreq driver controlling the frequencies, which overrides the hardware control. Perhaps the problem is with the acpi-cpufreq driver. My guess is that you didn't see any change when switching from Conservative to Performance because you don't have a driver, so you aren't really controlling anything. Anyone know how I can unload the acpi-cpufreq driver? I also recall some settings in the BIOS that I think were related to allowing the operating system to control frequency - though I might be imagining that now. I'll have to look again. Maybe I can disable software/OS control. Paul

Not sure I agree with your interpretation. The 1800x base clock is is 3.6GHz, and the single core boost is 4.0GHz. XFR adds 0.1GHz to both of those numbers. I'm getting 3.7Ghz all core, so I think XFR is working. To me, it seems the boost isn't working. Core Performance Boost (CPB) allows individual cores to boost up to 4GHz (4.1GHz with XFR) on the 1800x. This works on Windows. CPB support for early/first release 1800X's (which is what I have) was added to Linux kernel 4.14 (it was broken prior to that), and I've verified that the CPB flag is showing. I just did some more testing. Upgraded my BIOS to the latest, (which includes the new AGESA 1.0.0.1a) but that made no difference. Also tried disabling SVM in the BIOS, as I read somewhere that solved another user's problem with no single-core boost in Linux. Disabling SVM broke KVM (which I definitely need) and did not fix the boost issue anyway. My temps are fine, and my cooling is sufficient. I could easily overclock all cores to 4.0GHz, but I choose not to. Stability, longevity, and power conservation are more important to me. All the same, this CPU should boost a single core up to 4.1GHz, and it just won't do it under Linux. react, since you have a 2700x, what do you get for this command? root@Tower:~# cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to [email protected], please. analyzing CPU 0: driver: acpi-cpufreq CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: 4294.55 ms. hardware limits: 2.20 GHz - 3.60 GHz available frequency steps: 3.60 GHz, 3.20 GHz, 2.20 GHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil current policy: frequency should be within 2.20 GHz and 3.60 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 2.20 GHz (asserted by call to hardware). cpufreq stats: 3.60 GHz:6.43%, 3.20 GHz:4.04%, 2.20 GHz:89.53% (6269) Paul

I did a bit more testing. Rebooting to complete the 6.5.2 installation, I double-checked my BIOS to make sure all the settings appeared correct. While everything looks fine, I don't see an XFR setting at all in my BIOS. I think it is safe to assume it is enabled. In Tips & Tweaks, I found that if I change "Enable Intel Turbo Boost?" to Yes (I had it on No before), then my max core frequency changes to 3.7GHz, so that's a small improvement. I tried changing the "CPU Scaling Governor", and on Performance all cores basically idled at 3.7GHz. On Demand allows other cores to scale down, normally hitting around 2.1GHz, and I can see my 1 or 2 cores hitting 3.7GHz, but no higher. Power Saver basically forces all cores down to about 2.2GHz. The odd part here is that the motherboard/cpu easily hits an 8/16 core frequency of 3.7GHz, which is perfect, but refuses to boost a single core any higher than 3.7GHz, even if only a single core is loaded. I agree that I'm glad it is fixed, I'm just baffled that I'm not finding any additional info on the web regarding the problem or the solution. Paul

I can't believe that there isn't more discussion on this, as it certainly caught my attention. I have an 1800X, and my individual core's max out at 3.6GHz with unRAID, even when only a single core is loaded. I always figured that this was just a limitation of Linux, as in earlier testing I saw 4.1GHz in Windows. Now react comes along and shows boosting working perfectly on a 2700X, what's up with that? While I know that the 2xxx series on the X470 motherboards adds Precision Boost 2 and XFR2, these simply permit more aggressive boosting when more cores are loaded, and shouldn't be required for boosting when only a single core was loaded. I never see over 3.6GHz (I don't manually overclock). Is this the same experience of other Ryzen 1 users? Is there a change in Ryzen 2 or X470 that fixes this issue, or a system setting or BIOS setting I might have wrong? Paul

Hey gang, sorry I've been absent. Luckily I've been healthy, so no concerns there. Instead, I've been sidetracked by by work and other projects. My biggest sidetrack has been a new program I wrote to replace the old My Movies Media Center Plugin. That may be of some interest to unRAID users like me who store their large movie collections on their server: If you're interested, you can check it out here: MM Browser MM Browser was supposed to be a quick little 2-4 week programming project, just for myself, but then I went crazy and decided to sell it online, which required a ton more programming and a website. User support has been much more time consuming than I ever fathomed. I've easily spent 6 months full time on MM Browser. MM Browser pulled me away from my other project, the Chameleon Pinball Engine. I had planned to have it ready for the next big Southern Fried Gameroom Expo here in Atlanta. Somehow the time slipped away. The show is in 4 weeks, and I'm realizing that there's too much work to make up to make to the show. That's a big disappointment for me. I've also got a a small enterprise software suite that I've worked on for the past decade, and I'm currently working on my first big sale. Trying to sell enterprise software to, uhm, big enterprises, has been eye opening to say the least. So many hurdles, and I'm spending more time doing documentation than anything else. Right now this is my biggest priority. Plus I've got a full time consulting gig at the moment. Long story short, I just haven't a moment to spare. I would release the private beta, but to be honest it just didn't work well, so that version was scrapped. I have documented plans for a new version that hopefully would fix the problems of the private beta. Every so often I think about trying to knock it out, and I've come close to working on it a few times, but it just fell too low on my priority list. There's always a chance I may get to it soon, but I can't make any promises. I know that this isn't the answer anyone was looking for. Sorry. If anyone else wants to run with it, please feel free. You have my blessing. Paul

It's like you're inside my head! I added mine to the survey. Did you want us to include our Username? A couple notes on my build. My Ryzen server has always been highly susceptible to the stability issues, and I always had to disable C-state Global Control in the BIOS to solve them. For over a month now I've been running with "/usr/local/sbin/zenstates --c6-disable" in my go file, and C-state Global Control enabled, and I am running stable. I don't do GPU passthrough, but I am currently running 4 VM's. I'm running 3 VM's with Windows Server 2016 Standard, each with 4-cores and 16GB of RAM. These are running an enterprise web application that I am developing against. I'm also running a single VM with Windows Server 2016 Datacenter and SQL Server 2016, also with 4-cores and 16GB. This VMs is the database server for the other 3 application servers. Surprisingly, even with 16-cores and 64GB concurrently allocated, the unRAID server is running like a champ. I'm averaging about 37% CPU load, and 57% memory usage. The KVM technology is simply amazing. And I can still stream Blu-ray movies. I'm hammering these servers with requests, approximately 24 XML API requests per second per server, 40 HTTP requests per second per server, and 72 database queries per second, or 6.2 million DB queries per day. My current unRAID uptime is 32 days (when I installed 6.4.1), and the VM uptime is approaching 18 days during this stress test session. My only problem is that the database server ran out of disk storage as the audit table grew so large from all the DB requests... hehehe. Paul

Sorry, couldn't resist. What John stated is 100% correct, though some details were omitted. To build on John's statement, those maximum speeds are for Single-Rank DIMMs and typically only 2 populated slots. The most likely configuration this would achieve is 16GB of installed RAM (2x 8GB Single-Rank). If you are running Dual-Rank DIMMs (think 16GB DIMMs, though I believe some 8GB DIMMs can also be Dual-Rank), or have installed memory into all 4 DIMM slots, then the maximum supported frequency drops. And if you are running Dual-Rank DIMMs, AND have installed memory into all 4 DIMM slots, then the maximum supported frequency drops again. On my Ryzen 7 1800x build, I have installed 64GB, so that means I am running both Dual-Rank DIMMs and I have populated all 4 memory slots. While my memory is only running at DDR4-2400, technically that is overclocked, as maximum supported speed in this configuration is just DDR4-2133. While my ASRock board runs my memory at DDR4-2400 without any special configuration, I am running overclocked according to AMD Ryzen specifications. Depending upon the installed memory, Killer IQ may be overclocking the integrated Ryzen memory controller by nearly 50%. While initially all Ryzen boards had issues with RAM clocks, the situation has greatly improved over the past year, so that Killer IQ's assertion is no longer valid. As G.I. Joe says, "Knowing is half the battle." Paul

I was the original discoverer of the Ryzen stability issue and C-state solution. My server is extremely susceptible to the C-state issue, typically crashing in 4-8 hours when the issue is present. I'm running 6.4.0-rc7a, with C-states enabled, and my uptime is 52 days. I have avoided all of the recent 'Really Close' releases since 7a, as the changes just seemed too scary for me to be a guinea pig. I think it was the introduction of the block level device encryption. I don't plan to use it, but I have nightmares thinking that a beta version could somehow misbehave and accidentally encrypt my precious data, so that I never get it back. I know the odds of that happening are pretty much zilch, though if it could happen it would likely happen to me. I'm waiting for the next stable public release. Anyway, perhaps something has changed since 7a that lost the fix for the Ryzen C-state issue. Paul

Thanks Greygoose! I just reached 50+ hours uptime on 6.4.0-rc7a with C-states enabled. Looks like Lime-Tech may have solved the stability issue, good job guys! With C-states enabled, Idle wattage has dropped 10+ watts. My UPS only reports in 10.5w increments (which is 1% of the 1050w power rating), so actual savings are likely somewhere between 10.5w-21 watts. From earlier testing with a more accurate Kill-A-Watt, the actual delta between C-states enabled & disabled was between 12w-18w. Idle temps have dropped 2-3 degrees C on both CPU (41C) and System (36C). Not as much as I had hoped, but I think my expectations were off. I did a lot of initial testing with the case cover off, and temps have unsurprisingly increased simply from closing the case, as case fans are on lowest speed (three 120mm fans, 1000 RPM @ 35% PWM), and they have to suck air past the HD's, so very little airflow at idle. The CPU fan speed profile is set to 'Standard' in the BIOS. At max case fan speeds (2750 RPM), idle CPU temp easily drops to 35C and System to 30C, but the higher fan speeds consume an extra 10+ watts and make lots of noise. As a compromise, I just changed my minimum case fan speed to 1400 RPM @ 50% PWM, which is much more quiet and energy efficient than full blast, but still improves my idle temps a couple degrees over the slowest fan speeds: 39C CPU, 34C System. I'll probably change the CPU fan profile from Standard to Performance in the BIOS to see if that drops the 5C delta over ambient a bit, but other than that I think I'm done. I'm happy to have idle temps back in the 30's, at reasonable fan speeds/noise, and with idle watts back to a more reasonable level. Paul

That's not a bad idea. I've been using the same USB stick for 8+ years, since the beginning when I started with 4.5 beta4 (with its brand new 20-disk limit). How's that for a flashback. Though I've certainly wiped it on occasion over the years. Most recently I think for the 6.1 branch. Now that I've finally got things settled, I'm gonna let it chill as-is. If more problem crop up, this will be high on my trouble-shooting list. As far as dealing with the potential corruption, I might just have to start from scratch and rebuild my configuration if I wipe the drive. Otherwise, I'm simply restoring potentially corrupted files. Thanks. Paul

Okay, multiple findings. First, when I checked on my server this morning, I found a Kernel Panic on the console screen, and the system was fully hung. Here's a pic: I restarted in Safe Mode again, started the array, and checked the share.cfg file. shareCacheEnabled was still missing. I stopped the array and went to the Settings/Global Shares panel. I couldn't directly apply "Yes" to "Use cache disk:", as it was already on "Yes" and wouldn't let me Apply it. I set it to "No", Applied, then set back to "Yes" and Applied. Now the share.cfg file got updated with the shareCacheEnabled="Yes" line, plus what appears to be several additional lines that must have also been missing. Here's the new file contents: # Generated settings: shareDisk="e" shareUser="e" shareUserInclude="" shareUserExclude="" shareSMBEnabled="yes" shareNFSEnabled="no" shareNFSFsid="100" shareAFPEnabled="no" shareInitialOwner="Administrator" shareInitialGroup="Domain Users" shareCacheEnabled="yes" shareCacheFloor="2000000" shareMoverSchedule="40 3 * * *" shareMoverLogging="yes" fuse_remember="330" fuse_directio="auto" shareAvahiEnabled="yes" shareAvahiSMBName="%h" shareAvahiSMBModel="Xserve" shareAvahiAFPName="%h-AFP" shareAvahiAFPModel="Xserve" Expecting Mover to now work again, I restarted into normal mode, as I wanted my temperature and fan plugins to keep my drives cool while the Mover got busy. On reboot, I confirmed that shareCacheEnabled="yes" was still in the share.cfg file. I then manually started Mover. This time the logged message was "root: mover: started", and I can see disk activity so it appears that Mover really is working. So it appears that my Cache drive and Mover troubles are finally over - thank you Tom and all who helped. That said, assuming Lime-Tech is already here reading this, I'd like to take a moment to recount my experiences with the -rc6/-rc7a releases: Experienced 1 Kernel Panic while in Safe Mode on -rc7a (above), and possibly another in Safe Mode on -rc6 (speculation) The upgrade from 6.3.latest to 6.4.0-rc6 coincided with whacking some cache related configuration file parameters (can't rule out plug-ins as a contributing factor) Could not assign the cache drive under -rc6, though -rc7a fixed this Several -rc7a anomalies (cache drive showing unassigned even though it was assigned, multiple Stop/Starts/Restarts required to get system synced up & behaving correctly) Currently, Mover is working but only at about 36 MB/s peak. Never paid attention before, because Mover is normally running in the middle of the night, but this seems rather slow. Possibly because data is being written to a drive that is 96% full, so this may be nothing. Odd caching in new GUI under -rc7a (didn't notice on -rc6) in which sometimes I have to Shift-F5/Forced Refresh to get current data presented. An easy example is the UPS Summary on the Dashboard, which kept reporting 157 watts for 30 minutes after I spun the drives down. I finally forced a screen refresh, and the status updated to 84 watts. Another example (plugin related) is the Dynamix System Temps ticker at the bottom of the screen doesn't seem to be updating. I've got both Firefox and IE open on the Main screen, and the ticker has been frozen on both for 10+ minutes, and they don't match each other. If I click around the menus, sometimes the ticker updates, and sometimes it just disappears. The behavior seems worse on IE than Firefox. On the plus side, the new 4.12 kernel includes some drivers that were missing, so that's pretty nice. It will take a while to determine if the C-state issue is resolved. Thanks, Paul

Thanks for chiming in Tom. Yes, 7a. Okay, so I booted up in Safe Mode. With array stopped, first I check the Settings/Global Shares and see that Cache Enabled is showing 'Yes'. I don't touch anything here for the moment. I then go start the array. Whole GUI immediately becomes unresponsive, web pages returning "Server not found" errors. Telnet is unresponsive too. Can't find server by name or IP. This is the second time server has become unresponsive tonight, once on rc6, and now again on rc7a, both times in safe mode. Seems very odd that I have more problems in safe mode than in frivolous mode. There is a chance the server has hung from the C-state being re-enabled on my Ryzen, not sure at this time and don't want to rule anything out yet. One last thought: Mover worked great until I upgraded from 6.3.latest to 6.4.0-rc6, at which point it appears to have stopped working at the same time. Just thinking that might be a clue as to how the shareCacheEnabled var got whacked. That's enough for one night, gonna hit the sack.

Wow, rc7 has been a rollercoaster so far... After a brief heart attack thinking the upgrade killed the server (turns out the new 4.12 kernel finally includes the drivers for my primary NIC, so I had to move my network cable), I'm now running on rc7 with C-states enabled (fingers crossed). LOL, didn't think the cache drive situation could get worse. I was wrong. Now I can't even make the cache drive assignment. I can see the drive under UNASSIGNED DEVICES, and the pretty name is back: Samsung_SSD_960_EVO_1TB_<longstringnottypingit> - 1 TB (nvme0n1) I can see the same drive listed as a selectable option for the cache drive assignment. But I've been trying to select it for 5 minutes, using 3 different browsers. I simply can't select it, it always goes back to "no drive". Seems like something got touched in rc7 related to this, since the behavior is definitely different. But different bad, not different good. Hmmm, more weirdness: Just noticed that the disk.cfg file got updated with the cacheID of my new drive, but with the array running the drive is still showing as unassigned on the Main tab, and I have no cache drive. Decided to reboot and see if that made a difference, and when I Stopped the array I noticed that now the drive is showing as the assigned cache drive. Instead of rebooting, I Started the array again, and now I have a cache drive. Tried to run Mover, and got the dreaded "root: mover: cache not enabled, exit" log entry. Stopped the array and rebooted (forgot to grab diagnostics first, sorry), and for the first time in 4 months the cache drive booted up assigned!!! Then tried Mover again, and it is still not working. Attaching diagnostics showing Mover not working. Paul tower-diagnostics-20170727-2147.zip

Yes I had tried that, but just did it again to make sure. No dice. I can open disk.cfg in Notepad++, which automatically notifies me if the file changes and asks if I want to reload it. When I start the array, the file gets touched. But not when I change the cache drive assignment. No matter what I do, the cacheId value stays the same with the old cache drive info.

Just wanted to report that I'm still suffering from cache drive issues. No idea if this is a Ryzen issue, or now a 6.4.0-rc6 issue, as I've since upgraded from the 6.3.x branch. It was previously pointed out to me by johnnie.black that this was a common issue to users of Chrome, and he pointed me to the FAQ. Apparently, using another browser would address the issue. I have tried both Firefox and IE, no joy. Seems like the problem has actually gotten worse under 6.4.0-rc6. At the very least I am noticing some new issues related to the problem. Here's everything I've found. A) As before, every time I boot up, the cache drive is unassigned (unless I install the old SSD, then the old SSD boots up assigned) B) Looking in the disk.cfg file, I see the old SSD still being referenced (cacheId="Samsung_SSD_840_EVO_1TB_S1D9NSAF627019H") C) Stopping the array, assigning the cache drive, and restarting the array appears to partially fix the problem, as now caching and VM's work. D) Under 6.4.0-rc6, the Identification is ugly: "eui.0025385c61b08640 - 1 TB (nvme0n1)", back on 6.3.x it seems like it was pretty, something like "Samsung 960 1TB (nvme0n1)" E) I can browse to \\TOWER\cache and see my files, and "cache" shows up in the DISK SHARES list on the Shares tab, so both of those seem good F) But Mover is broken, always logging the issue "root: cache not enabled, exit" G) Looking at the backlog of files to move, Mover not has worked in a month (June 25 is oldest file) - appears to be when I upgraded from 6.3.x to 6.4.0.-rc6 Paul

The 18w delta closely aligns with what I see on my Ryzen 7 1800X. Also keep in mind that disabling C-states pumps those 18w directly into the CPU. Your 62w figure represents CPU + Memory + Motherboard/Chipset + Storage + Graphics + Fans. The CPU is probably only about 20w-30w of that 62w, so an extra 18w pumped into the CPU is close to doubling the CPU idle power consumption and heat generation. Which is exactly why I made such a big deal about the trade-offs when using the C-state workaround for stability. And also why I'm very excited to hear it is fixed! I keep scanning the Prerelease 6.4 Support area, but no sign of RC7 yet... Thanks for sharing. Paul