limetech Posted May 17, 2019 Share Posted May 17, 2019 This is a special release in light of the recent so-called Zombieland vulnerability revealed by Intel earlier this week. Normally we don't generate -rc stable patch releases, however in the interest of maintaining our, and the Community's sanity, this release is exactly the same as 6.7.0 except for updated Intel CPU microcode and corresponding Linux kernel (4.19.43). If 6.7.1-rc1 "fails" with something but 6.7.0 "works" then we can be fairly confident microcode/kernel to blame. We have released this on the next branch in order to get some testing before publishing for everyone on stable. Please post here in this topic, any issues you run across which are not present in 6.7.0 also - that is, issue that you think can be directly attributed to the microcode and/or kernel changes. Version 6.7.1-rc1 2019-05-17 Linux kernel: version: 4.19.43 intel-microcode: version 20190514a 2 2 Quote Link to comment
mattgob86 Posted May 18, 2019 Share Posted May 18, 2019 So if we are running AMD this won't have any change? Quote Link to comment
limetech Posted May 18, 2019 Author Share Posted May 18, 2019 10 minutes ago, mattgob86 said: So if we are running AMD this won't have any change? Unless something broke inadvertently in the kernel. Quote Link to comment
limetech Posted May 18, 2019 Author Share Posted May 18, 2019 A better article, complete with scary videos: https://www.bleepingcomputer.com/news/security/new-ridl-and-fallout-attacks-impact-all-modern-intel-cpus/ Quote Link to comment
strike Posted May 18, 2019 Share Posted May 18, 2019 (edited) Thanks for being on top of this. Off topic: These vulnerabilities are becoming a nightmare for the performance. Last time it affected the performance and I vaguely remember reading something about that this patch also would impact the performance. Soon our powerfull unraid servers will be a slow as hell, barely able to run as a plain NAS.. Edited May 18, 2019 by strike 1 1 Quote Link to comment
Dazog Posted May 18, 2019 Share Posted May 18, 2019 1 hour ago, limetech said: A better article, complete with scary videos: https://www.bleepingcomputer.com/news/security/new-ridl-and-fallout-attacks-impact-all-modern-intel-cpus/ And this: https://www.phoronix.com/scan.php?page=news_item&px=MDS-Zombieload-Initial-Impact Quote Link to comment
BRiT Posted May 18, 2019 Share Posted May 18, 2019 This isn't about unraid, more of general rant... At this point, I'd rather have an option to enable and disable the patches which are focused and directed on cloud-hosted servers or servers that would be targeted and have actual secure data to protect. Why should normal consumers be forced to run performance heavy security patches? They should be set as a compliance level, like DOD. General security patches are fine since they dont come with 30% perf hits. 1 Quote Link to comment
strike Posted May 18, 2019 Share Posted May 18, 2019 9 minutes ago, BRiT said: At this point, I'd rather have an option to enable and disable the patches Well, I think I have to agree now that I think about it. I'd like to have a powerful and a secure server but it seems that I can't have both anymore. Since unraid 6 my unraid server does so much in my home that I'd hate to lose any of it on account of performance. My data is valuable to me but not THAT valuable that I risk a 30-50% performance hit. Then I'd rather lose my data and restore from backup. Cause like all seasoned unraid users I do have backups. Quote Link to comment
limetech Posted May 18, 2019 Author Share Posted May 18, 2019 There are several ways to disable the various mitigations, both via kernel parameters and at run-time (though this will require us to include 'debugfs' which we can do). Google 'linux disable meltdown spectre zombieland mitigations' for a number of how-to articles. If it gets serious enough we'll probably add debugfs support and perhaps a config page. Having all these mitigations in place is mainly a C.Y.A. move in my opinion. 3 2 Quote Link to comment
limetech Posted May 18, 2019 Author Share Posted May 18, 2019 3 hours ago, Dazog said: And this: phoronix articles are good but the comments always make my eyes bleed 😱 1 Quote Link to comment
1812 Posted May 18, 2019 Share Posted May 18, 2019 5 hours ago, limetech said: There are several ways to disable the various mitigations, both via kernel parameters and at run-time (though this will require us to include 'debugfs' which we can do). Google 'linux disable meltdown spectre zombieland mitigations' for a number of how-to articles. If it gets serious enough we'll probably add debugfs support and perhaps a config page. Having all these mitigations in place is mainly a C.Y.A. move in my opinion. I would be interested in this, right after multiple cache pools and arrays! Quote Link to comment
ximian Posted May 18, 2019 Share Posted May 18, 2019 So when will someone start a class action suit against Intel. Seriously. If you purchase a processor for a certain performance / dollar value you expect it to be what you purchased? Now with the patches in regards to the list of vulnerabilities in the last couple of years there can easily be a 30% performance decrease. Which means the performance / dollar is no longer being respected. Seeing that Intel charges the big price for their processors it would be about time that they pay back their customers! I am a big fan of AMD but they fell behind in terms of performance. But seeing that the performance will now be at par with Intel, after patch installs, and the cost being cheaper... I think I will be going back to them. 1 Quote Link to comment
JonathanM Posted May 18, 2019 Share Posted May 18, 2019 1 hour ago, ximian said: I am a big fan of AMD but they fell behind in terms of performance. But seeing that the performance will now be at par with Intel, after patch installs, and the cost being cheaper... I think I will be going back to them. Unfortunately performance isn't the only metric. General compatibility and ease of implementation with various advanced functions like hardware pass through and such also tip on intel's side. I would say that you can eventually see progress and things get worked out, like the Ryzen and Threadripper issues, but it seems like by the time everything settles out and works well, the products are stale, and it's time to start the cycle of incompatibility and fixes again. "Maybe next time it will be different" isn't a comfortable way to approach tech. I'd rather spend the extra $/performance on a platform I don't have as much support time invested. If you love tinkering with it, fine. I'd rather buy it and stay hands off as much as possible. Quote Link to comment
CHBMB Posted May 18, 2019 Share Posted May 18, 2019 12 hours ago, BRiT said: This isn't about unraid, more of general rant... At this point, I'd rather have an option to enable and disable the patches which are focused and directed on cloud-hosted servers or servers that would be targeted and have actual secure data to protect. Why should normal consumers be forced to run performance heavy security patches? They should be set as a compliance level, like DOD. General security patches are fine since they dont come with 30% perf hits. I must admit I agree, I suspect the actual risk of me running a vulnerable home server is infinitismally small, however that isn't a criticism of LT, as I understand 110% they are kind of "duty-bound" to patch this stuff. Quote Link to comment
SimonF Posted May 18, 2019 Share Posted May 18, 2019 Updated my test server no issues seen so far. Quote Link to comment
BRiT Posted May 18, 2019 Share Posted May 18, 2019 Oh, absolutely. I don't and can't fault LT for the security patches. LT's prompt responses for security concerns are to be commended. It just seems like the entire software industry is jumping through so many hoops for what are infinitely small possible attack vectors on consumer devices. 1 Quote Link to comment
Dazog Posted May 18, 2019 Share Posted May 18, 2019 (edited) https://www.phoronix.com/scan.php?page=article&item=mds-zombieload-mit&num=5 "If looking at the geometric mean for the tests run today, the Intel systems all saw about 16% lower performance out-of-the-box now with these default mitigations and obviously even lower if disabling Hyper Threading for maximum security. The two AMD systems tested saw a 3% performance hit with the default mitigations. While there are minor differences between the systems to consider, the mitigation impact is enough to draw the Core i7 8700K much closer to the Ryzen 7 2700X and the Core i9 7980XE to the Threadripper 2990WX." Devastating. I hope everyone starts to buy threadripper/epyc2/ryzen3000. Intel does not deserve people's hard earned money. I guess we need switches to disable on our AMD systems. Due to intel's mistakes. So we aren't caught in the crossfire. Edited May 18, 2019 by Dazog Quote Link to comment
limetech Posted May 18, 2019 Author Share Posted May 18, 2019 47 minutes ago, Dazog said: I hope everyone starts to buy threadripper/epyc2/ryzen3000. Intel does not deserve people's hard earned money. At least Intel admits their mistakes, AMD has never chimed in on an obvious power issue with Ryzen: https://bugzilla.kernel.org/show_bug.cgi?id=196683#c602 50 minutes ago, Dazog said: I guess we need switches to disable on our AMD systems. Due to intel's mistakes. So we aren't caught in the crossfire. The Kernel knows what CPU you are using and adjusts appropriately. 1 Quote Link to comment
cybrnook Posted May 18, 2019 Share Posted May 18, 2019 (edited) 14 hours ago, limetech said: There are several ways to disable the various mitigations, both via kernel parameters and at run-time (though this will require us to include 'debugfs' which we can do). Google 'linux disable meltdown spectre zombieland mitigations' for a number of how-to articles. If it gets serious enough we'll probably add debugfs support and perhaps a config page. Having all these mitigations in place is mainly a C.Y.A. move in my opinion. I would also love to disable these as well. I got bigger fish to fry than cloud hosted VM vulnerabilities for my little home servers🙂 Much more important is my vm with pass through performance, Plex, and handbrake type conversions. Edited May 18, 2019 by cybrnook Quote Link to comment
ezhik Posted May 18, 2019 Share Posted May 18, 2019 (edited) You can also use this tool to check what vulnerabilities your CPU is susceptible to. Edited May 18, 2019 by ezhik Quote Link to comment
cybrnook Posted May 18, 2019 Share Posted May 18, 2019 (edited) 18 hours ago, limetech said: There are several ways to disable the various mitigations, both via kernel parameters and at run-time (though this will require us to include 'debugfs' which we can do). Google 'linux disable meltdown spectre zombieland mitigations' for a number of how-to articles. If it gets serious enough we'll probably add debugfs support and perhaps a config page. Having all these mitigations in place is mainly a C.Y.A. move in my opinion. @limetech Okay, so I did a bit a googling, and to build up, this is what I found based on Kernel level and availability (Leaving zombieload off the table for now since latest release does not yet mitigate it, that will be next): Kernel 4.15 Spectre v2 (CVE-2017-5715) - "nospectre_v2" Kernel 4.17 Spectre v4 (CVE-2018-3639) - "nospec_store_bypass_disable" Kernel 4.19 PR_SPEC_DISABLE_NOEXEC - During compilation Spectre v1 (CVE-2017-5753) - "nospectre_v1" Spectre v2 (CVE-2017-5715) - "nospectre_v2" Spectre v4 (CVE-2018-3639) - "nospec_store_bypass_disable" So checking uname on the latest 6.7.0 (non RC1), I can see we are at 4.19.41, so all three should be available nospectre_v1,v2, and store_bypass_disable. Of course seems there is a flag that can be set during compile, but that's not even worth getting into since that will never happen understandably. Then I found a phoronix forum thread that went back and forth, and I ultimately came out with this: pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier Where would be the best place to do this in Unraid during boot? Would it simple be an append in the syslinux file? append pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier EDIT: Seems zombieload (aka MDS) will be mds=off EDIT2: I am testing adding this to my unraid backup servers syslinux.cfg files boot section, like this: default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append initrd=/bzroot pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier label Unraid OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest So far I have rebooted with the change, and boot was successful. I am working now to validate these are actually disabled. EDIT3: Okay, it worked. Here is my before and after: BEFORE: cat /sys/devices/system/cpu/vulnerabilities/* Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable Mitigation: PTI Mitigation: Speculative Store Bypass disabled via prctl and seccomp Mitigation: __user pointer sanitization Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling AFTER: cat /sys/devices/system/cpu/vulnerabilities/* Mitigation: PTE Inversion; VMX: vulnerable Vulnerable Vulnerable Mitigation: __user pointer sanitization Vulnerable, IBPB: disabled, STIBP: disabled You will notice "conditional cache flushes", "SMT vulnerable", "PTI", "Speculative Store Bypass disabled via prctl and seccomp", "Full generic retpoline"* are all disabled now. 😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁😁 I don't know about you, but I don't care if someone is watching me play games on a vm, or is watching that I am encoding or decrypting a movie, big deal, not much going on at my house anyone would work hard enough to watch....... and if someone did make it that far to target me, I got bigger problems than speculative execution, like checking my firewall rules!! I will post this in another thread for better visibility if you all agree? Edited May 19, 2019 by cybrnook EDIT2: testing it 2 Quote Link to comment
glennv Posted May 18, 2019 Share Posted May 18, 2019 +1 Absolutely vital i would say to be able to disable all these mitigations for non exposed servers or whatever.Couldnt care less for my server. I just need as much perf as the cpu can deliver. But hats of for LT for beeing on top of this. Respect. Quote Link to comment
cybrnook Posted May 19, 2019 Share Posted May 19, 2019 Updated my post @BRiT and @glennv , got it working 1 Quote Link to comment
limetech Posted May 19, 2019 Author Share Posted May 19, 2019 1 hour ago, cybrnook said: I will post this in another thread for better visibility if you all agree? Nice work! Right, I think a better place would be in the Security Board. If you want to add the post there, I'll make it 'sticky'. Also, thanks to all testing this out. We have no choice but to keep marching on with new kernel releases. 1 1 Quote Link to comment
cybrnook Posted May 19, 2019 Share Posted May 19, 2019 Just now, limetech said: Nice work! Right, I think a better place would be in the Security Board. If you want to add the post there, I'll make it 'sticky'. Also, thanks to all testing this out. We have no choice but to keep marching on with new kernel releases. Will do, thanks! I will work on working it up a bit nicer and then post once it's super clear. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.