Zenbleed microcode update

[sticklyman]

Today the Zenbleed vulnrability has been made public, which allows data to be extracted from the YMM, etc, registers on a set of AMD Zen2 architecture CPUs. The high-level summary is that this requires either a microcode update (preferred), or a chicken bit to be set (not preferred due to performance hit). The microcode update has seemingly been released by AMD already, but it will take time for motherboard vendors to propagate this to the BIOS/UEFI. Considering that there are some relatively old CPUs affected, I can see some motherboard vendors not shipping the update.

 

Is there a plan to update the kernel shipped with unraid to include this microcode update? If so, when?

 

Thanks!

[sticklyman]

Update: Looks like AMD has only released the microcode fix for the Epyc CPUs and the rest are not coming until much later in the year. There is now a Linux kernel patch which effectively sets the chicken bit if the vulnrability has not been patched. Probably makes sense to cherry-pick this patch into the unraid kernel if it hasn't been backported already (seems to be in Linux 6.1.41, for example).

[bitcore]

Apparently you can set the following at boot and it disables the chicken bit (edit) without microcode update:

wrmsr -a 0xc0011029 $(($(rdmsr -c 0xc0011029) | (1<<9)))

Reference: https://news.ycombinator.com/item?id=36852699

Reference that it works better than a microcode update for one person's hardware combo: https://news.ycombinator.com/item?id=36854488

Quote

blinkingled 18 days ago | prev | next [–]

On my Zen2 / Renoir based system the PoC exploit continues to work albeit slowly even after updating the microcode (linked from TFA) that has the fix for this issue. The wrmsr stops it fully in its track.

Edit: just realized it must have been that the initramfs image is not updated with the manually updated firmware in /lib/firmware.

Edit2: Updated the initramfs and even if the benchmark.sh fails, ./zenbleed -v2 still picks out and prints strings which doesn't happen with the wrmsr solution.

I've not tested it myself yet but apparently it's rather well supported across the linux ecosystem.

 

I'm not sure of the best way to implement this automatically at boot in Unraid... perhaps someone else can chime in and add a recommended method?

I'm running threadripper and a lot of VMs, so this is a particularly desirable fix that I want to be reliable.

Edited August 12, 2023 by bitcore
mispelling

[itimpi]

2 minutes ago, bitcore said:

I'm not sure of the best way to implement this automatically at boot in Unraid... perhaps someone else can chime in and add a recommended method?

Assuming the fix works you can always add it to the config/go file on the flash drive to make it persistent.

[sticklyman]

1 hour ago, bitcore said:

Apparently you can set the following at boot and it disables the chicken but without microcode update:

wrmsr -a 0xc0011029 $(($(rdmsr -c 0xc0011029) | (1<<9)))

 

Thanks a lot for reminding me of this. I knew of MSR tools and just saw that intel had archived them - I'd mistakenly assumed they no longer worked.

 

I can confirm that I was able to get wrmsr and rdmsr to work on unraid. I ended up building the two binaries statically:

gcc -static -o wrmsr wrmsr.c
gcc -static -o rdmsr rdmsr.c

I did this on a different machine and transferred them across. Note that they can't reside in /boot as that is vfat and therefore doesn't support direct execution. For now, I've tested this with the binaries on my cache drive, and have set up a user script to run the above command "At First Array Start Only".

 

My script is the following:

#!/bin/bash
echo "About to battle the Spectral Chicken. Value before the battle:"
/mnt/fastcache/rdmsr -c 0xc0011029
/mnt/fastcache/wrmsr -a 0xc0011029 $(($(/mnt/fastcache/rdmsr -c 0xc0011029) | (1<<9)))
echo "The battle is over. Did we beat the Spectral Chicken? (The value should have the DE_CFG[9] bit set)"
/mnt/fastcache/rdmsr -c 0xc0011029

 

And the result is the following on my system:

About to battle the Spectral Chicken. Value before the battle:
0x3000310e08002
The battle is over. Did we beat the Spectral Chicken? (The value should have the DE_CFG[9] bit set)
0x3000310e08202

 

I have (so far at least) not confirmed if this stops the exploit or not, but having the bit set certainly makes me happier at the moment.

 

1 hour ago, itimpi said:

Assuming the fix works you can always add it to the config/go file on the flash drive to make it persistent.

Do you have a recomendation for where to put the rdmsr/wrmsr binaries such that they are both persistent across boot and available when the config/go script is executed?

[itimpi]

7 minutes ago, sticklyman said:

Do you have a recomendation for where to put the rdmsr/wrmsr binaries such that they are both persistent across boot and available when the config/go script is executed?

I would put them on the flash drive, and then in the ‘go’ file copy them to a runtime position and set the executable bit before running them.   That way everything is nicely self-contained, and it is easy to later undo this if you no longer want them.

 

an alternative is to do this in the User Script but then it only applies when the array starts and you may want it earlier?

 

[sticklyman]

10 minutes ago, itimpi said:

I would put them on the flash drive, and then in the ‘go’ file copy them to a runtime position and set the executable bit before running them.   That way everything is nicely self-contained, and it is easy to later undo this if you no longer want them.

 

an alternative is to do this in the User Script but then it only applies when the array starts and you may want it earlier?

 

Ah, yes. That makes sense. I definitely want to run it earlier, but just didn't understand enough about the unraid boot process to know how to best do that. Cheers!

 

I have also just confirmed that this does seem to stop the zenbleed POC (I'll leave it running for some hours to be sure).

[itimpi]

5 minutes ago, sticklyman said:

but just didn't understand enough about the unraid boot process to know how to best do that.

The boot process is covered in some detail here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page.

[bitcore]

Nice work @sticklyman! Thanks for the help @itimpi ! Glad to hear that it seems to stop the exploit POC!

 

@sticklyman, Would a community applications plugin be easy to build/deploy with your binaries to make this as easy as a few clicks for the entire unraid userbase? I've not made one of these before so I'm unsure if it's even possible to distribute binaries as part of a plugin...

[itimpi]

6 minutes ago, bitcore said:

I've not made one of these before so I'm unsure if it's even possible to distribute binaries as part of a plugin.

Plugins can definitely distribute binaries.   This is why it is important that they are maintained and validated against each Unraid release so they do not install any binaries that are incompatible with those included within any particular Unraid release.