Ryzen Build Chronicle


NathanR

Recommended Posts

Figured I could chronicle my journey a bit and help anyone in the future or just be a place for reference and input as I have no idea what I'm doing :)

 

Build:

Model: Custom
M/B: ASRockRack X570D4U-2L2T Version - 
BIOS: American Megatrends International, LLC. Version P1.40. Dated: 05/19/2021
BMS: 1.2
CPU: AMD Ryzen 9 3900X 12-Core @ 3800 MHz
HVM: Enabled
IOMMU: Enabled
Cache: 768 KiB, 6 MB, 64 MB
Memory: 64 GiB DDR4 Multi-bit ECC (max. installable capacity 128 GiB)
Network: bond0: fault-tolerance (active-backup), mtu 1500
 eth0: interface down
 eth1: 1000 Mbps, full duplex, mtu 1500
 eth2: interface down
 eth3: interface down
Kernel: Linux 5.15.46-Unraid x86_64
OpenSSL: 1.1.1o
Uptime: ~3hrs

 

Previous (noob getting started) thread:

 

 

---

 

Me trying to figure out how to pass through a NVMe Drive in 6.10.3

I might just give up and run xcp-ng with Unraid nested?

Or I guess I don't really need NVMe drives passthru I could just install a new Win10...but I have a working image with all my files, boo :(

Idk, will revisit slowly...we'll see what Ghost82 says.

 

 

 

Then I went down the why USB bootable drive and 64GB drive and which drive hole...

Ugh, please just make this bootable from a standard HDD and the USB nightmare can go away.

 

I read that 64GB drive Fat32 formatted with something like GUI Format.exe will work still.

I read that Samsung BAR Plus is the best drive (as of 2022-07-12) for this.

I get consternation given the reviews of the BAR Plus's longevity and lack of USB 2.0 drives.

I'd rather have a list of tested compatible drives or better-yet a USB drive sold by Lime Technology...but it's all good.

 

---

 

Current plan is to trade-for/buy a 5900X and run that. Using my old 3900X until then.

 

I want some cache drives... but I am seeing now that there are issues with even that.
Me complaining: Ugh... WHY is everything so complicated?!?! I read that Unraid was simple and 'just worked' FFFFFFF's in chat!!!

 

 

Seems this is the fix:

nvme_core.default_ps_max_latency_us=1500

 

---

 

Next is figuring out drive structure, folder structure, cache drives, etc.

 

---

 

I'd like some input on the most common first things y'all do on an install...

 

I've heard of zeroing or wiping or cleaning drives...whats that?

What dockers, apps, VM's do you typically run?

What's your folder/drive structure and why do you like it?

Etc.

Link to comment

@Arbadacarba was instrumental in solving my VM issues. Thank you :)

\/ \/ \/ \/ See fixes here \/ \/ \/ \/

 

Having BMC issues with my X570D4U-2L2T >:(

Somehow it lost the 1.2 image o.O

 

---

 

On happier notes I ordered:

-5900X

-2x 980 1T

 

On Prime Day :)

 

I plan on using 5900X for H265 encoding and the 980's to store VM's and do write cache etc.

Yes I plan on having 10G NIC's to utilize the throughput :)

 

---

 

Here are all the app's I have installed so far:

image.thumb.png.06d4bf9069383ac332b7df2dd4e65f50.png

 

I have long wanted to run Home-Assistant!!! I CAN FINALLY DO IT YEASSSSSSS PLZZZZZZ THANK YOU Unraid!

Plex is next I guess?

Link to comment

Plugins:
Community Applications - https://forums.unraid.net/topic/38582-plug-in-community-applications/
Unassigned Devices (And Plus) - https://forums.unraid.net/topic/92462-unassigned-devices-managing-disk-drives-and-remote-shares-outside-of-the-unraid-array/
Unassigned Devices Preclear - https://forums.unraid.net/topic/120567-unassigned-devices-preclear-a-utility-to-preclear-disks-before-adding-them-to-the-array/

Dynamix File Manager - https://forums.unraid.net/topic/120982-dynamix-file-manager/

 

Dockers:
DiskSpeed - https://forums.unraid.net/topic/70636-diskspeed-hard-drive-benchmarking-unraid-6-version-292/
ESPHome (until I get VM running) - https://forums.unraid.net/topic/72033-support-digiblurs-docker-template-repository/
Home-Assistant-Core - https://registry.hub.docker.com/r/homeassistant/home-assistant/
netdata - https://forums.unraid.net/topic/47828-support-data-monkey-netdata/

 

---

 

Home-Assistant-Core doesn't have the add-ons module (dockers) because it is a docker.

There's also the linuxsystems developed core (docker) version?

Idk, too many limitations; decided VM was best route.


Unfortunately HA is now setup as CORE and I need to move files to the VM... hOw? idk.

But I know that Dynamix is amazingly helping me with all that.

 

Unfortunately installing HA as a VM was rather troublesome...

qemu-img convert -p -f qcow2 -O raw -o preallocation=off "/mnt/user/domains/Home Assistant/haos_ova-8.2.qcow2" "/mnt/user/domains/Home Assistant/vdisk1.img"

 

Solved here though:

---

 

I am slowly condensing my data on old HDD's [1x 16T, 2x 10T, 2x 2T] while running pre-checks on the new drive(s).

Ordering 10G components (sw and svr already have 10G, but I need a card for my desktop (5950X/3070/64G,NVMe)

1T 980 NVMe Cache drives arrived [need to lookup the temp issue and apply the fix again]

 

Todo:

-5900X

-Setup cache

-Move HA-Core config

-Condense

-10G

-Move Data

-Preclear

-DNS docker

-test failure modes

 

Edited by NathanR
Added plugins & dockers
Link to comment
7 minutes ago, SimonF said:

Why did you not just use the qcow2 file?

No matter what I did I could not get the qcow2 file to launch after EFI-Shell came up.

 

I googled for hours finally finding that as the fix (after seeing stuff about CSM and Secure Boot all for older versions 4.xx/5.xx, ~2019)

Link to comment

Preclear signature was lost on the 20T :( Now it will have to do it again :(

I wonder if this has to do with Unassigned Devices or Dynamix File Manager being installed?

 

---

 

Finished my VM vs HW testing.

Interesting results... Overall, seems not a huge difference.
Fairly impressive IMO :)

 

compare.thumb.PNG.6184df26b14d77a0d06ecd09882fd9ac.PNG

 

Drive speed differences are rather strange.

(Strange because image 1 is the passthru and image 2 is hw, lol)

Who cares tho.

 

699361817_DiskBench-VM.PNG.805e8fc2196375364ba602e51855e6ed.PNG

1264059252_DiskBench-HW.PNG.fa871cfa5f03cdd6b6ca3e6cc8bbb09a.PNG

Edited by NathanR
Link to comment

Ran into the dreaded Ryzen Linux power states issue.
I was curious if I was going to have an issue with it or not.

 

I had BIOS set for typical idle current, but when I replaced IPMI/BMS chip I think it got reset somehow.

I haven't had a need for disabling global c-states yet, but I am curious once I have more going on if it will be necessary.

 

My Threadripper TrueNAS build definitely needed the fix and would reboot constantly before fixing those issues.

 

---

 

Solved via FAQ for v6, this is a very good post that quickly explains the various issues running Ryzen and acceptable parameters.

 

Link to comment
  • 2 weeks later...

Had a crash even after fixing Idle current.

Probably will disable global C-states now.

 

Upgraded to 6.11-RC2 to see if system is more stable for X570 builds.

Tried to get temp/sensors working.

Foundout the nerdtools plugin for Pearl is broken on 6.11 - hopefully we seen an update soon :)

 

 

https://www.reddit.com/r/unRAID/comments/w9647x/unraid_6110rc1_now_available_notes_in_comments/

Version 6.11.0-rc2 2022-07-27
Improvements
With this release there have been many base package updates including several CVE mitigations.

The Linux kernel update includes mitigation for Processor MMIO stale-data vulnerabilities.

The plugin system has been refactored so that 'plugin install' can proceed in the background. This alleviates issue where a user may think installation has crashed and closes the window, when actually it has not crashed.

Many other webGUI immprovements.

Bug fixes
Fixed issue in VM manager where VM log can not open when VM name has an embedded '#' character.

Fixed issue where Parity check pause/resume on schedule was broken.

Change Log vs. Unraid OS 6.10
Base distro:
aaa_base: version 15.1
aaa_glibc-solibs: version 2.35
aaa_libraries: version 15.1
adwaita-icon-theme: version 42.0
appres: version 1.0.6
at-spi2-core: version 2.44.1
atk: version 2.38.0
bind: version 9.18.5
btrfs-progs: version 5.18.1
ca-certificates: version 20220622
cifs-utils: version 6.15
coreutils: version 9.1
curl: version 7.84.0
dbus: version 1.14.0
dmidecode: version 3.4
docker: version 20.10.17 (CVE-2022-29526 CVE-2022-30634 CVE-2022-30629 CVE-2022-30580 CVE-2022-29804 CVE-2022-29162 CVE-2022-31030)
editres: version 1.0.8
etc: version 15.1
ethtool: version 5.18
file: version 5.42
findutils: version 4.9.0
freeglut: version 3.2.2
freetype: version 2.12.1
fribidi: version 1.0.12
fuse3: version 3.11.0
gdbm: version 1.23
gdk-pixbuf2: version 2.42.8
git: version 2.37.1
glib2: version 2.72.3
glibc: version 2.35
gnutls: version 3.7.6
gptfdisk: version 1.0.9
harfbuzz: version 5.0.1
hdparm: version 9.64
htop: version 3.2.1
icu4c: version 71.1
inotify-tools: version 3.22.6.0
iproute2: version 5.18.0
iptables: version 1.8.8
json-c: version 0.16_20220414
kernel-firmware: version: 20220725_150864a
kmod: version 30
libX11: version 1.8.1
libXcursor: version 1.2.1
libaio: version 0.3.113
libcap-ng: version 0.8.3
libdrm: version 2.4.110
libepoxy: version 1.5.10
libevdev: version 1.12.1
libgcrypt: version 1.10.1
libgpg-error: version 1.45
libidn: version 1.41
libjpeg-turbo: version 2.1.3
libmnl: version 1.0.5
libnetfilter_conntrack: version 1.0.9
libnfnetlink: version 1.0.2
libnftnl: 1.2.2
libnl3: version 3.6.0
libtiff: version 4.4.0
libtiff: version 4.4.0
liburcu: version 0.13.1
libusb: version 1.0.26
libxcb: version 1.15
libxkbcommon: version 1.4.1
libzip: version 1.9.2
libX11: version 1.8.1
listres: version 1.0.5
logrotate: version 3.20.1
lsof: version 4.95.0
lzip: version 1.23
mc: version 4.8.28
mcelog: version 184
mkfontscale: version 1.2.2
nano: version 6.3
nettle: version 3.8
nfs-utils: version 2.6.1
nghttp2: version 1.48.0
ntfs-3g: version 2022.5.17
oniguruma: version 6.9.8
openssh: version 9.0p1
openssl: version 1.1.1q (CVE-2022-1292 CVE-2022-2097 CVE-2022-2274)
openssl-solibs: version 1.1.1q (CVE-2022-1292)
pango: version 1.50.8
pciutils: version 3.8.0
pcre2: version 10.40
php: version 7.4.30 (CVE-2022-31625 CVE-2022-31626)
rsync: version 3.2.4
samba: version 4.16.4 (CVE-2022-2031 CVE-2022-32744 CVE-2022-32745 CVE-2022-32746 CVE-2022-32742)
setxkbmap: version 1.3.3
shared-mime-info: version 2.2
sqlite: version 3.39.2
sudo: version 1.9.11p3
sysfsutils: version 2.1.1
tdb: version 1.4.7
tevent: version 0.12.1
tree: version 2.0.2
util-linux: version 2.38
wget: version 1.21.3
xauth: version 1.1.2
xclock: version 1.1.1
xdpyinfo: version 1.3.3
xfsprogs: version 5.18.0
xkeyboard-config: version 2.36
xload: version 1.1.4
xmodmap: version 1.0.11
xsm: version 1.0.5
xterm: version 372
xwud: version 1.0.6
Linux kernel:
version 5.18.14 (CVE-2022-21123 (CVE-2022-21123 CVE-2022-21125 CVE-2022-21166)
oot: md/unraid: version 2.9.23
CONFIG_IOMMU_DEFAULT_PASSTHROUGH: Passthrough
CONFIG_VIRTIO_IOMMU: Virtio IOMMU driver
CONFIG_X86_AMD_PSTATE: AMD Processor P-State driver
CONFIG_FIREWIRE: FireWire driver stack
CONFIG_FIREWIRE_OHCI: OHCI-1394 controllers
CONFIG_FIREWIRE_SBP2: Storage devices (SBP-2 protocol)
CONFIG_FIREWIRE_NET: IP networking over 1394
CONFIG_INPUT_UINPUT: User level driver support
CONFIG_INPUT_JOYDEV: Joystick interface
CONFIG_INPUT_JOYSTICK: Joysticks/Gamepads
CONFIG_JOYSTICK_XPAD: X-Box gamepad support
CONFIG_JOYSTICK_XPAD_FF: X-Box gamepad rumble support
CONFIG_JOYSTICK_XPAD_LEDS: LED Support for Xbox360 controller 'BigX' LED
Management:
rc.nginx: enable OCSP stapling on certs which include an OCSP responder URL
rc.wireguard: add better troubleshooting for WireGuard autostart
rc.S: support early load of plugin driver modules
upc: version v1.3.0
webgui: Plugin system update
Detach frontend and backend operation
Use nchan as communication channel
Allow window to be closed while backend continues
Use SWAL as window manager
Added multi remove ability on Plugins page
Added update all plugins with details
webgui: docker: use docker label as primary source for WebUI
This makes the 'net.unraid.docker.webui' docker label the primary source when parsing the web UI address. If the docker label is missing, the template value will be used instead.
webgui: Update Credits.page
webgui: VM manager: Fix VM log can not open when VM name has an embedded '#'
webgui: Management Access page: add details for self-signed certs
webgui: Parity check: fix regression error
webgui: Remove session creation in scripts
webgui: Update ssh key regex
Add support for ed25519/sk-ed25519
Remove support for ecdsa (insecure)
Use proper regex to check for valid key types
webgui: misc. style updates
webgui: Management access: HTTP port setting should always be enabled
webgui: Fix: preserve vnc port settings
webgui: Fix regression error in plugin system
webgui: Fix issue installing registration keys
webgui: Highlight case selection when custom image is selected
webgui: fix(upc): v1.4.2 apiVersion check regression

 

Edited by NathanR
added 6.11-RC2 info
Link to comment
On 7/23/2022 at 2:11 PM, NathanR said:

Ran into the dreaded Ryzen Linux power states issue.
I was curious if I was going to have an issue with it or not.

 

I had BIOS set for typical idle current, but when I replaced IPMI/BMS chip I think it got reset somehow.

I haven't had a need for disabling global c-states yet, but I am curious once I have more going on if it will be necessary.

 

My Threadripper TrueNAS build definitely needed the fix and would reboot constantly before fixing those issues.

 

I only needed to set Typical Idle Current (and appropriate DRAM settings) for my 1st Gen Ryzen 1500X.  The 1st gen were the most notorious for Linux issues, but hasn't had one hang/freeze/crash for 1.5+ years since setting it correctly.

 

As for Nerdtools, it will get caught up eventually.  This is the price one pays to be an early adopter of a new revision of software (especially a release candidate).  If a test bed system, no worries.  For one you rely on, well... 😞

Link to comment
20 hours ago, ConnerVT said:

 

I only needed to set Typical Idle Current (and appropriate DRAM settings) for my 1st Gen Ryzen 1500X.  The 1st gen were the most notorious for Linux issues, but hasn't had one hang/freeze/crash for 1.5+ years since setting it correctly.

 

As for Nerdtools, it will get caught up eventually.  This is the price one pays to be an early adopter of a new revision of software (especially a release candidate).  If a test bed system, no worries.  For one you rely on, well... 😞

Thanks!

 

I haven't had time to look into why it crashed the second time.

Any pointers on where to look in the logs?

I read that the logs don't show pre-boot logs IIRC.

Idk, I'll investigate in the future.

 

Re: RC - I completely agree, I usually don't run RC stuff, but I saw all the significant improvements (in-general & x570/kernel) and wanted to try it out. My SVR right now is just a test-bench and is running things that aren't critical (yet) just so I can get used to Unraid in a relaxed environment. That said, I am in the process of moving computer cases, installing 10G, and swapping drives so I'll be ready soon to move the server towards production :)

 

Edit: Just read your story :D I think  know I am in good company xD

Edited by NathanR
Link to comment

From what I think I know, the normal logging for Unraid doesn't help much after a "crash" where you must boot the system to access anything.  Knock on wood - I haven't found myself there yet.  As Unraid loads and runs in memory (RAM), it also keeps its syslog there as well.  So when you boot, POOF it is gone.  You can set up a syslog server, which will write the log to disk.  Believe the intent of normal logging to memory is to reduce disk I/O, important for those who wish to keep disks spun down (or not hammering/filling the flash drive).

 

Though with Ryzen lock ups, there really isn't much of anything logged.  The CPU bug is in the chip, so when it happens, things just freeze up.  So not much help from the syslog, regardless of how you collect it.

 

Glad you like the story.  I did really intend just to put an appliance down in the basement and forget about it.  But it has grown a life of its own.  Since writing that, I procured another gently used 8TB drives - in cold store backup along with a 6TB waiting in the same drawer of the server desk.  Plus another portable drive for off-site backup storage.  Unraid is a pretty decent platform, and it amazes me how much (and how easily) my little 4c/8t 1st gen Ryzen server can handle.

Edited by ConnerVT
speeling
Link to comment
  • 1 month later...

Update.
Now running Version 6.11.0 2022-09-23

 

Pearl & iperf3 included with the kernel so...yay.

My 10G NIC's are confirmed working; woot!

I was having issues confirming speeds from within the W10 VM.

image.thumb.png.9a6617a6c267adb8ecbaadaf4feb8f44.png

 

Also, I have my 1T 980's in what I think is a Raid 1 pool. But I've read conflicting posts about whether this is operating properly or not.
Pressing the rebalance as Raid 1 does nothing.

image.thumb.png.e6cb21ad18b66d243575d0ae202bd889.png

 

I also modified the temp warning/errors to 85c so I didn't get the annoying Samsung 980 bug. I really wish they would do a firmware release.
BTW...how do I do a NVMe firmware update?

 

Oh! and I'm super happy that the temp sensors work out of the box now (well, there is a tiny config todo). Pearl or Linux Kernel with AMD tunings, whatever, don't care, I'm happy :)

image.png.f45a040b6ace134eae0a93d532e6fbf9.png

 

 

figured out exactly which by looking at IPMI & running P95 in a VM.

 

CPU @ 70c and x570 @ ~50c lol

image.png.9a369aee9c2d5c600748487e9c9c2690.png

Edited by NathanR
Link to comment

Brainstorming/thoughts post...

 

Looking into converting my .xva (xcp-ng) vms into Unraid format (qcow?)

Maybe this? - https://forums.unraid.net/topic/69244-convert-vmware-vm-to-kvm/

 

 

Looking into converting my physical disk into unraid drive.

https://kmwoley.com/blog/convert-a-windows-installation-into-a-unraid-kvm-virtual-machine/

https://kmwoley.com/blog/reduce-shrink-raw-image-img-size-of-a-windows-virtual-machine/

A little worried I can't shrink my 500GB NVMe drive down to a manageable 50GB lol

 

 

Looking into creating new W10 VM

I was looking here > https://wiki.unraid.net/Manual/VM_Management#Basic_VM_Creation

But that is wrong, this is correct steps to install HDD driver > https://wiki.unraid.net/index.php/UnRAID_Manual_6#Installing_a_Windows_VM

Latest virtio drivers: https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md

Link to comment
  • 2 weeks later...
  • 3 months later...

95 days, 1 hour, 58 minutes uptime on 6.11.0

Running:
1x W10 VM
Handful of dockers
Some apps

 

Just updated to 6.11.5

Needed a few reboots to get stats working properly.

Hopefully stable for a long time again :)


Found this useful command for USB Flash Drive (OS) backups from this reddit post: https://www.reddit.com/r/unRAID/comments/wp44pp/reminder_backup_your_unraid_usb_drive/

zip -r  /mnt/user/backups/unraidbackup/flash`date +%d-%m-%Y`.zip /boot

 

Fan 1 - CPU 80mm Noctua
Fan 2 - x570 Chipset 40mm Noctua
Fan 3 - Chassis Fan(s) 120mm arctic p12 PWM PST CO  [25% = 500rpm, 50% = 900rpm, 75% = 1200rpm, 100% = 1800rpm]
Fan 4 - HDD chassis fan 80mm arctic p8 PWM PST CO  [25% = 900rpm, 50% = 1500rpm, 75% = 2300rpm, 80% = 2500rpm, 100% = 2900rpm]
Fan 5 - future 120mm
Fan 6 - future 120mm

Edited by NathanR
Added Info
Link to comment
  • 1 year later...

Been running 6.11.5 for a while now.

Current uptime is 2 months 22 days.

 

Can't remember the last thing I did that required me to reboot it.

I remember now, added a USB 3.0 card so I could pass-through for my W10 VM so I could run LOR for my Christmas lights.

 

No crashes.

Been very happy.

 

Future Wishes:

  • VM cloning, backup, snapshot, etc.
  • Rebuild server with 13900K, X13SAE-F, 128GB ECC
    • quicksync (GPU) encoding for plex
    • more USB

 

 

Plugins:

  • Community
  • CA Backup / Restore Appdata
  • Dynamix Auto Fan Control
  • Dynamix File Manager
  • Dynamix System Statics
  • Dynamix System Temperature
  • Network UPS Tools (NUT)
  • Unassigned Devices
  • Unassigned Devices Plus
  • Unassigned Devices Preclear
  • Unraid Connect
  • User Scripts

Docker:

  • blueiris (not on)
  • DiskSpeed (not on)
  • docker-diag-tools (not on)
  • ESPHome
  • glances
  • Grafana-Unraid-Stack (not on)
  • luckyBackup (not on)
  • MongoDB (not on)
  • OpenProject
  • Phoronix-Test-Suite (not on)
  • plex
  • qbitorrent (not on)
  • syslog-ng (not on)
  • unifi-controller
  • UniFi-Protect-Backup
  • UptimeKuma

VMs

  • Dev-Home Assistant
  • Home Assistant
  • W10-SVR
  • Win10 (not on)

 

Decided to make a development home assistant VM to play around with things and not break my house.

Process still works in Feb of 2024

  1. Download Home Assistant .qcow2 file
  2. Copy to Array somehow
    • Shared folder
    • Upload via Dynamix File Manager app
  3. Create VM (linux)
    • 32GB HDD
  4. Delete the vdisk
    • vdisk1.img
    • /mnt/user/domains/Dev-Home Assistant/vdisk1.img
  5. Run the convert command
    • qemu-img convert -p -f qcow2 -O raw -o preallocation=off "/mnt/user/domains/Dev-Home Assistant/haos_ova-11.4.qcow2" "/mnt/user/domains/Dev-Home Assistant/vdisk1.img
  6. Boot the VM
  7. Launch VNC
  8. Get IP
  9. Login
  10. Woot/FIN

 

Edited by NathanR
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.