Jump to content
axipher

Issues with NVME Drives on Ryzen - Page Faults all over (SOLVED)

6 posts in this topic Last Reply

Recommended Posts

Posted (edited)

Hello Forum,

 

I'm having some major issues with what I believe might be my NVME drives I installed in my system as Cache drives to free up SATA ports for more HDD's for my Data Drives.  I've attached the Diagnostics Zip File along with some summary details below.

 

First a quick run-down of my system so you don't have to dig it out if diagnostics:

 

Unraid: 6.7.0 Stable

Motherboard: Asrock X370 Killer SLI

CPU: Ryzen 1700x

RAM: 2x 8GB 2400 MHz Patriot DDR4

USB: Sony Micro Vault 8GB

GPU1: Nvidia GTX 1050 Ti

Parity + Data: 3x SeaGate IronWolf 4TB + 2x Western Digital Blue 3TB

Cache: 2x Intel 660p 512 GB M.2 NVME

Power Supply: Seasonic Focus Plus Gold 550W

 

No add-in cards, plenty of fans and airflow.

 

 

 

Now to the problems, last week my server started just shutting down randomly on my when it was seemingly idle during the day and there was no log of any sort.

 

I turned on the syslog to flash option in settings so that I can see things a little better if there was an error and I am running an SSH session from a Linux desktop on the network with a tail command to the syslog so that if the Unraid Server dies, at least the syslog will be buffered on that other desktop.

 

Jump to this week on Tuesday and I was still worried about it so I moved as much as I could off the Cache drives, the only thing left is "system", "appdata" and "transcode" (for Plex) shares are left on the Cache.

 

I also decided to install a much better power supply and cooler in case power or CPU temperature were issues.  The system hasn't crashed/shutdown since the power supply upgrade, but this morning, I checked the syslog quickly to find a bunch of page faults on the NVME drives and the files in the Cache drives "trimmed" down in size to much smaller then I'm used to seeing.

 

My Scheduler is set to call SSD Trim at 2:45 which lines up with these Page Faults.

 

Here are the trim lines, what worries me is my Docker Image says it is 21.5 GB in the WebUI when browing the contents of the "system" share but from command line, docker.img is only 2.9 GB, and libvirt.img says it is 1.07 GB in the share browser, but only 102.5 MB from command line.  What do I need to be worried about and/or fix?

 

May 16 02:46:51 CACHE root: /etc/libvirt: 921.5 MiB (966287360 bytes) trimmed on /dev/loop3
May 16 02:46:51 CACHE root: /var/lib/docker: 17.1 GiB (18363129856 bytes) trimmed on /dev/loop2
May 16 02:46:51 CACHE root: /mnt/cache: 900.7 GiB (967107411968 bytes) trimmed on /dev/nvme0n1p1

 

 

 

And here is the massive log of page faults:

 

May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6194000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c19000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6211000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d674a000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6296000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d93e0000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x000000009fba5000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d080d000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x000000009e5ff000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000daef6000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb599000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x000000009e52c000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb9ec000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcbe2000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb886000 flags=0x0050]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000de46f000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x0000000093d47000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d66fa000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0d18000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d622e000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f37da000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5da8000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5cc3000 flags=0x0000]
May 16 02:45:01 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db7ea000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5c40000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000dc566000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000dc4dd000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fb6fe000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fc0f9000 flags=0x0000]
May 16 02:45:01 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fc3f4000 flags=0x0000]
May 16 02:45:06 CACHE kernel: amd_iommu_report_page_fault: 252 callbacks suppressed
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0d22000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5ea9000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f756d000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5bef000 flags=0x0000]
May 16 02:45:06 CACHE kernel: amd_iommu_report_page_fault: 213 callbacks suppressed
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fca88000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fbefd000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d601c000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5e9e000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ffe29000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d64c1000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f5f80000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6228000 flags=0x0000]
May 16 02:45:06 CACHE kernel: amd_iommu_report_page_fault: 455 callbacks suppressed
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000ffee0000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f8400000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc772000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a00bd000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc77c000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000daef3000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc499000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5c00000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc0b1000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5ed8000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc500000 flags=0x0000]
May 16 02:45:06 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb579000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f82b1000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000a68d7000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000dc54a000 flags=0x0050]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000da895000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009eeed000 flags=0x0000]
May 16 02:45:06 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000dc480000 flags=0x0000]
May 16 02:45:11 CACHE kernel: amd_iommu_report_page_fault: 260 callbacks suppressed
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fd29e000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d663b000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d621b000 flags=0x0000]
May 16 02:45:11 CACHE kernel: amd_iommu_report_page_fault: 274 callbacks suppressed
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db07c000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a66e0000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc5ef000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a79a1000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c42000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f7bbe000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d66a9000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db080000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c9b000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc533000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d632e000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0809000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fd4e8000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d66a9000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb8b7000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc45a000 flags=0x0000]
May 16 02:45:11 CACHE kernel: amd_iommu_report_page_fault: 524 callbacks suppressed
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fb6f6000 flags=0x0000]
May 16 02:45:11 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5ffa000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000ddcff000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fca9b000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6040000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000db3ea000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f8400000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f86d8000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000a66ce000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6254000 flags=0x0000]
May 16 02:45:11 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009fbb9000 flags=0x0000]
May 16 02:45:16 CACHE kernel: amd_iommu_report_page_fault: 182 callbacks suppressed
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f0a15000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dfedb000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0e03000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5e95000 flags=0x0000]
May 16 02:45:16 CACHE kernel: amd_iommu_report_page_fault: 224 callbacks suppressed
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc8dc000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d631f000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000da8a6000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dfedb000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6284000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f3a60000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc0c7000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d082e000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x000000009fb4e000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc275000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb6f1000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d9a9a000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c55000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc77e000 flags=0x0000]
May 16 02:45:16 CACHE kernel: amd_iommu_report_page_fault: 396 callbacks suppressed
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5c55000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f8168000 flags=0x0000]
May 16 02:45:16 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc3ce000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f87af000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5cd1000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000a66e0000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5cd1000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000dc5ef000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x0000000093d5b000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d0852000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000a66e0000 flags=0x0000]
May 16 02:45:16 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000dcdbd000 flags=0x0000]
May 16 02:45:21 CACHE kernel: amd_iommu_report_page_fault: 261 callbacks suppressed
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6212000 flags=0x0000]
May 16 02:45:21 CACHE kernel: amd_iommu_report_page_fault: 261 callbacks suppressed
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f83f5000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc5c8000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc5f5000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fda9d000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc1f2000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fab15000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fd4c3000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d080f000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcdc8000 flags=0x0050]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6655000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f800f000 flags=0x0050]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d61c8000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a8777000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6406000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x0000000093d48000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d620d000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db3d2000 flags=0x0050]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d61fe000 flags=0x0000]
May 16 02:45:21 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0cd1000 flags=0x0000]
May 16 02:45:21 CACHE kernel: amd_iommu_report_page_fault: 519 callbacks suppressed
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fbf80000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d65ae000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000dc880000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009eef7000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fd4df000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f868a000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f7a7f000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5d35000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6022000 flags=0x0000]
May 16 02:45:21 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009e514000 flags=0x0000]
May 16 02:45:26 CACHE kernel: amd_iommu_report_page_fault: 258 callbacks suppressed
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc3d7000 flags=0x0000]
May 16 02:45:26 CACHE kernel: amd_iommu_report_page_fault: 266 callbacks suppressed
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6244000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6404000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fbeff000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ff94d000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f79d8000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d620d000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0807000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a0837000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d66ab000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fff6f000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb5bc000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c2b000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c28000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000da362000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000926f1000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d637a000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fbeee000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d66e4000 flags=0x0000]
May 16 02:45:26 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc8ec000 flags=0x0000]
May 16 02:45:26 CACHE kernel: amd_iommu_report_page_fault: 507 callbacks suppressed
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5e3f000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6645000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009fb71000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000db376000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f9f8e000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fffcf000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d678f000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000a0818000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d0de1000 flags=0x0000]
May 16 02:45:26 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5c28000 flags=0x0000]
May 16 02:45:31 CACHE kernel: amd_iommu_report_page_fault: 216 callbacks suppressed
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc561000 flags=0x0000]
May 16 02:45:31 CACHE kernel: amd_iommu_report_page_fault: 227 callbacks suppressed
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d63bc000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb592000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d65f9000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcd80000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a73df000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a00b4000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcbd4000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d64c7000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fffb8000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5caa000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x000000009fb76000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d677d000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc1de000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc54c000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000da8bb000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f7052000 flags=0x0000]
May 16 02:45:31 CACHE kernel: amd_iommu_report_page_fault: 433 callbacks suppressed
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fb8cf000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000a00b4000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6254000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fca8a000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6302000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000da775000 flags=0x0000]
May 16 02:45:31 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fa73e000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fffaa000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d61ca000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fbaff000 flags=0x0050]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fd33e000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d0def000 flags=0x0000]
May 16 02:45:31 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5e58000 flags=0x0000]
May 16 02:45:36 CACHE kernel: amd_iommu_report_page_fault: 244 callbacks suppressed
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d626c000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d67bb000 flags=0x0000]
May 16 02:45:36 CACHE kernel: amd_iommu_report_page_fault: 244 callbacks suppressed
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f8fcf000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6028000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x000000009e4f9000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db35a000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000da880000 flags=0x0050]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fcab5000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db35a000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dc3ec000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fffcd000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5eb6000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fd25c000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5f88000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcbee000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f4648000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5c7e000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6403000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5f9e000 flags=0x0000]
May 16 02:45:36 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6190000 flags=0x0000]
May 16 02:45:36 CACHE kernel: amd_iommu_report_page_fault: 489 callbacks suppressed
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fd4ab000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fff7a000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fd4bd000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6205000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f850f000 flags=0x0050]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f2c12000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009fb35000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d66d6000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000daee2000 flags=0x0000]
May 16 02:45:36 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d0cfc000 flags=0x0000]
May 16 02:45:41 CACHE kernel: amd_iommu_report_page_fault: 292 callbacks suppressed
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0819000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6195000 flags=0x0000]
May 16 02:45:41 CACHE kernel: amd_iommu_report_page_fault: 292 callbacks suppressed
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb8ea000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f7c5d000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d9330000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d65ec000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d635a000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000926f3000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0c6b000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fca98000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d61e4000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x0000000093d5c000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0c45000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fd4e1000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d64ea000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb9ec000 flags=0x0000]
May 16 02:45:41 CACHE kernel: amd_iommu_report_page_fault: 563 callbacks suppressed
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6319000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb78d000 flags=0x0000]
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d6025000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb9fe000 flags=0x0000]
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000ff947000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f86c7000 flags=0x0000]
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000f5f7f000 flags=0x0000]
May 16 02:45:41 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fffce000 flags=0x0000]
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000db348000 flags=0x0000]
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d0d21000 flags=0x0000]
May 16 02:45:41 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f8699000 flags=0x0000]
May 16 02:45:42 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000f86a1000 flags=0x0000]
May 16 02:45:42 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5e42000 flags=0x0000]
May 16 02:45:42 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000fff73000 flags=0x0000]
May 16 02:45:46 CACHE kernel: amd_iommu_report_page_fault: 245 callbacks suppressed
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5ca6000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0c24000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5f6a000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ff93e000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f9278000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d0d2b000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f325e000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6420000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5bcc000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d5f6a000 flags=0x0000]
May 16 02:45:46 CACHE kernel: amd_iommu_report_page_fault: 279 callbacks suppressed
May 16 02:45:46 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc680000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000a79a4000 flags=0x0000]
May 16 02:45:46 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fca92000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fc165000 flags=0x0000]
May 16 02:45:47 CACHE kernel: amd_iommu_report_page_fault: 519 callbacks suppressed
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d66e0000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f6000000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000da869000 flags=0x0050]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5be8000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcd97000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f115f000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5ef2000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fd498000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d96a0000 flags=0x0000]
May 16 02:45:47 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000dcd97000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x000000009fbcf000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x0000000093d5d000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d0cd0000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d62e3000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=22:00.0 domain=0x0000 address=0x00000000d9a8b000 flags=0x0000]
May 16 02:45:47 CACHE kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0000 address=0x00000000d5cc5000 flags=0x0000]
May 16 02:46:05 CACHE kernel: amd_iommu_report_page_fault: 180 callbacks suppressed
May 16 02:46:05 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6045000 flags=0x0000]
May 16 02:46:13 CACHE kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d63a8000 flags=0x0000]
May 16 02:46:33 CACHE kernel: amd_iommu_report_page_fault: 187 callbacks suppressed
May 16 02:46:33 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db3df000 flags=0x0000]
May 16 02:46:38 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fbdbb000 flags=0x0000]
May 16 02:46:46 CACHE kernel: nvme 0000:22:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000fb6ec000 flags=0x0000]

 

cache-diagnostics-20190516-1033.zip

Edited by axipher
Mark as Solved

Share this post


Link to post
Posted (edited)

As far as I can tell, IOMMU is only required if I want to pass through hardware to a VM right?

 

My VM is running purely with QEMU emulated hardware as I needed a Windows Server VM to run some daily tasks and whatnot.

 

 

 

Questions:

 

1)  Should I disable IOMMU?  I do plan to add a Quadro P2000 to the system though to use for Plex Hardware Encoding, would I still need IOMMU enabled for Plex to use the Nvidia GPU?

 

2)  If I should disable IOMMU, what is the best way.  Just with the UnRaid boot parameters, or do I need to change anything in BIOS?

 

3)  Do I need to worry about the "trimmed" docker and libvirt files that are now much smaller on disk then they report to be?

 

4)  Do I need the scheduled TRIM to happen weekly?

 

 

Edited by axipher

Share this post


Link to post
21 minutes ago, axipher said:

I do plan to add a Quadro P2000 to the system though to use for Plex Hardware Encoding, would I still need IOMMU enabled for Plex to use the Nvidia GPU?

Not sure, you can check in the plugin thread.

 

22 minutes ago, axipher said:

2)  If I should disable IOMMU, what is the best way.  Just with the UnRaid boot parameters, or do I need to change anything in BIOS?

Disable it in the bios.

 

22 minutes ago, axipher said:

3)  Do I need to worry about the "trimmed" docker and libvirt files that are now much smaller on disk then they report to be?

No, that's the trimmed capacity, i.e., the free space.

 

22 minutes ago, axipher said:

4)  Do I need the scheduled TRIM to happen weekly?

At least once a week is good practice

 

Share this post


Link to post

No need for iommu to use the gpu in plex. Only need to install the unraid nvidia plugin and download the unraid nvidia build.

Share this post


Link to post
2 hours ago, johnnie.black said:

Not sure, you can check in the plugin thread.

 

Disable it in the bios.

 

No, that's the trimmed capacity, i.e., the free space.

 

At least once a week is good practice

Okay, thanks for the answers.  I had read somewhere this morning that BTRFS might not need TRIM enabled for SSD's, so wasn't sure if I should use it still or not.  Trimmed capacity being smaller than reported capacity makes me feel better, I was just so use to seeing a 20GB docker image use up 20GB of space that it had me worried.

 

1 minute ago, saarg said:

No need for iommu to use the gpu in plex. Only need to install the unraid nvidia plugin and download the unraid nvidia build.

Thanks for the reply as well.  I might just go and disable IOMMU completely then in the BIOS tonight since I'm not using it for any pass-through.

 

 

 

I'm going to try reverting to the default UnRaid boot parameters right now.  I can't recall why I was using "intel_pstate=disable intel_iommu=on" on the append line.

Share this post


Link to post

Just to update the thread, disabling IOMMU in BIOS fixed the Page Fault errors.  Just ran a Trim on the SSD's using the Dynamix SSD TRIM Plug-in and no PAge Faults this time.

 

Haven't done further testing on GPU pass-through in VM's as I only have a single Nvidia GPU in this box right now for Plex HW encoding.

 

Marking thread as solved.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.