Vr2Io

Never think that 👍

That's strange all show excellent state, but got slow writing speed.

Also some comment on this, 4TB disk usually not high performance if compare to high capacity disk. If almost full and writing at inner track then speed will drop a lot. How about the usage of writing disk ? Would you posible found a less usage disk and perform same test and verify any difference. Pls use disk share for test. I am at work now, once available will try to check diagnostic.

You have perform all necessary troubleshoot step and really clear and great. Does CPU reach 100% at any core ? Then pls attach diagnostic file.

Yes, you can reorder on the fly for single parity array.

Due to each HBA only connect maximum 8 disk, even for 4x PCIe 2.0 bandwidth still have ~2GB, so there are no bottleneck, and you can upgrade to PCIe3 HBA, the slot support up to PCIe4. For PCH to CPU, with DMI4 x8, there also huge bandwidth and no need concerns in bottleneck. Happy with that, but CPU will be less performance and huge power usage if compare to new main stream CPU. The reason for me use it was 8 ram slot, so basically not recommend for new build.

A bug found when change Docker setting "custom network type" from ipvlan to macvlan, still ipvlan.

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.101.10 netmask 255.255.255.0 broadcast 192.168.101.255 Client was 192.168.20.65, you test cross two different subnet.

Actually you don't need two 8x, 4x also fine. Suggest ASUS Prime Z790-P WiFi D4 ( I haven't that ) , it have four 16x slot in 16x 4x 4x 4x, so two GPU and two HBA fine. PS, My current main build was Asus Prime X299- A II, also price reasonable.

Yes Yes, for my understanding Z690 and Z790 mainly different on PCIe lane support,Z790 have more Gen4 lane ( chipset lane not CPU lane ). But this not means mobo will utilize it, different mobo have different design. That's great have 10G NIC built-in.

I haven't much solid experience on GPU passthrough and bifurcation ( currently haven't use either ), you need/plan many NVMe ? GPU PT and BF always are trouble stuff, quite depend on mobo. Pls double check does this mobo have bifurcation support. ( My Asrock Z390M-ITX have that after FW update but never mention in manual ) Pls note we usually talking bifurcation means Single slot support different PCIe lane grouping, i.e. 16x slot support in 16 / 8+8 / 8+4+4, but below talking about two PCIe slot support working in 16+0 or 8+8 by hardware PCIe switch, both are different thing.

I am not sure how true of that "Similar performance", but below power point really catch my eye. ( I m not sure E-Core support status ) Interesting, benchmark also trend for different TDP performance. https://www.club386.com/intel-core-i9-13900k-vs-amd-ryzen-9-7950x-at-125w-and-65w/

Pls post diagnosis, it look like Emulex chip solution. Check what have detect in BIOS and any setting provide.

Always true when you need performance. When system idle in fair light load, it won't have much different power usage for different profile. Seems system always in mid / high loading and you just limit the power usage by low power profile. Why so much non-stop log for docker, veth ? time="2022-11-08T00:10:37.527247514+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-11-08T00:10:37.527260213+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-11-08T00:10:37.527393578+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/0122d57f354ced477628c86783710eaa96cbea4de7b4b7f067c24dc6dcf46750 pid=1511 runtime=io.containerd.runc.v2 time="2022-11-08T00:10:37.953624726+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2022-11-08T00:10:37.953676404+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-11-08T00:10:37.953689363+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-11-08T00:10:37.953814318+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/2051608828d453978c1683d4bdcf71f25b9851e6a9be7765868363b4a1013e45 pid=1832 runtime=io.containerd.runc.v2 time="2022-11-08T00:10:38.383517946+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2022-11-08T00:10:38.383584264+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-11-08T00:10:38.383603893+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-11-08T00:10:38.383799705+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/77ee18d85243fb37dc0fc8031caa80d1d4f8a4dddf971670b8c683eb0ad0b894 pid=2088 runtime=io.containerd.runc.v2 time="2022-11-08T00:10:38.920248624+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2022-11-08T00:10:38.920298302+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-11-08T00:10:38.920310951+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-11-08T00:10:38.920430927+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ccbcff7381a2a705677e33801f833fd38170ef2510e46a90a7fe0b851b99c642 pid=2400 runtime=io.containerd.runc.v2 time="2022-11-08T00:10:39.258758726+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2022-11-08T00:10:39.258812364+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-11-08T00:10:39.258825623+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-11-08T00:10:39.258961448+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/27762cb3525d1d588b6ad9a0501cf1c463abcfe0ff974d7a281f8f97a7ef206e pid=2553 runtime=io.containerd.runc.v2 time="2022-11-08T00:10:44.707281914+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2022-11-08T00:10:44.707339361+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-11-08T00:10:44.707363450+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-11-08T00:10:44.707548513+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/04d2453b376e189c2e036ffe995260ad09d3ac33067c8078e54b20618665ac10 pid=3680 runtime=io.containerd.runc.v2 Nov 8 07:45:53 Starlight kernel: vetha220995: renamed from eth0 Nov 8 07:46:53 Starlight kernel: eth0: renamed from veth942f7d1 Nov 8 07:47:02 Starlight kernel: veth942f7d1: renamed from eth0 Nov 8 07:48:02 Starlight kernel: eth0: renamed from vethe7e728c Nov 8 07:48:11 Starlight kernel: vethe7e728c: renamed from eth0 Nov 8 07:49:11 Starlight kernel: eth0: renamed from veth3550bb2 Nov 8 07:49:20 Starlight kernel: veth3550bb2: renamed from eth0 Nov 8 07:50:20 Starlight kernel: eth0: renamed from veth5f799e7 Nov 8 07:50:28 Starlight kernel: veth5f799e7: renamed from eth0 Nov 8 07:51:29 Starlight kernel: eth0: renamed from veth9f1f980 Nov 8 07:51:37 Starlight kernel: veth9f1f980: renamed from eth0 Nov 8 07:52:37 Starlight kernel: eth0: renamed from veth70e116e Nov 8 07:52:46 Starlight kernel: veth70e116e: renamed from eth0 Nov 8 07:53:46 Starlight kernel: eth0: renamed from vethdc4c0d4 Nov 8 07:53:55 Starlight kernel: vethdc4c0d4: renamed from eth0 Nov 8 07:54:55 Starlight kernel: eth0: renamed from veth154de13 Nov 8 07:55:03 Starlight kernel: veth154de13: renamed from eth0 Nov 8 07:56:04 Starlight kernel: eth0: renamed from vethf5157df Nov 8 07:56:12 Starlight kernel: vethf5157df: renamed from eth0 Nov 8 07:57:12 Starlight kernel: eth0: renamed from vethf8e3bfa Nov 8 07:57:21 Starlight kernel: vethf8e3bfa: renamed from eth0 Nov 8 07:58:21 Starlight kernel: eth0: renamed from veth8739048 Nov 8 07:58:29 Starlight kernel: veth8739048: renamed from eth0 Nov 8 07:59:30 Starlight kernel: eth0: renamed from vethbb99a4f Nov 8 07:59:38 Starlight kernel: vethbb99a4f: renamed from eth0 Nov 8 08:00:38 Starlight kernel: eth0: renamed from veth07a647f Nov 8 08:00:47 Starlight kernel: veth07a647f: renamed from eth0 Nov 8 08:01:47 Starlight kernel: eth0: renamed from veth86d5ee0 Nov 8 08:01:56 Starlight kernel: veth86d5ee0: renamed from eth0 Nov 8 08:02:56 Starlight kernel: eth0: renamed from vethbbaa75d Nov 8 08:03:04 Starlight kernel: vethbbaa75d: renamed from eth0 Nov 8 08:04:05 Starlight kernel: eth0: renamed from veth5be5e89 Nov 8 08:04:13 Starlight kernel: veth5be5e89: renamed from eth0 Nov 8 08:05:13 Starlight kernel: eth0: renamed from vetheab3340 Nov 8 08:05:22 Starlight kernel: vetheab3340: renamed from eth0 Nov 8 08:06:22 Starlight kernel: eth0: renamed from veth9a03a46 Nov 8 08:06:31 Starlight kernel: veth9a03a46: renamed from eth0 Nov 8 08:07:31 Starlight kernel: eth0: renamed from veth27da58c Nov 8 08:07:39 Starlight kernel: veth27da58c: renamed from eth0 Nov 8 08:08:40 Starlight kernel: eth0: renamed from veth3596462 Nov 8 08:08:48 Starlight kernel: veth3596462: renamed from eth0

Crazy 👍 Agree

Ensure BIOS time was UTC, when large different with BIOS and NTP, it won't update.

You have 1TB memory ? Below is test on a Intel J1900 and 9700k system time dd if=/dev/zero bs=1024k of=tstfile count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.72748 s, 622 MB/s real 0m1.736s user 0m0.014s sys 0m1.708s /dev/shm# time dd if=/dev/zero bs=1024k of=tstfile count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.454901 s, 2.4 GB/s real 0m0.458s user 0m0.002s sys 0m0.454s How about to /dev/null ? time dd if=/dev/zero bs=2M of=/dev/null count=1024 1024+0 records in 1024+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.530346 s, 4.0 GB/s real 0m0.539s user 0m0.019s sys 0m0.516s time dd if=/dev/zero bs=2M of=/dev/null count=1024 1024+0 records in 1024+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.131551 s, 16.3 GB/s real 0m0.140s user 0m0.006s sys 0m0.134s

You can connect disks to external port same as internal.

You won't got benefit change to 9207, because it still 6gb HBA same as M1015, bottleneck not on PCIe2.0. You need concurrent upgrade PCIe3.0 and HBA and Expander to 12Gb. For 2.5 vs 3.5, this also apple and orange, 3.5 easy got high capacity CMR drive. If in terms of cost and power usage, it will endup no benefit using 2.5. For example, three 5TB 2.5 vs one 16TB 3.5 disk, 3.5 would all win. PS : Last mount I buy 18TB disk just 257usd, it even cheaper then shuck disk a lot. This only need one port instead 3 or 4 port. The drawback also as your mention, hard for cooling and noise ( cooling fan not disk itself ) and much longer parity check time. My current config are 9300-4i4e, 4i connect expander to 16disk and 4e connect expander to 12disk. All expander are 12Gb. This setup still have room to add one more expander for more disks.

Your real problem was stalls, it is quite common that small stalls (drop out) happen, but if too long period, this usually mean storage issue rather then network issue and hard to advice how to shoot.

Network fine now. 1. Does both end also 10G NIC ? otherwise bottleneck there. Highest transfer speed ? 2. Does actual writing happen in SSD instead array disk ? For array (spinner disk) storage performance increase, pls also turn on array reconstruct write mode.

Actually they are bonding in MAC address 18:c0:4d:b9:df:b1, pls try toggle the bonding setting or delete network-rules.cfg file, then setting the 10G NIC be eth0, and try again. bond0: flags=5443<UP,BROADCAST,RUNNING,PROMISC,MASTER,MULTICAST> mtu 1500 ether 18:c0:4d:b9:df:b1 txqueuelen 1000 (Ethernet) RX packets 329170 bytes 45232652 (43.1 MiB) RX errors 0 dropped 85 overruns 0 frame 0 TX packets 1895322 bytes 2715835227 (2.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0: flags=6147<UP,BROADCAST,SLAVE,MULTICAST> mtu 1500 ether 18:c0:4d:b9:df:b1 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500 ether 18:c0:4d:b9:df:b1 txqueuelen 1000 (Ethernet) RX packets 329170 bytes 45232652 (43.1 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1895322 bytes 2715835227 (2.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # PCI device 0x10ec:0x8168 (r8169) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="18:c0:4d:b9:df:b1", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0" # PCI device 0x15b3:0x1003 (mlx4_core) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="f4:52:14:87:bd:80", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

You shouldn't bonding 10G & 1G NIC.

Harder to say, I haven't ML10 gen9, but for my experiences, most machine support S3 well with Windows but may not well with Unraid. And some CPU power management BIOS setting will affect S3 success or not. BTW I only use S3 with Windows, same machine with Uniad will crash when execute S3. That means S3 not success / whole process doesn't completed. As mention, due to S3 not always success, not WOL problem.

In S3, machine will power down, do you got that. After that, would you can resume the server and all applications resume normal. You need enable S3 ( suspend to RAM ) in BIOS.