Prior to upgrading my fileserver to v7.0.0, my array would constantly have a disk write speed hovering around 120mb/s. After the upgrade, it hovers around 25mb/s. Watching htop run, the only file activity is the Mover itself, except for random short bursts of activity from my Home Assistant VM. If I stop the array, restart the array, then start Mover again, the speeds are back to normal, 120s, for maybe 30 seconds, then drop back to the 20's. Rebooting the server has the same effect as restarting the array. This machine runs a single Home Assistant VM, and the Cloudflare Docker. Nothing else is automatically running other than my backup scripts. When running Mover, the CPU activity stays around 25-50%. As soon as Mover stops, it drops back to 1-2% usage.

 

My array consists of a 15-bay HBA shelf, with 10 SATA drives, 4 to 12Tb, and 2 12Tb parity drives. The shelf is connected to an LSI 9207-8i HBA card in IT-mode. My array cache is a 4x1Tb ZFS Raid-10 spinning drives, connected to the motherboard SATA ports. The system cache is a 2x500Gb m.2 NVME on the motherboard, ZFS mirrored.

I do have 2 drives with failing helium levels, 1 parity and 1 array, but these drives have been showing that error for months prior to upgrading to v7, and the transfer speeds were fine under v6.12.14.

Diagnostics are attached

neo-diagnostics-20250210-0602.zip

Edited February 10, 20251 yr by malaki86
added info

Enable turbo write and retest.

11 hours ago, JorgeB said:

Enable turbo write and retest.

No change. It's writing to the array at the same speed.

I also noticed that the file size doesn't make any difference in speed. Sub-1mb files transfer at the same speed as 4gb+ files.

 

Post the diags while copying a large file to the array.

New diagnostics are attached.

The only changes I made to the system from these diagnostics compared to the first set are that I have Docker disabled now, and I made the Turbo Write setting you suggested.

I did run a DiskSpeed on my array a couple of days ago. The speeds shown in the results from it are basically the same as when I ran it probably 6 months ago.

neo-diagnostics-20250211-0514.zip

There's something else reading from disk6, stop all array activity and retest.

11 hours ago, JorgeB said:

There's something else reading from disk6, stop all array activity and retest.

I'll have to do that tomorrow morning. I'll kill the Docker & VM systems, and disable SMB & NFS.

 

This is after stopping the Docker, VM & NFS systems, with the server doing nothing except for running Mover

neo-diagnostics-20250212-0516.zip

No extra activity now, just slow, run the diskspeed docker test and post the results, for both the disk and controller tests.

I said in the original post that I had ran Diskspeed on v6 months prior to upgrading to v7, then I ran it again a few days ago. The direct drive access looks identical between the two.

This is the drive speed from Feb 3rd, under v7

And the controllers test?

3 hours ago, JorgeB said:

And the controllers test?

I don't see a controllers test

It's good to confirm there are no controller bottlenecks, it kooks like this:

 

I don't see an option anywhere on it to test the controllers. I see Rescan Controllers, which reads the data from the controllers and drives, and I see "Benchmark drives".

If there's a "Controller Test" or "Controller Benchmark" link somewhere on one of those two pages, I'm not seeing it.

I just ran another benchmark. At no point did I see anything whatsoever about controller tests.

 

I don't have the container installed to confirm, but IIRC you click the controller, then you'll see the test option.

I found it. There's a really small "I" in blue beside the controller name.

 

Tests do show a controller bottleneck, but still a lot faster than the speeds you are seeing, so something else is going on.

I'm just going to downgrade back to v6.12. It was fast and a lot more stable. 

17 hours ago, malaki86 said:

I'm just going to downgrade back to v6.12. It was fast and a lot more stable. 

Well, that didn't help, either. I'm on 6.12.15 now, and the speeds are still crap.

Read & Write speeds to the array-pool (4x1Tb 2.5" spinners, Raid-10) is fast, hovering around 240MB/s on both. The array, though... NSM, with is at 20MB/s or slower.

 

I've rebooted multiple times with the downgrade back to 6.12.14, then again to upgrade to 6.12.15, with no change. The only thing that I haven't done, though, is a complete shutdown/power-up. I have new SSDs due to arrive today, which I'm going to use to replace the spinners in my array-pool. So, it'll get a full shutdown at that point.

 

1 hour ago, malaki86 said:

and the speeds are still crap.

I suspected that the upgrade would not be the issue, if you transfer a large file over SMB directly to the array, do you see the same slow speed? Post the Windows explorer transfer graph.

I'll try that after I let Mover do a little work. Right now, I have a lot of files that I'm moving from the array to the system pool (VMs & appdata). It's absolutely screaming.

 

I had a 2Gb file that I transferred via SMB, written directly to the array, and it stayed at 105Mb/sec. I wasn't able to do a screenshot. My network is 1Gbit.