Suddenly slow read write speed

[larrytanjj]

My server use to be able to read and write at about 180-200. Only this few days I notice that when I am copying file from a share to my local workstation, the speed is very inconsistence. Most of the time is about 70-40 and occasionally it will drop to 0. The last hardware modification was to install an HBA card and 2 addition 12TB hard disk. I did some file transfer internally using unbalance to fill up the 2 new hard disk. I have did the following checks and tests but still not able to find out the reason why it is behaving like this. Really appreciate anyone can look into my diagnostics and figure out why this is happening.

 

• Run parity check with an avg speed of 183
• Test it against 2 different workstation
• Swap hard disk between onboard SATA controller or SAS breakout
• iperf in both direction with both workstation and its perfect with 0 retry.
• SMART Test on all drive and report healthy
• Stop all docker and VMs
• Reboot and Shutdown the server
• Run disk speed and the graph start at 250 and end at 150

 

tower-diagnostics-20230805-0114.zip

[JorgeB]

28 minutes ago, larrytanjj said:

and write at about 180-200.

If you mean array writes that is not possible with the default writing mode, as you are using, maybe you were using turbo write?

[larrytanjj]

8 hours ago, JorgeB said:

If you mean array writes that is not possible with the default writing mode, as you are using, maybe you were using turbo write?

Yup I am using the turbo write. However I am more concern about the read speed. It went from 200+ consistent when copy a single 50+gb or even larger to down to below 100 and inconsistent and occasionally drop to 0. This happen regardless which share I copy from to my workstation. I personally believe some to do with the software as most of the hardware test suggest that it is working fine.

[larrytanjj]

Attach is a screenshot that I transfer file between 2 disks using MC. The speed is at 30+

[JorgeB]

6 hours ago, larrytanjj said:

Yup I am using the turbo write

The diags you posted have turbo write disabled.

 

1 hour ago, larrytanjj said:

Attach is a screenshot that I transfer file between 2 disks using MC

Two array disks?

 

[larrytanjj]

1 minute ago, JorgeB said:

The diags you posted have turbo write disabled.

Apologize for the mismatch in the diags. Could be I have be troubleshooting everything I can and got confuse. However I do remember with or without the turbo write should be way fast than 30+

 

3 minutes ago, JorgeB said:

Two array disks?

I did a copy from 1 user share to another user share. Both share is using different array disk. This is to prevent reading and writing to the same array disk

[JorgeB]

2 minutes ago, larrytanjj said:

Both share is using different array disk.

That will always be slow because of the write speed, since parity needs to be updated, try reading from one disk to a cache or unassigned device.

[itimpi]

3 minutes ago, larrytanjj said:

Apologize for the mismatch in the diags. Could be I have be troubleshooting everything I can and got confuse. However I do remember with or without the turbo write should be way fast than 30+

Not quite true.   A speed of 30-40 MBs is not abnormal if you do not have Turbo Write enabled.

[larrytanjj]

4 minutes ago, JorgeB said:

That will always be slow because of the write speed, since parity needs to be updated, try reading from one disk to a cache or unassigned device.

This is the speed when I copy from user share from array disk to a user share which I store my VM which is on NVME.

[Kilrah]

If those are both in the array then 30-40MB/s is normal, have to live with it. And any other concurrent access to the array will slow it down further.

[larrytanjj]

This is what happen when I copy a couple of large file over a 10G network. It has this sudden burst and drop all the way down to ~30. This does not happen on my server all these while. I was able to copy from this same share at a much higher speed as mention in my first post.

Edited August 5, 2023 by larrytanjj
disk performance

[JorgeB]

Test copying to a different computer or device to make sure the destination is not the problem, if the same run a single stream iperf test in both directions.

[larrytanjj]

2 hours ago, JorgeB said:

Test copying to a different computer or device to make sure the destination is not the problem, if the same run a single stream iperf test in both directions.

 

[StatMatt]

I'm having a similar issue. Recently upgraded to 6.12.3, and now Mover basically never stops running just transferring ~200GB in client backups from the cache pool to the array. Like OP, I'm seeing a lot of times when the array appears to be idle.  I never had this problem on 6.11 or earlier.

[larrytanjj]

4 minutes ago, StatMatt said:

I'm having a similar issue. Recently upgraded to 6.12.3, and now Mover basically never stops running just transferring ~200GB in client backups from the cache pool to the array. Like OP, I'm seeing a lot of times when the array appears to be idle.  I never had this problem on 6.11 or earlier.

I wish I know what is wrong with my system but apparently I do not use mover. My protected cache pool is only for system, docker and vms.

[StatMatt]

I have a second server on 6.12 that I've been setting up for a server backup (eventually to be off-site). It's currently on my LAN (wired 1GB) and with parity disabled I'm getting single-digit (!!) average write speeds. Duplicacy is telling me a 6TB backup will take over six days. I disabled parity to try to speed it up but it's still giving single-digit write speeds.

[itimpi]

3 minutes ago, StatMatt said:

I have a second server on 6.12 that I've been setting up for a server backup (eventually to be off-site). It's currently on my LAN (wired 1GB) and with parity disabled I'm getting single-digit (!!) average write speeds. Duplicacy is telling me a 6TB backup will take over six days. I disabled parity to try to speed it up but it's still giving single-digit write speeds.

You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread.

[grumpy]

On 8/5/2023 at 12:34 PM, StatMatt said:

I have a second server on 6.12 that I've been setting up for a server backup (eventually to be off-site). It's currently on my LAN (wired 1GB) and with parity disabled I'm getting single-digit (!!) average write speeds. Duplicacy is telling me a 6TB backup will take over six days. I disabled parity to try to speed it up but it's still giving single-digit write speeds.

I do not think your issue is the same as the OP. You are doing a backup with encryption and the highest level of deduplication that Duplicacy can do. I would think if you increased the resources to this container it would speed it up. (no experience)

I was amazed how long the new PC I setup took to backup to Synology Active Backup compared to the original PC it was replacing. This was caused by the 90% CPU usage and transfer rates were dismal even though the Synology transfer rates are faster then my Unraid on testing file transfers.

[grumpy]

On 8/5/2023 at 12:16 PM, larrytanjj said:

I wish I know what is wrong with my system but apparently I do not use mover. My protected cache pool is only for system, docker and vms.

Are you not using a SSD cache drive for file transfers? You may want to read my quest to get faster writes. Simple answer @JorgeB suggested Turbo Writes and Cache drive makes it livable. Turbo Drive It improves writes and reads.

My Post

[larrytanjj]

6 hours ago, grumpy said:

Are you not using a SSD cache drive for file transfers? You may want to read my quest to get faster writes. Simple answer @JorgeB suggested Turbo Writes and Cache drive makes it livable. Turbo Drive It improves writes and reads.

My Post

Thanks for the suggestion but apparently that is not the issue I am facing. Before this happen everything was working as expected. The read speed which I am most concern was fine at 200MB/s. After installing LSI® SAS 9207-8i (IT mode) to add 2 more additional disk and this start to happen. Really appreciate anyone could help me as the server has been down for 4 days. With no docker running I manage to get 30MB/s via SMB. With docker running, SMB is pretty much unusable at 1-5MB/s.

[ajslim79]

On 8/5/2023 at 11:56 AM, Kilrah said:

If those are both in the array then 30-40MB/s is normal, have to live with it. And any other concurrent access to the array will slow it down further.

would be nice, if the transfer would be much faster then that - especially if you have a gen.3 nvme cache drive

Edited August 12, 2023 by ajslim79

[JorgeB]

You can enable turbo write, it should be noticeably faster at the expense of spinning up all disks for writes.

[StatMatt]

Thanks everyone for your thoughts on my issue - sorry to be so slow to reply. I am attaching diagnostics from the "new" server.  I recently used Unbalance to move about 1.77tb from one drive in the array to another, and it took more than two days at an reported average transfer rate of 9.46Mb/s.  This is with parity disabled.  Now the drive I was writing to is an SMR drive, but even so isn't that slow even for SMR?  In my previous post I mentioned a Duplicacy backup and I was getting similar speeds writing to a non-SMR drive. Below I'm pasting in the report from the unbalance log file. (No, I don't know why it says elapsed time 0s.)  I was occasionally checking the server while unbalance was running and never saw CPU at more than about 20%, and I had Mover basically disabled (set to monthly more than 20 days away) and also had Docker and VMs disabled.

Quote

 

unBALANCE - MOVE operation completed

Started: Aug 14, 2023 14:57:59

Ended: Aug 16, 2023 16:33:41

Elapsed: 0s

Move operation has finished

Transferred 1.77 TB at ~ 9.46 MB/s

349761 commands were executed.

 

 

I know a few people are thinking my issue is not the same as OP, so I'd be happy to move to a new thread if warranted.

unsmedley-diagnostics-20230818-1206.zip

[JorgeB]

5 minutes ago, StatMatt said:

Now the drive I was writing to is an SMR drive, but even so isn't that slow even for SMR?

SMR combined with the ZFS on array performance issue likely explains that speed.

[StatMatt]

Thanks for the link, JorgeB. I do have all of the array drives formatted ZFS - maybe I should drop the SMR drives back to XFS and see if that makes any difference?  I was also getting very disappointing write speeds from the one drive that isn't SMR, but it is formatted ZFS.  The platform is old (it's an Ivy Bridge i7), and maybe ZFS just has too much overheard for a 11-year-old CPU?